# Presentation: 182549

## Communication of Probability Forecasts for PWS

- Ken Mylne

- Ensemble Forecasting Manager, Met Office, UK
- Chair WMO Expert Team on EPS

## Outline

- Communication of Probability and Uncertainty
- Why? (1 slide!)
- Some concepts and principals
- Some pitfalls to avoid
- Common misconceptions
- Research on Understanding of Forecasts with Uncertainty Information
- Examples of Presentations

## Why Probability Forecasts?

## Why? The Effect of Chaos

- Communicating with an audience? Motivate them with a little basic science...

- Up to about 3 days ahead we can *usually* forecast the general pattern of the weather quite accurately
- Beyond 3 days Chaos becomes a major factor

- Fine details (eg rainfall) have shorter predictability

- The atmosphere is a chaotic system: *“... one flap of a seagull’s wing may forever change the future course of the weather”*, (Lorenz, 1963)

**Notes:**
Lorenz’s famous quote is probably an exaggeration for day-to-day forecasting, but nevertheless typifies the problem. We can never analyse the current state of the atmosphere perfectly, even on much larger scales than a seagull! The two charts shown are from quite a typical 4-day ECMWF ensemble forecast – a more extreme example, but certainly not a rare one. You may like to comment that the differences in initial conditions between them are too small for a forecaster to be able to analyse which is the better. The left chart predicts a deep area of low pressure near NW Ireland bringing strong winds and rain to much of the British Isles; the right chart predicts a much weaker low NE of Scotland, with a ridge of high pressure likely to bring fine weather to the southern half of the British Isles. Clearly in this situation a forecaster who has access to only a single model forecast is in danger of issuing a forecast which could go seriously wrong.

## Communication of Uncertainty

## Expressing uncertainty

- When?Where? How much? Are you sure?

- Forecasters have always dealt with uncertainty.

- ...at times...mainly in the NW...up to an inch in places...risk of heavy bursts

- Uncertainty expressed in many ways , mostly subjective.

## Probabilities - getting quantitative

- How can we improve on subjective description?

- People in London have a 50% chance of seeing a light shower this afternoon...

- ...but in the NW you have an 80% risk of a shower with a 20% chance of over an inch

- This is much more informative, but only if the figures are meaningful!

## Use of probability forecasts - Clear Definition

- What does this mean?
- 30% probability of rain in Florida
- 30% in Orlando?
- 30% somewhere along I-95?
- or 30% “somewhere in Florida”?
- How much?
- A trace? 5mm? Tropical cyclone quantity?!
- Probabilities must be *unambiguous* and *relevant* to the end user

## Clear Definition

- Examples of well-defined probability forecasts could be:
- 30% probability of more than 5mm of rain at Orlando Airport between 1200 and 1800.
- 70% probability of wind reaching 40kt in at least one place in Florida on Tuesday.
- 10% probability of wind sufficient to cause severe structural damage in Miami overnight.
- The last example shows how a low probability can give useful warning of a serious event
- A good test: *Can I verify this forecast?*

## Decision-making with probabilities

- Rational decision-making depends on the user’s sensitivity – illustrate with how we respond to low probabilities:
- 5% risk that a plane will crash - ***would you board it?***
- 5% risk of rain*** – would you play golf?***
- Decisions must be based on user’s Cost/Loss ratio
- users with low C/L should protect at low probabilities

## The 50% “problem”

- We are often told you should never say 50%
- “Covering your back”
- “You have no idea”
- “How can anyone make a decision?”
- etc, etc
- But...

## Probabilities in Context

- There’s a 50% prob of snow in London tomorrow

- 50% ? You mean you don’t know what will happen!

- Normally it only snows one day in 50 at this time of year - so 50% is a strong signal.

## Some days really are 50%!

- There is nothing wrong with 50% as part of a sequence of forecasts which change from day to day

- There’s a 50% prob of rain in Exeter on  Thursday

- 50% ? Rain’s not rare - you really haven’t a clue this time!

- No – on this occasion it’s very finely balanced. Yesterday it was 70%, the day before it was 20%. Today the forecast happens to be 50%.

## Hedging

- Should we swing one way or the other to *Help customers make decisions*?...NO!
- We can show categorically that hedging damages verification scores (eg Brier Skill Score, RPSS are *strictly proper*)
- Different customers have different sensitivities
- If a customer’s rational decision threshold is 60% and we hedge 50% to 60% they will make the wrong decision!
- Where the science says 50%, issue 50%!

## Research on Understanding of Forecasts including Uncertainty

- Thanks to Mark Roulston (ex-Met Office) and Todd Kaplan (University of Exeter)

## Methods

- Questionnaire about uncertainty in the 5-day temperature forecast placed on Met Office public website from Tuesday 13th June until Monday 19th June 2006. (1144 external responses).
- Experiments conducted in the experimental economics laboratory at Exeter University to test undergraduates’ understanding of format for 5-day temperature forecasts.

## Temperature Uncertainty Questionnaire

## Option 1

## Option 2

## Option 3

## Option 4

## Option 5

## Most Useful?

- Option 1

- Option 2

- Option 3

- Option 4

- Option 5

## Easiest Format?

- Option 1

- Option 2

- Option 3

- Option 4

- Option 5

## Option 5: Popular

- “Options 3 and 5 seem equivalent 5 looks best.”

- “If the graph type option 5 appeared regularly on the regional forecast pages we'd get used to it very quickly.”

- “Tried it on my kids too.  They also found Option 5 simplest.”

- “I really like Option 5 - very useful.”

- “option 5 is good actually but a person has to be an experienced graph reader or have time to devote to deciphering it.”

- “option 5 is MUCH better than the rest”

- “Option five looks particularly good”

## Option 5 – design features

**Option 5 – design features**

## Temperature Uncertainty Laboratory Experiments

## Experimental Design

- Experiments conducted at the Finance and Economics Experimental Economics Laboratory at Exeter University (FEELE) with Todd Kaplan.
- 153 undergraduates at Exeter University participated.
- Students studying business, economics, humanities and sciences/mathematics
- Group A (77) received no uncertainty information.
- Group B (76) received uncertainty information.
- Presented with a set of 20 “lotteries”.

## Example of a “lottery”

- Participants asked whether they would rather receive £0.50 (US$1.00) if
- A: Temperature at midday on Sunday is below 9oC
- B: Temperature at midday on Monday is below 12oC
- 20 questions of this type asked of each group.

**GROUP A**

**GROUP B**

- Questions in which someone assuming uniform uncertainty would make a *different* decision to someone with Format B were called “swing” questions.

## How did they do?

| Group | % correct (Format A) | % correct (Format B) |
| --- | --- | --- |
| Business/Econ | 69.6 | 85.7 |
| Science/Engr | 68.5 | 85.8 |
| Humanities | 66.5 | 83.8 |
| Male | 69.3 | 85.8 |
| Female | 66.8 | 83.7 |
| Overall | 68.5 | 85.2 |

**Average**** ****earnings £7.25**

**Average earnings £8.48**

## Statistical Analysis

| Predictor | Coeff | dF/dx | P-value |
| --- | --- | --- | --- |
| Question Number | 0.0060 | 0.0017 | 0.236 |
| Reverse order | -0.0871 | -0.0247 | 0.295 |
| Early Correct | -0.2446 | -0.0699 | 0.003 |
| Early Correct & Format B | -0.1560 | -0.0451 | 0.180 |
| Format B | 0.0244 | 0.0068 | 0.785 |
| Swing Question | -0.5287 | -0.1587 | 0.000 |
| Swing Question & Format A | -1.3672 | -0.4761 | 0.000 |
| Gender is Male | 0.0344 | 0.0096 | 0.587 |
| Humanities | -0.1478 | -0.0422 | 0.086 |
| Science/Engineering | -0.0906 | -0.0256 | 0.254 |
| Nationality is British | 0.3842 | 0.1207 | 0.001 |
| Prob Question Mistake | -0.1269 | -0.0361 | 0.082 |
| Sample Question Mistake | -0.0169 | -0.0047 | 0.793 |
| Constant | 0.9932 |  | 0.000 |

**Bias towards picking the later option.**

**People 16% less likely to get “swing” questions right – 63% less likely if they had no uncertainty information.**

**British people (aka native English speakers) were 12% more likely to get answers right.**

## How long did it take them?

- Getting faster as the experiment progresses.

- Not much difference between groups.

## Summary

- Laboratory experiments suggest people are able to correctly interpret uncertainty information.
- Laboratory experiments can be used to objectively evaluate how well people understand forecasts.

## Precipitation Uncertainty Web Survey

- Similar survey to temperature one. Only conclusion presented here – more details at end if you want to look afterwards.

## Two most popular presentations for precipitation

## Aspects of Effective Communication

- Use of words and colours

## Verbal descriptions of Uncertainty – the IPCC scale

- The IPCC have proposed a Likelihood Scale for communication of climate change predictions:

| Virtually Certain | >99% probability |
| --- | --- |
| Very Likely | >90% probability |
| Likely | >66% probability |
| About as likely as not | 33% to 66% probability |
| Unlikely | <33% probability |
| Very Unlikely | <10% probability |
| Exceptionally Unlikely | <1% probability |

## Verbal descriptions of Uncertainty – an alternative scale

- An alternative scale proposed for general use by WMO:

| Extremely Likely | >99% probability |
| --- | --- |
| Very Likely | 90-99% probability |
| Likely | 70-89% probability |
| Probable – more likely than not | 55-69% probability |
| Equally likely as not | 45-54% probability |
| Possible – less likely than not | 30-44% probability |
| Unlikely | 10-29% probability |
| Very Unlikely | 1-9% probability |
| Extremely Unlikely | <1% probability |

## Misinterpretation of words

- Psychology research shows that many people understand words differently, eg
- Probable
- Possible
- Chance
- People *think *they understand them, but one person’s understanding is different from another’s
- Increases with cultural difference eg my understanding as British compared to yours as Americans
- Need clear definitions and consistent use

## Colour on maps

- Use of colour can be effective on maps

## Probs of multiple categories

- Inclusion of probabilities for alternative scenarios can help to communicate that “*something else could happen”*
- Example from seasonal outlook
- Pie chart is also effective:

## Problems with coloursChoose colour scales carefully

- Colours either side of 50% give misleading impression of very small difference in forecast:
- Above/below normal
- Colours at opposite ends of scale very similar

- Revised version:
- Avoids ambiguity

## Confidence indices

- Confidence index is easy for users to understand
- Needs care because confidence can be very different for different elements of the forecast, eg.
- Temperature
- Precipitation
- Sunshine

## Presentation of Probability and Uncertainty

- Further examples of presentations for specific purposes

## Meteograms

- Original design by ECMWF
- Widely used by ensemble forecasters
- Spread of uncertainty for several variables

## Special Meteogram for Windstorms project

- Combines Meteogram and Fan-chart ideas to present wind speed and gust speed

## Wind roses – probs ofspeed and direction

## Stacked Probability Graph

- Probability graph for multiple severity thresholds
- Example of use for risk management in offshore oil industry
- Not very popular with users

## Questions and answers

## Precipitation Uncertainty Web Survey

- Details omitted from main presentation.

## Methods

- Questionnaire about uncertainty in the 5-day precipitation forecast placed on Met Office public website from 17th January  to 9th March (883 external responses).

## Option 1

## Option 2

## Option 3

## Option 4

## Option 5

## Most Useful?

- Option 1

- Option 2

- Option 3

- Option 4

- Option 5

## Easiest to Understand?

- Option 1

- Option 2

- Option 3

- Option 4

- Option 5

## Option 3: Most popular.

- “Option 3 is the most intuitively easy format to understand and interpret. It tells you when it's likely to rain and how much is likely to fall at a glance.”

- “Format ‘1' is the easiest to interpret quickly but ‘3‘ gives much more information and is also easy to interpret.”

- “I favour 3 because the principal information - amount of rain - is seen at a glance, probabilities become apparent on further inspection.”

- “The key for format 3 might be better colour coded with red green etc. I like the height of the bar representing amount of rain.”

## Option 1: Easiest to understand

- “Although I say 1 is easiest to understand that is because it provides least information. I definitely would not want to see that version used.”

- “I think ‘1' preferable to the others. The easy and quick to look at format gives the basic information that is needed by most of the public.”

- “Are these options mutually exclusive? Could users not be presented with the most simplified option 1 as a default and then request to see the other more complex options according to preference?”

- “This would be very useful to display in the Weathercall and Marinecall services and option A seems the clearest and easiest way of expressing the information.”