# Presentation: 305875

## Probabilistic forecasting: Statistical background and using probabilistic data when making forecasts

**Probabilistic forecasting: Statistical background and using probabilistic data when making forecasts**

- Session 3, 9 October 2007
- Bill Bua, UCAR/COMET
- and
- Rich Grumm, NOAA/NWS

**Notes:**
This web cast is a presentation of an Introduction to Ensemble Prediction, created by Dr. Bill Bua, Project Scientist at COMET.  The web cast supplements the recently released web-based module, Ensemble Forecasting Explained.

## Outline

- Deterministic forecasts and uncertainty.
- Definition of probabilities.
- Distributions:
- Frequency distributions.
- Measures of central tendency.
- Measures of dispersion.
- Measures of shapes.
- Statistics of probability distributions.
- Probability density functions.
- Cumulative density functions.
- Distributions used in meteorology
- Joint distributions
- Clustering of similar forecasts

## Deterministic Forecasts and Uncertainty

- From model jumpiness lecture by Rich
- Four consecutive GFS runs from 13Aug07 valid 12 UTC 18 Aug07
- Degree of “jumpiness”  sensitivity of model to initial conditions
- East coast S. Africa low appeared in GFS runs from 00 and 06 UTC 13Aug07 valid time 12 UTC 18Aug
- Shifts southwest in 12 UTC 13Aug07 run
- Just a weak trough by 18 UTC 13Aug07 run

- First you see it....

- Now you don’t!

- NWS

## Same forecast valid time from different NWP models

- Different models from same cycle have broadly similar forecasts for 12 UTC 18 Aug 2007
- Subtle model differences may result in significant sensible weather differences

- 1003
-    ***L***

- 1008
-    ***L***

-    ***L ***
- 1006

**NCEP**

## And the verification was...

- None of the forecasts we showed from 13 August were terribly good over S. Africa.
- D(prog)/Dt for the GFS was heading in the right direction.
- Slowing and weakening eastward moving trough from Atlantic

- NCEP

## Sources of Uncertainty

- Model formulation of dynamics and physics
- Dynamics  differencing schemes, resolution, treatment of boundary conditions
- Physics  parameterization of effect of sub-gridscale physical processes (e.g. convection) on resolved scales
- Initial conditions
- Forecast sensitive to *which* data is assimilated into models ... missing data may degrade forecast
- Computer hardware
- Different hardware configurations can lead to different forecasts!

## Probabilistic Forecasting

- Although weather forecasting appears to be a ***deterministic*** initial value problem based on the laws of physics, there is an ***inherent uncertainty*** in all weather forecasts.
- This uncertainty comes from data assimilation and modelling approximations and errors and from the chaos effect in the predictability aspect of the atmosphere.
- If this uncertainty can be assessed, it ought to be part of the weather forecasts.

## What we have covered so far ...

- There is uncertainty in the forecasts.
- No forecast is complete without an assessment of uncertainty.

**Any QUESTIONS?**

## Definition of “probabilistic”

- What do we mean by a “probabilistic” forecast?
- What does this statement mean to you?
- What does it mean to users of forecasts?

***The chance of showers today is 30%***

## Definition of probabilistic forecasts

- Calculus (and Venn Diagram) point of view:
- Methodology for computing, from given probabilities, other probabilities that depend upon the former in a more or less complex manner.
- The probability *P* of some event *E**i*, noted *P(E**i**)*, is defined with respect to all possible elementary events in such a way that *P* must satisfy three basic axioms (the Kolmogorov axioms)...

**The probability of an event *****E******i***** is**

**The probability of all events *****E******i***** is**

**If all *****E******i***** are mutually exclusive**

**Notes:**
The use of probabilities in weather forecasting is the most common way of conveying the chance of occurrence of a future event. It is generally thought that probabilities are either the product of past knowledge projected into the future or a personal degree of belief regarding the occurrence of the event. Although both interpretations stated here satisfy the conditions of probability calculus, one could wonder if these two interpretation are really describing the same thing. With the advent of ensemble forecasting, probability of occurrence of an event can be obtain as a direct result of numerical forecasting, implying that such probabilities may be linked to the intrinsic predictability of the event. But do probabilities express the uncertainty in the forecasts ?

The definition of probability is very wide and need only to satisfy the three Kolmogorov axioms : (1) probability is a number between 0 and 1 ; (2) the probability of all events is 1 and (3) if events are disjoint, the probabilities of the union of these events is the sum of the individual probability of occurrence of each event. From a calculus stand point, probabilities are simply numbers and other probabilities can be calculated from the former in a more or less complex manner following the Kolmogorov axioms from which the laws of probability can be derived. 

## Definition of probabilistic forecasts

- Classical interpretation:
- Based on the principle of indifference  each event is equally probable.
- Probabilities are estimated by the ratio of favorable cases against all possible cases (e.g. roll of dice, coin toss)
- The estimated probabilities coincide with frequencies of occurrence in a large set of trials
- Limitations:
- The hypothesis of equal probability may not apply
- Difficult to apply in meteorology because events are unique (although similar to others) and all events are not equally probable

**Notes:**
The classical interpretation of probability is based on the principle of indifference or symmetry theory which proposes that events are equiprobable. The probabilities of appearance of given combinations of numbers are estimated by the ratio of favorable cases against all possible cases. The estimated probabilities coincide with frequencies of occurrence obtained in a large set of trials. 

Amongst the limitations of the classical interpretation of probability is the fact that the hypothesis of equiprobability is not always tenable. For example the classical interpretation will not work with an unfair die (not equiprobable). Equiprobability in this case would lead to a biased estimate of the likelihood of occurrence of the event. 

It is difficult to apply the classical interpretation of probability in meteorology because all weather events are unique, although similar or analogous in nature. For example two different but similar low pressure systems may have different intrinsic probabilities for occurrences or precipitation depending on their fundamental nature and structure. One system may have more moisture and precipitable water than the other, the circulation may be different etc. One thing is for sure, the symmetry theory or the principle of indifference does not work in meteorology.

## Definition of probabilistic forecasts

- Frequentist interpretation:
- Based on frequency of occurrences
- Unambiguous
- Probability ***P*** of an event ***E******i*** is defined as:
- Limitations:
- The infinite limit is impossible to attain, so we’re always dealing with estimates using event sampling from *subsets of data*
- Different relative frequencies of the event may be obtained by using different subsets of the trials
- Not well suited for short records
- Confusion of definition and measurement of probability
- Difficult to apply in meteorology because events are unique (although similar to others)

**Notes:**
If one is not sure that equiprobability applies, then the only way to estimate probability is to study the relative frequency of occurrences of the events in a large set of trials. This is the frequentist interpretation of probability. This interpretation implies that the computations of the classical interpretation should always await the confirmation of the experiment. According to the frequentist stand point, the probability p of an event Ei is defined as the limit of the ratio of the number of occurrences of the event and the total number of trials, the total number of trials approaching infinity. 

According to the frequentist interpretation of probability a statement such as « there is 80% probability of precipitation » may be understood as « precipitation occurs 80% of the times this is forecast » which refers to the reliability of the probability forecasts (we will see what reliability is later on in this module). Or another interpretation of the statement is « there is an 80% chance that the forecast will verify against an actual occurrence of the forecast event ». This latter interpretation refers to the uncertainty on the forecast.

Amongst the limitations of the frequentist interpretation of probability there is the fact that it is not possible to attain the infinity in the number of trials to calculate probabilities. This implies that probabilities have to be calculated based on a finite number of trials, and consequently different probabilities can be achieved with different subsets of the trials or climatology. 

Although each event in meteorology is unique, the frequentist approach is widely used.

## Definition of probabilistic forecasts

- Subjective interpretation:
- Probability is a number (between 0 and 1) that represents the degree of belief of a person about the occurrence of an event, implying that probability *per se* may not exist.
- Allows the forecast of events that have no previous records.
- Limitations:
- No objective tool to measure the degree of belief, and not systematically reproducible.
- Quantifying numbers may seem arbitrary.
- Different persons (even experts!) may have conflicting beliefs about the likelihood of an event, but all are equally valid in principle.
- May be affected by non-rational behavior and cognitive illusions.
- Not applicable in meteorology  impossible to ensure reliability of forecasts.

**Notes:**
According to the subjective interpretation of probability, probability is a number that represents the degree of belief of a person about the occurrences of a given event which implies that probability per se does not exist. This interpretation may not differ greatly from what one can call « opinion ». 

One of the limitations of this viewpoint is that there is no objective tool to measure the degree of belief of a person. Also the quantification in numbers may seem arbitrary. Different persons may have conflicting beliefs about the likelihood of occurrence of an event, and all are equally valid in principle (although one would tend to believe whoever has the best record with respect to previous forecasts). It is also possible that even experts may be somewhat affected by non-rational behavior and cognitive illusions. The main advantage of this interpretation is the fact that it allows predictions of situations that have no previous record.

## Definition of probabilistic forecasts

- In meteorology, the frequentist approach is widely used:
- Different (although similar) events are combined to estimate probabilities of occurrences
- Forecast can be verified and calibrated based on results
- However, users are likely to understand probabilities as the forecaster’s estimate of the likelihood of occurrence of the event (subjective interpretation)

**Notes:**
In meteorology, the frequentist interpretation of probability is widely used. For example large samples of a predictand are gathered to generate correlations with predictors in the Perfect Prog (PP) or MOS statistical adaptation approaches. Different events, different in nature and unique, although similar to several others, are combined to derive statistics and probabilities. This is widely applied also in verification where large number of forecasts are combined to measure skill etc. However, every forecasters will know that forecast skill depends on each problem to solve: there are cases where the confidence is higher and hence the forecasts should show better skill than in other situations where the confidence is low.

Although probabilities may be calculated based on frequencies of occurrences in a finite sample, users are likely to interpret them as the forecasters’ estimate of the likelihood of occurrence of the event, which is the subjective interpretation of probability. 

From here on in this workshop, probability is interpreted according to the frequentist approach. But we have to be careful here because the ensemble prediction systems give the capability of having probabilities that are situation dependent (approaching the propensity interpretation of probability). 

## Definition of probabilistic forecasts

- A forecast that says that:can be interpreted as:
- A statement on the uncertainty of the event (frequentist interpretation).
- A forecaster’s estimate on the likelihood of occurrence of that event (subjective interpretation).
- A statement that says that the forecast has Y% chance of verifying against the actual occurrence of the event.

***The chance of showers today is 30%***

**Notes:**
There are different interpretations to a statement such as « the chance of showers today is 30% ». 

One is a statement on the likelihood of occurrence of the event. This interpretation can fit under the subjective, frequentist or propensity interpretations of probability. 

One interpretation is a forecaster’s estimate of the likelihood of occurrence of the event. This is under the subjective interpretation of probability and is what users are likely to do. 

Another one is a statement on the uncertainty of the forecast. This is within the frequentist paradigm. The probability in the forecast can be seen as a statement of the reliability of the forecasts: « there is 30% chance that the forecast will actually verify against an occurrence of the forecast event ». Such an interpretation is valid as long as the forecasts are reliable (reliability being measured or achieved by calibration over a large verification sample of forecasts). 

## Definition of probabilistic forecasts

- What is an event?
- Phenomena happening:
- At a location (point, area, region).
- Over a valid time period.
- Can be defined with threshold values.
- Can be defined with ranges of values.
- The definition of events is important:
- To avoid confusion in forecasts.
- For verification purposes.

**Notes:**
But what is an event? It is important to define what we call event first to make sure that forecasts are understood properly and second for verification purposes. An event is « something » happening « somewhere » over a given time period. Where the phenomena is happening can be a point, a city, a location, an area or a forecast region. But all these must be well defined. The phenomena can be defined with threshold values (more than 25 mm of precipitation or temperature below 0C for examples), with ranges of values (winds 15 to 30 km/h for example). 

## What we have covered so far ...

- There is uncertainty in the forecasts.
- No forecast is complete without an assessment of uncertainty.
- There are several interpretation of probabilities – the frequentist approach is widely used in meteorology.
- Probability is a measure and an expression of uncertainty.

**Any QUESTIONS?**

## Probabilities

- In meteorology, use the frequentist approach to the problem
- Probabilities are estimated from observed and/or forecast frequencies of occurrences of events:
- Frequency distributions
- Probabilities can be estimated from:
- Climatology distributions
- Statistical adaptation
- Ensemble prediction system:
- Several realizations of the modeled atmosphere
- Ensemble forecasting is like throwing the “weather dice” n times (where n = number of members)

**Notes:**
How do we estimate probabilities? As seen when we discussed the different interpretations of probability, probabilities are generally estimated from the frequencies of occurrences of events. This is the pure frequentist interpretation of probability. If climatology is used, then we can calculate probabilities of occurrences of events based on observed frequencies of those events during the past years. But it is more difficult to apply this approach in the forecast process since forecasters are dealing with one specific problem to solve (the problem of the day). Statistical adaptation such as Perfect Prog (PP) and/or MOS can help here in providing probabilities of occurrences of events, but these will be generated by combining a large number of cases that include several different events (although similar). These probabilities will not be situation dependent. Ensemble prediction systems can provide means to generate probabilities that will be situation or flow dependent. 

## Frequency distribution

- Total number of cases = 50

- Courtesy of R. Verret, CMC

**Notes:**
This graph shows a frequency distribution for temperatures at some location and time. Fifty cases are considered here. The horizontal axis is the temperature and the vertical axis is the number of cases in the sample corresponding to each temperature displayed along the horizontal axis. These could be from a 50-year climatology or from a 50-member ensemble prediction system. We will use this frequency distribution as an example to explain some statistics and probabilities that can be derived from such distributions. 

## Frequency distribution

- Total number of cases = 50

- P(16 ≤ T ≤ 18) = ?

- P(T > 19) = ?

- P(T < 15) = ?

- Courtesy of R. Verret, CMC

**Notes:**
From this frequency distribution, what is the probability that the temperature will be larger or equal to 16C and less than or equal to 18C? 

What is the probability that the temperatures will be above 19C? 

What is the probability that the temperatures will be below 15C?

## Frequency distribution

- Total number of cases = 50

- P(16 ≤ T ≤ 18) = 22/50 = 44%

- P(T > 19) = 7/50 = 14%

- P(T < 15) = 10/50 = 20%

- Courtesy of R. Verret, CMC

**Notes:**
Under the assumptions of the frequentist interpretation of probability, the probability of having the temperatures between 16 and 18C will be 44% since we have in the distribution 22 cases where the actual temperature is between the two limits over a total number of 50 cases. 

Similarly, the probability of having a temperature above 19C is 14% since there are 7 cases in the sample with a temperature above 19C over a total of 50 cases.

Again, the probability of having a temperature below 15C is 20% since there are 10 cases in the sample with a temperature below 15C over a total of 50 cases.

## Measures of central tendency

**Mode** = value that occurs with the greatest frequency
- Not sensitive to extreme
- Not necessarily unique (multi-modal distributions)

**Mean** = (frequency X values) / total number of cases
- Sensitive to extreme

- 1st moment

**16.7**

**Mean **

**Mode**

- Courtesy of R. Verret, CMC

**Notes:**
In order to be in a position of providing probabilistic information to users, we are interested in finding the characteristics of the distribution, such as its shape or how is the data distributed around the mean etc. Measures of central tendency provide information about the distribution. The mean of the distribution is the sum over all values multiplied by their respective frequency of occurrence divided by the total number of cases. This is the arithmetic mean of all cases. This is also referred to as the first moment of the distribution. It is important to note that the mean of the distribution is sensitive to extreme values and to outliers of the distribution. In the distribution here, the mean is 16.68.

Another measure of central tendency is the mode which is defined as the value that occurs with the greatest frequency. It is the value that has the higher probability of occurrence. Unlike the mean, the mode is not sensitive to extreme values of the distribution or outliers. It is important to note also that the mode is not necessarily unique. the distribution here has one mode only, and this mode is 17C.

## Measures of central tendency

- modes

- Courtesy of R. Verret, CMC

**Notes:**
This is an example of a distribution with three modes. In this case, we talk of a tri-modal distribution (two modes would indicate a bi-modal distribution). It is possible to find distributions with more than three modes. In those cases, we can talk of multi-modal distributions. In the example here, we have a mode at approximately -9.8C, a second mode at about -6.5C and a third one at approximately -3.5 C. Multi-modal distributions make the forecast decisions more difficult.

## Measures of dispersion

**Median** = the middle value or the arithmetic mean of the two middle values of a set of numbers arranged in order of magnitude
- Divides the sample in half
- 50% of the values are above the median
- and 50% are below

**Quartiles** = the values which divide the set into four equal parts
- Q1, 2, 3

**Terciles** = the values which divide the set into three equal parts
- T1, 2

**Deciles** = the values which divide the set into ten equal parts
- D1, 2, 3...9

**Percentiles** = the values which divide the set into
- one hundred equal parts
- P1, 2, 3... 99

**Q****1**

**Q****3**

**Q****2**

**Median**

- Courtesy of R. Verret, CMC

**Notes:**
The degree to which numerical data tend to spread about an average value is called the variation or dispersion of the data. Various measures of dispassion are available. 

The median is the value that splits the distribution into to halves, 50% of the data being above that value and 50% being below. It is the middle value or the arithmetic mean of the two middle values of a set of numbers arranged in order of magnitude. Of course, the median is not sensitive to extreme values, and the median is unique to a given distribution. 

There are other ways to split a distribution once the data has been arranged in order of magnitude. Terciles will split the distribution into three equal parts each containing 33.33% of the data. We need only two values to define the terciles (noted T1 and T2). The quartiles will divide the data set into four equal parts each containing 25% of the data. Quartiles are noted Q1, Q2 and Q3. The deciles will divide the sample into ten equal parts, each containing 10% of the data. The deciles are usually notes as D1,2,3...9. Finally the percentiles will divide the data into 100 equal parts each containing 1% of the data. The percentiles are noted as P1,2,3...99.

In the example here, the median is 17. The first quartile is 15 and the second quartile is equal to the median and is 17. The third quartile value is 18.75.

## Measures of dispersion

- The range is the difference between the largest value and the smallest one

- Courtesy of R. Verret, CMC

**Median**

**range**

**Q**

**Q****1**

**Q****3**

**Q****2**

**Notes:**
The mean deviation (MD) is the arithmetic average of the absolute difference between each value and the mean of the sample. Sometimes, the mean deviation is defined in terms of absolute deviations from the median or other average instead of the mean. An interesting property of the mean deviation is that the mean deviation is minimum when the reference is the median.

The semi-interquartile range (Q) is the mean of the first and third quartile.

The 10-90 percentile range (P10-90) is the 90th percentile minus the 10th percentile values.

The range of a set of values is the difference between the largest and smallest number in the set. In the example here, the range is 12 (22-10). 

## Measures of dispersion

- 2nd moment

- Courtesy of R. Verret, CMC

**1 ******

**1 ******

**16.7**

**Mean**

**Notes:**
The standard deviation measures the dispersion around the mean of the distribution. It is the root mean square of the averaged squared deviation from the mean. Note that usually, one is subtracted from the total number of cases when averaging the squared deviation from the mean because the resulting value is a better estimate of the standard deviation of a population from which the sample is taken. This also accounts for the loss of one degree of freedom because the mean of the sample has to be estimated first for the calculation of the standard deviation.

The variance is the square of the standard deviation.  The variance is the second moment of the distribution.

## Measures of shape

**Mode**

**Mode**

**Median**

**Median**

**Mean**

**Mean**

- 3rd moment

- Courtesy of R. Verret, CMC

**Notes:**
Skewness is the degree of asymmetry, or the departure from symmetry, of a distribution. For a variable which has a reasonably symmetric distribution, the mean and median will usually be fairly similar. Positively skewed distributions have a mean that is higher than the median, and the opposite for negatively skewed distributions. The difference between the mean and the median divided by the standard deviation provides one measure of the skewness of the distribution. Another measure of skewness is given by the equation here. This is referred to as the third moment of the distribution. For perfectly symmetrical distributions, skewness is zero.

## Statistics of probability distribution

| N | 50 |
| --- | --- |
|  | 16.68 |
| Mode | 17 |
| T1 | 16 |
| T2 | 18 |
| Q1 | 15 |
| Q2 | 17 |
| Q3 | 18.75 |
| Q | 1.875 |
|  | 2.69 |
| 2 | 7.24 |
| Minimum | 10 |
| Maximum | 22 |
| Range | 12 |

- Courtesy of R. Verret, CMC

**1 ******

**1 ******

**16.7**

**Mean**

**Median**

**Q****1**

**Q****3**

**50%**

**Notes:**
As a summary, here are the characteristics of the distribution that we have been looking at for a while. Note that P10-90 is not calculated here because it would be meaningless since we only have 50 cases in the frequency distribution. Note that 50% of the data is between Q1 and Q3. The mean is slightly to the left (smaller) of the median, meaning that this distribution is slightly negatively skewed. 

## What we have covered so far ...

- There is uncertainty in the forecasts.
- No forecast is complete without an assessment of uncertainty.
- There are several interpretation of probabilities – the frequentist approach is widely used in meteorology.
- Probability is a measure and an expression of uncertainty.

**Any QUESTIONS?**

## Probability density function (pdf)

**pdf** = a graph of the frequency of x for all values of x

- Courtesy of R. Verret, CMC

**Notes:**
Instead of having a graph with the number of cases in the ordinate, it is possible to generate a graph with relative frequencies in the ordinate (the number of cases is simply divided by the total number of cases). This will then be referred to as a probability density function. The blue curve in the graph here is the probability density function corresponding to the frequency distribution that we have been using as example so far. The red curve is a normal distribution fitted to the data.

## Probability density function (pdf)

**pdf** = a graph of the frequency of x for all values of x

- Courtesy of R. Verret, CMC

**Notes:**
Instead of having a graph with the number of cases in the ordinate, it is possible to generate a graph with relative frequencies in the ordinate (the number of cases is simply divided by the total number of cases). This will then be referred to as a probability density function. The blue curve in the graph here is the probability density function corresponding to the frequency distribution that we have been using as example so far. The red curve is a normal distribution fitted to the data.

## Probability density function (pdf)

- P(12 ≤ T ≤ 15)

- Courtesy of R. Verret, CMC

- Pe(12 ≤ T ≤ 15) = 13/50 = 26%

**Notes:**
The area under the PDF, or the integral of the PDf for a given variable x between x1 and x2 gives the probability that x will be between x1 and x2. x1 and or x2 can be the infinite. Hence the integral of the PDF from minus infinity to X or from X to plus infinity will give the probability that x is below or above the value X respectively. The normal distribution fitted to the data that we have been using so far is reproduced here. In this example, the probability that T is above or equal to 12 and less than or equal to 15 is 30.3%% as calculated from the area under the normal distribution (Note: since the temperatures are all integer, the area under the normal curve is integrated from 11.5 to 15.5). From the empirical data, there is 13 cases out of 50 with temperatures of 12, 13, 14 or 15 which implies that the empirical probability is 26%. 

## Cumulative density function (cdf)

- Courtesy of R. Verret, CMC

**Notes:**
If the probability density function (pdf) in integrated from minus infinity up to a value x, we would get the probability that the variable is less than the value x. The integral of the pdf as a function of x is called the cumulative probability density function (cdf). The cdf can be graphed much like the pdf. The blue curve here is the cdf corresponding to the pdf of the previous slide. The red curve is the cdf of the normal distribution that has been fitted to the data.

## Probability rules

***E******i***

***E******j***

- If *E**i* and *E**j* are mutually exclusive

- If *E**i* and *E**j* are independent

**Notes:**
The fundamental rules of probability can be derived from the tree Kolmogorov axioms. 

The probability of having event Ei and Ej together is equal to the conditional probability of Ei given Ej times the probability of having Ej. It is also equal to the conditional probability of having Ej given Ei time the probability of having Ei. 

The probability of having Ei or Ej is the sum of the individual probability of Ei and Ej minus the probability of having both Ei and Ej at the same time.

The probability of having Ei and Ej is zero if Ei and Ej are mutually exclusive. In the Venn diagram, there is no intersection of events Ei and Ej (or it can be said that the intersection is a null ensemble).

The probability of having Ei and Ej is the product of the individual probability of Ei and Ej if Ei and Ej are independent. The conditional probability of Ei given Ej is equal to the probability of Ei when Ei and Ej are independent (there is no relationship between Ei and Ej).

The  first two rules lead to the Baye’s theorem. 

## Theoretical Probability Distributions used in meteorology

| Normal | Unbounded continuous variables such as temperatures, geopotential heights, pressure. | Two parameter distribution (mean and standard deviation).
Symmetric (mean = median = mode).
Bell shaped. |
| --- | --- | --- |
| Gamma | Precipitation amount (continuous variables bounded below). | Two parameter distribution (« shape » and « spread »).
Positively skewed.
The cube-root of precipitation amount has been found to be approximately normally distributed. |
| Weibull | Wind speed (continuous variables bounded below). | Two parameter distribution.
Negatively or positively skewed depending on parameters. |
| Beta | Cloud amount (continuous variables bounded above and below). | Two parameter distribution (family of distributions including the U-shaped distribution).
Negatively or positively skewed depending on parameters. |
| Lognormal | Visibility (positive definite variables). | Normal distribution with logarithmic x-axis. |

- Courtesy of L. Wilson, RPN

**Notes:**
Instead of looking at distributions based on actual data (which is a sampling of the true population with all the approximations that that implies) or in other words looking at an empirical discrete distribution or discrete probability function it may be more convenient to use theoretical distribution that can be fitted to the data (continuous probability function). For example we have fitted a normal distribution to the data that we have been using so far. It is often easier to derive statistics and probabilities from the theoretical distribution rather that working with actual data. 

The normal distribution is convenient for unbounded continuous variables such as temperatures, pressure, geopotential heights. It is symmetric and has a bell shape. The Gamma distribution has been found to work well with precipitation amounts, which is a continuous variable bounded below. The Weibull distribution is useful for wind speed, another continuous variable bounded below. The Beta distribution is useful for cloud amounts or cloud opacity, a continuous variable bounded on both side. The Beta distribution can be U-shaped. The lognormal distribution works well for visibility, a positive definite variable bounded below. 

Click on the name of the distribution for an example. Click on the button « return » on the example slides to come back here. 

## Probability rules

***E******i***

***E******j***

- If *E**i* and *E**j* are mutually exclusive

- If *E**i* and *E**j* are independent

**Notes:**
The fundamental rules of probability can be derived from the three Kolmogorov axioms. 

The probability of having event Ei and Ej together is equal to the conditional probability of Ei given Ej times the probability of having Ej. It is also equal to the conditional probability of having Ej given Ei time the probability of having Ei. 

The probability of having Ei or Ej is the sum of the individual probability of Ei and Ej minus the probability of having both Ei and Ej at the same time.

The probability of having Ei and Ej is zero if Ei and Ej are mutually exclusive. In the Venn diagram, there is no intersection of events Ei and Ej (or it can be said that the intersection is a null ensemble).

The probability of having Ei and Ej is the product of the individual probability of Ei and Ej if Ei and Ej are independent. The conditional probability of Ei given Ej is equal to the probability of Ei when Ei and Ej are independent (there is no relationship between Ei and Ej).

The  first two rules lead to the Baye’s theorem. 

## Joint distributions

**Event A**

**Event B**

- P(A) = (a+c) / T
- P(B) = (a+b) / T
- P(A+B) = a / T
- P(A+not B) = c / T
- P(B+not A) = b / T
- P(not A+not B) = d / T

- P(A|B) = a / (a+b)
- P(B|A) = a / (a+c)
- P(A|not B) = c / (c+d)
- P(B|not A) = b / (b+d)

- Marginal probabilities

- Conditional probabilities

**Notes:**
It is sometimes important to look at the probability of two events occurring at the same time. For example, a forecaster may be interested at the probability that heavy rain falls with strong winds at the same time, or at the probability that the temperature will be below freezing and rain falling at the same time (hence freezing rain). A forecaster may be interested at the probability of having precipitation in a given period given that there is or not precipitation in the previous period (two consecutive 12-h periods for example). Two by two contingency tables such as this one here can be constructed. Such a table gives the number of cases (or relative frequencies) of both events occurring relative one to the other. We can refer to these contingency tables as joint probability tables and it becomes possible to calculate marginal probabilities and conditional probabilities for the events. This concept can be extended to more than two events occurring at the same time. With ensemble prediction system, the relative numbers of members forecasting either one or both events can be inserted in the contingency tables. 

The inner cells are related to the frequency of jointly occurring events, while the marginal cells are related to the total probability of the event. 

## Joint distributions

**Event A- wind speed > 30-kt**

**Event B – Rain > 25-mm/6hr**

- P(A) = ?
- P(B) = ?
- P(A+B) = ?
- P(A+not B) = ?
- P(B+not A) = ?
- P(not A+not B) = ?

- P(A|B) = ?
- P(B|A) = ?
- P(A|not B) = ?
- P(B|not A) = ?

- Marginal probabilities

- Conditional probabilities

- Total of 10 members:
-  7 members give rain > 25mm
-  3 members give rain ≤ 25mm
-  5 member give wind > 30-kt
-  5 members give wind ≤ 30-kt
-  3 members give rain and wind ≤ 30-kt

- R. Verret, CMC

**Notes:**
Suppose we have a 10 member ensemble. 6 members give rain while the remaining 4 members give no rain. 7 members forecast temperatures below 0C, while the 3 remaining members have temperatures above or equal to 0C. And finally, 2 members give rain with temperatures above or equal to 0C. From that information, how can we fill in the 2x2 contingency table that describes the joint distribution of rain event and temperatures below freezing event ? Once we have the contingency table filled in, we can calculate the marginal and conditional probabilities according to the definitions that we have here. The total number of cases (10) in normalized to one here (numbers are divided by 10).

## Joint distributions

- P(A) = ?
- P(B) = ?
- P(A+B) = ?
- P(A+not B) = ?
- P(B+not A) = ?
- P(not A+not B) = ?

- P(A|B) = ?
- P(B|A) = ?
- P(A|not B) = ?
- P(B|not A) = ?

- Marginal probabilities

- Conditional probabilities

- R. Verret, CMC

**Event A- wind speed > 30-kt**

**Event B – Rain > 25-mm/6hr**

- Total of 10 members:
-  7 members give rain > 25mm
-  3 members give rain ≤ 25mm
-  5 member give wind > 30-kt
-  5 members give wind ≤ 30-kt
-  3 members give rain and wind ≤ 30-kt

**Notes:**
The fact that 6 members out of 10 give rain indicates that the probability of having rain is 0.6 (R1). Consequently the probability of having no rain is 0.4 (R2) (4 members out of 10 give no rain). The probability of no rain can be deduced from 1 - 0.6. 

## Joint distributions

- P(A) = ?
- P(B) = ?
- P(A+B) = ?
- P(A+not B) = ?
- P(B+not A) = ?
- P(not A+not B) = ?

- P(A|B) = ?
- P(B|A) = ?
- P(A|not B) = ?
- P(B|not A) = ?

- Marginal probabilities

- Conditional probabilities

- After R. Verret, CMC

**Event A- wind speed > 30-kt**

**Event B – Rain > 25-mm/6hr**

- Total of 10 members:
-  7 members give rain > 25mm
-  3 members give rain ≤ 25mm
-  5 member give wind > 30-kt
-  5 members give wind ≤ 30-kt
-  3 members give rain and wind ≤ 30-kt

**Notes:**
Now 7 members out of 10 give temperatures below 0C, which means that the probability of having freezing temperatures is 0.7 (C1). Consequently the probability of having temperatures equal to 0C or above is 0.3 (C2) (or 1 - 0.7). 

## Joint distributions

- P(A) = ?
- P(B) = ?
- P(A+B) = ?
- P(A+not B) = ?
- P(B+not A) = ?
- P(not A+not B) = ?

- P(A|B) = ?
- P(B|A) = ?
- P(A|not B) = ?
- P(B|not A) = ?

- Marginal probabilities

- Conditional probabilities

- After R. Verret, CMC

**Event A- wind speed > 30-kt**

**Event B – Rain > 25-mm/6hr**

- Total of 10 members:
-  7 members give rain > 25mm
-  3 members give rain ≤ 25mm
-  5 member give wind > 30-kt
-  5 members give wind ≤ 30-kt
-  3 members give rain and wind ≤ 30-kt

**Notes:**
It is not possible to have the conditional probabilities from the marginal probabilities alone. We need at least one piece of information to fill in the inner cells of the contingency table. Here we have that 2 members out of 10 forecast both rain and temperatures equal to 0C or above at the same time. This indicates that the probability of having both rain and above freezing temperatures is 0.2 (cell b). Once we have that, it is easy to fill in the remaining cells.

## Joint distributions

- P(A) = 0.7
- P(B) = 0.5
- P(A+B) = 0.3
- P(A+not B) = 0.2
- P(B+not A) = 0.4
- P(not A+not B) = 0.1

- P(A|B) = 0.43
- P(B|A) = 0.60
- P(A|not B) = 0.67
- P(B|not A) = 0.80

- Marginal probabilities

- Conditional probabilities

- After R. Verret, CMC

**Event A- wind speed > 30-kt**

**Event B – Rain > 25-mm/6hr**

- Total of 10 members:
-  7 members give rain > 25mm
-  3 members give rain ≤ 25mm
-  5 member give wind > 30-kt
-  5 members give wind ≤ 30-kt
-  3 members give rain and wind ≤ 30-kt

**Notes:**
And the mariginal and conditional probabilities follow according to their definitions.

P(A) = (a+c) / T = 0.3+0.4 = 0.7

P(B) = (a+b) / T = 0.3+0.2 = 0.5

P(A+B) = a / T = 0.3

P(A+not B) = c / T = 0.4

P(B+not A) = b / T = 0.2

P(not A+not B) = d / T = 0.1

P(A|B) = a / (a+b) = 0.3/(0.3+0.4) = 0.43

P(B|A) = a / (a+c) = 0.3/(0.3+0.2) = 0.60

P(A|not B) = c / (c+d) = 0.2/(0.2+0.1) = 0.67

P(B|not A) = b / (b+d) = 0.4/(0.4+0.1) = 0.80

## What we’ve covered so far...

- There is uncertainty in the forecasts.
- No forecast is complete without an assessment of uncertainty.
- There are several interpretation of probabilities – the frequentist approach is widely used in meteorology.
- Probability is a measure and an expression of uncertainty.
- Probability density functions (pdf and/or cdf) provide all the information.

**Questions ?**

## Attributes of probabilistic forecasts

- Accuracy:
- Level of agreement between the forecasts and the observations. Only deterministic forecast may have perfect accuracy.
- Skill:
- Accuracy of the forecasts relative to some standard, such as long-term climatology.
- Reliability:
- Average agreement between forecast and observed values (bias). For probabilistic forecasts, measures the agreement between forecast probabilities and observed frequencies of the event.
- Resolution:
- Ability of the forecasts to sort or resolve the events into subsets with different frequency distributions. **No resolution**  same probability distribution of observed values for different forecast values
- Sharpness:
- Tendency to forecast toward extreme values (0 and 100% for probabilistic forecasts). For probabilistic forecasts, measures the tendency toward categorical deterministic forecasts.

**Notes:**
The attributes characterize the forecasts. They statistically relate the performance of a forecast system to verifying data in an abstract sense. They give insight on the various aspects of the forecast system. The main attributes are listed here : accuracy, skill, reliability, resolution and sharpness. The last three attributes are particularly important for probabilistic forecasts.

The accuracy is a general term indicating the level of agreement between forecast weather and true weather as represented by observations. The difference between the forecast value and the observed value is called the error. The smaller this difference is, the smaller the error is and the greater is the accuracy of the forecast. The Brier score for example, will measure the accuracy of probabilistic forecasts.

The skill or relative accuracy is defined as the accuracy of a forecast relative to the accuracy of a forecast produced by some standard procedure. Common standards, which are considered to have no skill, are climatology, persistence and chance. Skill scores provide means of accounting for variations in accuracy which have nothing to do with the forecaster’s ability to forecast.

Reliability is equivalent to bias and is simply the average agreement between the stated forecast value (label) of an element and the observed value (label). For continuous variables it measures the bias of the forecast. For example, reliability may be expressed as the ability to forecast maximum temperatures to within plus or minus 3 degrees C from observations. The reliability for categorical or probability forecasts can be quantified in a reliability table where for example, 70% probability of precipitation forecasts may correspond to an observed frequency of 90%. Such probability forecasts under-forecast the true frequency by 20%. Hence, for probabilistic forecasts, reliability measures the agreement between forecast probabilities and observed frequencies of the event. Reliability may be improved with statistical post-processing of the forecasts.

Resolution is the ability of the forecasts to sort or resolve the set of sample events into subsets with different frequency distributions. Resolution is related to the standard deviation or variance of the observations stratified by the forecasts. It measures the sensitivity of the conditional probability of a variable given the forecasts to different forecast values. If a forecast system leads to identical probability distributions of observed values for different forecast values, then the system has no resolution. Resolution is essential for a forecasting system to be able to discriminate between observable future events. 

Sharpness, an attribute of forecasts alone, refers to the tendency to forecast extreme values. For probability forecasts, this is the tendency toward forecasting 0% and 100% probability, that is, the tendency toward categorical deterministic forecasts. Sharpness is an attribute of the marginal distribution of the forecasts that measures how much the forecasts deviate from the mean climatological value/category for deterministic forecasts, or from the climatological mean probabilities for probabilistic forecasts. Sharpness can be increased by post-processing the forecast using mathematical procedures such as inflation, but at the expense of reliability.

## Attributes of probabilistic forecasts

- Over-confident

- Under-confident

- Perfect reliability

- Sharper forecasts

- Little sharpness

- No resolution

- Reliability

- Sharpness

- After R. Verret, CMC

**Notes:**
The diagram on the left-hand side shows two reliability curves for a given probabilistic forecast system. The observed frequencies of the event are plotted against each forecast probabilities. For all forecasts that say probability of Y%, the observed frequencies of occurrence of the event is measured from a verification sample and plotted on the attribute diagram. A forecast system with perfect reliability will have a reliability curve that is close to the diagonal. When the observed frequencies of the event is larger than the corresponding forecast probabilities, we have a forecast system that is under-confident (probabilities are under-estimated). A forecast system that is over-confident will have a reliability curve that is to the right of the diagonal (the probabilities are over-estimated with respect to the corresponding observed frequencies). The horizontal purple line indicates no resolution (corresponding to climatology).

The diagram on the right-hand side represents the sharpness of two different forecast systems. The horizontal axis is the forecast probabilities while the vertical axis is the relative frequencies or the number of cases in each forecast probability bin. A U-shape distribution would indicate a sharper forecast system. A forecast system with little sharpness will have a distribution with a bell-shape. Note that sharpness does not involve the observations. Sharpness is an attribute of forecasts alone.

## Attributes of probabilistic forecasts

- Forecast pdf is narrower than that of climatology.
- Hence, forecast is sharper than climatology.

- Forecast pdf

- Climatology pdf

- Verifying observation

- Courtesy S. Beauregard, CMC

**Notes:**
Sharpness can be measured with the width of the forecast probability density function (pdf). A pdf that has a variance or standard deviation smaller than that of climatology will be sharper than climatology. 

## Attributes of probabilistic forecasts

- Reliability

- Sharpness

- Courtesty J. Montpetit, CMC

**Notes:**
The left-hand side diagram gives the reliability curves for the UMOS 12-h probability of precipitation (PoP), at the ensemble of stations across Canada during Summer 2006 (June-July-August). The 0-12 hour forecasts are in blue and the 36-48 hour forecasts in red. The UMOS PoP forecasts are very reliable for probabilities below 50%. The 0-12 hour forecasts are slightly under-confident for probabilities above 50% while the 36-48 hour forecasts are slightly over-confident. The right-hand side diagram gives the sharpness (frequency distributions of the forecasts). The 0-12 hour forecasts (in blue) are sharper than the 36-48 hour ones (in red). One of the characteristics of UMOS is to maintain the reliability of the forecasts at the expense of sharpness. Note that perfect forecasts would only be in the two extreme lower and upper bins (deterministic). Furthermore, a U-shape frequency distribution is a better characteristic of sharp forecasts. Note also that the observations are not involved in the display of forecast sharpness. Sharpness is an internal attribute of forecasts alone.

## Attributes of probabilistic forecasts

- Reliability

- After R. Verret, CMC

**Notes:**
This attribute diagram shows the reliability term of the Brier score for a theoretical forecast system. This term measures the difference between the forecast probability bin and its corresponding observed frequency of the event. The smaller the reliability term is, the smaller the Brier score is. Remember that the Brier score is negatively oriented: the smaller the score is, the better the forecasts are. Perfect reliability implies that the reliability term of the Brier score will be 0. Essentially, the reliability term measures the size of the area in blue on the attribute diagram or how much the reliability curve departs from the diagonal. Climatology as a forecast will have perfect reliability.

## Attributes of probabilistic forecasts

- Resolution

- R. Verret, CMC

**Notes:**
This attribute diagram shows the resolution term of the Brier score. This term measures the difference between the observed event frequencies corresponding to each forecast probability bin and the over-all frequency of the event (or climatology). In other words, it measures the difference between the reliability curve and the horizontal line corresponding to climatological frequency of the event. Since there is a negative sign in front of the resolution term, the larger the resolution term is the lower the Brier score will be. Remember that the Brier score is negatively oriented. Resolution is the complement of reliability. Climatology as forecast will have no resolution (resolution term = 0).

## A PDF with no resolution

- Cold forecast and verification PDFs

- Warm forecast and verification
- PDFs

**Cold/warm forecasts do not resolve warm**
**and cold observations/verifications! Get back climatology for verification in both cases!**

## Attributes of probabilistic forecasts

- What does these reliability curves mean to you?

- Forecast system is generally under-confident

- Relatively rare events. Forecast system is very reliable up to 60% but never forecasts probabilities above 60%.

- Adapted from L. Wilson, RPN

**Notes:**
What can you tell about these reliability curves ? 

The reliability curve on the left-hand side indicates that the forecast system is under-confident. 

The attribute diagram on the right-hand side indicates that the forecast system is very reliable for forecast probabilities between 0 and 60%, but the system never forecast probability values above 60%.  Relatively rare events might look like this.

## Attributes of probabilistic forecasts

- What does these reliability curves mean to you?

- Forecast system is very unreliable and has no resolution. Skill score with respect to climatology will be negative.

- Forecast system appears unstable. Verification sample is likely too small.

- Adapted from L. Wilson, RPN

**Notes:**
What can you tell about these reliability curves ? 

On the left-hand side, the forecast system does not show either reliability or resolution. A skill score with respect to climatology such as the Brier Skill Score will be negative. 

The diagram on the right-hand side indicates that the forecast system is not stable. This may be due to the fact that the verification sample is not large enough to properly measure the reliability of the forecast system. 

## Attributes of probabilistic forecasts

- Reliability term:
- Measures closeness of forecast frequency values to observational frequency values. If PDFs overlap perfectly, reliability = 0.
- Can be statistically corrected based on verification samples assuming no climate trends.
- Has no intrinsic forecast value.
- Climate forecasts are perfectly reliable (if there is no climate trend) – reliability term = 0.
- Resolution  term:
- Reflects how well different forecast classes *F**k* can separate cases with subsequent observed events.
- Cannot be statistically corrected.
- Is an intrinsic property of the forecasts.
- Climate forecasts have no resolution – resolution term = 0.
- Reliability and resolution are two independent but complementary attributes.

**Notes:**
In summary, reliability and resolution are two terms of the Brier score. The reliability term measures the statistical consistency between each class of forecasts and the corresponding distributions of observations. Reliability can be corrected based on verification samples, assuming stationary processes. Climatology has perfect reliability. Resolution measures how well different forecast classes can separate cases with subsequent observed events. Resolution cannot be statistically corrected (forecasts do not appear explicitly in the resolution term). Climatology has no resolution. The uncertainty term does not depend on the forecasts at all. 

## Advantages of probabilistic forecasts

- Expression of uncertainty:
- If forecasts are reliable, probabilistic forecasts give the probabilities that the forecasts will actually verify against occurrences of the forecast events.
- PDF gives all the information.
- Probabilistic forecasts can be used in decision making processes and for risk management.
- Probabilistic forecasts have economic value (when used).

**Notes:**
Probabilistic forecasts express uncertainty. They may be seen as the probability of occurrence of an event or the probability that the forecasts will actually verify against the occurrence of that event. Probability density functions give complete information. Probabilistic forecasts can be used in decision making processes, which is not necessarily the case for deterministic forecasts. It can be shown that probabilistic forecasts have greater economic value than deterministic ones.

## What we’ve covered so far...

- There is uncertainty in the forecasts.
- No forecast is complete without an assessment of uncertainty.
- There are several interpretation of probabilities – the frequentist approach is widely used in meteorology.
- Probability is a measure and an expression of uncertainty.
- Probability density functions (pdf and/or cdf) provide all the information.
- Probability density functions have characteristic reliability and resolution, that can be used to determine their usefulness.

**Questions ?**

## Multimodal distributions

- Characteristics
- Frequent occurrence in meteorology
- What kind of phenomena might lead to clustering in near-surface variables like 2-m temperature?
- More than one grouping or “cluster” of likely forecast outcomes
- Ensemble mean and spread become less meaningful
- Relatively easy to (subjectively) see clusters in plume diagrams, spaghetti plots

- “warm” cluster

- “cool” cluster

## Clustering in larger scale as well...

- Chaotic systems with multiple preferred modes or “attractors”
- At right is simple “Lorenz model”
- Note at circle, high sensitivity to initial conditions can lead to two clusters of solutions in Regimes A and B
- Same can happen with short-waves and other synoptic scale features

- Ridge?

- Trough?

## Objective Clustering

- Separates data into groups whose identifying characteristics are not known in advance (useful because of groupings are likely flow-dependent)
- Uses concept of “distance” to determine how far away ensemble members are from each other
- Brings out groupings that may not be obvious
- Distance measure examples
- Root-mean-squared-difference (ie. Euclidean distance)
- Anomaly correlation (base grouping on distance from long-term climatology)
- Minimum variance (group together based on minimizing within-group variance)

## Summary

- NWP models are inherently uncertain
- Probabilistic NWP/ensembles make the best use of meteorology and computational power as they stand today
- Meteorology uses the “frequentist” point of view regarding probability
- Advantages
- Based on frequency of occurrence
- Relatively easy to understand, but.....
- Disadvantages
- Works best with large samples
- Not all events are alike (past will not perfectly reflect the present!)

## Summary

- Probability is a measurement of uncertainty surrounding a defined event
- A probability distribution function (PDF) can be described by measures of
- Central tendency (mean, median, mode)
- Dispersion (terciles, quartiles, etc., standard deviation)
- Shape (e.g. skewness)
- Theoretical probability distributions can be used to describe sample data
***If ***the sample fits the theoretical distribution closely enough
- When the physical characteristics of the process observed fit the theoretical distribution (e.g. unbounded, lower or upper bound, etc.)

## Summary

- Joint probabilities are used to describe the likelihood of concurrent events
- PDFs for forecast variables have attributes of
- Accuracy
- Skill (relative to some baseline like climatology)
- Reliability (forecast probabilities match observed probabilities)
- Resolution (forecast anomalies match up well in sign with observed anomalies)
- Sharpness (degree to which probability distribution tends to 0% or 100%, distinguishing between event and NO event occurrence)

## Summary

- Advantages of probabilistic forecasts
- PDF contains all the information needed
- Better fits our knowledge of how the atmosphere works
- PDFs can be used for decision-making and risk management
- Limitations
- Limited sample size
- Do we have enough data to *know* the probabilities?
- Mitigated somewhat if we understand physical processes
- Combining of events with different characteristics
- Relates to sample size problem