A forecast is consistent if the forecast probability is a true estimate of the forecaster s level of uncertainty. If forecasts are reliable , the forecaster s confidence is appropriate. If forecasts are consistent and reliable , the probability that the event will occur is the same as the forecast probability. Reliability and Attributes Diagrams For all forecasts of a given confidence, identify how often the event occurs. Probabilities should be sharp .
Reliability is a function of forecast accuracy .
Assuming the forecasts are reliable , sharpness is a function of predictability. Measures the mean-squared error of probability forecasts. The same as the Brier score, but for multiple categories. Verification measures for continuous probabilistic forecasts are experimental there are very few attempts to estimate the full probability distribution of possible outcomes. A miss may have a different size of impact than a false alarm .