Probabilistic Forecasts: Pinball Loss Function
By: Nicolas Vandeput
Tags: Probability, Demand Forecasting, Time Series Analysis, Supply Chain, Forecasting

Probabilistic Forecasts: Pinball Loss Function

Probabilistic forecasts are essential to optimize supply chains and inventory targets. How can we evaluate the quality of probabilistic forecasts? Nicolas Vandeput Aug 20·6 min read

Self-Assessment Quiz

Let’s start with a few questions. Read them first before going through the article. By the end of your reading, you should be able to answer them. (The answers are provided at the end as well as a Python implementation)

You want to forecast your product’s demand. Specifically, you want to predict a value for which the demand has an 80% probability of being under. What is the worst, over forecasting the actual demand or under forecasting it? You design a forecast model that reduces the absolute error (or MAE). Is your model aiming for the average demand or the median demand? You made a 95% quantile demand forecast, your forecast was 150, and the observed demand is 120. How would you assess the quality of your forecast? You sell high-margin products. What is the worst: overstocking or understocking them? When setting the safety stock target, should you aim for a high or a low demand quantile?

Quantile Forecasts

A quantile forecast is a probabilistic forecast aiming at a specific demand quantile (or percentile).

Definition: Quantile

The quantile α (α is a percentage, 0<α<1) of a random distribution is the value for which the probability for an occurrence of this distribution to be below this value is α.

P(x<=Quantile) = α

In other words, the quantile is the distribution’s cumulative distribution function evaluated at α.

Quantile α = F^{-1}(α)

In simple words, if you do an 80% quantile forecast of tomorrow’s weather, you would say, “There is an 80% probability that the temperature will be 20°C or lower.”

Side note: If you are used to inventory optimization and the usual safety stock formula Ss = z * σ * √(R+L) z is the gaussian alpha quantile z=Φ^(-1) (α). Somehow, you can see the safety stock formula as the α quantile forecast of the demand over the risk-horizon.

PS: Remember to include the review period in this formula.

Asymmetrical Penalties

How can we assess the accuracy of a quantile forecast? Let’s take another look at our above example: you forecast that tomorrow’s 80th temperature quantile is 20°C. The next day, the temperature is 16°C.

How accurate was your forecast?

Before discussing the math, let’s get an intuition on how we should evaluate (or penalize) a quantile forecast. Somehow, when we say, “We forecast the temperature to have 80% of chances to be below 20°C,” we expect the temperature to be lower than 20°C and would be surprised that the temperature would be 25°C.

In other words, this quantile forecast should get a higher penalty if it under forecasted the temperature than if it overpredicted it.

Moreover, as shown in the figure above, the penalty for over forecasting should be even higher for higher quantiles. For example, if your 95% temperature quantile forecast is 20°C, you would be very surprised if the actual temperature was 25°C.

Pinball Loss Function

Let’s formalize this forecasting penalty using the pinball loss function.

The pinball loss function L_α is computed for a quantile α, the quantile forecast f, and the demand d as

L_α (d,f) = (d-f) α if d≥f

(f-d)(1-α) if f>d

This loss function aims to provide a forecast with an α probability of under forecasting the demand and an (α-1) probability of over forecasting the demand.

Intuition and Examples

Let’s see how the pinball function works in practice. In the figure below, you can see an example with d=100 and α=90% (we want to forecast the demand 90th quantile) where the pinball loss function L_α is computed for different values of f.

As you can see, the pinball loss L_α is highly asymmetrical: it doesn’t increase at the same speed if you over forecast (low penalty) or under forecast (high penalty).

Let’s take an example: forecasting 20 units too much will result in a loss of 10%|f-d|=2 (that is to say that over forecasting is not much penalized). But, on the other side, forecasting 20 units too little will result in a loss of 90%|f-d|=18.

Basically, under forecasting is penalized by α|e| (where e is the forecast error), whereas over forecasting is penalized by (1-α)|e|.

Mean Absolute Error

In my forecasting KPIs article, I highlighted that optimizing the forecast mean absolute error (MAE) will ultimately aim at forecasting the median demand.

As you can see in the figure below, the 50% quantile pinball loss function corresponds to the regular absolute error — they are interchangeable. Keep in mind that, by definition, forecasting the 50% demand quantile is the same as predicting the demand median. This is another illustration that optimizing the MAE will result in a forecast aiming at the demand median (and not the demand average).

Self-Assessment Solutions

You want to forecast your product’s demand. Specifically, you want to predict a value for which the demand has an 80% probability of being under. What is the worst, over forecasting the actual demand or under forecasting it?

Underforecasting the demand is the worst. If you want to forecast the 80th quantile, you should aim for a value that is likely to be higher than the observed value.

2. You design a forecast model that reduces the absolute error (or MAE). Is your model aiming for the average demand or the median demand?

Optimizing the Mean Absolute Error will ultimately result in forecasting the expected median demand and not the expected mean demand.

3. You made a 95% quantile demand forecast, your forecast was 150, and the observed demand is 120. How would you assess the quality of your forecast?

You can compute the pinball loss of your quantile forecast. Your forecast has an absolute error of 30 units (=|150–120|), and it is an overprediction, so you pinball loss is 1.5 units (=30*0.05%).

4. You sell high-margin products. What is the worst: overstocking or understocking them? When setting the safety stock target, should you aim for a high or a low demand quantile?

You want to have a lot of stock of high-margin products, so understocking them is a bad decision. Henceforth, when setting the stock target you should aim for a high demand quantile.

🐍 Do It Yourself

You can easily implement the pinball loss function in Python in a single expression (with a few modifications compared to the original equation).