Exponential Smoothing Approaches In Time Series Forecasting | by Arun Jagota | Towards Data Science

In this post, we describe and explain certain classical algorithms that forecast future values of time series. These algorithms, called exponential smoothers, have been in wide use for many decades. We convey intuition with examples, some augmented with Python code in an appendix
Exponential smoothers continue to remain attractive in the modern age of deep machine learning. They are simple and easy to implement. They are also surprisingly effective for short-term forecasting. Especially on rapidly-changing time series.
Exponential smoothers also lend themselves to incremental learning, often a must-have on rapidly-changing time series. This is so that the algorithm can keep up with the changing characteristics. Such as a trending time series suddenly starting to oscillate.
Basic Concepts
A time series is a sequence of values over time. Such as daily average temperature in a city, the daily closing price of a particular stock, monthly sales of iPhone 11.
Our interest here is in forecasting future values of a time series from its historical values. (In a broader formulation, which we won’t cover here, we might include additional predictors, even additional time series.)
This problem is of immense interest. Businesses want to forecast future sales. Traders would like to forecast stock or index prices when possible. We all want to forecast the weather.
Next, a key concept. The forecast horizon, a positive integer h, is the number of steps ahead we’d like to forecast. So for a daily time series, a forecast horizon of 1 would forecast the next day’s value, a forecast horizon of 7 would forecast its value 7 days into the future.
We start with the naive forecaster (NF). While it’s too naive to use, it serves as a useful baseline to compare against the various exponential smoothers.
The naive forecaster forecasts x(t+h) to be x(t). Note that, regardless of what h is, the forecast is always x(t).
Now, on to our first exponential smoother.
Simple Exponential Smoothing (SES)
Let’s model our time series as follows:
x(t) = f(t) + noise
Here f(t) is a deterministic function of t, and noise is independently generated at each time step by sampling from a suitable distribution, e.g. standard normal. This model is both rich and intuitively appealing. f(t) models the deterministic component of the time series. This can be pretty elaborate, including multiple trends and multiple seasonalities if we like. noise models random fluctuations and other unmodeled effects.
Below are some examples
x(t) = t + noisex(t) = log(t) + noisex(t) = t2 + noisex(t) = sin(t) + noise
The main idea in SES is to estimate f(t) from x(t), x(t-1), ..., and then use this estimate to forecast future values of x.
SES’s estimate of f(t) is the exponentially-weighted average of x(t), x(t-1), x(t-2), etc. x(t) contributes the most, x(t-1) the second most, x(t-2) the third most, and so on. The contributions decay exponentially, at a rate controlled by a decay parameter a.
This may be expressed recursively as
f^(t) = ax(t) + (1-a)f^(t-1)
The ^ is there to remind us that this is an estimate of f. Not f itself, which is hidden from us.
The forecast now is simply x^(t+h) = f^(t). NF is a special case of SES in which f^(t) = x(t).
For its simplicity, SES works surprisingly well for 1-step forecasts, i.e. h=1. The reason for this is that SES not only smooths out noise, but it also fits f(t) locally, i.e. at time t, which is important when the time series is changing rapidly. NF also fits f(t) locally, however, it does not smooth out noise. SES is ineffective for longer-term forecasts, i.e. h > 1.
Simple Exponential Smoothing + Trend (TrES)
As mentioned earlier, SES is ineffective for longer-term forecasts. By adding a trending component to SES, we can improve the situation. The intuition is simple. If there is a local trend at x(t), a trend component can estimate it, and use it to forecast for somewhat longer horizons. (This is similar to the reasoning: if you know an object’s current position and velocity, you can predict where it will be a little later.)
Here is a simple example. Consider x(t) = t + noise. Say somehow we figure out that x(t) is growing by 1 in every time unit (noise aside). Knowing this we can forecast that x’s value h time units later will be its current value plus h.
Okay, let’s describe the algorithm more thoroughly. For this, let’s reexpress our time series as
x(t) = f(t-1) + df(t-1) + noise
Here f(t’) is a function of t’ and df(t’) is the local trend at time t’. We have written it as df(t’) because it is the discrete analog of the first derivative of f at t’.
This model is equivalent to our first model. We’ve just factored f(t) as f(t-1) + df(t-1).
Below are some refactored examples.
x(t) = (t-1) + 1 + noisex(t) = (t-1)2 + 2t + noise
What’s the point of this factoring? We can estimate the two terms f(t-1) and df(t-1) separately. This allows us to make sensible longer term forecasts on series in which df(t) can be accurately estimated. Such as in x(t) = t + noise. Under the factoring x(t) = (t-1) + 1 + noise we see that df(t) equals 1. Using this estimate lets us make sensible forecasts further out into the future.
How do we estimate f(t-1) and df(t-1)? Both via exponential smoothing. f(t)’s estimate is the exponentially-weighted average of x(t), x(t-1), .... df(t)’s estimate is the exponentially-weighted average of x(t) - x(t-1), x(t-1) - x(t-2), x(t-2) - x(t-3), ...
Let’s call these estimates f^(t) and df^(t) respectively. The forecast is now
x^(t+h) = f^(t) + h*df^(t)
Note that f^(t) and df^(t) are both local estimates. So the algorithm adapts to changes in f(t) as well as df(t). We can use a and b as the exponential-decay parameters for estimating f and df respectively. This gives us more knobs to tweak. (Following the example below, we will discuss tuning these knobs a bit.)
Example
t           1   2    3    4    5     ...x           1   2    3    4    5     ...f^(a=1⁄2)     2   2.5  3.25 ... df^(b=1⁄2)    1   1    1    ...
f^(2) is 2 because we have initialized it to x(2). df^(2) is 1 because we have initialized it to x(2)-x(1). We see that the algorithm has deduced the trend that the series grows by 1 in every time step.
Let’s calculate x^(t+1) at time t=4. It is f^(4) + df^(4) = 3.25 + 1 = 4.25. Close to the actual value x(5) = 5 though lagging a bit. The lag is a consequence of the exponential smoothing. We could reduce the lag by weighing recency higher, but that may incur a cost elsewhere. We discuss trade-offs involving this in the next paragraph.
Next, let’s calculate x^(t+3) at time t = 4. It is f^(4) + 3*df^(4) = 3.25 + 3*1 = 6.25. Clearly we have been able to exploit the trend to forecast further into the future! x(7) is 7. The longer horizon forecast is as accurate as the 1-step one in this case! The lag hasn’t increased.
The smoothing parameters offer us ways to control the lags. In our example, increasing the value of a will reduce the lag. On the other hand, on a noisy version of our example, an overly large value of a will not be able to smooth out the noise. So yes, the lag will reduce, but not the forecast quality. It will be as if we are chasing the most recent value, moving with the noise in it.
Similar reasoning applies to the parameter b. Except that it applies to df, i.e. to the slope. In our example, the slope was always the same, 1. So changing b would not have any effect. However, if the slope was greater than 1 and had multiplicative noise in it, e.g. df was 5*noise where noise has a mean of 1 and fluctuates around it, changing b would likely have an impact on how much df^ lags df versus how much df^ chases the noise in df.
The way to go here is to auto-tune these parameters on suitable slices of the input data. That is, machine-learn them to minimize a suitable loss function. We won’t go into the details here.
Simple Exponential Smoothing + Trend + Seasonality (TrSeES)
SeES, while effective on series that exhibit locally linear trends, is not always effective on series with cyclic structure.
Consider the time series
1, 2, 3, 1, 2, 3, 1, 2, 3, ... 
SeES will predict x^(3) to be close to 4. In reality, x(3) is 1. That’s a big difference. If we somehow knew that the time series repeats itself every three time steps we could use this information to improve the forecast.
How can we improve the TrES? Let’s assume the series has a single seasonality, i.e. a single repeating component, and we know its order, i.e. k. We can model such a time series as follows.
x(t) = tr(t) + s(t) + noise
s(t) = s(t-k)
Here s(t) is a repeating time series. x(t) is obtained by adding s(t) to another time series which we are calling tr(t). We are calling it tr(t) because we think of it as modeling x(t)’s trend component. That said, in reality, tr(t) models whatever is left over after removing s(t) from x(t). It may yet have other repeating components. tr(t) is just less seasonable than x(t). As will become clear when we see the algorithm below, even removing a single seasonality will be progress.
Let’s see an example expressed this way.
t   1  2  3  4  5  6  7  8  9s   1  2  3  1  2  3  1  2  3tr  1  2  3  4  5  6  7  8  9x   2  4  6  5  7  9  8  10 12
How do we estimate tr(t) and s(t)? First, we assume we know the order k. One sensible approach is to obtain a new time series yk(t) = x(t) - x(t - k) with the influence of the seasonality removed. We can then apply TrES to forecast yk^(t+h). We then add x(t+h - k) to yk^(t+h) which gives us a sensible forecast x^(t+h) of x(t + h).
Let’s see how this plays out in the example above.
t   1  2  3  4  5  6  7  8  9x   2  4  6  5  7  9  8  10 12y3  3  3  3  3  3  3
Great, y3 is easy to forecast on. Its value is always 3. Removing the seasonality’s influence helped a lot! TrES will quickly figure out that y3 is constant.
Let’s forecast x(t+3=12) at t=9. First we forecast y3(12) to be 3. Then we add x(9) = 12 to this forecast. We get x^(12) = 3+12 = 15. This forecast is accurate unless the future changes course.
We can do this so long as h is less than or equal to k. (If h were greater than k, t+h - k would be greater than t, i.e. into the future!)
Smoothing The Seasonality
We can improve on TrSeES further. In deriving x^(t+h) from y^k(t+h) instead of adding x(t+h - k) we can add a smoothed version of it. Smoothed in what way? By taking the exponentially-weighted average of x(t+h - k), x(t+h - 2k), x(t+h - 3k), ...
Why do we think smoothing the seasonality improves TrSeES further? Consider the case when the seasonality component has multiplicative noise in it. I.e.,
s(t) = noise*s(t-k)
where, for illustration purposes, let’s say this noise is a Gaussian with a mean of 1 and a standard deviation greater than 0 but much less than 1. Such a seasonality model is realistic.
In this case, x(t+h - k), x(t+h - 2k), x(t+h - 3k), ... will also be influenced by such multiplicative noise. TrES will address the additive noise we previously had. But not this multiplicative noise. Using an exponentially-smoothed estimate of x(t+h - k) from x(t+h - k), x(t+h - 2k), x(t+h - 3k), ..., rather than x(t+h - k) will help alleviate the effect of this noise on the forecast.
Python Code
I included it here so people can see some actual code. That said, it’s incomplete in parts. There might also be bugs. So be prepared to do some additional work if you want to get it all working.
Time series generator
import numpy as npnoise = np.random.normalf = lambda t: <some function of t>x = [f(t) + noise for t in range(n)]
By choosing f suitably we can generate a variety of time series. Including ones with trends, ones with cycles, and ones with both.
Next, we show the code for the various exponential smoothers. As the first statement in all of them, add
x = [1,2,3,4,5,6,7]
Or, better, use the time series generator to generate x.
SES algorithm
<initialize x here>f = x[0]y = {}a = 0.5for t in range(len(x)):   xhat_tplus1 = f        # forecast next value   f = a*x[t] + (1-a)*f
TrES algorithm
<initialize x here>f,df    = x[1], x[1]-x[0]y       = {}h, a, b = 3, 0.5, 0.5for t in range(2,len(x)):   x_hat_tplush = f + h*df # forecast h steps ahead   f  = a*x[t] + (1-a)*f   df = b*(x[t]-x[t-1]) + (1-b)*df
TrSeES
This needs more work to get it going end-to-end. It also does not smoothen the seasonality.
<initialize x here>yk = [x[t] — x[t-k] for t in range(k,n)]<Run TrES on yk># Forecast x. forecast_y(t+h) uses the forecast model built earlier on yk using TrES.xhat_tplus_h = forecast_y(t+h) + x[t+h-k]
Further Reading
https://otexts.com/fpp2/ Chap 7 is on exponential smoothing.
https://otexts.com/fpp2/ Chap 7 is on exponential smoothing.