A short tutorial on Fuzzy Time Series — Part III
By: Petrônio Silva
Tags: Probabilistic, Fuzzy Logic, Time Series Forecasting, Interval, Towards Data Science

Hi folks! It was a long time since I published the first and second parts of this tutorial. Meanwhile I had the opportunity to talk with many people that applied FTS methods on several distinct fields, and they helped me to improve pyFTS library by fixing some bugs and expanding its features and usability.

We have been dealing with uncertainty since the beginning of this tutorial. We found ways to model and describe the temporal behavior of the time series using fuzzy sets, but we did not take into account the uncertainty of our forecasts. The forecasting uncertainty is like a fog: if you are looking closer you will see blurred and it may not affect you too much, but if you want to look more ahead…

So, once again it is time to go deeper and explore more forecasting types and also new methods, this time I will focus on non-stationary time series, concept drifts, etc.

All examples of this tutorial are available on Google Colab, on http://bit.ly/short_tutorial_colab3 . Feel free to get in touch, give feedback and ask questions. Now, let’s move!

What is there behind a point forecast?

The greatest majority of the forecasting methods produce point forecasts, i.e., a single scalar forecast. Given a numeric time series Y and its individual values y(t) ∈ Y, a point forecast is usually a conditional expectation ŷ(t+1) = 𝔼[Y |y(t), …]. This number represents the best guess for Y at time t+1 given what we know about Y, its past values t, t-1, … Regardless how good our estimator is, it is still a guess and not an exact certain value. In other words, behind that number there is uncertainty hidden in it.

The problem with the point forecasting is exactly this: it does not inform the uncertainty around it. A good starting point is the conditional variance Var[ y(t+1) | y(t) ], but it is not usually informed or even known.

There are ways to deal with this limitation of the point forecasts, and they are the interval and probabilistic forecasts.

Interval Forecasts

A prediction interval 𝕀 is a tuple 𝕀 = [u, l], where u is the upper uncertainty bound and l is the lower uncertainty bound, asserting that the estimate will probably fall inside this range, ŷ(t+1) ∈ [u, l].

A confidence interval is a value σ associated with a confidence level α ∈ [0,1], where ŷ(t+1) ∈ [ ŷ(t+1)-σ, ŷ(t+1)+σ] with a probability α. The confidence intervals are usually parametric, i.e., they assume that the uncertainty around the predicted value ŷ(t+1) is described by a Gaussian pdf.

There are several ways to measure the accuracy of an interval forecast and the simplest one is the coverage (that indicates the percentage of true values that fell inside the interval), calibration, sharpness, etc. The Winkler Score is known to embody several aspects of the intervals in one measure

S(α,x,𝕀) = (u−l) +2α(l−x)𝟙(x< l) +2α(x−u)𝟙(x>u)

Where 𝕀 = [u, l], x is the true value, α is the confidence level and 𝟙(condition) = {1 if condition is true, 0 otherwise }.

Probabilistic Forecasts

A probabilistic forecast is a probability distribution P: U →[0, 1], where U is the Universe of Discourse of Y, constrained to ∑P(x) = 1, ∀x ∈ U. It means that we will estimate, for each possible value x of U, a probability π that ŷ(t+1) = x, or in other words: P(ŷ(t+1) = x) = π, 0 ≤ π ≤1 This is the most complete way to represent the uncertainty of a forecast, and give to us the landscape of the possible values that ŷ(t+1) can assume with its degrees of probability.

With the probability distribution P we also can calculate α confidence intervals (using α and 1-α quantiles of the cumulative distribution F) and produce a point forecast using the expected value 𝔼[P] = ∑xP(x), ∀x ∈ U.

The main accuracy measure for probabilistic forecasts is the Continuous Rank Probability Score (CRPS). In simple terms, the CRPS measures the area of the forecast probability curve before and after the true value x. This measurement informs how close to the true value the probability distribution was. The CRPS is a generalization of MAE to the case of probabilistic forecasts, then lower values of CRPS are better.

CRPS(F, x) = ∫ (F(y) -𝟙(y>x))² dy , y = -∞ ..+∞

Where F(x) = P(X ≤ x) is the cumulative distribution function and x is the true value.

Which FTS methods should I use?

First of all, on pyFTS library every model has boolean flags indicating which type of forecasting it is able to make. All models are point forecasting-enabled but not all of them are interval and probabilistic enabled. If you try to use one of these kinds of forecasting on a model without this skill an exception will be raised.

if model.has_interval_forecasting:

forecasts = model.predict(data, type=’interval’)

To generate interval forecasts in pyFTS you use the same method for the scalar forecasts — the method predict in the FTS class — and inform the parameter type=’interval’. The interval forecasting type on predict method generates a list of tuples, each tuple with two values corresponding to a prediction interval 𝕀 = [u, l]. Some models (such as EnsembleFTS and PWFTS) allow the specification of the method of interval.

ifts.IntervalFTS (IFTS): The most basic method for generating prediction intervals, it is an extension of the HighOrderFTS. The generated prediction intervals do not have some probabilistic mean, they just measure the upper and lower bounds of the fuzzy sets that were involved on forecasting process, i.e., the fuzzy uncertainty. The method is described here.

(IFTS): The most basic method for generating prediction intervals, it is an extension of the HighOrderFTS. The generated prediction intervals do not have some probabilistic mean, they just measure the upper and lower bounds of the fuzzy sets that were involved on forecasting process, i.e., the fuzzy uncertainty. The method is described here. ensemble.AllMethodEnsembleFTS: The EnsembleFTS is a meta model composed by several base models. The AllMethodEnsembleFTS creates one instance of each monovariate FTS method implemented in pyFTS and set them as its base models. The forecasting is computed from the forecasts of the base models. A brief description of the method can be found here. There is basically two ways to compute prediction intervals in EnsembleFTS: extremum and quantile (default). In extremum method the maximum and minimum values between the forecasts of the base models are chosen. On quantile method the alpha parameter must be informed and then the forecasts of the base models will be ordered and the quantile interval will be extracted.

from pyFTS.models.ensemble import ensemble part = Grid.GridPartitioner(data=train, npart=11) model = ensemble.AllMethodEnsembleFTS(partitioner=part) forecasts = model.predict(test, type='interval', mode='extremum') forecasts2 = model.predict(test, type='interval', mode='quantile', alpha=.1)

pwfts.ProbabilisticWeightedFTS (PWFTS): As its name says, this is the most complex method and still under review (on its way to be published). There is basically two ways to produce prediction intervals on PWFTS: heuristic (default) and quantile. In the heuristic method the interval bounds are calculated as the expected value of the fuzzy sets bounds and its empirical probabilities and the quantile method generates a full probability distribution and then extracts the quantiles (using the alpha parameter).

forecasts1 = model.predict(test, type='interval', method='heuristic') forecasts2 = model.predict(test, type='interval', method='quantile', alpha=.05)

multivariate.mvfts.MVFTS : This multivariate method uses the same approach of IFTS to produce prediction intervals.

: This multivariate method uses the same approach of IFTS to produce prediction intervals. multivariate.wmvfts.WeightedMVFTS: This weighted multivariate method uses the same approach of IFTS to produce prediction intervals.

In the module pyFTS.common.Util we can find the function plot_interval which allows us to easily draw the intervals:

Intervals generated by the monovariate methods (source)

Intervals generated by the multivariate methods (source)

The generated intervals try to demonstrate the range of possible variations that the model takes into account. You can see that some models generate wider intervals than others and sometimes (especially on the weighted models that have the thinner intervals) the original values fall outside the interval. The best intervals have balanced widths, neither too wide to show high uncertainty and neither too thin to not cover the real values.

To generate probabilistic forecasts in pyFTS you also use the method predict in the FTS class and inform the parameter type=’distribution’. The predict method will return a list of ProbabilityDistribution objects. Also, like the intervals the probabilistic forecasting has its own boolean flag to indicate which models are enabled to perform it:

if model.has_probabilistic_forecasting:

distributions = model.predict(test, type='distribution')

In contrast to interval forecasting, the probabilistic forecasting has its own class to represent a probability distribution — the class pyFTS.probability.ProbabilityDistribution. There are several ways to represent this probability distribution that is, by definition, a discrete probability distribution. Some methods of this class have especial interest for us now: density (returns the probability of the input value(s)), cumulative (returns the cumulative probability of the input value(s)), quantile (returns the quantile value of the input value(s)) and plot (plots the probability distribution on the input matplotlib axis).

Now let’s take a look on some probabilistic forecasting enabled methods on pyFTS:

pwfts.ProbabilisticWeightedFTS (PWFTS) : This method was entirely designed for probabilistic forecasting and is the best method for this on pyFTS. Its rules contains empirical probabilities associated with the fuzzy sets and also present a specific defuzzyfication rule that transforms an input (crisp) value on a probability distribution to the future value;

: This method was entirely designed for probabilistic forecasting and is the best method for this on pyFTS. Its rules contains empirical probabilities associated with the fuzzy sets and also present a specific defuzzyfication rule that transforms an input (crisp) value on a probability distribution to the future value; ensemble.EnsembleFTS: The before mentioned ensemble creates probabilistic forecastings using Kernel Density Estimators (KDE) over the point forecasts of the base models. The KDE also requires the specification of the kernel function and the width parameter.

Let’s see how the probability forecasting looks like:

The probabilistic forecasting of 4 days using the Util.plot_density function (source)

The probabilistic forecasting of 24 hours using ProbabilityDistribution.plot function (source)

In above pictures the probabilistic forecasting is shown in two different perspectives. The first picture is generated with the method plot_density in the module common.Util, where each probability distribution is plotted as a shade of blue and its intensity corresponds to the probability. This method allows to plot the original time series and the forecast probability distributions on top of it. The second picture shows each probability distribution individually in relation to the universe of discourse using the method plot of the class ProbabilityDistribution.

Of course it is not everything! We have to consider the interval and probabilistic forecasting for many steps ahead, which we expect to tell us how the uncertainty evolve as the prediction horizon increases. Yes! It is fascinating but I still have many things to show, so I will let it as an exercise for you, ok?

Let’s walk now on a trickiest road…

The land of the Non-Stationarities

You may remember that old and universally known quote:

“the only certainty is that nothing is certain”.

Yes. Forecasting may be unfair because things change all the time and we have to deal with it. Stationarity means, in layman terms, that the statistical properties of a stochastic process (like their expected value and variance) do not change along the time, whose the ultimate mean is stability. This is awesome for the forecasting models: the test data set will behave exactly as the train data set!

In other hand, the non-stationarity means that the statistical properties change. But not all non-stationarities are created equal. Some of them are predictable as trend and seasonality. Dealing with seasonality is not tricky, you can use High Order, Seasonal or Multivariate methods (you may remember our last tutorial). To deal with trends it is not too complicated either, you can de-trend the data using a difference transformation.

The original time series — NASDAQ — and the differentiated time series (source)

Suppose that we split the above time series in half, and call these subsets training and testing data. You can see that the test subset (after the instance number 2000) have values that did not appear before, in the train subset. This is a drawback for most FTS methods: what happens when the input data fall outside the known Universe of Discourse? The model never saw that region before and doesn’t know how to proceed, than it fails tragically. You can also see on the above image that the differentiated time series is much more well behaved and, indeed, it is stationary.

How can we use the Difference transformation on pyFTS? Just import the Transformations module from pyFTS.common and instantiate it. Don’t forget to inform the transformation to the partitioning method and also add it to the model (with the method append_transformation).

from pyFTS.data import NASDAQ

from pyFTS.models import chen

from pyFTS.partitioners import Grid

from pyFTS.common import Transformations diff = Transformations.Differential(1) train = data[:2000]

test = data[2000:] part = Grid.GridPartitioner(data=train, npart=15, transformation=diff) model = chen.ConventionalFTS(partitioner=part)

model.append_transformation(diff)

model.fit(train) forecasts = model.predict(test)

Look the behavior of the classical Chen’s model with and without the Differential transformation for the NASDAQ dataset:

The degradation effect of the FTS when the test data falls out of the known Universe of Discourse (source)

While the time series is still fluctuating inside the known Universe of Discourse both models performed well. But when the time series jumped out the Universe of Discourse of the training data, the model without the Differentiate transformation started to deteriorate because it does not know what to do with that unknown data.

Then the transformations help us not only with trending patterns, but also with the unknown ranges of the universe of discourse.

But some non-stationarities are unpredictable, and sometimes they are painful to deal with.

The nightmare of the Concept-Drift

Concept drifts are unforeseen changes (on mean, variance or both) which can happen gradually or suddenly. Some times these drifts occur in cycles (with irregular periods) and, in other scenario, the drift is temporary. There are some questions to answer when a concept drifts happen: Is it temporary? Is the change finished (established) or it will keep changing?

We have also to make the distinction between concept-drift and outlier (or a blip). Outliers are not change, they belong to the known signal but are rare events.

Concept drifts are nightmares — not only FTS methods, other computational intelligence and statistical techniques fear them too — but we need to learn how to live together them. Despite the complexity of the problem there are some simple (somehow expensive unfortunately) techniques to tackle them.

Time Variant Methods

All FTS methods we saw before are time invariant methods, which means that they assume that the future fluctuations of the time series will behave according to patterns already happened before. In other words: the behavior of the time series, which was described by the fuzzy temporal rules of the model, will not change in the future.

This is works fine for many time series (for example the environmental seasonal time series like we studied before) but fails terribly in others (for instance the stock exchange asset prices).

In that cases we need to apply time variant models,

incremental.TimeVariant.Retrainer: The Time Variant model is the simplest (but efficient) approach to tackle concept drifts and non stationarity. This class implement a metamodel, what means that you can choose any FTS method to be its base method, then at each batch_size inputs the metamodel retrain its internal model with the last window_length inputs. These are the main parameters of the model: the window_length and the batch_size. As an meta model you can also specify which FTS method to use (the fts_method parameter) and which partitioner you want to use inside him (the partitioner_method and partitioner_params parameters).

from pyFTS.models.incremental import TimeVariant model = TimeVariant.Retrainer(

partitioner_method=Grid.GridPartitioner,

partitioner_params={'npart': 35},

fts_method=hofts.WeightedHighOrderFTS,

fts_params={},

order=2 ,

batch_size=50,

window_length=200)

incremental.IncrementalEnsembleFTS: Works similarly to the TimeVariant but in a EnsembleFTS approach. In TimeVariant there is only one internal model that is recreated after n inputs (which means that the batch_size is the unique memory it has). On IncrementalEnsemble we also have the window_length and the batch_size parameters but also the num_models that says how many internal models to hold. As soon new models are created (with the incoming data) the older ones are dropped from the ensemble.

from pyFTS.models.incremental import IncrementalEnsemble model = IncrementalEnsemble.IncrementalEnsembleFTS(

partitioner_method=Grid.GridPartitioner,

partitioner_params={'npart': 35},

fts_method=hofts.WeightedHighOrderFTS,

fts_params={},

order=2 ,

batch_size=50,

window_length=200,

num_models=3)

nonstationary.nsfts.NonStationaryFTS (NSFTS): The non-stationary fuzzy sets are fuzzy sets that can be modified along the time, allowing them to adapt to changes in the data by translating and/or scaling its parameters. The NSFTS method is very similitar to time invariant FTS methods, with the exception that their fuzzy sets are not static: for each forecast performed by an NSFTS model the error is calculated and stored and the fuzzy sets are changed to fix that error. For this method, the error is a measure of how much the test data is different from the train data. This method is on this way to be published.

from pyFTS.models.nonstationary import partitioners as nspart

from pyFTS.models.nonstationary import nsfts part = nspart.simplenonstationary_gridpartitioner_builder(

data=train,

npart=35, transformation=None) model3 = nsfts.NonStationaryFTS(partitioner=part)

The pyFTS.data module contains a lot of non-stationary and concept drifted time series as NASDAQ, TAIEX, S&P 500, Bitcoin, Ethereum, etc. You can also use the class data.artificial.SignalEmulator to create synthetic and complex patterns. The SignalEmulater is designed to work as an method chain / fluent interface, so you can simulate complex signals by chaining methods that produce specific signals that are added to previous one or replacing it. The method stationary_signal creates a simple stationary signal with constant mean and variance, the method incremental_gaussian creates a signal where the mean and/or variance is incremented at each time, the method periodic_gaussian fluctuate the mead and/or variance in constant periods and the blip method adds an outlier on a random location. Every time one of this methods is called its effects are added to the previous signal except if you inform the start parameter — indicating when (which iteration) the method start to work — or set the boolean parameter additive to False, making the stop the previous signal and start this new one. To render the whole signal you just need to call the function run.

from pyFTS.data import artificial signal = artificial.SignalEmulator()\

.stationary_gaussian(mu=2.5,sigma=.1,length=100,it=10)\

.incremental_gaussian(mu=0.02,sigma=0.01,length=500,start=500)\

.run()

Now let’s put it all together, create 3 non-stationary time series, with concept drifts and employ the above presented methods to forecast them:

Performance of the time variant methods for artificial time series with concept drifts (source)

Time Variant methods have to balance some kind of exploitation and exploration when dealing with non-stationarities and concept drifts. To exploit what the model already knows — its memory, the last learned patterns from the data — or to explore new data and learn new patterns. Each method has its own mechanisms: the Retrainer is controlled by the window_length and batch_size, the Incremental Ensemble for the both and the num_models, the NSFTS uses the magnitude of its own errors to adjust the fuzzy sets . After all, the time spent to adapt to concept drifts is one of the most important aspects of the time variant methods. The same principle we saw in previous tutorials apply in this one: each FTS method has its own features and parameters and the best method will depend on the context.

Well guys! That’s enough for today, ok?

In these tutorials we have covered — even superficially — a good portion of the time series forecasting field, with their problems and solutions using FTS methods. We did not finished yet! We will always have problems to solve, new improved methods and optimizations.

In next tutorials I will cover some new approaches, like hyperparameter optimization and how to tackle big time series with distributed computing. See you there guys!