Removing non-linear trends from timeseries data | by Florin Andrei | Towards Data Science

Sometimes trends need to be removed from timeseries data, in preparation for the next steps, or part of the data cleaning process. If you can identify a trend, then simply subtract it from the data, and the result is detrended data.
If the trend is linear, you can find it via linear regression. But what if the trend is not linear? We’ll see what we can do about that in a few moments.
But first, the simple case.
Here’s timeseries data with a trend:
https://raw.githubusercontent.com/FlorinAndrei/misc/master/qdata.csv
Let’s load it up and see what does it look like:
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom sklearn.linear_model import LinearRegressionfrom sklearn.preprocessing import PolynomialFeaturesfrom sklearn.metrics import mean_squared_error, r2_scoreser = pd.read_csv('qdata.csv', index_col=0, squeeze=True)serx0      473.9177641       75.3248252     -306.9694793       53.2714764      372.966686         ...     95    4650.55047396    4604.57334497    4891.70463898    5265.94816299    5618.909339Name: y, Length: 100, dtype: float64plt.plot(ser)plt.show()
Okay, there is a trend there. Let’s assume it’s linear — let’s do linear regression and find out. This is a direct application of linear regression. The sklearn library, imported above, has everything we need to do regression.
X = ser.indexX = np.reshape(X, (len(X), 1))y = ser.valuesmodel = LinearRegression()model.fit(X, y)trend = model.predict(X)plt.plot(y)plt.plot(trend)plt.legend(['data', 'trend'])plt.show()
Seems to fit, but perhaps not very well. Let’s subtract the trend from the data and see how the detrended data looks like:
detr = [y[i] - trend[i] for i in range(0, len(y))]plt.plot(detr)plt.title('data detrended in a linear fashion')plt.show()
Not very convincing. There’s still a concave trend left in the data. Probably the original trend was not linear to begin with.
Let’s compute the RMSE and the R2-score between the data and the trend we’ve extracted. We will come back to these numbers later.
r2 = r2_score(y, trend)rmse = np.sqrt(mean_squared_error(y, trend))print('r2:', r2)print('rmse', rmse)r2: 0.8782399672701933rmse 553.6078593008505
If the trend is not linear, we could try to fit it with a polynomial curve. But here’s the thing: even if the curve we’re fitting is a high degree polynomial, it’s still linear regression that will be used to find it. How come?
Well, consider this quadratic expression:
y = a + bx + cx2
The values we’re looking for are a, b, and c, and they are all linear (to the power of 1). Forget the powers of x, what we’re looking at is the powers of a, b, and c. Remember why a generalized linear model is called “linear”? Because the coefficients are linear — the x values could have different powers just fine.
So linear regression it is — it just happens that we will have to do linear regression in multiple dimensions at once.
Let’s assume our data has a quadratic trend. Then we need to transform X in a quadratic form:
pf = PolynomialFeatures(degree=2)Xp = pf.fit_transform(X)Xp
Which looks like this:
array([[1.000e+00, 0.000e+00, 0.000e+00],       [1.000e+00, 1.000e+00, 1.000e+00],       [1.000e+00, 2.000e+00, 4.000e+00],       [1.000e+00, 3.000e+00, 9.000e+00],       [1.000e+00, 4.000e+00, 1.600e+01],       [1.000e+00, 5.000e+00, 2.500e+01],       [1.000e+00, 6.000e+00, 3.600e+01],...       [1.000e+00, 9.600e+01, 9.216e+03],       [1.000e+00, 9.700e+01, 9.409e+03],       [1.000e+00, 9.800e+01, 9.604e+03],       [1.000e+00, 9.900e+01, 9.801e+03]])
The first column is X to the power of 0. The second column is X. The third column is X to the power of 2. This is just like the quadratic expression (y = a + bx + cx2) shown above.
And now we will use the quadratic form to fit the data and generate a quadratic trend. The parameters of the quadratic expression are found using linear regression.
md2 = LinearRegression()md2.fit(Xp, y)trendp = md2.predict(Xp)
What does the trend look like?
plt.plot(X, y)plt.plot(X, trendp)plt.legend(['data', 'polynomial trend'])plt.show()
Much closer fit, isn’t it? Now let’s see the detrended data:
detrpoly = [y[i] - trendp[i] for i in range(0, len(y))]plt.plot(X, detrpoly)plt.title('polynomially detrended data')plt.show()
This is clearly better. There’s no trend left there that could be discerned visually. But let’s see what the numbers say about it:
r2 = r2_score(y, trendp)rmse = np.sqrt(mean_squared_error(y, trendp))print('r2:', r2)print('rmse', rmse)r2: 0.9343217231542871rmse 406.5937924291518
Compared to the linear trend, with the polynomial trend R2 has increased and RMSE has decreased. Both are good changes. Both mean polynomial is a better fit than linear.
You can pick a polynomial of any order you like by simply assigning different values to N here:
pf = PolynomialFeatures(degree=N)
Generally speaking, use low values for N. If you increase N and not much happens, go back to the smaller value.
A curve with one bend can probably be described by a quadratic function. A curve with two bends might be described by a cubic function. And so on. N-1 bends require an expression to the power of N.
If you increase N a lot, eventually your “best fit” curve will start to follow the noise in the data, instead of fitting the trend. You’ve overfitted the curve, and now it’s meaningless. Either reduce N, or add more data points.