Introduction to Trend Filtering with Applications in Python | by Ian Ho | Towards Data Science

Introduction to Trend FilteringThe Minimisation Problem behind Trend Filtering Distinctions between H-P & L1 Trend FilteringApplied Trend Filtering for Singapore COE Premiums in Python
Introduction to Trend Filtering
The Minimisation Problem behind Trend Filtering Distinctions between H-P & L1 Trend Filtering
Applied Trend Filtering for Singapore COE Premiums in Python
Recently, I’ve been working on a project that involves some time-series modelling for quantitative investment strategies. Specifically, a key component of the strategy involves developing a differentiated investment approach for different regimes. Such an identification is useful because when the underlying dynamics of the financial market shift, the strategies that work well in one regime may not work well in another.
As a simple example, it is well known that asset class correlations tend to increase significantly when the market is in distress. Therefore, models that factor in covariance in asset allocation will have to change their assumptions accordingly, which will necessarily affect optimal portfolio allocations. In order to develop a differentiated strategy, one could simply identify two different regimes in the financial markets: expansions and contractions.
In order for this approach to be useful, regime-based investment strategies, especially medium to long-term ones, need to address two questions:
What are the relevant regimes to consider?How do we segment different time periods according to these regimes?
What are the relevant regimes to consider?
How do we segment different time periods according to these regimes?
One simple yet powerful regime classification would be to segment a time series into periods of increase and decline. In fact, Mulvey & Han (2016) apply this very approach in their paper Identifying Economic Regimes: Reducing Downside Risks for University Endowments & Foundations. By conducting Monte-Carlo simulations and identifying ‘growth’ and ‘contraction’ regimes on a quarterly basis, they are able to demonstrate the improved optimality of adjusting spending rules according to regime type, which primarily works through the active and dynamic management of downside risks.
However, financial time series tend to be extremely noisy, and in order to achieve robust quarter-to-quarter regime identification, smoothening is required. This is where trend filtering comes in.
The goal of trend filtering is to smooth out a time series by filtering out the ‘noise’. The trend filtering algorithm faces a trade-off between two objectives. Firstly, it wants to minimise the residual ‘noise’ between the actual and smooth series. Think of this as the algorithm wanting to stay as true as possible to the original series. Secondly, it wants to maximise the smoothness of the filtered series, which tends to be at odds with the first objective.
Mathematically, we aim to minimise the following objective function:
y is the actual time series, while x is the estimated filtered time series. The first part of the loss function represents the objective of minimising the sum of squared residuals between the actual and fitted series. The second part of the loss function represents the desire for smoothness. Dx captures the smoothness between every set of three points.
Finally λ is the regularisation parameter. Think of it as capturing the trade off between our two objectives of minimising residuals and maximising smoothness. We will see examples later on of how λ affects the filtered trend.
The model laid out so far describes the Hodrick-Prescott (H-P) version of trend filtering. Alternatively, we can adopt L1 trend filtering. The key difference is the penalty for smoothness, which is changed from L2 to L1 norm. For those familiar with regressions, this is analogous to the distinction between regularised Lasso and Ridge Regression. Like H-P filtering, L1 filtering also helps to smooth out a time series. The key difference in outcome is that L1 filtering produces piecewise-linear estimates of the underlying trend. This will become clear when we look at the applied example later on.
Mathematically, the objective function now becomes:
As the authors of L1 trend filtering explain:
The kinks, knots or changes in slope of the estimated trend can be interpreted as abrupt changes or events in the underlying dynamics of the time series.
Mulvey & Han actually applied the L1 trend filtering algorithm to smoothen their financial time series, which is arguably more suited for the context of regime identification and regime change between quarters.
To illustrate what trend filtering actually does, and to understand the distinction between L1 and H-P trend filtering, let’s consider the time series of COE Premiums in Singapore. One of my ongoing projects has been to understand the dynamics of COE Premiums in Singapore using multivariate LSTMs, and so I thought it would be interesting to use this time series to provide a simple introduction to trend filtering.
For the examples below, I have taken the log of premiums as the underlying time series.
Python Implementation
import pandas as pdimport numpy as npimport cvxpy import scipyimport cvxopt import matplotlib.pyplot as plty = pd.read_csv('coe_premium.csv')y = y['premium'].to_numpy()y = np.log(y)n = y.sizeones_row = np.ones((1, n))D = scipy.sparse.spdiags(np.vstack((ones_row, -2*ones_row, ones_row)), range(3), n-2, n)
For those unfamiliar with spdiags, refer to the documentation here for more details.
In the code above, I have imported the underlying time series and converted to logs. Then, I defined the D matrix that is used as part of constructing the loss function / objective function.
If you want a copy of the ‘coe_premium.csv’ file to try this out for yourself, feel free to get in touch with me for a copy, as it is actually an engineered and interpolated version of the public data which I used for training neural networks.
I define a list of λ to experiment with, so that we can see the effects of the parameter on the estimated filtered trend.
lambda_list = [0, 0.1, 0.5, 1, 2, 5, 10, 50, 200, 500, 1000, 2000, 5000, 10000, 100000]
Finally, I loop over the list of λ to first estimate the filtered trend, and plot the results side by side to see the effects of changing λ:
solver = cvxpy.CVXOPTreg_norm = 2fig, ax = plt.subplots(len(lambda_list)//3, 3, figsize=(20,20))ax = ax.ravel()ii = 0for lambda_value in lambda_list:    x = cvxpy.Variable(shape=n)     # x is the filtered trend that we initialize    objective = cvxpy.Minimize(0.5 * cvxpy.sum_squares(y-x)                   + lambda_value * cvxpy.norm(D@x, reg_norm))    # Note: D@x is syntax for matrix multiplication    problem = cvxpy.Problem(objective)    problem.solve(solver=solver, verbose=False)    ax[ii].plot(np.arange(1, n+1), y, , linewidth=1.0, c='b')    ax[ii].plot(np.arange(1, n+1), np.array(x.value), 'b-', linewidth=1.0, c='r')    ax[ii].set_xlabel('Time')    ax[ii].set_ylabel('Log Premium')    ax[ii].set_title('Lambda: {}\nSolver: {}\nObjective Value: {}'.format(lambda_value, prob.status, round(obj.value, 3)))    ii+=1    plt.tight_layout()plt.savefig('results/trend_filtering_L{}.png'.format(reg_norm))plt.show()
And that’s it! The cvxpy library does most of the optimisation heavy lifting for you so all you need are these few lines of code.
That being said, picking out the right parameters is more of an art than a science, and it really depends on what problem you are trying to solve. Thus, it’s often useful to search across different parameters to understand how the model output changes accordingly, and pick the regularisation best suited for your project’s purpose.
These are the visualisations for H-P trend filtering:
From the above visualisations, we can see that as the parameter λ increases from 0 to 100,000, the priority of the algorithm shifts from fitting the original time series well to obtaining a smoother version of the trend. When λ is at its maximum value, the algorithm focuses so overwhelmingly on smoothing the series that it just estimates a straight (smoothest) line for the entire series. On the other hand, when λ = 0, the algorithm disregards the smoothing entirely and gives us the original time series to minimise the residuals.
To get the L1 version of the above procedure, we just have to change the following:
solver = cvxpy.ECOSreg_norm = 1
For reasons beyond the scope of this article, the optimiser (variable solver) has to be changed because different optimisers work for different kinds of optimisation problems.
These are the results for L1 trend filtering:
When we compare the results of L1 trend filtering to H-P trend filtering, we notice three key results. Firstly, just like in H-P trend filtering, increasing λ increases the smoothness of the estimated trend. Secondly, unlike H-P trend filter, there are more distinct kinks, or sudden sharper changes, in the estimated time series. As mentioned before, this is due to us changing the penalty on smoothing. The math behind this result is beyond the scope of this discussion but can be found in L1 trend filtering for those who are interested. Thirdly, we notice that the algorithm arrives at the ‘straight-line’ estimated trend more quickly than in H-P trend filtering when we increase the value of λ.
I hope this gentle applied introduction to trend filtering has been useful for those interested in time series analysis and forecasting!