Financial Signal Processing — Part 1 | by Yao Lei Xu | Towards Data Science

Algorithmic trading is hard. You can’t just stick a dozen technical analysis signals into a neural network and expect to become a millionaire next month. This is because processing financial data is all about the details. Setting aside all the potential biases and mistakes that beginners can make, most financial machine learning and algo trading projects fail because they ignore the subtle assumptions within a financial context.
To that end, today I will go back and discuss the fundamentals of financial signal processing. I will examine various properties of financial data, when and how to process them in a particular way, and how to analyze them through machine learning techniques.
(You can find the full code and additional resources here)
Most machine learning techniques assume stationarity in the data. However, stationarity is rarely found in finance, which can cause several issues.
Before diving into financial signals, it’s important to mention the concept of stationarity, as it is the fundamental assumption behind most modern machine learning techniques.
Setting aside the rigorous mathematical definition, what stationarity implies is that the statistics (e.g. mean and variance) of the underlying signals are constant over time.
For instance, consider ML problems involving the classification of cats and dogs, where the underlying biology of the animals doesn’t change everyday with drastic differences. The stationarity of the biological features allows the ML models to pick up patterns that will remain true over time, which allows the models to generalize well out-of-sample.
In finance however, stationarity is a huge issue. Consider for instance the price signal of Apple in the figure below. It doesn’t take rigorous statistical tests to see that the average price does not remain constant over time.
Consider a train-test split framework with the above data, where the price signal before 2019 is used for training and the remaining for testing. If one were to naively scale the price range from [100, 400] to [0, 1], it would result in look-ahead bias (leakage of test set information to the training set). This would result in an unrealistically better performance, since the ML model implicitely knows what the max price will be in the future from the scaling.
Intuitively, if you were implementing this ML model back in 2019, there is no way you could have known that the price of AAPL would end up around 400 in 2020 (and if you knew, you wouldn’t need ML in the first place).
What if we scaled the price range using the training set? Scaling the price range from [100, 250] to [0,1] would result in no look-ahead bias, but all the prices greater than 250 in the test set would be scaled to be greater than 1. This is particularly problematic for ML models that need the features to be within a certain scaled range, such as neural networks.
A much better solution is to process the above signal through log-differencing, as discussed below.
Log-differencing the price data can generate stationary signals suitable for machine learning techniques, which is often better than raw price signals.
Log-differencing computes the difference between log-values at time t-1 and t which is also often referred to as log-returns, as shown below.
A price signal processed through log-returns have a number of advantages: (1) It renders the original signal stationary, as shown in the figure below. By a simple inspection, we can see that the new signal has a constant mean of around 0, as well as a more constant variance. (2) Log-returns tend to be more normally distributed than the price data, which is great for classical statistical models that rely on the normality assumption. (3) Log-returns are time-additive, which allows us to compute backtests through simple additions (see the article here for a more detailed explanation).
The major draw back of log-returns is that it removes the memory component, which destroys the pricing information. In most applications, this is usually not an issue. For instance, a lot of quantitative finance literature in portfolio optimization, momentum strategies, mean-reversion strategies, etc. rely on how much do asset prices change together. Hence the comparison of log-returns is more meaningful than the comparison of prices.
However, for other financial instruments, such as commodity futures, the exact price value might have real-life impact on industries that rely on those commodities. So in that case, the pricing information can be more meaningful than log-returns.
Finally, there can be some additional subtleties in the computation of log-returns, which can be adjusted through volume based sampling and fractional differencing. Those techniques will be discussed in a future post.
Rolling statistics of log-returns can be processed through machine learning methods, offering market insights.
As discussed previously, log-returns tend to follow a nicer distribution. For a short-term window, it is often desirable to analyze different statistics associated with that distribution, such as mean, standard deviation, skew, and kurtosis, which tend to be stationary as well.
In Python, these can be computed easily through the rolling function in pandas:
w = 22 # number of trading days in a months1 = rs.rolling(w).mean() # moving averages2 = rs.rolling(w).std() # moving standard deviations3 = rs.rolling(w).skew() # moving skewnesss4 = rs.rolling(w).kurt() # moving kurtosissignals = pd.concat([s1, s2, s3, s4], axis=1)signals.columns = ['mean', 'std dev', 'skew', 'kurtosis']signals.plot(subplots=True, figsize=(10,7), legend=True);
In particular, the rolling computation of the standard deviation can be extremely useful in analysing market volatility.
Volatility has a special role in financial machine learning. Intuitively, volatility measures how much uncertainty/movement/chaos is present in the market. In addition, it can be used to measure the amount of “risk” present in the market (although alternative/better definitions of risk exist, such as “exposure”).
The concept of volatility is also tightly linked to the idea of market regimes, which can be used to identify different market conditions. For instance, we could run a clustering algorithm on the volatility levels of the SP500 at different times:
from sklearn.mixture import GaussianMixturew = 22vol = rs.rolling(w).std()vol = vol.dropna()labels = GaussianMixture(2).fit_predict(vol.values.reshape(-1,1))prices = prices.reindex(vol.index)prices[labels==0].plot(style='bo', alpha=0.2)prices[labels==1].plot(style='ro', alpha=0.2)plt.title('Volatility Regimes SPY')
By fitting a Gaussian Mixture on the observed volatility levels, the ML model automatically inferred bullish and bearish markets regimes. This is intuitive since market sell-offs are often driven by panic selling decisions, which can lead to sharp declines that increases the chaos in the market, resulting in higher volatility. In contrast, bullish markets are characterized by a steadier increase in value, which leads to lower volatility.
The identification of market regimes is especially important for financial machine learning, since a model trained during a bullish market is unlikely to do well in a bearish one and vice-versa.
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
I hope you enjoyed the article! Follow me if you would like to see more content like this.
Also, check out my website for the full code and additional resources.