AI in Finance: advanced idea research and evaluation beyond backtests
By: Alexandr Honchar
Tags: Research, Machine Learning, Artificial Intelligence, Data Science, Finance

AI in Finance: advanced idea research and evaluation beyond backtests

Feature importance: a financial research driver that won’t let you down

How do we usually do trading ideas research? We get some data with the hope of having some predictive value there, prepare it, extract potential alphas, train forecasting models to predict prices based on that alphas and after run some long-short strategy backtest based on the trained model. If it looks good, we are happy to invest our money there. Looks pretty legit, right? Although, not everyone agrees with it.

More and more practitioners in financial machine learning are diving deeper than checking backtest Sharpe and maximal drawdown alongside other popular metrics

That backtesting process itself is extremely tricky, you can check it yourself in Dr. De Prado’s book or his lecture slides and you most probably already know about it from your experience. But there is a way to check ideas before actually doing backtesting — via analyzing ML models behind and their feature importance. Why it can be beneficial?

Investment management researchers are often doing research stating the hypotheses as “strategy based on features X and model Y will get Sharpe ratio Z”. However, research is about forming the hypothesis about the cause-effect relationship between financial variables. Backtesting is just one of the tools for validating the business value of the confirmed hypothesis.

about the between financial variables. Backtesting is just one of the tools for validating the business value of the confirmed hypothesis. Since classical backtesting is performed on the historical data, we tend to explain random patterns and correlations (that don’t explain causations) of the simulated trades in the past to justify our assumptions. Later, based on these “findings” we try to “fix” our models and strategies several times, which leads to the next problem.

and correlations (that don’t explain causations) of the simulated trades in the past to justify our assumptions. Later, based on these “findings” we try to “fix” our models and strategies several times, which leads to the next problem. Backtesting is prone to multiple hypothesis testing — the more you try to optimize a strategy based on the backtest on a single security, the more likely you’re diving deep into the false discovery. Hence, it’s very easy to “overfit” predictive models to perform well on the backtest, but that’s what we definitely don’t to put our capital on.

In this blog, we will see what are the methods to evaluate trading ideas without backtests on the examples of several data sources. We will see, that this is not just a tool for machine learning practitioners, but a strong framework for checking business hypotheses apart of the “trading” or portfolio management metrics. For practitioners, as always, the source code of all the experiments is on my GitHub.

Practical benefits

Apart from the benefits that are related to research and development (as mathematical validation and correctness of the ideas), there are several side outcomes that might be handy for financial institutions. While working with our customers in Neurons Lab, we have experienced the following scenarios where we could apply feature importance analysis for direct benefit:

If you’re constantly evaluating new data sources and market and alternative data research for inner use or for sale, correct feature importance is crucial to identify what data has real predictive value ;

; In live trading, selecting only important features for making trades can serve as one of the ways to reduce risks related to machine learning models performance decrease;

performance decrease; Tracking feature importance evolution in time, we can explain economical factors dynamics better than conventional econometric models, this also serves for risk management and early trend detection.

Feature Importance

In machine learning and mathematical modeling, no one likes the “black box” things. Even for complex deep learning models in NLP and computer vision, there are ways to visualize their performance and flaws: