Package: rfcorr
rfcorr - Random Forest-based "Correlation" measuresThis library records an open research agenda related to alternative conceptions of correlation based on tree-based ensembles, i.e., "random forests."Author:Michael BommaritoProject Homepage:GitHubOriginal AnnouncementPyPIINSTALL$pipinstallrfcorrUSEimportrfcorr.random_forest# df = pandas.DataFrame of data with features/variables in columnsrfcorr.random_forest.get_pairwise_corr(df.values,num_trees=100,# number of trees in forest - bigger => tighter estimateslag=0,# lag feature-target variable => allows for asymmetric R(x,y) != R(y,x)method="regression",# estimate using regression or classification taskuse_permutation=True# permutation- or impurity-based importance estimates)WHY?Countless tasks rely on conceptions and formalizations of "correlation."  But in two decades of working in areas that utilize correlation
as key to their everyday operation, e.g., finance, I have found that few are measuring what their words reveal is their intuition.While others have offered alternative measures of correlation or dependence generally, this is my contribution - an approach based on tree-based ensembles that natively supports lagged correlation.Tree-based ensembles have a number of highly-favorable properties:Support for categorical features or mixed feature setsIntrinsic uncertainty estimation and intervals through ensemble constructionSupport for overfitting protectionHigh degree of flexibilityEstimation produces inferential models that can be used to make predictionsAdditionally, this library supports lagging inputs against targets in the supervised training process, enabling asymmetric correlation estimates
from the same data.There are, however, downsides:Slower estimation than other correlation methodsStochastic estimates (though this library supports fixing RNG)Question around interpretation of signedness or directionalityMore complex interpretation than linear correlation measuresScaling permutation-based estimatesEstimating covariance in asymmetric contextsFUNCTIONALITYRandom Forest (rfcorr.random_forest)get_corr_classification: Correlation from classification taskget_corr_regression: Correlation from regression taskget_corr: Convenience handler including lag support for (x, y)get_pairwise_corr: Convenience handler including lag support for full matrix XSupport for impurity-based or permutation-based importances (use_permutation=True)Extra Trees (rfcorr.extra_trees)get_corr_classification: Correlation from classification taskget_corr_regression: Correlation from regression taskget_corr: Convenience handler including lag support for (x, y)get_pairwise_corr: Convenience handler including lag support for full matrix XSupport for impurity-based or permutation-based importances (use_permutation=True)CatBoost (rfcorr.cat) (WIP)get_corr_classification: Correlation from classification taskget_corr_regression: Correlation from regression taskget_corr: Convenience handler including lag support for (x, y)get_pairwise_corr: Convenience handler including lag support for full matrix XSupport for GPU training and limited subset of catboost training parametersxgboost (rfcorr.xgboost) (WIP)get_corr_classification: Correlation from classification taskget_corr_regression: Correlation from regression taskget_corr: Convenience handler including lag support for (x, y)get_pairwise_corr: Convenience handler including lag support for full matrix XTODO: Histogram-based Gradient Boosting TreesTODO: Gradient-Boosting TreesTODO: Support exposing intervals (std, range) from permutation-based estimatesEXAMPLE USEThere are sample notebooks in thenotebooks/directory, including:notebooks/test_sector_etf.ipynb: "Correlation" and eigenvalue/spectral representations for SPDR Sector ETFs and SPYnotebooks/test_sector_etf_ts.ipynb: Rolling "Correlation" time series for SPDR Sector ETFs and SPYnotebooks/test_periodic_pathological.ipynb: Test of periodic (sin(x)) data with pathological results for Pearson/SpearmanSample usage looks like this:importnumpyimportpandasimportrfcorr.random_forest# create sample datax=numpy.arange(0,8*numpy.pi,0.1)y1=numpy.sqrt(x)y2=numpy.sin(x)# fix random state/RNGrs=numpy.random.RandomState(42)pandas.DataFrame(rfcorr.random_forest.get_pairwise_corr(df.values,num_trees=1000,lag=0,method="regression",use_permutation=True,random_state=rs),columns=["x","y1","y2"],index=["x","y1","y2"])"""x	y1	y2x	1.000000	1.919737	0.001276y1	1.965436	1.000000	0.003697y2	0.649579	0.628396	1.000000"""#NB: ~0 correlation for x~y2 and y1~y2# compare with pearsondf=pandas.DataFrame(zip(x,y1,y2),columns=["x","y1","y2"])df.corr(method="pearson")"""x	y1	y2x	1.000000	0.978639	-0.194091y1	0.978639	1.000000	-0.206973y2	-0.194091	-0.206973	1.000000"""# compare with spearmandf.corr(method="spearman")"""x	y1	y2x	1.000000	1.000000	-0.186751y1	1.000000	1.000000	-0.186751y2	-0.186751	-0.186751	1.000000"""HISTORY0.1.0, 2022-02-22: Initial PyPI release0.1.1, 2022-05-02: catboost support (available on GH as of 2022-03-28)0.1.2, 2022-05-02: xgboost supportLICENSEApache 2.0COLLABORATIONI'm currently working on a brief research note that should be on arxiv by March 2022.  I'd love to collaborate with anyone interested on the topic,
especially to bring broader perspective to backtesting, portfolio construction, and regime detection/timing applications.(back to top)