Package: tuneta
TuneTAoptimizes technical indicators using adistance correlationmeasure to a user defined target feature such as next day return.  Indicator parameter(s) are selected using clustering techniques to avoid "peak" or "lucky" values.  The set of tuned indicators can be pruned by choosing the most correlated with the target while minimizing correlation with each other (based on user defined maximum correlation). TuneTA maintains its state to add all tuned indicators to multiple data sets (train, validation, test).FeaturesGiven financial prices (OHLCV) and a target feature such as return, TuneTA optimizes the parameter(s) of technical indicator(s) using distance correlation to the target feature. Distance correlation captures both linear and non-linear strength and provides significant benefit over the popular Pearson correlation.Optimal indicator parameters are selected in a multi-step clustering process to avoid values which are not consistent with neighboring values, providing a more robust parameter selection.Prune indicators with a maximum correlation to each other.  This is helpful for machine learning models which generally perform better with lower feature intercorrelation.Supports tuning indicator(s) for single or multiple equities.  Multiple equities can be combined into a market basket where indicator parameters are optimized across the entire basket of equities.Multiple time ranges (ie: short, medium and long)Supports pruning preexisting featuresPersists state to generate identical indicators on multiple datasets (train, validation, test)Parallel processing for technical indicator optimization as well as correlation pruningSupports technical indicators produced from the following packages:Pandas TATA-LibFinTACorrelation report of target and featuresEarly stoppingOverviewTuneTA simplifies the process of optimizing many technical indicators while avoiding "peak" values, and selecting the best indicators with minimal correlation between each other (optional). At a high level, TuneTA performs the following steps:For each indicator,Optunasearches for parameter(s) which maximize its correlation to a user defined target (for example, next day return).After the specified Optuna trials are complete, a 3-step KMeans clustering method is used to select the optimal parameter(s):Each trial is placed in its nearest neighbor cluster based on its distance correlation to the target.  The optimal number of clusters is determined using the elbow method.  The cluster with the highest average correlation is selected with respect to its membership.  In other words, a weighted score is used to select the cluster with highest correlation but also with the most trials.After the best correlation cluster is selected, the parameters of the trials within the cluster are also clustered. Again, the best cluster of indicator parameter(s) are selected with respect to its membership.Finally, the centered best trial is selected from the best parameter cluster.Optionally, the tuned indicators can be pruned by selecting the indicators with a maximum correlation to the all other indicators.Finally, TuneTA generates all optimized indicators.InstallationNote: Forcing re-installation of TA-Lib as last step to ensure it's compiled correctly with environment.pipinstall-Ugit+https://github.com/jmrichardson/tunetapipinstall--force-reinstall--no-cache-dir--no-depsTA-LibInstall the latest release:pipinstall-Utunetapipinstall--force-reinstall--no-cache-dir--no-depsTA-LibInstall using Colab:!wgethttp://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz!tar-xzvfta-lib-0.4.0-src.tar.gz%cdta-lib!./configure--prefix=/usr!make!makeinstall!pipinstallTa-Lib!pipinstall-Ugit+https://github.com/jmrichardson/tuneta!pipinstall-Ugit+https://github.com/DistrictDataLabs/yellowbrick.git!pipinstallnumpy==1.20.3!pipinstallnumba==0.54.1!pipinstallpandas==1.3.4!pipinstallscikit-learn==1.0.1ExamplesTune RSI IndicatorTune Multiple IndicatorsTune and Prune all IndicatorsTuneTA fit usageTune MarketPrune Existing FeaturesTune RSI IndicatorFor simplicity, lets optimize a single indicator:RSI IndicatorTwo time periods (short and long term): 4-30 and 31-180Maximum of 100 trials per time period to search for the best indicator parameterStop after 20 trials per time period without improvementThe following is a snippet of the complete example found in the examples directory:tt=TuneTA(n_jobs=4,verbose=True)tt.fit(X_train,y_train,indicators=['tta.RSI'],ranges=[(4,30),(31,180)],trials=100,early_stop=20,)Two studies are created for each time period with up to 100 trials to test different indicator length values.  The correlation values are displayed based on the trial parameter.  The best trial with its respective parameter value is saved for both time ranges.To view the correlation of both indicators to the target return as well as each other:tt.report(target_corr=True,features_corr=True)IndicatorCorrelationtoTarget:Correlation----------------------------------tta_RSI_timeperiod_190.23393tta_RSI_timeperiod_360.227434IndicatorCorrelationtoEachOther:tta_RSI_timeperiod_19tta_RSI_timeperiod_36-------------------------------------------------------------------tta_RSI_timeperiod_1900.93175tta_RSI_timeperiod_360.931750To generate both RSI indicators on a data set:features=tt.transform(X_train)tta_RSI_timeperiod_19tta_RSI_timeperiod_36Date2011-10-03NaNNaN2011-10-04NaNNaN2011-10-05NaNNaN2011-10-06NaNNaN2011-10-07NaNNaN.........2018-09-2562.17326160.7130512018-09-2659.18566659.3627312018-09-2761.02623860.2102352018-09-2861.09479360.2418062018-10-0163.38482461.305540Tune Multiple IndicatorsBuilding from the previous example, lets optimize a handful of indicators:Basket of indicators from 3 different packages (TA-Lib,Pandas-TA,FinTA)One time period: 4-60tt.fit(X_train,y_train,indicators=['pta.slope','pta.stoch','tta.MACD','tta.MOM','fta.SMA'],ranges=[(4,60)],trials=100,early_stop=20,)You can view how long it took to optimize each indicator:tt.fit_times()IndicatorTimes--------------------1pta.stoch23.560tta.MACD12.032pta.slope6.824fta.SMA6.423tta.MOM5.7Let's have a look at each indicator's distance correlation to target as well as each other:tt.report(target_corr=True,features_corr=True)IndicatorCorrelationtoTarget:Correlation----------------------------------------------------------------tta_MACD_fastperiod_43_slowperiod_4_signalperiod_520.236575pta_stoch_k_57_d_29_smooth_k_20.231091pta_slope_length_150.215603tta_MOM_timeperiod_150.215603fta_SMA_period_300.080596IndicatorCorrelationtoEachOther:tta_MACD_fastperiod_43_slowperiod_4_signalperiod_52pta_stoch_k_57_d_29_smooth_k_2pta_slope_length_15tta_MOM_timeperiod_15fta_SMA_period_30-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------tta_MACD_fastperiod_43_slowperiod_4_signalperiod_5200.8862650.7797940.7797940.2209pta_stoch_k_57_d_29_smooth_k_20.88626500.6783110.6783110.110129pta_slope_length_150.7797940.678311010.167069tta_MOM_timeperiod_150.7797940.678311100.167069fta_SMA_period_300.22090.1101290.1670690.1670690Notice above that both slope(15) and mom(15) are perfectly correlated in the intercorrelation report (indicated by value of 1) as well as having the same correlation to the target.  Initially, I thought this had to be a bug, but they are indeed identically correlated on a different scale (notice the same heat color coding):Lets remove correlated indicators with a maximum threshold of .85 for demonstration purposes. Based on the above correlation report, the two indicator pairs that have a correlation of greater than .85 are MACD/Stoch and Slope/Mom.  We can easily remove the worst correlated to the target of each pair (removes Stoch as MACD is more correlated to the target and either slope or mom can be removed as they are both identically correlated to the target).  Notice that all indicators now have an intercorrelation less than .85:tt.prune(max_inter_correlation=.85)IndicatorCorrelationtoTarget:Correlation----------------------------------------------------------------tta_MACD_fastperiod_43_slowperiod_4_signalperiod_520.236576pta_slope_length_150.215603fta_SMA_period_60.099375IndicatorCorrelationtoEachOther:tta_MACD_fastperiod_43_slowperiod_4_signalperiod_52pta_slope_length_15fta_SMA_period_6-----------------------------------------------------------------------------------------------------------------------------------------------tta_MACD_fastperiod_43_slowperiod_4_signalperiod_5200.7797940.252834pta_slope_length_150.77979400.188658fta_SMA_period_60.2528340.1886580BackendTkAggisinteractivebackend.Turninginteractivemodeon.As in the previous example, we can easily create features:features=tt.transform(X_train)Tune and Prune all IndicatorsBuilding from the previous examples, lets optimize all available indicators.  Note the addition of min_target_correlation which removes indicators below target correlation threshold:tt.fit(X_train,y_train,indicators=['all'],ranges=[(4,30)],trials=500,early_stop=100,min_target_correlation=.05,)As in the previous examples we can see the correlation to the target with the report function:tt.report(target_corr=True,features_corr=False)For brevity, only showing the top 10 of the many results:IndicatorCorrelationtoTarget:Correlation---------------------------------------------------------------------------------------pta_natr_length_4_scalar_270.253049tta_NATR_timeperiod_60.247999tta_MACD_fastperiod_3_slowperiod_29_signalperiod_250.240217pta_macd_fast_3_slow_29_signal_250.240217pta_pgo_length_260.239584pta_tsi_fast_28_slow_2_signal_25_scalar_150.238303pta_smi_fast_29_slow_2_signal_20_scalar_260.238294fta_TSI_long_3_short_29_signal_260.234654tta_RSI_timeperiod_190.23393pta_rsi_length_19_scalar_260.23393...Let's prune the indicators to have a maximum of .7 correlation with any of the other indicators:tt.prune(max_inter_correlation=.7)Show the correlation for both target and intercorrelation after prune:tt.report(target_corr=True,features_corr=True)Again, showing only top 10 rows of each for brevity (intercorrelation omitted as well):Correlation----------------------------------------------------------------pta_natr_length_4_scalar_270.253049tta_MACD_fastperiod_3_slowperiod_29_signalperiod_250.240217pta_pvol_0.199302pta_kc_length_3_scalar_270.193162fta_VZO_period_200.171986fta_DMI_period_40.148614pta_pvo_fast_27_slow_28_signal_29_scalar_150.14692pta_cfo_length_28_scalar_260.141013fta_IFT_RSI_rsi_period_28_wma_period_40.140977pta_stc_fast_18_slow_270.140789...Tune MarketTuneTA supports tuning indicators across a market of equities. Simply, index the input dataframe with the date and symbol similar to the following.  Notice the dataframe still contains OHLCV but is indexed by both date and symbol (see tune_market.py in examples folder):Use TuneTA in the same way as the previous examplesPrune Existing FeaturesIf you have preexisting features in your dataframe (regardless if you use TuneTA to create new ones), I've added a helper prune_df function to prune the all of the features based on intercorrelation.  This is helpful, for example, if you have custom features that you would like to combine with TuneTA and select only the features with maximum correlation with minimal intercorrelation.  The prune_df helper function takes a dataframe and returns the column names of the appropriate features to keep.  The column names can then be used to filter your datasets:# Features to keepfeature_names=tt.prune_df(X_train,y_train,min_target_correlation=.05,max_inter_correlation=.7,report=False)# Filter datasetsX_train=X_train[feature_names]X_test=X_test[feature_names]See prune_dataframe.py in the examples folderTuneTA fit usagett.fit(X, y, indicators, ranges, trials, early_stop)Parameters:indicators: List of indicators to optimize['all']: All indicators['pta']: All pandas-ta indicators['tta']: All ta-lib indicators['fta']: All fin-ta indicators['tta.RSI']: RSI indicator from ta-libSee config.py for available indicators and the parameters that are optimizedranges: Time periods to optimize[(2-30)]: Single time period (2 to 30 days)[(2-30, 31-90)]: Two time periods (short and long term)trials: Number of trials to search for optimal parametersearly_stop: Max number of trials without improvementmin_target_correlation: Minimum correlation to target required