Package: target-statistic-encoding
target statistic encodingTable of contents:Installfrom pypifrom sourceWhat?Why?Benefits of this implementationHow?APIInstantiatefitprefer.fit_transformon your training setuse.transformon yournon-trainingsetCustom target statistic functionsGiven:Implement your own:Installfrom pypipip install target-statistic-encodingfrom sourcepython -m pip install git+https://github.com/CircArgs/target_statistic_encoding.gitWhat?There are many means to convert categorical features to numeric ones from one-hot to embeddings. Then there are target statistic methods. These methods take statistics based on the target feature.Why?Even within this simple technique there is variation in implementations. Some implement a time-mimicking approach such as Catboost to gain robustness over target leakage. However, one issue with this approach is that while it introduces some variation to the encoding, for a some samples the statistic is possibly excessively biased. This small package takes a different approach for this reason. Instead, it uses stratified folds of the training set and aggregates target statistics on each fold independently.Benefits of this implementationstratified split target statistic helps prevent target leakage thus making your models more robustcredibility factor allows categories with low support to be ignored additionally making your models more robustclean apivariety of target statistic functions in addition to allowing custom implemented oneseasy productionalization - everything is 100% serializable with pickle
ex.#save for prod/test time environmentpd.to_pickle(cat2num,"cat2num_for_production.pkl")#read into prod envcat2num=pd.read_pickle("cat2num_for_production.pkl")...model.predict(cat2num.transform(prod_data))How?This is just a simple utility library that performs the following sample operation:See this example notebookkeep in mind this is simply an example. The example target is random here so no real signal is expectedAPIInstantiateInitsignature:Cat2Num(cat_vars:List[str],target_var:str,stat_func:target_statistic_encoding.stat_funcs.stat_funcs._StatFunc=<functionmean.<locals>.stat_funcat0x7fea58a85950>,)Args:cat_vars(List[str]):alistofstringsrepresentingthecategoricalfeaturestobeencodedtarget_var(str):stringofthenameofthetargetfeaturein`data`stat_func(optionalFunction(*args,**kwargs)->Function({pd.Series,pd.DataFrameGroupBy})->{float,pd.Series})):functionwhichreturnsaclosuretoaggregatestatisticsovertarget-defaultstat_funcs.mean()fitprefer.fit_transformon your training setNote: running.fitfollowed by.transformon your training set is not equivalent to simply running.fit_transform. There wil be no differentiation amongst category statistics as they will all be mapped to the mean.cat2num.fit_transform(data:pandas.core.frame.DataFrame,split:str=None,n_splits:int=5,credibility:Union[float,int]=0,drop:bool=False,suffix:str='_Cat2Num',inplace:bool=False,)Args:data(pd.DataFrame):pandasdataframewithcategoricalfeaturestoconverttonumerictargetstatisticsplit(str):nameofacolumntouseinthedataforfoldingthedata.-ifthisisusethenn_splitsisignoredn_splits(int):numberofsplitstousefortargetstatisticcredibility(floatorint):-iffloatmustbein[0,1]as%offittingdataconsideredcredibletofitstatisticto-ifintmustbe>=0asnumberofrecordsinfittingdatalevelmustexistwithintobecredible-levelsnotabovethisthresholdwillbegiventheoveralltargetmeandrop(bool):droptheoriginalcolumnssuffix(str):astringtoappendtotheendofanencodedcolumn,default`'_Cat2Num'`inplace(bool):whetherthetransformationshouldbedoneinplaceorreturnthetransformeddata,default`False`Returns:thepasseddataframewithencodedcolumnsaddedifinplaceis`False`else`None`cat2num.fit(data:pandas.core.frame.DataFrame,credibility:Union[float,int]=0,)Args:data(pd.DataFrame):pandasdataframewithcategoricalfeaturestofitnumerictargetstatisticfromcredibility(floatorint):-iffloatmustbein[0,1]as%offittingdataconsideredcredibletofitstatisticto-ifintmustbe>=0asnumberofrecordsinfittingdatalevelmustexistwithintobecredible-levelsnotabovethisthresholdwillbegiventheoveralltargetmeanReturns:fitCat2Numinstanceuse.transformon yournon-trainingsetcat2num.transform(data:pandas.core.frame.DataFrame,drop:bool=False,suffix:str='_Cat2Num',inplace:bool=False,)Args:data(pd.DataFrame):pandasdataframewithcategoricalfeaturestoconverttonumerictargetstatisticdrop(bool):droptheoriginalcolumnssuffix(str):astringtoappendtotheendofanencodedcolumn,default`'_Cat2Num'`inplace(bool):whetherthetransformationshouldbedoneinplaceorreturnthetransformeddata,default`False`Returns:thepasseddataframewithencodedcolumnsaddedifinplaceis`False`else`None`Custom target statistic functionsYou may optionally opt for a target statistic based on a statistic other than the mean although this is usually unwanted/unnecessary.Several are included and you can implement your own with a few considerations.Given:mean (target_statistic_encoding.stat_funcs.Mean()) - the defaultmedian (target_statistic_encoding.stat_funcs.Median())std (target_statistic_encoding.stat_funcs.Std())var (target_statistic_encoding.stat_funcs.Var())quantile (target_statistic_encoding.stat_funcs.Quantile(quantile=0.5))Implement your own:You may optionally implement your own target statistic function. It must be a callable that operates on thepandas.core.groupby.DataFrameGroupbytype i.e. the result of apandas.DataFrame.groupbye.g.: something akin totargetX1a0.287356b0.298795c0.336879d0.287037