Package: treesmoothing
This above picture is from (https://www.blackandwhite.ie/mononeil/blurred-forest)Bayesian post-hoc regularization for random forestsMethod DescriptionRandom Forests are powerful ensemble learning algorithms widely used in various machine learning tasks. However, they have a tendency to overfit noisy or irrelevant features, which can result in decreased generalization performance. Post-hoc regularization techniques aim to mitigate this issue by modifying the structure of the learned ensemble after its training.Here, we propose Bayesian post-hoc regularization to leverage the reliable patterns captured by leaf nodes closer to the root, while potentially reducing the impact of more specific and potentially noisy leaf nodes deeper in the tree. This approach allows for a form of pruning that does not alter the general structure of the trees but rather adjusts the influence of leaf nodes based on their proximity to the root node. We have evaluated the performance of our method on various machine learning data sets. Our approach demonstrates competitive performance with the state-of-the-art methods and, in certain cases, surpasses them in terms of predictive accuracy and generalization.Code DescriptionAll classes inherit fromShrinkageEstimator, which extendssklearn.base.BaseEstimator.
Usage of these two classes is entirely analogous, and works just like any othersklearnestimator:__init__()parameters:base_estimator: the estimator around which we "wrap" hierarchical shrinkage. This should be a tree-based estimator:DecisionTreeClassifier,RandomForestClassifier, ... (analogous forRegressors)shrink_mode: 2 options:"hs": classical Hierarchical Shrinkage (from Agarwal et al. 2022)"beta": Bayesian post-hoc regularization (from Pfeifer 2023)lmb: lambda hyperparameteralpha: alpha hyperparameterbeta: beta hyperparameterrandom_state: random state for reproducibilityOther functions:fit(X, y),predict(X),predict_proba(X),score(X, y)work just like with any othersklearnestimator.UsageInstall the Python package treesmoothing via pippipinstalltreesmoothingand import the ShrinkageClassifier asfromtreesmoothingimportShrinkageClassifieror install locally import of main function from sourcepipinstall./treesmoothingfromtreesmootingimportShrinkageClassifierOther importsfromimodels.util.data_utilimportget_clean_datasetimportnumpyasnpfromsklearn.model_selectionimportcross_val_scoreimportmatplotlib.pyplotaspltfromsklearn.model_selectionimportGridSearchCVfromsklearn.treeimportDecisionTreeClassifier,DecisionTreeRegressorfromsklearn.ensembleimportRandomForestClassifierfromsklearn.model_selectionimporttrain_test_splitfromsklearn.metricsimportbalanced_accuracy_scorefromsklearn.metricsimportroc_auc_scoreimportsysExample data setclf_datasets=[("breast-cancer","breast_cancer","imodels")]# scoring#sc = "balanced_accuracy"sc="roc_auc"# number of treesntrees=10# Read in data setX,y,feature_names=get_clean_dataset('breast_cancer',data_source='imodels')# train-test splitX_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.20)scores={}scores["vanilla"]=[]scores["hs"]=[]scores["beta"]=[]Vanilla Random Forest# vanilla RF ##########################################print("Vanilla Mode")shrink_mode="vanilla"#######################################################clf=RandomForestClassifier(n_estimators=ntrees)clf.fit(X_train,y_train)ifsc=="balanced_accuracy":pred_vanilla=clf.predict(X_test)scores[shrink_mode].append(balanced_accuracy_score(y_test,pred_vanilla))ifsc=="roc_auc":pred_vanilla=clf.predict_proba(X_test)[:,1]scores[shrink_mode].append(roc_auc_score(y_test,pred_vanilla))Hierarchical Shrinkage from Agarwal et al. 2022# hs - Hierarchical Shrinkage #########################print("HS Mode")shrink_mode="hs"#######################################################param_grid={"lmb":[0.001,0.01,0.1,1,10,25,50,100,200],"shrink_mode":["hs"]}grid_search=GridSearchCV(ShrinkageClassifier(RandomForestClassifier(n_estimators=ntrees)),param_grid,cv=5,n_jobs=-1,scoring=sc)grid_search.fit(X_train,y_train)best_params=grid_search.best_params_print(best_params)clf=ShrinkageClassifier(RandomForestClassifier(n_estimators=ntrees),shrink_mode=shrink_mode,lmb=best_params.get('lmb'))clf.fit(X_train,y_train)ifsc=="balanced_accuracy":pred_hs=clf.predict(X_test)scores[shrink_mode].append(balanced_accuracy_score(y_test,pred_hs))ifsc=="roc_auc":pred_hs=clf.predict_proba(X_test)[:,1]scores[shrink_mode].append(roc_auc_score(y_test,pred_hs))Bayesian post-hoc regularization from Pfeifer 2023# beta - Bayesian post-hoc regularization #########################print("Beta Shrinkage")shrink_mode="beta"###################################################################param_grid={"alpha":[1500,1000,800,500,100,50,30,10,1],"beta":[1500,1000,800,500,100,50,30,10,1],"shrink_mode":["beta"]}grid_search=GridSearchCV(ShrinkageClassifier(RandomForestClassifier(n_estimators=ntrees)),param_grid,cv=5,n_jobs=-1,scoring=sc)grid_search.fit(X,y)best_params=grid_search.best_params_print(best_params)clf=ShrinkageClassifier(RandomForestClassifier(n_estimators=ntrees),shrink_mode=shrink_mode,alpha=best_params.get('alpha'),beta=best_params.get('beta'))clf.fit(X_train,y_train)ifsc=="balanced_accuracy":pred_beta=clf.predict(X_test)scores[shrink_mode].append(balanced_accuracy_score(y_test,pred_beta))ifsc=="roc_auc":pred_beta=clf.predict_proba(X_test)[:,1]scores[shrink_mode].append(roc_auc_score(y_test,pred_beta))Print the resultsprint(scores)AcknowledgementThe TreeSmoothing Python code was written by Bastian Pfeifer and Arne Gevaert.
It is based on the Hierarchical Shrinkage implementation within the Python package imodels (https://github.com/csinva/imodels).CitationIf you find the Bayesian post-hoc method useful please cite@misc{pfeifer2023bayesian,
      title={Bayesian post-hoc regularization of random forests}, 
      author={Bastian Pfeifer},
      year={2023},
      eprint={2306.03702},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}Bibtex References@inproceedings{agarwal2022hierarchical,
  title={Hierarchical Shrinkage: Improving the accuracy and interpretability of tree-based models.},
  author={Agarwal, Abhineet and Tan, Yan Shuo and Ronen, Omer and Singh, Chandan and Yu, Bin},
  booktitle={International Conference on Machine Learning},
  pages={111--135},
  year={2022},
  organization={PMLR}
}