Package: eBoruta
eBoruta-- anextendedBorutaalgorithmIntroductionBoruta is an "all-relevant" feature selection algorithm initially suggested for Random Forests [1]. It's categorized as a "wrapper" method since it uses an ML model to filter for features relevant to the model's learning objective.To decide what's relevant, Boruta first creates a copy of the data and scrambles the values, thus breaking the connection to the response variable. Such "shadow" features are used as a reference point to separate relevant variables from irrelevant ones. Indeed, if a real feature behaves in the same way as would random noise, it doesn't contribute anything useful, and one should consider discarding it.Thus, Boruta is an iterative procedure. First, it trains the model on the dataset augmented by shadow features and computes feature importance values. It then checks the importance distribution for the shadow features and how real features relate to this distribution. If, given these data, Boruta can't tell the difference between real and shadow features, it assigns a "miss" to this feature in the current iteration. Otherwise, it assigns a "hit".Accumulating "hits" and "misses," Boruta uses statistical testing to decide whether their composition at the current number of steps could have occurred by chance. Thus, Boruta accepts or rejects a feature consistently yielding  "hits" or "misses," resp. However, Boruta is not always confident about accepting or rejecting, leaving some features "tentative," i.e., requiring further testing. So the loop continues until reaching some set number of steps or accepting/rejecting the whole feature pool.Why making this library?Indeed, two Python implementations already exist:BorutaPyBorutaShapThe first introduced modifications that improved the original algorithm, while the second allowed Boruta to use theSHAP importance[2]. Cool! We combine these features into a more user-friendly, transparent, and flexible interface.So what isextended?DespiteBorutaShapexisting for quite a while, not many people seem to have realized the benefit of using SHAP importance in Boruta. Namely, it makes the method model-agnostic. Indeed, the original publication likely picked RF since it's general purpose and provides an optimal balance between speed and accuracy. However, nothing in the algorithm itself would justify hard dependency on RF. WhileBorutaShapsupports other ensemble models such asXGBoostorLightGBM, we explicitly state its potential to be fully model-agnostic. Hence,eBorutasupports any model as long as the method to compute the feature importance is defined. Want to trySVMwith permutation importance? Go ahead! Neural networks? We've got you covered! To heck with it, why use a single model when it's known that there isn't a single one good for everything? WitheBoruta, you can (and probably should) try various approaches.Currently,InstallationeBorutais a pure Python package installable via pip:pipinstalleborutaUsageBelow we present the simplest use case corresponding to the default RF-based algorithm.fromeBorutaimporteBorutafromsklearn.datasetsimportmake_classificationx,y=make_classification()selector=eBoruta()selector.fit(x,y)However, you're encouraged to explore documentationhere, moreusage examples.DisclaimereBorutais still undergoing development. Not all the model types were tested. We encourage you to raise an issue if you find a bug, an example where it fails, or you want to propose a new feature/interface improvement/whatever.References[1] Miron B. Kursa and Witold R. Rudnicki. “Feature Selection with the Boruta Package”. In: Journal of Statistical Software 36.11 (2010), pp. 1–13.
[2] Scott Lundberg and Su-In Lee. “A Uniﬁed Approach to Interpreting Model Predictions”. In: arXiv (2017). eprint: 1705.07874.