Package: rgf-python
rgf_pythonThe wrapper of machine learning algorithmRegularized Greedy Forest (RGF)[1]for Python.FeaturesScikit-learn interface and possibility of usage for multiclass classification problem.rgf_pythoncontains bothoriginal RGFfrom the paper[1]andFastRGFimplementations.Note that FastRGF is developed to be used with large (and sparse) datasets, so on small datasets it often shows poorer performance compared to vanilla RGF.Original RGF implementations are available only for regression and binary classification, butrgf_pythonis also available formulticlass classificationby “One-vs-Rest” method.Examplesfromsklearnimportdatasetsfromsklearn.utils.validationimportcheck_random_statefromsklearn.model_selectionimportStratifiedKFold,cross_val_scorefromrgf.sklearnimportRGFClassifieriris=datasets.load_iris()rng=check_random_state(0)perm=rng.permutation(iris.target.size)iris.data=iris.data[perm]iris.target=iris.target[perm]rgf=RGFClassifier(max_leaf=400,algorithm="RGF_Sib",test_interval=100,verbose=True)n_folds=3rgf_scores=cross_val_score(rgf,iris.data,iris.target,cv=StratifiedKFold(n_folds))rgf_score=sum(rgf_scores)/n_foldsprint('RGF Classifier score:{0:.5f}'.format(rgf_score))More examples of using RGF estimators could be foundhere.Examples of using FastRGF estimators could be foundhere.Software RequirementsPython (>= 3.7)joblibscikit-learn (>= 0.18)InstallationFromPyPIusingpip:pip install rgf_pythonor fromGitHub:git clone https://github.com/RGF-team/rgf.git
cd rgf/python-package
python setup.py installMacOS users,rgf_pythonafter the3.10.0version is built withg++-11and cannot be launched on systems withg++-10and earlier. You should update yourg++compiler if you don’t want to build from sources or installrgf_python3.10.0from PyPI which is the last version built withg++-10.MacOS users,rgf_pythonafter the3.8.0version is built withg++-10and cannot be launched on systems withg++-9and earlier. You should update yourg++compiler if you don’t want to build from sources or installrgf_python3.8.0from PyPI which is the last version built withg++-9.MacOS users,rgf_pythonafter the3.5.0version is built withg++-9and cannot be launched on systems withg++-8and earlier. You should update yourg++compiler if you don’t want to build from sources or installrgf_python3.5.0from PyPI which is the last version built withg++-8.MacOS users,rgf_pythonafter the3.1.0version is built withg++-8and cannot be launched on systems withg++-7and earlier. You should update yourg++compiler if you don’t want to build from sources or installrgf_python3.1.0from PyPI which is the last version built withg++-7.If you have any problems while installing by methods listed above, you shouldbuild RGF and FastRGF executable files from binaries on your own and place compiled executable filesinto directory which is included in environmental variable‘PATH’or into directory with installed package. Alternatively, you may specify actual locations of executable files and directory for placing temp files by corresponding flags in configuration file.rgfrc, which you should create into your home directory. The default values are the following:rgf_location=$HOME/rgf($HOME/rgf.exefor Windows),fastrgf_location=$HOME,temp_location=tempfile.gettempdir()(here is more details abouttempfile.gettempdir()). Please take a look at the example of the.rgfrcfile:rgf_location=C:/Program Files/RGF/bin/rgf.exe
fastrgf_location=C:/Program Files/FastRGF/bin
temp_location=C:/Program Files/RGF/tempNote that whilergf_locationshould point to a concrete RGF executablefile,fastrgf_locationshould point to afolderin whichforest_train.exeandforest_predict.exeFastRGF executable files are located.Also, you may directly specify installation without automatic compilation:pip install rgf_python --install-option=--nocompilationorgit clone https://github.com/RGF-team/rgf.git
cd rgf/python-package
python setup.py install --nocompilationsudo(or administrator privileges in Windows) may be needed to perform installation commands.Detailed guides how you can build executable files of RGF and FastRGF from source files could be found in their foldershereandhererespectively.Docker imageWe providedocker imagewith installedrgf_python.# Run docker image
docker run -it rgfteam/rgf /bin/bash
# Run RGF example
python ./rgf/python-package/examples/RGF/comparison_RGF_and_RF_regressors_on_boston_dataset.py
# Run FastRGF example
python ./rgf/python-package/examples/FastRGF/FastRGF_classifier_on_iris_dataset.pyTuning HyperparametersRGFYou can tune hyperparameters as follows.max_leaf: Appropriate values are data-dependent and usually varied from 1000 to 10000.test_interval: For efficiency, it must be either multiple or divisor of 100 (default value of the optimization interval).algorithm: You can select “RGF”, “RGF Opt” or “RGF Sib”.loss: You can select “LS”, “Log”, “Expo” or “Abs”.reg_depth: Must be no smaller than 1. Meant for being used withalgorithm= “RGF Opt” or “RGF Sib”.l2: Either 1, 0.1, or 0.01 often produces good results though with exponential loss (loss= “Expo”) and logistic loss (loss= “Log”), some data requires smaller values such as 1e-10 or 1e-20.sl2: Default value is equal tol2. On some data,l2/100 works well.normalize: If turned on, training targets are normalized so that the average becomes zero.min_samples_leaf: Smaller values may slow down training. Too large values may degrade model accuracy.n_iter: Number of iterations of coordinate descent to optimize weights.n_tree_search: Number of trees to be searched for the nodes to split. The most recently grown trees are searched first.opt_interval: Weight optimization interval in terms of the number of leaf nodes.learning_rate: Step size of Newton updates used in coordinate descent to optimize weights.Detailed instruction of tuning hyperparameters ishere.FastRGFn_estimators: Typical range is [100, 10000], and a typical value is 1000.max_depth: Controls the tree depth.max_leaf: Controls the tree size.tree_gain_ratio: Controls when to start a new tree.min_samples_leaf: Controls the tree growth process.loss: You can select “LS”, “MODLS” or “LOGISTIC”.l1: Typical range is [0, 1000], and a large value induces sparsity.l2: Use a relatively large value such as 1000 or 10000. The larger value is, the largern_estimatorsyou need to use: the resulting accuracy is often better with a longer training time.opt_algorithm: You can select “rgf” or “epsilon-greedy”.learning_rate: Step size of epsilon-greedy boosting. Meant for being used withopt_algorithm= “epsilon-greedy”.max_bin: Typical range for dense data is [10, 65000] and for sparse data is [10, 250].min_child_weight: Controls the process of discretization (creating bins).data_l2: Controls the degree of L2 regularization for discretization (creating bins).sparse_max_features: Typical range is [1000, 10000000]. Meant for being used with sparse data.sparse_min_occurences: Controls which feature will be selected. Meant for being used with sparse data.Using at Kaggle KernelsKaggle Kernels supportrgf_python. Please seethis page.TroubleshootingIf you meet any error, please try to runtest_rgf_python.pyto confirm successful package installation.Then feel free toopen new issue.Known IssuesFastRGF crashes if training dataset is too small (#data < 28). (rgf#92)FastRGFClassifierandFastRGFRegressordo not provide any built-in method to calculate feature importances. (rgf#109)FAQQ: Temporary files use too much space on my hard drive (Kaggle Kernels disc space is exhausted while fittingrgf_pythonmodel).A: Please seergf#75.Q: GridSearchCV/RandomizedSearchCV/RFECV or other scikit-learn tool withn_jobsparameter hangs/freezes/crashes when runs withrgf_pythonestimator.A: This is a known general problem of multiprocessing in Python. You should setn_jobs=1parameter of either estimator or scikit-learn tool.Licensergf_pythonis distributed under theMIT license. Please read fileLICENSEfor more information.Many thanks to Rie Johnson and Tong Zhang (the authors of RGF).OtherShamelessly, some part of the implementation is based on the followingcode. Thanks!References[1] Rie Johnson and Tong Zhang.Learning Nonlinear Functions Using Regularized Greedy Forest.IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(5):942-954, May 2014