Package: linear-tree
linear-treeA python library to build Model Trees with Linear Models at the leaves.linear-tree provides also the implementations ofLinearForestandLinearBoostinspired fromthese works.OverviewLinear Treescombine the learning ability of Decision Tree with the predictive and explicative power of Linear Models.
Like in tree-based algorithms, the data are split according to simple decision rules. The goodness of slits is evaluated in gain terms fitting Linear Models in the nodes. This implies that the models in the leaves are linear instead of constant approximations like in classical Decision Trees.Linear Forestsgeneralize the well known Random Forests by combining Linear Models with the same Random Forests. The key idea is to use the strength of Linear Models to improve the nonparametric learning ability of tree-based algorithms. Firstly, a Linear Model is fitted on the whole dataset, then a Random Forest is trained on the same dataset but using the residuals of the previous steps as target. The final predictions are the sum of the raw linear predictions and the residuals modeled by the Random Forest.Linear Boostingis a two stage learning process. Firstly, a linear model is trained on the initial dataset to obtain predictions. Secondly, the residuals of the previous step are modeled with a decision tree using all the available features. The tree identifies the path leading to highest error (i.e. the worst leaf). The leaf contributing to the error the most is used to generate a new binary feature to be used in the first stage. The iterations continue until a certain stopping criterion is met.linear-tree is developed to be fully integrable with scikit-learn.LinearTreeRegressorandLinearTreeClassifierare provided as scikit-learnBaseEstimatorto build a decision tree using linear estimators.LinearForestRegressorandLinearForestClassifieruse theRandomForestfrom sklearn to model residuals.LinearBoostRegressorandLinearBoostClassifierare available also asTransformerMixinin order to be integrated, in any pipeline, also for  automated features engineering. All the models available insklearn.linear_modelcan be used as base learner.Installationpipinstall--upgradelinear-treeThe module depends on NumPy, SciPy and Scikit-Learn (>=0.24.2). Python 3.6 or above is supported.MediaLinear Tree: the perfect mix of Linear Model and Decision TreeModel Tree: handle Data Shifts mixing Linear Model and Decision TreeExplainable AI with Linear TreesImprove Linear Regression for Time Series ForecastingLinear Boosting with Automated Features EngineeringImprove Random Forest with Linear ModelsUsageLinear Tree Regressionfromsklearn.linear_modelimportLinearRegressionfromlineartreeimportLinearTreeRegressorfromsklearn.datasetsimportmake_regressionX,y=make_regression(n_samples=100,n_features=4,n_informative=2,n_targets=1,random_state=0,shuffle=False)regr=LinearTreeRegressor(base_estimator=LinearRegression())regr.fit(X,y)Linear Tree Classificationfromsklearn.linear_modelimportRidgeClassifierfromlineartreeimportLinearTreeClassifierfromsklearn.datasetsimportmake_classificationX,y=make_classification(n_samples=100,n_features=4,n_informative=2,n_redundant=0,random_state=0,shuffle=False)clf=LinearTreeClassifier(base_estimator=RidgeClassifier())clf.fit(X,y)Linear Forest Regressionfromsklearn.linear_modelimportLinearRegressionfromlineartreeimportLinearForestRegressorfromsklearn.datasetsimportmake_regressionX,y=make_regression(n_samples=100,n_features=4,n_informative=2,n_targets=1,random_state=0,shuffle=False)regr=LinearForestRegressor(base_estimator=LinearRegression())regr.fit(X,y)Linear Forest Classificationfromsklearn.linear_modelimportLinearRegressionfromlineartreeimportLinearForestClassifierfromsklearn.datasetsimportmake_classificationX,y=make_classification(n_samples=100,n_features=4,n_informative=2,n_redundant=0,random_state=0,shuffle=False)clf=LinearForestClassifier(base_estimator=LinearRegression())clf.fit(X,y)Linear Boosting Regressionfromsklearn.linear_modelimportLinearRegressionfromlineartreeimportLinearBoostRegressorfromsklearn.datasetsimportmake_regressionX,y=make_regression(n_samples=100,n_features=4,n_informative=2,n_targets=1,random_state=0,shuffle=False)regr=LinearBoostRegressor(base_estimator=LinearRegression())regr.fit(X,y)Linear Boosting Classificationfromsklearn.linear_modelimportRidgeClassifierfromlineartreeimportLinearBoostClassifierfromsklearn.datasetsimportmake_classificationX,y=make_classification(n_samples=100,n_features=4,n_informative=2,n_redundant=0,random_state=0,shuffle=False)clf=LinearBoostClassifier(base_estimator=RidgeClassifier())clf.fit(X,y)More examples in thenotebooks folder.Check theAPI Referenceto see the parameter configurations and the available methods.ExamplesShow the linear tree learning path:Linear Tree Regressor at work:Linear Tree Classifier at work:Extract and examine coefficients at the leaves:Impact of the features automatically generated with Linear Boosting:Comparing predictions of Linear Forest and Random Forest:ReferencesRegression-Enhanced Random Forests. Haozhe Zhang, Dan Nettleton, Zhengyuan Zhu.Explainable boosted linear regression for time series forecasting. Igor Ilic, Berk Gorgulu, Mucahit Cevik, Mustafa Gokce Baydogan.