Repository: LAMDA-NJU/Deep-Forest (language: python)

Issue:
Feature Requests
This issue collects all features requests. Any one is welcomed to work on issues listed below, and do not forget to include your contributions and name in the `CHANGELOG.rst`.

If you want to work on a requested feature, please re-open the linked issue, and leave a comment below to let us know that you want to work on it.

### New features
* [x] `CascadeForestRegressor` class for regression problem (#4)
* [ ] `export_graphviz` method on visualizing decision trees in deep forest (#12)
* [ ] Export decision tree in DF21 to [SHAP](https://shap.readthedocs.io/en/latest/index.html) (#12)
* [x] Internal label encoder on the class labels (#13)
* [x] Better support on the input data (#19, #20)
* [ ] GPU Support (#27)
* [x] Support customized estimator in the layers (#29)
* [ ] `CascadeForestSurvAnalyzer` class for survival analysis (#71)
* [ ] Customized evaluation metrics in cascade forest (#98)

### Python package
* [ ] Build wheels for Python 2.7
* [x] Set up CI on building wheels for Mac-OS (#6, #32)

### New language wrappers:
* [ ] C/C++ Interface (#9)

### Fix
* [ ] Model fix on extremely small datasets (#103)


Discussion / hints:
I will work on the #4 regressor task
> I will work on the #4 regressor task

That would be really nice @tczhao! Adding the regressor requires many efforts, can you open a draft pull request and upload what you have done there? I am willing to take part in the development on this feature request and have some deeper discussions there.

 In addition, here are some things that may be helpful to you:
* For regression, the augmented features are out-of-bag predicted values from the cascade layer, which is **unbounded** features (In contrast, the augmented features for classification are bounded, i.e., the class vectors). This poses some problems if we want to use binning for acceleration, because the unbounded feature values after binning will be very sensitive to the boundary values.
* Use the `RandomForestRegressor` and `ExtraTreeRegressor` from Scikit-Learn first.  This version only includes the reduced version on classification trees. I am willing to optimize that for regression trees after we have a quick verification on the effectiveness on regression.
I'm working on #13 
> > I will work on the #4 regressor task
> 
> That would be really nice @tczhao! Adding the regressor requires many efforts, can you open a draft pull request and upload what you have done there? I am willing to take part in the development on this feature request and have some deeper discussions there.
> 
> In addition, here are some things that may be helpful to you:
> 
> * For regression, the augmented features are out-of-bag predicted values from the cascade layer, which is **unbounded** features (In contrast, the augmented features for classification are bounded, i.e., the class vectors). This poses some problems if we want to use binning for acceleration, because the unbounded feature values after binning will be very sensitive to the boundary values.
> * Use the `RandomForestRegressor` and `ExtraTreeRegressor` from Scikit-Learn first.  This version only includes the reduced version on classification trees. I am willing to optimize that for regression trees after we have a quick verification on the effectiveness on regression.

Thanks, will have a draft ready in 2 days

Fix patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 1dc5c83..9d5e7f4 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -31,6 +31,7 @@ Version 0.1.*
 .. |Fix| replace:: :raw-html:`<span class="badge badge-danger">Fix</span>` :raw-latex:`{\small\sc [Fix]}`
 .. |API| replace:: :raw-html:`<span class="badge badge-warning">API Change</span>` :raw-latex:`{\small\sc [API Change]}`
 
+- |Feature| support regression prediction (`#25 <https://github.com/LAMDA-NJU/Deep-Forest/issues/25>`__) @tczhao
 - |Fix| fix accepted data types on the :obj:`binner` (`#23 <https://github.com/LAMDA-NJU/Deep-Forest/pull/23>`__) @xuyxu
 - |Feature| implement the :meth:`get_forest` method for efficient indexing (`#22 <https://github.com/LAMDA-NJU/Deep-Forest/pull/22>`__) @xuyxu
 - |Feature| support class label encoding (`#18 <https://github.com/LAMDA-NJU/Deep-Forest/pull/18>`__) @NiMaZi
diff --git a/README.rst b/README.rst
index 06e9122..9c065db 100644
--- a/README.rst
+++ b/README.rst
@@ -61,6 +61,23 @@ Quickstart
     print("\nTesting Accuracy: {:.3f} %".format(acc))
     >>> Testing Accuracy: 98.667 %
 
+.. code-block:: python
+
+    from sklearn.datasets import load_boston
+    from sklearn.model_selection import train_test_split
+    from sklearn.metrics import mean_squared_error
+
+    from deepforest import CascadeForestRegressor
+
+    X, y = load_boston(return_X_y=True)
+    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
+    model = CascadeForestRegressor(random_state=1)
+    model.fit(X_train, y_train)
+    y_pred = model.predict(X_test)
+    mse = mean_squared_error(y_test, y_pred)
+    print("\nTesting MSE: {:.3f}".format(mse))
+    >>> Testing MSE: 8.068
+
 Resources
 ---------
 
diff --git a/deepforest/__init__.py b/deepforest/__init__.py
index fd72c17..dcc150c 100644
--- a/deepforest/__init__.py
+++ b/deepforest/__init__.py
@@ -1,14 +1,19 @@
-from .cascade import CascadeForestClassifier
-from .forest import RandomForestClassifier
-from .forest import ExtraTreesClassifier
-from .tree import DecisionTreeClassifier
-from .tree import ExtraTreeClassifier
+from .cascade import CascadeForestClassifier, CascadeForestRegressor
+from .forest import RandomForestClassifier, RandomForestRegressor
+from .forest import ExtraTreesClassifier, ExtraTreesRegressor
+from .tree import DecisionTreeClassifier, DecisionTreeRegressor
+from .tree import ExtraTreeClassifier, ExtraTreeRegressor
 
 
 __all__ = [
     "CascadeForestClassifier",
+    "CascadeForestRegressor",
     "RandomForestClassifier",
+    "RandomForestRegressor",
     "ExtraTreesClassifier",
+    "ExtraTreesRegressor",
     "DecisionTreeClassifier",
+    "DecisionTreeRegressor",
     "ExtraTreeClassifier",
+    "ExtraTreeRegressor",
 ]
diff --git a/deepforest/_estimator.py b/deepforest/_estimator.py
index 35917e5..501c5c9 100644
--- a/deepforest/_estimator.py
+++ b/deepforest/_estimator.py
@@ -3,10 +3,15 @@
 
 __all__ = ["Estimator"]
 
-from .forest import RandomForestClassifier, ExtraTreesClassifier
+from .forest import (
+    RandomForestClassifier,
+    ExtraTreesClassifier,
+    RandomForestRegressor,
+    ExtraTreesRegressor,
+)
 
 
-def make_estimator(
+def make_classifier_estimator(
     name,
     n_trees=100,
     max_depth=None,
@@ -39,6 +44,39 @@ def make_estimator(
     return estimator
 
 
+def make_regressor_estimator(
+    name,
+    n_trees=100,
+    max_depth=None,
+    min_samples_leaf=1,
+    n_jobs=None,
+    random_state=None,
+):
+    # RandomForestRegressor
+    if name == "rf":
+        estimator = RandomForestRegressor(
+            n_estimators=n_trees,
+            max_depth=max_depth,
+            min_samples_leaf=min_samples_leaf,
+            n_jobs=n_jobs,
+            random_state=random_state,
+        )
+    # ExtraTreesRegressor
+    elif name == "erf":
+        estimator = ExtraTreesRegressor(
+            n_estimators=n_trees,
+            max_depth=max_depth,
+            min_samples_leaf=min_samples_leaf,
+            n_jobs=n_jobs,
+            random_state=random_state,
+        )
+    else:
+        msg = "Unknown type of estimator, which should be one of {{rf, erf}}."
+        raise NotImplementedError(msg)
+
+    return estimator
+
+
 class Estimator(object):
     def __init__(
         self,
@@ -48,10 +86,28 @@ def __init__(
         min_samples_leaf=1,
         n_jobs=None,
         random_state=None,
+        is_classifier=True,
     ):
-        self.estimator_ = make_estimator(
-            name, n_trees, max_depth, min_samples_leaf, n_jobs, random_state
-        )
+
+        self.is_classifier = is_classifier
+        if self.is_classifier:
+            self.estimator_ = make_classifier_estimator(
+                name,
+                n_trees,
+                max_depth,
+                min_samples_leaf,
+                n_jobs,
+                random_state,
+            )
+        else:
+            self.estimator_ = make_regressor_estimator(
+                name,
+                n_trees,
+                max_depth,
+                min_samples_leaf,
+                n_jobs,
+                random_state,
+            )
 
     @property
     def oob_decision_function_(self):
@@ -64,8 +120,11 @@ def fit_transform(self, X, y, sample_weight=None):
         return X_aug
 
     def transform(self, X):
-
-        return self.estimator_.predict_proba(X)
+        if self.is_classifier:
+            return self.estimator_.predict_proba(X)
+        return self.estimator_.predict(X)
 
     def predict(self, X):
-        return self.estimator_.predict_proba(X)
+        if self.is_classifier:
+            return self.estimator_.predict_proba(X)
+        return self.estimator_.predict(X)
diff --git a/deepforest/_layer.py b/deepforest/_layer.py
index 2c80093..ee76956 100644
--- a/deepforest/_layer.py
+++ b/deepforest/_layer.py
@@ -4,7 +4,7 @@
 __all__ = ["Layer"]
 
 import numpy as np
-from sklearn.metrics import accuracy_score
+from sklearn.metrics import accuracy_score, mean_squared_error
 
 from . import _utils
 from ._estimator import Estimator
@@ -56,6 +56,7 @@ def __init__(
         n_jobs=None,
         random_state=None,
         verbose=1,
+        is_classifier=True,
     ):
         self.layer_idx = layer_idx
         self.n_classes = n_classes
@@ -68,7 +69,7 @@ def __init__(
         self.n_jobs = n_jobs
         self.random_state = random_state
         self.verbose = verbose
-
+        self.is_classifier = is_classifier
         # Internal container
         self.estimators_ = {}
 
@@ -93,6 +94,7 @@ def _make_estimator(self, estimator_idx, estimator_name):
             min_samples_leaf=self.min_samples_leaf,
             n_jobs=self.n_jobs,
             random_state=random_state,
+            is_classifier=self.is_classifier,
         )
 
         return estimator
@@ -113,7 +115,10 @@ def fit_transform(self, X, y, sample_weight=None):
         n_samples, _ = X.shape
 
         X_aug = []
-        oob_decision_function = np.zeros((n_samples, self.n_classes))
+        if self.is_classifier:
+            oob_decision_function = np.zeros((n_samples, self.n_classes))
+        else:
+            oob_decision_function = np.zeros((n_samples, 1))
 
         # A random forest and an extremely random forest will be fitted
         for estimator_idx in range(self.n_estimators // 2):
@@ -154,18 +159,29 @@ def fit_transform(self, X, y, sample_weight=None):
 
         # Set the OOB estimations and validation accuracy
         self.oob_decision_function_ = oob_decision_function / self.n_estimators
-        y_pred = np.argmax(oob_decision_function, axis=1)
-        self.val_acc_ = accuracy_score(y, y_pred, sample_weight=sample_weight)
+        if self.is_classifier:
+            y_pred = np.argmax(oob_decision_function, axis=1)
+            self.val_acc_ = accuracy_score(
+                y, y_pred, sample_weight=sample_weight
+            )
+        else:
+            y_pred = self.oob_decision_function_
+            self.val_acc_ = mean_squared_error(
+                y, y_pred, sample_weight=sample_weight
+            )
 
         X_aug = np.hstack(X_aug)
         return X_aug
 
-    def transform(self, X):
+    def transform(self, X, is_classifier):
         """
         Return the concatenated transformation results from all base
         estimators."""
         n_samples, _ = X.shape
-        X_aug = np.zeros((n_samples, self.n_classes * self.n_estimators))
+        if is_classifier:
+            X_aug = np.zeros((n_samples, self.n_classes * self.n_estimators))
+        else:
+            X_aug = np.zeros((n_samples, self.n_estimators))
         for idx, (key, estimator) in enumerate(self.estimators_.items()):
             if self.verbose > 1:
                 msg = "{} - Evaluating estimator = {:<5} in layer = {}"
@@ -175,15 +191,21 @@ def transform(self, X):
                 # Load the estimator from the buffer
                 estimator = self.buffer.load_estimator(estimator)
 
-            left, right = self.n_classes * idx, self.n_classes * (idx + 1)
-            X_aug[:, left:right] += estimator.transform(X)
+            if is_classifier:
+                left, right = self.n_classes * idx, self.n_classes * (idx + 1)
+            else:
+                left, right = idx, (idx + 1)
+            X_aug[:, left:right] += estimator.predict(X)
 
         return X_aug
 
-    def predict_full(self, X):
+    def predict_full(self, X, is_classifier):
         """Return the concatenated predictions from all base estimators."""
         n_samples, _ = X.shape
-        pred = np.zeros((n_samples, self.n_classes * self.n_estimators))
+        if is_classifier:
+            pred = np.zeros((n_samples, self.n_classes * self.n_estimators))
+        else:
+            pred = np.zeros((n_samples, self.n_estimators))
         for idx, (key, estimator) in enumerate(self.estimators_.items()):
             if self.verbose > 1:
                 msg = "{} - Evaluating estimator = {:<5} in layer = {}"
@@ -193,7 +215,10 @@ def predict_full(self, X):
                 # Load the estimator from the buffer
                 estimator = self.buffer.load_estimator(estimator)
 
-            left, right = self.n_classes * idx, self.n_classes * (idx + 1)
+            if is_classifier:
+                left, right = self.n_classes * idx, self.n_classes * (idx + 1)
+            else:
+                left, right = idx, (idx + 1)
             pred[:, left:right] += estimator.predict(X)
 
         return pred
diff --git a/deepforest/cascade.py b/deepforest/cascade.py
index c7480d2..8d25485 100644
--- a/deepforest/cascade.py
+++ b/deepforest/cascade.py
@@ -1,7 +1,7 @@
 """Implementation of Deep Forest."""
 
 
-__all__ = ["CascadeForestClassifier"]
+__all__ = ["CascadeForestClassifier", "CascadeForestRegressor"]
 
 import time
 import numbers
@@ -9,7 +9,8 @@
 from abc import ABCMeta, abstractmethod
 from sklearn.preprocessing import LabelEncoder
 from sklearn.utils.multiclass import type_of_target
-from sklearn.base import BaseEstimator, ClassifierMixin
+from sklearn.base import BaseEstimator, ClassifierMixin, RegressorMixin
+from sklearn.base import is_classifier
 
 from . import _utils
 from . import _io
@@ -25,7 +26,7 @@ def _get_predictor_kwargs(predictor_kwargs, **kwargs) -> dict:
     return predictor_kwargs
 
 
-def _build_predictor(
+def _build_classifier_predictor(
     predictor_name,
     n_estimators,
     n_outputs,
@@ -107,7 +108,89 @@ def _build_predictor(
     return predictor
 
 
-__model_doc = """
+def _build_regressor_predictor(
+    predictor_name,
+    n_estimators,
+    n_outputs,
+    max_depth=None,
+    min_samples_leaf=1,
+    n_jobs=None,
+    random_state=None,
+    predictor_kwargs={},
+):
+    """Build the predictor concatenated to the deep forest."""
+    predictor_name = predictor_name.lower()
+
+    # Random Forest
+    if predictor_name == "forest":
+        from .forest import RandomForestRegressor
+
+        predictor = RandomForestRegressor(
+            **_get_predictor_kwargs(
+                predictor_kwargs,
+                n_estimators=n_estimators,
+                max_depth=max_depth,
+                min_samples_leaf=min_samples_leaf,
+                n_jobs=n_jobs,
+                random_state=random_state,
+            )
+        )
+    # XGBoost
+    elif predictor_name == "xgboost":
+        try:
+            xgb = __import__("xgboost.sklearn")
+        except ModuleNotFoundError:
+            msg = (
+                "Cannot load the module XGBoost when building the predictor."
+                " Please make sure that XGBoost is installed."
+            )
+            raise ModuleNotFoundError(msg)
+
+        # The argument `tree_method` is always set as `hist` for XGBoost,
+        # because the exact mode of XGBoost is too slow.
+        objective = "reg:squarederror"
+        predictor = xgb.sklearn.XGBRegressor(
+            **_get_predictor_kwargs(
+                predictor_kwargs,
+                objective=objective,
+                n_estimators=n_estimators,
+                tree_method="hist",
+                n_jobs=n_jobs,
+                random_state=random_state,
+            )
+        )
+    # LightGBM
+    elif predictor_name == "lightgbm":
+        try:
+            lgb = __import__("lightgbm.sklearn")
+        except ModuleNotFoundError:
+            msg = (
+                "Cannot load the module LightGBM when building the predictor."
+                " Please make sure that LightGBM is installed."
+            )
+            raise ModuleNotFoundError(msg)
+
+        objective = "regression"
+        predictor = lgb.LGBMRegressor(
+            **_get_predictor_kwargs(
+                predictor_kwargs,
+                objective=objective,
+                n_estimators=n_estimators,
+                n_jobs=n_jobs,
+                random_state=random_state,
+            )
+        )
+    else:
+        msg = (
+            "The name of the predictor should be one of {{forest, xgboost,"
+            " lightgbm}}, but got {} instead."
+        )
+        raise NotImplementedError(msg.format(predictor_name))
+
+    return predictor
+
+
+__classifier_model_doc = """
     Parameters
     ----------
     n_bins : :obj:`int`, default=255
@@ -184,7 +267,7 @@ def _build_predictor(
 """
 
 
-__fit_doc = """
+__classifier_fit_doc = """
 
     .. note::
 
@@ -213,6 +296,111 @@ def _build_predictor(
         Sample weights. If ``None``, then samples are equally weighted.
 """
 
+__regressor_model_doc = """
+    Parameters
+    ----------
+    n_bins : :obj:`int`, default=255
+        The number of bins used for non-missing values. In addition to the
+        ``n_bins`` bins, one more bin is reserved for missing values. Its
+        value must be no smaller than 2 and no greater than 255.
+    bin_subsample : :obj:`int`, default=2e5
+        The number of samples used to construct feature discrete bins. If
+        the size of training set is smaller than ``bin_subsample``, then all
+        training samples will be used.
+    max_layers : :obj:`int`, default=20
+        The maximum number of cascade layers in the deep forest. Notice that
+        the actual number of layers can be smaller than ``max_layers`` because
+        of the internal early stopping stage.
+    n_estimators : :obj:`int`, default=2
+        The number of estimator in each cascade layer. It will be multiplied
+        by 2 internally because each estimator contains a
+        :class:`RandomForestRegressor` and a :class:`ExtraTreesRegressor`,
+        respectively.
+    n_trees : :obj:`int`, default=100
+        The number of trees in each estimator.
+    max_depth : :obj:`int`, default=None
+        The maximum depth of each tree. ``None`` indicates no constraint.
+    min_samples_leaf : :obj:`int`, default=1
+        The minimum number of samples required to be at a leaf node.
+    use_predictor : :obj:`bool`, default=False
+        Whether to build the predictor concatenated to the deep forest. Using
+        the predictor may improve the performance of deep forest.
+    predictor : :obj:`{"forest", "xgboost", "lightgbm"}`, default="forest"
+        The type of the predictor concatenated to the deep forest. If
+        ``use_predictor`` is False, this parameter will have no effect.
+    predictor_kwargs : :obj:`dict`, default={}
+        The configuration of the predictor concatenated to the deep forest.
+        Specifying this will extend/overwrite the original parameters inherit
+        from deep forest.
+        If ``use_predictor`` is False, this parameter will have no effect.
+    n_tolerant_rounds : :obj:`int`, default=2
+        Specify when to conduct early stopping. The training process
+        terminates when the validation performance on the training set does
+        not improve compared against the best validation performance achieved
+        so far for ``n_tolerant_rounds`` rounds.
+    delta : :obj:`float`, default=1e-5
+        Specify the threshold on early stopping. The counting on
+        ``n_tolerant_rounds`` is triggered if the performance of a fitted
+        cascade layer does not improve by ``delta`` compared against the best
+        validation performance achieved so far.
+    partial_mode : :obj:`bool`, default=False
+        Whether to train the deep forest in partial mode. For large
+        datasets, it is recommended to use the partial mode.
+
+        - If ``True``, the partial mode is activated and all fitted
+          estimators will be dumped in a local buffer;
+        - If ``False``, all fitted estimators are directly stored in the
+          memory.
+    n_jobs : :obj:`int` or ``None``, default=None
+        The number of jobs to run in parallel for both :meth:`fit` and
+        :meth:`predict`. None means 1 unless in a
+        :obj:`joblib.parallel_backend` context. ``-1`` means using all
+        processors.
+    random_state : :obj:`int` or ``None``, default=None
+
+        - If :obj:`int`, ``random_state`` is the seed used by the random
+          number generator;
+        - If ``None``, the random number generator is the RandomState
+          instance used by :mod:`np.random`.
+    verbose : :obj:`int`, default=1
+        Controls the verbosity when fitting and predicting.
+
+        - If ``<= 0``, silent mode, which means no logging information will
+          be displayed;
+        - If ``1``, logging information on the cascade layer level will be
+          displayed;
+        - If ``> 1``, full logging information will be displayed.
+"""
+
+__regressor_fit_doc = """
+
+    .. note::
+
+        Deep forest supports two kinds of modes for training:
+
+        - **Full memory mode**, in which the training / testing data and
+          all fitted estimators are directly stored in the memory.
+        - **Partial mode**, in which after fitting each estimator using
+          the training data, it will be dumped in the buffer. During the
+          evaluating stage, the dumped estimators are reloaded into the
+          memory sequentially to evaluate the testing data.
+
+        By setting the ``partial_mode`` to ``True``, the partial mode is
+        activated, and a local buffer will be created at the current
+        directory. The partial mode is able to reduce the running memory
+        cost when training the deep forest.
+
+    Parameters
+    ----------
+    X : :obj:`numpy.ndarray` of shape (n_samples, n_features)
+        The training data. Internally, it will be converted to
+        ``np.uint8``.
+    y : :obj:`numpy.ndarray` of shape (n_samples,)
+        The target of input samples.
+    sample_weight : :obj:`numpy.ndarray` of shape (n_samples,), default=None
+        Sample weights. If ``None``, then samples are equally weighted.
+"""
+
 
 def deepforest_model_doc(header, item):
     """
@@ -228,7 +416,13 @@ def deepforest_model_doc(header, item):
 
     def get_doc(item):
         """Return the selected item."""
-        __doc = {"model": __model_doc, "fit": __fit_doc}
+        __doc = {
+            "regressor_model": __regressor_model_doc,
+            "regressor_fit": __regressor_fit_doc,
+            "classifier_model": __classifier_model_doc,
+            "classifier_fit": __classifier_fit_doc,
+        }
+
         return __doc[item]
 
     def adddoc(cls):
@@ -298,8 +492,10 @@ def __getitem__(self, index):
 
     def _get_n_output(self, y):
         """Return the number of output inferred from the training labels."""
-        n_output = np.unique(y).shape[0]  # classification
-        return n_output
+        if is_classifier(self):
+            n_output = np.unique(y).shape[0]  # classification
+            return n_output
+        return 1  # this parameter are not used in regression
 
     def _get_layer(self, layer_idx):
         """Get the layer from the internal container according to the index."""
@@ -460,6 +656,13 @@ def _handle_early_stopping(self):
             msg = "{} The optimal number of layers: {}"
             print(msg.format(_utils.ctime(), self.n_layers_))
 
+    def _if_improved(self, new_pivot, pivot, delta, is_classifier):
+        """
+        Return true if new vlidation result is better than previous"""
+        if is_classifier:
+            return new_pivot >= pivot + delta
+        return new_pivot <= pivot - delta
+
     @abstractmethod
     def _repr_performance(self, pivot):
         """Format the printting information on training performance."""
@@ -474,7 +677,9 @@ def predict(self, X):
 
     @property
     def n_aug_features_(self):
-        return 2 * self.n_estimators * self.n_outputs_
+        if is_classifier(self):
+            return 2 * self.n_estimators * self.n_outputs_
+        return 2 * self.n_estimators
 
     # flake8: noqa: E501
     def fit(self, X, y, sample_weight=None):
@@ -514,13 +719,16 @@ def fit(self, X, y, sample_weight=None):
             self.n_jobs,
             self.random_state,
             self.verbose,
+            is_classifier(self),
         )
 
         if self.verbose > 0:
             print("{} Fitting cascade layer = {:<2}".format(_utils.ctime(), 0))
 
         tic = time.time()
-        X_aug_train_ = layer_.fit_transform(X_train_, y, sample_weight)
+        X_aug_train_ = layer_.fit_transform(
+            X_train_, y, sample_weight=sample_weight
+        )
         toc = time.time()
         training_time = toc - tic
 
@@ -586,6 +794,7 @@ def fit(self, X, y, sample_weight=None):
                 self.n_jobs,
                 self.random_state,
                 self.verbose,
+                is_classifier(self),
             )
 
             X_middle_train_ = self.buffer_.cache_data(
@@ -598,7 +807,7 @@ def fit(self, X, y, sample_weight=None):
 
             tic = time.time()
             X_aug_train_ = layer_.fit_transform(
-                X_middle_train_, y, sample_weight
+                X_middle_train_, y, sample_weight=sample_weight
             )
             toc = time.time()
             training_time = toc - tic
@@ -622,7 +831,9 @@ def fit(self, X, y, sample_weight=None):
             # training stage will terminate before reaching the maximum number
             # of layers.
 
-            if new_pivot >= pivot + self.delta:
+            if self._if_improved(
+                new_pivot, pivot, self.delta, is_classifier(self)
+            ):
 
                 # Update the cascade layer
                 self._set_layer(layer_idx, layer_)
@@ -667,16 +878,28 @@ def fit(self, X, y, sample_weight=None):
 
         # Build the predictor if `self.use_predictor` is True
         if self.use_predictor:
-            self.predictor_ = _build_predictor(
-                self.predictor_name,
-                self.n_trees,
-                self.n_outputs_,
-                self.max_depth,
-                self.min_samples_leaf,
-                self.n_jobs,
-                self.random_state,
-                self.predictor_kwargs,
-            )
+            if is_classifier(self):
+                self.predictor_ = _build_classifier_predictor(
+                    self.predictor_name,
+                    self.n_trees,
+                    self.n_outputs_,
+                    self.max_depth,
+                    self.min_samples_leaf,
+                    self.n_jobs,
+                    self.random_state,
+                    self.predictor_kwargs,
+                )
+            else:
+                self.predictor_ = _build_regressor_predictor(
+                    self.predictor_name,
+                    self.n_trees,
+                    self.n_outputs_,
+                    self.max_depth,
+                    self.min_samples_leaf,
+                    self.n_jobs,
+                    self.random_state,
+                    self.predictor_kwargs,
+                )
 
             binner_ = Binner(
                 n_bins=self.n_bins,
@@ -874,7 +1097,8 @@ def clean(self):
 
 
 @deepforest_model_doc(
-    """Implementation of the deep forest for classification.""", "model"
+    """Implementation of the deep forest for classification.""",
+    "classifier_model",
 )
 class CascadeForestClassifier(BaseCascadeForest, ClassifierMixin):
     def __init__(
@@ -957,7 +1181,7 @@ def _repr_performance(self, pivot):
         return msg.format(pivot * 100)
 
     @deepforest_model_doc(
-        """Build a deep forest using the training data.""", "fit"
+        """Build a deep forest using the training data.""", "classifier_fit"
     )
     def fit(self, X, y, sample_weight=None):
 
@@ -1000,7 +1224,7 @@ def predict_proba(self, X):
                 print(msg.format(_utils.ctime(), layer_idx))
 
             if layer_idx == 0:
-                X_aug_test_ = layer.transform(X_test)
+                X_aug_test_ = layer.transform(X_test, is_classifier(self))
             elif layer_idx < self.n_layers_ - 1:
                 binner_ = self._get_binner(layer_idx)
                 X_aug_test_ = self._bin_data(
@@ -1009,7 +1233,9 @@ def predict_proba(self, X):
                 X_middle_test_ = _utils.merge_array(
                     X_middle_test_, X_aug_test_, self.n_features_
                 )
-                X_aug_test_ = layer.transform(X_middle_test_)
+                X_aug_test_ = layer.transform(
+                    X_middle_test_, is_classifier(self)
+                )
             else:
                 binner_ = self._get_binner(layer_idx)
                 X_aug_test_ = self._bin_data(
@@ -1021,7 +1247,9 @@ def predict_proba(self, X):
 
                 # Skip calling the `transform` if not using the predictor
                 if self.use_predictor:
-                    X_aug_test_ = layer.transform(X_middle_test_)
+                    X_aug_test_ = layer.transform(
+                        X_middle_test_, is_classifier(self)
+                    )
 
         if self.use_predictor:
 
@@ -1039,7 +1267,7 @@ def predict_proba(self, X):
             predictor = self.buffer_.load_predictor(self.predictor_)
             proba = predictor.predict_proba(X_middle_test_)
         else:
-            proba = layer.predict_full(X_middle_test_)
+            proba = layer.predict_full(X_middle_test_, is_classifier(self))
             proba = _utils.merge_proba(proba, self.n_outputs_)
 
         return proba
@@ -1062,3 +1290,139 @@ def predict(self, X):
         proba = self.predict_proba(X)
         y = self._decode_class_labels(np.argmax(proba, axis=1))
         return y
+
+
+@deepforest_model_doc(
+    """Implementation of the deep forest for regression.""", "regressor_model"
+)
+class CascadeForestRegressor(BaseCascadeForest, RegressorMixin):
+    def __init__(
+        self,
+        n_bins=255,
+        bin_subsample=2e5,
+        bin_type="percentile",
+        max_layers=20,
+        n_estimators=2,
+        n_trees=100,
+        max_depth=None,
+        min_samples_leaf=1,
+        use_predictor=False,
+        predictor="forest",
+        predictor_kwargs={},
+        n_tolerant_rounds=2,
+        delta=1e-5,
+        partial_mode=False,
+        n_jobs=None,
+        random_state=None,
+        verbose=1,
+    ):
+        super().__init__(
+            n_bins=n_bins,
+            bin_subsample=bin_subsample,
+            bin_type=bin_type,
+            max_layers=max_layers,
+            n_estimators=n_estimators,
+            n_trees=n_trees,
+            max_depth=max_depth,
+            min_samples_leaf=min_samples_leaf,
+            use_predictor=use_predictor,
+            predictor=predictor,
+            predictor_kwargs=predictor_kwargs,
+            n_tolerant_rounds=n_tolerant_rounds,
+            delta=delta,
+            partial_mode=partial_mode,
+            n_jobs=n_jobs,
+            random_state=random_state,
+            verbose=verbose,
+        )
+
+    def _repr_performance(self, pivot):
+        msg = "Val Acc = {:.3f}"
+        return msg.format(pivot)
+
+    @deepforest_model_doc(
+        """Build a deep forest using the training data.""", "regressor_fit"
+    )
+    def fit(self, X, y, sample_weight=None):
+        super().fit(X, y, sample_weight)
+
+    def predict(self, X):
+        """
+        Predict regression target for X.
+
+        Parameters
+        ----------
+        X : :obj:`numpy.ndarray` of shape (n_samples, n_features)
+            The input samples. Internally, its dtype will be converted to
+            ``np.uint8``.
+
+        Returns
+        -------
+        y : :obj:`numpy.ndarray` of shape (n_samples,)
+            The predicted values.
+        """
+        if not self.is_fitted_:
+            raise AttributeError("Please fit the model first.")
+        self._check_input(X)
+
+        if self.verbose > 0:
+            print("{} Start to evalute the model:".format(_utils.ctime()))
+
+        binner_ = self._get_binner(0)
+        X_test = self._bin_data(binner_, X, is_training_data=False)
+        X_middle_test_ = _utils.init_array(X_test, self.n_aug_features_)
+
+        for layer_idx in range(self.n_layers_):
+            layer = self._get_layer(layer_idx)
+
+            if self.verbose > 0:
+                msg = "{} Evaluating cascade layer = {:<2}"
+                print(msg.format(_utils.ctime(), layer_idx))
+
+            if layer_idx == 0:
+                X_aug_test_ = layer.transform(X_test, is_classifier(self))
+            elif layer_idx < self.n_layers_ - 1:
+                binner_ = self._get_binner(layer_idx)
+                X_aug_test_ = self._bin_data(
+                    binner_, X_aug_test_, is_training_data=False
+                )
+                X_middle_test_ = _utils.merge_array(
+                    X_middle_test_, X_aug_test_, self.n_features_
+                )
+                X_aug_test_ = layer.transform(
+                    X_middle_test_, is_classifier(self)
+                )
+            else:
+                binner_ = self._get_binner(layer_idx)
+                X_aug_test_ = self._bin_data(
+                    binner_, X_aug_test_, is_training_data=False
+                )
+                X_middle_test_ = _utils.merge_array(
+                    X_middle_test_, X_aug_test_, self.n_features_
+                )
+
+                # Skip calling the `transform` if not using the predictor
+                if self.use_predictor:
+                    X_aug_test_ = layer.transform(
+                        X_middle_test_, is_classifier(self)
+                    )
+
+        if self.use_predictor:
+
+            if self.verbose > 0:
+                print("{} Evaluating the predictor".format(_utils.ctime()))
+
+            binner_ = self._get_binner(self.n_layers_)
+            X_aug_test_ = self._bin_data(
+                binner_, X_aug_test_, is_training_data=False
+            )
+            X_middle_test_ = _utils.merge_array(
+                X_middle_test_, X_aug_test_, self.n_features_
+            )
+
+            predictor = self.buffer_.load_predictor(self.predictor_)
+            _y = predictor.predict(X_middle_test_)
+        else:
+            _y = layer.predict_full(X_middle_test_, is_classifier(self))
+            _y = _y.sum(axis=1) / _y.shape[1]
+        return _y
diff --git a/deepforest/forest.py b/deepforest/forest.py
index 030c5d7..203bdb8 100644
--- a/deepforest/forest.py
+++ b/deepforest/forest.py
@@ -6,7 +6,12 @@
 """
 
 
-__all__ = ["RandomForestClassifier", "ExtraTreesClassifier"]
+__all__ = [
+    "RandomForestClassifier",
+    "RandomForestRegressor",
+    "ExtraTreesClassifier",
+    "ExtraTreesRegressor",
+]
 
 import numbers
 from warnings import warn
@@ -22,7 +27,8 @@
 from sklearn.base import clone
 from sklearn.base import BaseEstimator
 from sklearn.base import MetaEstimatorMixin
-from sklearn.base import ClassifierMixin, MultiOutputMixin
+from sklearn.base import is_classifier
+from sklearn.base import ClassifierMixin, RegressorMixin, MultiOutputMixin
 from sklearn.utils import check_random_state, compute_sample_weight
 from sklearn.exceptions import DataConversionWarning
 from sklearn.utils.fixes import _joblib_parallel_args
@@ -32,7 +38,12 @@
 from . import _cutils as _LIB
 from . import _forest as _C_FOREST
 
-from .tree import DecisionTreeClassifier, ExtraTreeClassifier
+from .tree import (
+    DecisionTreeClassifier,
+    DecisionTreeRegressor,
+    ExtraTreeClassifier,
+    ExtraTreeRegressor,
+)
 from .tree._tree import DOUBLE
 
 
@@ -96,6 +107,8 @@ def _parallel_build_trees(
     n_samples_bootstrap,
     sample_weight,
     out,
+    mask,
+    is_classifier,
     lock,
 ):
     """
@@ -123,14 +136,17 @@ def _parallel_build_trees(
         value = np.ascontiguousarray(value)
 
     value = np.squeeze(value, axis=1)
-    value /= value.sum(axis=1)[:, np.newaxis]
+
+    if is_classifier:
+        value /= value.sum(axis=1)[:, np.newaxis]
 
     # Set the OOB predictions
     oob_prediction = _C_FOREST.predict(
         X[~sample_mask, :], feature, threshold, children, value
     )
-
     with lock:
+
+        mask += ~sample_mask
         out[~sample_mask, :] += oob_prediction
 
     return feature, threshold, children, value
@@ -194,7 +210,6 @@ def _partition_estimators(n_estimators, n_jobs):
 def _accumulate_prediction(feature, threshold, children, value, X, out, lock):
     """This is a utility function for joblib's Parallel."""
     prediction = _C_FOREST.predict(X, feature, threshold, children, value)
-
     with lock:
         if len(out) == 1:
             out[0] += prediction
@@ -284,8 +299,9 @@ def _make_estimator(self, append=True, random_state=None):
         )
 
         # Pass the inferred class information to avoid redudant finding.
-        estimator.classes_ = self.classes_
-        estimator.n_classes_ = np.array(self.n_classes_, dtype=np.int32)
+        if is_classifier(estimator):
+            estimator.classes_ = self.classes_
+            estimator.n_classes_ = np.array(self.n_classes_, dtype=np.int32)
 
         if random_state is not None:
             _set_random_states(estimator, random_state)
@@ -411,6 +427,12 @@ def fit(self, X, y, sample_weight=None):
         if getattr(y, "dtype", None) != DOUBLE or not y.flags.contiguous:
             y = np.ascontiguousarray(y, dtype=DOUBLE)
 
+        if expanded_class_weight is not None:
+            if sample_weight is not None:
+                sample_weight = sample_weight * expanded_class_weight
+            else:
+                sample_weight = expanded_class_weight
+
         # Get bootstrap sample size
         n_samples_bootstrap = _get_n_samples_bootstrap(
             n_samples=X.shape[0], max_samples=self.max_samples
@@ -427,9 +449,13 @@ def fit(self, X, y, sample_weight=None):
         ]
 
         # Pre-allocate OOB estimations
-        oob_decision_function = np.zeros(
-            (n_samples, self.classes_[0].shape[0])
-        )
+        if is_classifier(self):
+            oob_decision_function = np.zeros(
+                (n_samples, self.classes_[0].shape[0])
+            )
+        else:
+            oob_decision_function = np.zeros((n_samples, 1))
+        mask = np.zeros(n_samples)
 
         lock = threading.Lock()
         rets = Parallel(
@@ -444,11 +470,12 @@ def fit(self, X, y, sample_weight=None):
                 n_samples_bootstrap,
                 sample_weight,
                 oob_decision_function,
+                mask,
+                is_classifier(self),
                 lock,
             )
             for i, t in enumerate(trees)
         )
-
         # Collect newly grown trees
         for feature, threshold, children, value in rets:
 
@@ -460,17 +487,22 @@ def fit(self, X, y, sample_weight=None):
             self.values.append(value)
 
         # Check the OOB predictions
-        if (oob_decision_function.sum(axis=1) == 0).any():
+        if (
+            is_classifier(self)
+            and (oob_decision_function.sum(axis=1) == 0).any()
+        ):
             warn(
                 "Some inputs do not have OOB predictions. "
                 "This probably means too few trees were used "
                 "to compute any reliable oob predictions."
             )
-
-        prediction = (
-            oob_decision_function
-            / oob_decision_function.sum(axis=1)[:, np.newaxis]
-        )
+        if is_classifier(self):
+            prediction = (
+                oob_decision_function
+                / oob_decision_function.sum(axis=1)[:, np.newaxis]
+            )
+        else:
+            prediction = oob_decision_function / mask.reshape(-1, 1)
 
         self.oob_decision_function_ = prediction
 
@@ -701,3 +733,206 @@ def __init__(
         self.max_features = max_features
         self.min_impurity_decrease = min_impurity_decrease
         self.min_impurity_split = min_impurity_split
+
+
+class ForestRegressor(RegressorMixin, BaseForest, metaclass=ABCMeta):
+    """
+    Base class for forest of trees-based regressors.
+
+    Warning: This class should not be used directly. Use derived classes
+    instead.
+    """
+
+    @abstractmethod
+    def __init__(
+        self,
+        base_estimator,
+        n_estimators=100,
+        *,
+        estimator_params=tuple(),
+        n_jobs=None,
+        random_state=None,
+        verbose=0,
+        max_samples=None
+    ):
+        super().__init__(
+            base_estimator,
+            n_estimators=n_estimators,
+            estimator_params=estimator_params,
+            n_jobs=n_jobs,
+            random_state=random_state,
+            verbose=verbose,
+            max_samples=max_samples,
+        )
+
+    def predict(self, X):
+        """
+        Predict regression target for X.
+
+        The predicted regression target of an input sample is computed as the
+        mean predicted regression targets of the trees in the forest.
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            The input samples. Internally, its dtype will be converted to
+            ``dtype=np.float32``. If a sparse matrix is provided, it will be
+            converted into a sparse ``csr_matrix``.
+
+        Returns
+        -------
+        y : ndarray of shape (n_samples,) or (n_samples, n_outputs)
+            The predicted values.
+        """
+        check_is_fitted(self)
+
+        # Assign chunk of trees to jobs
+        n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)
+
+        # avoid storing the output of every estimator by summing them here
+        y_hat = np.zeros((X.shape[0], 1), dtype=np.float64)
+
+        # Parallel loop
+        lock = threading.Lock()
+        Parallel(
+            n_jobs=n_jobs,
+            verbose=self.verbose,
+            **_joblib_parallel_args(require="sharedmem")
+        )(
+            delayed(_accumulate_prediction)(
+                self.features[i],
+                self.thresholds[i],
+                self.childrens[i],
+                self.values[i],
+                X,
+                [y_hat],
+                lock,
+            )
+            for i in range(self.n_estimators)
+        )
+
+        y_hat /= self.n_estimators
+        return y_hat
+
+    @staticmethod
+    def _get_oob_predictions(tree, X):
+        """Compute the OOB predictions for an individual tree.
+
+        Parameters
+        ----------
+        tree : DecisionTreeRegressor object
+            A single decision tree regressor.
+        X : ndarray of shape (n_samples, n_features)
+            The OOB samples.
+
+        Returns
+        -------
+        y_pred : ndarray of shape (n_samples, 1, n_outputs)
+            The OOB associated predictions.
+        """
+        y_pred = tree.predict(X, check_input=False)
+        if y_pred.ndim == 1:
+            # single output regression
+            y_pred = y_pred[:, np.newaxis, np.newaxis]
+        else:
+            # multioutput regression
+            y_pred = y_pred[:, np.newaxis, :]
+        return y_pred
+
+
+class RandomForestRegressor(ForestRegressor):
+    @_deprecate_positional_args
+    def __init__(
+        self,
+        n_estimators=100,
+        *,
+        criterion="mse",
+        max_depth=None,
+        min_samples_split=2,
+        min_samples_leaf=1,
+        min_weight_fraction_leaf=0.0,
+        max_features="auto",
+        min_impurity_decrease=0.0,
+        min_impurity_split=None,
+        n_jobs=None,
+        random_state=None,
+        verbose=0,
+        max_samples=None
+    ):
+        super().__init__(
+            base_estimator=DecisionTreeRegressor(),
+            n_estimators=n_estimators,
+            estimator_params=(
+                "criterion",
+                "max_depth",
+                "min_samples_split",
+                "min_samples_leaf",
+                "min_weight_fraction_leaf",
+                "max_features",
+                "min_impurity_decrease",
+                "min_impurity_split",
+                "random_state",
+            ),
+            n_jobs=n_jobs,
+            random_state=random_state,
+            verbose=verbose,
+            max_samples=max_samples,
+        )
+
+        self.criterion = criterion
+        self.max_depth = max_depth
+        self.min_samples_split = min_samples_split
+        self.min_samples_leaf = min_samples_leaf
+        self.min_weight_fraction_leaf = min_weight_fraction_leaf
+        self.max_features = max_features
+        self.min_impurity_decrease = min_impurity_decrease
+        self.min_impurity_split = min_impurity_split
+
+
+class ExtraTreesRegressor(ForestRegressor):
+    @_deprecate_positional_args
+    def __init__(
+        self,
+        n_estimators=100,
+        *,
+        criterion="mse",
+        max_depth=None,
+        min_samples_split=2,
+        min_samples_leaf=1,
+        min_weight_fraction_leaf=0.0,
+        max_features="auto",
+        min_impurity_decrease=0.0,
+        min_impurity_split=None,
+        n_jobs=None,
+        random_state=None,
+        verbose=0,
+        max_samples=None
+    ):
+        super().__init__(
+            base_estimator=ExtraTreeRegressor(),
+            n_estimators=n_estimators,
+            estimator_params=(
+                "criterion",
+                "max_depth",
+                "min_samples_split",
+                "min_samples_leaf",
+                "min_weight_fraction_leaf",
+                "max_features",
+                "min_impurity_decrease",
+                "min_impurity_split",
+                "random_state",
+            ),
+            n_jobs=n_jobs,
+            random_state=random_state,
+            verbose=verbose,
+            max_samples=max_samples,
+        )
+
+        self.criterion = criterion
+        self.max_depth = max_depth
+        self.min_samples_split = min_samples_split
+        self.min_samples_leaf = min_samples_leaf
+        self.min_weight_fraction_leaf = min_weight_fraction_leaf
+        self.max_features = max_features
+        self.min_impurity_decrease = min_impurity_decrease
+        self.min_impurity_split = min_impurity_split
diff --git a/deepforest/tree/__init__.py b/deepforest/tree/__init__.py
index 3fdfb3c..e1b0767 100644
--- a/deepforest/tree/__init__.py
+++ b/deepforest/tree/__init__.py
@@ -1,6 +1,13 @@
 from .tree import BaseDecisionTree
 from .tree import DecisionTreeClassifier
+from .tree import DecisionTreeRegressor
 from .tree import ExtraTreeClassifier
+from .tree import ExtraTreeRegressor
 
-
-__all__ = ["BaseDecisionTree", "DecisionTreeClassifier", "ExtraTreeClassifier"]
+__all__ = [
+    "BaseDecisionTree",
+    "DecisionTreeClassifier",
+    "DecisionTreeRegressor",
+    "ExtraTreeClassifier",
+    "ExtraTreeRegressor",
+]
diff --git a/deepforest/tree/tree.py b/deepforest/tree/tree.py
index c73792b..ead6dc2 100644
--- a/deepforest/tree/tree.py
+++ b/deepforest/tree/tree.py
@@ -6,7 +6,12 @@
 """
 
 
-__all__ = ["DecisionTreeClassifier", "ExtraTreeClassifier"]
+__all__ = [
+    "DecisionTreeClassifier",
+    "DecisionTreeRegressor",
+    "ExtraTreeClassifier",
+    "ExtraTreeRegressor",
+]
 
 import numbers
 import warnings
@@ -18,6 +23,7 @@
 
 from sklearn.base import BaseEstimator
 from sklearn.base import ClassifierMixin
+from sklearn.base import RegressorMixin
 from sklearn.base import is_classifier
 from sklearn.base import MultiOutputMixin
 from sklearn.utils import check_array
@@ -43,6 +49,7 @@
 DOUBLE = _tree.DOUBLE
 
 CRITERIA_CLF = {"gini": _criterion.Gini, "entropy": _criterion.Entropy}
+CRITERIA_REG = {"mse": _criterion.MSE}
 
 DENSE_SPLITTERS = {
     "best": _splitter.BestSplitter,
@@ -77,7 +84,7 @@ def __init__(
         min_impurity_decrease,
         min_impurity_split,
         class_weight=None,
-        presort="deprecated"
+        presort="deprecated",
     ):
         self.criterion = criterion
         self.splitter = splitter
@@ -164,7 +171,7 @@ def fit(
         self.n_outputs_ = y.shape[1]
 
         # `classes_` and `n_classes_` were set by the forest.
-        if not hasattr(self, "classes_"):
+        if not hasattr(self, "classes_") and is_classifier(self):
             check_classification_targets(y)
             y = np.copy(y)
 
@@ -323,9 +330,14 @@ def fit(
         # Build tree
         criterion = self.criterion
         if not isinstance(criterion, Criterion):
-            criterion = CRITERIA_CLF[self.criterion](
-                self.n_outputs_, self.n_classes_
-            )
+            if is_classifier(self):
+                criterion = CRITERIA_CLF[self.criterion](
+                    self.n_outputs_, self.n_classes_
+                )
+            else:
+                criterion = CRITERIA_REG[self.criterion](
+                    self.n_outputs_, n_samples
+                )
 
         SPLITTERS = DENSE_SPLITTERS
 
@@ -339,7 +351,17 @@ def fit(
                 random_state,
             )
 
-        self.tree_ = Tree(self.n_features_, self.n_classes_, self.n_outputs_)
+        if is_classifier(self):
+            self.tree_ = Tree(
+                self.n_features_, self.n_classes_, self.n_outputs_
+            )
+        else:
+            self.tree_ = Tree(
+                self.n_features_,
+                # TODO: tree should't need this in this case
+                np.array([1] * self.n_outputs_, dtype=np.int32),
+                self.n_outputs_,
+            )
 
         builder = DepthFirstTreeBuilder(
             splitter,
@@ -409,7 +431,12 @@ def predict(self, X, check_input=True):
         X = self._validate_X_predict(X, check_input)
         proba = self.tree_.predict(X)
 
-        return self.classes_.take(np.argmax(proba, axis=1), axis=0)
+        # Classification
+        if is_classifier(self):
+            return self.classes_.take(np.argmax(proba, axis=1), axis=0)
+        # Regression
+        else:
+            return proba[:, 0]
 
 
 class DecisionTreeClassifier(ClassifierMixin, BaseDecisionTree):
@@ -428,7 +455,7 @@ def __init__(
         min_impurity_decrease=0.0,
         min_impurity_split=None,
         class_weight=None,
-        presort="deprecated"
+        presort="deprecated",
     ):
 
         super().__init__(
@@ -472,6 +499,49 @@ def predict_proba(self, X, check_input=True):
         return proba
 
 
+class DecisionTreeRegressor(RegressorMixin, BaseDecisionTree):
+    @_deprecate_positional_args
+    def __init__(
+        self,
+        *,
+        criterion="mse",
+        splitter="best",
+        max_depth=None,
+        min_samples_split=2,
+        min_samples_leaf=1,
+        min_weight_fraction_leaf=0.0,
+        max_features=None,
+        random_state=None,
+        min_impurity_decrease=0.0,
+        min_impurity_split=None,
+        presort="deprecated",
+    ):
+        super().__init__(
+            criterion=criterion,
+            splitter=splitter,
+            max_depth=max_depth,
+            min_samples_split=min_samples_split,
+            min_samples_leaf=min_samples_leaf,
+            min_weight_fraction_leaf=min_weight_fraction_leaf,
+            max_features=max_features,
+            min_impurity_decrease=min_impurity_decrease,
+            min_impurity_split=min_impurity_split,
+            random_state=random_state,
+        )
+
+    def fit(
+        self, X, y, sample_weight=None, check_input=True, X_idx_sorted=None
+    ):
+
+        return super().fit(
+            X,
+            y,
+            sample_weight=sample_weight,
+            check_input=check_input,
+            X_idx_sorted=X_idx_sorted,
+        )
+
+
 class ExtraTreeClassifier(DecisionTreeClassifier):
     @_deprecate_positional_args
     def __init__(
@@ -487,7 +557,7 @@ def __init__(
         random_state=None,
         min_impurity_decrease=0.0,
         min_impurity_split=None,
-        class_weight=None
+        class_weight=None,
     ):
 
         super().__init__(
@@ -503,3 +573,34 @@ def __init__(
             min_impurity_split=min_impurity_split,
             random_state=random_state,
         )
+
+
+class ExtraTreeRegressor(DecisionTreeRegressor):
+    @_deprecate_positional_args
+    def __init__(
+        self,
+        *,
+        criterion="mse",
+        splitter="random",
+        max_depth=None,
+        min_samples_split=2,
+        min_samples_leaf=1,
+        min_weight_fraction_leaf=0.0,
+        max_features=None,
+        random_state=None,
+        min_impurity_decrease=0.0,
+        min_impurity_split=None,
+    ):
+
+        super().__init__(
+            criterion=criterion,
+            splitter=splitter,
+            max_depth=max_depth,
+            min_samples_split=min_samples_split,
+            min_samples_leaf=min_samples_leaf,
+            min_weight_fraction_leaf=min_weight_fraction_leaf,
+            max_features=max_features,
+            min_impurity_decrease=min_impurity_decrease,
+            min_impurity_split=min_impurity_split,
+            random_state=random_state,
+        )
diff --git a/docs/api_reference.rst b/docs/api_reference.rst
index db221bf..be7476c 100644
--- a/docs/api_reference.rst
+++ b/docs/api_reference.rst
@@ -12,3 +12,13 @@ CascadeForestClassifier
     :show-inheritance:
     :no-undoc-members:
     :member-order: bysource
+
+CascadeForestRegressor
+-----------------------
+
+.. autoclass:: deepforest.CascadeForestRegressor
+    :members:
+    :inherited-members:
+    :show-inheritance:
+    :no-undoc-members:
+    :member-order: bysource
diff --git a/docs/experiments.rst b/docs/experiments.rst
index 79ba98a..803c2be 100644
--- a/docs/experiments.rst
+++ b/docs/experiments.rst
@@ -2,7 +2,7 @@ Experiments
 ===========
 
 Baseline
-********
+--------
 For all experiments, we used 5 popular tree-based ensemble methods as baselines. Details on the baselines are listed in the following table:
 
 +------------------+---------------------------------------------------------------+
@@ -20,7 +20,7 @@ For all experiments, we used 5 popular tree-based ensemble methods as baselines.
 +------------------+---------------------------------------------------------------+
 
 Environment
-***********
+-----------
 For all experiments, we used a single linux server. Details on the specifications are listed in the table below. All processors were used for training and evaluating.
 
 +------------------+-----------------+--------+
@@ -30,9 +30,12 @@ For all experiments, we used a single linux server. Details on the specification
 +------------------+-----------------+--------+
 
 Setting
-*******
+-------
 We kept the number of decision trees the same across all baselines, while remaining hyper-parameters were set to their default values. Running scripts on reproducing all experiment results are available, please refer to this `Repo`_.
 
+Classification
+--------------
+
 Dataset
 *******
 
@@ -137,6 +140,60 @@ Some observations are listed as follow:
 * Histogram-based GBDT (e.g., :class:`HGBDT`, :class:`XGB HIST`, :class:`LightGBM`) are typically faster mainly because decision tree in GBDT tends to have a much smaller tree depth;
 * With the number of input dimensions increasing (e.g., on mnist and fashion-mnist), random forest and deep forest can be faster.
 
+Regression
+----------
+
+Dataset
+*******
+
+We have also collected four datasets on univariate regression for a comparison on the regression problem.
+
++------------------+------------+-----------+------------+
+|       Name       | # Training | # Testing | # Features |
++==================+============+===========+============+
+|     `abalone`_   |    2,799   |   1,378   |      8     |
++------------------+------------+-----------+------------+
+|    `cpusmall`_   |    5,489   |   2,703   |     12     |
++------------------+------------+-----------+------------+
+|     `boston`_    |     379    |    127    |     13     |
++------------------+------------+-----------+------------+
+|    `diabetes`_   |     303    |    139    |     10     |
++------------------+------------+-----------+------------+
+
+Testing Mean Squared Error
+**************************
+
+The table below shows the testing mean squared error of each method, with the best result on each dataset **bolded**. Each experiment was conducted over 5 independently trials, and the average result was reported.
+
++----------+-----------+---------+-----------+----------+----------+-------------+
+|   Name   |     RF    |  HGBDT  | XGB EXACT | XGB HIST | LightGBM | Deep Forest |
++==========+===========+=========+===========+==========+==========+=============+
+|  abalone |    4.79   |   5.40  |    5.73   |   5.75   |   5.60   |   **4.66**  |
++----------+-----------+---------+-----------+----------+----------+-------------+
+| cpusmall |    8.31   |   9.01  |    9.86   |   11.82  |   8.99   |   **7.15**  |
++----------+-----------+---------+-----------+----------+----------+-------------+
+|  boston  | **16.61** |  20.68  |   20.61   |   19.65  |   20.27  |    19.87    |
++----------+-----------+---------+-----------+----------+----------+-------------+
+| diabetes |  3796.62  | 4333.66 |  4337.15  |  4303.96 |  4435.95 | **3431.01** |
++----------+-----------+---------+-----------+----------+----------+-------------+
+
+Runtime
+*******
+
+Runtime in seconds reported in the table below covers both the training stage and evaluating stage.
+
++----------+------+-------+-----------+----------+----------+-------------+
+|   Name   |  RF  | HGBDT | XGB EXACT | XGB HIST | LightGBM | Deep Forest |
++==========+======+=======+===========+==========+==========+=============+
+|  abalone | 0.53 |  1.57 |    0.47   |   0.50   |   0.17   |     1.29    |
++----------+------+-------+-----------+----------+----------+-------------+
+| cpusmall | 1.87 |  3.59 |    1.71   |   1.25   |   0.36   |     2.06    |
++----------+------+-------+-----------+----------+----------+-------------+
+|  boston  | 0.70 |  1.75 |    0.19   |   0.22   |   0.20   |     1.45    |
++----------+------+-------+-----------+----------+----------+-------------+
+| diabetes | 0.37 |  0.66 |    0.14   |   0.18   |   0.06   |     1.09    |
++----------+------+-------+-----------+----------+----------+-------------+
+
 .. _`Random Forest`: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
 
 .. _`HGBDT`: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html
@@ -170,3 +227,11 @@ Some observations are listed as follow:
 .. _`mnist`: https://keras.io/api/datasets/mnist/
 
 .. _`fashion mnist`: https://keras.io/api/datasets/fashion_mnist/
+
+.. _`abalone`: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html#abalone
+
+.. _`cpusmall`: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html#cpusmall
+
+.. _`boston`: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html
+
+.. _`diabetes`: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html
diff --git a/docs/index.rst b/docs/index.rst
index 7c6fe9a..eceacaf 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -53,6 +53,23 @@ Quickstart
     print("Testing Accuracy: {:.3f} %".format(acc))
     >>> Testing Accuracy: 98.667 %
 
+.. code-block:: python
+
+    from sklearn.datasets import load_boston
+    from sklearn.model_selection import train_test_split
+    from sklearn.metrics import mean_squared_error
+
+    from deepforest import CascadeForestRegressor
+
+    X, y = load_boston(return_X_y=True)
+    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
+    model = CascadeForestRegressor(random_state=1)
+    model.fit(X_train, y_train)
+    y_pred = model.predict(X_test)
+    mse = mean_squared_error(y_test, y_pred)
+    print("\nTesting MSE: {:.3f}".format(mse))
+    >>> Testing MSE: 8.068
+
 Resources
 ---------
 


Test patch:
diff --git a/tests/test_forest.py b/tests/test_forest.py
index ab09d71..0274980 100644
--- a/tests/test_forest.py
+++ b/tests/test_forest.py
@@ -2,11 +2,13 @@
 
 from deepforest import RandomForestClassifier
 from deepforest import ExtraTreesClassifier
+from deepforest import RandomForestRegressor
+from deepforest import ExtraTreesRegressor
 from deepforest.forest import _get_n_samples_bootstrap
 
 # Load utils
 from sklearn.ensemble._hist_gradient_boosting.binning import _BinMapper
-from sklearn.datasets import load_iris, load_wine
+from sklearn.datasets import load_iris, load_wine, load_boston
 from sklearn.ensemble._forest import (
     _get_n_samples_bootstrap as sklearn_get_n_samples_bootstrap,
 )
@@ -51,7 +53,7 @@ def test_n_samples_bootstrap_invalid_type():
 
 
 @pytest.mark.parametrize("load_func", [load_iris, load_wine])
-def test_forest_workflow(load_func):
+def test_forest_classifier_workflow(load_func):
 
     n_estimators = 100  # to avoid oob warning
     random_state = 42
@@ -77,3 +79,32 @@ def test_forest_workflow(load_func):
 
     model.fit(X_binned, y)
     model.predict(X_binned)
+
+
+@pytest.mark.parametrize("load_func", [load_boston])
+def test_forest_regressor_workflow(load_func):
+
+    n_estimators = 100  # to avoid oob warning
+    random_state = 42
+
+    X, y = load_func(return_X_y=True)
+
+    # Data binning
+    binner = _BinMapper(random_state=random_state)
+    X_binned = binner.fit_transform(X)
+
+    # Random Forest
+    model = RandomForestRegressor(
+        n_estimators=n_estimators, random_state=random_state
+    )
+
+    model.fit(X_binned, y)
+    model.predict(X_binned)
+
+    # Extremely Random Forest
+    model = ExtraTreesRegressor(
+        n_estimators=n_estimators, random_state=random_state
+    )
+
+    model.fit(X_binned, y)
+    model.predict(X_binned)
diff --git a/tests/test_layer_estimator.py b/tests/test_layer_estimator.py
index 9673955..425f93c 100644
--- a/tests/test_layer_estimator.py
+++ b/tests/test_layer_estimator.py
@@ -5,7 +5,7 @@
 
 # Load utils
 from sklearn.ensemble._hist_gradient_boosting.binning import _BinMapper
-from sklearn.datasets import load_digits
+from sklearn.datasets import load_digits, load_boston
 from sklearn.model_selection import train_test_split
 
 
@@ -43,11 +43,11 @@
 }
 
 
-def test_layer_properties_after_fitting():
+def test_classifier_layer_properties_after_fitting():
 
     layer = Layer(**layer_kwargs)
     X_aug = layer.fit_transform(X_train, y_train)
-    y_pred_full = layer.predict_full(X_test)
+    y_pred_full = layer.predict_full(X_test, is_classifier=True)
 
     # n_trees
     assert (
@@ -61,6 +61,32 @@ def test_layer_properties_after_fitting():
     assert y_pred_full.shape[1] == expect_dim
 
 
+def test_regressor_layer_properties_after_fitting():
+    # Load data and binning
+    X, y = load_boston(return_X_y=True)
+    binner = _BinMapper(random_state=142)
+    X_binned = binner.fit_transform(X)
+
+    X_train, X_test, y_train, y_test = train_test_split(
+        X_binned, y, test_size=0.42, random_state=42
+    )
+    layer = Layer(**layer_kwargs)
+    layer.is_classifier = False
+    X_aug = layer.fit_transform(X_train, y_train)
+    y_pred_full = layer.predict_full(X_test, is_classifier=False)
+
+    # n_trees
+    assert (
+        layer.n_trees_
+        == 2 * layer_kwargs["n_estimators"] * layer_kwargs["n_trees"]
+    )
+
+    # Output dim
+    expect_dim = 2 * layer_kwargs["n_estimators"]
+    assert X_aug.shape[1] == expect_dim
+    assert y_pred_full.shape[1] == expect_dim
+
+
 @pytest.mark.parametrize(
     "param", [(0, {"n_estimators": 0}), (1, {"n_trees": 0})]
 )
diff --git a/tests/test_model.py b/tests/test_model_classifier.py
similarity index 95%
rename from tests/test_model.py
rename to tests/test_model_classifier.py
index 582af0f..b304de4 100644
--- a/tests/test_model.py
+++ b/tests/test_model_classifier.py
@@ -170,8 +170,8 @@ def test_model_sample_weight():
     assert_array_equal(y_pred_no_sample_weight, y_pred_equal_sample_weight)
 
     model = CascadeForestClassifier(**case_kwargs)
-    sample_weight = np.where(y_train == 0, 0.1, y_train)
-    model.fit(X_train, y_train, sample_weight=y_train)
+    sample_weight = np.where(y_train == 0, 1, 10)
+    model.fit(X_train, y_train, sample_weight=sample_weight)
     y_pred_skewed_sample_weight = model.predict(X_test)
 
     # Make sure the different predictions with None and equal sample_weight
@@ -237,13 +237,15 @@ def test_model_invalid_training_params(param):
 
 
 @pytest.mark.parametrize("predictor", ["forest", "xgboost", "lightgbm"])
-def test_predictor_normal(predictor):
-    deepforest.cascade._build_predictor(predictor, n_estimators=1, n_outputs=2)
+def test_classifier_predictor_normal(predictor):
+    deepforest.cascade._build_classifier_predictor(
+        predictor, n_estimators=1, n_outputs=2
+    )
 
 
-def test_predictor_unknown():
+def test_classifier_predictor_unknown():
     with pytest.raises(NotImplementedError) as excinfo:
-        deepforest.cascade._build_predictor(
+        deepforest.cascade._build_classifier_predictor(
             "unknown", n_estimators=1, n_outputs=2
         )
     assert "name of the predictor should be one of" in str(excinfo.value)
diff --git a/tests/test_model_regressor.py b/tests/test_model_regressor.py
new file mode 100644
index 0000000..1243e09
--- /dev/null
+++ b/tests/test_model_regressor.py
@@ -0,0 +1,231 @@
+import copy
+import pytest
+import shutil
+import numpy as np
+from numpy.testing import assert_array_equal
+from sklearn.datasets import load_boston
+from sklearn.model_selection import train_test_split
+
+import deepforest
+from deepforest import CascadeForestRegressor
+from deepforest.cascade import _get_predictor_kwargs
+
+
+save_dir = "./tmp"
+
+# Load data
+X, y = load_boston(return_X_y=True)
+X_train, X_test, y_train, y_test = train_test_split(
+    X, y, test_size=0.42, random_state=42
+)
+
+# Parameters
+toy_kwargs = {
+    "n_bins": 10,
+    "bin_subsample": 2e5,
+    "max_layers": 10,
+    "n_estimators": 1,
+    "n_trees": 100,
+    "max_depth": 3,
+    "min_samples_leaf": 1,
+    "use_predictor": True,
+    "predictor": "forest",
+    "predictor_kwargs": {},
+    "n_tolerant_rounds": 2,
+    "delta": 1e-5,
+    "n_jobs": -1,
+    "random_state": 0,
+    "verbose": 2,
+}
+
+kwargs = {
+    "n_bins": 255,
+    "bin_subsample": 2e5,
+    "max_layers": 10,
+    "n_estimators": 2,
+    "n_trees": 100,
+    "max_depth": None,
+    "min_samples_leaf": 1,
+    "use_predictor": True,
+    "predictor": "forest",
+    "predictor_kwargs": {},
+    "n_tolerant_rounds": 2,
+    "delta": 1e-5,
+    "n_jobs": -1,
+    "random_state": 0,
+    "verbose": 2,
+}
+
+
+@pytest.mark.parametrize(
+    "test_input,expected",
+    [
+        (
+            {"predictor_kwargs": {}, "n_job": 2},
+            {"n_job": 2},
+        ),
+        (
+            {"predictor_kwargs": {"n_job": 3}, "n_job": 2},
+            {"n_job": 3},
+        ),
+        (
+            {"predictor_kwargs": {"iter": 4}, "n_job": 2},
+            {"iter": 4, "n_job": 2},
+        ),
+    ],
+)
+def test_predictor_kwargs_overwrite(test_input, expected):
+    assert _get_predictor_kwargs(**test_input) == expected
+
+
+def test_model_properties_after_fitting():
+    """Check the model properties after fitting a deep forest model."""
+    model = CascadeForestRegressor(**toy_kwargs)
+    model.fit(X_train, y_train)
+
+    assert len(model) == model.n_layers_
+
+    assert model[0] is model._get_layer(0)
+
+    with pytest.raises(ValueError) as excinfo:
+        model._get_layer(model.n_layers_)
+    assert "The layer index should be in the range" in str(excinfo.value)
+
+    with pytest.raises(RuntimeError) as excinfo:
+        model._set_layer(0, None)
+    assert "already exists in the internal container" in str(excinfo.value)
+
+    with pytest.raises(ValueError) as excinfo:
+        model._get_binner(model.n_layers_ + 1)
+    assert "The binner index should be in the range" in str(excinfo.value)
+
+    with pytest.raises(RuntimeError) as excinfo:
+        model._set_binner(0, None)
+    assert "already exists in the internal container" in str(excinfo.value)
+
+
+def test_model_workflow_partial_mode():
+    """Run the workflow of deep forest with a local buffer."""
+
+    case_kwargs = copy.deepcopy(kwargs)
+    case_kwargs.update({"partial_mode": True})
+
+    model = CascadeForestRegressor(**case_kwargs)
+    model.fit(X_train, y_train)
+
+    # Predictions before saving
+    y_pred_before = model.predict(X_test).astype(np.float32)
+
+    # Save and Reload
+    model.save(save_dir)
+
+    model = CascadeForestRegressor(**case_kwargs)
+    model.load(save_dir)
+
+    # Predictions after loading
+    y_pred_after = model.predict(X_test).astype(np.float32)
+
+    # Make sure the same predictions before and after model serialization
+    assert_array_equal(y_pred_before, y_pred_after)
+
+    model.clean()  # clear the buffer
+    shutil.rmtree(save_dir)
+
+
+def test_model_workflow_in_memory():
+    """Run the workflow of deep forest with in-memory mode."""
+
+    case_kwargs = copy.deepcopy(kwargs)
+    case_kwargs.update({"partial_mode": False})
+
+    model = CascadeForestRegressor(**case_kwargs)
+    model.fit(X_train, y_train)
+
+    # Predictions before saving
+    y_pred_before = model.predict(X_test).astype(np.float32)
+
+    # Save and Reload
+    model.save(save_dir)
+
+    model = CascadeForestRegressor(**case_kwargs)
+    model.load(save_dir)
+
+    # Make sure the same predictions before and after model serialization
+    y_pred_after = model.predict(X_test).astype(np.float32)
+
+    assert_array_equal(y_pred_before, y_pred_after)
+
+    shutil.rmtree(save_dir)
+
+
+@pytest.mark.parametrize(
+    "param",
+    [
+        (0, {"max_layers": 0}),
+        (1, {"n_tolerant_rounds": 0}),
+        (2, {"delta": -1}),
+    ],
+)
+def test_model_invalid_training_params(param):
+    case_kwargs = copy.deepcopy(toy_kwargs)
+    case_kwargs.update(param[1])
+
+    model = CascadeForestRegressor(**case_kwargs)
+
+    with pytest.raises(ValueError) as excinfo:
+        model.fit(X_train, y_train)
+
+    if param[0] == 0:
+        assert "max_layers" in str(excinfo.value)
+    elif param[0] == 1:
+        assert "n_tolerant_rounds" in str(excinfo.value)
+    elif param[0] == 2:
+        assert "delta " in str(excinfo.value)
+
+
+@pytest.mark.parametrize("predictor", ["forest", "xgboost", "lightgbm"])
+def test_regressor_predictor_normal(predictor):
+    deepforest.cascade._build_regressor_predictor(
+        predictor, n_estimators=1, n_outputs=2
+    )
+
+
+def test_regressor_predictor_unknown():
+    with pytest.raises(NotImplementedError) as excinfo:
+        deepforest.cascade._build_regressor_predictor(
+            "unknown", n_estimators=1, n_outputs=2
+        )
+    assert "name of the predictor should be one of" in str(excinfo.value)
+
+
+def test_model_n_trees_non_positive():
+    case_kwargs = copy.deepcopy(toy_kwargs)
+    case_kwargs.update({"n_trees": 0})
+    model = CascadeForestRegressor(**case_kwargs)
+    with pytest.raises(ValueError) as excinfo:
+        model._set_n_trees(0)
+    assert "should be strictly positive." in str(excinfo.value)
+
+
+def test_model_n_trees_auto():
+    case_kwargs = copy.deepcopy(toy_kwargs)
+    case_kwargs.update({"n_trees": "auto"})
+    model = CascadeForestRegressor(**case_kwargs)
+
+    n_trees = model._set_n_trees(0)
+    assert n_trees == 100
+
+    n_trees = model._set_n_trees(2)
+    assert n_trees == 300
+
+    n_trees = model._set_n_trees(10)
+    assert n_trees == 500
+
+
+def test_model_n_trees_invalid():
+    case_kwargs = copy.deepcopy(toy_kwargs)
+    case_kwargs.update({"n_trees": [42]})
+    model = CascadeForestRegressor(**case_kwargs)
+    with pytest.raises(ValueError) as excinfo:
+        model._set_n_trees(0)
+    assert "Invalid value for n_trees." in str(excinfo.value)
diff --git a/tests/test_tree regressor.py b/tests/test_tree regressor.py
new file mode 100644
index 0000000..6baeb33
--- /dev/null
+++ b/tests/test_tree regressor.py
@@ -0,0 +1,71 @@
+import pytest
+from sklearn.datasets import load_boston
+from sklearn.ensemble._hist_gradient_boosting.binning import _BinMapper
+
+from deepforest import DecisionTreeRegressor
+
+
+X, y = load_boston(return_X_y=True)
+
+# Data binning
+binner = _BinMapper(random_state=42)
+X_binned = binner.fit_transform(X)
+
+
+def test_tree_properties_after_fitting():
+    tree = DecisionTreeRegressor()
+    tree.fit(X_binned, y)
+
+    assert tree.get_depth() == tree.tree_.max_depth
+    assert tree.n_leaves == tree.tree_.n_leaves
+    assert tree.n_internals == tree.tree_.n_internals
+
+
+def test_tree_fit_invalid_dtype():
+    tree = DecisionTreeRegressor()
+
+    with pytest.raises(RuntimeError) as execinfo:
+        tree.fit(X, y)
+    assert "The dtype of `X` should be `np.uint8`" in str(execinfo.value)
+
+
+def test_tree_fit_invalid_training_params():
+    tree = DecisionTreeRegressor(min_samples_leaf=0)
+    with pytest.raises(ValueError) as execinfo:
+        tree.fit(X_binned, y)
+    assert "min_samples_leaf must be at least 1" in str(execinfo.value)
+
+    tree = DecisionTreeRegressor(min_samples_leaf=0.6)
+    with pytest.raises(ValueError) as execinfo:
+        tree.fit(X_binned, y)
+    assert "or in (0, 0.5]" in str(execinfo.value)
+
+    tree = DecisionTreeRegressor(min_samples_split=1)
+    with pytest.raises(ValueError) as execinfo:
+        tree.fit(X_binned, y)
+    assert "min_samples_split must be an integer" in str(execinfo.value)
+
+    tree = DecisionTreeRegressor(max_features="unknown")
+    with pytest.raises(ValueError) as execinfo:
+        tree.fit(X_binned, y)
+    assert "Invalid value for max_features." in str(execinfo.value)
+
+    tree = DecisionTreeRegressor()
+    with pytest.raises(ValueError) as execinfo:
+        tree.fit(X_binned, y[:1])
+    assert "Number of labels=" in str(execinfo.value)
+
+    tree = DecisionTreeRegressor(min_weight_fraction_leaf=0.6)
+    with pytest.raises(ValueError) as execinfo:
+        tree.fit(X_binned, y)
+    assert "min_weight_fraction_leaf must in [0, 0.5]" in str(execinfo.value)
+
+    tree = DecisionTreeRegressor(max_depth=0)
+    with pytest.raises(ValueError) as execinfo:
+        tree.fit(X_binned, y)
+    assert "max_depth must be greater than zero." in str(execinfo.value)
+
+
+if __name__ == "__main__":
+
+    test_tree_properties_after_fitting()
diff --git a/tests/test_tree.py b/tests/test_tree_classifier.py
similarity index 100%
rename from tests/test_tree.py
rename to tests/test_tree_classifier.py
diff --git a/tests/test_tree_same.py b/tests/test_tree_same.py
index d380b05..a7909bf 100644
--- a/tests/test_tree_same.py
+++ b/tests/test_tree_same.py
@@ -9,18 +9,23 @@
 from sklearn.tree import (
     DecisionTreeClassifier as sklearn_DecisionTreeClassifier,
 )
+from sklearn.tree import (
+    DecisionTreeRegressor as sklearn_DecisionTreeRegressor,
+)
 from sklearn.tree import ExtraTreeClassifier as sklearn_ExtraTreeClassifier
+from sklearn.tree import ExtraTreeRegressor as sklearn_ExtraTreeRegressor
 
 # Load utils
 from sklearn.model_selection import train_test_split
 from sklearn.ensemble._hist_gradient_boosting.binning import _BinMapper
 
 # Toy classification datasets
-from sklearn.datasets import load_iris, load_wine
+from sklearn.datasets import load_iris, load_wine, load_boston
 
 from deepforest import DecisionTreeClassifier
 from deepforest import ExtraTreeClassifier
-
+from deepforest import DecisionTreeRegressor
+from deepforest import ExtraTreeRegressor
 
 test_size = 0.42
 random_state = 42
@@ -81,3 +86,54 @@ def test_extra_tree_classifier_proba(load_func):
 
     assert_array_equal(actual_pred, expected_pred)
     assert_array_equal(actual_proba, expected_proba)
+
+
+@pytest.mark.parametrize("load_func", [load_boston])
+def test_tree_regressor_pred(load_func):
+
+    X, y = load_func(return_X_y=True)
+    X_train, X_test, y_train, y_test = train_test_split(
+        X, y, test_size=test_size, random_state=random_state
+    )
+
+    # Data binning
+    binner = _BinMapper(random_state=random_state)
+    X_train_binned = binner.fit_transform(X_train)
+    X_test_binned = binner.transform(X_test)
+
+    # Ours
+    model = DecisionTreeRegressor(random_state=random_state)
+    model.fit(X_train_binned, y_train)
+    actual_pred = model.predict(X_test_binned)
+
+    # Sklearn
+    model = sklearn_DecisionTreeRegressor(random_state=random_state)
+    model.fit(X_train_binned, y_train)
+    expected_pred = model.predict(X_test_binned)
+
+    assert_array_equal(actual_pred, expected_pred)
+
+
+@pytest.mark.parametrize("load_func", [load_boston])
+def test_extra_tree_regressor_pred(load_func):
+    X, y = load_func(return_X_y=True)
+    X_train, X_test, y_train, y_test = train_test_split(
+        X, y, test_size=test_size, random_state=random_state
+    )
+
+    # Data binning
+    binner = _BinMapper(random_state=random_state)
+    X_train_binned = binner.fit_transform(X_train)
+    X_test_binned = binner.transform(X_test)
+
+    # Ours
+    model = ExtraTreeRegressor(random_state=random_state)
+    model.fit(X_train_binned, y_train)
+    actual_pred = model.predict(X_test_binned)
+
+    # Sklearn
+    model = sklearn_ExtraTreeRegressor(random_state=random_state)
+    model.fit(X_train_binned, y_train)
+    expected_pred = model.predict(X_test_binned)
+
+    assert_array_equal(actual_pred, expected_pred)