Repository: scikit-learn/scikit-learn (Python)

Issue:
Bug in Gradient Boosting: Feature Importances do not sum to 1
#### Description

I found conditions when Feature Importance values do not add up to 1 in ensemble tree methods, like Gradient Boosting Trees or AdaBoost Trees.  

This error occurs once the ensemble reaches a large number of estimators.  The exact conditions depend variously.  For example, the error shows up sooner with a smaller amount of training samples.  Or, if the depth of the tree is large.  

When this error appears, the predicted value seems to have converged.  But it’s unclear if the error is causing the predicted value not to change with more estimators.  In fact, the feature importance sum goes lower and lower with more estimators thereafter.  

Consequently, it's questionable if the tree ensemble code is functioning as expected.  

Here's sample code to reproduce this:

``` python
import numpy as np
from sklearn import datasets
from sklearn.ensemble import GradientBoostingRegressor

boston = datasets.load_boston()
X, Y = (boston.data, boston.target)

n_estimators = 720
# Note: From 712 onwards, the feature importance sum is less than 1

params = {'n_estimators': n_estimators, 'max_depth': 6, 'learning_rate': 0.1}
clf = GradientBoostingRegressor(**params)
clf.fit(X, Y)

feature_importance_sum = np.sum(clf.feature_importances_)
print("At n_estimators = %i, feature importance sum = %f" % (n_estimators , feature_importance_sum))
```

_Output:_

```
At n_estimators = 720, feature importance sum = 0.987500
```

In fact, if we examine the tree at each staged prediction, we'll see that the feature importance goes to 0 after we hit a certain number of estimators.  (For the code above, it's 712.)

Here's code to describe what I mean:

``` python
for i, tree in enumerate(clf.estimators_):
    feature_importance_sum = np.sum(tree[0].feature_importances_)
    print("At n_estimators = %i, feature importance sum = %f" % (i , feature_importance_sum))
```

_Output:_

```
...
At n_estimators = 707, feature importance sum = 1.000000
At n_estimators = 708, feature importance sum = 1.000000
At n_estimators = 709, feature importance sum = 1.000000
At n_estimators = 710, feature importance sum = 1.000000
At n_estimators = 711, feature importance sum = 0.000000
At n_estimators = 712, feature importance sum = 0.000000
At n_estimators = 713, feature importance sum = 0.000000
At n_estimators = 714, feature importance sum = 0.000000
At n_estimators = 715, feature importance sum = 0.000000
At n_estimators = 716, feature importance sum = 0.000000
At n_estimators = 717, feature importance sum = 0.000000
At n_estimators = 718, feature importance sum = 0.000000
...
```

I wonder if we’re hitting some floating point calculation error. 

BTW, I've posted this issue on the mailing list [Link](https://mail.python.org/pipermail/scikit-learn/2016-September/000508.html).  There aren't a lot of discussion, but others seem to think there's a bug here too.

Hope we can get this fixed or clarified.

Thank you!
-Doug
#### Versions

Windows-7;'Python', '2.7.9 ;'NumPy', '1.9.2';'SciPy', '0.15.1';'Scikit-Learn', '0.16.1' 



Discussion:
thanks for the report. ping @pprett ?

Observation:
The estimators with feature importance sum 0 have only 1 node which is being caused by the following [code](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/_tree.pyx#L228-L229)
`is_leaf = (is_leaf or (impurity <= min_impurity_split))`

`is_leaf` is changing to 1 in the first iteration for such trees.

For estimator number 710
`impurity = 0.00000010088593634382
min_impurity_split = 0.00000010000000000000`

For estimator number 711 (which has 0 feature importance sum)
`impurity = 0.00000009983122597550
min_impurity_split = 0.00000010000000000000`

The same behavior is there for all subsequent estimators.

Versions:
Ubuntu 16.04, Python: 2.7.10, Numpy: 1.11.2, Scipy: 0.18.1, Scikit-Learn: 0.19.dev0

ping @jmschrei maybe?

If the gini impurity at the first node is that low, you're not gaining anything by having more trees. I am unsure if this is a bug, except that it should maybe only look at the impurity of trees where a split improves things. 

So should these extra trees (containing only one node) be considered while computing feature importance?

@Naereen skipping them would be fine I guess, and would make them sum to one again?

IMO, while computing trees, when gini impurity of the first node of a tree is lower than the threshold, the program shouldn't move forward. The actual number of trees computed should be communicated to the user. In example above, program should stop at 711 and number of estimators should be 710.

on master, the first part of the issue seems fixed. This part of the code now returns 1.0 as expected.
```python
feature_importance_sum = np.sum(clf.feature_importances_)
print("At n_estimators = %i, feature importance sum = %f" % (n_estimators , feature_importance_sum))
```
It has been fixed in #11176.
However, for the second part of the issue, the problem is still present. the trees that add up to 0.0 actually contain only 1 node and `feature_importances_` does not really make sens for those trees. Maybe something should be done to avoid having those trees.
By 1 node, you mean just a single leaf node, or a single split node? Yes, a tree with a single leaf node should not be contributing to feature importances.
1 is the value returned by `node_count`. According to the docstring, it's internal nodes + leaves. So those trees are just a single leaf node.



Patch:
diff --git a/doc/whats_new/_contributors.rst b/doc/whats_new/_contributors.rst
index b9c5aee1164b2..bc99bc89f572d 100644
--- a/doc/whats_new/_contributors.rst
+++ b/doc/whats_new/_contributors.rst
@@ -174,4 +174,4 @@
 
 .. _Thomas Fan: https://github.com/thomasjpfan
 
-.. _Nicolas Hug: https://github.com/NicolasHug
\ No newline at end of file
+.. _Nicolas Hug: https://github.com/NicolasHug
diff --git a/doc/whats_new/v0.21.rst b/doc/whats_new/v0.21.rst
index d8fa26e42c9ae..1e3bb6d0af010 100644
--- a/doc/whats_new/v0.21.rst
+++ b/doc/whats_new/v0.21.rst
@@ -199,14 +199,16 @@ Support for Python 3.4 and below has been officially dropped.
   :class:`ensemble.RandomForestClassifier`,
   :class:`ensemble.RandomForestRegressor`,
   :class:`ensemble.ExtraTreesClassifier`,
-  :class:`ensemble.ExtraTreesRegressor`, and
-  :class:`ensemble.RandomTreesEmbedding`) now:
+  :class:`ensemble.ExtraTreesRegressor`,
+  :class:`ensemble.RandomTreesEmbedding`,
+  :class:`ensemble.GradientBoostingClassifier`, and
+  :class:`ensemble.GradientBoostingRegressor`) now:
 
   - sum up to ``1``
   - all the single node trees in feature importance calculation are ignored
   - in case all trees have only one single node (i.e. a root node),
     feature importances will be an array of all zeros.
-  :issue:`13636` by `Adrin Jalali`_.
+  :issue:`13636` and :issue:`13620` by `Adrin Jalali`_.
 
 - |Fix| Fixed a bug in :class:`ensemble.GradientBoostingClassifier` and
   :class:`ensemble.GradientBoostingRegressor`, which didn't support
diff --git a/sklearn/ensemble/gradient_boosting.py b/sklearn/ensemble/gradient_boosting.py
index 2a1f4937bb2a5..b4199526bb4e3 100644
--- a/sklearn/ensemble/gradient_boosting.py
+++ b/sklearn/ensemble/gradient_boosting.py
@@ -1709,17 +1709,26 @@ def feature_importances_(self):
         Returns
         -------
         feature_importances_ : array, shape (n_features,)
+            The values of this array sum to 1, unless all trees are single node
+            trees consisting of only the root node, in which case it will be an
+            array of zeros.
         """
         self._check_initialized()
 
-        total_sum = np.zeros((self.n_features_, ), dtype=np.float64)
-        for stage in self.estimators_:
-            stage_sum = sum(tree.tree_.compute_feature_importances(
-                normalize=False) for tree in stage) / len(stage)
-            total_sum += stage_sum
-
-        importances = total_sum / total_sum.sum()
-        return importances
+        relevant_trees = [tree
+                          for stage in self.estimators_ for tree in stage
+                          if tree.tree_.node_count > 1]
+        if not relevant_trees:
+            # degenerate case where all trees have only one node
+            return np.zeros(shape=self.n_features_, dtype=np.float64)
+
+        relevant_feature_importances = [
+            tree.tree_.compute_feature_importances(normalize=False)
+            for tree in relevant_trees
+        ]
+        avg_feature_importances = np.mean(relevant_feature_importances,
+                                          axis=0, dtype=np.float64)
+        return avg_feature_importances / np.sum(avg_feature_importances)
 
     def _validate_y(self, y, sample_weight):
         # 'sample_weight' is not utilised but is used for


Test patch:
diff --git a/sklearn/ensemble/tests/test_gradient_boosting.py b/sklearn/ensemble/tests/test_gradient_boosting.py
index 821124b73cb1c..18b8b625d8c08 100644
--- a/sklearn/ensemble/tests/test_gradient_boosting.py
+++ b/sklearn/ensemble/tests/test_gradient_boosting.py
@@ -1440,3 +1440,12 @@ def test_early_stopping_n_classes():
     # No error if we let training data be big enough
     gb = GradientBoostingClassifier(n_iter_no_change=5, random_state=0,
                                     validation_fraction=4)
+
+
+def test_gbr_degenerate_feature_importances():
+    # growing an ensemble of single node trees. See #13620
+    X = np.zeros((10, 10))
+    y = np.ones((10,))
+    gbr = GradientBoostingRegressor().fit(X, y)
+    assert_array_equal(gbr.feature_importances_,
+                       np.zeros(10, dtype=np.float64))