Repository: scikit-learn/scikit-learn
Pull request: FIX switch to binary_crossentropy if a binary problem
Fixes #14858.

Raise a warning and switch to `binary_crossentropy` if it's a binary classification problem and `categorical_crossentropy` is given.

Problem statement:
HGBC with categorical_crossentropy fails silently on binary classification
```python
import numpy as np
from sklearn.experimental import enable_hist_gradient_boosting
from sklearn.ensemble import HistGradientBoostingClassifier

X = [[1, 0],
     [1, 0],
     [1, 0],
     [0, 1],
     [1, 1]]
y = [1, 1, 1, 0, 1]
gb = HistGradientBoostingClassifier(loss='categorical_crossentropy',
                                    min_samples_leaf=1)
gb.fit(X, y)
print(gb.predict([[1, 0]]))
print(gb.predict([[0, 1]]))
```

gives:

```
[0]
[0]
```

And `binary_crossentropy` works fine. `categorical_crossentropy` should either generalize or raise an error on binary classification.

Ping @NicolasHug @ogrisel

Hints from issue discussion:
Thanks for the report, I think changing 

```py
self.n_trees_per_iteration_ = 1 if n_classes <= 2 else n_classes
```
to
```
self.n_trees_per_iteration_ = n_classes
```

would make categorical-crossentropy behave like the log loss.

But I think we want to error in this case: categorical-crossentropy will be twice as slow. This is a bug right now

Submit a PR Adrin?
I'm happy to submit a PR, but do we want to have two trees for binary classification and categorical crossentropy and one tree for binary crossentropy? And then raise a warning if categorical crossentropy is used for binary classification?
> do we want to have two trees for binary classification and categorical crossentropy and one tree for binary crossentropy? 

I'd say no, we should just error. But no strong opinion. I don't see the point of allowing categorical crossentropy in binary classification when the log loss is equivalent and twice as fast. Especially considering the loss='auto' which is the default.

Commit message: switch to raising an error

Patch to review:
diff --git a/doc/whats_new/v0.22.rst b/doc/whats_new/v0.22.rst
index ce3174218679f..f935ad5d04056 100644
--- a/doc/whats_new/v0.22.rst
+++ b/doc/whats_new/v0.22.rst
@@ -136,6 +136,9 @@ Changelog
     avoid any data leak. :pr:`13933` by `Nicolas Hug`_.
   - |Fix| Fixed a bug where early stopping would break with string targets.
     :pr:`14710` by :user:`Guillaume Lemaitre <glemaitre>`.
+  - |Fix| :class:`ensemble.HistGradientBoostingClassifier` now raises an error
+    if ``categorical_crossentropy`` loss is given for a binary classification
+    problem. :pr:`14869` by `Adrin Jalali`_.
 
   Note that pickles from 0.21 will not work in 0.22.
 
diff --git a/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py b/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py
index 70a507d09c1c6..9b74c6ba93d8e 100644
--- a/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py
+++ b/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py
@@ -1045,4 +1045,10 @@ def _get_loss(self):
             else:
                 return _LOSSES['categorical_crossentropy']()
 
+        if (self.loss == 'categorical_crossentropy' and
+                self.n_trees_per_iteration_ == 1):
+            raise ValueError("'categorical_crossentropy' is not suitable for "
+                             "a binary classification problem. Please use "
+                             "'auto' or 'binary_crossentropy' instead.")
+
         return _LOSSES[self.loss]()
diff --git a/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py b/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py
index 5de49ef740295..7d01940ba5ad5 100644
--- a/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py
+++ b/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py
@@ -417,6 +417,17 @@ def test_infinite_values_missing_values():
     assert stump_clf.fit(X, y_isnan).score(X, y_isnan) == 1
 
 
+def test_crossentropy_binary_problem():
+    # categorical_crossentropy should only be used if there are more than two
+    # classes present. PR #14869
+    X = [[1], [0]]
+    y = [0, 1]
+    gbrt = HistGradientBoostingClassifier(loss='categorical_crossentropy')
+    with pytest.raises(ValueError,
+                       match="'categorical_crossentropy' is not suitable for"):
+        gbrt.fit(X, y)
+
+
 @pytest.mark.parametrize("scoring", [None, 'loss'])
 def test_string_target_early_stopping(scoring):
     # Regression tests for #14709 where the targets need to be encoded before


Code review comments:
diff_hunk: @@ -1045,4 +1045,10 @@ def _get_loss(self):
             else:
                 return _LOSSES['categorical_crossentropy']()
 
+        if (self.loss == 'categorical_crossentropy' and
path: sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py
text: @user1:
Nit: error checking at the beginning of the `_get_loss`?