Repository: scikit-learn/scikit-learn
Pull request: [MRG+1] Catch cases for different class size in MLPClassifier with warm start (#7976) 
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->
Fixes #7976 

#### What does this implement/fix? Explain your changes.
This provides a test for different cases that throws an error when warm_start = True for MLPClassifier. Currently, vague errors are thrown when class size is different between the current fit and the previous fit. This fix will throw a clearer error message. 

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->



Problem statement:
MLPClasiffier produce error when trying to re-fit
Hi,

I am training a MLPClasiffier model twice, each time on a different data-set.
On the second iteration the fit method produce an error. Every time a different error.
I did the processes on various models, this is the only one that produce an error. 

this are the errors i get (each time a different one) - 

> _lbfgsb.error: failed in converting 7th argument `g' of _lbfgsb.setulb to C/Fortran array

> ValueError: total size of new array must be unchanged

 > ValueError: operands could not be broadcast together with shapes (154,100) (25,) (154,100) 

thanks

Hints from issue discussion:
Could you please provide a code snippet that produces this error?

On 5 December 2016 at 01:30, yairbr <notifications@github.com> wrote:

> Hi,
>
> I am training a MLPClasiffier model twice, each time on a different
> data-set.
> On the second iteration the fit method produce an error. Every time a
> different error.
> I did the processes on various models, this is the only one that produce
> an error.
>
> this are the errors i get (each time a different one) -
>
> _lbfgsb.error: failed in converting 7th argument `g' of _lbfgsb.setulb to
> C/Fortran array
>
> ValueError: total size of new array must be unchanged
>
> ValueError: operands could not be broadcast together with shapes (154,100)
> (25,) (154,100)
>
> thanks
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/7976>, or mute the
> thread
> <https://github.com/notifications/unsubscribe-auth/AAEz6y6977DM8kYwAKypCHGIQHhF0G-oks5rEs6igaJpZM4LDmPb>
> .
>

```python
import numpy as np
from random import randint
import pandas as pd
from sklearn.neural_network import MLPClassifier

model = MLPClassifier(hidden_layer_sizes=100, activation='logistic', solver='lbfgs',
                                  learning_rate='invscaling', power_t=0.7, warm_start=True, momentum=0.7,
                                  nesterovs_momentum=True)

df = pd.read_csv(<some csv source>)
data_set = df.values

no_lines = len(data_set )
test_rows_lst = []
for _ in range(0, 2):
    nrow = np.floor(0.1 * no_lines)
    n_rows_lst = []
    while nrow > 0:
        i = randint(0, no_lines - 1)
        if i not in n_rows_lst:
            nrow -= 1
            n_rows_lst.append(i)
    test_rows_lst.append(n_rows_lst)

# build train set
new_data_set = np.delete(data_set, test_rows_lst[0], axis=0)
X = new_data_set[:, :-1]
y = new_data_set[:, -1]

# build test set
rows_to_delete = data_set[test_rows_lst[0], :]
y_deleted_row = rows_to_delete[:, -1]
tst = rows_to_delete[:, :-1]

model.fit(X, y)
p = model.predict(tst)
print(p)

# build train set
new_data_set = np.delete(data_set, test_rows_lst[1], axis=0)
X = new_data_set[:, :-1]
y = new_data_set[:, -1]

# build test set
rows_to_delete = data_set[test_rows_lst[1], :]
y_deleted_row = rows_to_delete[:, -1]
tst = rows_to_delete[:, :-1]

model.fit(X, y)
p = model.predict(tst)
print(p)
```

Surely it doesn't depend on reading a CSV in. If possible, please confirm the error on some particular dataset (even a trivial one) and provide it.
yeah, it seems like the errors come from scypy and numpy (as googling them provide).
the error occurs on the second **model.fit(X, y)**.

this is a sample from my data set, still produce the error - 

> data_set = np.asarray([[51.4, 82283.0, -10.0, 115.0, 86.9, 175.0, 180.0, 240.0, 142.0], [53.8, 110996.0, -9.0, 95.0, 96.0, 200.0, 196.0, 300.0, 86.0], [56.6, 102971.0, -12.0, 80.0, 96.5, 180.0, 180.0, 300.0, 207.0], [53.6, 70571.0, -13.0, 140.0, 89.1, 190.0, 193.0, 300.0, 101.0], [59.4, 71836.0, -16.0, -350.0, 76.9, -150.0, -148.0, -25.0, -94.0], [59.1, 81301.0, -16.0, -25.0, 74.2, 120.0, 116.0, 300.0, 56.0], [58.1, 100279.0, -14.0, 145.0, 81.6, 210.0, 216.0, 350.0, 215.0], [54.2, 107822.0, -22.0, 130.0, 91.5, 197.0, 207.0, 330.0, 108.0], [54.8, 100067.0, -8.0, 175.0, 91.2, 250.0, 251.0, 350.0, 193.0], [56.7, 87437.0, -22.0, 100.0, 86.7, 204.0, 208.0, 300.0, 243.0], [55.2, 64975.0, -18.0, 150.0, 88.9, 145.0, 194.0, 300.0, 197.0], [57.3, 59688.0, -11.0, 165.0, 87.4, 200.0, 202.0, 250.0, 138.0], [54.4, 53216.0, -14.0, 125.0, 79.1, 170.0, 171.0, 250.0, 75.0], [53.8, 67576.0, -15.0, 55.0, 84.9, 175.0, 178.0, 300.0, 121.0], [54.7, 37178.0, -12.0, 71.0, 84.7, 141.0, 142.0, 225.0, 113.0], [54.5, 65278.0, -8.0, 89.0, 82.0, 125.0, 123.0, 180.0, 128.0], [52.9, 100135.0, -11.0, 50.0, 85.4, 120.0, 119.0, 200.0, 151.0], [51.2, 69177.0, -9.0, 72.0, 93.6, 123.0, 118.0, 180.0, 92.0], [49.5, 76773.0, -11.0, 40.0, 92.1, 100.0, 106.0, 190.0, 132.0], [51.4, 54643.0, -7.0, 0.0, 91.7, 100.0, 96.0, 164.0, 167.0], [49.3, 62975.0, -5.0, 20.0, 96.9, 150.0, 149.0, 225.0, 111.0], [52.3, 84414.0, -14.0, 38.0, 91.3, 95.0, 98.0, 165.0, 97.0], [50.9, 48997.0, -14.0, 70.0, 88.4, 130.0, 134.0, 240.0, 180.0], [54.7, 70672.0, -14.0, 40.0, 87.1, 120.0, 100.0, 168.0, 88.0], [55.0, 71115.0, -10.0, 45.0, 88.3, 132.0, 133.0, 224.0, 157.0], [56.0, 55726.0, -6.0, 85.0, 85.3, 125.0, 122.0, 155.0, 132.0], [52.9, 79489.0, -12.0, 35.0, 83.4, 100.0, 100.0, 140.0, 19.0], [52.0, 71739.0, -15.0, 15.0, 83.4, 100.0, 98.0, 175.0, 110.0], [50.9, 63114.0, -14.0, 10.0, 80.9, 85.0, 83.0, 125.0, 166.0], [50.8, 73140.0, -16.0, -5.0, 76.1, 80.0, 83.0, 195.0, 94.0], [50.7, 74986.0, -19.0, 5.0, 78.4, 76.0, 72.0, 160.0, -107.0], [48.3, 83091.0, -27.0, -110.0, 70.8, 20.0, 13.0, 70.0, -63.0], [48.6, 90015.0, -33.0, -150.0, 62.6, -75.0, -77.0, -18.0, -20.0], [47.7, 44416.0, -19.0, 34.0, 75.5, 71.0, 68.0, 106.0, 18.0], [49.6, 103522.0, -31.0, -150.0, 59.8, -60.0, -58.0, -10.0, -49.0], [50.2, 81770.0, -37.0, -130.0, 56.4, -60.0, -60.0, 48.0, -62.0], [50.0, 103312.0, -33.0, -150.0, 61.2, -75.0, -72.0, 0.0, -51.0], [49.9, 88736.0, -30.0, -150.0, 63.0, -75.0, -77.0, -40.0, -84.0]])

replace the dataset that reads from csv with this. 
I am occasionally getting the error with that dataset. Whether it errors is non-deterministic and independent of MLPClassifier's `random_state`. :(

Error is either:
```
sklearn/neural_network/multilayer_perceptron.py in _forward_pass(self, activations)
    103             activations[i + 1] = safe_sparse_dot(activations[i],
    104                                                  self.coefs_[i])
--> 105             activations[i + 1] += self.intercepts_[i]
    106
    107             # For the hidden layers

ValueError: operands could not be broadcast together with shapes (35,100) (34,) (35,100)
```
or
```
sklearn/neural_network/multilayer_perceptron.py in _fit_lbfgs(self, X, y, activations, deltas, coef_grads, intercept_grads, layer_units)
    467             iprint=iprint,
    468             pgtol=self.tol,
--> 469             args=(X, y, activations, deltas, coef_grads, intercept_grads))
    470
    471         self._unpack(optimal_parameters)

scipy/optimize/lbfgsb.py in fmin_l_bfgs_b(func, x0, fprime, args, approx_grad, bounds, m, factr, pgtol, epsilon, iprint, maxfun, maxiter, disp, callback, maxls)
    191
    192     res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,
--> 193                            **opts)
    194     d = {'grad': res['jac'],
    195          'task': res['message'],

scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, **unknown_options)
    319         _lbfgsb.setulb(m, x, low_bnd, upper_bnd, nbd, f, g, factr,
    320                        pgtol, wa, iwa, task, iprint, csave, lsave,
--> 321                        isave, dsave, maxls)
    322         task_str = task.tostring()
    323         if task_str.startswith(b'FG'):

error: failed in converting 7th argument `g' of _lbfgsb.setulb to C/Fortran array
```
I've not yet found "ValueError: total size of new array must be unchanged"
@yairbr, I note that you have `warm_start=True`, but you may fit with a different set of classes each time. Is that the case?
I hadn't noticed the randomisation in the data. This appears to be the source of the non-determinism.

The error occurs when there are different numbers of classes in the two calls to `fit`.

We should require (indeed we could have a common test) that the classes in `y` are a subset of `classes_` if `warm_start=True`. We do not require this, and land up in bad places...
great. thanks
@yairbr Just to clarify, for your case that means you'll get a better error message, but it'll still fail.
Can I help out with this?

Commit message: fixed pep8 violation

Patch to review:
diff --git a/sklearn/neural_network/multilayer_perceptron.py b/sklearn/neural_network/multilayer_perceptron.py
index b3b657e09c956..c6aff72d29bc3 100644
--- a/sklearn/neural_network/multilayer_perceptron.py
+++ b/sklearn/neural_network/multilayer_perceptron.py
@@ -908,6 +908,13 @@ def _validate_input(self, X, y, incremental):
             self._label_binarizer = LabelBinarizer()
             self._label_binarizer.fit(y)
             self.classes_ = self._label_binarizer.classes_
+        elif self.warm_start:
+            classes = unique_labels(y)
+            if set(classes) != set(self.classes_):
+                raise ValueError("warm_start can only be used where `y` has "
+                                 "the same classes as in the previous "
+                                 "call to fit. Previously got %s, `y` has %s" %
+                                 (self.classes_, classes))
         else:
             classes = unique_labels(y)
             if np.setdiff1d(classes, self.classes_, assume_unique=True):
@@ -939,6 +946,28 @@ def predict(self, X):
 
         return self._label_binarizer.inverse_transform(y_pred)
 
+    def fit(self, X, y):
+        """Fit the model to data matrix X and target(s) y.
+
+        Parameters
+        ----------
+        X : array-like or sparse matrix, shape (n_samples, n_features)
+            The input data.
+
+        y : array-like, shape (n_samples,) or (n_samples, n_outputs)
+            The target values (class labels in classification, real numbers in
+            regression).
+
+        Returns
+        -------
+        self : returns a trained MLP model.
+        """
+        if self.warm_start and hasattr(self, "classes_"):
+            incremental = True
+        else:
+            incremental = False
+        return self._fit(X, y, incremental=incremental)
+
     @property
     def partial_fit(self):
         """Fit the model to data matrix X and target y.
diff --git a/sklearn/neural_network/tests/test_mlp.py b/sklearn/neural_network/tests/test_mlp.py
index e54a02d31cc05..e01fcbe7ef6d5 100644
--- a/sklearn/neural_network/tests/test_mlp.py
+++ b/sklearn/neural_network/tests/test_mlp.py
@@ -12,7 +12,7 @@
 
 from numpy.testing import assert_almost_equal, assert_array_equal
 
-from sklearn.datasets import load_digits, load_boston
+from sklearn.datasets import load_digits, load_boston, load_iris
 from sklearn.datasets import make_regression, make_multilabel_classification
 from sklearn.exceptions import ConvergenceWarning
 from sklearn.externals.six.moves import cStringIO as StringIO
@@ -23,7 +23,8 @@
 from sklearn.preprocessing import StandardScaler, MinMaxScaler
 from scipy.sparse import csr_matrix
 from sklearn.utils.testing import (assert_raises, assert_greater, assert_equal,
-                                   assert_false, ignore_warnings)
+                                   assert_false, ignore_warnings,
+                                   assert_raise_message)
 
 
 np.seterr(all='warn')
@@ -49,6 +50,11 @@
 Xboston = StandardScaler().fit_transform(boston.data)[: 200]
 yboston = boston.target[:200]
 
+iris = load_iris()
+
+X_iris = iris.data
+y_iris = iris.target
+
 
 def test_alpha():
     # Test that larger alpha yields weights closer to zero
@@ -556,3 +562,58 @@ def test_adaptive_learning_rate():
     clf.fit(X, y)
     assert_greater(clf.max_iter, clf.n_iter_)
     assert_greater(1e-6, clf._optimizer.learning_rate)
+
+
+def test_warm_start():
+    X = X_iris[:150]
+    y = y_iris[:150]
+
+    y_2classes = np.array([0] * 75 + [1] * 75)
+    y_3classes = np.array([0] * 50 + [1] * 50 + [2] * 50)
+    y_3classes_alt = np.array([0] * 50 + [1] * 50 + [3] * 50)
+    y_4classes = np.array([0] * 37 + [1] * 37 + [2] * 38 + [3] * 38)
+    y_5classes = np.array([0] * 30 + [1] * 30 + [2] * 30 + [3] * 30 + [4] * 30)
+
+    with ignore_warnings(category=Warning):
+
+        # Test with 2 unique labels
+        clf = MLPClassifier(hidden_layer_sizes=2, solver='lbfgs',
+                            warm_start=True)
+        clf.fit(X, y)
+        message = ('warm_start can only be used where `y` has the same '
+                   'classes as in the previous call to fit.'
+                   ' Previously got [0 1 2], `y` has [0 1]')
+        assert_raise_message(ValueError, message, clf.fit, X, y_2classes)
+
+        # Test with 3 unique labels that are the same as original fit.
+        clf = MLPClassifier(hidden_layer_sizes=2, solver='lbfgs',
+                            warm_start=True)
+        clf.fit(X, y)
+        clf.fit(X, y_3classes)
+
+        # Test with 3 unique labels that are different from original fit.
+        clf = MLPClassifier(hidden_layer_sizes=2, solver='lbfgs',
+                            warm_start=True)
+        clf.fit(X, y)
+        message = ('warm_start can only be used where `y` has the same '
+                   'classes as in the previous call to fit.'
+                   ' Previously got [0 1 2], `y` has [0 1 3]')
+        assert_raise_message(ValueError, message, clf.fit, X, y_3classes_alt)
+
+        # Test with 4 unique labels
+        clf = MLPClassifier(hidden_layer_sizes=2, solver='lbfgs',
+                            warm_start=True)
+        clf.fit(X, y)
+        message = ('warm_start can only be used where `y` has the same '
+                   'classes as in the previous call to fit.'
+                   ' Previously got [0 1 2], `y` has [0 1 2 3]')
+        assert_raise_message(ValueError, message, clf.fit, X, y_4classes)
+
+        # Test with 5 unique labels. This is a non-regression error.
+        clf = MLPClassifier(hidden_layer_sizes=2, solver='lbfgs',
+                            warm_start=True)
+        clf.fit(X, y)
+        message = ('warm_start can only be used where `y` has the '
+                   'same classes as in the previous call to fit.'
+                   ' Previously got [0 1 2], `y` has [0 1 2 3 4]')
+        assert_raise_message(ValueError, message, clf.fit, X, y_5classes)


Code review comments:
diff_hunk: @@ -939,6 +946,28 @@ def predict(self, X):
 
         return self._label_binarizer.inverse_transform(y_pred)
 
+    def fit(self, X, y):
+        """Fit the model to data matrix X and target(s) y.
+
+        Parameters
+        ----------
+        X : array-like or sparse matrix, shape (n_samples, n_features)
+            The input data.
+
+        y : array-like, shape (n_samples,) or (n_samples, n_outputs)
+            The target values (class labels in classification, real numbers in
+            regression).
+
+        Returns
+        -------
+        self : returns a trained MLP model.
+        """
+        if self.warm_start and hasattr(self, "classes_"):
+            incremental = True
+        else:
+            incremental = False
+        return self._fit(X, y, incremental=incremental)
path: sklearn/neural_network/multilayer_perceptron.py
text: @user1:
can we put this in one line

```py
        return self._fit(X, y, incremental=(self.warm_start and
                                            hasattr(self, "classes_"))
```

diff_hunk: @@ -556,3 +562,58 @@ def test_adaptive_learning_rate():
     clf.fit(X, y)
     assert_greater(clf.max_iter, clf.n_iter_)
     assert_greater(1e-6, clf._optimizer.learning_rate)
+
+
+def test_warm_start():
+    X = X_iris[:150]
path: sklearn/neural_network/tests/test_mlp.py
text: @user1:
Why? `X_iris.shape[0]` is 150...

diff_hunk: @@ -556,3 +562,58 @@ def test_adaptive_learning_rate():
     clf.fit(X, y)
     assert_greater(clf.max_iter, clf.n_iter_)
     assert_greater(1e-6, clf._optimizer.learning_rate)
+
+
+def test_warm_start():
+    X = X_iris[:150]
+    y = y_iris[:150]
+
+    y_2classes = np.array([0] * 75 + [1] * 75)
+    y_3classes = np.array([0] * 50 + [1] * 50 + [2] * 50)
path: sklearn/neural_network/tests/test_mlp.py
text: @user1:
This is same as y_iris... Could you change the order / shuffle it maybe?

`np.array([0] * 40, [1] * 40, [2] * 70)`