Repository: scikit-learn/scikit-learn
Pull request: MAINT: Remove redundant code in MLP
Closes https://github.com/scikit-learn/scikit-learn/issues/14807
Closes https://github.com/scikit-learn/scikit-learn/issues/11952

Alternative to https://github.com/scikit-learn/scikit-learn/pull/10665


This makes sure that it's possible to use a custom activation function registering new ones in `sklearn.neural_network.multilayer_perceptron.{ACTIVATIONS,DERIVATIVES}`.

It was almost working before, except for a manual check of supported activation (instead of using the `ACTIVATIONS` variable), so this mostly adds a test.

Not sure if we want to document it (and if so where).  

Problem statement:
Can I use leaky-relu function for MLPs?
The Leaky-ReLU activation function for MLPs (max(x, alpha *x) where alpha is a small number around 0.01) is rumoured to work better than ReLU in some cases. Unfortunately, I din't find any way to use if tor MPLClassifier or MLPRegressor. I can try to implement it myself, but I am still not familiar enough with SciKit-Learn source code to do it quickly. Maybe there is a proper way to use it?



Custom Activation Function added in Sklearn _base.py but getting error
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-19-63a02213db29> in <module>
      6 labels=['relu' , 'sigmoid', 'tanh', 'LeakyRelu']
      7 for clf, label in zip(clfs, labels):
----> 8     clf.fit(X_train, y_train)
      9 #     print('TRAINING FINISH go for TESTING')
     10 

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in fit(self, X, y)
    975         """
    976         return self._fit(X, y, incremental=(self.warm_start and
--> 977                                             hasattr(self, "classes_")))
    978 
    979     @property

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in _fit(self, X, y, incremental)
    317 
    318         # Validate input parameters.
--> 319         self._validate_hyperparameters()
    320         if np.any(np.array(hidden_layer_sizes) <= 0):
    321             raise ValueError("hidden_layer_sizes must be > 0, got %s." %

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in _validate_hyperparameters(self)
    418             raise ValueError("The activation '%s' is not supported. Supported "
    419                              "activations are %s." % (self.activation,
--> 420                                                       supported_activations))
    421         if self.learning_rate not in ["constant", "invscaling", "adaptive"]:
    422             raise ValueError("learning rate %s is not supported. " %

ValueError: The activation 'LeakyRelu' is not supported. Supported activations are ('identity', 'logistic', 'tanh', 'relu').
```

Hints from issue discussion:
See #10665
Please provide a fully reproducible code snippet.

I have tried to add Leaky Relu function in _base.py file . I am attaching it.

```py
"""Utilities for the neural network modules
"""

# Author: Issam H. Laradji <issam.laradji@gmail.com>
# License: BSD 3 clause

import numpy as np

from scipy.special import expit as logistic_sigmoid


def identity(X):
    """Simply return the input array.

    Parameters
    ----------
    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        Data, where n_samples is the number of samples
        and n_features is the number of features.

    Returns
    -------
    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        Same as the input data.
    """
    return X


def logistic(X):
    """Compute the logistic function inplace.

    Parameters
    ----------
    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        The input data.

    Returns
    -------
    X_new : {array-like, sparse matrix}, shape (n_samples, n_features)
        The transformed data.
    """
    return logistic_sigmoid(X, out=X)


def tanh(X):
    """Compute the hyperbolic tan function inplace.

    Parameters
    ----------
    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        The input data.

    Returns
    -------
    X_new : {array-like, sparse matrix}, shape (n_samples, n_features)
        The transformed data.
    """
    return np.tanh(X, out=X)


def relu(X):
    """Compute the rectified linear unit function inplace.

    Parameters
    ----------
    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        The input data.

    Returns
    -------
    X_new : {array-like, sparse matrix}, shape (n_samples, n_features)
        The transformed data.
    """
    np.clip(X, 0, np.finfo(X.dtype).max, out=X)
    return X

def leakyrelu(X):
    """Compute the leaky rectified linear unit function inplace.
    leakyrely(x) = x if x > 0
    leakyrelu(x) = 0.01*x if x <= 0
    """
    # np.clip(X, 0.01, np.finfo(X.dtype).max, out=X)
    return np.max(0.01*X, X)

# def elu():
# 	"""Compute the ELU function inplace.
# 		elu(x) = x if x>0
# 		elu(x) = alpha*(exp(x)-1) if x <= 0
# 		generally alpha=0.01
# 	"""

# 	return 


def softmax(X):
    """Compute the K-way softmax function inplace.

    Parameters
    ----------
    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        The input data.

    Returns
    -------
    X_new : {array-like, sparse matrix}, shape (n_samples, n_features)
        The transformed data.
    """
    tmp = X - X.max(axis=1)[:, np.newaxis]
    np.exp(tmp, out=X)
    X /= X.sum(axis=1)[:, np.newaxis]

    return X


ACTIVATIONS = {'identity': identity, 'tanh': tanh, 'logistic': logistic,
               'relu': relu, 'softmax': softmax, 'leakyrelu': leakyrelu}


def inplace_identity_derivative(Z, delta):
    """Apply the derivative of the identity function: do nothing.

    Parameters
    ----------
    Z : {array-like, sparse matrix}, shape (n_samples, n_features)
        The data which was output from the identity activation function during
        the forward pass.

    delta : {array-like}, shape (n_samples, n_features)
         The backpropagated error signal to be modified inplace.
    """
    # Nothing to do


def inplace_logistic_derivative(Z, delta):
    """Apply the derivative of the logistic sigmoid function.

    It exploits the fact that the derivative is a simple function of the output
    value from logistic function.

    Parameters
    ----------
    Z : {array-like, sparse matrix}, shape (n_samples, n_features)
        The data which was output from the logistic activation function during
        the forward pass.

    delta : {array-like}, shape (n_samples, n_features)
         The backpropagated error signal to be modified inplace.
    """
    delta *= Z
    delta *= (1 - Z)


def inplace_tanh_derivative(Z, delta):
    """Apply the derivative of the hyperbolic tanh function.

    It exploits the fact that the derivative is a simple function of the output
    value from hyperbolic tangent.

    Parameters
    ----------
    Z : {array-like, sparse matrix}, shape (n_samples, n_features)
        The data which was output from the hyperbolic tangent activation
        function during the forward pass.

    delta : {array-like}, shape (n_samples, n_features)
         The backpropagated error signal to be modified inplace.
    """
    delta *= (1 - Z ** 2)


def inplace_relu_derivative(Z, delta):
    """Apply the derivative of the relu function.

    It exploits the fact that the derivative is a simple function of the output
    value from rectified linear units activation function.

    Parameters
    ----------
    Z : {array-like, sparse matrix}, shape (n_samples, n_features)
        The data which was output from the rectified linear units activation
        function during the forward pass.

    delta : {array-like}, shape (n_samples, n_features)
         The backpropagated error signal to be modified inplace.
    """
    delta[Z == 0] = 0

def inplace_leakyrelu_derivative(Z, delta):
    """Apply the derivative of the leaky relu function.

    It exploits the fact that the derivative is a simple function of the output
    value from leaky rectified linear units activation function.

    Parameters
    ----------
    Z : {array-like, sparse matrix}, shape (n_samples, n_features)
        The data which was output from the rectified linear units activation
        function during the forward pass.

    delta : {array-like}, shape (n_samples, n_features)
         The backpropagated error signal to be modified inplace.
    """
    delta[Z >= 0] = 0.01


DERIVATIVES = {'identity': inplace_identity_derivative,
               'tanh': inplace_tanh_derivative,
	       'logistic': inplace_logistic_derivative,
               'relu': inplace_relu_derivative,
               'leakyrelu': inplace_leakyrelu_derivative}


def squared_loss(y_true, y_pred):
    """Compute the squared loss for regression.

    Parameters
    ----------
    y_true : array-like or label indicator matrix
        Ground truth (correct) values.

    y_pred : array-like or label indicator matrix
        Predicted values, as returned by a regression estimator.

    Returns
    -------
    loss : float
        The degree to which the samples are correctly predicted.
    """
    return ((y_true - y_pred) ** 2).mean() / 2


def log_loss(y_true, y_prob):
    """Compute Logistic loss for classification.

    Parameters
    ----------
    y_true : array-like or label indicator matrix
        Ground truth (correct) labels.

    y_prob : array-like of float, shape = (n_samples, n_classes)
        Predicted probabilities, as returned by a classifier's
        predict_proba method.

    Returns
    -------
    loss : float
        The degree to which the samples are correctly predicted.
    """
    y_prob = np.clip(y_prob, 1e-10, 1 - 1e-10)

    if y_prob.shape[1] == 1:
        y_prob = np.append(1 - y_prob, y_prob, axis=1)

    if y_true.shape[1] == 1:
        y_true = np.append(1 - y_true, y_true, axis=1)

    return -np.sum(y_true * np.log(y_prob)) / y_prob.shape[0]


def binary_log_loss(y_true, y_prob):
    """Compute binary logistic loss for classification.

    This is identical to log_loss in binary classification case,
    but is kept for its use in multilabel case.

    Parameters
    ----------
    y_true : array-like or label indicator matrix
        Ground truth (correct) labels.

    y_prob : array-like of float, shape = (n_samples, n_classes)
        Predicted probabilities, as returned by a classifier's
        predict_proba method.

    Returns
    -------
    loss : float
        The degree to which the samples are correctly predicted.
    """
    y_prob = np.clip(y_prob, 1e-10, 1 - 1e-10)

    return -np.sum(y_true * np.log(y_prob) +
                   (1 - y_true) * np.log(1 - y_prob)) / y_prob.shape[0]


LOSS_FUNCTIONS = {'squared_loss': squared_loss, 'log_loss': log_loss,
                  'binary_log_loss': binary_log_loss}
```
@ArijitN5 It preferable to use code formatting on github with https://help.github.com/en/articles/creating-and-highlighting-code-blocks

Currently setting a custom activation function wouldn't work, but a fix is proposed https://github.com/scikit-learn/scikit-learn/pull/14815 (that should work without having to edit scikit-learn files)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-530a8c6bb846> in <module>
     12 mlp = MLPClassifier(activation='identity2')
     13 
---> 14 mlp.fit([[0, 1], [1, 1]], [0, 1])

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in fit(self, X, y)
    980         """
    981         return self._fit(X, y, incremental=(self.warm_start and
--> 982                                             hasattr(self, "classes_")))
    983 
    984     @property

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in _fit(self, X, y, incremental)
    316 
    317         # Validate input parameters.
--> 318         self._validate_hyperparameters()
    319         if np.any(np.array(hidden_layer_sizes) <= 0):
    320             raise ValueError("hidden_layer_sizes must be > 0, got %s." %

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in _validate_hyperparameters(self)
    417             raise ValueError("The activation '%s' is not supported. Supported "
    418                              "activations are %s." % (self.activation,
--> 419                                                       supported_activations))
    420         if self.learning_rate not in ["constant", "invscaling", "adaptive"]:
    421             raise ValueError("learning rate %s is not supported. " %

**ValueError: The activation 'identity2' is not supported. Supported activations are ('identity', 'logistic', 'tanh', 'relu').**
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-58f6cbc117fc> in <module>
      1 mlp=MLPClassifier(activation='leakyrelu')
----> 2 mlp.fit(X_train,y_train)

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in fit(self, X, y)
    980         """
    981         return self._fit(X, y, incremental=(self.warm_start and
--> 982                                             hasattr(self, "classes_")))
    983 
    984     @property

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in _fit(self, X, y, incremental)
    316 
    317         # Validate input parameters.
--> 318         self._validate_hyperparameters()
    319         if np.any(np.array(hidden_layer_sizes) <= 0):
    320             raise ValueError("hidden_layer_sizes must be > 0, got %s." %

~/anaconda3/lib/python3.7/site-packages/sklearn/neural_network/multilayer_perceptron.py in _validate_hyperparameters(self)
    417             raise ValueError("The activation '%s' is not supported. Supported "
    418                              "activations are %s." % (self.activation,
--> 419                                                       supported_activations))
    420         if self.learning_rate not in ["constant", "invscaling", "adaptive"]:
    421             raise ValueError("learning rate %s is not supported. " %

**ValueError: The activation 'leakyrelu' is not supported. Supported activations are ('identity', 'logistic', 'tanh', 'relu').**

#Custom Activation Function
from sklearn.neural_network.multilayer_perceptron import(ACTIVATIONS, DERIVATIVES, MLPClassifier)

def leakyrelu(X):
    """Compute the leaky rectified linear unit function inplace.
    leakyrely(x) = x if x > 0
    leakyrelu(x) = 0.01*x if x <= 0
    """
    # np.clip(X, 0.01, np.finfo(X.dtype).max, out=X)
    return np.max(0.01*X, X)

def inplace_leakyrelu_derivative(Z, delta):
    """Apply the derivative of the leaky relu function.

    It exploits the fact that the derivative is a simple function of the output
    value from leaky rectified linear units activation function.

    Parameters
    ----------
    Z : {array-like, sparse matrix}, shape (n_samples, n_features)
        The data which was output from the rectified linear units activation
        function during the forward pass.

    delta : {array-like}, shape (n_samples, n_features)
         The backpropagated error signal to be modified inplace.
    """
    delta = 0.01 *(Z > 0)
    
    
ACTIVATIONS['leakyrelu']= leakyrelu

DERIVATIVES['leakyrelu']= inplace_leakyrelu_derivative

Commit message: Don't publicaly re-export ACTIVATIONS variable from sklearn.neural_network._base

Patch to review:
diff --git a/sklearn/neural_network/multilayer_perceptron.py b/sklearn/neural_network/multilayer_perceptron.py
index 0cb94de71b3a6..09f2f884f2966 100644
--- a/sklearn/neural_network/multilayer_perceptron.py
+++ b/sklearn/neural_network/multilayer_perceptron.py
@@ -15,7 +15,7 @@
 
 from ..base import BaseEstimator, ClassifierMixin, RegressorMixin
 from ..base import is_classifier
-from ._base import ACTIVATIONS, DERIVATIVES, LOSS_FUNCTIONS
+from ._base import LOSS_FUNCTIONS
 from ._stochastic_optimizers import SGDOptimizer, AdamOptimizer
 from ..model_selection import train_test_split
 from ..preprocessing import LabelBinarizer
@@ -97,6 +97,10 @@ def _forward_pass(self, activations):
         activations : list, length = n_layers - 1
             The ith element of the list holds the values of the ith layer.
         """
+        # local import to allow custom activations set via this
+        # private variable in _base
+        from ._base import ACTIVATIONS
+
         hidden_activation = ACTIVATIONS[self.activation]
         # Iterate over the hidden layers
         for i in range(self.n_layers_ - 1):
@@ -215,6 +219,10 @@ def _backprop(self, X, y, activations, deltas, coef_grads,
         coef_grads : list, length = n_layers - 1
         intercept_grads : list, length = n_layers - 1
         """
+        # local import to allow custom activations set via this
+        # private variable in _base
+        from ._base import DERIVATIVES
+
         n_samples = X.shape[0]
 
         # Forward propagate
@@ -376,6 +384,10 @@ def _fit(self, X, y, incremental=False):
         return self
 
     def _validate_hyperparameters(self):
+        # local import to allow custom activations set via this
+        # private variable in _base
+        from ._base import ACTIVATIONS
+
         if not isinstance(self.shuffle, bool):
             raise ValueError("shuffle must be either True or False, got %s." %
                              self.shuffle)
@@ -414,11 +426,10 @@ def _validate_hyperparameters(self):
                              % self.n_iter_no_change)
 
         # raise ValueError if not registered
-        supported_activations = ('identity', 'logistic', 'tanh', 'relu')
-        if self.activation not in supported_activations:
+        if self.activation not in ACTIVATIONS:
             raise ValueError("The activation '%s' is not supported. Supported "
-                             "activations are %s." % (self.activation,
-                                                      supported_activations))
+                             "activations are %s."
+                             % (self.activation, list(sorted(ACTIVATIONS))))
         if self.learning_rate not in ["constant", "invscaling", "adaptive"]:
             raise ValueError("learning rate %s is not supported. " %
                              self.learning_rate)
diff --git a/sklearn/neural_network/tests/test_mlp.py b/sklearn/neural_network/tests/test_mlp.py
index 819d183f57c2f..cf2f94d7c9b06 100644
--- a/sklearn/neural_network/tests/test_mlp.py
+++ b/sklearn/neural_network/tests/test_mlp.py
@@ -710,3 +710,35 @@ def test_early_stopping_stratified():
             ValueError,
             match='The least populated class in y has only 1 member'):
         mlp.fit(X, y)
+
+
+def test_custom_activation(monkeypatch):
+    import sklearn.neural_network._base
+
+    X = X_digits_binary[:100]
+    y = y_digits_binary[:100]
+
+    monkeypatch.setattr(sklearn.neural_network._base, "ACTIVATIONS", {})
+
+    mlp = MLPClassifier()
+    # check that monkeypatching works as expected.
+    msg = r"activation 'relu' is not supported. Supported activations are \[\]"
+    with pytest.raises(ValueError, match=msg):
+        mlp.fit(X, y)
+
+    monkeypatch.setattr(
+        sklearn.neural_network._base,
+        "ACTIVATIONS",
+        # logistic is necessary for the last layer in any case
+        {'identity2': lambda x: x, 'logistic': lambda x: x}
+    )
+
+    monkeypatch.setattr(
+        sklearn.neural_network._base,
+        "DERIVATIVES",
+        # logistic is necessary for the last layer in any case
+        {'identity2': lambda x, _: 1.0, 'logistic': lambda x, _: 1.0}
+    )
+
+    mlp = MLPClassifier(activation='identity2')
+    mlp.fit(X, y)


Code review comments:
diff_hunk: @@ -97,6 +97,10 @@ def _forward_pass(self, activations):
         activations : list, length = n_layers - 1
             The ith element of the list holds the values of the ith layer.
         """
+        # local import to allow custom activations set via this
+        # private variable in _base
+        from ._base import ACTIVATIONS
path: sklearn/neural_network/multilayer_perceptron.py
text: @user1:
I'm confused, why do we need a local import exactly?

@author:
First to remove `ACTIVATIONS` and `DERIVATIVES` from a public import path. Second to allow monkeypatching it. If you import it at the top level, it's imported together with `MLPClassifier`, and then changing `neural_network._base.ACTIVATION` no longer has an effect on `MLPClassifier`. 

@user1:
I don't understand the second part: base.ACTIVATION is a singleton dict, it's never copied anywhere?

@author:
You are right. Actually the above example will work even with top level import. The problem is pytest monkeypatching, if I don't do local imports tests using it will fail, since I monkeypatch `ACTIVATIONS` instead of just updating it to be sure not to affect the global state or other tests.

I removed the comment as it was not accurate, but I guess in any case it doesn't hurt not to expose those variables at top level?

@user1:
can we add a test that doesn't require monkey patching? I'd be fine with `del ACTIVATIONS[key]` at the end of the test. It's not great, but having a local import so that we can write a test in a certain way sounds like a strange design to me

@author:
> It's not great, but having a local import so that we can write a test in a certain way sounds like a strange design to me

Fair enough, I can also just revert https://github.com/scikit-learn/scikit-learn/pull/14815/commits/7519f1412eeb6ab99d5ba0dfdafc1f893642ca9a and we will be back where we started but with an example in the issue using the private API :)