Repository: scikit-learn/scikit-learn
Pull request: [MRG] DOC: improve docstring form 'normalize'
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->
Fixes: [#4480](https://github.com/scikit-learn/scikit-learn/issues/4480)

#### What does this implement/fix? Explain your changes.
Clarifies the behavior of normalize in Ridge. Adding in that mean subtraction and division by l2-norm are performed. I also made the line about it being ignored if `fit_intercept` is False to the beginning, as that seems important.
I also removed a reference to claim that normalization, as opposed to standardization, has reduced dependence on number of rows. I am not aware of a reference for that claim. I think it may come from the discussion on [#2601](https://github.com/scikit-learn/scikit-learn/issues/2601). I do not see how either the empirical or theoretical justification works in that thread. The range for alpha in both graphs is approximately an order of magnitude, and I don't understand the theoretical discussion at all. I am happy to be corrected on that point, however, I think a claim like that should reference some justification rather than being briefly mentioned in the docstring.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Documentation for 'normalize' in Ridge regression somewhat unclear
For linear models with regularization, the input variables should be standardized because the solution is not scale invariant. I noticed that in the ridge regression source code, the input variables are centered by mean subtraction but normalization by std deviation is a flag.

My questions are:
1) Under what circumstances should normalized be set to True so that both centering and normalizing by std. dev are turned on?

2) Does normalize=True affect prediction? When predicting from new X, should X be centered and normalized?

3) Does normalizing have any affect on the range of alpha the regularization parameter? It would be nice to be able to restrict alpha to [0, 1].

It would be great if the documentation made this clear. Thanks.

Hints from issue discussion:
1) I think our models are not consistent in this, but normalization should never hurt.
2) It affects the model. The data will not be expected to be normalized, the model will take care of this
3) I don't see why it should be possible to restrict alpha to [0, 1]. Why do you want that? alpha>1 is pretty rare, but I don't see why it shouldn't be allowed.

> 2) It affects the model. The data will not be expected to be normalized, the model will take care of this

Could you elaborate a little bit on how it affects the model? I don't see the coefficients being re-scaled back to original scale, so I am a little concerned about prediction. 

> I don't see why it should be possible to restrict alpha to [0, 1]. Why do you want that? alpha>1 is pretty rare, but I don't see why it shouldn't be allowed.

It would be nice to run GCV on alphas in np.logspace(-3, 0). When alpha is not bounded, your search space is much bigger.

In prediction, the data will be scaled the same way as during training.

Only because parameters are valid, doesn't mean you have to search them.

Sorry, I don't see where the data is scaled for prediction - 

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/base.py#L138

It's perfectly clear to me: centering is always done, normalization is optional:

```
Parameters
----------
normalize : boolean, optional, default False
    If True, the regressors X will be normalized before regression.
```

Hum, looking at the code it is not so clear to me any more.
maybe @agramfort or @GaelVaroquaux can say something about that?
I thought the data was also transformed in `decision_function` but that doesn't seem to be the case.

the scaling and centering gets used to correct the .coef_ and .intercept_ 

see

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/base.py#L162

it's more efficient than storing X_mean and X_std

ok?

yep, missed that (I didn't think that `set_intercept` would change `coef_` ;)

> yep, missed that (I didn't think that set_intercept would change coef_ ;)

point taken... maybe _post_fit would have been a better name...

Since @ConvexHull doesn't seem to be responding on his PR, I'm taking this up.

Commit message: add missing path reference

Patch to review:
diff --git a/sklearn/linear_model/base.py b/sklearn/linear_model/base.py
index 7ac614a1cd7fe..546aaaa544919 100644
--- a/sklearn/linear_model/base.py
+++ b/sklearn/linear_model/base.py
@@ -410,12 +410,10 @@ class LinearRegression(LinearModel, RegressorMixin):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
         This parameter is ignored when `fit_intercept` is set to False.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
         with `normalize=False`.
 
diff --git a/sklearn/linear_model/bayes.py b/sklearn/linear_model/bayes.py
index 542e763612ad2..4ac3e06a1ebfd 100644
--- a/sklearn/linear_model/bayes.py
+++ b/sklearn/linear_model/bayes.py
@@ -64,12 +64,10 @@ class BayesianRidge(LinearModel, RegressorMixin):
         Default is True.
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
         This parameter is ignored when `fit_intercept` is set to False.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
         with `normalize=False`.
 
@@ -329,12 +327,10 @@ class ARDRegression(LinearModel, RegressorMixin):
         Default is True.
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
         This parameter is ignored when `fit_intercept` is set to False.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
         with `normalize=False`.
 
diff --git a/sklearn/linear_model/coordinate_descent.py b/sklearn/linear_model/coordinate_descent.py
index f2449c68ebd85..958bfe4fd1175 100644
--- a/sklearn/linear_model/coordinate_descent.py
+++ b/sklearn/linear_model/coordinate_descent.py
@@ -65,13 +65,11 @@ def _alpha_grid(X, y, Xy=None, l1_ratio=1.0, fit_intercept=True,
         Whether to fit an intercept or not
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     copy_X : boolean, optional, default True
@@ -546,13 +544,11 @@ class ElasticNet(LinearModel, RegressorMixin):
         data is assumed to be already centered.
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     precompute : True | False | array-like
@@ -794,13 +790,11 @@ class Lasso(ElasticNet):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     copy_X : boolean, optional, default True
@@ -1282,13 +1276,11 @@ class LassoCV(LinearModelCV, RegressorMixin):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     copy_X : boolean, optional, default True
@@ -1436,13 +1428,11 @@ class ElasticNetCV(LinearModelCV, RegressorMixin):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     copy_X : boolean, optional, default True
@@ -1568,13 +1558,11 @@ class MultiTaskElasticNet(Lasso):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     copy_X : boolean, optional, default True
@@ -1754,13 +1742,11 @@ class MultiTaskLasso(MultiTaskElasticNet):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     copy_X : boolean, optional, default True
@@ -1891,13 +1877,11 @@ class MultiTaskElasticNetCV(LinearModelCV, RegressorMixin):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     copy_X : boolean, optional, default True
@@ -2057,13 +2041,11 @@ class MultiTaskLassoCV(LinearModelCV, RegressorMixin):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If ``True``, the regressors X will be normalized before regression.
-        This parameter is ignored when ``fit_intercept`` is set to ``False``.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
-        :class:`preprocessing.StandardScaler` before calling ``fit`` on an estimator
+        This parameter is ignored when ``fit_intercept`` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
+        :class:`sklearn.preprocessing.StandardScaler` before calling ``fit`` on an estimator
         with ``normalize=False``.
 
     copy_X : boolean, optional, default True
diff --git a/sklearn/linear_model/least_angle.py b/sklearn/linear_model/least_angle.py
index c06720a25299c..70a0391acf439 100644
--- a/sklearn/linear_model/least_angle.py
+++ b/sklearn/linear_model/least_angle.py
@@ -514,12 +514,10 @@ class Lars(LinearModel, RegressorMixin):
         Sets the verbosity amount
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
         This parameter is ignored when `fit_intercept` is set to False.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
         with `normalize=False`.
 
diff --git a/sklearn/linear_model/omp.py b/sklearn/linear_model/omp.py
index 5328a2ed81707..887592965140e 100644
--- a/sklearn/linear_model/omp.py
+++ b/sklearn/linear_model/omp.py
@@ -558,12 +558,10 @@ class OrthogonalMatchingPursuit(LinearModel, RegressorMixin):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
-        This parameter is ignored when `fit_intercept` is set to `False`.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        This parameter is ignored when `fit_intercept` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
         with `normalize=False`.
 
@@ -696,12 +694,10 @@ def _omp_path_residues(X_train, y_train, X_test, y_test, copy=True,
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
-        This parameter is ignored when `fit_intercept` is set to `False`.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        This parameter is ignored when `fit_intercept` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
         with `normalize=False`.
 
@@ -763,12 +759,10 @@ class OrthogonalMatchingPursuitCV(LinearModel, RegressorMixin):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
-        This parameter is ignored when `fit_intercept` is set to `False`.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        This parameter is ignored when `fit_intercept` is set to False.
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
         with `normalize=False`.
 
diff --git a/sklearn/linear_model/ridge.py b/sklearn/linear_model/ridge.py
index 3b67128ac8a54..b0d3a1f1327b6 100644
--- a/sklearn/linear_model/ridge.py
+++ b/sklearn/linear_model/ridge.py
@@ -532,12 +532,10 @@ class Ridge(_BaseRidge, RegressorMixin):
         by scipy.sparse.linalg. For 'sag' solver, the default value is 1000.
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
         This parameter is ignored when `fit_intercept` is set to False.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
         with `normalize=False`.
 


Code review comments:
diff_hunk: @@ -410,12 +410,10 @@ class LinearRegression(LinearModel, RegressorMixin):
         (e.g. data is expected to be already centered).
 
     normalize : boolean, optional, default False
-        If True, the regressors X will be normalized before regression.
         This parameter is ignored when `fit_intercept` is set to False.
-        When the regressors are normalized, note that this makes the
-        hyperparameters learnt more robust and almost independent of the number
-        of samples. The same property is not valid for standardized data.
-        However, if you wish to standardize, please use
+        If True, the regressors X will be normalized before regression by
+        subtracting the mean and dividing by the l2-norm.
+        If you wish to standardize, please use
         `preprocessing.StandardScaler` before calling `fit` on an estimator
path: sklearn/linear_model/base.py
text: @user1:
Can you please fix those one also?
I count 8 possible fixes of `preprocessing.StandardScaler`, in the line immediately after your diff

@user1:
I'll merge it afterward