Repository: scikit-learn/scikit-learn
Pull request: [MRG] DOC Mention StandardScaler ddof

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Fixes #7757
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Expands the documentation so it's clear that the estimate of the standard deviation in StandardScaler is the biased one (equivalent to numpy.sqrt(numpy.var(x, ddof=0))).

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
StandardScaler that supports option to use unbiased estimator of variance
#### Description

It would be nice to provide StandardScaler with an option to chose unbiased estimator of variance,

It it sometimes convenient to use unbiased estimator of variance: numpy.var(X, ddof=1)
instead of default: numpy.var(X, ddof=0)

So far StandardScaler does not allow to set ddof.
#### Versions

Linux-4.4.0-45-generic-x86_64-with-Ubuntu-16.04-xenial
('Python', '2.7.12 (default, Jul  1 2016, 15:12:24) \n[GCC 5.4.0 20160609]')
('NumPy', '1.11.2')
('SciPy', '0.18.1')
('Scikit-Learn', '0.19.dev0')

<!-- Thanks for contributing! -->

Hints from issue discussion:
Does it really matter? It seems to me that it's one of those things that
people (academics in particular) love to argue, but has no impact on
something like predictive accuracy in realistic settings.

PS: don't read me wrong, I am an academic. I love to argue. But I try to
do that in papers, not software.

No problem.
Just to tell the whole story. I've moved my 'pipeline' code from matlab to scikit and spent couple of hours trying to figure out why the matlab and scikit produce different results (selected features and classification performance).

It was due to the normalization in StandardScaler.
Providing this option I was just trying to save someone else those couple of hours.

```
It seems to me that it's one of those things that people (academics in particular) love to argue, but has no impact on something like predictive accuracy in realistic settings.
```

Yes and no :). 
I'm sorry, I'm new here, do not want to start an academic discussions. This was only suggestion. 

You can leave it as it is. I have created my own dirty StandardScaler :)

@spilkjir just out of curiosity: how many samples do you have that that made a difference?

This dataset: https://www.kaggle.com/c/melbourne-university-seizure-prediction
Seizure prediction from EEG.

features matrix ~ [2350x500] instances x features
number of normal 2200
number of pathological 150.

Logistic regression (LR) with l1 penalty. (3-GroupKFold CV) (C was kept small to rather promote sparsity in feature vector).

My conclusion was that as the data are high dimensional a small change in normalization causes different features to be selected with different perf.
The bad thing is that I don't know, which solution (standardization with ddof=0 or ddof=1) was more right or wrong. Only from theory, ddof should be 1 as population mean is not known. 

From user point of view I would like to have an easy option to chose ddof in StandardScaler.

> My conclusion was that as the data are high dimensional a small change in
> normalization causes different features to be selected with different perf.
> The bad thing is that I don't know, which solution (standardization with ddof=0
> or ddof=1) was more right or wrong. Only from theory, ddof should be 1 as
> population mean is not known.

Neither are right or wrong: l1 penalties for variable selection is
unstable and does not garanty in any way some form of control on
recovery. If you're interested in that, I would more advise you to
bootstrap the data and look at what's stable.

Yeah if a change in scale of 0.0004 changes your outcome drastically, your method is probably not very robust.

IMHO, any difference that it creates in your results is in the error bars
(and they may be very large in high-dimensional settings) and you
shouldn't trust results in these error bars. They will probably not be
reproduce on new data.

ok, thanks for the info.
I'm sorry for the spam

It's true that using slightly biased stdev estimator by itself does not make much of a difference, and if it does one should look into how to make the model more robust.  The bigger problem is that I was trying to reproduce some results that I obtained earlier in R and could not.  Yes, the differences are small and do not really impact the predictive power.  What it does impact is the time spent trying to understand where that difference comes from in the situations where one expects none, just in case there is a bigger issue lurking underneath. Thanks.
@atolpin, could our documentation be improved? Please offer a PR to clarify
the docs.

Commit message: Add message explaning it's unlikely that the choice of estimator would affect model performance.

Patch to review:
diff --git a/sklearn/preprocessing/data.py b/sklearn/preprocessing/data.py
index b78f1c11fab96..529a2c76ebbff 100644
--- a/sklearn/preprocessing/data.py
+++ b/sklearn/preprocessing/data.py
@@ -478,7 +478,10 @@ class StandardScaler(BaseEstimator, TransformerMixin):
 
     where `u` is the mean of the training samples or zero if `with_mean=False`,
     and `s` is the standard deviation of the training samples or one if
-    `with_std=False`.
+    `with_std=False`. Note that `s` is a biased estimator of the standard
+    deviation, equivalent to numpy.sqrt(numpy.var(x, ddof=0)), and that it is
+    unlikely that using this estimator as opposed its unbiased counterpart
+    will affect model performance.
 
     Centering and scaling happen independently on each feature by computing
     the relevant statistics on the samples in the training set. Mean and


Code review comments:
diff_hunk: @@ -478,7 +478,10 @@ class StandardScaler(BaseEstimator, TransformerMixin):
 
     where `u` is the mean of the training samples or zero if `with_mean=False`,
     and `s` is the standard deviation of the training samples or one if
-    `with_std=False`.
+    `with_std=False`. Note that `s` is a biased estimator of the standard
+    deviation, equivalent to numpy.sqrt(numpy.var(x, ddof=0)), and that it is
path: sklearn/preprocessing/data.py
text: @user1:
use ``numpy.std`` instead?