Repository: scikit-learn/scikit-learn
Pull request: [MRG] Updating SKF user_guide example
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Fixes #14714
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Updated the example of `StratifiedKFold` in the user guide and compared with `KFold` split.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Improve example on StratifiedKFold in user guide
The user guide currently has the following example:
```python
    >>> from sklearn.model_selection import StratifiedKFold
  
    >>> X = np.ones(10)
    >>> y = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
    >>> skf = StratifiedKFold(n_splits=3)
    >>> for train, test in skf.split(X, y):
    ...     print("%s %s" % (train, test))
    [2 3 6 7 8 9] [0 1 4 5]
    [0 1 3 4 5 8 9] [2 6 7]
    [0 1 2 4 5 6 7] [3 8 9]
```
I think it would be much more instructive to show the class distributions for training and test set instead, and maybe create a slightly larger dataset to show the effect more clearly? Or compare against what KFold would do (which would be pretty disastrous in this case)

Hints from issue discussion:
see #14560 for an example from the regression case.
@amueller 
I made an attempt to create a good example. Please give your comments.

```
from sklearn.datasets import make_classification
from sklearn.model_selection import StratifiedKFold, KFold
from collections import Counter
X, y = make_classification(n_samples=100, n_classes=3,
                           n_informative=3,
                           weights=[0.5, 0.3, 0.2], random_state=1)

print('n_samples in StratifiedKFold\n')
skf = StratifiedKFold(n_splits=3, random_state=1)
for train, test in skf.split(X, y):
    print('train - ', Counter(y[train]),'\n',
            ' test - ', Counter(y[test]),'\n')

print('n_samples in KFold\n')
kf = KFold(n_splits=3, random_state=4)
for train, test in kf.split(X, y):
    print('train - ', Counter(y[train]),'\n',
            ' test - ', Counter(y[test]),'\n')
```
```
n_samples in StratifiedKFold

train -  Counter({0: 33, 1: 20, 2: 13}) 
  test -  Counter({0: 17, 1: 10, 2: 7}) 

train -  Counter({0: 33, 1: 20, 2: 13}) 
  test -  Counter({0: 17, 1: 10, 2: 7}) 

train -  Counter({0: 34, 1: 20, 2: 14}) 
  test -  Counter({0: 16, 1: 10, 2: 6}) 

n_samples in KFold

train -  Counter({0: 33, 1: 21, 2: 12}) 
  test -  Counter({0: 17, 1: 9, 2: 8}) 

train -  Counter({0: 32, 1: 22, 2: 13}) 
  test -  Counter({0: 18, 1: 8, 2: 7}) 

train -  Counter({0: 35, 1: 17, 2: 15}) 
  test -  Counter({0: 15, 1: 13, 2: 5}) 
```
I would probably use ``np.bincount`` instead of ``Counter``.
Do you think there's a clear visible difference between the two approaches in your example?
@amueller 
Sure. Thanks for your comments. Please find my updated example:
```
from sklearn.datasets import make_classification
from sklearn.model_selection import StratifiedKFold, KFold
from collections import Counter
X, y = make_classification(n_samples=50, n_classes=3,
                           n_informative=3,
                           weights=[0.7, 0.1, 0.2], random_state=1)

print('n_samples in StratifiedKFold\n')
skf = StratifiedKFold(n_splits=3, random_state=1)
for train, test in skf.split(X, y):
    print('train - ', np.bincount(y[train]),' | ',
            ' test - ', np.bincount(y[test]),'\n')

print('n_samples in KFold\n')
kf = KFold(n_splits=3, random_state=9)
for train, test in kf.split(X, y):
    print('train - ', np.bincount(y[train]),' | ',
            ' test - ', np.bincount(y[test]),'\n')
```

```
n_samples in StratifiedKFold

train -  [23  3  6]  |   test -  [12  2  4] 

train -  [23  3  7]  |   test -  [12  2  3] 

train -  [24  4  7]  |   test -  [11  1  3] 

n_samples in KFold

train -  [24  5  4]  |   test -  [11  0  6] 

train -  [21  5  7]  |   test -  [14  0  3] 

train -  [25  0  9]  |   test -  [10  5  1] 
```

@venkyyuvy can you send a pull request?
@amueller Can I make a pr?
Hi @amueller , is this stll an open issue since I am interested to contribute?

Commit message: More description

Patch to review:
diff --git a/doc/modules/cross_validation.rst b/doc/modules/cross_validation.rst
index e42d7f8514438..dedb7df8ed984 100644
--- a/doc/modules/cross_validation.rst
+++ b/doc/modules/cross_validation.rst
@@ -534,19 +534,35 @@ Stratified k-fold
 folds: each set contains approximately the same percentage of samples of each
 target class as the complete set.
 
-Example of stratified 3-fold cross-validation on a dataset with 10 samples from
-two slightly unbalanced classes::
+Example of stratified 3-fold cross-validation on a dataset with 50 samples from
+two unbalanced classes and comparing with :class:`KFold`::
 
-  >>> from sklearn.model_selection import StratifiedKFold
+  >>> from sklearn.model_selection import StratifiedKFold, KFold
+  >>> import numpy as np
+
+  >>> X, y = np.ones((50, 1)), np.hstack(([0] * 45, [1] * 5))
 
-  >>> X = np.ones(10)
-  >>> y = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
   >>> skf = StratifiedKFold(n_splits=3)
-  >>> for train, test in skf.split(X, y):
-  ...     print("%s %s" % (train, test))
-  [2 3 6 7 8 9] [0 1 4 5]
-  [0 1 3 4 5 8 9] [2 6 7]
-  [0 1 2 4 5 6 7] [3 8 9]
+  >>> print('Number of samples in each class for StratifiedKFold')
+  Number of samples in each class for StratifiedKFold 
+  >>> for train, test in skf.split(X, y):  
+  ...     print('train -  {}   |   test -  {}'.format(
+  ...         np.bincount(y[train]), np.bincount(y[test])))
+  train -  [30  3]   |   test -  [15  2]
+  train -  [30  3]   |   test -  [15  2]
+  train -  [30  4]   |   test -  [15  1]
+  >>> kf = KFold(n_splits=3)
+  >>> print('Number of samples in each class for KFold')
+  Number of samples in each class for KFold
+  >>> for train, test in kf.split(X, y):
+  ...     print('train -  {}   |   test -  {}'.format(
+  ...         np.bincount(y[train]), np.bincount(y[test])))
+  train -  [28  5]   |   test -  [17]
+  train -  [28  5]   |   test -  [17]
+  train -  [34]   |   test -  [11  5]
+
+We can see that :class:StratifiedKFold preserves the class ratios 
+(approximately 1 / 10) in both train and test dataset.
 
 Here is a visualization of the cross-validation behavior.
 


Code review comments:
diff_hunk: @@ -534,19 +534,35 @@ Stratified k-fold
 folds: each set contains approximately the same percentage of samples of each
 target class as the complete set.
 
-Example of stratified 3-fold cross-validation on a dataset with 10 samples from
-two slightly unbalanced classes::
+Example of stratified 3-fold cross-validation on a dataset with 50 samples from
+two unbalanced classes and comparing with :class:`KFold`::
 
-  >>> from sklearn.model_selection import StratifiedKFold
+  >>> from sklearn.model_selection import StratifiedKFold, KFold
+  >>> import numpy as np
+
+  >>> X, y = np.ones((50, 1)), np.hstack(([0] * 45, [1] * 5))
 
-  >>> X = np.ones(10)
-  >>> y = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
   >>> skf = StratifiedKFold(n_splits=3)
-  >>> for train, test in skf.split(X, y):
-  ...     print("%s %s" % (train, test))
-  [2 3 6 7 8 9] [0 1 4 5]
-  [0 1 3 4 5 8 9] [2 6 7]
-  [0 1 2 4 5 6 7] [3 8 9]
+  >>> print('Number of samples in each class for StratifiedKFold')
+  Number of samples in each class for StratifiedKFold 
+  >>> for train, test in skf.split(X, y):  
+  ...     print('train -  {}   |   test -  {}'.format(
+  ...         np.bincount(y[train]), np.bincount(y[test])))
+  train -  [30  3]   |   test -  [15  2]
+  train -  [30  3]   |   test -  [15  2]
+  train -  [30  4]   |   test -  [15  1]
+  >>> kf = KFold(n_splits=3)
+  >>> print('Number of samples in each class for KFold')
+  Number of samples in each class for KFold
+  >>> for train, test in kf.split(X, y):
+  ...     print('train -  {}   |   test -  {}'.format(
+  ...         np.bincount(y[train]), np.bincount(y[test])))
+  train -  [28  5]   |   test -  [17]
+  train -  [28  5]   |   test -  [17]
+  train -  [34]   |   test -  [11  5]
+
+We can see that :class:StratifiedKFold preserves the class ratios 
path: doc/modules/cross_validation.rst
text: @user1:
```suggestion
We can see that :class:`StratifiedKFold` preserves the class ratios 
```

diff_hunk: @@ -534,19 +534,35 @@ Stratified k-fold
 folds: each set contains approximately the same percentage of samples of each
 target class as the complete set.
 
-Example of stratified 3-fold cross-validation on a dataset with 10 samples from
-two slightly unbalanced classes::
+Example of stratified 3-fold cross-validation on a dataset with 50 samples from
+two unbalanced classes and comparing with :class:`KFold`::
 
-  >>> from sklearn.model_selection import StratifiedKFold
+  >>> from sklearn.model_selection import StratifiedKFold, KFold
+  >>> import numpy as np
+
+  >>> X, y = np.ones((50, 1)), np.hstack(([0] * 45, [1] * 5))
 
-  >>> X = np.ones(10)
-  >>> y = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
   >>> skf = StratifiedKFold(n_splits=3)
-  >>> for train, test in skf.split(X, y):
-  ...     print("%s %s" % (train, test))
-  [2 3 6 7 8 9] [0 1 4 5]
-  [0 1 3 4 5 8 9] [2 6 7]
-  [0 1 2 4 5 6 7] [3 8 9]
+  >>> print('Number of samples in each class for StratifiedKFold')
+  Number of samples in each class for StratifiedKFold 
+  >>> for train, test in skf.split(X, y):  
+  ...     print('train -  {}   |   test -  {}'.format(
+  ...         np.bincount(y[train]), np.bincount(y[test])))
+  train -  [30  3]   |   test -  [15  2]
+  train -  [30  3]   |   test -  [15  2]
+  train -  [30  4]   |   test -  [15  1]
+  >>> kf = KFold(n_splits=3)
+  >>> print('Number of samples in each class for KFold')
path: doc/modules/cross_validation.rst
text: @user1:
That doesn't render really well (bad advice from my part, sorry).

Maybe we can just change the text above (see other comment)

@author:
No problem. Thanks for your inputs.