Repository: scikit-learn/scikit-learn
Pull request: DOC Add example of using CV iterator to split train test
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
closes #9193 

#### What does this implement/fix? Explain your changes.
Add example of using 'group' CV iterator to split train test subset. `train_test_split` wraps around `ShuffleSplit` and thus cannot account for groups.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
add support for groups in train_test_split
`train_test_split` has support for options 'stratify' and 'shuffle' but not 'groups'.

I'm interested in adding support for 'groups' to `train_test_split` (so that all samples from the same group will be in either train or test but not both).  In 0.18.1 there is support for the option 'stratify' and in master there is recently added support for 'shuffle'.  If others think it might be useful I'd like to make a PR.

The rules starts to get a little complicated with options for 'stratify', 'shuffle', and 'groups' interacting.  It makes sense to throw an error when `groups` is not `None` and `shuffle` is `False` (there is similar logic for `stratify` and `shuffle` in master).  And it makes sense for a `ValueError` to be raised if `groups` is not `None` and `stratify` is not `None` since there is a class `GroupShuffleSplit` and `StratifyShuffleSplit` but no `StratifyGroupShuffleSplit`.

The rules look something like this:

| `stratify` | `shuffle` | `groups` | behavior |
|------------|------------|--------------|-------------|
| None | True | None | use `ShuffleSplit` |
| None | False | None | no shuffling, just splits on `n_train` |
| not None | True | None | use `StratifiedShuffleSplit` |
| not None | False | None | raise ValueError |
| None | True | not None | use `GroupShuffleSplit` (proposed) |
| None | False | not None | raise ValueError (proposed) |
| not None | True | not None | raise ValueError (proposed) |
| not None | False | not None | raise ValueError (proposed) |

Another possibility is for `train_test_split` to be explicitly passed a cross-validator class (rather than figuring it out), but that might be adding more burden on the caller, considering this is a convenience function.

If this is easier to discuss in the form of a PR, I'd be happy to submit one.  And if I'm missing a simpler solution to this, I'd be happy to learn that.

thanks,
Dennis

Hints from issue discussion:
Related to #8844 and #8845 
Hm I'm not sure we want to go overboard with this helper.
You can do
```python
train_inds, test_inds = GroupShuffleSplit().split(X, y, goups).next()
X_train, X_test, y_train, y_test = X[train_inds], X[test_inds], y[train_inds], y[test_inds]
```
Okay, makes sense.  That snippet is much more concise than the changes I made locally to `train_test_split` to support groups.
Hey, I would like to work on this. May I ?
I don't think we particularly want this feature. are you offering to
implement it because you want to use it?

On 17 Jul 2017 7:18 pm, "JustAnotherHomer" <notifications@github.com> wrote:

> Hey, I would like to work on this. May I ?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/9193#issuecomment-315704551>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAEz68FtqoA2jmKz1JT1F62d35XAGejiks5sOybIgaJpZM4OAVkz>
> .
>

@jnothman Yeah, I thought that it would be simpler to use it. But if you feel it's unnecessary, then ok.  
I'd review a PR if you made one.

On 18 July 2017 at 02:08, JustAnotherHomer <notifications@github.com> wrote:

> @jnothman <https://github.com/jnothman> Yeah, I thought that it would be
> simpler to use it. But if you feel it's unnecessary, then ok.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/9193#issuecomment-315790772>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAEz65AzdkrreRRpwKwOvqFtuop58hMVks5sO4WWgaJpZM4OAVkz>
> .
>

I think it is unnecessary to add another helper. We could add to the train_test_split docs that this can be done for any splitter using my snippet.
In case anyone else is using Python3 and got `AttributeError: 'generator' object has no attribute 'next'` from the code:

`train_inds, test_inds = GroupShuffleSplit().split(X, groups=groups).next()
`

...the following works in Python3:

`train_inds, test_inds = next(GroupShuffleSplit().split(X, groups=groups))
`

yes, should have been a function call.

Commit message: set random state

Patch to review:
diff --git a/doc/modules/cross_validation.rst b/doc/modules/cross_validation.rst
index 61ea8c7ef4248..c8466a69ce25b 100644
--- a/doc/modules/cross_validation.rst
+++ b/doc/modules/cross_validation.rst
@@ -744,6 +744,35 @@ e.g. when searching for hyperparameters.
 For example, when using a validation set, set the ``test_fold`` to 0 for all
 samples that are part of the validation set, and to -1 for all other samples.
 
+Using cross-validation iterators to split train and test
+--------------------------------------------------------
+
+The above group cross-validation functions may also be useful for spitting a
+dataset into training and testing subsets. Note that the convenience
+function :func:`train_test_split` is a wrapper around :func:`ShuffleSplit`
+and thus only allows for stratified splitting (using the class labels)
+and cannot account for groups.
+
+To perform the train and test split, use the indices for the train and test
+subsets yielded by the generator output by the `split()` method of the
+cross-validation iterator. For example::
+
+  >>> import numpy as np
+  >>> from sklearn.model_selection import GroupShuffleSplit
+
+  >>> X = np.array([0.1, 0.2, 2.2, 2.4, 2.3, 4.55, 5.8, 0.001])
+  >>> y = np.array(["a", "b", "b", "b", "c", "c", "c", "a"])
+  >>> groups = np.array([1, 1, 2, 2, 3, 3, 4, 4])
+  >>> train_indx, test_indx = next(
+  ...     GroupShuffleSplit(random_state=7).split(X, y, groups)
+  ... )
+  >>> X_train, X_test, y_train, y_test = \
+  ...     X[train_indx], X[test_indx], y[train_indx], y[test_indx]
+  >>> X_train.shape, X_test.shape
+  ((6,), (2,))
+  >>> np.unique(groups[train_indx]), np.unique(groups[test_indx])
+  (array([1, 2, 4]), array([3]))
+
 .. _timeseries_cv:
 
 Cross validation of time series data


Code review comments:
diff_hunk: @@ -744,6 +744,35 @@ e.g. when searching for hyperparameters.
 For example, when using a validation set, set the ``test_fold`` to 0 for all
 samples that are part of the validation set, and to -1 for all other samples.
 
+Using cross-validation iterators to split train and test
+--------------------------------------------------------
+
+The above group cross-validation functions may also be useful for spitting a
+dataset into training and testing subsets. Note that the convenience
+function :func:`train_test_split` is a wrapper around :func:`ShuffleSplit`
+and thus only allows for stratified splitting (using the class labels)
+and cannot account for groups.
+
+To perform the train and test split, use the indices for the train and test
+subsets yielded by the generator output by the `split()` method of the
+cross-validation iterator. For example::
path: doc/modules/cross_validation.rst
text: @user1:
The cross-validation object is not iterator itself.

```suggestion
cross-validation splitter. For example::
```

@author:
thanks!