Repository: scikit-learn/scikit-learn
Pull request: [MRG+1] Repeated K-Fold and Repeated Stratified K-Fold
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
Fixes #7948 


#### What does this implement/fix? Explain your changes.
Implements RepeatedKFold and RepeatedStratifiedKFold

#### Any other comments?
For previous discussion on this, please refer to #7960 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
repeated k-fold
We should add repeated k-fold cross-validation. It's something a lot of people use, and it's tricky to implement in scikit-learn. In particular with the recent changes to cross-validation,

```python
cv = KFold(shuffle=True)
results = []
for i in range(10):
    results.append(cross_val_score(est, X, y, cv=cv))
```
yields repeated cross-validation in 0.17 and 0.18, but not in 0.18.1 or dev (I think?)

We could do this with a wrapper that sets the random_state or we could explicitly write ``RepeatedKFold`` and ``RepeatedStratifiedKFold``.

Hints from issue discussion:
Do you want to be repeating the *same* splits? In all of 0.17, 0.18.0,
0.18.1 and master this will produce different splits in each call to
cross_val_score, but not if random_state is an int. The difference in
0.18.0 is that in GridSearchCV each parameter would use different splits. I
think #7935 would change this to perform repeated KFold with the *same*
splits. These differences are clearly very subtle but important
semantically.

On 29 November 2016 at 10:26, Andreas Mueller <notifications@github.com>
wrote:

> We should add repeated k-fold cross-validation. It's something a lot of
> people use, and it's tricky to implement in scikit-learn. In particular
> with the recent changes to cross-validation,
>
> cv = KFold(shuffle=True)
> results = []for i in range(10):
>     results.append(cross_val_score(est, X, y, cv=cv))
>
> yields repeated cross-validation in 0.17 and 0.18, but not in 0.18.1 or
> dev (I think?)
>
> We could do this with a wrapper that sets the random_state or we could
> explicitly write RepeatedKFold and RepeatedStratifiedKFold.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/7948>, or mute the
> thread
> <https://github.com/notifications/unsubscribe-auth/AAEz63mM7vzvCPzHC-VwnaNcYS9BKtzMks5rC2M7gaJpZM4K-YtH>
> .
>

You are right, this is still the same as it has been in 0.17, but would change with #7935.

> We could do this with a wrapper that sets the random_state or we could
> explicitly write RepeatedKFold and RepeatedStratifiedKFold.

+1 for explicit RepeatedKFold and RepeatedStratifiedKFold.


Yeah that's where I'm leaning to, too.
I would like to work on this one.
@neerajgangwar go ahead!
@amueller I pushed initial commit. Can you please check and see if I am going in the right direction? :)

Commit message: Remove blank line, put testcases for deterministic split in loop and add StopIteration check in testcase

Patch to review:
diff --git a/doc/modules/classes.rst b/doc/modules/classes.rst
index 3aee8f258b9d1..3101488fd6661 100644
--- a/doc/modules/classes.rst
+++ b/doc/modules/classes.rst
@@ -170,6 +170,8 @@ Splitter Classes
    model_selection.LeavePGroupsOut
    model_selection.LeaveOneOut
    model_selection.LeavePOut
+   model_selection.RepeatedKFold
+   model_selection.RepeatedStratifiedKFold
    model_selection.ShuffleSplit
    model_selection.GroupShuffleSplit
    model_selection.StratifiedShuffleSplit
diff --git a/doc/modules/cross_validation.rst b/doc/modules/cross_validation.rst
index 3f7cf95e59417..085893e9262d9 100644
--- a/doc/modules/cross_validation.rst
+++ b/doc/modules/cross_validation.rst
@@ -263,6 +263,32 @@ Thus, one can create the training/test sets using numpy indexing::
   >>> X_train, X_test, y_train, y_test = X[train], X[test], y[train], y[test]
 
 
+Repeated K-Fold
+---------------
+
+:class:`RepeatedKFold` repeats K-Fold n times. It can be used when one
+requires to run :class:`KFold` n times, producing different splits in
+each repetition.
+
+Example of 2-fold K-Fold repeated 2 times::
+
+  >>> import numpy as np
+  >>> from sklearn.model_selection import RepeatedKFold
+  >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
+  >>> random_state = 12883823
+  >>> rkf = RepeatedKFold(n_splits=2, n_repeats=2, random_state=random_state)
+  >>> for train, test in rkf.split(X):
+  ...     print("%s %s" % (train, test))
+  ...
+  [2 3] [0 1]
+  [0 1] [2 3]
+  [1 2] [0 3]
+  [0 3] [1 2]
+
+Similarly, :class:`RepeatedStratifiedKFold` repeats Stratified K-Fold n times
+with different randomization in each repetition.
+
+
 Leave One Out (LOO)
 -------------------
 
@@ -409,6 +435,10 @@ two slightly unbalanced classes::
   [0 1 3 4 5 8 9] [2 6 7]
   [0 1 2 4 5 6 7] [3 8 9]
 
+:class:`RepeatedStratifiedKFold` can be used to repeat Stratified K-Fold n times
+with different randomization in each repetition.
+
+
 Stratified Shuffle Split
 ------------------------
 
diff --git a/sklearn/model_selection/__init__.py b/sklearn/model_selection/__init__.py
index f5ab0d7526ccf..73c842e706df8 100644
--- a/sklearn/model_selection/__init__.py
+++ b/sklearn/model_selection/__init__.py
@@ -7,6 +7,8 @@
 from ._split import LeaveOneOut
 from ._split import LeavePGroupsOut
 from ._split import LeavePOut
+from ._split import RepeatedKFold
+from ._split import RepeatedStratifiedKFold
 from ._split import ShuffleSplit
 from ._split import GroupShuffleSplit
 from ._split import StratifiedShuffleSplit
@@ -36,6 +38,8 @@
            'LeaveOneOut',
            'LeavePGroupsOut',
            'LeavePOut',
+           'RepeatedKFold',
+           'RepeatedStratifiedKFold',
            'ParameterGrid',
            'ParameterSampler',
            'PredefinedSplit',
diff --git a/sklearn/model_selection/_split.py b/sklearn/model_selection/_split.py
index b2ed060e31717..af2ade65f07c1 100644
--- a/sklearn/model_selection/_split.py
+++ b/sklearn/model_selection/_split.py
@@ -41,6 +41,8 @@
            'LeaveOneOut',
            'LeavePGroupsOut',
            'LeavePOut',
+           'RepeatedStratifiedKFold',
+           'RepeatedKFold',
            'ShuffleSplit',
            'GroupShuffleSplit',
            'StratifiedKFold',
@@ -397,6 +399,8 @@ class KFold(_BaseKFold):
         classification tasks).
 
     GroupKFold: K-fold iterator variant with non-overlapping groups.
+
+    RepeatedKFold: Repeats K-Fold n times.
     """
 
     def __init__(self, n_splits=3, shuffle=False,
@@ -553,6 +557,9 @@ class StratifiedKFold(_BaseKFold):
     All the folds have size ``trunc(n_samples / n_splits)``, the last one has
     the complementary.
 
+    See also
+    --------
+    RepeatedStratifiedKFold: Repeats Stratified K-Fold n times.
     """
 
     def __init__(self, n_splits=3, shuffle=False, random_state=None):
@@ -913,6 +920,165 @@ def get_n_splits(self, X, y, groups):
         return int(comb(len(np.unique(groups)), self.n_groups, exact=True))
 
 
+class _RepeatedSplits(with_metaclass(ABCMeta)):
+    """Repeated splits for an arbitrary randomized CV splitter.
+
+    Repeats splits for cross-validators n times with different randomization
+    in each repetition.
+
+    Parameters
+    ----------
+    cv : callable
+        Constructor of cross-validator.
+
+    n_repeats : int, default=5
+        Number of times cross-validator needs to be repeated.
+
+    random_state : None, int or RandomState, default=None
+        Random state to be used to generate random state for each
+        repetition.
+
+    **cvargs : additional params
+        Constructor parameters for cv. Must not contain random_state
+        and shuffle.
+    """
+    def __init__(self, cv, n_repeats=5, random_state=None, **cvargs):
+        if not isinstance(n_repeats, (np.integer, numbers.Integral)):
+            raise ValueError("Number of repetitions must be of Integral type.")
+
+        if n_repeats <= 1:
+            raise ValueError("Number of repetitions must be greater than 1.")
+
+        self.cv = cv
+        self.n_repeats = n_repeats
+        self.random_state = random_state
+        self.cvargs = cvargs
+
+    def split(self, X, y=None, groups=None):
+        """Generates indices to split data into training and test set.
+
+        Parameters
+        ----------
+        X : array-like, shape (n_samples, n_features)
+            Training data, where n_samples is the number of samples
+            and n_features is the number of features.
+
+        y : array-like, of length n_samples
+            The target variable for supervised learning problems.
+
+        groups : array-like, with shape (n_samples,), optional
+            Group labels for the samples used while splitting the dataset into
+            train/test set.
+
+        Returns
+        -------
+        train : ndarray
+            The training set indices for that split.
+
+        test : ndarray
+            The testing set indices for that split.
+        """
+        n_repeats = self.n_repeats
+        rng = check_random_state(self.random_state)
+
+        for idx in range(n_repeats):
+            random_state = rng.randint(np.iinfo(np.int32).max)
+            cv = self.cv(random_state=random_state, shuffle=True,
+                         **self.cvargs)
+            for train_index, test_index in cv.split(X, y, groups):
+                yield train_index, test_index
+
+
+class RepeatedKFold(_RepeatedSplits):
+    """Repeated K-Fold cross validator.
+
+    Repeats K-Fold n times with different randomization in each repetition.
+
+    Read more in the :ref:`User Guide <cross_validation>`.
+
+    Parameters
+    ----------
+    n_splits : int, default=3
+        Number of folds. Must be at least 2.
+
+    n_repeats : int, default=5
+        Number of times cross-validator needs to be repeated.
+
+    random_state : None, int or RandomState, default=None
+        Random state to be used to generate random state for each
+        repetition.
+
+    Examples
+    --------
+    >>> from sklearn.model_selection import RepeatedKFold
+    >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
+    >>> y = np.array([0, 0, 1, 1])
+    >>> rkf = RepeatedKFold(n_splits=2, n_repeats=2, random_state=2652946)
+    >>> for train_index, test_index in rkf.split(X):
+    ...     print("TRAIN:", train_index, "TEST:", test_index)
+    ...     X_train, X_test = X[train_index], X[test_index]
+    ...     y_train, y_test = y[train_index], y[test_index]
+    ...
+    TRAIN: [0 2] TEST: [1 3]
+    TRAIN: [1 3] TEST: [0 2]
+    TRAIN: [1 2] TEST: [0 3]
+    TRAIN: [0 3] TEST: [1 2]
+
+    See also
+    --------
+    RepeatedStratifiedKFold: Repeates Stratified K-Fold n times.
+    """
+    def __init__(self, n_splits=3, n_repeats=5, random_state=None):
+        super(RepeatedKFold, self).__init__(
+            KFold, n_repeats, random_state, n_splits=n_splits)
+
+
+class RepeatedStratifiedKFold(_RepeatedSplits):
+    """Repeated Stratified K-Fold cross validator.
+
+    Repeats Stratified K-Fold n times with different randomization in each
+    repetition.
+
+    Read more in the :ref:`User Guide <cross_validation>`.
+
+    Parameters
+    ----------
+    n_splits : int, default=3
+        Number of folds. Must be at least 2.
+
+    n_repeats : int, default=5
+        Number of times cross-validator needs to be repeated.
+
+    random_state : None, int or RandomState, default=None
+        Random state to be used to generate random state for each
+        repetition.
+
+    Examples
+    --------
+    >>> from sklearn.model_selection import RepeatedStratifiedKFold
+    >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
+    >>> y = np.array([0, 0, 1, 1])
+    >>> rskf = RepeatedStratifiedKFold(n_splits=2, n_repeats=2,
+    ...     random_state=36852745)
+    >>> for train_index, test_index in rskf.split(X, y):
+    ...     print("TRAIN:", train_index, "TEST:", test_index)
+    ...     X_train, X_test = X[train_index], X[test_index]
+    ...     y_train, y_test = y[train_index], y[test_index]
+    ...
+    TRAIN: [1 3] TEST: [0 2]
+    TRAIN: [0 2] TEST: [1 3]
+    TRAIN: [0 2] TEST: [1 3]
+    TRAIN: [1 3] TEST: [0 2]
+
+    See also
+    --------
+    RepeatedKFold: Repeats K-Fold n times.
+    """
+    def __init__(self, n_splits=3, n_repeats=5, random_state=None):
+        super(RepeatedStratifiedKFold, self).__init__(
+            StratifiedKFold, n_repeats, random_state, n_splits=n_splits)
+
+
 class BaseShuffleSplit(with_metaclass(ABCMeta)):
     """Base class for ShuffleSplit and StratifiedShuffleSplit"""
 
diff --git a/sklearn/model_selection/tests/test_split.py b/sklearn/model_selection/tests/test_split.py
index 601e9b259c537..d4a9685078618 100644
--- a/sklearn/model_selection/tests/test_split.py
+++ b/sklearn/model_selection/tests/test_split.py
@@ -42,6 +42,8 @@
 from sklearn.model_selection import check_cv
 from sklearn.model_selection import train_test_split
 from sklearn.model_selection import GridSearchCV
+from sklearn.model_selection import RepeatedKFold
+from sklearn.model_selection import RepeatedStratifiedKFold
 
 from sklearn.linear_model import Ridge
 
@@ -804,6 +806,76 @@ def test_leave_one_p_group_out_error_on_fewer_number_of_groups():
                          LeavePGroupsOut(n_groups=3).split(X, y, groups))
 
 
+def test_repeated_cv_value_errors():
+    # n_repeats is not integer or <= 1
+    for cv in (RepeatedKFold, RepeatedStratifiedKFold):
+        assert_raises(ValueError, cv, n_repeats=1)
+        assert_raises(ValueError, cv, n_repeats=1.5)
+
+
+def test_repeated_kfold_determinstic_split():
+    X = [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
+    random_state = 258173307
+    rkf = RepeatedKFold(
+        n_splits=2,
+        n_repeats=2,
+        random_state=random_state)
+
+    # split should produce same and deterministic splits on
+    # each call
+    for _ in range(3):
+        splits = rkf.split(X)
+        train, test = next(splits)
+        assert_array_equal(train, [0, 3])
+        assert_array_equal(test, [1, 2, 4])
+
+        train, test = next(splits)
+        assert_array_equal(train, [1, 2, 4])
+        assert_array_equal(test, [0, 3])
+
+        train, test = next(splits)
+        assert_array_equal(train, [3, 4])
+        assert_array_equal(test, [0, 1, 2])
+
+        train, test = next(splits)
+        assert_array_equal(train, [0, 1, 2])
+        assert_array_equal(test, [3, 4])
+
+        assert_raises(StopIteration, next, splits)
+
+
+def test_repeated_stratified_kfold_determinstic_split():
+    X = [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
+    y = [1, 1, 1, 0, 0]
+    random_state = 1944695409
+    rskf = RepeatedStratifiedKFold(
+        n_splits=2,
+        n_repeats=2,
+        random_state=random_state)
+
+    # split should produce same and deterministic splits on
+    # each call
+    for _ in range(3):
+        splits = rskf.split(X, y)
+        train, test = next(splits)
+        assert_array_equal(train, [0, 3])
+        assert_array_equal(test, [1, 2, 4])
+
+        train, test = next(splits)
+        assert_array_equal(train, [1, 2, 4])
+        assert_array_equal(test, [0, 3])
+
+        train, test = next(splits)
+        assert_array_equal(train, [2, 4])
+        assert_array_equal(test, [0, 1, 3])
+
+        train, test = next(splits)
+        assert_array_equal(train, [0, 1, 3])
+        assert_array_equal(test, [2, 4])
+
+        assert_raises(StopIteration, next, splits)
+
+
 def test_train_test_split_errors():
     assert_raises(ValueError, train_test_split)
     assert_raises(ValueError, train_test_split, range(3), train_size=1.1)


Code review comments:
diff_hunk: @@ -913,6 +920,165 @@ def get_n_splits(self, X, y, groups):
         return int(comb(len(np.unique(groups)), self.n_groups, exact=True))
 
 
+class _RepeatedSplits(with_metaclass(ABCMeta)):
+    """Repeated splits for an arbitrary randomized CV splitter.
+
+    Repeats splits for cross-validators n times with different randomization
+    in each repetition.
+
+    Parameters
+    ----------
+    cv : callable
+        Constructor of cross-validator.
+
+    n_repeats : int, default=5
+        Number of times cross-validator needs to be repeated.
+
+    random_state : None, int or RandomState, default=None
+        Random state to be used to generate random state for each
+        repetition.
+
+    **cvargs : additional params
+        Constructor parameters for cv. Must not contain random_state
+        and shuffle.
+    """
+    def __init__(self, cv, n_repeats=5, random_state=None, **cvargs):
+        if not isinstance(n_repeats, (np.integer, numbers.Integral)):
+            raise ValueError("Number of repetitions must be of Integral type.")
+
+        if n_repeats <= 1:
+            raise ValueError("Number of repetitions must be greater than 1.")
+
+        self.cv = cv
+        self.n_repeats = n_repeats
+        self.random_state = random_state
+        self.cvargs = cvargs
+
+    def split(self, X, y=None, groups=None):
+        """Generates indices to split data into training and test set.
+
+        Parameters
+        ----------
+        X : array-like, shape (n_samples, n_features)
+            Training data, where n_samples is the number of samples
+            and n_features is the number of features.
+
+        y : array-like, of length n_samples
+            The target variable for supervised learning problems.
+
+        groups : array-like, with shape (n_samples,), optional
+            Group labels for the samples used while splitting the dataset into
+            train/test set.
+
+        Returns
+        -------
+        train : ndarray
+            The training set indices for that split.
+
+        test : ndarray
+            The testing set indices for that split.
+        """
+        n_repeats = self.n_repeats
+        rng = check_random_state(self.random_state)
+
+        for idx in range(n_repeats):
+            random_state = rng.randint(np.iinfo(np.int32).max)
path: sklearn/model_selection/_split.py
text: @user1:
Maybe I'm missing something but why directly send `rng` to `random_state` ?

@author:
Integer random state generated by `rng` is sent as `random_state`. Do you have any other way in mind?

@user1:
Sorry I was not clear.
Can you remove the line `random_state = rng.randint(np.iinfo(np.int32).max)` and change `random_state` by `rng` later? 

@author:
Do you mean after creating the object for `cv`? If yes, how would it make a difference?

@user1:
I mean : `cv = self.cv(random_state=rng, shuffle=True, **self.cvargs)`

@author:
`rng` is being used to generate different random states for each repetition. Passing it directly to `cv` constructor will not solve the purpose and might have unexpected behavior.

@user1:
Your code is similar to the code following :
```python
In [1]: from sklearn.utils import check_random_state

In [2]: class Foo:
   ...:     def __init__(self, random_state):
   ...:         self.rng = check_random_state(random_state)
   ...:         
   ...:     def fit(self):
   ...:         print(self.rng.randint(1000))
   ...:         

In [3]: rng = check_random_state(0)

In [4]: f1 = Foo(rng)

In [5]: f2 = Foo(rng)

In [6]: f1.fit()
684

In [7]: f1.fit()
559

In [8]: f2.fit()
629

In [9]: f2.fit()
192
```
`rng` is an object and it will be modified every time you call `cv.split`. So for me is not necessary to generate a specific `random_state` at each iteration. Maybe I am missing something ?

@user1:
`check_random_state` does not create a copy of `rng` if `rng` is a `random_state`.

@author:
I can't find any case which we'll miss by this approach. But I think current implementation keeps the use of `random_state` clean. I am not really sure.

@user2 thoughts?

@user2:
I think passing rng directly should be okay. If it's not okay, we need to be able to construct a test case that proves so!

@author:
I am not able to find any such testcase. So making the changes. Thanks!