Repository: scikit-learn/scikit-learn
Pull request: [MRG+1] Change CV defaults to 5
#### Reference Issues/PRs
Fixes #11129 and takes over stalled PR #11139 

#### What does this implement/fix? Explain your changes.
Add warning for models that do not specify an explicit value for`cv` or `n_splits` to prepare for a future deprecation of the default to 3 and an update of the default value to 5.

Problem statement:
Change cv default to 5
I think it's commonly accepted that kfold cross-validation should be done with k=5 or k=10. We do 3 fold by default. Actually in a recent talk Max Kuhn said "never do threefold" and I think he was not even aware that this could have been a jibe.

Again, not sure if we should tag this 1.0 and make it a breaking change or just deprecate it because really no-one should have been using the default anyway.

If someone wants to see data, I'm not opposed to collecting data on this. Actually I think @GaelVaroquaux has some pretty in-depth studies of cross-validation, maybe he even has data.

Hints from issue discussion:
It would be a good thing.

What do you propose in terms of deprecation path, to avoid surprising
users?

Standard deprecation warning, two versions. I don't like deprecation warnings but if they get a warning they've been doing it wrong.
(I didn't tag as "help wanted" before there is consensus)
Yes, let's deprecate, with warning. When we get to version 1.0 (!) we will
automatically complete all deprecation processes, no?​

I am willing to push a PR on it if no one is yet assigned. Thanks!
@jnothman I'd say so.
The question is if the target for this one is 0.22 or 1.0.
0.22. 
+1 for switching to `cv=5` by default (with the usual FutureWarning cycle).
Question: Should we also change the default cv of other similar classes? It seems strange from my side that we change the default cv to 5 in KFold but still use 3 as default cv in e.g., StratifiedKFold.
We should definitely be changing check_cv. We should also probably be
changing all of those KFold splitters.

Commit message: fix doctests

Patch to review:
diff --git a/doc/glossary.rst b/doc/glossary.rst
index 6a3780ba0c054..533a8eac63d04 100644
--- a/doc/glossary.rst
+++ b/doc/glossary.rst
@@ -1414,7 +1414,8 @@ functions or non-estimator constructors.
         - An iterable yielding train/test splits.
 
         With some exceptions (especially where not using cross validation at
-        all is an option), the default is 3-fold.
+        all is an option), the default is 3-fold and will change to 5-fold
+        in version 0.22.
 
         ``cv`` values are validated and interpreted with :func:`utils.check_cv`.
 
diff --git a/doc/modules/cross_validation.rst b/doc/modules/cross_validation.rst
index c6d301f8edb60..f678dc6f96673 100644
--- a/doc/modules/cross_validation.rst
+++ b/doc/modules/cross_validation.rst
@@ -138,10 +138,10 @@ validation iterator instead, for instance::
 
   >>> from sklearn.model_selection import ShuffleSplit
   >>> n_samples = iris.data.shape[0]
-  >>> cv = ShuffleSplit(n_splits=3, test_size=0.3, random_state=0)
+  >>> cv = ShuffleSplit(n_splits=5, test_size=0.3, random_state=0)
   >>> cross_val_score(clf, iris.data, iris.target, cv=cv)
   ...                                                     # doctest: +ELLIPSIS
-  array([0.97..., 0.97..., 1.        ])
+  array([0.977..., 0.955..., 0.977..., 0.888..., 0.977...])
 
 
 .. topic:: Data transformation with held out data
@@ -168,7 +168,7 @@ validation iterator instead, for instance::
       >>> clf = make_pipeline(preprocessing.StandardScaler(), svm.SVC(C=1))
       >>> cross_val_score(clf, iris.data, iris.target, cv=cv)
       ...                                                 # doctest: +ELLIPSIS
-      array([0.97..., 0.93..., 0.95...])
+      array([0.977..., 0.933..., 0.955..., 0.933..., 0.977...])
 
     See :ref:`combining_estimators`.
 
@@ -462,15 +462,16 @@ generator.
 Here is a usage example::
 
   >>> from sklearn.model_selection import ShuffleSplit
-  >>> X = np.arange(5)
-  >>> ss = ShuffleSplit(n_splits=3, test_size=0.25,
+  >>> X = np.arange(10)
+  >>> ss = ShuffleSplit(n_splits=5, test_size=0.25,
   ...     random_state=0)
   >>> for train_index, test_index in ss.split(X):
   ...     print("%s %s" % (train_index, test_index))
-  ...
-  [1 3 4] [2 0]
-  [1 4 3] [0 2]
-  [4 0 2] [1 3]
+  [9 1 6 7 3 0 5] [2 8 4]
+  [2 9 8 0 6 7 4] [3 5 1]
+  [4 5 1 0 6 9 7] [2 3 8]
+  [2 7 5 8 0 3 4] [6 1 9]
+  [4 1 0 6 8 9 3] [5 2 7]
 
 :class:`ShuffleSplit` is thus a good alternative to :class:`KFold` cross
 validation that allows a finer control on the number of iterations and
diff --git a/doc/modules/linear_model.rst b/doc/modules/linear_model.rst
index bb4a9e4e57f36..dada7db1095c9 100644
--- a/doc/modules/linear_model.rst
+++ b/doc/modules/linear_model.rst
@@ -138,9 +138,9 @@ as GridSearchCV except that it defaults to Generalized Cross-Validation
 (GCV), an efficient form of leave-one-out cross-validation::
 
     >>> from sklearn import linear_model
-    >>> reg = linear_model.RidgeCV(alphas=[0.1, 1.0, 10.0])
+    >>> reg = linear_model.RidgeCV(alphas=[0.1, 1.0, 10.0], cv=3)
     >>> reg.fit([[0, 0], [0, 0], [1, 1]], [0, .1, 1])       # doctest: +SKIP
-    RidgeCV(alphas=[0.1, 1.0, 10.0], cv=None, fit_intercept=True, scoring=None,
+    RidgeCV(alphas=[0.1, 1.0, 10.0], cv=3, fit_intercept=True, scoring=None,
         normalize=False)
     >>> reg.alpha_                                      # doctest: +SKIP
     0.1
diff --git a/doc/tutorial/statistical_inference/model_selection.rst b/doc/tutorial/statistical_inference/model_selection.rst
index 3feba26c6a77d..43989cc48c8a4 100644
--- a/doc/tutorial/statistical_inference/model_selection.rst
+++ b/doc/tutorial/statistical_inference/model_selection.rst
@@ -60,19 +60,21 @@ of the chosen cross-validation strategy.
 This example shows an example usage of the ``split`` method.
 
     >>> from sklearn.model_selection import KFold, cross_val_score
-    >>> X = ["a", "a", "b", "c", "c", "c"]
-    >>> k_fold = KFold(n_splits=3)
+    >>> X = ["a", "a", "a", "b", "b", "c", "c", "c", "c", "c"]
+    >>> k_fold = KFold(n_splits=5)
     >>> for train_indices, test_indices in k_fold.split(X):
     ...      print('Train: %s | test: %s' % (train_indices, test_indices))
-    Train: [2 3 4 5] | test: [0 1]
-    Train: [0 1 4 5] | test: [2 3]
-    Train: [0 1 2 3] | test: [4 5]
+    Train: [2 3 4 5 6 7 8 9] | test: [0 1]
+    Train: [0 1 4 5 6 7 8 9] | test: [2 3]
+    Train: [0 1 2 3 6 7 8 9] | test: [4 5]
+    Train: [0 1 2 3 4 5 8 9] | test: [6 7]
+    Train: [0 1 2 3 4 5 6 7] | test: [8 9]
 
 The cross-validation can then be performed easily::
 
     >>> [svc.fit(X_digits[train], y_digits[train]).score(X_digits[test], y_digits[test])
     ...          for train, test in k_fold.split(X_digits)]  # doctest: +ELLIPSIS
-    [0.934..., 0.956..., 0.939...]
+    [0.963..., 0.922..., 0.963..., 0.963..., 0.930...]
 
 The cross-validation score can be directly calculated using the
 :func:`cross_val_score` helper. Given an estimator, the cross-validation object
@@ -86,7 +88,7 @@ Refer the :ref:`metrics module <metrics>` to learn more on the available scoring
 methods.
 
     >>> cross_val_score(svc, X_digits, y_digits, cv=k_fold, n_jobs=-1)
-    array([0.93489149, 0.95659432, 0.93989983])
+    array([0.96388889, 0.92222222, 0.9637883 , 0.9637883 , 0.93036212])
 
 `n_jobs=-1` means that the computation will be dispatched on all the CPUs
 of the computer.
@@ -96,7 +98,7 @@ scoring method.
 
     >>> cross_val_score(svc, X_digits, y_digits, cv=k_fold,
     ...                 scoring='precision_macro')
-    array([0.93969761, 0.95911415, 0.94041254])
+    array([0.96578289, 0.92708922, 0.96681476, 0.96362897, 0.93192644])
 
    **Cross-validation generators**
 
@@ -214,9 +216,9 @@ estimator during the construction and exposes an estimator API::
     >>> from sklearn.model_selection import GridSearchCV, cross_val_score
     >>> Cs = np.logspace(-6, -1, 10)
     >>> clf = GridSearchCV(estimator=svc, param_grid=dict(C=Cs),
-    ...                    n_jobs=-1)
+    ...                    n_jobs=-1, cv=3)
     >>> clf.fit(X_digits[:1000], y_digits[:1000])        # doctest: +ELLIPSIS
-    GridSearchCV(cv=None,...
+    GridSearchCV(cv=3,...
     >>> clf.best_score_                                  # doctest: +ELLIPSIS
     0.925...
     >>> clf.best_estimator_.C                            # doctest: +ELLIPSIS
@@ -229,7 +231,8 @@ estimator during the construction and exposes an estimator API::
 
 By default, the :class:`GridSearchCV` uses a 3-fold cross-validation. However,
 if it detects that a classifier is passed, rather than a regressor, it uses
-a stratified 3-fold.
+a stratified 3-fold. The default will change to a 5-fold cross-validation in
+version 0.22.
 
 .. topic:: Nested cross-validation
 
@@ -261,12 +264,12 @@ scikit-learn exposes :ref:`cross_validation` estimators that set their
 parameter automatically by cross-validation::
 
     >>> from sklearn import linear_model, datasets
-    >>> lasso = linear_model.LassoCV()
+    >>> lasso = linear_model.LassoCV(cv=3)
     >>> diabetes = datasets.load_diabetes()
     >>> X_diabetes = diabetes.data
     >>> y_diabetes = diabetes.target
     >>> lasso.fit(X_diabetes, y_diabetes)
-    LassoCV(alphas=None, copy_X=True, cv=None, eps=0.001, fit_intercept=True,
+    LassoCV(alphas=None, copy_X=True, cv=3, eps=0.001, fit_intercept=True,
         max_iter=1000, n_alphas=100, n_jobs=1, normalize=False, positive=False,
         precompute='auto', random_state=None, selection='cyclic', tol=0.0001,
         verbose=False)
diff --git a/doc/whats_new/v0.20.rst b/doc/whats_new/v0.20.rst
index 8ed142b6367ea..1ec32bdf1b246 100644
--- a/doc/whats_new/v0.20.rst
+++ b/doc/whats_new/v0.20.rst
@@ -194,7 +194,7 @@ Metrics
   :func:`metrics.roc_auc_score`. :issue:`3273` by
   :user:`Alexander Niederbühl <Alexander-N>`.
 
-- Added control over the normalization in 
+- Added control over the normalization in
   :func:`metrics.normalized_mutual_information_score` and
   :func:`metrics.adjusted_mutual_information_score` via the ``average_method``
   parameter. In version 0.22, the default normalizer for each will become
@@ -333,7 +333,7 @@ Preprocessing
 - :class:`preprocessing.RobustScaler` and :func:`preprocessing.robust_scale`
   can be fitted using sparse matrices.
   :issue:`11308` by :user:`Guillaume Lemaitre <glemaitre>`.
-  
+
 Model evaluation and meta-estimators
 
 - A scorer based on :func:`metrics.brier_score_loss` is also available.
@@ -542,12 +542,12 @@ Classifiers and regressors
   per-tree basis. The previous behavior over-weighted the Gini importance of
   features that appear in later stages. This issue only affected feature
   importances. :issue:`11176` by :user:`Gil Forsyth <gforsyth>`.
-  
-- Fixed a bug in :class:`tree.MAE` to ensure sample weights are being used 
-  during the calculation of tree MAE impurity. Previous behaviour could 
-  cause suboptimal splits to be chosen since the impurity calculation 
+
+- Fixed a bug in :class:`tree.MAE` to ensure sample weights are being used
+  during the calculation of tree MAE impurity. Previous behaviour could
+  cause suboptimal splits to be chosen since the impurity calculation
   considered all samples to be of equal weight importance.
-  :issue:`11464` by :user:`John Stott <JohnStott>`.  
+  :issue:`11464` by :user:`John Stott <JohnStott>`.
 
 Decomposition, manifold learning and clustering
 
@@ -730,10 +730,10 @@ API changes summary
 
 Classifiers and regressors
 
-- The default value of the ``n_estimators`` parameter of 
-  :class:`ensemble.RandomForestClassifier`, :class:`ensemble.RandomForestRegressor`, 
-  :class:`ensemble.ExtraTreesClassifier`, :class:`ensemble.ExtraTreesRegressor`, 
-  and :class:`ensemble.RandomTreesEmbedding` will change from 10 in version 0.20 
+- The default value of the ``n_estimators`` parameter of
+  :class:`ensemble.RandomForestClassifier`, :class:`ensemble.RandomForestRegressor`,
+  :class:`ensemble.ExtraTreesClassifier`, :class:`ensemble.ExtraTreesRegressor`,
+  and :class:`ensemble.RandomTreesEmbedding` will change from 10 in version 0.20
   to 100 in 0.22. A FutureWarning is raised when the default value is used.
   :issue:`11542` by :user:`Anna Ayzenshtat <annaayzenshtat>`.
 
@@ -797,9 +797,9 @@ Metrics
   :issue:`9851` by :user:`Hanmin Qin <qinhanmin2014>`.
 
 - In :func:`metrics.normalized_mutual_information_score` and
-  :func:`metrics.adjusted_mutual_information_score`, 
+  :func:`metrics.adjusted_mutual_information_score`,
   warn that ``average_method``
-  will have a new default value. In version 0.22, the default normalizer for each 
+  will have a new default value. In version 0.22, the default normalizer for each
   will become the *arithmetic* mean of the entropies of each clustering. Currently,
   :func:`metrics.normalized_mutual_information_score` uses the default of
   ``average_method='geometric'``, and :func:`metrics.adjusted_mutual_information_score`
@@ -901,6 +901,13 @@ Preprocessing
   will be from ``True`` to ``False`` in 0.22.
   :issue:`10655` by :user:`Guillaume Lemaitre <glemaitre>`.
 
+Model selection
+
+- The default number of cross-validation folds ``cv`` and the default number of
+  splits ``n_splits`` in the :class:`model_selection.KFold`-like splitters will change
+  from 3 to 5 in 0.22 as 3-fold has a lot of variance.
+  :issue:`11557` by :user:`Alexandre Boucaud <aboucaud>`.
+
 Changes to estimator checks
 ---------------------------
 
@@ -918,7 +925,7 @@ These changes mostly affect library developers.
   ``check_set_params`` test which checks that ``set_params`` is equivalent to
   passing parameters in ``__init__`` and warns if it encounters parameter
   validation. :issue:`7738` by :user:`Alvin Chiang <absolutelyNoWarranty>`
-  
+
 - Add invariance tests for clustering metrics. :issue:`8102` by :user:`Ankita
   Sinha <anki08>` and :user:`Guillaume Lemaitre <glemaitre>`.
 
diff --git a/examples/compose/plot_compare_reduction.py b/examples/compose/plot_compare_reduction.py
index 89d4d417290ec..1eca8e40a0727 100755
--- a/examples/compose/plot_compare_reduction.py
+++ b/examples/compose/plot_compare_reduction.py
@@ -63,7 +63,7 @@
 ]
 reducer_labels = ['PCA', 'NMF', 'KBest(chi2)']
 
-grid = GridSearchCV(pipe, cv=3, n_jobs=1, param_grid=param_grid)
+grid = GridSearchCV(pipe, cv=5, n_jobs=1, param_grid=param_grid)
 digits = load_digits()
 grid.fit(digits.data, digits.target)
 
@@ -114,7 +114,7 @@
                        memory=memory)
 
 # This time, a cached pipeline will be used within the grid search
-grid = GridSearchCV(cached_pipe, cv=3, n_jobs=1, param_grid=param_grid)
+grid = GridSearchCV(cached_pipe, cv=5, n_jobs=1, param_grid=param_grid)
 digits = load_digits()
 grid.fit(digits.data, digits.target)
 
diff --git a/examples/ensemble/plot_gradient_boosting_oob.py b/examples/ensemble/plot_gradient_boosting_oob.py
index dfae1ad9b8a98..ea38b326ce5c9 100644
--- a/examples/ensemble/plot_gradient_boosting_oob.py
+++ b/examples/ensemble/plot_gradient_boosting_oob.py
@@ -74,7 +74,7 @@ def heldout_score(clf, X_test, y_test):
     return score
 
 
-def cv_estimate(n_splits=3):
+def cv_estimate(n_splits=None):
     cv = KFold(n_splits=n_splits)
     cv_clf = ensemble.GradientBoostingClassifier(**params)
     val_scores = np.zeros((n_estimators,), dtype=np.float64)
diff --git a/sklearn/calibration.py b/sklearn/calibration.py
index 4bc376cc506a7..d2b5157604625 100644
--- a/sklearn/calibration.py
+++ b/sklearn/calibration.py
@@ -75,6 +75,10 @@ class CalibratedClassifierCV(BaseEstimator, ClassifierMixin):
         If "prefit" is passed, it is assumed that base_estimator has been
         fitted already and all data is used for calibration.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     Attributes
     ----------
     classes_ : array, shape (n_classes)
@@ -99,7 +103,7 @@ class CalibratedClassifierCV(BaseEstimator, ClassifierMixin):
     .. [4] Predicting Good Probabilities with Supervised Learning,
            A. Niculescu-Mizil & R. Caruana, ICML 2005
     """
-    def __init__(self, base_estimator=None, method='sigmoid', cv=3):
+    def __init__(self, base_estimator=None, method='sigmoid', cv='warn'):
         self.base_estimator = base_estimator
         self.method = method
         self.cv = cv
diff --git a/sklearn/covariance/graph_lasso_.py b/sklearn/covariance/graph_lasso_.py
index 2ed9e86c7829a..97bd099bd4149 100644
--- a/sklearn/covariance/graph_lasso_.py
+++ b/sklearn/covariance/graph_lasso_.py
@@ -497,6 +497,10 @@ class GraphicalLassoCV(GraphicalLasso):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     tol : positive float, optional
         The tolerance to declare convergence: if the dual gap goes below
         this value, iterations are stopped.
@@ -566,7 +570,7 @@ class GraphicalLassoCV(GraphicalLasso):
     be close to these missing values.
     """
 
-    def __init__(self, alphas=4, n_refinements=4, cv=None, tol=1e-4,
+    def __init__(self, alphas=4, n_refinements=4, cv='warn', tol=1e-4,
                  enet_tol=1e-4, max_iter=100, mode='cd', n_jobs=1,
                  verbose=False, assume_centered=False):
         super(GraphicalLassoCV, self).__init__(
@@ -900,6 +904,10 @@ class GraphLassoCV(GraphicalLassoCV):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     tol : positive float, optional
         The tolerance to declare convergence: if the dual gap goes below
         this value, iterations are stopped.
diff --git a/sklearn/feature_selection/rfe.py b/sklearn/feature_selection/rfe.py
index 31c0133b2c694..0690bc07cd936 100644
--- a/sklearn/feature_selection/rfe.py
+++ b/sklearn/feature_selection/rfe.py
@@ -312,6 +312,10 @@ class RFECV(RFE, MetaEstimatorMixin):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     scoring : string, callable or None, optional, default: None
         A string (see model evaluation documentation) or
         a scorer callable object / function with signature
@@ -382,7 +386,7 @@ class RFECV(RFE, MetaEstimatorMixin):
            for cancer classification using support vector machines",
            Mach. Learn., 46(1-3), 389--422, 2002.
     """
-    def __init__(self, estimator, step=1, cv=None, scoring=None, verbose=0,
+    def __init__(self, estimator, step=1, cv='warn', scoring=None, verbose=0,
                  n_jobs=1):
         self.estimator = estimator
         self.step = step
diff --git a/sklearn/linear_model/coordinate_descent.py b/sklearn/linear_model/coordinate_descent.py
index 38650b887f5b2..1e2a0d297dc21 100644
--- a/sklearn/linear_model/coordinate_descent.py
+++ b/sklearn/linear_model/coordinate_descent.py
@@ -1052,7 +1052,7 @@ class LinearModelCV(six.with_metaclass(ABCMeta, LinearModel)):
     @abstractmethod
     def __init__(self, eps=1e-3, n_alphas=100, alphas=None, fit_intercept=True,
                  normalize=False, precompute='auto', max_iter=1000, tol=1e-4,
-                 copy_X=True, cv=None, verbose=False, n_jobs=1,
+                 copy_X=True, cv='warn', verbose=False, n_jobs=1,
                  positive=False, random_state=None, selection='cyclic'):
         self.eps = eps
         self.n_alphas = n_alphas
@@ -1311,6 +1311,10 @@ class LassoCV(LinearModelCV, RegressorMixin):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     verbose : bool or integer
         Amount of verbosity.
 
@@ -1381,7 +1385,7 @@ class LassoCV(LinearModelCV, RegressorMixin):
 
     def __init__(self, eps=1e-3, n_alphas=100, alphas=None, fit_intercept=True,
                  normalize=False, precompute='auto', max_iter=1000, tol=1e-4,
-                 copy_X=True, cv=None, verbose=False, n_jobs=1,
+                 copy_X=True, cv='warn', verbose=False, n_jobs=1,
                  positive=False, random_state=None, selection='cyclic'):
         super(LassoCV, self).__init__(
             eps=eps, n_alphas=n_alphas, alphas=alphas,
@@ -1464,6 +1468,10 @@ class ElasticNetCV(LinearModelCV, RegressorMixin):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     copy_X : boolean, optional, default True
         If ``True``, X will be copied; else, it may be overwritten.
 
@@ -1573,7 +1581,7 @@ class ElasticNetCV(LinearModelCV, RegressorMixin):
 
     def __init__(self, l1_ratio=0.5, eps=1e-3, n_alphas=100, alphas=None,
                  fit_intercept=True, normalize=False, precompute='auto',
-                 max_iter=1000, tol=1e-4, cv=None, copy_X=True,
+                 max_iter=1000, tol=1e-4, cv='warn', copy_X=True,
                  verbose=0, n_jobs=1, positive=False, random_state=None,
                  selection='cyclic'):
         self.l1_ratio = l1_ratio
@@ -1994,6 +2002,10 @@ class MultiTaskElasticNetCV(LinearModelCV, RegressorMixin):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     copy_X : boolean, optional, default True
         If ``True``, X will be copied; else, it may be overwritten.
 
@@ -2048,11 +2060,11 @@ class MultiTaskElasticNetCV(LinearModelCV, RegressorMixin):
     Examples
     --------
     >>> from sklearn import linear_model
-    >>> clf = linear_model.MultiTaskElasticNetCV()
+    >>> clf = linear_model.MultiTaskElasticNetCV(cv=3)
     >>> clf.fit([[0,0], [1, 1], [2, 2]],
     ...         [[0, 0], [1, 1], [2, 2]])
     ... #doctest: +NORMALIZE_WHITESPACE
-    MultiTaskElasticNetCV(alphas=None, copy_X=True, cv=None, eps=0.001,
+    MultiTaskElasticNetCV(alphas=None, copy_X=True, cv=3, eps=0.001,
            fit_intercept=True, l1_ratio=0.5, max_iter=1000, n_alphas=100,
            n_jobs=1, normalize=False, random_state=None, selection='cyclic',
            tol=0.0001, verbose=0)
@@ -2079,7 +2091,7 @@ class MultiTaskElasticNetCV(LinearModelCV, RegressorMixin):
 
     def __init__(self, l1_ratio=0.5, eps=1e-3, n_alphas=100, alphas=None,
                  fit_intercept=True, normalize=False,
-                 max_iter=1000, tol=1e-4, cv=None, copy_X=True,
+                 max_iter=1000, tol=1e-4, cv='warn', copy_X=True,
                  verbose=0, n_jobs=1, random_state=None, selection='cyclic'):
         self.l1_ratio = l1_ratio
         self.eps = eps
@@ -2164,6 +2176,10 @@ class MultiTaskLassoCV(LinearModelCV, RegressorMixin):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     verbose : bool or integer
         Amount of verbosity.
 
@@ -2225,7 +2241,7 @@ class MultiTaskLassoCV(LinearModelCV, RegressorMixin):
 
     def __init__(self, eps=1e-3, n_alphas=100, alphas=None, fit_intercept=True,
                  normalize=False, max_iter=1000, tol=1e-4, copy_X=True,
-                 cv=None, verbose=False, n_jobs=1, random_state=None,
+                 cv='warn', verbose=False, n_jobs=1, random_state=None,
                  selection='cyclic'):
         super(MultiTaskLassoCV, self).__init__(
             eps=eps, n_alphas=n_alphas, alphas=alphas,
diff --git a/sklearn/linear_model/least_angle.py b/sklearn/linear_model/least_angle.py
index f0409545a26bc..9afa2d7fc1a1c 100644
--- a/sklearn/linear_model/least_angle.py
+++ b/sklearn/linear_model/least_angle.py
@@ -1013,6 +1013,10 @@ class LarsCV(Lars):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     max_n_alphas : integer, optional
         The maximum number of points on the path used to compute the
         residuals in the cross-validation
@@ -1071,7 +1075,7 @@ class LarsCV(Lars):
     method = 'lar'
 
     def __init__(self, fit_intercept=True, verbose=False, max_iter=500,
-                 normalize=True, precompute='auto', cv=None,
+                 normalize=True, precompute='auto', cv='warn',
                  max_n_alphas=1000, n_jobs=1, eps=np.finfo(np.float).eps,
                  copy_X=True, positive=False):
         self.max_iter = max_iter
@@ -1222,6 +1226,10 @@ class LassoLarsCV(LarsCV):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     max_n_alphas : integer, optional
         The maximum number of points on the path used to compute the
         residuals in the cross-validation
@@ -1297,7 +1305,7 @@ class LassoLarsCV(LarsCV):
     method = 'lasso'
 
     def __init__(self, fit_intercept=True, verbose=False, max_iter=500,
-                 normalize=True, precompute='auto', cv=None,
+                 normalize=True, precompute='auto', cv='warn',
                  max_n_alphas=1000, n_jobs=1, eps=np.finfo(np.float).eps,
                  copy_X=True, positive=False):
         self.fit_intercept = fit_intercept
diff --git a/sklearn/linear_model/logistic.py b/sklearn/linear_model/logistic.py
index 0f79b9a60bb78..9b557315cac62 100644
--- a/sklearn/linear_model/logistic.py
+++ b/sklearn/linear_model/logistic.py
@@ -1408,12 +1408,16 @@ class LogisticRegressionCV(LogisticRegression, BaseEstimator,
         Specifies if a constant (a.k.a. bias or intercept) should be
         added to the decision function.
 
-    cv : integer or cross-validation generator
+    cv : integer or cross-validation generator, default: None
         The default cross-validation generator used is Stratified K-Folds.
         If an integer is provided, then it is the number of folds used.
         See the module :mod:`sklearn.model_selection` module for the
         list of possible cross-validation objects.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     dual : bool
         Dual or primal formulation. Dual formulation is only implemented for
         l2 penalty with liblinear solver. Prefer dual=False when
@@ -1570,7 +1574,7 @@ class LogisticRegressionCV(LogisticRegression, BaseEstimator,
 
     """
 
-    def __init__(self, Cs=10, fit_intercept=True, cv=None, dual=False,
+    def __init__(self, Cs=10, fit_intercept=True, cv='warn', dual=False,
                  penalty='l2', scoring=None, solver='lbfgs', tol=1e-4,
                  max_iter=100, class_weight=None, n_jobs=1, verbose=0,
                  refit=True, intercept_scaling=1., multi_class='ovr',
diff --git a/sklearn/linear_model/omp.py b/sklearn/linear_model/omp.py
index 09d58e72b886a..3c1bd41be48f1 100644
--- a/sklearn/linear_model/omp.py
+++ b/sklearn/linear_model/omp.py
@@ -785,6 +785,10 @@ class OrthogonalMatchingPursuitCV(LinearModel, RegressorMixin):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     n_jobs : integer, optional
         Number of CPUs to use during the cross validation. If ``-1``, use
         all the CPUs
@@ -822,7 +826,7 @@ class OrthogonalMatchingPursuitCV(LinearModel, RegressorMixin):
 
     """
     def __init__(self, copy=True, fit_intercept=True, normalize=True,
-                 max_iter=None, cv=None, n_jobs=1, verbose=False):
+                 max_iter=None, cv='warn', n_jobs=1, verbose=False):
         self.copy = copy
         self.fit_intercept = fit_intercept
         self.normalize = normalize
diff --git a/sklearn/linear_model/ridge.py b/sklearn/linear_model/ridge.py
index d3f845b4442b6..10748921879c6 100644
--- a/sklearn/linear_model/ridge.py
+++ b/sklearn/linear_model/ridge.py
@@ -1091,7 +1091,7 @@ def identity_estimator():
 class _BaseRidgeCV(LinearModel):
     def __init__(self, alphas=(0.1, 1.0, 10.0),
                  fit_intercept=True, normalize=False, scoring=None,
-                 cv=None, gcv_mode=None,
+                 cv='warn', gcv_mode=None,
                  store_cv_values=False):
         self.alphas = np.asarray(alphas)
         self.fit_intercept = fit_intercept
@@ -1343,7 +1343,7 @@ class RidgeClassifierCV(LinearClassifierMixin, _BaseRidgeCV):
     """
 
     def __init__(self, alphas=(0.1, 1.0, 10.0), fit_intercept=True,
-                 normalize=False, scoring=None, cv=None, class_weight=None,
+                 normalize=False, scoring=None, cv='warn', class_weight=None,
                  store_cv_values=False):
         super(RidgeClassifierCV, self).__init__(
             alphas=alphas, fit_intercept=fit_intercept, normalize=normalize,
diff --git a/sklearn/linear_model/tests/test_least_angle.py b/sklearn/linear_model/tests/test_least_angle.py
index f5e5b2222d244..a11ed4d611b14 100644
--- a/sklearn/linear_model/tests/test_least_angle.py
+++ b/sklearn/linear_model/tests/test_least_angle.py
@@ -428,6 +428,7 @@ def test_lars_cv():
     assert_false(hasattr(lars_cv, 'n_nonzero_coefs'))
 
 
+@pytest.mark.filterwarnings('ignore::FutureWarning')
 def test_lars_cv_max_iter():
     with warnings.catch_warnings(record=True) as w:
         X = diabetes.data
diff --git a/sklearn/model_selection/_search.py b/sklearn/model_selection/_search.py
index effee28cb20ca..98760d46e79d5 100644
--- a/sklearn/model_selection/_search.py
+++ b/sklearn/model_selection/_search.py
@@ -413,7 +413,7 @@ class BaseSearchCV(six.with_metaclass(ABCMeta, BaseEstimator,
     @abstractmethod
     def __init__(self, estimator, scoring=None,
                  fit_params=None, n_jobs=1, iid='warn',
-                 refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs',
+                 refit=True, cv='warn', verbose=0, pre_dispatch='2*n_jobs',
                  error_score='raise-deprecating', return_train_score=True):
 
         self.scoring = scoring
@@ -887,6 +887,10 @@ class GridSearchCV(BaseSearchCV):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     refit : boolean, or string, default=True
         Refit an estimator using the best found parameters on the whole
         dataset.
@@ -938,10 +942,10 @@ class GridSearchCV(BaseSearchCV):
     >>> iris = datasets.load_iris()
     >>> parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
     >>> svc = svm.SVC(gamma="scale")
-    >>> clf = GridSearchCV(svc, parameters) # doctest: +SKIP
+    >>> clf = GridSearchCV(svc, parameters, cv=3) # doctest: +SKIP
     >>> clf.fit(iris.data, iris.target) # doctest: +SKIP
     ...                             # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
-    GridSearchCV(cv=None, error_score=...,
+    GridSearchCV(cv=3, error_score=...,
            estimator=SVC(C=1.0, cache_size=..., class_weight=..., coef0=...,
                          decision_function_shape='ovr', degree=..., gamma=...,
                          kernel='rbf', max_iter=-1, probability=False,
@@ -1091,7 +1095,7 @@ class GridSearchCV(BaseSearchCV):
     """
 
     def __init__(self, estimator, param_grid, scoring=None, fit_params=None,
-                 n_jobs=1, iid='warn', refit=True, cv=None, verbose=0,
+                 n_jobs=1, iid='warn', refit=True, cv='warn', verbose=0,
                  pre_dispatch='2*n_jobs', error_score='raise-deprecating',
                  return_train_score="warn"):
         super(GridSearchCV, self).__init__(
@@ -1227,6 +1231,10 @@ class RandomizedSearchCV(BaseSearchCV):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     refit : boolean, or string default=True
         Refit an estimator using the best found parameters on the whole
         dataset.
@@ -1404,7 +1412,7 @@ class RandomizedSearchCV(BaseSearchCV):
     """
 
     def __init__(self, estimator, param_distributions, n_iter=10, scoring=None,
-                 fit_params=None, n_jobs=1, iid='warn', refit=True, cv=None,
+                 fit_params=None, n_jobs=1, iid='warn', refit=True, cv='warn',
                  verbose=0, pre_dispatch='2*n_jobs', random_state=None,
                  error_score='raise-deprecating', return_train_score="warn"):
         self.param_distributions = param_distributions
diff --git a/sklearn/model_selection/_split.py b/sklearn/model_selection/_split.py
index ded8b66b3af3d..d5c8149d273fb 100644
--- a/sklearn/model_selection/_split.py
+++ b/sklearn/model_selection/_split.py
@@ -49,6 +49,17 @@
            'check_cv']
 
 
+NSPLIT_WARNING = (
+    "You should specify a value for 'n_splits' instead of relying on the "
+    "default value. The default value will change from 3 to 5 "
+    "in version 0.22.")
+
+CV_WARNING = (
+    "You should specify a value for 'cv' instead of relying on the "
+    "default value. The default value will change from 3 to 5 "
+    "in version 0.22.")
+
+
 class BaseCrossValidator(with_metaclass(ABCMeta)):
     """Base class for all cross-validators
 
@@ -358,6 +369,9 @@ class KFold(_BaseKFold):
     n_splits : int, default=3
         Number of folds. Must be at least 2.
 
+        .. deprecated:: 0.20
+            ``n_splits`` default value will change from 3 to 5 in v0.22.
+
     shuffle : boolean, optional
         Whether to shuffle the data before splitting into batches.
 
@@ -406,8 +420,11 @@ class KFold(_BaseKFold):
     RepeatedKFold: Repeats K-Fold n times.
     """
 
-    def __init__(self, n_splits=3, shuffle=False,
+    def __init__(self, n_splits='warn', shuffle=False,
                  random_state=None):
+        if n_splits is 'warn':
+            warnings.warn(NSPLIT_WARNING, FutureWarning)
+            n_splits = 3
         super(KFold, self).__init__(n_splits, shuffle, random_state)
 
     def _iter_test_indices(self, X, y=None, groups=None):
@@ -440,6 +457,9 @@ class GroupKFold(_BaseKFold):
     n_splits : int, default=3
         Number of folds. Must be at least 2.
 
+        .. deprecated:: 0.20
+            ``n_splits`` default value will change from 3 to 5 in v0.22.
+
     Examples
     --------
     >>> from sklearn.model_selection import GroupKFold
@@ -472,7 +492,10 @@ class GroupKFold(_BaseKFold):
         For splitting the data according to explicit domain-specific
         stratification of the dataset.
     """
-    def __init__(self, n_splits=3):
+    def __init__(self, n_splits='warn'):
+        if n_splits is 'warn':
+            warnings.warn(NSPLIT_WARNING, FutureWarning)
+            n_splits = 3
         super(GroupKFold, self).__init__(n_splits, shuffle=False,
                                          random_state=None)
 
@@ -530,6 +553,9 @@ class StratifiedKFold(_BaseKFold):
     n_splits : int, default=3
         Number of folds. Must be at least 2.
 
+        .. deprecated:: 0.20
+            ``n_splits`` default value will change from 3 to 5 in v0.22.
+
     shuffle : boolean, optional
         Whether to shuffle each stratification of the data before splitting
         into batches.
@@ -567,7 +593,10 @@ class StratifiedKFold(_BaseKFold):
     RepeatedStratifiedKFold: Repeats Stratified K-Fold n times.
     """
 
-    def __init__(self, n_splits=3, shuffle=False, random_state=None):
+    def __init__(self, n_splits='warn', shuffle=False, random_state=None):
+        if n_splits is 'warn':
+            warnings.warn(NSPLIT_WARNING, FutureWarning)
+            n_splits = 3
         super(StratifiedKFold, self).__init__(n_splits, shuffle, random_state)
 
     def _make_test_folds(self, X, y=None):
@@ -687,17 +716,20 @@ class TimeSeriesSplit(_BaseKFold):
     n_splits : int, default=3
         Number of splits. Must be at least 1.
 
+        .. deprecated:: 0.20
+            ``n_splits`` default value will change from 3 to 5 in v0.22.
+
     max_train_size : int, optional
         Maximum size for a single training set.
 
     Examples
     --------
     >>> from sklearn.model_selection import TimeSeriesSplit
-    >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
-    >>> y = np.array([1, 2, 3, 4])
-    >>> tscv = TimeSeriesSplit(n_splits=3)
+    >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
+    >>> y = np.array([1, 2, 3, 4, 5, 6])
+    >>> tscv = TimeSeriesSplit(n_splits=5)
     >>> print(tscv)  # doctest: +NORMALIZE_WHITESPACE
-    TimeSeriesSplit(max_train_size=None, n_splits=3)
+    TimeSeriesSplit(max_train_size=None, n_splits=5)
     >>> for train_index, test_index in tscv.split(X):
     ...    print("TRAIN:", train_index, "TEST:", test_index)
     ...    X_train, X_test = X[train_index], X[test_index]
@@ -705,6 +737,8 @@ class TimeSeriesSplit(_BaseKFold):
     TRAIN: [0] TEST: [1]
     TRAIN: [0 1] TEST: [2]
     TRAIN: [0 1 2] TEST: [3]
+    TRAIN: [0 1 2 3] TEST: [4]
+    TRAIN: [0 1 2 3 4] TEST: [5]
 
     Notes
     -----
@@ -713,7 +747,10 @@ class TimeSeriesSplit(_BaseKFold):
     with a test set of size ``n_samples//(n_splits + 1)``,
     where ``n_samples`` is the number of samples.
     """
-    def __init__(self, n_splits=3, max_train_size=None):
+    def __init__(self, n_splits='warn', max_train_size=None):
+        if n_splits is 'warn':
+            warnings.warn(NSPLIT_WARNING, FutureWarning)
+            n_splits = 3
         super(TimeSeriesSplit, self).__init__(n_splits,
                                               shuffle=False,
                                               random_state=None)
@@ -1270,27 +1307,31 @@ class ShuffleSplit(BaseShuffleSplit):
     Examples
     --------
     >>> from sklearn.model_selection import ShuffleSplit
-    >>> X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
-    >>> y = np.array([1, 2, 1, 2])
-    >>> rs = ShuffleSplit(n_splits=3, test_size=.25, random_state=0)
+    >>> X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [3, 4], [5, 6]])
+    >>> y = np.array([1, 2, 1, 2, 1, 2])
+    >>> rs = ShuffleSplit(n_splits=5, test_size=.25, random_state=0)
     >>> rs.get_n_splits(X)
-    3
+    5
     >>> print(rs)
-    ShuffleSplit(n_splits=3, random_state=0, test_size=0.25, train_size=None)
+    ShuffleSplit(n_splits=5, random_state=0, test_size=0.25, train_size=None)
     >>> for train_index, test_index in rs.split(X):
     ...    print("TRAIN:", train_index, "TEST:", test_index)
     ...  # doctest: +ELLIPSIS
-    TRAIN: [3 1 0] TEST: [2]
-    TRAIN: [2 1 3] TEST: [0]
-    TRAIN: [0 2 1] TEST: [3]
-    >>> rs = ShuffleSplit(n_splits=3, train_size=0.5, test_size=.25,
+    TRAIN: [1 3 0 4] TEST: [5 2]
+    TRAIN: [4 0 2 5] TEST: [1 3]
+    TRAIN: [1 2 4 0] TEST: [3 5]
+    TRAIN: [3 4 1 0] TEST: [5 2]
+    TRAIN: [3 5 1 0] TEST: [2 4]
+    >>> rs = ShuffleSplit(n_splits=5, train_size=0.5, test_size=.25,
     ...                   random_state=0)
     >>> for train_index, test_index in rs.split(X):
     ...    print("TRAIN:", train_index, "TEST:", test_index)
     ...  # doctest: +ELLIPSIS
-    TRAIN: [3 1] TEST: [2]
-    TRAIN: [2 1] TEST: [0]
-    TRAIN: [0 2] TEST: [3]
+    TRAIN: [1 3 0] TEST: [5 2]
+    TRAIN: [4 0 2] TEST: [1 3]
+    TRAIN: [1 2 4] TEST: [3 5]
+    TRAIN: [3 4 1] TEST: [5 2]
+    TRAIN: [3 5 1] TEST: [2 4]
     """
 
     def _iter_indices(self, X, y=None, groups=None):
@@ -1502,20 +1543,22 @@ class StratifiedShuffleSplit(BaseShuffleSplit):
     Examples
     --------
     >>> from sklearn.model_selection import StratifiedShuffleSplit
-    >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
-    >>> y = np.array([0, 0, 1, 1])
-    >>> sss = StratifiedShuffleSplit(n_splits=3, test_size=0.5, random_state=0)
+    >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
+    >>> y = np.array([0, 0, 0, 1, 1, 1])
+    >>> sss = StratifiedShuffleSplit(n_splits=5, test_size=0.5, random_state=0)
     >>> sss.get_n_splits(X, y)
-    3
+    5
     >>> print(sss)       # doctest: +ELLIPSIS
-    StratifiedShuffleSplit(n_splits=3, random_state=0, ...)
+    StratifiedShuffleSplit(n_splits=5, random_state=0, ...)
     >>> for train_index, test_index in sss.split(X, y):
     ...    print("TRAIN:", train_index, "TEST:", test_index)
     ...    X_train, X_test = X[train_index], X[test_index]
     ...    y_train, y_test = y[train_index], y[test_index]
-    TRAIN: [1 2] TEST: [3 0]
-    TRAIN: [0 2] TEST: [1 3]
-    TRAIN: [0 2] TEST: [3 1]
+    TRAIN: [5 2 3] TEST: [4 1 0]
+    TRAIN: [5 1 4] TEST: [0 2 3]
+    TRAIN: [5 0 2] TEST: [4 3 1]
+    TRAIN: [4 1 0] TEST: [2 3 5]
+    TRAIN: [0 5 1] TEST: [3 4 2]
     """
 
     def __init__(self, n_splits=10, test_size="default", train_size=None,
@@ -1859,7 +1902,7 @@ def split(self, X=None, y=None, groups=None):
             yield train, test
 
 
-def check_cv(cv=3, y=None, classifier=False):
+def check_cv(cv='warn', y=None, classifier=False):
     """Input checker utility for building a cross-validator
 
     Parameters
@@ -1880,6 +1923,9 @@ def check_cv(cv=3, y=None, classifier=False):
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value will change from 3-fold to 5-fold in v0.22.
+
     y : array-like, optional
         The target variable for supervised learning problems.
 
@@ -1893,7 +1939,8 @@ def check_cv(cv=3, y=None, classifier=False):
         The return value is a cross-validator which generates the train/test
         splits via the ``split`` method.
     """
-    if cv is None:
+    if cv is 'warn':
+        warnings.warn(CV_WARNING, FutureWarning)
         cv = 3
 
     if isinstance(cv, numbers.Integral):
diff --git a/sklearn/model_selection/_validation.py b/sklearn/model_selection/_validation.py
index 5a46ca0bf7929..95efc08872ac6 100644
--- a/sklearn/model_selection/_validation.py
+++ b/sklearn/model_selection/_validation.py
@@ -38,7 +38,7 @@
            'permutation_test_score', 'learning_curve', 'validation_curve']
 
 
-def cross_validate(estimator, X, y=None, groups=None, scoring=None, cv=None,
+def cross_validate(estimator, X, y=None, groups=None, scoring=None, cv='warn',
                    n_jobs=1, verbose=0, fit_params=None,
                    pre_dispatch='2*n_jobs', return_train_score="warn",
                    return_estimator=False):
@@ -93,6 +93,10 @@ def cross_validate(estimator, X, y=None, groups=None, scoring=None, cv=None,
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     n_jobs : integer, optional
         The number of CPUs to use to do the computation. -1 means
         'all CPUs'.
@@ -175,7 +179,8 @@ def cross_validate(estimator, X, y=None, groups=None, scoring=None, cv=None,
 
     Single metric evaluation using ``cross_validate``
 
-    >>> cv_results = cross_validate(lasso, X, y, return_train_score=False)
+    >>> cv_results = cross_validate(lasso, X, y, cv=3,
+    ...                             return_train_score=False)
     >>> sorted(cv_results.keys())                         # doctest: +ELLIPSIS
     ['fit_time', 'score_time', 'test_score']
     >>> cv_results['test_score']    # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
@@ -184,7 +189,7 @@ def cross_validate(estimator, X, y=None, groups=None, scoring=None, cv=None,
     Multiple metric evaluation using ``cross_validate``
     (please refer the ``scoring`` parameter doc for more information)
 
-    >>> scores = cross_validate(lasso, X, y,
+    >>> scores = cross_validate(lasso, X, y, cv=3,
     ...                         scoring=('r2', 'neg_mean_squared_error'),
     ...                         return_train_score=True)
     >>> print(scores['test_neg_mean_squared_error'])      # doctest: +ELLIPSIS
@@ -255,7 +260,7 @@ def cross_validate(estimator, X, y=None, groups=None, scoring=None, cv=None,
     return ret
 
 
-def cross_val_score(estimator, X, y=None, groups=None, scoring=None, cv=None,
+def cross_val_score(estimator, X, y=None, groups=None, scoring=None, cv='warn',
                     n_jobs=1, verbose=0, fit_params=None,
                     pre_dispatch='2*n_jobs'):
     """Evaluate a score by cross-validation
@@ -299,6 +304,10 @@ def cross_val_score(estimator, X, y=None, groups=None, scoring=None, cv=None,
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     n_jobs : integer, optional
         The number of CPUs to use to do the computation. -1 means
         'all CPUs'.
@@ -339,7 +348,7 @@ def cross_val_score(estimator, X, y=None, groups=None, scoring=None, cv=None,
     >>> X = diabetes.data[:150]
     >>> y = diabetes.target[:150]
     >>> lasso = linear_model.Lasso()
-    >>> print(cross_val_score(lasso, X, y))  # doctest: +ELLIPSIS
+    >>> print(cross_val_score(lasso, X, y, cv=3))  # doctest: +ELLIPSIS
     [0.33150734 0.08022311 0.03531764]
 
     See Also
@@ -612,7 +621,7 @@ def _multimetric_score(estimator, X_test, y_test, scorers):
     return scores
 
 
-def cross_val_predict(estimator, X, y=None, groups=None, cv=None, n_jobs=1,
+def cross_val_predict(estimator, X, y=None, groups=None, cv='warn', n_jobs=1,
                       verbose=0, fit_params=None, pre_dispatch='2*n_jobs',
                       method='predict'):
     """Generate cross-validated estimates for each input data point
@@ -654,6 +663,10 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None, n_jobs=1,
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     n_jobs : integer, optional
         The number of CPUs to use to do the computation. -1 means
         'all CPUs'.
@@ -714,7 +727,7 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None, n_jobs=1,
     >>> X = diabetes.data[:150]
     >>> y = diabetes.target[:150]
     >>> lasso = linear_model.Lasso()
-    >>> y_pred = cross_val_predict(lasso, X, y)
+    >>> y_pred = cross_val_predict(lasso, X, y, cv=3)
     """
     X, y, groups = indexable(X, y, groups)
 
@@ -888,7 +901,7 @@ def _index_param_value(X, v, indices):
     return safe_indexing(v, indices)
 
 
-def permutation_test_score(estimator, X, y, groups=None, cv=None,
+def permutation_test_score(estimator, X, y, groups=None, cv='warn',
                            n_permutations=100, n_jobs=1, random_state=0,
                            verbose=0, scoring=None):
     """Evaluate the significance of a cross-validated score with permutations
@@ -939,6 +952,10 @@ def permutation_test_score(estimator, X, y, groups=None, cv=None,
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     n_permutations : integer, optional
         Number of times to permute ``y``.
 
@@ -1025,8 +1042,8 @@ def _shuffle(y, groups, random_state):
 
 
 def learning_curve(estimator, X, y, groups=None,
-                   train_sizes=np.linspace(0.1, 1.0, 5), cv=None, scoring=None,
-                   exploit_incremental_learning=False, n_jobs=1,
+                   train_sizes=np.linspace(0.1, 1.0, 5), cv='warn',
+                   scoring=None, exploit_incremental_learning=False, n_jobs=1,
                    pre_dispatch="all", verbose=0, shuffle=False,
                    random_state=None):
     """Learning curve.
@@ -1085,6 +1102,10 @@ def learning_curve(estimator, X, y, groups=None,
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     scoring : string, callable or None, optional, default: None
         A string (see model evaluation documentation) or
         a scorer callable object / function with signature
@@ -1266,7 +1287,7 @@ def _incremental_fit_estimator(estimator, X, y, classes, train, test,
 
 
 def validation_curve(estimator, X, y, param_name, param_range, groups=None,
-                     cv=None, scoring=None, n_jobs=1, pre_dispatch="all",
+                     cv='warn', scoring=None, n_jobs=1, pre_dispatch="all",
                      verbose=0):
     """Validation curve.
 
@@ -1318,6 +1339,10 @@ def validation_curve(estimator, X, y, param_name, param_range, groups=None,
         Refer :ref:`User Guide <cross_validation>` for the various
         cross-validation strategies that can be used here.
 
+        .. deprecated:: 0.20
+            ``cv`` default value if None will change from 3-fold to 5-fold
+            in v0.22.
+
     scoring : string, callable or None, optional, default: None
         A string (see model evaluation documentation) or
         a scorer callable object / function with signature
diff --git a/sklearn/model_selection/tests/test_search.py b/sklearn/model_selection/tests/test_search.py
index 9bd4c475d1908..7d7c23e82ac17 100644
--- a/sklearn/model_selection/tests/test_search.py
+++ b/sklearn/model_selection/tests/test_search.py
@@ -179,6 +179,8 @@ def test_parameter_grid():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
+
 def test_grid_search():
     # Test that the best estimator contains the right value for foo_param
     clf = MockClassifier()
@@ -249,6 +251,7 @@ def test_grid_search_fit_params_deprecation():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_fit_params_two_places():
     # NOTE: Remove this test in v0.21
 
@@ -311,6 +314,7 @@ def test_grid_search_no_score():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_score_method():
     X, y = make_classification(n_samples=100, n_classes=2, flip_y=.2,
                                random_state=0)
@@ -341,6 +345,7 @@ def test_grid_search_score_method():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_groups():
     # Check if ValueError (when groups is None) propagates to GridSearchCV
     # And also check if groups is correctly passed to the cv object
@@ -375,9 +380,10 @@ def test_return_train_score_warn():
     y = np.array([0] * 5 + [1] * 5)
     grid = {'C': [1, 2]}
 
-    estimators = [GridSearchCV(LinearSVC(random_state=0), grid, iid=False),
+    estimators = [GridSearchCV(LinearSVC(random_state=0), grid,
+                               iid=False, cv=3),
                   RandomizedSearchCV(LinearSVC(random_state=0), grid,
-                                     n_iter=2, iid=False)]
+                                     n_iter=2, iid=False, cv=3)]
 
     result = {}
     for estimator in estimators:
@@ -406,6 +412,7 @@ def test_return_train_score_warn():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_classes__property():
     # Test that classes_ property matches best_estimator_.classes_
     X = np.arange(100).reshape(10, 10)
@@ -434,6 +441,7 @@ def test_classes__property():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_trivial_cv_results_attr():
     # Test search over a "grid" with only one point.
     clf = MockClassifier()
@@ -447,6 +455,7 @@ def test_trivial_cv_results_attr():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_no_refit():
     # Test that GSCV can be used for model selection alone without refitting
     clf = MockClassifier()
@@ -478,6 +487,7 @@ def test_no_refit():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_error():
     # Test that grid search will capture errors on data with different length
     X_, y_ = make_classification(n_samples=200, n_features=100, random_state=0)
@@ -488,6 +498,7 @@ def test_grid_search_error():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_one_grid_point():
     X_, y_ = make_classification(n_samples=200, n_features=100, random_state=0)
     param_dict = {"C": [1.0], "kernel": ["rbf"], "gamma": [0.1]}
@@ -503,6 +514,7 @@ def test_grid_search_one_grid_point():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_when_param_grid_includes_range():
     # Test that the best estimator contains the right value for foo_param
     clf = MockClassifier()
@@ -516,6 +528,7 @@ def test_grid_search_when_param_grid_includes_range():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_bad_param_grid():
     param_dict = {"C": 1.0}
     clf = SVC(gamma='auto')
@@ -546,6 +559,7 @@ def test_grid_search_bad_param_grid():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_sparse():
     # Test that grid search works with both dense and sparse matrices
     X_, y_ = make_classification(n_samples=200, n_features=100, random_state=0)
@@ -568,6 +582,7 @@ def test_grid_search_sparse():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_sparse_scoring():
     X_, y_ = make_classification(n_samples=200, n_features=100, random_state=0)
 
@@ -604,6 +619,7 @@ def f1_loss(y_true_, y_pred_):
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_precomputed_kernel():
     # Test that grid search works when the input features are given in the
     # form of a precomputed kernel matrix
@@ -633,6 +649,7 @@ def test_grid_search_precomputed_kernel():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_precomputed_kernel_error_nonsquare():
     # Test that grid search returns an error with a non-square precomputed
     # training kernel matrix
@@ -671,6 +688,7 @@ def test_refit():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_gridsearch_nd():
     # Pass X as list in GridSearchCV
     X_4d = np.arange(10 * 5 * 3 * 2).reshape(10, 5, 3, 2)
@@ -741,6 +759,7 @@ def check_series(x):
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_unsupervised_grid_search():
     # test grid-search with unsupervised estimator
     X, y = make_blobs(random_state=0)
@@ -768,6 +787,7 @@ def test_unsupervised_grid_search():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_gridsearch_no_predict():
     # test grid-search with an estimator without predict.
     # slight duplication of a test from KDE
@@ -1121,6 +1141,7 @@ def compare_refit_methods_when_refit_with_acc(search_multi, search_acc, refit):
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_search_cv_results_rank_tie_breaking():
     X, y = make_blobs(n_samples=50, random_state=42)
 
@@ -1153,6 +1174,7 @@ def test_search_cv_results_rank_tie_breaking():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_search_cv_results_none_param():
     X, y = [[1], [2], [3], [4], [5]], [0, 0, 0, 0, 1]
     estimators = (DecisionTreeRegressor(), DecisionTreeClassifier())
@@ -1230,6 +1252,7 @@ def test_grid_search_correct_score_results():
                 assert_almost_equal(correct_score, cv_scores[i])
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_fit_grid_point():
     X, y = make_classification(random_state=0)
     cv = StratifiedKFold(random_state=0)
@@ -1259,6 +1282,7 @@ def test_fit_grid_point():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_pickle():
     # Test that a fit search can be pickled
     clf = MockClassifier()
@@ -1277,6 +1301,7 @@ def test_pickle():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_with_multioutput_data():
     # Test search with multi-output estimator
 
@@ -1410,6 +1435,7 @@ def get_cand_scores(i):
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_grid_search_failing_classifier_raise():
     # GridSearchCV with on_error == 'raise' raises the error
 
@@ -1462,6 +1488,7 @@ def test_parameters_sampler_replacement():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_stochastic_gradient_loss_param():
     # Make sure the predict_proba works when loss is specified
     # as one of the parameters in the param_grid.
@@ -1493,6 +1520,7 @@ def test_stochastic_gradient_loss_param():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_search_train_scores_set_to_false():
     X = np.arange(6).reshape(6, -1)
     y = [0, 0, 0, 1, 1, 1]
@@ -1586,6 +1614,7 @@ def _pop_time_keys(cv_results):
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')  # 0.22
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_transform_inverse_transform_round_trip():
     clf = MockClassifier()
     grid_search = GridSearchCV(clf, {'foo_param': [1, 2, 3]}, verbose=3)
diff --git a/sklearn/model_selection/tests/test_split.py b/sklearn/model_selection/tests/test_split.py
index 98a1f808b4d74..129e9247cd261 100644
--- a/sklearn/model_selection/tests/test_split.py
+++ b/sklearn/model_selection/tests/test_split.py
@@ -1,7 +1,7 @@
 """Test the split module"""
 from __future__ import division
 import warnings
-
+import pytest
 import numpy as np
 from scipy.sparse import coo_matrix, csc_matrix, csr_matrix
 from scipy import stats
@@ -23,6 +23,7 @@
 from sklearn.utils.testing import assert_warns
 from sklearn.utils.testing import assert_raise_message
 from sklearn.utils.testing import ignore_warnings
+from sklearn.utils.testing import assert_no_warnings
 from sklearn.utils.validation import _num_samples
 from sklearn.utils.mocking import MockDataFrame
 
@@ -50,6 +51,8 @@
 from sklearn.model_selection._split import _validate_shuffle_split
 from sklearn.model_selection._split import _CVIterableWrapper
 from sklearn.model_selection._split import _build_repr
+from sklearn.model_selection._split import CV_WARNING
+from sklearn.model_selection._split import NSPLIT_WARNING
 
 from sklearn.datasets import load_digits
 from sklearn.datasets import make_classification
@@ -199,6 +202,7 @@ def test_cross_validator_with_default_params():
                          lpo.get_n_splits, None, y, groups)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_2d_y():
     # smoke test for 2d y and multi-label
     n_samples = 30
@@ -1398,6 +1402,7 @@ def test_time_series_max_train_size():
     _check_time_series_max_train_size(splits, check_splits, max_train_size=2)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_nested_cv():
     # Test if nested cross validation works with different combinations of cv
     rng = np.random.RandomState(0)
@@ -1423,6 +1428,26 @@ def test_train_test_default_warning():
                  train_size=0.75)
 
 
+def test_nsplit_default_warn():
+    # Test that warnings are raised. Will be removed in 0.22
+    assert_warns_message(FutureWarning, NSPLIT_WARNING, KFold)
+    assert_warns_message(FutureWarning, NSPLIT_WARNING, GroupKFold)
+    assert_warns_message(FutureWarning, NSPLIT_WARNING, StratifiedKFold)
+    assert_warns_message(FutureWarning, NSPLIT_WARNING, TimeSeriesSplit)
+
+    assert_no_warnings(KFold, n_splits=5)
+    assert_no_warnings(GroupKFold, n_splits=5)
+    assert_no_warnings(StratifiedKFold, n_splits=5)
+    assert_no_warnings(TimeSeriesSplit, n_splits=5)
+
+
+def test_check_cv_default_warn():
+    # Test that warnings are raised. Will be removed in 0.22
+    assert_warns_message(FutureWarning, CV_WARNING, check_cv)
+
+    assert_no_warnings(check_cv, cv=5)
+
+
 def test_build_repr():
     class MockSplitter:
         def __init__(self, a, b=0, c=None):
diff --git a/sklearn/model_selection/tests/test_validation.py b/sklearn/model_selection/tests/test_validation.py
index f0a6564270b7b..e6025e6efac00 100644
--- a/sklearn/model_selection/tests/test_validation.py
+++ b/sklearn/model_selection/tests/test_validation.py
@@ -240,6 +240,7 @@ def get_params(self, deep=False):
 
 
 @pytest.mark.filterwarnings('ignore: From version 0.22, errors during fit')
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 # FIXME issue in error_score parameter
 def test_cross_val_score():
     clf = MockClassifier()
@@ -281,6 +282,7 @@ def test_cross_val_score():
     assert_raises(ValueError, cross_val_score, clf, X_3d, y2)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_validate_invalid_scoring_param():
     X, y = make_classification(random_state=0)
     estimator = MockClassifier()
@@ -402,7 +404,7 @@ def test_cross_validate_return_train_score_warn():
     result = {}
     for val in [False, True, 'warn']:
         result[val] = assert_no_warnings(cross_validate, estimator, X, y,
-                                         return_train_score=val)
+                                         return_train_score=val, cv=5)
 
     msg = (
         'You are accessing a training score ({!r}), '
@@ -509,6 +511,7 @@ def check_cross_validate_multi_metric(clf, X, y, scores):
             assert np.all(cv_results['score_time'] < 10)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_score_predict_groups():
     # Check if ValueError (when groups is None) propagates to cross_val_score
     # and cross_val_predict
@@ -529,6 +532,7 @@ def test_cross_val_score_predict_groups():
 
 
 @pytest.mark.filterwarnings('ignore: Using or importing the ABCs from')
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_score_pandas():
     # check cross_val_score doesn't destroy pandas dataframe
     types = [(MockDataFrame, MockDataFrame)]
@@ -566,6 +570,7 @@ def test_cross_val_score_mask():
     assert_array_equal(scores_indices, scores_masks)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_score_precomputed():
     # test for svm with precomputed kernel
     svm = SVC(kernel="precomputed")
@@ -592,6 +597,7 @@ def test_cross_val_score_precomputed():
                   linear_kernel.tolist(), y)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_score_fit_params():
     clf = MockClassifier()
     n_samples = X.shape[0]
@@ -785,6 +791,7 @@ def test_cross_val_score_multilabel():
     assert_almost_equal(score_samples, [1, 1 / 2, 3 / 4, 1 / 2, 1 / 4])
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_predict():
     boston = load_boston()
     X, y = boston.data, boston.target
@@ -834,6 +841,7 @@ def split(self, X, y=None, groups=None):
                          X, y, method='predict_proba', cv=KFold(2))
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_predict_decision_function_shape():
     X, y = make_classification(n_classes=2, n_samples=50, random_state=0)
 
@@ -880,6 +888,7 @@ def test_cross_val_predict_decision_function_shape():
                         cv=KFold(n_splits=3), method='decision_function')
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_predict_predict_proba_shape():
     X, y = make_classification(n_classes=2, n_samples=50, random_state=0)
 
@@ -894,6 +903,7 @@ def test_cross_val_predict_predict_proba_shape():
     assert_equal(preds.shape, (150, 3))
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_predict_predict_log_proba_shape():
     X, y = make_classification(n_classes=2, n_samples=50, random_state=0)
 
@@ -908,6 +918,7 @@ def test_cross_val_predict_predict_log_proba_shape():
     assert_equal(preds.shape, (150, 3))
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_predict_input_types():
     iris = load_iris()
     X, y = iris.data, iris.target
@@ -954,6 +965,7 @@ def test_cross_val_predict_input_types():
 
 
 @pytest.mark.filterwarnings('ignore: Using or importing the ABCs from')
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 # python3.7 deprecation warnings in pandas via matplotlib :-/
 def test_cross_val_predict_pandas():
     # check cross_val_score doesn't destroy pandas dataframe
@@ -972,6 +984,7 @@ def test_cross_val_predict_pandas():
         cross_val_predict(clf, X_df, y_ser)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_score_sparse_fit_params():
     iris = load_iris()
     X, y = iris.data, iris.target
@@ -1325,11 +1338,13 @@ def check_cross_val_predict_with_method(est):
         assert_array_equal(predictions, predictions_ystr)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_predict_with_method():
     check_cross_val_predict_with_method(LogisticRegression())
 
 
 @pytest.mark.filterwarnings('ignore: max_iter and tol parameters')
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_predict_method_checking():
     # Regression test for issue #9639. Tests that cross_val_predict does not
     # check estimator methods (e.g. predict_proba) before fitting
@@ -1338,6 +1353,7 @@ def test_cross_val_predict_method_checking():
 
 
 @pytest.mark.filterwarnings('ignore: The default of the `iid`')
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_gridsearchcv_cross_val_predict_with_method():
     est = GridSearchCV(LogisticRegression(random_state=42),
                        {'C': [0.1, 1]},
@@ -1365,6 +1381,7 @@ def get_expected_predictions(X, y, cv, classes, est, method):
     return expected_predictions
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_cross_val_predict_class_subset():
 
     X = np.arange(200).reshape(100, 2)
@@ -1406,6 +1423,7 @@ def test_cross_val_predict_class_subset():
         assert_array_almost_equal(expected_predictions, predictions)
 
 
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_score_memmap():
     # Ensure a scalar score of memmap type is accepted
     iris = load_iris()
@@ -1434,6 +1452,7 @@ def test_score_memmap():
 
 
 @pytest.mark.filterwarnings('ignore: Using or importing the ABCs from')
+@pytest.mark.filterwarnings('ignore: You should specify a value')
 def test_permutation_test_score_pandas():
     # check permutation_test_score doesn't destroy pandas dataframe
     types = [(MockDataFrame, MockDataFrame)]


Code review comments:
diff_hunk: @@ -214,9 +216,9 @@ estimator during the construction and exposes an estimator API::
     >>> from sklearn.model_selection import GridSearchCV, cross_val_score
     >>> Cs = np.logspace(-6, -1, 10)
     >>> clf = GridSearchCV(estimator=svc, param_grid=dict(C=Cs),
-    ...                    n_jobs=-1)
+    ...                    n_jobs=-1, cv=3)
path: doc/tutorial/statistical_inference/model_selection.rst
text: @user1:
I think I'd rather use 5 because 3 is really weird.

@author:
It was simply to make the doctests pass because I was not aware of `# doctest: SKIP`