Repository: automl/auto-sklearn

Issue:
Fix BaseCV objects early crashes
When specifying customized cross validation objects,  AutoML process tends to crash in its early stage.
The corresponding debug message shown below
```
[DEBUG] [2020-06-25 13:06:54,163:AutoMLSMBO(2176371645)::129e649232a63f02b1825a7f17c56883]
Return: Status: <StatusType.CRASHED: 3>, cost: 1.000000, time: 0.059507, 
additional: {'traceback': 'Traceback      (most recent call last):
\n  File "/home/charles/anaconda3/envs/autoskdev/lib/python3.7/site-packages/autosklearn/evaluation/__init__.py", line 29, 
in fit_predict_try_except_decorator\n    return ta(queue=queue, **kwargs)\n  
File "/home/charles/anaconda3/envs/autoskdev/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py", 
line 1237, in eval_cv\n    budget_type=budget_type,\n  
File "/home/charles/anaconda3/envs/autoskdev/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py",
 line 180, in __init__\n    self.splitter = self.get_splitter(self.datamanager)\n  
File  "/home/charles/anaconda3/envs/autoskdev/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py", 
line 952, in get_splitter\n    cv = copy.deepcopy(self.resampling_strategy)(**init_dict)
\nTypeError: __init__() got an unexpected keyword argument \'cost_for_crash\'\n', 'error': 
'TypeError("__init__() got an unexpected keyword argument \'cost_for_crash\'")', 
'configuration_origin': 'Random initial design.'}
```


It is basically caused by deepcopying _resampling_strategy_args_ into _init_dict_ in **train_evaluator.py**. It obtains extra parameters that are not for initializing cross validation objects. Therefore, I did some modification to make sure init_dict only contain defaults arguments defined in the beginning of **train_evaluator.py**. Besides, due to the change mentioned, PredefinedSplits also got default arguments to avoid failure in unit test.


The results of same resampling_strategy for example KFold and 'cv' should get the same results with same arguments and seeds. However, they have quite different outcomes in cv_results_, pipeline, and hyper-parameters in the end('cv' apparently overfitted and KFold did not). Not sure what caused this. I will do more research on it later this week.


Discussion:
Hi @mfeurer, can you do a small code review?  I think the problem is now fixed.
Regarding the different results from same fold of 'cv' and KFold I mentioned above was just because 'cv' without shuffle parameter will be default set to True in **train_evaluator.py**(code snippet below). I rechecked them with same folds and set shuffle both as False and got the same cv results in the end. 

Thank you!

```python
962   y = D.data['Y_train']
963   shuffle = self.resampling_strategy_args.get('shuffle', True)
```
Hey @charlesfu4 I'm currently busy attending the ICML conference. I'll do my best to have a look at this PR next week.
# [Codecov](https://codecov.io/gh/automl/auto-sklearn/pull/897?src=pr&el=h1) Report
> Merging [#897](https://codecov.io/gh/automl/auto-sklearn/pull/897?src=pr&el=desc) into [development](https://codecov.io/gh/automl/auto-sklearn/commit/2786d636e92507323b21be7692fbbf8b3f37f7f3&el=desc) will **decrease** coverage by `0.09%`.
> The diff coverage is `66.66%`.

[![Impacted file tree graph](https://codecov.io/gh/automl/auto-sklearn/pull/897/graphs/tree.svg?width=650&height=150&src=pr&token=5Cs17RkYm1)](https://codecov.io/gh/automl/auto-sklearn/pull/897?src=pr&el=tree)

```diff
@@               Coverage Diff               @@
##           development     #897      +/-   ##
===============================================
- Coverage        84.98%   84.89%   -0.10%     
===============================================
  Files              128      128              
  Lines             9439     9442       +3     
===============================================
- Hits              8022     8016       -6     
- Misses            1417     1426       +9     
```


| [Impacted Files](https://codecov.io/gh/automl/auto-sklearn/pull/897?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [autosklearn/evaluation/train\_evaluator.py](https://codecov.io/gh/automl/auto-sklearn/pull/897/diff?src=pr&el=tree#diff-YXV0b3NrbGVhcm4vZXZhbHVhdGlvbi90cmFpbl9ldmFsdWF0b3IucHk=) | `72.58% <66.66%> (-0.05%)` | :arrow_down: |
| [...eline/components/feature\_preprocessing/fast\_ica.py](https://codecov.io/gh/automl/auto-sklearn/pull/897/diff?src=pr&el=tree#diff-YXV0b3NrbGVhcm4vcGlwZWxpbmUvY29tcG9uZW50cy9mZWF0dXJlX3ByZXByb2Nlc3NpbmcvZmFzdF9pY2EucHk=) | `91.30% <0.00%> (-6.53%)` | :arrow_down: |
| [...mponents/feature\_preprocessing/nystroem\_sampler.py](https://codecov.io/gh/automl/auto-sklearn/pull/897/diff?src=pr&el=tree#diff-YXV0b3NrbGVhcm4vcGlwZWxpbmUvY29tcG9uZW50cy9mZWF0dXJlX3ByZXByb2Nlc3NpbmcvbnlzdHJvZW1fc2FtcGxlci5weQ==) | `85.29% <0.00%> (-5.89%)` | :arrow_down: |
| [...ine/components/classification/gradient\_boosting.py](https://codecov.io/gh/automl/auto-sklearn/pull/897/diff?src=pr&el=tree#diff-YXV0b3NrbGVhcm4vcGlwZWxpbmUvY29tcG9uZW50cy9jbGFzc2lmaWNhdGlvbi9ncmFkaWVudF9ib29zdGluZy5weQ==) | `91.89% <0.00%> (-0.91%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/automl/auto-sklearn/pull/897?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/automl/auto-sklearn/pull/897?src=pr&el=footer). Last update [2786d63...bf5b187](https://codecov.io/gh/automl/auto-sklearn/pull/897?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> Hey @charlesfu4 I'm currently busy attending the ICML conference. I'll do my best to have a look at this PR next week.

Thanks a lot, hope you enjoy it! By the way, do you know why my codecov build keeps staying at 2 month ago? Is it a normal bug or just something I forget to set up?
Hey @charlesfu4 I just merged #908 to allow custom resampling strategies. As you said, this PR also allows using multi-output regression with custom resampling strategies. Would you like to create a new PR allowing custom resampling strategies for multi-output regression?
> Hey @charlesfu4 I just merged #908 to allow custom resampling strategies. As you said, this PR also allows using multi-output regression with custom resampling strategies. Would you like to create a new PR allowing custom resampling strategies for multi-output regression?

Sure, thank you! I will do it soon.

Resolving patch:
diff --git a/autosklearn/evaluation/train_evaluator.py b/autosklearn/evaluation/train_evaluator.py
--- a/autosklearn/evaluation/train_evaluator.py
+++ b/autosklearn/evaluation/train_evaluator.py
@@ -14,6 +14,8 @@
 from autosklearn.constants import (
     CLASSIFICATION_TASKS,
     MULTILABEL_CLASSIFICATION,
+    REGRESSION_TASKS,
+    MULTIOUTPUT_REGRESSION
 )
 
 
@@ -909,7 +911,13 @@ def get_splitter(self, D):
                     raise ValueError('Unknown CrossValidator.')
                 ref_arg_dict = __baseCrossValidator_defaults__[class_name]
 
-                y = D.data['Y_train'].ravel()
+                y = D.data['Y_train']
+                if (D.info['task'] in CLASSIFICATION_TASKS and
+                   D.info['task'] != MULTILABEL_CLASSIFICATION) or \
+                   (D.info['task'] in REGRESSION_TASKS and
+                   D.info['task'] != MULTIOUTPUT_REGRESSION):
+
+                    y = y.ravel()
                 if class_name == 'PredefinedSplit':
                     if 'test_fold' not in self.resampling_strategy_args:
                         raise ValueError('Must provide parameter test_fold'
@@ -922,7 +930,7 @@ def get_splitter(self, D):
                         raise ValueError('Must provide parameter groups '
                                          'for chosen CrossValidator.')
                     try:
-                        if self.resampling_strategy_args['groups'].shape != y.shape:
+                        if self.resampling_strategy_args['groups'].shape[0] != y.shape[0]:
                             raise ValueError('Groups must be array-like '
                                              'with shape (n_samples,).')
                     except Exception:
@@ -930,7 +938,7 @@ def get_splitter(self, D):
                                          'with shape (n_samples,).')
                 else:
                     if 'groups' in self.resampling_strategy_args:
-                        if self.resampling_strategy_args['groups'].shape != y.shape:
+                        if self.resampling_strategy_args['groups'].shape[0] != y.shape[0]:
                             raise ValueError('Groups must be array-like'
                                              ' with shape (n_samples,).')
 


Test patch:
diff --git a/test/test_evaluation/test_train_evaluator.py b/test/test_evaluation/test_train_evaluator.py
--- a/test/test_evaluation/test_train_evaluator.py
+++ b/test/test_evaluation/test_train_evaluator.py
@@ -26,7 +26,8 @@
 from autosklearn.constants import BINARY_CLASSIFICATION, \
     MULTILABEL_CLASSIFICATION,\
     MULTICLASS_CLASSIFICATION,\
-    REGRESSION
+    REGRESSION,\
+    MULTIOUTPUT_REGRESSION
 from autosklearn.metrics import accuracy, r2, f1_macro
 
 this_directory = os.path.dirname(__file__)
@@ -1149,6 +1150,50 @@ def test_get_splitter(self, te_mock):
         self.assertIsInstance(cv, sklearn.model_selection._split.KFold)
         self.assertFalse(cv.shuffle)
 
+        # multioutput regression, shuffle split
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = 'holdout'
+        evaluator.resampling_strategy_args = {}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv,
+                              sklearn.model_selection._split.ShuffleSplit)
+
+        # multioutput regression, no shuffle
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = 'holdout'
+        evaluator.resampling_strategy_args = {'shuffle': False}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv,
+                              sklearn.model_selection._split.PredefinedSplit)
+
+        # multioutput regression cv, KFold
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = 'cv'
+        evaluator.resampling_strategy_args = {'folds': 5}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, sklearn.model_selection._split.KFold)
+        self.assertTrue(cv.shuffle)
+
+        # multioutput regression cv, KFold, no shuffling
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = 'cv'
+        evaluator.resampling_strategy_args = {'folds': 5, 'shuffle': False}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, sklearn.model_selection._split.KFold)
+        self.assertFalse(cv.shuffle)
+
     @unittest.mock.patch.object(TrainEvaluator, "__init__")
     def test_get_splitter_cv_object(self, te_mock):
         te_mock.return_value = None
@@ -1173,8 +1218,61 @@ def test_get_splitter_cv_object(self, te_mock):
         evaluator = TrainEvaluator()
         evaluator.resampling_strategy = GroupKFold
         evaluator.resampling_strategy_args = None
-        with self.assertRaises(ValueError):
-            evaluator.get_splitter(D)
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
+
+        # GroupKFold, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = GroupKFold
+        evaluator.resampling_strategy_args = {'folds': 2, 'groups': np.array([1, 1, 2, 1, 2, 2])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, GroupKFold)
+        self.assertEqual(cv.get_n_splits(groups=evaluator.resampling_strategy_args['groups']), 2)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # GroupKFold, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = GroupKFold
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
+
+        # GroupKFold, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = GroupKFold
+        evaluator.resampling_strategy_args = {'folds': 2, 'groups': np.array([1, 1, 2, 1, 2, 2])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, GroupKFold)
+        self.assertEqual(cv.get_n_splits(groups=evaluator.resampling_strategy_args['groups']), 2)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # GroupKFold, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = GroupKFold
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
 
         # KFold, classification with args
         D.data['Y_train'] = np.array([0, 0, 0, 1, 1, 1])
@@ -1186,7 +1284,7 @@ def test_get_splitter_cv_object(self, te_mock):
         self.assertIsInstance(cv, KFold)
         self.assertEqual(cv.get_n_splits(
             groups=evaluator.resampling_strategy_args['groups']), 4)
-        self.assertEqual(cv.shuffle, True)
+        self.assertTrue(cv.shuffle)
         self.assertEqual(cv.random_state, 5)
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
@@ -1200,7 +1298,71 @@ def test_get_splitter_cv_object(self, te_mock):
         self.assertIsInstance(cv, KFold)
         self.assertEqual(cv.get_n_splits(
             groups=evaluator.resampling_strategy_args['groups']), 3)
-        self.assertEqual(cv.shuffle, False)
+        self.assertFalse(cv.shuffle)
+        self.assertIsNone(cv.random_state)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # KFold, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = KFold
+        evaluator.resampling_strategy_args = {'folds': 4, 'shuffle': True,
+                                              'random_state': 5}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, KFold)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 4)
+        self.assertTrue(cv.shuffle)
+        self.assertEqual(cv.random_state, 5)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # KFold, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = KFold
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, KFold)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 3)
+        self.assertFalse(cv.shuffle)
+        self.assertIsNone(cv.random_state)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # KFold, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = KFold
+        evaluator.resampling_strategy_args = {'folds': 4, 'shuffle': True,
+                                              'random_state': 5}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, KFold)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 4)
+        self.assertTrue(cv.shuffle)
+        self.assertEqual(cv.random_state, 5)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # KFold, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = KFold
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, KFold)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 3)
+        self.assertFalse(cv.shuffle)
         self.assertIsNone(cv.random_state)
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
@@ -1220,8 +1382,59 @@ def test_get_splitter_cv_object(self, te_mock):
         evaluator = TrainEvaluator()
         evaluator.resampling_strategy = LeaveOneGroupOut
         evaluator.resampling_strategy_args = None
-        with self.assertRaises(ValueError):
-            evaluator.get_splitter(D)
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
+
+        # LeaveOneGroupOut, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeaveOneGroupOut
+        evaluator.resampling_strategy_args = {'groups': np.array([1, 1, 2, 1, 2, 2])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeaveOneGroupOut)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # LeaveOneGroupOut, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeaveOneGroupOut
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
+
+        # LeaveOneGroupOut, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeaveOneGroupOut
+        evaluator.resampling_strategy_args = {'groups': np.array([1, 1, 2, 1, 2, 2])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeaveOneGroupOut)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # LeaveOneGroupOut, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeaveOneGroupOut
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
 
         # LeavePGroupsOut, classification with args
         D.data['Y_train'] = np.array([0, 0, 0, 1, 1, 1])
@@ -1240,8 +1453,63 @@ def test_get_splitter_cv_object(self, te_mock):
         evaluator = TrainEvaluator()
         evaluator.resampling_strategy = LeavePGroupsOut
         evaluator.resampling_strategy_args = None
-        with self.assertRaises(ValueError):
-            evaluator.get_splitter(D)
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
+
+        # LeavePGroupsOut, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeavePGroupsOut
+        evaluator.resampling_strategy_args = {'n_groups': 1,
+                                              'groups': np.array([1, 1, 2, 1, 2, 2])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeavePGroupsOut)
+        self.assertEqual(cv.n_groups, 1)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # LeavePGroupsOut, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeavePGroupsOut
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
+
+        # LeavePGroupsOut, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeavePGroupsOut
+        evaluator.resampling_strategy_args = {'n_groups': 1,
+                                              'groups': np.array([1, 1, 2, 1, 2, 2])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeavePGroupsOut)
+        self.assertEqual(cv.n_groups, 1)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # LeavePGroupsOut, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeavePGroupsOut
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
 
         # LeaveOneOut, classification
         D.data['Y_train'] = np.array([0, 0, 0, 1, 1, 1])
@@ -1253,6 +1521,29 @@ def test_get_splitter_cv_object(self, te_mock):
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
 
+        # LeaveOneOut, regression
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeaveOneOut
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeaveOneOut)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # LeaveOneOut, multi-output regression
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeaveOneOut
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeaveOneOut)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
         # LeavePOut, classification with args
         D.data['Y_train'] = np.array([0, 0, 0, 1, 1, 1])
         evaluator = TrainEvaluator()
@@ -1275,6 +1566,55 @@ def test_get_splitter_cv_object(self, te_mock):
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
 
+        # LeavePOut, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeavePOut
+        evaluator.resampling_strategy_args = {'p': 3}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeavePOut)
+        self.assertEqual(cv.p, 3)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # LeavePOut, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeavePOut
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeavePOut)
+        self.assertEqual(cv.p, 2)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # LeavePOut, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeavePOut
+        evaluator.resampling_strategy_args = {'p': 3}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeavePOut)
+        self.assertEqual(cv.p, 3)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # LeavePOut, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = LeavePOut
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, LeavePOut)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
         # PredefinedSplit, classification with args
         D.data['Y_train'] = np.array([0, 0, 0, 1, 1, 1])
         evaluator = TrainEvaluator()
@@ -1290,8 +1630,59 @@ def test_get_splitter_cv_object(self, te_mock):
         evaluator = TrainEvaluator()
         evaluator.resampling_strategy = PredefinedSplit
         evaluator.resampling_strategy_args = None
-        with self.assertRaises(ValueError):
-            evaluator.get_splitter(D)
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter test_fold for class PredefinedSplit.',
+            evaluator.get_splitter,
+            D)
+
+        # PredefinedSplit, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = PredefinedSplit
+        evaluator.resampling_strategy_args = {'test_fold': np.array([0, 1, 0, 1, 0, 1])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, PredefinedSplit)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # PredefinedSplit, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = PredefinedSplit
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter test_fold for class PredefinedSplit.',
+            evaluator.get_splitter,
+            D)
+
+        # PredefinedSplit, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = PredefinedSplit
+        evaluator.resampling_strategy_args = {'test_fold': np.array([0, 1, 0, 1, 0, 1])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, PredefinedSplit)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # PredefinedSplit, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = PredefinedSplit
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter test_fold for class PredefinedSplit.',
+            evaluator.get_splitter,
+            D)
 
         # RepeatedKFold, classification with args
         D.data['Y_train'] = np.array([0, 0, 0, 1, 1, 1])
@@ -1322,6 +1713,70 @@ def test_get_splitter_cv_object(self, te_mock):
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
 
+        # RepeatedKFold, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = RepeatedKFold
+        evaluator.resampling_strategy_args = {'folds': 4, 'n_repeats': 3,
+                                              'random_state': 5}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, RepeatedKFold)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 4*3)
+        self.assertEqual(cv.n_repeats, 3)
+        self.assertEqual(cv.random_state, 5)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # RepeatedKFold, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = RepeatedKFold
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, RepeatedKFold)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 5*10)
+        self.assertEqual(cv.n_repeats, 10)
+        self.assertIsNone(cv.random_state)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # RepeatedKFold, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = RepeatedKFold
+        evaluator.resampling_strategy_args = {'folds': 4, 'n_repeats': 3,
+                                              'random_state': 5}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, RepeatedKFold)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 4*3)
+        self.assertEqual(cv.n_repeats, 3)
+        self.assertEqual(cv.random_state, 5)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # RepeatedKFold, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = RepeatedKFold
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, RepeatedKFold)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 5*10)
+        self.assertEqual(cv.n_repeats, 10)
+        self.assertIsNone(cv.random_state)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
         # RepeatedStratifiedKFold, classification with args
         D.data['Y_train'] = np.array([0, 0, 0, 1, 1, 1])
         evaluator = TrainEvaluator()
@@ -1361,7 +1816,7 @@ def test_get_splitter_cv_object(self, te_mock):
         self.assertIsInstance(cv, StratifiedKFold)
         self.assertEqual(cv.get_n_splits(
             groups=evaluator.resampling_strategy_args['groups']), 2)
-        self.assertEqual(cv.shuffle, True)
+        self.assertTrue(cv.shuffle)
         self.assertEqual(cv.random_state, 5)
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
@@ -1375,11 +1830,41 @@ def test_get_splitter_cv_object(self, te_mock):
         self.assertIsInstance(cv, StratifiedKFold)
         self.assertEqual(cv.get_n_splits(
             groups=evaluator.resampling_strategy_args['groups']), 3)
-        self.assertEqual(cv.shuffle, False)
+        self.assertFalse(cv.shuffle)
         self.assertIsNone(cv.random_state)
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
 
+        # TimeSeriesSplit, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = TimeSeriesSplit
+        evaluator.resampling_strategy_args = {'folds': 4, 'max_train_size': 3}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, TimeSeriesSplit)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 4)
+        self.assertEqual(cv.max_train_size, 3)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # TimeSeriesSplit, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = TimeSeriesSplit
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, TimeSeriesSplit)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 3)
+        self.assertIsNone(cv.max_train_size)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
         # TimeSeriesSplit, regression with args
         D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
         D.info['task'] = REGRESSION
@@ -1417,7 +1902,7 @@ def test_get_splitter_cv_object(self, te_mock):
         self.assertIsInstance(cv, StratifiedKFold)
         self.assertEqual(cv.get_n_splits(
             groups=evaluator.resampling_strategy_args['groups']), 3)
-        self.assertEqual(cv.shuffle, False)
+        self.assertFalse(cv.shuffle)
         self.assertIsNone(cv.random_state)
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
@@ -1443,8 +1928,71 @@ def test_get_splitter_cv_object(self, te_mock):
         evaluator = TrainEvaluator()
         evaluator.resampling_strategy = GroupShuffleSplit
         evaluator.resampling_strategy_args = None
-        with self.assertRaises(ValueError):
-            evaluator.get_splitter(D)
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
+
+        # GroupShuffleSplit, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = GroupShuffleSplit
+        evaluator.resampling_strategy_args = {'folds': 2, 'test_size': 0.3,
+                                              'random_state': 5,
+                                              'groups': np.array([1, 1, 2, 1, 2, 2])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, GroupShuffleSplit)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 2)
+        self.assertEqual(cv.test_size, 0.3)
+        self.assertEqual(cv.random_state, 5)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # GroupShuffleSplit, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = GroupShuffleSplit
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
+
+        # GroupShuffleSplit, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = GroupShuffleSplit
+        evaluator.resampling_strategy_args = {'folds': 2, 'test_size': 0.3,
+                                              'random_state': 5,
+                                              'groups': np.array([1, 1, 2, 1, 2, 2])}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, GroupShuffleSplit)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 2)
+        self.assertEqual(cv.test_size, 0.3)
+        self.assertEqual(cv.random_state, 5)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # GroupShuffleSplit, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = GroupShuffleSplit
+        evaluator.resampling_strategy_args = None
+        self.assertRaisesRegex(
+            ValueError,
+            'Must provide parameter groups for chosen CrossValidator.',
+            evaluator.get_splitter,
+            D)
 
         # StratifiedShuffleSplit, classification with args
         D.data['Y_train'] = np.array([0, 0, 0, 1, 1, 1])
@@ -1505,6 +2053,70 @@ def test_get_splitter_cv_object(self, te_mock):
         next(cv.split(D.data['Y_train'], D.data['Y_train'],
                       groups=evaluator.resampling_strategy_args['groups']))
 
+        # ShuffleSplit, regression with args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = ShuffleSplit
+        evaluator.resampling_strategy_args = {'folds': 2, 'test_size': 0.3,
+                                              'random_state': 5}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, ShuffleSplit)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 2)
+        self.assertEqual(cv.test_size, 0.3)
+        self.assertEqual(cv.random_state, 5)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # ShuffleSplit, regression no args
+        D.data['Y_train'] = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5])
+        D.info['task'] = REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = ShuffleSplit
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, ShuffleSplit)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 10)
+        self.assertIsNone(cv.test_size)
+        self.assertIsNone(cv.random_state)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # ShuffleSplit, multi-output regression with args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = ShuffleSplit
+        evaluator.resampling_strategy_args = {'folds': 2, 'test_size': 0.3,
+                                              'random_state': 5}
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, ShuffleSplit)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 2)
+        self.assertEqual(cv.test_size, 0.3)
+        self.assertEqual(cv.random_state, 5)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
+        # ShuffleSplit, multi-output regression no args
+        D.data['Y_train'] = np.array([[0.0, 0.1], [0.2, 0.3], [0.4, 0.5],
+                                     [1.0, 1.1], [1.2, 1.3], [1.4, 1.5]])
+        D.info['task'] = MULTIOUTPUT_REGRESSION
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = ShuffleSplit
+        evaluator.resampling_strategy_args = None
+        cv = evaluator.get_splitter(D)
+        self.assertIsInstance(cv, ShuffleSplit)
+        self.assertEqual(cv.get_n_splits(
+            groups=evaluator.resampling_strategy_args['groups']), 10)
+        self.assertIsNone(cv.test_size)
+        self.assertIsNone(cv.random_state)
+        next(cv.split(D.data['Y_train'], D.data['Y_train'],
+                      groups=evaluator.resampling_strategy_args['groups']))
+
     @unittest.mock.patch.object(TrainEvaluator, "__init__")
     def test_holdout_split_size(self, te_mock):
         te_mock.return_value = None