Repository: automl/auto-sklearn

Issue:
Floating point issue when choosing holdout set train size
This line of [code](https://github.com/automl/auto-sklearn/blob/master/autosklearn/evaluation/train_evaluator.py#L549) will introduce floating point issue, e.g., when `train_size=0.7` will yield `0.30000000000000004` instead of `0.3`, thus causing the following error message
 
```
[ERROR] [2018-08-27 16:34:59,972:AutoML(1):cb3d3adc8ee73def12eda5903e98a85d] Error creating dummy predictions: {'traceback': 'Traceback (most recent call last):\n  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/evaluation/__init__.py", line 30, in fit_predict_try_except_decorator\n    return ta(queue=queue, **kwargs)\n  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/evaluation/train_evaluator.py", line 644, in eval_holdout\n    init_params=init_params\n  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/evaluation/train_evaluator.py", line 93, in __init__\n    self.cv = self.get_splitter(self.datamanager)\n  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/evaluation/train_evaluator.py", line 568, in get_splitter\n    raise e\n  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/evaluation/train_evaluator.py", line 562, in get_splitter\n    next(test_cv.split(y, y))\n  File "/root/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 1204, in split\n    for train, test in self._iter_indices(X, y, groups):\n  File "/root/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 1534, in _iter_indices\n    self.train_size)\n  File "/root/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 1710, in _validate_shuffle_split\n    \'train_size.\' % (n_train + n_test, n_samples))\nValueError: The sum of train_size and test_size = 17221, should be smaller than the number of samples 17220. Reduce test_size and/or train_size.\n', 'error': "ValueError('The sum of train_size and test_size = 17221, should be smaller than the number of samples 17220. Reduce test_size and/or train_size.',)", 'configuration_origin': 'DUMMY'} 
[ERROR] [2018-08-27 16:35:01,245:__main__] Unexpected Exception
Traceback (most recent call last):
  File "/var/www/html/MC_PY/mc_modeling/modeling.py", line 26, in main
    alg.fit()
  File "/var/www/html/MC_PY/mc_modules/mc_automl/base.py", line 456, in fit
    self.train()
  File "/var/www/html/MC_PY/mc_modules/mc_automl/classification.py", line 60, in train
    self.automl.fit(self.X_train.copy(), self.y_train.copy(), metric=self.metric, feat_type=self.feature_type)
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/estimators.py", line 466, in fit
    dataset_name=dataset_name,
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/estimators.py", line 248, in fit
    self._automl.fit(*args, **kwargs)
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/automl.py", line 954, in fit
    only_return_configuration_space=only_return_configuration_space,
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/automl.py", line 199, in fit
    only_return_configuration_space,
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/automl.py", line 461, in _fit
    _proc_smac.run_smbo()
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/smbo.py", line 501, in run_smbo
    smac.optimize()
  File "/root/anaconda3/lib/python3.6/site-packages/smac/facade/smac_facade.py", line 400, in optimize
    incumbent = self.solver.run()
  File "/root/anaconda3/lib/python3.6/site-packages/smac/optimizer/smbo.py", line 180, in run
    challengers = self.choose_next(X, Y)
  File "/root/anaconda3/lib/python3.6/site-packages/smac/optimizer/smbo.py", line 247, in choose_next
    incumbent_value = self.runhistory.get_cost(self.incumbent)
  File "/root/anaconda3/lib/python3.6/site-packages/smac/runhistory/runhistory.py", line 271, in get_cost
    config_id = self.config_ids[config]
KeyError: None
 ```

My current workaround that prevents this issue is 
```
accuracy = 2 #will only work when the train_size has 2 or less accuracy, it can be increased if you want to
n = 10**accuracy
tr_sz = float(str(train_size)[:2+accuracy])*n
test_size = (n-tr_sz)/n
```

Can this be fixed?


Discussion:
I've also had some problems with this. 
It raises an error in this part of [sklearn's code](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/model_selection/_split.py)
```
    if np.asarray(test_size).dtype.kind == 'f':
        n_test = ceil(test_size * n_samples)
    elif np.asarray(test_size).dtype.kind == 'i':
        n_test = float(test_size)

    if train_size is None:
        n_train = n_samples - n_test
    elif np.asarray(train_size).dtype.kind == 'f':
        n_train = floor(train_size * n_samples)
    else:
        n_train = float(train_size)

    if test_size is None:
        n_test = n_samples - n_train
```
For instance, in @kaiweiang  example:
With n_samples = 900; 
train_size = 0.7
test_size = 0.30000000000000004
n_test = 271, n_train = 630 (total samples 901)

I also had errors with other float numbers:
train_size = 0.72
test_size = 0.28
Here, it does not have a floating point issue when subtracting 1 - train_size, but when multiplying n_samples * test_size = 252.00000000000003 -> n_test = 253

I guess sklearn is using ceil and floor respectively to account the cases where the n_samples* test_size or n_samples* train_size are not integers, but when the ratio is a float and the result it's theoretically an integer, it is raising an error.


To account for both cases and still manage to use ceil and floor I only use up to 4 decimals and then provide the train_size and test_size as the number of samples to use and not the ratio (which is also handled by sklearn's code):
```
from math import ceil, floor

n_samples = len(y)
test_size = ceil(float("%.4f" % (n_samples * test_size)))
train_size = floor(float("%.4f" % (n_samples * train_size)))
```


Resolving patch:
diff --git a/autosklearn/evaluation/train_evaluator.py b/autosklearn/evaluation/train_evaluator.py
--- a/autosklearn/evaluation/train_evaluator.py
+++ b/autosklearn/evaluation/train_evaluator.py
@@ -540,30 +540,32 @@ def get_splitter(self, D):
 
                 return cv
 
-        y = D.data['Y_train'].ravel()
+        y = D.data['Y_train']
         shuffle = self.resampling_strategy_args.get('shuffle', True)
         train_size = 0.67
         if self.resampling_strategy_args:
             train_size = self.resampling_strategy_args.get('train_size',
                                                            train_size)
-        test_size = 1 - train_size
+        test_size = float("%.4f" % (1 - train_size))
+
         if D.info['task'] in CLASSIFICATION_TASKS and \
                         D.info['task'] != MULTILABEL_CLASSIFICATION:
 
+            y = y.ravel()
             if self.resampling_strategy in ['holdout',
                                             'holdout-iterative-fit']:
+
                 if shuffle:
                     try:
                         cv = StratifiedShuffleSplit(n_splits=1,
-                                                    train_size=train_size,
                                                     test_size=test_size,
                                                     random_state=1)
                         test_cv = copy.deepcopy(cv)
                         next(test_cv.split(y, y))
                     except ValueError as e:
                         if 'The least populated class in y has only' in e.args[0]:
-                            cv = ShuffleSplit(n_splits=1, train_size=train_size,
-                                              test_size=test_size, random_state=1)
+                            cv = ShuffleSplit(n_splits=1, test_size=test_size,
+                                              random_state=1)
                         else:
                             raise e
                 else:
@@ -588,8 +590,8 @@ def get_splitter(self, D):
                                             'holdout-iterative-fit']:
                 # TODO shuffle not taken into account for this
                 if shuffle:
-                    cv = ShuffleSplit(n_splits=1, train_size=train_size,
-                                      test_size=test_size, random_state=1)
+                    cv = ShuffleSplit(n_splits=1, test_size=test_size,
+                                      random_state=1)
                 else:
                     tmp_train_size = int(np.floor(train_size * y.shape[0]))
                     test_fold = np.zeros(y.shape[0])


Test patch:
diff --git a/test/test_evaluation/test_train_evaluator.py b/test/test_evaluation/test_train_evaluator.py
--- a/test/test_evaluation/test_train_evaluator.py
+++ b/test/test_evaluation/test_train_evaluator.py
@@ -23,7 +23,10 @@
     eval_holdout, eval_iterative_holdout, eval_cv, eval_partial_cv
 from autosklearn.util import backend
 from autosklearn.util.pipeline import get_configuration_space
-from autosklearn.constants import *
+from autosklearn.constants import BINARY_CLASSIFICATION, \
+    MULTILABEL_CLASSIFICATION,\
+    MULTICLASS_CLASSIFICATION,\
+    REGRESSION
 from autosklearn.metrics import accuracy, r2, f1_macro
 
 this_directory = os.path.dirname(__file__)
@@ -1226,6 +1229,112 @@ def test_get_splitter_cv_object(self, te_mock):
         next(cv.split(D.data['Y_train'], D.data['Y_train']
                       , groups=evaluator.resampling_strategy_args['groups']))
 
+    @unittest.mock.patch.object(TrainEvaluator, "__init__")
+    def test_holdout_split_size(self, te_mock):
+        te_mock.return_value = None
+        D = unittest.mock.Mock(spec=AbstractDataManager)
+        D.feat_type = []
+
+        evaluator = TrainEvaluator()
+        evaluator.resampling_strategy = 'holdout'
+
+        # Exact Ratio
+        D.data = dict(Y_train=np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1]))
+        D.info = dict(task=BINARY_CLASSIFICATION)
+        evaluator.resampling_strategy_args = {'shuffle': True,
+                                              'train_size': 0.7}
+        cv = evaluator.get_splitter(D)
+
+        self.assertEqual(cv.get_n_splits(), 1)
+        train_samples, test_samples = next(cv.split(D.data['Y_train'],
+                                                    D.data['Y_train']))
+        self.assertEqual(len(train_samples), 7)
+        self.assertEqual(len(test_samples), 3)
+
+        # No Shuffle
+        evaluator.resampling_strategy_args = {'shuffle': False,
+                                              'train_size': 0.7}
+        cv = evaluator.get_splitter(D)
+
+        self.assertEqual(cv.get_n_splits(), 1)
+        train_samples, test_samples = next(cv.split(D.data['Y_train'],
+                                                    D.data['Y_train']))
+        self.assertEqual(len(train_samples), 7)
+        self.assertEqual(len(test_samples), 3)
+
+        # Rounded Ratio
+        D.data = dict(Y_train=np.array([0, 0, 0, 0, 0, 1, 1, 1, 1]))
+
+        evaluator.resampling_strategy_args = {'shuffle': True,
+                                              'train_size': 0.7}
+        cv = evaluator.get_splitter(D)
+
+        self.assertEqual(cv.get_n_splits(), 1)
+        train_samples, test_samples = next(cv.split(D.data['Y_train'],
+                                                    D.data['Y_train']))
+        self.assertEqual(len(train_samples), 6)
+        self.assertEqual(len(test_samples), 3)
+
+        # Rounded Ratio No Shuffle
+        evaluator.resampling_strategy_args = {'shuffle': False,
+                                              'train_size': 0.7}
+        cv = evaluator.get_splitter(D)
+
+        self.assertEqual(cv.get_n_splits(), 1)
+        train_samples, test_samples = next(cv.split(D.data['Y_train'],
+                                                    D.data['Y_train']))
+        self.assertEqual(len(train_samples), 6)
+        self.assertEqual(len(test_samples), 3)
+
+        # More data
+        evaluator.resampling_strategy_args = {'shuffle': True,
+                                              'train_size': 0.7}
+
+        D.data = dict(Y_train=np.zeros((900, 1)))
+        cv = evaluator.get_splitter(D)
+        self.assertEqual(cv.get_n_splits(), 1)
+        train_samples, test_samples = next(cv.split(D.data['Y_train'],
+                                                    D.data['Y_train']))
+        self.assertEqual(len(train_samples), 630)
+        self.assertEqual(len(test_samples), 270)
+
+        evaluator.resampling_strategy_args = {'train_size': 0.752}
+        D.data = dict(Y_train=np.zeros((900, 1)))
+        cv = evaluator.get_splitter(D)
+        self.assertEqual(cv.get_n_splits(), 1)
+        train_samples, test_samples = next(cv.split(D.data['Y_train'],
+                                                    D.data['Y_train']))
+        self.assertEqual(len(train_samples), 676)
+        self.assertEqual(len(test_samples), 224)
+
+        # Multilabel Exact Ratio
+        D.data = dict(Y_train=np.array([[0, 0], [0, 1], [1, 1], [1, 0], [1, 1],
+                                        [1, 1], [1, 1], [1, 0], [1, 1], [1, 1]]
+                                       ))
+        D.info = dict(task=MULTILABEL_CLASSIFICATION)
+        evaluator.resampling_strategy_args = {'shuffle': True,
+                                              'train_size': 0.7}
+        cv = evaluator.get_splitter(D)
+
+        self.assertEqual(cv.get_n_splits(), 1)
+        train_samples, test_samples = next(cv.split(D.data['Y_train'],
+                                                    D.data['Y_train']))
+        self.assertEqual(len(train_samples), 7)
+        self.assertEqual(len(test_samples), 3)
+
+        # Multilabel No Shuffle
+        D.data = dict(Y_train=np.array([[0, 0], [0, 1], [1, 1], [1, 0], [1, 1],
+                                        [1, 1], [1, 1], [1, 0], [1, 1]]))
+        evaluator.resampling_strategy_args = {'shuffle': False,
+                                              'train_size': 0.7}
+        cv = evaluator.get_splitter(D)
+
+        self.assertEqual(cv.get_n_splits(), 1)
+        train_samples, test_samples = next(cv.split(D.data['Y_train'],
+                                                    D.data['Y_train']))
+        self.assertEqual(len(train_samples), 6)
+        self.assertEqual(len(test_samples), 3)
+
 
 class FunctionsTest(unittest.TestCase):
     def setUp(self):