Repository: automl/auto-sklearn

Issue:
Memory Exception- unable to escape from memory exception 
I try to using auto-Sklearn on my own dataset,
The shape of my dataset is :` Train X_train/y_train :  (1400014, 17) (1400014,)` and running 5-fold cv (doing by myself)
Here is how I calling auto-Sklearn : 
`automl = autosklearn.classification.AutoSklearnClassifier(include_estimators = ['k_nearest_neighbors'],time_left_for_this_task =3600, ml_memory_limit = 16384 )`

I using **only one estimator** and **set memory_limit to 16384 MB**,
But always failed due to Memory problem shown below:
```
[WARNING] [2018-10-08 20:20:16,950:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,032:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2018-10-08 20:20:17,111:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,194:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 12
[WARNING] [2018-10-08 20:20:17,268:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,348:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 6
[WARNING] [2018-10-08 20:20:17,422:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,491:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 3
[WARNING] [2018-10-08 20:20:17,569:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,651:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 1
[WARNING] [2018-10-08 20:20:17,726:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[CRITICAL] [2018-10-08 20:20:17,806:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- Unable to escape from memory exception
```

Not sure is the data are too large or wrong using the auto-Sklearn API ?
Memory Exception- unable to escape from memory exception 
I try to using auto-Sklearn on my own dataset,
The shape of my dataset is :` Train X_train/y_train :  (1400014, 17) (1400014,)` and running 5-fold cv (doing by myself)
Here is how I calling auto-Sklearn : 
`automl = autosklearn.classification.AutoSklearnClassifier(include_estimators = ['k_nearest_neighbors'],time_left_for_this_task =3600, ml_memory_limit = 16384 )`

I using **only one estimator** and **set memory_limit to 16384 MB**,
But always failed due to Memory problem shown below:
```
[WARNING] [2018-10-08 20:20:16,950:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,032:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2018-10-08 20:20:17,111:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,194:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 12
[WARNING] [2018-10-08 20:20:17,268:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,348:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 6
[WARNING] [2018-10-08 20:20:17,422:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,491:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 3
[WARNING] [2018-10-08 20:20:17,569:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,651:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 1
[WARNING] [2018-10-08 20:20:17,726:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[CRITICAL] [2018-10-08 20:20:17,806:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- Unable to escape from memory exception
```

Not sure is the data are too large or wrong using the auto-Sklearn API ?


Discussion:
I had the same error and by reducing the number of different classes the error disappear. (My dataset size `(982840, 5)`)
Will be fixed in the upcoming release via ab45a25a.
@mfeurer Thank you for your work! I was wondering when should we expect next release ? I saw the `4.0.1` came out today. 
@mfeurer I still encounter this error when running the newest version on mnist dataset which is on LibSVM website. Is it normal?
I didn't have the time to try this. I'm re-opening this issue now to remember us to look into this.
k_nearest_neighbors eats memory  ：P
may try smaller dataset or less time_left_for_this_task 
@huangwei2013 Thanks for your suggestion. It seems it doesn't help.
But I just use half of the mnist dataset (with 30,000 instances and  780 features) and 1 classifier (SVC).
Is there anything I get wrong?
code from the error as below：
```
            if safe_ensemble_script.exit_status is pynisher.MemorylimitException:
                # if ensemble script died because of memory error,
                # reduce nbest to reduce memory consumption and try it again
                if self.ensemble_nbest == 1:
                    self.logger.critical("Memory Exception -- Unable to escape from memory exception")
```

while you didn't set ensemble_nbest & ensemble_size (both defaulted ：1)， so only tip I can get is reduce time_left_for_this_task &  ml_memory_limit to whatever you can run
Hi, I just tryed with the last version, just downloaded now as in "install explanations".

I do have the same issue:
```
[WARNING] [2019-07-24 23:53:01,502:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2019-07-24 23:53:02,188:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 12
/root/anaconda3/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
/root/anaconda3/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
[WARNING] [2019-07-25 00:02:10,870:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 6
```
I See upside that there was a fix to correct that.
And I see that I actually do have enaugh memory. :octopus: 
```
(base) root@123456789:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          23543        2752       14498           0        6292       20407
```
I have some questions, I hope you will forgive my ignorance :+1: 
- Do the last install include this fix ?
- Should I put some parameters to profit of that fix ? here is my current config :
``` # current parameters are OK ?
    clf = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=int(3600*15),
        per_run_time_limit=int(3600*1.5),
        include_estimators=['MLPClassifier'],ml_memory_limit=22000
    )
```
- Note: I am training with some MLPClassifier
- You have others advices ?  I imagine that the more we have ensemble, the best...

Unfortunately, I've got the same issue. I am using the Dataset of the BPI Challenge 2019 (https://data.4tu.nl/repository/uuid:d06aff4b-79f0-45e6-8ec8-e19730c248f1), sort it by 
`df = df.sort_values(by=["case concept:name", "event time:timestamp"])`
restrict it by 
`df = df[["case concept:name", "event concept:name"]]`
label encode it with 
`df.iloc[:, 1] = labelencoder.fit_transform(df.iloc[:, 1])`
transform/unstack it with 
`df_activities_trans = df.set_index(
    ['case concept:name', df.groupby('case concept:name').cumcount() + 1]).unstack()`
and fill the NaNs with 
`final_df = final_df.fillna(999)`
such that I receive a dataframe with 251734 rows and 990 columns. However, I iteratively compute classification models by selecting e.g. 
`X = df_activities_trans_ml.iloc[:,0:2]`  (first two columns of the dataset as features) and 
`Y = df_activities_trans_ml.iloc[:,2]` (third column as output) such that I use a dataset with 251734 rows and 3 columns. 
I defined the classifier as follows: 
`classifier = autosklearn.classification.AutoSklearnClassifier(time_left_for_this_task=3600,
                                                                  per_run_time_limit=360, ml_memory_limit=10240, n_jobs=4)`
and used the following train/test split
`X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, Y, random_state=1)`
However, I always get the following error when calling classifier.fit(X_train, y_train): 

```
[WARNING] [2020-02-04 17:01:49,717:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:49,754:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2020-02-04 17:01:49,947:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:49,967:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 12
[WARNING] [2020-02-04 17:01:50,190:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,215:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 6
[WARNING] [2020-02-04 17:01:50,444:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,465:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 3
[WARNING] [2020-02-04 17:01:50,628:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,649:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 1
[CRITICAL] [2020-02-04 17:03:53,982:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- Unable to escape from memory exception
```
My system is a vm running Ubuntu 16.04 with 4 cores and 15032 MB RAM. 
```
              total        used        free      shared  buff/cache   available
Mem:          15032         176       12367           8        2487       14484
Swap:             0           0           0
```

EDIT: Sorry, it seems like I confused the ml_memory_limit with ensemble_memory_limit. Increasing the ensemble_memory_limit seems to work for me. 

I had the same error and by reducing the number of different classes the error disappear. (My dataset size `(982840, 5)`)
Will be fixed in the upcoming release via ab45a25a.
@mfeurer Thank you for your work! I was wondering when should we expect next release ? I saw the `4.0.1` came out today. 
@mfeurer I still encounter this error when running the newest version on mnist dataset which is on LibSVM website. Is it normal?
I didn't have the time to try this. I'm re-opening this issue now to remember us to look into this.
k_nearest_neighbors eats memory  ：P
may try smaller dataset or less time_left_for_this_task 
@huangwei2013 Thanks for your suggestion. It seems it doesn't help.
But I just use half of the mnist dataset (with 30,000 instances and  780 features) and 1 classifier (SVC).
Is there anything I get wrong?
code from the error as below：
```
            if safe_ensemble_script.exit_status is pynisher.MemorylimitException:
                # if ensemble script died because of memory error,
                # reduce nbest to reduce memory consumption and try it again
                if self.ensemble_nbest == 1:
                    self.logger.critical("Memory Exception -- Unable to escape from memory exception")
```

while you didn't set ensemble_nbest & ensemble_size (both defaulted ：1)， so only tip I can get is reduce time_left_for_this_task &  ml_memory_limit to whatever you can run
Hi, I just tryed with the last version, just downloaded now as in "install explanations".

I do have the same issue:
```
[WARNING] [2019-07-24 23:53:01,502:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2019-07-24 23:53:02,188:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 12
/root/anaconda3/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
/root/anaconda3/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
[WARNING] [2019-07-25 00:02:10,870:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 6
```
I See upside that there was a fix to correct that.
And I see that I actually do have enaugh memory. :octopus: 
```
(base) root@123456789:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          23543        2752       14498           0        6292       20407
```
I have some questions, I hope you will forgive my ignorance :+1: 
- Do the last install include this fix ?
- Should I put some parameters to profit of that fix ? here is my current config :
``` # current parameters are OK ?
    clf = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=int(3600*15),
        per_run_time_limit=int(3600*1.5),
        include_estimators=['MLPClassifier'],ml_memory_limit=22000
    )
```
- Note: I am training with some MLPClassifier
- You have others advices ?  I imagine that the more we have ensemble, the best...

Unfortunately, I've got the same issue. I am using the Dataset of the BPI Challenge 2019 (https://data.4tu.nl/repository/uuid:d06aff4b-79f0-45e6-8ec8-e19730c248f1), sort it by 
`df = df.sort_values(by=["case concept:name", "event time:timestamp"])`
restrict it by 
`df = df[["case concept:name", "event concept:name"]]`
label encode it with 
`df.iloc[:, 1] = labelencoder.fit_transform(df.iloc[:, 1])`
transform/unstack it with 
`df_activities_trans = df.set_index(
    ['case concept:name', df.groupby('case concept:name').cumcount() + 1]).unstack()`
and fill the NaNs with 
`final_df = final_df.fillna(999)`
such that I receive a dataframe with 251734 rows and 990 columns. However, I iteratively compute classification models by selecting e.g. 
`X = df_activities_trans_ml.iloc[:,0:2]`  (first two columns of the dataset as features) and 
`Y = df_activities_trans_ml.iloc[:,2]` (third column as output) such that I use a dataset with 251734 rows and 3 columns. 
I defined the classifier as follows: 
`classifier = autosklearn.classification.AutoSklearnClassifier(time_left_for_this_task=3600,
                                                                  per_run_time_limit=360, ml_memory_limit=10240, n_jobs=4)`
and used the following train/test split
`X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, Y, random_state=1)`
However, I always get the following error when calling classifier.fit(X_train, y_train): 

```
[WARNING] [2020-02-04 17:01:49,717:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:49,754:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2020-02-04 17:01:49,947:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:49,967:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 12
[WARNING] [2020-02-04 17:01:50,190:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,215:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 6
[WARNING] [2020-02-04 17:01:50,444:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,465:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 3
[WARNING] [2020-02-04 17:01:50,628:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,649:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 1
[CRITICAL] [2020-02-04 17:03:53,982:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- Unable to escape from memory exception
```
My system is a vm running Ubuntu 16.04 with 4 cores and 15032 MB RAM. 
```
              total        used        free      shared  buff/cache   available
Mem:          15032         176       12367           8        2487       14484
Swap:             0           0           0
```

EDIT: Sorry, it seems like I confused the ml_memory_limit with ensemble_memory_limit. Increasing the ensemble_memory_limit seems to work for me. 


Resolving patch:
diff --git a/autosklearn/automl.py b/autosklearn/automl.py
--- a/autosklearn/automl.py
+++ b/autosklearn/automl.py
@@ -88,7 +88,7 @@ def __init__(self,
                  ensemble_size=1,
                  ensemble_nbest=1,
                  max_models_on_disc=1,
-                 ensemble_memory_limit=1000,
+                 ensemble_memory_limit: Optional[int] = 1024,
                  seed=1,
                  ml_memory_limit=3072,
                  metadata_directory=None,
diff --git a/autosklearn/ensemble_builder.py b/autosklearn/ensemble_builder.py
--- a/autosklearn/ensemble_builder.py
+++ b/autosklearn/ensemble_builder.py
@@ -45,7 +45,7 @@ def __init__(
             max_iterations: int = None,
             precision: int = 32,
             sleep_duration: int = 2,
-            memory_limit: int = 1000,
+            memory_limit: Optional[int] = 1024,
             read_at_most: int = 5,
             random_state: Optional[Union[int, np.random.RandomState]] = None,
             queue: multiprocessing.Queue = None
@@ -100,8 +100,8 @@ def __init__(
                 precision of floats to read the predictions
             sleep_duration: int
                 duration of sleeping time between two iterations of this script (in sec)
-            memory_limit: int
-                memory limit in mb
+            memory_limit: Optional[int]
+                memory limit in mb. If ``None``, no memory limit is enforced.
             read_at_most: int
                 read at most n new prediction files in each iteration
         """
@@ -243,8 +243,12 @@ def run(self):
                 # reduce nbest to reduce memory consumption and try it again
                 if isinstance(self.ensemble_nbest, numbers.Integral) and \
                         self.ensemble_nbest == 1:
-                    self.logger.critical("Memory Exception --"
-                                         " Unable to escape from memory exception")
+                    self.logger.critical(
+                        "Memory Exception -- Unable to further reduce the number of ensemble "
+                        "members -- please restart Auto-sklearn with a higher value for the "
+                        "argument 'ensemble_memory_limit' (current limit is {} MB)."
+                        "".format(self.memory_limit)
+                    )
                 else:
                     if isinstance(self.ensemble_nbest, numbers.Integral):
                         self.ensemble_nbest = int(self.ensemble_nbest / 2)
diff --git a/autosklearn/estimators.py b/autosklearn/estimators.py
--- a/autosklearn/estimators.py
+++ b/autosklearn/estimators.py
@@ -26,7 +26,7 @@ def __init__(
         ensemble_size: int = 50,
         ensemble_nbest=50,
         max_models_on_disc=50,
-        ensemble_memory_limit=1024,
+        ensemble_memory_limit: Optional[int] = 1024,
         seed=1,
         ml_memory_limit=3072,
         include_estimators=None,
@@ -90,6 +90,7 @@ def __init__(
             Memory limit in MB for the ensemble building process.
             `auto-sklearn` will reduce the number of considered models
             (``ensemble_nbest``) if the memory limit is reached.
+            If ``None``, no memory limit is enforced.
 
         seed : int, optional (default=1)
             Used to seed SMAC. Will determine the output file names.


Test patch: