Repository: EpistasisLab/tpot

Issue #1148: Issue with cuML: "Current best internal CV score: -inf"
Created: 2020-12-30T11:57:23Z
(EDIT: see comment below for quicker repro steps)

I'm trying to make use of a GPU-accelerated environment to train a TPOT Regression pipeline.
After two generations, when tpot computes the first CV score, the result goes straight to negative infinity:
Generation 1 - Current best internal CV score: -inf
After that, it outputs this rather unintelligible error:
RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly. If you enabled PyTorch estimators, please check the data requirements in the online documentation: https://epistasislab.github.io/tpot/using/

Environment
I spun a fresh g4dn.xlarge machine on AWS. Ubuntu 18.04, Python 3.7. This is what nvidia-smi is outputting:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   73C    P0    67W /  70W |   1522MiB / 15109MiB |     93%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      9058      C   ...vs/rapids-0.17/bin/python     1519MiB |
+-----------------------------------------------------------------------------+

I installed cuML with this command:
conda create -n rapids-0.17 -c rapidsai -c nvidia -c conda-forge     -c defaults rapids-blazing=0.17 python=3.7 cudatoolkit=11.0

That generated a conda environment. I then installed tpot and related dependencies, in this way:
conda activate rapids-0.17
conda install -c conda-forge tpot dask dask-ml scikit-mdr skrebate

Pytorch has not been installed in this conda environment.
Process to reproduce the issue
The following Hello World will reproduce the bug. I tried to change the number of samples and features, and the size of the CV, with no avail.
from tpot import TPOTRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

NSAMPLES = 15000
NFEATURES = 150
SEED = 12

# For cuML with TPOT, you must use CPU data (such as NumPy arrays)
X, y = make_regression(
    n_samples=NSAMPLES,
    n_features=NFEATURES,
    n_informative=NFEATURES,
    random_state=SEED,
    noise=200,
)

X = X.astype("float32")

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=12)

# TPOT setup
GENERATIONS = 5
POP_SIZE = 100
CV = 5

tpot = TPOTRegressor(
    generations=GENERATIONS,
    population_size=POP_SIZE,
    random_state=SEED,
    config_dict="TPOT cuML",
    n_jobs=1, # cuML requires n_jobs=1
    cv=CV,
    verbosity=2,
)

tpot.fit(X_train, y_train)

preds = tpot.predict(X_test)
print(r2_score(y_test, preds))

print(tpot.export())

Full error print out
Below, a full copy of the error:
Optimization Progress: 33%
200/600 [11:11<12:39, 1.90s/pipeline]

Generation 1 - Current best internal CV score: -inf

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/anaconda3/envs/rapids-0.17/lib/python3.7/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
    741                     per_generation_function=self._check_periodic_pipeline,
--> 742                     log_file=self.log_file_
    743                 )

~/anaconda3/envs/rapids-0.17/lib/python3.7/site-packages/tpot/gp_deap.py in eaMuPlusLambda(population, toolbox, mu, lambda_, cxpb, mutpb, ngen, pbar, stats, halloffame, verbose, per_generation_function, log_file)
    280         if per_generation_function is not None:
--> 281             per_generation_function(gen)
    282 

~/anaconda3/envs/rapids-0.17/lib/python3.7/site-packages/tpot/base.py in _check_periodic_pipeline(self, gen)
   1051         """
-> 1052         self._update_top_pipeline()
   1053         if self.periodic_checkpoint_folder is not None:

~/anaconda3/envs/rapids-0.17/lib/python3.7/site-packages/tpot/base.py in _update_top_pipeline(self)
    837                         break
--> 838                 raise RuntimeError('There was an error in the TPOT optimization '
    839                                    'process. This could be because the data was '

RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly. If you enabled PyTorch estimators, please check the data requirements in the online documentation: https://epistasislab.github.io/tpot/using/

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-e01323d79c09> in <module>
     37 )
     38 
---> 39 tpot.fit(X_train, y_train)
     40 
     41 preds = tpot.predict(X_test)

~/anaconda3/envs/rapids-0.17/lib/python3.7/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
    771                     # raise the exception if it's our last attempt
    772                     if attempt == (attempts - 1):
--> 773                         raise e
    774             return self
    775 

~/anaconda3/envs/rapids-0.17/lib/python3.7/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
    762                         self._pbar.close()
    763 
--> 764                     self._update_top_pipeline()
    765                     self._summary_of_best_pipeline(features, target)
    766                     # Delete the temporary cache before exiting

~/anaconda3/envs/rapids-0.17/lib/python3.7/site-packages/tpot/base.py in _update_top_pipeline(self)
    836                                                     error_score="raise")
    837                         break
--> 838                 raise RuntimeError('There was an error in the TPOT optimization '
    839                                    'process. This could be because the data was '
    840                                    'not formatted properly, or because data for '

RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly. If you enabled PyTorch estimators, please check the data requirements in the online documentation: https://epistasislab.github.io/tpot/using/

Additional info
By using the default config file, everything run smoothly. Using dask also doesn't give any relevant problem (but it's 2 to 3 times slower, as expected).

Duplicate issue #1163: cv in TPOTClassifier
Created: 2021-01-11T11:39:58Z
I faced an issue with tpot:
I am working with ~70000 x 36 features and predicting a multi-class target.
In particular, I am playing with Tanzania water wells dataset
Target is imbalanced :
functional                 0.54
non functional             0.38
functional needs repair    0.07
To process such data in a reasonable time, I am using the tpot - cuml configuration.
I created TPOTClassifier object with the following parameters:
tpt=TPOTClassifier(generations=100, population_size=100,
                          offspring_size=None, mutation_rate=0.9,
                          crossover_rate=0.1,
                          scoring='accuracy', cv=3,
                          subsample=1.0, n_jobs=1,
                          max_time_mins=None, max_eval_time_mins=5,
                          random_state=None, config_dict='TPOT cuML',
                          template=None,
                          warm_start=False,
                          memory=None,
                          use_dask=False,
                          periodic_checkpoint_folder=PREFIX+'tpot_checkpoints',
                          early_stop=None,
                          verbosity=3,
                          disable_update_check=False,
                          log_file=PREFIX+'tpot_checkpoints/log_cuml.txt',
                          
                          )

After a few hundreds of iterations, it falls with traceback:
/usr/local/lib/python3.6/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 2 members, which is less than n_splits=3.
  % (min_groups, self.n_splits)), UserWarning)

I believe that this kind of traceback indicates that there is no minor class present in one of the splits.
Can you please confirm that when I am setting cv= 3,5, 10  - tpot uses stratified folds?

Closing comment (2021-01-11T11:51:22Z): TPOTClassifier uses stratified kfold when cv=3,5 or 10.
I think this issue maybe related to #1148. If so, you need update TPOT or downgrade sklearn to 0.23.2.