Repository: autogluon/autogluon

Issue:
fit() error
Hi, 
The dataset used was like this:

```
import io
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import scipy.stats
import scipy.special
data = """Grain size (mm)	Spiders
0.245	absent
0.247	absent
0.285	present
0.299	present
0.327	present
0.347	present
0.356	absent
0.36	present
0.363	absent
0.364	present
0.398	absent
0.4	present
0.409	absent
0.421	present
0.432	absent
0.473	present
0.509	present
0.529	present
0.561	absent
0.569	absent
0.594	present
0.638	present
0.656	present
0.816	present
0.853	present
0.938	present
1.036	present
1.045	present
"""
df = pd.read_table(io.StringIO(data))
df.Spiders = df.Spiders == 'present'
df.head()
```
![image](https://user-images.githubusercontent.com/34644984/85224256-8a7f8e80-b3fb-11ea-9d6c-692eca6a5bc3.png)
I am trying to perform the 'likelihood ratio test' to evaluate the regression model. it includes a null model and fitted model. in the null model, we do the below in fit()
**CODE:**
```
k1 = np.zeros_like(df['Spiders']).reshape(-1,1)
k2 = pd.DataFrame(k1, columns=['n'])
k3 = df['Spiders']
k4 = pd.DataFrame(k3)
k5 = k2.join(k4)
label='Spiders'
pred2 = task.fit(train_data=k5,label=label, hyperparameters={'LR':{}}, eval_metric='log_loss')
```
---------------------------------------------------------------------------------------------------------
The above code basically takes a column of 'false' values(0's) as dependent variable and the independent variable label as 0's and 1's. But it gives me the following error, can u pls say why?
**_ERROR:_**
```
No output_directory specified. Models will be saved in: AutogluonModels/ag-20200621_121945/
Beginning AutoGluon training ...
AutoGluon will save models to AutogluonModels/ag-20200621_121945/
Train Data Rows:    28
Train Data Columns: 2
Preprocessing data ...
Here are the 2 unique label values in your data:  [False, True]
AutoGluon infers your prediction problem is: binary  (because only two unique label-values observed).
If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])

Warning: Updated label_count_threshold from 10 to 9 to avoid cutting too many classes.
Selected class <--> label mapping:  class 1 = True, class 0 = False
Train Data Class Count: 2
Feature Generator processed 28 data points with 0 features
Original Features:
Generated Features:
	int features: 0
All Features:
	int features: 0
	Data preprocessing and feature engineering runtime = 0.03s ...
AutoGluon will gauge predictive performance using evaluation metric: log_loss
This metric expects predicted probabilities rather than predicted class labels, so you'll need to use predict_proba() instead of predict()
To change this, specify the eval_metric argument of fit()
AutoGluon will early stop models using evaluation metric: log_loss
Fitting model: LinearModel ...
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 261, in train_and_save
    model = self.train_single(X_train, y_train, X_test, y_test, model, kfolds=kfolds, k_fold_start=k_fold_start, k_fold_end=k_fold_end, n_repeats=n_repeats, n_repeat_start=n_repeat_start, level=level, time_limit=time_limit)
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 246, in train_single
    model.fit(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, time_limit=time_limit, **model_fit_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/models/lr/lr_model.py", line 123, in fit
    X_train = self.preprocess(X_train, is_train=True, vect_max_features=hyperparams['vectorizer_dict_size'], model_specific_preprocessing=True)
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/models/lr/lr_model.py", line 72, in preprocess
    self.preprocess_train(X, feature_types, vect_max_features)
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/models/lr/lr_model.py", line 105, in preprocess_train
    self.pipeline = FeatureUnion(transformer_list=transformer_list)
  File "/usr/local/lib/python3.6/dist-packages/sklearn/pipeline.py", line 814, in __init__
    self._validate_transformers()
  File "/usr/local/lib/python3.6/dist-packages/sklearn/pipeline.py", line 845, in _validate_transformers
    names, transformers = zip(*self.transformer_list)
Warning: Exception caused LinearModel to fail during training... Skipping this model.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 261, in train_and_save
    model = self.train_single(X_train, y_train, X_test, y_test, model, kfolds=kfolds, k_fold_start=k_fold_start, k_fold_end=k_fold_end, n_repeats=n_repeats, n_repeat_start=n_repeat_start, level=level, time_limit=time_limit)
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 246, in train_single
    model.fit(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, time_limit=time_limit, **model_fit_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/models/lr/lr_model.py", line 123, in fit
    X_train = self.preprocess(X_train, is_train=True, vect_max_features=hyperparams['vectorizer_dict_size'], model_specific_preprocessing=True)
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/models/lr/lr_model.py", line 72, in preprocess
    self.preprocess_train(X, feature_types, vect_max_features)
  File "/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/models/lr/lr_model.py", line 105, in preprocess_train
    self.pipeline = FeatureUnion(transformer_list=transformer_list)
  File "/usr/local/lib/python3.6/dist-packages/sklearn/pipeline.py", line 814, in __init__
    self._validate_transformers()
  File "/usr/local/lib/python3.6/dist-packages/sklearn/pipeline.py", line 845, in _validate_transformers
    names, transformers = zip(*self.transformer_list)
ValueError: not enough values to unpack (expected 2, got 0)
not enough values to unpack (expected 2, got 0)
No base models to train on, skipping weighted ensemble...

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-37-002c89d9f4c6> in <module>
      5 k5 = k2.join(k4)
      6 label='Spiders'
----> 7 pred2 = task.fit(train_data=k5,label=label, hyperparameters={'LR':{}}, eval_metric='log_loss')

/usr/local/lib/python3.6/dist-packages/autogluon/task/tabular_prediction/presets_configs.py in _call(**kwargs)
      7         @functools.wraps(f)
      8         def _call(**kwargs):
----> 9             return f(**g(**kwargs))
     10         return _call
     11     return _unpack_inner

/usr/local/lib/python3.6/dist-packages/autogluon/task/tabular_prediction/tabular_prediction.py in fit(train_data, label, tuning_data, time_limits, output_directory, presets, problem_type, eval_metric, stopping_metric, auto_stack, hyperparameter_tune, feature_prune, holdout_frac, num_bagging_folds, num_bagging_sets, stack_ensemble_levels, hyperparameters, num_trials, scheduler_options, search_strategy, search_options, nthreads_per_trial, ngpus_per_trial, dist_ip_addrs, visualizer, verbosity, **kwargs)
    586                     hyperparameter_tune=hyperparameter_tune, feature_prune=feature_prune,
    587                     holdout_frac=holdout_frac, num_bagging_folds=num_bagging_folds, num_bagging_sets=num_bagging_sets, stack_ensemble_levels=stack_ensemble_levels,
--> 588                     hyperparameters=hyperparameters, time_limit=time_limits_orig, save_data=cache_data, save_bagged_folds=save_bagged_folds, verbosity=verbosity)
    589 
    590         predictor = TabularPredictor(learner=learner)

/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/learner/default_learner.py in fit(self, X, X_test, scheduler_options, hyperparameter_tune, feature_prune, holdout_frac, num_bagging_folds, num_bagging_sets, stack_ensemble_levels, hyperparameters, time_limit, save_data, save_bagged_folds, verbosity)
    101         self.save()
    102         trainer.train(X, y, X_test, y_test, hyperparameter_tune=hyperparameter_tune, feature_prune=feature_prune, holdout_frac=holdout_frac,
--> 103                       hyperparameters=hyperparameters)
    104         self.save_trainer(trainer=trainer)
    105         time_end = time.time()

/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/trainer/auto_trainer.py in train(self, X_train, y_train, X_test, y_test, hyperparameter_tune, feature_prune, holdout_frac, hyperparameters)
     31             if (y_test is None) or (X_test is None):
     32                 X_train, X_test, y_train, y_test = generate_train_test_split(X_train, y_train, problem_type=self.problem_type, test_size=holdout_frac, random_state=self.random_seed)
---> 33         self.train_multi_and_ensemble(X_train, y_train, X_test, y_test, models, hyperparameter_tune=hyperparameter_tune, feature_prune=feature_prune)

/usr/local/lib/python3.6/dist-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py in train_multi_and_ensemble(self, X_train, y_train, X_test, y_test, models, hyperparameter_tune, feature_prune)
    503         self.train_multi_levels(X_train, y_train, X_test, y_test, models=models, hyperparameter_tune=hyperparameter_tune, feature_prune=feature_prune, level_start=0, level_end=self.stack_ensemble_levels)
    504         if len(self.get_model_names_all()) == 0:
--> 505             raise ValueError('AutoGluon did not successfully train any models')
    506 
    507     def train_multi_levels(self, X_train, y_train, X_test, y_test, models: List[AbstractModel], hyperparameter_tune=True, feature_prune=False, level_start=0, level_end=0):

ValueError: AutoGluon did not successfully train any models
```
Note: This error doesn't occur if I use fit() using sklearn without using autogluon. But I want to implement using autogluon, please help!
**Please help me to solve, thanks in advance!**



Discussion:
A tip for future issues: Please use the code block functionality of GitHub when pasting code into your issue, this simplifies viewing and makes things look a lot cleaner. I have edited your original comment to use code blocks.

I will plan to take a look at this sometime Monday.
Oh sure, noted. Thanks. Waiting for your reply on the issue!
Your data transformation was not correct, and led to the 'n' column consisting of only 0 values. Since this is the only feature and it contains no information (thus getting filtered out during preprocessing), AutoGluon crashes.

This modified code works:

```
import io
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import scipy.stats
import scipy.special
from autogluon import TabularPrediction as task

data = """Grain size (mm)	Spiders
0.245	absent
0.247	absent
0.285	present
0.299	present
0.327	present
0.347	present
0.356	absent
0.36	present
0.363	absent
0.364	present
0.398	absent
0.4	present
0.409	absent
0.421	present
0.432	absent
0.473	present
0.509	present
0.529	present
0.561	absent
0.569	absent
0.594	present
0.638	present
0.656	present
0.816	present
0.853	present
0.938	present
1.036	present
1.045	present
"""
df = pd.read_table(io.StringIO(data))
# df.Spiders = df.Spiders == 'present'
# print(df.head())

# k1 = np.zeros_like(df['Spiders']).reshape(-1, 1)
# k2 = pd.DataFrame(k1, columns=['n'])
# k3 = df['Spiders']
# k4 = pd.DataFrame(k3)
# k5 = k2.join(k4)
label = 'Spiders'
pred2 = task.fit(train_data=df, label=label, hyperparameters={'LR': {}}, eval_metric='log_loss')
```

Code Output:

```
No output_directory specified. Models will be saved in: AutogluonModels/ag-20200623_003617/
Beginning AutoGluon training ...
AutoGluon will save models to AutogluonModels/ag-20200623_003617/
Train Data Rows:    28
Train Data Columns: 2
Preprocessing data ...
Here are the 2 unique label values in your data:  ['absent', 'present']
AutoGluon infers your prediction problem is: binary  (because only two unique label-values observed).
If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])
Warning: Updated label_count_threshold from 10 to 9 to avoid cutting too many classes.
Selected class <--> label mapping:  class 1 = present, class 0 = absent
Train Data Class Count: 2
Feature Generator processed 28 data points with 1 features
Original Features:
	float features: 1
	object features: 0
Generated Features:
	int features: 0
All Features:
	float features: 1
	object features: 0
	int features: 0
	Data preprocessing and feature engineering runtime = 0.04s ...
AutoGluon will gauge predictive performance using evaluation metric: log_loss
This metric expects predicted probabilities rather than predicted class labels, so you'll need to use predict_proba() instead of predict()
To change this, specify the eval_metric argument of fit()
AutoGluon will early stop models using evaluation metric: log_loss
Fitting model: LinearModel ...
	-0.5258	 = Validation log_loss score
	1.86s	 = Training runtime
	0.01s	 = Validation runtime
Fitting model: weighted_ensemble_k0_l1 ...
	-0.5258	 = Validation log_loss score
	0.01s	 = Training runtime
	0.0s	 = Validation runtime
AutoGluon training complete, total runtime = 1.95s ...
```

For future issues, please confirm that your data being sent into AutoGluon is correct before creating the issue, in order to save time.

Best,
Nick
No, to perform likelihood ratio test we use fit() with only zeros as train_data and dependent variable. This error doesn't occur when we do the same with scikit learn fit(). 
I understand what you mean now, sorry for misunderstanding. While this is technically a valid use-case, AutoML may not be the best solution to go about creating a null model. AutoGluon crashes because it filters out features which have all the same value across all rows, and since no features exist, no models can be trained. This could be changed in future to instead return a dummy model which just predicts a constant value.
Thank you!

Resolving patch:
diff --git a/autogluon/utils/tabular/features/abstract_feature_generator.py b/autogluon/utils/tabular/features/abstract_feature_generator.py
--- a/autogluon/utils/tabular/features/abstract_feature_generator.py
+++ b/autogluon/utils/tabular/features/abstract_feature_generator.py
@@ -51,6 +51,7 @@ def __init__(self):
         self.pre_memory_usage_per_row = None
         self.post_memory_usage = None
         self.post_memory_usage_per_row = None
+        self.is_dummy = False
 
     @property
     def feature_types_metadata(self):
@@ -155,6 +156,7 @@ def fit_transform(self, X: DataFrame, y=None, banned_features=None, fix_categori
             X_features = self.minimize_ngram_memory_usage(X_features=X_features)
         if self.features_binned:
             X_features = self.minimize_binned_memory_usage(X_features=X_features)
+        self.feature_type_family_generated['int'] += self.features_binned
 
         self.post_memory_usage = self.get_approximate_df_mem_usage(X_features, sample_ratio=0.2).sum()
         self.post_memory_usage_per_row = self.post_memory_usage / X_len
@@ -166,8 +168,13 @@ def fit_transform(self, X: DataFrame, y=None, banned_features=None, fix_categori
             logger.warning(f'Warning: Data size post feature transformation consumes {round(post_memory_usage_percent*100, 1)}% of available memory. Consider increasing memory or subsampling the data to avoid instability.')
 
         X_features.index = X_index
+        if len(list(X_features.columns)) == 0:
+            self.is_dummy = True
+            logger.warning(f'WARNING: No useful features were detected in the data! AutoGluon will train using 0 features, and will always predict the same value. Ensure that you are passing the correct data to AutoGluon!')
+            X_features['__dummy__'] = 0
+            self.feature_type_family_generated['int'] = ['__dummy__']
+
         self.features = list(X_features.columns)
-        self.feature_type_family_generated['int'] += self.features_binned
         self.fit = True
 
         logger.log(20, 'Feature Generator processed %s data points with %s features' % (X_len, len(self.features)))
@@ -205,6 +212,10 @@ def transform(self, X: DataFrame):
         X_features = self.generate_features(X)
         for column in self.features_binned:
             X_features[column] = self.bin_column(series=X_features[column], mapping=self.features_binned_mapping[column])
+        if self.is_dummy:
+            X_features.index = X_index
+            X_features['__dummy__'] = 0
+            return X_features
         X_features = X_features[self.features]
         for column in self.features_categorical_final:
             X_features[column].cat.set_categories(self.features_categorical_final_mapping[column], inplace=True)
diff --git a/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_model.py b/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_model.py
--- a/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_model.py
+++ b/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_model.py
@@ -577,8 +577,8 @@ def _get_types_of_features(self, df, skew_threshold, embed_min_categories, use_n
         if not use_ngram_features:
             vectorizers_featnames = self.__get_feature_type_if_present('vectorizers')
             nlp_featnames = self.__get_feature_type_if_present('nlp')
-            self.feature_types_metadata['int'] = [feature for feature in self.feature_types_metadata['int'] if feature not in vectorizers_featnames]
-            self.feature_types_metadata['object'] = [feature for feature in self.feature_types_metadata['object'] if feature not in nlp_featnames]
+            self.feature_types_metadata['int'] = [feature for feature in self.__get_feature_type_if_present('int') if feature not in vectorizers_featnames]
+            self.feature_types_metadata['object'] = [feature for feature in self.__get_feature_type_if_present('object') if feature not in nlp_featnames]
 
         categorical_featnames = self.__get_feature_type_if_present('object') + self.__get_feature_type_if_present('bool')
         continuous_featnames = self.__get_feature_type_if_present('float') + self.__get_feature_type_if_present('int') + self.__get_feature_type_if_present('datetime')


Test patch: