Repository: automl/auto-sklearn

Issue:
Should baseline DummyClassifier ever be the winning ensemble?
I tried auto-sklearn using the data from a [Kaggle competition](https://www.kaggle.com/c/santander-customer-satisfaction/data), and no matter how I configure the classifier, sklearn's `DummyClassifier` is always the "winning" ensemble.

```
cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=180,
    tmp_folder='/tmp/autosklearn_tmp',
    output_folder='/tmp/autosklearn_out')

cls.fit(X_train, y_train)

cls.show_models()
[(1.000000, MyDummyClassifier(configuration=2, random_states=None)),
]
```

I would expect _any_ other ensemble to be chosen as the output over the DummyClassifier, regardless of training accuracy.



Discussion:
Your time_left_for_this_task=180 (overall time for fitting) is lower then the default time for each model fitting:
http://automl.github.io/auto-sklearn/stable/api.html#api

> per_run_time_limit : int, optional (default=360)     Time limit for a single call to machine learning model.

Have you tried to set  time_left_for_this_task to be at least 360 seconds? Could it be that literally no model had enough time to complete under your 180 seconds time_left_for_this_task ?

Is there a reason you are not going with the default values here?

Actually from my experience it is worth it to set time_left_for_this_task to 900 or even 9000 seconds and per_run_time_limit at least 4 or 5 times that just to see if any model goes through at all. Other things to look for is the ml_memory_limit - set it to as much as you have to sort this factor out. Yet another quick check is include_estimators=('sgd',) . SGDClassifier is one of the fastest there is, usually completes within a minute for very large datasets.

I should have been more clear.  

I'm using `auto-sklearn==0.0.2.dev0`.

I actually started with the AutoSklearnClassifier defaults, setting the `time_left_for_this_task` to 3600 (one hour), and I still got the same result.   I only reduced the time during debugging.

I tried increasing the `ml_memory_limit` as well, but I still end up with the DummyClassifier as the resulting ensemble.

```
cls = autosklearn.classification.AutoSklearnClassifier(
    ml_memory_limit=6000,
    tmp_folder='/tmp/autosklearn_santander_tmp',
    output_folder='/tmp/autosklearn_santander_out')

cls.fit(X_train, y_train)

print(cls.show_models())
[(1.000000, MyDummyClassifier(configuration=2, random_states=None)),]
```

It could just be the [dataset](https://www.kaggle.com/c/santander-customer-satisfaction/data), which contains continuous features that I have not done any preprocessing on.  This is not blocking me, as I'm just kicking the tires.  But it's just an unexpected result to have a DummyClassifier outperform all other ensembles when using the default settings.

Do you see any successfully completed model run e.g. via `grep -h "Finished evaluating" /tmp/autosklearn_santander_tmp/*`

I can think of two failure modes, one is that the dataset is too big for the settings, which basically prevents auto-sklearn from finishing a single configuration in time, the other one is the use of accuracy as the target metric. There a few things I would try:
- have a look into the output at /tmp/autosklearn_santander_tmp/ as @Motorrat suggested.
- subsample the dataset so you only have about 1.000 datapoints and see if it works then.
- convert the data into sparse representation with the `scipy.sparse` module
- find out how long it takes to train a scikit-learn random forest with 100 trees on the unprocessed data. That's basically the default auto-sklearn uses. If you give it less time, it's very likely that it won't work.
- Maximize AUC, the target metric of the competition. The data has less than 4% positive data points. Thus, the dummy classifier will have an accuracy of 96%, something which is very hard to beat for a different classifier. If none of the other models tried in the beginning does so, auto-sklearn will create an ensemble consisting of the dummy classifier . You can find the documentation [here](http://automl.github.io/auto-sklearn/stable/api.html#autosklearn.classification.AutoSklearnClassifier.fit)

@Motorrat, @mfeurer Thanks for all the suggestions.

Models were completing:  

```
grep -h "Finished evaluating" /tmp/autosklearn_santander_tmp/*

[INFO] [2016-09-23 22:49:11,582:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 3. configuration on SUBSET. Duration 4.203074; loss 0.088989; status 1; additional run info: ;duration: 4.203074216842651;num_run:00003 
[INFO] [2016-09-23 22:49:12,244:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 4. configuration on SUBSET. Duration 0.453758; loss 0.717289; status 1; additional run info: ;duration: 0.45375823974609375;num_run:00004 
[INFO] [2016-09-23 22:49:13,382:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 5. configuration on SUBSET. Duration 0.949710; loss 0.542900; status 1; additional run info: ;duration: 0.9497101306915283;num_run:00005 
[INFO] [2016-09-23 22:49:14,045:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 6. configuration on SUBSET. Duration 0.458671; loss 1.749576; status 1; additional run info: ;duration: 0.45867085456848145;num_run:00006 
[INFO] [2016-09-23 22:50:21,239:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 7. configuration. Duration 19.398112; loss 0.092277; status 1; additional run info: ;duration: 19.398112058639526;num_run:00007 
[INFO] [2016-09-23 22:50:32,256:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 8. configuration. Duration 10.827285; loss 2.000000; status 5; additional run info: Memout 
[INFO] [2016-09-23 22:51:46,777:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 9. configuration. Duration 74.305234; loss 0.080518; status 1; additional run info: ;duration: 74.30523371696472;num_run:00009 
[INFO] [2016-09-23 22:52:09,707:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 10. configuration. Duration 22.741419; loss 0.079123; status 1; additional run info: ;duration: 22.741419315338135;num_run:00010 
[INFO] [2016-09-23 22:52:43,575:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 11. configuration. Duration 32.233451; loss 0.101146; status 1; additional run info: ;duration: 32.23345112800598;num_run:00011 
[INFO] [2016-09-23 22:53:01,776:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 12. configuration. Duration 16.998820; loss 0.171898; status 1; additional run info: ;duration: 16.998819828033447;num_run:00012 
[INFO] [2016-09-23 22:53:02,536:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 13. configuration. Duration 0.555217; loss 2.000000; status 5; additional run info: Memout 
[INFO] [2016-09-23 22:57:47,756:AutoMLSMBO(1)::0d79f8daa63cc07c033e3b8d698a3267] Finished evaluating 14. configuration. Duration 284.745253; loss 2.000000; status 5; additional run info: Memout 
```

Also, @mfeurer is correct that the dummy classifier accuracy is 96% when I do not preprocess the data.  Following his suggestions:
- taking subsample of 1,000 data points yielded the dummy classifier model
- converting to sparse matrix yielded the dummy classifier model
- RandomForestClassifier with 100 trees took just 44 seconds (with approximately 96% accuracy)
- using the AUC metric did the trick, producing

```
cls.fit(X_train, y_train, metric='auc_metric')

print(cls.show_models())
[(0.740000, SimpleClassificationPipeline(configuration={
  'balancing:strategy': 'none',
  'classifier:__choice__': 'adaboost',
  'classifier:adaboost:algorithm': 'SAMME',
  'classifier:adaboost:learning_rate': 1.0081104516473922,
  'classifier:adaboost:max_depth': 6,
  'classifier:adaboost:n_estimators': 468,
  'imputation:strategy': 'mean',
  'one_hot_encoding:use_minimum_fraction': 'False',
  'preprocessor:__choice__': 'liblinear_svc_preprocessor',
  'preprocessor:liblinear_svc_preprocessor:C': 1.1828431725901418,
  'preprocessor:liblinear_svc_preprocessor:dual': 'False',
  'preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True',
  'preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1,
  'preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge',
  'preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr',
  'preprocessor:liblinear_svc_preprocessor:penalty': 'l1',
  'preprocessor:liblinear_svc_preprocessor:tol': 0.0022792606924326923,
  'rescaling:__choice__': 'min/max'})),
(0.180000, SimpleClassificationPipeline(configuration={
  'balancing:strategy': 'none',
  'classifier:__choice__': 'random_forest',
  'classifier:random_forest:bootstrap': 'True',
  'classifier:random_forest:criterion': 'gini',
  'classifier:random_forest:max_depth': 'None',
  'classifier:random_forest:max_features': 1.0,
  'classifier:random_forest:max_leaf_nodes': 'None',
  'classifier:random_forest:min_samples_leaf': 1,
  'classifier:random_forest:min_samples_split': 2,
  'classifier:random_forest:min_weight_fraction_leaf': 0.0,
  'classifier:random_forest:n_estimators': 100,
  'imputation:strategy': 'mean',
  'one_hot_encoding:minimum_fraction': 0.01,
  'one_hot_encoding:use_minimum_fraction': 'True',
  'preprocessor:__choice__': 'no_preprocessing',
  'rescaling:__choice__': 'min/max'})),
(0.060000, SimpleClassificationPipeline(configuration={
  'balancing:strategy': 'none',
  'classifier:__choice__': 'random_forest',
  'classifier:random_forest:bootstrap': 'True',
  'classifier:random_forest:criterion': 'gini',
  'classifier:random_forest:max_depth': 'None',
  'classifier:random_forest:max_features': 1.0,
  'classifier:random_forest:max_leaf_nodes': 'None',
  'classifier:random_forest:min_samples_leaf': 1,
  'classifier:random_forest:min_samples_split': 2,
  'classifier:random_forest:min_weight_fraction_leaf': 0.0,
  'classifier:random_forest:n_estimators': 100,
  'imputation:strategy': 'mean',
  'one_hot_encoding:minimum_fraction': 0.01,
  'one_hot_encoding:use_minimum_fraction': 'True',
  'preprocessor:__choice__': 'no_preprocessing',
  'rescaling:__choice__': 'min/max'})),
(0.020000, SimpleClassificationPipeline(configuration={
  'balancing:strategy': 'weighting',
  'classifier:__choice__': 'sgd',
  'classifier:sgd:alpha': 0.0001,
  'classifier:sgd:average': 'True',
  'classifier:sgd:eta0': 0.01,
  'classifier:sgd:fit_intercept': 'True',
  'classifier:sgd:learning_rate': 'optimal',
  'classifier:sgd:loss': 'hinge',
  'classifier:sgd:n_iter': 5,
  'classifier:sgd:penalty': 'l2',
  'imputation:strategy': 'mean',
  'one_hot_encoding:minimum_fraction': 0.1,
  'one_hot_encoding:use_minimum_fraction': 'True',
  'preprocessor:__choice__': 'no_preprocessing',
  'rescaling:__choice__': 'min/max'})),
]

predictions = cls.predict(X_test)

from sklearn.metrics import accuracy_score
print("Accuracy score", accuracy_score(y_test, predictions))
Accuracy score 0.95665614312
```

So I consider this an exercise in how to use auto-sklearn properly.  Closing.

I am having the same issue - the DummyClassifier always ends up being 90+% of the final ensemble. I am trying to do a pretty imbalanced classification problem, and tried:

- Subsampling the data down to 2000 points (original is closer to 1 million)
- Trying `roc_auc`, `f1_micro`,` f1_macro`
- Removing kernel methods from the search space since they can be slow
- Increasing `time_left_for_this_task` and `per_run_time_limit` to several hours each 
- Increasing the memory limit to 10000
- Both holdout and 10-fold cross-validation

I have not been able to get any parameter settings that give a useful estimator. Training a random forest with default values takes something like 2 seconds with the subsampled dataset, but if I search for `Status` in the logfile, it is always either crashing or timing out. The only result for `grep -h "Finish"` in the logfile is `Finished creating dummy predictions.`

Has anything changed in the last year that would change the suggestions above?
If you are willing to experiment a little further you could try out https://github.com/paypal/autosklearn-zeroconf. It’s a wrapper around autosklearn that does its configuration for you. 
There is no guarantee that it’ll give a different result but at least you’ll get a “second opinion “ on your configuration. 
It seems that we introduced this bug when switching from our internal metrics to the scikit-learn metrics. As the metrics in scikit-learn are not normalized to be zero in the case of random predictions, our check for random models does no longer exclude the dummy classifier. I will work on a fix for this.

Resolving patch:
diff --git a/autosklearn/ensemble_builder.py b/autosklearn/ensemble_builder.py
--- a/autosklearn/ensemble_builder.py
+++ b/autosklearn/ensemble_builder.py
@@ -203,7 +203,7 @@ def main(self):
                 num_run = int(match.group(2))
 
                 if self.ensemble_nbest is not None:
-                    if score <= 0.001:
+                    if score <= 0.001 or num_run == 1:
                         self.logger.info('Model only predicts at random: ' +
                                          model_name + ' has score: ' + str(score))
                         backup_num_runs.append((automl_seed, num_run))
@@ -243,7 +243,7 @@ def main(self):
 
                 else:
                     # Load all predictions that are better than random
-                    if score <= 0.001:
+                    if score <= 0.001 or num_run == 1:
                         # include_num_runs.append(True)
                         self.logger.info('Model only predicts at random: ' +
                                          model_name + ' has score: ' +


Test patch: