Repository: automl/auto-sklearn

Issue:
Auto-sklearn predict_proba is too slow on very large test datatset
I'm using a big test dataset (12500 k rows) for prediction. `predict_proba` takes about one day using autosklearn model, formed by only two pipelines, meanwhile it's takes only 4 hours with Random Forest sklearn classifier. I tried to use more than one job but i faced memory issue "Memory error", knowing i have   250 GB of memory. 

Have you please any idea to boost prediction function ?



Discussion:
Here is the tail of the stacktraces: 

> /work/home/auto-sk/as_env/lib/python3.5/site-packages/autosklearn/pipeline/components/data_preprocessing/variance_threshold/variance_threshold.py in transform(self=VarianceThreshold(random_state=None), X=array([[0.00000000e+00, 1.00000000e+00, 0.000000...3.90000000e+01, 2.40000000e+01, 2.50000000e+03]]))
>      20         return self
>      21 
>      22     def transform(self, X):
>      23         if self.preprocessor is None:
>      24             raise NotImplementedError()
> ---> 25         return self.preprocessor.transform(X)
>         self.preprocessor.transform = <bound method SelectorMixin.transform of VarianceThreshold(threshold=0.0)>
>         X = array([[0.00000000e+00, 1.00000000e+00, 0.000000...3.90000000e+01, 2.40000000e+01, 2.50000000e+03]])
>      26 
>      27     @staticmethod
>      28     def get_properties(dataset_properties=None):
>      29         return {
> 
> ...........................................................................
> /work/home/auto-sk/as_env/lib/python3.5/site-packages/sklearn/feature_selection/base.py in transform(self=VarianceThreshold(threshold=0.0), X=array([[0.00000000e+00, 1.00000000e+00, 0.000000...3.90000000e+01, 2.40000000e+01, 2.50000000e+03]]))
>      79                  " too noisy or the selection test too strict.",
>      80                  UserWarning)
>      81             return np.empty(0).reshape((X.shape[0], 0))
>      82         if len(mask) != X.shape[1]:
>      83             raise ValueError("X has a different shape than during fitting.")
> ---> 84         return X[:, safe_mask(X, mask)]
>         X = array([[0.00000000e+00, 1.00000000e+00, 0.000000...3.90000000e+01, 2.40000000e+01, 2.50000000e+03]])
>         mask = array([False,  True,  True,  True,  True,  True,... True,
>         True,  True,  True,  True,  True])
>      85 
>      86     def inverse_transform(self, X):
>      87         """
>      88         Reverse the transformation operation
> 
> MemoryError: 
Could you please paste your model? In case the two models are expensive for prediction, it can happen that they are much slower than scikit-learn's random forest.
> [(0.9, SimpleClassificationPipeline({'classifier:random_forest:max_depth': 'None', 'classifier:random_forest:min_samples_leaf': 1, 'categorical_encoding:__choice__': 'one_hot_encoding', 'classifier:random_forest:max_leaf_nodes': 'None', 'rescaling:__choice__': 'standardize', 'categorical_encoding:one_hot_encoding:use_minimum_fraction': 'True', 'classifier:__choice__': 'random_forest', 'preprocessor:__choice__': 'no_preprocessing', 'classifier:random_forest:min_impurity_decrease': 0.0, 'classifier:random_forest:criterion': 'gini', 'categorical_encoding:one_hot_encoding:minimum_fraction': 0.01, 'imputation:strategy': 'mean', 'classifier:random_forest:bootstrap': 'True', 'balancing:strategy': 'none', 'classifier:random_forest:min_samples_split': 2, 'classifier:random_forest:max_features': 0.5, 'classifier:random_forest:n_estimators': 100, 'classifier:random_forest:min_weight_fraction_leaf': 0.0},
> dataset_properties={
>   'multiclass': False,
>   'multilabel': False,
>   'signed': False,
>   'target_type': 'classification',
>   'task': 1,
>   'sparse': False})), (0.1, SimpleClassificationPipeline({'classifier:random_forest:max_depth': 'None', 'classifier:random_forest:min_samples_leaf': 20, 'categorical_encoding:__choice__': 'one_hot_encoding', 'classifier:random_forest:max_leaf_nodes': 'None', 'rescaling:__choice__': 'quantile_transformer', 'categorical_encoding:one_hot_encoding:use_minimum_fraction': 'True', 'rescaling:quantile_transformer:output_distribution': 'uniform', 'rescaling:quantile_transformer:n_quantiles': 1000, 'classifier:__choice__': 'random_forest', 'preprocessor:__choice__': 'no_preprocessing', 'classifier:random_forest:min_impurity_decrease': 0.0, 'classifier:random_forest:criterion': 'entropy', 'categorical_encoding:one_hot_encoding:minimum_fraction': 0.002615346832354839, 'imputation:strategy': 'most_frequent', 'classifier:random_forest:bootstrap': 'True', 'balancing:strategy': 'weighting', 'classifier:random_forest:min_samples_split': 15, 'classifier:random_forest:max_features': 0.7884268823432835, 'classifier:random_forest:n_estimators': 100, 'classifier:random_forest:min_weight_fraction_leaf': 0.0},
> dataset_properties={
>   'multiclass': False,
>   'multilabel': False,
>   'signed': False,
>   'target_type': 'classification',
>   'task': 1,
>   'sparse': False}))]
Is it possible that you either share your data or code to generate similar data? Then we could cook up a minimal working example to see what's going (wr)on(g).
I'm using 

> predictions = automl.predict_proba(test_set_X, batch_size=1000000, n_jobs=8)[:,1]

I reduced batch size to 1million to prevent memory error but it takes too much time (about 16 hours)
I'm using a dataset of 12,5 million rows with  25 features, only 7 of them are numerical.

Many thanks in advance

We'll have a look, it might take a bit, though.
as I understand current predict methods first compute predictions for all fitted models and then use them for ensemble (even if their weight = 0), if we compute predictions only for non zero weights it may reduce this timings dramatically.
Nope, Auto-sklearn uses only the models in the ensemble to predict, see [here](https://github.com/automl/auto-sklearn/blob/master/autosklearn/automl.py#L533). Furthermore, @yMorakkam only uses two models, and both have a weight larger than zero. Still, I don't know where this issue originates.
I'm not sure that it's true.

We select models and send them to ensemble. At this moment we fix this models in ensemble.identifiers_.

If we want to have ensemble of size 50, have for example 100 best models from nbest, but after selection use only one model all 50 times (so it have weight 1) we have zero weights for other models but still compute all predictions (since ensemble.get_identifiers() returns all ensemble.identifiers_ of length nbest or so)

Then we print models we use function ensemble.get_models_with_weights(), so we do not see these zero weighted models.
Thanks a lot for pointing this out. This is indeed an issue which needs to be fixed.
Thanks @VladimirGl for pointing this issue.
As suggested by  `example_parallel.py`, i used `fit_ensemble` function to construct my ensemble. Then, i save my model using `joblib.dump`. However, my `model.pkl` has neither `models_` nor `ensemble_` objects. So, i think the line below should be added to `fit_ensemble` in `automl.py`
> self._load_models()

So, when predicting using model.pkl, `predict()` function looks for models and ensemble in tmp folder  

Resolving patch:
diff --git a/autosklearn/automl.py b/autosklearn/automl.py
--- a/autosklearn/automl.py
+++ b/autosklearn/automl.py
@@ -406,7 +406,7 @@ def _fit(self, datamanager, metric, only_return_configuration_space=False):
             del self._datamanager
         except Exception:
             pass
-            
+
         # => RUN SMAC
         smac_task_name = 'runSMAC'
         self._stopwatch.start_task(smac_task_name)
@@ -554,7 +554,7 @@ def predict(self, X, batch_size=None, n_jobs=1):
         # Each process computes predictions in chunks of batch_size rows.
         all_predictions = joblib.Parallel(n_jobs=n_jobs)(
             joblib.delayed(_model_predict)(self, X, batch_size, identifier)
-            for identifier in self.ensemble_.get_model_identifiers())
+            for identifier in self.ensemble_.get_model_selected_identifiers())
 
         if len(all_predictions) == 0:
             raise ValueError('Something went wrong generating the predictions. '
diff --git a/autosklearn/ensembles/ensemble_selection.py b/autosklearn/ensembles/ensemble_selection.py
--- a/autosklearn/ensembles/ensemble_selection.py
+++ b/autosklearn/ensembles/ensemble_selection.py
@@ -89,7 +89,7 @@ def _fast(self, predictions, labels):
                     task_type=self.task_type,
                     metric=self.metric,
                     all_scoring_functions=False)
-            
+
             all_best = np.argwhere(scores == np.nanmin(scores)).flatten()
             best = np.random.choice(all_best)
             ensemble.append(predictions[best])
@@ -194,7 +194,8 @@ def _bagging(self, predictions, labels, fraction=0.5, n_bags=20):
         return np.array(order_of_each_bag)
 
     def predict(self, predictions):
-        for i, weight in enumerate(self.weights_):
+        non_null_weights = (weight for  weight in self.weights_ if weight > 0)
+        for i, weight in enumerate(non_null_weights):
             predictions[i] *= weight
         return np.sum(predictions, axis=0)
 
@@ -224,5 +225,17 @@ def get_models_with_weights(self, models):
     def get_model_identifiers(self):
         return self.identifiers_
 
+    def get_model_selected_identifiers(self):
+        output = []
+
+        for i, weight in enumerate(self.weights_):
+            identifier = self.identifiers_[i]
+            if weight > 0.0:
+                output.append(identifier)
+
+        output.sort(reverse=True, key=lambda t: t[0])
+
+        return output
+
     def get_validation_performance(self):
-        return self.trajectory_[-1]
\ No newline at end of file
+        return self.trajectory_[-1]


Test patch: