Repository: automl/auto-sklearn

Issue:
AutoSklearn2Classifier issues being pickle-ed
## Describe the bug ##
I've been testing the `AutoSklearnClassifier` and `AutoSklearn2Classifier` classifiers. 

Using the v1, I get expected results in training the model. However, on saving, it crashes with the following details:

> /home/bot/.local/lib/python3.6/site-packages/sklearn/linear_model/_stochastic_gradient.py:557: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.
  ConvergenceWarning)
/home/bot/.local/lib/python3.6/site-packages/sklearn/base.py:197: FutureWarning: From version 0.24, get_params will raise an AttributeError if a parameter cannot be retrieved as an instance attribute. Previously it would return None.
  FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python3.6/pickle.py", line 918, in save_global
    obj2, parent = _getattribute(module, name)
  File "/usr/lib/python3.6/pickle.py", line 266, in _getattribute
    .format(name, obj))
AttributeError: Can't get local attribute 'get_smac_object_callback.<locals>.get_smac_object' on <function get_smac_object_callback at 0x7ff884638378>
During handling of the above exception, another exception occurred:
joblib.dump(model, osp.join(base_path, model_type + "_model.pkl"))
_pickle.PicklingError: Can't pickle <function get_smac_object_callback.<locals>.get_smac_object at 0x7ff8427e7f28>: it's not found as autosklearn.experimental.askl2.get_smac_object_callback.<locals>.get_smac_object



From what I've gathered, the model cannot be pickled.  


## To Reproduce ##
// Make a classifier using: 
from autosklearn.classification import AutoSklearnClassifier
import joblib
model = AutoSklearnClassifier()
model.fit(X,y)
joblib.dump(model, "model.pkl")

## Expected behavior ##
Both v1 and v2 classifiers to be able to fit, save and re-load.

## Actual behavior, stacktrace or logfile ##
See above

## Environment and installation: ##

Please give details about your installation:
Versions in docker container:
sklearn: 0.22.2.post1
auto-sklearn: 0.8.0
python: 3.6



Discussion:
AutoSK2 Classifier promptly throws an error after starting. Guess it is the same problem as the OP...

Seems like the tool is trying to pickle a selector into a system library folder, instead of into my current folder 
(it also ignores the "output_folder" parameter in the Class constructor)

Example code:
```
from autosklearn.experimental.askl2 import AutoSklearn2Classifier
model = AutoSklearn2Classifier( 
    include_estimators=["random_forest", ], exclude_estimators=None,
    include_preprocessors=["no_preprocessing", ], exclude_preprocessors=None,
    n_jobs=8,
    output_folder='.')    
```

Error output:
```
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-84-90b488fce84f> in <module>
      3 import sklearn.metrics
      4 
----> 5 from autosklearn.experimental.askl2 import AutoSklearn2Classifier
      6 model = AutoSklearn2Classifier( #)
      7     include_estimators=["random_forest", ], exclude_estimators=None,

/usr/local/lib/python3.8/site-packages/autosklearn/experimental/askl2.py in <module>
     40         maxima=maxima_for_methods,
     41     )
---> 42     with open(selector_file, 'wb') as fh:
     43         pickle.dump(selector, fh)
     44 

PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.8/site-packages/autosklearn/experimental/selector.pkl'
```
Thanks a lot @23pointsNorth and @Atharex for reporting these issues. I can confirm both issues and we'll work on fixing these.
@mfeurer,

The error encountered by @23pointsNorth was due to python's standard `pickle` or `joblib` not being able to serialise functions. Switching to [dill](https://github.com/uqfoundation/dill) resolves this issue. 

However, just switching library will not be enough. Another error will come up: `Runtime Error: Lock objects should only be shared between processes through inheritance.`

The fix to that subsequent error is in [#955](https://github.com/automl/auto-sklearn/pull/955), by making the `lock` object to be global, together with `CALLBACK_COUNTER`.

Resolving patch:
diff --git a/autosklearn/experimental/askl2.py b/autosklearn/experimental/askl2.py
--- a/autosklearn/experimental/askl2.py
+++ b/autosklearn/experimental/askl2.py
@@ -1,27 +1,43 @@
+import hashlib
 import json
 import os
+import pathlib
 import pickle
 from typing import Any, Dict, Optional, Union
 
 from ConfigSpace import Configuration
 import numpy as np
 import pandas as pd
+import sklearn
 
+import autosklearn
 from autosklearn.classification import AutoSklearnClassifier
 import autosklearn.experimental.selector
 from autosklearn.metrics import Scorer
 
-this_directory = os.path.abspath(os.path.dirname(__file__))
-selector_file = os.path.join(this_directory, 'selector.pkl')
-training_data_file = os.path.join(this_directory, 'askl2_training_data.json')
+this_directory = pathlib.Path(__file__).resolve().parent
+training_data_file = this_directory / 'askl2_training_data.json'
 with open(training_data_file) as fh:
     training_data = json.load(fh)
+    fh.seek(0)
+    m = hashlib.md5()
+    m.update(fh.read().encode('utf8'))
+training_data_hash = m.hexdigest()[:10]
+sklearn_version = sklearn.__version__
+autosklearn_version = autosklearn.__version__
+selector_file = pathlib.Path(
+    os.environ.get(
+        'XDG_CACHE_HOME',
+        '~/.cache/auto-sklearn/askl2_selector_%s_%s_%s.pkl'
+        % (autosklearn_version, sklearn_version, training_data_hash),
+    )
+).expanduser()
 metafeatures = pd.DataFrame(training_data['metafeatures'])
 y_values = np.array(training_data['y_values'])
 strategies = training_data['strategies']
 minima_for_methods = training_data['minima_for_methods']
 maxima_for_methods = training_data['maxima_for_methods']
-if not os.path.exists(selector_file):
+if not selector_file.exists():
     selector = autosklearn.experimental.selector.OneVSOneSelector(
         configuration=training_data['configuration'],
         default_strategy_idx=strategies.index('RF_SH-eta4-i_holdout_iterative_es_if'),
@@ -34,12 +50,17 @@
         minima=minima_for_methods,
         maxima=maxima_for_methods,
     )
+    selector_file.parent.mkdir(exist_ok=True, parents=True)
     with open(selector_file, 'wb') as fh:
         pickle.dump(selector, fh)
 
 
-def get_smac_object_callback(portfolio):
-    def get_smac_object(
+class SmacObjectCallback:
+    def __init__(self, portfolio):
+        self.portfolio = portfolio
+
+    def __call__(
+        self,
         scenario_dict,
         seed,
         ta,
@@ -56,7 +77,7 @@ def get_smac_object(
 
         initial_configurations = [
             Configuration(configuration_space=scenario.cs, values=member)
-            for member in portfolio.values()]
+            for member in self.portfolio.values()]
 
         rh2EPM = RunHistory2EPM4LogCost
         return SMAC4AC(
@@ -70,11 +91,17 @@ def get_smac_object(
             n_jobs=n_jobs,
             dask_client=dask_client,
         )
-    return get_smac_object
 
 
-def get_sh_object_callback(budget_type, eta, initial_budget, portfolio):
-    def get_smac_object(
+class SHObjectCallback:
+    def __init__(self, budget_type, eta, initial_budget, portfolio):
+        self.budget_type = budget_type
+        self.eta = eta
+        self.initial_budget = initial_budget
+        self.portfolio = portfolio
+
+    def __call__(
+        self,
         scenario_dict,
         seed,
         ta,
@@ -91,10 +118,10 @@ def get_smac_object(
         scenario = Scenario(scenario_dict)
         initial_configurations = [
             Configuration(configuration_space=scenario.cs, values=member)
-            for member in portfolio.values()]
+            for member in self.portfolio.values()]
 
         rh2EPM = RunHistory2EPM4LogCost
-        ta_kwargs['budget_type'] = budget_type
+        ta_kwargs['budget_type'] = self.budget_type
 
         smac4ac = SMAC4AC(
             scenario=scenario,
@@ -106,9 +133,9 @@ def get_smac_object(
             run_id=seed,
             intensifier=SuccessiveHalving,
             intensifier_kwargs={
-                'initial_budget': initial_budget,
+                'initial_budget': self.initial_budget,
                 'max_budget': 100,
-                'eta': eta,
+                'eta': self.eta,
                 'min_chall': 1,
             },
             dask_client=dask_client,
@@ -118,7 +145,6 @@ def get_smac_object(
             len(scenario.cs.get_hyperparameters()) / 2
         )
         return smac4ac
-    return get_smac_object
 
 
 class AutoSklearn2Classifier(AutoSklearnClassifier):
@@ -236,15 +262,15 @@ def fit(self, X, y,
         else:
             resampling_strategy_kwargs = None
 
-        portfolio_file = os.path.join(this_directory, 'askl2_portfolios', '%s.json' % automl_policy)
+        portfolio_file = this_directory / 'askl2_portfolios' / ('%s.json' % automl_policy)
         with open(portfolio_file) as fh:
             portfolio_json = json.load(fh)
         portfolio = portfolio_json['portfolio']
 
         if setting['fidelity'] == 'SH':
-            smac_callback = get_sh_object_callback('iterations', 4, 5.0, portfolio)
+            smac_callback = SHObjectCallback('iterations', 4, 5.0, portfolio)
         else:
-            smac_callback = get_smac_object_callback(portfolio)
+            smac_callback = SmacObjectCallback(portfolio)
 
         self.resampling_strategy = resampling_strategy
         self.resampling_strategy_arguments = resampling_strategy_kwargs


Test patch:
diff --git a/test/test_automl/test_estimators.py b/test/test_automl/test_estimators.py
--- a/test/test_automl/test_estimators.py
+++ b/test/test_automl/test_estimators.py
@@ -651,7 +651,7 @@ def test_regression_methods_returns_self(self):
 
 class AutoSklearn2ClassifierTest(unittest.TestCase):
     # Currently this class only tests that the methods of AutoSklearnClassifier
-    # which should return self actually return self.
+    # which should return self actually return self and can be pickled.
     def test_classification_methods_returns_self(self):
         X_train, y_train, X_test, y_test = putil.get_dataset('iris')
         automl = AutoSklearn2Classifier(time_left_for_this_task=60, ensemble_size=0,)
@@ -665,6 +665,11 @@ def test_classification_methods_returns_self(self):
         automl_refitted = automl.refit(X_train.copy(), y_train.copy())
         self.assertIs(automl, automl_refitted)
 
+        predictions = automl_fitted.predict(X_test)
+        self.assertGreaterEqual(sklearn.metrics.accuracy_score(y_test, predictions), 2 / 3)
+
+        pickle.dumps(automl_fitted)
+
 
 if __name__ == "__main__":
     unittest.main()