Repository: sktime/sktime

Issue:
Error when running TimeSeriesForestClassifier() function
**Describe the bug**
Encounter an error when running the code in the example. Specifically, TimeSeriesForestClassifier() function.
PicklingError: Can't pickle <function Pipeline.predict_proba at 0x7fab65da6158>: it's not the same object as sklearn.pipeline.Pipeline.predict_proba

**To Reproduce**
tsf = TimeSeriesForestClassifier(base_estimator=base_estimator, 
                                 n_estimators=100,
                                 criterion='entropy',
                                 bootstrap=True, 
                                 oob_score=True, 
                                 random_state=1)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)

**Expected behavior**
Get a score.

**Additional context**
<!--
Add any other context about the problem here.
-->

**Versions**
Linux-3.10.0-693.17.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core
Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
NumPy 1.17.2
SciPy 1.3.1
sktime 0.3.0

<!-- Thanks for contributing! -->

Error when running TimeSeriesForestClassifier() function
**Describe the bug**
Encounter an error when running the code in the example. Specifically, TimeSeriesForestClassifier() function.
PicklingError: Can't pickle <function Pipeline.predict_proba at 0x7fab65da6158>: it's not the same object as sklearn.pipeline.Pipeline.predict_proba

**To Reproduce**
tsf = TimeSeriesForestClassifier(base_estimator=base_estimator, 
                                 n_estimators=100,
                                 criterion='entropy',
                                 bootstrap=True, 
                                 oob_score=True, 
                                 random_state=1)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)

**Expected behavior**
Get a score.

**Additional context**
<!--
Add any other context about the problem here.
-->

**Versions**
Linux-3.10.0-693.17.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core
Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
NumPy 1.17.2
SciPy 1.3.1
sktime 0.3.0

<!-- Thanks for contributing! -->



Discussion:
Hi and thanks for the bug report! 

I've just run this and it seems to work: 
```python
from sktime.classifiers.compose import TimeSeriesForestClassifier
from sktime.datasets import load_gunpoint
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sktime.transformers.summarise import RandomIntervalFeatureExtractor
from sktime.pipeline import Pipeline
from sktime.utils.time_series import time_series_slope

X_train, y_train = load_gunpoint("TRAIN", return_X_y=True)
X_test, y_test = load_gunpoint("TEST", return_X_y=True)

features = [np.mean, np.std, time_series_slope]
steps = [
    ('transform', RandomIntervalFeatureExtractor(n_intervals='sqrt', features=features)),
    ('clf', DecisionTreeClassifier())
]
base_estimator = Pipeline(steps)

tsf = TimeSeriesForestClassifier(
    base_estimator=base_estimator,
    n_estimators=100,
    criterion='entropy',
    bootstrap=True,
    oob_score=True,
    random_state=1
)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)
```

I've seen this error before but thought we've solved it. Could you provide more details on the `base_estimator` and data that you're using?
Hi Markus,
Thank you for your replay.
I try to re-run the exact code your show in the Email but it reports the same error I met before.
Information about the environment:

Linux-3.10.0-693.17.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core

Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09)

[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

NumPy 1.17.2

SciPy 1.3.1

sktime 0.3.0

sklearn 0.21.3


The reported error content:

PicklingErrorTraceback (most recent call last)

<ipython-input-2-f5d357e3d880> in <module>()

     15 )

     16 tsf.fit(X_train, y_train)

---> 17 tsf.score(X_test, y_test)



/opt/intel/intelpython3/lib/python3.6/site-packages/sklearn/base.py in score(self, X, y, sample_weight)

    355         """

    356         from .metrics import accuracy_score

--> 357         return accuracy_score(y, self.predict(X), sample_weight=sample_weight)

    358

    359



/opt/intel/intelpython3/lib/python3.6/site-packages/sklearn/ensemble/forest.py in predict(self, X)

    543             The predicted classes.

    544         """

--> 545         proba = self.predict_proba(X)

    546

    547         if self.n_outputs_ == 1:



/opt/intel/intelpython3/lib/python3.6/site-packages/sktime/classifiers/compose/ensemble.py in predict_proba(self, X)

    387         n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)

    388

--> 389         all_proba = Parallel(n_jobs=n_jobs, verbose=self.verbose)(delayed(e.predict_proba)(X) for e in self.estimators_)

    390

    391         return np.sum(all_proba, axis=0) / len(self.estimators_)



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in __call__(self, iterable)

    777             # was dispatched. In particular this covers the edge

    778             # case of Parallel used with an exhausted iterator.

--> 779             while self.dispatch_one_batch(iterator):

    780                 self._iterating = True

    781             else:



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)

    618

    619         with self._lock:

--> 620             tasks = BatchedCalls(itertools.islice(iterator, batch_size))

    621             if len(tasks) == 0:

    622                 # No more tasks available in the iterator: tell caller to stop.



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in __init__(self, iterator_slice)

    125

    126     def __init__(self, iterator_slice):

--> 127         self.items = list(iterator_slice)

    128         self._size = len(self.items)

    129



/opt/intel/intelpython3/lib/python3.6/site-packages/sktime/classifiers/compose/ensemble.py in <genexpr>(.0)

    387         n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)

    388

--> 389         all_proba = Parallel(n_jobs=n_jobs, verbose=self.verbose)(delayed(e.predict_proba)(X) for e in self.estimators_)

    390

    391         return np.sum(all_proba, axis=0) / len(self.estimators_)



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in delayed(function, check_pickle)

    181     # using with multiprocessing:

    182     if check_pickle:

--> 183         pickle.dumps(function)

    184

    185     def delayed_function(*args, **kwargs):



PicklingError: Can't pickle <function Pipeline.predict_proba at 0x7f54a0802d08>: it's not the same object as sklearn.pipeline.Pipeline.predict_proba


Thank you!

Regards,
Evelyn


发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用

________________________________
From: Markus <notifications@github.com>
Sent: Thursday, October 3, 2019 9:21:20 PM
To: alan-turing-institute/sktime <sktime@noreply.github.com>
Cc: i3evelyn <i3evelyn@hotmail.com>; Author <author@noreply.github.com>
Subject: Re: [alan-turing-institute/sktime] Error when running TimeSeriesForestClassifier() function (#160)


Hi and thanks for the bug report!

I've just run this and it seems to work:

from sktime.classifiers.compose import TimeSeriesForestClassifier
from sktime.datasets import load_gunpoint
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sktime.transformers.summarise import RandomIntervalFeatureExtractor
from sktime.pipeline import Pipeline
from sktime.utils.time_series import time_series_slope

X_train, y_train = load_gunpoint("TRAIN", return_X_y=True)
X_test, y_test = load_gunpoint("TEST", return_X_y=True)

features = [np.mean, np.std, time_series_slope]
steps = [
    ('transform', RandomIntervalFeatureExtractor(n_intervals='sqrt', features=features)),
    ('clf', DecisionTreeClassifier())
]
base_estimator = Pipeline(steps)

tsf = TimeSeriesForestClassifier(
    base_estimator=base_estimator,
    n_estimators=100,
    criterion='entropy',
    bootstrap=True,
    oob_score=True,
    random_state=1
)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)

I've seen this error before but thought we've solved it. Could you provide more details on the base_estimator and data that you're using?

―
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/alan-turing-institute/sktime/issues/160?email_source=notifications&email_token=AD7YBAM5SIIOERUVAAESCKLQMXWVBA5CNFSM4I45QN22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAIFX6Y#issuecomment-537943035>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AD7YBAMFS6OVAUNRKSA2ICLQMXWVBANCNFSM4I45QN2Q>.

I still can't reproduce the error and I'm not sure what causes it. I will later try to run it on a different linux system than we use for unit testing. 

You could try to install the latest development version (cloning and installing from the dev branch as described in the README file) and see if the error still comes up. 
Hi, I am experiencing another issue with this function on macOS 10.14.6 with Python 3.8.0. It looks easily fixable, attached you can find the log from the code @mloning suggested to run (same behavior for the univariate tutorial) and the package versions in my environment. Thank you for this extremely useful package!
[sktime_bug.log](https://github.com/alan-turing-institute/sktime/files/3926922/sktime_bug.log)
[sktime_env.txt](https://github.com/alan-turing-institute/sktime/files/3926923/sktime_env.txt)

UPDATE: setting `bootstrap=False` and `oob_score=False` the code runs fine
Hi @simone-pignotti, thanks for giving us feedback! The code still runs with no errors for me, have you tried it with Python 3.7? There's also a newer sktime version available via pip (0.3.1).  
Thank you for the quick answer! Unfortunately I run into the same issue with the following environment (still macOS 10.14.6 and Python 3.7.3, which btw is installed in a conda environment if that matters).
[sktime_env_37.txt](https://github.com/alan-turing-institute/sktime/files/3927315/sktime_env_37.txt)
I'll test on linux asap!
Why is there no Python in the environment files that you send? What do you see if you run `conda list -n <your_env> | grep python`?
That was the output of pip freeze, here's the full `conda list`.
[sktime_env_conda.txt](https://github.com/alan-turing-institute/sktime/files/3931574/sktime_env_conda.txt)

Also, same error on Debian 9.
Okay, strange, will look into it, but of course would also really appreciate a PR if you have time since I can't really reproduce the error at the moment. :-)
Hi and thanks for the bug report! 

I've just run this and it seems to work: 
```python
from sktime.classifiers.compose import TimeSeriesForestClassifier
from sktime.datasets import load_gunpoint
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sktime.transformers.summarise import RandomIntervalFeatureExtractor
from sktime.pipeline import Pipeline
from sktime.utils.time_series import time_series_slope

X_train, y_train = load_gunpoint("TRAIN", return_X_y=True)
X_test, y_test = load_gunpoint("TEST", return_X_y=True)

features = [np.mean, np.std, time_series_slope]
steps = [
    ('transform', RandomIntervalFeatureExtractor(n_intervals='sqrt', features=features)),
    ('clf', DecisionTreeClassifier())
]
base_estimator = Pipeline(steps)

tsf = TimeSeriesForestClassifier(
    base_estimator=base_estimator,
    n_estimators=100,
    criterion='entropy',
    bootstrap=True,
    oob_score=True,
    random_state=1
)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)
```

I've seen this error before but thought we've solved it. Could you provide more details on the `base_estimator` and data that you're using?
Hi Markus,
Thank you for your replay.
I try to re-run the exact code your show in the Email but it reports the same error I met before.
Information about the environment:

Linux-3.10.0-693.17.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core

Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09)

[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

NumPy 1.17.2

SciPy 1.3.1

sktime 0.3.0

sklearn 0.21.3


The reported error content:

PicklingErrorTraceback (most recent call last)

<ipython-input-2-f5d357e3d880> in <module>()

     15 )

     16 tsf.fit(X_train, y_train)

---> 17 tsf.score(X_test, y_test)



/opt/intel/intelpython3/lib/python3.6/site-packages/sklearn/base.py in score(self, X, y, sample_weight)

    355         """

    356         from .metrics import accuracy_score

--> 357         return accuracy_score(y, self.predict(X), sample_weight=sample_weight)

    358

    359



/opt/intel/intelpython3/lib/python3.6/site-packages/sklearn/ensemble/forest.py in predict(self, X)

    543             The predicted classes.

    544         """

--> 545         proba = self.predict_proba(X)

    546

    547         if self.n_outputs_ == 1:



/opt/intel/intelpython3/lib/python3.6/site-packages/sktime/classifiers/compose/ensemble.py in predict_proba(self, X)

    387         n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)

    388

--> 389         all_proba = Parallel(n_jobs=n_jobs, verbose=self.verbose)(delayed(e.predict_proba)(X) for e in self.estimators_)

    390

    391         return np.sum(all_proba, axis=0) / len(self.estimators_)



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in __call__(self, iterable)

    777             # was dispatched. In particular this covers the edge

    778             # case of Parallel used with an exhausted iterator.

--> 779             while self.dispatch_one_batch(iterator):

    780                 self._iterating = True

    781             else:



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)

    618

    619         with self._lock:

--> 620             tasks = BatchedCalls(itertools.islice(iterator, batch_size))

    621             if len(tasks) == 0:

    622                 # No more tasks available in the iterator: tell caller to stop.



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in __init__(self, iterator_slice)

    125

    126     def __init__(self, iterator_slice):

--> 127         self.items = list(iterator_slice)

    128         self._size = len(self.items)

    129



/opt/intel/intelpython3/lib/python3.6/site-packages/sktime/classifiers/compose/ensemble.py in <genexpr>(.0)

    387         n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)

    388

--> 389         all_proba = Parallel(n_jobs=n_jobs, verbose=self.verbose)(delayed(e.predict_proba)(X) for e in self.estimators_)

    390

    391         return np.sum(all_proba, axis=0) / len(self.estimators_)



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in delayed(function, check_pickle)

    181     # using with multiprocessing:

    182     if check_pickle:

--> 183         pickle.dumps(function)

    184

    185     def delayed_function(*args, **kwargs):



PicklingError: Can't pickle <function Pipeline.predict_proba at 0x7f54a0802d08>: it's not the same object as sklearn.pipeline.Pipeline.predict_proba


Thank you!

Regards,
Evelyn


发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用

________________________________
From: Markus <notifications@github.com>
Sent: Thursday, October 3, 2019 9:21:20 PM
To: alan-turing-institute/sktime <sktime@noreply.github.com>
Cc: i3evelyn <i3evelyn@hotmail.com>; Author <author@noreply.github.com>
Subject: Re: [alan-turing-institute/sktime] Error when running TimeSeriesForestClassifier() function (#160)


Hi and thanks for the bug report!

I've just run this and it seems to work:

from sktime.classifiers.compose import TimeSeriesForestClassifier
from sktime.datasets import load_gunpoint
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sktime.transformers.summarise import RandomIntervalFeatureExtractor
from sktime.pipeline import Pipeline
from sktime.utils.time_series import time_series_slope

X_train, y_train = load_gunpoint("TRAIN", return_X_y=True)
X_test, y_test = load_gunpoint("TEST", return_X_y=True)

features = [np.mean, np.std, time_series_slope]
steps = [
    ('transform', RandomIntervalFeatureExtractor(n_intervals='sqrt', features=features)),
    ('clf', DecisionTreeClassifier())
]
base_estimator = Pipeline(steps)

tsf = TimeSeriesForestClassifier(
    base_estimator=base_estimator,
    n_estimators=100,
    criterion='entropy',
    bootstrap=True,
    oob_score=True,
    random_state=1
)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)

I've seen this error before but thought we've solved it. Could you provide more details on the base_estimator and data that you're using?

―
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/alan-turing-institute/sktime/issues/160?email_source=notifications&email_token=AD7YBAM5SIIOERUVAAESCKLQMXWVBA5CNFSM4I45QN22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAIFX6Y#issuecomment-537943035>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AD7YBAMFS6OVAUNRKSA2ICLQMXWVBANCNFSM4I45QN2Q>.

I still can't reproduce the error and I'm not sure what causes it. I will later try to run it on a different linux system than we use for unit testing. 

You could try to install the latest development version (cloning and installing from the dev branch as described in the README file) and see if the error still comes up. 
Hi, I am experiencing another issue with this function on macOS 10.14.6 with Python 3.8.0. It looks easily fixable, attached you can find the log from the code @mloning suggested to run (same behavior for the univariate tutorial) and the package versions in my environment. Thank you for this extremely useful package!
[sktime_bug.log](https://github.com/alan-turing-institute/sktime/files/3926922/sktime_bug.log)
[sktime_env.txt](https://github.com/alan-turing-institute/sktime/files/3926923/sktime_env.txt)

UPDATE: setting `bootstrap=False` and `oob_score=False` the code runs fine
Hi @simone-pignotti, thanks for giving us feedback! The code still runs with no errors for me, have you tried it with Python 3.7? There's also a newer sktime version available via pip (0.3.1).  
Thank you for the quick answer! Unfortunately I run into the same issue with the following environment (still macOS 10.14.6 and Python 3.7.3, which btw is installed in a conda environment if that matters).
[sktime_env_37.txt](https://github.com/alan-turing-institute/sktime/files/3927315/sktime_env_37.txt)
I'll test on linux asap!
Why is there no Python in the environment files that you send? What do you see if you run `conda list -n <your_env> | grep python`?
That was the output of pip freeze, here's the full `conda list`.
[sktime_env_conda.txt](https://github.com/alan-turing-institute/sktime/files/3931574/sktime_env_conda.txt)

Also, same error on Debian 9.
Okay, strange, will look into it, but of course would also really appreciate a PR if you have time since I can't really reproduce the error at the moment. :-)

Resolving patch:
diff --git a/sktime/classifiers/compose/ensemble.py b/sktime/classifiers/compose/ensemble.py
--- a/sktime/classifiers/compose/ensemble.py
+++ b/sktime/classifiers/compose/ensemble.py
@@ -14,6 +14,7 @@
 from sklearn.ensemble.forest import MAX_INT
 from sklearn.ensemble.forest import _generate_sample_indices
 from sklearn.ensemble.forest import _generate_unsampled_indices
+from sklearn.ensemble.forest import _get_n_samples_bootstrap
 from sklearn.ensemble.base import _partition_estimators
 from sklearn.utils._joblib import Parallel, delayed
 from sklearn.tree._tree import DOUBLE
@@ -146,6 +147,13 @@ class TimeSeriesForestClassifier(ForestClassifier):
         For multi-output, the weights of each column of y will be multiplied.
         Note that these weights will be multiplied with sample_weight (passed
         through the fit method) if sample_weight is specified.
+    max_samples : int or float, default=None
+        If bootstrap is True, the number of samples to draw from X
+        to train each base estimator.
+        - If None (default), then draw `X.shape[0]` samples.
+        - If int, then draw `max_samples` samples.
+        - If float, then draw `max_samples * X.shape[0]` samples. Thus,
+          `max_samples` should be in the interval `(0, 1)`.
 
     Attributes
     ----------
@@ -190,7 +198,8 @@ def __init__(self,
                  random_state=None,
                  verbose=0,
                  warm_start=False,
-                 class_weight=None):
+                 class_weight=None,
+                 max_samples=None):
 
         if base_estimator is None:
             features = [np.mean, np.std, time_series_slope]
@@ -213,7 +222,8 @@ def __init__(self,
         self.max_leaf_nodes = max_leaf_nodes
         self.min_impurity_decrease = min_impurity_decrease
         self.min_impurity_split = min_impurity_split
-        
+        self.max_samples = max_samples
+
         # Rename estimator params according to name in pipeline.
         estimator = base_estimator.steps[-1][0]
         estimator_params = {
@@ -240,7 +250,8 @@ def __init__(self,
             random_state=random_state,
             verbose=verbose,
             warm_start=warm_start,
-            class_weight=class_weight
+            class_weight=class_weight,
+            max_samples=max_samples
         )
 
         # Assign random state to pipeline.
@@ -311,6 +322,12 @@ def fit(self, X, y, sample_weight=None):
             else:
                 sample_weight = expanded_class_weight
 
+        # Get bootstrap sample size
+        n_samples_bootstrap = _get_n_samples_bootstrap(
+            n_samples=X.shape[0],
+            max_samples=self.max_samples
+        )
+
         # Check parameters
         self._validate_estimator()
 
@@ -352,7 +369,8 @@ def fit(self, X, y, sample_weight=None):
             trees = Parallel(n_jobs=self.n_jobs, verbose=self.verbose)(
                 delayed(_parallel_build_trees)(
                     t, self, X, y, sample_weight, i, len(trees),
-                    verbose=self.verbose, class_weight=self.class_weight)
+                    verbose=self.verbose, class_weight=self.class_weight,
+                    n_samples_bootstrap=n_samples_bootstrap)
                 for i, t in enumerate(trees))
 
             # Collect newly grown trees
@@ -433,9 +451,13 @@ def _set_oob_score(self, X, y):
         predictions = [np.zeros((n_samples, n_classes_[k]))
                        for k in range(self.n_outputs_)]
 
+        n_samples_bootstrap = _get_n_samples_bootstrap(
+            n_samples, self.max_samples
+        )
+
         for estimator in self.estimators_:
             unsampled_indices = _generate_unsampled_indices(
-                estimator.random_state, n_samples)
+                estimator.random_state, n_samples, n_samples_bootstrap)
             p_estimator = estimator.predict_proba(X.iloc[unsampled_indices, :])
 
             if self.n_outputs_ == 1:
@@ -465,7 +487,8 @@ def _set_oob_score(self, X, y):
 
 
 def _parallel_build_trees(tree, forest, X, y, sample_weight, tree_idx, n_trees,
-                          verbose=0, class_weight=None):
+                          verbose=0, class_weight=None,
+                          n_samples_bootstrap=None):
     """Private function used to fit a single tree in parallel, adjusted for pipeline trees."""
     if verbose > 1:
         print("building tree %d of %d" % (tree_idx + 1, n_trees))
@@ -480,7 +503,8 @@ def _parallel_build_trees(tree, forest, X, y, sample_weight, tree_idx, n_trees,
         else:
             curr_sample_weight = sample_weight.copy()
 
-        indices = _generate_sample_indices(tree.random_state, n_samples)
+        indices = _generate_sample_indices(tree.random_state, n_samples,
+                                           n_samples_bootstrap)
         sample_counts = np.bincount(indices, minlength=n_samples)
         curr_sample_weight *= sample_counts
 


Test patch: