Repository: sktime/sktime

Issue:
Error when running TimeSeriesForestClassifier() function
**Describe the bug**
Encounter an error when running the code in the example. Specifically, TimeSeriesForestClassifier() function.
PicklingError: Can't pickle <function Pipeline.predict_proba at 0x7fab65da6158>: it's not the same object as sklearn.pipeline.Pipeline.predict_proba

**To Reproduce**
tsf = TimeSeriesForestClassifier(base_estimator=base_estimator, 
                                 n_estimators=100,
                                 criterion='entropy',
                                 bootstrap=True, 
                                 oob_score=True, 
                                 random_state=1)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)

**Expected behavior**
Get a score.

**Additional context**
<!--
Add any other context about the problem here.
-->

**Versions**
Linux-3.10.0-693.17.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core
Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
NumPy 1.17.2
SciPy 1.3.1
sktime 0.3.0

<!-- Thanks for contributing! -->



Discussion:
Hi and thanks for the bug report! 

I've just run this and it seems to work: 
```python
from sktime.classifiers.compose import TimeSeriesForestClassifier
from sktime.datasets import load_gunpoint
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sktime.transformers.summarise import RandomIntervalFeatureExtractor
from sktime.pipeline import Pipeline
from sktime.utils.time_series import time_series_slope

X_train, y_train = load_gunpoint("TRAIN", return_X_y=True)
X_test, y_test = load_gunpoint("TEST", return_X_y=True)

features = [np.mean, np.std, time_series_slope]
steps = [
    ('transform', RandomIntervalFeatureExtractor(n_intervals='sqrt', features=features)),
    ('clf', DecisionTreeClassifier())
]
base_estimator = Pipeline(steps)

tsf = TimeSeriesForestClassifier(
    base_estimator=base_estimator,
    n_estimators=100,
    criterion='entropy',
    bootstrap=True,
    oob_score=True,
    random_state=1
)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)
```

I've seen this error before but thought we've solved it. Could you provide more details on the `base_estimator` and data that you're using?
Hi Markus,
Thank you for your replay.
I try to re-run the exact code your show in the Email but it reports the same error I met before.
Information about the environment:

Linux-3.10.0-693.17.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core

Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09)

[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

NumPy 1.17.2

SciPy 1.3.1

sktime 0.3.0

sklearn 0.21.3


The reported error content:

PicklingErrorTraceback (most recent call last)

<ipython-input-2-f5d357e3d880> in <module>()

     15 )

     16 tsf.fit(X_train, y_train)

---> 17 tsf.score(X_test, y_test)



/opt/intel/intelpython3/lib/python3.6/site-packages/sklearn/base.py in score(self, X, y, sample_weight)

    355         """

    356         from .metrics import accuracy_score

--> 357         return accuracy_score(y, self.predict(X), sample_weight=sample_weight)

    358

    359



/opt/intel/intelpython3/lib/python3.6/site-packages/sklearn/ensemble/forest.py in predict(self, X)

    543             The predicted classes.

    544         """

--> 545         proba = self.predict_proba(X)

    546

    547         if self.n_outputs_ == 1:



/opt/intel/intelpython3/lib/python3.6/site-packages/sktime/classifiers/compose/ensemble.py in predict_proba(self, X)

    387         n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)

    388

--> 389         all_proba = Parallel(n_jobs=n_jobs, verbose=self.verbose)(delayed(e.predict_proba)(X) for e in self.estimators_)

    390

    391         return np.sum(all_proba, axis=0) / len(self.estimators_)



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in __call__(self, iterable)

    777             # was dispatched. In particular this covers the edge

    778             # case of Parallel used with an exhausted iterator.

--> 779             while self.dispatch_one_batch(iterator):

    780                 self._iterating = True

    781             else:



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)

    618

    619         with self._lock:

--> 620             tasks = BatchedCalls(itertools.islice(iterator, batch_size))

    621             if len(tasks) == 0:

    622                 # No more tasks available in the iterator: tell caller to stop.



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in __init__(self, iterator_slice)

    125

    126     def __init__(self, iterator_slice):

--> 127         self.items = list(iterator_slice)

    128         self._size = len(self.items)

    129



/opt/intel/intelpython3/lib/python3.6/site-packages/sktime/classifiers/compose/ensemble.py in <genexpr>(.0)

    387         n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)

    388

--> 389         all_proba = Parallel(n_jobs=n_jobs, verbose=self.verbose)(delayed(e.predict_proba)(X) for e in self.estimators_)

    390

    391         return np.sum(all_proba, axis=0) / len(self.estimators_)



/opt/intel/intelpython3/lib/python3.6/site-packages/joblib/parallel.py in delayed(function, check_pickle)

    181     # using with multiprocessing:

    182     if check_pickle:

--> 183         pickle.dumps(function)

    184

    185     def delayed_function(*args, **kwargs):



PicklingError: Can't pickle <function Pipeline.predict_proba at 0x7f54a0802d08>: it's not the same object as sklearn.pipeline.Pipeline.predict_proba


Thank you!

Regards,
Evelyn


发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用

________________________________
From: Markus <notifications@github.com>
Sent: Thursday, October 3, 2019 9:21:20 PM
To: alan-turing-institute/sktime <sktime@noreply.github.com>
Cc: i3evelyn <i3evelyn@hotmail.com>; Author <author@noreply.github.com>
Subject: Re: [alan-turing-institute/sktime] Error when running TimeSeriesForestClassifier() function (#160)


Hi and thanks for the bug report!

I've just run this and it seems to work:

from sktime.classifiers.compose import TimeSeriesForestClassifier
from sktime.datasets import load_gunpoint
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sktime.transformers.summarise import RandomIntervalFeatureExtractor
from sktime.pipeline import Pipeline
from sktime.utils.time_series import time_series_slope

X_train, y_train = load_gunpoint("TRAIN", return_X_y=True)
X_test, y_test = load_gunpoint("TEST", return_X_y=True)

features = [np.mean, np.std, time_series_slope]
steps = [
    ('transform', RandomIntervalFeatureExtractor(n_intervals='sqrt', features=features)),
    ('clf', DecisionTreeClassifier())
]
base_estimator = Pipeline(steps)

tsf = TimeSeriesForestClassifier(
    base_estimator=base_estimator,
    n_estimators=100,
    criterion='entropy',
    bootstrap=True,
    oob_score=True,
    random_state=1
)
tsf.fit(X_train, y_train)
tsf.score(X_test, y_test)

I've seen this error before but thought we've solved it. Could you provide more details on the base_estimator and data that you're using?

―
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/alan-turing-institute/sktime/issues/160?email_source=notifications&email_token=AD7YBAM5SIIOERUVAAESCKLQMXWVBA5CNFSM4I45QN22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAIFX6Y#issuecomment-537943035>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AD7YBAMFS6OVAUNRKSA2ICLQMXWVBANCNFSM4I45QN2Q>.

I still can't reproduce the error and I'm not sure what causes it. I will later try to run it on a different linux system than we use for unit testing. 

You could try to install the latest development version (cloning and installing from the dev branch as described in the README file) and see if the error still comes up. 

Resolving patch:
diff --git a/sktime/classifiers/compose/ensemble.py b/sktime/classifiers/compose/ensemble.py
--- a/sktime/classifiers/compose/ensemble.py
+++ b/sktime/classifiers/compose/ensemble.py
@@ -14,6 +14,7 @@
 from sklearn.ensemble.forest import MAX_INT
 from sklearn.ensemble.forest import _generate_sample_indices
 from sklearn.ensemble.forest import _generate_unsampled_indices
+from sklearn.ensemble.forest import _get_n_samples_bootstrap
 from sklearn.ensemble.base import _partition_estimators
 from sklearn.utils._joblib import Parallel, delayed
 from sklearn.tree._tree import DOUBLE
@@ -146,6 +147,13 @@ class TimeSeriesForestClassifier(ForestClassifier):
         For multi-output, the weights of each column of y will be multiplied.
         Note that these weights will be multiplied with sample_weight (passed
         through the fit method) if sample_weight is specified.
+    max_samples : int or float, default=None
+        If bootstrap is True, the number of samples to draw from X
+        to train each base estimator.
+        - If None (default), then draw `X.shape[0]` samples.
+        - If int, then draw `max_samples` samples.
+        - If float, then draw `max_samples * X.shape[0]` samples. Thus,
+          `max_samples` should be in the interval `(0, 1)`.
 
     Attributes
     ----------
@@ -190,7 +198,8 @@ def __init__(self,
                  random_state=None,
                  verbose=0,
                  warm_start=False,
-                 class_weight=None):
+                 class_weight=None,
+                 max_samples=None):
 
         if base_estimator is None:
             features = [np.mean, np.std, time_series_slope]
@@ -213,7 +222,8 @@ def __init__(self,
         self.max_leaf_nodes = max_leaf_nodes
         self.min_impurity_decrease = min_impurity_decrease
         self.min_impurity_split = min_impurity_split
-        
+        self.max_samples = max_samples
+
         # Rename estimator params according to name in pipeline.
         estimator = base_estimator.steps[-1][0]
         estimator_params = {
@@ -240,7 +250,8 @@ def __init__(self,
             random_state=random_state,
             verbose=verbose,
             warm_start=warm_start,
-            class_weight=class_weight
+            class_weight=class_weight,
+            max_samples=max_samples
         )
 
         # Assign random state to pipeline.
@@ -311,6 +322,12 @@ def fit(self, X, y, sample_weight=None):
             else:
                 sample_weight = expanded_class_weight
 
+        # Get bootstrap sample size
+        n_samples_bootstrap = _get_n_samples_bootstrap(
+            n_samples=X.shape[0],
+            max_samples=self.max_samples
+        )
+
         # Check parameters
         self._validate_estimator()
 
@@ -352,7 +369,8 @@ def fit(self, X, y, sample_weight=None):
             trees = Parallel(n_jobs=self.n_jobs, verbose=self.verbose)(
                 delayed(_parallel_build_trees)(
                     t, self, X, y, sample_weight, i, len(trees),
-                    verbose=self.verbose, class_weight=self.class_weight)
+                    verbose=self.verbose, class_weight=self.class_weight,
+                    n_samples_bootstrap=n_samples_bootstrap)
                 for i, t in enumerate(trees))
 
             # Collect newly grown trees
@@ -433,9 +451,13 @@ def _set_oob_score(self, X, y):
         predictions = [np.zeros((n_samples, n_classes_[k]))
                        for k in range(self.n_outputs_)]
 
+        n_samples_bootstrap = _get_n_samples_bootstrap(
+            n_samples, self.max_samples
+        )
+
         for estimator in self.estimators_:
             unsampled_indices = _generate_unsampled_indices(
-                estimator.random_state, n_samples)
+                estimator.random_state, n_samples, n_samples_bootstrap)
             p_estimator = estimator.predict_proba(X.iloc[unsampled_indices, :])
 
             if self.n_outputs_ == 1:
@@ -465,7 +487,8 @@ def _set_oob_score(self, X, y):
 
 
 def _parallel_build_trees(tree, forest, X, y, sample_weight, tree_idx, n_trees,
-                          verbose=0, class_weight=None):
+                          verbose=0, class_weight=None,
+                          n_samples_bootstrap=None):
     """Private function used to fit a single tree in parallel, adjusted for pipeline trees."""
     if verbose > 1:
         print("building tree %d of %d" % (tree_idx + 1, n_trees))
@@ -480,7 +503,8 @@ def _parallel_build_trees(tree, forest, X, y, sample_weight, tree_idx, n_trees,
         else:
             curr_sample_weight = sample_weight.copy()
 
-        indices = _generate_sample_indices(tree.random_state, n_samples)
+        indices = _generate_sample_indices(tree.random_state, n_samples,
+                                           n_samples_bootstrap)
         sample_counts = np.bincount(indices, minlength=n_samples)
         curr_sample_weight *= sample_counts
 
diff --git a/sktime/transformers/pca.py b/sktime/transformers/pca.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformers/pca.py
@@ -0,0 +1,86 @@
+__author__ = "Patrick Rockenschaub"
+__all__ = "PCATransformer"
+
+import pandas as pd
+
+from sklearn.utils.validation import check_is_fitted
+from sklearn.decomposition import PCA
+
+from sktime.transformers.base import BaseTransformer
+from sktime.utils.validation.supervised import validate_X, check_X_is_univariate
+from sktime.utils.data_container import tabularise, detabularise, check_equal_index
+
+
+class PCATransformer(BaseTransformer):
+    """ Transformer that applies Principle Components Analysis to a univariate time series.
+
+    Provides a simple wrapper around ``sklearn.decomposition.PCA``.
+
+    Parameters
+    ----------
+    n_components : int, float, str or None (default None)
+        Number of principle components to retain. By default, all components are retained. See
+        ``sklearn.decomposition.PCA`` documentation for a detailed description of all options.
+    **kwargs
+        Additional parameters passed on to ``sklearn.decomposition.PCA``. See ``sklearn.decomposition.PCA``
+        documentation for a detailed description of all options.
+    """
+
+    def __init__(self, n_components=None, **kwargs):
+        self.pca = PCA(n_components, **kwargs)
+
+    def fit(self, X, y=None):
+        """
+        Fit transformer, finding all principal components.
+
+        Parameters
+        ----------
+        X : nested pandas DataFrame of shape [n_samples, 1]
+            Nested dataframe with univariate time-series in cells.
+
+        Returns
+        -------
+        self : an instance of self.
+        """
+
+        validate_X(X)
+        check_X_is_univariate(X)
+
+        # Transform the time series column into tabular format and
+        # apply PCA to the tabular format
+        Xtab = tabularise(X)
+        self.pca.fit(Xtab)
+
+        return self
+
+    def transform(self, X, y=None):
+        """
+        Transform X, transforms univariate time-series using sklearn's PCA class
+
+        Parameters
+        ----------
+        X : nested pandas DataFrame of shape [n_samples, 1]
+            Nested dataframe with univariate time-series in cells.
+
+        Returns
+        -------
+        Xt : pandas DataFrame
+          Transformed pandas DataFrame with the same number of rows and the (potentially reduced) PCA transformed
+          column. Time indices of the original column are replaced with 0:(n_components - 1).
+        """
+
+        check_is_fitted(self.pca, 'n_components_')
+        validate_X(X)
+        check_X_is_univariate(X)
+
+        # Transform X using the fitted PCA
+        Xtab = tabularise(X)
+        Xpca = pd.DataFrame(data=self.pca.transform(Xtab),
+                            index=Xtab.index,
+                            columns=Xtab.columns[:self.pca.n_components_])
+
+        # Back-transform into time series data format
+        Xt = detabularise(Xpca, index=X.index)
+        Xt.columns = X.columns
+
+        return Xt
diff --git a/sktime/transformers/shapelets.py b/sktime/transformers/shapelets.py
--- a/sktime/transformers/shapelets.py
+++ b/sktime/transformers/shapelets.py
@@ -100,7 +100,7 @@ def fit(self, X, y=None):
         self : FullShapeletTransform
             This estimator
         """
-        if type(self) is ContractedShapeletTransform and self.time_limit <= 0:
+        if type(self) is ContractedShapeletTransform and self.time_limit_in_mins <= 0:
             raise ValueError("Error: time limit cannot be equal to or less than 0")
 
         X_lens = np.array([len(X.iloc[r,0]) for r in range(len(X))]) # note, assumes all dimensions of a case are the same length. A shapelet would not be well defined if indices do not match!
@@ -313,13 +313,13 @@ def _round_robin(*iterables):
                         shapelet_heaps_by_class[this_class_val].pop()
 
                 # Takes into account the use of the MAX shapelet calculation time to not exceed the time_limit (not exact, but likely a good guess).
-                if hasattr(self,'time_limit') and self.time_limit > 0:
+                if hasattr(self,'time_limit_in_mins') and self.time_limit_in_mins > 0:
                     time_now = time_taken()
                     time_this_shapelet = (time_now - time_last_shapelet)
                     if time_this_shapelet > max_time_calc_shapelet:
                         max_time_calc_shapelet = time_this_shapelet
                     time_last_shapelet = time_now
-                    if (time_now + max_time_calc_shapelet) > self.time_limit:
+                    if (time_now + max_time_calc_shapelet) > self.time_limit_in_mins * 60:
                         if self.verbose > 0:
                             print("No more time available! It's been {0:02d}:{1:02}".format(int(round(time_now / 60, 3)), int((round(time_now / 60, 3) - int(round(time_now / 60, 3))) * 60)))
                         time_finished = True
@@ -327,18 +327,18 @@ def _round_robin(*iterables):
                     else:
                         if self.verbose > 0:
                             if candidate_rejected is False:
-                                print("Candidate finished. {0:02d}:{1:02} remaining".format(int(round((self.time_limit - time_now) / 60, 3)),
-                                                                                            int((round((self.time_limit - time_now) / 60, 3) - int(round((self.time_limit - time_now) / 60, 3))) * 60)))
+                                print("Candidate finished. {0:02d}:{1:02} remaining".format(int(round(self.time_limit_in_mins - time_now / 60, 3)),
+                                                                                            int((round(self.time_limit_in_mins - time_now / 60, 3) - int(round(self.time_limit_in_mins - time_now / 60, 3))) * 60)))
                             else:
-                                print("Candidate rejected. {0:02d}:{1:02} remaining".format(int(round((self.time_limit - time_now) / 60, 3)),
-                                                                                            int((round((self.time_limit - time_now) / 60, 3) - int(round((self.time_limit - time_now) / 60, 3))) * 60)))
+                                print("Candidate rejected. {0:02d}:{1:02} remaining".format(int(round(self.time_limit_in_mins - time_now / 60, 3)),
+                                                                                            int((round(self.time_limit_in_mins - time_now / 60, 3) - int(round(self.time_limit_in_mins - time_now / 60, 3))) * 60)))
 
             # stopping condition: in case of iterative transform (i.e. num_cases_to_sample have been visited)
             #                     in case of contracted transform (i.e. time limit has been reached)
             case_idx += 1
 
             if case_idx >= num_series_to_visit:
-                if hasattr(self,'time_limit') and time_finished is not True:
+                if hasattr(self,'time_limit_in_mins') and time_finished is not True:
                     case_idx = 0
             elif case_idx >= num_series_to_visit or time_finished:
                 if self.verbose > 0:
@@ -676,18 +676,15 @@ def __init__(
             remove_self_similar = True
     ):
 
-        self.min_shapelet_length = min_shapelet_length
-        self.max_shapelet_length = max_shapelet_length
-        self.max_shapelets_to_store_per_class = max_shapelets_to_store_per_class
         self.num_candidates_to_sample_per_case = num_candidates_to_sample_per_case
-        self.time_limit = time_limit_in_mins*60
-        self.random_state = random_state
-        self.verbose = verbose
-        self.remove_self_similar = remove_self_similar
+        self.time_limit_in_mins = time_limit_in_mins
 
         self.predefined_ig_rejection_level = 0.05
         self.shapelets = None
 
+        super().__init__(min_shapelet_length, max_shapelet_length, max_shapelets_to_store_per_class, random_state,
+                         verbose, remove_self_similar)
+
 
 class RandomEnumerationShapeletTransform(ShapeletTransform):
     pass


Test patch:
diff --git a/sktime/transformers/tests/test_PCATransformer.py b/sktime/transformers/tests/test_PCATransformer.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformers/tests/test_PCATransformer.py
@@ -0,0 +1,101 @@
+from sklearn.decomposition import PCA
+from sklearn.exceptions import NotFittedError
+from sktime.transformers.pca import PCATransformer
+from sktime.utils.testing import generate_df_from_array
+from sktime.utils.data_container import tabularize, detabularize, get_time_index
+import pytest
+import pandas as pd
+import numpy as np
+
+
+# Check that exception is raised for bad input args.
+@pytest.mark.parametrize("bad_components", ['str', 1.2, -1.2, -1, 11])
+def test_bad_input_args(bad_components):
+    X = generate_df_from_array(np.ones(10), n_rows=10, n_cols=1)
+
+    if isinstance(bad_components, str):
+        with pytest.raises(TypeError):
+            PCATransformer(n_components=bad_components).fit(X)
+    else:
+        with pytest.raises(ValueError):
+            PCATransformer(n_components=bad_components).fit(X)
+
+
+# Test that keywords can be passed to PCA
+@pytest.mark.parametrize("kwargs", [{'copy': False},
+                                    {'whiten': True},
+                                    {'svd_solver': 'arpack'},
+                                    {'tol': 10e-6},
+                                    {'iterated_power': 10},
+                                    {'random_state': 42}])
+def test_pca_kwargs(kwargs):
+    np.random.seed(42)
+    X = detabularize(pd.DataFrame(data=np.random.randn(10, 5)))
+    pca = PCATransformer(n_components=1, **kwargs)
+    pca.fit_transform(X)
+
+
+# Test that PCATransformer fails if attempt to transform before fit
+def test_early_trans_fail():
+    X = generate_df_from_array(np.ones(10), n_rows=1, n_cols=1)
+    pca = PCATransformer(n_components=1)
+
+    with pytest.raises(NotFittedError):
+        pca.transform(X)
+
+
+# Test output format and dimensions.
+@pytest.mark.parametrize(
+    "n_instances,len_series,n_components", [
+        (5,  2, 1),
+        (5, 10, 1),
+        (5, 10, 3),
+        (5, 10, 5),
+    ])
+def test_output_format_dim(len_series, n_instances, n_components):
+    np.random.seed(42)
+    X = detabularize(pd.DataFrame(data=np.random.randn(n_instances, len_series)))
+
+    trans = PCATransformer(n_components=n_components)
+    Xt = trans.fit_transform(X)
+
+    # Check number of rows and output type.
+    assert isinstance(Xt, pd.DataFrame)
+    assert Xt.shape[0] == X.shape[0]
+
+    # Check number of principal components in the output.
+    assert tabularize(Xt).shape[1] == min(n_components, tabularize(X).shape[1])
+
+
+# Check that the returned values agree with those produced by ``sklearn.decomposition.PCA``
+@pytest.mark.parametrize("n_components", [1, 5, 0.9, 'mle'])
+def test_pca_results(n_components):
+    np.random.seed(42)
+
+    # sklearn
+    X = pd.DataFrame(data=np.random.randn(10, 5))
+    pca = PCA(n_components=n_components)
+    Xt1 = pca.fit_transform(X)
+
+    # sktime
+    Xs = detabularize(X)
+    pca_transform = PCATransformer(n_components=n_components)
+    Xt2 = pca_transform.fit_transform(Xs)
+
+    assert np.allclose(np.asarray(Xt1), np.asarray(tabularize(Xt2)))
+
+
+# Check output indices (row indices and columns the same, time indices start from 0)
+@pytest.mark.parametrize("n_components", [1, 5, None])
+def test_indices(n_components):
+    np.random.seed(42)
+    X = detabularize(pd.DataFrame(data=np.random.randn(10, 5)))
+    X.columns = pd.CategoricalIndex(['col_0'])
+    X.index = pd.Int64Index([i+10 for i in range(10)])
+
+    pca = PCATransformer(n_components=n_components)
+    Xt = pca.fit_transform(X)
+
+    assert X.columns.equals(Xt.columns)
+    assert X.index.equals(Xt.index)
+    assert get_time_index(Xt).equals(pd.Int64Index(range(pca.pca.n_components_)))