Repository: sktime/sktime

Issue:
[BUG] sporadic bug in time series forest? manylinux builds sporadically failing
The `manylinux` builds are sporadically failing with a `numpy` casting error (dimension 1 vs 2) in `SupervisedTimeSeriesForest`.

I suspect this is a sporadic bug due to randomization and incomplete seed-setting in the classifier, e.g., a partition becomes size 1 or 0 or similar.

I haven't made a screenshot of the error because the build restarted, but we should watch out for it appearing again.


Discussion:
Here it is again, got it! Elusive critter...

[pytest_log.txt](https://github.com/alan-turing-institute/sktime/files/6717330/pytest_log.txt)


Have a look at this, @TonyBagnall - what is this? Interesting stuff starts at line 5205:

```
2021-06-25T14:07:24.8077061Z =================================== FAILURES ===================================
2021-06-25T14:07:24.8078353Z ______________ test_classifier_output[SupervisedTimeSeriesForest] ______________
2021-06-25T14:07:24.8079203Z 
2021-06-25T14:07:24.8081314Z Estimator = <class 'sktime.classification.interval_based._stsf.SupervisedTimeSeriesForest'>
2021-06-25T14:07:24.8082268Z 
2021-06-25T14:07:24.8082945Z     @pytest.mark.parametrize("Estimator", CLASSIFIERS)
2021-06-25T14:07:24.8083710Z     def test_classifier_output(Estimator):
2021-06-25T14:07:24.8084470Z         estimator = _construct_instance(Estimator)
2021-06-25T14:07:24.8085245Z         X_train, y_train = _make_args(estimator, "fit", n_classes=N_CLASSES)
2021-06-25T14:07:24.8086022Z         estimator.fit(X_train, y_train)
2021-06-25T14:07:24.8086667Z     
2021-06-25T14:07:24.8087318Z         X_new = _make_args(estimator, "predict")[0]
2021-06-25T14:07:24.8088187Z     
2021-06-25T14:07:24.8088789Z         # check predict
2021-06-25T14:07:24.8089464Z >       y_pred = estimator.predict(X_new)
2021-06-25T14:07:24.8090002Z 
2021-06-25T14:07:24.8091034Z Estimator  = <class 'sktime.classification.interval_based._stsf.SupervisedTimeSeriesForest'>
2021-06-25T14:07:24.8091902Z X_new      =                                                 var_0
2021-06-25T14:07:24.8092807Z 0   0    -0.733648
2021-06-25T14:07:24.8093461Z 1     0.320080
2021-06-25T14:07:24.8094222Z 2    -0.010786
2021-06-25T14:07:24.8095656Z 3...
2021-06-25T14:07:24.8098107Z 1   0     ...602657
2021-06-25T14:07:24.8099216Z 3...
2021-06-25T14:07:24.8099942Z 18  0     0.357356
2021-06-25T14:07:24.8100790Z 1    -0.075322
2021-06-25T14:07:24.8101210Z 2     0.060190
2021-06-25T14:07:24.8101595Z 3...
2021-06-25T14:07:24.8101973Z 19  0     0.790637
2021-06-25T14:07:24.8102540Z 1    -0.447664
2021-06-25T14:07:24.8102944Z 2     0.813162
2021-06-25T14:07:24.8103329Z 3...
2021-06-25T14:07:24.8103797Z X_train    =                                                 var_0
2021-06-25T14:07:24.8104304Z 0   0     100.003351
2021-06-25T14:07:24.8104709Z 1     100.983767
2021-06-25T14:07:24.8105119Z 2      99.46...
2021-06-25T14:07:24.8105541Z 1   0     ...  200.65...
2021-06-25T14:07:24.8105955Z 18  0     199.341296
2021-06-25T14:07:24.8106559Z 1     199.236967
2021-06-25T14:07:24.8106952Z 2     200.98...
2021-06-25T14:07:24.8107368Z 19  0     200.032719
2021-06-25T14:07:24.8107763Z 1     199.115088
2021-06-25T14:07:24.8108168Z 2     199.94...
2021-06-25T14:07:24.8108716Z estimator  = SupervisedTimeSeriesForest(n_estimators=3, n_jobs=1, random_state=None)
2021-06-25T14:07:24.8109262Z y_train    = 0     1
2021-06-25T14:07:24.8109642Z 1     0
2021-06-25T14:07:24.8110021Z 2     2
2021-06-25T14:07:24.8111516Z 3     0
2021-06-25T14:07:24.8112117Z 4     0
2021-06-25T14:07:24.8112553Z 5     1
2021-06-25T14:07:24.8113040Z 6     0
2021-06-25T14:07:24.8113389Z 7     0
2021-06-25T14:07:24.8113722Z 8     1
2021-06-25T14:07:24.8114077Z 9     1
2021-06-25T14:07:24.8114411Z 10    2
2021-06-25T14:07:24.8114931Z 11    0
2021-06-25T14:07:24.8115292Z 12    1
2021-06-25T14:07:24.8115669Z 13    1
2021-06-25T14:07:24.8116029Z 14    0
2021-06-25T14:07:24.8116404Z 15    2
2021-06-25T14:07:24.8116761Z 16    1
2021-06-25T14:07:24.8117139Z 17    2
2021-06-25T14:07:24.8117502Z 18    2
2021-06-25T14:07:24.8117876Z 19    2
2021-06-25T14:07:24.8118247Z dtype: int64
2021-06-25T14:07:24.8118517Z 
2021-06-25T14:07:24.8119492Z /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/sktime/series_as_features/tests/test_all_panel_estimators.py:105: 
2021-06-25T14:07:24.8120492Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2021-06-25T14:07:24.8121494Z /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/sktime/classification/interval_based/_stsf.py:186: in predict
2021-06-25T14:07:24.8122140Z     proba = self.predict_proba(X)
2021-06-25T14:07:24.8122683Z         X          =                                                 var_0
2021-06-25T14:07:24.8123386Z 0   0    -0.733648
2021-06-25T14:07:24.8123821Z 1     0.320080
2021-06-25T14:07:24.8124354Z 2    -0.010786
2021-06-25T14:07:24.8124760Z 3...
2021-06-25T14:07:24.8125151Z 1   0     ...602657
2021-06-25T14:07:24.8125527Z 3...
2021-06-25T14:07:24.8125921Z 18  0     0.357356
2021-06-25T14:07:24.8126473Z 1    -0.075322
2021-06-25T14:07:24.8126892Z 2     0.060190
2021-06-25T14:07:24.8127258Z 3...
2021-06-25T14:07:24.8127648Z 19  0     0.790637
2021-06-25T14:07:24.8128188Z 1    -0.447664
2021-06-25T14:07:24.8128606Z 2     0.813162
2021-06-25T14:07:24.8128971Z 3...
2021-06-25T14:07:24.8129494Z         self       = SupervisedTimeSeriesForest(n_estimators=3, n_jobs=1, random_state=None)
2021-06-25T14:07:24.8130494Z /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/sktime/classification/interval_based/_stsf.py:225: in predict_proba
2021-06-25T14:07:24.8131175Z     output = np.sum(y_probas, axis=0) / (
2021-06-25T14:07:24.8131968Z         X          = array([[-7.33648398e-01,  3.20079950e-01, -1.07861350e-02,
2021-06-25T14:07:24.8132933Z          7.45276023e-01, -6.93378748e-01,  6.48461235e-01,
2021-06-25T14:07:24.8133600Z ...-9.79251532e-02,
2021-06-25T14:07:24.8134295Z         -6.69767423e-01, -4.00957689e-01, -3.07072737e-01,
2021-06-25T14:07:24.8135033Z         -7.63543500e-01, -1.75539009e-02]])
2021-06-25T14:07:24.8135892Z         X_d        = array([[ 1.05372835, -0.33086609,  0.75606216, -1.43865477,  1.34183998,
2021-06-25T14:07:24.8136862Z         -0.50037525, -0.96176866,  0.18642599...296 , -0.0172857 ,  0.91812058, -0.44719371, -0.57184227,
2021-06-25T14:07:24.8137736Z          0.26880973,  0.09388495, -0.45647076,  0.7459896 ]])
2021-06-25T14:07:24.8138602Z         X_p        = array([[2.21867130e-32, 3.58362521e-01, 7.12498905e-01, 1.69109286e-01,
2021-06-25T14:07:24.8139621Z         6.79661669e-02, 1.69078505e+00, 2.1669...44120888e-01, 2.12991418e-01, 2.42178657e-01, 1.23594571e+00,
2021-06-25T14:07:24.8140496Z         6.28137094e-01, 1.23545256e-01, 7.88665624e-02]])
2021-06-25T14:07:24.8141122Z         _          = array([0.  , 0.05, 0.1 , 0.15, 0.2 , 0.25, 0.3 , 0.35, 0.4 , 0.45, 0.5 ])
2021-06-25T14:07:24.8141799Z         self       = SupervisedTimeSeriesForest(n_estimators=3, n_jobs=1, random_state=None)
2021-06-25T14:07:24.8142368Z         y_probas   = [array([[1., 0.],
2021-06-25T14:07:24.8142914Z        [1., 0.],
2021-06-25T14:07:24.8143324Z        [1., 0.],
2021-06-25T14:07:24.8143807Z        [1., 0.],
2021-06-25T14:07:24.8144279Z        [1., 0.],
2021-06-25T14:07:24.8144648Z        [1., 0.],
2021-06-25T14:07:24.8145148Z        [1., 0.]...1., 0., 0.],
2021-06-25T14:07:24.8145590Z        [1., 0., 0.],
2021-06-25T14:07:24.8145885Z        [1., 0., 0.],
2021-06-25T14:07:24.8146166Z        [1., 0., 0.],
2021-06-25T14:07:24.8146460Z        [1., 0., 0.],
2021-06-25T14:07:24.8146942Z        [1., 0., 0.]])]
2021-06-25T14:07:24.8147465Z <__array_function__ internals>:6: in sum
2021-06-25T14:07:24.8147823Z     ???
2021-06-25T14:07:24.8148173Z         args       = ([array([[1., 0.],
2021-06-25T14:07:24.8148542Z        [1., 0.],
2021-06-25T14:07:24.8148857Z        [1., 0.],
2021-06-25T14:07:24.8149578Z        [1., 0.],
2021-06-25T14:07:24.8149968Z        [1., 0.],
2021-06-25T14:07:24.8150369Z        [1., 0.],
2021-06-25T14:07:24.8150776Z        [1., 0...., 0., 0.],
2021-06-25T14:07:24.8151215Z        [1., 0., 0.],
2021-06-25T14:07:24.8151620Z        [1., 0., 0.],
2021-06-25T14:07:24.8152063Z        [1., 0., 0.],
2021-06-25T14:07:24.8152467Z        [1., 0., 0.],
2021-06-25T14:07:24.8152893Z        [1., 0., 0.]])],)
2021-06-25T14:07:24.8153560Z         kwargs     = {'axis': 0}
2021-06-25T14:07:24.8154084Z         relevant_args = ([array([[1., 0.],
2021-06-25T14:07:24.8154536Z        [1., 0.],
2021-06-25T14:07:24.8154926Z        [1., 0.],
2021-06-25T14:07:24.8155331Z        [1., 0.],
2021-06-25T14:07:24.8155719Z        [1., 0.],
2021-06-25T14:07:24.8156124Z        [1., 0.],
2021-06-25T14:07:24.8156523Z        [1., 0.... 0.],
2021-06-25T14:07:24.8156979Z        [1., 0., 0.],
2021-06-25T14:07:24.8157387Z        [1., 0., 0.],
2021-06-25T14:07:24.8157806Z        [1., 0., 0.],
2021-06-25T14:07:24.8158202Z        [1., 0., 0.],
2021-06-25T14:07:24.8158639Z        [1., 0., 0.]])], None)
2021-06-25T14:07:24.8159474Z /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2242: in sum
2021-06-25T14:07:24.8160237Z     initial=initial, where=where)
2021-06-25T14:07:24.8160714Z         a          = [array([[1., 0.],
2021-06-25T14:07:24.8161145Z        [1., 0.],
2021-06-25T14:07:24.8161552Z        [1., 0.],
2021-06-25T14:07:24.8161943Z        [1., 0.],
2021-06-25T14:07:24.8162351Z        [1., 0.],
2021-06-25T14:07:24.8162739Z        [1., 0.],
2021-06-25T14:07:24.8163167Z        [1., 0.]...1., 0., 0.],
2021-06-25T14:07:24.8163589Z        [1., 0., 0.],
2021-06-25T14:07:24.8164011Z        [1., 0., 0.],
2021-06-25T14:07:24.8164429Z        [1., 0., 0.],
2021-06-25T14:07:24.8164854Z        [1., 0., 0.],
2021-06-25T14:07:24.8165278Z        [1., 0., 0.]])]
2021-06-25T14:07:24.8165848Z         axis       = 0
2021-06-25T14:07:24.8166306Z         dtype      = None
2021-06-25T14:07:24.8166742Z         initial    = <no value>
2021-06-25T14:07:24.8167204Z         keepdims   = <no value>
2021-06-25T14:07:24.8167638Z         out        = None
2021-06-25T14:07:24.8168084Z         where      = <no value>
2021-06-25T14:07:24.8168615Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2021-06-25T14:07:24.8169004Z 
2021-06-25T14:07:24.8169381Z obj = [array([[1., 0.],
2021-06-25T14:07:24.8169803Z        [1., 0.],
2021-06-25T14:07:24.8170190Z        [1., 0.],
2021-06-25T14:07:24.8170595Z        [1., 0.],
2021-06-25T14:07:24.8170996Z        [1., 0.],
2021-06-25T14:07:24.8171389Z        [1., 0.],
2021-06-25T14:07:24.8171817Z        [1., 0.]...1., 0., 0.],
2021-06-25T14:07:24.8172242Z        [1., 0., 0.],
2021-06-25T14:07:24.8172663Z        [1., 0., 0.],
2021-06-25T14:07:24.8173067Z        [1., 0., 0.],
2021-06-25T14:07:24.8173797Z        [1., 0., 0.],
2021-06-25T14:07:24.8174317Z        [1., 0., 0.]])]
2021-06-25T14:07:24.8175933Z ufunc = <ufunc 'add'>, method = 'sum', axis = 0, dtype = None, out = None
2021-06-25T14:07:24.8176829Z kwargs = {'initial': <no value>, 'keepdims': <no value>, 'where': <no value>}
2021-06-25T14:07:24.8177413Z passkwargs = {}
2021-06-25T14:07:24.8177797Z 
2021-06-25T14:07:24.8178328Z     def _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs):
2021-06-25T14:07:24.8178971Z         passkwargs = {k: v for k, v in kwargs.items()
2021-06-25T14:07:24.8179517Z                       if v is not np._NoValue}
2021-06-25T14:07:24.8179959Z     
2021-06-25T14:07:24.8180396Z         if type(obj) is not mu.ndarray:
2021-06-25T14:07:24.8180880Z             try:
2021-06-25T14:07:24.8181338Z                 reduction = getattr(obj, method)
2021-06-25T14:07:24.8181862Z             except AttributeError:
2021-06-25T14:07:24.8182316Z                 pass
2021-06-25T14:07:24.8182745Z             else:
2021-06-25T14:07:24.8183479Z                 # This branch is needed for reductions like any which don't
2021-06-25T14:07:24.8184057Z                 # support a dtype.
2021-06-25T14:07:24.8184557Z                 if dtype is not None:
2021-06-25T14:07:24.8185126Z                     return reduction(axis=axis, dtype=dtype, out=out, **passkwargs)
2021-06-25T14:07:24.8185683Z                 else:
2021-06-25T14:07:24.8186192Z                     return reduction(axis=axis, out=out, **passkwargs)
2021-06-25T14:07:24.8186679Z     
2021-06-25T14:07:24.8187137Z >       return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
2021-06-25T14:07:24.8187781Z E       ValueError: could not broadcast input array from shape (20,2) into shape (20)
2021-06-25T14:07:24.8188168Z 
2021-06-25T14:07:24.8188548Z axis       = 0
2021-06-25T14:07:24.8188941Z dtype      = None
2021-06-25T14:07:24.8189733Z kwargs     = {'initial': <no value>, 'keepdims': <no value>, 'where': <no value>}
2021-06-25T14:07:24.8190458Z method     = 'sum'
2021-06-25T14:07:24.8190920Z obj        = [array([[1., 0.],
2021-06-25T14:07:24.8191364Z        [1., 0.],
2021-06-25T14:07:24.8191759Z        [1., 0.],
2021-06-25T14:07:24.8192184Z        [1., 0.],
2021-06-25T14:07:24.8192575Z        [1., 0.],
2021-06-25T14:07:24.8192980Z        [1., 0.],
2021-06-25T14:07:24.8193392Z        [1., 0.]...1., 0., 0.],
2021-06-25T14:07:24.8193838Z        [1., 0., 0.],
2021-06-25T14:07:24.8194248Z        [1., 0., 0.],
2021-06-25T14:07:24.8194668Z        [1., 0., 0.],
2021-06-25T14:07:24.8195072Z        [1., 0., 0.],
2021-06-25T14:07:24.8195499Z        [1., 0., 0.]])]
2021-06-25T14:07:24.8195954Z out        = None
2021-06-25T14:07:24.8196347Z passkwargs = {}
2021-06-25T14:07:24.8196940Z ufunc      = <ufunc 'add'>
2021-06-25T14:07:24.8197230Z 
2021-06-25T14:07:24.8197992Z /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/numpy/core/fromnumeric.py:87: ValueError
2021-06-25T14:07:24.8198694Z =============================== warnings summary ===============================
2021-06-25T14:07:24.8199307Z forecasting/fbprophet.py::sktime.forecasting.fbprophet.Prophet
2021-06-25T14:07:24.8200789Z   /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/Cython/Build/Inline.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
2021-06-25T14:07:24.8201717Z     import imp
2021-06-25T14:07:24.8201959Z 
2021-06-25T14:07:24.8202385Z forecasting/tests/test_all_forecasters.py: 47 warnings
2021-06-25T14:07:24.8202900Z tests/test_all_estimators.py: 2 warnings
2021-06-25T14:07:24.8204088Z   /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/statsmodels/tsa/holtwinters/model.py:429: FutureWarning: After 0.13 initialization must be handled at model creation
2021-06-25T14:07:24.8204825Z     FutureWarning,
2021-06-25T14:07:24.8205082Z 
2021-06-25T14:07:24.8205538Z forecasting/tests/test_all_forecasters.py: 12 warnings
2021-06-25T14:07:24.8206917Z   /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py:383: DeprecationWarning: No supported index is available. In the next version, calling this method in a model without a supported index will result in an exception.
2021-06-25T14:07:24.8208168Z     ' exception.', DeprecationWarning)
2021-06-25T14:07:24.8208495Z 
2021-06-25T14:07:24.8208991Z transformations/tests/test_all_transformers.py::test_all_transformers[HampelFilter]
2021-06-25T14:07:24.8210147Z   /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages/numpy/lib/nanfunctions.py:1114: RuntimeWarning: All-NaN slice encountered
2021-06-25T14:07:24.8210830Z     overwrite_input=overwrite_input)
2021-06-25T14:07:24.8211139Z 
2021-06-25T14:07:24.8211776Z -- Docs: https://docs.pytest.org/en/stable/warnings.html
2021-06-25T14:07:24.8212638Z ---------------- generated xml file: /io/junit/test-results.xml ----------------
2021-06-25T14:07:24.8213043Z 
2021-06-25T14:07:24.8213744Z ---------- coverage: platform linux, python 3.6.13-final-0 -----------
2021-06-25T14:07:24.8214303Z Coverage HTML written to dir htmlcov
2021-06-25T14:07:24.8214794Z Coverage XML written to file coverage.xml
2021-06-25T14:07:24.8215094Z 
2021-06-25T14:07:24.8238868Z Coverage.py warning: No data was collected. (no-data-collected)
```


hi @fkiraly what code generates this so I can try reproduce it? 
`test_classifier_output(SupervisedTimeSeriesForest, some_args)`, as the second line tells you, you have to fill in `some_args` from the `mark.parametrize` decorator in the test file.
@MatthewMiddlehurst could you take a look, think this is yours? 
FYI, it's sporadic - feels like you have to run it 100 times or so to get the error.
Yeah this is mine, I tried to reproduce it before but couldn't. I will look at the code and try again at some point, even though running it 100 times to find out whats changing sounds painful :)
well, what you have to do is repeat and log or repeat and break imo

Resolving patch:
diff --git a/sktime/classification/interval_based/_stsf.py b/sktime/classification/interval_based/_stsf.py
--- a/sktime/classification/interval_based/_stsf.py
+++ b/sktime/classification/interval_based/_stsf.py
@@ -77,7 +77,7 @@ class SupervisedTimeSeriesForest(ForestClassifier, BaseClassifier):
 
     def __init__(
         self,
-        n_estimators=500,
+        n_estimators=200,
         n_jobs=1,
         random_state=None,
     ):
@@ -94,6 +94,7 @@ def __init__(
 
         # The following set in method fit
         self.n_classes = 0
+        self.n_instances = 0
         self.estimators_ = []
         self.intervals_ = []
         self.classes_ = []
@@ -127,7 +128,7 @@ def fit(self, X, y):
             coerce_to_numpy=True,
         )
         X = X.squeeze(1)
-        n_instances, _ = X.shape
+        self.n_instances, _ = X.shape
 
         rng = check_random_state(self.random_state)
 
@@ -141,7 +142,7 @@ def fit(self, X, y):
         X_d = np.diff(X, 1)
 
         balance_cases = np.zeros(0, dtype=np.int32)
-        average = math.floor(n_instances / self.n_classes)
+        average = math.floor(self.n_instances / self.n_classes)
         for i, c in enumerate(cls):
             if class_counts[i] < average:
                 cls_idx = np.where(y == c)[0]
@@ -155,9 +156,7 @@ def fit(self, X, y):
                 X_p,
                 X_d,
                 y,
-                np.concatenate(
-                    (rng.choice(n_instances, size=n_instances), balance_cases)
-                ),
+                balance_cases,
                 i,
             )
             for i in range(self.n_estimators)
@@ -183,8 +182,13 @@ def predict(self, X):
         -------
         output : array of shape = [n_test_instances]
         """
-        proba = self.predict_proba(X)
-        return np.asarray([self.classes_[np.argmax(prob)] for prob in proba])
+        rng = check_random_state(self.random_state)
+        return np.array(
+            [
+                self.classes_[int(rng.choice(np.flatnonzero(prob == prob.max())))]
+                for prob in self.predict_proba(X)
+            ]
+        )
 
     def predict_proba(self, X):
         """Find probability estimates for each class for all cases in X.
@@ -338,20 +342,26 @@ def _supervised_interval_search(
                 end,
             )
 
-    def _fit_estimator(self, X, X_p, X_d, y, bag, idx):
+    def _fit_estimator(self, X, X_p, X_d, y, balance_cases, idx):
         """Fit an estimator - a clone of base_estimator - on input data (X, y).
 
         Transformed using the supervised intervals for each feature and representation.
         """
-        n_instances = bag.shape[0]
-        bag = bag.astype(int)
-
         estimator = clone(self.base_estimator)
         rs = 5465 if self.random_state == 0 else self.random_state
         rs = None if self.random_state is None else rs * 37 * (idx + 1)
         estimator.set_params(random_state=rs)
-
         rng = check_random_state(rs)
+
+        class_counts = np.zeros(0)
+        while class_counts.shape[0] != self.n_classes:
+            bag = np.concatenate(
+                (rng.choice(self.n_instances, size=self.n_instances), balance_cases)
+            )
+            _, class_counts = np.unique(y[bag], return_counts=True)
+        n_instances = bag.shape[0]
+        bag = bag.astype(int)
+
         transformed_x = np.zeros((n_instances, 0), dtype=np.float32)
 
         intervals = self._get_intervals(X[bag], y[bag], rng)


Test patch:
diff --git a/sktime/classification/interval_based/tests/test_stsf.py b/sktime/classification/interval_based/tests/test_stsf.py
--- a/sktime/classification/interval_based/tests/test_stsf.py
+++ b/sktime/classification/interval_based/tests/test_stsf.py
@@ -1,23 +1,29 @@
 # -*- coding: utf-8 -*-
+"""STSF test code."""
 import numpy as np
-from sklearn.model_selection import train_test_split
+from numpy import testing
 
 from sktime.classification.interval_based import SupervisedTimeSeriesForest
 from sktime.datasets import load_gunpoint, load_italy_power_demand
 
 
-def test_y_proba_on_gunpoint():
-    X, y = load_gunpoint(return_X_y=True)
-    X_train, X_test, y_train, y_test = train_test_split(
-        X, y, test_size=0.1, random_state=42
-    )
-    stsf = SupervisedTimeSeriesForest(random_state=42, n_estimators=20)
-    stsf.fit(X_train, y_train)
-    actual = stsf.predict_proba(X_test)
-    np.testing.assert_array_equal(actual, expected)
+def test_stsf_on_gunpoint():
+    """Test of STSF on gun point."""
+    # load gunpoint data
+    X_train, y_train = load_gunpoint(split="train", return_X_y=True)
+    X_test, y_test = load_gunpoint(split="test", return_X_y=True)
+    indices = np.random.RandomState(0).permutation(10)
+
+    stsf = SupervisedTimeSeriesForest(n_estimators=20, random_state=0)
+    stsf.fit(X_train.iloc[indices], y_train[indices])
+
+    # assert probabilities are the same
+    probas = stsf.predict_proba(X_test.iloc[indices])
+    testing.assert_array_equal(probas, stsf_gunpoint_probas)
 
 
 def test_stsf_on_power_demand():
+    """Test of STSF on italy power demand."""
     # load power demand data
     X_train, y_train = load_italy_power_demand(split="train", return_X_y=True)
     X_test, y_test = load_italy_power_demand(split="test", return_X_y=True)
@@ -31,88 +37,47 @@ def test_stsf_on_power_demand():
     assert score >= 0.92
 
 
-# expected y_proba
-expected = np.array(
+stsf_gunpoint_probas = np.array(
     [
         [
-            0.95,
-            0.05,
-        ],
-        [
-            1.0,
-            0.0,
-        ],
-        [
-            0.95,
-            0.05,
-        ],
-        [
-            0.9,
-            0.1,
-        ],
-        [
-            0.0,
-            1.0,
-        ],
-        [
-            0.9,
-            0.1,
+            0.15,
+            0.85,
         ],
         [
-            0.05,
-            0.95,
+            0.65,
+            0.35,
         ],
         [
-            1.0,
-            0.0,
+            0.7,
+            0.3,
         ],
         [
-            1.0,
-            0.0,
+            0.3,
+            0.7,
         ],
         [
             0.2,
             0.8,
         ],
         [
-            0.85,
-            0.15,
-        ],
-        [
-            1.0,
-            0.0,
-        ],
-        [
-            1.0,
-            0.0,
-        ],
-        [
-            0.15,
-            0.85,
-        ],
-        [
-            1.0,
-            0.0,
-        ],
-        [
-            0.8,
-            0.2,
+            0.55,
+            0.45,
         ],
         [
-            1.0,
-            0.0,
+            0.35,
+            0.65,
         ],
         [
-            0.95,
-            0.05,
+            0.5,
+            0.5,
         ],
         [
-            0.05,
-            0.95,
+            0.4,
+            0.6,
         ],
         [
-            0.0,
-            1.0,
+            0.35,
+            0.65,
         ],
     ]
 )
@@ -130,11 +95,11 @@ def test_stsf_on_power_demand():
 #
 #
 # if __name__ == "__main__":
-#     X, y = load_gunpoint(return_X_y=True)
-#     X_train, X_test, y_train, y_test = train_test_split(
-#         X, y, test_size=0.1, random_state=42
-#     )
-#     estimator = SupervisedTimeSeriesForest(random_state=42, n_estimators=20)
-#     estimator.fit(X_train, y_train)
-#     probas = estimator.predict_proba(X_test)
+#     X_train, y_train = load_gunpoint(split="train", return_X_y=True)
+#     X_test, y_test = load_gunpoint(split="test", return_X_y=True)
+#     indices = np.random.RandomState(0).permutation(10)
+#
+#     stsf = SupervisedTimeSeriesForest(n_estimators=20, random_state=0)
+#     stsf.fit(X_train.iloc[indices], y_train[indices])
+#     probas = stsf.predict_proba(X_test.iloc[indices])
 #     print_array(probas)