Repository: sktime/sktime

Issue:
[BUG] LSTM deep learning estimators failing CI on windows
Since recently, a failure of two deep learning estimators has been appearing on windows CI:
```
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[MLPClassifier-1-ClassifierFitPredictMultivariate-predict]
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[LSTMFCNClassifier-0-ClassifierFitPredictMultivariate-predict]
```

Note that the failure appears to be only on python 3.9, but this is due to the matrix design which spreads estimators across version/OS combinations.

In theory, it could be a 3.9 specific failure, but I think that is less likely than windows specifity (although it may be worth to test that by turning the `matrixdesign` flag off in the CI).


Discussion:
This does not seem clearly related to any recent PR merge to `main`.

@achieveordie, @AurumnPegasus, @jnrusson1, @solen0id, do you have a clue what is happening here?
traceback is not very informative:

```
 ================================== FAILURES ===================================
_____________________ sktime/tests/test_all_estimators.py _____________________
[gw0] win32 -- Python 3.9.12 C:\Miniconda\envs\test\python.exe
worker 'gw0' crashed while running 'sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[MLPClassifier-1-ClassifierFitPredictMultivariate-predict]'
_____________________ sktime/tests/test_all_estimators.py _____________________
[gw1] win32 -- Python 3.9.12 C:\Miniconda\envs\test\python.exe
worker 'gw1' crashed while running 'sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[LSTMFCNClassifier-0-ClassifierFitPredictMultivariate-predict]'
```
`LSTMFCN` was added last week, right? I suspect the problem is arising from that estimator.

We have seen similar problems back when `TapNet` was ported, `OutOfMemory/MemoryError` error arising from one worker can crash the entire test suite. Hence we see `MLPClassifier` failing, just because it happened to be tested on the same system.

The reason I think `LSTMFCN` is causing the failures is that its [`get_test_params()`](https://github.com/sktime/sktime/blob/main/sktime/classification/deep_learning/lstmfcn.py#L206) currently only reduces the number of epochs, the size of the model remains the same.

Let me take this up and diagnose if there are any more problems in it.

Resolving patch:
diff --git a/sktime/classification/deep_learning/lstmfcn.py b/sktime/classification/deep_learning/lstmfcn.py
--- a/sktime/classification/deep_learning/lstmfcn.py
+++ b/sktime/classification/deep_learning/lstmfcn.py
@@ -70,7 +70,7 @@ class LSTMFCNClassifier(BaseDeepClassifier):
 
     def __init__(
         self,
-        n_epochs=100,
+        n_epochs=2000,
         batch_size=128,
         dropout=0.8,
         kernel_sizes=(8, 5, 3),
@@ -84,7 +84,6 @@ def __init__(
 
         super(LSTMFCNClassifier, self).__init__()
 
-        # calced in fit
         self.classes_ = None
         self.input_shape = None
         self.model_ = None
@@ -232,11 +231,19 @@ def get_test_params(cls, parameter_set="default"):
             `create_test_instance` uses the first (or only) dictionary in `params`.
         """
         param1 = {
-            "n_epochs": 50,
+            "n_epochs": 25,
+            "batch_size": 4,
+            "kernel_sizes": (3, 2, 1),
+            "filter_sizes": (2, 4, 2),
         }
 
         param2 = {
-            "n_epochs": 100,
+            "n_epochs": 5,
+            "batch_size": 2,
+            "kernel_sizes": (3, 2, 1),
+            "filter_sizes": (2, 4, 2),
+            "lstm_size": 2,
+            "attention": True,
         }
 
         return [param1, param2]


Test patch:
diff --git a/sktime/tests/_config.py b/sktime/tests/_config.py
--- a/sktime/tests/_config.py
+++ b/sktime/tests/_config.py
@@ -94,7 +94,6 @@
     ],
     "MLPClassifier": [
         "test_fit_idempotent",
-        "test_methods_have_no_side_effects",  # unknown cause, see bug report #4033
     ],
     # sth is not quite right with the RowTransformer-s changing state,
     #   but these are anyway on their path to deprecation, see #2370