Repository: sktime/sktime

Issue:
[BUG] `CNNRegressor` missing essential initializing parameters
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
`CNNRegressor` misses parameters like `optimizer` in its constructor. There might be more things which don't coincide with `CNNClassifier` which should be added but since I'm not sure what all things might be missing, I'm opening this issue to bring it to notice.



Discussion:
apart from interface homogeneity, are there other reasons to have these parameters?

does it break the save/load interface of neural networks?
There are three reasons we should have an `optimizer` parameter.
1. save/load interface tries to find and load it, so it does break. We could modify load/save such that it doesn't fail when optimizer is not found but I'd suggest keeping that as it is if we consider points 2 and 3 below.
2. Interface Homogeneity.
3. More flexibility for the user.
I don't undestand what you are saying. Can you please explain and be more precise here? What object is the problem with?

How is that related to having an `optimizer` parameter. And what object is "we" that should have an `optimizer` parameter?
Ah, I must have made that comment in a rush, leading to some confusion. Let me start over.

If you compare `CNNClassifier` and `CNNRegressor`, you'd see that parameters like `optimizer` is not present in the latter. On the surface, this implies that the current interface of `CNNClassifier` and `CNNRegressor` is not homogenous and that the user doesn't have the freedom to choose a different optimizer for `CNNRegressor`, they are stuck with the default one. This is what point 2 and 3 was meant to convey.

IMO, these two reasons together warrant us to add an `optimizer` parameter (and any other that may be missing). But there's an additional reason. Save/load assumes that an `optimizer` parameter exists for all DL estimators even if it is `None` (see `__setstate__()` for reference). We probably could modify it to not complain, but then bugs like this might go unnoticed.

I hope things are more clear now.

not sure, still.
Could you please outline precisely, (explanation or a statement that you aren't entirely clear about it)

1. the precise property that you think `CNNRegressor` does not have and should have
2. the reason why it should have that property, e.g., to confirm with an interface requirement
3. how that relates to the test failure in in #3425, e.g., interface requirement tested by a spefici test?
> Save/load assumes that an `optimizer` parameter exists for all DL estimators even if it is `None` (see `__setstate__()` for reference).

Is that an interface assumption we want to make for all deep learning estimators?

This might not be a good idea, since some deep learning estimators might hard-code the optimizer or not have a parameter like that.
> the precise property that you think `CNNRegressor` does not have and should have

`CNNRegressor` should have an `optimizer` parameter in its constructor; even if it's `None`.

> the reason why it should have that property

It gives the user more flexibility to choose which optimizer to use for training and not being stuck with `Adam`.

> how that relates to the test failure

save/load interface expects `optimizer` to be present in it's constructor. I could change it to not complain when not found but the downside would be bugs like this would go unnoticed (at least in the case of `CNNRegressor` it is certainly undesirable to not have an `optimizer`).

Even though this issue is related to `CNNRegressor`, could you give me an example of a DL estimator that doesn't require an optimizer AND is also supported by the current interface that `sktime` provides? I'm failing to see those cases.
Hey @achieveordie ,
So the reason for difference in `CNNClassifier` was because of the issue/PR at #2935
I had added some parameters which we could add and give user more chances for customisations.
I started of with classifiers and did not do it for regressors as my plan was to finish the migration of classifiers first, but since the migration looks at a good place currently, either me or you can create an issue/pr based on #2935  and update the structure of `CNNRegressor` as well 
I see, thank you. So let me take up this issue. Do you suggest more changes that I should incorporate?

Resolving patch:
diff --git a/sktime/regression/deep_learning/cnn.py b/sktime/regression/deep_learning/cnn.py
--- a/sktime/regression/deep_learning/cnn.py
+++ b/sktime/regression/deep_learning/cnn.py
@@ -1,9 +1,11 @@
 # -*- coding: utf-8 -*-
 """Time Convolutional Neural Network (CNN) for regression."""
 
-__author__ = ["AurumnPegasus"]
+__author__ = ["AurumnPegasus", "achieveordie"]
 __all__ = ["CNNRegressor"]
 
+from sklearn.utils import check_random_state
+
 from sktime.networks.cnn import CNNNetwork
 from sktime.regression.deep_learning.base import BaseDeepRegressor
 from sktime.utils.validation._dependencies import _check_dl_dependencies
@@ -34,7 +36,12 @@ class CNNRegressor(BaseDeepRegressor):
         whether to output extra information
     loss            : string, default="mean_squared_error"
         fit parameter for the keras model
-    optimizer       : keras.optimizer, default=keras.optimizers.Adam(),
+    activation      : keras.activations or string, default `linear`
+        function to use in the output layer.
+    optimizer       : keras.optimizers or string, default `None`.
+        when `None`, internally uses `keras.optimizers.Adam(0.01)`
+    use_bias        : bool, default=True
+        whether to use bias in the output layer.
     metrics         : list of strings, default=["accuracy"],
 
     Notes
@@ -58,7 +65,10 @@ def __init__(
         verbose=False,
         loss="mean_squared_error",
         metrics=None,
-        random_seed=0,
+        random_state=0,
+        activation="linear",
+        use_bias=True,
+        optimizer=None,
     ):
         _check_dl_dependencies(severity="error")
         super(CNNRegressor, self).__init__(
@@ -73,7 +83,11 @@ def __init__(
         self.verbose = verbose
         self.loss = loss
         self.metrics = metrics
-        self.random_seed = random_seed
+        self.random_state = random_state
+        self.activation = activation
+        self.use_bias = use_bias
+        self.optimizer = optimizer
+        self.history = None
         self._network = CNNNetwork()
 
     def build_model(self, input_shape, **kwargs):
@@ -96,7 +110,7 @@ def build_model(self, input_shape, **kwargs):
         import tensorflow as tf
         from tensorflow import keras
 
-        tf.random.set_seed(self.random_seed)
+        tf.random.set_seed(self.random_state)
 
         if self.metrics is None:
             metrics = ["accuracy"]
@@ -105,13 +119,23 @@ def build_model(self, input_shape, **kwargs):
 
         input_layer, output_layer = self._network.build_network(input_shape, **kwargs)
 
-        output_layer = keras.layers.Dense(units=1, activation="sigmoid")(output_layer)
+        output_layer = keras.layers.Dense(
+            units=1,
+            activation=self.activation,
+            use_bias=self.use_bias,
+        )(output_layer)
+
+        self.optimizer_ = (
+            keras.optimizers.Adam(learning_rate=0.01)
+            if self.optimizer is None
+            else self.optimizer
+        )
 
         model = keras.models.Model(inputs=input_layer, outputs=output_layer)
 
         model.compile(
             loss=self.loss,
-            optimizer=keras.optimizers.Adam(),
+            optimizer=self.optimizer_,
             metrics=metrics,
         )
         return model
@@ -136,6 +160,7 @@ def _fit(self, X, y):
         # Transpose to conform to Keras input style.
         X = X.transpose(0, 2, 1)
 
+        check_random_state(self.random_state)
         self.input_shape = X.shape[1:]
         self.model_ = self.build_model(self.input_shape)
         if self.verbose:


Test patch: