Repository: sktime/sktime

Issue:
[ENH] Changing and Adding features to CNN Classifier
**Is your feature request related to a problem? Please describe.**
As discussed in the daily-stand up on 05-07-12, I wanted to propose some enhancements / changes to CNN Classifier (and hence to Deep Learning based estimators)

**Describe the solution you'd like**
- [ ] Moving the `convert_y_to_keras` function in `BaseDeepClassifier` into a utils function
- [x] Enable returning the `label_encoder` and `one hot encoder` to the user from `convert_y_to_keras` (it can be useful for the users in case they want to use it to encode other data / or to decode data later on)
- [x] Add option to have multiple linear layers in output layer / different activation function. Also add the `use_bias` parameter to output layer. (simply as a customisation option to users, multiple linear layers are useful to create deeper networks depending on dataset / choice of activation function)
- [x] option to choose optimiser (usually its Adam but just for customisation's sake we can give this as option as well). Also should allow specifying `lr` either within the optimiser or as parameter to the class itself.
- [x] Accept parameter as input in `CNNClassifier` to specify `random seed`
- [ ] Train/fit the model on GPU depending on users choice (maybe take an input parameter for this as well)
- [x] Option to view / return the loss of fit (either as a graph or simply as a list / array ) / A better idea is to add a summary function which can return model summary as well as `fit` history. 

**Additional context**
These are some additional features / changes I wanted to add to the `CNNClasifier`. Any inputs / additional comments are appreciated  



Discussion:
Yes, great additions, great suggestions!

> Also should allow specifying `lr` either within the optimiser or as parameter to the class itself.

I think this one is unnecessary, because you can pass the `lr` to the optimizer, which is an object you would then pass to the classifier class constructor. We do not need two places to pass it?

Or, do you see a good argument why it would be useful to the user to pass `lr` separately? Or am I simply misunderstanding the suggestion?
all sounds like a good idea, can we keep the defaults the same to maintain reproducibility of results? 
> can we keep the defaults the same to maintain reproducibility of results?

Changing defaults would require deprecation, so I would advise against changing defaults.

Resolving patch:
diff --git a/sktime/classification/deep_learning/base.py b/sktime/classification/deep_learning/base.py
--- a/sktime/classification/deep_learning/base.py
+++ b/sktime/classification/deep_learning/base.py
@@ -66,6 +66,18 @@ def build_model(self, input_shape, n_classes, **kwargs):
         """
         ...
 
+    def summary(self):
+        """
+        Summary function to return the losses/metrics for model fit.
+
+        Returns
+        -------
+        history: dict,
+            Dictionary containing model's train/validation losses and metrics
+
+        """
+        return self.history.history
+
     def _predict(self, X, **kwargs):
         probs = self._predict_proba(X, **kwargs)
         rng = check_random_state(self.random_state)
@@ -100,23 +112,14 @@ def _predict_proba(self, X, **kwargs):
         probs = probs / probs.sum(axis=1, keepdims=1)
         return probs
 
-    def convert_y_to_keras(self, y, label_encoder=None, onehot_encoder=None):
+    def convert_y_to_keras(self, y):
         """Convert y to required Keras format."""
-        if (label_encoder is None) and (onehot_encoder is None):
-            # make the encoders and store in self
-            self.label_encoder = LabelEncoder()
-            self.onehot_encoder = OneHotEncoder(sparse=False, categories="auto")
-            # categories='auto' to get rid of FutureWarning
-
-            y = self.label_encoder.fit_transform(y)
-            self.classes_ = self.label_encoder.classes_
-            self.n_classes_ = len(self.classes_)
-
-            y = y.reshape(len(y), 1)
-            y = self.onehot_encoder.fit_transform(y)
-        else:
-            y = label_encoder.fit_transform(y)
-            y = y.reshape(len(y), 1)
-            y = onehot_encoder.fit_transform(y)
-
+        self.label_encoder = LabelEncoder()
+        y = self.label_encoder.fit_transform(y)
+        self.classes_ = self.label_encoder.classes_
+        self.n_classes_ = len(self.classes_)
+        y = y.reshape(len(y), 1)
+        self.onehot_encoder = OneHotEncoder(sparse=False, categories="auto")
+        # categories='auto' to get rid of FutureWarning
+        y = self.onehot_encoder.fit_transform(y)
         return y
diff --git a/sktime/classification/deep_learning/cnn.py b/sktime/classification/deep_learning/cnn.py
--- a/sktime/classification/deep_learning/cnn.py
+++ b/sktime/classification/deep_learning/cnn.py
@@ -3,6 +3,7 @@
 
 __author__ = ["James-Large", "TonyBagnall"]
 __all__ = ["CNNClassifier"]
+
 from sklearn.utils import check_random_state
 
 from sktime.classification.deep_learning.base import BaseDeepClassifier
@@ -37,6 +38,14 @@ class CNNClassifier(BaseDeepClassifier):
         fit parameter for the keras model
     optimizer       : keras.optimizer, default=keras.optimizers.Adam(),
     metrics         : list of strings, default=["accuracy"],
+    activation      : string or a tf callable, default="sigmoid"
+        Activation function used in the output linear layer.
+        List of available activation functions:
+        https://keras.io/api/layers/activations/
+    use_bias        : boolean, default = True
+        whether the layer uses a bias vector.
+    optimizer       : keras.optimizers object, default = Adam(lr=0.01)
+        specify the optimizer and the learning rate to be used.
 
     Notes
     -----
@@ -46,6 +55,16 @@ class CNNClassifier(BaseDeepClassifier):
 
     Adapted from the implementation from Fawaz et. al
     https://github.com/hfawaz/dl-4-tsc/blob/master/classifiers/cnn.py
+
+    Examples
+    --------
+    >>> from sktime.classification.deep_learning.cnn import CNNClassifier
+    >>> from sktime.datasets import load_unit_test
+    >>> X_train, y_train = load_unit_test(split="train", return_X_y=True)
+    >>> X_test, y_test = load_unit_test(split="test", return_X_y=True)
+    >>> cnn = CNNClassifier()
+    >>> cnn.fit(X_train, y_train)
+    CNNClassifier(...)
     """
 
     def __init__(
@@ -59,6 +78,10 @@ def __init__(
         verbose=False,
         loss="mean_squared_error",
         metrics=None,
+        random_state=None,
+        activation="sigmoid",
+        use_bias=True,
+        optimizer=None,
     ):
         _check_dl_dependencies(severity="error")
         super(CNNClassifier, self).__init__()
@@ -71,6 +94,11 @@ def __init__(
         self.verbose = verbose
         self.loss = loss
         self.metrics = metrics
+        self.random_state = random_state
+        self.activation = activation
+        self.use_bias = use_bias
+        self.optimizer = optimizer
+        self.history = None
         self._network = CNNNetwork()
 
     def build_model(self, input_shape, n_classes, **kwargs):
@@ -92,22 +120,31 @@ def build_model(self, input_shape, n_classes, **kwargs):
         -------
         output : a compiled Keras Model
         """
+        import tensorflow as tf
         from tensorflow import keras
 
+        tf.random.set_seed(self.random_state)
+
         if self.metrics is None:
             metrics = ["accuracy"]
         else:
             metrics = self.metrics
         input_layer, output_layer = self._network.build_network(input_shape, **kwargs)
 
-        output_layer = keras.layers.Dense(units=n_classes, activation="sigmoid")(
-            output_layer
+        output_layer = keras.layers.Dense(
+            units=n_classes, activation=self.activation, use_bias=self.use_bias
+        )(output_layer)
+
+        self.optimizer_ = (
+            keras.optimizers.Adam(learning_rate=0.01)
+            if self.optimizer is None
+            else self.optimizer
         )
 
         model = keras.models.Model(inputs=input_layer, outputs=output_layer)
         model.compile(
             loss=self.loss,
-            optimizer=keras.optimizers.Adam(),
+            optimizer=self.optimizer_,
             metrics=metrics,
         )
         return model
@@ -128,6 +165,7 @@ def _fit(self, X, y):
         """
         if self.callbacks is None:
             self._callbacks = []
+
         y_onehot = self.convert_y_to_keras(y)
         # Transpose to conform to Keras input style.
         X = X.transpose(0, 2, 1)


Test patch: