Repository: sktime/sktime

Issue:
[ENH] Adding CNTC DL model for classification
**Is your feature request related to a problem? Please describe.**
In line with discussions and #2894 , adding CNTC model for classification from sktime-dl



Discussion:
Adding some notes here to help me better understand the model (and hopefully someone else if they were to work on it someday):
[Reference Paper](https://reader.elsevier.com/reader/sd/pii/S0925231219316364?token=EF73B97B3D6DD3191D54F3117B5AFF1E0C1D115379A39493893BD9A0EA0A099ACAC3A4405ABB3CDAFB4E6DC2D10C4E1D&originRegion=eu-west-1&originCreation=20220802110553)

1. Idea about shaping the data according to the paper: "Time series data is fed into a CLSTM and CCNN networks simultaneously and is perceived differently. In the CLSTM block, the input data is viewed as a multivariate time series with a single time stamp. In contrast, the CCNN block receives univariate data with numerous time stamps"
2. The reason for taking sliding window and mean from the paper: "We let M denote the mean of an i-th sliding window [52] with size j of our time series. Given a time series s1, s2, s3, ..., sn−1, sn we denote the i-th sliding window of size j as Li,j. By setting the windows size to be five, the corresponding windows are as follows; L1,5 = {s1, s2, s3, s4, s5}, L2,5 = {s2, s3, s4, s5, s6}, ..., Ln−1,5 = {sn−5, sn−4, sn−3, sn−2, sn−1}, Ln,5 = {sn−4, sn−3, sn−2, sn−1, sn}. Therefore, the means of windows L1,5, L2,5,...,Ln−1,5, Ln,5 are M1, M2, ..., Mn−1, Mn respectively. Let Pd be the mean vectors equivalent to the size of a window. For instance, a time series with a chosen window of length 3 will have vectors Pd as follows; P1 = {M1, M2, M3}, P2 = {M4, M5, M6}, ..., Pn−1 = {Mn−5, Mn−4, Mn−3}, Pn = {Mn−2, Mn−1, Mn}. Hence, we have vector P = {P1, P2, P3, . . . , Pn−1, Pn}. The CLSTM module refers to the mean of windows vector P as the contextual features"
Hey @James-Large  (tagging you since tony recommended I ask you about this):
I had a doubt with respect to the prepare_input function in the network. Can you just help me out by telling the expected shape of input X to the function? The file I am looking at is this: https://github.com/sktime/sktime-dl/blob/b565b7499f58f43da7314f1bf26eccce94e88134/sktime_dl/networks/_cntc.py#L10

also, in the model creation why are there 3 input layers? Regarding this doubt. there are 2 arms which are suggested in the paper, one is of CCNN and another is CLSTM. In the code written by the author (and in sktime-dl) the first arm has cnn-rnn-cnn arm (conv1, rnn1 which becomes conc1, and then conv2) and another is lstm arm (lstm1), both of which are then merged. From my understanding atleast, it seems that the arms are supposed to be cnn and lstm only (or maybe cnn-rnn and lstm? there was a concept of recurrant convolution mentioned in one of the references), but the current model doesnt seem to follow that. Is there something wrong in my understanding or is there a change in code? Please let me know!
Hi @AurumnPegasus, to predicate this answer I've been out of the loop on sktime/-dl for some time. It looks like that architecture was imported by Jack Russon, who if I recall was a masters student last year and so maybe I'm easier to contact. 

On the model details, I can't have a read of the paper since I don't have academic institute access any more and it doesn't look like there's an arxiv/free version of the paper, however from the [original contextual CCN paper](https://www.researchgate.net/profile/Joongbo-Shin/publication/325423879_Contextual-CNN_A_Novel_Architecture_Capturing_Unified_Meaning_for_Sentence_Classification/links/5b8e9b0345851540d1c870fb/Contextual-CNN-A-Novel-Architecture-Capturing-Unified-Meaning-for-Sentence-Classification.pdf), the general idea is "combining recurrent connections with convolutional layer". I annoyingly can't find code for that original paper, but the architecture defined by CNTC looks like it follows it (RNN and CNN merged to make a CCNN as the first arm). I think in the [CNTC figures](https://raw.githubusercontent.com/AmaduFullah/CNTC_MODEL/master/model2.png) they've just simplified this into a single arm for figure-drawing purposes. 

The [CLSTM](https://arxiv.org/pdf/1602.06291.pdf) part ("To adapt the LSTM cell that takes words to a CLSTM cell that takes as input both words and topics,") in a TSC context seems to just be an LSTM with an expanded input to act as the contextual topics. In the original CNTC code this is defined [here](https://github.com/AmaduFullah/CNTC_MODEL/blob/master/cntc.ipynb) (it's a notebook so no line number links)
![image](https://user-images.githubusercontent.com/44509982/182811027-083ae869-244d-4935-ae19-86f7bdcef41a.png)
and then in the sktime-dl code defined in the [prepare_input function](https://github.com/sktime/sktime-dl/blob/b565b7499f58f43da7314f1bf26eccce94e88134/sktime_dl/networks/_cntc.py#L120).

So ultimately, to the best of my knowledge there being 3 input layers seems reasonable to me. The CNTC paper aggregated the CCNN architecture into one 'arm' for their model, but that itself is composed of the RNN and CNN inputs. I'd say also that regardless, faithfulness to the author's code is what (I think, others feel free to correct me) is important when integrating known/defined models that have code attached. If that's the architecture they implemented to get the results they reported and published, we should replicate the same. 

____________

On the input format to prepare_input, it looks like prepare_input is called in as a data processing step in the fit and predicts of the implementing model classes, [_after_](https://github.com/sktime/sktime-dl/blob/b565b7499f58f43da7314f1bf26eccce94e88134/sktime_dl/classification/_cntc.py#L157) the [call](https://github.com/sktime/sktime-dl/blob/b565b7499f58f43da7314f1bf26eccce94e88134/sktime_dl/classification/_cntc.py#L145) to [check_and_clean_data](https://github.com/sktime/sktime-dl/blob/b565b7499f58f43da7314f1bf26eccce94e88134/sktime_dl/utils/_data.py#L10)

check_and_clean_data takes (what I assume to still be) the sktime data format and converts it to a 3d np array of the form [instances][dimensions][timepoints], and so it looks like that's what prepare_input is expecting



Hope that helps, I won't be able to spend much more time on this today but it was a neat little foray back into deep learning literature for me, thanks

Resolving patch:
diff --git a/sktime/classification/deep_learning/cntc.py b/sktime/classification/deep_learning/cntc.py
new file mode 100644
--- /dev/null
+++ b/sktime/classification/deep_learning/cntc.py
@@ -0,0 +1,282 @@
+# -*- coding: utf-8 -*-
+"""Contextual Time-series Neural Classifier for TSC."""
+
+__author__ = ["James-Large", "TonyBagnall", "AurumnPegasus"]
+__all__ = ["CNTCClassifier"]
+from sklearn.utils import check_random_state
+
+from sktime.classification.deep_learning.base import BaseDeepClassifier
+from sktime.networks.cntc import CNTCNetwork
+from sktime.utils.validation._dependencies import _check_dl_dependencies
+
+_check_dl_dependencies(severity="warning")
+
+
+class CNTCClassifier(BaseDeepClassifier):
+    """Contextual Time-series Neural Classifier (CNTC), as described in [1].
+
+    Parameters
+    ----------
+    n_epochs       : int, default = 2000
+        the number of epochs to train the model
+    batch_size      : int, default = 16
+        the number of samples per gradient update.
+    filter_sizes    : tuple of shape (2), default = (16, 8)
+        filter sizes for CNNs in CCNN arm.
+    kernel_sizes     : two-tuple, default = (1, 1)
+        the length of the 1D convolution window for
+        CNNs in CCNN arm.
+    rnn_size        : int, default = 64
+        number of rnn units in the CCNN arm.
+    lstm_size       : int, default = 8
+        number of lstm units in the CLSTM arm.
+    dense_size      : int, default = 64
+        dimension of dense layer in CNTC.
+    random_state    : int or None, default=None
+        Seed for random number generation.
+    verbose         : boolean, default = False
+        whether to output extra information
+    loss            : string, default="mean_squared_error"
+        fit parameter for the keras model
+    optimizer       : keras.optimizer, default=keras.optimizers.Adam(),
+    metrics         : list of strings, default=["accuracy"],
+
+    Notes
+    -----
+    Adapted from the implementation from Fullah et. al
+    https://github.com/AmaduFullah/CNTC_MODEL/blob/master/cntc.ipynb
+
+    References
+    ----------
+    .. [1] Network originally defined in:
+        @article{FULLAHKAMARA202057,
+        title = {Combining contextual neural networks for time series classification},
+        journal = {Neurocomputing},
+        volume = {384},
+        pages = {57-66},
+        year = {2020},
+        issn = {0925-2312},
+        doi = {https://doi.org/10.1016/j.neucom.2019.10.113},
+        url = {https://www.sciencedirect.com/science/article/pii/S0925231219316364},
+        author = {Amadu {Fullah Kamara} and Enhong Chen and Qi Liu and Zhen Pan},
+        keywords = {Time series classification, Contextual convolutional neural
+            networks, Contextual long short-term memory, Attention, Multilayer
+            perceptron},
+       }
+
+    Examples
+    --------
+    >>> from sktime.classification.deep_learning.cntc import CNTCClassifier
+    >>> from sktime.datasets import load_unit_test
+    >>> X_train, y_train = load_unit_test(split="train", return_X_y=True)
+    >>> X_test, y_test = load_unit_test(split="test", return_X_y=True)
+    >>> cntc = CNTCClassifier() # doctest: +SKIP
+    >>> cntc.fit(X_train, y_train) # doctest: +SKIP
+    CNTCClassifier(...) # doctest: +SKIP
+    """
+
+    def __init__(
+        self,
+        n_epochs=2000,
+        batch_size=16,
+        filter_sizes=(16, 8),
+        kernel_sizes=(1, 1),
+        rnn_size=64,
+        lstm_size=8,
+        dense_size=64,
+        callbacks=None,
+        verbose=False,
+        loss="categorical_crossentropy",
+        metrics=None,
+        random_state=0,
+    ):
+        _check_dl_dependencies(severity="error")
+        super(CNTCClassifier, self).__init__()
+        self.kernel_sizes = kernel_sizes  # used plural
+        self.filter_sizes = filter_sizes  # used plural
+        self.rnn_size = rnn_size
+        self.lstm_size = lstm_size
+        self.dense_size = dense_size
+        self.callbacks = callbacks
+        self.n_epochs = n_epochs
+        self.batch_size = batch_size
+        self.verbose = verbose
+        self.loss = loss
+        self.metrics = metrics
+        self.random_state = random_state
+        self._network = CNTCNetwork()
+
+    def build_model(self, input_shape, n_classes, **kwargs):
+        """Construct a compiled, un-trained, keras model that is ready for training.
+
+        In sktime, time series are stored in numpy arrays of shape (d,m), where d
+        is the number of dimensions, m is the series length. Keras/tensorflow assume
+        data is in shape (m,d). This method also assumes (m,d). Transpose should
+        happen in fit.
+
+        Parameters
+        ----------
+        input_shape : tuple
+            The shape of the data fed into the input layer, should be (m,d)
+        n_classes: int
+            The number of classes, which becomes the size of the output layer
+
+        Returns
+        -------
+        output : a compiled Keras Model
+        """
+        from tensorflow import keras
+
+        metrics = ["accuracy"] if self.metrics is None else self.metrics
+        input_layer, output_layer = self._network.build_network(input_shape, **kwargs)
+
+        output_layer = keras.layers.Dense(units=n_classes, activation="softmax")(
+            output_layer
+        )
+
+        model = keras.models.Model(inputs=input_layer, outputs=output_layer)
+        model.compile(
+            loss=self.loss,
+            optimizer=keras.optimizers.Adam(),
+            metrics=metrics,
+        )
+        return model
+
+    def prepare_input(self, X):
+        """
+        Prepare input for the CLSTM arm of the model.
+
+        According to the paper:
+            "
+                Time series data is fed into a CLSTM and CCNN networks simultaneously
+                and is perceived differently. In the CLSTM block, the input data is
+                viewed as a multivariate time series with a single time stamp. In
+                contrast, the CCNN block receives univariate data with numerous time
+                stamps
+            "
+
+        Arguments
+        ---------
+        X: tuple of shape = (series_length (m), n_dimensions (d))
+            The shape of the data fed into the model.
+
+        Returns
+        -------
+        trainX: tuple,
+            The input to be fed to the two arms of CNTC.
+        """
+        import numpy as np
+        import pandas as pd
+        from tensorflow import keras
+
+        if X.shape[2] == 1:
+            # Converting data to pandas
+            trainX1 = X.reshape([X.shape[0], X.shape[1]])
+            pd_trainX = pd.DataFrame(trainX1)
+
+            # Taking rolling window
+            window = pd_trainX.rolling(window=3).mean()
+            window = window.fillna(0)
+
+            trainX2 = np.concatenate((trainX1, window), axis=1)
+            trainX2 = keras.backend.variable(trainX2)
+            trainX2 = keras.layers.Dense(
+                trainX1.shape[1], input_shape=(trainX2.shape[1:])
+            )(trainX2)
+            trainX2 = keras.backend.eval(trainX2)
+            trainX = trainX2.reshape((trainX2.shape[0], trainX2.shape[1], 1))
+        else:
+            trainXs = []
+            for i in range(X.shape[2]):
+                trainX1 = X[:, :, i]
+                pd_trainX = pd.DataFrame(trainX1)
+
+                window = pd_trainX.rolling(window=3).mean()
+                window = window.fillna(0)
+
+                trainX2 = np.concatenate((trainX1, window), axis=1)
+                trainX2 = keras.backend.variable(trainX2)
+                trainX2 = keras.layers.Dense(
+                    trainX1.shape[1], input_shape=(trainX2.shape[1:])
+                )(trainX2)
+                trainX2 = keras.backend.eval(trainX2)
+
+                trainX = trainX2.reshape((trainX2.shape[0], trainX2.shape[1], 1))
+                trainXs.append(trainX)
+
+            trainX = np.concatenate(trainXs, axis=2)
+        return trainX
+
+    def _fit(self, X, y):
+        """Fit the classifier on the training set (X, y).
+
+        Parameters
+        ----------
+        X : np.ndarray of shape = (n_instances (n), n_dimensions (d), series_length (m))
+            The training input samples.
+        y : np.ndarray of shape n
+            The training data class labels.
+
+        Returns
+        -------
+        self : object
+        """
+        if self.callbacks is None:
+            self._callbacks = []
+        y_onehot = self.convert_y_to_keras(y)
+        # Transpose to conform to Keras input style.
+        X = X.transpose(0, 2, 1)
+
+        check_random_state(self.random_state)
+        self.input_shape = X.shape[1:]
+        self.model_ = self.build_model(self.input_shape, self.n_classes_)
+        X2 = self.prepare_input(X)
+        if self.verbose:
+            self.model_.summary()
+        self.history = self.model_.fit(
+            [X2, X, X],
+            y_onehot,
+            batch_size=self.batch_size,
+            epochs=self.n_epochs,
+            verbose=self.verbose,
+            callbacks=self._callbacks,
+        )
+        return self
+
+    def _predict(self, X, **kwargs):
+        import numpy as np
+
+        probs = self._predict_proba(X, **kwargs)
+        rng = check_random_state(self.random_state)
+        return np.array(
+            [
+                self.classes_[int(rng.choice(np.flatnonzero(prob == prob.max())))]
+                for prob in probs
+            ]
+        )
+
+    def _predict_proba(self, X, **kwargs):
+        """Find probability estimates for each class for all cases in X.
+
+        Parameters
+        ----------
+        X : an np.ndarray of shape = (n_instances, n_dimensions, series_length)
+            The training input samples.
+
+        Returns
+        -------
+        output : array of shape = [n_instances, n_classes] of probabilities
+        """
+        import numpy as np
+
+        # Transpose to work correctly with keras
+        X = X.transpose((0, 2, 1))
+        X2 = self.prepare_input(X)
+        probs = self.model_.predict([X2, X, X], self.batch_size, **kwargs)
+
+        # check if binary classification
+        if probs.shape[1] == 1:
+            # first column is probability of class 0 and second is of class 1
+            probs = np.hstack([1 - probs, probs])
+        probs = probs / probs.sum(axis=1, keepdims=1)
+        return probs
diff --git a/sktime/networks/base.py b/sktime/networks/base.py
--- a/sktime/networks/base.py
+++ b/sktime/networks/base.py
@@ -11,6 +11,10 @@
 class BaseDeepNetwork(BaseObject, ABC):
     """Abstract base class for deep learning networks."""
 
+    _tags = {
+        "python_dependencies": "tensorflow",
+    }
+
     @abstractmethod
     def build_network(self, input_shape, **kwargs):
         """Construct a network and return its input and output layers.
diff --git a/sktime/networks/cntc.py b/sktime/networks/cntc.py
new file mode 100644
--- /dev/null
+++ b/sktime/networks/cntc.py
@@ -0,0 +1,193 @@
+# -*- coding: utf-8 -*-
+"""a combination of CCNN and CLSTM as feature extractors."""
+
+__author__ = ["James-Large", "Withington", "TonyBagnall", "AurumnPegasus"]
+
+from sktime.networks.base import BaseDeepNetwork
+from sktime.utils.validation._dependencies import (
+    _check_dl_dependencies,
+    _check_soft_dependencies,
+)
+
+_check_dl_dependencies(severity="warning")
+_check_soft_dependencies(
+    package_import_alias={"keras-self-attention": "keras_self_attention"},
+    severity="warning",
+)
+
+
+class CNTCNetwork(BaseDeepNetwork):
+    """Combining contextual neural networks for time series classification.
+
+    Adapted from the implementation used in [1]
+
+    Parameters
+    ----------
+    kernel_size     : int, default = 7
+        specifying the length of the 1D convolution window
+    avg_pool_size   : int, default = 3
+        size of the average pooling windows
+    n_conv_layers   : int, default = 2
+        the number of convolutional plus average pooling layers
+    filter_sizes    : array of int, shape = (nb_conv_layers)
+    activation      : string, default = sigmoid
+        keras activation function
+    random_state    : int, default = 0
+        seed to any needed random actions
+
+    Notes
+    -----
+    Adapted from the implementation from Fullah et. al
+    https://github.com/AmaduFullah/CNTC_MODEL/blob/master/cntc.ipynb
+
+    References
+    ----------
+    .. [1] Network originally defined in:
+        @article{FULLAHKAMARA202057,
+        title = {Combining contextual neural networks for time series classification},
+        journal = {Neurocomputing},
+        volume = {384},
+        pages = {57-66},
+        year = {2020},
+        issn = {0925-2312},
+        doi = {https://doi.org/10.1016/j.neucom.2019.10.113},
+        url = {https://www.sciencedirect.com/science/article/pii/S0925231219316364},
+        author = {Amadu {Fullah Kamara} and Enhong Chen and Qi Liu and Zhen Pan},
+        keywords = {Time series classification, Contextual convolutional neural
+            networks, Contextual long short-term memory, Attention, Multilayer
+            perceptron},
+       }
+    """
+
+    _tags = {"python_dependencies": ["tensorflow", "keras_self_attention"]}
+
+    def __init__(
+        self,
+        random_state=0,
+        rnn_layer=64,
+        filter_sizes=(16, 8),
+        kernel_sizes=(1, 1),
+        lstm_size=8,
+        dense_size=64,
+    ):
+        super(CNTCNetwork, self).__init__()
+        _check_dl_dependencies(severity="error")
+        _check_soft_dependencies(
+            package_import_alias={"keras-self-attention": "keras_self_attention"},
+            severity="warning",
+        )
+        self.random_state = random_state
+        self.rnn_layer = rnn_layer
+        self.filter_sizes = filter_sizes
+        self.kernel_sizes = kernel_sizes
+        self.lstm_size = lstm_size
+        self.dense_size = dense_size
+
+    def build_network(self, input_shape, **kwargs):
+        """Construct a network and return its input and output layers.
+
+        Arguments
+        ---------
+        input_shape: tuple
+            The shape of the data fed into the input layer, should be (m,d)
+
+        Returns
+        -------
+        input_layer: a keras layer
+        output_layer: a keras layer
+        """
+        from keras_self_attention import SeqSelfAttention
+        from tensorflow import keras
+
+        input_layers = []
+
+        # CNN Arm
+        input_layers.append(keras.layers.Input(input_shape))
+        input_layers.append(keras.layers.Input(input_shape))
+        self.dropout = 0.2
+
+        conv1 = keras.layers.Conv1D(
+            self.filter_sizes[0],
+            self.kernel_sizes[0],
+            activation="relu",
+            use_bias=True,
+            kernel_initializer="glorot_uniform",
+        )(input_layers[0])
+        conv1 = keras.layers.BatchNormalization()(conv1)
+        conv1 = keras.layers.Dropout(self.dropout)(conv1)
+        conv1 = keras.layers.Dense(
+            input_shape[1],
+            input_shape=(input_shape[0], keras.backend.int_shape(conv1)[2]),
+        )(conv1)
+
+        # RNN for CNN Arm (CCNN)
+        rnn1 = keras.layers.SimpleRNN(
+            self.rnn_layer * input_shape[1],
+            activation="relu",
+            use_bias=True,
+            kernel_initializer="glorot_uniform",
+        )(input_layers[1])
+        rnn1 = keras.layers.BatchNormalization()(rnn1)
+        rnn1 = keras.layers.Dropout(self.dropout)(rnn1)
+        rnn1 = keras.layers.Reshape((64, input_shape[1]))(rnn1)
+
+        # Combining CNN and RNN
+        conc1 = keras.layers.Concatenate(
+            axis=-2, name="contextual_convolutional_layer1"
+        )([conv1, rnn1])
+
+        # Final CNN for C-CNN (WHY)
+        conv2 = keras.layers.Conv1D(
+            self.filter_sizes[1],
+            self.kernel_sizes[1],
+            activation="relu",
+            kernel_initializer="glorot_uniform",
+            name="standard_cnn_layer",
+        )(conc1)
+        conv2 = keras.layers.Dense(
+            input_shape[1],
+            input_shape=(input_shape[0], keras.backend.int_shape(conv2)[2]),
+        )(conv2)
+        conv2 = keras.layers.BatchNormalization()(conv2)
+        conv2 = keras.layers.Dropout(0.1)(conv2)
+
+        # CLSTM Arm
+        input_layers.append(keras.layers.Input(input_shape))
+        lstm1 = keras.layers.LSTM(
+            self.lstm_size * input_shape[1],
+            return_sequences=False,
+            kernel_initializer="glorot_uniform",
+            activation="relu",
+        )(input_layers[2])
+        lstm1 = keras.layers.Reshape((self.lstm_size, input_shape[1]))(lstm1)
+        lstm1 = keras.layers.Dropout(self.dropout)(lstm1)
+        merge = keras.layers.Concatenate(
+            axis=-2, name="contextual_convolutional_layer2"
+        )([conv2, lstm1])
+
+        # Output calculation based on combination
+        avg = keras.layers.MaxPooling1D(pool_size=1, strides=None, padding="valid")(
+            merge
+        )
+        avg = keras.layers.Dropout(0.1)(avg)
+
+        # Adding self attention
+        att = SeqSelfAttention(
+            attention_width=10,
+            attention_activation="sigmoid",
+            name="Attention",
+            attention_type="multiplicative",
+        )(avg)
+        att = keras.layers.Dropout(0.1)(att)
+
+        # Adding ouutput MLP Layer
+        mlp1 = keras.layers.Dense(
+            self.dense_size, kernel_initializer="glorot_uniform", activation="relu"
+        )(att)
+        mlp1 = keras.layers.Dropout(0.1)(mlp1)
+        mlp2 = keras.layers.Dense(
+            self.dense_size, kernel_initializer="glorot_uniform", activation="relu"
+        )(mlp1)
+        mlp2 = keras.layers.Dropout(0.1)(mlp2)
+        flat = keras.layers.Flatten()(mlp2)
+        return input_layers, flat


Test patch:
diff --git a/sktime/tests/_config.py b/sktime/tests/_config.py
--- a/sktime/tests/_config.py
+++ b/sktime/tests/_config.py
@@ -96,6 +96,9 @@
     "MLPClassifier": [
         "test_fit_idempotent",
     ],
+    "CNTCClassifier": [
+        "test_fit_idempotent",
+    ],
     # sth is not quite right with the RowTransformer-s changing state,
     #   but these are anyway on their path to deprecation, see #2370
     "SeriesToPrimitivesRowTransformer": ["test_methods_do_not_change_state"],