dataesr/scientific-paragraphs-categorization huggingface dataset:
To better understand the mechanism of these networks, consider an ESN with input dimensionality M , neurons in the reservoir N r and an output dimensionality K equal to the number of classes in the dataset. Let X(t) ∈ R M , I(t) ∈ R Nr and Ŷ (t) ∈ R K denote the vectors of the input M -dimensional MTS, the internal (or hidden) state and the output unit activity for time t respectively. Further let W in ∈ R Nr×M and W ∈ R Nr×Nr and W out ∈ R C×Nr denote respectively the weight matrices for the input time series, the internal connections and the output connections as seen in Figure  4. The internal unit activity I(t) at time t is updated using the internal state at time step t -1 and the input time series element at time t. Formally the hidden state can be computed using the following recurrence: I(t) = f (W in X(t) + W I(t -1)) | ∀ t ∈ [1, T ] (8) with f denoting an activation function of the neurons, a common choice is tanh(•) applied elementwise  (Tanisaro and Heidemann, 2016).
The output can be computed according to the following equation: Ŷ (t) = W out I(t) (9) thus classifying each time series element X(t). Note that ESNs depend highly on the initial values of the reservoir that should satisfy a pre-determined hyperparameter: the spectral radius. Figure  4  shows an example of an ESN with a univariate input time series to be classified into K classes. Finally, we should note that for all types of DNNs, a set of techniques was proposed by the deep learning community to enhance neural networks' generalization capabilities. Regularization methods such as l2-norm weight decay  (Bishop, 2006)  or Dropout  (Srivastava et al. , 2014)  aim at reducing overfitting by limiting the activation of the neurons. Another popular technique is data augmentation, which tackles the problem of overfitting a small dataset by increasing the number of training instances  (Baird, 1992).
Discriminative models A discriminative deep learning model is a classifier (or regressor) that directly learns the mapping between the raw input of a time series (or its hand engineered features) and outputs a probability distribution over the class variables in a dataset. Several discriminative deep learning architectures have been proposed to solve the TSC task, but we found that this type of model could be further sub-divided into two groups: (1) deep learning models with hand engineered features and (2) endto-end deep learning models. The most frequently encountered and computer vision inspired feature extraction method for hand engineering approaches is the transformation of time series into images using specific imaging methods such as Gramian fields  (Wang and Oates, 2015b,a) , recurrence plots  (Hatami et al.
(2017b); Zhao et al. (2017) , traditional CNNs were also validated on the UCR/UEA archive. More recently in  Wang et al. (2018) , the architectures proposed in  Wang et al. (2017b)  were modified to leverage a filter initialization technique based on the Daubechies 4 Wavelet values  (Rowe and Abbott, 1995). Outside of the UCR/UEA archive, deep learning has reached state-of-the-art performance on several datasets in different domains  (Längkvist et al. , 2014). For spatio-temporal series forecasting problems, such as meteorology and oceanography, DNNs were proposed in  Ziat et al. (2017). Strodthoff and Strodthoff (2019)  proposed to detect myocardial infractions from electrocardiography data using deep CNNs. For human activity recognition from wearable sensors, deep learning is replacing the feature engineering approaches  (Nweke et al.
, 2018)  for the considered task. We mention one final type of hybrid architectures that showed promising results for the TSC task on the UCR/UEA archive datasets, where mainly CNNs were combined with other types of architectures such as Gated Recurrent Units  (Lin and Runger, 2018)  and the attention mechanism  (Serrà et al. , 2018). The reader may have noticed that CNNs appear under Auto Encoders as well as under End-to-End learning in Figure  5. This can be explained by the fact that CNNs when trained as Auto Encoders have a complete different objective function than CNNs that are trained in an end-to-end fashion. Now that we have presented the taxonomy for grouping DNNs for TSC, we introduce in the following section the different approaches that we have included in our experimental evaluation.
, 2015b; Bagnall et al. , 2017)  and/or on the MTS archive  (Baydogan, 2015). Finally, we chose to work with approaches that do not try to solve a sub task of the TSC problem such as in  Geng and Luo (2018)  where CNNs were modified to classify imbalanced time series datasets. To justify this choice, we emphasize that imbalanced TSC problems can be solved using several techniques such as data augmentation  (Ismail Fawaz et al. , 2018a)  and modifying the class weights  (Geng and Luo, 2018). However, any deep learning algorithm can benefit from this type of modification. Therefore if we did include modifications for solving imbalanced TSC tasks, it would be much harder to determine if it is the choice of the deep learning classifier or the modification itself that improved the accuracy of the model.
Multi Layer Perceptron The MLP, which is the most traditional form of DNNs, was proposed in  Wang et al. (2017b)  as a baseline architecture for TSC. The network contains 4 layers in total where each one is fully connected to the output of its previous layer. The final layer is a softmax classifier, which is fully connected to its previous layer's output and contains a number of neurons equal to the number of classes in a dataset. All three hidden FC layers are composed of 500 neurons with ReLU as the activation function. Each layer is preceded by a dropout operation  (Srivastava et al. , 2014)  with a rate equal to 0. 1, 0. 2, 0. 2 and 0. 3 for respectively the first, second, third and fourth layer. Dropout is one form of regularization that helps in preventing overfitting  (Srivastava et al.
Fully Convolutional Neural Network Fully Convolutional Neural Networks (FCNs) were first proposed in  Wang et al. (2017b)  for classifying univariate time series and validated on 44 datasets from the UCR/UEA archive. FCNs are mainly convolutional networks that do not contain any local pooling layers which means that the length of a time series is kept unchanged throughout the convolutions. In addition, one of the main characteristics of this architecture is the replacement of the traditional final FC layer with a Global Average Pooling (GAP) layer which reduces drastically the number of parameters in a neural network while enabling the use of the CAM  (Zhou et al. , 2016)  that highlights which parts of the input time series contributed the most to a certain classification. The architecture proposed in  Wang et al.
We can see that FCN does not hold any pooling nor a regularization operation. In addition, one of the advantages of FCNs is the invariance (denoted by #invar in Table  1 ) in the number of parameters for 4 layers (out of 5) across time series of different lengths. This invariance (due to using GAP) enables the use of a transfer learning approach where one can train a model on a certain source dataset and then fine-tune it on the target dataset  (Ismail Fawaz et al. , 2018c).
Residual Network The third and final proposed architecture in  Wang et al. (2017b)  is a relatively deep Residual Network (ResNet). For TSC, this is the deepest architecture with 11 layers of which the first 9 layers are convolutional followed by a GAP layer that averages the time series across the time dimension. The main characteristic of ResNets is the shortcut residual connection between consecutive convolutional layers. Actually, the difference with the usual convolutions (such as in FCNs) is that a linear shortcut is added to link the output of a residual block to its input thus enabling the flow of the gradient directly through these connections, which makes training a DNN much easier by reducing the vanishing gradient effect  (He et al. , 2016). The network is composed of three residual blocks followed by a GAP layer and a final softmax classifier whose number of neurons is equal to the number of classes in a dataset.
Each residual block is first composed of three convolutions whose output is added to the residual block's input and then fed to the next layer. The number of filters for all convolutions is fixed to 64, with the ReLU activation function that is preceded by a batch normalization operation. In each residual block, the filter's length is set to 8, 5 and 3 respectively for the first, second and third convolution. Similarly to the FCN model, the layers (except the final one) in the ResNet architecture have an invariant number of parameters across different datasets. That being said, we can easily pre-train a model on a source dataset, then transfer and fine-tune it on a target dataset without having to modify the hidden layers of the network. As we have previously mentioned and since this type of transfer learning approach can give an advantage for certain types of architecture, we leave the exploration of this area of research for future work.
Encoder Originally proposed by  Serrà et al. (2018) , Encoder is a hybrid deep CNN whose architecture is inspired by FCN  (Wang et al. , 2017b)  with a main difference where the GAP layer is replaced with an attention layer. In  Serrà et al. (2018) , two variants of Encoder were proposed: the first approach was to train the model from scratch in an end-to-end fashion on a target dataset while the second one was to pre-train this same architecture on a source dataset and then fine-tune it on a target dataset. The latter approach reached higher accuracy thus benefiting from the transfer learning technique. On the other hand, since almost all approaches can benefit to certain degree from a transfer learning method, we decided to implement only the end-to-end approach (training from scratch) which already showed high performance in the author's original paper.
More precisely, to implement this technique, the input MTS is multiplied with a second MTS of the same length and number of channels, except that the latter has gone through the softmax function. Each element in the second MTS will act as a weight for the first MTS, thus enabling the network to learn the importance of each element (time stamp). Finally, a traditional softmax classifier is fully connected to the latter layer with a number of neurons equal to the number of classes in the dataset. In addition to replacing the GAP layer with the attention layer, Encoder differs from FCN in three main core changes: (1) the PReLU activation function where an additional parameter is added for each filter to enable learning the slope of the function, (2) the dropout regularization technique and (3) the max pooling operation.
The output of each convolution in the first convolutional stage is concatenated to form the input of the subsequent convolutional layer. Following this second layer, an FC layer is deployed with 256 neurons using the sigmoid activation function. Finally, the usual softmax classifier is used with a number of neurons equal to the number of classes in the dataset. Note that each convolution in this network uses 256 filters with the sigmoid as an activation function, followed by a max pooling operation. Two architecture hyperparameters are cross-validated, using a grid search on an unseen split from the training set: the filter length and the pooling factor which determines the pooling size for the max pooling operation. The total number of layers in this network is 4, out of which only the first two convolutional layers are invariant (transferable).
For the first convolution, 5 filters are used and followed by a max pooling of length equal to 2. The second convolution uses 20 filters followed by a max pooling of length equal to 4. Thus, for an input time series of length l, the resulting output of these two convolutions will divide the length of the time series by 8 = 4 × 2. The convolutional blocks are followed by a non-linear fully connected layer which is composed of 500 neurons, each one using the ReLU activation function. Finally, similarly to all previous architectures, the number of neurons in the final softmax classifier is equal to the number of classes in a dataset. Unlike ResNet and FCN, this approach does not have much invariant layers (2 out of 4) due to the use of an FC layer instead of a GAP layer, thus increasing drastically the number of parameters needed to be trained which also depends on the length of the input time series.
Thus, the transferability of this network is limited to the first two convolutions whose number of parameters depends solely on the number and length of the chosen filters. We should note that t-LeNet is one of the approaches adopting a data augmentation technique to prevent overfitting especially for the relatively small time series datasets in the UCR/UEA archive. Their approach uses two data augmentation techniques: WS and Window Warping (WW). The former method is identical to MCNN's data augmentation technique originally proposed in  Cui et al. (2016). As for the second data augmentation technique, WW employs a warping technique that squeezes or dilates the time series. In order to deal with multi-length time series the WS method is adopted to ensure that subsequences of the same length are extracted for training the network.
Multi Channel Deep Convolutional Neural Network Multi Channel Deep Convolutional Neural Network (MCDCNN) was originally proposed and validated on two multivariate time series datasets  (Zheng et al. , 2014 (Zheng et al. , , 2016)). The proposed architecture is mainly a traditional deep CNN with one modification for MTS data: the convolutions are applied independently (in parallel) on each dimension (or channel) of the input MTS. Each dimension for an input MTS will go through two convolutional stages with 8 filters of length 5 with ReLU as the activation function. Each convolution is followed by a max pooling operation of length 2. The output of the second convolutional stage for all dimensions is concatenated over the channels axis and then fed to an FC layer with 732 neurons with ReLU as the activation function.
Another unique characteristic of this architecture is that the final classifier is fully connected directly to the output of the second convolution, which removes completely the GAP layer without replacing it with an FC non-linear layer. The network is composed of two consecutive convolutional layers with respectively 6 and 12 filters followed by a local average pooling operation of length 3. The convolutions adopt the sigmoid as the activation function. The network's output consists of an FC layer with a number of neurons equal to the number of classes in the dataset.
During test time, for each element of an input test time series, the already trained Ridge classifier will output a probability distribution over the classes in a dataset. Then the a posteriori probability for each class is averaged over all time series elements, thus assigning for each input test time series the label for which the averaged probability is maximum. Following the original paper of  Tanisaro and Heidemann (2016) , using a grid-search on an unseen split (20%) from the training set, we optimized TWIESN's three hyperparameters: the reservoir's size, sparsity and spectral radius.
Experimental setup We first start by presenting the datasets' properties we have adopted in this empirical study. We then describe in details our developed open-source framework of deep learning for time series classification. In order to have a thorough and fair experimental evaluation of all approaches, we tested each algorithm on the whole UCR/UEA archive  (Chen et al. , 2015b; Bagnall et al. , 2017)  which contains 85 univariate time series datasets. The datasets possess different varying characteristics such as the length of the series which has a minimum value of 24 for the ItalyPowerDemand dataset and a maximum equal to 2,709 for the HandOutLines dataset. One important characteristic that could impact the DNNs' accuracy is the size of the training set which varies between 16 and 8926 for respectively DiatomSizeReduction and ElectricDevices datasets.
We should note that twenty datasets contains a relatively small training set (50 or fewer instances) which surprisingly was not an impediment for obtaining high accuracy when applying a very deep architecture such as ResNet. Furthermore, the number of classes varies between 2 (for 31 datasets) and 60 (for the ShapesAll dataset). Note that the time series in this archive are already z-normalized  (Bagnall et al. , 2017). Other than the fact of being publicly available, the choice of validating on the UCR/UEA archive is motivated by having datasets from different domains which have been broken down into seven different categories (Image Outline, Sensor Readings, Motion Capture, Spectrographs, ECG, Electric Devices and Simulated Data) in  Bagnall et al. (2017). Further statistics, which we do not repeat for brevity, were conducted on the UCR/UEA archive in  Bagnall et al.
Multivariate archive We also evaluated all deep learning models on Baydogan's archive  (Baydogan, 2015)  that contains 13 MTS classification datasets. For memory usage limitations over a single GPU, we left the MTS dataset Performance Measurement System (PeMS) out of our experimentations. This archive also exhibits datasets with different characteristics such as the length of the time series which, unlike the UCR/UEA archive, varies among the same dataset. This is due to the fact that the datasets in the UCR/UEA archive are already re-scaled to have an equal length among one dataset  (Bagnall et al. , 2017). In order to solve the problem of unequal length time series in the MTS archive we decided to linearly interpolate the time series of each dimension for every given MTS, thus each time series will have a length equal to the longest time series' length.
This form of pre-processing has also been used by  Ratanamahatana and Keogh (2005)  to show that the length of a time series is not an issue for TSC problems. This step is very important for deep learning models whose architecture depends on the length of the input time series (such as a MLP) and for parallel computation over the GPUs. We did not z-normalize any time series, but we emphasize that this traditional pre-processing step  (Bagnall et al. , 2017)  should be further studied for univariate as well as multivariate data, especially since normalization is known to have a huge effect on DNNs' learning capabilities  (Zhang et al. , 2017). Note that this process is only true for the MTS datasets whereas for the univariate benchmark, the time series are already z-normalized. Since the data is pre-processed using the same technique for all nine classifiers, we can safely say, to some extent, that the accuracy improvement of certain models can be solely attributed to the model itself.
Experiments For each dataset in both archives (97 datasets in total), we have trained the nine deep learning models (presented in the previous Section) with 10 different runs each. Each run uses the same original train/test split in the archive but with a different random weight initialization, which enables us to take the mean accuracy over the 10 runs in order to reduce the bias due to the weights' initial values. In total, we have performed 8730 experiments for the 85 univariate and 12 multivariate TSC datasets. Thus, given the huge number of models that needed to be trained, we ran our experiments on a cluster of 60 GPUs. These GPUs were a mix of four types of Nvidia graphic cards: GTX 1080 Ti, Tesla K20, K40 and K80. The total sequential running time was approximately 100 days, that is if the computation has been done on a single GPU.
Results for univariate time series We provide on the companion GitHub repository the raw accuracies over the 10 runs for the nine deep learning models we have tested on the 85 univariate time series datasets: the UCR/UEA archive  (Chen et al. , 2015b; Bagnall et al. , 2017). The corresponding critical difference diagram is shown in Figure  7. The ResNet significantly outperforms the other approaches with an average rank of almost 2. ResNet wins on 50 problems out of 85 and significantly outperforms the FCN architecture. This is in contrast to the original paper's results where FCN was found to outperform ResNet on 18 out of 44 datasets, which shows the importance of validating on a larger archive in order to have a robust statistical significance. We believe that the success of ResNet is highly due to its deep flexible architecture.
First of all, our findings are in agreement with the deep learning for computer vision literature where deeper neural networks are much more successful than shallower architectures  (He et al. , 2016). In fact, in a space of 4 years, neural networks went from 7 layers in AlexNet 2012  (Krizhevsky et al. , 2012)  to 1000 layers for ResNet 2016  (He et al. , 2016). These types of deep architectures generally need a huge amount of data in order to generalize well on unseen examples  (He et al. , 2016). Although the datasets used in our experiments are relatively small compared to the billions of labeled images (such as ImageNet  (Russakovsky et al. , 2015)  and OpenImages  (Krasin et al. , 2017)  challenges), the deepest networks did reach competitive accuracies on the UCR/UEA archive benchmark.
We give two potential reasons for this high generalization capabilities of deep CNNs on the TSC tasks. First, having seen the success of convolutions in classification tasks that require learning features that are spatially invariant in a two dimensional space (such as width and height in images), it is only natural to think that discovering patterns in a one dimensional space (time) should be an easier task for CNNs thus requiring less data to learn from. The other more direct reason behind the high accuracies of deep CNNs on time series data is its success in other sequential data such as speech recognition  (Hinton et al. , 2012)  and sentence classification  (Kim, 2014)  where text and audio, similarly to time series data, exhibit a natural temporal ordering. The MCNN and t-LeNet architectures yielded very low accuracies with only one win for the Earthquakes dataset.
Although MCDCNN and Time-CNN were originally proposed to classify MTS datasets, we have evaluated them on the univariate UCR/UEA archive. The MCDCNN did not manage to beat any of the classifiers except for the ECG5000 dataset which is already a dataset where almost all approaches reached the highest accuracy. This low performance is probably due to the non-linear FC layer that replaces the GAP pooling of the best performing algorithms  (FCN and ResNet). This FC layer reduces the effect of learning time invariant features which explains why MLP, Time-CNN and MCDCNN exhibit very similar performance. One approach that shows relatively high accuracy is Encoder  (Serrà et al. , 2018). The statistical test indicates a significant difference between Encoder, FCN and ResNet. FCN wins on 36 datasets whereases Encoder wins only on 17 which suggests the superiority of the GAP layer compared to Encoder's attention mechanism.
Note that for the ten different random initializations of ResNet, the pairwise statistical test always failed to find any significance between ResNet and COTE/HIVE-COTE. PF, ST, BOSS and ResNet showed similar performances according to the Wilcoxon signed-rank test, but the fact that ResNet is not significantly different than COTE suggests that more datasets would give a better insight into these performances  (Demšar, 2006). NN-DTW-WW and EE showed the lowest average rank suggesting that these methods are no longer competitive with current state-of-the-art algorithms for TSC. It is worthwhile noting that cliques formed by the Wilcoxon Signed Rank Test with Holm's alpha correction do not necessary reflect the rank order  (Lines et al. , 2018). For example, if we have three classifiers (C 1 , C 2 , C 3 ) with average ranks (C 1 > C 2 > C 3 ), one can still encounter a case where C 1 is not significantly worse than C 2 and C 3 with C 2 and C 3 being significantly different.