dataesr/scientific-paragraphs-categorization huggingface dataset:
Multivariate Time Series (MTS) classification has gained importance over the past decade with the increase in the number of temporal datasets in multiple domains. The current state-of-the-art MTS classifier is a heavyweight deep learning approach, which outperforms the second-best MTS classifier only on large datasets. Moreover, this deep learning approach cannot provide faithful explanations as it relies on post hoc model-agnostic explainability methods, which could prevent its use in numerous applications. In this paper, we present XCM, an eXplainable Convolutional neural network for MTS classification. XCM is a new compact convolutional neural network which extracts information relative to the observed variables and time directly from the input data. Thus, XCM architecture enables a good generalization ability on both large and small datasets, while allowing the full exploitation of a faithful post hoc model-specific explainability method (Gradient-weighted Class Activation Mapping) by precisely identifying the observed variables and timestamps of the input data that are important for predictions.
We first show that XCM outperforms the state-of-the-art MTS classifiers on both the large and small public UEA datasets. Then, we illustrate how XCM reconciles performance and explainability on a synthetic dataset and show that XCM enables a more precise identification of the regions of the input data that are important for predictions compared to the current deep learning MTS classifier also providing faithful explainability. Finally, we present how XCM can outperform the current most accurate state-of-the-art algorithm on a real-world application while enhancing explainability by providing faithful and more informative explanations.
However, MLSTM-FCN outperforms the second-best MTS classifier (Bag-of-Words method WEASEL+MUSE  [6] ) only on the large datasets (relatively to the public UEA archive  [7] -training set size ≥ 500). This deep learning approach contains a significant number of trainable parameters, which could be an important reason for its poor performance on small datasets. Moreover, for many applications, the adoption of machine learning methods cannot rely solely on their prediction performance. For example, the European Union's General Data Protection Regulation (GDPR), which became enforceable on 25 May 2018, introduces a right to explanation for all individuals so that they can obtain "meaningful explanations of the logic involved" when automated decision making has "legal effects" on individuals or similarly "significantly affecting" them (https://ec.
europa. eu/info/law/law-topic/data-protection_en (accessed on 1 November 2021)). As far as we have seen, an architecture concatenating a LSTM network with a CNN such as MLSTM-FCN, or a classifier based on unigrams/bigrams extraction following a Symbolic Fourier Approximation  [8]  such as WEASEL+MUSE, cannot provide perfectly faithful explanations as they rely solely on post hoc model-agnostic explainability methods  [9] , which could prevent their use in numerous applications. Faithfulness is critical as it corresponds to the level of trust an end-user can have in the explanations of model predictions, i. e. , the level of relatedness of the explanations to what the model actually computes. Hence, we propose a new compact (in terms of the number of parameters) and explainable deep learning approach for MTS classification that performs well on both large and small datasets while providing faithful explanations.
CNNs along with post hoc model-specific saliency methods such as Gradient-weighted Class Activation Mapping-Grad-CAM  [10] -have the potential to have a compact architecture while enabling faithful explanations  [11]. A recent CNN, MTEX-CNN  [12]  proposes using 2D and 1D convolution filters in sequence to extract key MTS information, i. e. , information relative to the observed variables and time, respectively. However, as confirmed by our experiments, the features related to time which are extracted from the output features of the first stage (relative to each observed variable) cannot fully incorporate the timing information from the input data and subsequently yield poor performance compared to the state-of-the-art MTS classifiers. In addition, the significant number of trainable parameters of MTEX-CNN affects its generalization ability on small datasets.
Finally, MTEX-CNN requires upsampling processes on feature maps when applying Grad-CAM, which can lead to an imprecise identification of the regions of the input data that are important for predictions. Therefore, we propose a new faithfully eXplainable CNN method for MTS classification (XCM) which improves MTEX-CNN in three substantial ways: (i) it generates features by extracting information relative to the observed variables and timestamps in parallel and directly from the input data; (ii) it enhances the generalization ability by adopting a compact architecture (in terms of the number of parameters); and (iii) it allows precise identification of the observed variables and timestamps of the input data that are important for predictions by avoiding upsampling processes. Summarizing our main contributions:.
• We show that XCM outperforms the state-of-the-art MTS classifiers on both the large and small UEA datasets  [7] ; • We illustrate on a synthetic dataset that XCM enables a more precise identification of the regions of the input data that are important for predictions compared to the current faithfully explainable deep learning MTS classifier MTEX-CNN; • We show that XCM outperforms the current most accurate state-of-the-art algorithm on a real-world application while enhancing explainability by providing faithful and more informative explanations. The rest of this paper is organized as follows: Section 2 presents the related work concerning MTS classification and explainability; Section 3 details XCM architecture; Section 4 presents our evaluation method; and finally, Section 5 discusses our results.
Background We address the issue of supervised learning for classification. Classification consists of learning a function that maps an input data to its label: given an input space X, an output space Y, an unknown distribution P over X × Y, a training set sampled from P, and a 0-1 loss function 0-1 compute function h * as follows: h * = arg min h E (x,y)∼P [ 0-1 (h, (x, y))] (1) In this study, classification is performed on multivariate time series datasets. A Multivariate Time Series (MTS) M = {x 1 ,. , x d } ∈ R d * l is an ordered sequence of d ∈ N streams with x i = (x i,1 ,. , x i,l ), where l is the length of the time series and d is the number of multivariate dimensions. We address MTS generated from automatic sensors with a fixed and synchronized sampling along all dimensions. An example of an MTS with two dimensions and a length of 100 is given at the top of Figure  5.
DTW D uses a similar calculation with single-dimensional time series; it considers the squared Euclidean accumulated distance over the multiple dimensions. Next, feature-based methods can be categorized into two families: shapelet-based and Bag-of-Words (BoW) classifiers. Shapelets models (gRSF  [21]  and UFS  [22] ) use subsequences (shapelets) to transform the original time series into a lower-dimensional space that is easier to classify. On the other hand, BoW models (LPS  [23] , mv-ARF  [24] , SMTS  [25]  and WEASEL+MUSE  [6] ) convert time series into a bag of discrete words and use a histogram of words representation to perform the classification. WEASEL+ MUSE shows better results compared to gRSF, LPS, mv-ARF, SMTS and UFS on average (20 MTS datasets). WEASEL+MUSE generates a BoW representation by applying various sliding windows with different sizes on each discretized dimension (Symbolic Fourier Approximation) to capture features (unigrams, bigrams and dimension identification).
Following a feature selection with chi-square test, it classifies the MTS based on a logistic regression. Finally, deep learning methods (FCN  [26] , MLSTM-FCN  [5] , MTEX-CNN  [12] , ResNet  [27] , TapNet  [28]  and TST  [29] ) use Long-Short Term Memory (LSTM), Convolutional Neural Networks (CNN) or Transformers. According to the results published and our experiments, the current state-of-the-art model (MLSTM-FCN) is proposed in  [5]  and consists of a LSTM layer and a stacked CNN layer along with squeeze-and-excitation blocks to generate latent features. A recent network, TapNet  [28] , also consists of a LSTM layer and a stacked CNN layer, followed by an attentional prototype network. However, TapNet shows lower accuracy results (https://github. com/xuczhang/xuczhang. github. io/blob/ master/papers/aaai20_tapnet_full.
pdf (accessed on 1 November 2021)) on average on the 30 public UEA MTS datasets compared to MLSTM-FCN (MLSTM-FCN results presented in Table  3 ). There is no basis of comparison for MLSTM-FCN with MTEX-CNN  [12]  as MTEX-CNN has not been evaluated on public datasets. As illustrated in Figure  1 , MTEX-CNN is a two-stage CNN network which first extracts information relative to each feature with 2D convolution filters and then extracts information relative to time with 1D convolution filters. The output feature map is fed into fully connected layers for classification. Therefore, in this work, we choose to benchmark XCM to the best-in-class for each similarity-based, feature-based and deep learning category (DTW D /DTW I , WEASEL+ MUSE and MLSTM-FCN classifiers). We also include MTEX-CNN in the benchmark to demonstrate the superiority of our approach as MTEX-CNN has not been evaluated on the public UEA datasets.
Explainability In addition to their prediction performance, machine learning methods have to be assessed on how they can support their decisions with explanations. Two levels of explanations are generally distinguished: global and local  [30]. Global explainability means that explanations concern the overall behavior of the model across the full dataset, while local explainability informs the user about a particular prediction. As previously introduced with the example of the GDPR, our new CNN approach needs to be able to support each individual prediction. Thus, we present in this section the local explainability methods for CNNs. CNNs classifiers do not provide explainability-by-design at the local level. Thus, some post hoc model-agnostic explainability methods could be used. These methods provide explanations for any machine learning model.
However, the precision of the explanations provided by Grad-CAM, i. e. , the fraction of explanations that are relevant to a prediction, can vary across CNN architectures as Grad-CAM is sensitive to the upsampling processes on feature maps to match the input data dimensions. Therefore, we support the predictions of our new CNN model XCM with Grad-CAM, a post hoc model-specific explainability method which provides faithful explanations at local level. The design of our network architecture avoids upsampling processes and enables Grad-CAM to identify the observed variables and timestamps of the input data that are important for predictions more precisely as compared to what the current explainable deep learning MTS classifier MTEX-CNN give. Table  1  presents an overview of the challenges addressed by the state-of-the-art MTS classifiers and how we position our new method XCM.
Architecture Our approach aims to design a new compact and explainable CNN architecture that performs well on both the large and small UEA datasets. As illustrated in Figure  1 , a recent explainable CNN, MTEX-CNN  [12] , proposes to use 2D and 1D convolution filters in sequence to extract key MTS information, i. e. , information relative to the observed variables and time, respectively. However, CNN architectures such as MTEX-CNN have significant limitations. The use of 2D and 1D convolution filters in sequence means that the features related to time (features maps from 1D convolution filters) are extracted from the processed features related to observed variables (features maps from 2D convolution filters). Therefore, features related to time cannot fully incorporate the timing information from the input data and can only partially reflect the necessary information to discriminate between the different classes.
We then upsample it to match the input data dimensions T × D with a bilinear interpolation in order to obtain the attribution map. This operation does not alter the time attribution results as the padding on the 1D convolution filters ensured that the feature extraction over the time dimension has kept the time series length. Therefore, the upsampling only replicates the results over the observed variables. An example of observed variables and time attribution maps on a synthetic dataset is presented in Section 5. 2. Before discussing the performance and explainability results of XCM, we present in the next section the evaluation setting.
Algorithms We compare our algorithm XCM implemented in Python 3. 6 (code available on GitHub https://github. com/XAIseries/XCM) to the state-of-the-art MTS classifiers, as detailed in Section 2. 2, and to the variant XCM-Seq: ). We used the same setting as XCM. All the networks that we implemented (XCM, XCM-Seq and MTEX-CNN) were trained with 100 epochs, the categorical crossentropy loss and the Adam optimization (computing infrastructure: Debian 8 operating system, GPU NVIDIA GeForce RTX 2080 Ti with 11Gb GRAM and 96Gb of RAM). Concerning Grad-CAM, we used the implementation available for Keras (https://github. com/jacobgil/keras-grad-cam (accessed on 1 November 2021)).
Metrics For each dataset, we computed the classification accuracy-the metric used to benchmark the MTS classifiers on the public UEA datasets  [7]. Then, we presented the average rank and the number of wins/ties to compare the different classifiers on the same datasets. Finally, we presented the critical difference diagram  [47] , the statistical comparison of multiple classifiers on multiple datasets based on the nonparametric Friedman test, to show the overall performance of XCM. We used the implementation available in R package scmamp (https://www. rdocumentation. org/packages/scmamp/versions/0. 2. 55/topics/ plotCD (accessed on 1 November 2021)).
Performance The accuracy results on the public UEA test sets of XCM and the other MTS classifiers are presented in Table  3. A blank in the table indicates that the approach ran out of memory. The best accuracy for each dataset is denoted in boldface. Firstly, we observe that XCM obtains the best average rank and the lowest rank variability across the datasets (rank: 2. 3, standard error: 0. 4), followed by MLSTM-FCN in second position (rank: 3. 5, standard error: 0. 5) and WEASEL+MUSE in third position (rank: 4. 0, standard error: 0. 5). Using the categorization of the datasets published in the archive website (http://www. timeseriesclassification. com/dataset. php (accessed on 1 November 2021)), we do not see any influence from the different train set sizes, MTS lengths, number of dimensions, number of classes and dataset types on XCM performance relative to the other classifiers on the UEA datasets.
More specifically, XCM exhibits better performance than MLSTM-FCN and WEASEL +MUSE on both the large (rank: 1. 9, MLSTM-FCN rank: 2. 1, WEASEL+MUSE rank: 4. 6train size ≥ 500, 23% of the datasets) and small datasets (rank: 2. 4, MLSTM-FCN rank: 4. 0, WEASEL+MUSE rank: 3. 9-train size < 500, 77% of the datasets). We can assume that the more compact architecture of XCM compared to the other deep learning classifiers provides a better generalization ability on the UEA datasets (average rank on the number of trainable parameters: XCM 1. 7, MLSTM-FCN: 1. 9, MTEX-CNN: 2. 0). Furthermore, the results confirm the superiority of the XCM approach based on the extraction in parallel and directly from the input data of features relative to the observed variables and time compared to the sequential approaches.
XCM outperforms both XCM-Seq and MTEX-CNN on average on the UEA datasets (rank: 2. 3, XCM-Seq: 5. 0, MTEX-CNN: 7. 2). With regard to the hyperparameter Window Size of XCM, Figure  3  shows the average relative drop in performance across the datasets when using the other time window sizes than the one used in the best configuration given in Table  3. In order to evaluate the relative impact with respect to the range of performance, we defined four categories of datasets: datasets with XCM original accuracy< 50%, datasets with 50% ≤ accuracy < 75%, datasets with 75% ≤ accuracy < 90% and datasets with accuracy ≥ 90%. First, as expected, we observe that the average relative impact of using suboptimal time window sizes is higher when XCM level of performance is low (average relative drop in accuracy: 13.
1% when XCM accuracy < 50% versus 3. 0% when XCM accuracy ≥ 90%). Then, the average relative drop in accuracy when using suboptimal time window sizes is not negligible but remains limited in all the cases. This drop is below 15% on average on the category where XCM has the lowest level of accuracy (13. 1% ± 3. 2%) and below 10% on average across all the datasets (7. 0% ± 1. 3%). 3. The performance drop is presented across four categories of datasets, defined according to XCM levels of accuracy shown in Table  3. Abbreviation: Acc-Accuracy. Finally, we performed a statistical test to evaluate the performance of XCM compared to the other MTS classifiers. We present in Figure  4  the critical difference plot with alpha equals to 0. 05 from results shown in Table  3. The values correspond to the average rank, and the classifiers linked by a bar do not have a statistically significant difference.
Explainability In this section, we illustrate how our approach XCM reconciles performance and explainability and show that XCM enables a more precise identification of the regions of the input data that are important for predictions compared to the current deep learning MTS classifier also providing faithful explainability-MTEX-CNN. We perform the comparison on a synthetic dataset as the construction of such a dataset allows us to know the expected explanations, with such information not being available in the public UEA datasets. Concerning the evaluation of the results, we adopt the intersection-over-union as a metric, i. e. , the extent of overlap between the predicted and expected explanations. The synthetic dataset is composed of 20 MTS (50%/50% train/test split) with a length of 100, two dimensions, and two balanced classes.
The offline dataset consists of 15. 5k MTS samples of length 4 with seven variables: the body temperature variable and six activity variables (rumination, ingestion, rest, standing up, over activity and other activity). A time series corresponds to a 4 day period (MTS length 4): the day of estrus (Day 0) and the previous 3 days. The labels are set with the ground truth in estrus detection-progesterone dosage in whole milk. We compare XCM with Grad-CAM to a reference commercial solution (HeatPhone  [53] ) and the most accurate state-of-the-art MTS classifier of our benchmark (see Section 4. 2) on this real-world application-MLSTM-FCN-with SHAP  [32]. As far as we have seen, an architecture concatenating a LSTM network with a CNN such as MLSTM-FCN can only rely on post hoc model-agnostic explainability methods to support its predictions.
Conclusions We have presented XCM, a new compact and explainable convolutional neural network for MTS classification, which extracts information relative to the observed variables and time directly from the input data. XCM exhibits a better average rank than the state-of-theart classifiers on both the large and small public UEA datasets. Moreover, it was designed to enable faithful explainability based on Grad-CAM method and the precise identification of the regions of the input data that are important for predictions. Following the illustration of the performance and explainability of XCM on a synthetic dataset, we showed how XCM can outperform the current most accurate state-of-the-art algorithm MLSTM-FCN on a real-world application while enhancing explainability by providing faithful and more informative explanations.
The complex intrinsic and extrinsic forces from the body and environment push the brain into nonequilibrium. The arrow of time, central to thermodynamics in physics, is a hallmark of non-equilibrium and serves to distinguish between reversible and non-reversible dynamics in any system. Here, we use a deep learning Temporal Evolution NETwork (TENET) framework to discover the asymmetry in the flow of events, 'arrow of time', in human brain signals, which provides a quantification of how the brain is driven by the interplay of the environment and internal processes. Specifically, we show in large-scale HCP neuroimaging data from a thousand participants that the levels of nonreversibility/non-equilibrium change across time and cognitive state with higher levels during tasks than when resting. The level of non-equilibrium also differentiates brain activity during the seven different cognitive tasks.
Furthermore, using the large-scale UCLA neuroimaging dataset of 265 participants, we show that the TENET framework can distinguish with high specificity and sensitivity resting state in control and different neuropsychiatric diseases (schizophrenia, bipolar disorders and ADHD) with higher levels of non-equilibrium found in health. Overall, the present thermodynamicsbased machine learning framework provides vital new insights into the fundamental tenets of brain dynamics for orchestrating the interactions between behaviour and brain in complex environments.
Specifically, for whole-brain data we extracted the normal forward timeseries as well as constructing the time reversal of the backward timeseries for a given parcellation. This procedure provides a clear arrow of time for a given timeseries for which a deep learning classifier (here named Temporal Evolution NETwork, TENET) could be trained to identify the forward and the time reversal for a given timeseries of any length. The performance of TENET provides a reliable measure of the degree of non-reversibility and non-equilibrium at different levels of global and local brain organisation. We used TENET and the concept of reversibility in thermodynamics to assess the level of nonequilibrium and arrow of time in brain dynamics in resting and seven different tasks from the large HCP neuroimaging fMRI dataset of 1003 healthy human participants.
We also used the same approach to study time's arrow in health and disease in a UCLA dataset of 265 neuropsychiatric patients (ADHD, schizophrenia and bipolar disorder as well as controls). The TENET framework turns out to be highly capable of distinguishing different tasks and different diseases and could potentially serve as a biomarker in future. Overall, the key idea of estimating the brain's arrow of time provides the basis for a detailed analysis of the non-stationary and non-equilibrium nature of brain dynamics in health and disease. As such, the present framework and findings provide vital new insights into the fundamental tenets of how the level of non-equilibrium of brain dynamics directly reflects the interactions with the environment, in other words how the environment drives the brain to survive and thrive.
Significant differences in global levels of non-equilibrium/non-reversibility in HCP rest and seven tasks For the global level of analysis of how the environment is driving the brain out of equilibrium, we extracted BOLD time series from the DK80 parcellation covering the whole brain in rest and the seven tasks. For each of HCP participant, we extracted forward and backward patterns in sliding windows with a length of 20 TRs (of 0. 72sec), which were then shifted 3 TRs forward. Each of the sliding windows consisted of two input patterns containing 1) forward and 2) time reversed backward sliding windowed timeseries, which was each labelled with an output class label of forward and backward, respectively. For the training of TENET, in order to perfectly balance the data and avoid any potential source of bias, we used 890 HCP participants with the longest possible duration available in all conditions (176 TRs).