Shedding Light on Time Series Classification using Interpretability Gated Networks
Authors: Yunshi Wen
Tengfei Ma
Ronny Luss
Debarun Bhattacharjya
Achille Fokoue
Anak Agung Julius
Venue: ICLR 2025 Poster (2025)
Keywords: Interpretability
Time-series
Shapelet
Primary area: interpretability and explainable AI
TL;DR: A framework to integrate interpretable models with deep neural networks for interpretable time-series classification.

Abstract: In time-series classification, interpretable models can bring additional insights but be outperformed by deep models since human-understandable features have limited expressivity and flexibility. In this work, we present InterpGN, a framework that integrates an interpretable model and a deep neural network. Within this framework, we introduce a novel gating function design based on the confidence of the interpretable expert, preserving interpretability for samples where interpretable features are significant while also identifying samples that require additional expertise. For the interpretable expert, we incorporate shapelets to effectively model shape-level features for time-series data. We introduce a variant of Shapelet Transforms to build logical predicates using shapelets. Our proposed model achieves comparable performance with state-of-the-art deep learning models while additionally providing interpretable classifiers for various benchmark datasets. We further show that our models improve on quantitative shapelet quality and interpretability metrics over existing shapelet-learning formulations. Finally, we show that our models can integrate additional advanced architectures and be applied to real-world tasks beyond standard benchmarks such as the MIMIC-III and time series extrinsic regression datasets.

Published as a conference paper at ICLR 2025
SHEDDING LIGHT ON TIME SERIES CLASSIFICATION USING INTERPRETABILITY GATED NETWORKS
Yunshi Wen'; Tengfei Ma²; Ronny Luss?, Debarun Bhattachariya', Achille Fokoue", Agung Julius'
Rensselaer Polytechnic Institute, Stony Brook University, 3IBM Research 1nyy2erpi.edu. tengfei ii...... ma@ st onybrook. edu {rluss, debarunb, achille S ibm. com sgung@ecse.rpi.edu edi
ABSTRACT
In time-series classification, interpretable models can bring additional insights but be outperformed by deep models since uuman-understandablee features have limited expressivity and flexibility. In this work, we present InterpGNN, a framework that in- tegrates an interpretable model and deep neural network. Within this framework, we introduce a novel gating function design based on the confidence of the inter- pretable expert, preserving interpretability for samples where interpretable features are significant while also identifying samples that require additional expertise, For the interpretable expert, we incorporate shapelets to effectively model shape-level features for time-series data. We introduce variant of Shapelet Transforms to build logical predicates using shapelets. Our proposed model achieves comparable performance with state-of-the art deep learning models while additionally provid- ing interpretable classifiers for various benchmark datasets. We further show that models improve on quantitative shapelet quality and interpretability metrics over existing shapelet-leariin formulations. Finally, we show that our models can integrate additional advanced architectures and be applied to real-world tasks beyond standard benchmarks such as the MIMIC-III and time series extrinsic regression datasets.
INTRODUCTION
Time series classification is fundamental task for time series (TS) data. Depending the number of observed variables, TS classification problems can be categorized as either univariate or multivariate. Both categories are involved in wide range of applications. Univariate TS data includes image outlines, sound, and spectrographs, and multivariate TS data include electroencephalogramn (EEG), electrocardiogram (ECG), and human activity recognition (HAR). These types of TS data appear in essential applications such as healthcare, neuroscience, and automation. In recent years, there has been an increasing trend of employing deep learning models to TS classification problems. However, despite achieving state-of-the-art performance, deep models typically lack interpretability.
As motivating example for interpretable TS classification, consider the problem of classifying ECG data using machine learning (Neves et al. 2021] Abdullah et al.) [2033] Aziz et al.) (02T). While models might be accurate, doctors may hesitate to rely on them without understanding why patient's 'ss heartbeat as measured by an ECG was classified as being indicative of reduced bloodflow to the heart (known as myocardial ischemia). Our model seeks to provide such trust by discovering classifiers based on understandable concepts, such as those illustrated in Figure (middle)) where heartbeats are classified according to the existence of patterns known as shapelets. In this example, shapelet 811, representing downward trend, and 812, representing upward trend, serve as discriminative patterns that classify normal and ischemic heartbeats. doctor may gain confidence in seeing such patterns with straightforward interpretations, such as in Figure (right), that were deemed important to the predictions.
Most of the work was done while at IBM Research. Corresponding to Yunshi Wen (weny28 @rri Arrpp.ee .edu)
and Tengfei Ma (tengfei. ma@st.nybrook edu).

---

Published as a conference paper at ICLR 2025
Shapelets
Classifier
Interpretations
Longth:
10%6
30%
50%
50%s
schenaa
SG
11.56
11,42
Local Explanation: Il lame-deriess
+0.47
F4
+1.00
bacheeria bacoatzue car Laino
Pa
(Ome small importance)
4
M
Narma samples should include 512. whie
sociaale syg and sq.
20
2. tachamia sumples should include syg and while mediadee
Figure 1: An example of shapelet-basee interpretable TS classification on the ECG200 dataset. (Left) The set of shapelets with various lengths to capture features with different scales. (Middle) A classifier that associates the shapelets with categories, assigning importance scores to them. (Right) Two methods to interpret the classifier result in rule-like explanations.
Interpretability can be significant from two perspectives. Firstly, in applications such as healtheare and physical systems where safety is essential, interpreting a model allows us to validate reliability, trustworthiness, and fairness. Secondly, in challenging tasks, we can gain knowledge about the problem by interpreting patterns found by machine learning models during training. Existing interpretable time series methods (Zhao et al. 2023) include post hoc explainability which explain the behavior of a trained deep learning model often using surrogate model (Queen et al. 2023). and self-explainable models for which model predictions are directly interpretable.
In this study, we focus self-explainable models. A recent self- self-explainable TS model, Neuro- Symbolic Time Series Classification (NSTSC) (Yan et al. 2022). combines a neuro-symbolic model (Riegel et al. 2020) with signal temporal logic (Mehdipour al. 202I); soft-logic predicates are defined on each time stamp. and logic rules are learned to describe the classes. Although such approaches are defined to be interpretable, identifying the logical rules when applied to complex tasks can result in numerous predicates and logical operators. Furthermore, it is known that interpretability may lead to lower classification performance due incomplete coverage of interpretable concepts or limited expressivity (Koh et al. 2020: Oikarinen et al.) [022] Havasi et al. 2022); such is the case for Concept Bottleneck Models (CBMs) (Koh al.] 2020). popular interpretable model in computer vision. Similarly in TS, time-domain features (such as shapes and trends) are easy to understand but may not be able to capture other essential features such as those the frequency domain.
In addition to post-hoc explanations, the literature on combining black-box models with interpretable models includes approaches such as performing logical reasoning on concepts discovered by deep models (Lee et al. 2022) using deep models to select from a set of interpretable primitives, which are then composed to form the final predictions (Okajima & Sadamasa 2019). In this paper, with the goal of maintaining interpretability when appropriate while allowing for additional complexity where concept-level information is not significant, we propose an Interpretability Gated Network (InterpGN) that combines an interpretable expert with deep neural network (DNN) expert. InterpGN uses novel gating function to decide whether to use the interpretable expert's prediction or to rely on the DNN's expertise (as likely needed on challenging samples). Our interpretable expert, termed the Shapelet Bottleneck Model (SBM), introduces a variant of the Shapelet Transform (Lines et al. 2012) build expressive yet interpretable predicates based learned shapelets (Ye & Keogh 201 such as the patterns in Figure [ (left). Our interpretable expert holds several advantages over existing methods such as fewer parameters than ceuro-symboli method NSTSC, improved shapelet selection due to our Shapelet Transform, and better accuracy due to gating on challenging samples.
Our main contributions are summarized as follows: (1) a novel gating function that assigns samples to experts based on the confidence level of the interpretable expert, (2) novel variant of the Shapelet Transform that improves interpretability compared to existing shapelet-based methods, and (3) quantitative metrics for interpretabilityy and shapelet quality, where we show that our models improve in both metrics over existing shapelet-learning formulations. We further demonstrate that InterpGN outperforms state-of-the-an methods the UEA multivariate TS classification archive (Bagnall jet al., 2018). illustrate the interpretability of InterpGN on multivariate TS classification datasets, and finally apply framework on a real-world healthcare dataset, MIMIC-III (Johnson et al. 2016).
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Machine Learning Methods for TS Analysis Classical deep learning models such as Multilayer Perceptron (MLP), Fully Convolutional Network (FCN), and Residual Network (ResNet) (Wang) et al. 2017) are usually considered as baselines. ROCKET (Dempster et al. 2020) uses random convolution kernels with various lengths to transform TS data and trains a linear classifier. Another direction is representation learning that uses unsupervised approaches to learn from positive and negative samples (Franceschi et al., 2019). For instance, TapNet (Zhang al., 2020) proposes an attention prototype network to learn multivariate TS representations and address the issue of limited labeled data. Time Series Transformer (TST) (Zerveas et al. (02T) reconstructs TS from masked data to learn pre-trained model that can be used for multiple types of downstream tasks. Recent works also find that when combined with deep neural networks such as Transformers (Vaswani et al. 2017). the patch-based modeling, which segments TS into channel-independenn patches, is more effective than tinestamp-based modeling in various tasks (Zuo et al. 2023 Nie et al. 2023) Wu et al. 2023). Although the above methods may achieve state-of-the-ans performance, they lack interpretability.
Interpretable Methods Pattern-based methods usually have some level of interpretability. (Zhao al. 2023) surveys many methods for interpretable TS classification. A subset of pattem-based methods use shapelets, defined as discriminatory subsequences of TS data (Ye & Keogh, (011)) selected based on information gain. ShapeNet (Li et al.) (02T) learns embeddings that encode subsequences with different lengths into unified space using triplet loss, and then selects the most representative shapelets to use with Shapelet Transforms. Shapelets can also be learned using gradient-based methods (Grabocka er al., 2014), relaxing the definition of shapelets as sequences with certain patterns. However, with the original formulation, the learned shapelets have poor quality as they may not look similar to the actual subsequences. ADSN (Ma et al., 2020) improves shapelet quality using adversarial training. ShapeConv (Qu et al.) 2024) initializes the shapelets as the cluster means of TS subsequence and uses additional regulari eggulariatiin to keep shapelets close to the actual subsequence during training. Besides shapelet-basee methods, a recent pattern-based method RLPAM (Gao et al. 2022) first transform TS to sequence of patterns and uses reinforcement learning to select the most informative patterns for classification.
3 PROBLEM FORMULATIONS
Notation Let = {1 Dilli = .,....N) N} denotes a TS classification dataset with N samples, where x; E MxT is a TS sample and y E {I...., . C} the class label. Here, M the number of variables, T' is the length in timestamps, and C' is the number of categories. A multivariate TS sample 2; can be viewed as set of univariate TS samples where x;"h e Rº denotes the TS of the mh variable. :i,4, denotes a subsequence of x;" between timestamps l and fz.
Shapelets Shapelets were originally defined as discriminative TS subsequences to distinguish different categories (Ye Keogh 201T). Learning Time-series Shapelets (LTS) (Grabocka et al. 2014) combines Shapelet Transform with gradient- based optimization to learn a set of shapelets from scratch, where the definition of shapelets is relaxed be short sequences with certain patterns. In this work, we follow the definition of LTS where the shapelets are the model's parameter to train.
Shapelet-based TS Modeling Independently modeling each channel of multivariate TS has been demonstrated be effective approach (Wu et al. 2023 Nie et al. 2023). In this paper, we consider a similar approach by formulating univariate shapelets for each variable separately. Let
S = (sx
m. lk 1.... K;m 1,..., M;l E T.)
denote a set of shapelets, where s IR denotes a shapelet on the mh channel with length / and index k. i.e., we learn K shapelets for each possible length and channel. In this paper, we consider shapelets with multiple lengths where L = {max([oT") 3,3) 0.05, 0.1, 0.2, 0.3, 0.5, 0.8}. Therefore, S has total of M K shapelets.
Interpretability Existing methods gain different levels of interpretability by inputting interpretable
features (Zuo al. 2023) into simple model such linear layer (Ma et al. 2020 Qu al. 2024
3

---

Published as a conference paper at ICLR 2025
Interpeetbble Shapelet Botteeeek Model
Shapelats
Hyard
Lus
by
Predicanss
Contidence
Channel Norm.
Pi 1r7e:
Ft
Gasing
a, R
Deep Neural Network Expert
Figure 2: Overview of InterpGN.
or SVM (Li et al., 2021). However, such approaches usually fail to provide explanations of their predictions based on distances features. For the interpretable expert, we build logical predicates using shapelets and the classifier directly provides rule-like explanations. Combining interpretable features with logic allows this approach to select essential shapelets since model weights associated with shapelets represent corresponding importance to predictions (Riegel et al. 2020).
4 THE MIXTURE-OF-ExeI
FRAMEWORK: INTERPGN
In this section, we present the details of Interpretability Gated Networks (InterpGN). We first discuss the proposed variant of Shapelet Transforms to build logical predicates and the formulations of the Shapelet Bottleneck Model (SBM), and then discuss how explanations can be derived from SBM. Lastly, we present the novel gating function used by InterpGNN along with the training procedure. An overview of InterpGN is given in Figure
4.1 BUILDING LOGICAL PREDICATES USING SHAPELETS
A straightforward way to express predictions with logical classifiers is to predicates built on fea- tures such as distances. Mueen et al. (201T) con- structed logical predicate with threshold dis- tance; however, when combined with LTS, we find this approach reduces shapelet quality (essential shapelets are not similar to the TS) as well as in- terpretability (notion of distance can vary across datasets) since the threshold distance does not di- rectly reflect the existence of shapelet (see Figure 3 for example). Therefore, analogous to con- cepts in CBM, we view shapelets as interpretable concepts for TS and build logical predicates directly from shapelet distances. Consider the distance between
RBF-based
Threshold-basee based
w
22
100
200
300
100
300
Figure 3: Comparison between shapelets
learned by (left) radial basis function (RBF) based predicates and (right) threshold based predicates. The three most important shapelets are visualized.
8% m,d and x: defined as:
min
dist(s m. <(ie)))
where di st
m m ( the Euclidean distance between the two sequences. We introduce a variant the Shapelet Transform, formally defined by
md max e- (e-dist(eM) (a = P.k :=
(2)
using a Gaussian radial basis function on the distance to measure the likelihood that s exists in wi where is scaling parameter that controls the steepness of the kernel. We say that the
4

---

Published as a conference paper at ICLR 2025
set of shapelets & transforms each sample I; E D to predicates Pa E R RMKI where p: m.d
PaR
k 1, K; m = mon==,... I,...,,;;; M; (ELE
4.2 SHAPELET BOTTLENECK MODEL- THE INTERPRETABLE EXPERT
When constructing the SBM, we adopt similar approach LTS and CBM preserve interpretability and produce rule-like classifiers. Formally, the predicates p; are fed into a linear layer to compute the outputs
Ti.c
Σ
mdd mnd We PER Vc E {l.....C).
(3)
Jom 2..., Kmm MHEL.
The SBM's output is r, RCC [ri, 1i,1,.... ric). An overview of SBM given in Figure 2 Note that the different shapelet lengths are illustrated vertically under the Shapelet Bottleneck; each row allows shapelets of length I across all variables.
Training SBM is end-to-end differentiable and trained using gradient-based optimization. The m.d and the shapelets % ... both of 2 can be
parameters to be trained include the linear classifier weights w'ckk which are randomly initialized. In practice, gradients of the max operator in Equation substituted by gradients of so soft using straight-throus estimation. The objective function contains the following three loss functions:
Classification loss: softmax cross- cross-entropy loss Lee on (ri, Vi),
Shapelet diversity loss (Ma et al. 2020) to regulate learning redundant shapelets:
Laiv
ΣΣΣ
-1l - -
2K T.
L.1 regularization freg on the classifier weights w m.d tive concepts, and produce simple classifiers.
wo to encourage sparsity, select the most informa-
The overall loss function is Lim = Lee + Can + Areg where Adiv, and A are hyperparameters.
4.3 INTERPRETATIONS
m.. m. is logical predicate whose value measures the existence of shapelet sx A concept probability Pi,
The linear classifier behaves like weighted linear logic analogous to the operators in Riegel et al. (2020), where weights wa,k specify the importance of shapelet s class We propose two ways interpret the model.
Local Explainability Local explanations answer why the classification decision for a sample x; made. In SBM, the explanation has the form of "sample x; belongs class because o; contains m. m.d shapelet sx mn whose corresponding wick Pi,k is significant. Consider Figure in the Introduction m.1 for each class. Such interpretations where we visualize the two shapelets with the largest with Pt,k
allow to visually validate the correctness of learned shapelets and classifiers. Note that, the best of our knowledge, most existing shapelet-based methods (Qu et al. 2024) and post-hoc explanation methods (Queen al. 2023) only provide local explainability.
Global Explainability Global explanations provide knowledge about the classification problem, expressed in SBM by inductive logics without pertaining to particular samples. SBM offers the following global explanation: (1) w.. 0: samples class should contain s m,,
(2) w.. m.8
m, is unrelated class (3) wi m. < 0: samples in class should not contain 8 m,,5
8%
Most figures in this paper visualize shapelets with the highest weights in the linear classifier, validating the correctness of the global explanation. For some challenging tasks, we use local explanations to provide visually better explanations for specific samples. Note the difference between local and global explainability: local explanations discuss what does occur a sample whereas global explanations discuss what should occur in sample.
5

---

Published as a conference paper at ICLR 2025
4.4 MIXTURE-OF-ENTE EXPERTS GATING
In the previous section, the shapelet bottleneck transforms TS into low-dimensional representations in the form of logical predicates. While the max operator in Equation (2) makes the classifier more interpretable, it also reduces expressiveness. For example, the predicates can only measure the existence of shapelets but lose information about the number of occurrences and corresponding timestamps. In TS with discriminative features in other domains, such as the frequeney domain instead of the time domain, the shapelet bottleneck may not capture essential features for classification. To address this limitation, we employ a Mixture-of of--Exerts approach to build iartiall-inneepretable hybrid model.
A Mfxture-of ofpextrts (MoE) combines the output of multiple expert models with gating network (Shazeer et al. 2017). Experts have different structures to capture different aspects of the data and the gating network assigns the work to the experts. IME (Ismail et al. 2023) views simple model (such as linear regression or soft decision tree) as an interpretable gating network. From a different perspective, the interpretable expert itself can serve the gating network, which reduce the number of parameters and improves interpretability. We introduce a gating function to assign the work to experts based on the confidence level of the interpretable expert.
The outputs r; minimizes Lee during training. Denote f; = softmax(r;) with components Áiç corresponding Equation (3). Then the optimal i; is one-hot vector with fi.e 1 if y: c. Diversity of fi.e thus measures confidence of the model. When combining an interpretable model and DNN with MoE, the use of DNN should be inverse proportional the confidence level of the interpretable model to maximize interpretability. Therefore, we design the gating function
(xx,) C- CE (P,o) C-
C-1
which is a modified Gini Index that measures the diversity of variables in f.. Intuitively, 77(x;) = when values in é; are identical, and n(æ;) when f; is a one-hot vector. During training, the output of the hybrid model is then a mixture of the outputs of the interpretable expert and the DNN with ratio s)(:;)::
(6) where z; is the output of the DNN with input zy. During inference, we activate only the SBM and discard the DNN if the confidence of SBM is high. Specifically, given a gating value z. we set
h r, (Ix,) + 2, (1 -=(2)),
(xx))) = if m)(1)) --
4.5 TRAINING OF INTERPGN
The overall objective of the InterpGN framework is minimize Lnybrit 3Lim + Lee- Here, Lce is another softmax cross-entropy loss on (h; 35) to optimize the overall performance of the hybrid model. 3 E [0, 1] is a scalar to weight the training of the interpretable expert, which is set to be either constant hyperparameter or decaying with cosine schedule. Intuitively, the Lint term directly optimizes the SBM component, which ensure that the SBM is actively encouraged to find meaningful shapelets. This design prevents InterpGN from collapsing into a pure DNN model whenever the shapelet-based features are useful (see Figure 18 and Figure The hybrid model is still end-to-end differentiable and trained using gradient-basee optimizations methods.
5 EXPERIMENTS
We first compare InterpGN against various methods on the UEA multivariate TS classification archive, presenting insights gained from visualizing concept representations. Results from SBM without the additional expert are also given. We then demonstrate the benefits of our models on the real-world application of in-hospital mortality early prediction (Harutyunyan et al.) (019) using the MIMIC III dataset. We conclude with ablation studies on the effects of each component in our model. As basic InterpGN design, we combine the SBM with a Fully Convolutional Network (FCN) since has best performance among the three basic deep models for TS classification (Wang et al. 2017). Details of the experimental setup are provided Appendix
Code available http://///haiiic///oo/aaaaaaaaaa

---

Published as a conference paper at ICLR 2025
Table 1: Comparison on 30 datasets from the UEA archive. The best accuracy for each dataset is marked with bold, the second-bess is marked with italic, and the third-best is marked with underline.
Classical
Deep Leamiing
With Interpeecable Features
Our
Datnset
STRF
DTW
FCN
TS2Nec
TimesNer
TST
PrcchTST
SVP-T
ShupeNet
RIPPA
StupeCroyy
SRM
InterpiN
Arg. Accuracy
0,645
0,650
0,246
0.704
0,702
0.709
0.611
0.730
0.6997
0,740
0,743
0,726
0.760
Avg. Rank
9,700
9.138
5,000
7,533
7.517
6.778
10.414
5.262
6.567
5,400
5.500
5:733
3,500
Top-1
8
Num. Top-3
WinsDrows
27
25
Losses
3
11
5
10
10
10
Wileoxon p-value
0.000
0.000
0.003
0.000
0.000
0.001
0.000
0.066
0.001
0.112
0.009
0.003
5.1 MULTIVARIATE TIME SERIES CLASSIFICATION
First, evaluate the performance of InterpGN on 30 datasets from the UEA multivariate TS
classification archive and compare with baseline methods including (1) classical search-based method:
Shapelet Transform with Random Forest Classifier (STRF) and DTW (Chen al., 2013). (2) black-
box deep learning methods: FCN (Wang et al. 2017), TS2Vec Yue et al. (2022). TimesNet (Wu)
al. 2023), TST (Zerveas et al. 2021), PatchTST, (Nie al.
2023). SVP-T (Zuo et al. 2023
and (3) methods with interpretable features: ShapeNet (Li et al.
2021). RLPAM (Gao et al. 2022)
ShapeConv (Qu et al.) 2024).
Table [ presents average classification accuracy across the UEA datasets; the full results are included in Table 5 of Appendix [.] Based on the average rank across the 30 datasets, InterpGN outperforms the baseline methods and SBM achieves comparable performance without help from the FCN.
5.1.1 INTERPRETABILITY
In addition to its effective performance, SBM provides interpretable logical classifiers. Figure visualizes shapelets learned by SBM on multivariate TS dataset from global explanations, where the blue shapelets have the highest positive weights and the red shapelets have the lowest negative weights. Figure visually confirms that the models capture discriminative shapelet features.
Straight
Straight Right
Straight Up
Straight
CCW Circle
Angle
Square
Down
Circle
Mmp
(a)
(b)
(c)
(d)
(e)
(
(g)
(h)
Figure 4: Visualization of the UGL dataset from the UEA archive with accelerometer readings of eight gestures. For example, in (g), the classifier can be interpreted as: "CW Circle" gesture should contain the blue shapelet but not the red shapelet in channel 1.
5.1.2 SHAPELET BOTTLENECK REPRESENTATION
We explore the effectiveness of the shapelet predicates by viewing probabilities p. as representations for TS data. Figure 5 displays the EP dataset using the dimensionality reduction technique t-SNE (van der Maaten & Hinton 2008). Representations learned by SBM and InterpGN are clearly more separable than the raw TS data. While a few samples are mixed into incorrect clusters (c.g.. blue mixed into green), the DNN in the InterpGN model can make corrections and result in better accuracy than SBM. Shading intensity of data points in Figure 5 (c) represents (a,)) Intuitively, predictions of shaded points are interpretable
they are based on SBM while lighter points rely more on the DNN. We observe that most transparent points lie along the cluster boundaries, meaning InterpGN learns to use interpretable logical classifier for casier-fo-classify samples while relying on the DNN to classify the difficult boundary samples which can be identified as requiring additional expertise. In Figure 17, further present the behavior of InterpGN by visualizing the distribution of n7xz). We can conclude that the predictions of InterpGN is consistent with SBM when 17(2;) is high, preserving interpretability for those samples.
7

---

Published as a conference paper at ICLR 2025
Z;
Pi
and
7(21)
25
10
0.0
-25
-10
-10
(a)
(b)
(c)
Figure 5: t-SNE dimensionality reduction of (a) raw data wa. (b) P. of SBM, and (c) P. of InterpGN on the EP dataset. Each color represents samples with a specific label. Samples with low (7x;)) from InterpGN lie on the boundary, i.e., are more difficult predict and can benefit from additional expertise.
5.2 PREDICTING IN-HOSPITAL MORTALITY
The MIMIC-III dataset contains electronic healthcare records from ICU patients. We follow the procedure in (Harutyunyan et al. 2019) to preprocess and generate data to predict in-hospital mortality based on patient records from the first 48 hours after entering an ICU, Setup details are provided in Appendix A.2
Table shows results illustrating how InterpGN helps SBM on the difficult samples.
Interpretability Clinical records from the MIMIC-III dataset are challenging due sam- ple diversity and missing values. Figure O high- lights an example of local explanations on two samples. For better representation, we visualize the five most essential shapelets on their cor- responding channels and omit other channels. The model captures long-term discriminative trends in three channels, HR, MBP and OS. The shapelets and classifier can be interpreted as: the first patient passed away due to decreasing HR, increasing MBP, and drop in OS; whereas the second survived because of increasing HR after local drop, rapid decrease in MBP, and steady OS. In real-world scenarios, these interpretations could bring more insights to clinical applications than DNNs that only provide black-box predictions.
Table Results for in mn-hospital mortality predica- tion on the MIMIC-III dataset.
Metric
STRF
FCN
SBM
InterpGN
Accuracy
0.653
0.693
0.658
0.703
F1
0.639
0.675
0.657
0.657
Recall
0.615
0.634
0.657
0.569
Precision
0.666
0.734
0.659
0.784
ROC-AUC
0.653
0.698
0.658
0.703
Patient Did Not Survive
Patient Survived
20
05
temp
684
25
25
(hours)
thours)
Figure 6: Local explanations of SBM for in-hospital mortality predication. The variables in the figure are: Heart Rate (HR), Mean Blood Pressure (MBP), Oxygen Saturation (OS), Respiratory Rate (RR), and Temperature (Temp).
5.3 EFFECTS OF HYPERPARAMETER ON INTERPRETABILITY AND SHAPELET QUALITY
Analogous to the results presented in Figure I and Section 5.1.1 existing interpretable models mainly demonstrate qualitative results on interpretability by visualizing the interpretable features (Ma) et al. 2020 Qu et al. 2024). However, there remains lack of quantitative metries measure the level of interpretability and quality features. Therefore, we introduce two additional metrics to quantitatively assess the interpretable models.
8

---

Published as a conference paper at ICLR 2025
Interpretability metrie: Naturally, the prediction from the model is considered more interpretable if fewer features are needed to simulate and understand the prediction. With a linear classifier taking logical predicates as inputs, interpretability of SBM can be measured by the sparsity of the classifier weights since fewer shapelets contribute to the prediction as weight sparsity increases. As metric for sparsity, choose three empirical thresholds and compute the ratio of weights above each threshold. Formally, the sparsity metrics are defined as
ratio
1 IW]
Σ WEW
1([w] > 1), for E 01,0.5,0.1..
Additionally, as a relative sparsity measurement, we also compute the Gini coefficient of |w].
Shapelet quality metric: In order to produce reliable representations in shapelet-based models, the shapelets need to be "close" to the actual TS subsequences. Therefore, shapelet quality can be measured by shapelet distances (defined by Equation D). In a prediction, more important shapelets should be expected to have higher quality. Therefore, for our SBM, the shapelet quality metric is defined as
[D)
(5 = vb
ΣΣΣ max(0, w m.d "d ,8 m (ri, Vi) E Diest )d
M
K
shapelet error
(8)
MK
m= IEL k=1
which measures the shapelet distances for samples that are correctly classified, averaged over all shapelets in SBM and weighted by their corresponding classifier weight if wa,k
mn >> 0. Low shapelet
error reflects that the essential shapelets for the prediction are close to the actual TS subsequences which represent high shapelet quality.
Results and Analysis We conduct an ablation study on the effect of hyperparameters on classi- fication accuracy, interpretability, shapelet quality, and interpretable expert utility. Details for the ablation study setup are summarized in Appendix A.4 The results are summarized in Tables 6 7
and Figures 20
22 23
which lead to the following respective conclusions:
Number of shapelet K determines the expressiveness of SBM. T that increasing K improving accuracy while additionally improving shapelet quality since the shapelet could capture more sample-specife patterns. Although the classifier weights also become sparser, the absolute number of weights above the threshold values may also increase. Therefore, the effect of K on interpretability may not be clearly concluded. In InterpGN, the SBM utility rate r) also increase K increase since the SBM becomes more expressive.
Schedule of of loss weighting 3 controls the training of the interpretable expert in InterpGN. If 3 decays with a cosine schedule, the SBM no longer trains on samples that cannot be confidently predicted as the models get trained. Table 7 shows that, compared
constant schedule, models trained with the cosine schedule result in slightly worse accuracy but improve in both shapelet quality and interpretability since the SBMs focus more on samples where shapelet features are useful. As less samples are fitted, 17 also decreases.
Predicate type defines the method of building logical predicates from shapelet distances. In our models, the SBM uses the RBF predicates defined in Equation[]] Another commonly used formulation a linear predicate (Mueen al., 201 Yan et al., (2022) which computes threshold- based predicates m.d = si sim i (c m. d' where ct 2n is the threshold value to Iearn. Table shows that our by Pi,k
RBF predicates outperform linear predicates accuracy, shapelet quality, and interpretability.
affects the steepness of the RBF kernels. A large defines a tight kernel and pushes the shapelet to be close to the TS data. Figure 20 demonstrates that increasing e improves shapelet quality in the SBM but significantly reduces the accuracy for 2 5. = offers the best balance between shapelet quality and accuracy.
Weight regularization Xreg affects the sparsity of classifier weights and the number of shapelets contributing to the predictions. Figure 23 shows that increasing X improves interpretability without significantly affecting the accuracy for Aveg I 1.
Shapelet diversity regularization Aai encourages learning different shapelets. Figure 22 leads us conclude that Xai does not have significant effect on the three metrics.

---

Published as a conference paper at ICLR 2025
Gating value !/ controls the threshold of discarding the DNN during inference. Prediction of samples with 7(24)) Y only rely the SBM, maximizing their interpretability. Figure 21 shows that reducing the gating value results in slight degrade in accuracy, where the difference in average accuracy between using ? = 0.5 and ! 1 is 0.0035.
5.4 EXTENDED EXPERIMENTS
Finally, we provide preliminary results for more extensive experiments, including the following:
Architecture of the DNN expert within the InterpGN: We experiment with four supervised deep learning models: FCN. Transformer, PatchTST, and TimesNet (see Appendix C.T). Among the tested architectures, InterpGN with FCN achieves the highest accuracy on most datasets. Interestingly, we observe that the choice of DNN significantly affects for each dataset, reflecting the compatibility between the SBM and the DNN. This highlights a complex relationship between the features learned by the SBM and those captured by different DNN architectures, which may be worth more in-depth study.
Shapelet distance metric and classifier architecture in the SBM: For the shapelet distance function dist, we further compare cosine similarity and Pearson correlation with the conventional Euclidean distance (see Appendix C.2). For the classifier in SBM, compare bi-linear and attention- prrention-based classifiers, which could model more complex relationships between shapelets, with the conventional linear classifier (see Appendix C.3 These variants sometimes yield better performance on some datasets, but not consistently, indicating that they could be meaningful architecture choices to tune for each dataset.
We follow Alvarez Melis & Jaakkola (2018) to assess the faithfulness of shapelets learned by the SBM, which measures the correlation between the presence of essential shapelets and the predictions of SBM (see Appendix B.2.1). Over the 30 UEA datasets, we observe high faithfulness on most of the datasets. Notably, all datasets have faithfulness greater than 0, indicating that the learned shapelets are always positively correlated with the predictions.
Faithfulness of shapelets learned by SBM:
Time series extrinsic regression: To further examine the generalizability and limitations of the proposed methods, we apply them time series extrinsic regression tasks (Tan et al.) (02T). In order to leverage classification models for regression, we train and evaluate them using ordinal regression framework, employing the Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery 2007) the performance metric. Additional details are provided in Appendix C.4
6 CONCLUSION AND LIMITATIONS
In this paper, we address the limitations of existing TS classification methods. We introduce InterpGN, which leverages the trade-off between interpretability and performance. We present a variant of the Shapelet Transform build logical predicates from shapelets and propose a novel gating function design based on confidence of the interpretable model. Our experiments demonstrate that InterpGiN outperforms existing methods while preserving interpretability for most samples. We also interpreted and analyzed patterns and classifiers learned by the models on various datasets, and applied our models the MIMIC-III dataset to provide interpretable features and discover knowledge on real-world task. In the future, we plan to extend the shapelet- based approaches to other time series analysis tasks (Ansari al., 2024) and build scalable interpretable models (Wen al., 2024).
We note two limitations of our study. Firstly, in some classification rules, the most important shapelet for class e may not look similar to subsequences of TS class c. Such rules tend to have interpretations with form "z; belongs to class since it does not contain the essential shapelets for + c" or "x; belongs to class e since other c' + с do not contain shapelet s". For example, the blue shapelet in Figure (a) may not capture the essential trend for the "Angle" movement but it can still be correctly categorized since it does not contain the essential shapelets for other categories. Secondly, note that have studied simple design which includes one interpretable expert and one DNN. Analogous to the MoE (Shazeer et al. 2017) framework, InterpGN can scale up by integrating more experts. For example, multiple interpretable models be included using hierarchical gating approach, while multiple DNN models can be combined using the original MoF approach.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work funded through the IBM-RPI Future of Computing Research Collaboration. We thank the teams at UCR, UEA, and Monash University for their efforts in creating and publicly sharing the benchmark datasets. We also appreciate Wu al. for providing the open-source time series benchmarking repository. Finally, we are grateful to the anonymous reviewers of NeurlPS 2024 and ICLR 2025 for their insightful and constructive feedback, which greatly enhanced the quality and scope of our work.
REFERENCES
Talal AA Abdullah, Mohd Soperi Mohd Zahid, Waleed Ali, and Shahab UI Hassan. B-lime: An improvement of lime for interpretable deep learning classification of cardiac arrhythmia from ecg signals. In Processes, 2023.
Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Gift-eval: A benchmark for general time series forecasting model evaluation. arXiv preprint arXiv:2410. 10393, 2024.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explainnng neural networks. Advances in neural information processing systems, 31, 2018
Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sehastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider and Bernie Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Saira Aziz, Sajid Ahmed, and Mohamed-Slin Alouini. Ecg-based machine-leariing algorithms for heartbeat classification. In Nature: Scientific Reports, volume 11, 2021.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018. 2018.
arXiv preprint 1rXiv:1811.000
Yanping Chen, Bing Hu, Eamonn Keogh, and Gustavo EAPA Batista. DTW-D: time series semi- supervised learning from single example. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 383-391, 2013.
Angus Dempster, François Petitjean, and Geoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5): 34(5):1454-1495) 2020.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
Ge Gao, Qitong Gao, Xi Yang. Miroslav Pajic, and Min Chi. A reinforcement leaming-informed pattern mining framework for multivariate time series classification. In Ist International Joint Conference on Artificial Intelligence (IJCAI), 2022.
Tilmann Gneiting and Adrian Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association. 102(477):359-377 2007.
Josif Grabocka, Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thieme. Learning time-series shapelets. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 392-401, 2014.
Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.
Marton Havasi, Sonali Parbhoo, and Finale Doshi- Velez. Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35:23386-23397 2022.

---

Published as a conference paper at ICLR 2025
Aya Abdelsalam Ismail, Sercan O Arik, Jinsung Yoon, Ankur Taly, Soheil Feizi, and Tomas Pfister. Interpretable mixture of experts. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Data descriptor: Mimic-iii, a freely accessible critical care database. Scientific data, 3(160035)119, 2016.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338 5348. PMLR, 2020.
Seungeon Lee, Xiting Wang, Sungwon Han, Xiaoyuan Yi, Xing Xie, and Meeyoung Cha. Self- explaining deep models with logic rule reasoning. Advances Neural Information Processing Systems, 35:3203-3216 2022.
Guozhong Li, Byron Choi, Jianliang Xu, Sourav S Bhowmick, Kwok-Pan Chun, and Grace Hung Wong. Shapenet: A shapelet-neural network approach for multivariate time series classification. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 8375-8383, 2021.
Jason Lines, Luke Davis, Jon Hills, and Anthony Bagnall. shapelet transform for time series classification. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 289-297, 2012.
Qianli Ma. Wanqing Zhuang. Sen Li, Desen Huang, and Garrison Cottrell. Adversarial dynamic shapelet networks. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5509950778 2020.
Noushin Mehdipour, Cristian-loan Vasile, and Calin Belta. Specifying user preferences using weighted signal temporal logic. IEEE Control Systems Letters, 5(6): 2011, 2021. doi: 10.1 WWw.CCSSS...00..444...
Abdullah Mueen, Eamonn Keogh, and Neal Young. Logical-shapeless: an expressive primitive for time series classification. In Proceedings of the 17th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 1154-1162, 2011.
I. Neves, D. Folgado, Santos, M. Barandas, A. Campagner, L. Ronzio, F. Cabitza, and H. Gamboa. Interpretable heartbeat classification using local model model-agnostic explanations on ecgs. In Computers in Biology and Medicine, 2021.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2022.
Yuzuru Okajima and Kunihiko Sadamasa. Deep neural networks constrained by decision rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2496-2505, 2019.
Eric Qu, Yansen Wang, Xufang Luo, Wenqiang He, Kan Ren, and Dongsheng Li. Cnn kernels be the best shapelets. In The Twelfth International Conference on Learning Representations, 2024.
Owen Queen, Tom Hartvigsen, Teddy Koker, Huan He, Theodoros Tsiligkaridis, and Marinka Zitnik. Encoding time-series explanations through self-supervised model behavior consistency. In Advances in Neural Information Processing Systems, volume 36, pp. 32129-32159, 2023.
Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian. Ronald Fagin, Francisco Barahona, Udit Sharma, Shajith Ikbal, Hima Karanam, Sumit Neelam, Ankita Likhyani, and Santosh Srivastava. Logical neural networks, 2020.
12

---

Published as a conference paper at ICLR 2025
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-experss layer. In International Conference Learning Representations, 2017.
Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey Webb. Time series extrinsic regression. Data Mining and Knowledge Discovery, pp. 1-29, 2021.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. In Journal of Machine 2008.
Learning Research, volume 9, pp.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong bascline. In 2017 International joint conference on neural networks (IJCNN), pp. 1578-1585, JEEE, 2017.
Yunshi Wen, Tengfei Ma. Tsui-Wei Weng, Lam M. Nguyen, and Anak Agung Julius. Abstracted shapes as tokens a generalizable and interpretable model for time-series classification. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet:
Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
Ruixuan Yan, Tengfei Ma, Achille Fokoue, Maria Chang, and Agung Julius. Neuro-symbolic models for interpretable time series classification using temporal logic description. In 2022 IEEE International Conference on Data Mining (ICDM), pp. 618-627. IEEE, 2022.
Lexiang Ye and Eamonn Keogh. Time series shapelets: a novel technique that allows accurate, interpretable and fast classification. Data mining and knowledge discovery, 22:149- 182, 2011.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ve2:ec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery data mining, pp. 21 14-2144, 2021.
Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Tien Lu. Tapnet: Multivariate time series classification with attentional prototypical network. In Proceedings of the AAAI conference on artificiall intelligence, volume 34, pp. 6845-6852, 2020.
Ziqi Zhao, Yucheng Shi, Shushan Wu, Fan Yang. Wenzhan Song, and Ninghao Liu. Interpretation of time-series deep models: A survey. arXiv preprint arXiv:2305.. /4582, 2023.
Rundong Zuo, Guozhong Li, Byron Choi, Sourav S Bhowmick, Daphne Ngar-yin Mah, and Grace LHH Wong. Svp-t: a shape-level variable- variable-position transformer for multivariate time series classification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 11497-11505 2023.
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL SETUP
Environment. All the experiments are implemented using Python 3.11 and PyTorch 2.4.0. All the experiments run a NVIDIA V100-SXMM-32GG GPUs. The SBM and InterpGN are trained single GPU. The STRF is implemented using Random Shapelet Transforms from the Aeon Toolkit and Random Forest Classifiers from the Scikir-lears package, which runs on CPU using 64 threads.
A.1 UEA MULTIVARIATE TIME-SERIES CLASSIFICATION ARCHIVE
The UEA multivariate TS classification archive (Bagnall et al., 2018) contains 30 datasets col- lected from various domains. Details of the 30 datasets are provided in Table 4 Data pre- processing and training procedure follow the benchmarking repository provided by [Wu et al.) (2023 The default hyperparameter to produce results in Tan Table alre summarized in Table3 For each dataset, we train the models on the default training split with 5 different random seeds and report the average accuracy on the test split. Statistics on experiment variations could refer to Appendix B.4
Table 3: Default hyperparameters.
Hyperprameter
Default value
Number of shapelet
K = 10
Steepness of RBF kernels
Diversity loss weight Sparsity loss weight 3 schedule Inference gating value Optimizer
Adi- = 0.1
Areg = 0.1
constant
=
Adam
Learning rate
0.005
Weight decay
Batch size
32
The metrics in Table I and Table 5 are:
Avg. Accuracy: Average test accuracy.
Avg. Rank: Average ranking of the method among all baseline methods.
Num. Top-1: Number of datasets the method performs the best among all baseline methods.
Num. Top-3: Number of datasets the method achieves top-3 accuracy among all baseline datasets.
Wins/Draws: Number of datasets that InterpGN achieves better/same accuracy compared to other baseline methods.
Looses: Number of datasets that InterpGN achieves worse accuracy compared to other baseline methods.
Wilcoxon p-value: p-value in the Wilcoxon signed-rank test, which measures if the perfor- mance of InterpGN is statistically significant compared to the baseline methods. A small p-value indicates significance, while large value indicates the InterpGN and the baseline method have comparable performance on the UEA datasets.
"The Acon Toolkit is available at www. -ww.
"The benchmarking repository available g wwtpb/owo/tttt//tc///mc hww.mm/meeeeeee--iiiirrre
14

---

Published as a conference paper at ICLR 2025
Table 4: Information of the 30 benchmark datasets from the UEA archive.
Dataset
Dataset Code
Train Size
Test Size
Variables
Length
Categories
Nerin
Nus
M
T
C
WordRcoontonnn
AWR
275
300
9
144
25
AtrialFibrillation
AF
15
15
2
640
BasicMotions
BM
40
40
6
100
CharacterTrajectoriis
CT
1422
1436
119
Cricket
CK
108
72
6
1197
12
DuckDuckGeese
DDG
50
50
1345
270
ERing
ER
30
270
4
65
6
Eigen Worms
EW
128
131
6
17948
Epilepsy
137
138
206
EthamolConsconrattion
EC
261
263
3
1751
FaceDetection
FD
5890
3524
144
62
2
FingerMovements
FM
316
100
28
50
2
HandMovementDirecttin
HM
160
74
10
400
Handwriting
HW
150
580
52
Heartbeat
HB
204
205
61
405
InsectWingheat
IW
25000
25000
200
22
10
Japanese Vowels
IV
270
370
12
29
9
LSST
LSST
2459
2466
6
36
14
Libras
LB
180
45
MotorImagery
278
100
64
3000
NATOPS
NT
180
180
24
51
6
PEMS-SP
PM
267
173
263
144
7
PenDigits
PD
7494
3498
2
8
10
PhonemeSpectra
PH
3315
3353
217
RacketSports
RS
151
152
6
30
SelfRegulationSCPP
SCPI
268
293
6
896
2
SelfRegulationSCPP2
SCP2
200
180
7
1152
SpokenArrabiciigiss
SAD
6599
2199
13
93
10
StandWalkJump
SWJ
12
15
4
2500
UWaveGesturel library
UGL
120
320
315
A.2 PREDICTING IN-HOSPITAL MORTALITY ON MIMIC-III
From the original MIMIC-II dataset, we first extract time-series and impute missing values using the method proposed by Harutyunyan al. (019). For predicting in-hospital mortality, the data highly imbalance, with more than 80% positive samples (patient survived). Therefore we randomly select subset of 1500 positive and 1500 negative samples evaluate our models. A sample in the dataset is TS with M = 9 and T = 48. We further divide the subset into 80% training and 20% validation to evaluate the performance using 5-fold cross validation. Omitting categorical variables in the MIMIC-III datasets, the TS data has the following variabels:
Diastolic blood pressure (DBP),
2. Fraction inspired oxygen (FIO),
3. Heart Rate (HR),
4. Mean blood pressure (MBP),
5. Oxygen saturation (OS),
6. Respiratory rate (RR),
7. Systolic blood pressure (SBP),
8. Temperature (Temp),
9. Glucose (Glu).
15

---

Published as a conference paper at ICLR 2025
A.3 LIMITATION IN IMPLEMENTATION
When combining shapelet learning with modern deep learning frameworks such as PyTorch, the learning suffers from memory overhead. Intuitively, the transformations in Equation and Equation convolution layer with
2 slide 8g over z; and take the minimum/maximmm,, which is analogous pooling. In practice, for efficiency concern, the memory cost of this operation is O(T 1). When T is large (e.g. T> 2000) this operation requires numerous GPU memory, mainly due the unnecessary intermediate variables and gradients are retained However, the traditional convolution operation does not suffer from this because of low-level optimizations from CUDA, which does not retain intermediate gradients. Note that ShapeConv (Qu et al. 2024 makes connection between shapelet transform and convolution by:
d(a th mil Sk = -2 j=1,
m.L 5k
12
27
max
3p
2;
m.t where & x."h is the convolution (cross-correlation) operation. However, found that, in actual implementations, the additional norm term still suffers from the memory overhead.
To address this memory limitation on very long sequences, we provide implementations for the time-memory trade-off:
A time-efficient implementation using large matrix operations:
u.unfodd shapelet peapelet___nnth, 1)
x x.permute (U, 2, 1, 31) unsqueeze (2)
d dist (x shapelets) mean (dim=-1)
2. A memory-efficient implementation using loops:
class ShapeletDistanceFune (torch. nuutograd.Functionn def forward (ctx, x, s):
ctx.save_foct tex.:aaw__oor_bcckaaadll backward s)
output = corch.ccte ([dist (s - x(:, ;, s:its.shape (-11])
for i in range (x. shapeee[-1] return output def backward (ctx, grad_output):
s.shap dim=1)
$ Compute gradient of s return 0, grad_s
def ShapeletDistance s): return ShapeeetDiis.meeeess...... (x, s)
d ShapeletDistance (x, shapelet)
On the largest datasets, this memory-efficient implementation takes less than 12 GB of GPU memory, but requires significantly more time to compute.
Another method to mitigate this issue is by using stride. For example, compared traditional convo- mn lution operation Convid(), and transforms v; into Pi,k using Equation 2 with in-channels and di lation = 1.
out_channell. 1, kerneleee Memory cost reduces as stride increase. When is large, this would not bring significant effects since the learned shapelets are abstracted patterns. In practice, we find that setting stride log(!) results in the same level of performance as stride = I on most datasets.
= l, striii di 1, padd ng
16

---

Published as a conference paper at ICLR 2025
A.4 ABLATION STUDY SETUP CHOICES AND EXPLANATIONS
Main experiment:
Unlike existing methods that perform dataset-specific hyperparameter the same set of default hyperparameters
tuning achieve optimal performance, across all datasets, and later study the effect of each hyperparameter on the metrics in ablation study.
Ablation study: Ablation study in Section 5.3 is conducted on the 30 UEA datasets. For each ablation case, we only change the hyperparameter of interest and keeping other hyperparameters the default values. A summary of the ablation results are presented in Section 5.3 and detailed results are included in Appendix [B.4]
Diversity loss: In the SBM training, we incorporate a regularization (Ma et al., 2020) to encourage the learning of more diverse shapelets (Equation 4 However, the ablation study, we find that the weight of the diversity regularization Adiv does not introduce substantial effects the metries of interests (see Figure 22). These results suggest that the diversity loss could be omitted, considering its high computational complexity. Nevertheless, we chose retain it because is used in other shapelet-based literature (Qu al., (024) and may still promote favorable properties for shapelet learning, which would require further investigations.
Number of shapelets: In the ablation study, we consider the number shapelets K 10 and K = 5 which are rather compact. Although using large K would test the scalability of SBM, we believe that interpretability favors a compact model with a small number of concepts whenever possible. Using a large K would compromise interpretability by making the SBM more complex and less plausible for human understanding. Furthermore, the features of dataset is so complex that require large K to express, we prefer to leverage the DNN component of InterpGN for such cases.
17

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 FULL RESULTS ON 30 DATASETS FROM THE UEA ARCHIVE
For the performance of baseline methods, we reproduce STRF, FCN, TimesNet and PatchTST to obtain their performance on the UEA datasets. Performance of DTW and Tec are obtained from Yue et al. (2022). Performance of TST and SVP-T are obtained from Zuo et al. (2023). Performance of ShapeNet, RLPAM, and ShapeConv are obtained from the corresponding paper.
Due to the limitations in model architectures, some method may fail on certain datasets. For example, TimesNer and PatchTST fail on the FigenWorms datasets as the time series exceed their input length limits. For fair comparison, in Table 5 we additionally report the statistics of the methods with "N/A" dropped, keeping the datasets where all the method have valid result.
Table 5: Accuracy and comparison of the methods on 30 datasets from the UEA archive. The best accuracy for each dataset is marked with bold, the second- -estt is marked with italic, and the third-best is marked with underline.
Clasical
Deep Learning
With Interpeetable l'eatures
Ours
Datiot
STRF
DTW
PCN
TS2Ns:
TimesNer
TST
ParcTTTT
SVP-T
ShapeNec
RIPAAM
StopeCoom
SBM
Interp(NN
Articalrry Wandeyyyynnneoeeiiiiee
0:917
0.967
0.990
0.987
0.983
0:943
0.918
0.902
0.967
0.025
0.994
0. 6.992
0.991
0.267
0.200
0.383
0.280
0.507
0.200
0.467
0.400
0.400
0.733
0.521
0.455
0.333
RasicMotions
0.925
0.975
1,000
0.955
0.65
0.965
0.560
1,800
1,000
1,000
0.997
1,800
1,000
Tharacerinjjeeiiiiss
0.849
0.989
0.998
0.995
0.980
N/A
0.859
0.590
0,960
0.788
0.981
0.963
0.997
Cricket
0.944
1,000
0.997
0.922
0.89
0.958
0.883
1,000
0.966
1,000
0.998
0.981
1,000
DueeDaclGeese
0,350
0.600
0.668
0.680
0.404
0.450
NA
0.700
0.725
0.700
0.648
0.384
0,480
EigenWorms
0.672
0,618
0.759
0.47
N/A
NAA
0.415
0.925
0.878
0.909
0.302
0.582
0,835
Epilppy
0.97%
0.964
0.975
0.664
0.36
0.92
0.565
0.86
0.967
0.978
0.972
0.993
0.99
BRing
0.589
0.133
0.394
0374
0.914
0391
0.572
0.37
0.133
0.319
0.774
0.967
0.933
EdhanoComnenrariic
0.677
0.323
0.297
0.308
0.300
0.337
0.263
0.331
0.312
0.369
0.253
0.361
0.29
DeceDoteztion
0.567
0,529
0.584
0.501
0.686
07687
0.559
0.512
0.600
0.61
0.635
0.657
0.699
Fingar Movements
0,500
0.530
0.622
0.480
0.605
0.776
0.564
0.400
0.550
0.640
0.587
0.600
0.616
DineMMovemenDDrretiinn
0.419
0,231
0.466
0.358
0.416
0.608
0.568
0.392
0,318
0.635
0.413
0.475
0.459
Hanchening
0.104
0,256
0,61
0.155
0.259
0.305
0.112
0433
0.452
0.527
0.357
0.617
Hourtheat
0.746
0,717
0.780
0.883
0.739
0.712
0.724
0.790
0.764
0
0.651
0.466
0.393
0.684
0.276
0.384
0,250
0.309
0.633
Japanese Nowels
0.676
0,949
0.991
0984
0.972
0.991
0.876
0978
0.964
0.955
0.993
0.563
0,950
Lihus
0,817
0.953
0.806
0.844
0.642
0.387
0.969
LSST
0.491
0.354
0.40
0.315
0AUN
0.597
Manvelmagery
0.510
0.584
0.510
0.14
0.500
NATOPS
0.754
0.904
0.730
0.977
0.925
0.524
0.764
0.367
0.801
PenDigits
Specter
0.155
0.003
0233
0.19
0.192
RacketSports
0.842
0.903
0.555
0.829
0.728
0,842
0.863
0.801
SellRegulatonSPPI
0.846
0.902
0.122
0.995
0.925
0.825
0.358
SellRepulaiinSCCP
0.489
0.777
0.578
0.599
0.589
0.559
0.600
0.57%
0.624
0.578
SpokenAniicDiiia
0.679
0.955
0.988
0.977
0.993
0.748
03%6
0.975
0979
0.996
0,997
Sand/WalkJmmp
0.467
0.200
0.493
0.467
0.600
0.267
0.507
0.477
0.533
0.587
0.507
0.762
0,903
0.589
0.986
0.12
0.901
0.778
0.941
0.906
0.944
0.356
0.915
0.911
Avg. Accuracy
0.645
0.765
0.704
0.702
0,709
0.611
0.730
0.743
Aug. Rank
9.700
9.138
5.000
7.533
7.517
6.778
10.444
6.567
5,000
5.500
3,500
Num. Top-1
Num. Top-3
10
WineTTrass
28
23
25
18
27
20
Losss
2
11
10
10
Wilcoxon P value
0,000
0,000
0.003
0.000
0.000
0,001
0,000
0.066
0,001
0.112
0.009
0.003
Statisties without
Avg. Accuracy
0.669
0.646
7,9
0.705
0,710
0,719
0,622
0.738
0.699
0.747
0.7.8
0,738
0.768
Avg. Rank
9,520
9,250
3,000
7.920
7.640
6,900
10,520
5,480
6,760
$,360
5,600
5,430
3,520
Num. Top-1
2
3
1
0
1
8
Num. Top-3
2
12
3
8
1
11
9.
14
WineThraus
23
24
21
24
21
17
23
17
22
17
Lasuss
4
8
8
Wilcowat poralue
0,000
0,000
0.001
0,000
00000
0,003
0,000
0,030
0.000
0.138
0,007
0.019
18

---

Published as a conference paper at ICLR 2025
B.2 ADDITIONAL INTERPRETABILITY ANALYSIS
B.2.1 FAITHFULNESS
We follow Alvarez Melis & Jaakkola 2018 to asses the faithfulness of shapelets learned by the SBM. Specifically, for each concept (i.e., shapelet predicate p), we measure its relevance by calculating the change in the probability of certain prediction before and after removing it (i.e., set 0). The faithfulness score is then determined by the correlation between the relevance of shapelet concept and its assigned importance
Formally, given a sample (Ir,3s = c), the SBM first provides the set of shapelet predicates p; 6 RR = Ip.. Papss and their corresponding classifier weights w | [wi.ev.... W|s| E {l....,C), CH]. Then, the predicted class probabilities are computed by f softmax(w p;) and f, fcl. Then, we have
(k) A (k) Ai.e Fie where !( softmax(wW) w P; (K)
(10)
P (R) [pts |j (l....., 1S,. 0f
Here A, measures the changes in the predicted class probability with the absence of shapelet concept 8t- The faithfulness of the sample H; then quantified by the correlation coefficient between
the vectors of relevance values A; _A A(..)).((())) and the importance weights of the predicted class w,:
Faithfulness(2:)) = p (Ww,,44 A.).
(11)
where p denotes the Pearson correlation coefficient. Aggregating Faithfulness(z;; across all (2t, 9:) E D, generate the box plots shown in Figure On most datasets, regardless of the varying performance levels the SBM, we observe consistently high faithfulness estimates (i.e., greater than 0.5).
1.0
Falthfulness Estimate
0.0
AWR
AF
BM
CT
CK
DDG
ER
EW
EP
EC
FD
FM
HM
HW
HB
Dataset
3.0
Faithfulness Estimate
0.8
0.8
0.4
0000
800
IW
LSST
LB
MI
NT
PM
PD
PH
RS
SCP1
SCP2
SAD
SW]
UGL
Dataset
Figure 7: Faithfulness estimate: aggregated correlation statistics between shapelet concept relevance and importance.
19

---

Published as a conference paper at ICLR 2025
B.2.2 VISUALIZATION OF EXPLANATIONS
Global explanations We visualize the global explanations of the UWaveGestureLibrag BasicMo- tions, SelfRegulationSCPI, and SellRegulationSCP) datasets. We choose these datasets since their dimensions are favored for visualizations (i.e., M and C' are not too high). Lyryery and BasicMotions are representitives for human motion data; SelfRegulationSCII and SelfRegu- lationSCP2 are representatives for EEG data. each figure, for each category, we visualize the time-series samples (left, in gray), the top-5 positively relevant shapelets (middle, in blue), and the top-5 negatively relevant shapelets (right, red).
UNNowGentttnn/trrryrr Caingary
Top-5 Postiaee Belecari Shapelets
Top-3 Negatize Belezent Shupalets
Jan c0
on.s"
B on
(1e -105
100
SAAveeGetteeiirrrr Cacegory
Thp-5 Posttive Belecant Shapsiets
Top-5 Regetiive Releuumt Shapelets
Ray 120
Rai
Soe Welaea
Sai ia on
Sue welii
You 510
Arolw 0.7 on
(AbveGGetturnllbnng Caregory 3
Top-5 Roskive Relevart Shapelets
Top- Negatlve Relevant Shapelets
Toe
UwwweGesturd.berrgg bry Canegory
Top-5 Rostive Shapelets
Top- Nogative Relevant Shapelets a ona
Figure 8: Global explanations of the SBM on the UWaveGestureLibraayy dataset. (Part I)
20

---

Published as a conference paper at ICLR 2025
Uwbveeenuruurary.c Caregery
Toe Postave Relevrt Shopsitss Se w- 1
Top 5 Negative Reievan: Shapelets
Sum
Plcie 19tmn
Rorie 11.0--100 mn.
Purta 036
Na on.
Pacla 033
on.
UkwwtGrrarctiran Catepory
Top 5 Postive Rel evant Shapslets
Top- Negative Re evant Shap dless
13
-25
MW
Carngory
N
Posttiee Relevant Shapslets Sua N- 1o Auple
Top Nogothve Relevant Shapelets Sa ie ma
Sou Ne
6/2ma
Figure 9: Global explanations of the SBM on the UWaveGesturelLibaa dataset. (Part 2)
21

---

Published as a conference paper at ICLR 2025
iile st
Suscee
430
BoseMeters Catepers
Tee Peliss Receant Mapeles
Tel herelive le Stesden
Wyp
Bediiettone Caragory
Top Sortin Belevee Stapeter
Tp. Megulize Beleverr
Sou ot
MW
Pe -0.
DuseMettere: Categery
The Pedtue Reeaent Shopolets
To hepativ Sa esstt Shpdimt
MVVV
W
W
W
N
Kicew 000.00033000 a
Beston
Figure 10: Global explanations of the SBM on the BasicMotions dataset.
22

---

Published as a conference paper at ICLR 2025
SelffegubtionSCPP: Category
Positive Relevant Shapelets
Negstive Re Shape
MMW
And te
Wmm
Mmy My N MMM
10 -0
mmy wom.Aww Mommmaan
M
SofReguratonSCFI: Category
Top Posttive Relevant Shapelets
Top-5 Negative Relevant Shapslets MWW
0.23
www.
Myy Aww
you MMWW
O
on
mn
Figure 11: Global explanations of the SBM the SelfRegulationSCPI dataset. For clarity,
visualize 30 time-series samples.
23

---

Published as a conference paper at ICLR 2025
Seffeepuater5(PP: Cakegory
Top-5 Positive Relevant Shapelets
Top-5 Negative Belevart Shapdlets
0 on.a"
es on
Se/fRegu/ttonSCFP Category
Top-S Positive Relevant Shapelets
Top-5 Negutive Relevart Shappleess
42 onie!
1.5
32
Figure 12: Global explanations of the SBM on the SelfRegulationSSCP2 dataset. For clarity, we visualize 30 time-serie samples.
24

---

Published as a conference paper at ICLR 2025
Local explanations. We visualize the local explanations for the four datasets. In each Figure, fug-fiiuue visualizes a time-series sample (in gray) along with the top-5 most important shapelets. The color of each shapelet indicates its importance, with blue representing the highest importance, followed by orange, green, red, and purple for progressively lower importance.
UWaveGesturell.eo Category
UNavaGesturel larary. Category
UwwavaGesturellbrrry Cotegory
Uwb.waiistuurllrrrryy Category
3.
UWaveGesturel Brran Category
J/NavaGesturel.ie iaray Category
LMMavadasth.eiibrrry Category
UndiveGerbumiibbrry Category
35-
25
UNaveCGesturel.com Brans Category
UNaveGescure ihrary Category
UylvvaGesttreiire Category
Univeeieesttetbrrrrr Category
UWeeeGGesturell.oo Category
UWNveGGestrreli Category
UWlaeeGesturellbrrry Categoty
Caleggor
20
19
UNeveGesturel. Brary. Category
UWaveGesture ibrary Category
Wiveesst/rebbaay
(WlveGest.reubbery Category
UWaveGesturel brary Category
U/WaveCeruuee lbraay Category
UlbveeGettureibbrayy Category
6
UWbeeeGstturrlbbbyy Category
UWaveGesturtLDaag
UNbveGGsturelibrng Category
LWWlvssasttureibbray Category
UVbsveGesturrubbrayy Category
10
82
80
UWbveGesturstlbbrrg Category
Udbivaaactuutiibrrry Categery
34
60
84
24
89
6
64
66
Figure 13: Local explanations of the SBM on the UWaveGesttreLiibbaay dataset.
25

---

Published as a conference paper at ICLR 2025
PasicMations
Naticnotionns
Calegory!
MasicMocions: Calegory
34
2.5
M Man
25-
-25
-34
22-
My
-25
82-
Categooyy
SasicMocions: Category
3.
MAWUAAM
-hil Mann
2.
www
20-
2
9.
ML
wMwMMMMMM
RasicMations: Category
Category
Category
BesicMotions: Categary
M
25
wwwmmm
M
1 3.
3
235
-35
23
Category
Category
Category
DenicMocion:: Category
AMM M.M
3.00
w Mp
wwwm
www
M
MAMAMA
*..
3
25 2
Ammm
10MM
M0MM
Figure 14: Local explanations of the SBM on the BasicMotions dataset.
26

---

Published as a conference paper at ICLR 2025
The local explanations reflect the limitation discussed in Section 6 where explanations for certain categories may appear counterfactual. For example, in the BasicMotions dataset, the most important shapelets for Category may not represent the most characteristic features of that category. Despite this, the SBM can still make accurate predictions due to the strong feature representation for other categories. This issue becomes more pronounced in datasets with two categories, such as SellRegulationSCPI and SellRegulationSCP2. These findings suggest that for binary classification tasks, using single binary classifier with one unified "rule" might be more effective and reasonable design, rather than assigning separate "rule" to each class.
SelitegpdeiieSSPP: Category
SeliinuultionScC Category
SollleeuulaionSCPP Cavagory
Se (llegu elerrSPP: Catlegory
25
Av
SefRegubtonSCP1 Category
ationCCP::
-35-
c4-
a
Figure 15: Local explanations of the SBM on the SellRegulationSCPI dataset.
27

---

Published as a conference paper at ICLR 2025
SoffRegu abionSCP2: Category
SefRegularionCCP2 Category 1
5e Sc/RoculatinSCP2 Category
SeffRegu ationSCP2: Category
1000 Mm
-28
25
20
Soltheuu abonSCP2: Category
SSP Category 2
NR.ReuaaiionSCP2 Category
Sefflegu ation Category
23
30
2.
www. Mui
2
22-
-1
251
3.
-25
30
-20
-5
1m
Figure 16: Local explanations of the SBM on the SelfRegulationSCP2 dataset.
28

---

Published as a conference paper at ICLR 2025
B.3 BEHAVIOR OF THE INTERPRETABILII GATING FUNCTION
B.3.1 DISTRIBUTION OF THE GATING VALUES
In Figure 17 we present the relationship between values of 77(;;) and predictions of InterpGN to fur- ther study the behavior of the proposed interpretability gating function. We conclude that InterpGN's predictions is consistent with SBM's predictions when 77x;;) is high, preserving interpretability for those samples. The DNN makes corrections when 77(x;; low to improve performance. Note that there cases where the DNN changes SBM's predictions incorrect, suggesting that InterpGN's performance can be further improved using more advanced DNNs.
Fpilepsy
Handwriknng
700
25
30
600
900
400
300
200
160
0
Heamteet
Jepanesevowels
Prediction 38M IntorpoN
Predicion: SBM Interphh
Correct Inzemect
175
Coreet Incorrect
60
inoorect to Comect
nnormct Incomect
130
incomect insomeet
Incomest Correct
50
Conrect Unchan ged
Coneec Unchanged
incar rect Unchanged
Incomect Unichange
100
20
10
25
0.0
0.2
0.6
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
ol63
n/60)
Se/RegulatonSCP1
SelfReguration ation55P2
Prediction S3K
ImtrrNN
Predictor:: InterpoN
Conrect Incomeet
Conrect Correct
incemrect
30
20
Unchanged
Correct Unchangedi
Unchanngdd
urchanged
Count
40
10
0.0
0.8
SpaienArabicD
100 Predicton: UWWreeesstunLLBBrayy
2000
Predciao:: SBM Interpdi
1750
Incemest Cornec
incomet
Insomeet Gorrect
Connect Incorrect
1500
Incereet neonect
Incoment Insommett
Conrect Unchanged
Correct Unchanged
1250
Incoreett unchanged
incoreet Unchanged
1000
0.4
1.0
0.0
0.
0001
(1x0
Figure 17: Histogram of values 17(2;), and the prediction consistency between the interpretable expert and InterpGN.
29

---

Published as a conference paper at ICLR 2025
B.3.2 TRAINING CURVES OF INTERPGN
visualize the loss and 7) during the training of InterpGN. The figures present how the loss function design Lnybess = Git L could encourage the use of SBM during training, preventing the InterpGN from collapsing into a pure DNN when possible.
In Figure [8 and Figure 19
1.0
1.0
Loss
Training Loss
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
10
30
50
10
20
30
50
60
Num. Samples Trained (10?)
Num. Samples Trained (10*)
Figure 18: Training curves InterpGiN on the BasicMotions dataset: (left) Inybria and (right) average
2.5
1.0
Loss
Training Loss
0.8
1.5
1.0
0.2
0.0
0
10
20
Num. Samples Trained (10?)
Num. Samples Trained (10?)
Figure 19: Training curves for InterpGN on the UWaveGestureLibbrayy dataset: (left) Laybria and (right) average 1).
30

---

Published as a conference paper at ICLR 2025
B.4 ABLATION STUDY RESULTS
Table 6: Number of shapelet
Table 7: 3 schedule of InterpGN
quality
Ginl)n))
quality
0.5
0.1
Glel(
5133
18723
11:033
0.030
18.84
0.523
0.480
commlani
0.781
0.879
0.034
0.339
11429
0-222
0.405
18728
0.014
0.025
0.467
conment
01760
0.061
0.14
0.420
0.44
0.761
0.039
0.015
0334
SBM
Table 8: Predicate types in SBM2
8.00
0.2
InterpGN
Accuracy
0.46
8:60
048
accn.
quality
[w] >
[w] > 0.5
|u] >
o66
RBF
5
0.728
0.101
0.051
0.141
0.513
RDF
10
0.734
0.073
0.027
0.083
0.405
SBM
0.199
140
InterpGN
684
Linear
10
0.722
0.121
0.054
0.125
0.433
830
18
05
50
12.3
Figure 20: Ablation
0.74
424
6.70
SBM: -
Accuracy
Percentage
<.4
Accuracy
8.24
SBM:
InterpGN:
8.00
Ratio
InterpGN: (Ml + 0.5
6:25
0.1
SBM InterpCN
31
Gating Value
Gating Value
Adw
Adte
Figure 21: Ablation on !
Figure 22: Ablation on Adiv
SBM InterpGN
SBM: [M] 1
6.25
SBM InterpGN
Accuracy
3.76
0.33
SBM: [M] 0.5
InterpGN:
Gini Coeff.
6.76
Ratio
IntorpGN: (w]
0:35
0:38
0.45
0.00
6.18
Areg
Areg
Areg
Figure 23: Ablation on Areg
Table 9: Full results and variations on number of shapelets K
model
K
accu.
quanlity
[w] 0.5
0.1
SBM
0.72 1 0.014
0.103
0.023
0.05
0.017
0.144
0.051
0.523
0.109
SBM
10
0.726
0.011
0.074 0.014
0.026
0.008
0.084 0.022
0.412
0.085
InterpGN
0.761
0.012
0.079
0.019
0.034
0.011
0.109
0.036
0.429
0.105
0.405 0.091
InterpGN
10
0.76
0.01
0.061 0.014
0.014
0.006
0.06 1 0.024
0.343
0.092
0.441 1 0.112
Results in this table are averaged over 28 datasets since the models with linear predicates fail on the datasets Figen Worms and MotorImagery
31

---

Published as a conference paper at ICLR 2025
Table 10: Full results and variations on schedule in InterpGN
schedule
K
uccu,
quanlity
[t| >
[r] > 0.5
[w] > 0.1
1
constant
5
0.761 +: 0.012
0.079 + 0.019
0.034 0.011
0.109 = 0.036
0.429 = 0.105
0.405 = 0.091
constant
10
0.76 + 0.01
0.061
0.014
0.014 0.006
0.06 0.024
0.343 = 0.092
0,441 = 0.112
cosine
5
0.759
0.016
0.077
0.019
0.031 0.044
0.096 0.028
04427 0.103
0.374 0.099
cosine
10
0.761
+ 0.011
0.089 0.014
0.015 + 0.007
0.06 0.02
0.334 0.09
0.435 + 0.1
Table 11: Full results and variations on
model
quanlity
[v] 0.5
|e|| > 0.1
"
SBM
0.5
0.715
1 0.014
0.136
0.03
0.033 0.008
0.089 0.028
0.383 0.091
SBM
1.0
0.726 0.011
0.074 0.014
0.026 + 0.008
0,084 : 0.022
0.412 1 0.085
SBM
2.0
0.723 0.00
0.061 = 0.011
0.029 + 0.008
0.111 0.034
0.034
0.465 + 0.095
SBM
5.0
0.586 0.029
0.015 0.004
0,031 0.009
0.063 0.016
0.138 +:38+00000 0.022
SBM
10.0
0.415 0.026
0.004
0.023 0.008
0.042 0.014
0.072 0.011
InterpGN
0.749
0.055
+ 0.025
0.01611.0004
0.05
0.251 0.093
InterpGN
1.0
0.01
0.061 = 0.014
0.014
0,343
0,441 0.112
InterpGN
2.0
0.76 0.009
0.057 0.01
0.023 0.009
0,097
0.431 + 0.094
0.488 + 0.098
InterpGN
5.0
0.75 + 0.012
0.014 = 0.004
0.029 + 0.008
0.054 - 0.016
0.136 + 0.028
0.263 + 0.059
InterpGiN
10.0
0.752
0.012
0.009 0.003
0.024 0.007
0.04 0.014
0.072 + 0.017
0,118 + 0.038
Table 12: Full results and variations on Areg
model
Àreg
accu,
quanlity
[u] >
[u] 0.5
[w] > 0.1
"
SBM
0.0
0.726 1: 0.01
0.113 + 0.023
0.046 1: 0.01
0.121 +: 0.03
0.487 1 0.09
0.726
0.074 : 0.014
0.026 0.008
0.084 0.022
0.412 + 0.085
SBM
0.726
0.099 0.012
0.017
0.059
0.338 1 0.073
SBM
1.0
0.726 0.011
0.048 = 0.007
0.014 0.004
0.047
0.259 + 0.064
SBM
5.0
0.022 = 0.004
0.007 + 0.002
0.024 + 0.008
0.1 + 0.028
SBM
10.0
0.012 0.701
0.014 + 0.003
0.005
0.001
0.014 0.004
0.058 + 0.015
InterpGN
0.0
0.067 0.02
0.017 0.011
0.031
0.381 +: 0.093
0.463 1 0.114
InterpGN
0.06
0.061 = 0.014
0.004
0.02
0.092
0,441 0.112
InterpGN
0.01
0.049:0000
0.01
0.271 0.016
0.104
InterpGN
1.0
0.759 0.011
0.042 = 0.009
0.009 0.004
0.038
0.221 + 0.069
0.432
InterpNN
5.0
0.757
0.01
0.02 + 0.004
0.004 + 0.002
0.018 + 0.006
0.006
0.09 + 0.036
0.358 + 0.093
InterpGN
10.0
0.755
+: 0.013
0.013 + 0.003
0.004 0.002
0.011
0.005
0.047 1: 0.02
0.299 1 0.092
Table 13: Full results and variations on Aav
model
Aas
aceu.
quanlity
w
[w] > 0.5
[w| 0.1
7
SBM
0.0
0.726 + 0.011
0.072
0.013
0.025
0.005
0.085 : 0.019
0.413 0.093
SBM
0.1
0.726 + 0.011
0.074 0.014
0.026
+ 0.008
0.084 + 0.022
0.412 + 0.085
SBM
1.0
0.726
+: 0.01
0.079
0.016
0.026
0.00
0.083 : 0.022
0.417 - 0.089
SBM
10.0
0.725 0.01
0.089 0.017
0.027 0.009
0.085 0.024
0.42 = 0.079
SBM
100.0
0.724 0.011
0.097 0.019
0.028
0.022
0.406 0.091
InterpGN
0.0
0.762
0.012
0.062 0.014
0.016 0.007
1: 0.021
0.362 0.102
0.451 0.112
InterpGN
0.1
0.76+
0.061 0.014
0.004
0.06 0.024
0.343 0.092
0.441 + 0.112
InterpCNN
1.0
0.76
+ 0.011
0.065 + 0.013
0.017 + 0.006
0.063 + 0.021
0.338 0.084
0.443 + 0.107
InterpGN
10.0
0.758
+ 0.011
0.073 1: 0.016
0.018 0.009
0.064 1: 0.027
0.331 1: 0.086
0.477 1 0.109
InterpGN
100.0
0.762
-: 0.011
0.081 1 0.018
0.018
0.008
1 0.023
0.331 0.099
0.437 1 0.113
32

---

Published as a conference paper at ICLR 2025
C PRELIMINARY RESULTS ON EXTENDED DESIGNS
We further provide preliminary results on extended designs for SBM and InterpGN, where we hope they could provide insights for future research. For simplicity, we experiment with a subset of the UEA archive containing 10 datasets that are commonly used in evaluating deep learning methods (Wu et al. 2023).
C.I ARCHITECTURE OF THE DNN EXPERT IN INTERPGN
Table 14: Ablation on the architecture of the DNN expert in InterpGN. The best performance for each dataset is highlighted with bold.
DNN Expert of InterpGN:
FCN
Transformer
PatchTST
TimesNet
Dataset
accu.
accu.
1
accu.
7]
accu.
1
EthanolConneentration
0.298
0.008
0.320
0.236
0.276
0.833
0.302
0.401
FaceDetection
0.663
0.709
0.657
0.568
0.652
0.756
0.670
0.517
Handwriting
0.612
0.022
0.127
0.545
0.115
0.266
0.214
0.024
Heartbeat
0.779
0.246
0.735
0.671
0.729
0.954
0.740
0.576
Japanese Vowels
0.991
0.758
0.970
0.920
0.919
0,871
0.974
0.831
PEMS-SF
0.861
0.480
0.800
0.671
0.206
0.980
0.834
0.560
SelfRegulationSCCPI
0.917
0.428
0.878
0.448
0.851
0.407
0.902
0.437
SelfRegulationSCPP
0.580
0.036
0.559
0.700
0.576
0.587
0.578
0.363
SpokenArrabiDDigiss
0.998
0.959
0.993
0.984
0.991
0.985
0.992
0.969
UWaeeGestureLLbbrary
0.909
0.646
0.803
0.365
0.791
0.295
0.829
0.205
C.2 SHAPELET DISTANCE MEASUREMENTS
In this paper, as defined in Equation I we follow the conventional definition of shapelet distance and use Euclidean distance, but the shapelet distance can be general and replaced by other metrics. Here we explore two alternative distance measures: cosine similarity and Pearson correlation. These metries may offer favorable properties compared Euclidean distance in some cases. For instance, cosine similarity can capture shapelets that are invariant to changes in offset. Table 15 presents the classification accuracy of SBM and InterpGN variants utilizing these three distance measures. The certain
results indicate that cosine similarity and Pearson correlation can yield better performance datasets, but no metric can universally better than others. This suggests that the choice of shapelet distance metric could possibly serve hyperparameter requiring dataset-specific tuning depending on the needs. For example, motion datasets such as WaaveGestureeLibraa may benefit from cosine similarity due to the presence of offset-invariant shapelet features. In Figure 24 visualize example of local explanations of SBM using euclidean distance and cosine similarity as the shapelet distance measurement, which suggests that they both could learn high-quality shapelets.
UWaveGestureLibrary Category
UWaveeGssurrLibbrryy Category
x
0.2-
22.
3.5
04
02
74
06
08
(a) SBM with Euclidean Distance
(b) SBM with Cosine Similarity
Figure 24: Local explanations of SBM using (a) euclidean distance and (b) cosine similarity as the shapelet distance measurement.
33

---

Published as a conference paper at ICLR 2025
Table 15:
Classification accuracy of the SBM using three shapelet distance metrics. The best accuracy for each dataset is marked with bold. The training of SBM using Pearson correlation fails on some datasets due to infinite gradients.
Model:
SBM
InterpGN
Distance metric:
Euclidean
Cosine
Pearson
Euclidean
Cosine
Pearson
EthanolConcentratioon
0,301
0.297
0.304
0.298
0.304
0.291
FaceDetection
0.657
0.680
0.682
0.659
0.683
0.682
Handwriting
0.357
0.395
0.042
0.617
0.613
0.042
Heartbeat
0.736
0.738
0.740
0.779
0.789
0.779
Japanese Vowels
0.963
0.966
0.084
0.990
0,989
0.084
PEMS-SF
0.850
0.830
0.173
0.853
0.853
0.173
SelJRegulationSCCP
0.864
0,857
0.855
0.917
0,900
0,901
SellRegulationSCPP2
0,522
0,553
0.556
0,578
0.578
0.579
SpokenAArabiDiigtss
0.994
0.991
0.100
0.997
0.995
0.100
UWaaeGGsturreLibraay
0.913
0.946
0.125
0.911
0.936
0.125
C.3 CLASSIFIER ARCHITECTURE IN SBM
In this paper, we focus on using a linear classifier with shapelet predicates, in line with existing shapelet-based methods. While the linear classifier is often preferred for its interpretability. its simplicity introduces limitations such as neglect the interaction/correltion among shapelets, as discussed in Section 6 To mitigate the issues, we present two additional classifier architectures: a bi-linear classifier and an attention-based classifier. Here, for simplicity, we denote the set of shapelets predicates as p, = P..1.....1. Pt, s|]. Specifically, they are formulated as
Bi-linear classifier:
S
SS]
SS|
Tie
We Pi,k +
1(k + j)w3,8
Pi,k Pi,k Ph.t
(12)
1=
k=
j=1
Attention-based classifier:
Ti.c Σ (tttenttoo(((1)) PE, ((.))
PEE,
(13)
where attention is the scaled dot-produc, attention, PE denotes the positional em- beddings, Q and K are learnable linear projections. In our implementations, we keep low-dimensional projections Q R - R 16 and K Ri 16
Compared to the linear classifier, the bi-linear and attention- classifiers can capture relationships between shapelet concepts while retaining some level of interpretability. For example, the bi-linear classifier could express "shapelet and 9j both exist in æ;"" Although the attention mechanism (i.e., Q and K projections) may be considered fully interpretable, actention(Q(p:. PE, K(p:) + PE,Pi. Pi,k) produces predicates that combines shapelet predicates based on attention weights, which preserves some level interpretability
In Table 16 we compare the three SBM classifier architectures on a subset of the UEA datasets. Our results indicate that the advanced classifiers improve performance for both SBM and InterpGiN on certain datasets. Considering the simplicity, good interpretability of linear classifier and its comparable performance to more advanced classifiers on most datasets, we suggest that still favorable to use linear classifier in practice; however it is good to try more advanced classifiers to achieve better performance. We will also leave the exploration other advanced classifiers (such as energy-buct based graph-based ones) for future work.
34

---

Published as a conference paper at ICLR 2025
Table 16: Classification accuracy for the three SBM classifier architectures. The best accuracy for each dataset and model is marked with bold. The PEMS-SF dataset is omitted since due to the quadratic memory cost in bi-linear and attention-bar based classifiers and its high dimensionality. which fail to run.
Model:
SBM
InterpGN
Classifier:
Linear
Bi-linear
Attention
Linear
Bi-linear
Attention
EbhanolConcentration
0.301
0.299
0.302
0.298
0.307
0.290
FaceDetection
0.657
0.638
0.644
0.659
0.642
0.642
Handwriting
0.357
0.314
0.282
0.617
0.605
0.609
Heartbeat
0.736
0.738
0.733
0.779
0.736
0.745
Japanese Vowels
0.963
0.951
0.936
0.990
0.898
0.989
SeliRegulationSCPI
0.864
0,859
0.866
0.917
0.883
0,914
SellRegulationSCP2
0.522
0,530
0.566
0.578
0.562
0.582
SpokenAArabiCDigits
0.994
0.990
0.987
0.997
0.989
0.995
UWaveGestureLLibrary
0.913
0.873
0.854
0.911
0.902
0.902
35

---

Published as a conference paper at ICLR 2025
C.4 TIME SERIES EXTRINSIC REGRESSION
To further investigate the generalizability and limitations of the proposed methods, we apply them to time series extrinsic regression tasks using datasets from the Monash, UEA, and UCR Time Series Extrinsic Regression Repository (Tan et al. 2021). Because the proposed methods are inherently classification models, we recast the regression tasks as ordinal regression problems. Moreover, given that some datasets exhibit highly imbalanced target value distributions, we discretize the target variable into 10 equally sized bins spanning the minimum and maximum target values of the training split for each dataset.
Since the models produce probabilistic predictions over the ordinal categories (i.e., the bins), we employ the Continuous Ranked Probability Score (CRPS) (neiiting & Raftery (007)) which is also widely used in probabilistic forecasting (Aksu et al. 2024), as the performance metric for training and evaluation. Formally, the CRPS is defined
CRPS(f, Σ
Σ f3 1
(14)
where is the predicted probabilities for each bins, is the true target bin, and the total number of bins. Table summarizes the results obtained for the SBM, InterpGi,, and three deep learning models. It should be noted, however, that this experimental setup is preliminary and may require further refinement; our primary objective is to demonstrate the feasibility of extending our proposed methods to regression tasks.
Table 17: Continuous Ranked Probability Score (CRPS) of the methods the time series extrinsic regression datasets. The best result for each dataset marked with bold, and the second best is marked with italic.
Model:
SBM
InterpGN
Deep Learning
DNN Type:
FCN
PatchTST
TimesNet
FCN
PStchTST
TimesNet
AppliancesEnergy
0.844
0.834
1.274
0.937
0.878
1.331
1.135
BIDMC32HR
0.275
0.550
0.369
0.378
0,981
2.026
1.505
BeijinPMI1QQuliiy
0.549
0.425
0.835
0.441
0.415
1.157
0.424
BeijinPM25Quulity
0.477
0.332
0.569
0.332
0.313
1.440
0.317
BenzeneConcentration
0.966
1.038
2.997
1.250
1.031
2.234
1.258
Covid3Month
0.013
0.014
0.017
0.017
0.000
0.000
0.000
H
0.006
0.006
0.008
0.008
0.000
0.000
0.000
HoushholdPowerronsssppioolll
0.456
0.405
1.318
0.935
0.409
1,079
0.807
IEEEPPG
1.122
2,285
2.896
2,862
2.250
2.619
3.077
0.491
0.501
0.649
0.648
0.494
0.644
0.644
NewsHeeadlineSemimeen
0.001
0,000
0.000
0.000
0.000
0.000
0.000
36


Reviews:
[{"review_id": "aev1moz50x", "reviewer": "Reviewer_Vz3P", "summary": "This paper introduces the Interpretability Gated Network (InterpGN), a time-series classification framework that implements interpretability with sufficient performance guarantees. InterpGN integrates a deep neural network and a concept-based shapelet bottleneck model (SBM) to model patterns for classification using interpretable shapelets (i.e., unique time-series subsequences). Here, linear classifiers are simple to interpret because the importance of each feature is directly reflected in the magnitude of its weight. By integrating predicates into a mixture of experts (MoE) framework, InterpGN uses a gating function to assign tasks based on the model's confidence.", "questions": "1. How do we get shapelets? Is it a learnable parameter?\n\n2. Why is the length of the shapelet considered as in the paper? Why is the minimum length fixed at 3?\n\n3. \bCan InterpGN be applied to other time-series applications? I'm really curious to see if InterpGN performs well in time-series anomaly detection (since it's a one-class classification) or some future prediction tasks.\n\n4. Where and how were the hyperparameters introduced in Eq. (5) chosen?\n\n5. Fig. 5 (c) is interesting, but I have a question: low $\\eta$ (low transparency; low confidence of expert) data relies on the DNN output. However, line 388 states, 'Intuitively, predictions of opaque points are interpretable as they are based on SBM, while transparent points rely more on the DNN,' which is confusing.\n\n6. Considering that the MIMIC-III in-hospital mortality dataset is imbalanced, how does InterpGN address potential issues in capturing representative shapelets for the minority class, and does it risk overfitting the majority class?\n\n7. Why was an ablation study on the effect of L (length candidates of predicates)?\n\n8. As the authors highlighted in the limitations section, a drawback of InterpGN is that its interpretability is somewhat rigid (e.g., sequences belonging to a certain class must have or must not have certain predicates). Given this, could InterpGN be utilized as a feature selection tool to enhance classification performance rather than the interpretability tool?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The gated mechanism is novel. Such a gating function based on SBM's confidence level preserves interpretability for simple cases while calling on the DNN for complex samples.\n\n2. The motivation for adding interpretability to time-series classification is compelling, specifically since the link between capturing temporal patterns and classification outcomes is often not intuitive.\n\n3. New quantitative metrics are used to assess shapelet quality and interoperability, demonstrating improvements over existing models.\n\n4. InterpGN achieves comparable or superior accuracy on benchmark datasets.\n\n5. The manuscript is well-written, with a clear presentation of concepts and methodologies. Additionally, the authors effectively visualize the schematic concepts of the paper (Fig. 1, 2, and 4). These illustrations clearly present the core ideas.", "weaknesses": "1. Though interpretability is a primary focus, InterpGN reintroduces black-box elements due to the integration of DNN outputs. This can override shapelet-based explanations in challenging cases. The interpretability is particularly compromised when SBM fails to capture patterns adequately, and the model defaults to the DNN expert without sufficient transparency regarding how the shapelet information influences the final decision.\n\n2. The authors claim that creating predicates using the RBF kernel allows for more flexible matching with the original time-series compared to the threshold distance-based shapelet approach. However, I'm not fully convinced that Fig. 3 effectively illustrates this intuition. In MTS classification, there could be relatively less important features (i.e., those that do not impact the classification results), so what's the justification for requiring a shapelet for every variable?\n\n3. Interpretation of Fig. 6 may seem forced; the claim that the \"OS\" variable is steady appears subjective, as the left side actually seems steadier than the right side.\n\n4. I'm entirely persuaded of the usefulness of \"global explanation.\" As the authors explain well, local explanation is valuable as a post-hoc method for identifying the shapelets that most influenced the classification outcome. However, compared to recent time-series classifications that achieve high performance, what is the necessity of a global explanation? Further intuition on this point would be helpful.\n\n5. The performance improvement in terms of average accuracy in Tab. 1 seems too small. Since InterpGN uses both FCN and SBM, i.e., more parameters, we expect the performance improvement to be significant. However, the improvement is only 0.6% in terms of average accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3MU1J4kUVx", "reviewer": "Reviewer_Jqxe", "summary": "This article proposes Shapelet Bottleneck Model  (SBM) a framework for time-series classification that produces an interpretable model. This framework is based on the adaptation of an existing model (Learning Time-Series Shapelets, LTS). The main idea is to modify shapelet learning by introducing a Gaussian kernel to measure the distance between the time-series and the shapelet, as well as a cost function to ensure diversity among the learned shapelets and greater sparsity in the model's use of shapelets. The learned shapelets are then provided to a linear classifier. An analysis of the weights allows identifying the shapelets that are important for classification. The second contribution is the introduction of a Mixture-of-Experts model and a gating function to ensure good classification performance when the first model is insufficient. The approach is evaluated on 30 common datasets compared to state-of-the-art algorithms.", "questions": "Please see weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper addresses an important topic regarding the explainability of classification in time series.\n* The evaluations conducted are numerous and compared to a sufficient number of state-of-the-art algorithms.\n* The experiments are well analyzed, and the qualitative analyses are welcome.", "weaknesses": "* The state of the art is relatively brief, while there is a wealth of literature on the subject. However, the main works are well cited.\n* The model section of the paper is relatively difficult to read. For SBM, even though it is an iteration of LTS, it would have been useful to say a few words about end-to-end gradient shapelet learning (at least a reference to the LTS paper in this section).\n* The Mixture-of-Experts (MoE) section is really hard to follow due to the very limited space given to it. Very few details are provided about this part.\n* The main contribution of the paper, given the stated objective — an interpretable time-series classification model — is the proposal of a Gaussian kernel distance rather than an Euclidean distance in the LTS approach, which seems rather modest.\n* On the experimental side, given the similarity of the approach with LTS, why not use LTS as a baseline as well?\n* Some results are unclear: what exactly do \"Wins/Draws\" and \"Losses\" mean in Table 1? What does the p-value refer to?\n* Unless I am mistaken, there is no comparison with other interpretable methods for the interpretability results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKolwn9eKX", "reviewer": "Reviewer_Ausa", "summary": "The paper proposes Interpretability Gated Network InterpGN, a gated network that combines a DNN with a  Shapelet Bottleneck Model (SBM). This is a concept bottleneck model that makes predictions based on the presence or the absence of different shapelet.\n \n# Method\n### Shapelet-based TS Modeling\n- a multivariate TS is modeled as independent univariate channels.\n- they learn learn $K$ shapelets for each possible length $L$ and channel $M$ i.e each shapelet can be viewed as  $s_{k}^{m,l}$\n- the paper introduces a distance metric $d_{i,k}^{m,l}$ between the shapelet $s_{k}^{m,l}$ and input $x_{i,t:t+l}^{m}$ which is the Euclidean distance between the two. \n- they also introduce Shapelet Transform on the distance to measure the likelihood that $s_{k}^{m,l}$  exists in $x_{i}^{m}$. This is defined as predicates $p_{i,k}^{m,l}$\n###  Shapelet Bottleneck Model\n- the predicates $p_{i}$ are fed into a linear layer to compute the outputs $r_{i,c}$ for each channel $c$\n- the final loss is cross entropy loss between $(r_i,y_i)$, Shapelet diversity loss and  L1 regularization loss on classifier weight $\\mathcal{L}_{\\text{int}}$.\n### InterpGN\n- The paper introduces a DNN expert along with the SBM and a gated network that chooses to use either the SBM or the DNN.\n- To choose whether to use the SBM or DNN, the paper proposed measuring the confidence of the SBM by measuring the diversity of $\\hat{r_i}=\\text{softmax}(r_i)$ they use a modified Gini Index that measures the diversity of variables in $\\hat{r_i}$ which is given in equation 6  $\\eta(x_i)$.\n- The final output is a  hybrid between DNN output $z_i$ and the SBM output $r_i$ i.e $h_i = r_i . \\eta(x_i) + z_i . (1-\\eta(x_i))$\n- The final loss is $\\mathcal{L}_{\\text{hybrid}}$ which is the weighted sum of SBM loss and the cross-entropy between $(h_i,y_i)$\n\n# Experiments  \n- **Classification** they evaluate the performance of InterpGN and SBN on 30 datasets from the UEA multivariate TS, overall InterpGN outperforms the baseline methods and SBM achieves comparable performance.\n- **Interpretability** SBM offers global-level explanations by looking at weights of different shapelets in the linear layer for a given class. They also show that samples for InterpGN that rely most on DNN are usually on cluster boundaries, meaning InterpGN learns to use SBM for easier-to-classify samples while relying on the DNN to classify difficult boundary samples.\n- **MIMIC-III dataset** The paper shows that InterpGN outperforms DNNs in terms of accuracy.\n\n### Additional Metrics:\n- **Interpretability metric** this measures the interpretability of SBM by measuring how sparse the weights of the linear model are.\n**Shapelet quality metric** This measures how close the shapelets are to the actual time series.  \n\n# Results and conclusions \nThrough ablation studies, they found the following:\n\n- Increasing the number of shapelets (K) in SBM enhances accuracy and shapelet quality by capturing more specific patterns; it reduces $\\eta$ but seems to reduce sparsity, reducing overall interpretability. \n- The cosine decay schedule of loss weighting $\\beta$ in InterpGN results in slightly worse accuracy but improves shapelet quality and interpretability by focusing SBM training on confidently predictable samples, though it reduces the utility rate $\\eta$.\n- Using RBF to get the predicate outperforms linear ones.\n -  The parameter  ϵ influences the steepness of RBF kernels, with larger  ϵ values improving shapelet quality but reducing accuracy significantly.\n- There is an interpretability accuracy trade-off when increasing Weight regularization.\n- Shapelet diversity regularization does not add any benefits.", "questions": "- How are the shapelets learned? i.e., how do you get $s_{k}^{m,l}$? Are they randomly initialized for each channel and learned through back prob?\n \n- Text is unclear in section 3; in the paper, lines 156-160, it is mentioned \"Existing methods gain different levels of interpretability by inputting interpretable features (Zuo et al., 2023) into a simple model such as a linear layer (Ma et al., 2020; Qu et al., 2024)\nor SVM (Li et al., 2021). However, such approaches usually fail to provide explanations of their  predictions based on distance features. For the interpretable expert, we build logical predicates using shapelets, and the classifier directly provides rule-like explanations.\" From this paragraph, one can assume that the classifier on top of the shapelets is not a linear classifier but something else. But in equation 3, it was mentioned that a linear classifier was used on top of the shapelets, so it is a bit confusing... \n\n- In figure Figure 5, what do different colors correspond to? Are they different classes?\n\n- Typos:\n    -  line 98 and line 285 interpretablity \n    -  line 437 intepretability \n    -  line 533 interoperable", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "## Originality -- High\n-  The paper propose a shapelet bottleneck model (SBM) which is original extension to CBM for time series.\n- The gating method used to decide between DNN and SBM in equations 6 and 7 original and quite smart to take into account how confident the SBM for gating.\n- SBM offers several forms of interpretability that might be very useful for downstream applications.\n\n## Quality -- High\n- Strong empirical results and through experiments.\n- Excellent ablation studies.", "weaknesses": "## Significance -- Low\n- My main issue with the paper is that InterpGN is not interpretable at all. It makes sense that it will outperform regular DNNs because the prediction for a single sample comes from both SBM and DNN. This is unlike IME (Ismail et al., 2023), which also combines interpretable models with DNNs, but in IME case, it could still say there is a level of interpretability since for a single sample, only a single expert is used, but combining outputs from different models removes any forms of interpretability. So the paper up SBM was great, and it showed very useful interpretability forms, but InterpGN makes the model a black box again...\n\n- There are also some justified choices, for example:\n\n    -  Why have $\\mathcal{L}_{\\text{div}}$ while ablation studies show it didn't really help in any way?\n     -  Why use $\\eta$ as a gating function and not a linear model?\n\n- The code was not provided to replicate the experiments.\n\n## Clarity  -- Medium\n- Please see the questions section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WHXw3tPKgP", "reviewer": "Reviewer_zkUe", "summary": "This paper proposes a self-explaining framework for time-series classification using shapelets as the form of explanations. While the concepts of CBM (Concept Bottleneck Models) and shapelets are not novel individually, successfully combining them into an end-to-end framework adds value. The overall model design is reasonable, though some details are questionable. The paper presents evaluation results showing similar or superior predictive performance but lacks sufficient experiments on explainability, even though explainability is its main selling point. I believe that the authors will address these shortcomings in the discussion period.", "questions": "**Related Work**\n\n1. I recommend including a discussion about papers that propose CBM-like self-explaining frameworks that use logic rules (e.g., [2] and [3]).\n\n**Method**\n\n2. Why do you use Euclidean distance for measuring the \"distance\" between shapelets and inputs? In time-series data, two signals might look similar but have different offsets, leading to a large Euclidean distance. Did you also try cosine similarity or Pearson correlation?\n\n3. The current model only considers the existence of the shapelet. However, isn't a shapelet that appears at the end of the input more important, considering the characteristics of time-series data? Is there any way to include the location of the shapelet in the explanation?\n\n**Experiment**\n\n4. An ablation study on the model components is needed. What is the performance of the DNN-only model and the SBM-only model?\n\n5. The suggested interpretability metric using $w$ is unconvincing. The scale of the weights might change according to the training setup or dataset, and the thresholds are set empirically. Measuring the skewness of $w$ using the Gini index (which would be similar to $\\eta$) would be more convincing.\n\n6. While the sparsity of weight $w$ might increase interpretability, it is just one aspect. A more comprehensive evaluation of interpretability is needed, possibly through a user study.\n\n7. In the XAI domain, \"faithfulness\" - the degree to which an explanation reflects the model's decision - is an important criterion for evaluating explanations. An evaluation of faithfulness is needed. It would be beneficial if the authors followed metrics from existing literature [4].\n\n**Minor Questions**\n\n8. Why do you refer to $p$ as a \"predicate\"? Typically, a predicate is a condition of a logic rule. It seems that $p$ is a value, not a condition or \"predicate.\" Is this terminology commonly used in other papers?\n\n9. Why do you consider SBM a \"rule-like classifier\"? I could not find a direct connection between SBM and a rule-like concept.\n\n\\\n[3] Deep Neural Networks Constrained by Decision Rules (AAAI 2019)\n\n[4] Towards Robust Interpretability with Self-Explaining Neural Networks (NeurIPS 2018)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. High predictive performance\n\n2. The exploitation of shapelets as \"concepts\" of CBMs.\n\n3. The proposed framework provides both local and global explanations.\n\n4. The overall manuscript is clearly written.", "weaknesses": "1. In the worst case, the model can collapse (i.e., fail to find good shapelets), and $\\eta$ appears to be always $0$ (e.g., Handwriting dataset in Figure 10). In this scenario, the model fully relies on the performance of unexplainable DNN module, while providing the explanation from SBM that does not affect the final model decision. Do you provide the value of $\\eta$ in your explanation? Is there any design consideration to prevent such a collapse? Please see [1] and [2] for references explaining why it is problematic if there is another direct pathway from input to prediction.\n\n2. The computational complexity of the shapelet diversity loss seems to be $K^2 \\times M \\times L$. This might incur a huge computational burden. While using a small K may reduce the complexity, this introduces another weakness.\n\n3. A small number of shaplets (K) are used in the implementation, but this significantly limits the expressiveness of the model and may cause it to miss important features in practice. The experiments only consider $K=5$ and $K=10$; the authors should present more extensive result with larger values of $K$, such as $K=100$ or $K=1000$, if possible. \n\n4. This framework cannot capture the interaction between shapelets, since it uses a simple linear layer for the final prediction. However, the co-existence of relevant shapelets can meaningfully increase the model confidence in practice.\n\n\\\n[1] Faithful Reasoning Using Large Language Models (Arxiv 2022)\n\n[2] Self-explaining deep models with logic rule reasoning (NeurIPS 2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QL9Z45LQRu", "reviewer": "Reviewer_x6CP", "summary": "This paper introduces InterpGN, which stands for the interpretable time series classification framework, that brings together:\n* A novel Shapelet Bottleneck Model (SBM) that explicitly uses shapelets as easily interpretable features\n* A gating mechanism that determines the conditions under which interpretable predictions should be used and under which conditions deep neural network predictions should be used\n* A Shapelet Transform variant for the construction of logical predicates\n* Quantitative measures to assess interpretability and quality of shapelets", "questions": "* Why this form of gating function η(xi)? Were other confidence measures evaluated?\n* How sensitive is the gating mechanism to noise in the interpretable expert's predictions? What if the expert is confidently wrong?\n* What is the basis of using FCN as the DNN expert? Would other architectures like Transformers provide better results?\n* How was the maximum channel number chosen as Kmax=40? What happens if this value is increased/decreased?\n* Given a memory overhead of O(T·l), what is the practical sequence length limit? Will this ever change?\n* What happens to performance as the number of variables, M, increases? What is computational complexity given M?\n* How was the balance parameter β between interpretable and DNN experts determined? Is there a theoretical basis for its selection?\n* Why were the particular shapelet lengths in L chosen? How sensitive is the model to these choices?\n* How do you handle the case where important patterns occur at lengths not covered by the pre-specified shapelet lengths?\n* A deeper comparison with post-hoc explanation methods is probably welcome. What exactly is the trade-off between the approach and methods like LIME?\n* Have there been datasets on which InterpGN performed worse than others? What are the characteristics of such datasets?\n* Maybe one can provide more detailed guidance on the hyperparameter choice for new datasets. How sensitive is this across different hyperparameters?\n* How was the RBF kernel parameter ε chosen? What effect does that have on the quality of the shapelets?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novel gating mechanism based on confidence of interpretable model\n* Improved shapelet transform variant using RBF-based predicates\n* Quantitative metrics for interpretability and shapelet quality\n* Theoretically sound integration of physical constraints\n\n* Clean separation between interpretable expert (SBM) and DNN\n* Adaptive layer normalization for condition injection\n* Unified masking strategy handling multiple tasks\n* Flexible handling of varying input dimensions\n\n* Local explanations: Sample-specific shapelet importance\n* Global explanations: Rule-based class characteristics \n* Visual validation of learned patterns\n* Quantitative interpretability metrics", "weaknesses": "* High memory overhead O(T·l) w.r.t. very long sequences\n* Limited to a fixed maximum number of channels (Kmax=40)\n* Limited scalability analysis w.r.t very long sequences\n* Linear classifier might be too simple, when relations are complex\n\n* Limited analysis concerning failure cases\n* Only one type of DNN expert (FCN only)\n* Very restricted real-world applications (MIMIC-III only)\n* Lacking comparison w.r.t. some very recent methods\n\n* Some counterintuitive shapelet interpretations\n* No formal evaluation of explanation quality\n* Small number of studies regarding users and interpretability\n\n* Lack of discussions on adversarial cases. Limitations regarding \n* Fixed shapelet lengths; single-stage training process; design of a simple gating mechanism\n* Lack of realization based on physical constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)