Reproducibility Study of "Learning Perturbations to Explain Time Series Predictions"
Authors: Jiapeng Fan
Luke Cadigan
Paulius Skaisgiris
Sebastian Uriel Arias
Venue: Accepted by TMLR ()

Abstract: In this work, we attempt to reproduce the results of Enguehard (2023), which introduced ExtremalMask, a mask-based perturbation method for explaining time series data. We investigated the key claims of this paper, namely that (1) the model outperformed other models in several key metrics on both synthetic and real data, and (2) the model performed better when using the loss function of the preservation game relative to that of the deletion game. Although discrepancies exist, our results generally support the core of the original paper’s conclusions. Next, we interpret ExtremalMask’s outputs using new visualizations and metrics and discuss the insights each interpretation provides. Finally, we test whether ExtremalMask create out of distribution samples, and found the model does not exhibit this flaw on our tested synthetic dataset. Overall, our results support and add nuance to the original paper’s findings.

Published in Transactions on Machine Learning Research (06/2024)
Reproducibility Study of "Learning Perturbations to Explain Time Series Predictions"
Jiapeng Fan" Luke Cadigan"
Paulius Skaisgiris"
Sebastian Arias*
Reviewed OpenReview:
Abstract
In this work, we attempt to reproduce the results of Enguchard (023)) which introduced ExtremalMask, a mask-based perturbation method for explaining time series data. We investigated the key claims of this paper, namely that (1) the model outperformed other models several key metrics on both synthetic and real data, and (2) the model performed better when using the loss function of the preservation game relative to that of the deletion game. Although discrepancies exist, results generally support the of the original paper's conclusions. Next, interpret ExtremalMask's outputs using visualizations and metries and discuss the insights cach interpretation provides. Finally, we test whether ExtremalMask create out of distribution samples, and found the model does not exhibit this flaw on our tested synthetic dataset. Overall, our results support and add muance to the original paper's 's findings. Code available this link
1 Introduction
Machine learning (ML) methods are commonly applied to analyze time series data in critical situations, such as predicting patient survival using vital sign readings (Perla et al. 2021) and forecasting crime (Safat et al. 2021). However, these methods often act as black boxes, obfuscating errors and biases in their decision mak- ing. Interpretability methods attempt to explain how these models make their decisions. These explanations allow greater involvement of practitioners in the decision making process, necessity for adoption in many contexts /Vellido 2020).
Enguehard 2023) introduced ExtremalMask, perturbatiion-based machine learning method for time series data that builds upon DynaMask (Crabbé & Van Der Schaar 2021 Tn this paper, investigate its repro- ducibility by validating the three primary claims posed in it. Beyond reproducing the original experiment, this work makes the following contributions:
Explores the assumptions underlying the implementation of the optimization problems in the original paper.
Proposes an alternative saliency metric which takes into account the strength of the perturbations and analyzed its implications.
Investigates whether the perturbations learned by ExtremalMask are realistic.
2 Scope of reproducibility
We test the three main claims found in Enguchard 2023
Claim On synthetic dataset (HMM, Section 3.2.1
EatremalMask best identified the non-salient features compared to 9 other methods, as measured by all relevant tested metrics specified in Section 3.4.1 besides AUP.
"Equal contribution. Author ordering determined randomly.

---

Published in Transactions on Machine Learning Research (06/2024)
Claim 2 On real-life dataset (MIMIC-III, Section 3.2.3), EatremalMask best identified the salient and non-salient features compared to 6 other methods, measured by the relevant Lested metrics specified in Section 3.4.1
Claim BatremalMask achieves belter performance in the preservation game (ezplained in Section 3.1 compared to the deletion game both datasets.
To expand on the original paper, we first propose a more suitable saliency metric the optimization problem solved and investigate its implications on ExtremalMask following extension:
We propose an alternative metric for measuring data saliency. Using EatremalMosk, we re- evatuale the information and entropy (defined in Section 3.4.1) of allernative saliency on HMM. Additionally, we perform a comparative study between identified salient data using the mask and alternative saliency metrics on MIMIC-III.
Extension 1
Then we investigate the practicality of ExtremalMask with the following extension:
Extension 2 Using synthelic dalaset (HMM modified, Section 3.2.2), we explove the plausibilily of the perturbations created with ExtremalMask by testing their probability relative to the original distribution.
3 Methodology
Enguehard
2023 provided open-source implementation of their proposed approach as part of the Python
tint library The repository includes implementations of all methods used in this study, namely DynaMask
Crabbé & Van Der Schaar 2021), Augmented Occlusion ( (Tonekaboni et al., 2020), DeepLift (Shrikumar
ct al. 2017) FIT (Tjoa & Guan 2020), GradientShap (Lundberg & Loc (017). Integrated Gradients (Sun-
darara.jan el al. 2017), Lime (Ribeiro el al., 2016), Occlusion (Zeiler & Fergus 2014), and Retain (Choi
et al. 2016).
3.1 Model descriptions
ExtremalMask quantifies saliency of data in a multivariate time-series dataset for any predictive model f IR + 12 in which is the time horizon, D the number features and every prediction resides the metric space (2, C). To identify such data, the model solves the following optimization problem, referred to the deletion game:
arg min M.O
Σ 1 1 M.l + NN/NNXX, () Xnl (((x,,
(1)
where, X E NxTxD is the time-scrics dataset comprising of samples, M E [0, NxTxxD is the associated mask, NN RTXD > TTDD is an arbitrary neural network (we refer to the network's 's output as noise), and (XX,; M,) is the perturbation function defined as:
M, + (1 M,)NN((,;;O) 9).
(2)
Intuitively, the deletion game rewards small perturbation that cause large changes in the output. Specifically, in Equation
The combination of >1|1 - M,, and >z|NN(X,,; θ) - X,i encourages the perturbed data $(X,, M,)
to be close to the original data X,, incentivizing small perturbations.
[I) M,))) encourages the perturbed output f(0(X,, M,)) to be far from the orig- inal output f(X,,,, incentivizing large changes in the output after perturbation.
http:///hhhlbb..momoooooiigggggg inteeprr
2

---

Published in Transactions on Machine Learning Research (06/2024)
Note that the deletion game implemented in the original paper replaces:
X2,; )) Xn, with ; O) By the triangle inequality, we have that ||NN(X,; θ) Xnl (X,; Ii $ X Thus, ensuring this modified optimization problem that, on average (across n), Xn is geometrically sufficiently close to 0 more accurately estimates the solution to the deletion game.
that (((( M,)) with ((,,,, M,)) S ( (exX,,MM))) on average, S(X,, should be geometrically sufficiently far away from f(0) for it be a good approximation. This change could also be motivated by the potential numerical problem associated with the negative distance being unbounded from below.
Since, by the triangle incquality, it holds
The preservation game, as the counterpart to the deletion game, is defined as:
N Σ n=1
aig min
M Nn|NN((,; X.1 + ((O))) M.))..
(3)
M
In contrast to the deletion game, the preservation game rewards large perturbations that cause small changes in the prediction. To address the analogous numerical problem as in the deletion game, the original paper replaces ~ NN -) X,, with X2 NN(X,,; O) Again, by triangle inequality, this solution again comes at the cost of the additional assumption that X,, average being sufficiently far away from O. Note that unlike the deletion game, the preservation game implemented in the original paper does not assume anything regarding f.
3.2 Datasets
The original paper utilized two datasets: a synthetic dataset generated by Hidden Markov Model (HMM) and the MIMIC-III dataset, which both pose classification problem. Only the training dataset is used to train the classifier f. ExtremalMask and other explainers (interpretability methods) aim to identify salicency of data for f's predictions on the test dataset.
3.2.1 HMM (reproducibility study)
Enguehard 2023 used the implementation of the HMM dataset from Crabbé & Van Der Schaar (2021) However, we noticed some differences between the HMM datasct's description in the DynaMask paper and its implementation in tint. To foster reproducibility, we summarize the implemented dataset here, with the specific parameters described in [Section A.1.T.
Let < 6 T. Consider a 2-state HMM with hidden state at time as 8, E 10, For a single sample X E R TxD, the data at time Xt, sampled from normal distribution with mean Ms, and covariance matrix Σx, Furthermore, the ground-truth label at time f, ye, is sampled from Bernoulli distribution with probability:
(1 exp(-2(x))2)) (st = 00) (1+ (xp(-2(x))s))-
(st - 1)
We generate datasct containing 1000 such samples with T 200 and D 3, i.c. X R 10000x20xx3 and Y E 10000x200 This dalaset is split into 800 training samples and 200 Lest samples.
3.2.2 HMM (extension)
In Extension 2 we modify the HMM dataset introduced in Section 3.2.1 to facilitates the use of the forward algorithm (explained in Section A.2 to compute the probability of perturbed data occurrences within our HMM dataset. to assess how likely the generated perturbations are. Specifically, we ensure that the Markovian
3

---

Published in Transactions on Machine Learning Research (06/2024)
property is satisfied and reduce the time horizon to 50. Additionally, we rectify the asymmetry issue in the decay of transition probabilities present in the original HMM dataset. The modifications are detailed in Section A.I.I
3.2.3 MIMIC-II
The MIMIC-III (Johnson et al. 2016 dataset includes vital sign information for over 40k intensive care unit patients at Beth Israel Deaconess Medical Center. From this dataset, we trained the classifier on 18,390 train samples and ExtremalMask on 4,598 test samples. Each sample contains a binary mortality outcome (sampled patients had 9% mortality rate) and vital signs measured over 48 hour-long time steps. In order to replace missing values, we use the previous values when possible and a standard value otherwise.
3.3 Hyperparameters
We use the default hyperparameters provided by the tint library. The classifier and the NN within Ex- tremalMask both utilize GRU (Cho al., 2014) architecture across all our experiments. These choices are consistent with the original paper.
3.4
Experimental setup and code
3.4.1
Metrics
Suppose there exists an index set A of X E NxTxD such that A, is the index set of the ground truth non-salient features for X,, To assess the minimal impact of the identified non-salient data on predictions, Enguehard 2023) employs the following four metrics when perturbing the non-salient data:
Area Under Precision (AUP) and Area Under Recall (AUR) These metrics gauge the similarity between predictions ions on perturbed and original data. A higher value indicates better performance, signifying perturbing non-salient features marginally impacts the predicted classes.
Information (I) Defined as (A) = - -EN Edtdee log(1 - MMaa), I is higher when the perturbed data on average places more weight on X. higher information is desired, signifying a more informative mask.
Entropy EE) Defined as ( E(r.jee Mard log(Mara) + (1 Muta) log(1 Mnaa). Entropy is low for masks with extreme values and high for masks with values around 0.5. Therefore, a lower entropy is desired, signilying the learned mask being more confident. the identified salient or non-salient. data.
For real-life datasct like MIMIC-III, the same metrics cannot be used as the ground-truth salient data is unknown. Instead, Enguchard (2023) considers the 20% of the data with the highest mask values to be the identified salient data, where we refer to the 20% as the saliency ratio henceforth, and introduces the following metrics:
Accuracy (Acc) Measures whether perturbing identified salient data results in accurate prediction. A lower value is preferred, signifying more important features being perturbed.
Cross-Entropy (CE) Measures change in predicted probability for the correct class. A higher value is preferred, suggesting perturbed data is salient as it decreases the probability of the original prediction.
Comprehensiveness (Comp) and Sufficiency (Suff) Both metrics defined the average distance between the probability of the correct class on original and perturbed data. However, when calculating sufficiency, the 80% of data with the lowest mask values (identified non-salient data) are perturbed instead. A higher value desired, signifying that perturbing salient data influences the probability of predicting the correct class.
Perturbing MIMIC-III data refers to replacing a feature in data-point with the feature's average over the 48 hours sample unless otherwise specified.
4

---

Published in Transactions on Machine Learning Research (06/2024)
3.4.2 Reproducibility study
The repository from the original paper includes code for training the classifier and explainers as well as code for outputting results. To facilitate comparisons with the original paper, we made minimal edits to the original code. These edits include code for saving and loading chockpoints of neural networks.
In all our experiments, we seed PyTorch and other libraries to improve experimental reproducibility. The authors mentioned in our communication that they opted not do this in their paper to reduce experiment runtime. In an additional change, unfortunately, for Jor[Claim 2 chose not to retrieve the results for certain computationally intensive methods. To enable others to reproduce our results, we provide job scripts for executing our experiments on a computer cluster, and shell scripts for running the code locally.
3.4.3 Additional experiments
In this scction, we describe the approaches of tackling each of the extensions introduced in Section 2
Extension The original saliency metric considers only the mask. However, the mask alone fails to effectively represent the magnitude of the difference between the perturbed and original data. Specifically, both NN(X,),a being arbitrarily close to being distant from X E IR could result in low values of the mask in the preservation game. To tackle this, propose the following saliency metric S given X:
Snid def
Xntd o(X,,, Mn)ed
(4)
X, $(X,,, Mn)
A higher value indicates a smaller perturbation, identifying salient data in the preservation game. Note that compared to the mask, we would always expect alternative saliency to be at least informative about the perturbation size, and therefore also at least as informative about the learned saliency of the data by ExtremalMask.. We calculate the information and entropy of this metric to evaluate the informativeness of the perturbations learned by ExtremalMask.
Furthermore, we conduct a comparative analysis between the salient data identified by the two saliency metries in the preservation game. We examine differences local and global scale. Since interested in the data that led to the classifier to correctly classify patient dead, exclusively consider accurately classified dead patients within MIMIC-IIT in this study.
For a local analysis, we pick an arbitrary correctly classified dead patient in MIMIC-III and plot the saliency results. In one plot, we classified data points as salient they were predicted to have a saliency value in the top 20% in sample. This choice of percentage for hard classification of salient points is somewhat arbitrary but aids us in visually assessing data with multiple features and time steps. When employing these visualizations, we recommend exploring different top percentages and recognize that the ideal percentage might vary depending dataset size. Then, we contrast the heatmap based the mask and alternative saliency values for that patient. We refer to these heatmaps as saliency maps.
For a global analysis, we calculate the weighted averages of the mask and the alternative saliency for every feature over time and across all patients in the MIMIC-III test dataset. Enguehard (2023) found that data from later timestops (closer to the patient's death) had greater saliency than data from carlier periods. To ascertain which features carry more saliency in these later time steps, we compute three variants of weighted averages with the following weights applied to saliency values for data at time step (1 < t T):
1. No decay: 1.
2. Linear decay: T
3. Exponential decay: e Tpp
Moreover, we employ Monte Carlo's 95% confidence intervals for the attributions.
5

---

Published in Transactions on Machine Learning Research (06/2024)
Perturbed data generated through perturbation methods may extend beyond their original distribution, becoming out of distribution (OOD). The classifier's s decisions on these unrealistic perturbations are less meaningful, since these perturbations are unlikely to occur during the classifier's training process or its deployment. As a result, we wanted to test ExtremalMask produced less realistic, and therefore lower quality, perturbations.
Extension 2
Specifically, we conducted an experiment for ExtremalMask on the modified HMM dataset (Section 3.2.2) with the test dataset X 2200x50x3, Consider the stochastic process Z (Z,) such that Z, (H D.,). We classify perturbed data X², 550x3 derived from the raw data X.,, OOD when:
log fz (X,,, < median((Iog. Jz (xx))) X IQR((log (x(X)))
where, fz the probability density function of Z and X 2 controls the tolerance for identifying OOD instances. This approach follows the same principle as using Z-scores assess the probability of a datapoint belonging to normal distribution. However, instead of relying the conventional mcan and standard deviation, we use the more robust median and interquartile range (IQR) measures. Using the forward algorithm (Section A.2p that leverages the Markovian assumption to confine the search space, we could evaluate (z) for any a E 150x3, To prevent numerical underflow, we use a shortened time horizon of 50 timesteps.
In our codebase, we provide Jupyter notebooks that enable reproduction of our additional experiments.
3.5 Computational requirements
We used NVIDIA A100 GPU and nine CPUs on a computer cluster for all of our reproducibility experi- ments. Our experiments took total of around 81 hours to run, with the time used per claim specified in Section A.3
We ran all of our extensions CPU (Intel 17-4720HQ), taking negligible time.
4 Results
Our reproduction study confirmed all three claims laid out in Section 2 with the exception of the AUR result in CClamm I and Claim 3 on the HMM datasct. Specifically, ExtremalMask outperforms the other methods on all metrics for the MIMIC-III dataset, and has strong performance on the HMM dataset, with notably strong values for information and entropy. Note that as the results presented in Enguchard 2023 were obtained without seeding, drawing comparative conclusions with our seeded results poses challenges. Our extensions provide a deeper empirical intuition into the perturbation methods and further confidence why these methods reasonable for explaining ML. models.
Our results reported the HMM dataset average + standard deviation over 5 folds. Columns labeled with the sulfix -R denote diff-to-std ratios, obtained by dividing the difference between our and the original paper's results by the standard deviation from the original paper. Differences falling outside 2 standard deviations are underlined.
Table 1:
Method
AUP 1
AUP-R
AUR 1
AUR-R
11
J-R
E-R
Aug. Occlusion
E
1.48
3.08E+4
601
1.65
# 3.40E+4
0.877
DecpLift
0.931 0.003
0.579
0.328 1 0.009
11.5
2.98E+1
890
1.45
3.00E = 256
0.879
DynaMask (MSE)
0.376 + 0.003
16.7
0.771
0.002
0.308
1.05E+5 346
2.60
2.53E44
59.2
2.34
DynaMask (CE)
0.341 + 0.003
18.5
0.569 + 0.002
7.46
1.23E+5 + 1.22E-3
6.77
9.32E+3 + 161
39.3
Fit
0.174 + 0.034
1.08
0.576 4 0.067
1.59
7/2E+4 8.51E-3
0.117
3.30E 4: 1.62E-3
0.916
GradientShap
0.886 + 0.004
1.23
0.283 + 0.007
8.73
2.56 + 697
1.06
2.78E+4 + 282
0.806
IG
0.931 0.003
0.684
0.323 1 0.008
11.9
2.92E+44 817
1.40
2,99E+44 + 293
0.919
Lime
0.950 + 0.003
1.06
0.301 - 0.007
17.1
2.73E 638
1.16
2.84E 4 + 259
0.430
Occlusion
0.919 + 0.003
1.66
0.283 + 0.005
18.3
2.56F+4 530
1.54
2.77 + 212
1.48
Retain
0.681 + 0.113
0.409
0.280 1 0.049
4.15
2.23E+4 + 4.85E-3
4.33
2.95E
2,67E-3
3.45
ExtremalMask (MSE)
0.904 + 0.010
0.633
0.760
1 0.004
1.62
2.93E
4.51E+3
1.07
7.51E+3 = 183
0.927
ExtremelMask (CE)
0.913 + 0.009
0.933
0.666 + 0.012
8.85
1.54E45 + 6.58E-3
10.4
1,59E + 436
16.7

---

Published in Transactions on Machine Learning Research (06/2024)
4.1 Claim ExtremalMask best (besides in AUP) identifies non-salient features on HMM
Unlike all other methods, only DynaMask and ExtremalMask used the mean squared error (MSE) loss function on HMM in the provided code. Notably, those two methods used cross-entropy (CE) loss on MIMIC-II1. Therefore, to facilitate fair comparison with other methods, we also test the results of these two methods trained on HMM with the cross-entropy (CE) loss.
Table corroborates that ExtremalMask (MSE) outperformed other methods in information and entropy: whereas the original paper found ExtremalMask had the best performance in AUR, we found that had the second best performance for both the MSE and CE variants. Thus, our reproducibility results only partially support CCaim 1 In addition, we found that ExtremalMask (MSE) outperforms ExtremalMask (CE) in both information and entropy. Nevertheless, with ExtremalMask (CE), it still had the best performance in information and entropy compared to the other methods.
In addition, Table shows that our calculated information and entropy differed by two orders of magnitude from the original paper. We have reported this discrepancy to the authors, who indicated that their paper included mistake and sent us the updated values for these metrics.
Using these updates values, in Table we present the absolute difference between our results and those of the authors, divided by the standard deviation reported in the original paper. We refer to these as the diff- to-std ratios henceforth. For AUR and AUP, we used the standard deviations reported in the original paper, whereas we use the updated standard deviations for information and entropy updated by the authors. Many of our results do not lie within 2 times the standard deviation. The significant magnitudes of some of the diff-to-std ratios, such as that of DynaMask, suggest that randomness an unlikely cause. The diff-to-std ratios of ExtremalMask (CE) seem suggest that the values in the original paper are obtained by training ExtremalMask using MSE.
4.2
Claim 2
ExtremalMask best identifies salient and non-salient features on MIMIC-3
we observe that ExtremalMask outperformed all other tested methods on the MIMIC-III dataset across all four relevant metrics, thereby substantiating Claim 2 In addition, Section A.5 shows that this result held across variations of saliency ratios of 10%, 30% 40%, and 60%, as well as, replacing the identified salient data with instead of its average. The diff-to-std ratios in Table 2 once again highlight a significant deviation of our DynaMask result from those reported in the original paper, when compared to other methods.
In Table 2|
Our results reported on the MIMIC-III dataset as average + standard deviation over 5 folds. Columns labeled with the suffix -R denote diff-to-std ratios, obtained by dividing the difference between and the original paper's 's results by the standard deviation from the original paper. Differences falling outside 2 standard deviations are underlined.
Table 2:
Method
Acc
Acc-R
Comp 1
Comp-R
CE 1
CE-R
Suff [
Sul--R
Ang. Occlusion
0.987 = 0.002
2,00
L68E-03 7.21E-04
1.17
9.83E-02 = 6.75E-03
0.105
2.27E-03
2.12E-03
0.201
DeepLift
0.987 = 0.002
0.500
8.98E-01
1.65E-03
1.11
9.71E-02 = 7.15E-03
0.106
3.21E-03
7.88E-01
0.302
DynaMask
0.990 = 0,002
0
1.31E-I3 + 9.52E-01
5.33
0.1 + 0.006
0.697
5.40E-01
1.48E-03
1.71
IG
0.986 = 0.003
0.667
1.50E-03 + 1.92E-03
0.851
9.84E-02 + 7. 7.19E-03
0.13
2.62E-03
+: 5.42E-04
0.339
Occlusion
0.987 1 0.002
1.00
-3.60E 04 1. 9.90E-04
2.02
9.62E-02
1 6.73E-03
0.194
4.58E. 03
1.68E-08
7.52E-03
Retain
0.987 = 0.003
2,00
22,08E-03
1.57E-03
9.49E-02
= 7.33E-03
0.337
7.61E-03
8.91-00
9.91E-02
ExtremaiMask
0.983 0.003
0.633
1.23E-02 + 2.16E-03
1.62
0.114 0.009
1.07
-8.07E-03
2.21E-03
0.927
Table 3: Our results reported on the HMM dataset for the two optimization problems as average + standard deviation over 5 folds. Columns labeled with the suffix -R denole diff-to-std ratios, obtained by dividing the difference between our and the original paper's results by the standard deviation from the original paper. Differences falling outside 2 standard deviations are underlined.
Game
AUP
AUP-R
AUR
AUR-R
J-R.
E
E-R
Preservation
0.904 = 0.010
0.633
0.760 + 0.004
1.62
2.93E+5 + 4.51E+3
1.07
7.51E+3 + 183
0.927
Deletion
0.341 1 0.001
1.67
0.898 1 0.002
2.92
2.52E 1 1.56E
0.781
1.21+4 191
0.220
7

---

Published in Transactions on Machine Learning Research (06/2024)
Table 4: Our results reported on the MIMIC-I1I dataset for the two optimization problems as average = standard deviation over folds. Columns labeled with the suffix -R denote diff-to-std ratios, obtained by dividing the difference between our and the original paper's results by the standard deviation from the original paper. Differences falling outside 2 standard deviations underlined.
Game
Aoc
Acc-R
Comp f
Comp-R
CE 1
CE-R
Sulf +
Suff-R
Preservation 0.984 0.003
0.75
1.14c-02 1 0.001
1
0.11
0.006
0.875
-8.38e-03 1 0.001
Deletion
0.996 + 0.001
2.5
-0.003 0.001
0.5
0.088
0.003
0.455
0.010 + 0.004
0.5
Table Information and entropy of the two saliency metrics over 5 folds.
Saliency metric
M
2.90 4.51 3.33E+5 = 2.99E+3
7.51++3 1888+2 7.29E+3 1.27E+2
4.3
Claim 3
ExtremalMask performs better in preservation than deletion game both datasets
Table 4 illustrates that the preservation game outperforms the deletion game across all metrics except for AUR on HMM. On the other hand, the presorvation game outperforms the deletion game across all metrics on MIMIC-III. A possible explanation for the preservation game's 's superiority lies in its avoidance of additional assumptions regarding the classifier, as highlighted in [Section 3.1 Concluding which assumption regarding the data holds true for the two methods requires a rigorous definition of what it means for the data to sufficiently close or far away from 0. Formalizing these assumptions would help determine which implemented optimization problem is more suitable for specifie dataset, supplementing less ges-generalliaable empirical study.
4.4 Extension 1
Comparative study between saliency metrics
In Table 5] we observe that the perturbations learned by ExtremalMask, as quantified by the alternative saliency metric (S); is more informative for identifying the salient data compared to its proxy, i.e. the mask (M).
Figure I visualizes the saliency maps and the salient data generated using the two saliency metrics for an arbitrarily-selecter correctly-classifee dead patient, specifically patient 64 from fold 0. Our visual analysis yielded the following conclusions:
The alternative saliency values tend to be significantly higher and less polarized than the mask values. This suggests that the noise learned (output of NN) tends be close to the original data, resulting in limited perturbations.
2. The alternative saliency values tend to be less sporadic than the mask values. This suggests that features identified as salient at one point in time persist in their saliency over extended period with alternative saliency. This stability in salient features over time allows for simpler and more interpretable explanations.
Next, we performed global analysis to identify the most salient features. Figure shows substantial differences in magnitudes among attributions calculated using different saliency metrics. One would expect that lower mask values lead to larger perturbations and lower alternative saliency values. In turn, this would result in the mask being similar in value as alternative saliency. The observed discrepancy, however, is justifiable when the noise (output of NN) learned by ExtremalMask is, on average, close to the original data, limiting the impact of the mask on the size of the perturbations.
Morcover, we note that the confidence intervals for the mask values per feature are larger than those for alternative saliency. This indicates that the mask values tend to be more polarized over time compared alternative saliency, which aligns with analysis for the saliency maps of patient 64.
8

---

Published in Transactions on Machine Learning Research (06/2024)
Mask
Altemative saliency
1.0
Feature number
Feature number
0.4
(a) Saliency maps generated by the two different saliency metrics.
Mask
Alternative saliency
Festure number
Feature number
Salient sal-salient
Time
Time
(b) Binary saliency maps where only top 20% highest saliency values are considered as salient.
Figure Saliency maps of two saliency metrics for a correctly classified dead patient (pationt 64, fold 0). These maps show saliency of different features different times for predicting the patient's 's mortality.
Alternative saliency
decay
0.30
Uneer decay Esporantial decay
0.575
5.25
0.950
0.925
0.15
0.005
0.50
8.55
020
0.000
lore 433.
Aleca car
NEMONI ae antted
Figure 2: Attribution for feature is the weighted averages of its saliency over time across all patients in the MIMIC-III test. dataset, where the weights correspond to different form of decays introduced in Section 3.2.1 The error bars are the 95% confidence intervals of the Student's 's t-distribution. A higher value attribution implies an identified greater importance for the classifier's prediction.

---

Published in Transactions on Machine Learning Research (06/2024)
We found that some features seemed to be sensitive the decay used. Certain features had higher attribution when using exponential or linear decay relative to decay, indicating that they grew in salience over time. Other features were relatively invariant- their attributions overlapped using 3 forms of decays. These findings may enable practitioners to learn from or to debug the classifier. For example, we found that temperature became less relevant over time. This conflicted with our expectation that. lemperature measurements later in the time series (occurring closer to death) would be more important. Equally surprisingly, we found that gender became more salient over time, whereas we had expected gender salience to be time-invariant. These findings may reflect an error in either the classifier in naively interpreting the results of ExtremalMask.
4.5 Extension 2
Are the perturbed data of ExtremalMask realistic?
Finally, found that the samples perturbed by ExtremalMask lend to be within distribution. Figure 3 shows that most of the perturbed samples more probable than their original unperturbed counterparts, In fact, we observe that for any À 2 0 in Equation 4 every perturbed data point is not considered as OOD.
-160
-17
-180
-190
-200
Log
-220
230
-20
Original
Perturbed
Figure 3: Boxplot of the log-pre log-probability of the original and perturbed test data of the modified HMM. The higher the value, the more probable the datapoint.
Centered values
1-
Original NSF 1
Original NSF Perturbed NSF Perturbed NSF
Figure Boxplot of the difference between non-salient features and the mean of the fcature's normal distribution for the original and perturbed test data of the modified HMM dataset. NSF stands for non- salient feature, where NSF is the first feature (which remains non-salient throughout time) and NSF 2 is the combination of nal-salient parts of feature 2 and 3 (see Section A.1.1 for ground-truth label generation).
10

---

Published in Transactions on Machine Learning Research (06/2024)
Figure 4 provides explanation for why the perturbed data became more probable. Feature 1 of the perturbed data is clustered noticeably closer to the mean of its distribution, as evidenced by the small width of the feature's zero-centered IQR. On the other hand, we had anticipated that ExtremalMask would identify the non-salient parts of features 2 and 3, and that, after perturbation, these values would have a similarly narrow TQR. However, the IQR of these parts appear to be relatively large the figure, suggesting that ExtremalMask struggled to identify which parts of these features were non-salient.
In summary, ExtremalMask does not seem to generate OOD perturbations the modified HMM dataset, thereby reinforcing the credibility of the associated salient data identified. This implies that ExtremaLMask may generate comparable realistic perturbed data on analogous, relatively straightforward distributions.
5 Discussion
This paper presents a reproducibility study Enguehard 2023 and our findings generally support the conclusions of the original paper. With our extensions, we (1) proposed a new saliency metric and offered an analysis on the relation between noise and the original data on MIMIC-III, and (2) found out that perturbed data learned by ExtremalMask on the modified HMM dataset more probable than the original data.
We would suggest the following areas for future research:
Formalizing sufficiently close or far away that are present as assumptions for the preservation and deletion game introducce Enguehard (2023).
Domain experts could be consulted to validate whether the generated explanations appropriately identify salient features. For instance, considering the top-k identified salient feat from Figure 2 would doctor also agree that these features are most important for determining patient mortality?
It seems to be possible to get rid of the mask and instead just add the noise to the data as the expressiveness of ExtromalMask with this modification would remain unchanged. In this caso, the only natural saliency metric would be allernative saliency.
5.1 What was easy
Overall, the authors maintained very well-structured and cellmmouularized code base with good documen- fation. They maintain this code base publicly available resource for time series experiments, testament to their dedication to reproducibility. To run the HMM experiment, we simply had to follow the documen- tation to create conda environment (using the provided yml file) and run the provided experiment shell script. It was also easy to imagine extensions because the original paper had a relatively straightforward premise with deep implications.
5.2 What was difficult
We spent some days debugging the difference between our results for entropy and information and the incorrect values reported in the original paper. We also had some difficulty loading check-pointed models due to differences between CUDA and CPU devices well the relatively old Porch version used in the tint library's environment.
Finally, like most new methods, the theoretical basis of this method is not fully developed. This made it diffieult. to build upon previous theory in order to create a rigorous analysis.
5.3 Communication with original authors
We contacted the authors twice by email, who gave us swift and thorough foedback to several lengthy theoretical and reproducibility questions. The authors have also briefly reviewed this manuscript before our submission and did not point out major issues.
11

---

Published in Transactions on Machine Learning Research (06/2024)
References
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statisti- cal machine translation, 2014.
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. RETAIN: in- terpretable predictive model in healthcare using reverse time attention mechanism. CoRR, abs/1608.05745, 2016. URL http://arxiv..ort www.aaa........./..
Jonathan Crabbé and Mihaela Van Der Schaar. Explaining time series predictions with dynamic masks, In International Conference on Machine Learning, pp. 2166-2177. PMLR, 2021.
Joseph Enguchard. Learning perturbations to explain time series predictions.
arXiv preprint
arXiv:2305.1880 18840, 2023.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Francesca Perla, Ronald Richman, Salvatore Scognamiglio, and Mario V Wüthrich. Time-series forecasting of mortality rates using deep learning. Scandinavian Actuarial Journal, 2021(7):572 598, 2021.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ' "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135 1135-1144 1144, 2016.
Wajiha Safat, Sohail Asghar, and Saira Andleeb Gillani. Empirical analysis for crime prediction and fore- casting using machine learning and deep learning techniques. IEEE access, 9:70080 000000 2021.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145 3153. PMLR, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319 3328. PMLR, 2017.
Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. 1EEE transactions on neural networks and learning systems, 31(11):479334411 2020.
Sana Tonekaboni, Shalmali Joshi, Kieran Campbell, David K Duvenaud, and Anna Goldenberg, What went wrong and when? instance-wise feature importance for time-series black-box models. Advances in Neural Information Processing Systems, 03:799-8003 2020.
Alfredo Vellido. The importance of interpretability and visualization in machine learning for applications in medicine and health care. Newral computing and applications, 12(24):18069-188 2020.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Rision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 818 833. Springer, 2014.
12

---

Published in Transactions on Machine Learning Research (06/2024)
A
Appendix
A.1
HMM dataset
In this section, we describe the specific parameters used for the two HMM datasets described in Section A.I.I and Section A.1.2
A.1.1
Reproducibility study
Recall the 2-state HMM with hidden state at time t 8, E {0, as described Section 3.2.1 initial distribution vector and transition matrix time t, E,, are defined as
The associated
0.95 (((s:) 0.95
0.05
== (0.5,0.5)
E.
((sx)) 500
0.05 + 500
where, for 1 1 m t, ((;)) = m and only if s, 1 for all t m S i and 8t-m-1 = 0. To generate single time-sories sample x E Tx we have X2 NN ,,,) with
0..1
0.1
0.801
0
0.801
0.01
N4
1.6
0.4
Eg
0.801
0.01
EE
0.01
0.801
0.5
1.5
0.01
0.801
0
0.801
A.1.2
Additional details for Section 3.2.2
The modified HMM differs from the original in the following properties:
Shortened sequence length, T=50
2. New hidden state transition probabilities:
0..95 Pt
0.05 P:
Σ,
0.05 + Pt 0.95 Pt
where
Pt 4/500
A.2 The Forward algorithm (FA)
FA is a dynamical programming algorithm used to calculate the marginal probability of a sequence of labels generated by a HMM. This probability marginal in the sense that it's calculated by integrating over all possible hidden state sequences. FA reduces the time complexity of calculating the marginal label probability from exponential (in hidden state sequence length) to linear.
Let N be the number of different hidden states, K the number of labels, and M the sequence length of HMM. Furthermore let (f and (Hx) be the random variables representing the labels and hidden states respectively. Let (44) be the observed label sequence. We assume that the hidden states take values in N7. Let p represent the joint distribution of all hidden states and labels. The main idea of the FA is the definition:
def =
au
p(L1 = L2 = L,,,,., = 4, H. =
(6)
And the following set of equations:
01, = nTp(<1 = 1 H1 =
(7)
0.j =
Σ
0 0-11op(H, = - -
(8)
13

---

Published in Transactions on Machine Learning Research (06/2024)
Where xj the initial hidden state probabilities. The second and third term inside the sum in equation g are called the transition and emission probabilities. The FA consists of filling the M N matrix a row by row. The final result
pLLi = 11 L2 12, ..., LM - IM)
(9)
is given by the sum of its last row.
A.3 Time used per claim
Hours
Claim 40
Claim 2
40
Claim 3
1
Table 6: Approximated number of hours used per claim Snellius supercluster
A.4 Weights for different decays in Section 3.4.3
1.0
0.8
0.6
Weight
0.4
Linear decay
Exponential decay
10
40
Figure 5: Weights of the different forms of decays described in Section 3.4.3 with time horizon 48.
14

---

Published in Transactions on Machine Learning Research (06/2024)
A.5 Complete metrics of MIMIC-III experiment
Mask
Average Mask
Comprehensiiveeee
0.10
0.015
Denenal Masc
Retain
0.010
Occlision
0.05
0.005
0.00
Entropy
0.12
0.11
0.2
0.09
0.990
0.96
0.905
0.94
0.980
0.02
Suffciency
0.00
0.005
-0.02
0.000
00.04
-0.110
0.1
0.2
0.3
0.4
0.5
0.6
0.1
0.2
0.3
0.4
0.5
0.6
Percentage
Percentage
Figure Results on MIMIC-III using different variations of the experiment. Each row corresponds to a different metric. For the left column, masked the top x% of values using zero as a mask. For the right column, used the feature's sample average. Tn all variations, MxtremalMask had the best performance (lowest accuracy and sufficiency, highest comprehensiveness and cross entropy).
15

---

Published in Transactions on Machine Learning Research (06/2024)
A.6 Additional visualizations of MIMIC-III experiment
Patient 33, prediction = survive
Method: ExtremalMask
Method: ExtremalMask, alternative saliency
Feature number
Feature number
1.0
Saliency Mask intensity
0.6
Time
Time
Method: DynaMask
Method: DynaMask, alternative saliency
0.4
0.2
Feature number
Feature number
Time
Figure 7: The left-hand side plots of this figure show original mask saliencies retrieved for ExtremalMask and DynaMask methods. The right-hand hand side plots of this plot show alternative saliency. These masks are shown for correctly predicted patient who survived. The green the point, the more important it for f's prediction for this sample.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Patient 64, prediction
Method: ExtremalMask
Method: DynaMask
Feature number
Feature number
Time
Time
Method: Deep Lift
Method: Augmented Occlusion
0.8
Saliency Mask intensity
Feature number
Feature number
0.6
8100 1ES--/-----------
0.2
Time
Time
Method: Retain
Method: Integrated gradients
Feature number
B00 51 240 tor to BOR
Feature number
40
Time
Time
Figure 8: Comparison of mask-only saliency results from different methods. We show sample 64 from MIMIC-III, seed 42, fold 1. The patient was correctly predicted by the classifier f to die. The more green the point, the more important it for f's prediction for this sample.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Patient 33, prediction survive
Method: ExtremalMask
Method: DynaMask
Feature number
Feature number
Time
Time
Method: Deep Lift
Method: Augmented Occlusion
0.8
Saliency Mask intensity
Feature number
Feature number
0.6
8100 Www.://1------------
0.2
Time
Time
Method: Retain
Method: Integrated gradients
Feature number
Feature number
40
Time
Time
Figure 9: Comparison of the mask-only saliency results from different methods. We show sample 64 from MIMIC-III, seed 42, fold The patient was correctly predicted by the classifier f to survive, the more green the point, the more important it was for the classifcation.
18

---

Published in Transactions on Machine Learning Research (06/2024)
Patient 64, prediction die. Top most important features
ANION GAP
PLATELET BICARBONATE
1.0
Saliency Mask intensity
0.6
Sy58P
0.0
2
spo2
Perturbed data
Original
18
20
21
22
24
25
28
30
Figure 10: Detailed plot showing the original data, predicted perturbation by ExtremalMask, and the
predicted mask-only saliency by ExtremalMask. The results are shown for top 5 most important features as identified by Figure?
Patient 64, prediction die. Top most important features. Alternative saliency
ANION GAP
BICARBONATE
1.0
Saliency Mask intensity
0.8
PLATELET
0.4
SysBP
0.2
0.0
Sp02
Perturbed data
Original data
10
15
20
21
24
25
26
27
28
30
31
Figure 11: Detailed plot showing the original data, predicted perturbation by ExtremalMask, and the
alternative saliency computed in Equation 4
The results shown for top most important features
identified by Figure 2
19

---

Published in Transactions on Machine Learning Research (06/2024)
Patient sample 64, prediction die, feature ANION GAP
Learned mask/saliency: m
1.0
0.5
0.0
30
Learned perturbation: NN(x)
0.0075
0.0050
0.0025
10
20
30
40
m X
2
1
10
20
30
40
(1 - m) NN(x)
0.005
0.000
10
20
30
40
Perturbed vs Original data
2
x + (1- m) NN())
Original data
0
10
20
30
40
Time
Figure 12: Visualization Equation Equation 2 components. As in the main text, the 64th sample from MIMIC-III was used with seed 42 and fold 0. We focus the 2nd feature (as e.g. in Figure la and Figure 10).
20

---

Published in Transactions on Machine Learning Research (06/2024)
A.7
Visualizations of whitebox HMM experiments
Mask coefficients time
Method: true
Method: extremal
1.0
Saliency Mask intensity
Feature number
Feature number
0.8
0.6
0.4
0.2
27
30
45
54
63
72
81
90
99
108
117
126
135
144
153
162
171
180
189
198
18
36
23
54
63
72
90
99
108
117
126
135
144
153
162
171
180
189
196
Time
Time
Figure 13: True and predicted saliency by ExtremalMask for HMM dataset. We show sample 12 for seed
42 and fold
Predicted saliency
Mar My
My MM
m-r" La Worn
Feature
1.
Saliency Mask intensity
2.5
Feature Feature
0.8
0.0
-2.5
0.2
Perturbed data Original data
M
0.0
-2.5
0
40
88
82
100
108
112
116
150
128
136
140
144
148
152
156
160
164
168
172
176
190
184
188
192
196
Figure 14: Predicted saliency and perturbation by ExtremalMask for HMM datasct. We show sample 12 for
seed = 42 and fold 0.
True saliency
Feature 0
My
M W
Saliency Mask intersity
0.0
0.4
2.5
Perturbed data Original data
0.0
22
60
44
48
60
5
88
g
96
100
104
108
112
116
136
140
104
148
152
156
180
164
168
172
176
180
188
192
196
Figure 15: True saliency and perturbation by MatremalMask for HMM dataset.
We show sample 12 for seed
= 42 and fold 0.
21


Reviews:
[{"review_id": "BmUIB9rlEK", "reviewer": "Reviewer_xnfk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a high quality reproduction, and the discussion and motivation (and the details) of the original work are well-presented. I like the additional ablations, and discussion of the modifications made. I like the intuition presented behind various formulations. \n\nOverall, the topic is of course interesting, and the paper adds some nice value beyond the original work. \n\nThe writing is high quality, though I sometimes feel the emphasis is on intuition rather than mathematical rigor (especially in the early part).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a reproducibility study of an algorithm that optimizes over sequence perturbations to identify influence in time-series predictions done by neural nets. The authors reproduce the original work, and experiment with a few extensions they propose, and list the key implications of the research and explain what was more difficult to reproduce.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Seems like very good quality reproduction to me. However, I like the comparison with the choices made in the original paper, but perhaps move that a bit earlier?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7g8MVO5FNQ", "reviewer": "Reviewer_6VRP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Though I've never reviewed a reproducibility study before, this paper comes across to me as a model for how to do it. The original paper's claims are clearly laid out and thoroughly tested. The setup and methodology is carefully described, along with the original motivation. Where the authors' design choices deviate from this, they give clear reasoning for each modification they make to the to original method/evaluation. They further propose several extensions or possible improvements, and they run ablations to compare.\n\nI was especially impressed to see a conclusion which laid out very clearly which elements of the reproduction offered difficulty and why, as this seems invaluable to someone who may be trying to reproduce the idea themselves.\n\nIn terms of weakness, I would only give one minor word of caution: some of the language in the expository sections can be a little loose and informal when discussing precise mathematical equations. Unfortunately I think this is not uncommon in the explainability literature, but I recommend the authors take care to try to avoid this. As one example, see the discussion after Equation (2). The first sentence is quite vague, and the \"justification\" later on feels very hand-wavy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a detailed reproducibility study of a particular recent method which optimizes over sequence perturbations to highlight influence in time-series predictions of neural networks. The authors establish an explicit set of claims by the authors of the original work and test each of these claims, as well as experimenting with a few extensions/improvements of their own. They conclude with a summary of the important takeaways as well as a discussion of which elements of the reproduction were easy/difficult and why.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None, see recommendation above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JU1XiPDh8i", "reviewer": "Reviewer_6YMW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors reproduce the experiments in the original paper with an attempt to facilitate fair comparison with other methods. For example, in Claim 1, the proposed method (ExtremalMask) and DynaMask were tested after training with CE loss, although only MSE loss was provided in the original code. \n- The three claims made from the original paper were well supported through the reproduced experimental results and detailed explanations. The major discrepancies were also well addressed, such as the information and entropy differing by two orders of magnitude being reported to the authors to update the values. \n- The two extensions are very important additions to the original paper, because the first one conducts a comparative study between the proposed and original saliency metrics, and the second one conducts experimentation on if the perturbed data of ExtremalMask is realistic. The qualitative analysis in both extensions are easy to follow and supports the effectiveness of each extension. \n\nWeaknesses:\n- The visualizations in Figure 1 are not straight-forward to understand, although it can be inferred that the Alternative saliency is more informative than the mask. More detailed explanation on what the visualizations depict would be helpful. \n- Although there were resource limitations mentioned by the authors, it is still important to match all experimental settings with the original paper. Therefore, like the original paper which conducted five folds, the authors should have conducted five folds instead of three.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a reproducibility study of the paper “Learning Perturbations to Explain Time Series Predictions” with detailed performance evaluations and extensions that develop many areas of the original paper, such as the metric, distribution analysis of the proposed perturbations, and potential future research directions. During the reproduced performance evaluations, the authors were able to confirm with the original authors mistaken reports in the original paper, and provided valuable insights with a comparative study between the original and proposed saliency metric.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More detailed explanations of the components in the Figure 1 visualizations would strengthen the work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6SYTqcgfVm", "reviewer": "Reviewer_GHn8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nAuthors adequately provide context of the main paper before the methods, results, experiments\nProvides code for the redone experiments as well as the newly run experiments\n\nWeaknesses:\nSee requested changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript is a reproducibility study of Enguehard’s 2023 Learning Perturbations to Explain Time Series Predictions. The authors found general agreement on synthetic and real datasets with the original work and provide novel visualizations of the results and offer nuanced interpretations of the results. They also contribute an alternative metric for assessing saliency and extend the experiments to investigate how practical ExtremalMask is.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions:\n- In the original paper there were 5 folds run. In this reproducibility study there were only 3. One of the metrics reported was the difference normalized by the standard deviation of the metric. How meaningful is this if the standard deviation is being estimated with only 3 samples? (even if there were 5 would there be much reason to trust the standard deviation estimate very much?)\n- “To replicate the results in the original paper, we adopt the default hyperparameters provided by the tint library” Are these the same hyperparameters that were used in the original paper?\n- How was 20% chosen as the threshold for saliency when selecting features? Wouldn’t the appropriate thresholding value be dataset dependent?\n- In what setting would you expect the newly introduced saliency metric to be more informative than the one used in the original Enguehard paper? In what settings would it be less informative?\n- I would appreciate more discussion of the results shown in Figure 2. The values for the alternate metric tend to be higher than for the mask. Should there be equal attribution in totality (across all features) across both metrics? Apologies if I am misunderstanding the plot/attribution\n\nMinor: \n- Figure 2 is a little bit difficult to read. The lines overlap each other so the ranges of values is not always clear/easy to read at a glance. The font in the figure is also small\n- Occasional typos, grammatical errors", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attempts to reproduce the results in an earlier paper. The results obtained by this paper support the claims in the original paper. The paper was reviewed by 4 reviewers, who are all positive after the rebuttal.
Decision: Accept (poster)