Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The aim of this work is to improve interpretability in time series prediction. To do so, they propose to use a relatively post-hoc procedure which learns a sparse representation informed by gradients of the prediction objective under a trained model. In particular, given a trained next-step classifier, they propose to train a sparse autoencoder with a combined objective of reconstruction and classification performance (while keeping the classifier fixed), so as to expose which features are useful for time series prediction.  Sparsity, and sparse auto-encoders, have been widely used for the end of interpretability. In this sense, the crux of the approach is very well motivated by the literature.

* Pros
	* The work provides extensive comparison to a battery of other methods for model prediction interpretation. 
	* The method is conceptually simple and is easy to implement. It is also general and can be applied to any prediction model (though this is more property of the sparse auto-encoder).
	* Despite its simplicity and generality, the method is shown to perform well on average, though it sometimes performs significantly worse than simple baselines.

* Cons
	* The method itself is not explained very well. The authors use language such as “attach the auto encoder to the classifier”, which is a bit vague and could mean a number of things. It would be helpful if they provided either a formal definition of the model or a architectural diagram.
	* Though the quantitative evaluation is not entirely flattering, the authors should not be punished for providing experiments on many datasets. That said, if their contribution is then rather one of technical novelty, i.e. a sparse-autoencoder-based framework for time series interpretability, it would be helpful for them to 
		* More formally define their framework / class of solutions
		* Provide a more in depth study of possible variants of the method (this is elaborated on in the “Questions” section)
		* More strongly argue the novelty of their method
	* The authors provide a discussion on automatic hyper-parameter tuning that seems a bit out of place in the main method section, since it is not mentioned much thereafter and is claimed to not bring benefits.
	* The qualitative evaluation made by authors is rather vague:
		* "Alongside the numbers, TSInsight was also able to produce the most plausible explanations”
	
* Additional Remarks
	* Why not train things jointly? Does this have to be done post-hoc? The authors state that they “should expect a drop in performance since the input distribution changes” -> so why not at least try fine-tune and study the effect of training the classifier with sparse representations end-to-end? Exploring whether things can be trained jointly, or in other configurations, might allow the authors to frame their work as more of a general technical contribution.
	* It would be nice to have the simple baseline of a classifier with a sparsity constraint, i.e. 
		* I.e. ablate the reconstruction loss

I’ve given a reject because 1) the explanation of the method is not very precise and could be greatly improved, 2) the quantitative evaluation is not sufficiently convincing, given the lack of technical novelty), and 3) the qualitative evaluation is hand-wavy. 
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper presents a new approach for improving the interpretability of deep learning methods used for time series. The is mainly concerned with classification tasks for time series. First, the classifier is learned in a usual way. Subsequently, a sparse auto-encoder is used that encodes the last layer of the classifier. For training the auto-encoder the classifier is fixed and there is a decoding loss as well as a sparsity loss. The sparse encoding of the last layer is supposed to increase the interpretability of the classification as it indicates which features are important for the classification.

In general, I think this paper needs to be considerably improved in order to justify a publication. I am not convinced about the interpretability of the sparse extracted feature vector. It is not clear to me why this should be more interpretable then other methods. The paper contains many plots where the compare to other attributes methods, but it is not clear why the presented results should be any better as other methods (for example Fig 3). The paper is missing also a lot of motivation, the results are not well explained (e.g. Fig 3) and it needs to be improved in terms of writing. Equation 1 is not motivated (which is the main part of the paper) and it is not clear how Figure 2a has been generated and why this represented should be " an interesting one, doesn’t help with the interpretability of the model". The authors have to improve the motivation part as well as the discussion of the results.

More comments below:
- The method seems to suffer from a severe parameter tuning problem, which makes it hard to use in practise.
- It is unclear to me why the discriminator is fixed during training the encoder and decoder. Shouldnt it improve performance to also adapt the discriminator to the new representation.
- Why can we not just add a layer with a sparsity constraint one layer before the "encoded" layer such that we have the same architecture  and optimize that end to end? At least comparison to such an approach would be needed to justify something more complex. 
- The plots need to be better explained. While the comparisons seems to be exhaustive, it is already too many plots and it is very easy to get lost. Also, the quality of the plots need to be improved (e.g. font size)


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed an algorithm for identifying important inputs for the time-series data as an explanation of the model's output.
Given a fixed model, the authors proposed to put an auto-encoder to the input of the model, so that the input data is first transformed through the auto-encoder, and the transformed input is then fed to the model.
In the proposed algorithm, the auto-encoder is trained so that (i) the prediction loss on the model's output to be small, (ii) the reconstruction loss of the auto-encoder to be small, and (iii) the transformed input of the auto-encoder to be sufficiently sparse (i.e. it has many zeros).

I am not very sure if the proposed algorithm can generate reasonable explanation, for the following two reasons.

First, the auto-encoder transforms the input into sparse, which can completely differ from any of the "natural" data, as shown in Fig1(b).
I am not very sure whether studying the performance of the model for such an "outlying" input is informative.

Second, it seems the authors implicitly assumed that zero input is irrelevant to the output of the model.
However, zero input can have a certain meaning to the model, and thus naively introducing the sparsity to the model input may bias the model's output.

I think the paper lacks deeper considerations on the use of sparsity.
Thus, to me, the soundness of the proposed approach is not very clear to me.


### Updated after author response ###
Through the communication with the author, I found that the author seems to be confusing the two different approaches *sparsifying the feature importance* and *sparsifying the input data*.
This paper considers the latter approach, which can be biased as I mentioned above.
The authors tried to justify the approach by raising related studies, which is however the mix of the two different approaches.
I expect the author to clarify the distinction between the two approaches, and provide strong evidences that the sparsity is not a harm (if there is any).
review_title: Official Blind Review #1

Meta-review: Main content:

Blind review #2 summarizes it well:

The aim of this work is to improve interpretability in time series prediction. To do so, they propose to use a relatively post-hoc procedure which learns a sparse representation informed by gradients of the prediction objective under a trained model. In particular, given a trained next-step classifier, they propose to train a sparse autoencoder with a combined objective of reconstruction and classification performance (while keeping the classifier fixed), so as to expose which features are useful for time series prediction.  Sparsity, and sparse auto-encoders, have been widely used for the end of interpretability. In this sense, the crux of the approach is very well motivated by the literature.

--

Discussion:

All reviews had difficulties understanding the significance and novelty, which appears to have in large part arisen from the original submission not having sufficiently contextualized the motivation and strengths of the approach (especially for readers not already specialized in this exact subarea).

--

Recommendation and justification:

The reviews are uniformly low, probably due to the above factors, and while the authors' revisions during the rebuttal period have improved the objections, there are so many strong submissions that it would be difficult to justify override the very low reviewer scores.
Decision: Reject