Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a new method computing the importance of features in time series, called Feed Forward Counterfactual (FFC).
In previous work, the explainability problem in time series was tackled with feature occlusion (FO), sensitivity analyses (SA) methods. However, previous counterfactual based methods do not carefully consider appropriate conditional distribution and generate out-of-distribution counterfactuals.
The proposed FFC method addresses this issue by leveraging a generative model which learns the underlying dynamics and generates a realistic counterfactual given the past observations. FFC is evaluated on simulated and real datasets and shows that it is better at localizing important observations over time compared to the other baselines.
In summary, this paper introduces a way of defining the feature importance at every time point. The main idea of this paper follows in line with [Chang et al. 2019] which address the problem of out-of-distribution counterfactuals. Although the experiment shows successes of the proposed method on several datasets, the major weakness of this paper is the lack of technical novelty and detailed analysis of the proposed method. For example,
If the time series is non-stationary, this could incur a different amount of the change in the model output and proposed time importance might not work. How about this?
Did the authors consider trying out with varying size of training data or generator model?
Minor
On page 2, p(\mathbf{x_{t,i}|\mathbf{X_{0:{t-1}}}}) -> p(x_{i,t}|\mathbf{X_{0:{t-1}}}})
On page 6, affected by feature and -> affected by feature 1 and

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: --- Overall ---

This paper proposes a method for evaluating the influence of individual observations on the output of a time series prediction model by replacing each (discrete time) observation with its conditional expectation given the other observations. They evaluate this method qualitatively on synthetic, healthcare, and climate datasets. I reviewed this paper for NeurIPS and was happy to see that the authors have made substantial improvements to the presentation and evaluation of the method. With that said, I think that the methodological contribution is incremental (sampling from a conditional rather than marginal distribution), there is at least one major correctness issue that needs to be addressed, and the analysis of the experiments fails explain why the models perform differently.

--- Major comments ---

1. The Montecarlo approximation in Algorithm 1 does not approximate Imp(i,t). Specifically, because the averaging is done before the absolute value, Algorithm 1 approximates |F(X_{0:t}) - E[F(X_{0:-t},x_{-i,t},\hat{x}_{i,t}]|. This is also a valid measure of feature importance and it is not clear from the paper why we should prefer one over the other.

2. I think the paper needs to do a better job explaining why sampling from the conditional leads to better explanations than sampling from the marginal. The second paragraph makes an argument based on variance, but it is not clear that low variance translates to better explanations. In particular, using mean imputation has very low variance, but I would expect it to give poor explanations. I recommend using a toy example to make this point. For example, in a healthcare context, doctors are reacting to changes relative to a particular patient's baseline. A conditional model can capture this baseline but a marginal model cannot.

3. In general, I thought that the experiments were well done, but stops short of explaining *why* the methods perform differently. Put differently, I think it is really important to clearly explain why certain methods fail while others succeed. For example, the authors demonstrate the sensitivity analysis fails on the synthetic data, but never explain why. I am looking for a statement of the form: "Sensitivity analysis fails on this data because... FFC solves this weakness by doing... which is reflected in the experimental results."

4. In 4.2.1, it is very unsurprising to me that a model that samples from an approximation of the conditional has higher likelihood under the conditional than samples from the marginal, but why should we expect this to lead to better explanations?

5. I thought the idea of looking at feature importance just before clinician intervention was a very clever evaluation, but I wanted the qualitative evaluation to go one step further. That is, does bicarbonate being the most important feature just before administration norepinephrine and fluids make clinical sense? Is this picking up on a specific condition and if so what condition? A clinician could tell you what they are typically reacting to when they administer fluids or vasopressors and you can compare what they say to what the model says. I was surprised to see the top features all being lab measurements as opposed to vital signs. In particular, in a vacuum, I would expect systolic blood pressure to be the most important feature in both of these cases. Is it possible that the frequency of measurements affects which features are selected as important?

6. I thought GHG experiment was *much* better and clearer in this version of the paper. Well done.

7. I recommend moving the notation from the appendix to the main paper. I don't think a reader should have to reference another document to follow notation.

--- Minor comments ---

1. Pg. 3 "The magnitude of our...": I call the authors' definition of feature importance absolute not relative. I would expect a "relative importance" to be a ratio of some sort (e.g. relative risk). 

2. Pg. 5: Figure 8 --> Figure 3
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes an extension of feature occlusion (FO) [Suresh et al., 2017] in which they sample from a pre-trained generative model for replacing the observed variables. Technically, it is in the category of saliency maps, only with possibly larger perturbations defined by the generative model. Thus, it most likely should inherit the same properties as the saliency maps.

The authors experiment on both synthetic and real-world datasets. They use log-probabilities of the generated samples as a metric for the quality of the counterfactuals; however, because not all baselines are based on counterfactuals, this approach has limited usefulness. Also, it seems that the log-probabilities are too small, indicating that most likely the authors have reported the sum log-probabilities instead of average probabilities.

Given the similarity to the saliency maps, the authors should have tested the proposed method in the sanity checks in [1]. Also, the authors should have examined the robustness of the proposed explanations given the adversarial vulnerability phenomenon.

Despite sampling from a generative model, because of the univariate nature of the counterfactuals used in this paper, the process might create invalid data points. For example, increased blood sugar is usually correlated with increased blood pressure. However, this method does not account for the correlation among the features.

Finally, this method should be very slow to run. The authors should have compared the run-time speed of the algorithms in the experiment section, too.

[1] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In NeurIPS.
review_title: Official Blind Review #1

Meta-review: The paper proposes a definition of and an algorithm for computing the importance                                                   
of features in time series classification / regression.                                                                            
The importance is defined as a finite difference version of standard sensitivity                                                   
analysis, where the distribution over finite perturbations is given by a                                                           
learned time series model.                                                                                                         
The approach is tested on simulated and real-world data sets.                                                                      
                                                                                                                                   
The reviewers note a lack of novelty in the paper and deem the contribution                                                        
somewhat incremental, although exposition and experiments have improved compared                                                   
to previous versions of the manuscript.                                                                                            
                                                                                                                                   
I recommend to reject this paper in its current form, taking into account on the reviews and my own                                
reading, mostly due t a lack of novelty.                                                                                           
Furthermore, the authors call their method a "counterfactual" approach.                                                            
I don't agree with this terminology.                                                                                               
No attempt is made to justify is by linking it to the relevant causal literature                                                   
on counterfactuals.                                                                                                                
The authors do indeed motivate their algorithm by considering how the classifier                                                   
output would change "had an observation been different" (a counterfactual), but                                                    
mathematical in their model this the same as asking "what changes if the observation is                                            
different" (interventional query). 
Decision: Reject