Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to provide salient time steps for probabilistic time-series forecasting tasks. It builds on Miller's framework of explanations and establishes a link between counterfactual reasoning and saliency-based explanation technique; it proposes a unifying definition of information-theoretic saliency for forecasting tasks. Also, the paper empirically shows that the proposed method recovers the true salient timesteps for a synthetic dataset.
strength_and_weaknesses: Strength:

This paper proposes a reasonable and principle solution to provide salient time steps for time-series forecasting tasks.

It considers a probabilistic forecasting model which is critical for robust predictions.

The proposed method does not need explicit training but obtains a closed-form solution which easy to compute.

Weakness:

The paper lacks a quantitative evaluation of the quality of the salient maps. 

The paper did not compare with existing methods, e.g., gradient-based methods and attention-based methods. They are quite straightforward to be extended to the forecasting tasks.

More datasets are expected to show the effectiveness of the method.
clarity,_quality,_novelty_and_reproducibility: The presentation of the paper is clear and the proposed method is technically sound. However, it lacks rigorous empirical evaluations.
summary_of_the_review: This paper proposes a new method to obtain salient timesteps for forecasting tasks. The proposed method is simple and has a closed-form solution. However, the paper lacks rigorous empirical evaluation with existing methods to demonstrate the effectiveness of the proposed method.

[After rebuttal] I do not think the updated draft has addressed the concerns about the empirical evaluation part. Therefore, I keep my original score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission provides a new information-theoretic approach to inferring salient temporal sequences for counterfactual explanations. Conceptually the authors closely follow the organization framework in (Miller, 2019), partitioning kinds of explanations into analogous categories to the rungs in Pearl's causal hierarchy. The authors go on to present and justify the use of conditional differential entropy as the basis for constructing such a saliency measure. Finally, the authors show the application of the proposed approach on a synthetic conversation dynamics dataset.
strength_and_weaknesses: Strengths:

Overall the paper is extremely well written, and a real pleasure to read. Very nicely done! In particular this submission does a very nice job of arguing why one should use the approach presented, and not just how it works. I suppose we should expect nothing less from a paper about causal explanations.

Weaknesses:

My primary concern with this paper is in the empirical evaluation. While I understand that the problem statement itself is somewhat new, it is still very important to compare the proposed method against (even potentially very weak) baselines. Otherwise, it is impossible to determine whether the proposed method actually delivers on its promise. Additionally, while the figures presented are helpful for their explanatory power, it is very important to somehow quantify performance in terms of some metric. Unfortunately I don't have any concrete suggestions on such a metric.

While it's not necessarily a weakness per se, I am somewhat concerned about the use of the term "counterfactual" here when the authors do not hold exogenous noise constant between factual and counterfactual worlds, as described in (Pearl, 2009) Chapter 7. It would be very helpful for the authors to clarify exactly how the abduction-action-prediction process ala Pearl is reflected in this work. Related, it would helpful to discuss some of the common issues in counterfactual reasoning here, e.g. necessary assumptions for identifiability.

clarity,_quality,_novelty_and_reproducibility: As mentioned above, the submission is very clearly written, and is of very high quality over all. The work appears novel, although saliency methods are a bit outside my core area of expertise so it is difficult for me to assess.
summary_of_the_review: Overall the paper is well written, and presents a well motivated and plausible approach to determining temporal saliency. I have some concerns that currently make it challenging to recommend acceptance, but I am also happy to revise my opinion during the discussion.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to compute temporal saliency in probabilistic forecasting. The saliency of each timestamp in time-series data is measured based on its gradient w.r.t. the output entropy. Authors construct a synthesized dataset to verify the effectiveness of the proposed method.
strength_and_weaknesses: 
[Strength]
1. This paper focused on an important topic, i.e., explaining the model’s forecast on time-series data.

[Weakness]
1. The introduction of this paper is a little dense, and several concepts are not formulated mathematically, including “associative”, “interventionist”, “counterfactual”, “speculation”, “surprisal”, etc. In the field of explainable AI, researchers are expected to define the mentioned concepts in rigorous mathematical language. Maybe the above concepts are meaningful, but none of them are analyzed mathematically. Therefore, the discussion on these concepts will hardly make sense to readers. Please provide formal mathematical definitions for these concepts, and then discuss their relationship to the proposed method.

2. There lacks evidence to support the claim that “the proposed method is grounded in preattentive human cognition.” First, as mentioned above, please mathematically define components in preattentive human cognition. Second, please provide a theoretical proof to verify that the proposed method indeed aligns with the theory of neuroscience.

3. The actual setting of $\phi$ in Section 3.2 is inconsistent with the introduced physical meaning of $\phi$ in Section 2. In Section 2, $\phi$ is defined as a mapping from image locations to image features. In this case, modeling the distribution of image features as $p_\Phi$ makes sense. It is also meaningful to measure the amount of information using $-\log p_\Phi$. However, in Section 3.2, $\phi$ is defined as a mapping to the entropy of the output. Then, what is the purpose of modeling “the distribution of entropy as $p_\Phi$?” The entropy has its own physical meaning for measuring uncertainty, but modeling the distribution of entropy is meaningless. I suggest authors provide more clarifications on this issue.

4. Authors are encouraged to discuss more about the link between counterfactual reasoning and their method of saliency.
(1) First, I suggest authors mathematically define the “counterfactual explanation” for forecasting settings, instead of only giving a description as “simulating alternative causes to see whether the event still happens” (in Table 1 of Section 1.1). In this way, the connection between the proposed method and counterfactual reasoning can be explained in a more rigorous way.
(2) Second, I do not find experimental results on such counterfactual explanations in Section 4, though authors mention in Section 1.3 that they “examine changes in the future density predicted by M in response to different realizations of real-world sequences $t_{obs}$.” Please conduct some experiments to verify that the proposed method is able to provide counterfactual explanations. 

5. Experiments in this paper are insufficient.
(1) No baseline method is compared. Authors are suggested to compare the proposed method with existing widely used attribution methods, including Saliency [cite 1], Occlusion [cite 2], Input x Gradient [cite 3], LIME [cite 4], LRP [cite 5], Shapley value [cite 6], DeepSHAP [cite 7], etc.
(2) Authors also do not verify the effectiveness of the proposed method on benchmark datasets for the time-series forecasting task.

6. About experimental details.
(1) What is the specific form of the future and observed features, $i.e.$, $X$ and $Y$, corresponding to Algorithm 1?
(2) Which one of the three approaches introduced in Section 3.3 is chosen for the experiment to compute the differential entropy?
(3) Moreover, how well is the estimation of the distribution Y|X based on the Gaussian distribution and autoregressive decoders? Please provide a metric to quantify the estimation error, and verify the accuracy of such estimation.
(4) The saliency map in Figure 3 is problematic. Figure 3 shows that the saliency of frames where the nod is not in motion has a zero saliency value. However, the information of the position of the current speaker also affects possible futures. Thus, the saliency of frames where the nod is not in motion may be low, but not zero.
(5) Minor: “Saliency” in the last third line of page 8 is supposed to be “entropy”. According to Figure 3, when the nod begins, the entropy drops, and the saliency increases.

7. About the limitation of the proposed method.
(1) For the time-series forecasting task, the proposed attribution method simply computes the square of the gradient’s L2-norm. Is it equivalent or similar to existing gradient-based explanation methods, such as Saliency [cite 2]?
(2) In Section 2, $\phi$ is restricted to a differentiable function. However, the mapping from image position to feature vector is usually not differentiable, because the image position is a discrete variable. Please provide some clarification for this issue.

[cite 1] Simonyan, Karen, Andrea Vedaldi, and Andrew Zisserman. "Deep inside convolutional networks: Visualising image classification models and saliency maps." arXiv preprint arXiv:1312.6034 (2013).

[cite 2] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European conference on computer vision. Springer, Cham, 2014.

[cite 3] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. In arXiv: 1605.01713, 2016.

[cite 4] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “Why should I trust you?” explaining the predictions of any classifier. In KDD, 2016.

[cite 5] Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, Klaus-Robert Muller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks (ICANN), 2016.

[cite 6] Lloyd S Shapley. A value for n-person games. In Contributions to the Theory of Games, 2(28): 307–317, 1953.

[cite 7] Lundberg, Scott M., and Su-In Lee. "A unified approach to interpreting model predictions." Proceedings of the 31st international conference on neural information processing systems. 2017.
clarity,_quality,_novelty_and_reproducibility: This paper proposes a method to compute temporal saliency in probabilistic forecasting. However, several concepts in this paper are not defined and verified mathematically, which hurt the clarity of this paper. What is more, experiments in this paper are also insufficient.
summary_of_the_review: This paper focuses on an important topic. However, several concepts are not formulated in a rigorous way, and some claims are not proved or verified. There are also problems concerning the formulation of the proposed method. Moreover, experiments in this paper are insufficient. Therefore, I recommend rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a theoretical framework to explain probabilistic forecasting using a saliency map. They start from Miller's framework to link counterfactual reasoning and saliency-based XAI techniques. Then, they extend the generic notion of saliency to the probabilistic forecasting setting by inferring a closed-form solution for commonly used density functions. The proposed framework is qualitatively evaluated on one synthetic dataset.
strength_and_weaknesses: Strengths:
- XAI for time series forecasting is an extremely interesting and underexplored topic
- The authors try to build a formal theory behind saliency maps for time series forecasting


Weaknesses:
- The work is extremely abstract and theoretical and sometimes hard to follow
- The proposed framework is tested qualitatively on only one dataset 
- Conclusions are missing
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written and discussed; however, some practical examples would significantly increase the readability and ease of understanding
Quality: the paper seems to be of good quality; however, my expertise in this theoretical field is limited
Novelty: good, the approach is novel.
Reproducibility: poor; code implementation is not referenced nor discussed, and the synthetic dataset is not publicly available
summary_of_the_review: The work is extremely abstract and theoretical and sometimes hard to follow. Some practical examples would greatly increase the readability and ease of understanding. The authors should better explain possible applications of the proposed approach and recap their contribution and future work in a Conclusion section. 
The proposed framework is tested qualitatively on only one dataset; therefore, it is quite hard to assess its performance and usefulness in real-world applications. The authors should at least provide some quantitative metrics on their synthetic dataset, comparing their framework with other approaches or, if not possible, at least with random or naive approaches.

Other comments:
- page 3: "Our framework addresses all the aforementioned concerns we identified with saliency-based approaches in forecasting: (i) the counterfactuals or alternate 'causes' in our framework pertain to real preceding windows of actual observed features, and are therefore consistent with the underlying data generation process and semantics preserving". To me, this statement is not obvious. Sampling perturbations from past observations does not ensure semantic consistency with future observations. For example, a past observation could only be consistent in certain time series timesteps around specific patterns.
- pag 5, 3.2: "salent" instead of "salient".

In summary, the work is fascinating, but the evaluation part is extremely lacking, and its applicability in real-case scenarios is not adequately discussed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an information-theoretic saliency-based framework for counterfactual reasoning in probabilistic forecasting, and constructs and validates the approach on a small synthetic dataset. Reviewers agreed it’s an interesting, relevant topic, and found it overall well-written and nicely presented.

However, all reviewers expressed the same major concerns regarding the lack of rigorous empirical evaluations and comparisons to other methods. There was also no demonstration of how this technique can be applied to real-world datasets. Authors emphatically maintained throughout the rebuttal period that they are unable to make any sort of comparisons with other methods, and that it’d be impossible to apply to real-world datasets. 

I’m not convinced by their arguments that you can’t compare to other XAI methods. They claim it’s because it’d be an unfair comparison as 

>any comparable method would need to compute saliency by changing observed timesteps one-at-a-time and evaluate change in the information in the probablistic forecast. 

They then argue that there are many differences between these approaches and theirs. However, it seems like comparison is not impossible, it’s simply imperfect (“It may seem like we can simply keep the model fixed and visually evaluate the saliency maps. Here any interpretations would suffer from well-established issues of biases in the observer's assessment.”) This is not a reason to not do the comparison at all. I’d suggest first performing the comparisons, and then clarifying why these are imperfect comparisons.

I’m also not convinced by their reasoning that they can’t apply this to real-world datasets. They state that  

>given the forecasting model's predictions, our framework retrieves the salient timesteps when, crucially,
there are human-verifiable ground-truth salient timesteps that predict a future outcome, and
the underlying forecasting model is not flawed (defending against garbage in - garbage out, see introduction to Sec. 4).

At the same time, they also claim that there are obstacles to applying to real-world datasets because:

>Ground-truthing saliency towards future outcomes in human behavior datasets is especially hard because there often exists both supporting and opposing evidence in social psychology for predictive relationships between behaviors [R1, p.5; R2, p.22]. So we have a chicken-egg problem surrounding the evaluation on real-world data: if we had a way of identifying such ground-truth salient predictive relationships, we wouldn’t need this paper.

However, if this is the case, then doesn’t that mean that their method, which relies on human-verifiable ground-truth saliency, simply can’t be used in practical situations? Doesn’t this then just become a theoretical exercise?

>Even if we somehow obtain such ground-truth (which is not straightforward), we would then need to train a forecasting model on the real-world data. As discussed above, this would fall into the same issues where validation is based on an imperfect underlying forecasting model. Besides, on real-world data it is impossible to assume a perfect model to isolate any potential issues to our framework. This requires computing the true distribution over futures for any sequence in the data which is not possible for real-world data.

Again, just because the metric would be imperfect or would have inaccuracies doesn't mean that one shouldn't try, if only to at the very least demonstrate that the approach is applicable in principle.

I’m sympathetic to the authors’ frustration, but acceptance at ICLR requires at least answering the questions “what do I expect my readers to get out of this?” “How can others apply this to their own work / datasets?” “Why should people apply this approach as opposed to what they’re currently doing to solve this problem?” Once authors reframe the paper to more directly answer these questions, I think they’ll find more receptiveness from reviewers at these types of conferences.

As all reviewers have the same major concerns, I am unable to recommend acceptance.

justification_for_why_not_higher_score: All reviewers expressed major concerns and recommended reject, and I am not inclined to override this consensus, based on my reasons stated above.
Decision: Reject