Why Did This Model Forecast This Future? Information-Theoretic Saliency for Counterfactual Explanations of Probabilistic Regression Models
Authors: Chirag Raman
Alec Nonnemaker
Amelia Villegas-Morcillo
Hayley Hung
Marco Loog
Venue: NeurIPS 2023 poster (2023)
Keywords: Probabilistic Forecasting
Saliency
Explainability
XAI
Probabilistic Regression
TL;DR: We propose an information-theoretic saliency-based framework for counterfactual reasoning in probabilistic forecasting. For common distributions, we obtain a closed-form expression for the saliency of observed timesteps towards a model's forecasts.

Abstract: We propose a post hoc saliency-based explanation framework for counterfactual reasoning in probabilistic multivariate time-series forecasting (regression) settings. Building upon Miller's framework of explanations derived from research in multiple social science disciplines, we establish a conceptual link between counterfactual reasoning and saliency-based explanation techniques. To address the lack of a principled notion of saliency, we leverage a unifying definition of information-theoretic saliency grounded in preattentive human visual cognition and extend it to forecasting settings. Specifically, we obtain a closed-form expression for commonly used density functions to identify which observed timesteps appear salient to an underlying model in making its probabilistic forecasts. We empirically validate our framework in a principled manner using synthetic data to establish ground-truth saliency that is unavailable for real-world data. Finally, using real-world data and forecasting models, we demonstrate how our framework can assist domain experts in forming new data-driven hypotheses about the causal relationships between features in the wild.

Why Did This Model Forecast This Future? Iiformation-Thhorett Saliency for Counterfactual Explanations of Probabilistic Regression Models
Chirag Raman
Alec Nonnemaker
Amelia Villegas-Morcillo Hayley Hung
Marco Loog
Delft University of Technology, Delft, The Netherlands
{c. raman, a. a...v.lllaaaooorriiiiiiiooo willeggamorcilloo h. h.hung, m. m.loog)@
a m. www.ommmaokrrrsuudett tudelft.nl
Abstract
We propose a post hoc saliency-based explanation framework for counterfactual reasoning in probabilistic multivariate time-series forecasting (regression) settings. Building upon Miller's framework of explanations derived from research in multiple social science disciplines, establish a conceptual link between counterfactual reasoning and saliency-based explanation techniques. To address the lack of a principled notion of saliency, we leverage unifying definition of information- theoretic saliency grounded in preattentive human visual cognition and extend to forecasting settings. Specifically, we obtain closed-form expression for commonly used density functions to identify which observed timesteps appear salient to an underlying model in making its probabilistic forecasts. We empirically validate our framework in a principled manner using synthetic data to establish ground-truth saliency that is unavailable for real-world data. Finally, using real-world data and forecasting models, we demonstrate how our framework can assist domain experts in forming new data-driven hypotheses about the causal relationships between features in the wild.
Introduction
As we go about our daily lives, engaging in conversations, walking down the street, or driving car, we rely our ability to anticipate the future actions and states of those around us 2). However, the numerous unknowns, such as hidden thoughts and intentions, make our predictions of the future inherently uncertain [2] To reflect this uncertainty, several machine learning methods in such settings forecast a full distribution over plausible futures, rather than making a single point prediction ( 4. Identifying the factors that influence such a model's 's forecasts is particularly useful for domain experts seeking to understand the causal relationships guiding complex real-world behaviors, especially in situations where the future is uncertain. In this work, we introduce and address novel research question toward counterfactual reasoning in multivariate probabilistic regression settings: how can identify the observed timesteps that salient for model's probabilistic forecasts over specific future window? Specifically, we introduce the first post hoc, model-agnostic, saliency-based explanation framework for probabilistic time-series forecasting.
We begin with a fundamental observation about human social cognition: we are averse to uncertainty and strive to minimize it (2). Consider the scenario where pedestrian is approaching you on the street. Initially, there is uncertainty about which direction each of you will take avoid a collision. As one of you changes direction, the other observes and takes the opposite direction, ultimately avoiding a collision. Concretely, the thesis of this work formalize the following notion of saliency: the timestep that changes the uncertainty of predicted future is salient toward predicting that future.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

For instance, in the aforementioned scenario, we posit that the moment when one pedestrian changes direction is salient toward forecasting the future trajectories of the pedestrians.
Our notion of saliency is grounded in preattentive human cognition and related to the concept of surprisal or information associated with observations [3] []. Preattentive saliency captures what the brain subconsciously finds informative before conscious, or attentive, processing occurs. An unexpected surprising observation is considered salient in this context. However, when applied to forecasting, the idea of surprisal or informativeness must be linked to the fitture outcome. Conse- quently, we propose that a timestep that alters an observer's s certainty about the future is surprising, and therefore, salient. Crucially, our unifying "bottom-up' perspective treats a forecasting model like a human observer, providing a principled definition of saliency that not arbitrarily tied to task-specific error metrics. In contrast, the 'top-down' or task-specific notions of saliency common in post hoc explainable artificial intelligence (XAI) literature suffer from several drawbacks. Computed saliency maps may not measure the intended saliency, and even be independent of both the model and data generating process 779. Moreover, what constitutes good explanation is subject to the biases, intuition, the visual assessment of the human observer I0] phenomenon we refer to as the interpretation being in the eye of the beholder. Finally, as Barredo Arrieta et al. [ Sec. 5.3] note, "there is absolutely no consistency behind what is known saliency maps, salient masks, heatmaps, neuron activations, attribution, and other approaches alike."
To the best of our knowledge, no existing work addresses the specific task of obtaining post hoc model-agnostic explanations for probabilistic forecasts. Existing XAI methods for time-series data have predominantly focused on sequence classification, as we discuss in Section and Appendix A For regression, instead of post hoc explainability, researchers have emphasized interpretability by design [1] or intrinsic interpretability [12). where interpretability stems from the simple structure of models or coefficients of predefined basis functions [13 [4]. Against this backdrop, we present the following key contributions:
Conceptual Grounding: We establish the conceptual foundation for linking saliency- based explana- tions with counterfactual reasoning. We draw upon insights from Miller's [10) work on explanations
in artificial intelligence, highlighting the contrastive nature of explanations (Section 3
Information- Theooreii Framework: We extend Loog's [5] framework of bottom-up preattentive saliency to the domain of probabilistic forecasting. Specifically, we introduce a novel expression of saliency based on the differential entropy of the predicted future distribution, providing a closed-form solution for commonly used density functions in the literature (Section (
Empirical Validation: We empirically evaluate framework using synthetic and real-world data. In the synthetic setting, we achieve full accuracy in retrieving salient timesteps with known ground truth saliency. In real-world scenarios without ground truth saliency, we demonstrate the utility of our framework in explaining forecasts of social nonverbal behavior and vehicle trajectories, showcasing its effectiveness in complex and dynamic contexts (Section 5).
2 Related Work
XAI Techniques for Time-Series Data. The taxonomy commonly used for explainability methods categorizes techniques based on three criteria: (i) intrinsic or post hoc, (ii) model-specific or model- agnostic, and (iii) local or global [1]. In the context of time-series regression, existing techniques predominantly focus on non-probabilistic settings and fall into the category of intrinsic and model- specific approaches. These include: (i) incorporating inductive biases through internal basis functions [14] (also extended to the probabilistic setting [33,, (ii) utilizing self-attention mechanisms in the model [1],, and (iii) adapting saliency maps from computer vision to measure the contribution of features to the final forecast [16] 17 For a comprehensive review of XAI methods across domains and time-series tasks, please refer to Appendix
Saliency-Based Explanations and Drawbacks. Saliency maps gained popularity as post hoc explanation tools for image classification [16 [8]. However, the lack of consistency in defining saliency has led to diverse interpretations, including occlusion sensitivity, gradient-based attribution heatmaps, and neuron activations [1] [2]. Nevertheless, these maps are typically computed by perturbing different parts of the input and observing the resulting change in the prediction error or output class. Several issues arise with the current use of saliency maps explanations: (i) the

---

feature-level manipulations used for saliency maps may distort the sample in ways that deviate from the real-world data manifold and destroy semanties 77: ; (ii) given the arbitrary definitions. evaluating saliency maps becomes challenging and is subject to observer biases [12] Sec.10.3.3 which can lead to maps appearing correct even when they are insensitive to the model and data [7]: (iii) for forecasting, Pan et al.'s 's [17] notion of saliency based on the error between the point prediction and ground truth future is arbitrary and relies on ground truths unavialable during testing; and (iv) the saliency map is explicitly retrained for a single observed-futue sequence, failing to capture salient patterns across similar observed sequences that result in divergent but plausible futures [7].
Model-Agnostie Techniques. The SHAP framework, which integrates ideas from Shapley Values, LIME, LRP, and DeepLIFT. has gained popularity as a model-agnostic approach [19]. However, adapting these techniques to time- series tasks poses several challenges. Firstly, the Shapley methods rely on functions with real-valued codomains, such as a regression function fz [9 see Eq. 4, 8], while our focus is on probabilistic models that output the distribution py instead of some y y= ft) to handle future uncertainty. Adapting these methods to deal with full predicted distributions is nontrivial. Similarly, gradient-based approaches compute gradients with respect to single output instead of a full distribution. Secondly, these methods provide feature importance measures for a single output, whereas in time-series analysis, an observed timestep for an entire future sequence. That is, the joint consideration of the entire future sequence when computing input importance measures challenging. As Pan al. [17] note, in evaluating single-time predictions, these methods "ignore crucial temporal information and are insufficient for forecasting interpretation". Finally, similar to perturbation-basee saliency methods, the sampling of features from arbitrary background samples in methods like Shapley/SHAP can lead to Frankenstein Monster instances [2] Sec. 9.5.3.3] that may not be valid samples on the data manifold. This undermines the semantics of the data, particularly in scenarios like motion trajectories, where randomly replacing features can result in physically impossible or glitchy motions.
interested in identifying the importance of
3 Conceptual Grounding: Linking Saliency-Based Explanations to Counterfactual Reasoning
Given the challenges in XAI where speculations are often presented in the guise of explanations 20). argue for grounding the concept of explanation within established frameworks of how humans define, generate, and present explanations. Turning to research in philosophy, psychology, and cogni- tive science, Miller [10] emphasized the importance of causality in explanatory questions. Drawing upon Pearl and Mackenzie's s Ladder of Causation (1 he proposed the following categorization:
Associative (What?): Reason about which unobserved events could have occurred given the observed events. Interventionist (Howw?): Simulate change in the situation to see if the event still happens. Counterfactual (Why?): Simulate alternative causes to see whether the event still happens.
To apply Miller's framework in the context of forecasting, one needs to define the abstract notions of "events" and "causes'. Consider a model M that predicts features over future window fut by observing features over a window tubs- We assert that the intrinsic interpretability methods involving inductive biases [13 [4] and attention mechanisms [15], fall under associative reasoning. These methods assess the (unobserved) importance of features over lohs using model parameters or attention coefficients based on a single prediction from M (the 'event') for fixed lint and single saliency methods can support counterfactual
Lobs. In contrast, we posit that the perturbation- reasoning. They perturb different parts of the imput over tobs simulating alternative 'causes' from M's perspective, and observe the effect on an error metric (the 'event"). However, the current application of these methods encounters issues outlined in Section [
To address the aforementioned challenges, we employ a unifying information-theorett concept of bottom-up saliency grounded in preattentive human cognition [5] 6 as discussed in Section ] Concretely, we propose the following implication that links this saliency to counterfactual reasoning:
observing the features timestep /. tobs results in change in M's information about the future Tua over the given biut. + 1 is salient.
(l)
3

---

Note that the antecedent (on the left of the im-
EM
plication) is counterfactual statement. We
formally express the implication using causal
Data
+ M
M M
graphs 22 in Figure The generic graph ex-
presses relationships between the random vari- ables prior to training the forecasting model M. The exogenous variable eM captures the ran- domness in the training process and modeling choices, including the distribution family for representing the forecasts. The exogenous vari- able eœ captures the randomness in the human observer's choice of observed and future win- dows examine the model. Our central idea is to evaluate the information in the model's predicted distribution denoted by futt Specifically, we propose posing the following counterfactual question: What information would M have about the future over a fixed tiu if observed the features over lobs? ? The modified graph for evaluating this question is in Figure 1b Once the model M has been trained and the windows lobs and liut have been chosen, the effect of the exogenous variables on the variable Lutt disappears. This allows us to evaluate the change in the information about the future in response to different realizations of tobe and tjut, facilitating counterfactual analysis. Note that we assume the modified graph is already available, as our focus on the explanation phase. While the procedure starting from training the model in the generic graph implicitly follows Pearl's abduct-action-ppe pertiionpreeiction process [2 p. 207], estimating the distribution over the exogenous variables from the abduction step is conceptually not applicable in this setting.
Tobs
Tobs
tobs
fut
Lfut =
EH
Tiut
Tiut
+uut
(a) Generic Graph
(b) Modified Graph
Figure 1: Causal Graphs for Explaining Forecasts
Note that these graphs are not meant describe relationships between random variables in the data for specific hypothesis, as is typical in causal inference literature: for instance, the effect of [rotating toward the speaker] on [turn changes] in conversations. Rather, they describe the process of a human generating contrastive explanations for a given pretrained forecasting model M- in--rrespective of whether or not it is the true model-foo some sequences in the data (tube, tiut). Further, the notion of counterfactuals, as used within the context of contrastive explanations, is also distinct from that in causal inference. As Millerr [T] Sec. 2.3] points out, "it is important to note that this is not the same counterfactual that one refers to when determining causality. For causality, the counterfactuals are hypothetical 'noncauses'. ... whereas in contrastive explanation, counterfactuals are hypothetical outcomes." Miller's ' point is that why explanations entail contrastive reasoning which involves comparing 'outcomes' in response to alternate 'causes'. In our work, this 'outcome" relates to the information in M's predicted distribution, the what-if question being "would the information in M's prediction for the window Litt change if it had observed features over a different (contrastive) Lohs ?". Contrast this to associative reasoning which uses features from single tobe to generate the attribution map. A longer discussion is in Appendix E
Considering the information in forecasts in implication I Tlinks counterfactual reasoning to a more principled notion of saliency than has been used in XAI literature. Note that the implication entails that for the antecedent to be true t must be salient. However, knowing the antecedent is false is not sufficient to conclude that is not salient, i.e. there can be other notions of saliency that make salient. However, for less speculative evaluation, it is crucial that we use a unifying notion of saliency that is not arbitrarily defined based on task-specific error metrics or model gradients []] [6 Preattentive saliency, as we formalize in Section 4.1 is based on what is informative for the brain before conscious processing, making it more objective in nature.
Our framework addresses all the concerns associated with saliency- based approaches described in Section 2 (i) the counterfactuals in our framework are real observed features rather than random input perturbations, preserving the semantics of the real-world data; (ii) our use of information- theoretic preattentive saliency is principled and objective; (iii) our framework allows for saliency computation on unseen test data where the ground-truth future is unavailable, relying solely on the underlying model; and (iv) our approach considers the distribution over possible futures for a single input, capturing the structural predictive relationships between features across multiple samples. An additional advantage of our framework that it does not require any training to compute the saliency and can be applied any model that outputs distribution over futures.

---

Methodology: Closed-Form Saliency for Probabilistic Forecasting
4.1 Preliminary: Information Theoretic Preattentive Saliency
Loog [5] developed a general closed-form expression for saliency based on computational visual perception that unifies different definitions of saliency encountered in the literature. The framework was illustrated on images and employed surprisal-base In this framework, an image is represented by a feature mapping function e that relates each location in the image to a set of features. The saliency of a location is determined by the information or surprise associated with its corresponding feature vector ((x) compared to other feature vectors extracted from the same image. The saliency measure is defined as follows:
operational definition of bottom-up attention.
S(x) > S(z') + - 1o (0())) > -log loggo (ó(x'))
(2)
Here, Po represents the probability density function over all feature vectors, while px captures any prior knowledge that influences the saliency of different image locations.
Contrary to approaches that determine saliency maps through explicit data-driven density estima- tion [16 18 23/25], once the feature mapping e is fixed, glosed-form expression for saliency can be obtained. The information content log pa can be obtained from log px through simple change of variables [26] from x to 6(z). The saliency ((x) is then given by the expression:
log pa (o(x)) log (z) + log det (J)(J(())
2
where Jæ denotes the Jacobian matrix of o, and indicates matrix transposition. Since monotonic transformation does not essentially alter the map, Loog (5) simplifies the saliency map definition to
S(x) =
This formulation of saliency offers several advantages. It provides a principled and objective measure that captures the informativeness of features for human perception. Moreover, the saliency computation is purely local to an image, making it independent of previously observed data.
4.2 Defining & in Terms of the Uncertainty over the Future Window tut
Let tobs := [ol, o2, .... of'' represent window of consecutively increasing observed timesteps, and fT] denote an unobserved future time window, where f1 > oT. Consider a set of interacting agents, and let X [b]: t E tobsl/=1 and Y = [b] E t | represent their features over tobs and tut respectively. Here, by captures multimodal features from agent i at time t. The forecasting task is to predict the density pyJx. Given model that outputs pylx. x. our task is to compute the saliency S(tobs) of an observed tobs with respect fixed choice of tiut.
tat =
To extend Loog's (5] framework forecasting settings, we need to choose an appropriate d. We formalize the implication in Equation I and map tobs to the differential entropy of the model's predicted future distribution over tjut- Specifically, we define Φ tobs + h(Y X), where the conditional differential entropy of Y given (X = X} is defined as
h(Y|X= = X) X)= = f (O
(5)
Our framework is summarized in Algorithm I Consider that domain expert selects specific tra corresponding high-order semantic behavior they wish to analyze. This could be speaking-tury change [27 28) an interaction termination (29 30), or synchronous behavior event [3I]. Given an underlying forecasting model M and look-back period before fuut. we compute h X) for different observed multivariate features X corresponding to different locations of a sliding tobs- The computed differential entropy values are then inserted into Equutiin 4 to obtain the saliency of different tobs locations towards the future over the chosen tru In Appendix B we discuss other favorable properties of differential entropy that make a suitable choice as Φ.
Explanation Using the Running Example. Within our running example from Section I fru corresponds to the two pedestrians passing each other while avoiding collision. In this example, let us assume M 's training data contains examples of pedestrians passing others to both the left and the
5

---

Algorithm Temporal Saliency in Probabilistic Forecasting
Input:
The probability density function Py fixed tlue of interest, a sequence of m preceding observed windows [loba..... di and the behavioral features X for every t.
Output: The saliency map S(O) over the observed windows
I: for each E/ E do
2: Compute the feature mapping ((t))))) h(Y X()
3: end for
4: Compute saliency S(tora) + det(J]{(tta)ds(tuan)))
right. Consequently, for a lots containing the pedestrians approaching each other in a straight line, the predicted distribution PYxx over fuue encapsulates both possibilities of each pedestrian passing to the left as well as the right of the other. So the entropy h X) is high for this tobs. Only once M is fed as input with the trajectories from the fobs containing the pedestrians choosing one of the two directions to pass, the predicted pyjx is certain in terms of the pedestrians continuing along the chosen direction. (Note that in this case, we assume M has been trained by maximizing likelihood over the dataset containing only these two direction changes for avoiding collision.) Consequently, the entropy h(Y X) drops only once this moment of the pedestrians committing to direction seen by the model and would be considered salient for our algorithm.
4.3 Computing h(|]X X)
Typically, the density Pvjx is modeled as a multivariate Gaussian distribution 4 34]. When the decoding of the future is non-autoregressive, the parameters of the distributions for all E lut estimated at once, and the differential entropy has a closed-form expression, given by (see Cover and Theorem 8.4.1))
Thomas 35
h( = X) h(Na(,, K)) = (2rre) det( (et (K))-
(6)
A common choice is to set K to be diagonal, i.e. the predicted distribution is factorized over agents and features. In this case, we can simply sum the log of the individual variances to obtain the feature mapping Φb. Note that from Equation 6) for multivariate Gaussian distribution, the differential entropy only depends on the covariance, or the spread the distribution, aligning with the notion of differential entropy expressions for large number of commonly employed probability density functions.)
measure of total uncertainty. (See (35] Tab. 17.1: 33) for closed-fom
In cases where probabilistic autoregressive de- we do not have access to the full joint distribution "y...l.,yrr X for the timesteps trut. This is because infer- ring the density function Py often involves sampling: a specific sample Y, is taken from the predicted density at each t E tiut, and passed back as input to the decoder for estimating the
coders are used 33 37 38
Pram (11))
density at timestep t + 1 [37] [8). Therefore, the density at L + depends on the randomness introduced in sampling Y, Figure 2 illustrate the concept for two timesteps. Here, single forecast would only output the shaded red distri- bution for 12. In such cases, computing the joint entropy 1(YY,, Y2) directly is challenging in the absence of the full joint distribution PY,,Y;-
Pv
Y2
Y1
To address this, we have two options. The sim- pler option is redefine our feature-mapping as Φ Lohs + Eretpm M(Y,)y<<.. X), i.e. we approximate the total uncertainty over the pre- dicted sequence by summing the differential en- tropies of the individual densities estimated at
Pi
Figure Illustrating predicted densities under greedy autoregressive decoding for two timesteps. For simplicity, we depict a joint Gaussian distribu- tion and omit the conditioning on X everywhere.

---

each timestep. Note that following the chain rule for differential entropy (see Cover and Thomas B5 Eq. 8.62J), the joint entropy can indeed be written as the sum of individual conditionals. However,
k = x) = Σ MV x) Σ MYY. X).
(7)
tEfrue
rEfT
And yet, training autoregressive decoders by maximizing likelihood actually assumes the inequality in Equutionn to be approximately equal (see [39] Sec. 2: 4: Eq. 5]). The approximation relies on the observation that, for autoregressive decoding, the parameters of the predicted distribution for Y, computed deterministic function of the decoder hidden state. That is, Y, is conditionally independent of Yz: given the hidden state of the decoder 8, at timestep t. The underlying assumption is that for well-trained decoder, 8, encodes all relevant information from other timesteps to infer the distribution of Y.- So at inference, despite being function of the single sample Y-1,. the predicted distribution conditioned on 8; provides reasonable estimate of the uncertainty in Y. This assumption allows us again obtain a closed-form expression for the saliency map when each Y; is modeled using density function with known closed-form expression for differential entropy
335 Tab. 17.1; 36). For the common choice of modeling Y2 using a Gaussian mixture [37
approximations that approach the true differential entropy can also be obtained efficiently 41
to
directly compute the feature mapping ø.
The second option is to estimate h(Y = X) using sampling or other non-parametric approaches when analytical expressions or computationally efficient approximations are not available [44 These sampling-ban -aaeeiggg-eeddd--- methods provide approximations that converge to the true entropy, although they may be computationally more expensive than parametric methods. Overall, the choice of modeling the future density and the approach for estimating the differential entropy depends on the specific scenario and the available resources.
5 Experiments
The common evaluation of saliency-based explanations relies on qualitative visual assessment, which is subjective and prone to observer biases (7) [T 12]. Meanwhile, establishing reliable ground truth for the salient relationship between the observed window tobs and the future window true is challenging in real- wol-w data due to conflicting domain evidence predictive relationships [28] 48). Furthermore, fair validation of a model agnostic, post hoc method requires evaluating it independently of imperfections in the underlying forecasting model. To address these challenges we conduct two types of empirical evaluation: one using synthetic data to establish ground truth predictive saliency and validate the framework, and another demonstrate empirical utility in real-world scenarios where perfect forecasts and ground truth saliency are unavailable.
No existing benchmarks or post hoc explanation frameworks exist for probabilistic time-series regression that meet the necessary requirements for a meaningful empirical comparison. Nevertheless, provide results by adapting several explainability frameworks in our experiments. Specifically, we considered DeepSHAP and GradienSSHAP [19], and IntegratedGradients and SmoothGrad 49]. It is important to note that we do not imply that these are fair comparisons; they are not (see Section 2). However, the comparisons are meant to characterize results from popular tools that practitioners are likely to use in the absence of our proposed framework. Implementation details for the following experiments and additional results for the real-world scenarios are in Appendices C and D. respectively.
5.1 Empirical Validation using Synthesized Ground Truth Saliency
Dataset. We simulate a group conversation that emulates real behavior patterns. Listeners typically focus on the speaker, while the speaker looks at different listeners [50]. Additionally, head gestures and gaze patterns predict the next speaker (51)54). In our simplified simulation, the speaker rotates towards the center when speaking, and listeners nod to trigger turn handover. We use real-valued quaternions to represent 3D head poses, commonly used for human motion and pose representation 4,55 (6). Following the notation in Section 4.2 b (9w, Qr gy ss] where ss denotes binary speaking status. We simulate the turn changes to occur once clockwise and once anticlockwise. The ground truth salient timestep is when listener initiates head nod to trigger turn handover, ensuring a certain future turn change. [Figure 3 iillustrates this mechanism. The code, dataset, and animated visualization are available in the Supplement.
7

---

listener head-nod begins
observed
obs
obs
possible fuburess
tuu
fut
Figure 3: Illustrating the synthetic conversation dynamics dataset. Speakers are denoted in orange and listeners in green. For a fixed tuu we depict preceding tobs windows. By construction, when observing stable speaking turn over fohs> two valid futures are possible over fut. These correspond to turn handover the immediate left or right the current speaker. Over t to when a listener nods to indicate the desire take the floor, the future over Litt becomes certain, corresponding to the listener successfully taking over the speaking turn. Here 12 consequently more salient than Loss towards forecasting the turn change over tru (Best viewed as video, see Supplement.)
Qualeemion
Value
0.0
Qustertoon
Volue
0.3
0.60
Entrayy
Endcay
Attr (mean)
0.25
-22
-2.30-
Sallency
22-
Saliency
Attr. 1400
230
125
-45
345
155
130
115
140
345
1.0
130
1.35
144
145
350
Timestep
Timestep
Timestep
(a) Ours (Perfect forecast)
(b) Ours (Trained model)
(c) Attributions (Trained model)
Figure 4: Computing Saliency. The top plots show the quatemnion dimensions qx and qy for the listener that nods over t. in Figure 3. The gray dotted line indicates the true salient timestep 138 when the head nod begins, making the future show the (a) entropy over future values of all participants (middle), and saliency map obtained using our framework (bottom), considering perfect forecasts; (b) entropy and saliency for the forecasts from a Social Process model; and (c) mean attributions across features per timestep from different explainability frameworks (DeepSHAP, GradientSHAP, IntegratedGradient, and SmoothGrad) for the predicted mean and std. of the same forecast from the Social Process model.
timesteps 183 228 (ffut) certain. The rest of plots
Empirical Validation. To validate our framework in isolation, we assume perfect forecasting model that predicts the true distribution over the possible future quaternion trajectories. The forecast- ing model focuses solely low-level features and does not incorporate any high-order semantics of turn-taking. The saliency map generated by our framework, as shown Figure 4a accurately identifies the ground truth salient timesteps at frames 138 and 139 where the head nod begins. The saliency decreases once the nod is already motion, indicating that it does not provide additional information about the future. This empirically validates our framework.
We evaluate our framework using real underlying forecasting model trained on synthetic data. employ a Social Process model [4] for ability to capture relative partner behavior and adapt to specific group dynamics. As shown in Figure 4b our framework identifies the true salient timesteps with higher salieney values. Conversely, the attributions provided by other explainability frameworks in Figure 4c for the predicted mean and
Introducing a Real Forecasting Model.

---

Time Hor Lum for
Sallency Rototion (deprees)
Zvengle
-190
Saliency of observed in tex towards future over But
0.0001
sallbgr towordsruture
Sailenyy
0.0000
tax over fat
SHAP [mean)
0.0
0.25
SWAP (twwan)
SHAP valurs cheerved lowards faure
0.00
-0.25
S+ Grad ents-up
SHAP (st))
Doest-i0
Sh-AP (wid)
310
Timestep
Timestop
(a) Group leaving behavior (MatchNMingle dataset)
(b) Vehicle trajectory (nuScenes dataset)
Figure 5: (a) Analysis of the group leaving instance at 12:11 on Day Cam 12 in the MatchNMingle dataset. Row 1: Video frames and overlaid arrows denoting the head orientation of participants. Orange indicates the person leaving the group over the tiut- Row 2: Head orientation of the leaver plotted as 2D horizontal rotation. Row 3: Saliency map from running predictions from the Attentive Social Process model through framework. The timesteps salient towards the model's forecasts correspond to the leaver making sweeping gazes away from the group. Rows 4-5: mean DeePSHAP and GradientSHAP across features per timestep for the predicted mean and std. of the same forecast. (b) Analysis of the vehicle turn making instance on Scene 3 in the nuScenes dataset. Row I: Video frames showing the bus and surrounding cars from the camera. Row 2: Future predictions for the bus position (circled) from the Trajectron++ model (ground truth in white, predicted mean in black and variance in red). Row Saliency map from running predictions through our framework. The timesteps salient correspond to the model being more certain that the bus will make turn. Rows 4-5: mean DeePSHAP values across features per timestep for the predicted mean and std. of the same forecast. Best viewed as video (see Supplement).
standard deviation of the same forecast fail to capture the salient predictive relationships in the data. This comparison underscores the effectiveness of our framework in capturing meaningful and interpretable saliency, even in conjunction with imperfect forecasting model.
5.2
Empirical Evaluation in Real-World Scenarios
5.2.1 Group Leaving Behavior in the Wild
The study of group-leaving behavior has garnered interest in social psychology and the development of conversational agents [29] 30 Recent approaches employ data-driven models to predict future non-verbal cues, capturing general predictive patterns in the data [4]. In this study, demonstrate how our framework can assist domain experts in hypothesizing about the causal relationships between behavioral patterns and group leaving. We leverage the publicly available MatchNMingle dataset [57], which features natural interactions of 92 individuals during cocktail party. We use an Atlentive Social Process model to forecast continuous head pose, body pose, and binary speaking status.
Through our analysis (see Figure 5ap, we find that the salient timesteps in the model's forecasts correspond to instances when a person about to leave directs their gaze away from the shared group space (o-space [[] by rotating their head. This observation leads to the following hypothesis:
gazing away from the o-space of conversing group is predictive of group leaving.

---

While this hypothesis aligns with established leave-taking patterns [ 58 and the sweeping gaze behavior associated with seeking new interaction partners (59] requires further validation through subsequent studies and rigorous statistical testing with the involvement of domain experts. Nonethe- less, our experiment demonstrates how the framework can unveil data-driven insights into patterns that, in other cases, may have been overlooked by humans but captured by the forecasting model. By contrast, we do not observe any discernible intuitive patterns in the features associated with the trends in DeepSHAP and GradientSHAP values for the predicted mean and standard deviation.
5.2.2 Vehicle Trajectory Forecasting
The accurate forecasting of pedestrian and vehicle trajectories is crucial for safe and socially- aware autonomous navigation of vehicles 37
60-62). In this study, we utilize our framework to investigate vehicle dynamics in real driving scenarios. Specifically, we leverage the muScenes dataset, a multimodal dataset for autonomous driving 63 and the Trajectron++ forecasting model [37
Figure 5b) illustrates our analysis of vehicle dynamics at an intersection. Notably, our framework identifies a salient timestep for the Trajectron++ model precisely when it becomes more confident that the bus will make a turn instead of continuing straight. This coincides with the model's 's increased certainty that the point-of-view vehicle will decelerate as a new vehicle enters the scene from the left. Although there are no relevant domain-speen specific theories in this case to interpret this saliency, these identified patterns align with expected driving behavior. In contrast, the DeepSHAP values fail to capture the model's 's change in certainty about the bus making the turn instead of continuing straight. Moreover, we also do not identify any intuitive patterns in the predictions associated with the DeepSHAP trends. Thus, our framework serves as a valuable tool for sanity-checking model forecasts in real-world driving scenarios. It helps identify instances where the model's predictions align or misalign with established norms and expectations.
Conclusion
We have proposed a computational framework that provides counterfactual explanations of model forecasts based on principled notion of bottom-up task-agnostic saliency. We derive closed-fom expression to compute this saliency for commonly used probability density functions to represent 38 62] To validate our framework, we conduct empirical experiments using synthetic setup, enabling quantitative validation and mitigating observer biases associated with visual assessment of saliency maps. Additionally, we demonstrate the practical utility of our framework in two real-world scenarios involving the prediction of nonverbal social behavior and vehicle trajectories. By identifying salient timesteps towards predicted future through counterfactual reasoning, our framework can support domain experts in formulating data-driven hypotheses regarding the predictive causal patterns involving the features present at those salient timesteps. These hypotheses can then be tested through subsequent controlled experiments, establishing a human-in -uman-in-the-loop toop Explainable AI (XAI) methodology. For more comprehensive discussion, please refer to Appendix
forecasts 4 337
7 Limitations and Potential Negative Societal Impact
While our framework provides a closed-form or efficient solution for most probability density functions, limitations arise when an analytic expression for differential entropy is unavailable. As discussed in Scction 4.3 alternative approaches like sampling or nonparametric methods can be employed to approximate the entropy, albeit at an increased computational cost.
Our work here is an upstream methodological contribution. However, when applied downstream to human behavior or healthcare data, ethical considerations arise naturally. Here, care must be taken that such methods are not applied for gaining insights into behavior in a way that violates the privacy of people. Our framework enables domain experts to derive data-driven insights and hypotheses about predictive causal patterns. However, hypotheses should be rigorously tested, using controlled experiments and peer review, before being considered valid statements about human behavior. Collaboration among researchers, practitioners, and policymakers across disciplines is crucial to mitigate such societal risks and ensure ethical deployment of AI technologics.
10

---

Acknowledgments
The authors would like thank Jesse Krijthe, Rickard Karlsson, David Tax, Yeshwanth Napolean, and Megha Khosla for the thoughtful discussions.
References
Adam Kendon. Conducting Interaction: Patterns of Behavior in Focused Encounters. Number in Studies in Interactional Sociolinguistics. Cambridge University Press, Cambridge New York, 1990. ISBN
978-0-521-38036- 978-0-221-38938.
[2] Oriel FeldmanHall and Amitai Shenhav. Resolving uncertainty in a social world. Nature human behaviour, 3(5):426-335, 2019.
[3] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-STGENN: Social Opatio-Temport Graph Convolutional Neural Network for Human Trajectory Prediction, 20Xx:20002.11227 [cs), February 2020.
[4] Chirag Raman, Hayley Hung, and Marco Loog. Social Processes: Self-Supervised Meta-I Meta-I.erriig over Conversational Groups for Forecasting Nonverbal Social Cues. arXiv:2107.13576 [cx], July 2021.
[5] Marco Loog. Information theoretic preattentive saliency: A closed-form solution. In 2011 IEEE Inter- national Conference on Computer Vision Workshops (ICCV Workshops), pages 1424, Barcelona, Spain, November 2011. IEEE, ISBN 978-1-4677--006---993338 0700008---697----------000--0-90 978-99-799------3--00111---- doi: 10. 10.1109//CVW 1,1220011.611044 1.6130417.
[6] AHC vandertteijden. Perception for selection, selection for action, and action for perception, 1996.
[7] Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing syxtems, 31, 2018.
[8] Sebastian Lapuschkin. Stephan Wäldchen, Alexander Binder. Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking Clever Hans Predictors and Assessing What Machines Really Learn. Na- Communications, 10(1):1096, December 2019. ISSN 2041-1723. doi: 10. 40738/444477-011--99977744
[9] Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning, Fehruary 2020.
[10] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1-38, February 2019. ISSN 0004-3702. doi: (..10166j.rtiint2211. 2018..07.. .007.
[11] Alejandro Barredo Arricta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alherto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (XAT): Concepts, taxonomies, opportunitics and challenges toward responsible AL. Information Fusion, 58:82-115 2020. ISSN 1566-2535. doi: 2001//iinfuus.219.12...
[12] Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
[13] David Rügamer, Philipp F. M. Baumann, Thomas Kneib, and Torsten Hothom. Probabilistic Time Series Forecasts with Autoregressive Transformation Models. arXiv:2110.08248 [cx], February 2022.
[14] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv:1905.10433 10437 [cs, star), February 2020.
[15] Bryan Lim, Serean O. Arik, Nicolas Loeff, and Tomas Pfister. Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. arXiv:1912.09363 [cs, stat], September 2020.
[16] Piotr Dabkowski Yarin Gal. Real time image saliency black box classifiers. Advances in neural information processing systems, 30, 2017.
[17] Qingyi Pan, Wenbo Hu, and Ning Chen. Two Birds with One Stone: Series Saliency for Accurate and Interpretable Multivariate Time Series Forecasting. In Twenty-Ninth International Joint Conference on Artificial Intelligence, volume 3, pages 2884-2891, August 2021. doi: 10. (0.24963/jjcai.20221. 2021/397.
[18] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.603. 2013.
11

---

[19] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[20] Zachary C. Lipton and Jacob Steinhardt. Troubling Trends in Machine Learning Scholarship. July 2018.
[21] Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Basic books, 2018.
[22] Judea Pearl. Causality. Cambridge university press, 2009.
[23] Antonio Torralba. Modeling glohal scene factors in attention. JOSA A, 20(7): 1407-118, 2003.
[24] Neil DB Bruce. Features that draw visual attention: An information theoretic perspective. Neurocomputing. 135-33,, 2005.
[25] Lai Jiang, 7he Wang, Mai Xu, and Zalin Wang. Image salieney prediction transformed domain: deep complex neural network method. In Proceedings of the AAAI Conference on Artificial Intelligncce volume 33, pages 8521-8528, 2019.
[26] WM Boothby. An introduction to differential geometry and riemannian manifolds. 1975.
[27] Anne Keitel, Moritz M Daum, et al. The use of intonation for turn anticipation in observed conversations without visual signals as source of information. Frontiers in psychology. 6 6(108):265- 273, 2015.
[28] Stephen C. Levinson and Francisco Torreira. Timing in tur-takiin and its implications for processing models language. Frontiers Psychology, 6, June 2015. ISSN 1664-1078. doi: 10.3389/fpsy..200 2015. 00731.
[29] Dan Bohus and Fric Horvitz. Managing Human- Engagement with Forecasts and... um... Hesita- tions. Proceedings of the 16th International Conference on Multimodal Interaction, page 8, 2014.
[30] Felix van Doorn Rituals of Leaving: Predictive Modelling of Leaving Behaviour in Conversation. Master of Science Thesis, Delft University of Technology, 2018
[31] Sanjay Bilakhia, Stavros Petridis, and Maja Pantic. Audiovisual Detection of Behavioural Mimicry. In 2013 Humaine Asxociation Conference Affective Computing and Intelligent Interaction, pages 123-128. Geneva, Switzerland, September 2013. IEEE, ISBN 778-0-7695-5048- doi: 10.1109/AC11.203..27
[32] David Salinas, Michael dimensional multivariate forecasting with low-rank Gaussian Copula Processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Laurent Callot, Roberto Medico, and Jan Gasthaus. High-
[33] David Salinas, Valentin Flunkert, and Jan Gasthaus. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. arXiv:1704.041100 [cs, stat], February 2019.
[34] Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.. 2018
[35] Thomas M Cover and Joy A Thomas. Elements of Information Theory. Wiley, 2006.
[36] A Verdugo Lazo and P Rathie. On the entropy of continuous probability distributions (corresp.). IEEE Transactions on Information Theory, 24012:2 1978.
[37] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- Feasible Trajectory Forecasting With Heterogeneous Data. arXiv:2001.03993 [cx], January 2021.
[38] David Ha and Douglas Eck. A Neural Representation of Sketch Drawings. 1704v:1704.03477 [cs, stat], May 2017.
[39] Ilia Kulikov, Sean Welleck, and Kyunghyun Cho. Mode recovery in neural autoregressive sequence modeling, arXiv preprint arXiv:2106.05455 2021.
[40] Raphaël Dang-Nhu, Gagandeep Singh, Pavol Bielik, and Martin Vechev. Adversarial attacks on prob- abilistic autoregressive forecasting models. In International Conference on Machine Learning, pages 2356-220 2365.266..26 PMLR, 2020.
12

---

[41] Marco F. Huber, Tim Bailey, Hugh Durrant-Whyte, and Uwe D. Hanebeck. On entropy approximation for Gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181-188, Seoul, August 2008, IEEE. ISBN 978-1-4244-2143-5. doi: 10.1 1..11099/F2200 2648666
[42] Chenchen Zhang and Yuan Luo. Approximating the differential entropy of gaussian mixtures. In GLOBECOM 20/7-2017 TEEE Global Communications Conference, pages 1-6, IEEE, 2017.
[43] Joseph V. Michalowicz, Jonathan M. Nichols, and Frank Bucholtz. Calculation of Differential Entropy for a Mixed Gaussian Distribution. Entropy, 10(3):200 206, September 2008. ISSN 1099-4300, doi: 10.3 00.3390/entropy-eelle 110030000.
[44] Gil Ariel and Yoram I Louzoun. Estimating differential entropy using recursive copula splitting. Entropy. An International and Interdisciplinary Journal of Entropy and Information Studies, 22(2):236, 2020.
[45] Brendon J Brewer. Computing entropies with nested sampling. Entropy. An International and Interdisci- plinary Journal of Entropy and Information Studies, 19(8):422, 2017.
[46] Jifí Aigl and Miroslav Šimandl. Differential entropy estimation by particles. IFAC Proceedings Volumes, 44(1): (4(():1119111111 1991-11999, January 2011. ISSN 1474-6670. doi: 10.3182/20100288--111 1002.0010
[47] Jan Beirlant, Edward Dudewicz, László Györfi, Edward C Van der Meulen, Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39 1997.
[48] Akko Kalma. Gazing in triads: A powerful signal in floor apportionment. British Journal of Social Psychology. 31(1):21-39, March 1992.
[49] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Insernational conference machine learning, pages 3319-3328. PMIRR 2017.
[50] Hayley Hung, Dinesh Babu Jayagopi. Sileye Ba, Jean-Mare Odobez, and Daniel Gatica-Peree Perez. Investigating automatic dominance estimation in groups from visual attention and speaking activity. In Proceedings the 10th international conference on Multimodal interfaces, pages 233-236, 2008
[51] Volha Petukhova and Harry Bunt. Who's next? Speaker-selcction mechanisms in multiparty dialogue. In Workshop on the Semantics and Pragmatics of Dialogue, 2009.
[S2] Iwan De Kok and Dirk Heylen. Multimodal end-of-turn prediction in multi-party meetings. Proceedings of the 2009 International Conference on Multimodal Interfaces, pages 91 91-98, 2009.
[53] Usman Malik, Julien Saunier, Kotaro Funakoshi, and Alexandre Pauchet. Who speaks next? Turn change and next speaker prediction in multimodal multiparty interaction. In 2020 JEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAT), pages 349-354, IEBE, 2020.
[54] Ryo Ishii, Shiro Kumano, and Kazuhiro Otsuka. Prediction of Next- Utteracee Timing using Head Movement in Multi- Aulti-Party Meetings. In Proceedings of the 5th International Conference Human Agent Interaction, HAI 17., pages 181-187, New York, NY, USA. October 2017. Association for Computing doi: 11114////57993111111
Machinery. ISBN 978- 778-1-45003-511-2
[55] German Barquero, Johnny Núñez, Sergio Escalera, Zhen Xu, Wei-Wei Tu, Isabelle Guyon, and Cristina Palmero. Didn't see that coming: A survey on non-verbal social human behavior forecasting. arXiv:2203.0 224002000000000 [ex], March 2022.
[56] Dario Pavllo, David Grangier, and Michael Auli, QuaterNet: A Quaternion-based Recurrent Model for Human Motion. / [cs], July 2018
[57] Laura Cabrera-Quires Andrew Demetriou, Ekin Gedik, Leander van der Meij. and Hayley Hung. The matchnmingle dataset: novel multi-sensor resource for the analysis of social interactions and group dynamics in the- during free-standing conversations and speed dates. IEEE Transactions on Affective Computing, 2018
[58] Mark L. Knapp, Roderick Hart, Gustav W Friedrich, and Gary M Shulman. The rhctoric of goodbye: Verbal and nonverbal correlates of human leave-taking. Communications Monogruphs, 40(3):182-198, 1973.
[59] Monica M. Moore. Nonverbal courtship patterns in women: Context and consequences. Ethology and Sociobiology, 6(4):237-244 224,, January 1985. ISSN 0195- doi: 10.1016/0162- 10.101660662-3355888900016---
13

---

[60] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou. Bogdan Stanciuleseu, and Fabien Moutarde. THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling. arXiv:2110.06607 [cs]. January 2022.
[61] Sandra Carrasco, David Feráández Llorca, and Miguel Ángel Sotelo. SCOUT: Sncially- and Un- dersTandable Graph Attention Network for Trajectory Prediction of Vehicles and VRUs. arXiv:2/02.0633) [cs], May 2021.
[62] Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research, (9()): 895-935, 2020.
[63] Holger Caesar, Varun Bankiti, Alex H. Lang. Sourabh Vora, Venice Frin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multimodal Dataset for Autonomous Driving. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recugnition (CVPR), pages 11618-1 1628,, Seattle, WA, USA, June 2020. IEEE. ISBN 778-1-72817-168-5 doi: 10. 10911109422020002222 2020.01 164.
[64] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11: 1803-1831, 2010.
[65] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.0388 2017.
[66] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation differences. In International conference on machine learning, pages 3145-3153, PMLR, 2017.
[67] Sarah Adel Bargal, Andrea Zunino, Donghyun Kim, Jianming Zhang, Vittorio Murino, and Stan Sclaroff. Excitation Backprop for RNNs. arXiv:1711.06778 [es]. March 2018.
[68] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pages 818 833. Springer, 2014.
[69] Felix Mujkanovic, Vanja Doskoč, Martin Schimeck. Patrick Schäfer, and Tobias Friedrich. timeXplain A Framework for Explaining the Predictions of Time Series Classifiers. urXiv:2007.07606 [cs, stat], July 2020.
[70] Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Clinical intervention prediction and understanding using deep networks. arXiv preprint arXiv:1705.08-40 2017.
[71] Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi. Benchmarking Decp Learning Interpretability in Time Series Predictions. arXiv:2010. 12220000.19924 13924 (cs, stat], October 2020.
[72] Edward Choi. Mohammad Taha Bahadori, Joshua A. Kulas, Andy Schuetz, Walter Stewart, and Jimeng Sun. RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism. arXiv: 1608,057755 [es), August 2016.
[73] Ahmed M Alaa and Mihaela van der Schaar. Attentive state-space modeling of disease progression. Advances in neural information processing systems, 32, 2019.
[74] Madhurima Panja, Uttam Kumar, and Tanujit Chakraborty. An Interpretable Probabilistic Autoregressive Neural Network Model for Time Series Forecasting, arXiv:2204, 09640 (cs, stat), April 2022.
[75] Longyuan Li. Junchi Yan, Xiaokang Yang, and Yaohui Jin. Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting. arXiv:2102.00397 [cs, January 2021.
[76] Shalmali Joshi, Sonali Parbhoo, and Finale Doshi-Velez, Pre-emptive learning-to-defer for sequential medical decision decision-making under uncertainty, September 2021.
14

---

A Broader Related Work: Explainable Methods for Time-Series Data Across Tasks and Domains
The larger focus of explainability techniques involving time-series data has been on the task of classifying time-series. The goal has been to estimate the relevance of each input feature given timestep towards each output class. Here, saliency approaches often overlap with techniques developed for image data and can be categorized into:
Pradient-Based Techniques. The broad approach involves evaluating the gradient of the output class with respect to the input [64]. Several variants have been proposed [19 49 65-67
Perturbation- Based Techniques. The idea to examine how the output changes in response to some perturbation of the input. Perturbations are implemented by either occluding contiguous regions
of the input [68, 69]; performing an ablation of the features [70]; or randomly permuting features Ismail et al. 71 provide a benchmark of subset of these techniques.
[2]
Attention- Based Techniques. These incorporate an attention mechanism into the model that is trained to attribute importance to different parts of the input sequence towards prediction at each future timestep. Such techniques have been extensively utilized for healthcare data, Early methods applied reverse-time attention [72). Later methods applied the attention to probabilistic state-space representations [73].
Some of these broad ideas have been applied to the regression setting to make interpretable forecasts of future time-series features. Lim al. [5] leveraged self-attention layers for capturing long-term dependencies. Pan et al. [17] recently proposed computing saliency as mixup strategy between series images and their perturbed version with learnable mask for each sample. They view saliency in terms of minimizing the mean squared error between the predictions and ground-truthy for a particular instance. Focusing on the univariate forecssin problem, Oreshkin al. [14] proposed injecting inductive biases by computing the forecast as a combination of a trend and seasonality model. They argue that this decomposition makes the outputs more interpretable.
Developing explainable techniques for the probabilistic forecasting setting remains largely unexplored and subject to non-overlapping notions of explainability. Rügamer et al. [13] transform the forecast using predefined basis functions such as Bernstein polynomials. They relate interpretability to the coefficients of these basis functions (a notion similar to that of Oreshkin et al. [14). Panja al. [74] embed the classical linear ARIMA model into non-lincar autoregressive neural network for univariate probabilistic forecasting. As before, the explainability here also stems from the 'white-box' nature of the linear ARIMA component. Li et al. [75] propose an automatic relevance determination network to identify useful exogenous variables (i.e. variables that can affect the forecast without being part of the time-series data). To the best of our knowledge, saliency-based methods have not yet been considered within this setting.
B Favorable Properties of Differential Entropy
Differential entropy possesses favorable properties that make it suitable choice á for computing the saliency map. First, the scale of the forecast density does not affect the resulting saliency map (see Cover and Thomas [35 Theorem 8.6.4)):
h(aY) h(Y) + log |a), for * 0, and
(8)
M(AY) M(Y) + log (det(A)), when A is square matrix.
(9)
That is, scaling the distribution changes the differential entropy by only constant factor. So the saliency map resulting from inserting the entropy into [qqatton 4 remains unaffected since the Jacobian term only depends on the relative change in entropy across different choices of lobs- Similarly, translating the predicted density leaves the saliency map unaffected (see Cover and Thomas 35 Theorem 8.6.3]):
h(Y + +c)=+(())
(10)

---

C Implementation Details for Experiments
C.1 Other Explainability Methods
For DeepSHAP and GradientSHAP, we used the official implementation of SHAP: https::/ttt:tiiiiee com/slundberg/shun For IntegratedGradients and SmoothGrad, we used the Captum framework:
https://captum. ai/ We reiterate that these are fair comparisons, for the reasons have discussed in Section 2 One crucial reason is that no existing method is designed to handle a predicted distribution. To apply them in this context, we need compute attribution values for the predicted mean and standard deviation for every feature at every E tiut in isolation. In contrast, by measuring differential entropy, our method jointly accounts for the parameters of the distribution and captures the information content of the distribution. Despite these limitations, we include these comparisons to provide readers with a contextual understanding of the results obtained from commonly used explainability tools.
Computational Efficiency. For further insight, we measured the execution time of our saliency method compared to DeepSHAP for both real-world scenarios (see Table T). The main takeaway is that our method is an order of magnitude faster at computing the saliency map compared to DeepSHAP. This is in part because DeepSHAP computes values for every feature at every timestep in the output independently. Even if this is parallelized, DeepSHAP requires multiple forward passes and gradient computations including samples from background set to compute the reference values. The computation time scales with the size of the background set.
Table 1: Comparing Computational Efficiency, We compare the practical execution time of our method to running DeepSHAP. For reasonably fair characterization of DeepSHAP, report execution time for computing the SHAP values associated with a single predicted parameter, the mean of the future distribution.
Saliency
DeepSHAP
Scenario
Forward Pass
Compute Saliency
Init
Compute Values
Group leaving (MatchNMingle dataset)
ms 2.5
2 ms
50 ms
13.5 min
Vehicle trajectory (nuScenes dataset)
33 ms
68 ms
26.3 min
C.2 Empirical Validation using Synthetic Data
We model the future distribution using a Gaussian function for simplicity (setting std. to 10-10 for the single future), but a more complex distribution that predicts the appropriate change in variance would also work in practice. We now implement Algorithm [ as follows. We identify window where a turn change occurs in the data (frames 183-228) and denote this 45 frame window as the bit of interest. While we manually identify an interesting event for illustration, such window could also correspond to an interesting prediction by a model. We generate a set of candidate lobs by sliding a 30 frame window over horizon of 100 frames prior tjut, with stride of frame. For every observed tobs. we fit Gaussian density to the quaternion and speaking status features of all participants over the futures that can occur during tiut. We then the entropy of this Gaussian density as the feature & for that tobs. For the experiments with real forecasting model, we employ the [recurrent, uniform attention] variant of the Social Process family given its ability to capture dynamic movements 4
C.3 Forecasting Group Leaving Behavior
We used the pretrained model on the MatchNMingle dataset provided with the official implementation of Social Processes 4: https://githut www.cco//cc//eeeee chtragraaann
Specifically, we employed the [recurrent, dot- cot-attentionl variant of the Attentive Social Process family (ASP-GRU-dot). We set 4 to correspond to a second window (3 frames the data, which is at 1 Hz) containing individual leaving conversing group. We obtained forecasts from the model corresponding to a rolling 5 second labs within a 20 second preceding horizon. For computing DeepSHAP and GradientSHAP values, used the entire observed time horizon the background

---

dataset. This ensures that the expected values are computed within reasonably similar context for given Eabs.
C.4 Vehicle Trajectory Forecasting
We trained the Trajectron++ model 37 on the nuScenes dataset (mini version) 63 using the default command provided in the official implementation: https://gi com/StanfordASL.. Trajectron- plus-plus In particular, we used the int_ee model which incorporates the agent's system dynamics to produce dynamically-feasee from scene index 3 (scene. name 757). We set tiue to correspond to window of 6 timesteps (14-19) with lookback horizon from timesteps to 13. For computing DeepSHAP values we again used the entire observed horizon as the background set. The features in the observed sequence contained several NaN entries, which resulted in NaM DeepSHAP values. We ignored these when aggregating values per timestep.
trajectories. For analysis, we used sequence
D Additional Results
Time HorDon
Saleere Rosaton rdsgress
Head Oirection In anglee
Rotation (dogroos)
Head Direction
Saliency Lin tax lowards Ex:
Sailency of observed in tas towards future over tex
Sailency
SHAP values diverved towards future over
SHAF
SHAP (moan)
SHAP (moar))
0.50
DespOraP GrediniShip
DeepSmeP GradeenSiuP
SHAPP (ced)
SHAP (2ne)
0.5
-0.1
-025
1828
(a) 14:27 Day 1, Cam 12
(b) 25:42 Day 2, Cam 5
Figure 6: Analysis of two sequences the Match dataset. Row Video frames and overlaid arrows denoting the head orientation of the participant of interest. Row 2: Head orientation plotted as 2D horizontal rotation. Rows 3-5: Saliency map from running predictions from the Attentive Social Process model through framework, well as the mean DeepSHAP and GradientSHAP values across features per timestep for the predicted mean and std. of the same forecast. Our saliency framework succeeds in identifying the salient timesteps in both cases. In (a), the timestep in which the participant of interest is looking away is identified as salient towards predicting the other participant leaving the dyadic interaction. In (b), the timestep which the participant of interest suddenly stops actively participating in the conversation (not nodding or looking at the speakers) is identified as salient towards predicting their group leaving.
3

---

Saliency<pf observed toss towards future pver
Sallency
0.04
SHAP values of observed tats towards future over Fiu
SHAP (mean)
-0.5
DeepSHAP
SHAP (std)
12
Timestep
Figure 7: Analysis of sequence on Scene 0 in the nuScenes dataset. Row I: Video frames showing the black turning car and surrounding cars from the camera. Row 2: Future predictions for the black car position (circled) from the Trajectron++ model (ground truth in white, predicted mean black and variance in red). Rows 3-5: Saliency map from running predictions through our framework, well as the mean DeepSHAP values across features per timestep for the predicted mean and std. of the same forecast. Our framework reveals that the silver and point-of-view cars slowing is salient for the model in predicting that the black car completes the turn.

---

E Broader Discussion: Saliency & XAI with Domain Experts in the Loop
We begin this broader discussion by revisiting the different notions of saliency, make the case for why our proposed framework is suitable for forecasting tasks. Rather than defining saliency in a top-down manner as a function of some task-specific error metric, we have started from a more fundamental conception of bottom-up, or task-agnostic, saliency. Loog's s [ original definition pertains to preattentive saliency, which captures what is perceived to be subconsciously informative before conscious (attentive) processing by the brain. Here, surprising or unexpected observation is salient. For instance, in a large white image with a single black pixel, the black pixel is salient. The direct application of this concept to time-series data would involve identifying surprising task-agnostic temporal events. For instance, imagine viewing a static landscape where bird suddenly flies in. The entry of the bird into the scene is unexpected, and therefore salient.
When applied to forecasting tasks, however, this idea of surprisal (or unexpectedness or informa- tiveness) that saliency represents needs to be tied to the future outcome. The saliency computed by most methods working on given specific realization of the future. While not explicitly stated by these works, we argue that this notion of saliency is related to the surprisal in PXlY for some specific Y. We therefore interpret these methods as being associative in nature within Miller's [10] categorization in Section 3 In contrast, our approach is counterfactual because we examine alternate future outcomes, while conceptualizing saliency more naturally defined in terms of the changes in the uncertainty in Py x in response to different realizations of observed sequences. However, rather than corresponding to random occlusions perturbations of the input, the different realizations of X in our framework correspond to real features (or behaviors) preceding a future, which is more suitable to present to domain experts candidate causes.
tasks deals with which past features are surprising
Loog's 's [ unifying framework subsumes all forms of saliency, although identifying the appropriate for specific domain is non-trivial. In this work we have established both theoretically and empirically how expressing & in terms of the information about the future enables principled counterfactual reasoning in forecasting settings. Nevertheless, we reiterate that the salient timesteps retrieved by our framework ought to be treated as candidate causes until subsequently examined along with a domain expert. Our stance human-in-the-lopg XAI also aligns with rescarch on saliency-based and general XAI in other domains g (6).
In principle, when it is possible to have access to the true pr|x, the salient timesteps identified by our framework reflect the true predictive structural relationships captured by the underlying model across the entire data. However, estimating this density analytically entails identifying the multiple futures in the data corresponding to every occurrence of the same observed features. In practice, subtle variations in behaviors and sensor measurement errors make it infeasible to estimate Py analytically, so a model trained to capture generalized patterns from the given data. In these cases, our framework identifies the sequences that the model considers salient for its forecasts given the data. Consequently, subsequent causal analysis of the features over the salient timesteps is crucial, especially in the healthcare and human behavior domains to avoid potential prejudices against certain behaviors, or worse, misdiagnoses of affective conditions.
5


Reviews:
[{"review_id": "1kFlFxcZnr", "reviewer": "Reviewer_VPQB", "summary": "This paper presents a method for detecting salient points in time+feature space for models which make predictions from timeseries data. Salient points are defined as points that once observed produce a large change in the model’s distribution of predicted future outcomes (measured as differential entropy). The method is evaluated on a toy dataset, in which the factors that lead to future events are known, and in two real-world datasets where these factors are unknown. The proposed method finds the true salient features in toy data and finds plausible salient features in real-world data.", "questions": "- I’m unclear why you can drop the prior (-log p_x(x)) in between equations 3 and 4. I assume Loog dropped this due to assuming a uniform prior, but is p_x uniform here? (Also, in line 195 – it seems like the saliency is only “independent of previously observed data” if you assume a uniform p_x, which is not always a good assumption for real-world image saliency tasks.)\n- What are the \"low-level features\" (line 294) used to model the synthetic dataset? Pixels? Edges?", "limitations": "This is adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written with a clear explanation of the theory, method, and limitations.\n- The approach is novel, at least for this specific application (information-theoretic models of saliency are common in other areas, such as predicting human eye movements in single images).\n- The proposed method is straightforward and works well.", "weaknesses": "- The evaluation could be more detailed (e.g., evaluating the effect of different model design choices or how sensitive the model is to time parameters t_obs and t_fut, which have to be set by the user).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and your comments, VPQB! We are glad that you found the paper to be well-written and clearly explained.\n\n1a\\.\n>I’m unclear why you can drop the prior (-log p_x(x)) between equations 3 and 4… is p_x uniform here... which is not always a good assumption for real-world image saliency tasks.\n\nThis term can indeed be dropped as the prior is assumed to be uniform, and consequently, does not essentially alter the saliency map (a monotonic transformation preserves relative saliency of the image locations, L191-2.) In Loog's work, this seems the principled choice in the absence of any a priori information on which locations may be more surprising than others. (However, irrespective of the choice of prior, the saliency still does not require density estimation from multiple images as implied in your parenthetical note; please see 1b below).\n\nIn our setting with time series data, a uniform prior over timesteps corresponds to the intuition that, in the absence of any prior information, all timesteps in the $t_{\\mathrm{obs}}$ horizon are equally salient for the model’s forecasts over the given $t_{\\mathrm{fut}}$. Since the framework captures *preattentive* saliency before conscious processing (in this case by the forecasting model M), we believe this is a reasonable prior assumption.\n\nHowever, a nonuniform prior can be incorporated trivially, by replacing Eq. 4 with Eq. 3 in Step 4 of Algorithm 1. This may be relevant in specific use cases if researchers have domain knowledge that would *a priori* weigh some timesteps as being more salient than others. (E.g. prioritizing short-term over long-term time dependencies when studying specific behavioral phenomena would make timesteps immediately before $t_{\\mathrm{fut}}$ more salient a priori.) We’ll clarify this in the manuscript.\n\n1b\\.\n> Also, in line 195 (“Moreover, the saliency computation is purely local to an image, making it independent of previously observed data.”) - it seems like the saliency is only “independent of previously observed data” if you assume a uniform p_x.\n\nThere seems to be a misunderstanding. *The statement holds irrespective of the choice of prior*, for the full expression of saliency (our Eq.3, [5, Eq.4]). In Loog's original setting, the idea is that we can calculate the pre-attentive saliency for a given image without reference to any previously observed images in a dataset. Quoting directly Loog's original paper:  \n\n“The (Eq.4) implies that the computation of saliency can be performed directly, based on purely “local” measurements and, surprisingly, without the need to refer to (previously) observed data or any explicit density estimate from these, something all prior formulations rely on [2,12,16,37].” - paragraph following [5, Eq.4]\n\nThe point is that information-theoretic formulations of saliency prior to Loog’s [5] involved a step to approximate a density estimate from images in the dataset. (E.g. Torralba’s formulation [5, quoted ref. 37] required estimating the distribution of local features given global scene features/background over the training set.) The significance of Loog’s formulation in our opinion is that such density estimates are not required since it can be computed exactly: it only depends on the local features within a single image at a location described by $\\phi$.\n\n2\\.\n> What are the \"low-level features\" (line 294) used to model the synthetic dataset?\n\nHere we were referring to the 5D feature vectors described in L285-288: 4D real-valued quaternions (qw, qx, qy, qz) to represent 3D head poses commonly used for human motion and pose representation [4,52,53], and binary speaking status. (In contrast, ‘high-level’ features would capture behavioral semantics in this low-level data stream e.g. an explicit action label such as 'head nod' or 'floor grab' as listeners try to take over the conversation) \n\n3\\.\n>...how sensitive the model is to time parameters t_obs and t_fut, which have to be set by the user\n\nNote that choosing $t_{\\mathrm{obs}}$ and $t_{\\mathrm{fut}}$ corresponds to a human selecting sequences of data that are of some interest for analysis, rather than setting some hyperparameters that affect performance in the traditional sense of training a model. (We’re not sure if the ‘model’ refers to the underlying forecasting model we take as input or our framework which only requires the predicted distribution for computing entropy without additional hyperparams). Nevertheless, in our Global Response, we have added analyses for additional sequences from the real-world datasets (corresponding to different $t_{\\mathrm{fut}}$ and several preceding $t_{\\mathrm{obs}}$. We interpreted this comment to be in the same spirit as 82xZ’s comment about more real-world examples.\n"}, {"review_id": "ENcCsGZ02E", "reviewer": "Reviewer_L4MH", "summary": "This paper proposes a method for counterfactual explanation using Information-Theoretic Saliency. The method trains a model using data collected up to the present moment and applies it to predict the outcomes caused by events occurring in the future time. This paper provides experiments on synthetic and real-world data, and the results demonstrate the effectiveness of this method to a certain extent.", "questions": "* This paper addresses the problem of time-series forecasting (regression). How does this problem differ from time-series modelling?\n* This paper builds upon Miller's framework, but it lacks a detailed introduction to Miller's framework, which hinders the smooth flow of the article.\n* What is the necessity of using information-theoretic methods to solve the problem?\n* In line 168, the paper mentions replacing counterfactuals from random perturbation by observation, which makes me confused. Intuitively, observation should refer to factual data. Or it seems the authors are just emphasizing the inclusion of some real features.\n* The design of the causal graph shown in Figure 1 raises some doubts: \n\n(i) In the field of causality, specific variables are generally set as causal nodes, while the model is designed as the causal path between nodes. In our environment, we can only intervene in variables, not the causal effects between variables. Exogenous variables should act on endogenous variables, not the model itself. If we adopt the causal graph depicted in Figure 1, the implicit implication is that the model is also a manipulable factor. However, the model should be used to describe the relationship between t_obs and t_fut, which should remain constant in the environment. In other words, the relationship between t_obs and t_fut data should not change based on the effectiveness of the trained model. \n\n(ii) The proposed method relies on the correctness of causal graph shown in Figure 1a. My concern is that exogenous variables should not only exist in t_obs, t_fut, and M (assuming M's position in the causal graph is reasonable), but more importantly, they should also exist in I_fut. If you assume that I_fut is not influenced by exogenous variables or confounders, then the abduction-action-prediction process is unnecessary for the method you have designed, and the problem degenerates into a simple time series prediction problem.\n* Based on the aforementioned issues, the model should remain constant in the environment, and it is necessary to estimate the model because it is unknown. In the process of estimating the model, since t_obs is used for training, how can we ensure the identifiability of the estimated model? In other words, how can we guarantee that the learned model is the true model?\n* Could the authors explain why \\hat{Y} is introduced in Section 4.3? What confuses me is that from lines 228-233, it seems that \\hat{Y} is sampled from the distribution as a single value. However, in Equation 7 and the subsequent descriptions, \\hat{Y} appears to become a variable, which makes me feel that the mathematics used here is not very rigorous.\n* Did the experiments include baselines for comparison with other relevant methods?", "limitations": "The author presents innovative information-theoretic saliency metrics; however, the rigour of this paper could be improved.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper presents a method which uses information-theoretic saliency metrics, the motivation is interesting. The experiment results show the effectiveness of the proposed method on both synthetic and real-world data.", "weaknesses": "In my opinion, the biggest issue with this paper is that, despite the authors claiming to integrate the tasks of future forecasting and counterfactual reasoning, I don't see any actual utilization of counterfactual reasoning techniques in this article. Firstly, the design of the causal graph does not adhere to the standards of causal work. Secondly, the paper mentions the use of Pearl's abduction-action-prediction steps, but I fail to see where abduction (inference of exogenous variables) and action (intervention) are applied. I think the authors should delve deeper into understanding the relationship between counterfactual reasoning and the problem addressed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Clarifying Central Misinterpretations\nThanks for the detailed review, L4MH! There seem to be some misinterpretations (perhaps expecting alignment with traditional causal inference rather than XAI literature):\n\n1\\.\n> ..the model is designed as the causal path between nodes. Exogenous variables should not act on the model…model should remain constant…guarantee model is true model?\n\nWe suspect that the reviewer might be mistaking the forecasting model M in our graph with the causal model as is typically defined in causal inference literature [22, p. 206].\n\nOur causal graph is not meant to describe relationships between random variables in the data as is typical in causal inference (e.g. effect of [head rotation] on [group leaving]). Rather, it describes the *process of a human generating explanations* for a given pretrained forecasting model—irrespective of whether or not it is the true model—for some sequences in the data ($t_\\mathrm{obs}$, $t_\\mathrm{fut}$). Given that different models can be chosen for explanation, $M$ is a natural random variable here. This is consistent with how causal graphs have been used in explainability works [9, Fig. 2a] (although [9] doesn’t discuss exogenous factors at all, which we’ve tried to correct here).\n\n2\\. \n> The relationship between t_obs and t_fut data should not change based on model. Exogenous variables in I_fut\n\nActually, the relationship between the data over $t_\\mathrm{obs}$ and $t_\\mathrm{fut}$ can change when $M$ changes: different models (potentially) yield different forecasts, and consequently different information in the predicted distribution $I_\\mathrm{fut}$. Our approach exploits this connection to help an expert to get to data-driven hypotheses, which, ultimately, are formed through those very forecasting models. Once M is chosen, its predicted distribution for a $t_\\mathrm{fut}$ given a $t_\\mathrm{obs}$ is fixed, so $I_\\mathrm{fut}$ doesn’t have any other source of exogenous randomness.\n\n3\\.\n> I don't see any utilization of counterfactual reasoning\n\nPlease note that counterfactuals as used in XAI literature are distinct from that in causal inference:\n\n>“It is important to note that this is not the same counterfactual that one refers to when determining causality. For causality, the counterfactuals are hypothetical ‘noncauses’...whereas in contrastive explanation, counterfactuals are hypothetical outcomes.” [10, Sec. 2.3]\n\nMiller’s point is that ‘why’ explanations are contrastive (L62, L123) and require comparing ‘outcomes’ in response to alternate ‘causes’. In our work, this ‘outcome’ relates to the information in M’s predicted distribution, the what-if question being “would the information in M’s prediction for the window $t_\\mathrm{fut}$ change if it had observed features over a different (contrastive) $t_\\mathrm{obs}$?”. So, the logical implication in Eq. 1 is a counterfactual statement. (Contrast this to associative reasoning (L124-33, L663-66 in App. D, and [10, Tab. 3]) which would use only a single $t_\\mathrm{obs}$ to generate the attribution map).\n\n4\\.\n> ..the use of Pearl's abduction-action-prediction steps…the process is unnecessary.\n\nFirstly, we haven’t claimed to explicitly *use* the steps but only meant to put our work in context of these steps; we explicitly echo the reviewer in L157-60: “...estimating the distribution over the exogenous variables *from the abduction step is conceptually not applicable in this setting*”\n\n### Questions\n\n1\\.\n> What is the necessity of using information-theoretic methods…?\n\nIn a way, information theory may not be strictly necessary, as there are other formulations, e.g. in terms of energies or losses. To us, however, the language of information theory seems natural in the context of probabilistic models. Moreover, it acknowledges the link with previous works like [5] and [24].\n\n2\\.\n> The paper mentions replacing counterfactuals from random perturbation by observation.. observation should refer to factual data.\n\n(For the distinction between counterfactuals as hypothetical unobservables in causal inference vs their use in XAI, please see clarification #3). The operationalization of counterfactuals as data samples is consistent with several explainability techniques for time-series classification [R1,R2,R3].\n\nOne reason to move away from random perturbations as counterfactuals is that these perturbed samples may not be realistic alternatives (i.e. they may not preserve semantics or lie on the manifold of valid data; L87-9). Our information-theoretic approach shows that we can, in fact, use derivatives of the observables (which may still be interpreted as a particular infinitesimal perturbation in time, but represent valid data) to come to a principled definition of saliency.\n\n3\\.\n> ​​..why \\hat{Y} is introduced in Section 4.3?\n\nComputing $h(Y|X)$ requires the joint distribution over the full decoded future timesteps. In the commonly used scheme of autoregressive decoding [4,33,3,38] (L226-7), the decoder doesn’t actually compute the full joint over the future (Fig.2), but relies on samples at each timestep $\\hat{Y}$ as input to the next (non-bold consistently denotes random variable, bold being the sampled array). Training such a decoder by maximizing likelihood actually assumes the inequality in Eq. 7 to be approximately equal (see ‘Approximate decoding’ [R4, Sec. 2], Eq. 5 [R5]). This poses problems for computing the entropy of the distribution, and so we explicitly redefine $\\phi$ in L242 to address this scenario.\n\n### REFERENCES\n\n[R1] E. Ates et al. \"Counterfactual explanations for multivariate time series.\"\n\n[R2] J. Lang et al. \"Generating sparse counterfactual explanations for multivariate time series.\"\n\n[R3] E. Delaney et al. \"Instance-based counterfactual explanations for time series classification.\"\n\n[R4] I. Kulikov et al. \"Mode recovery in neural autoregressive sequence modeling.\"\n\n[R5] R. Dang-Nhu et al. \"Adversarial attacks on probabilistic autoregressive forecasting models.\""}, {"review_id": "OZcBdLGHIU", "reviewer": "Reviewer_N3J3", "summary": "The paper proposes a computational XAI framework for counterfactual reasoning in probabilistic multivariate time-series forecasting. The key idea of the paper is to establish the conceptual foundation for linking saliency-based explanations with counterfactual reasoning. Specifically, the paper leverages a unifying definition of information theoretic saliency grounded in preattentive human visual cognition and extends it to forecasting settings. Experiments on synthetic and real data demonstrate the utility of the proposed framework.", "questions": "Please see Weaknesses", "limitations": "Authors adequately addressed the limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The key idea of the paper is well-motivated. Deriving the conceptual link between counterfactual reasoning and saliency-based explanation techniques from Miller’s framework is interesting. Also, it is well-explained in section 3. Experiments sufficiently justify the key conclusions made in the paper. The evaluations and analysis have a wide range practical applicability in multiple real world problems.", "weaknesses": "I do not find any major weaknesses with the work. Here are some minor weaknesses: I find it hard to completely follow the use of information-theoretic preattentive in section 4 (and section 4.3 in particular). Improving the readability with multiple examples could help readers to understand the key intuitions. Also, It would be nice to have some quantitative evaluation metrics for the evaluations made in section 5.2.1 and 5.2.2. \n\nMissing related work:\n\nAkula, A., Wang, S., & Zhu, S. C. (2020, April). Cocox: Generating conceptual and counterfactual explanations via fault-lines. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 03, pp. 2594-2601).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review, N3J3. We are glad that you found our derivation of the conceptual link between saliency-based explanations and counterfactual reasoning interesting and well-explained.  \n\n### Addressing minor weakness comments\n\n1\\.\n>  … (and section 4.3 in particular). Improving the readability with multiple examples could help readers to understand the key intuitions.\n\nThanks for the suggestion, reviewer 82xZ also suggested including running examples. We’re not allowed a revision this year but will have an extra content page if accepted, and will add the running example we used in the introduction (pedestrians avoiding collision) to expand Sec. 4.3 to aid in understanding. We’ll also use this to further include our explanation of why the $\\hat{Y}$ is relevant in response to reviewer L4MH’s comments.\n\n2\\.\n>  Also, It would be nice to have some quantitative evaluation metrics for the evaluations made in section 5.2.1 and 5.2.2.\n\nThis was a topic of extended consideration amongst the authors already during writing, and we’ve explained the conceptual issues in adopting or deriving a meaningful quantitative metric in our Global Response. Please let us know what you think, and if you have suggestions about metrics that would avoid these conceptual issues we’ve mentioned, we’d be happy to update the results during the author-reviewer discussion week. \n\nMeanwhile, we have also added more qualitative examples from the real-world scenario for readers to get more insight into how the method can be useful in Global Response.\n\n3\\.\n> Missing reference\n\nThanks! We will add this paper to the final version."}, {"review_id": "4Ufh2gv0om", "reviewer": "Reviewer_82xZ", "summary": "The authors create a saliency method for time-series data, where they quantify the impact that a previous time-step has on the model's prediction of a future time-step. They deploy this method on synthetic and real-world video datasets, and show that this method is able to pick up on key points in the time-series process. ", "questions": "Is it possible to use such a formulation for sequential but non-time series data (for example, understanding important tokens in the prompt for an NLP setting?)", "limitations": "The authors do discuss limitations to their work. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think this is a strong paper that is relevant to the community. The formulation seems quite elegant, and the results on both synthetic and natural experiments are strong (especially in comparison to DeepShap). In particular, I thought the vehicle trajectory was quite interesting, and would encourage the authors to spend more time discussing those results beyond a single example, perhaps by shortening Section 3.", "weaknesses": "Clarity: There were some aspects of the methodology that were hard to parse (in particular Section 4.3). One thing that might help is including a running example (perhaps the pedestrian one from the intro) throughout the methodology. \n\nBaselines: Can you include another baseline beyond just DeepShap (at least for the synthetic setting) e.g., Integrated Gradients or SmoothGrad.\n\nThe discussion of the real-world examples just talk about a few examples. I would suggest expanding this section, and including some quantitative metrics over the entire dataset. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and comments, reviewer 82xZ! We are delighted that you found our formulation to be elegant (we especially worked hard on this aspect), and our experimental results to be strong.\n\n### Answering Questions\n\n1\\.\n> Is it possible to use such a formulation for sequential but non-time series data (for example, understanding important tokens in the prompt for an NLP setting?)\n\nYes! Note that we have considered seq2seq tasks here, the key idea being evaluating how the entropy of the predicted distribution changes based on observing different positions in the input sequence (tokens in the case of NLP settings), so the formulation lends itself naturally to NLP data as well.\n\nWe were actually considering adding an experiment along the lines of recovering sensitivity to shortcuts, similar to the setting Bastings et al. considered in [R1]. But in the end we didn’t have space to do it justice; the two real-world settings we already considered formed a more coherent narrative in the available space.\n\nHowever, note that while computing entropy in such settings, Eq. 7 can be relevant if autoregressive decoding is employed. (where a decoded token from the predicted distribution over the vocabulary becomes the input to the next timestep). In such cases we recommend using the definition of $\\phi$ in line 242.\n\n### Addressing Weaknesses\n\n- **Baselines**: Done. We have included comparisons against Integrated Gradients, Smooth Grad, and Gradient SHAP in addition to the DeepSHAP results we had already included in the paper. Please refer to our Global Response for the results. (Following the rebuttal instructions we have shared the updated figures in our 1 page pdf and explained it in the response). The conclusions and take-away message do not change. \n\n- **More real-world examples**: Done. We had already discussed the results in more details in Appendix C. We’ve now added more examples for both real-world tasks. The figures are in the 1 page pdf in the Global Response, and we’ve discussed the cases in the text of the response. We will include these in the final version of the paper since we get an extra content page if accepted. \n\n- **Quantitative Metrics**: This was a matter of extended discussion amongst the authors when writing the paper. We have included the summary of conceptual issues in deriving a reasonable metric in the absence of ground truth saliency in real-world data, and why we came up with the synthetic setting to quantitatively validate the model in the Global Response. Please let us know your thoughts and we would be happy to compute them during the discussion phase and include them in the final version.\n\n- **Running example in Sec. 4.3**: Done. Thanks for the suggestion! We’re not allowed a revision during discussion this year; but in the final manuscript, we will include the running example of pedestrians to indicate what the notations mean, which should also help address the confusion reviewer L4MH mentioned about $\\hat{Y}$ in 4.3.\n\n### REFERENCE\n\n[R1] Bastings, Jasmijn, et al. \"Will You Find These Shortcuts? A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)\n"}, {"review_id": "jyPhtyFHaN", "reviewer": "Reviewer_f9yh", "summary": "This paper proposes a framework to provide counterfactual explanations for time-series forecasting models. This method attempts to detect the salient timestep which is critical to a future event. The method is built based on the framework of bottom-up preattentive saliency. Experiments are conducted on one synthetic and two real-world datasets, highlighting that the proposed model is able to localize the salient time step.", "questions": "L295, should it be Figure 4b? In Figure 4a, the perfect forecast model is considered. But there is a plateau in the saliency, i.e., more than one frame is labeled as salient. If this is the case, does the algorithm still work in this scenario?", "limitations": "The novelty of the proposed method needs to be improved. The method is built on the previous work from Loog et al. and causality models. The authors should address their novelty to highlight their contributions. Overall, this paper needs a non-trivial revision to address all weaknesses and to make the contribution clear.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This project tackles a relevant and not well-studied problem: explaining time-series forecasting models. The authors motivate their method using psychology and cognitive science, which bridges the gap between pure algorithmic explanation design and the need of human users.", "weaknesses": "(1)\tHow to understand that the explanation (salient timestep) is a counterfactual explanation? This framework is “to compute the saliency of an observed t with respect to a fixed choice of t” (L202), which cannot reflect the counterfactual. Does it infer that the “fixed choice of t” is the event that did not happen?\n\n(2)\tThe current method functions more as a “critical event detector” rather than an explainer. For instance, in the two experiments on real-world datasets, the tasks can be done by an action localization model studied in the video understanding area. The authors need to explain why these are research challenges that can be solved by model explanations. \n\n(3)\tThe experiment settings need to be clearer to address the advantages of the proposed explanations. As pointed out in (2), the current experiments are too similar to critical event detection rather than an explanation task. The authors should reconsider a task where a counterfactual explanation is generated and compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and comments, f9yh!\n\n### Clarifying Misinterpretation: Distinguishing from critical event detection and action localization\n\n> (2) The current method functions more as a “critical event detector”...the tasks can be done by an action localization model.\n\n[In L38-42 and App. D (L654-669) we discuss the difference between detecting a salient event in a time series and identifying preceding timesteps that are *salient towards a specific future forecast*. Here we use additional running examples to clarify.]\n\nPlease note that in event-detection literature the task of detecting critical events *does not involve forecasting*: it is effectively a classification task, involving labelling timesteps in a time-series as containing the event of interest or not [R1-R5]. Even when saliency is explicitly considered [R6], the goal is to see if a word is *salient towards some (action) label, not towards predicting if the action occurs later*. Using our real-world experiment as a running example (Fig. 5a), let’s say the critical event is [group leaving]. The ‘critical event detection’ version of the question would be **“does this video frame contain a person leaving a group?”** This would certainly be possible to achieve with an action localizer trained for [group leaving].\n\nHowever, this is not the goal of our paper. Given some forecasting model M, the question in the running example is **“which preceding timesteps made M predict the person leaving the group in the future?”** In our results, we found that timesteps corresponding to [turning away from group] was salient for the model predicting features corresponding to [group leaving]. However, such predictive relationships are often complex and unknown in the domain so the task isn’t simply to run an event detector to localize a pre-known action. Rather, our method enables the discovery of such predictive relationships using the forecasting model.\n\nMoreover, **the output of a critical event detector can be the starting point to using our proposed method**. In the running example, an F-formation detector [R7, R8] can be used to localize frames 728-30 as being the end of the group, which is then set as $t_{\\mathrm{fut}}$ for analysis before running Algorithm 1.\n\n### Addressing Other Questions and Weaknesses\n\n1\\.\n> This framework…cannot reflect the counterfactual. Does it infer that the “fixed choice of t” is the event that did not happen?\n\nLet’s say a social scientist is interested in ‘What behaviors predict group leaving?’ The fixed choice of $t_{\\mathrm{fut}}$ corresponds to timesteps 728-30 containing an instance of [group leaving]. For a given $t_{\\mathrm{obs}}$, the counterfactual reasoning involves the question: *what-if the model had seen different $t_{\\mathrm{obs}}$ (counterfactual), does it become more/less certain that the person will leave over 728-30?* (Also please see clarification #3 about counterfactuals in response to L4MH: https://openreview.net/forum?id=IrEYkhuxup&noteId=cXNtr2xBAI) \n\n2\\.\n> ..address the advantages of the proposed explanations [in experiments]\n\nThere are two advantages:\n\n\n- **Data-driven theory building and generating new domain knowledge [R9]**: As we demonstrate for group conversation, our method can be used by domain experts to build new data-driven hypotheses about causal relationships between behaviors (see App. D for discussion on human-in-the-loop XAI). Typically, hypotheses in social psychology are generated by the researcher based on their own experiences or theory [R10, Sec. 2.1].\n\n\n- **To improve/sanity-check the forecasting model**: When there is an obvious salient predictive relationship between timesteps (e.g. vehicle trajectory results), our method can be used to sanity-check that the model captures it.\n\n3\\.\n> L295, should it be Figure 4b? Does the algorithm still work \n\nWe believe Fig. 4a is the correct figure. Fig. 4b deals with the introduction of a trained forecasting model, which we discuss in L299-306. The analysis in L292-8 deals with evaluating our framework independent of any imperfections in the forecasting model (guarding against ‘garbage-in garbage-out'). The algorithm works; the Jacobian in Eq. 4 for the middle graph captures that only at frames 138 and 139 is there a local change in entropy, and consequently, both are salient.\n\n4\\.\n> The authors should address their novelty to highlight their contributions.\n\nWe have stated our contributions in L60-71. Please note:\n\n- No existing framework exists for identifying salient observed timesteps towards probabilistic forecasts in regression settings. So even the task is novel. Moreover, as our empirical validation experiments demonstrate, popular existing explainability techniques (DeepSHAP, GradientSHAP, IntegratedGradients, SmoothGrad) fail to retrieve true salient timesteps like our method does.\n\n- Conceptually, no work has linked counterfactual explanations to saliency-based techniques. Moreover, the existing saliency-based methods suffer from drawbacks (L82-95) that our method does not.\n\n### REFERENCES\n\n[R1] Zhang, Yunyi, et al. \"Unsupervised key event detection from massive text corpora.\"\n\n[R2] T. Nugent et al. \"A comparison of classification models for natural disaster and critical event detection from news\" \n\n[R3] Kosan, Mert, et al. \"Event detection on dynamic graphs.\"\n\n[R4] ML Jamil et al. “Detection of dangerous events on social media: a critical review.”\n\n[R5] Kosan, Mert et al. \"Graph Macro Dynamics with Self-Attention for Event Detection.\" \n\n[R6] Jian Liu et al. “Saliency as Evidence: Event Detection with Trigger Saliency Attribution.”\n\n[R7] ​​S. Thompson et al. “Conversational Group Detection with Graph Neural Networks.”\n\n[R8] E. Gedik and H. Hung. “Detecting Conversing Groups Using Social Dynamics from Wearable Acceleration: Group Size Awareness.”\n\n[R9] J. Radford et al. \"Theory in, theory out: the uses of social theory in machine learning for social science.\"\n\n[R10] E. Aronson et al. “Social Psychology”"}]

Author rebuttal:
## Updates: Additional Baselines, Real-World Examples (82xZ, VPQB, N3J3)

Following the suggestions, we have added the following (see attachment):

1\.
(82xZ). We added results from IntegratedGradients (IG) and SmoothGrad in Fig. R1. We also added GradientSHAP. The takeaway remains the same (the caveat in L278-9 also applies to these methods).

2\.
(82xZ, VPQB, N3J3). We added more examples from the real-world data (Fig. R2 and R3):

- Fig. R2a: MnM. The participant of interest looking away is salient for the model to predict interaction termination over the future window.
- Fig. R2b: MnM. The participant of interest stopping actively participating in the conversation (not nodding or looking at the speakers) is salient towards the model predicting them leaving.
- Fig. R3: nuScenes. The timesteps where the silver and point-of-view cars slow down while the black car advances are increasingly salient until the model is certain in predicting that the black car will complete the turn.

## Considerations for meaningful quantitative metrics in real-world experiments (82xZ, N3J3)

Quantitative evaluation was a matter of extended discussion amongst the authors at writing. Here we discuss why deriving a suitable quantitative metric in the absence of ground-truth saliency for real-world experiments entails conceptual issues and is consequently non-trivial.

The challenges and desiderata for meaningful evaluation of our work are summarized in L265-73:
- Ground-truth saliency for real-world data does not exist for quantitative evaluation.
- Qualitative evaluation can be subject to observer biases [7,11,12]
- The post-hoc explanation framework needs to be evaluated in isolation of any imperfections in the given underlying forecasting model (guarding against ‘garbage in, garbage out’).

### Recap: Current Evaluation Scheme (for the benefit of all readers)

Our empirical evaluation scheme in light of these challenges:
- To *validate* our method, we designed our synthetic setting to establish ground-truth saliency and found that
    - When the underlying model is the true forecasting model - our framework retrieves the salient timesteps perfectly (Fig. 4a).
    - For a real-world forecasting model trained on the data - our method outperforms other commonly used ones (Fig. 4b,c, Fig. R1)
- To *demonstrate empirical utility* we used real-world forecasting models in two use cases. We found that compared to DeepSHAP our method retrieves salient timesteps that align better with domain knowledge in social psychology (Sec. 5.2.1) or capture readily intuitive patterns suitable for sanity-checking models (Sec. 5.2.2).
 
### Possible options for quantitative metrics for real-world data (82xZ, N3J3)

In the absence of ground-truth saliency, having a reliable quantitative metric for real-world settings is an open question.

Here, we considered modifying existing metrics from classification such as Time Relevance Score (TRS) [R1] (which inspired AUCORR [17] for regression), but found them to be inadequate. These metrics hinge on masking or perturbing a progressively larger proportion of the salient timesteps, passing the perturbed samples as input, and evaluating the drop in accuracy (requiring ground-truth labels unavailable at test time) [R1] or correlation with predictions [17]. There are two conceptual problems:

- *Distortion interventions*: Random perturbation/masking of salient timesteps may destroy real-world semantics, constituting ‘distortion’ interventions (L88-9). E.g. in 5.2.1, if we randomly perturb the person’s head rotation when they look away from the group, it may result in the people glitching, which is not possible in the real world.
- *Notion of saliency*: These metrics capture a notion of saliency attached to task performance in accuracy or prediction metrics (i.e. top-down saliency, see L43-8 and App. D). Standardizing such metrics reinforces the false idea that all saliency maps are conceptually comparable, which is an issue in the field (L50-2).

Together, such conceptual issues often lead to using quantitative metrics to establish superiority of methods even if it may not be meaningful; for instance:

> “In general, although we can’t guarantee any selected feature values will have representative interpretation in qualitative analysis, the performance of our method is quantitatively superior to other competitors.” [17, Sec. 3.3]

We can fix the first issue by passing time-shifted real sequences in the data back as input (by moving the window to exclude the salient timesteps instead of masking or perturbing, keeping valid samples), but this is precisely how our method computes saliency in the first place, so by definition, such a strawman metric would make for a poor test! Moreover, the preattentive saliency framework is independent of specific tasks (L37-43, Sec. 3) and guaranteed to capture saliency for any $\phi$ in Eq. 3. What needs empirical validation is our proposed choice of $\phi$, which we have in Sec. 5.1.

So, actually, what the reviewers are asking for is a summary measure of how meaningful the saliency map is across the dataset. However, as we’ve stated in our argument for a human-in-the-loop methodology (App. D), evaluating the meaningfulness of saliency maps is best handled through confirmatory experiments with domain experts to avoid negative societal impact (Sec. 7). It is more responsible to treat the results on real data as hypotheses about the data rather than as quantitative truths cemented by conceptually problematic metrics.

Nevertheless, our existing empirical validation is still rigorous: since the method is validated  and also yields a closed-form expression, it serves as the benchmark for subsequent methods. If the reviewers have suggestions on a specific metric that does not suffer the discussed issues, we are happy to update during the discussion phase.

### REFERENCE

[R1] Ismail et al. “Benchmarking deep learning interpretability in time series predictions”

Meta-review: The paper received “accept, weak accept, accept, weak accept, and borderline accept”. While there is still room for it to be a stronger paper (please refer to the review comments), reviewers liked the idea and experimental results are solid. They agreed that it could be a worthy contribution to the NeurIPS community.  After considering the paper, reviews, and rebuttal, the area chair agrees with the reviewers' comments. We congratulate the authors on the acceptance of the paper!
Decision: Accept (poster)