SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chirag Raman Alec Nonnemaker Amelia Villegas-Morcillo Hayley Hung Marco Loog
Delft University of Technology, Delft, The Netherlands {c.a.raman, a.o.villegasmorcillo, h.hung, m.loog}@tudelft.nl a.m.nonnemaker@student.tudelft.nl
We propose a post hoc saliency-based explanation framework for counterfactual reasoning in probabilistic multivariate time-series forecasting (regression) settings. Building upon Miller's framework of explanations derived from research in multiple social science disciplines, we establish a conceptual link between counterfactual reasoning and saliency-based explanation techniques. To address the lack of a principled notion of saliency, we leverage a unifying definition of information-theoretic saliency grounded in preattentive human visual cognition and extend it to forecasting settings. Specifically, we obtain a closed-form expression for commonly used density functions to identify which observed timesteps appear salient to an underlying model in making its probabilistic forecasts. We empirically validate our framework in a principled manner using synthetic data to establish ground-truth saliency that is unavailable for real-world data. Finally, using real-world data and forecasting models, we demonstrate how our framework can assist domain experts in forming new data-driven hypotheses about the causal relationships between features in the wild.
As we go about our daily lives, engaging in conversations, walking down the street, or driving a car, we rely on our ability to anticipate the future actions and states of those around us [1, 2]. However, the numerous unknowns, such as hidden thoughts and intentions, make our predictions of the future inherently uncertain [2]. To reflect this uncertainty, several machine learning methods in such settings forecast a full distribution over plausible futures, rather than making a single point prediction [3, 4]. Identifying the factors that influence such a model's forecasts is particularly useful for domain experts seeking to understand the causal relationships guiding complex real-world behaviors, especially in situations where the future is uncertain. In this work, we introduce and address a novel research question toward counterfactual reasoning in multivariate probabilistic regression settings: how can we identify the observed timesteps that are salient for a model's probabilistic forecasts over a specific future window? Specifically, we introduce the first post hoc, model-agnostic, saliency-based explanation framework for probabilistic time-series forecasting.
We begin with a fundamental observation about human social cognition: we are averse to uncertainty and strive to minimize it [2]. Consider the scenario where a pedestrian is approaching you on the street. Initially, there is uncertainty about which direction each of you will take to avoid a collision. As one of you changes direction, the other observes and takes the opposite direction, ultimately avoiding a collision. Concretely, the thesis of this work is to formalize the following notion of saliency: the timestep that changes the uncertainty of a predicted future is salient toward predicting that future.
For instance, in the aforementioned scenario, we posit that the moment when one pedestrian changes direction is salient toward forecasting the future trajectories of the pedestrians.
Our notion of saliency is grounded in preattentive human cognition and related to the concept of surprisal or information associated with observations [5, 6]. Preattentive saliency captures what the brain subconsciously finds informative before conscious, or attentive, processing occurs. An unexpected or surprising observation is considered salient in this context. However, when applied to forecasting, the idea of surprisal or informativeness must be linked to the future outcome. Consequently, we propose that a timestep that alters an observer's certainty about the future is surprising, and therefore, salient. Crucially, our unifying 'bottom-up' perspective treats a forecasting model like a human observer, providing a principled definition of saliency that is not arbitrarily tied to task-specific error metrics. In contrast, the 'top-down' or task-specific notions of saliency common in post hoc explainable artificial intelligence (XAI) literature suffer from several drawbacks. Computed saliency maps may not measure the intended saliency, and even be independent of both the model and data generating process [7-9]. Moreover, what constitutes a good explanation is subject to the biases, intuition, or the visual assessment of the human observer [7, 10]; a phenomenon we refer to as the interpretation being in the eye of the beholder. Finally, as Barredo Arrieta et al. [11, Sec. 5.3] note, "there is absolutely no consistency behind what is known as saliency maps, salient masks, heatmaps, neuron activations, attribution, and other approaches alike."
To the best of our knowledge, no existing work addresses the specific task of obtaining post hoc model-agnostic explanations for probabilistic forecasts. Existing XAI methods for time-series data have predominantly focused on sequence classification, as we discuss in Section 2 and Appendix A. For regression, instead of post hoc explainability, researchers have emphasized interpretability by design [11] or intrinsic interpretability [12], where interpretability stems from the simple structure of models or coefficients of predefined basis functions [13, 14]. Against this backdrop, we present the following key contributions:
- Conceptual Grounding: We establish the conceptual foundation for linking saliency-based explanations with counterfactual reasoning. We draw upon insights from Miller's [10] work on explanations in artificial intelligence, highlighting the contrastive nature of explanations (Section 3).
- Information-Theoretic Framework: We extend Loog's [5] framework of bottom-up preattentive saliency to the domain of probabilistic forecasting. Specifically, we introduce a novel expression of saliency based on the differential entropy of the predicted future distribution, providing a closed-form solution for commonly used density functions in the literature (Section 4).
- Empirical Validation: We empirically evaluate our framework using synthetic and real-world data. In the synthetic setting, we achieve full accuracy in retrieving salient timesteps with known ground truth saliency. In real-world scenarios without ground truth saliency, we demonstrate the utility of our framework in explaining forecasts of social nonverbal behavior and vehicle trajectories, showcasing its effectiveness in complex and dynamic contexts (Section 5).
XAI Techniques for Time-Series Data. The taxonomy commonly used for explainability methods categorizes techniques based on three criteria: (i) intrinsic or post hoc, (ii) model-specific or model-agnostic, and (iii) local or global [12]. In the context of time-series regression, existing techniques predominantly focus on non-probabilistic settings and fall into the category of intrinsic and model-specific approaches. These include: (i) incorporating inductive biases through internal basis functions [14] (also extended to the probabilistic setting [13]), (ii) utilizing self-attention mechanisms in the model [15], and (iii) adapting saliency maps from computer vision to measure the contribution of features to the final forecast [16, 17]. For a comprehensive review of XAI methods across domains and time-series tasks, please refer to Appendix A.
Saliency-Based Explanations and Drawbacks. Saliency maps gained popularity as post hoc explanation tools for image classification [16, 18]. However, the lack of consistency in defining saliency has led to diverse interpretations, including occlusion sensitivity, gradient-based attribution heatmaps, and neuron activations [11, 12]. Nevertheless, these maps are typically computed by perturbing different parts of the input and observing the resulting change in the prediction error or output class. Several issues arise with the current use of saliency maps as explanations: (i) the
feature-level manipulations used for saliency maps may distort the sample in ways that deviate from the real-world data manifold and destroy semantics [7-9]; (ii) given the arbitrary definitions, evaluating saliency maps becomes challenging and is subject to observer biases [12, Sec.10.3.2], which can lead to maps appearing correct even when they are insensitive to the model and data [7]; (iii) for forecasting, Pan et al.'s [17] notion of saliency based on the error between the point prediction and ground truth future is arbitrary and relies on ground truths unavailability during testing; and (iv) the saliency map is explicitly retrained for a single observed-future sequence, failing to capture salient patterns across similar observed sequences that result in divergent but plausible futures [17].
Model-Agnostic Techniques. The SHAP framework, which integrates ideas from Shapley Values, LIME, LRP, and DeepLIFT, has gained popularity as a model-agnostic approach [19]. However, adapting these techniques to time-series tasks poses several challenges. Firstly, the Shapley methods rely on functions with real-valued codomains, such as a regression function  $f_{x}$  [19, see Eq. 4, 8], while our focus is on probabilistic models that output the distribution  $p_{Y|X}$  instead of some  $y = f_{x}(\cdot)$  to handle future uncertainty. Adapting these methods to deal with full predicted distributions is nontrivial. Similarly, gradient-based approaches compute gradients with respect to a single output instead of a full distribution. Secondly, these methods provide feature importance measures for a single output, whereas in time-series analysis, we are interested in identifying the importance of an observed timestep for an entire future sequence. That is, the joint consideration of the entire future sequence when computing input importance measures is challenging. As Pan et al. [17] note, in evaluating single-time predictions, these methods "ignore crucial temporal information and are insufficient for forecasting interpretation". Finally, similar to perturbation-based saliency methods, the sampling of features from arbitrary background samples in methods like Shapley/SHAP can lead to Frankenstein Monster instances [12, Sec. 9.5.3.3] that may not be valid samples on the data manifold. This undermines the semantics of the data, particularly in scenarios like motion trajectories, where randomly replacing features can result in physically impossible or glitchy motions.
3 Conceptual Grounding: Linking Saliency-Based Explanations to Counterfactual Reasoning
Given the challenges in XAI where speculations are often presented in the guise of explanations [20], we argue for grounding the concept of explanation within established frameworks of how humans define, generate, and present explanations. Turning to research in philosophy, psychology, and cognitive science, Miller [10] emphasized the importance of causality in explanatory questions. Drawing upon Pearl and Mackenzie's Ladder of Causation [21], he proposed the following categorization:
- Associative (What?): Reason about which unobserved events could have occurred given the observed events.
- Interventionist (How?:) Simulate a change in the situation to see if the event still happens.
- Counterfactual (Why?:) Simulate alternative causes to see whether the event still happens.
To apply Miller's framework in the context of forecasting, one needs to define the abstract notions of 'events' and 'causes'. Consider a model  $\mathbf{M}$  that predicts features over a future window  $t_{\mathrm{fut}}$  by observing features over a window  $t_{\mathrm{obs}}$ . We assert that the intrinsic interpretability methods involving inductive biases [13, 14] and attention mechanisms [15], fall under associative reasoning. These methods assess the (unobserved) importance of features over  $t_{\mathrm{obs}}$  using model parameters or attention coefficients based on a single prediction from  $\mathbf{M}$  (the 'event') for a fixed  $t_{\mathrm{fut}}$  and single  $t_{\mathrm{obs}}$ . In contrast, we posit that the perturbation-based saliency methods can support counterfactual reasoning. They perturb different parts of the input over  $t_{\mathrm{obs}}$  simulating alternative 'causes' from  $\mathbf{M}$ 's perspective, and observe the effect on an error metric (the 'event'). However, the current application of these methods encounters issues outlined in Section 2.
To address the aforementioned challenges, we employ a unifying information-theoretic concept of bottom-up saliency grounded in preattentive human cognition [5, 6] as discussed in Section 1. Concretely, we propose the following implication that links this saliency to counterfactual reasoning:
observing the features at a timestep  $t \in \pmb{t}_{\mathrm{obs}}$  results in a change in  $\mathbf{M}$ 's information about the future  $I_{\mathrm{fut}}$  over the given  $\pmb{t}_{\mathrm{fut}} \Rightarrow t$  is salient. (1)
Note that the antecedent (on the left of the implication) is a counterfactual statement. We formally express the implication using causal graphs [22] in Figure 1. The generic graph expresses relationships between the random variables prior to training the forecasting model  $M$ . The exogenous variable  $\epsilon_{M}$  captures the randomness in the training process and modeling choices, including the distribution family for representing the forecasts. The exogenous variable  $\epsilon_{H}$  captures the randomness in the human observer's choice of observed and future windows to examine the model. Our central idea is to evaluate the information in the model's
(a) Generic Graph
Figure 1: Causal Graphs for Explaining Forecasts
(b) Modified Graph
predicted distribution denoted by  $I_{\mathrm{fut}}$ . Specifically, we propose posing the following counterfactual question: What information would  $\mathbf{M}$  have about the future over a fixed  $t_{\mathrm{fut}}$  if it observed the features over  $t_{\mathrm{obs}}$ ? The modified graph for evaluating this question is in Figure 1b. Once the model  $\mathbf{M}$  has been trained and the windows  $t_{\mathrm{obs}}$  and  $t_{\mathrm{fut}}$  have been chosen, the effect of the exogenous variables on the variable  $I_{\mathrm{fut}}$  disappears. This allows us to evaluate the change in the information about the future in response to different realizations of  $t_{\mathrm{obs}}$  and  $t_{\mathrm{fut}}$ , facilitating counterfactual analysis. Note that we assume the modified graph is already available, as our focus is on the explanation phase. While the procedure starting from training the model in the generic graph implicitly follows Pearl's abduct-action-prediction process [22, p. 207], estimating the distribution over the exogenous variables from the abduction step is conceptually not applicable in this setting.
Note that these graphs are not meant to describe relationships between random variables in the data for a specific hypothesis, as is typical in causal inference literature: for instance, the effect of [rotating toward the speaker] on [turn changes] in conversations. Rather, they describe the process of a human generating contrastive explanations for a given pretrained forecasting model  $M$  —irrespective of whether or not it is the true model—for some sequences in the data ( $t_{\mathrm{obs}}$ ,  $t_{\mathrm{fut}}$ ). Further, the notion of counterfactuals, as used within the context of contrastive explanations, is also distinct from that in causal inference. As Miller [10, Sec. 2.3] points out, "it is important to note that this is not the same counterfactual that one refers to when determining causality. For causality, the counterfactuals are hypothetical 'noncauses'… whereas in contrastive explanation, counterfactuals are hypothetical outcomes." Miller's point is that why explanations entail contrastive reasoning which involves comparing 'outcomes' in response to alternate 'causes'. In our work, this 'outcome' relates to the information in  $M$ 's predicted distribution, the what-if question being "would the information in  $M$ 's prediction for the window  $t_{\mathrm{fut}}$  change if it had observed features over a different (contrastive)  $t_{\mathrm{obs}}$ ?". Contrast this to associative reasoning which uses features from a single  $t_{\mathrm{obs}}$  to generate the attribution map. A longer discussion is in Appendix E.
Considering the information in forecasts in implication 1 links counterfactual reasoning to a more principled notion of saliency than has been used in XAI literature. Note that the implication entails that for the antecedent to be true  $t$  must be salient. However, knowing the antecedent is false is not sufficient to conclude that  $t$  is not salient, i.e. there can be other notions of saliency that make  $t$  salient. However, for less speculative evaluation, it is crucial that we use a unifying notion of saliency that is not arbitrarily defined based on task-specific error metrics or model gradients [11, 16-18]. Preattentive saliency, as we formalize in Section 4.1, is based on what is informative for the brain before conscious processing, making it more objective in nature.
Our framework addresses all the concerns associated with saliency-based approaches described in Section 2: (i) the counterfactuals in our framework are real observed features rather than random input perturbations, preserving the semantics of the real-world data; (ii) our use of information-theoretic preattentive saliency is principled and objective; (iii) our framework allows for saliency computation on unseen test data where the ground-truth future is unavailable, relying solely on the underlying model; and (iv) our approach considers the distribution over possible futures for a single input, capturing the structural predictive relationships between features across multiple samples. An additional advantage of our framework is that it does not require any training to compute the saliency and can be applied to any model that outputs a distribution over futures.
4 Methodology: Closed-Form Saliency for Probabilistic Forecasting
4.1 Preliminary: Information Theoretic Preattentive Saliency
Loog [5] developed a general closed-form expression for saliency based on computational visual perception that unifies different definitions of saliency encountered in the literature. The framework was illustrated on images and employed a surprisal-based operational definition of bottom-up attention. In this framework, an image is represented by a feature mapping function  $\phi$  that relates each location in the image to a set of features. The saliency of a location  $x$  is determined by the information or surprise associated with its corresponding feature vector  $\phi(x)$  compared to other feature vectors extracted from the same image. The saliency measure is defined as follows:
$$
S (x) > S \left(x ^ {\prime}\right) \Longleftrightarrow - \log p _ {\Phi} (\phi (x)) > - \log p _ {\Phi} (\phi \left(x ^ {\prime}\right)). \tag {2}
$$
Here,  $p_{\Phi}$  represents the probability density function over all feature vectors, while  $p_{X}$  captures any prior knowledge that influences the saliency of different image locations.
Contrary to approaches that determine saliency maps through an explicit data-driven density estimation [16, 18, 23-25], once the feature mapping  $\phi$  is fixed, a closed-form expression for saliency can be obtained. The information content -  $\log p_{\Phi}$  can be obtained from  $\log p_X$  through a simple change of variables [26] from  $x$  to  $\phi (x)$ . The saliency  $S(x)$  is then given by the expression:
$$
- \log p _ {\Phi} (\phi (x)) = - \log p _ {X} (x) + \frac {1}{2} \log \det  \left(J _ {\phi} ^ {t} (x) J _ {\phi} (x)\right), \tag {3}
$$
where  $J_{\phi}$  denotes the Jacobian matrix of  $\phi$ , and  ${}_{-}^{t}$  indicates matrix transposition. Since a monotonic transformation does not essentially alter the map, Loog [5] simplifies the saliency map definition to
$$
S (x) := \det  \left(J _ {\phi} ^ {t} (x) J _ {\phi} (x)\right), \tag {4}
$$
This formulation of saliency offers several advantages. It provides a principled and objective measure that captures the informativeness of features for human perception. Moreover, the saliency computation is purely local to an image, making it independent of previously observed data.
4.2 Defining  $\phi$  in Terms of the Uncertainty over the Future Window  $t_{\mathrm{fut}}$
Let  $\pmb{t}_{\mathrm{obs}} \coloneqq [o1, o2, \dots, oT]$  represent a window of consecutively increasing observed timesteps, and  $\pmb{t}_{\mathrm{fut}} \coloneqq [f1, f2, \dots, fT]$  denote an unobserved future time window, where  $f1 > oT$ . Consider a set of  $n$  interacting agents, and let  $\pmb{X} \coloneqq [\pmb{b}_t^i; t \in \pmb{t}_{\mathrm{obs}}]_{i=1}^n$  and  $\pmb{Y} \coloneqq [\pmb{b}_t^i; t \in \pmb{t}_{\mathrm{fut}}]_{i=1}^n$  represent their features over  $\pmb{t}_{\mathrm{obs}}$  and  $\pmb{t}_{\mathrm{fut}}$  respectively. Here,  $\pmb{b}_t^i$  captures multimodal features from agent  $i$  at time  $t$ . The forecasting task is to predict the density  $p_{Y|X}$ . Given a model that outputs  $p_{Y|X}$ , our task is to compute the saliency  $S(\pmb{t}_{\mathrm{obs}})$  of an observed  $\pmb{t}_{\mathrm{obs}}$  with respect to a fixed choice of  $\pmb{t}_{\mathrm{fut}}$ .
To extend Loog's [5] framework to forecasting settings, we need to choose an appropriate  $\phi$ . We formalize the implication in Equation 1 and map  $t_{\mathrm{obs}}$  to the differential entropy of the model's predicted future distribution over  $t_{\mathrm{fut}}$ . Specifically, we define  $\phi : t_{\mathrm{obs}} \mapsto h(Y|X = X)$ , where the conditional differential entropy of  $Y$  given  $\{X = X\}$  is defined as
$$
h (Y | X = \boldsymbol {X}) := - \int p _ {Y | X} (\boldsymbol {Y} | \boldsymbol {X}) \log p _ {Y | X} (\boldsymbol {Y} | \boldsymbol {X}) d \boldsymbol {Y}. \tag {5}
$$
Our framework is summarized in Algorithm 1. Consider that a domain expert selects a specific  $t_{\mathrm{fut}}$  corresponding to a high-order semantic behavior they wish to analyze. This could be a speaking-turn change [27, 28] an interaction termination [29, 30], or a synchronous behavior event [31]. Given an underlying forecasting model  $M$  and look-back period before  $t_{\mathrm{fut}}$ , we compute  $h(Y|X = X)$  for different observed multivariate features  $X$  corresponding to different locations of a sliding  $t_{\mathrm{obs}}$ . The computed differential entropy values are then inserted into Equation 4 to obtain the saliency of different  $t_{\mathrm{obs}}$  locations towards the future over the chosen  $t_{\mathrm{fut}}$ . In Appendix B we discuss other favorable properties of differential entropy that make it a suitable choice as  $\phi$ .
Explanation Using the Running Example. Within our running example from Section 1,  $t_{\mathrm{fut}}$  corresponds to the two pedestrians passing each other while avoiding collision. In this example, let us assume  $M$ 's training data contains examples of pedestrians passing others to both the left and the
Algorithm 1 Temporal Saliency in Probabilistic Forecasting
Input: The probability density function  $p_{Y|X}$ , a fixed  $t_{\mathrm{fut}}$  of interest, a sequence of  $m$  preceding observed windows  $O = [t_{\mathrm{obs}}^1, \ldots, t_{\mathrm{obs}}^m]$ , and the behavioral features  $X^j$  for every  $t_{\mathrm{obs}}^j$
Output: The saliency map  $S(O)$  over the observed windows
1: for each  $t_{\mathrm{obs}}^j \in O$  do
2: Compute the feature mapping  $\phi (\pmb{t}_{\mathrm{obs}}^j)\gets h(Y|X = \pmb{X}^j)$
3: end for
4: Compute saliency  $S(\pmb{t}_{\mathrm{obs}}) \gets \operatorname*{det}(J_{\phi}^{t}(\pmb{t}_{\mathrm{obs}}) J_{\phi}(\pmb{t}_{\mathrm{obs}}))$