Title: Why Did This Model Forecast This Future? Information-Theoretic Saliency for Counterfactual Explanations of Probabilistic Regression Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Probabilistic Forecasting

Saliency

Explainability

XAI

Probabilistic Regression
Abstract: We propose a post hoc saliency-based explanation framework for counterfactual reasoning in probabilistic multivariate time-series forecasting (regression) settings. Building upon Miller's framework of explanations derived from research in multiple social science disciplines, we establish a conceptual link between counterfactual reasoning and saliency-based explanation techniques. To address the lack of a principled notion of saliency, we leverage a unifying definition of information-theoretic saliency grounded in preattentive human visual cognition and extend it to forecasting settings. Specifically, we obtain a closed-form expression for commonly used density functions to identify which observed timesteps appear salient to an underlying model in making its probabilistic forecasts. We empirically validate our framework in a principled manner using synthetic data to establish ground-truth saliency that is unavailable for real-world data. Finally, using real-world data and forecasting models, we demonstrate how our framework can assist domain experts in forming new data-driven hypotheses about the causal relationships between features in the wild.

Why Did This Model Forecast This Future? Information-Theoretic Saliency for Counterfactual Explanations of Probabilistic Regression Models

Abstract

We propose a post hoc saliency-based explanation framework for counterfactual reasoning in probabilistic multivariate time-series forecasting (regression) settings. Building upon Miller's framework of explanations derived from research in multiple social science disciplines, we establish a conceptual link between counterfactual reasoning and saliency-based explanation techniques. To address the lack of a principled notion of saliency, we leverage a unifying definition of informationtheoretic saliency grounded in preattentive human visual cognition and extend it to forecasting settings. Specifically, we obtain a closed-form expression for commonly used density functions to identify which observed timesteps appear salient to an underlying model in making its probabilistic forecasts. We empirically validate our framework in a principled manner using synthetic data to establish ground-truth saliency that is unavailable for real-world data. Finally, using real-world data and forecasting models, we demonstrate how our framework can assist domain experts in forming new data-driven hypotheses about the causal relationships between features in the wild.

1 Introduction

As we go about our daily lives, engaging in conversations, walking down the street, or driving a car, we rely on our ability to anticipate the future actions and states of those around us [1, 2]. However, the numerous unknowns, such as hidden thoughts and intentions, make our predictions of the future inherently uncertain [2]. To reflect this uncertainty, several machine learning methods in such settings forecast a full distribution over plausible futures, rather than making a single point prediction [3, 4]. Identifying the factors that influence such a model's forecasts is particularly useful for domain experts seeking to understand the causal relationships guiding complex real-world behaviors, especially in situations where the future is uncertain. In this work, we introduce and address a novel research question toward counterfactual reasoning in multivariate probabilistic regression settings: how can we identify the observed timesteps that are salient for a model's probabilistic forecasts over a specific future window? Specifically, we introduce the first post hoc, model-agnostic, saliency-based explanation framework for probabilistic time-series forecasting.

We begin with a fundamental observation about human social cognition: we are averse to uncertainty and strive to minimize it [2]. Consider the scenario where a pedestrian is approaching you on the street. Initially, there is uncertainty about which direction each of you will take to avoid a collision. As one of you changes direction, the other observes and takes the opposite direction, ultimately avoiding a collision. Concretely, the thesis of this work is to formalize the following notion of saliency: the timestep that changes the uncertainty of a predicted future is salient toward predicting that future.

For instance, in the aforementioned scenario, we posit that the moment when one pedestrian changes direction is salient toward forecasting the future trajectories of the pedestrians.

Our notion of saliency is grounded in preattentive human cognition and related to the concept of surprisal or information associated with observations [5, 6]. Preattentive saliency captures what the brain subconsciously finds informative before conscious, or attentive, processing occurs. An unexpected or surprising observation is considered salient in this context. However, when applied to forecasting, the idea of surprisal or informativeness must be linked to the future outcome . Consequently, we propose that a timestep that alters an observer's certainty about the future is surprising, and therefore, salient. Crucially, our unifying 'bottom-up' perspective treats a forecasting model like a human observer, providing a principled definition of saliency that is not arbitrarily tied to task-specific error metrics. In contrast, the 'top-down' or task-specific notions of saliency common in post hoc explainable artificial intelligence (XAI) literature suffer from several drawbacks. Computed saliency maps may not measure the intended saliency, and even be independent of both the model and data generating process [7-9]. Moreover, what constitutes a good explanation is subject to the biases, intuition, or the visual assessment of the human observer [7, 10]; a phenomenon we refer to as the interpretation being in the eye of the beholder . Finally, as Barredo Arrieta et al. [11, Sec. 5.3] note, 'there is absolutely no consistency behind what is known as saliency maps, salient masks, heatmaps, neuron activations, attribution, and other approaches alike.'

To the best of our knowledge, no existing work addresses the specific task of obtaining post hoc model-agnostic explanations for probabilistic forecasts. Existing XAI methods for time-series data have predominantly focused on sequence classification, as we discuss in Section 2 and Appendix A. For regression, instead of post hoc explainability, researchers have emphasized interpretability by design [11] or intrinsic interpretability [12], where interpretability stems from the simple structure of models or coefficients of predefined basis functions [13, 14]. Against this backdrop, we present the following key contributions:

Conceptual Grounding: We establish the conceptual foundation for linking saliency-based explanations with counterfactual reasoning. We draw upon insights from Miller's [10] work on explanations in artificial intelligence, highlighting the contrastive nature of explanations (Section 3).

Information-Theoretic Framework: We extend Loog's [5] framework of bottom-up preattentive saliency to the domain of probabilistic forecasting. Specifically, we introduce a novel expression of saliency based on the differential entropy of the predicted future distribution, providing a closed-form solution for commonly used density functions in the literature (Section 4).

Empirical Validation: We empirically evaluate our framework using synthetic and real-world data. In the synthetic setting, we achieve full accuracy in retrieving salient timesteps with known ground truth saliency. In real-world scenarios without ground truth saliency, we demonstrate the utility of our framework in explaining forecasts of social nonverbal behavior and vehicle trajectories, showcasing its effectiveness in complex and dynamic contexts (Section 5).

2 Related Work

XAI Techniques for Time-Series Data. The taxonomy commonly used for explainability methods categorizes techniques based on three criteria: (i) intrinsic or post hoc, (ii) model-specific or modelagnostic, and (iii) local or global [12]. In the context of time-series regression, existing techniques predominantly focus on non-probabilistic settings and fall into the category of intrinsic and modelspecific approaches. These include: (i) incorporating inductive biases through internal basis functions [14] (also extended to the probabilistic setting [13]), (ii) utilizing self-attention mechanisms in the model [15], and (iii) adapting saliency maps from computer vision to measure the contribution of features to the final forecast [16, 17]. For a comprehensive review of XAI methods across domains and time-series tasks, please refer to Appendix A.

Saliency-Based Explanations and Drawbacks. Saliency maps gained popularity as post hoc explanation tools for image classification [16, 18]. However, the lack of consistency in defining saliency has led to diverse interpretations, including occlusion sensitivity, gradient-based attribution heatmaps, and neuron activations [11, 12]. Nevertheless, these maps are typically computed by perturbing different parts of the input and observing the resulting change in the prediction error or output class. Several issues arise with the current use of saliency maps as explanations: (i) the

feature-level manipulations used for saliency maps may distort the sample in ways that deviate from the real-world data manifold and destroy semantics [7-9]; (ii) given the arbitrary definitions, evaluating saliency maps becomes challenging and is subject to observer biases [12, Sec.10.3.2], which can lead to maps appearing correct even when they are insensitive to the model and data [7]; (iii) for forecasting, Pan et al.'s [17] notion of saliency based on the error between the point prediction and ground truth future is arbitrary and relies on ground truths unavialable during testing; and (iv) the saliency map is explicitly retrained for a single observed-future sequence, failing to capture salient patterns across similar observed sequences that result in divergent but plausible futures [17].

Model-Agnostic Techniques. The SHAP framework, which integrates ideas from Shapley Values, LIME, LRP, and DeepLIFT, has gained popularity as a model-agnostic approach [19]. However, adapting these techniques to time-series tasks poses several challenges. Firstly, the Shapley methods rely on functions with real-valued codomains, such as a regression function f x [19, see Eq. 4, 8], while our focus is on probabilistic models that output the distribution p Y | X instead of some y = f x ( · ) to handle future uncertainty. Adapting these methods to deal with full predicted distributions is nontrivial. Similarly, gradient-based approaches compute gradients with respect to a single output instead of a full distribution. Secondly, these methods provide feature importance measures for a single output, whereas in time-series analysis, we are interested in identifying the importance of an observed timestep for an entire future sequence . That is, the joint consideration of the entire future sequence when computing input importance measures is challenging. As Pan et al. [17] note, in evaluating single-time predictions, these methods 'ignore crucial temporal information and are insufficient for forecasting interpretation'. Finally, similar to perturbation-based saliency methods, the sampling of features from arbitrary background samples in methods like Shapley/SHAP can lead to Frankenstein Monster instances [12, Sec. 9.5.3.3] that may not be valid samples on the data manifold. This undermines the semantics of the data, particularly in scenarios like motion trajectories, where randomly replacing features can result in physically impossible or glitchy motions.

3 Conceptual Grounding: Linking Saliency-Based Explanations to Counterfactual Reasoning

Given the challenges in XAI where speculations are often presented in the guise of explanations [20], we argue for grounding the concept of explanation within established frameworks of how humans define, generate, and present explanations. Turning to research in philosophy, psychology, and cognitive science, Miller [10] emphasized the importance of causality in explanatory questions. Drawing upon Pearl and Mackenzie's Ladder of Causation [21], he proposed the following categorization:

Associative ( What? ): Reason about which unobserved events could have occurred given the observed events.

Interventionist ( How? ): Simulate a change in the situation to see if the event still happens.

Counterfactual ( Why? ): Simulate alternative causes to see whether the event still happens.

To apply Miller's framework in the context of forecasting, one needs to define the abstract notions of 'events' and 'causes'. Consider a model M that predicts features over a future window t fut by observing features over a window t obs . We assert that the intrinsic interpretability methods involving inductive biases [13, 14] and attention mechanisms [15], fall under associative reasoning. These methods assess the (unobserved) importance of features over t obs using model parameters or attention coefficients based on a single prediction from M (the 'event') for a fixed t fut and single t obs . In contrast, we posit that the perturbation-based saliency methods can support counterfactual reasoning. They perturb different parts of the input over t obs simulating alternative 'causes' from M 's perspective, and observe the effect on an error metric (the 'event'). However, the current application of these methods encounters issues outlined in Section 2.

To address the aforementioned challenges, we employ a unifying information-theoretic concept of bottom-up saliency grounded in preattentive human cognition [5, 6] as discussed in Section 1. Concretely, we propose the following implication that links this saliency to counterfactual reasoning:

observing the features at a timestep t ∈ t obs results in a change in M 's information about the future I fut over the given t fut = ⇒ t is salient. (1)

Note that the antecedent (on the left of the implication) is a counterfactual statement. We formally express the implication using causal graphs [22] in Figure 1. The generic graph expresses relationships between the random variables prior to training the forecasting model M . The exogenous variable ϵ M captures the randomness in the training process and modeling choices, including the distribution family for representing the forecasts. The exogenous variable ϵ H captures the randomness in the human observer's choice of observed and future windows to examine the model. Our central idea is to evaluate the information in the model's

Data

M

ϵ

M

T

obs

T

fut

ϵ

H

I

fut

M

=

M

T

obs

=

t

obs

T

fut

=

t

fut

I

fut

= I

(b) Modified Graph

predicted distribution denoted by I fut . Specifically, we propose posing the following counterfactual question: What information would M have about the future over a fixed t fut if it observed the features over t obs ? The modified graph for evaluating this question is in Figure 1b. Once the model M has been trained and the windows t obs and t fut have been chosen, the effect of the exogenous variables on the variable I fut disappears. This allows us to evaluate the change in the information about the future in response to different realizations of t obs and t fut , facilitating counterfactual analysis. Note that we assume the modified graph is already available, as our focus is on the explanation phase. While the procedure starting from training the model in the generic graph implicitly follows Pearl's abduct-action-prediction process [22, p. 207], estimating the distribution over the exogenous variables from the abduction step is conceptually not applicable in this setting.

Note that these graphs are not meant to describe relationships between random variables in the data for a specific hypothesis, as is typical in causal inference literature: for instance, the effect of [rotating toward the speaker] on [turn changes] in conversations. Rather, they describe the process of a human generating contrastive explanations for a given pretrained forecasting model M -irrespective of whether or not it is the true model-for some sequences in the data ( t obs , t fut ). Further, the notion of counterfactuals, as used within the context of contrastive explanations, is also distinct from that in causal inference. As Miller [10, Sec. 2.3] points out, 'it is important to note that this is not the same counterfactual that one refers to when determining causality. For causality, the counterfactuals are hypothetical 'noncauses'. . . whereas in contrastive explanation, counterfactuals are hypothetical outcomes.' Miller's point is that why explanations entail contrastive reasoning which involves comparing 'outcomes' in response to alternate 'causes'. In our work, this 'outcome' relates to the information in M 's predicted distribution, the what-if question being 'would the information in M 's prediction for the window t fut change if it had observed features over a different (contrastive) t obs ?'. Contrast this to associative reasoning which uses features from a single t obs to generate the attribution map. A longer discussion is in Appendix E.

Considering the information in forecasts in implication 1 links counterfactual reasoning to a more principled notion of saliency than has been used in XAI literature. Note that the implication entails that for the antecedent to be true t must be salient. However, knowing the antecedent is false is not sufficient to conclude that t is not salient, i.e. there can be other notions of saliency that make t salient. However, for less speculative evaluation, it is crucial that we use a unifying notion of saliency that is not arbitrarily defined based on task-specific error metrics or model gradients [11, 16-18]. Preattentive saliency, as we formalize in Section 4.1, is based on what is informative for the brain before conscious processing, making it more objective in nature.

Our framework addresses all the concerns associated with saliency-based approaches described in Section 2: (i) the counterfactuals in our framework are real observed features rather than random input perturbations, preserving the semantics of the real-world data; (ii) our use of informationtheoretic preattentive saliency is principled and objective; (iii) our framework allows for saliency computation on unseen test data where the ground-truth future is unavailable, relying solely on the underlying model; and (iv) our approach considers the distribution over possible futures for a single input, capturing the structural predictive relationships between features across multiple samples. An additional advantage of our framework is that it does not require any training to compute the saliency and can be applied to any model that outputs a distribution over futures.

4 Methodology: Closed-Form Saliency for Probabilistic Forecasting

4.1 Preliminary: Information Theoretic Preattentive Saliency

Loog [5] developed a general closed-form expression for saliency based on computational visual perception that unifies different definitions of saliency encountered in the literature. The framework was illustrated on images and employed a surprisal-based operational definition of bottom-up attention. In this framework, an image is represented by a feature mapping function ϕ that relates each location in the image to a set of features. The saliency of a location x is determined by the information or surprise associated with its corresponding feature vector ϕ ( x ) compared to other feature vectors extracted from the same image. The saliency measure is defined as follows:

Here, p Φ represents the probability density function over all feature vectors, while p X captures any prior knowledge that influences the saliency of different image locations.

Contrary to approaches that determine saliency maps through an explicit data-driven density estimation [16, 18, 23-25], once the feature mapping ϕ is fixed, a closed-form expression for saliency can be obtained. The information content -log p Φ can be obtained from log p X through a simple change of variables [26] from x to ϕ ( x ) . The saliency S ( x ) is then given by the expression:

where J ϕ denotes the Jacobian matrix of ϕ , and _ t indicates matrix transposition. Since a monotonic transformation does not essentially alter the map, Loog [5] simplifies the saliency map definition to

This formulation of saliency offers several advantages. It provides a principled and objective measure that captures the informativeness of features for human perception. Moreover, the saliency computation is purely local to an image, making it independent of previously observed data.

4.2 Defining ϕ in Terms of the Uncertainty over the Future Window t fut

Let t obs := [ o 1 , o 2 , ..., oT ] represent a window of consecutively increasing observed timesteps, and t fut := [ f 1 , f 2 , ..., f T ] denote an unobserved future time window, where f 1 > oT . Consider a set of n interacting agents, and let X := [ b i t ; t ∈ t obs ] n i =1 and Y := [ b i t ; t ∈ t fut ] n i =1 represent their features over t obs and t fut respectively. Here, b i t captures multimodal features from agent i at time t . The forecasting task is to predict the density p Y | X . Given a model that outputs p Y | X , our task is to compute the saliency S ( t obs ) of an observed t obs with respect to a fixed choice of t fut .

To extend Loog's [5] framework to forecasting settings, we need to choose an appropriate ϕ . We formalize the implication in Equation 1 and map t obs to the differential entropy of the model's predicted future distribution over t fut . Specifically, we define ϕ : t obs ↦→ h ( Y | X = X ) , where the conditional differential entropy of Y given { X = X } is defined as

Our framework is summarized in Algorithm 1. Consider that a domain expert selects a specific t fut corresponding to a high-order semantic behavior they wish to analyze. This could be a speaking-turn change [27, 28] an interaction termination [29, 30], or a synchronous behavior event [31]. Given an underlying forecasting model M and look-back period before t fut , we compute h ( Y | X = X ) for different observed multivariate features X corresponding to different locations of a sliding t obs . The computed differential entropy values are then inserted into Equation 4 to obtain the saliency of different t obs locations towards the future over the chosen t fut . In Appendix B we discuss other favorable properties of differential entropy that make it a suitable choice as ϕ .

Explanation Using the Running Example. Within our running example from Section 1, t fut corresponds to the two pedestrians passing each other while avoiding collision. In this example, let us assume M 's training data contains examples of pedestrians passing others to both the left and the

Algorithm 1 Temporal Saliency in Probabilistic Forecasting

Input: The probability density function p Y | X , a fixed t fut of interest, a sequence of m preceding observed windows O = [ t 1 obs , . . . , t m obs ] , and the behavioral features X j for every t j

obs Output: The saliency map S ( O ) over the observed windows 1: for each t j obs ∈ O do 2: Compute the feature mapping ϕ ( t j obs ) ← h ( Y | X = X j ) 3: end for 4: Compute saliency S ( t obs ) ← det( J t ϕ ( t obs ) J ϕ ( t obs ))

right. Consequently, for a t obs containing the pedestrians approaching each other in a straight line, the predicted distribution p Y | X over t fut encapsulates both possibilities of each pedestrian passing to the left as well as the right of the other. So the entropy h ( Y | X = X ) is high for this t obs . Only once M is fed as input with the trajectories from the t obs containing the pedestrians choosing one of the two directions to pass, the predicted p Y | X is certain in terms of the pedestrians continuing along the chosen direction. (Note that in this case, we assume M has been trained by maximizing likelihood over the dataset containing only these two direction changes for avoiding collision.) Consequently, the entropy h ( Y | X = X ) drops only once this moment of the pedestrians committing to a direction is seen by the model and would be considered salient for our algorithm.

4.3 Computing h ( Y | X = X )

Typically, the density p Y | X is modeled as a multivariate Gaussian distribution [4, 32-34]. When the decoding of the future is non-autoregressive, the parameters of the distributions for all t ∈ t fut are estimated at once, and the differential entropy has a closed-form expression, given by (see Cover and Thomas [35, Theorem 8.4.1])

A common choice is to set K to be diagonal, i.e. the predicted distribution is factorized over agents and features. In this case, we can simply sum the log of the individual variances to obtain the feature mapping ϕ . Note that from Equation 6, for a multivariate Gaussian distribution, the differential entropy only depends on the covariance, or the spread of the distribution, aligning with the notion of differential entropy as a measure of total uncertainty. (See [35, Tab. 17.1; 36] for closed-form expressions for a large number of commonly employed probability density functions.)

In cases where probabilistic autoregressive decoders are used [4, 33, 37, 38], we do not have access to the full joint distribution p Y f 1 ,...,Y fT | X for the timesteps in t fut . This is because inferring the density function p Y | X often involves sampling: a specific sample ̂ Y t is taken from the predicted density at each t ∈ t fut , and passed back as input to the decoder for estimating the density at timestep t + 1 [37, 38]. Therefore, the density at t +1 depends on the randomness introduced in sampling ̂ Y t . Figure 2 illustrates the concept for two timesteps. Here, a single forecast would only output the shaded red distribution for Y 2 . In such cases, computing the joint entropy h ( Y 1 , Y 2 ) directly is challenging in the absence of the full joint distribution p Y 1 ,Y 2 .

To address this, we have two options. The simpler option is to redefine our feature-mapping as ϕ : t obs ↦→ ∑ t ∈ t fut h ( Y t | ̂ Y <t , X ) , i.e. we approximate the total uncertainty over the predicted sequence by summing the differential entropies of the individual densities estimated at

PYaIY(1E)

PYi,Y2

Y2

Y1

Yi

each timestep. Note that following the chain rule for differential entropy (see Cover and Thomas [35, Eq. 8.62]), the joint entropy can indeed be written as the sum of individual conditionals. However,

̸

And yet, training autoregressive decoders by maximizing likelihood actually assumes the inequality in Equation 7 to be approximately equal (see [39, Sec. 2; 40, Eq. 5]). The approximation relies on the observation that, for autoregressive decoding, the parameters of the predicted distribution for Y t are computed as a deterministic function of the decoder hidden state. That is, Y t is conditionally independent of Y <t given the hidden state of the decoder s t at timestep t . The underlying assumption is that for a well-trained decoder, s t encodes all relevant information from other timesteps to infer the distribution of Y t . So at inference, despite being a function of the single sample ̂ Y t -1 , the predicted distribution conditioned on s t provides a reasonable estimate of the uncertainty in Y t . This assumption allows us to again obtain a closed-form expression for the saliency map when each Y t is modeled using a density function with a known closed-form expression for differential entropy [35, Tab. 17.1; 36]. For the common choice of modeling Y t using a Gaussian mixture [37, 38], approximations that approach the true differential entropy can also be obtained efficiently [41-43] to directly compute the feature mapping ϕ .

The second option is to estimate h ( Y | X = X ) using sampling or other non-parametric approaches when analytical expressions or computationally efficient approximations are not available [44-47]. These sampling-based methods provide approximations that converge to the true entropy, although they may be computationally more expensive than parametric methods. Overall, the choice of modeling the future density and the approach for estimating the differential entropy depends on the specific scenario and the available resources.

5 Experiments

The common evaluation of saliency-based explanations relies on qualitative visual assessment, which is subjective and prone to observer biases [7, 11, 12]. Meanwhile, establishing a reliable ground truth for the salient relationship between the observed window t obs and the future window t fut is challenging in real-world data due to conflicting domain evidence on predictive relationships [28, 48]. Furthermore, fair validation of a model agnostic, post hoc method requires evaluating it independently of imperfections in the underlying forecasting model. To address these challenges we conduct two types of empirical evaluation: one using synthetic data to establish ground truth predictive saliency and validate the framework , and another to demonstrate empirical utility in real-world scenarios where perfect forecasts and ground truth saliency are unavailable.

No existing benchmarks or post hoc explanation frameworks exist for probabilistic time-series regression that meet the necessary requirements for a meaningful empirical comparison. Nevertheless, we provide results by adapting several explainability frameworks in our experiments. Specifically, we considered DeepSHAP and GradientSHAP [19], and IntegratedGradients and SmoothGrad [49]. It is important to note that we do not imply that these are fair comparisons; they are not (see Section 2). However, the comparisons are meant to characterize results from popular tools that practitioners are likely to use in the absence of our proposed framework. Implementation details for the following experiments and additional results for the real-world scenarios are in Appendices C and D, respectively.

5.1 Empirical Validation using Synthesized Ground Truth Saliency

Dataset. We simulate a group conversation that emulates real behavior patterns. Listeners typically focus on the speaker, while the speaker looks at different listeners [50]. Additionally, head gestures and gaze patterns predict the next speaker [51-54]. In our simplified simulation, the speaker rotates towards the center when speaking, and listeners nod to trigger a turn handover. We use real-valued quaternions to represent 3D head poses, commonly used for human motion and pose representation [4, 55, 56]. Following the notation in Section 4.2, b = [ q w , q x , q y , q z , ss ] where ss denotes binary speaking status. We simulate the turn changes to occur once clockwise and once anticlockwise. The ground truth salient timestep is when a listener initiates a head nod to trigger a turn handover, ensuring a certain future turn change. Figure 3 illustrates this mechanism. The code, dataset, and animated visualization are available in the Supplement.

-listenerhead-nodbegins

observed

obs

2

obs

possible futures

0.00

0.05

0.10

Value

Quaternion

qx

qy

20

15

Entropy

130

135

140

145

150

Timestep

0

10

20

Saliency

(a) Ours (Perfect forecast)

0.00

0.05

0.10

Value

Quaternion

qx

qy

2.30

2.25

Entropy

130

135

140

145

150

Timestep

0.0000

0.0005

Saliency

(b) Ours (Trained model)

0.00

0.05

0.10

Value

Quaternion

qx

qy

0.25

0.00

0.25

Attr. (mean)

130

135

140

145

150

Timestep

0.5

0.0

Attr. (std)

DeepSHAP

GradientSHAP

IntegratedGradients

SmoothGrad

(c) Attributions (Trained model)

Empirical Validation. To validate our framework in isolation, we assume a perfect forecasting model that predicts the true distribution over the possible future quaternion trajectories. The forecasting model focuses solely on low-level features and does not incorporate any high-order semantics of turn-taking. The saliency map generated by our framework, as shown in Figure 4a, accurately identifies the ground truth salient timesteps at frames 138 and 139 where the head nod begins. The saliency decreases once the nod is already in motion, indicating that it does not provide additional information about the future. This empirically validates our framework.

Introducing a Real Forecasting Model. We evaluate our framework using a real underlying forecasting model trained on synthetic data. We employ a Social Process model [4] for its ability to capture relative partner behavior and adapt to specific group dynamics. As shown in Figure 4b, our framework identifies the true salient timesteps with higher saliency values. Conversely, the attributions provided by other explainability frameworks in Figure 4c for the predicted mean and

Head Direction in Z-angle

150

100

50

0

50

Rotation (degrees)

0.0000

0.0001

Saliency

Saliency of observed t in

t

obs

towards future over

t

fut

0.25

0.00

0.25

SHAP (mean)

SHAP values of observed t in

t

obs

towards future over

t

fut

705

710

715

720

725

730

Timestep

0.4

0.2

0.0

0.2

SHAP (std)

DeepSHAP

GradientSHAP

Time Horizon for

t

obs

...

...

Time Horizon for

t

obs

...

...

...

(a) Group leaving behavior (

MatchNMingle

dataset)

...

0.00

0.05

0.10

Saliency

Saliency of observed t in

t

obs

towards future over

t

fut

0.50

0.25

0.00

SHAP (mean)

SHAP values of observed t in

t

obs

towards future over

t

fut

6

7

8

9

10

11

12

Timestep

0.5

0.0

SHAP (std)

DeepSHAP

(b) Vehicle trajectory (

nuScenes

dataset)

standard deviation of the same forecast fail to capture the salient predictive relationships in the data. This comparison underscores the effectiveness of our framework in capturing meaningful and interpretable saliency, even in conjunction with an imperfect forecasting model.

5.2 Empirical Evaluation in Real-World Scenarios

5.2.1 Group Leaving Behavior in the Wild

The study of group-leaving behavior has garnered interest in social psychology and the development of conversational agents [29, 30]. Recent approaches employ data-driven models to predict future non-verbal cues, capturing general predictive patterns in the data [4]. In this study, we demonstrate how our framework can assist domain experts in hypothesizing about the causal relationships between behavioral patterns and group leaving. We leverage the publicly available MatchNMingle dataset [57], which features natural interactions of 92 individuals during a cocktail party. We use an Attentive Social Process model [4] to forecast continuous head pose, body pose, and binary speaking status.

Through our analysis (see Figure 5a), we find that the salient timesteps in the model's forecasts correspond to instances when a person about to leave directs their gaze away from the shared group space ( o-space [1]) by rotating their head. This observation leads to the following hypothesis:

gazing away from the o-space of a conversing group is predictive of group leaving.

While this hypothesis aligns with established leave-taking patterns [1, 58] and the sweeping gaze behavior associated with seeking new interaction partners [59], it requires further validation through subsequent studies and rigorous statistical testing with the involvement of domain experts. Nonetheless, our experiment demonstrates how the framework can unveil data-driven insights into patterns that, in other cases, may have been overlooked by humans but captured by the forecasting model. By contrast, we do not observe any discernible intuitive patterns in the features associated with the trends in DeepSHAP and GradientSHAP values for the predicted mean and standard deviation.

5.2.2 Vehicle Trajectory Forecasting

The accurate forecasting of pedestrian and vehicle trajectories is crucial for safe and sociallyaware autonomous navigation of vehicles [37, 60-62]. In this study, we utilize our framework to investigate vehicle dynamics in real driving scenarios. Specifically, we leverage the nuScenes dataset, a multimodal dataset for autonomous driving [63], and the Trajectron++ forecasting model [37].

Figure 5b illustrates our analysis of vehicle dynamics at an intersection. Notably, our framework identifies a salient timestep for the Trajectron++ model precisely when it becomes more confident that the bus will make a turn instead of continuing straight. This coincides with the model's increased certainty that the point-of-view vehicle will decelerate as a new vehicle enters the scene from the left. Although there are no relevant domain-specific theories in this case to interpret this saliency, these identified patterns align with expected driving behavior. In contrast, the DeepSHAP values fail to capture the model's change in certainty about the bus making the turn instead of continuing straight. Moreover, we also do not identify any intuitive patterns in the predictions associated with the DeepSHAP trends. Thus, our framework serves as a valuable tool for sanity-checking model forecasts in real-world driving scenarios. It helps identify instances where the model's predictions align or misalign with established norms and expectations.

6 Conclusion

We have proposed a computational framework that provides counterfactual explanations of model forecasts based on a principled notion of bottom-up task-agnostic saliency. We derive a closed-form expression to compute this saliency for commonly used probability density functions to represent forecasts [4, 37, 38, 62]. To validate our framework, we conduct empirical experiments using a synthetic setup, enabling quantitative validation and mitigating observer biases associated with visual assessment of saliency maps. Additionally, we demonstrate the practical utility of our framework in two real-world scenarios involving the prediction of nonverbal social behavior and vehicle trajectories. By identifying salient timesteps towards a predicted future through counterfactual reasoning, our framework can support domain experts in formulating data-driven hypotheses regarding the predictive causal patterns involving the features present at those salient timesteps. These hypotheses can then be tested through subsequent controlled experiments, establishing a human-in-the-loop Explainable AI (XAI) methodology. For a more comprehensive discussion, please refer to Appendix E.

7 Limitations and Potential Negative Societal Impact

While our framework provides a closed-form or efficient solution for most probability density functions, limitations arise when an analytic expression for differential entropy is unavailable. As discussed in Section 4.3, alternative approaches like sampling or nonparametric methods can be employed to approximate the entropy, albeit at an increased computational cost.

Our work here is an upstream methodological contribution. However, when applied downstream to human behavior or healthcare data, ethical considerations arise naturally. Here, care must be taken that such methods are not applied for gaining insights into behavior in a way that violates the privacy of people. Our framework enables domain experts to derive data-driven insights and hypotheses about predictive causal patterns. However, hypotheses should be rigorously tested, using controlled experiments and peer review, before being considered valid statements about human behavior. Collaboration among researchers, practitioners, and policymakers across disciplines is crucial to mitigate such societal risks and ensure ethical deployment of AI technologies.

Acknowledgments

The authors would like to thank Jesse Krijthe, Rickard Karlsson, David Tax, Yeshwanth Napolean, and Megha Khosla for the thoughtful discussions.

References

Adam Kendon. Conducting Interaction: Patterns of Behavior in Focused Encounters . Number 7 in Studies in Interactional Sociolinguistics. Cambridge University Press, Cambridge ; New York, 1990. ISBN 978-0-521-38036-2 978-0-521-38938-9.

Oriel FeldmanHall and Amitai Shenhav. Resolving uncertainty in a social world. Nature human behaviour , 3(5):426-435, 2019.

Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction. arXiv:2002.11927 [cs] , February 2020.

Chirag Raman, Hayley Hung, and Marco Loog. Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues. arXiv:2107.13576 [cs] , July 2021.

Marco Loog. Information theoretic preattentive saliency: A closed-form solution. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) , pages 1418-1424, Barcelona, Spain, November 2011. IEEE. ISBN 978-1-4673-0063-6 978-1-4673-0062-9 978-1-4673-0061-2. doi: 10.1109/ICCVW.2011.6130417.

AHC vanderHeijden. Perception for selection, selection for action, and action for perception, 1996.

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems , 31, 2018.

Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking Clever Hans Predictors and Assessing What Machines Really Learn. Nature Communications , 10(1):1096, December 2019. ISSN 2041-1723. doi: 10.1038/s41467-019-08987-4.

Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning, February 2020.

Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence , 267:1-38, February 2019. ISSN 0004-3702. doi: 10.1016/j.artint.2018.07.007.

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion , 58:82-115, 2020. ISSN 1566-2535. doi: 10.1016/j.inffus.2019.12.012.

Christoph Molnar. Interpretable machine learning . Lulu. com, 2020.

David Rügamer, Philipp F. M. Baumann, Thomas Kneib, and Torsten Hothorn. Probabilistic Time Series Forecasts with Autoregressive Transformation Models. arXiv:2110.08248 [cs] , February 2022.

Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv:1905.10437 [cs, stat] , February 2020.

Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister. Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. arXiv:1912.09363 [cs, stat] , September 2020.

Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in neural information processing systems , 30, 2017.

Qingyi Pan, Wenbo Hu, and Ning Chen. Two Birds with One Stone: Series Saliency for Accurate and Interpretable Multivariate Time Series Forecasting. In Twenty-Ninth International Joint Conference on Artificial Intelligence , volume 3, pages 2884-2891, August 2021. doi: 10.24963/ijcai.2021/397.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 , 2013.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems , 30, 2017.

Zachary C. Lipton and Jacob Steinhardt. Troubling Trends in Machine Learning Scholarship, July 2018.

Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect . Basic books, 2018.

Judea Pearl. Causality . Cambridge university press, 2009.

Antonio Torralba. Modeling global scene factors in attention. JOSA A , 20(7):1407-1418, 2003.

Neil DB Bruce. Features that draw visual attention: An information theoretic perspective. Neurocomputing , 65:125-133, 2005.

Lai Jiang, Zhe Wang, Mai Xu, and Zulin Wang. Image saliency prediction in transformed domain: A deep complex neural network method. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 8521-8528, 2019.

WMBoothby. An introduction to differential geometry and riemannian manifolds, 1975.

Anne Keitel, Moritz M Daum, et al. The use of intonation for turn anticipation in observed conversations without visual signals as source of information. Frontiers in psychology , 6(108):265-273, 2015.

Stephen C. Levinson and Francisco Torreira. Timing in turn-taking and its implications for processing models of language. Frontiers in Psychology , 6, June 2015. ISSN 1664-1078. doi: 10.3389/fpsyg.2015. 00731.

Dan Bohus and Eric Horvitz. Managing Human-Robot Engagement with Forecasts and. . . um. . . Hesitations. Proceedings of the 16th International Conference on Multimodal Interaction , page 8, 2014.

Felix van Doorn. Rituals of Leaving: Predictive Modelling of Leaving Behaviour in Conversation. Master of Science Thesis, Delft University of Technology , 2018.

Sanjay Bilakhia, Stavros Petridis, and Maja Pantic. Audiovisual Detection of Behavioural Mimicry. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction , pages 123-128, Geneva, Switzerland, September 2013. IEEE. ISBN 978-0-7695-5048-0. doi: 10.1109/ACII.2013.27.

David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. Highdimensional multivariate forecasting with low-rank Gaussian Copula Processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

David Salinas, Valentin Flunkert, and Jan Gasthaus. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. arXiv:1704.04110 [cs, stat] , February 2019.

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Thomas M Cover and Joy A Thomas. Elements of Information Theory . Wiley, 2006.

A Verdugo Lazo and P Rathie. On the entropy of continuous probability distributions (corresp.). IEEE Transactions on Information Theory , 24(1):120-122, 1978.

Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: DynamicallyFeasible Trajectory Forecasting With Heterogeneous Data. arXiv:2001.03093 [cs] , January 2021.

David Ha and Douglas Eck. A Neural Representation of Sketch Drawings. arXiv:1704.03477 [cs, stat] , May 2017.

Ilia Kulikov, Sean Welleck, and Kyunghyun Cho. Mode recovery in neural autoregressive sequence modeling. arXiv preprint arXiv:2106.05459 , 2021.

Raphaël Dang-Nhu, Gagandeep Singh, Pavol Bielik, and Martin Vechev. Adversarial attacks on probabilistic autoregressive forecasting models. In International Conference on Machine Learning , pages 2356-2365. PMLR, 2020.

Marco F. Huber, Tim Bailey, Hugh Durrant-Whyte, and Uwe D. Hanebeck. On entropy approximation for Gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems , pages 181-188, Seoul, August 2008. IEEE. ISBN 978-1-4244-2143-5. doi: 10.1109/MFI.2008.4648062.

Chenchen Zhang and Yuan Luo. Approximating the differential entropy of gaussian mixtures. In GLOBECOM 2017-2017 IEEE Global Communications Conference , pages 1-6. IEEE, 2017.

Joseph V. Michalowicz, Jonathan M. Nichols, and Frank Bucholtz. Calculation of Differential Entropy for a Mixed Gaussian Distribution. Entropy , 10(3):200-206, September 2008. ISSN 1099-4300. doi: 10.3390/entropy-e10030200.

Gil Ariel and Yoram Louzoun. Estimating differential entropy using recursive copula splitting. Entropy. An International and Interdisciplinary Journal of Entropy and Information Studies , 22(2):236, 2020.

Brendon J Brewer. Computing entropies with nested sampling. Entropy. An International and Interdisciplinary Journal of Entropy and Information Studies , 19(8):422, 2017.

Jiˇ rí Ajgl and Miroslav Šimandl. Differential entropy estimation by particles. IFAC Proceedings Volumes , 44(1):11991-11996, January 2011. ISSN 1474-6670. doi: 10.3182/20110828-6-IT-1002.01404.

Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences , 6(1):17-39, 1997.

Akko Kalma. Gazing in triads: A powerful signal in floor apportionment. British Journal of Social Psychology , 31(1):21-39, March 1992.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning , pages 3319-3328. PMLR, 2017.

Hayley Hung, Dinesh Babu Jayagopi, Sileye Ba, Jean-Marc Odobez, and Daniel Gatica-Perez. Investigating automatic dominance estimation in groups from visual attention and speaking activity. In Proceedings of the 10th international conference on Multimodal interfaces , pages 233-236, 2008.

Volha Petukhova and Harry Bunt. Who's next? Speaker-selection mechanisms in multiparty dialogue. In Workshop on the Semantics and Pragmatics of Dialogue , 2009.

Iwan De Kok and Dirk Heylen. Multimodal end-of-turn prediction in multi-party meetings. In Proceedings of the 2009 International Conference on Multimodal Interfaces , pages 91-98, 2009.

Usman Malik, Julien Saunier, Kotaro Funakoshi, and Alexandre Pauchet. Who speaks next? Turn change and next speaker prediction in multimodal multiparty interaction. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI) , pages 349-354. IEEE, 2020.

Ryo Ishii, Shiro Kumano, and Kazuhiro Otsuka. Prediction of Next-Utterance Timing using Head Movement in Multi-Party Meetings. In Proceedings of the 5th International Conference on Human Agent Interaction , HAI '17, pages 181-187, New York, NY, USA, October 2017. Association for Computing Machinery. ISBN 978-1-4503-5113-3. doi: 10.1145/3125739.3125765.

German Barquero, Johnny Núñez, Sergio Escalera, Zhen Xu, Wei-Wei Tu, Isabelle Guyon, and Cristina Palmero. Didn't see that coming: A survey on non-verbal social human behavior forecasting. arXiv:2203.02480 [cs] , March 2022.

Dario Pavllo, David Grangier, and Michael Auli. QuaterNet: A Quaternion-based Recurrent Model for Human Motion. arXiv:1805.06485 [cs] , July 2018.

Laura Cabrera-Quiros, Andrew Demetriou, Ekin Gedik, Leander van der Meij, and Hayley Hung. The matchnmingle dataset: a novel multi-sensor resource for the analysis of social interactions and group dynamics in-the-wild during free-standing conversations and speed dates. IEEE Transactions on Affective Computing , 2018.

Mark L Knapp, Roderick P Hart, Gustav W Friedrich, and Gary M Shulman. The rhetoric of goodbye: Verbal and nonverbal correlates of human leave-taking. Communications Monographs , 40(3):182-198, 1973.

Monica M. Moore. Nonverbal courtship patterns in women: Context and consequences. Ethology and Sociobiology , 6(4):237-247, January 1985. ISSN 0162-3095. doi: 10.1016/0162-3095(85)90016-0.

Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling. arXiv:2110.06607 [cs] , January 2022.

Sandra Carrasco, David Fernández Llorca, and Miguel Ángel Sotelo. SCOUT: Socially-COnsistent and UndersTandable Graph Attention Network for Trajectory Prediction of Vehicles and VRUs. arXiv:2102.06361 [cs] , May 2021.

Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research , 39(8): 895-935, 2020.

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multimodal Dataset for Autonomous Driving. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11618-11628, Seattle, WA, USA, June 2020. IEEE. ISBN 978-1-72817-168-5. doi: 10.1109/CVPR42600.2020.01164.

David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research , 11: 1803-1831, 2010.

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning , pages 3145-3153. PMLR, 2017.

Sarah Adel Bargal, Andrea Zunino, Donghyun Kim, Jianming Zhang, Vittorio Murino, and Stan Sclaroff. Excitation Backprop for RNNs. arXiv:1711.06778 [cs] , March 2018.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision , pages 818-833. Springer, 2014.

Felix Mujkanovic, Vanja Doskoˇ c, Martin Schirneck, Patrick Schäfer, and Tobias Friedrich. timeXplain - A Framework for Explaining the Predictions of Time Series Classifiers. arXiv:2007.07606 [cs, stat] , July 2020.

Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Clinical intervention prediction and understanding using deep networks. arXiv preprint arXiv:1705.08498 , 2017.

Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi. Benchmarking Deep Learning Interpretability in Time Series Predictions. arXiv:2010.13924 [cs, stat] , October 2020.

Edward Choi, Mohammad Taha Bahadori, Joshua A. Kulas, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism. arXiv:1608.05745 [cs] , August 2016.

Ahmed M Alaa and Mihaela van der Schaar. Attentive state-space modeling of disease progression. Advances in neural information processing systems , 32, 2019.

Madhurima Panja, Uttam Kumar, and Tanujit Chakraborty. An Interpretable Probabilistic Autoregressive Neural Network Model for Time Series Forecasting. arXiv:2204.09640 [cs, stat] , April 2022.

Longyuan Li, Junchi Yan, Xiaokang Yang, and Yaohui Jin. Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting. arXiv:2102.00397 [cs, stat] , January 2021.

Shalmali Joshi, Sonali Parbhoo, and Finale Doshi-Velez. Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty, September 2021.

A Broader Related Work: Explainable Methods for Time-Series Data Across Tasks and Domains

The larger focus of explainability techniques involving time-series data has been on the task of classifying time-series. The goal has been to estimate the relevance of each input feature at a given timestep towards each output class. Here, saliency approaches often overlap with techniques developed for image data and can be categorized into:

Gradient-Based Techniques. The broad approach involves evaluating the gradient of the output class with respect to the input [64]. Several variants have been proposed [19, 49, 65-67].

Perturbation-Based Techniques. The idea is to examine how the output changes in response to some perturbation of the input. Perturbations are implemented by either occluding contiguous regions of the input [68, 69]; performing an ablation of the features [70]; or randomly permuting features [12]. Ismail et al. [71] provide a benchmark of a subset of these techniques.

Attention-Based Techniques. These incorporate an attention mechanism into the model that is trained to attribute importance to different parts of the input sequence towards a prediction at each future timestep. Such techniques have been extensively utilized for healthcare data. Early methods applied a reverse-time attention [72], Later methods applied the attention to probabilistic state-space representations [73].

Some of these broad ideas have been applied to the regression setting to make interpretable forecasts of future time-series features. Lim et al. [15] leveraged self-attention layers for capturing long-term dependencies. Pan et al. [17] recently proposed computing saliency as a mixup strategy between series images and their perturbed version with a learnable mask for each sample. They view saliency in terms of minimizing the mean squared error between the predictions and ground-truths for a particular instance. Focusing on the univariate point-forecasting problem, Oreshkin et al. [14] proposed injecting inductive biases by computing the forecast as a combination of a trend and seasonality model. They argue that this decomposition makes the outputs more interpretable.

Developing explainable techniques for the probabilistic forecasting setting remains largely unexplored and subject to non-overlapping notions of explainability. Rügamer et al. [13] transform the forecast using predefined basis functions such as Bernstein polynomials. They relate interpretability to the coefficients of these basis functions (a notion similar to that of Oreshkin et al. [14]). Panja et al. [74] embed the classical linear ARIMA model into a non-linear autoregressive neural network for univariate probabilistic forecasting. As before, the explainability here also stems from the 'white-box' nature of the linear ARIMA component. Li et al. [75] propose an automatic relevance determination network to identify useful exogenous variables (i.e. variables that can affect the forecast without being a part of the time-series data). To the best of our knowledge, saliency-based methods have not yet been considered within this setting.

B Favorable Properties of Differential Entropy

Differential entropy possesses favorable properties that make it a suitable choice as ϕ for computing the saliency map. First, the scale of the forecast density does not affect the resulting saliency map (see Cover and Thomas [35, Theorem 8.6.4]):

̸

That is, scaling the distribution changes the differential entropy by only a constant factor. So the saliency map resulting from inserting the entropy into Equation 4 remains unaffected since the Jacobian term only depends on the relative change in entropy across different choices of t obs . Similarly, translating the predicted density leaves the saliency map unaffected (see Cover and Thomas [35, Theorem 8.6.3]):

C Implementation Details for Experiments

C.1 Other Explainability Methods

For DeepSHAP and GradientSHAP, we used the official implementation of SHAP: https://github. com/slundberg/shap . For IntegratedGradients and SmoothGrad, we used the Captum framework: https://captum.ai/ . We reiterate that these are not fair comparisons, for the reasons we have discussed in Section 2. One crucial reason is that no existing method is designed to handle a predicted distribution. To apply them in this context, we need to compute attribution values for the predicted mean and standard deviation for every feature at every t ∈ t fut in isolation. In contrast, by measuring differential entropy, our method jointly accounts for the parameters of the distribution and captures the information content of the distribution. Despite these limitations, we include these comparisons to provide readers with a contextual understanding of the results obtained from commonly used explainability tools.

Computational Efficiency. For further insight, we measured the execution time of our saliency method compared to DeepSHAP for both real-world scenarios (see Table 1). The main takeaway is that our method is an order of magnitude faster at computing the saliency map compared to DeepSHAP. This is in part because DeepSHAP computes values for every feature at every timestep in the output independently. Even if this is parallelized, DeepSHAP requires multiple forward passes and gradient computations including samples from a background set to compute the reference values. The computation time scales with the size of the background set.

C.2 Empirical Validation using Synthetic Data

We model the future distribution using a Gaussian function for simplicity (setting std. to 10 -10 for the single future), but a more complex distribution that predicts the appropriate change in variance would also work in practice. We now implement Algorithm 1 as follows. We identify a window where a turn change occurs in the data (frames 183 -228 ) and denote this 45 frame window as the t fut of interest. While we manually identify an interesting event for illustration, such a window could also correspond to an interesting prediction by a model. We generate a set of candidate t obs by sliding a 30 frame window over a horizon of 100 frames prior to t fut , with a stride of 1 frame. For every observed t obs , we fit a Gaussian density to the quaternion and speaking status features of all participants over the futures that can occur during t fut . We then set the entropy of this Gaussian density as the feature ϕ for that t obs . For the experiments with a real forecasting model, we employ the [recurrent, uniform attention] variant of the Social Process family given its ability to capture dynamic movements [4].

C.3 Forecasting Group Leaving Behavior

We used the pretrained model on the MatchNMingle dataset provided with the official implementation of Social Processes [4]: https://github.com/chiragraman/social-processes . Specifically, we employed the [recurrent, dot-attention] variant of the Attentive Social Process family (ASP-GRU-dot). We set t fut to correspond to a 3 second window ( 3 frames in the data, which is at 1 Hz) containing an individual leaving a conversing group. We obtained forecasts from the model corresponding to a rolling 5 second t obs within a 20 second preceding horizon. For computing DeepSHAP and GradientSHAP values, we used the entire observed time horizon as the background

dataset. This ensures that the expected values are computed within a reasonably similar context for a given t obs .

C.4 Vehicle Trajectory Forecasting

We trained the Trajectron++ model [37] on the nuScenes dataset (mini version) [63] using the default command provided in the official implementation: https://github.com/StanfordASL/ Trajectron-plus-plus . In particular, we used the int_ee model which incorporates the agent's system dynamics to produce dynamically-feasible trajectories. For analysis, we used a sequence from scene index 3 ( scene.name 757 ). We set t fut to correspond to a window of 6 timesteps ( 14 -19 ) with a lookback horizon from timesteps 6 to 13 . For computing DeepSHAP values we again used the entire observed horizon as the background set. The features in the observed sequence contained several NaN entries, which resulted in NaN DeepSHAP values. We ignored these when aggregating values per timestep.

D Additional Results

Time

Horizon for

t

obs

50

0

50

Rotation

(degrees)

Head

Direction

in

Z-angle

0.00000

0.00025

0.00050

0.00075

Saliency

Saliency

of

observed

t

in

t

obs

towards

future

over

t

fut

0.25

0.00

0.25

SHAP

(mean)

SHAP

values

of

observed

t

in

t

obs

towards

future

over

t

fut

842

847

852

857

862

867

Timestep

0.25

0.00

0.25

SHAP

(std)

DeepSHAP

GradientSHAP

...

...

(a)

14

:

27

at

Day 1, Cam 12

...

...

50

25

0

25

50

Rotation

(degrees)

Head

Direction

in

Z-angle

0.00000

0.00005

0.00010

Saliency

Saliency

of

observed

t

in

t

obs

towards

future

over

t

fut

0.25

0.00

0.25

SHAP

(mean)

SHAP

values

of

observed

t

in

t

obs

towards

future

over

t

fut

1520

1525

1530

1535

1540

Timestep

0.25

0.00

0.25

0.50

SHAP

(std)

DeepSHAP

GradientSHAP

(b)

25

:

42

at

Day 2, Cam 5

...

0.00

0.02

0.04

Saliency

Saliency

of

observed

t

in

t

obs

towards

future

over

t

fut

0.5

0.0

SHAP

(mean)

SHAP

values

of

observed

t

in

t

obs

towards

future

over

t

fut

6

8

10

12

14

16

Timestep

0.0

0.5

SHAP

(std)

DeepSHAP

E Broader Discussion: Saliency & XAI with Domain Experts in the Loop

We begin this broader discussion by revisiting the different notions of saliency, to make the case for why our proposed framework is suitable for forecasting tasks. Rather than defining saliency in a top-down manner as a function of some task-specific error metric, we have started from a more fundamental conception of bottom-up, or task-agnostic, saliency. Loog's [5] original definition pertains to preattentive saliency, which captures what is perceived to be subconsciously informative before conscious (attentive) processing by the brain. Here, a surprising or unexpected observation is salient. For instance, in a large white image with a single black pixel, the black pixel is salient. The direct application of this concept to time-series data would involve identifying surprising task-agnostic temporal events. For instance, imagine viewing a static landscape where a bird suddenly flies in. The entry of the bird into the scene is unexpected, and therefore salient.

When applied to forecasting tasks, however, this idea of surprisal (or unexpectedness or informativeness) that saliency represents needs to be tied to the future outcome. The saliency computed by most methods working on point-forecasting tasks deals with which past features are surprising given a specific realization of the future. While not explicitly stated by these works, we argue that this notion of saliency is related to the surprisal in p X | Y for some specific Y . We therefore interpret these methods as being associative in nature within Miller's [10] categorization in Section 3. In contrast, our approach is counterfactual because we examine alternate future outcomes, while conceptualizing saliency more naturally defined in terms of the changes in the uncertainty in p Y | X in response to different realizations of observed sequences. However, rather than corresponding to random occlusions or perturbations of the input, the different realizations of X in our framework correspond to real features (or behaviors) preceding a future, which is more suitable to present to domain experts as candidate causes.

Loog's [5] unifying framework subsumes all forms of saliency, although identifying the appropriate ϕ for a specific domain is non-trivial. In this work we have established both theoretically and empirically how expressing ϕ in terms of the information about the future enables principled counterfactual reasoning in forecasting settings. Nevertheless, we reiterate that the salient timesteps retrieved by our framework ought to be treated as candidate causes until subsequently examined along with a domain expert. Our stance on human-in-the-loop XAI also aligns with research on saliency-based and general XAI in other domains [9, 76].

In principle, when it is possible to have access to the true p Y | X , the salient timesteps identified by our framework reflect the true predictive structural relationships captured by the underlying model across the entire data. However, estimating this density analytically entails identifying the multiple futures in the data corresponding to every occurrence of the same observed features. In practice, subtle variations in behaviors and sensor measurement errors make it infeasible to estimate p Y | X analytically, so a model is trained to capture generalized patterns from the given data. In these cases, our framework identifies the sequences that the model considers salient for its forecasts given the data . Consequently, subsequent causal analysis of the features over the salient timesteps is crucial, especially in the healthcare and human behavior domains to avoid potential prejudices against certain behaviors, or worse, misdiagnoses of affective conditions.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework to provide counterfactual explanations for time-series forecasting models. This method attempts to detect the salient timestep which is critical to a future event. The method is built based on the framework of bottom-up preattentive saliency. Experiments are conducted on one synthetic and two real-world datasets, highlighting that the proposed model is able to localize the salient time step.\n\nSTRENGTHS:\nThis project tackles a relevant and not well-studied problem: explaining time-series forecasting models. The authors motivate their method using psychology and cognitive science, which bridges the gap between pure algorithmic explanation design and the need of human users.\n\nWEAKNESSES:\n(1)\tHow to understand that the explanation (salient timestep) is a counterfactual explanation? This framework is “to compute the saliency of an observed t with respect to a fixed choice of t” (L202), which cannot reflect the counterfactual. Does it infer that the “fixed choice of t” is the event that did not happen?\n\n(2)\tThe current method functions more as a “critical event detector” rather than an explainer. For instance, in the two experiments on real-world datasets, the tasks can be done by an action localization model studied in the video understanding area. The authors need to explain why these are research challenges that can be solved by model explanations. \n\n(3)\tThe experiment settings need to be clearer to address the advantages of the proposed explanations. As pointed out in (2), the current experiments are too similar to critical event detection rather than an explanation task. The authors should reconsider a task where a counterfactual explanation is generated and compared.\n\nQUESTIONS:\nL295, should it be Figure 4b? In Figure 4a, the perfect forecast model is considered. But there is a plateau in the saliency, i.e., more than one frame is labeled as salient. If this is the case, does the algorithm still work in this scenario?\n\nLIMITATIONS:\nThe novelty of the proposed method needs to be improved. The method is built on the previous work from Loog et al. and causality models. The authors should address their novelty to highlight their contributions. Overall, this paper needs a non-trivial revision to address all weaknesses and to make the contribution clear.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors create a saliency method for time-series data, where they quantify the impact that a previous time-step has on the model's prediction of a future time-step. They deploy this method on synthetic and real-world video datasets, and show that this method is able to pick up on key points in the time-series process.

STRENGTHS:
I think this is a strong paper that is relevant to the community. The formulation seems quite elegant, and the results on both synthetic and natural experiments are strong (especially in comparison to DeepShap). In particular, I thought the vehicle trajectory was quite interesting, and would encourage the authors to spend more time discussing those results beyond a single example, perhaps by shortening Section 3.

WEAKNESSES:
Clarity: There were some aspects of the methodology that were hard to parse (in particular Section 4.3). One thing that might help is including a running example (perhaps the pedestrian one from the intro) throughout the methodology. 

Baselines: Can you include another baseline beyond just DeepShap (at least for the synthetic setting) e.g., Integrated Gradients or SmoothGrad.

The discussion of the real-world examples just talk about a few examples. I would suggest expanding this section, and including some quantitative metrics over the entire dataset.

QUESTIONS:
Is it possible to use such a formulation for sequential but non-time series data (for example, understanding important tokens in the prompt for an NLP setting?)

LIMITATIONS:
The authors do discuss limitations to their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The paper proposes a computational XAI framework for counterfactual reasoning in probabilistic multivariate time-series forecasting. The key idea of the paper is to establish the conceptual foundation for linking saliency-based explanations with counterfactual reasoning. Specifically, the paper leverages a unifying definition of information theoretic saliency grounded in preattentive human visual cognition and extends it to forecasting settings. Experiments on synthetic and real data demonstrate the utility of the proposed framework.

STRENGTHS:
The key idea of the paper is well-motivated. Deriving the conceptual link between counterfactual reasoning and saliency-based explanation techniques from Miller’s framework is interesting. Also, it is well-explained in section 3. Experiments sufficiently justify the key conclusions made in the paper. The evaluations and analysis have a wide range practical applicability in multiple real world problems.

WEAKNESSES:
I do not find any major weaknesses with the work. Here are some minor weaknesses: I find it hard to completely follow the use of information-theoretic preattentive in section 4 (and section 4.3 in particular). Improving the readability with multiple examples could help readers to understand the key intuitions. Also, It would be nice to have some quantitative evaluation metrics for the evaluations made in section 5.2.1 and 5.2.2. 

Missing related work:

Akula, A., Wang, S., & Zhu, S. C. (2020, April). Cocox: Generating conceptual and counterfactual explanations via fault-lines. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 03, pp. 2594-2601).

QUESTIONS:
Please see Weaknesses

LIMITATIONS:
Authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for counterfactual explanation using Information-Theoretic Saliency. The method trains a model using data collected up to the present moment and applies it to predict the outcomes caused by events occurring in the future time. This paper provides experiments on synthetic and real-world data, and the results demonstrate the effectiveness of this method to a certain extent.\n\nSTRENGTHS:\nThis paper presents a method which uses information-theoretic saliency metrics, the motivation is interesting. The experiment results show the effectiveness of the proposed method on both synthetic and real-world data.\n\nWEAKNESSES:\nIn my opinion, the biggest issue with this paper is that, despite the authors claiming to integrate the tasks of future forecasting and counterfactual reasoning, I don't see any actual utilization of counterfactual reasoning techniques in this article. Firstly, the design of the causal graph does not adhere to the standards of causal work. Secondly, the paper mentions the use of Pearl's abduction-action-prediction steps, but I fail to see where abduction (inference of exogenous variables) and action (intervention) are applied. I think the authors should delve deeper into understanding the relationship between counterfactual reasoning and the problem addressed in this paper.\n\nQUESTIONS:\n* This paper addresses the problem of time-series forecasting (regression). How does this problem differ from time-series modelling?\n* This paper builds upon Miller's framework, but it lacks a detailed introduction to Miller's framework, which hinders the smooth flow of the article.\n* What is the necessity of using information-theoretic methods to solve the problem?\n* In line 168, the paper mentions replacing counterfactuals from random perturbation by observation, which makes me confused. Intuitively, observation should refer to factual data. Or it seems the authors are just emphasizing the inclusion of some real features.\n* The design of the causal graph shown in Figure 1 raises some doubts: \n\n(i) In the field of causality, specific variables are generally set as causal nodes, while the model is designed as the causal path between nodes. In our environment, we can only intervene in variables, not the causal effects between variables. Exogenous variables should act on endogenous variables, not the model itself. If we adopt the causal graph depicted in Figure 1, the implicit implication is that the model is also a manipulable factor. However, the model should be used to describe the relationship between t_obs and t_fut, which should remain constant in the environment. In other words, the relationship between t_obs and t_fut data should not change based on the effectiveness of the trained model. \n\n(ii) The proposed method relies on the correctness of causal graph shown in Figure 1a. My concern is that exogenous variables should not only exist in t_obs, t_fut, and M (assuming M's position in the causal graph is reasonable), but more importantly, they should also exist in I_fut. If you assume that I_fut is not influenced by exogenous variables or confounders, then the abduction-action-prediction process is unnecessary for the method you have designed, and the problem degenerates into a simple time series prediction problem.\n* Based on the aforementioned issues, the model should remain constant in the environment, and it is necessary to estimate the model because it is unknown. In the process of estimating the model, since t_obs is used for training, how can we ensure the identifiability of the estimated model? In other words, how can we guarantee that the learned model is the true model?\n* Could the authors explain why \\hat{Y} is introduced in Section 4.3? What confuses me is that from lines 228-233, it seems that \\hat{Y} is sampled from the distribution as a single value. However, in Equation 7 and the subsequent descriptions, \\hat{Y} appears to become a variable, which makes me feel that the mathematics used here is not very rigorous.\n* Did the experiments include baselines for comparison with other relevant methods?\n\nLIMITATIONS:\nThe author presents innovative information-theoretic saliency metrics; however, the rigour of this paper could be improved.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

comments: SUMMARY:
This paper presents a method for detecting salient points in time+feature space for models which make predictions from timeseries data. Salient points are defined as points that once observed produce a large change in the model’s distribution of predicted future outcomes (measured as differential entropy). The method is evaluated on a toy dataset, in which the factors that lead to future events are known, and in two real-world datasets where these factors are unknown. The proposed method finds the true salient features in toy data and finds plausible salient features in real-world data.

STRENGTHS:
- The paper is well-written with a clear explanation of the theory, method, and limitations.
- The approach is novel, at least for this specific application (information-theoretic models of saliency are common in other areas, such as predicting human eye movements in single images).
- The proposed method is straightforward and works well.

WEAKNESSES:
- The evaluation could be more detailed (e.g., evaluating the effect of different model design choices or how sensitive the model is to time parameters t_obs and t_fut, which have to be set by the user).

QUESTIONS:
- I’m unclear why you can drop the prior (-log p_x(x)) in between equations 3 and 4. I assume Loog dropped this due to assuming a uniform prior, but is p_x uniform here? (Also, in line 195 – it seems like the saliency is only “independent of previously observed data” if you assume a uniform p_x, which is not always a good assumption for real-world image saliency tasks.)
- What are the "low-level features" (line 294) used to model the synthetic dataset? Pixels? Edges?

LIMITATIONS:
This is adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11