Why Did This Model Forecast This Future? Information-Theoretic Temporal Saliency for Counterfactual Explanations of Probabilistic Forecasts
Authors: Chirag Raman
Hayley Hung
Marco Loog
Venue: Submitted to ICLR 2023 (2023)
Keywords: probabilistic forecasting
saliency
explainability

Abstract: Probabilistic forecasting of multivariate time series is significant to several research domains where multiple futures exist for a single observed sequence. Identifying the observations on which a well-performing model bases its forecasts can enable domain experts to form data-driven hypotheses about the causal relationships between features. Consequently, we begin by revisiting the question: what constitutes a causal explanation? One hurdle in the landscape of explainable artificial intelligence is that what constitutes an explanation is not well-grounded. We build upon Miller's framework of explanations derived from research in multiple social science disciplines, and establish a conceptual link between counterfactual reasoning and saliency-based explanation techniques. However, the complication is a lack of a consistent and principled notion of saliency. Also, commonly derived saliency maps may be inconsistent with the data generation process and the underlying model. We therefore leverage a unifying definition of information-theoretic saliency grounded in preattentive human visual cognition and extend it to forecasting settings. In contrast to existing methods that require either explicit training of the saliency mechanism or access to the internal parameters of the underlying model, we obtain a closed-form solution for the resulting saliency map for commonly used density functions in probabilistic forecasting. To empirically evaluate our explainability framework in a principled manner, we construct a synthetic dataset of conversation dynamics and demonstrate that our method recovers the true salient timesteps for a forecast given a well-performing underlying model.

Under review as a conference paper at ICLR 2023
WHY DID THIS MODEL FORECAST THIS FUTURE? INFORMATION-THEORETII TEMPORAL SALIENCY FOR COUNTERFACTUAL EXPLANATIONS OF PROBABILISTIC FORECASTS
Anonymous authors authorss Paper under double-blind review
ABSTRACT
Probabilistic forecasting of multivariate time series is significant to several research domains where multiple futures exist for a single observed sequence. Identifying the observations on which a well-performing model bases its forecasts can enable domain experts to form data-driven hypotheses about the causal relationships be- tween features. Consequently, we begin by revisiting the question: what constitutes a causal explanation? ? One hurdle in the landscape of explainable artificial intelli- gence is that what constitutes cxplanation is not well-gre goolddroddddd. We build upon Miller's s framework explanations derived from research in multiple social science disciplines, and establish conceptual link between counterfactual reasoning and saliency-basee based explanation techniques. However, the complication is a lack of a consistent and principled notion of saliency. Also, commonly derived saliency maps may be inconsistent with the data generation process and the underlying model. We therefore leverage unifying definition of information-Lee -theorrticcc saliency grounded in preattentive human visual cogmition and extend it to forecasting settings. In contrast to existing methods that require either explicit training of the saliency mechanism access to the internal parameters of the underlying model, we obtain closed-fi -orm solution for the resulting saliency map for commonly used density functions in probabilistic forecasting. To empirically evaluate our explainability framework in principled manner, we construct a synthetic dataset of conversation dynamics and demonstrate that our method recovers the true salient timesteps for forecast given a well well-performing underlying model.
INTRODUCTION
The existence of multiple valid futures for given observed sequence is crucial attribute of several
forecasting tasks, especially surrounding the dynamics of low-level human behavior. These tasks
include the forecasting of trajectories of pedestrians (Huang et al. 2019 Mohamed et al. 2020
Rudenko et al. 2020; Salzmann et al., 2021: Zhang et al. 2019). vehicles (Carrasco et al.
2021
Gilles et al.
|ll.|222.222 [022] Zeng et al. 20200] Zhao al. 2020), and autonomous robots (Ivanovic et al. 2021
Vemula et al. 2017). or other more general nonverbal cues of humans (Adeli et al. 2020 Barquero
adl.)222) guyen & Celiktutan 2022 Raman al. 202T Yao et al. 20188) and artificial virtual
agents (Ahuja et al. 2019) in group conversation settings. Consequently, rather than making single (i.e. point) predictions, several machine learning methods in these settings have attempted forecast distribution over plausible futures (Mohamed et al.) 2020] Raman et al. 2021). In this work, we insights into such
introduce and address a novel research question towards gaining
forecasts: given a reliable underlying model, how can we identify the observed timesteps that are salient for the model's probabilistic forecasts over particular future window?
1.1 NOTIONS OF INTERPRETABILITY IN FORECASTING TASKS & DRAWBACKS
Recently, several works have proposed techniques for making interpretable non con-probabilistic pre- dictions for point forecasting tasks (Lim al. 2020 Oreshkin al. 2020 Pan et al. 2021). The

---

Under review as a conference paper at ICLR 2023
Table 1: Classes of Explanatory Question and the Reasoning Required to Answer. Reproduced from Miller 2019 Table 3)
Question
Reasoning
Description
What?
Associative
Reason about which unohserved events could have occurred given the observed events
How?
Interventionist
Simulate change in the situation to see if the event still happens Simulating alternative causes to see whether the event still happens
Why?
Counterfactual
general approach has been to train architectures to produce forecasts that are not only accurate but also interpretable. However, in what [ipttnn what[Lipton (2017) terms The Mythos of Interpretability, the notion of what renders these models interpretable is often not well-grounded and subject to presenting spec- ulation in the guise of explanation Lipton & Steinhardt (2018). Examples of these operationallizaionn of interpretabillty] for forecasting tasks include: (i) injecting a suitable inductive bias into the model through a set of basis functions and identifying how they combine to produce an output (Oreshkin it al. 2020) (an approach that has recently been applied to the probabilistic setting as well (Rügamer al. )022);; ); (ii) employing a self-attention mechanism learn temporal patterns while attending to common set of features (Lim et al. (2020); and (iii) applying the notion of saliency maps from computer vision (Dabkowski & Gal G to simesseries data as a measure of how much each feature contributes to the final forecast (Pan et al. 2021). A broader review of notions of interpretability across domains is in Appendixx A
Irrespective of the notion of interpretability, these methodologies are underpinned by two common attributes: (a) the interpretability mechanism needs explicit training as part of the model architecture, and (b) what constitutes a good explanation is subject to the biases, intuition, the visual assessment of the human observer (Adebayo et al. 2018 Miller 2019); a phenomenon we refer to as the interpretation being in the eye of the beholder. This is especially true for when saliency maps have been used as tools for post-hoc explanations: the computed map may not measure the intended saliency, and even be independent of both the model and data generating process (Adebayo et al.. methods is further
2018; Atrey et al. 2020 Lapuschkin et al. 2019). Research on saliency- confounded by the lack of a common notion of saliency. As Barredo Arrieta et al. (2020) Sec. 5.3) point out, "there is absolutely no consistency behind what is known as saliency maps, salient masks, heatmaps, neuron activations, attribution, and other approaches alike." "
1.2 SALIENCY- EXPLANATIONS FOR COUNTERFACTUAL REASONING & DRAWBACKS
Due to the inconsistencies mentioned above, we argue that the premise of what constitutes an explanation needs to be well-grounded within existing frameworks of how people define, generate, and present explanations. MMillrr (2019) recently turned to the vast body of research on the topic in philosophy, psychology, and cognitive science and highlighted the importance of causality in explanation. Specifically, in Table we reproduce the categorization
explanatory questions he proposed based on Pearl and Mackenzie's 's Ladder of Causation (Pearl & Mackenzie, 2018). Using an abstract notion of an 'event' that needs explaining, Miller argues that the what-questions involve associative reasoning. How questions require interventionist reasoning to determine the set of causes that, if removed, would prevent the event from happening. The why-questions are the most challenging, as they require counterfactual reasoning to undo events and simulate other events. These also require associative and interventionist reasoning.
To apply Miller's (2019) framework to forecasting, consider a model M that predicts features over a future window brue by observing features over a window tobs- We argue that most of the existing interpretability approaches- unproaches-iicludinng the two discussed categories involving the injection of inductive biases and attention based mechanisms- are associative in nature. For a fixed tha and single fabs: they reason about the (unobserved) importance of features over tohs using model parameters or attention coefficients based on one prediction from M (the 'event'). In contrast, we argue that the perturbation-base saliency methods have the potential to support counterfactual reasoning. The saliency masks are commonly learned by perturbing different parts of the input over
Barredo Arrieta et al. 2020 argue for the importance of distinguishing between interpretability and explainability as different concepts. the latter denoting any active action procedure taken by model with the intent of clarifying detailing its internal functions. From this perspective, what the eited works term interpretabiliity is eloser to the notion of explainability.
2

---

Under review as a conference paper at ICLR 2023
tobs- boberby simulating alternative 'causes' from M's perspective and observing the effect on an error metric (the 'event'). However, while saliency-based approaches can, in theory, be promising in answering such why-questions, for deriving explanations: (i) the feature-level manipulations used to produce saliency maps produce distortion interventions that may be inconsistent with the data generation process (Adebayo et al. 2018; Lapuschkin et al. (2119) and not lie on the real-world data manifold or preserve real-world semantics (Atrey et al. 2020); (ii) the notion of saliency in terms of minimizing an error metric is an arbitrary choice, requiring the ground-truth truth future that may not be available at test; and (iii) for forecasting applications, the saliency map is explicitly retrained for every single snserved-future
identify several issues with how such methods are being applied
sequence (Pan et al. 2021) and does not capture salient structural patterns across samples.
1.3
OUR APPROACH: BUILDING UPON A UNIFYING FRAMEWORK OF BOTTOM- UP SALIENCY
Our broad conceptual contribution is to link counterfactual reasoning with principled no- tion of saliency. In Figure we formally ex-
eM
press counterfactual reasoning for explaining forecasts using causal graphs (Pearl 2009). The generic graph expresses relationships be- tween the random variables prior to training the forecasting model denoted by M. The exoge- nous variable cad captures the randomness in the training process and modeling choices in- cluding the distribution family for representing probabilistic forecasts. The exogenous variable eи captures the the randomness in the human observer's s choice of observed and future win- dows to examine the model. Our central idea is to evaluate the information in the model's predicted distribution denoted by fut- Specifically, we propose posing the the following counterfactual ques- tion: What information would M have about the future over given tus if it observed the features over tohs? ? The modified graph for evaluating this question is depicted in Figure Ib] Once the model M has been trained and the windows lohs and litt have been chosen, the effect of the exogenous variables on the variable Lut disappears. This yields a deterministic system where the change in the information about the future can be evaluated for different values of tobs and trut- Given that the focus of this work is the explanation phase, we assume the modified graph is already available. Note that while the procedure starting from training the forecasting model in the generic graph implicitly follows Pearl's abduct-action preeiiitioriiiiionnn process (Pearl 2009, p.207), estimating the distribution over the exogenous variables from the abduction step is conceptually not applicable in this setting.
Data
I M
M = M
Tobs
T obs
tobs
fut
fut
EH
T iut
Fut
tru
(a) Generie Graph
(b) Modified Graph
Figure 1: Causal Graphs for Explaining Forecasts
To relate such counterfactual reasoning to saliency, we propose the following implication:
observing the features at a timestep E lobs results in a change in the model's information about the future Lut over the given window t/ut + t is salient.
This notion of saliency related to surprisal information associated with observations is grounded in preattentive human cognition (Loog. 2011 vanderHeijden 1996). Concretely, the statement entails that an observation that changes the certainty of the future surprising, and thereby salient. Note that the antecedent (on the left of the implication) is a counterfactual statement. The implication entails that for the antecedent be true must be salient. However, knowing the antecedent false is not sufficient to conclude that t is not salient, i.e. there can be other notions of saliency that make t salient. In this work, rather than defining saliency in a top-down manner as a function of some task- error metric that cannot be computed at test, as other works have done (Pan et al., 2021 we build our computational framework upon a general expression of bottom-up, or task-agnostic. preattentive saliency (Loog] 2011). Specifically, we derive closed-form solution for identifying salient timesteps using the differential entropy in M's forecast over tuu. We then empirically validate the framework using the counterfactual reasoning illustrated in Figure T for a forecasting task where suman-verifiable ground- -trtthhhhhhhhhh saliency exists.
Preattentive saliency captures what is perceived to be subconsciously informative before conscious, or attentive, processing by the brain. Here, a surprising or unexpected observation is salient. When applied to forecasting tasks, this idea of surprisal (or unexpectedness informativeness) needs
3

---

Under review as a conference paper at ICLR 2023
to be tied the future outcome, which guides our proposal of evaluating the information in the predicted future density. Our approach is counterfactual because we examine changes in this future density in response different realizations of real-world sequences tots. In contrast, the only existing saliency-based method for time-series forecasting follows associative reasoning within Miller's (2019) categorization: |an et al. (202T) retrain saliency map for every single Lobs, from scratch, at time, using a single (point) prediction over the tuu instead of distribution.
Our framework addresses all the aforementioned concerns we identified with saliency-based ap- proaches in forecasting: (i) the counterfactuals in our framework pertain to windows of real observed features rather than random perturbations of the input, and therefore preserve real-world semanties; (ii) our information -heooreticcc perspective of saliency is principled, and given the underlying model, can be used to compute saliency for unseen test data where the ground-truth future is unavailable; (iii) we utilize a distribution over the futures that the model believes could have occurred for a single input: this distribution captures structural predictive relationships between features across multiple samples. Moreover, our framework involves no training for computing the saliency, and can be applied to any underlying model that outputs a distribution over futures.
2 BACKGROUND: INFORMATION THEORETIC PREATTENTIVG SALIENCY
Loog (2011) developed a general closed-form expression for saliency using surprisal-basee op- erational defimition of bottom-up attention from the field of computational visual perception. Let L T Rdd be general n-dimensional d-valued image. A continuously differentiable feature mapping Φ RR + R relates every image location x E R from L. to N features. Further, let px denote the probability density function over image locations x and pa denote the probability density function over all feature vectors 6(x). The distribution px captures any prior knowledge that would make location more salient than another. the absence of such prior knowledge, px typically chosen to be uniform.
The saliency S(x) of a location r is then defined in terms of the amount of information- or sur- prise-of of its associated feature vector ((x) relative to the other feature vectors extracted from the same image. The intuition is that the larger the information content of certain combination of features, - log ()()), the more salient the location
S(x) S(x')
-
log Pa (é(x'))
This general definition unifies seemingly different definitions of saliency encountered in the literature. relates to salient observations being considered unexpected, rare, or surprising (Garcia et al.
2001: [tti & Koch, 2001 Torralba 2003; Walker al. 1998). It also incorporates the notion of saliency considered in decision-theoretio settings (Lingyun et al. 2007 Torralba 2003), where bottom-up saliency increases with increase in the information associated with feature vectors (Gao
& Vasconcelos 2007 Rosenholtz 1999).
Contrary to approaches that determine saliency maps through explicit data-driven density estima- tion (Bruce, 2005) Dabkowski & Gal 2017 Fong & Vedaldi 2017 Gao & Vasconcelos 2007]] Huang
Gao 2020 Itti & Baldi][0005| 2005| Jiang al. 2019 Li Yu 2016 Li 2013 Simonyan et al.
2013) Torralba 2003: Zhou et al. 2016), a closed form expression for saliency can be given once the feature mapping @ is fixed. When @ is continuously differentiable, the information content log p C R over all image features can be obtained from log through simple change of variables (Boothby 1975) from z to Φ(x). The amount of information for the feature vector ((x) at
on ((TR2)
every location and thereby the saliency S(x), is then given by the expression:
(x(x)
log pa (p(())) log
(3)
det (1)
log + log det (J (J)()))
where Jg. R. +R Nx" denotes the Jacobian matrix of o. and indicates matrix transposition.
crucial implication of this result is that the computation of saliency can be performed based on purely local measurements and without the need to refer to previously observed data or any explicit
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Temporal Saliency in Probabilistic Forecasting
Require: The probability density function PyІx
Input: fixed luu of interest, sequence of m preceding observed windows [Lobe: obal I.
and the
behavioral features X for every E'
Output: The saliency map SO) over the observed windows
1: for each tbee E O do
2: Compute the feature mapping ((thh))) +- 1( x
x')
3: end for
4: Compute the saliency map S(foba)
(tobs))
density estimate of these. Following Equation 4 Loog simplifies the definition of the saliency map to S(x) = det(J)(z))d((tt)
(5)
given that monotonic transformation of the saliency map does not essentially alter the map.
3 CLOSED-FORM TEMPORAL SALIENCY FOR PROBABILISTIC FORECASTING
3.1 SETUP AND NOTATION
Given the relevance to forecasting tasks involving human behavior, we borrow the notation from the task formulation of Social Cue Forecasting (SCF) (Raman al. (021). Let tobs = [ol. 02, ..., oT] denote window of consecutively increasing observed timesteps, and tfus == f2.
denote an unobserved future time window, with fl > oT. Given a set of n interacting participants, let us denote their features over labs and liut respectively as X = [b E foba]]==1 and Y = [b]: 4 E f The vector bú encapsulates the multimodal cues of interest from participant i at time I, such as head and body posc, facial expressions, gestures, etc. The task is to forecast the density Pyix, denoting a distribution over possible future features given an observed sequence X of the same features. Here Y and X denote the multivariate random variables associated with the future and observed sequences respectively. While the original formulation of SCF dealt with conversation settings, it general enough apply other tasks such as pedestrian trajectory forecasting, where the features are simply the locations of individuals. In the rest of this work, for simplicity we denote the feature array an individual timestep in tobs and tru as X; and Y, respectively, even though they correspond to the same feature types. Given x, the focus of this work is to compute the saliency S (fobba) of an observed tobs towards the future occurring over a fixed choice of fuut-
3.2 DEFINING & IN TERMS OF THE UNCERTAINTY OVER THE FUTURE WINDOW fitut
The closed-form expression for information- baased spatial saliency in Equation 4 makes it explicit that the choice of the feature mapping determines the actual form of the saliency map. To extend the saliency framework to forecasting settings, we formalize the logical implication in [Equaaton I by mapping the window tobs the differential entropy of the model's predicted future distribution over litt. conditioned on the observed features X. That is, we define Φ Lobs + ((Y|X X), where the conditional differential entropy of Y given {X X} defined as
A(Y = X)
x)= = - py P (X PY
(XXXXXXXXX (YIX)dY.
(6)
Following Equation 6 our framework for computing the saliency map is summarized in Algorithm
Consider that a domain expert selects specific tuu corresponding to a high-order semantic behavior they wish to analyse. Examples include a speaking-tur -urn change (Keitel et al.) 2015 Levinson & Torreira 2015) an interaction termination (Bohus Horvitz, 2014] van Doomn 2018), or synchronous behavior event (Bilakhia et al. 2013). Given an underlying forecasting model and looking back in time before trut. we compute h( = X) for different observed multivariate features X corresponding to different locations of sliding tobs. The computed differential entropy values are then inserted into Equation to obtain the saliency of different tybs locations towards the future over the chosen fut-
5

---

Under review as a conference paper at ICLR 2023
Differential entropy possesses favorable properties that make it suitable choice é for computing the saliency map. First, the scale of the forecast density does not affect the resulting saliency map (see Cover & Thomas) Theorem 8.6.4):
h(aY) h(Y) + log |a|: for # 0, and
M(AY) = h(Y) + log (A)), when A is square matrix.
(8)
That is, scaling the distribution changes the differential entropy by only a constant factor. So the saliency map resulting from inserting the entropy into [Equation 5] remains unaffected since the Jacobian term only depends on the relative change in entropy across different choices of toss- Similarly, translating the predicted density leaves the saliency map unaffected (see Cover & Thomas Theorem 8.6.3):
h(Y + c) = h(Y).
3.3 COMPUTING X = X)
To compute the differential entropy of the future distribution, we need the density function py Typically, this density is modeled as a multivariate Gaussian distribution (Raman et al. 2021 Ranga puram et al. 2018 Salinas et al.) 2019a b). When the decoding of the future is non-autoregreesivve the parameters of the distributions for all luu are estimated in parallel. In these cases, the differential entropy has a closed-form expression. Assuming a d-dimensional predicted Gaussian distribution with mean fi and covariance matrix K, the expression for the entropy of a multivariate Gaussian distribution is given by (see Cover& & Thomas, Theorem 8.4.1)
4 = X) = h(Na(u, = log[(2 det det (K)].
(10)
When K is diagonal, so that the predicted distribution is factorized over participants and features, we can simply sum the log of the individual variances obtain the feature mapping Note that from Equation 10 for a multivariate Gaussian distribution, the differential entropy only depends on the covariance, or the spread of the distribution, aligning with the notion of differential entropy as a measure of total uncertainty. (See (Cover& & Thomas Tab. 17.1: Lazo La.i:Lazo & Rathie 1978) for closed-form expressions for large number of commonly employed probability density functions.)
Another common approach for inferring the future density function is the use of probabilistic autoregressive decoders (Ha Eck 2017 Raman et al. 2021 Salinas et al., 2019b Salzmann et al., 2021). Here, one possible decoding approach (Ha & Eck Salmmaal et al. 2021 involves
Salzmaan
taking a specific sample Y; from the density predicted each / Luus, and passing back input to the decoder for estimating the density at timestep + 1. Therefore, the density at L + 1 depends on the randomness introduced in sampling Y: Figure 2 illustrates the concept for two timesteps. Here, a single forecast would only output the shaded red distribution for Yz. This precludes the direct computation of h( h(Yi, Y2) that requires the full joint distribution PY1.2.. In such cases, we have two broad options: using a simplifying assumption to retain computational simplicity, or approximating the differential entropy by sampling.
The simpler option is to redefine sum the differential entropy of the individual densities estimated at each timestep as an approximation of the total uncertainty over the predicted sequence. Note that following the chain rule for differential entropy (see Cover & Thomas, Eq. 8.62), the joint entropy can indeed be written as the sum of individual conditionals. However, in general,
Ceature-mapping as Φ tobs EtEt... h <t X), i.e.
x
Σ MY.Y X) rEtra
MAMYE& c6, X).
(11)
tEtt
Employing this approximation (of summing the entropies across all timesteps) relies on the ob- servation that for autoregressive decoding, the parameters of the predicted distribution for Y, is computed as deterministic function of the decoder hidden state. That is, Y, is conditionally inde- pendent of Yo given the hidden state of the decoder 8; timestep t. The underlying assumption is that for a well-trained decoder, 8, encodes all relevant information from other timesteps to infer the distribution of Y, So at inference, despite being function of the single sample Y,-1, the predicted distribution conditioned on 8; provides a reasonable estimate of the the uncertainty in Y,.

---

Under review as a conference paper at ICLR 2023
The benefit of employing this assumption is that when each Y, is modeled using a density function that has a closed-form expression for differential entropy (Cover & Thomas] Tab. Lazo Lazo & Rathie, 1978). every item in the sum can be computed anlaytically, and obtain closed-form expression for the saliency map. Apart from multivariate Gaussian, the other common choice for modeling Y, is using a Gaussian mixture (Ha & Eck, 2017, Salzmann al. 2021). While closed-form expression for the differential entropy of a Gaussian mixture is not known, approximations that approach the true differential entropy can be obtained efficiently (Huber et al. 2008
Prape (1))
Pri.
Y2
Y1
Michalowiez et al., 2008: Zhang & Luo 2017) to directly compute the feature mapping Φ.
Yi
Figure 2: Illustrating predicted densities un- der the greedy autoregressive decoding ap- proach for two timesteps. For simplicity. we depict a joint Gaussian distribution and omit the conditioning on X everywhere.
What the differential entropy for Y2 does not have an analytical expression computationally efficient approx- imation? In such cases, h(Y X = X) can be estimated using sampling or other non-parametre approaches (Aigl
& Simandl [201 AAriel & Louzoun 2020; Beirlant et al. 1997 Brewer 2017). Such sampling- based
methods can provide approximations that converges to the true entropy.
4 EMPIRICAL VALIDATION OF THE PROPOSED FRAMEWORK
How do we empirically validate that our framework retrieves verifiable salient timesteps towards future outcome? Crucially, to avoid issues arising from qualitative visual assessment of saliency maps (Adebayo et al., 2018 [Barredo Arrieta et al. 2020), human-verifiable salient relationship between tobs and tuu is needed to serve as ground truth. However, this is non-trivia for real-world data; ground-truthing such saliency in behavior datasets is especially hard because there often exists both supporting and opposing evidence in social psychology for predictive relationships between behaviors (Kalma 1992. Levinson & Torreira| 2015). So we have chicken-egg problem surrounding evaluation on real-world data: if we had a way of computing such ground-trutt predictive saliency, we would not need this research to begin with. Furthermore, even if this is addressed, using real-world data would also entail training a forecasting model on the data. This model is bound to be arbitrarily imperfect, resulting in a correspondingly imperfect saliency map from our framework (garbage in, garbage out). So, when using an imperfect model, do we attribute any flaws in the saliency map to the explainability method to the imperfect underlying forecasts? This also makes it non-trivial to compare against other gradient and attention based XAI techniques that require training a specific model access its parameters. As such, for a fair empirical comparison, any candidate XAI method would need to i. compute saliency by changing observed timesteps one-at-a- time (counterfactual reasoning); and ii evaluate change in the information in the probablistic forecast to measure the same notion of saliency. No such directly comparable saliency-based explanation frameworks exist for probabilistic forecasts. The closest saliency-based method (Pan et al. 202T) computes saliency maps using random perturbations for non-probabilistic forecasts and requires access to model parameters.
Consequently, our experimental setup is as follows. First, we construct a synthetic dataset of conversation dynamics where high-order semantic predictive relationship exists between behaviors. Next we assume a perfect underlying forecasting model trained over this dataset: the best a well- trained model can do is to predict the true variance over plausible futures given observation. Note that the model does not need to have any conception of saliency or high-order semantic concepts. The task is to predict the distribution PY|x over low-level head pose features denoted by real-valued quaternions, matching the real-world scenario (Barquero et al. 2022 Raman et al.) 2021). Our goal is to evaluate if our framework empirically retrieves the timesteps that are salient by construction.
4.1 THE DATASET
We choose to synthesize turn-tat turn-taking dynamics in multi-party conversations, setting that has received extensive domain interest over the last decade (De Kok & Heylen 2009; Ishii et al.) 2013 2017 Keitel al. 2015 Malik al. 2020 [Petukhova & Bunt 2009 Rochet-Capellll & Fuchs 2014).
7

---

Under review as a conference paper at ICLR 2023
listener head-nod begins
observed
to
obs
obs
possible fatures
tiut
tfu
Figure 3: Illustrating the synthetic conversation dynamics dataset. Speakers are denoted in orange and listeners in green. For fixed tau we depict two preceding tobs windows. By construction, when observing a stable speaking turn over fobe. two valid futures are possible over tus. These correspond to turn handover the immediate left or right of the current speaker. Over t when listener nods to indicate the desire take the floor, the future over tuu becomes certain, corresponding to listener successfully taking over speaking turn. Here ta is consequently more salient than Lab towards forecasting the turn change over tia..
Specifically, we synthesize group conversation following established patterns of social behavior. First, the visual focus of attention of listeners is usually the speaker, while the speaker might look at different listeners over the speaking turn (Hung et al., (2008). Second, head gestures and gaze patterns are predictive of the next speaker (De Kok & Heylen 2009 Ishii et al., 2017: Malik et al.) 2020 Petukhova & Bunt 2009). We implement a simplified version of these dynamics as follows. The speaker rotates towards the geometric center of the formation after acquiring a speaking turn, while the listeners orient towards the speaker. A listener nods their head to indicate desire to acquire the floor, following which the current speaker rotates to look at the listener and hands over the speaking turn. We represent 3D head poses with quaternions given their common in representing human motion and pose (Pavllo et al., 2018 Raman et al. 202T). Concretely following the notation in subsection 3.1 que Qa Vy Qaj- We simulate the changes occur once clockwise and once anticlockwiss around the group so that each participant yields the floor once each to the participant on their immediate left and right. We provide code, the dataset consisting the head-pose and speaking status features, and an animated visualization in the Supplementary material.
4.2 COMPUTING TEMPORAL SALIENCY
0.10-
Quaternion Dim
0.05
By construction, for a human observer it is impos- sible to guess the future with certainty by simply observing the current speaker speaking: there exist two valid examples of the future head behavior in the dataset for such sequence. Only the commence- ment a head nod by a listener makes the future turn handover certain. Fiiure 3jillustrates this mech- anism. Therefore, the timestep corresponding to a head nod is salient towards forecasting the features over future window involving a turn change in this dataset. Can our framework identify this timestep as salient? A perfect model would predict the true variance over the two valid future quaternion trajecto- ries. This variance vanishes when the future becomes certain as soon as one of the listeners starts nodding. We model the future distribution using Gaussian function for simplicity (setting std. to 10 10 for the single future), but more complex distribution that predicts the appropriate change in variance would also work in practice.
Entropy
-25
listener head nad begins
-20-
Saltency
130
335
140
145
150
Frame
Figure Computing Saliency. We plot the quaternion dimensions @x and qy for the listener that nods over t2r. in Figure (top) The observa- tion of the head-nod beginning in frame 138 makes the future over frames 183 228 (frut) certain. This is reflected in the reduction the mean en- tropy over future values of all participants (middle). The saliency map obtained using our framework (bottom) correctly identifies the beginning of the head-nod frame 138 as the salient timestep to- wards forecasting the future over trus.
8

---

Under review as a conference paper at ICLR 2023
We now implement Algorithm I as follows. We identify window where turn change occurs in the data (frames 183- 183-228) and denote this 45 frame window as the tfut of interest. While we manually identify an interesting event for illustration, such window could also correspond to an interesting prediction by a model. We generate a set of candidate toss by sliding a 30 frame window over horizon of 100 frames prior fut. with stride of 1 frame. For every observed Lobs, we fit Gaussian density to the quaternion and speaking status features of all participants over the futures that can occur during fut- We then set the entropy of this Gaussian density as the feature Φ for that tobs. We obtain the saliency map using Equation 5] and plot the listener features, entropy. and saliency across timesteps in Figuree 4] The ground truth salient timesteps here are frames 138 and 139 where the head nod begins. Our framework retrieves these timesteps as salient with 100% accuracy, empirically validating our framework, Note that once the nod is already in motion, the saliency drops as expected since the future is then already certain given the data. So observing head nod in motion does not provide any additional information about the future.
DISCUSSION AND CONCLUSION: SALIENCY & XAI WITH DOMAIN EXPERTS IN THE LOOP
We have proposed computational framework obtaining counterfactual explanations of model fore- casts and established its link to a principled notion of bottom-up task-agnostic saliency. Specifically, we obtain convenient closed-form expression of the saliency of observed timesteps towards future outcome for commonly used probability density functions to represent forecasts (Ha & Eck] 2017: Raman et al., 2021; Rudenko al., 2020) [Salmaann et al. [02I). We empirically validate our framework using a synthetic experiment that allows for quantitative validation, avoiding estab- lished issues arising from observer biases in visual assessment of saliency maps (Adebayo al.)
2018: Barredo Arrieta et al. 2020 Our goal with this work is to aid domain experts in forming new hypothses from real-world complex data. Given a model that has learned structural predictive relationships from data, our framework identifies timesteps salient towards the model's 's forecasts through counterfactual reasoning. We thereby envision a coman--n-the-loog XAI methodology where the domain expert can form hypotheses surrounding the features occurring at these salient timesteps. These hypotheses can then be verified through subsequent controlled experiments.
Loog's S (011) unifying framework subsumes all forms of saliency, although identifying the appro- priate @ for a specific domain is non-trivial. In this work we have established both theoretically and empirically how expressing ø in terms of the information about the future enables principled counterfactual reasoning in forecasting settings. Nevertheless, we reiterate that the salient timesteps retrieved by our framework ought be treated as candidate causes until subsequently examined along with a domain expert. Our stance on human-in-the-loop XAI also aligns with research on saliency- and general XAI in other domains (Atrey et al. 2020 Joshi al. 202T
In principle, when it is possible to have access to the true py xX the salient timesteps identified by our framework reflect the true predictive structural relationships captured by the underlying model across the entire data. However, estimating this density analytically entails identifying the multiple futures in the data corresponding to every occurrence of the same observed features. In practice, subtle variations in behaviors and sensor measurement errors make it infeasible to estimate Py analytically, so a model trained to capture generalized patterns from the given data. In these cases, our framework identifies the sequences that the model considers salient for its forecasts given the data. Consequently, subsequent causal analysis of the features over the salient timesteps is crucial, especially in the healthcare and human behavior domains to avoid potential prejudices against certain behaviors, or worse, misdiagnoses of affective conditions.
REFERENCES
Julius Adebayo. Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Vida Adeli, Ehsan Adeli, Ian Reid, Juan Carlos Niebles, and Hamid Rezatofighi. Socially and Contextually Aware Human Motion and Pose Forecasting, MXXv:2007.06843 [cs], July 2020.

---

Under review as a conference paper at ICLR 2023
Chaitanya Ahuja, Shugao Ma. Louis-Philippe Morency, and Yaser Sheikh. To React to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations. arXiv:1910.0218/ [es]. October 2019.
Jiří Aigl and Miroslav Simandl. Differential entropy estimation particles. IFAC Proceedings Volumes, 44(1): 11991-11996, January 2011. ISSN 1474-6670. doi: 10.3182220110828-61TT. 1000000000000011100000000004044 1002.04444.
Ahmed M Alaa and Mihacla der Schaar. Attentive state-space modeling of disease progression. Advances in nearal information processing systems, 32, 2019.
Gil Ariel and Yoram Louzoun. Estimating differential entropy using recursive copula splitting. Entropy. An International and Interdisciplinary Journal of Entropy and Information Studiex, 22(2):236, 2020.
Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning, February 2020.
David Baehrens. Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, and Katja Hansen. How to Explain Individual Classification Decisions. pp. 29.
Sarah Adel Bargal, Andrea Zunino, Donghyun Kim, Jianming Zhang, Vittorio Murino, and Stan Sclaroff.
Excitation Backprop for RNNs. arXiv: [ex], March 2018.
German Barquero, Johnny Núñez. Sergio Escalera, Zhen Xu, Wei-Wet Tu, Isabelle Guyon, and Cristina Palmero. Didn't see that coming: A survey on non-verbal social human behavior forecasting. MxXiv:2203.02488 [cs], March 2022.
Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barhado, Salvador Garcia, Sergio Gil-Loppez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible Information Fusion, 58:82-188 115. 2020. ISSN 1566-2535. doi: 10.1016/j.ifffus.2000 2019..2.12.
Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Sanjay Bilakhia, Stavros Petridis, and Maja Pantic. Audiovisual Detection of Behavioural Mimicry. In 2013 Humaine Association Conference Affective Computing and Intelligent Interaction, pp. 123-128, Geneva, Switzerland, Seplember 2013. IEEE, ISBN 978-0-7695-5048-0, doi: 10.1 1011///I111111....
Dan Bohus and Eric Horvitz. Managing Hamann-Robot Engagement with Forecasts and... Hesitations. Proceedings of the 16th International Conference on Multimodal Interaction, pp. 8, 2014.
WM Boothby, An introduction to differential geometry and riemannian manifolds, 1975.
Brendon Brewer. Computing entropies with nested sampling. Entropy. An International and Interdisciplinary Journal of Entropy and Information Studies, 19(8),42,, 2017.
Neil DB Bruce Features that draw visual attention: An information theoretic perspective. Neurocomputing, 65: 125-133, 2005.
Sandra Carrasco, David Fernández. Llorea, and Miguel Ángel Sotelo. SCOUT: Socially-COnsistent and Tandable Graph Attention Network for Trajectory Prediction of Vehicles and VRUs. arXiv:2102.06361 [cs]. May 2021.
Edward Choi, Mohammad Taha Bahadori, Joshua A. Kulas, Andy Schuctz, Walter F. Stewart, and Jimeng Sun. RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism. arXiv: arXiv:1608.00 010.6008.000.0 05745 [cs], August 2016.
Thomas M Cover and Joy A Thomas. Elements of Information Theory.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances neural information processing systems, 30, 2017.
Iwan De Kok and Dirk Heylen. Multimodal end-of-trm prediction in multi-party meetings. In Proceedings of the 2009 International Conference on Multimodal Interfaces, pp. 91-98, 2009.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3429-3437, 2017.
Dashan Gao and Nuno Vasconcelos. Bottom-up saliency is discriminant process. In 2007 IEEE 11 International Conference on Computer Vision, pp. 1-6. IEFE, 2007.
10

---

Under review as a conference paper at ICLR 2023
Jose A. Garcia, Joaquín Valzivaa, Xose R Fdee- Vidal, and Rosa Rodriguez-Sancher Information theoretic measure for visual target distinctness. IEEE Transactions Pattern Analysis and Machine Intelligence, 23 (4):362-383, 383, 2001.
Thomas Gilles, Stefano Sahatini, Damitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. THOMAS: Trajectory Heatmap Output with learned Aglti-Agent Sampling. arXiv:2110.06607 [cx]. Jamuary 2022.
David Ha and Douglas Eck. A Neural Representation of Sketch Drawings. arXiv:1704.03477 [cs, stat), May 2017.
Kun Huang and Shenghua Gao. Image saliency detection via multi-scale itcrative CNN. The Visual Computer, 36(7):1355-1333 1367, 2020.
Yingfan Huang. Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. STGAT: Modeling Spatial-Tempont Interactions for Human Trajectory Prediction. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 6271-6280, Scoul, Korea (South), Octoher 2019. IEEE ISBN 978-1-72814-800-8. doi: 1011009..22220000006
Marco F. Huber, Tim Bailey, Hugh Durrant- Durrant-Whyte, and Uwe D. Hanebeck. On entropy approximation for Gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 181-188188 188, Scoul, August 2008. IEEE. ISBN 978- 978-1-4244-2243-5 doi: 10. 10.1109//F.2008.. 4648062.
Hayley Hung. Dinesh Babu Jayagopi, Sileye Ba, Jean-Marc Odobez, and Daniel Gatica-Perez. Investigating automatic dominance estimation in groups from visual attention and speaking activity, pp. 233, ACM Press, 2008. ISBN 978-1 177858- 198-9, doi: 10.1145//422322.145444
Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Masafumi Matsuda, and Junji Yamato. Predicting next speaker and timing from gaze transition patterns in multi-pane party meetings. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 79-86, 2013.
Ryo Ishii, Shiro Kumano, and Kazuhiro Otsuka. Prediction of Next-Uterance Timing using Head Movement in Multi-Party Meetings. In Proceedings of the 5th International Conference on Human Agent Interaction, HAI 17, pp. 181-187, New York, NY, USA, October 2017. Association for Computing Machinery. ISBN 978 11----5-3---1333 doi: 10.1145/3125739.3222
Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi. Benchmarking Deep learning Interpretability in Time Series Predictions. arXiv:2010. 13924 [cs, stat], Octoher 2020.
Laurent Itti and Pierre Baldi. principted approach detecting surprising events in video. In 2005 IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition (CVPR'05) volume 1, pp. 631-637, IEEE, 2005.
Laurent Itti and Christof Koch. Computational modelling of visual attention. Nature reviews neursccience, 2(3): 194-203. 2001.
Boris Ivanovic, Yifeng Lin, Shubham Shrivastava, Punarjay Chakravarty, and Marco Pavone. Propagating state uncertainty through trajectory forccasting, arXiv preprint 1:X0v:211..03267. 2021.
Lai Jiang, Zhe Wang, Mai Xu, and Zulin Wang. Image saliency prediction in transformed domain: A deep complex neural network method. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp.
2019.
Shalmali Joshi, Sonali Parbhoo, and Finale Doshi-Velez. Pre-emptive learning-tor to-tg-tto-ffee for sequential medical decision-making under uncertainty, September 2021.
Akko Kalma. Gazing in triads: A powerful signal floor apportionment. British Journal of Social Psychology. 31(1):21-39, March 1992.
Anne Keitel, Moritz M Daum, et al. The use of intonation for tum anticipation in observed conversations without visual signals as source of information. Frontiers in psychology, 6( 6(108):265-273, 2015.
Sebastian Lapuschkin. Stephan Waldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus- Robert Müller. Unmasking Clever Hans Predictors and Assessing What Machines Really Learn. Nature Communications, 10(1):1096, December 2019. ISSN 2041-1723, doi: 10. 017-4441---11--8887-788
A Verdugo Laze and P Rathie. On the entropy of continuous probability distributions (corresp.). IEFE Transactions Information Theory, 120- 1978.

---

Under review as a conference paper at ICLR 2023
Stephen C. Levinson and Francisco Tomeiaa. Timing in turn-taking and its implications for processing models of language. Frontiers Psychology, 6, June 2015. ISSN 1664-1078 doi: 1.3389/fpsyg.2015..00
Guanbin Li and Yizhou Yu. Visual saliency detection based on multiscale deep CNN features. IEEE transactions on image processing. 25(11):5012-500 5024, 2016.
Longyuan Li, Junchi Yan, Xiaokang Yang, and Yaohui Jin. Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting. arXiv:2102 102.003977 [ex, stat], January 2021.
Xiaohui Li, Huchuan Lu, Lihe Zhang, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via dense and sparse reconstruction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 2976-2983, 2013.
Bryan Lim, Sercan Arik, Nicolas Loeff, and Tomas Pfister. Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. arXiv: 1912.09363 [cs, stat], September 2020.
Zhang Lingyun, Matthew H Tong, and Garrison Cottrell, Information attracts attention: probabilistic account of the cross-race advantage in visual search. In Proceedings of the Annual Meeting of the Cognitive Science Sociery, volume 29, 2007.
Zachary C. Lipton. The Mythos of Model Interpretability, March 2017.
Zachary C. Lipton and Jacob Steinhardt. Troubling Trends in Machine Learning Scholarship. July 2018.
Marco Loog. Information theoretic preattentive saliency: closed-form solution. 201/ IEFE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1418-1424, Barcelona, Spain, November 2011. IEEE. ISBN 978-1-4673-00663- 978-1 978---0--6--7----7777777 978-1-4633-0061-
doi: 10. 10911////C/
6130417.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Usman Malik, Julien Saunier, Kotaro Funakoshi, and Alexandre Pauchet. Who speaks next? Turn change and next speaker prediction multimodal multiparty interaction. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 349-354. IEEE, 2020.
Joseph V. Michalowiez, Jonathan M. Nichols, and Frank Bucholtz. Calculation of Differential Entropy for Mixed Gaussian Distribution. Entropy, 10(3):200-206, September 2008. ISSN 1099-4300, doi: 10.3390/
100300y-100302200.
Tim Miller. Explanation artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:
1 1-38, February 2019, ISSN 0004-3702. doi: 10. a.00///..riint.01...... 8.00007.
Abduallah Mohamed, Kun Qian. Mohamed Elhoseiny, and Christian Claudel. Social-STGCNN: A Social Temporal Graph Convolutional Neural Network for Human Trajectory Prediction. arXiv:2002. /cs], February 2020.
Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
Felix Mujkanovic, Vanja Doskoč, Martin Schimeck, Patrick Schäfer, and Tobias Friedrich. timeXplain A Framework for Explaining the Predictions of Time Series Classifiers. WxXi:2007.07660 [cs, stat]. July 2020.
Tan Viet Tuyen Nguyen and Oya Celiktutan. Context-Aware Human Behaviour Forecasting in Dyadic Interac- tions. Understanding Social Behavior in Dyadic and Small Group Interactions, pp. 1-19, 2022.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv:1905.10433 [es, stat], February 2020.
Qingyi Pan, Wenbo Hu, and Ning Chen. Two Birds with One Stone: Series Saliency for Accurate and Interpretable Multivariate Time Series Forecasting. In Twenty-Ninth International Joint Conference Artificial Intelligence, volume 3, pp. 2884 2891, August 2021. doi: 0.24963//jcai.2221/339,
Madhurima Panja, Uttam Kumar, and Tanujit Chakraborty. An Interpretable Probabilistic Autoregressive Neural Network Model Time Series Forecasting. arXiv:2204. 09640 [cs, stat], April 2022.
Dario Pavllo, David Grangier, and Michael Auli. QuaterNet: Quaternion-based Recurrent Model for Human Motion. 805. 06485 [cs]. July 2018
Judea Pearl, Causality. Cambridge university press, 2009.
12

---

Under review as a conference paper at ICLR 2023
Judea Pearl and Dana Mackenzie. The Book Why: The New Science of Cause and Effect. Basic books, 2018.
Volha Petukhova and Harry Bunt. Who's next? Speaker-sclection mechanisms in multiparty dialogue. In Workshop on the Semantics and Pragmatics of Dialogue, 2009.
Chirag Raman, Hayley Hung, and Marco Loog. Social Pracesses: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues. arXiv:2107.13576 [cs]. July 2021.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forccasting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Amélie Rochet-Capellan and Susanne Fuchs. Take breath and take the turn: How breathing meets turns in spontaneous dialogue. Philoxophical Transactions of the Royal Society B: Biological Sciences, 369( (1658) December 2014. ISSN 0962-8436. doi: 10.1098/rstb.201 2tt.:sstt.201....33. 13.9999..
Ruth Rosenholtz. A simple saliency model predicts number of motion popout phenomena. Vision research, 39 (19):3157-3163, 1999.
Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research, 39(8)8895-935, 2020.
David Rügamer, Philipp F. M. Baumann, Thomas Kneib, and Torsten Hothorn. Probabilistic Time Series Forccasts with Autoregressive Transformation Models. arXiv:21 Jes], February 2022.
David Salinas, Michael Sohhke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank Gaussian Copula Processes. In H. Wallach, H. Larochelle, A. Beygelz- imer, dAlché-Bue. E. Fox, and Garnett (eds.), Advances in Neural Information Processing Systems. volume 32. Curran Associates, Inc., 2019a.
David Salinas, Valentin Flunkert, and Jan Gasthaus. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. arXiv: 10 fes, stat], February 2019b.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- Trajectory Forecasting With Heterogeneous Data. arXiv:2001.0300 [cs], January 2021.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. International conference on machine learning. pp. 3145-3153. PMLR, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint 1rXiv:1312.6034, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: (rXXiv1706.03822. 03825, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pp. 3319-3328. PMLR, 2017.
Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Clinical intervention prediction and understanding using deep networks. arXiv preprint arXiv: 1ee.:-005..0000 705.08498. 2017.
Antonio Torralba. Modeling global scene factors in attention. JOSA A, 20(7):1407-1418, 2003.
Felix van Doorn Rituals Leaving: Predictive Modelling of Leaving Behaviour in Conversation, Master of Science Thesis, Delfi University of Technology. 2018
AHC vanderHeijden. Perception for selection, selection for action, and action for perception, 1996.
Anirudh Vemula, Katharina Muelling, and Jean Oh. Modeling cooperative navigation in dense human crowds. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1685-1692. IEEE, 2017.
Kevin N Walker. Timothy F Cootes, and Christopher Taylor. Locating salient object features. In BMVC, volume 98, pp. 557-566. Citeseer, 1998.
Taiping Yao, Minsi Wang. Bingbing Ni, Huawei Wei, and Xiaokang Yang. Multiple Granularity Group Interaction Prediction. In 2018 JEFE/CVF Conference Computer Vision and Pattern Recognition, pp. 2246-2254, Salt Lake City, UT, June 2018. TEEF. ISBN 973-1-9386- doi: 10. 1109//VR.20188.000
13

---

Under review as a conference paper at ICLR 2023
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pp. 818-833. Springer, 2014.
Wenyuan Zeng. Shenlong Wang. Renjie Liao, Yun Chen, Bin Yang, and Raquel Urtasun. DSDNet: Deep Structured self- Saff-Driving Network. arXiv:2008. 06041 [cs], August 2020.
Chenchen Zhang and Yuan Luo. Approximating the differential entropy of gaussian mixtures. In GLOBECOM 2017-2017 IEEE Global Communications Conference, pp. 1-6. IEEE, 2017.
Pu Zhang, Wanli Ouyang. Pengfei Zhang, Jianru Xue, and Nanning Zheng. SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction. urXiv:1903.02793 [cs], March 2019.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, Congcong Li, and Dragomir Anguelov. TNT: Target-driveN Trajectory Prediction. PrXiv:2008.08294 [cs], August 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Patern Recognition, pp. 2921-2929 2929, 2016.
14

---

Under review as a conference paper at ICLR 2023
A BROADER RELATED WORK: EXPLAINABLE METHODS FOR TIME-SERIES TASKS ACROSS DOMAINS
The larger focus of explainability techniques involving time-series data has been the task of classifying time-series. The goal has been to estimate the relevance each input feature at a given timestep towards each output class. Here, saliency approaches often overlap with techniques developed for image data and can be categorized into:
Gradient-Based Techniques. The broad approach involves evaluating the gradient of the output class with respect to the input (Baehrens et al.). Several variants have been proposed (Bargal et al. 2018 Lundberg & Lee
2017 Shrikumar et 2017 Smilkov et al. 2017: Sundararajan et 2017).
Perturbation-Bassd Techniques. The idea is to examine how the output changes in response to some perturbation of the input. Perturbations are implemented by either occluding contiguous regions of the input (Mujkanovic et al.] [2020] Zeiler & Fergus] 2014): performing an ablation of the features
(Suresh et al. 2017); or randomly permuting features (Molnar 2020). [smail et al. (2020) provide
benchmark of subset of these techniques.
Attention- Based Techniques. These incorporate an attention mechanism into the model that trained to attribute importance to previous parts of the input towards output at each timestep. Such techniques have been specially employed for healthcare data, with early methods applying reverse-time attention (Choi et al. (2016), and later ones applying the attention to probabilistic state-space representations (Alaa & van der Schaar 2019).
Some of these broad ideas have been applied to the regression setting to make interpretable forecasts of future time-series features. [im et al. (2020) leveraged self-attention layers for capturing long-term dependencies. Pan al. (2021) recently proposed computing saliency as a mixup strategy between series images and their perturbed version with learnable mask for each sample. They view saliency in terms of minimizing the mean squared error between the predictions and ground -rrutss for particular instance. Focusing on the univariate point-forecasting problem, Oreshkin et al. (2020) proposed injecting inductive biases by computing the forecast as a combination of trend and seasonality model. They argue that this decomposition make the outputs more interpretable.
Developing explainable techniques for the probabilistic forecasting setting remains largely unexplored and subject to non-overlapping notions of explainability. |Rügamer et al. (2022) transform the forecast using predefined basis functions such as Bernstein polynomials. They relate interpretability to the coefficients of these basis functions (a notion similar to that Oreshkin et al. (2020)). Panja et al. (2022) embed the classical linear ARIMA model into a non- non-linear autoregressive neural network for univariate probabilistic forecasting. As before, the explainability here also stems from the 'white-box' nature of the linear ARIMA component. Li et al. (021) propose an automatic relevance determination network identify useful exogenous variables (i.e. variables that affect the forecast without being a part of the time-series data). To the best of our knowledge, saliency-based methods have not yet been considered within this setting.


Reviews:
[{"review_id": "c0PgIyt14TO", "reviewer": "Reviewer_YgxY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to compute temporal saliency in probabilistic forecasting. The saliency of each timestamp in time-series data is measured based on its gradient w.r.t. the output entropy. Authors construct a synthesized dataset to verify the effectiveness of the proposed method.", "summary_of_the_review": "This paper focuses on an important topic. However, several concepts are not formulated in a rigorous way, and some claims are not proved or verified. There are also problems concerning the formulation of the proposed method. Moreover, experiments in this paper are insufficient. Therefore, I recommend rejection.", "strength_and_weaknesses": "\n[Strength]\n1. This paper focused on an important topic, i.e., explaining the model’s forecast on time-series data.\n\n[Weakness]\n1. The introduction of this paper is a little dense, and several concepts are not formulated mathematically, including “associative”, “interventionist”, “counterfactual”, “speculation”, “surprisal”, etc. In the field of explainable AI, researchers are expected to define the mentioned concepts in rigorous mathematical language. Maybe the above concepts are meaningful, but none of them are analyzed mathematically. Therefore, the discussion on these concepts will hardly make sense to readers. Please provide formal mathematical definitions for these concepts, and then discuss their relationship to the proposed method.\n\n2. There lacks evidence to support the claim that “the proposed method is grounded in preattentive human cognition.” First, as mentioned above, please mathematically define components in preattentive human cognition. Second, please provide a theoretical proof to verify that the proposed method indeed aligns with the theory of neuroscience.\n\n3. The actual setting of $\\phi$ in Section 3.2 is inconsistent with the introduced physical meaning of $\\phi$ in Section 2. In Section 2, $\\phi$ is defined as a mapping from image locations to image features. In this case, modeling the distribution of image features as $p_\\Phi$ makes sense. It is also meaningful to measure the amount of information using $-\\log p_\\Phi$. However, in Section 3.2, $\\phi$ is defined as a mapping to the entropy of the output. Then, what is the purpose of modeling “the distribution of entropy as $p_\\Phi$?” The entropy has its own physical meaning for measuring uncertainty, but modeling the distribution of entropy is meaningless. I suggest authors provide more clarifications on this issue.\n\n4. Authors are encouraged to discuss more about the link between counterfactual reasoning and their method of saliency.\n(1) First, I suggest authors mathematically define the “counterfactual explanation” for forecasting settings, instead of only giving a description as “simulating alternative causes to see whether the event still happens” (in Table 1 of Section 1.1). In this way, the connection between the proposed method and counterfactual reasoning can be explained in a more rigorous way.\n(2) Second, I do not find experimental results on such counterfactual explanations in Section 4, though authors mention in Section 1.3 that they “examine changes in the future density predicted by M in response to different realizations of real-world sequences $t_{obs}$.” Please conduct some experiments to verify that the proposed method is able to provide counterfactual explanations. \n\n5. Experiments in this paper are insufficient.\n(1) No baseline method is compared. Authors are suggested to compare the proposed method with existing widely used attribution methods, including Saliency [cite 1], Occlusion [cite 2], Input x Gradient [cite 3], LIME [cite 4], LRP [cite 5], Shapley value [cite 6], DeepSHAP [cite 7], etc.\n(2) Authors also do not verify the effectiveness of the proposed method on benchmark datasets for the time-series forecasting task.\n\n6. About experimental details.\n(1) What is the specific form of the future and observed features, $i.e.$, $X$ and $Y$, corresponding to Algorithm 1?\n(2) Which one of the three approaches introduced in Section 3.3 is chosen for the experiment to compute the differential entropy?\n(3) Moreover, how well is the estimation of the distribution Y|X based on the Gaussian distribution and autoregressive decoders? Please provide a metric to quantify the estimation error, and verify the accuracy of such estimation.\n(4) The saliency map in Figure 3 is problematic. Figure 3 shows that the saliency of frames where the nod is not in motion has a zero saliency value. However, the information of the position of the current speaker also affects possible futures. Thus, the saliency of frames where the nod is not in motion may be low, but not zero.\n(5) Minor: “Saliency” in the last third line of page 8 is supposed to be “entropy”. According to Figure 3, when the nod begins, the entropy drops, and the saliency increases.\n\n7. About the limitation of the proposed method.\n(1) For the time-series forecasting task, the proposed attribution method simply computes the square of the gradient’s L2-norm. Is it equivalent or similar to existing gradient-based explanation methods, such as Saliency [cite 2]?\n(2) In Section 2, $\\phi$ is restricted to a differentiable function. However, the mapping from image position to feature vector is usually not differentiable, because the image position is a discrete variable. Please provide some clarification for this issue.\n\n[cite 1] Simonyan, Karen, Andrea Vedaldi, and Andrew Zisserman. \"Deep inside convolutional networks: Visualising image classification models and saliency maps.\" arXiv preprint arXiv:1312.6034 (2013).\n\n[cite 2] Zeiler, Matthew D., and Rob Fergus. \"Visualizing and understanding convolutional networks.\" European conference on computer vision. Springer, Cham, 2014.\n\n[cite 3] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. In arXiv: 1605.01713, 2016.\n\n[cite 4] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “Why should I trust you?” explaining the predictions of any classifier. In KDD, 2016.\n\n[cite 5] Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, Klaus-Robert Muller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks (ICANN), 2016.\n\n[cite 6] Lloyd S Shapley. A value for n-person games. In Contributions to the Theory of Games, 2(28): 307–317, 1953.\n\n[cite 7] Lundberg, Scott M., and Su-In Lee. \"A unified approach to interpreting model predictions.\" Proceedings of the 31st international conference on neural information processing systems. 2017.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aRipErdMgO", "reviewer": "Reviewer_7H98", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to provide salient time steps for probabilistic time-series forecasting tasks. It builds on Miller's framework of explanations and establishes a link between counterfactual reasoning and saliency-based explanation technique; it proposes a unifying definition of information-theoretic saliency for forecasting tasks. Also, the paper empirically shows that the proposed method recovers the true salient timesteps for a synthetic dataset.", "summary_of_the_review": "This paper proposes a new method to obtain salient timesteps for forecasting tasks. The proposed method is simple and has a closed-form solution. However, the paper lacks rigorous empirical evaluation with existing methods to demonstrate the effectiveness of the proposed method.\n\n[After rebuttal] I do not think the updated draft has addressed the concerns about the empirical evaluation part. Therefore, I keep my original score.", "strength_and_weaknesses": "Strength:\n\nThis paper proposes a reasonable and principle solution to provide salient time steps for time-series forecasting tasks.\n\nIt considers a probabilistic forecasting model which is critical for robust predictions.\n\nThe proposed method does not need explicit training but obtains a closed-form solution which easy to compute.\n\nWeakness:\n\nThe paper lacks a quantitative evaluation of the quality of the salient maps. \n\nThe paper did not compare with existing methods, e.g., gradient-based methods and attention-based methods. They are quite straightforward to be extended to the forecasting tasks.\n\nMore datasets are expected to show the effectiveness of the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "NjPV1ZyMMbj", "reviewer": "Reviewer_PbXQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission provides a new information-theoretic approach to inferring salient temporal sequences for counterfactual explanations. Conceptually the authors closely follow the organization framework in (Miller, 2019), partitioning kinds of explanations into analogous categories to the rungs in Pearl's causal hierarchy. The authors go on to present and justify the use of conditional differential entropy as the basis for constructing such a saliency measure. Finally, the authors show the application of the proposed approach on a synthetic conversation dynamics dataset.", "summary_of_the_review": "Overall the paper is well written, and presents a well motivated and plausible approach to determining temporal saliency. I have some concerns that currently make it challenging to recommend acceptance, but I am also happy to revise my opinion during the discussion.", "strength_and_weaknesses": "Strengths:\n\nOverall the paper is extremely well written, and a real pleasure to read. Very nicely done! In particular this submission does a very nice job of arguing why one should use the approach presented, and not just how it works. I suppose we should expect nothing less from a paper about causal explanations.\n\nWeaknesses:\n\nMy primary concern with this paper is in the empirical evaluation. While I understand that the problem statement itself is somewhat new, it is still very important to compare the proposed method against (even potentially very weak) baselines. Otherwise, it is impossible to determine whether the proposed method actually delivers on its promise. Additionally, while the figures presented are helpful for their explanatory power, it is very important to somehow quantify performance in terms of some metric. Unfortunately I don't have any concrete suggestions on such a metric.\n\nWhile it's not necessarily a weakness per se, I am somewhat concerned about the use of the term \"counterfactual\" here when the authors do not hold exogenous noise constant between factual and counterfactual worlds, as described in (Pearl, 2009) Chapter 7. It would be very helpful for the authors to clarify exactly how the abduction-action-prediction process ala Pearl is reflected in this work. Related, it would helpful to discuss some of the common issues in counterfactual reasoning here, e.g. necessary assumptions for identifiability.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6e47q-wkfQ", "reviewer": "Reviewer_bLo2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a theoretical framework to explain probabilistic forecasting using a saliency map. They start from Miller's framework to link counterfactual reasoning and saliency-based XAI techniques. Then, they extend the generic notion of saliency to the probabilistic forecasting setting by inferring a closed-form solution for commonly used density functions. The proposed framework is qualitatively evaluated on one synthetic dataset.", "summary_of_the_review": "The work is extremely abstract and theoretical and sometimes hard to follow. Some practical examples would greatly increase the readability and ease of understanding. The authors should better explain possible applications of the proposed approach and recap their contribution and future work in a Conclusion section. \nThe proposed framework is tested qualitatively on only one dataset; therefore, it is quite hard to assess its performance and usefulness in real-world applications. The authors should at least provide some quantitative metrics on their synthetic dataset, comparing their framework with other approaches or, if not possible, at least with random or naive approaches.\n\nOther comments:\n- page 3: \"Our framework addresses all the aforementioned concerns we identified with saliency-based approaches in forecasting: (i) the counterfactuals or alternate 'causes' in our framework pertain to real preceding windows of actual observed features, and are therefore consistent with the underlying data generation process and semantics preserving\". To me, this statement is not obvious. Sampling perturbations from past observations does not ensure semantic consistency with future observations. For example, a past observation could only be consistent in certain time series timesteps around specific patterns.\n- pag 5, 3.2: \"salent\" instead of \"salient\".\n\nIn summary, the work is fascinating, but the evaluation part is extremely lacking, and its applicability in real-case scenarios is not adequately discussed.", "strength_and_weaknesses": "Strengths:\n- XAI for time series forecasting is an extremely interesting and underexplored topic\n- The authors try to build a formal theory behind saliency maps for time series forecasting\n\n\nWeaknesses:\n- The work is extremely abstract and theoretical and sometimes hard to follow\n- The proposed framework is tested qualitatively on only one dataset \n- Conclusions are missing", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject