SlowGuess/NeurIPS_2023_OCR huggingface dataset:
right. Consequently, for a  $t_{\mathrm{obs}}$  containing the pedestrians approaching each other in a straight line, the predicted distribution  $p_{Y|X}$  over  $t_{\mathrm{fut}}$  encapsulates both possibilities of each pedestrian passing to the left as well as the right of the other. So the entropy  $h(Y|X = X)$  is high for this  $t_{\mathrm{obs}}$ . Only once  $M$  is fed as input with the trajectories from the  $t_{\mathrm{obs}}$  containing the pedestrians choosing one of the two directions to pass, the predicted  $p_{Y|X}$  is certain in terms of the pedestrians continuing along the chosen direction. (Note that in this case, we assume  $M$  has been trained by maximizing likelihood over the dataset containing only these two direction changes for avoiding collision.) Consequently, the entropy  $h(Y|X = X)$  drops only once this moment of the pedestrians committing to a direction is seen by the model and would be considered salient for our algorithm.
4.3 Computing  $h(Y|X = X)$
Typically, the density  $p_{Y|X}$  is modeled as a multivariate Gaussian distribution [4, 32-34]. When the decoding of the future is non-autoregressive, the parameters of the distributions for all  $t \in t_{\mathrm{fut}}$  are estimated at once, and the differential entropy has a closed-form expression, given by (see Cover and Thomas [35, Theorem 8.4.1])
$$
h (Y | X = \boldsymbol {X}) = h \left(\mathcal {N} _ {d} (\boldsymbol {\mu}, \boldsymbol {K})\right) = \frac {1}{2} \log \left[ (2 \pi e) ^ {d} \det (\boldsymbol {K}) \right]. \tag {6}
$$
A common choice is to set  $\pmb{K}$  to be diagonal, i.e. the predicted distribution is factorized over agents and features. In this case, we can simply sum the log of the individual variances to obtain the feature mapping  $\phi$ . Note that from Equation 6, for a multivariate Gaussian distribution, the differential entropy only depends on the covariance, or the spread of the distribution, aligning with the notion of differential entropy as a measure of total uncertainty. (See [35, Tab. 17.1; 36] for closed-form expressions for a large number of commonly employed probability density functions.)
In cases where probabilistic autoregressive decoders are used [4, 33, 37, 38], we do not have access to the full joint distribution  $p_{Y_{f1},\dots,Y_{fT}|X}$  for the timesteps in  $\pmb{t}_{\mathrm{fut}}$ . This is because inferring the density function  $p_{Y|X}$  often involves sampling: a specific sample  $\widehat{\pmb{Y}}_t$  is taken from the predicted density at each  $t \in \pmb{t}_{\mathrm{fut}}$ , and passed back as input to the decoder for estimating the density at timestep  $t + 1$  [37, 38]. Therefore, the density at  $t + 1$  depends on the randomness introduced in sampling  $\widehat{\pmb{Y}}_t$ . Figure 2 illustrates the concept for two timesteps. Here, a single forecast would only output the shaded red distribution for  $Y_{2}$ . In such cases, computing the joint entropy  $h(Y_{1},Y_{2})$  directly is challenging in the absence of the full joint distribution  $p_{Y_1,Y_2}$ .
To address this, we have two options. The simpler option is to redefine our feature-mapping as  $\phi : \pmb{t}_{\mathrm{obs}} \mapsto \sum_{t \in t_{\mathrm{fut}}} h(Y_t | \widehat{\pmb{Y}}_{<t}, \pmb{X})$ , i.e. we approximate the total uncertainty over the predicted sequence by summing the differential entropies of the individual densities estimated at
Figure 2: Illustrating predicted densities under greedy autoregressive decoding for two timesteps. For simplicity, we depict a joint Gaussian distribution and omit the conditioning on  $X$  everywhere.
each timestep. Note that following the chain rule for differential entropy (see Cover and Thomas [35, Eq. 8.62]), the joint entropy can indeed be written as the sum of individual conditionals. However,
$$
h (Y | X = \boldsymbol {X}) = \sum_ {t \in \boldsymbol {t} _ {\mathrm {f u t}}} h \left(Y _ {t} \mid Y _ {<   t}, \boldsymbol {X}\right) \neq \sum_ {t \in \boldsymbol {t} _ {\mathrm {f u t}}} h \left(Y _ {t} \mid \widehat {\boldsymbol {Y}} _ {<   t}, \boldsymbol {X}\right). \tag {7}
$$
And yet, training autoregressive decoders by maximizing likelihood actually assumes the inequality in Equation 7 to be approximately equal (see [39, Sec. 2; 40, Eq. 5]). The approximation relies on the observation that, for autoregressive decoding, the parameters of the predicted distribution for  $Y_{t}$  are computed as a deterministic function of the decoder hidden state. That is,  $Y_{t}$  is conditionally independent of  $Y_{< t}$  given the hidden state of the decoder  $s_{t}$  at timestep  $t$ . The underlying assumption is that for a well-trained decoder,  $s_{t}$  encodes all relevant information from other timesteps to infer the distribution of  $Y_{t}$ . So at inference, despite being a function of the single sample  $\hat{Y}_{t-1}$ , the predicted distribution conditioned on  $s_{t}$  provides a reasonable estimate of the uncertainty in  $Y_{t}$ . This assumption allows us to again obtain a closed-form expression for the saliency map when each  $Y_{t}$  is modeled using a density function with a known closed-form expression for differential entropy [35, Tab. 17.1; 36]. For the common choice of modeling  $Y_{t}$  using a Gaussian mixture [37, 38], approximations that approach the true differential entropy can also be obtained efficiently [41-43] to directly compute the feature mapping  $\phi$ .
The second option is to estimate  $h(Y|X = X)$  using sampling or other non-parametric approaches when analytical expressions or computationally efficient approximations are not available [44-47]. These sampling-based methods provide approximations that converge to the true entropy, although they may be computationally more expensive than parametric methods. Overall, the choice of modeling the future density and the approach for estimating the differential entropy depends on the specific scenario and the available resources.
The common evaluation of saliency-based explanations relies on qualitative visual assessment, which is subjective and prone to observer biases [7, 11, 12]. Meanwhile, establishing a reliable ground truth for the salient relationship between the observed window  $t_{\mathrm{obs}}$  and the future window  $t_{\mathrm{fut}}$  is challenging in real-world data due to conflicting domain evidence on predictive relationships [28, 48]. Furthermore, fair validation of a model agnostic, post hoc method requires evaluating it independently of imperfections in the underlying forecasting model. To address these challenges we conduct two types of empirical evaluation: one using synthetic data to establish ground truth predictive saliency and validate the framework, and another to demonstrate empirical utility in real-world scenarios where perfect forecasts and ground truth saliency are unavailable.
No existing benchmarks or post hoc explanation frameworks exist for probabilistic time-series regression that meet the necessary requirements for a meaningful empirical comparison. Nevertheless, we provide results by adapting several explainability frameworks in our experiments. Specifically, we considered DeepSHAP and GradientSHAP [19], and IntegratedGradients and SmoothGrad [49]. It is important to note that we do not imply that these are fair comparisons; they are not (see Section 2). However, the comparisons are meant to characterize results from popular tools that practitioners are likely to use in the absence of our proposed framework. Implementation details for the following experiments and additional results for the real-world scenarios are in Appendices C and D, respectively.
5.1 Empirical Validation using Synthesized Ground Truth Saliency
Dataset. We simulate a group conversation that emulates real behavior patterns. Listeners typically focus on the speaker, while the speaker looks at different listeners [50]. Additionally, head gestures and gaze patterns predict the next speaker [51-54]. In our simplified simulation, the speaker rotates towards the center when speaking, and listeners nod to trigger a turn handover. We use real-valued quaternions to represent 3D head poses, commonly used for human motion and pose representation [4, 55, 56]. Following the notation in Section 4.2,  $\pmb{b} = [q_w, q_x, q_y, q_z, ss]$  where  $ss$  denotes binary speaking status. We simulate the turn changes to occur once clockwise and once anticlockwise. The ground truth salient timestep is when a listener initiates a head nod to trigger a turn handover, ensuring a certain future turn change. Figure 3 illustrates this mechanism. The code, dataset, and animated visualization are available in the Supplement.
Figure 3: Illustrating the synthetic conversation dynamics dataset. Speakers are denoted in orange and listeners in green. For a fixed  $t_{\mathrm{fut}}$  we depict two preceding  $t_{\mathrm{obs}}$  windows. By construction, when observing a stable speaking turn over  $t_{\mathrm{obs}}^{1}$ , two valid futures are possible over  $t_{\mathrm{fut}}$ . These correspond to a turn handover to the immediate left or right of the current speaker. Over  $t_{\mathrm{obs}}^{2}$ , when a listener nods to indicate the desire to take the floor, the future over  $t_{\mathrm{fut}}$  becomes certain, corresponding to the listener successfully taking over the speaking turn. Here  $t_{\mathrm{obs}}^{2}$  is consequently more salient than  $t_{\mathrm{obs}}^{1}$  towards forecasting the turn change over  $t_{\mathrm{fut}}$ . (Best viewed as video, see Supplement.)
(a) Ours (Perfect forecast)
(b) Ours (Trained model)
(c) Attributions (Trained model)
Figure 4: Computing Saliency. The top plots show the quaternion dimensions  $qx$  and  $qy$  for the listener that nods over  $t_{\mathrm{obs}}^2$  in Figure 3. The gray dotted line indicates the true salient timestep 138 when the head nod begins, making the future over timesteps  $183 - 228$  ( $t_{\mathrm{fut}}$ ) certain. The rest of plots show the (a) entropy over future values of all participants (middle), and saliency map obtained using our framework (bottom), considering perfect forecasts; (b) entropy and saliency for the forecasts from a Social Process model; and (c) mean attributions across features per timestep from different explainability frameworks (DeepSHAP, GradientSHAP, IntegratedGradients, and SmoothGrad) for the predicted mean and std. of the same forecast from the Social Process model.
Empirical Validation. To validate our framework in isolation, we assume a perfect forecasting model that predicts the true distribution over the possible future quaternion trajectories. The forecasting model focuses solely on low-level features and does not incorporate any high-order semantics of turn-taking. The saliency map generated by our framework, as shown in Figure 4a, accurately identifies the ground truth salient timesteps at frames 138 and 139 where the head nod begins. The saliency decreases once the nod is already in motion, indicating that it does not provide additional information about the future. This empirically validates our framework.
Introducing a Real Forecasting Model. We evaluate our framework using a real underlying forecasting model trained on synthetic data. We employ a Social Process model [4] for its ability to capture relative partner behavior and adapt to specific group dynamics. As shown in Figure 4b, our framework identifies the true salient timesteps with higher saliency values. Conversely, the attributions provided by other explainability frameworks in Figure 4c for the predicted mean and
(a) Group leaving behavior (MatchNSingle dataset)
Figure 5: (a) Analysis of the group leaving instance at 12:11 on Day 1, Cam 12 in the MatchNMingle dataset. Row 1: Video frames and overlaid arrows denoting the head orientation of participants. Orange indicates the person leaving the group over the  $t_{\mathrm{fut}}$ . Row 2: Head orientation of the leaver plotted as 2D horizontal rotation. Row 3: Saliency map from running predictions from the Attentive Social Process model through our framework. The timesteps salient towards the model's forecasts correspond to the leaver making sweeping gazes away from the group. Rows 4-5: mean DeepSHAP and GradientSHAP across features per timestep for the predicted mean and std. of the same forecast. (b) Analysis of the vehicle turn making instance on Scene 3 in the nuScenes dataset. Row 1: Video frames showing the bus and surrounding cars from the camera. Row 2: Future predictions for the bus position (circled) from the Trajectory++ model (ground truth in white, predicted mean in black and variance in red). Row 3: Saliency map from running predictions through our framework. The timesteps salient correspond to the model being more certain that the bus will make a turn. Rows 4-5: mean DeepSHAP values across features per timestep for the predicted mean and std. of the same forecast. Best viewed as video (see Supplement).
(b) Vehicle trajectory (nuScenes dataset)
standard deviation of the same forecast fail to capture the salient predictive relationships in the data. This comparison underscores the effectiveness of our framework in capturing meaningful and interpretable saliency, even in conjunction with an imperfect forecasting model.
5.2 Empirical Evaluation in Real-World Scenarios
5.2.1 Group Leaving Behavior in the Wild
The study of group-leaving behavior has garnered interest in social psychology and the development of conversational agents [29, 30]. Recent approaches employ data-driven models to predict future non-verbal cues, capturing general predictive patterns in the data [4]. In this study, we demonstrate how our framework can assist domain experts in hypothesizing about the causal relationships between behavioral patterns and group leaving. We leverage the publicly available MatchNMingle dataset [57], which features natural interactions of 92 individuals during a cocktail party. We use an Attentive Social Process model [4] to forecast continuous head pose, body pose, and binary speaking status.
Through our analysis (see Figure 5a), we find that the salient timesteps in the model's forecasts correspond to instances when a person about to leave directs their gaze away from the shared group space ( $o$ -space [1]) by rotating their head. This observation leads to the following hypothesis:
gazing away from the o-space of a conversing group is predictive of group leaving.
While this hypothesis aligns with established leave-taking patterns [1, 58] and the sweeping gaze behavior associated with seeking new interaction partners [59], it requires further validation through subsequent studies and rigorous statistical testing with the involvement of domain experts. Nonetheless, our experiment demonstrates how the framework can unveil data-driven insights into patterns that, in other cases, may have been overlooked by humans but captured by the forecasting model. By contrast, we do not observe any discernible intuitive patterns in the features associated with the trends in DeepSHAP and GradientSHAP values for the predicted mean and standard deviation.
5.2.2 Vehicle Trajectory Forecasting
The accurate forecasting of pedestrian and vehicle trajectories is crucial for safe and socially-aware autonomous navigation of vehicles [37, 60-62]. In this study, we utilize our framework to investigate vehicle dynamics in real driving scenarios. Specifically, we leverage the nuScenes dataset, a multimodal dataset for autonomous driving [63], and the Trajectory++ forecasting model [37].
Figure 5b illustrates our analysis of vehicle dynamics at an intersection. Notably, our framework identifies a salient timestep for the Trajectory++ model precisely when it becomes more confident that the bus will make a turn instead of continuing straight. This coincides with the model's increased certainty that the point-of-view vehicle will decelerate as a new vehicle enters the scene from the left. Although there are no relevant domain-specific theories in this case to interpret this saliency, these identified patterns align with expected driving behavior. In contrast, the DeepSHAP values fail to capture the model's change in certainty about the bus making the turn instead of continuing straight. Moreover, we also do not identify any intuitive patterns in the predictions associated with the DeepSHAP trends. Thus, our framework serves as a valuable tool for sanity-checking model forecasts in real-world driving scenarios. It helps identify instances where the model's predictions align or misalign with established norms and expectations.
We have proposed a computational framework that provides counterfactual explanations of model forecasts based on a principled notion of bottom-up task-agnostic saliency. We derive a closed-form expression to compute this saliency for commonly used probability density functions to represent forecasts [4, 37, 38, 62]. To validate our framework, we conduct empirical experiments using a synthetic setup, enabling quantitative validation and mitigating observer biases associated with visual assessment of saliency maps. Additionally, we demonstrate the practical utility of our framework in two real-world scenarios involving the prediction of nonverbal social behavior and vehicle trajectories. By identifying salient timesteps towards a predicted future through counterfactual reasoning, our framework can support domain experts in formulating data-driven hypotheses regarding the predictive causal patterns involving the features present at those salient timesteps. These hypotheses can then be tested through subsequent controlled experiments, establishing a human-in-the-loop Explainable AI (XAI) methodology. For a more comprehensive discussion, please refer to Appendix E.
7 Limitations and Potential Negative Societal Impact
While our framework provides a closed-form or efficient solution for most probability density functions, limitations arise when an analytic expression for differential entropy is unavailable. As discussed in Section 4.3, alternative approaches like sampling or nonparametric methods can be employed to approximate the entropy, albeit at an increased computational cost.
Our work here is an upstream methodological contribution. However, when applied downstream to human behavior or healthcare data, ethical considerations arise naturally. Here, care must be taken that such methods are not applied for gaining insights into behavior in a way that violates the privacy of people. Our framework enables domain experts to derive data-driven insights and hypotheses about predictive causal patterns. However, hypotheses should be rigorously tested, using controlled experiments and peer review, before being considered valid statements about human behavior. Collaboration among researchers, practitioners, and policymakers across disciplines is crucial to mitigate such societal risks and ensure ethical deployment of AI technologies.
The authors would like to thank Jesse Krijtthe, Rickard Karlsson, David Tax, Yeshwanth Napolean, and Megha Khosla for the thoughtful discussions.
A Broader Related Work: Explainable Methods for Time-Series Data Across Tasks and Domains
The larger focus of explainability techniques involving time-series data has been on the task of classifying time-series. The goal has been to estimate the relevance of each input feature at a given timestep towards each output class. Here, saliency approaches often overlap with techniques developed for image data and can be categorized into:
Gradient-Based Techniques. The broad approach involves evaluating the gradient of the output class with respect to the input [64]. Several variants have been proposed [19, 49, 65-67].
Perturbation-Based Techniques. The idea is to examine how the output changes in response to some perturbation of the input. Perturbations are implemented by either occluding contiguous regions of the input [68, 69]; performing an ablation of the features [70]; or randomly permuting features [12]. Ismail et al. [71] provide a benchmark of a subset of these techniques.
Attention-Based Techniques. These incorporate an attention mechanism into the model that is trained to attribute importance to different parts of the input sequence towards a prediction at each future timestep. Such techniques have been extensively utilized for healthcare data. Early methods applied a reverse-time attention [72], Later methods applied the attention to probabilistic state-space representations [73].
Some of these broad ideas have been applied to the regression setting to make interpretable forecasts of future time-series features. Lim et al. [15] leveraged self-attention layers for capturing long-term dependencies. Pan et al. [17] recently proposed computing saliency as a mixup strategy between series images and their perturbed version with a learnable mask for each sample. They view saliency in terms of minimizing the mean squared error between the predictions and ground-truths for a particular instance. Focusing on the univariate point-forecasting problem, Oreshkin et al. [14] proposed injecting inductive biases by computing the forecast as a combination of a trend and seasonality model. They argue that this decomposition makes the outputs more interpretable.
Developing explainable techniques for the probabilistic forecasting setting remains largely unexplored and subject to non-overlapping notions of explainability. Rügamer et al. [13] transform the forecast using predefined basis functions such as Bernstein polynomials. They relate interpretability to the coefficients of these basis functions (a notion similar to that of Oreshkin et al. [14]). Panja et al. [74] embed the classical linear ARIMA model into a non-linear autoregressive neural network for univariate probabilistic forecasting. As before, the explainability here also stems from the 'white-box' nature of the linear ARIMA component. Li et al. [75] propose an automatic relevance determination network to identify useful exogenous variables (i.e. variables that can affect the forecast without being a part of the time-series data). To the best of our knowledge, saliency-based methods have not yet been considered within this setting.
B Favorable Properties of Differential Entropy
Differential entropy possesses favorable properties that make it a suitable choice as  $\phi$  for computing the saliency map. First, the scale of the forecast density does not affect the resulting saliency map (see Cover and Thomas [35, Theorem 8.6.4]):
$$
h (a Y) = h (Y) + \log | a |, \text {f o r} a \neq 0, \text {a n d} \tag {8}
$$
$$
h (\boldsymbol {A} Y) = h (Y) + \log | \det (\boldsymbol {A}) |, \text {w h e n} \boldsymbol {A} \text {i s a s q u a r e m a t r i x .} \tag {9}
$$
That is, scaling the distribution changes the differential entropy by only a constant factor. So the saliency map resulting from inserting the entropy into Equation 4 remains unaffected since the Jacobian term only depends on the relative change in entropy across different choices of  $t_{\mathrm{obs}}$ . Similarly, translating the predicted density leaves the saliency map unaffected (see Cover and Thomas [35, Theorem 8.6.3]):
$$
h (Y + c) = h (Y). \tag {10}
$$
C Implementation Details for Experiments
C.1 Other Explainability Methods
For DeepSHAP and GradientSHAP, we used the official implementation of SHAP: https://github.com/slundberg/shap. For IntegratedGradients and SmoothGrad, we used the Captum framework: https://captum.ai/. We reiterate that these are not fair comparisons, for the reasons we have discussed in Section 2. One crucial reason is that no existing method is designed to handle a predicted distribution. To apply them in this context, we need to compute attribution values for the predicted mean and standard deviation for every feature at every  $t \in t_{\mathrm{fut}}$  in isolation. In contrast, by measuring differential entropy, our method jointly accounts for the parameters of the distribution and captures the information content of the distribution. Despite these limitations, we include these comparisons to provide readers with a contextual understanding of the results obtained from commonly used explainability tools.