SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Computational Efficiency. For further insight, we measured the execution time of our saliency method compared to DeepSHAP for both real-world scenarios (see Table 1). The main takeaway is that our method is an order of magnitude faster at computing the saliency map compared to DeepSHAP. This is in part because DeepSHAP computes values for every feature at every timestep in the output independently. Even if this is parallelized, DeepSHAP requires multiple forward passes and gradient computations including samples from a background set to compute the reference values. The computation time scales with the size of the background set.
Table 1: Comparing Computational Efficiency. We compare the practical execution time of our method to running DeepSHAP. For a reasonably fair characterization of DeepSHAP, we report execution time for computing the SHAP values associated with a single predicted parameter, the mean of the future distribution.
Scenario	Saliency	DeepSHAP
Forward Pass	Compute Saliency	Init	Compute Values
Group leaving (MatchNMingle dataset)	50 ms	2 ms	50 ms	13.5 min
Vehicle trajectory (nuScenes dataset)	2.5 s	33 ms	68 ms	26.3 min
C.2 Empirical Validation using Synthetic Data
We model the future distribution using a Gaussian function for simplicity (setting std. to  $10^{-10}$  for the single future), but a more complex distribution that predicts the appropriate change in variance would also work in practice. We now implement Algorithm 1 as follows. We identify a window where a turn change occurs in the data (frames 183-228) and denote this 45 frame window as the  $\pmb{t}_{\mathrm{fut}}$  of interest. While we manually identify an interesting event for illustration, such a window could also correspond to an interesting prediction by a model. We generate a set of candidate  $\pmb{t}_{\mathrm{obs}}$  by sliding a 30 frame window over a horizon of 100 frames prior to  $\pmb{t}_{\mathrm{fut}}$ , with a stride of 1 frame. For every observed  $\pmb{t}_{\mathrm{obs}}$ , we fit a Gaussian density to the quaternion and speaking status features of all participants over the futures that can occur during  $\pmb{t}_{\mathrm{fut}}$ . We then set the entropy of this Gaussian density as the feature  $\phi$  for that  $\pmb{t}_{\mathrm{obs}}$ . For the experiments with a real forecasting model, we employ the [recurrent, uniform attention] variant of the Social Process family given its ability to capture dynamic movements [4].
C.3 Forecasting Group Leaving Behavior
We used the pretrained model on the MatchNMingle dataset provided with the official implementation of Social Processes [4]: https://github.com/chiragraman/social-processes. Specifically, we employed the [recurrent, dot-attention] variant of the Attentive Social Process family (ASP-GRU-dot). We set  $t_{\mathrm{fut}}$  to correspond to a 3 second window (3 frames in the data, which is at 1 Hz) containing an individual leaving a conversing group. We obtained forecasts from the model corresponding to a rolling 5 second  $t_{\mathrm{obs}}$  within a 20 second preceding horizon. For computing DeepSHAP and GradientSHAP values, we used the entire observed time horizon as the background
dataset. This ensures that the expected values are computed within a reasonably similar context for a given  $\pmb{t}_{\mathrm{obs}}$ .
C.4 Vehicle Trajectory Forecasting
We trained the Trajectory++ model [37] on the nuScenes dataset (mini version) [63] using the default command provided in the official implementation: https://github.com/StanfordASL/Trajectory-plus-plus. In particular, we used the int_ee model which incorporates the agent's system dynamics to produce dynamically-feasible trajectories. For analysis, we used a sequence from scene index 3 (scene.name 757). We set  $t_{\mathrm{fut}}$  to correspond to a window of 6 timesteps (14-19) with a lookback horizon from timesteps 6 to 13. For computing DeepSHAP values we again used the entire observed horizon as the background set. The features in the observed sequence contained several NaN entries, which resulted in NaN DeepSHAP values. We ignored these when aggregating values per timestep.
(a) 14:27 at Day 1, Cam 12
(b) 25:42 at Day 2, Cam 5
Figure 6: Analysis of two sequences in the MatchNMingle dataset. Row 1: Video frames and overlaid arrows denoting the head orientation of the participant of interest. Row 2: Head orientation plotted as 2D horizontal rotation. Rows 3-5: Saliency map from running predictions from the Attentive Social Process model through our framework, as well as the mean DeepSHAP and GradientSHAP values across features per timestep for the predicted mean and std. of the same forecast. Our saliency framework succeeds in identifying the salient timesteps in both cases. In (a), the timestep in which the participant of interest is looking away is identified as salient towards predicting the other participant leaving the dyadic interaction. In (b), the timestep in which the participant of interest suddenly stops actively participating in the conversation (not nodding or looking at the speakers) is identified as salient towards predicting their group leaving.
Figure 7: Analysis of a sequence on Scene 0 in the nuScenes dataset. Row 1: Video frames showing the black turning car and surrounding cars from the camera. Row 2: Future predictions for the black car position (circled) from the Trajectory++ model (ground truth in white, predicted mean in black and variance in red). Rows 3-5: Saliency map from running predictions through our framework, as well as the mean DeepSHAP values across features per timestep for the predicted mean and std. of the same forecast. Our framework reveals that the silver and point-of-view cars slowing is salient for the model in predicting that the black car completes the turn.
E Broader Discussion: Saliency & XAI with Domain Experts in the Loop
We begin this broader discussion by revisiting the different notions of saliency, to make the case for why our proposed framework is suitable for forecasting tasks. Rather than defining saliency in a top-down manner as a function of some task-specific error metric, we have started from a more fundamental conception of bottom-up, or task-agnostic, saliency. Loog's [5] original definition pertains to preattentive saliency, which captures what is perceived to be subconsciously informative before conscious (attentive) processing by the brain. Here, a surprising or unexpected observation is salient. For instance, in a large white image with a single black pixel, the black pixel is salient. The direct application of this concept to time-series data would involve identifying surprising task-agnostic temporal events. For instance, imagine viewing a static landscape where a bird suddenly flies in. The entry of the bird into the scene is unexpected, and therefore salient.
When applied to forecasting tasks, however, this idea of surprisal (or unexpectedness or informativeness) that saliency represents needs to be tied to the future outcome. The saliency computed by most methods working on point-forecasting tasks deals with which past features are surprising given a specific realization of the future. While not explicitly stated by these works, we argue that this notion of saliency is related to the surprisal in  $p_{X|Y}$  for some specific  $Y$ . We therefore interpret these methods as being associative in nature within Miller's [10] categorization in Section 3. In contrast, our approach is counterfactual because we examine alternate future outcomes, while conceptualizing saliency more naturally defined in terms of the changes in the uncertainty in  $p_{Y|X}$  in response to different realizations of observed sequences. However, rather than corresponding to random occlusions or perturbations of the input, the different realizations of  $X$  in our framework correspond to real features (or behaviors) preceding a future, which is more suitable to present to domain experts as candidate causes.
Loog's [5] unifying framework subsumes all forms of saliency, although identifying the appropriate  $\phi$  for a specific domain is non-trivial. In this work we have established both theoretically and empirically how expressing  $\phi$  in terms of the information about the future enables principled counterfactual reasoning in forecasting settings. Nevertheless, we reiterate that the salient timesteps retrieved by our framework ought to be treated as candidate causes until subsequently examined along with a domain expert. Our stance on human-in-the-loop XAI also aligns with research on saliency-based and general XAI in other domains [9, 76].
In principle, when it is possible to have access to the true  $p_{Y|X}$ , the salient timesteps identified by our framework reflect the true predictive structural relationships captured by the underlying model across the entire data. However, estimating this density analytically entails identifying the multiple futures in the data corresponding to every occurrence of the same observed features. In practice, subtle variations in behaviors and sensor measurement errors make it infeasible to estimate  $p_{Y|X}$  analytically, so a model is trained to capture generalized patterns from the given data. In these cases, our framework identifies the sequences that the model considers salient for its forecasts given the data. Consequently, subsequent causal analysis of the features over the salient timesteps is crucial, especially in the healthcare and human behavior domains to avoid potential prejudices against certain behaviors, or worse, misdiagnoses of affective conditions.