Title: INTERPRETING VIDEO FEATURES: A COMPARISON OF 3D CONVOLUTIONAL NETWORKS AND CONVOLU-TIONAL LSTM NETWORKS

Abstract: A number of techniques for interpretability have been presented for deep learning in computer vision, typically with the goal of understanding what the networks have actually learned underneath a given classification decision. However, when it comes to deep video architectures, interpretability is still in its infancy and we do not yet have a clear concept of how to decode spatiotemporal features. In this paper, we present a study comparing how 3D convolutional networks and convolutional LSTM networks, respectively, learn features across temporally dependent frames. This is the first comparison of two video models that both convolve to learn spatial features but that have principally different methods of modeling time. Additionally, we extend the concept of meaningful perturbation introduced by Fong & Vedaldi (2017) to the temporal dimension to search for the most meaningful part of a sequence for a classification decision.

1. INTRODUCTION:

Two standard approaches to deep learning for sequential image data are 3D Convolutional Neural Networks (3D CNNs), e.g. the I3D model (Carreira & Zisserman (2017) ), and recurrent neural networks (RNNs). Among the RNNs, the convolutional long short-term memory network (C-LSTM) (Shi et al. (2015) ) is especially suited for sequences of images, since it learns both spatial and temporal dependencies simultaneously. Although both methods can capture aspects of the semantics pertaining to the temporal dependencies in a video clip, there is a fundamental difference in how 3D CNNs treat time compared to C-LSTMs. In 3D CNNs the time axis is treated just like a third spatial axis, whereas C-LSTMs only allow for information flow in the direction of increasing time, complying with the second law of thermodynamics. More concretely, C-LSTMs maintain a hidden state representing the current video frame when forward-traversing the input video sequence, and are able to model non-linear transitions in time. 3D CNNs instead convolve (i.e. take a weighted average) over both the temporal and spatial dimensions of the sequence.

The question investigated in this paper is whether this difference has consequences for how the two models compute spatiotemporal features. We present a qualitative study of how 3D CNNs and C-LSTMs respectively compute video features: what do they learn, and how do they differ from one another?

As outlined in Section 2, there is a large body of work on evaluating video architectures on spatial and temporal correlations, but significantly fewer investigations of what parts of the data the networks have used and what semantics relating to the temporal dependencies they have extracted from it. Deep neural networks are known to be large computational models, whose inner workings are difficult to overview for a human. For video models, the number of parameters is typically significantly higher which makes their interpretability all the more pressing.

We will evaluate these two types of models (3D CNN and C-LSTM) on tasks where temporal order is crucial. The 20BN-Something-something-V2 dataset (Mahdisoltani et al. (2018) , hereon Something-something) will be central to our investigations; it contains time-critical classes, agnostic to object appearance, such as move something from left to right or move something from right to left. We additionally evaluate the models on the smaller KTH actions dataset (Schuldt et al. (2004) ).

• We present the first comparison of 3D CNNs and C-LSTMs in terms of temporal modeling abilities and highlight the essential difference between their assumptions concerning temporal dependencies in the data.

• We extend the concept of meaningful perturbation introduced by Fong & Vedaldi (2017) to the temporal dimension to search for the most critical part of a sequence for classification.

2. RELATED WORK:

The field of interpretability in the context of deep neural networks is still young but has made considerable progress for single image networks, owing to works such as Zeiler & Fergus (2013) , Simonyan et al. (2014) and Montavon et al. (2018) . One can distinguish between data centric and network centric methods for interpretability. Activity maximization, first coined by Erhan et al. (2009) , is network centric in the sense that specific units of the network are being studied. By casting the maximization of the activation of a certain unit as an optimization problem in terms of the input, one can compute the optimal input for that particular unit by gradient ascent. In data centric interpretability methods, the focus is instead on the input to the network, to reveal which patterns of the data that the network has discerned.

Grad-CAM (Selvaraju et al. ( 2017)) and the meaningful perturbations explored in Fong & Vedaldi (2017) , which form the basis for our experiments, belong to the data centric category. These two methods are further explained in Section 3. Layer-wise relevance propagation (LRP) (Montavon et al. (2018) ) as well as Excitation backprop (Zhang et al. (2016) ) are two other examples of data centric backpropagation techniques designed for interpretability, where the excitation backprop method follows from a simpler parameter setting of LRP. Building on excitation backprop by Zhang et al. (2016 ), Adel Bargal et al. (2018) produce saliency maps for video RNNs without the use of gradients. Instead, products of forward weights and activations are normalized in order to be used as conditional probabilities, which are back-propagated. We have chosen to use Grad-CAM for our experiments, since it is one of the saliency methods in Adebayo et al. (2018) that passes the article's sanity checks, as well as for its simplicity of implementation and wide usage.

Limited works have been published with their focus on interpretability for video models (Feichtenhofer et al. (2018 ), Sigurdsson et al. (2017 ), Huang et al. (2018) , Ghodrati et al. (2018) ). Other works have treated it, but with less extensive experimentation (Chattopadhyay et al. (2017)) , while for example mainly presenting a new spatiotemporal architecture (Dwibedi et al. (2018) , Zhou et al. (2018) ). We build on the work by Ghodrati et al. (2018) , where the aim is to measure a network's ability to model video time directly, instead of via the proxy task of action classification, which is most commonly seen. Three defining properties of video time are defined in the paper: temporal symmetry, temporal continuity and temporal causality, and are each presented accompanied by a measurable task. The third property is measured using the classification accuracy on the Somethingsomething dataset. An important contribution of ours with respect to this work is that we compare 3D CNNs and C-LSTMs, whereas Ghodrati et al. (2018) compare 3D CNNs to standard LSTMs. Their comparison can be argued as slightly unfair, as standard LSTM layers only take 1D input, and thus needs to collapse each image frame in the video to a vector, which removes some spatial dependencies in the pixel grid.

Similar to our work, Dwibedi et al. (2018) investigate the temporal modeling capabilities of convolutional RNNs (Convolutional Gated Recurrent Units) trained on Something-something. The authors find that recurrent models perform well for the task, and a qualitative analysis of the learned hidden states of their trained model is presented. For each class of the dataset, they obtain the hidden states of the network corresponding to the frames of one clip and display its nearest neighbors from other clips' per-frame hidden state representations. These hidden states had encoded information about the relevant frame ordering for the classes. Sigurdsson et al. (2017) examined video architectures and datasets on a number of qualitative attributes. Huang et al. (2018) investigate how much the actual motion in a clip contributes the classification performance of a video architecture. To measure this, they perform classification experiments varying the number of sub-sampled frames used for a clip to examine how much the accuracy changes as a result.

In a search-based precursor to our temporal mask experiments, Satkin & Hebert (2010) crop sequences temporally to obtain the most discriminative sub-sequence for a certain class. The cropping corresponding to the highest classification confidence is selected as being the most discriminative sub-sequence. Feichtenhofer et al. (2018) present the first network centric interpretability work for video models.

The authors investigate spatiotemporal features using activity maximization. Zhou et al. (2018) introduce the Temporal Relational Network (TRN) which learns temporal dependencies between frames through sampling the semantically relevant frames for a particular action class. The TRN module is put on top of a convolutional layer and consists of a fully connected network between the sampled frame features and the output. Similar to Dwibedi et al. (2018) , they perform temporal alignment of clips from the same class, using the frames considered most representative for the clip by the network. They verify the conclusion previously made by Xie et al. (2017) , that temporal order is crucial on Something-something and show that their architecture is sensitive to that. They also investigate which classes of Something-something show the strongest sensitivity to temporal order.

3. APPROACH:



3.1. TEMPORAL MASKS:

The proposed temporal mask method aims to extend the interpretability of deep networks into the temporal dimension, utilizing meaningful perturbation of the input, as shown effective in the spatial dimension by Fong & Vedaldi (2017) . When adopting this approach, it is necessary to define what constitutes a meaningful perturbation. In the mentioned paper, a mask that blurs the input as little as possible is learned for a single image, while still maximizing the decrease in class score. Our proposed method applies this concept of a learned mask to the temporal dimension. The perturbation, in this setting, is a noise mask approximating either a 'freeze' operation, which removes motion data through time, or a 'reverse' operation that inverses the sequential order of the frames. This way, we aim to identify which frames are potentially most critical for the network's classification decision.

The perturbing temporal mask is defined as a vector of values on the interval [0,1] with the same length as the input sequence. For the 'freeze' type mask, a value of 1 for a frame at index t duplicates the value from the previous frame at t -1 onto the input sequence at t. The pseudocode for this procedure is given below. for i in maskIndices do

perturbedInput i ← (1 -mask i ) * originalInput i + mask i * perturbedInput i-1 end for

For the 'reverse' mask type, all indices of the mask m that are activated are first identified (threshold 0.1). These indices are then looped through to find all contiguous sections, which are treated as sub-masks, m i . For each sub-mask, the frames at the active indices in the sub-mask are reversed. For example (binary for clarity), an input indexed as t 1:16 perturbed with a mask with the value [0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0] would result in the sequence with frame indices [1, 2, 3, 8, 7, 6, 5, 4, 9, 10, 11, 12, 13, 15, 14, 16] .

In order to learn the mask, we define a loss function (Eq. 1) to be minimized using gradient descent, similar to the approach in Fong & Vedaldi (2017) .

EQUATION

)

where m is the mask expressed as a vector m ∈ [0, 1] t , • 1 1 is the L 1 norm, • β β is the Total Variation (TV) norm, λ 1,2 are weighting factors, and F c is the class score given by the model for the perturbed input. The L 1 norm punishes long masks, in order to identify only the most important frames in the sequence. The TV norm penalizes masks that are not contiguous. This approach allows our method to automatically learn masks that identify one or several contiguous sequences in the input. The mask is initialized centered at the middle of the sequence. To keep the perturbed input class score differentiable w.r.t. the mask, the optimizer operates on a mask vector that has values in R. A sigmoid function is applied to the mask before using it for the perturbing operation in order to keep its values in the [0,1] range.

The ADAM optimizer is then used to learn the mask through 300 iterations of gradient descent. After the mask has converged, it is thresholded for visualisation purposes.

3.2. GRAD-CAM:

Grad-CAM (Selvaraju et al. ( 2017)) is a method for producing visual explanations in the form of class-specific saliency maps for CNNs. One saliency map, L c t , is produced for each image input based on the activations from k filters, A k ij , at the final convolutional layer. In order to adapt the method to sequences of images, the activations for all timesteps t in the sequences are considered.

EQUATION

)

where Z is a normalizing constant and F c is the network output for the class c. Since the aim of the method is to identify which activations had the highest contribution to the class score, only positive values of the linear combination of activations are considered, as areas with negative values are likely to belong to other classes. By up-sampling these saliency maps to the resolution of the original input image, the aim is to examine what spatial data in specific frames contributed most to the predicted class.

4. EXPERIMENTS:



4.1. DATASETS:

The Something-something dataset contains over 220,000 sequences from 174 classes in a resolution of 224x224 pixels. The duration of the data is more than 200 hours, and the videos were recorded against varying backgrounds from different perspectives. The classes are action-oriented and objectagnostic. Each class is defined as performing some action with one or several arbitrary objects, such as closing something or folding something. This encourages the classifier to learn the template actions, since object recognition does not give enough information for the classifying task. We train and validate according to the provided split. Fig. 2 shows sequences from the validation set.

The KTH Actions dataset consists of 25 subjects performing six different actions (boxing, waving, clapping, walking, jogging, running) in four different settings, resulting in a total of 2391 sequences, with a total duration of almost three hours. The videos are provided with a resolution of 160x120 pixels at 25 fps. They are filmed against a homogeneous background with the different settings exhibiting varying lighting, distance to the subject and clothing of the participants. For this dataset, we trained on subjects 1-16 and evaluated on subjects 17-25 (Fig. 3 ). Both datasets have sequences varying from one to almost ten seconds. As 3D CNNs require a fixed sequence length, all input sequences from both datasets were sub-sampled to cover the entire sequence in 16 frames for Something-something and 32 frames for KTH Actions. The same set of sub-sampled frames was then used as input to each architecture for each dataset.

4.2. ARCHITECTURES AND EXPERIMENT DETAILS:

Hyperparameters are listed in the appendix. Any remaining settings can be found in the code which will be made public in both Pytorch and Tensorflow.

I3D consists of three 3D convolutional layers, nine Inception modules and four max pooling layers (Fig. 1 ). In the original setting, the temporal dimension of the output is down-sampled to two frames. In order to achieve a higher temporal resolution in the produced Grad-CAM images, the strides of the first convolutional layer and the second max pooling layer were reduced to 1x2x2, producing eight activations in the temporal dimension for the 16 frame inputs. The Grad-CAM images are produced from the gradients of the class scores w.r.t. the final Inception module.

The C-LSTM architecture used for Something-something consisted of three C-LSTM layers, each followed by batch normalization and max pooling layers. The convolutional kernels used for each layer had size 5x5 and stride 2x2 with 32 filters. The C-LSTM layers return the entire transformed sequence as input to the next layer, including the last layer before the fully connected layer, used for the predictions. For KTH, the C-LSTM model had two layers with 32 hidden units each and dropout between the layers (p = 0.5). These architectures were chosen as the best performing models after empirical experimentation with the number of layers, hidden units, stride and regularization. When calculating the Grad-CAM maps for the C-LSTM, the final C-LSTM layer was used.

We note that there is a substantial difference in the number of parameters for each resulting model, with 12, 465, 614 for I3D and 1, 324, 014 and for the three-layer C-LSTM. When introduced, the I3D architecture achieved state-of-the-art performance on several video recognition datasets. These properties combined suggest that I3D should have an advantage in performance over the two models. This was confirmed on Something-something, where the C-LSTM architecture could not reach the same overall performance as I3D (Table 1 ). Other architectural variants of the C-LSTM model with a larger amount of parameters (up to five layers) were evaluated as well, but no significant increase in performance was observed. Also, due to the computational complexity of back-propagation through time (BPTT), the C-LSTM variants were significantly more time demanding to train and evaluate than their I3D counterparts. With this in mind, in order to make the comparison as fair as possible, classes were chosen for which the performance of the two architectures were similar. The labels of these classes and their F1 scores for each architecture are shown in Section 5.1 and in the appendix.

5. RESULTS:

The F1-scores for both architectures and datasets are shown in Table 1 . In order to investigate how reliant the two models are on the temporal order of the input frames, a further test was conducted with the input sequences reversed. On Something-something, both the C-LSTM and I3D models were affected drastically, with their top-1 classification scores dropping by 78% and 79%, respectively. This suggests that both models are in fact sensitive to the temporal direction, to almost the same degree, when the test sequence is entirely reversed. In Sections 5.1-5.2, we present results for when only the most salient portion of a sequence is reversed.

For both models, the highest scoring class after reversing the sequence was turning something upside down. This is perhaps not surprising, as the semantic meaning of the action holds even when played backward. The classes with the largest drop in score after reversal for both models were those containing movement in a specific direction, such as turning the camera left while filming something or pushing something from left to right. Both models performed well when reversing the KTH Actions dataset. Most likely, this is because this dataset's classes have distinct spatial features. In this section, we present the Grad-CAM heatmaps and temporal masks generated for each architecture. We display eight sequences in Fig. 2 , but have included more examples in the appendix. The classes of this section's analysis are as follows (I3D F1/C-LSTM F1): moving something and something away from each other (0.76/0.58), moving something and something closer to each other (0.77/0.57), moving something and something so they pass each other (0.37/0.31), moving something up (0.43/0.4), pretending to take something from somewhere (0.1/0.07), moving the camera down while filming something (0.67/0.56), and moving the camera up while filming something (0.81/0.73).

First, we note that the Something-something classes can be ambiguous (one class may contain another class) and for a few samples, arguably, even be incorrectly labeled. The latter can be seen for example in Sequence #2, where I3D's classification was moving something and something so they collide with each other and the C-LSTM model predicted pushing something with something.

Although the two objects in the sequence do move closer to each other, they also touch at the very end, making both predictions technically correct. Another case of understandable confusion can be seen in Sequence #5, where I3D's classification was taking one of many similar things on the table.

In this case the surface seen in the image is a tiled floor, and the object is a transparent ruler. Once the temporal mask activates during the lifting motion in the last four frames, the Grad-CAM images show that the model also focuses on two of the lines on the floor. These could be considered similar to the lines caused by the outline of the ruler, which could explain the incorrect classification.

A difference between the architectures is that the I3D model often focuses on contiguous, centered blobs, while the C-LSTM model attempts to find relevant spatial features in multiple smaller areas (Table 2 ). Examples can be seen in Sequences #1 and #3 of Fig. 2 , where I3D focuses on a single region covering both objects while the C-LSTM has activations for both of the objects and the surface affected by the movement. The I3D model also has a bias of starting its focus around the middle of the screen as can be seen in Sequences #1 to #8, often even before the motion starts. The typical behavior for C-LSTM is instead to remain agnostic until the action actually starts (Sequence #7). For Sequence #7, the I3D maintains its foveal focus even after the green, round object is out of frame. For Sequence #8, the focus actually splits midway to cover both the moped and some features on the wall, while the C-LSTM model focuses mainly on numerous features along the wall, as it usually does in classes where the camera turns. The C-LSTM also seems to pay more attention to hands appearing in the clips, rather than objects, as can be seen in Sequences #1 to #4. The average distance to the center of the image for each blob in each frame is shorter for I3D (Table 2 ).

Given the same number of iterations for the optimization of the temporal mask, the two models typically reached different losses. Generally, I3D obtained a lower loss. For this reason, we consider the difference and ratio1 between the score drops after the freeze and reverse perturbations as the most relevant measures of how sensible one model was for the reverse perturbation. We observe that, in general, the drop caused by the reverse perturbation is smaller for the C-LSTM than for I3D. However, the freeze-reverse drops' difference and ratio are considerably higher for the I3D compared to C-LSTM (Table 2 ), suggesting that I3D is less sensitive to the salient reverse perturbation.

Last, we note that the average temporal mask is shorter for the I3D model (Table 2 ). This suggests that it has learned to react to short, specific events in the sequences. As an example, the temporal mask of Sequence #3 is active only on the frames where the objects first pass each other, and in Sequence #2, it is active on the frames leading to the objects touching.

Table 2 : Statistics for the temporal masks and Grad-CAM maps on a subset of the Somethingsomething validation set (791 samples). The analysis was run on 150 clips each (fewer if there were not 150 samples of that class in the validation set) of the seven classes presented in this section. The 'blobs' were computed per frame, using the blob detection tool from OpenCV (Bradski (2000) ).

Drop ratio is defined as OS-FS OS-RS and drop difference as (OS -FS) -(OS -RS) (Fig. 2 ).

Mask length Drop ratio Drop diff. Blob count Blob size Center distance I3D

6.1 ± 3.3 3.7 ± 9.7 0.2 ± 0.3 1.5 ± 0.35 33.9 ± 19.7 54.8 ± 32.9 C-LSTM 9.6 ± 4.2 2.1 ± 3.9 0.08 ± 0.2 3.6 ± 0.6 25.9 ± 23.4 96.8 ± 34.7

Figure 2 : Best displayed in Adobe Reader where the figures can be played as videos. I3D (left) and C-LSTM (right) results for validation sequences from Something-something. The three columns show, from left to right, the original input, the Grad-CAM result, and the input as perturbed by the temporal freeze mask. The third column also visualizes when the mask is on (red) or off (green), with the current frame highlighted. OS: original score (softmax output) for the guessed class, FS: freeze score, RS: reverse score and CS: score for the ground truth class when there was a misclassification.

5.2. INTERPRETABILITY RESULTS ON THE KTH ACTIONS DATASET:

In Fig. 3 , we observe results for the class 'handclapping'. Interestingly, the mask of each model covers at least one entire cycle of the action. The mask is smaller for C-LSTM and for that reason does not lower its score as much as for I3D, whose freeze score is very low compared to the original and reverse score. This can be further explained by watching the frozen sequence and observing that no full cycle remains from the action. The reverse perturbation affects both models very little since one action cycle is symmetrical in time. For the 'running' class, we see that the temporal mask identifies the frames in which the subject is in-frame as the most salient for both models. However, the Grad-CAM results show that the I3D model places more focus on the subject's legs than the C-LSTM version. This is also reflected in the temporal mask for I3D, which activates first when it has started to shift its focus to the legs. 

6. CONCLUSIONS AND FUTURE WORK:

In this work we have presented a comparison of the spatiotemporal information used by 3D CNN and C-LSTM based models to perform video classification on two datasets, aiming to narrow down what they learn, and how they differ from one another. We analyzed the spatial information used by each model using the Grad-CAM method, and proposed the temporal mask method to investigate which video segments are most important for the classification. The comparison suggests that the 3D CNN on average focuses on shorter and more specific sequences than the C-LSTM model. On average, the 3D CNN also tends to focus on fewer or a single contiguous spatial patch, instead of smaller areas on several objects like the C-LSTM. Also, when comparing the effect of removing motion either through 'freezing' the most salient frames or reversing their order, the C-LSTM experiences a relatively higher decrease in prediction confidence than I3D upon reversal. We have also seen that the proposed temporal mask is capable of identifying salient frames in sequences, such as one cycle of a repetitive motion, or the instance of a passing motion.

There is still much to explore in the patterns lying in temporal dependencies. The compared architectures had a difference in performance on the more difficult Something-something dataset. If an established C-LSTM architecture that performs equally well becomes available in the future, it would be of interest to revisit this comparison. Likewise, it would be of interest to extend the study to other datasets where temporal information is important, e.g. Charades (Sigurdsson et al. (2016) ).

Other possible future work includes evaluating the effect of other noise types beyond 'freeze' and 'reverse'. We also believe that in the future it would be of interest to gain further insight into stateof-the-art models performing video classification benchmarks by utilizing the proposed tools.

A APPENDIX:

A.1 TRAINING HYPERPARAMETERS 

:

For the drop ratio, if the denominator OS-RS ≤ 0.01, the sample was filtered out since its ratio would explode. The OS-FS ≤ 0.01 were also excluded for balance. Without any filtering, the results were 323 ± 8640 (I3D) and

22.0 ± 588 (C-LSTM).

Figures and Tables:

Figure 1: I3D network (figure from Carreira & Zisserman (2017)) and C-LSTM network (right).

Figure 3: The figures can be displayed as videos in Adobe Reader. Same structure as Fig. 2.

FURTHER SEQUENCE EXAMPLES: SOMETHING-SOMETHINGBelow, we present results for 22 additional randomly selected sequences (two from each class) from the Something-something dataset. As mentioned in the main article, we selected eleven classes where the two models had comparable performance. The four classes not appearing above are (I3D F1-score/C-LSTM F1 score): moving something and something so they collide with each other (0.16/0.03), burying something in something (0.1/0.06), turning the camera left while filming something (0.94/0.79) and turning the camera right while filming something (0.91/0.8).OS: 1.000 FS: 0.015 RS: 0.003 Sequence #9: Turning the camera downwards while filming something. OS: 0.373 FS: 0.547 RS: 0.224

F1-score of each model on the datasets KTH Actions and Something-something.

Table content: <table><tr><td>Model</td><td colspan="3">KTH Actions (Top-1) Smth-Smth (Top-1) Smth-Smth (Top-5)</td></tr><tr><td>C-LSTM</td><td>0.84</td><td>0.23</td><td>0.48</td></tr><tr><td>C-LSTM (reversed)</td><td>0.78</td><td>0.05</td><td>0.17</td></tr><tr><td>I3D</td><td>0.86</td><td>0.43</td><td>0.73</td></tr><tr><td>I3D (reversed)</td><td>0.80</td><td>0.09</td><td>0.27</td></tr></table>

Hyperparameters used for training the two models on each dataset.

Table content: <table><tr><td>Hyperparameter</td></tr></table>

Hyperparameters used for optimizing the temporal mask.

Table content: <table><tr><td/><td/><td/><td colspan="2">Hyperparameter</td></tr><tr><td>Dataset</td><td>λ 1</td><td>λ 1</td><td colspan="3">Optimizer Iterations Learning rate</td></tr><tr><td colspan="3">Smth-smth 0.01 0.02</td><td>ADAM</td><td>300</td><td>0.001</td></tr><tr><td>KTH</td><td colspan="2">0.02 0.04</td><td>ADAM</td><td>300</td><td>0.001</td></tr></table>

Peer review (iclr2020):
This paper presents a paradigm for generating saliency maps for video models, specifically, I3D (3D CNN) and C-LSTM. It extends Fong & Vedaldi, 2017 to generate a temporal mask and introduces two types of *meaningful perturbations* for videos: freezing and reversing frames; they use Grad-CAM (with no modifications) for generating spatial masks. The problem is well-motivated, as saliency maps have been extensively studied for image classification models, but rarely for video classification. Quantitatively, they demonstrate that frame-reversal is meaningful for the Something-something dataset but less for KTH because those actions rely more on spatial information than temporal (i.e., running, clapping). Qualitatively, they show their spatial and temporal masks on both datasets and suggest the a few insights: * I3D*s Grad-CAM visualizations show a center, default bias * I3D is less sensitive to the reverse perturbation * I3D temporal masks are typically shorter. I currently rate this paper as a weak reject (though closer to borderline) paper for the following reasons (all of which can be improved in rebuttal): 1. Quality of technique While the temporal masks are novel and qualitatively *make sense* (though this is subjective), the generation of the spatial masks is not novel, is unconnected to the temporal mask generation, and often doesn*t make sense because of lack of temporal smoothness / cohesion, particularly for C-LSTM, where the visualizations when the mask is on appear quite *jumpy* (see Fig 2, Seq 1). It would be great to see more innovation on the temporal mask generation to address some of these issues (one natural approach that comes to mind would be learn spatial masks as done in Fong & Vedaldi, 2017, possibly with a temporal smoothness term between spatial masks and possibly combining temporal + spatial masks for freezing operation, i.e., only freeze spatial pixels) -- that said, I realize that this may be out of scope for a rebuttal. 2. Lack of support for qualitative claims The paper makes 3 claims based on qualitative examples shown (see above bullets in summary); these claims could be easily substantiated qualitatively (by evaluating over the dataset or a subset of it). 3. Lack of discussion on limitations/benefits of technique + how to use/interpret technique * There are no baseline comparisons for the proposed temporal mask generation. natural ones would be visualizing saliency methods (i.e., gradient, SmoothGrad, occlusion [Zeiler & Fergus, 2014], RISE [BMVC 2018], for instance, as examples of a few easy-to-implement, representative methods for backprop and perturbation methods) w.r.t. to temporal dimensions and then thresholding and applying those baseline temporal masks and demonstrating that the proposed temporal mask generation is. * There*s no discussion (or experiments) on the benefits & limitations of their approach; this is important as we*ve seen from papers like Mahendran & Vedaldi, ECCV 2016, Adebayo et al., NeurIPS 2018 and Kindermans et al., arXiv 2017 that some saliency methods fail to meet basic desirata (i.e., specificity to output class and model weights, etc.). Ideally, the authors would show that their temporal mask generation meets some desired criteria (as well as compare with baseline methods) to justify their approach. * One limitation of Fong & Vedaldi 2017 is the difficulty of finding global optimum for the different hyper-parameter terms (Fong et al, ICCV 2019 addresses this, which might be of interest to the authors). There*s no discussion about whether this problem persists for this work. Also -- it*d be interesting to see whether the reverse loss function (i.e., maximizing class score) yields similar results. * More discussion can be added about how to interpret results / use the technique (i.e., what is this technique useful/not useful for? what do results mean?). For instance, the claim that I3D temporal mask is shorter suggests a complex phenomenon -- that the necessary temporal evidence is smaller. This is a bit surprising to me, as I3D performs better overall, so I would have expected it to encode more redundancy (this can be checked by exploring the classes in which I3D does perform better) -- however, this interpretation differs from that of the authors (*This is especially visible in the temporal mask of Sequence #3, where it is active specifically*). In addition to responding to the above with relevant text + preliminary experimental results, I also had the following questions/asks: * For misclassified classes in Fig 2, are you optimizing w.r.t. the top predicted class or the ground truth class? Do they differ substantially when optimizing for different output classes? * What were the lambda hyperparameters from Eq. 1 (and how were they chosen)? Are these relatively stable or are their instances of technique failure due to the difficulty in balancing these terms? * Show Table 1 on only the classes that were focused on (i.e., the ones w comparable performance between the two models); I*m wondering if the impact of reversal on I3D is less than that on C-LSTM, as claimed by the authors (it*s hard to tell when their baseline performance is different)