Interpreting video features: a comparison of 3D convolutional networks and convolutional LSTM networks
Authors: Joonatan Mänttäri*
Sofia Broomé*
John Folkesson
Hedvig Kjellström
Venue:  (2020)
Keywords: interpretability
spatiotemporal
video
features
saliency
temporal

Abstract: A number of techniques for interpretability have been presented for deep learning
in computer vision, typically with the goal of understanding what it is that the networks
have actually learned underneath a given classification decision. However,
when it comes to deep video architectures, interpretability is still in its infancy and
we do not yet have a clear concept of how we should decode spatiotemporal features.
In this paper, we present a study comparing how 3D convolutional networks
and convolutional LSTM networks respectively learn features across temporally
dependent frames. This is the first comparison of two video models that both
convolve to learn spatial features but that have principally different methods of
modeling time. Additionally, we extend the concept of meaningful perturbation
introduced by Fong & Vedaldi (2017) to the temporal dimension to search for the
most meaningful part of a sequence for a classification decision.

Under review as a conference paper at ICLR 2020
INTERPRETING VIDEO FEATURES: A COMPARISON OF 3D CONVOLUTIONAL NETWORKS AND CONVOLU- TIONAL LSTM NETWORKS
Anonymous authors Paper under double-bling boindd review
ABSTRACT
A number of techniques for interpretability have been presented for deep learning in computer vision, typically with the goal of understanding what the networks have actually learned underneath given classification decision. However, when it comes to deep video architectures, interpretability is still in its infancy and we do not yet have clear concept of how to decode spatiotemporal features. In this paper, we present a study comparing how 3D convolutional networks and con- volutional LSTM networks, respectively, learn features across temporally depen- dent frames. This is the first comparison of two video models that both convolve to learn spatial features but that have principally different methods of modeling time. Additionally, we extend the concept of meaningful perturbation introduced by byoong & Vedaldi (2017) the temporal dimension to search for the most mean- ingful part of sequence for a classification decision.
INTRODUCTION
Two standard approaches to deep learning for sequential image data are 3D Convolutional Neural Networks (3D CNNs), e.g. the I3D model (Carreira & Zisserman (2017)) and recurrent neural net- works (RNNs). Among the RNNs, the convolutional long short-term memory network (C-LSTM) (hhi et al. )0155) is especially suited for sequences of images, since it learns both spatial and tem- poral dependencies simultaneously. Although both methods can capture aspects of the semantics pertaining to the temporal dependencies in video clip, there is fundamental difference in how 3D CNNs treat time compared to C-LSTMs. In 3D CNNs the time axis is treated just like a third spatial axis, whereas C-LSTMs only allow for information flow in the direction of increasing time, complying with the second law of thermodynamics. More concretely. C-LSTMs maintain a hidden state representing the current video frame when forward-traversing the input video sequence, and are able to model non-linear transitions in time. 3D CNNs instead convolve (i.e. take a weighted average) over both the temporal and spatial dimensions of the sequence.
The question investigated in this paper is whether this difference has consequences for how the two models compute spatiotemporal features. We present qualitative study of how 3D CNNs and C- LSTMs respectively compute video features: what do they learn, and how do they differ from one another?
As outlined in Section 2. there is a large body of work on evaluating video architectures spatial and temporal correlations, but significantly fewer investigations of what parts of the data the net- works have used and what semantics relating to the temporal dependencies they have extracted from it. Deep neural networks are known to be large computational models, whose inner workings are dif- ficult to overview for human. For video models, the number of parameters is typically significantly higher which makes their interpretability all the more pressing.
We will evaluate these two types of models (3D CNN and C-LSTM) on tasks where temporal order is crucial. The 20BN-Something soweethiinnsmmetiingg V2 dataset (Mahdisoltani et al. (2018). hereon somthiiggssmmhhingggg will be central our investigations; it contains time-critical classes, agnos- tic to object appearance, such as move something from left to right or move something from right to left. We additionally evaluate the models on the smaller KTH actions dataset (Schuldt et al. (0044) )

---

Under review as a conference paper at ICLR 2020
Our contributions are listed as follows.
We present the first comparison of 3D CNNs and C-LSTMs in terms of temporal model- ing abilities and highlight the essential difference between their assumptions concerning temporal dependencies in the data.
We extend the concept of meaningful perturbation introduced by Fong & Vedaldi (2117) to the temporal dimension search for the most critical part of sequence for classification.
2 RELATED WORK
The field of interpretabilityy in the context of deep neural networks is still young but has made con- siderable progress for single image networks, owing to works such as Zeiler & Fergus (2013), [ii- monyan et al. (2014) and Montavon et al. (2018). One can distinguish between data centric and network centric methods for interpretability. Activity maximization, first coined by Erhhan et al. (2009), is network centric in the sense that specific units of the network are being studied. By cast- ing the maximization of the activation of certain unit as an optimization problem terms of the input, one can compute the optimal input for that particular unit by gradient ascent. In data centric interpretability methods, the focus is instead on the input to the network, to reveal which patterns of the data that the network has discerned.
Grad-CAM (Selvaraju al. (2017)) and the meaningful perturbations explored in inoon Vedaldi (2177). which form the basis for our experiments, belong to the data centric category. These two methods are further explained in Section 3 Layer-wise relevance propagation (LRP) (Montavon)
et (0188) as well as Excitation backprop (Zang et al. (2016)) are two other examples of data tric backpropagation techniques designed for interpretability. where the excitation backprop method
follows from a simpler parameter setting of LRP. Building excitation backprop by Zhang et al. (016)) Adel Bargal et al. 2018) produce saliency maps for video RNNs without the use of gra- dients. Instead, products of forward weights and activations
normalized in order to be used as conditional probabilities, which are back-propagated. We have chosen to use Grad-CAM for our experiments, since it is one of the saliency methods in [Adebayo et al. (2018) that passes the article's sanity checks, as well as for its simplicity of implementation and wide usage.
Limited works have been published with their focus on interpretability for video models (Feicht- enhofer et al. (2018), Sigurdsson et al. (2017). Huang et al. (2018), Ghodrati et al. (2018)) Other works have treated it, but with less extensive experimentation (Chattopadhyay et al. (2017)). ). while for example mainly presenting a new spatiotemporal architecture (Dwibedi et al. (2018) [Zou et al. (2018). We build on the work by Ghoorati et al.] (2018), where the aim is measure a network's ability to model video time directly, instead of via the proxy task of action classification, which is most commonly seen. Three defining properties of video time are defined in the paper: temporal symmetry, temporal continuity and temporal causality, and are each presented accompanied by measurable task. The third property is measured using the classification accuracy on the Something- something dataset. An important contribution of ours with respect this work is that compare 3D CNNs and C-LSTMM, whereas Ghodrati et al.] (2018) compare 3D CNNs to standard LSTMs. Their comparison can be argued slightly unfair, as standard LSTM layers only take ID input, and thus needs to collapse each image frame in the video to a vector, which removes some spatial dependencies in the pixel grid.
Similar to our work, Dwibedi et al. (208)) investigate the temporal modeling capabilities of convolu- tional RNNs (Convolutional Gated Recurrent Units) trained on segethinngsommttinn. The authors find that recurrent models perform well for the task, and a qualitative analysis of the learned hidden states of their trained model is presented. For each class of the dataset, they obtain the hidden states of the network corresponding to the frames of one clip and display its nearest neighbors from other clips' per-frame hidden state representations. These hidden states had encoded information about the relevant frame ordering for the classes. (iigurdsson al. (2017) examined video architectures and datasets on a number of qualitative attributes. Huang et al. (2018) investigate how much the ac- tual motion in clip contributes the classification performance of video architecture. To measure this, they perform classification experiments varying the number sub-sampled frames used for a clip to examine how much the accuracy changes result.
2

---

Under review as a conference paper at ICLR 2020
In a search-based precursor to temporal mask experiments, (aatkin & Hebert (2010) crop quences temporally to obtain the most discriminative sub-sequence for a certain class. The cropping corresponding to the highest classification confidence is selected being the most discriminative sub-sequence.
Feichtenhofer al. (018 present the first network centric interpretability work for video models. The authors investigate spatiotemporal features using activity maximization. [Zhou et al.] (2018) introduce the Temporal Relational Network (TRN) which learns temporal dependencies between frames through sampling the semantically relevant frames for a particular action class. The TRN module is put on top of convolutional layer and consists of a fully connected network between the sampled frame features and the output. Similar to Dwibedi et al. (2018), they perform temporal alignment of clips from the same class, using the frames considered most representative for the clip by the network. They verify the conclusion previously made by Xie et al. (2017), that temporal order crucial on momeethng--smmethinnn and show that their architecture sensitive to that. They also investigate which classes of Nomething-somethingg show the strongest sensitivity to temporal order.
3
APPROACH
3.1
TEMPORAL MASKS
The proposed temporal mask method aims to extend the interpretability of deep networks into the temporal dimension, utilizing meaningful perturbation of the input, as shown effective in the spatial dimension by Fong & Vedaldi (2017). When adopting this approach, it is necessary to define what constitutes meaningful perturbation. In the mentioned paper, mask that blurs the input as little as possible is learned for a single image, while still maximizing the decrease in class score. Our pro- posed method applies this concept of learned mask to the temporal dimension. The perturbation, in this setting, is a noise mask approximating either 'freeze' operation, which removes motion data through time, or a 'reverse" operation that inverses the sequential order of the frames. This way, we aim to identify which frames are potentially most critical for the network's 's classification decision.
The perturbing temporal mask is defined as vector of values on the interval [0,1] with the same length the input sequence. For the 'freeze' type mask, value of for a frame at index / duplicates the value from the previous frame at l onto the input sequence at l. The pseudocode for this procedure is given below.
for i in maskIndices do + (1 mask;) * ariginal/npul + mask; perturbedI ripu;
end for
For the 'reverse' mask type, all indices of the mask m that are activated are first identified (thresh- old 0.1). These indices are then looped through to find all contiguous sections, which are treated as sub-masks, m,, For each sub-mask, the frames at the active indices the sub-mask are re- versed. For example (binary for clarity), an input indexed as 41:16 perturbed with mask with the value [0,00, 0, 1, 1, 1, 0, 0,0, 0, 1, would result in the sequence with frame indices [1, 2, 8, 7, 6, 5, 4, 9, 10. 11, 12, 13, 15, 14, 16).
In order to learn the mask, we define a loss function (Eq. I to be minimized using gradient descent, similar to the approach in Fong & Vedaldi (017))
L = A1 Allm As A2||m||3 Fe,
where m is the mask expressed vector m E [0, 1]s, |1-IIt is the L norm, I1-11/ is the Total Variation (TV) norm, λ1 weighting factors, and F, is the class score given by the model for the perturbed input. The L norm punishes long masks, in order to identify only the most important frames in the sequence. The TV norm penalizes masks that are not contiguous.
This approach allows our method to automatically learn masks that identify one several contigu- ous sequences in the input. The mask is initialized centered at the middle of the sequence. To keep the perturbed input class score differentiable w.r.t. the mask, the optimizer operates mask vector that has values in R. A sigmoid function is applied to the mask before using it for the perturbing operation in order keep its values in the [0. range.
3

---

Under review as a conference paper at ICLR 2020
The ADAM optimizer is then used to learn the mask through 300 iterations of gradient descent. After the mask has converged, it is thresholded for visualisation purposes.
3.2 GRAD-CAM
Grad-CAM (Selvaraju et al. (017)) is a method for producing visual explanations in the form of class-specifie saliency maps for CNNs. One saliency map, Li. is produced for each image input based on the activations from k filters. A at the final convolutional layer. In order to adapt the method to sequences of images, the activations for all timesteps t in the sequences are considered.
Lige Σ
w A
wi
1
O
7
DA
where is a normalizing constant and 1e is the network output for the class Since the aim of the method is to identify which activations had the highest contribution to the class score, only positive values of the linear combination of activations are considered, as areas with negative values are likely to belong to other classes. By up-sampling these saliency maps to the resolution of the original input image, the aim is to examine what spatial data in specific frames contributed most to the predicted class.
4 EXPERIMENTS
4.1 DATASETS
The Something-something dataset contains over 220,000 sequences from 174 classes in a resolution of 224x224 pixels. The duration of the data more than 200 hours, and the videos were recorded against varying backgrounds from different perspectives. The classes are action-oriented and object- agnostic. Each class is defined as performing some action with one or several arbitrary objects, such as closing something or folding something. This encourages the classifier to learn the template actions, since object recognition does not give enough information for the classifying task. We train and validate according the provided split. Fig. 2 shows sequences from the validation set.
The KTH Actions dataset consists of 25 subjects performing six different actions (boxing, waving, clapping, walking, jogging, running) in four different settings, resulting in total of 2391 sequences, with a total duration of almost three hours. The videos are provided with a resolution of 160x 120 pixels 25 fps. They are filmed against homogeneous background with the different settings exhibiting varying lighting, distance to the subject and clothing of the participants. For this dataset, we trained on subjects 1-16 and evaluated on subjects 17-25 (Fig. 3). Both datasets have sequences varying from one to almost ten seconds. As 3D CNNs require fixed sequence length, all in- put sequences from both datasets were sub-sampled to cover the entire sequence in 16 frames for Something Something-sometting something and 32 frames for KTT Actions. The same set of sub-sampled frames was then used as input to each architecture for each dataset.
4.2 ARCHITECTURES AND EXPERIMENT DETAILS
Hyperparameters are listed in the appendix. Any remaining settings can be found in the code which will be made public in both Pytorch and Tensorflow.
I3D consists of three 3D convolutional layers, nine Inception modules and four max pooling layers (Fig. []. In the original setting, the temporal dimension of the output is down-sampled to two frames. In order to achieve higher temporal resolution in the produced Grad-CAM images, the strides of the first convolutional layer and the second max pooling layer eight activations in the temporal dimension for the 16 frame inputs. The Grad-CAM images are produced from the gradients of the class scores w.r.t. the final Inception module.
reduced to 1x2x2, producing
The C-LSTM architecture used for Something-somen something consisted of three C-LSTM layers, each followed by batch normalization and max pooling layers. The convolutional kernels used for each layer had size 5x5 and stride 2x2 with 32 filters. The C-LSTM layers return the entire transformed sequence input to the next layer, including the last layer before the fully connected layer, used for
4

---

Under review as a conference paper at ICLR 2020
Inflated Inception-V1
C-LSTM Model
7,51-11
11,27.27
Video
CaPn
Coey
Red Faedt
14
12
Predictions
Al
The: Paet:
Hoo eu
50.219220
Predictions
Argi
Cor
Figure I: 13D network (figure from Carreira & Zisserman (2017) and C-LSTM network (right).
the predictions. For KTH, the C-LSTM model had two layers with 32 hidden units each and dropout between the layers (p 0.5). These architectures were chosen as the best performing models after empirical experimentation with the number of layers, hidden units, stride and regularization. When calculating the Grad-CAM maps for the C-LSTM, the final C-LSTM layer was used.
We note that there is substantial difference in the number of parameters for each resulting model, with 12, 465, 614 for I3D and 324, 014 and for the three-layer C-LSTM. When introduced, the I3D architecture achieved state-of-the-an performance on several video recognition datasets. These properties combined suggest that I3D should have an advantage in performance over the two models. This was confirmed Something soooeiiingssmttinng... where the C-LSTM architecture could not reach the same overall performance as I3D (Table D]. Other architectural variants the C-LSTM model with a larger amount of parameters (up to five layers) were evaluated as well, but significant increase in performance was observed. Also, due to the computational complexity of back lack-prooagation through time (BPTT), the C-LSTM variants were significantly more time demanding to train and evaluate than their I3D counterparts. With this in mind, in order to make the comparison as fair as possible, classes chosen for which the performance of the two architectures were similar. The labels of these classes and their F1 scores for each architecture are shown in Section 5.1 and in the appendix.
RESULTS
The Fl-scores for both architectures and datasets are shown in Table [ In order to investigate how reliant the two models with the input sequences reversed. On Something-somethhe were affected drastically, with their top-1 classification scores dropping by 78% and 79%. respec- tively. This suggests that both models are in fact sensitive to the temporal direction, to almost the present results for same degree, when the test sequence entirely reversed. In Sections 5.1 5.2] when only the most salient portion of a sequence is reversed.
the temporal order of the input frames, further test conducted both the C-LSTM and I3D models
For both models, the highest scoring class after reversing the sequence was turning something upside down. This is perhaps not surprising, as the semantic meaning of the action holds even when played backward. The classes with the largest drop score after reversal for both models were those containing movement in a specific direction, such as turning the camera left while filming something or pushing something from left to right. Both models performed well when reversing the KTH Actions dataset. Most likely, this is because this dataset's classes have distinct spatial features.
Table 1: F1-score of each model on the datasets KTH Actions and somnethnng-something
Model
KTH Actions (Top-1)
Smth-Smth (Top-1)
Smth-Smth (Top-5)
C-LSTM
0.84
0.23
0.48
C-LSTM (reversed)
0.78
0.05
0.17
BD
0.86
0.43
0.73
BD (reversed)
0.80
0.09
0.27
5

---

Under review as a conference paper at ICLR 2020
5.1 INTERPRETABILITE RESULTS ON COMETHING-SOMETHING SOMETHING
In this section, we present the Grad-CAM heatmaps and temporal masks generated for each archi- tecture. We display eight sequences in Fig. 2, but have included more examples in the appendix. The classes of this section's 's analysis are as follows (I3D FI/C-LSTM FI): moving something and something away from each other (0.76/0.58), moving something and something closer to each other (0.77/0.57), moving something and something so they pass each other (0.37/0.31) moving something up (0.43/0.4), pretending take something from somewhere (0.1/0.07), moving the camera down while filming something (0.67/0.56), and moving the camera up while filming something (0. (0.81/0.73).
First, we note that the Something-sommthinn classes can be ambiguous (one class may contain an- other class) and for a few samples, arguably. even be incorrectly labeled. The latter can be seen for example in Sequence #2, where I3D's classification was moving something and something so they collide with each other and the C-LSTM model predicted pushing something with something. Although the two objects in the sequence do move closer to each other, they also touch at the very end, making both predictions technically correct. Another case of understandable confusion can be seen in Sequence #5, where I3D's classification was taking one of many similar things on the table. In this case the surface seen in the image is a tiled floor, and the object is a transparent ruler. Once the temporal mask activates during the lifting motion in the last four frames, the Grad-CAM images show that the model also focuses on two of the lines on the floor. These could be considered similar to the lines caused by the outline of the ruler, which could explain the incorrect classification.
A difference between the architectures is that the 13D model often focuses on contiguous, centered blobs, while the C-LSTM model attempts to find relevant spatial features in multiple smaller areas ( Examples can be seen in Sequences #1 and #3 of Fig. 2, where I3D focuses a single region covering both objects while the C-LSTM has activations for both of the objects and the surface affected by the movement. The I3D model also has bias of starting its focus around the middle of the screen as can be seen in Sequences #1 to #8, often even before the motion starts. The typical behavior for C-LSTM is instead to remain agnostic until the action actually starts (Sequence #7). For Sequence #7, the 13D maintains its foveal focus even after the green, round object is out of frame. For Sequence #8, the focus actually splits midway cover both the moped and some features on the wall, while the C-LSTM model focuses mainly on numerous features along the wall, as it usually does classes where the camera turns. The C-LSTM also seems to pay more attention to hands appearing in the clips, rather than objects, as can be seen in Sequences #1 to #4. The average distance to the center of the image for each blob in each frame is shorter for 13D (Table
Given the same number of iterations for the optimization of the temporal mask, the two models typi- cally reached different losses. Generally, I3D obtained a lower loss. For this reason, we consider the difference and ratic] between the score drops after the freeze and reverse perturbations as the most relevant measures of how sensible one model was for the reverse perturbation. We observe that, in general, the drop caused by the reverse perturbation is smaller for the C-LSTM than for I3D. How- ever, the freeze-reverse drops' difference and ratio are considerably higher for the I3D compared to C-LSTM (Table 2p. suggesting that 13D is less sensitive to the salient reverse perturbation.
Last, we note that the average temporal mask is shorter for the I3D model (Table 2). This suggests that it has learned to react to short, specific events in the sequences. As an example, the temporal mask of Sequence #3 is active only on the frames where the objects first pass each other, and in Sequence #2, it is active on the frames leading to the objects touching.
OS: 0.994
OS: 0.312 FS: 0.186
FS: 0.083
RS: 0.856
RS: 0.125
Sequence #1: Moving something and something away from each other.
For the drop ratio, if the denominator OS- RS < 0.01, the sample was filtered out since its ratio would explode. The OS-FS < 0.01 were also excluded for balance. Without any filtering, the results were 323 + 8640 (I3D) and 22.0 + 588 (C-LSTM).

---

Under review as a conference paper at ICLR 2020
Table 2: Statistics for the temporal masks and Grad-CAM maps subset of the Something- something validation set (791 samples). The analysis was run on 150 clips each (fewer if there were not 150 samples of that class in the validation set) of the seven classes presented in this section. The 'blobs' were computed per frame, using the blob detection tool from OpenCV (Bradski (2000). Drop ratio defined as OS-RS
0S-FF and drop difference (OS FS) (OS RS) (Fig. 2).
Mask length
Drop ratio
Drop diff.
Blob count
Blob size
Center distance
13D
6.1
3.3
3.7
9.7
0.2 + 0.3
1.5 + 0.35
33.9 + 19.7
54.8 + 32.9
C-LSTM
9.6
4.2
2.1
3.9
0.08 + 0.2
3.6 + 0.6
25.9 + 23.4
96.8 + 34.7
OS: 0.547
OS: 0.257
FS: 0.028
FS: 0.079
RS: 0,053
RS: 0.122
CS: 0,186
CS: 0.002
P: 38
Sequence Moving something and something closer to each other.
P: 135
OS: 0.999 FS: 0.002
OS: 0.788
FS: 0.392
RS: 0.414
RS: 0.537
Sequence #3: Moving something and something so they pass each other.
OS: 0.804 FS: 0.016
OS: 0.546 FS: 0.121
RS: 0.667
RS: 0.764
Sequence #4: Moving something
OS: 0.685
OS: 0.221
FS: 0.003
FS: 0.182
RS: 0.048
RS: 0.350
CS: 0.001
CS: 0.005
P: 146
Sequence #5: Moving something up.
P: 100
OS: 0.600
OS: 0.284 FS: 0.003
FS: 0.167
RS: 0.088
RS: 0.006
CS: 0.004
P: 27
Sequence #6: Pretending take something from somewhere.
OS: 1,000 FS: 0.001
OS: 0.158 FS: 0.063
RS: 0.011
RS: 0.093
Sequence #7: Turning the camera downwards while filming something.
OS: 0.990 FS: 0.001
OS: 0.806 FS: 0. 1.7
RS: 0.000
RS: 0.181
Sequence #8: Turning the camera upwards while filming something.

---

Under review as a conference paper at ICLR 2020
Figure 2: Best displayed in Adobe Reader where the figures can be played as videos. 13D (left) and C-LSTM (right) results for validation sequences from Something-somettingg The three columns show, from left right, the original input, the Grad-CAM result, and the input perturbed by the temporal freeze mask. The third column also visualizes when the mask is on (red) or off (green), with the current frame highlighted. OS: original score (softmax output) for the guessed class, FS: freeze score, RS: reverse score and CS: score for the ground truth class when there was misclassification.
5.2
INTERPRETABILITE RESULTS THE KTH ACTIONS DATASET
In Fig. 3, we observe results for the class 'handclapping', Interestingly, the mask of each model covers at least one entire cycle of the action. The mask is smaller for C-LSTM and for that reason does not lower its score as much as for I3D, whose freeze score is very low compared to the original and reverse score. This can be further explained by watching the frozen sequence and observing that no full cycle remains from the action. The reverse perturbation affects both models very little since one action eycle symmetrical in time. For the 'running' class, we see that the temporal mask identifies the frames in which the subject is in-frame the most salient for both models. However, the Grad-CAM results show that the I3D model places more focus on the subject's legs than the C-LSTM version. This is also reflected in the temporal mask for I3D, which activates first when it has started to shift its focus to the legs.
OS: 0.999
OS: 0.996 FS: 0.997
FS: 0.026
RS: 0.999
RS: 0.996
Handclapping, subject 18.
OS: 0.993 FS: 0.208 RS: 0.999
OS: 0.669 FS: 0.339
RS: 0.605
Rumning, subject 25.
Figure 3: The figures can be displayed as videos in Adobe Reader. Same structure Fig. 2.
6 CONCLUSIONS AND FUTURE WORK
In this work we have presented a comparison of the spatiotemporal information used by 3D CNN and C-LSTM based models to perform video classification on two datasets, aiming to narrow down what they learn, and how they differ from one another. We analyzed the spatial information used by each method, and proposed the temporal mask method to investigate which
model using the Grad-CAr video segments are most important for the classification. The comparison suggests that the 3D CNN on average focuses on shorter and more specific sequences than the C-LSTM model. On average, the 3D CNN also tends focus on fewer or a single contiguous spatial patch, instead of smaller areas on several objects like the C-LSTM. Also, when comparing the effect of removing motion either through 'freezing' the most salient frames reversing their order, the C-LSTM experiences a relatively higher decrease in prediction confidence than I3D upon reversal. We have also seen that the proposed temporal mask capable of identifying salient frames in sequences, such as one cycle of repetitive motion, or the instance of passing motion.
There is still much to explore in the patterns lying in temporal dependencies. The compared ar- chitectures had difference in performance on the more difficult Something-somen an established C-LSTM architecture that performs equally well becomes available in the future, would be of interest to revisit this comparison. Likewise, it would be of interest to extend the study to other datasets where temporal information is important, e.g. Charades (Sigurdsson et al. (0166). Other possible future work includes evaluating the effect of other noise types beyond 'freeze' and 'reverse'. We also believe that in the future it would be of interest to gain further insight into state- of-the-art models performing video classification benchmarks by utilizing the proposed tools.
dataset. If
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Julius Adcbayo, Justin Gilmer, Michael Muclly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for salieney maps. CoRR, www////1...a.. 180.0222,,, 2018. URLhttp:://aaa /atttp::/aaxx..or///bb///// 1abbs//8100 1810. 03292
Sarah Adel Bargal, Andrea Zunino, Donghyun Kim, Jianming Zhang, Vittorio Murino, and Stan Selaroff. Excitation backprop for mnss. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2.18.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-CAM++: Generalized Gradient- based Visual Explanations Deep Convolutional Networks. 2017. doi: 10.1109/ WACV
/ttt::://aaattee wttpa/awwww...aaaaaa/aa/aaaaaaaa 11163
Debidatta Dwibedi, Pierre Sermanet, and Jonathan Tompson. Temporal reasoning in videos using convolu- tional gated recurrent units. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Bernoulli, (1341):1-13, 2009. URL UtL.hctpp///grraa........ asu. /dd//file watfii/evviiewee network.
Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes, and Andrew Zässerman. What have we learned from deep representations for action recognition? pp. 1-64, 2018. doi: 10. 10.11000/0P.. URL.
http://arxxiv.or/abbs..
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In
The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
A. Ghodrati, E. Gavves, and C. G. M. Snoek. Video time: Properties. encoders and evaluation. In British Ma- chine Vision Conference, 2018.
Ghodrat 10MVC2011
De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Fei-Fei, and Juan Carlos Niebles. What makes a video a video: Analyzing temporal information in video understanding models and datasets. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR). June 2018.
Farzaneh Mahdisoltani, Guillaume Berger, Waseem Gharbieh, David J. Fleet, and Roland Memisevic. Fine- grained video classification and captioning. CoRR, tbs//100.00925, 2018. URL/ttp:://aaee htttp:///rxxvvorrgg/
008/18044 09235
Grégoire Montavon, Wojeiech Samek, and Klaus Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing: A Review Journal, 73:1-15, 2018. ISSN 10512004. doi:
10. 10.10016/.dsp..00 (tt.:.dsp..01..1.... 2017.10.011. /ttpst:s::// http:p:://doiio wrgg 1e.pg//1..111111 dsp. 2017 10.011
Scott Satkin and Martial Hebert. Modeling the temporal extent of actions. In Eumppan Conference Com- puter Vision, September 2010.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In Proceedings of 17th International Conference on Pattern Recognition, 2004, ICPR 2004., volume 3, pp. 92-36. JEEE, 2004.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.
Visual Explanations from Deep Networks via Gradient Based Localization. Proceedings of the IEEE International Conference on Computer Vision, 2017- 626, 2017. ISSN 15505499.
doi: N0.1109//CC22211..244 URLbtttp//aaxx org abs abs/1610. 02391
Xingjian Shi, Zhourong Chen, and Hao Wang. Convolutional LSTM Network A Machine Learning Approach for Precipitation Nowcasting arXiv: 1506. 04214v1 [ cs CV | 13 Jun 2015. pp. 1-11, 2015.
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference Computer Vision, 2016.

---

Under review as a conference paper at ICLR 2020
Gunnar A. Sigurdsson, Olga Russakovsky. and Abhinav Gupta. What Actions Needed for Understanding 02696
Human Actions Videos? 2017. /R...ttps//a......rr////a///
Karen Simonyan. Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. pp. 1-8, 2014. URL http::/aaxxv..rr///bbbb
1312. 2.2..6 6034
Saining Xie, Chen Sun, Jonathan Huang. Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal fea- ture learning for vidco understanding. CoRR, abs/1712.04851,, 2017. URL http://arxiv.ow wwgp/bbb.org//////oo//
04851
Matthew D. Zeiler and Rob Fergus. Visualizing understanding convolutional networks. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinfor- matics), 8689 LNCS(PART 1):818-833, 2013. ISSN 161 13349. doi: 10.1007/978-3-311 10590-111011111 {
Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. CoRR, 0bs/1608.005001 2016.
00507
Bolei Zhou, Alex Andonian. Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. European Conference on Computer Vision, 2018.
A APPENDIX
A.1 TRAINING HYPERPARAMETERS
Table 3: Hyperparameters used for training the two models on each dataset.
Hyperparameter
Model (Dataset)
Dropout Rate
Weight Decay
Optimizer
Epochs
Momentum
I3D (Smth-smth)
0.5
0
ADAM
13
I3D (KTH)
0.7
5E-5
ADAM
30
C-LSTM (Smth-smth)
0.0
SGD
105
0.2
C-LSTM (KTH)
0.5
IE-4
SGD
21
0.2
Table 4: Hyperparameters used for optimizing the temporal mask.
Hyperparameter
Dataset
A1
A1
Optimizer
Iterations
Learning rate
Smth-smth
0.01
0.02
ADAM
300
0.001
KTH 0.02
0.04
ADAM
300
0.001
A.2 FURTHER SEQUENCE EXAMPLES: DOMETHING-SOMETHIEE
Below, we present results for 22 additional randomly selected sequences (two from each class) from the Something something-someething dataset. As mentioned in the main article, we selected eleven classes where the two models had comparable performance. The four classes not appearing above are (I3D F1 F1-score/C-LSTM F1 score): moving something and something so they collide with each other (0.16/0.03), burying something something (0. (0.1/0.06), turning the camera left while filming some- thing (0.94/0.79) and turning the camera right while filming something (0.91/0.8).
OS: 1.000
OS: 0.373
FS: 0.015
FS: 0.547
RS: 0.003
RS: 0.224
Sequence #9; Turning the camera downwards while filming something.
10

---

Under review as a conference paper at ICLR 2020
OS: 0.999
OS: 0.921 FS: 0.238
FS: 0.000
RS: 0.000
RS: 0.460
Sequence #10: Turning the camera downwards while filming something.
OS: 0.997 FS: 0.012
OS: 0.988 FS: 0.183
RS: 0,014
RS: 0.105
Sequence #11: Turning the camera left while filming something.
OS: 0.999 FS: 0.001
OS: 0.985 FS: 0.229
RS: 0,451
RS: 0.094
Sequence #12: Turning the camera left while filming something.
PATIING
PLRVIN TUE
CLBIE TOO
AURING
OS: 0.192
OS: 0.940 FS: 0.106
THE
FS: 0.261
RS: 0.140
RS: 0.017
CS: 0.103
P: 157
Sequence #13: Turning the camera right while filming something.
OS: 0.947 FS: 0.005
OS: 0.708
FS: 0.093
RS: 0.188
RS: 0.119
Sequence #14: Turning the camera right while filming something.
OS: 0.999 FS: 0.001
OS: 0.687 FS: 0.205
RS: 0.002
RS: 0.149
Sequence #15: Turning the camera upwards while filming something.
OS: 0.997 FS: 0.002
OS: 0.689 FS: 0.108
RS: 0.064
RS: 0.129
Sequence #16: Turning the camera upwards while filming something.
OS: 0.297
OS: 0.917 FS: 0.058
FS: 0.155
RS: 0.294
RS: 0.071
CS: 0.085
P: 121
Sequence #17: Moving something and something away from each other.
11

---

Under review as a conference paper at ICLR 2020
OS: 0.259
OS: 0.991 FS: 0.022
FS: 0.081
RS: 0.146
RS: 0.956
CS: 0.008
P: 130
Sequence #18: Moving something and something away from each other.
OS: 0.273
OS: 0.123
FS: 0.004
FS: 0.375
RS: 0,245
RS: 0.200
CS: 0.001
CS: 0.012
P: 173
Sequence #19: Moving something and something closer to each other something.
P: 100
OS: 0.932
OS: 0.453
FS: 0.002
FS: 0.063
RS: 0.007
RS: 0.198
Sequence #20: Moving something and something closer to each other
something.
OS: 0.686 FS: 0.003
OS: 0.620
FS: 0.145
RS: 0.000
RS: 0.129
Sequence #21: Moving something and something so they collide with each other.
OS: 0.333
OS: 0.810 FS: 0.055
FS: 0.119
RS: 0.276
RS: 0.419
CS: 0.030
Sequence #22: Moving something and something so they collide with each other.
3:
OS: 0.997 FS: 0.007
OS: 0.737
FS: 0.490
RS: 0.974
RS: 0.140
Sequence #23: Moving something and something so they pass each other.
OS: 0.694
OS: 0.813
FS: 0.010
FS: 0.227
RS: 0.003
RS: 0.830
CS: 0.273
CS: 0.142
P: 37
P: 37
Sequence #24: Moving something and something so they pass each other.
12

---

OS: 0.973 FS: 0.978 Under review as a conference paper at ICLR 2020
OS: 0.619 FS: 0.010
OS: 0.1298
FS: 0.079
RS: 0,216
RS: 0.262
CS: 0.020
CS: 0.001
P: 145
P: 157
Sequence #25: Burying something in something.
OS: 0.177
OS: 0.112
FS: 0.007
FS: 0.147
RS: 0.130
RS: 0.327
CS: 0.027
CS: 0.002
P: 106
Sequence #26: Burying something in something.
P:
OS: 0.848
OS: 0.229
FS: 0.065
FS: 0.102
RS: 0.380
RS: 0.269
CS: 0.003
CS: 0.003
P: 27
P: 100
Sequence #27: Moving something up.
OS: 0.230
OS: 0.755
FS: 0.146
FS: 0.012
RS: 0.200
RS: 0.032
CS: 0.033
P: 100
Sequence #28: Moving something up.
OS: 0.810
OS: 0.179
FS: 0.019
FS: 0.073
RS: 0,682
RS: 0.162
CS: 0.000
CS: 0.004
P: 160
P: 160
Sequence #29: Pretending take something from somewhere.
OS: 0.325
OS: 0.418
FS: 0.012
FS: 0.062
RS: 0.126
RS: 0.266
CS: 0.047
CS: 0.011
P: 145
P: 160
Sequence #30: Pretending to take something from somewhere.
A.3 FURTHER KTH ACTION EXAMPLES
Below, we present results for six additional randomly selected sequences (one from each class) from KTH Actions dataset.
OS: 1.000 FS: 1.000
RS: 1.000
RS: 0.973
Boxing, subject 17.
OS: 0,946 FS: 0.009
OS: 0.715 FS: 0.507
RS: 0.982
RS: 0.765
Hand clapping, subject 17.
13

---

Under review as a conference paper at ICLR 2020
OS: 1.000 FS: 0.012 RS: 1.000
OS: 0.896
FS: 0.349
RS: 0.761
Hand waving, subject 18.
OS: 0.662
OS: 0.991
FS: 0.354
FS: 0.353
RS: 0.706
RS: 0.002
CS: 0.172
Jogging, subject 25.
4
OS: 1,000 FS: 0.025
OS: 0.524
FS: 0.274
RS: 1.000
RS: 0.932
Running, subject 24.
OS: 1.000
OS: 0.724 FS: 0.333
FS: 0.010
RS: 0.775
RS: 0.654
Walking, subject 25.
B SOMETHING-SOMETIE CLASSES
"Approaching something with your camera":"O", "Ar" "Attaching something to something"""1" "Bend- ing something that it deforms":"" "Bending something until it breaks":'3", "Burying something in something": """4" "Closing something":"""" "Covering something with something". """."""n""""""..." "Digging something out of something": "Dropping something behind something":""" "Dropping some- thing front of something": """"" "Dropping something into something" ""oiin"""""""""" "10", "Dropping some- thing next to "mmmtthin"""111""" "Dropping something onto something". "1"", "Failing to put some- thing into something because something does not fit" "Folding something" :""" "Hitting something with something"."15: "Holding something" "0"" "Holding something behind some- 1"""" "Holding something in front of something" 18""" "Holding something next to some- thing".""19" "Holding something over something":
"Laying something on the table on its side, not "pright":""1"" "Letting something roll along flat surface":""2", "Letting something roll down slanted "urface":"23"" "Letting something roll up a slanted surface, so rolls back down":"24". "Lifting surface with something on it but not enough for it to slide down":"25" "Lifting surface with something on until starts sliding down":"26" "Lifting something up completely without letting it drop down":"2?". "Lifting something up completely, then letting it drop down":"28"" "Lift- ing something with something on it":" "Lifting up one end of something without letting it drop down":"30". "Lifting up end of something, then letting drop down":"" "Moving away from something with your camera":"33 "Moving part of something":.33. surface until it falls down":"34" "Moving something across surface without falling down":"35", "Moving something and something away from each other":"36", "Moving something and some- thing closer each other"."37", "Moving something and something they collide with each """""" "Moving something and something so they pass each """"r","3"", "Moving something away from something""""40" "Moving something away from the camera":"41" "Moving something closer to Nomething"""444 ""2"", "Moving something down":"43". "Moving something towards the cam- era":"44". "Moving something up": "4""" "Opening nomething":"455 "Picking something "4"""47". "Piling something "8", "Plugging something into something"."" """""" "Plugging something into something but pulling it right out as you remove your hand":"50", "Poking a hole into some sub- Mancc""""11 "Poking hole into something "5"""."52", "Poking stack of something the stack collapses": """33" "Poking a stack something without the stack collapsing":""5 ":"5"" "Poking some- thing it slightly moves":"55" "Poking something so lightly that it doesn't almost doesn't 't mov""."56". "Poking something so that it falls "'""","57", "Poking something so that spins ar""nd"""58" "Pouring something into something"."59" "Pouring something into something until
"Moving something across a
14

---

Under review as a conference paper at ICLR 2020
it overflows"""60" "Pouring something onto something"""61". "Pouring something out of some- thing":" "Pretending or failing to wipe something off of something": :'"3", "Pretending or trying and failing to twist something""" """4" "Pretending to be tearing something that is not tearable":"65 "Pretending to close something without actually closing it":"66", "Pretending to open something without actually opening it":"67", "Pretending pick something up":"68", "Pretending to poke something": "6"9" "Pretending to pour something out of something, but something is empty":"70", "Pretending to put something behind something":"". "7"", "Pretending to put something into some- thing"."72" "Pretending to put something next to something" """3", "Pretending to put something on a surface": "Pretending to put something onto something""""55
"Pretending to put something underneath something": "76"", "Pretending to scoop something up with something":"77" "Pretend- ing to spread air onto "omething"""7""" "Pretending to sprinkle air onto something": "Pretend- ing to squeeze something": ""00""""""""""""""""""""""""","" "Pretending to take something from somewhere": "81"" "Pretending to take something out of something"
"Pretending to throw something": "833" "Pretending to turn something upside down"""84", "Pulling something from behind of something"."85" "Pulling something from left to right":"86", "Pulling something from right to left":"87", "Pulling something onto something": "8"""" "Pulling something out of something". """"", "Pulling two ends of some- thing but nothing happens": "900", "Pulling two ends of something so that it gets stretched"""91" "Pulling two ends of something so that it separates into two pieces":"92", "Pushing something from left to right"""'3", "Pushing something from right to left":"94" "Pushing something off of something" " ""95", "Pushing something onto something" "'"96", "Pushing something so it spins":"97", "Pushing something so that it almost falls off but doesn't":"99 "Pushing something so that it falls off the table":"99". "Pushing something so that slightly moves"." "Pushing something with something" "101", "Putting number of something onto something". "102"" "Putting something and something on the table":"103" "Putting something behind something": """O""", "Putting something in front of something" 11005", "Putting something into something": "oouthhing""""o"" "066", "Putting something next to something"""'000 '17"", "Putting something on a flat surface without letting it roll"."108" "Putting surface"""109" "Putting something
the edge of something so it is not supported and falls down":"110" "Putting something onto a slanted surface but it doesn't 't glide down":"1111 "Putting something onto something":"| 11"", "Putting something onto something else that cannot support it so it falls down":"1 "13", "Putting something similar to other things that are already on the "11""" "Putting something that can't 't roll onto a slanted surface, so it slides down":"115", "Putting something that can't 't roll onto a slanted surface, so it stays where it is"."116", "Putting something that cannot actually stand upright upright on the table, so it falls on its side":"117", "Putting something underneath something":"| 111"", "Putting something upright on the tab'""."119"" "Putting something, something and something on the ""ble"."120""" "Removing something, re- vealing something behind""""21"" "Rolling something on flat surface":"122" "Scooping some- thing up with something"."" 1223"" "Showing a photo of something to the camera":"124" "Show-
something
ing something behind something". "aanething"."neee ","1 "Showing something next to something":"122" "Show-
ing something on top of something":"1277 "Showing something to the camera":" "Show-
ing that something is "1"9"" "Showing that something is inside something": "1300" "Some-
thing being deflected from something". "131"" "Something colliding with something and both are being deflected"""132" "Something colliding with something and both come
halt"="133", "Something falling like a feather or "1p'"" "Something falling like a 11c5"""135", "Spilling something behind something"""13o" something onto something" "oaliing"""""" 118"", "Spinning something so it continues spinning":"133 1339"" "Spin- ning something that quickly stops spinning": 1'140", "Spreading something onto something":"14: "Sprinkling something onto something"." "'4"", "Squeezing comething"""1444 "Stacking number "Stuffing something into something"
"Spilling "Spilling something next to something"."
"Taking one of many simi- lar things on the table":"146" "Taking something from somewhere":" "1"47"" "Taking something out of something") " """44"" 148", "Tearing something into two pieces":"149", "Tearing something just a "Throwing something": 1511", "Throwing something against something":" """5"", "Throwing something in the air and catching it"""" "Throwing something in the air and let- ting it 1554" "Throwing something onto a surface":" 1555" "Tilting something with some- thing on it slightly so it doesn't 't fall down"."156"" "Tilting something with something on it until it falls "15"" "Tipping something "158" "TTipiig something with something in it over, so something in it falls out":"159" "Touching (without moving) part of something":: 1160"" "Trying but failing to attach something to something because doesn't 't "tick".""66 161", "Trying to bend something unbendable so nothing happens":" "1"62",, "Trying to pour something into something, but missing so it spills next to it":" "Turning something upside down":"164" "Turning the camera downwards
of something".""144
little bit":"
15

---

Under review as a conference paper at ICLR 2020
while filming something":"165" "Turning the camera left while filming something"."166" "Turn- ing the camera right while filming something"""167" "Turning the camera upwards while filming something". ""on""""""""""""""""""""""""""." "Twisting (wringing) something wet until water comes out":"16",, "Twisting something"" "Uncovering something"."1711 "Unfolding something":"172" "Wiping some- thing off of something"."173
C KTH ACTIONS CLASSES
"Boxing": "0", "Handclapping": "1", "Handwaving"""2"" "Jogging" ""3", "Running":"4", "Walk- ing":"5"
16


Reviews:
[{"review_id": "HJetz7Yp5r", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a paradigm for generating saliency maps for video models, specifically, I3D (3D CNN) and C-LSTM. It extends Fong & Vedaldi, 2017 to generate a temporal mask and introduces two types of \"meaningful perturbations\" for videos: freezing and reversing frames; they use Grad-CAM (with no modifications) for generating spatial masks. The problem is well-motivated, as saliency maps have been extensively studied for image classification models, but rarely for video classification. Quantitatively, they demonstrate that frame-reversal is meaningful for the Something-something dataset but less for KTH because those actions rely more on spatial information than temporal (i.e., running, clapping). Qualitatively, they show their spatial and temporal masks on both datasets and suggest the a few insights: \n* I3D's Grad-CAM visualizations show a center, default bias * I3D is less sensitive to the reverse perturbation * I3D temporal masks are typically shorter.\n\nI currently rate this paper as a weak reject (though closer to borderline) paper for the following reasons (all of which can be improved in rebuttal):\n1. Quality of technique\nWhile the temporal masks are novel and qualitatively \"make sense\" (though this is subjective), the generation of the spatial masks is not novel, is unconnected to the temporal mask generation, and often doesn't make sense because of lack of temporal smoothness / cohesion, particularly for C-LSTM, where the visualizations when the mask is on appear quite \"jumpy\" (see Fig 2, Seq 1). It would be great to see more innovation on the temporal mask generation to address some of these issues (one natural approach that comes to mind would be learn spatial masks as done in Fong & Vedaldi, 2017, possibly with a temporal smoothness term between spatial masks and possibly combining temporal + spatial masks for freezing operation, i.e., only freeze spatial pixels) -- that said, I realize that this may be out of scope for a rebuttal.\n\n2. Lack of support for qualitative claims\nThe paper makes 3 claims based on qualitative examples shown (see above bullets in summary); these claims could be easily substantiated qualitatively (by evaluating over the dataset or a subset of it).\n\n3. Lack of discussion on limitations/benefits of technique + how to use/interpret technique\n* There are no baseline comparisons for the proposed temporal mask generation. natural ones would be visualizing saliency methods (i.e., gradient, SmoothGrad, occlusion [Zeiler & Fergus, 2014], RISE [BMVC 2018], for instance, as examples of a few easy-to-implement, representative methods for backprop and perturbation methods) w.r.t. to temporal dimensions and then thresholding and applying those baseline temporal masks and demonstrating that the proposed temporal mask generation is.\n* There's no discussion (or experiments) on the benefits & limitations of their approach; this is important as we've seen from papers like Mahendran & Vedaldi, ECCV 2016, Adebayo et al., NeurIPS 2018 and Kindermans et al., arXiv 2017 that some saliency methods fail to meet basic desirata (i.e., specificity to output class and model weights, etc.). Ideally, the authors would show that their temporal mask generation meets some desired criteria (as well as compare with baseline methods) to justify their approach.\n* One limitation of Fong & Vedaldi 2017 is the difficulty of finding global optimum for the different hyper-parameter terms (Fong et al, ICCV 2019 addresses this, which might be of interest to the authors). There's no discussion about whether this problem persists for this work. Also -- it'd be interesting to see whether the reverse loss function (i.e., maximizing class score) yields similar results.\n* More discussion can be added about how to interpret results / use the technique (i.e., what is this technique useful/not useful for? what do results mean?). For instance, the claim that I3D temporal mask is shorter suggests a complex phenomenon -- that the necessary temporal evidence is smaller. This is a bit surprising to me, as I3D performs better overall, so I would have expected it to encode more redundancy (this can be checked by exploring the classes in which I3D does perform better) -- however, this interpretation differs from that of the authors (\"This is especially visible in the temporal mask of Sequence #3, where it is active specifically\").\n\nIn addition to responding to the above with relevant text + preliminary experimental results, I also had the following questions/asks:\n* For misclassified classes in Fig 2, are you optimizing w.r.t. the top predicted class or the ground truth class? Do they differ substantially when optimizing for different output classes?\n* What were the lambda hyperparameters from Eq. 1 (and how were they chosen)? Are these relatively stable or are their instances of technique failure due to the difficulty in balancing these terms?\n* Show Table 1 on only the classes that were focused on (i.e., the ones w comparable performance between the two models); I'm wondering if the impact of reversal on I3D is less than that on C-LSTM, as claimed by the authors (it's hard to tell when their baseline performance is different)\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJetEzXCur", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows a way to compare what is learned by two very different networks trained for a video classification task. The two architectures are state-of-the-art methods, one relying on 3d-CNNs (time= one dimension), the other on conv-LSTMs (time is treated sequentially, using hidden states to pass information). The idea of the authors is (i) to provide saliency maps for each of them, and (ii) to create interesting perturbations in order to measure the influence on the networks. The results indicate that these complex networks are usually focused on interesting features, and as we would imagine, LSTMs is more learning from temporal coherence than CNNs.\n\nOverall, I think the paper is worth to be published at ICLR, even if I am not aware of the recent publications in this field. The contribution in terms of method is small, but I think that such careful studies can be very fruitful to the community.\n\nPositive aspects:\n- A significant effort has been put to creating meaningful perturbations for this particular task, i.e. temporal dependance and coherence.\n- The effort to compare as best as possible two different approaches is fruitful, and very useful for the community as this is a real question to be raised.\n- The experiments are made also with care, on 2 different datasets, and large efforts were made on explaining the different results.\n\nQuestions/remarks:\n- As I was under Linux, I did not manage to view the videos, and I have not seen a link for downloading them. It would be best to add it as supplementary materials?\n- What do you mean by 'The TV norm penalizes masks that are not coherent'? what is coherent?\n- '...the mask is defined as a vector of values between [0,1]' : I understood that it was a real value between 0 and 1, but I think you meant more 'a binary vector'?\n- Have you looked at the importance of the sub-sampling in the CNN framework? i.e., since the LSTM framework does not have that, maybe the differences in activation also depend on the length of the sequence. Maybe the large sequences, where CNN and LSTM would have the same number of frames, are less different?\n- You are saying that the code will be made public, is it possible to make it public now?\n\nSmall remarks:\n- '3D CNNs instead instead convolve'\n- 'As outlined in section 2' when we are in the introduction\n- '(Mahdisoltani et al.(2018) contains...'. parenthesis.\n- Figure 1: the first figure is in too poor quality.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJlJ2kGpKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work attempts to experimentally investigate the difference between Conv3Ds and ConvLSTMs with the aid of visualization. The visualization focuses on two aspects: (1) Temporal masking for identifying key frames and key segments and (2) GradCAM for spatial saliency. Both visualization techniques are illustrated on two public available video classification datasets.\n\nThis work has some major issues:\n1. The quantitative results are not very relevant. It does not validate any of the following cases: (1) if Conv3Ds are more powerful, one should use the same number of parameters and compare classification results, or (2), in order to achieve the same level of accuracy, one of the two models is more parameter efficient. It is not clear which argument Table 1 is validating against if there is any as both parameters and accuracies vary. \n2. Qualitative results from Section 5.2 is not substantiated and underwhelming. For instance, it is hard to see why \"Conv3Ds has a bias around the center while ConvLSTMs find relevant spatial features in multiple smaller areas\". This is most likely due to visualization techniques other than the choice of models. \n3. The fundamental issue of this work is that it does not establish a hypothesis from the beginning and design experiments around it. Plenty of hand-wavy observations are made without further investigating the root of them, leaving readers unsatisfied, and quite often confused.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses interpretability in the video data domain. The authors study and compare the saliency maps for 3D CNNs and convolutional LSTMs networks, analysing what they learn, and how do they differ from one another when capturing temporal information. To search for the most informative part in a video sequence, the authors propose to adapt the meaningful perturbations approach by Fong & Vedaldi (2017) to the video domain using temporal mask perturbations. 
While all reviewers and AC acknowledge the importance and potential usefulness of studying and comparing different generative models in continual learning, they raised several important concerns that place this paper below the acceptance bar: 
(1) in an empirical study paper, an in-depth analysis and insightful evaluations are required to better understand the benefits and shortcomings of the available and proposed models (R5 and R2). Specifically: 
(i) providing a baseline comparison to assess the benefits of the proposed approach -- please see R5’s suggestions on the baseline methods; 
(ii) analyzing how the proposed approach can elucidate meaningful differences between 3D CNNs and LSTMs (R5, R2). The authors discussed in their rebuttal some of these questions, but a more detailed analysis is required to fully understand the benefits of this study. 
(2) R5 and R2 raised an important concern that the temporal mask generation developed in this work is grounded on the generation of the spatial masks, which is counterintuitive when analysing the temporal dynamics of the NNs - see R5’s suggestions on how to improve. 
Also R5 has raised concerns regarding the qualitative analysis of the Grad-CAM visualizations. Happy to report that the authors have addressed these concerns in the rebuttal, namely reporting the results in Table 2 and providing an updated discussion. R1 has raised a concern about the importance of the sub-sampling in the CNN framework, which was partially addressed in the rebuttal. 
To conclude, the AC suggest that in its current state the manuscript is not ready for a publication and needs a major revision before submitting for another round of reviews. We hope the reviews are useful for improving and revising the paper.

Decision: Reject