Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces intervolution, a new feature transform for neural networks that combines both content-to-content and content-to-relation interaction. According to the authors, this feature transform captures both spatio-temporal dynamics and visual information. The authors recast convolution, self attention and dynamic convolution into a unified framework and compare their method to these techniques under the proposed formulation. The paper indicates that dynamic transforms are limited for video representation because of the absence of content-to-content interaction, and they claim that their work is a step towards solving this problem. It features an in-depth analysis of intervolution, its empirical effects on 4 datasets, and an ablation analysis that sheds light on the impact of each component.
main_review: ## General thoughts
The paper is generally sound and proposes an interesting, although slightly convoluted feature transform. The general formulation unifying descriptions of convolution, self-attention, involution and lambda functions is elegant and easy to understand. The proposed transform does seem to introduce positive performance differentials, and the choice of datasets to address the proposed claim is adequate.

## Pros
- Interesting re-interpretation of dynamic feature transforms
- Strong performance on challenging datasets
- Detailed experiments: 4 datasets, comparison to multiple alternative networks, specific comparison to other spatio-temporal transforms
- In-depth ablation studies
- Good work done on improving the efficiency of the technique
- Strong additional information in supplemental, and code was submitted.

## Cons
- The re-interpretation of dynamic feature transforms takes a few liberties: 

1. The involution framework has a non-linearlity and batchnorm in its kernel generation process: it's calculatin is  $W_1\sigma(W_0 x_n)$ instead of $x_nP^T$ (eq. 3)
2. lambda convolution's $P^T$ matrix is also more than just a learnable matrix (l.134): it's actually a sum of a normalized key matrix and a position embedding matrix, where the key matrix is built through a projection of the context: in a nutshell, following the authors notation, $P^T$ should take $X_n^K$ as input. This makes it not as close to involution as the authors might suggest .

- The paper focuses on motion-centric datasets, but indicates that the intervolution framework is general enough to outperform existing alternatives in capturing both visual information and motion dynamics. If that's the case, testing the method on other data where visual information is a bit more prevalent would be interesting; however, there is no indication of how this technique would perform in other typical video datasets like Kinetics, Moments in Time or Jester (Jester would be a good dataset to analyze in this scenario)

- The idea appears a bit incremental and convoluted: the framework is a combination of several existing methods, and the resulting computational graph lacks the simplicity of its counterparts. This yields large computational complexity (although the authors address it quite well) and questions about the validity of each submodule. The ablation studies in table 4c appear to show that just using $\kappa_n^R$ and $X_n^R$ is enough to get extremely close to top performance: do we need the full setup?

- Section 5.3 is generally weak. l. 269: "softmax forces the kernel weights to be positive, so that the kernel may not compute gradients across frames" -> What do you mean by this?

- Table 3 is strong, but disorganized: missing multiple values, comparisons between networks that have different pre-trainings and objectives are bundled together, different networks are used for different datasets
 

## Small corrections
- In Fig. 2, the $1(X_n^Q)^T$ seems incorrectly placed (should go to the left of the element-wise)
- Section 5.3 has several typos and mistakes (e.g. l. 268 "it demonstrates the softmax" -> "it demonstrates that the softmax"
- In l. 173 and 174, it seems that there is a confusion between Value and Key (the text says value but the equation and figure show $X_n^K$

## Questions
- Why are the FLOPs almost equal for all alternatives in table 4a? specifically when comparing one kernel and one context vs two kernels and two contexts
- Any examples / justifications for the claim in paragraph at l. 180?
- In Table 4d, are you still using the TSN-ResNet50 network for this analysis?
- Why are you using ImageNet pretraining instead of Kinetics, Moments in Time or other large video-based dataset?
- L. 227: What is L? not introduced in the text before
- Any insight on the choice of backbone? Why TSN ResNet50 instead of a CSN or I3D?


### In general, good paper. If the cons are reasonably addressed and the small errors fixed, I believe this paper can be an adequate addition to Neurips.
limitations_and_societal_impact: Both limitations and societal impact have dedicated sections in the supplemental. In terms of societal impact, the authors outline potential risks in applications such as video surveillance. In terms of limitations, the authors outline the following:
- Computational complexity is still an issue and can be improved
- A more generalized dynamic transform could capture both visual appearance and relational features in a more natural way
- Further work is needed to extend this work to image understanding or NLP

Apart from that, I would point to the following additional limitations:
- The work here focuses on motion-centric datasets, but the proposed framework is introduced as general enough to beat and supersede other dynamic transforms that are successful in more general domains. I understand that this is a conscious choice by the authors, but it might point to a limitation: does this technique work on more general video datasets, such as Kinetics and Moments in Time?
- The technique lacks the simplicity that characterizes other dynamic transforms such as lambda layers. Is the combination of all 4 terms of intervolution really necessary? As mentioned above, the ablation studies in table 4c appear to show that just using $\kappa_n^R$ and $X_n^R$ is enough to get extremely close to top performance.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a relational feature transform that captures rich relational patterns for action recognition and compared to other dynamic feature transforms in learning motion dynamics in videos. Experimental results outperform state-of-the-art methods on standard motion-centric action recognition benchmarks.

main_review: **Strengths:**
1) The main idea of the Intervolution feature transform is well-motivated and straightforward.
2) The conducted experiments have shown the effectiveness of the proposed method compared to state-of-the-art methods.
3) The paper is well organized.


**Weaknesses:**

**Writing.** Although the paper is clearly organized, the reading is not smooth, and the figures are overloaded with equations, which does not make it easy to follow. Additionally, at least to me, the paper introduces a module for video action recognition and not video understanding (which is a broader aspect than recognizing actions). Still, I can live with the author's choice.

**Experiments.** Table 2 is missing a few crucial points. First, Non-local [1] and Long-Term feature bank [2] networks, which are full self-attention approaches, were proposed for capturing motion and long-range dependencies directly by computing interactions, and thus I expect to have a direct comparison with them. Second, by using only 8 frames for comparison, the authors prevent the other models from performing better. 

Additionally, in line 268, the authors mentioned, “It (Table 2) demonstrates the softmax function restricts the expressive ability of the kernel”. The self-attention is not permutation invariant if you use a positional encoding. Did the authors try to use self-attention with positional encoding (as in vision-transformers), and still, the softmax performed worse?.

Last, in line 289, the authors wrote, “Finally, our ensemble model with 2 clips achieves 56.1% and 67.7%, which set the new state-of-the-art on SS-v1 and v2 with reasonable computational cost“. Unfortunately, Table 3 does not include the number parameters, so I do not know if this is true. I believe that #params are essential for a fair comparison (at what costs do we improve?).


[1] Non-local Neural Networks

[2] Long-Term Feature Banks for Detailed Video Understanding


**Temporal understanding.** I would expect a model that claims that “re-interpreting recent dynamic feature transforms in a unified way” (line 44) to show more evidence of the learned dynamics. Can the authors present the kernel visualizations compared to the other methods or show the current top 5 actions compared to SlowFast? It is crucial to understand which actions the model recognized better compared to the existing methods. I also could not clearly understand Figure 3 in the Supplementary.


My main concern is the missing experiments which are crucial to validate this network compared to existing work (Non-Local and Long-term feature bank are also self-attention modules and should be compared in the same manner). On the other hand, I liked this paper, its contribution, and its motivation. I am open to the authors' feedback and other reviewers' opinions.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors tackle the problem of video classification. Motivated by the limitations of the recent video transformer methods, the authors propose a new relational feature transform called intervolution. The main idea behind this new module is that it should capture spatiotemporal relationships in video more effectively than the standard self-attention operator. The authors report strong performance on "temporally heavy" datasets such as Something-Something-V2, Diving-48, and FineGym while also having lower computational cost and using less labeled data in comparison to prior work.
main_review: Strengths:
- Current self-attention based video models such as TimeSformer [20] struggle to learn effective spatiotemporal features in settings that require strong temporal reasoning. Therefore, the proposed approach is timely, and well motivated. 
- The proposed approach achieves strong performance on a variety of temporally heavy datasets.
- The proposed approach is efficient and it also doesn't require large amount of external datasets for pre-training.
- The ablation studies are very thorough, providing evidence that every component is beneficial to the overall performance of the system.


Weaknesses:
- In my view, the readers who are not as familiar with recent self-attention based methods in the video domain, might find the motivation of this work not fully clear. It would be good if the authors could provide more intuitive examples for motivating their technical approach. For examples, the section in L149-153, which explains some of the limitations of standard self-attention in video domain is overly technical, and lacks simple and intuitive examples to understand why the proposed approach is needed.
- Furthermore, the subsection that explains the concept of the relational kernel can be quite difficult to understand at times. I think the authors could do a better job motivating the need for some of the equations/components, and also by simplifying the description a bit. For example, Equation 5 could be expressed using a simple matrix multiplication, but instead it contains additional terms/operations that are unnecessary in my view. In most recent papers, the query matrix is assumed to be the same as the context matrix. It seems that in practice that's the case in this paper as well. Therefore, I'm not sure if there's a need to introduce additional terms/operations and make the paper seem more complicated than it is. For instance, I think the paragraph in L186-L193 is written clearly, concisely, and with a good intuitive motivation in mind. Therefore, the rest of the draft should follow the same presentation style. In particular, I would also suggest to present the approach more in a top-down fashion (starting from Equation 7), instead of just listing all the components one after the other.
- Section 5.3 doesn't list the dataset that was used, which would be useful.
- Typo in L296.

Questions:
- Do all baselines in Table 2 use the same backbone, pre-training dataset, etc? Or is it completely different implementations of different methods?
- Have you tried running your method on Kinetics? I understand that Kinetics is a spatially biased dataset, but just curious if there's any improvement with respect to the baseline. 
- I was confused about the experiment in Table 4c). Which spatiotemporal kernel is M referring to? Is this in the bottleneck of the ResNet backbone? Is spatiotemporal kernel M related to the context size M? If not did you ablate the context size M? It's a bit confusing that both Kernel Size and Context Size is referred to as M. 
limitations_and_societal_impact: I believe so.
review_title: null

Meta-review: The paper unanimously receives positive rates. The authors’ rebuttal is through and excellent too. As the reviews show satisfaction on the authors’ feedback, the final draft needs to respect it accordingly. 
Decision: Accept (Poster)