Relational Self-Attention: What's Missing in Attention for Video Understanding
Authors: Manjin Kim
Heeseung Kwon
Chunyu Wang
Suha Kwak
Minsu Cho
Venue: NeurIPS 2021 Poster (2021)
Keywords: Action recognition
Video understanding
Motion analysis
Self-attention

Abstract: Convolution has been arguably the most important feature transform for modern neural networks, leading to the advance of deep learning.  Recent emergence of Transformer networks, which replace convolution layers with self-attention blocks,  has revealed the limitation of stationary convolution kernels and opened the door to the era of dynamic feature transforms. The existing dynamic transforms, including self-attention, however, are all limited for video understanding where correspondence relations in space and time, i.e., motion information, are crucial for effective representation. In this work, we introduce a relational feature transform, dubbed the relational self-attention (RSA), that leverages rich structures of spatio-temporal relations in videos by dynamically generating relational kernels and aggregating relational contexts. Our experiments and ablation studies show that the RSA network substantially outperforms convolution and self-attention counterparts, achieving the state of the art on the standard motion-centric benchmarks for video action recognition, such as Something-Something-V1&V2, Diving48, and FineGym. 

Relational Self-Attention:
What's Missing in Attention for Video Understanding
Manjin Kim
Heeseung Kwon
Chunyu Wang? Suha Kwak
Minsu Cho
POSTECH
Microsoft Research Asia
http://cvlab. htt.cc/labpppstcctteeoooo posteech....kr/reeeerrre
Abstract
Convolution has been arguably the most important feature transform for modern neural networks, leading to the advance of deep learning. Recent emergence of Transformer networks, which replace convolution layers with self-attention blocks, has revealed the limitation of stationary convolution kernels and opened the door to the era of dynamic feature transforms. The existing dynamic transforms, including self-attention, however, are all limited for video understanding where correspon- dence relations in space and time, i.e., motion information, are crucial for effective representation. In this work, we introduce relational feature transform, dubbed the relational self-attention (RSA), that leverages rich structures of spatio-temporal relations in videos by dynamically generating relational kemnels and aggregating lational contexts. Our experiments and ablation studies show that the RSA network substantially outperforms convolution and self-attention counterparts, achieving the state the art on the standard motion-centric benchmarks for video action recognition, such Something-Something V1 & Diving48, and FineGym.
1 Introduction
Convolution [13, 25] is a feature transform that is ubiquitous in modern neural networks for visual recognition, and has driven the development of deep learning in the past decade. The stationarity of convolution kernels, however, may limit its expressivity and hinder adaptation to diverse com- positional possibilities of visual concepts [18]. Dynamic convolution [20] and self-attention [52] that construct kernels or attentions according to input contents have emerged as an alternative to static convolution this context, being followed by further studies for dynamic feature trans- forms [6, 27, 33, 57]. The effectiveness of self-attention has been demonstrated by the success of Transformer variants on different image understanding tasks such as image classification [7, 51, 59], object detection [4], and semantic segmentation [43]. Recently, it has been further extended for video understanding, replacing spatio-temporal convolution [5, 48, 50] with spatio-temporal self- attention [1, 3, 35].
Despite their recent progress in the video domain, the existing dynamic feature transforms still leave much room for improvement in terms of learning relational patterns in space and time, i.e., motion information, which known to be essential for video understanding [23, 53]. For example, spatio- temporal self-attention [55] often fails to learn motion representation without positional embeddings as demonstrated in [32], and even those with positional embeddings [1. 3, 35] turn out to be not effective motion-centric action recognition benchmarks such
-ooietiinnn [14].
In this work, we introduce relational dynamic feature transform, dubbed relational self-attention (RSA), to address the limitation of existing methods, RSA leverages rich structures of spatio-tempore
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

relations in videos by dynamically generating relational kernels and aggregating relational contexts. Combining these relational components with ordinary dynamic kernels and context features, our RSA learns rich video representation that effectively captures both visual appearance and spatio-temporal motion dynamics.
The main contribution of this paper is three-fold:
We re-interpret recent dynamic feature transforms in unified way, and provide in-depth analysis on their capability of learning video representation.
We introduce a new dynamic feature transform, i.e., RSA, which effectively captures both motion dynamics for video understanding.
visual appearance and
Our RSA network substantially outperforms convolution and self-attention counterparts, achieving the state of the art on SS-VI&V2, Diving48, and FineGymm.
2 Related Work
Convolution and its variants. Convolution [13, 25] has been used as dominant neural primitive in modern neural architectures [17. 34, 38, 42, 45, 47]. Since convolution often suffers from its limited expressivity due to its static kernel, dynamic convolution operators have been studied for enhancing composability by dynamically adjusting the convolution kernel according to input features [6, 20, 27, 33, 57]. One example is involution [27], which generates lightweight dynamic kernel using input content and substantially outperforms convolution with less computational cost on image classification. Our RSA differs from these kernel generation methods in that it leverages relational patterns for learning a rich video representation.
Self-attention and its variants. Self-attention [52] was originally introduced for neural machine translation to capture long-range interactions, and now has been widely adopted in many different domains thanks to its versatility and expandability. Following local attention methods [18, 37] that employ self-attention as an alternative to convolution, ViT models have demonstrated impressive performance on a variety of image understanding tasks [4, 7, 43, 59]. On the other hand, some of previous work have attempted to design novel dynamic transforms through attention [2, 60]. Zhao
et al. [60] expand the spatial attention to include the channel attention. Bello [2] proposes lambda layer that focuses on interaction between visual contents and a relative position embedding without softmax, which outperforms self-attention counterparts on image classification. The proposed RSA is an extension of these techniques, yet focuses on learning rich relational features for video understanding.
Convolution and self-attention for video understanding. Action recognition is the most funda- mental task for video understanding, which aims at classifying a video clip into pre-defined action classes. key to success of this task to capture temporal dynamics across multiple video frames, and apatio-temporal (3D) convolution [48] has been de facto for modeling the temporal dynam- ies [8, 10, 31, 49, 50]. On the other hand, self-attention was used as an attachable module for 3D CNNs in the earliest method [55], yet nowadays becomes the major building block of video under- standing networks such as video VIT [1, 3, 35). However, both 3D CNNs and ViTs are not sufficient for modeling temporal dynamics due to the lack of inter-feature relations within the spatio-temporal context. Our RSA overcomes their limitation by explicitly utilizing relational features that imply temporal dynamics.
Learning motion for video understanding. Early approaches in video understanding rely on exter- nal motion extractors like optical flow for motion feature learning [5, 12, 41], but recent models aim to learn motion representation internally. To this end, they estimate feature-level optical flow inside their networks [9, 36] or compute subtraction between consecutive frame-wise features [21, 26, 29, 44]. Meanwhile, correlation-banl
methods [23, 24, 53] use inter-pixel relations as motion likelihood maps and achieve sate-offthheart on motion-centric action recognition benchmarks. Inspired by the correlation-basse methods, we apply dynamic relational kernels to the correlation tensors (i.e., relational context), leading to effective motion feature learning.

---

additoo X
doi-proddcc
Vec
redorizalion +6
softmma
p:
Yn
Yn
Yn
(C)
a
(M)
(C)
(x (M)
P
PTxy
(MCC)
VBC
(Ar)
(C
X
X
X ()
Xx (C)
X (c) Involution Lambda convolution
X
(MC g (C)
(M x C)
()
(a) Convolution
(b) Self-attention
Figure I: Computational graphs of different feature transforms. See text for details.
3 Background
In this section, we revisit three instances of feature transform and analyze them in a simplified form. In context, the role of feature transform is update feature map while preserving its order in space and/or time. To this end, it processes each position in space and/or time using features of its neighborhood. For example, given an input feature map of video X RTTxHXWxC for each spatio-temporal position (t, h, w), a feature transform takes the individual feature 2r,b as targer and transforms using its neighborhood features (æt M we (ee ,h' d.,,
its context. For the sake of notational convenience, let us denote the target by F. E IRC, where n represents specific position space and/or time, and its context by Xn E RMxx where M is the size of neighborhood, and the corresponding output by yn E Rº_ Now, without loss of generality, we can view the transform as applying each position function with learnable weights W that maps target F, to output Yn using context X, typically implemented as neural layer that processes all the targets in parallel.
f Xx, n. W). Note that such transform is
In the following, we describe recent feature transform functions in computer vision as well as the conventional one, convolution. Figure I illustrates the computational graphs of the transforms.
Convolution. Convolution [13, 25] is a static and translation-equivariann feature transform that updates each target æn by applying static kernels W MCxC on local context X,,
yo W vee X,)).
While extracting different visual patterns using multiple kernels, convolution remains static in the sense that the kernel is not affected by the target En, Le.., feature of interest (Figure la). This station- arity may not be effective in adapting to diverse compositional possibilities of visual concepts [18], and the channel-dependent weights can
inter-channel redundancy in the kernel [19].
Self-attention. Self-attention [52] is a dynamic transform that generates an attention map of context Xn using target H, and then aggregates the context using the attention map dynamic kernel (Figure 1b). The process starts by adapting inputs for query-key-value attention computation; using three learnable embedding matrices, E EK EV E CCC, it embeds the target 2n into the query Q via xº = x, EQ and also projects the context X,, into the key X and the value X via EK and x X,, respectively. The basic attention map is computed via the softmaxed product between query x and key X which can be viewed as content-to-come to-content interaction. To reinforce this content-only attention with positional information of the context, learnable matrix P E IR MC which encodes relative positions of individual features in the context, optionally included content-to-pee to-psittooo interaction. The self-attention transform aggregates the value embeddings X. of context using the attention map as kernel weights shared for all channels:
X
yx ( +x +
where (-) denote the softmax function. This transform cranslationeequivriann position em- bedding P is local and relative with respect to the target, and becomes content-to-position interaction is removed. Unlike convolution, dynamic the sense that the kernel or ( KyT x PT), which is used to aggregate the context, depends on the target content.
if

---

It allows more flexibility in adapting to input and also consumes fewer parameters by sharing the kernel across the channels [18].
Involution & lambda convolution. Involution [27] is a light- dynamic transform that lever- ages content-to-posiiic interaction only. It dynamically generates a kernel H RRM by projecting target v,, using learnable matrix P E IR IRMC xC via K. P,,, 1 and then use the kernel to aggregate the context X,, (Figure Ic, left):
yn
- Fr P X..
where the matrix P plays the role of converting the target to a kernel in position-sensitivy manner. In a similar spirit, lambda convolution of the lambda networks [2] uses content-to-positiee interaction between the target and the context position. As in self-attention, using two learnable embedding matrices, E EV RCxC it starts by projecting the target &n and the context X,, into the query 2, and the value X via x. = F, EQ and X = EVE,V respectively. The lambda convolution abstracts the context value X to a contextual matrix XØ using a learnable matrix P RMMxC via de- PTXX) E RCxC, which in turn is used for updating the target query 2, (Figure Ic, right):
ya o 2 2. 1.
Despite the differences in the operational procedures and the concepts, the lambda convolution has effectively the same form with involution except for additional key and value embeddings. Note that unlike self-attention, involution and lambda convolution both have no softmax nonlinearity. In fact, the absence of softmax reduces the computational cost and also increases the expressive ability by allowing negative activations [27]. Both involution and lambda convolution are shown to outperform convolution and self-attention counterparts in image classification [2, 27], demonstrating the effectiveness of dynamic content-aware kernels.
Limitation of existing dynamic transforms.
The aforementioned dynamic transforms are com- monly based on leveraging content-to-content and/or content-to-position interactions in constructing kernels, where the target content (input feature) is used as the source of the dynamic transform. While the methods are effective for learning image representation indeed, they are all limited for learning video representation; as will be shown in the experimental section 5.3, we have found that existing dynamic transforms show only marginal or virtually no improvement over the static transform of convolution on the motion-centric action recognition benchmark. The main reason lies interactions as a relational content in representation
in the missing structure of learning. While content-to-content interactions are considered in existing dynamic transforms, they are immediately broken into individuals without being used as whole. For example, self- self-attention computes query-to-key correlation x (XX)T but uses the individual elements of the correlation only for aggregating informative contents from the context. The content-to-position interaction 29 pl does not help in capturing the structure of interactions either since it has no access to the correlation as a whole. In videos, such structural patterns contain informative spatio-temporal contents, i.e., different levels of generic motion information, thus being crucial in video understanding.
4 Our approach
In this section, we introduce a new dynamic transform, dubbed relational self-attention (RSA), which is designed to learn rich spatio-temporal interaction patterns across input contents. Figure illustrates the computational graph of RSA. On top of the basic kernel and context in a dynamic transform, builds relational kernel and context and processes all the kemnel-context combinations. Here we describe the main components of RSA and their integrated form, and explain an efficient implementation of RSA.
As in self-attention, we start by adapting inputs for query-key-value interactions; using three learnable embedding matrices, EQ, Ek E R CCC target E, and context X,, are embedded into query Q key X and value X via 2 = x; E X = X,, and X = X, respectively. xx X and X V are then L2-normalized, and we omit the normalization term for the simplicity.
"This transform corresponds to the position lambda where the extent of the context x V local neighborhood of the target 2 [2].
is restricted to the

---

n Basic kemel
Relatinal kernel
Relstional context
K
(M)
1xx,)
(MC)
VBc
(M)
M c)
(X G
(M x ()
Basic context
(C A
dol-product
hadamardd product
P (M X C)
X (M C)
II (MC M)
X (M C)
column one veclor
(M X ()
vectorization
Figure 2: Computational graph of RSA. RSA consists of two types of kernels (basic and relational kernel) and two contexts (basic and relational context). See text for details.
4.1 Relational self-attention (RSA)
Relational kernel. The relational kernel is designed predict the relevance of context based on the structure of content-to-content interactions. To generate the relational kernel, we compute the dot- product correlation of the query and the key, i.e., x (X E AR, and then project the correlation vector using learnable matrix H E R.MMM
12
R K
(
The role of H in this relational kernel corresponds to that of P in the involution kernel (Eq, 3); while P predicts the kernel weights from the C-dinensional query vector, H predicts them from the M-dimensional query-key correlation vector. The resultant dynamic kernel aggregates the context depending how the individual contents are related to each other in space and time, thus being particularly effective in learning notion-rrllttd patterns in videos. Note that, when we set to an identity matrix I and add the softmax operator into Eq. 5, the relational kernel k is equivalent the self-attention kernel without content-to-positien interaction, i.e., o (x (x))))
Since the dot-product correlation contracts all channel dimensions of the query and the keys, may lose semantic information, which may help in generating an effective relational kernel. We thus take the Hadamard product [22] instead so that we can leverage channel-wiss query-ker correlations for producing the relational kernel. Using learnable kernel projection matrix H E I MfC M Eq. 5 can be reformulated as
(6) where @ denotes the Hadamard product. H E MCXM predicts the kernel weights from an MC- dimensional channel-wise query-key correlation vector. Furthermore, the absence of the softmax operator allows the relational kernel to have negative activations so that it better learns relative features where the subtractive interactions are beneficial for capturing relative changes of contents over time [21, 26, 29].
vec(1 O X
Relational context. The relational context is designed to provide the relational pattern of content- to-content (i.e.,
interactions for the kernels to aggregate. To this end, we use self-correlation [40], which effectively describes spatio-temporal intra-structure including motion information [23, 24, 53]. We first construct the self-correlation matrix X (XV) XVTT E I x and then project it using learnable matrix G E IR a xC' into the relational context X
= XXX)
where the self-correlation X XX reveals content-to-conted interaction patterns within the context, and the matrix G maps it to the relational context so that the output has the same size with the basic context X.V
Combining of different types of kernels and contexts. The proposed transform, RSA. integrates the relational kernel and context into a dynamic transform. As illustrated in Figure 2, it consists of two types of kernels, N.Y. and v.R. and two types of contexts, X and R n Note that the basic kernel n computed as QP We combine the kernels and the contexts in the RSA transform:
yn (x)) + (X + XR)
(8)
K.) XV + R X + Ko R n
K R
5

---

Table Complexity of RSA. N,N, M, D, denotes batch size, input size, context size, channel dimension, latent channel dimension, and number of queries, respectively, We simplify the complexity terms using M >> L, D < C/L. for ease description.
operation
time complexity
space complexity
RSA
C(
O(BNM(M +C) M²C O + MC9) O(BNC / BNC /CP/
+ Efficient RSA
O(BNMC3) O(BN (LA)
+ Multi-query (L.)
which contains four different dynamic transforms, KN X Y K XY K XR and K X The first sub-transform, An X corresponds to the involution (Eq. 3). The second and third sub-transform, ks X and KN X both capture content-to ceckenttto-relattee ro-relation interactions, but their effects are different; while R x aggregates the basic context with considering the content tovent-to-comn to-content interactions, K. X aggregates the relational patterns with the content-to-pon popossiionn interactions. The last sub-transfome R X is fully dependent on the relational information. The relational kernel dynamically aggregates relational patterns, generating the deeper relational interactions. These tub-transforms altogether encode rich relational interactions, leading to comprehensive video representation learning.
n
4.2 Improving efficiency of RSA
Efficient relational kernel. The projection matrix H MMM of the relational kernel increases the cost of computation and memory quadratically with the context size M, causing a computational bottleneck. We reduce the complexity the linear one by decomposing H Hi H such that H, E RCMCD and H2 E RMXDD where D is a latent channel dimension, which is smaller than M. Furthermore, we dramatically reduce the memory footprint by switching the computation orders. When r r(H))= E RM MxCxD is the reshaped H1 e IR MCXD the kernel equation is re-formulated as
K, R = veo(1 O H
= 1 @ r H where (x. @ ((HH))od = E(x ( (10)
m
Note that x @ r(Hi) can be efficiently implemented by a channel-win convolution. Our exper- iments show that this modification achieves a good computation-complextty trade-off (Table 4b). Please refer to the pseudo code Fig. 1 in our Supp. for more details.
Efficient RSA. The vanilla RSA may require high cost of computation and memory when generating contexts and aggregating the kernels with the contexts. To reduce them, we decompose P to H, P such that H2 E R xxD P1 E R PxC, where H₂ is the same one obtained by decomposing H. The RSA equation is then re-formulated as
Yn (K) + + X
(H))) +XXX))
(11)
HX)
= =P + x r H H ) (X)) where P= H₂P..
(12)
Note that X @ r(H)), H and (XX) G can be efficiently implemented with convolutions. As shown in Table 1, the space and time complexities are respectively reduced to O(BN MC²) and O + M 22), which are both linear to M. Note that the space complexity is proportional to N and M separately, so that it facilitates efficient training with sufficiently a large volume of context size M, e.g., 5 x 7 7. Please refer to the pseudo code Fig. 2 in our Supp. for more details.
Multi-query RSA. We adopt the multi-query setting [2] for RSA, which sets number of multiple queries and applies the same key and value context to each query, where the size of channel dimension of each query C becomes C/T. While the multi-head setting [52] maintains the time complexity and increases the space complexity, multi-query setting significantly reduces both of them (Table l).

---

Table 2: Performance comparison with other spatio-temporal feature transform methods on SS-v1. W (.).) indicates a sequential transform of W, followed by W,. denotes softmax.
feature transform
kemel
context
FLOPs
params.
top-1
top-5
2D convolution
W attandaddd
X,
32.5G
24.3 M
19.7
46.6
3D convolution [48]
WID.sum standard
X,
57.0
39.3 M
43.3
72.2
(2+1)D convolution [50]
W WiDatanand O W 2Dstandadd
X,
37.3
26.8 M
44.1
72.9
Self-attention [37]
(a ( [ + (PT))
x
32.2
23.4 M
41.6
70.9
Self-attention variant [37]
(x T)
X
32.1 G
23.4 M
25.9
56.3
Self-attention variant [37]
( PP))
X
32.1
23,4 M
41.3
70.6
Self-attention variant [37]
m pT
X
32.1 G
23.4 M
44.3
73.7
RSA (ours)
k + r
X
32.6
23.6 M
45.7
74.8
RSA (ours)
K2 + Kin
X + X
33.2 G
23.6 M
47.0
75.7
Experiments
5.1
Implementation details
Architecture details. use TSN-ResNer50 [54] as our backbone and replace the standard spatial
convolution layers by spatio-temporal RSA layers for every two ResNet bottlenecks [17]. Unless specified otherwise, we replace RSA layers, there are RSA layers total where L = 8, = CO M=5x7x7, x7. We set the input and output channel dimensions of RSA layers to be equal to those of spatial convolution layers TSN-ResNet50.
Training & testing details. For initialization, we randomly initialize the weights of bottlenecks including RSA layers with the MSRA method [16] and use ImageNet pre-trained weights for all the other layers. We set the gamma parameter of the last batch normalization layer to zero. For training, we sample 8 or frames from each video using the segment-ban testing, we sample one or two clips consisting of 8 or 16 frames using the segment-baseds sampling, and average softmax scores for final prediction. Refer to Sec. in our Supp. for more details.
sampling strategy [54]. For
5.2 Datasets
Something-something v1 & v2 (SS-V1 & V2) [14] are both large-scale action recognition bench- marks, including 108k and 220k action clips, respectively. Both datasets share the same motion- centric action classes, e,g. 'pushing something from left to right,' so thus capturing fine-grained motion crucial to achieve the better performance.
Diving-48 [30] a fine-grained action benchmark that is heavily dependent on temporal modeling [3], containing 18k videos with 48 diving classes. Due to the incorrect label issue, we only compare our result with the results on the modified version of Diving-48.
FineGym [39] is motion-centric benchmark that includes gymnastics action classes. We report results on two subsets of Gym288 and Gym99 that contain 288 and 99 action classes, respectively.
5.3 Comparison with other transform methods.
In this experiment, we evaluate the temporal modeling capability of different transform methods: spatio-temporal convolutions [48, 49, 50], self-attention [37] with its variants, and RSA. We replace a single 3 x 3 spatial convolution layer in ReN-Re [54] with single spatio-temporal transform layer. We analyze the ability of modeling temporal dependency of each transform layer with an apple-to-apple comparison. We use frames the input, and the kernel size M of all spatio-temporal transforms is set to 5x7x7.
Table 2 summarizes the results. 2D convolution baseline without modeling temporal dependency shows the lowest accuracy 19.7%. While 3D [48] and (2+1)D [50] convolutions, which use static spatio-temporal transforms, show the top-1 accuracy of 43.3% and 44.1 1, respectively, self-attention only achieves 41.6%. To find out the reasons behind the bad result, we ablate each component of self- attention, e.g., content-to-conten interaction (x Ω (X TT, content-to-postnenn -ossiiioon interaction (z (T)

---

Table 3: Performance comparison on SS-v1&v2, FineGym, and Diving-48.
(a) SS.VI&VZ. IN and IN2IK and K400 denote ImageNet- Ik, ImageNet- 21K. and Kinetics-400 dataset, respectively. Our method achieves a new Late-of-the-we accuracy on both datasets.
(b) Diving-48. Top-1 accuracy,
FLOPs are shown. Results in the upper compartment are from [3].
model
prc- trained
#frame
FLOPs xelips
top-1
SS-VI
top-5
top-1
SS-V2
top-5
model
FLOPss xclliss
top-1
13D [56)
ISM-R50 [31]
IN
IN
16
65 Gx1
47.2
72.2
77.1
63.4
88.5
Slow 8101
TimeSformer
TimeSformer-HR [9]
1703
213 Gx3
Gx3
75.0
77.6
78.0
ir-C5N-152 [49]
97 Gx 10
49.3
TimeSformer-1 [3]
2380
81.0
Slow 88-888 [11]
K400
32
67 Gx3
61.7
86.9
CT:Ner-R50 [28]
IN
16
75 G
52.5
80.9
64.5
89.3
2550
22GN2
84.2
STM-R50 [21]
IN
16
67 Gx:30
50.7
80.4
64.2
89.8
(c) FineGym. The averaged per- class accuracy (%) is reported. All results in the upper compart- ment are from [39].
CorrNet-R [53]
32
187 1x
50.9
TEA|29]
IN
81,9
89.9
MSNNI-TSM-R50 (23)
IN
16
67 Gx1
$2.1
82.3
64.7
89.4
NL-I3 [55] from [56]
IN
76.0
TimeSformer-HR [3]
IN
1703 Gx
62.2
TineSformee-L [3]
IN
96
2380 GX3
62.4
model
Gym288
Gym19
[1]
& K400
IN21K
32
N/Ax4
65.4
89.8
TRN [61]
33.1
68.7
13D [S]
27.9
63.2
RSANet-R50 (ours)
IN
Gx1
51.9
79.6
64.8
89.1
TSM [31]
34.8
70.6
RSANet-R50 (ours)
IN
16
72 Gx
54.0
81.1
66.0
89.8
TSM mee- [31]
46.5
81.2
RSANcI-R50g (ours)
IN
82.6
RSANet-R50gs (ours)
IN
8+16
108 Gx2
56.1
82.8
67.7
91.1
RSANct-R50
50.9
86.4
and softmax (o)) one by one. We observe that the performance of self-attention without x, pT significantly decreases, while that of self-attention without ) is comparable to the original one. It indicates that the temporal modeling capability of the self-attention is actually dependent on the content-to- content-to-position interaction rather than the content-to-comtet so thus it is hard learn position-speciffe features, e.g.
interaction; the self-attention
mechanism itself permutation-invarian motion [32] without the positional embedding. After we remove the softmax non non-linearity, where the kernel equivalent to the basic kernel (xx x Q PT), outperforms both the self-attention and the standard static transforms. It demonstrates the softmax function restricts the expressive ability of the kernel [27]; the softmax forces the kernel weights to be positive, so that the kernel may not compute gradients across frames, which are effective in learning motion. Please refer to the Fig. 3 and Fig. 3 in our Supp. for the qualitative results.
While the composability of the basic kemel depends on the query content, our relational kernel depends on the local pattern of query-key correlation. As we add the relational kernel K, k.y we improve the top-1 accuracy by 1.4%p. The result demonstrates that leveraging local query-key interactions are effective in aggregating informative context for learning motion. At last, by adding relational context X R to basic context X RSA achieves the best top-1 accuracy of 47.0%.
5.4 Comparison to the state-of-the-art methods
Table 3a compares our method to the state-of-the-ee methods on SS-VI and V2 datasets. The first com- partment shows the results of the 3D CNNs. The second compartment contains the motion-modeling methods: both STM [21] and TEA [29] compute frame-wise differences, and both CorrNet [53] and MSNet [23] compute inter-frame pixel-wise correlations to learn motion features. The third compartment reports the results of global self-attention- models. NI.I3D [55] inserts non-local blocks to the 3D CNN for capturing long-range spatio-temporal dependencies. TimeSformer-L. TimeSformer-HH HR [3] and VIVIT [1] are transformer architectures that learn video representations via factorized apatio-temporss self-attention. Our method, RSANet-R50, achieves 51.9% and 64.8% at top-1 accuracy SS-VI and V2 datasets, respectively, which are already competitive to most of existing methods while using 8 frames only. When we use 16 frames, our method outperforms other existing models by achieving 54.0% and 66.0% at top-1 accuracy on SS-VI and V2, respectively. Fi- nally, our ensemble model with 2 clips achieves 56. 1% and 67.7%, which set the new state-of-the-an on SS-V1 and V2 with reasonable computational cost.
Table 3c and Table 3b present the results on Diving-48 [30] and FineGym [39]. For Diving-48, our model achieves 84.2%, substantially outperforming the state-of-the -fete--rt 3D CNN and transformer
8

---

Table Ablation studies on SS-v1 dataset. All models use TSN- TSN-ResNet50 [54] the backbone. Top-1, top-5 accuracy (%), FLOPs (G) and paramaters (M) are shown.
(a) Combinations of different kernels and context.
(b) Latent dimension Decomposing H signifi-
single RSA layer is inserted into stage4.
cantly reduces the computation cost. OOM is an abhre-
biation of out-of-memory 8 video clips per single
kernel
context
FLOPs
parums.
top-1
top-5
GPU machine are used.
32.3
23.4
44.8
73.8
32.7G
23.6
45.4
73.9
D
FLOPs
perams.
memory
top-1
top-5
X
32.7G
M
45.7
74.8
62.9 G
32.0 M
0OM
0OM
0OM
X
32.7G
23.4
46.2
75.4
8
32.2
20.5 M
8.8 GB
50.1
78.8
X
G
23.6
46.5
75.6
34.7G
9.2
51.3
78.8
X
33.2
23.6 M
46.7
75.6
32
39.6
21.7 M
10.2 GB
50,9
79.0
X + R
32.7G
23.4 M
46.5
75.6
C9/2
32.9
21.1 M
8.8 GB
51.1
79.1
+
33.2G
23.6 M
46.8
75.6
C
35.9
22.0 M
9.6 GB
51.5
79.2
Rg +
R
X
+
33.2G
23.6 M
47.0
75.7
(c) Kernel size M. In most cases, larger kernel results the higher accuracy.
(d) Group Hadamard product = C performs the highest accuracy. Note that FLOPs are consistent with varying G due to the switched computation order.
kemel size M
FLOPs
parums.
lop-1
top-5
3
28.5 G
20.3 M
49.4
77.6
Groups G
FLOP
params.
top-1
top 5
X
30.2 G
20.7 M
50.5
78.7
35.9 G
20.2
50.4
78.9
32.6
21.2
50.7
G
20.2
50.9
78.9
3
22.0 M
51.1
79.1
35.9
20.3 M
51.2
78.9
3
22.0 M
51.5
79.2
35.9
20.5
51.2
9
G
2
51.2
78.9
09
35.9
22.0
51.5
79.2
architectures. For FineGym, our model outperforms other methods in the averaged per-class accuracy of 50.9% and 86.4% given 288 and 99 classes, respectively.
5.5 Ablation studies
We conduct ablation experiments to validate the effectiveness of RSA. We use 8 frames for all experiments. Other training and testing details are in Sec. 5.1.
Combinations of different kernels and contexts. In Table 4a, we compare the performance of single RSA layer with different combinations of dynamic kernels and contexts. We first vary different types of the kernels from xY, K to x + R while the context is fixed as X Compared x X Kig X improves the accuracy by 0.6%. It indicates that predicting kernel from the composition of local query-key correlation is effective in modeling temporal dependencies. As we obtain additional improvements
both of the basic and relational transforms, (x.) = by 0.3%p and 0.9%p at top-1 I and top-5 accuracy, respectively. We also vary different types of the context such X. XR. and X + +X while the kernel is fixed. We observe the consistent improvements across different types of contexts, which means that aggregating the relational context beneficial for learning relational information. Finally, as we combine all dynamic transforms, (R=) + RR (XY RR), we achieve the highest accuracy of 47.0%.
Latent dimension D. In Table 4b, validate the effectiveness of decomposing H with latent channel dimension D. Without decomposing H, the TSN-ResNet50 with 7 RSA layers requires 62.9 GFLOPs and 32.0 M parameters, resulting in out-of-memory error. After we decompose H with small D. we significantly reduce FL.OPs, the number of parameters, and the memory footprint. We set D C the default that performs the highest accuracy.
Kernel sizes. In Table 4c, we compare the effect of the kernel size M. In most cases, larger spatio- temporal kernel improves the accuracy, except the case of M 5 9 x 9. Considering the static convolution counterparts, the RSA effectively enlarges the spatio-tempon smaller FLOPs and parameters. We choose M 5 x 7 as the default kernel size that shows the best computation -ccuuraccccccc trade-off.
kernel size, requiring
Correlation computation. In Table 4d, we validate the effect of the group-wise correlation 58), which splits the query and the key embeddings into groups and computes product between each group. While the dot product correlation contracts all channel dimensions into scalar correlation

---

Input Video
self-Artention kernel
Basic Kernel (AC))
040
kernel (Kf))
(a) "Moving something down.' (origin)
(b) 'Moving something up.' (reversed order)
Figure 3: Kernel visualization results on SS-V1. From the top to the bottom in each subfigure, we visualize the input RGB frames, the self-attention kernels, the basic kernels, and the relational kernels. The query position and the context are marked as red and yellow in RGB frames, respectively. The size of spatio-temporal kernel M is set as m, X my X my = 5 7 (L 8) are shown for each transform.
and kernels out of kernels
score the dot-product self-attention [52]. the group-wise correlation outputs G-dimensional correlation vector, providing richer semantic information. As a result, the Hadamard product, i.e., G' = Cº, that computes full element-wise correlations achieves the best performance. We thus set the Hadamard product as the default correlation function. Note that the computational complexity for generating the relational kernel remains the same while the size of G varies since switch the orders of computation as Eq. 10.
5.6 Kernel Visualization
In Figure 3, we visualize the dynamic kernels of self-attention and RSA from the learned models in 4th and 7th rows in Table 3, respectively. In visualization, we observe that both the basic kernels and relational kernels resemble edge detectors, e.g., Sobel filters Laplacian filters [46], along temporal axis that compute discrete approximation of spatio-temporal gradients across frames. When reverse the temporal order of an input video clip, the relational kernel dynamically varies according to whether the object moves up or down but the basic kernel remains the same. Considering that motion information is related to the relative changes along the temporal axis, the results indicate that the RSA kernels effectively capture motion patterns within the context, whereas the self-attention kernels limited to aggregating the local context based the query-key similarities.
Conclusion
We have presented the RSA feature transform, which captures rich relational patterns for video understanding, and validated that it outperforms other dynamic feature transforms in learning motion dynamics in videos. The proposed RSANet outperforms the state-of-the methods standard motion-centric action benchmarks. While we have focused on vidco representation learning this work, we believe the RSA will also benefit image understanding and natural language processing. We leave this for future work.
Acknowledgments and Disclosure of Funding
This work was supported by the NRF grants ( 2021R1A2C3012729 and the IITP grant (No.2019-0-01906, AI Graduate School Program POSTECH) funded by Ministry Science and ICT, Korea. This work was done while Manjin working as intern Microsoft Research Asia.
NRF-
10

---

References
[1] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid. Vivit: video vision transformer. arXiv preprint arXiv:2103. /5691, 2021.
[2] I. Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08802, 2021.
[3] G. Bertasius, H. Wang, and L. Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095, 2021.
[4] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In Proc. European Conference on Computer Vision (ECCV), 2020.
[5] Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[6] Y. Chen, X. Dai, M. Liu, D. Chen, L. Yuan, and Z. Liu. Dynamic convolution: Attention convolution kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1130-11033, 2020.
[7] A. Dosovitskiy, L.. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19999, 2020.
[8] L.. Fan, S. Buch, G. Wang, R. Cao, Y. Zhu, J. C. Niebles, and I.. Fei-Fei. Rubiksnet: Learnable 3d-shift for efficient video action recognition. In Proc. European Conference Computer Vision (ECCV), 2020.
[9] I.. Fan, W. Huang, C. Gan, S. Ermon, B. Gong, and J. Huang, End-to-end learning of motion representation for video understanding. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[10] C. Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[11] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
[12] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[13] K. Fukushima and S. Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets. 1982.
[14] R. Goyal, S. E, Kahou, V. Michalski, Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense, In Proc. IEEE International Conference on Computer Vision (ICCV), 2017.
[15] X. Guo, K. Yang, W. Yang, X. Wang, and H. Li. Group-wise correlation stereo network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019.
[16] K. Hc, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[18] H. Hu, Z. Zhang, Z. Xie, and S. Lin. Local relation networks for image recognition. In Proceedings of the IEFE International Conference on Computer Vision, pages 3464-3473, 2019.
11

---

[19] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint urXvv1405.33866 2014.
[20] X. Jia, B. De Brabandere, T. Tuytelaars, and L. Van Gool. Dynamic filter networks. In NIPS, 2016.
[21] B. Jiang, M. Wang, W. Gan, W. Wu, and J. Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
[22] J.-H. Kim, K.-W. On, W. Lim, J. Kim, J.-W. Ha, and B.-T. Zhang. Hadamard product for low-rank bilinear pooling, arXiv preprint I 2016.
[23] H. Kwon, M. Kim, S. Kwak, and M. Cho. Motionsqueeze: Neural motion feature learning for video understanding. arXiv preprint vrXiv:2007.099333 2020.
[24] H. Kwon, M. Kim, S. Kwak, and M. Cho. Learning self-similarity in space and time as generalized motion for action recognition. arXiv preprint arXiv:2102.07092, 2021.
[25] Y. LeCun, L. Bottou, Y. Bengio, and Haffner. Gradient-base learning applied document recognition. Proceedings of the IEEE, 1998.
[26] M. Lee, S. Lee, S. Son, G. Park, and N. Kwak. Motion feature network: Fixed motion filter for action recognition. In Proc. European Conference on Computer Vision (ECCV), 2018.
[27] D. J. Hu, C. Wang, X. Li, Q. She, L. Zhu, T. Zhang, and Q Chen. Involution: Inverting the inherence of convolution for visual recognition. arXiv preprint arXiv:2103.06255, 2021.
[28] K. Li, X. Li, Y. Wang. J. Wang, and Y. Qiao. (net: Channel tensorization network for video classification. In International Conference on Learning Representations, 2021.
(29] Y. Li, B. Ji, X. Shi, J. Zhang, B. Kang, and L. Wang. Tea: Temporal excitation and aggregation for action recognition. In Proc. IEEÉ Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[30] Y. Li, Y. Li, and N. Vasconcelos. Resound: Towards action recognition without representation bias. In Proc. European Conference on Computer Vision (ECCV), 2018.
[31] Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
[32] X. Liu, J.-Y. Lee, and H. Jin. Learning video representations from correspondence proposals. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[33] N. Ma, X. Zhang. Huang. and J. Sun. Weightnet: Revisiting the design space of weight
networks. arXiv preprint 1rXiv:2007.11823; 2020.
[34] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufflenet v2: Practical guidelines for efficient cnn
architecture design. In Proc. European Conference on Computer Vision (ECCV), 2018.
[35] D. Neimark, O. Bar, M. Zohar, and D. Asselmann. Video transformer network. arXiv preprint arXiv:2102.00719, 2021.
[36] A. Piergiovanni and M. S. Ryoo. Representation flow for action recognition. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[37] P. Ramachandran, N. Parmar, A. Vaswani, I, Bello, A. Levskaya, and J. Shlens. Stand-alone self-attention in vision models. In Proc. Neural Information Processing Systems (NeurIPS), 2019.
[38] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[39] D. Shao, Y. Zhao, B. Dai, and D. Lin. Finegym: A hierarchical video dataset for fine-grained action understanding. Proc. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
12

---

[40] E. Shechtman and M. Irani. Matching local self seff-similarities across images and videos. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
[41] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Proc. Neural Information Processing Systems (NeurIPS). 2014.
[42] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[43] R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for semantic segmen- tation, arXiv preprint arXiv:2 1ete220050006333 2021.
[44] S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed. D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[46] R. Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
[47] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proc. International Conference on Machine Learning (ICML). 2019.
[48] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proc. IEEE International Conference Computer Vision (ICCV), 2015.
[49] D. Tran, H. Wang, L. Torresani, and M. Feiszli. Video classification with channel-separated convolutional networks. In Proc. IEEE International Conference Computer Vision (ICCV), 2019.
[50] D. Tran, H. Wang. L. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[51] A. Vaswani, P. Ramachandran, A. Srinivas, N. Parmar, B. Hechtman, and J. Shlens. Scaling local self-attention for parameter efficient visual backbones. arXiv preprint arXiv:2103.12331, 2021.
[52] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. In Proc. Neural Information Processing Systems (NeurIPS), 2017.
[53] H. Wang, D. Tran, L. Torresani, and M. Feiszli. Video modeling with correlation networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[54] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Proc. European Conference on Computer Vision (ECCV), 2016.
Conference Computer Vision and Pattern Recognition (CVPR), 2018. [55] X. Wang, R. Girshick, Gupta, and K. He. Non-local neural networks. Proc. IEEE
[56] X. Wang and A. Gupta. Videos space-time region graphs. In Proc. European Conference Computer Vision (ECCV), pages 399-417, 2018.
convoll [57] B. Yang, G. Bender, Q V. Le, and J. Ngiam. Condcony: Conditionally parameterized convolu- tions for efficient inference. arXiv preprint arXiv: 04971,, 2019.
[58] G. Yang and D. Ramanan. Volumetric correspondence networks for optical flow. Proc. Neural Information Processing Systems (NeurIPS), 2019.
13

---

[59] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens-to-ton vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2001. 1986, 2021.
[60] H. Zhao, J. Jia, and V. Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10076-10085, 2020.
[61] B. Zhou, A. Andonian, A. Oliva, and A. Torralba. Temporal relational reasoning in videos. In Proc. European Conference on Computer Vision (ECCV), 2018.
14


Reviews:
[{"review_id": "qqlPFyMdHoz", "reviewer": "Reviewer_hB4H", "summary": "The authors tackle the problem of video classification. Motivated by the limitations of the recent video transformer methods, the authors propose a new relational feature transform called intervolution. The main idea behind this new module is that it should capture spatiotemporal relationships in video more effectively than the standard self-attention operator. The authors report strong performance on \"temporally heavy\" datasets such as Something-Something-V2, Diving-48, and FineGym while also having lower computational cost and using less labeled data in comparison to prior work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Current self-attention based video models such as TimeSformer [20] struggle to learn effective spatiotemporal features in settings that require strong temporal reasoning. Therefore, the proposed approach is timely, and well motivated. \n- The proposed approach achieves strong performance on a variety of temporally heavy datasets.\n- The proposed approach is efficient and it also doesn't require large amount of external datasets for pre-training.\n- The ablation studies are very thorough, providing evidence that every component is beneficial to the overall performance of the system.\n\n\nWeaknesses:\n- In my view, the readers who are not as familiar with recent self-attention based methods in the video domain, might find the motivation of this work not fully clear. It would be good if the authors could provide more intuitive examples for motivating their technical approach. For examples, the section in L149-153, which explains some of the limitations of standard self-attention in video domain is overly technical, and lacks simple and intuitive examples to understand why the proposed approach is needed.\n- Furthermore, the subsection that explains the concept of the relational kernel can be quite difficult to understand at times. I think the authors could do a better job motivating the need for some of the equations/components, and also by simplifying the description a bit. For example, Equation 5 could be expressed using a simple matrix multiplication, but instead it contains additional terms/operations that are unnecessary in my view. In most recent papers, the query matrix is assumed to be the same as the context matrix. It seems that in practice that's the case in this paper as well. Therefore, I'm not sure if there's a need to introduce additional terms/operations and make the paper seem more complicated than it is. For instance, I think the paragraph in L186-L193 is written clearly, concisely, and with a good intuitive motivation in mind. Therefore, the rest of the draft should follow the same presentation style. In particular, I would also suggest to present the approach more in a top-down fashion (starting from Equation 7), instead of just listing all the components one after the other.\n- Section 5.3 doesn't list the dataset that was used, which would be useful.\n- Typo in L296.\n\nQuestions:\n- Do all baselines in Table 2 use the same backbone, pre-training dataset, etc? Or is it completely different implementations of different methods?\n- Have you tried running your method on Kinetics? I understand that Kinetics is a spatially biased dataset, but just curious if there's any improvement with respect to the baseline. \n- I was confused about the experiment in Table 4c). Which spatiotemporal kernel is M referring to? Is this in the bottleneck of the ResNet backbone? Is spatiotemporal kernel M related to the context size M? If not did you ablate the context size M? It's a bit confusing that both Kernel Size and Context Size is referred to as M. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe so.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "TtJDbVeEWE6", "reviewer": "Reviewer_L4t3", "summary": "The paper introduces intervolution, a new feature transform for neural networks that combines both content-to-content and content-to-relation interaction. According to the authors, this feature transform captures both spatio-temporal dynamics and visual information. The authors recast convolution, self attention and dynamic convolution into a unified framework and compare their method to these techniques under the proposed formulation. The paper indicates that dynamic transforms are limited for video representation because of the absence of content-to-content interaction, and they claim that their work is a step towards solving this problem. It features an in-depth analysis of intervolution, its empirical effects on 4 datasets, and an ablation analysis that sheds light on the impact of each component.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## General thoughts\nThe paper is generally sound and proposes an interesting, although slightly convoluted feature transform. The general formulation unifying descriptions of convolution, self-attention, involution and lambda functions is elegant and easy to understand. The proposed transform does seem to introduce positive performance differentials, and the choice of datasets to address the proposed claim is adequate.\n\n## Pros\n- Interesting re-interpretation of dynamic feature transforms\n- Strong performance on challenging datasets\n- Detailed experiments: 4 datasets, comparison to multiple alternative networks, specific comparison to other spatio-temporal transforms\n- In-depth ablation studies\n- Good work done on improving the efficiency of the technique\n- Strong additional information in supplemental, and code was submitted.\n\n## Cons\n- The re-interpretation of dynamic feature transforms takes a few liberties: \n\n1. The involution framework has a non-linearlity and batchnorm in its kernel generation process: it's calculatin is  $W_1\\sigma(W_0 x_n)$ instead of $x_nP^T$ (eq. 3)\n2. lambda convolution's $P^T$ matrix is also more than just a learnable matrix (l.134): it's actually a sum of a normalized key matrix and a position embedding matrix, where the key matrix is built through a projection of the context: in a nutshell, following the authors notation, $P^T$ should take $X_n^K$ as input. This makes it not as close to involution as the authors might suggest .\n\n- The paper focuses on motion-centric datasets, but indicates that the intervolution framework is general enough to outperform existing alternatives in capturing both visual information and motion dynamics. If that's the case, testing the method on other data where visual information is a bit more prevalent would be interesting; however, there is no indication of how this technique would perform in other typical video datasets like Kinetics, Moments in Time or Jester (Jester would be a good dataset to analyze in this scenario)\n\n- The idea appears a bit incremental and convoluted: the framework is a combination of several existing methods, and the resulting computational graph lacks the simplicity of its counterparts. This yields large computational complexity (although the authors address it quite well) and questions about the validity of each submodule. The ablation studies in table 4c appear to show that just using $\\kappa_n^R$ and $X_n^R$ is enough to get extremely close to top performance: do we need the full setup?\n\n- Section 5.3 is generally weak. l. 269: \"softmax forces the kernel weights to be positive, so that the kernel may not compute gradients across frames\" -> What do you mean by this?\n\n- Table 3 is strong, but disorganized: missing multiple values, comparisons between networks that have different pre-trainings and objectives are bundled together, different networks are used for different datasets\n \n\n## Small corrections\n- In Fig. 2, the $1(X_n^Q)^T$ seems incorrectly placed (should go to the left of the element-wise)\n- Section 5.3 has several typos and mistakes (e.g. l. 268 \"it demonstrates the softmax\" -> \"it demonstrates that the softmax\"\n- In l. 173 and 174, it seems that there is a confusion between Value and Key (the text says value but the equation and figure show $X_n^K$\n\n## Questions\n- Why are the FLOPs almost equal for all alternatives in table 4a? specifically when comparing one kernel and one context vs two kernels and two contexts\n- Any examples / justifications for the claim in paragraph at l. 180?\n- In Table 4d, are you still using the TSN-ResNet50 network for this analysis?\n- Why are you using ImageNet pretraining instead of Kinetics, Moments in Time or other large video-based dataset?\n- L. 227: What is L? not introduced in the text before\n- Any insight on the choice of backbone? Why TSN ResNet50 instead of a CSN or I3D?\n\n\n### In general, good paper. If the cons are reasonably addressed and the small errors fixed, I believe this paper can be an adequate addition to Neurips.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both limitations and societal impact have dedicated sections in the supplemental. In terms of societal impact, the authors outline potential risks in applications such as video surveillance. In terms of limitations, the authors outline the following:\n- Computational complexity is still an issue and can be improved\n- A more generalized dynamic transform could capture both visual appearance and relational features in a more natural way\n- Further work is needed to extend this work to image understanding or NLP\n\nApart from that, I would point to the following additional limitations:\n- The work here focuses on motion-centric datasets, but the proposed framework is introduced as general enough to beat and supersede other dynamic transforms that are successful in more general domains. I understand that this is a conscious choice by the authors, but it might point to a limitation: does this technique work on more general video datasets, such as Kinetics and Moments in Time?\n- The technique lacks the simplicity that characterizes other dynamic transforms such as lambda layers. Is the combination of all 4 terms of intervolution really necessary? As mentioned above, the ablation studies in table 4c appear to show that just using $\\kappa_n^R$ and $X_n^R$ is enough to get extremely close to top performance. ", "review_text": "", "extra_scores": {"ethical_concerns": "No specific ethical concerns, as this is an architecture-focused paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "5UBUgeqpAl", "reviewer": "Reviewer_1j5f", "summary": "This paper proposes a relational feature transform that captures rich relational patterns for action recognition and compared to other dynamic feature transforms in learning motion dynamics in videos. Experimental results outperform state-of-the-art methods on standard motion-centric action recognition benchmarks.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n1) The main idea of the Intervolution feature transform is well-motivated and straightforward.\n2) The conducted experiments have shown the effectiveness of the proposed method compared to state-of-the-art methods.\n3) The paper is well organized.\n\n\n**Weaknesses:**\n\n**Writing.** Although the paper is clearly organized, the reading is not smooth, and the figures are overloaded with equations, which does not make it easy to follow. Additionally, at least to me, the paper introduces a module for video action recognition and not video understanding (which is a broader aspect than recognizing actions). Still, I can live with the author's choice.\n\n**Experiments.** Table 2 is missing a few crucial points. First, Non-local [1] and Long-Term feature bank [2] networks, which are full self-attention approaches, were proposed for capturing motion and long-range dependencies directly by computing interactions, and thus I expect to have a direct comparison with them. Second, by using only 8 frames for comparison, the authors prevent the other models from performing better. \n\nAdditionally, in line 268, the authors mentioned, “It (Table 2) demonstrates the softmax function restricts the expressive ability of the kernel”. The self-attention is not permutation invariant if you use a positional encoding. Did the authors try to use self-attention with positional encoding (as in vision-transformers), and still, the softmax performed worse?.\n\nLast, in line 289, the authors wrote, “Finally, our ensemble model with 2 clips achieves 56.1% and 67.7%, which set the new state-of-the-art on SS-v1 and v2 with reasonable computational cost“. Unfortunately, Table 3 does not include the number parameters, so I do not know if this is true. I believe that #params are essential for a fair comparison (at what costs do we improve?).\n\n\n[1] Non-local Neural Networks\n\n[2] Long-Term Feature Banks for Detailed Video Understanding\n\n\n**Temporal understanding.** I would expect a model that claims that “re-interpreting recent dynamic feature transforms in a unified way” (line 44) to show more evidence of the learned dynamics. Can the authors present the kernel visualizations compared to the other methods or show the current top 5 actions compared to SlowFast? It is crucial to understand which actions the model recognized better compared to the existing methods. I also could not clearly understand Figure 3 in the Supplementary.\n\n\nMy main concern is the missing experiments which are crucial to validate this network compared to existing work (Non-Local and Long-term feature bank are also self-attention modules and should be compared in the same manner). On the other hand, I liked this paper, its contribution, and its motivation. I am open to the authors' feedback and other reviewers' opinions.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper unanimously receives positive rates. The authors’ rebuttal is through and excellent too. As the reviews show satisfaction on the authors’ feedback, the final draft needs to respect it accordingly. 
Decision: Accept (Poster)