Relational Self-Attention: What's Missing in Attention for Video Understanding
Authors: Manjin Kim
Heeseung Kwon
Chunyu Wang
Suha Kwak
Minsu Cho
Venue: NeurIPS 2021 Poster (2021)
Keywords: Action recognition
Video understanding
Motion analysis
Self-attention

Abstract: Convolution has been arguably the most important feature transform for modern neural networks, leading to the advance of deep learning.  Recent emergence of Transformer networks, which replace convolution layers with self-attention blocks,  has revealed the limitation of stationary convolution kernels and opened the door to the era of dynamic feature transforms. The existing dynamic transforms, including self-attention, however, are all limited for video understanding where correspondence relations in space and time, i.e., motion information, are crucial for effective representation. In this work, we introduce a relational feature transform, dubbed the relational self-attention (RSA), that leverages rich structures of spatio-temporal relations in videos by dynamically generating relational kernels and aggregating relational contexts. Our experiments and ablation studies show that the RSA network substantially outperforms convolution and self-attention counterparts, achieving the state of the art on the standard motion-centric benchmarks for video action recognition, such as Something-Something-V1&V2, Diving48, and FineGym. 

Relational Self-Attention:
What's Missing in Attention for Video Understanding
Manjin Kim
Heeseung Kwon
Chunyu Wang? Suha Kwak
Minsu Cho
POSTECH
Microsoft Research Asia
http://cvlab. htt.c//labpppstcctteeoooo posteech....kr/reeeerrre
Abstract
Convolution has been arguably the most important feature transform for modern neural networks, leading to the advance of deep learning. Recent emergence of Transformer networks, which replace convolution layers with self-attention blocks, has revealed the limitation of stationary convolution kernels and opened the door to the era of dynamic feature transforms. The existing dynamic transforms, including self-attention, however, are all limited for video understanding where correspon- dence relations in space and time, i.e., motion information, are crucial for effective representation. In this work, we introduce relational feature transform, dubbed the relational self-attention (RSA), that leverages rich structures of spatio-temporal relations in videos by dynamically generating relational kemnels and aggregating lational contexts. Our experiments and ablation studies show that the RSA network substantially outperforms convolution and self-attention counterparts, achieving the state the art on the standard motion-centric benchmarks for video action recognition, such Something-Something -1 & Diving48, and FineGym.
1 Introduction
Convolution [13, 25] is a feature transform that is ubiquitous in modern neural networks for visual recognition, and has driven the development of deep learning in the past decade. The stationarity of convolution kernels, however, may limit its expressivity and hinder adaptation to diverse com- positional possibilities of visual concepts [18]. Dynamic convolution [20] and self-attention [52] that construct kernels or attentions according to input contents have emerged as an alternative to static convolution this context, being followed by further studies for dynamic feature trans- forms [6, 27, 33, 57]. The effectiveness of self-attention has been demonstrated by the success of Transformer variants on different image understanding tasks such as image classification [7, 51, 59], object detection [4], and semantic segmentation [43]. Recently, it has been further extended for video understanding, replacing spatio-temporal convolution [5, 48, 50] with spatio-temporal self- attention [1, 3, 35].
Despite their recent progress in the video domain, the existing dynamic feature transforms still leave much room for improvement in terms of learning relational patterns in space and time, i.e., motion information, which known to be essential for video understanding [23, 53]. For example, spatio- temporal self-attention [55] often fails to learn motion representation without positional embeddings as demonstrated in [32], and even those with positional embeddings [1. 3, 35] turn out to be not effective motion-centric action recognition benchmarks such
-ooietiinnn [14].
In this work, we introduce relational dynamic feature transform, dubbed relational self-attention (RSA), to address the limitation of existing methods, RSA leverages rich structures of spatio-tempore
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

relations in videos by dynamically generating relational kernels and aggregating relational contexts. Combining these relational components with ordinary dynamic kernels and context features, our RSA learns rich video representation that effectively captures both visual appearance and spatio-temporal motion dynamics.
The main contribution of this paper is three-fold:
We re-interpret recent dynamic feature transforms in unified way, and provide in-depth analysis on their capability of learning video representation.
We introduce a new dynamic feature transform, i.e., RSA, which effectively captures both motion dynamics for video understanding.
visual appearance and
Our RSA network substantially outperforms convolution and self-attention counterparts, achieving the state of the art on SS-VI&V2, Diving48, and FineGymm.
2 Related Work
Convolution and its variants. Convolution [13, 25] has been used as dominant neural primitive in modern neural architectures [17. 34, 38, 42, 45, 47]. Since convolution often suffers from its limited expressivity due to its static kernel, dynamic convolution operators have been studied for enhancing composability by dynamically adjusting the convolution kernel according to input features [6, 20, 27, 33, 57]. One example is involution [27], which generates lightweight dynamic kernel using input content and substantially outperforms convolution with less computational cost on image classification. Our RSA differs from these kernel generation methods in that it leverages relational patterns for learning a rich video representation.
Self-attention and its variants. Self-attention [52] was originally introduced for neural machine translation to capture long-range interactions, and now has been widely adopted in many different domains thanks to its versatility and expandability. Following local attention methods [18, 37] that employ self-attention as an alternative to convolution, ViT models have demonstrated impressive performance on a variety of image understanding tasks [4, 7, 43, 59]. On the other hand, some of previous work have attempted to design novel dynamic transforms through attention [2, 60]. Zhao
et al. [60] expand the spatial attention to include the channel attention. Bello [2] proposes lambda layer that focuses on interaction between visual contents and a relative position embedding without softmax, which outperforms self-attention counterparts on image classification. The proposed RSA is an extension of these techniques, yet focuses on learning rich relational features for video understanding.
Convolution and self-attention for video understanding. Action recognition is the most funda- mental task for video understanding, which aims at classifying a video clip into pre-defined action classes. key to success of this task to capture temporal dynamics across multiple video frames, and apatio-temporal (3D) convolution [48] has been de facto for modeling the temporal dynam- ies [8, 10, 31, 49, 50]. On the other hand, self-attention was used as an attachable module for 3D CNNs in the earliest method [55], yet nowadays becomes the major building block of video under- standing networks such as video VIT [1, 3, 35). However, both 3D CNNs and ViTs are not sufficient for modeling temporal dynamics due to the lack of inter-feature relations within the spatio-temporal context. Our RSA overcomes their limitation by explicitly utilizing relational features that imply temporal dynamics.
Learning motion for video understanding. Early approaches in video understanding rely on exter- nal motion extractors like optical flow for motion feature learning [5, 12, 41], but recent models aim to learn motion representation internally. To this end, they estimate feature-level optical flow inside their networks [9, 36] or compute subtraction between consecutive frame-wise features [21, 26, 29, 44]. Meanwhile, correlation-bant
methods [23, 24, 53] use inter-pixel relations as motion likelihood maps and achieve sate-offthheart on motion-centric action recognition benchmarks. Inspired by the correlation-basse methods, we apply dynamic relational kernels to the correlation tensors (i.e., relational context), leading to effective motion feature learning.

---

additoo X
doi-proddcc
Vec
redorizalion +6
softmma
p:
Yn
Yn
Yn
(C)
a
(M)
(C)
(x (M)
P
PTxy
(MCC)
VBC
(Ar)
(C
X
X
X ()
Xx (C)
X (c) Involution Lambda convolution
X
(MC g (C)
(M x C)
()
(a) Convolution
(b) Self-attention
Figure I: Computational graphs of different feature transforms. See text for details.
3 Background
In this section, we revisit three instances of feature transform and analyze them in a simplified form. In context, the role of feature transform is update feature map while preserving its order in space and/or time. To this end, it processes each position in space and/or time using features of its neighborhood. For example, given an input feature map of video X RTTxHXWxC for each spatio-temporal position (t, h, w), a feature transform takes the individual feature 2r,b as targer and transforms using its neighborhood features (æt M we (ee ,h' d.,,
its context. For the sake of notational convenience, let us denote the target by F. E IRC, where n represents specific position space and/or time, and its context by Xn E RMxx where M is the size of neighborhood, and the corresponding output by yn E Rº_ Now, without loss of generality, we can view the transform as applying each position function with learnable weights W that maps target F, to output Yn using context X, typically implemented as neural layer that processes all the targets in parallel.
f Xx, n. W). Note that such transform is
In the following, we describe recent feature transform functions in computer vision as well as the conventional one, convolution. Figure I illustrates the computational graphs of the transforms.
Convolution. Convolution [13, 25] is a static and translation-equivariann feature transform that updates each target æn by applying static kernels W MCxC on local context X,,
yo W vee X,)).
While extracting different visual patterns using multiple kernels, convolution remains static in the sense that the kernel is not affected by the target En, Le.., feature of interest (Figure la). This station- arity may not be effective in adapting to diverse compositional possibilities of visual concepts [18], and the channel-dependent weights can
inter-channel redundancy in the kernel [19].
Self-attention. Self-attention [52] is a dynamic transform that generates an attention map of context Xn using target H, and then aggregates the context using the attention map dynamic kernel (Figure 1b). The process starts by adapting inputs for query-key-value attention computation; using three learnable embedding matrices, E EK EV E CCC, it embeds the target 2n into the query Q via xº = x, EQ and also projects the context X,, into the key X and the value X via EK and x X,, respectively. The basic attention map is computed via the softmaxed product between query x and key X which can be viewed as content-to-come to-content interaction. To reinforce this content-only attention with positional information of the context, learnable matrix P E IR MC which encodes relative positions of individual features in the context, optionally included content-to-pee to-psittooo interaction. The self-attention transform aggregates the value embeddings X. of context using the attention map as kernel weights shared for all channels:
X
yx (x +x +
where (-) denote the softmax function. This transform cranslationeeuuivriann position em- bedding P is local and relative with respect to the target, and becomes germutation-equivarinn if content-to-position interaction is removed. Unlike convolution, dynamic the sense that the kernel or ( KyT x PT), which is used to aggregate the context, depends on the target content.

---

It allows more flexibility in adapting to input and also consumes fewer parameters by sharing the kernel across the channels [18].
Involution & lambda convolution. Involution [27] is a light- dynamic transform that lever- ages content-to-posiiic interaction only. It dynamically generates a kernel H RRM by projecting target v,, using learnable matrix P E IR IRMC xC via K. P,,, 1 and then use the kernel to aggregate the context X,, (Figure Ic, left):
yn
- Fr P X
where the matrix P plays the role of converting the target to a kernel in position-sensitivy manner. In a similar spirit, lambda convolution of the lambda networks [2] uses content-to-positiee interaction between the target and the context position. As in self-attention, using two learnable embedding matrices, E EV RCxC it starts by projecting the target &n and the context X,, into the query 2, and the value X via x. F, EQ and X = EVE,V respectively. The lambda convolution abstracts the context value X to a contextual matrix XØ using a learnable matrix P RMMxC via de- PTXX) E RCxC, which in turn is used for updating the target query 2, (Figure Ic, right):
ya o 2 2. 1.
Despite the differences in the operational procedures and the concepts, the lambda convolution has effectively the same form with involution except for additional key and value embeddings. Note that unlike self-attention, involution and lambda convolution both have no softmax nonlinearity. In fact, the absence of softmax reduces the computational cost and also increases the expressive ability by allowing negative activations [27]. Both involution and lambda convolution are shown to outperform convolution and self-attention counterparts in image classification [2, 27], demonstrating the effectiveness of dynamic content-aware kernels.
Limitation of existing dynamic transforms.
The aforementioned dynamic transforms are com- monly based on leveraging content-to-content and/or content-to-position interactions in constructing kernels, where the target content (input feature) is used as the source of the dynamic transform. While the methods are effective for learning image representation indeed, they are all limited for learning video representation; as will be shown in the experimental section 5.3, we have found that existing dynamic transforms show only marginal or virtually no improvement over the static transform of convolution on the motion-centric action recognition benchmark. The main reason lies interactions as a relational content in representation
in the missing structure of learning. While content-to-content interactions are considered in existing dynamic transforms, they are immediately broken into individuals without being used as whole. For example, self- self-attention computes query-to-key correlation x (XX)T but uses the individual elements of the correlation only for aggregating informative contents from the context. The content-to-position interaction 29 pl does not help in capturing the structure of interactions either since it has no access to the correlation as a whole. In videos, such structural patterns contain informative spatio-temporal contents, i.e., different levels of generic motion information, thus being crucial in video understanding.
4 Our approach
In this section, we introduce a new dynamic transform, dubbed relational self-attention (RSA), which is designed to learn rich spatio-temporal interaction patterns across input contents. Figure illustrates the computational graph of RSA. On top of the basic kernel and context in a dynamic transform, builds relational kernel and context and processes all the kemnel-context combinations. Here we describe the main components of RSA and their integrated form, and explain an efficient implementation of RSA.
As in self-attention, we start by adapting inputs for query-key-value interactions; using three learnable embedding matrices, EQ, Ek E R CCC target E, and context X,, are embedded into query Q key X and value X via 2 = x; E X = X,, and X = X, respectively. xx X and X V are then L2-normalized, and we omit the normalization term for the simplicity.
"This transform corresponds to the position lambda where the extent of the context x V local neighborhood of the target 2 [2].
is restricted to the

---

n Basic kemel
Relatinal kernel
Relstional context
K
(M)
1xx,)
(MC)
VBc
(M)
M c)
(X G
(M x ()
Basic context
(C A
dol-product
hadamardd product
P (M X C)
X (M C)
II (MC M)
X (M C)
column one veclor
(M X ()
vectorization
Figure 2: Computational graph of RSA. RSA consists of two types of kernels (basic and relational kernel) and two contexts (basic and relational context). See text for details.
4.1 Relational self-attention (RSA)
Relational kernel. The relational kernel is designed predict the relevance of context based on the structure of content-to-content interactions. To generate the relational kernel, we compute the dot- product correlation of the query and the key, i.e., x (X E AR, and then project the correlation vector using learnable matrix H E R.MMM
12
R K
The role of H in this relational kernel corresponds to that of P in the involution kernel (Eq, 3); while P predicts the kernel weights from the C-dinensional query vector, H predicts them from the M-dimensional query-key correlation vector. The resultant dynamic kernel aggregates the context depending how the individual contents are related to each other in space and time, thus being particularly effective in learning notion-rrllttd patterns in videos. Note that, when we set to an identity matrix I and add the softmax operator into Eq. 5, the relational kernel k is equivalent the self-attention kernel without content-to-positien interaction, i.e., o (x (x))))
Since the dot-product correlation contracts all channel dimensions of the query and the keys, may lose semantic information, which may help in generating an effective relational kernel. We thus take the Hadamard product [22] instead so that we can leverage channel-wiss query-ker correlations for producing the relational kernel. Using learnable kernel projection matrix H E I MfC M Eq. 5 can be reformulated as
(6) where @ denotes the Hadamard product. H E MCXM predicts the kernel weights from an MC- dimensional channel-wise query-key correlation vector. Furthermore, the absence of the softmax operator allows the relational kernel to have negative activations so that it better learns relative features where the subtractive interactions are beneficial for capturing relative changes of contents over time [21, 26, 29].
vec(1 O X
Relational context. The relational context is designed to provide the relational pattern of content- to-content (i.e.,
interactions for the kernels to aggregate. To this end, we use self-correlation [40], which effectively describes spatio-temporal intra-structure including motion information [23, 24, 53]. We first construct the self-correlation matrix X (XV) XVTT E I x and then project it using learnable matrix G E IR a xC' into the relational context X
= XXX)
where the self-correlation X XX reveals content-to-conted interaction patterns within the context, and the matrix G maps it to the relational context so that the output has the same size with the basic context X.V
Combining of different types of kernels and contexts. The proposed transform, RSA. integrates the relational kernel and context into a dynamic transform. As illustrated in Figure 2, it consists of two types of kernels, N.Y. and v.R. and two types of contexts, X and R n Note that the basic kernel n computed as QP We combine the kernels and the contexts in the RSA transform:
yn (x)) + XX + XR)
(8)
K.) XV + R X + Ko R n
K R
5

---

Table Complexity of RSA. N,N, M, D, denotes batch size, input size, context size, channel dimension, latent channel dimension, and number of queries, respectively, We simplify the complexity terms using M >> L, D < C/L. for ease description.
operation
time complexity
space complexity
RSA
C(
O(BNM(M +C) M²C O + MC9) O(BNC / BNC /CP/
+ Efficient RSA
O(BNMC3) O(BN (LA)
+ Multi-query (L.)
which contains four different dynamic transforms, KN X Y K XY K XR and K X The first sub-transform, An X corresponds to the involution (Eq. 3). The second and third sub-transform, ks X and KN X both capture content-to ceckenttto-relattee ro-relation interactions, but their effects are different; while R x aggregates the basic context with considering the content tovent-to-comn to-content interactions, K. X aggregates the relational patterns with the content-to-pon popossiionn interactions. The last sub-transfome R X is fully dependent on the relational information. The relational kernel dynamically aggregates relational patterns, generating the deeper relational interactions. These tub-transforms altogether encode rich relational interactions, leading to comprehensive video representation learning.
n
4.2 Improving efficiency of RSA
Efficient relational kernel. The projection matrix H MMM of the relational kernel increases the cost of computation and memory quadratically with the context size M, causing a computational bottleneck. We reduce the complexity the linear one by decomposing H Hi H such that H, E RCMCD and H2 E RMXDD where D is a latent channel dimension, which is smaller than M. Furthermore, we dramatically reduce the memory footprint by switching the computation orders. When r r(H))= E RM MxCxD is the reshaped H1 e IR MCXD the kernel equation is re-formulated as
K, R = veo(1 O H
= 1 @ r H where (x. @ ((HH))od = E(x ( (10)
m
Note that x @ r(Hi) can be efficiently implemented by a channel-win convolution. Our exper- iments show that this modification achieves a good computation-complextty trade-off (Table 4b). Please refer to the pseudo code Fig. 1 in our Supp. for more details.
Efficient RSA. The vanilla RSA may require high cost of computation and memory when generating contexts and aggregating the kernels with the contexts. To reduce them, we decompose P to H, P such that H2 E R xxD P1 E R PxC, where H₂ is the same one obtained by decomposing H. The RSA equation is then re-formulated as
Yn (K) + + X
(H))) +XXX))
(11)
HX)
= =P + x r H H ) (X)) where P= H₂P..
(12)
Note that X @ r(H)), H and (XX) G can be efficiently implemented with convolutions. As shown in Table 1, the space and time complexities are respectively reduced to O(BN MC²) and O + M 22), which are both linear to M. Note that the space complexity is proportional to N and M separately, so that it facilitates efficient training with sufficiently a large volume of context size M, e.g., 5 x 7 7. Please refer to the pseudo code Fig. 2 in our Supp. for more details.
Multi-query RSA. We adopt the multi-query setting [2] for RSA, which sets number of multiple queries and applies the same key and value context to each query, where the size of channel dimension of each query C becomes C/T. While the multi-head setting [52] maintains the time complexity and increases the space complexity, multi-query setting significantly reduces both of them (Table l).

---

Table 2: Performance comparison with other spatio-temporal feature transform methods on SS-v1. W (.).) indicates a sequential transform of W, followed by W,. denotes softmax.
feature transform
kemel
context
FLOPs
params.
top-1
top-5
2D convolution
W attandaddd
X,
32.5G
24.3 M
19.7
46.6
3D convolution [48]
WID.sum standard
X,
57.0
39.3 M
43.3
72.2
(2+1)D convolution [50]
W WiDatanand O W 2Dstandadd
X,
37.3
26.8 M
44.1
72.9
Self-attention [37]
(a ( [ + )PT))
x
32.2
23.4 M
41.6
70.9
Self-attention variant [37]
(x T)
X
32.1 G
23.4 M
25.9
56.3
Self-attention variant [37]
( PP))
X
32.1
23,4 M
41.3
70.6
Self-attention variant [37]
m pT
X
32.1 G
23.4 M
44.3
73.7
RSA (ours)
k + r
X
32.6
23.6 M
45.7
74.8
RSA (ours)
K2 + Kin
X + X
33.2 G
23.6 M
47.0
75.7
Experiments
5.1
Implementation details
Architecture details. use TSN-ResNer50 [54] as our backbone and replace the standard spatial
convolution layers by spatio-temporal RSA layers for every two ResNet bottlenecks [17]. Unless specified otherwise, we replace RSA layers, there are RSA layers total where L = 8, = CO M=5x7x7, x7. We set the input and output channel dimensions of RSA layers to be equal to those of spatial convolution layers TSN-ResNet50.
Training & testing details. For initialization, we randomly initialize the weights of bottlenecks including RSA layers with the MSRA method [16] and use ImageNet pre-trained weights for all the other layers. We set the gamma parameter of the last batch normalization layer to zero. For training, we sample 8 or frames from each video using the segment-ban testing, we sample one or two clips consisting of 8 or 16 frames using the segment-baseds sampling, and average softmax scores for final prediction. Refer to Sec. in our Supp. for more details.
sampling strategy [54]. For
5.2 Datasets
Something-something v1 & v2 (SS-V1 & V2) [14] are both large-scale action recognition bench- marks, including 108k and 220k action clips, respectively. Both datasets share the same motion- centric action classes, e,g. 'pushing something from left to right,' so thus capturing fine-grained motion crucial to achieve the better performance.
Diving-48 [30] a fine-grained action benchmark that is heavily dependent on temporal modeling [3], containing 18k videos with 48 diving classes. Due to the incorrect label issue, we only compare our result with the results on the modified version of Diving-48.
FineGym [39] is motion-centric benchmark that includes gymnastics action classes. We report results on two subsets of Gym288 and Gym99 that contain 288 and 99 action classes, respectively.
5.3 Comparison with other transform methods.
In this experiment, we evaluate the temporal modeling capability of different transform methods: spatio-temporal convolutions [48, 49, 50], self-attention [37] with its variants, and RSA. We replace a single 3 x 3 spatial convolution layer in ReN-Re [54] with single spatio-temporal transform layer. We analyze the ability of modeling temporal dependency of each transform layer with an apple-to-apple comparison. We use frames the input, and the kernel size M of all spatio-temporal transforms is set to 5x7x7.
Table 2 summarizes the results. 2D convolution baseline without modeling temporal dependency shows the lowest accuracy 19.7%. While 3D [48] and (2+1)D [50] convolutions, which use static spatio-temporal transforms, show the top-1 accuracy of 43.3% and 44.1 1, respectively, self-attention only achieves 41.6%. To find out the reasons behind the bad result, we ablate each component of self- attention, e.g., content-to-conten interaction (x Ω (X TT, content-to-postnenn -ossiiioon interaction (z (T)

---

Table 3: Performance comparison on SS-v1&v2, FineGym, and Diving-48.
(a) SS.VI&VZ. IN and IN2IK and K400 denote ImageNet- Ik, ImageNet- 21K. and Kinetics-400 dataset, respectively. Our method achieves a new Late-of-the-we accuracy on both datasets.
(b) Diving-48. Top-1 accuracy,
FLOPs are shown. Results in the upper compartment are from [3].
model
prc- trained
#frame
FLOPs xelips
top-1
SS-VI
top-5
top-1
SS-V2
top-5
model
FLOPss xclliss
top-1
13D [56)
ISM-R50 [31]
IN
IN
16
65 Gx1
47.2
72.2
77.1
63.4
88.5
Slow 8101
TimeSformer
TimeSformer-HR [9]
1703
213 Gx3
Gx3
75.0
77.6
78.0
ir-C5N-152 [49]
97 Gx 10
49.3
TimeSformer-1 [3]
2380
81.0
Slow 88-888 [11]
K400
32
67 Gx3
61.7
86.9
CT:Ner-R50 [28]
IN
16
75 G
52.5
80.9
64.5
89.3
2550
22GN2
84.2
STM-R50 [21]
IN
16
67 Gx:30
50.7
80.4
64.2
89.8
(c) FineGym. The averaged per- class accuracy (%) is reported. All results in the upper compart- ment are from [39].
CorrNet-R [53]
32
187 1x
50.9
TEA|29]
IN
81,9
89.9
MSNNI-TSM-R50 (23)
IN
16
67 Gx1
$2.1
82.3
64.7
89.4
NL-I3 [55] from [56]
IN
76.0
TimeSformer-HR [3]
IN
1703 Gx
62.2
TineSformee-L [3]
IN
96
2380 GX3
62.4
model
Gym288
Gym19
[1]
& K400
IN21K
32
N/Ax4
65.4
89.8
TRN [61]
33.1
68.7
13D [S]
27.9
63.2
RSANet-R50 (ours)
IN
Gx1
51.9
79.6
64.8
89.1
TSM [31]
34.8
70.6
RSANet-R50 (ours)
IN
16
72 Gx1
54.0
81.1
66.0
89.8
TSM mee- [31]
46.5
81.2
RSANcI-R50g (ours)
IN
82.6
RSANet-R50gs (ours)
IN
8+16
108 Gx2
56.1
82.8
67.7
91.1
RSANct-R50
50.9
86.4
and softmax (o)) one by one. We observe that the performance of self-attention without x, pT significantly decreases, while that of self-attention without ) is comparable to the original one. It indicates that the temporal modeling capability of the self-attention is actually dependent on the content-to- content-to-position interaction rather than the content-to-comtet so thus it is hard learn position-speciffe features, e.g.
interaction; the self-attention
mechanism itself permutation-invarian motion [32] without the positional embedding. After we remove the softmax non non-linearity, where the kernel equivalent to the basic kernel (xx x Q PT), outperforms both the self-attention and the standard static transforms. It demonstrates the softmax function restricts the expressive ability of the kernel [27]; the softmax forces the kernel weights to be positive, so that the kernel may not compute gradients across frames, which are effective in learning motion. Please refer to the Fig. 3 and Fig. 3 in our Supp. for the qualitative results.
While the composability of the basic kemel depends on the query content, our relational kernel depends on the local pattern of query-key correlation. As we add the relational kernel K, k.y we improve the top-1 accuracy by 1.4%p. The result demonstrates that leveraging local query-key interactions are effective in aggregating informative context for learning motion. At last, by adding relational context X R to basic context X RSA achieves the best top-1 accuracy of 47.0%.
5.4 Comparison to the state-of-the-art methods
Table 3a compares our method to the state-of-the-ee methods on SS-VI and V2 datasets. The first com- partment shows the results of the 3D CNNs. The second compartment contains the motion-modeling methods: both STM [21] and TEA [29] compute frame-wise differences, and both CorrNet [53] and MSNet [23] compute inter-frame pixel-wise correlations to learn motion features. The third compartment reports the results of global self-attention- models. NI.I3D [55] inserts non-local blocks to the 3D CNN for capturing long-range spatio-temporal dependencies. TimeSformer-L. TimeSformer-HH HR [3] and VIVIT [1] are transformer architectures that learn video representations via factorized apatio-temporss self-attention. Our method, RSANet-R50, achieves 51.9% and 64.8% at top-1 accuracy SS-VI and V2 datasets, respectively, which are already competitive to most of existing methods while using 8 frames only. When we use 16 frames, our method outperforms other existing models by achieving 54.0% and 66.0% at top-1 accuracy on SS-VI and V2, respectively. Fi- nally, our ensemble model with 2 clips achieves 56. 1% and 67.7%, which set the new state-of-the-an on SS-V1 and V2 with reasonable computational cost.
Table 3c and Table 3b present the results on Diving-48 [30] and FineGym [39]. For Diving-48, our model achieves 84.2%, substantially outperforming the state-of-the -fete--rt 3D CNN and transformer
8

---

Table Ablation studies on SS-v1 dataset. All models use TSN- TSN-ResNet50 [54] the backbone. Top-1, top-5 accuracy (%), FLOPs (G) and paramaters (M) are shown.
(a) Combinations of different kernels and context.
(b) Latent dimension Decomposing H signifi-
single RSA layer is inserted into stage4.
cantly reduces the computation cost. OOM is an abhre-
biation of out-of-memory 8 video clips per single
kernel
context
FLOPs
parums.
top-1
top-5
GPU machine are used.
32.3
23.4
44.8
73.8
32.7G
23.6
45.4
73.9
D
FLOPs
perams.
memory
top-1
top-5
X
32.7G
M
45.7
74.8
62.9 G
32.0 M
0OM
0OM
0OM
X
32.7G
23.4
46.2
75.4
8
32.2
20.5 M
8.8 GB
50.1
78.8
X
G
23.6
46.5
75.6
34.7G
9.2
51.3
78.8
X
33.2
23.6 M
46.7
75.6
32
39.6
21.7 M
10.2 GB
50,9
79.0
X + R
32.7G
23.4 M
46.5
75.6
C9/2
32.9
21.1 M
8.8 GB
51.1
79.1
+
33.2G
23.6 M
46.8
75.6
C
35.9
22.0 M
9.6 GB
51.5
79.2
Rg +
R
X
+
33.2G
23.6 M
47.0
75.7
(c) Kernel size M. In most cases, larger kernel results the higher accuracy.
(d) Group Hadamard product = C performs the highest accuracy. Note that FLOPs are consistent with varying G due to the switched computation order.
kemel size M
FLOPs
parums.
lop-1
top-5
3
28.5 G
20.3 M
49.4
77.6
Groups G
FLOP
params.
top-1
top 5
X
30.2 G
20.7 M
50.5
78.7
35.9 G
20.2
50.4
78.9
32.6
21.2
50.7
G
20.2
50.9
78.9
3
22.0 M
51.1
79.1
35.9
20.3 M
51.2
78.9
3
22.0 M
51.5
79.2
35.9
20.5
51.2
9
G
2
51.2
78.9
09
35.9
22.0
51.5
79.2
architectures. For FineGym, our model outperforms other methods in the averaged per-class accuracy of 50.9% and 86.4% given 288 and 99 classes, respectively.
5.5 Ablation studies
We conduct ablation experiments to validate the effectiveness of RSA. We use 8 frames for all experiments. Other training and testing details are in Sec. 5.1.
Combinations of different kernels and contexts. In Table 4a, we compare the performance of single RSA layer with different combinations of dynamic kernels and contexts. We first vary different types of the kernels from xY, K to x + R while the context is fixed as X Compared x X Kig X improves the accuracy by 0.6%. It indicates that predicting kernel from the composition of local query-key correlation is effective in modeling temporal dependencies. As we obtain additional improvements
both of the basic and relational transforms, (x.) = by 0.3%p and 0.9%p at top-1 I and top-5 accuracy, respectively. We also vary different types of the context such X. XR. and X + +X while the kernel is fixed. We observe the consistent improvements across different types of contexts, which means that aggregating the relational context beneficial for learning relational information. Finally, as we combine all dynamic transforms, (R=) + RR (XY RR), we achieve the highest accuracy of 47.0%.
Latent dimension D. In Table 4b, validate the effectiveness of decomposing H with latent channel dimension D. Without decomposing H, the TSN-ResNet50 with 7 RSA layers requires 62.9 GFLOPs and 32.0 M parameters, resulting in out-of-memory error. After we decompose H with small D. we significantly reduce FL.OPs, the number of parameters, and the memory footprint. We set D C the default that performs the highest accuracy.
Kernel sizes. In Table 4c, we compare the effect of the kernel size M. In most cases, larger spatio- temporal kernel improves the accuracy, except the case of M 5 9 x 9. Considering the static convolution counterparts, the RSA effectively enlarges the spatio-tempon smaller FLOPs and parameters. We choose M 5 x 7 as the default kernel size that shows the best computation -ccuuraccccccc trade-off.
kernel size, requiring
Correlation computation. In Table 4d, we validate the effect of the group-wise correlation 58), which splits the query and the key embeddings into groups and computes product between each group. While the dot product correlation contracts all channel dimensions into scalar correlation

---

Input Video
self-Artention kernel
Basic Kernel (AC))
040
kernel (Kf))
(a) "Moving something down.' (origin)
(b) 'Moving something up.' (reversed order)
Figure 3: Kernel visualization results on SS-V1. From the top to the bottom in each subfigure, we visualize the input RGB frames, the self-attention kernels, the basic kernels, and the relational kernels. The query position and the context are marked as red and yellow in RGB frames, respectively. The size of spatio-temporal kernel M is set as m, X my X my = 5 7 (L 8) are shown for each transform.
and kernels out of kernels
score the dot-product self-attention [52]. the group-wise correlation outputs G-dimensional correlation vector, providing richer semantic information. As a result, the Hadamard product, i.e., G' = Cº, that computes full element-wise correlations achieves the best performance. We thus set the Hadamard product as the default correlation function. Note that the computational complexity for generating the relational kernel remains the same while the size of G varies since switch the orders of computation as Eq. 10.
5.6 Kernel Visualization
In Figure 3, we visualize the dynamic kernels of self-attention and RSA from the learned models in 4th and 7th rows in Table 3, respectively. In visualization, we observe that both the basic kernels and relational kernels resemble edge detectors, e.g., Sobel filters Laplacian filters [46], along temporal axis that compute discrete approximation of spatio-temporal gradients across frames. When reverse the temporal order of an input video clip, the relational kernel dynamically varies according to whether the object moves up or down but the basic kernel remains the same. Considering that motion information is related to the relative changes along the temporal axis, the results indicate that the RSA kernels effectively capture motion patterns within the context, whereas the self-attention kernels limited to aggregating the local context based the query-key similarities.
Conclusion
We have presented the RSA feature transform, which captures rich relational patterns for video understanding, and validated that it outperforms other dynamic feature transforms in learning motion dynamics in videos. The proposed RSANet outperforms the state-of-the methods standard motion-centric action benchmarks. While we have focused on vidco representation learning this work, we believe the RSA will also benefit image understanding and natural language processing. We leave this for future work.
Acknowledgments and Disclosure of Funding
This work was supported by the NRF grants ( 2021R1A2C3012729 and the IITP grant (No.2019-0-01906, AI Graduate School Program POSTECH) funded by Ministry Science and ICT, Korea. This work was done while Manjin working as intern Microsoft Research Asia.
NRF-
10

---

References
[1] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid. Vivit: video vision transformer. arXiv preprint arXiv:2103. /5691, 2021.
[2] I. Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08802, 2021.
[3] G. Bertasius, H. Wang, and L. Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095, 2021.
[4] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In Proc. European Conference on Computer Vision (ECCV), 2020.
[5] Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[6] Y. Chen, X. Dai, M. Liu, D. Chen, L. Yuan, and Z. Liu. Dynamic convolution: Attention convolution kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1130-11033, 2020.
[7] A. Dosovitskiy, L.. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19999, 2020.
[8] L.. Fan, S. Buch, G. Wang, R. Cao, Y. Zhu, J. C. Niebles, and I.. Fei-Fei. Rubiksnet: Learnable 3d-shift for efficient video action recognition. In Proc. European Conference Computer Vision (ECCV), 2020.
[9] I.. Fan, W. Huang, C. Gan, S. Ermon, B. Gong, and J. Huang, End-to-end learning of motion representation for video understanding. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[10] C. Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[11] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
[12] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[13] K. Fukushima and S. Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets. 1982.
[14] R. Goyal, S. E, Kahou, V. Michalski, Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense, In Proc. IEEE International Conference on Computer Vision (ICCV), 2017.
[15] X. Guo, K. Yang, W. Yang, X. Wang, and H. Li. Group-wise correlation stereo network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019.
[16] K. Hc, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[18] H. Hu, Z. Zhang, Z. Xie, and S. Lin. Local relation networks for image recognition. In Proceedings of the IEFE International Conference on Computer Vision, pages 3464-3473, 2019.
11

---

[19] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint urXv:1405.38666 2014.
[20] X. Jia, B. De Brabandere, T. Tuytelaars, and L. Van Gool. Dynamic filter networks. In NIPS, 2016.
[21] B. Jiang, M. Wang, W. Gan, W. Wu, and J. Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
[22] J.-H. Kim, K.-W. On, W. Lim, J. Kim, J.-W. Ha, and B.-T. Zhang. Hadamard product for low-rank bilinear pooling, arXiv preprint I 2016.
[23] H. Kwon, M. Kim, S. Kwak, and M. Cho. Motionsqueeze: Neural motion feature learning for video understanding. arXiv preprint vrXiv:2007.099333 2020.
[24] H. Kwon, M. Kim, S. Kwak, and M. Cho. Learning self-similarity in space and time as generalized motion for action recognition. arXiv preprint arXiv:2102.07092, 2021.
[25] Y. LeCun, L. Bottou, Y. Bengio, and Haffner. Gradient-base learning applied document recognition. Proceedings of the IEEE, 1998.
[26] M. Lee, S. Lee, S. Son, G. Park, and N. Kwak. Motion feature network: Fixed motion filter for action recognition. In Proc. European Conference on Computer Vision (ECCV), 2018.
[27] D. J. Hu, C. Wang, X. Li, Q. She, L. Zhu, T. Zhang, and Q Chen. Involution: Inverting the inherence of convolution for visual recognition. arXiv preprint arXiv:2103.06255, 2021.
[28] K. Li, X. Li, Y. Wang. J. Wang, and Y. Qiao. (net: Channel tensorization network for video classification. In International Conference on Learning Representations, 2021.
(29] Y. Li, B. Ji, X. Shi, J. Zhang, B. Kang, and L. Wang. Tea: Temporal excitation and aggregation for action recognition. In Proc. IEEÉ Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[30] Y. Li, Y. Li, and N. Vasconcelos. Resound: Towards action recognition without representation bias. In Proc. European Conference on Computer Vision (ECCV), 2018.
[31] Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
[32] X. Liu, J.-Y. Lee, and H. Jin. Learning video representations from correspondence proposals. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[33] N. Ma. X. Zhang. Huang, and J. Sun. Weightnet: Revisiting the design space of weight
networks. arXiv preprint 1rXiv:2007.11823; 2020.
[34] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufflenet v2: Practical guidelines for efficient cnn
architecture design. In Proc. European Conference on Computer Vision (ECCV), 2018.
[35] D. Neimark, O. Bar, M. Zohar, and D. Asselmann. Video transformer network. arXiv preprint arXiv:2102.00719, 2021.
[36] A. Piergiovanni and M. S. Ryoo. Representation flow for action recognition. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[37] P. Ramachandran, N. Parmar, A. Vaswani, I, Bello, A. Levskaya, and J. Shlens. Stand-alone self-attention in vision models. In Proc. Neural Information Processing Systems (NeurIPS), 2019.
[38] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[39] D. Shao, Y. Zhao, B. Dai, and D. Lin. Finegym: A hierarchical video dataset for fine-grained action understanding. Proc. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
12

---

[40] E. Shechtman and M. Irani. Matching local self seff-similarities across images and videos. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
[41] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Proc. Neural Information Processing Systems (NeurIPS). 2014.
[42] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[43] R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for semantic segmen- tation, arXiv preprint arXiv:2 1ete220050006333 2021.
[44] S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed. D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[46] R. Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
[47] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proc. International Conference on Machine Learning (ICML). 2019.
[48] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proc. IEEE International Conference Computer Vision (ICCV), 2015.
[49] D. Tran, H. Wang, L. Torresani, and M. Feiszli. Video classification with channel-separated convolutional networks. In Proc. IEEE International Conference Computer Vision (ICCV), 2019.
[50] D. Tran, H. Wang. L. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[51] A. Vaswani, P. Ramachandran, A. Srinivas, N. Parmar, B. Hechtman, and J. Shlens. Scaling local self-attention for parameter efficient visual backbones. arXiv preprint arXiv:2103.12331, 2021.
[52] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. In Proc. Neural Information Processing Systems (NeurIPS), 2017.
[53] H. Wang, D. Tran, L. Torresani, and M. Feiszli. Video modeling with correlation networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[54] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Proc. European Conference on Computer Vision (ECCV), 2016.
Conference Computer Vision and Pattern Recognition (CVPR), 2018. [55] X. Wang, R. Girshick, Gupta, and K. He. Non-local neural networks. Proc. IEEE
[56] X. Wang and A. Gupta. Videos space-time region graphs. In Proc. European Conference Computer Vision (ECCV), pages 399-417, 2018.
convoll [57] B. Yang, G. Bender, Q V. Le, and J. Ngiam. Condcony: Conditionally parameterized convolu- tions for efficient inference. arXiv preprint arXiv: 04971,, 2019.
[58] G. Yang and D. Ramanan. Volumetric correspondence networks for optical flow. Proc. Neural Information Processing Systems (NeurIPS), 2019.
13

---

[59] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens-to-toke vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2001. 1986, 2021.
[60] H. Zhao, J. Jia, and V. Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10076-10085, 2020.
[61] B. Zhou, A. Andonian, A. Oliva, and A. Torralba. Temporal relational reasoning in videos. In Proc. European Conference on Computer Vision (ECCV), 2018.
14


Reviews:
[{"review_id": "zx89PsBCWqv", "reviewer": "Reviewer_iQAh", "summary": "This paper proposes a new transform block called Intervolution for video understanding. Specifically, the Intervolution block is based on the recent dynamic transform blocks and further models the content-to-content relations in video data. The new designs of relational kernel and relational context improve the ability of dynamic transform to capture spatio-temporal relations. The experiments are conducted on three public datasets, i.e., SS-V1&V2, FineGym, and Diving-48. The experimental results show that the networks with Intervolution achieve higher accuracy on action recognition than other approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is mostly clear and easy to follow.\n+ The utilization of dynamic transform on video understanding is interesting.\n+ The performances are high on public datasets.\n \n\nWeaknesses:\n- Why Intervolution for video data. The design of Intervolution is based the dynamic transforms for image understanding. And the architecture of Intervolution is not specifically designed for video data. I am not sure why this paper only validate Intervolution on video recognition task. Although the paper claims that the relation modelling in Intervolution could help capture motion information in video, I think relation is also important for image understanding. It is not a convincing reason why the paper only experimented on video data. Meanwhile, without the results on image data, it is hard to directly compare Intervolution with other dynamic transforms.\n- Comparison with Involution and Lambda convolution. To me, the architecture of Intervolution combines the several designs in self-attention, Involution and Lambda convolution. However, in Table 2, only self-attention (LSA) is compared. It is not surprising that Intervolution is higher than LSA since Involution and Lambda convolution have already been proven to be better than LSA.  I understand that some settings in the ablation study are similar as Involution or Lambda convolution. The more direct comparisons with these two blocks should be given.\n- Dataset. The experiments are mainly conducted on the motion-centric datasets. It is acceptable since these datasets could better validate the ability of Intervolution to capture motion information. However, I still wonder how Intervolution performs on the datasets focusing more on object/scene (e.g., Kinetics dataset). The authors could give some results on such datasets or discuss these datasets to further provide some valuable insights on dataset bias.\n- Backbone. The experiments in this paper mainly utilize ResNet50 as backbone. The results on more backbones could make the conclusions more convincing.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations of this work. The potential negative societal impact is not discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "tAqe-hEuHr", "reviewer": "Reviewer_1xyW", "summary": "The paper proposes a dynamic feature transform method named intervolution. While bearing similarities with involution and lambda convolution, the proposed intervolution proposes to add content-to-content relational learning in both dynamic kernel generation and feature value transforms. Alongside the basic kernel and context transforms, additional learnable parameters are introduced for the relational kernel and context learning. To further improve the efficiency of intervolution, decomposition of the projection matrix and the use of multiple queries are proposed. The experiments use TSN-ResNet50 baseline and replace the 3D convolution layers with intervolution layers. Experiments on SS-v1 and v2, Diving-48 and FineGym datasets show that intervolution provides a reasonable efficiency-accuracy trade-off.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The proposed relational kernel and context learning is effective in learning spatial-temporal interactions compared to the static as well as existing dynamic feature transform methods. \n\n* Intervolution introduces not much of extra FLOPs, number of parameters while achieving consistent improvements on different action recognition benchmarks.\n\n* What does memory usage look like in Table 2? \n\n* It seems like the current method exploits CUDA implementation. Does it affect FLOPs or memory usage without CUDA modification? (e.g. pure Pytorch style implementation).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "YFgIyPzhAPe", "reviewer": "Reviewer_rPEM", "summary": "The authors propose a novel approach for action recognition. They integrate the ideas from current popular operators: convolution, involution, and self-attention into a relational feature transform. The new method can capture the spatial-temporal relations for video understanding, and it achieves state-of-the-art results on the motion-centric action recognition benchmark. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The authors address the problems in current popular operators for learning video representations. The motivation is clear and the idea is interesting. \n2. The method is lightweight and effective. It can reduce the computation cost and improve the accuracy of action recognition at the same time. \n3. The proposed approach can achieve SOTA results on public datasets like SS-V1&V2 with very light computation costs (GFLOPs). The experiments on motion-centric benchmarks and the ablation study are sufficient. \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. I can tell the authors try very hard to describe the motivation of intervolution from convolution, self-attention, and involution. But it still takes me some time to figure out the idea of the relational kernel, relational context, and some other concepts like content-to-content and content-to-position. I am wondering whether the author could simplify the presentation of the method so that the readers can easily interpret and reproduce the method. \n2. This paper is targeted to improve the relational representation; thus, the experiments are mostly applied on motion-centric datasets. However, the readers would like to see the comparison results on the other action recognition datasets like Kinetics. \n3. I know there is a huge literature working on this topic. However, please cover some representative methods: TRN [1], TPN [2] as well as the recent ones: MoviNet [3], etc. \n \n[1] Zhou, Bolei, et al. \"Temporal relational reasoning in videos.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018. \n[2] Yang, Ceyuan, et al. \"Temporal pyramid network for action recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. \n[3] Kondratyuk, Dan, et al. \"Movinets: Mobile video networks for efficient video recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "9kzVFSQPrbm", "reviewer": "Reviewer_KG6i", "summary": "The paper introduces a module that can process the spatio-temporal relations videos. The authors formalise the method as predicting for each point (target) in a feature map the output feature vector by aggregating a neighbourhood, acting as a context (of size $M \\times C, M = T \\times H \\times W$) according to a dynamic kernel (of size M). Many models fit this framework, including convolution, self-attention, but the proposed method is shown experimentally to be superior. The paper introduces two main components: a *relational kernel* for aggregating the context, and *relational context* for improving the context, and shows that both help the performance of the model.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points**:\n\n* The paper clearly presents the relation to prior work. The method is sound and the technical aspects are clearly explained.  \n\n* The method achieves superior performance on multiple datasets. The datasets are suitable for evaluating video processing methods and require modeling complex movements (e.g. for Smt-smt the motion is essential, the class cannot be predicted only from the general context) \n\n\n* The paper presents good ablation studies, showing the importance of the proposed modules. The fair comparisons with other methods from Table 2 (especially local attention) are also appreciated.\n\n\n**Weak Points**:\n\n* Since omitting the softmax in the kernel seems to improve the performance, it will be useful to test LSA with the full kernel but without softmax (in table 2, just the variant with only content-to-position kernel is compared).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The method shares the same Societal impact as any image/video classification method and it is addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received three borderlines (1 negative, 2 positive) and one accept ratings. The main weaknesses were:
1. The results on Kinetics are weak.
2.  The ideas are kind of incremental.
3.  The presentation of this paper is unclear.

* The AC does not consider 1 an important concern. It would be nice to have strong results on Kinetics as well, but kinetics relies more on appearance rather than motion and long-range interactions  (the focus of this paper). 
* The AC shares the same concern as 2 and 3. The technical contributions and the writing quality do not meet the bar of NeurIPS. 

Due to these issues, the AC recommends rejection. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Accept (Poster)