Learning Self-Similarity in Space and Time as a Generalized Motion for Action Recognition
Authors: Heeseung Kwon
Manjin Kim
Suha Kwak
Minsu Cho
Venue:  (2021)
Keywords: Action recognition
Video understanding
Motion analysis

Abstract: Spatio-temporal convolution often fails to learn motion dynamics in videos and thus an effective motion representation is required for video understanding in the wild. In this paper, we propose a rich and robust motion representation method based on spatio-temporal self-similarity (STSS). Given a sequence of frames, STSS represents each local region as similarities to its neighbors in space and time. By converting appearance features into relational values, it enables the learner to better recognize structural patterns in space and time. We leverage the whole volume of STSS and let our model learn to extract an effective motion representation from it. 
The proposed method is implemented as a neural block, dubbed SELFY, that can be easily inserted into neural architectures and learned end-to-end without additional supervision. With a sufficient volume of the neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition.
Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolution. On the standard action recognition benchmarks, Something-Something-V1 & V2, Diving-48, and FineGym, the proposed method achieves the state-of-the-art results.  

Under review as a conference paper at ICLR 2021
LEARNING SELF-SIMILARITY IN SPACE AND TIME AS A GENERALIZED MOTION FOR ACTION RECOGNITION
An Anonymous authors Paper under double-blind review
ABSTRACT
Spatio-temporal convolution often fails to learn motion dynamics in videos and thus an effective motion representation is required for video understanding in the wild. In this paper, propose rich and robust motion representation method based on spatio-temporal self-similarity (STSS). Given a sequence frames, STSS represents each local region similarities its neighbors in space and time. By converting appearance features into relational values, enables the learner to better recognize structural patterns in space and time. We leverage the whole vol- ume of STSS and let our model learn to extract an effective motion representation from it. The proposed method is implemented as a neural block, dubbed SELFY, that can be easily inserted into neural architectures and learned end-to-end with- out additional supervision. With a sufficient volume of the neighborhood space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition. Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complemen- tarity to spatio-temporal features from direct convolution. the standard action recognition benchmarks, Something Sooeiining V1 & V2, Diving-48 and Fine- Gym, the proposed method achieves the state-of-the-ar results.
INTRODUCTION
Learning spatio-temporal dynamics is the key to video understanding. To this end, extending convo- lutional neural networks (CNNs) with spatio-temporal convolution has been actively investigated in recent years (Tran al. 2015 Carreira & Zisserman 2017 Tran al., 2018). The empirical results so far indicate that spatio-temports convolution alone is not sufficient for grasping the whole picture; it often learns irrelevant context bias rather than motion information (Materzynska al. (0200) and thus the additional use of optical flow turns out to boost the performance in most cases (Carreira & Zisserman 2017 Lin al. 2019). Motivated by this, recent action recognition methods learn to extract explicit motion, i.e., flow or correspondence, between feature maps of adjacent frames and they improve the performance indeed (Li et al., [2020c] Kwon et al. 2020). But, is it essential to extract such an explicit form of flows or correspondences? How can we learn richer and more robust form of motion information for videos in the wild?
Spazeelfine
.uerry
Short-term reighborhood
STSS tensor
ST8S reprssntation
Spetial neighboroodd
Long-tam nei negntorhood
Figure 1: Spatio-temporal self-similarity (STSS) representation learning. STSS represents each spatio-temporal position (query) as its similarities (STSS tensor) with its neighbors space and time (ncighborhood). STSS allows take generalized, far-sighted view motion. i.e.. both short-term and long-term, both forward and backward, as well as spatial self-motion. Our method learns to extract rich and effective motion representation from STSS without additional supervision.

---

Under review as a conference paper at ICLR 2021
In this paper, we propose to learn spatio-temporal self-similarity (STSS) representation for video understanding. Self-similarity is a relational descriptor for an image that effectively captures intra- structures by representing each local region similarities its spatial neighbors (Shechtman & Trani 2007). Given a sequence of frames, iee, a video, it extends along the temporal dimension and thus represents each local region similarities its neighbors in space and time. By converting appearance features into relational values, STSS enables a learner to better recognize structural patterns in space and time. For neighbors at the same frame it computes a spatial self-similarity map, while for neighbors at a different frame it extracts a motion likelihood map. If we fix our attention to the similarity map to the very next frame within STSS and attempt to extract a single displacement vector to the most likely position at the frame, the problem reduces to optical flow, which partieular type of motion information. In contrast, we leverage the whole volume of STSS and let our model learn to extract an effective motion representation from it in an end-to- end manner. With a sufficient volume of neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition.
We introduce a neural block for STSS representation, dubbed SELFY, that can be easily inserted into neural architectures and learned end-to-end without additional supervision. Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its com- plementarity to spatio-temporal features from direct convolutions. On the standard benchmarks for action recognition, Someehing-SSmettiiee V1& V2, Diving-48, and FineGym, the proposed method achieves the sfate-of-thh-ar results.
2 RELATED WORK
Video action recognition. Video action recognition is a task to categorize videos into pre-defined action classes. One of the conventional topics in action recognition is to capture temporal dynam- ics in videos. In deep learning, many approaches attempt to learn temporal dynamics in different
ways: Two-stream networks with external optical flows (Simonyan & Zisserman 2014
Wang et al.]
2016), recurrent networks (Donahue et al. 2015), and 3D CNNs (Tran et al. 2015 Carreira & Zisserman, 2017). Recent approaches have introduced the advanced 3D CNNs (Tran et al.) 2018 Feichteeitt 2020 Lin et al. 2019 Fan et al. 2020) and show the effectiveness of captur- ing spatio-temporal features, so that 3D CNNs now become a de facto approach to learn temporal dynamics in the video. However,
convolution is vulnerable unless relevant features are well aligned across frames within the fixed-sized kernel. To address this issue, a few methods adaptively translate the kernel offsets with deformable convolutions (Zhao et al.] 2018 Li et al., 2020a), while several methods (Feichtenhofer et al.] 2019 Li et al. (020b) modulate the other hyper-parameters, e.g., higher frame rate or larger spatial receptive fields. Unlike these methods, we address the problem of the spatio-temporal convolution by sufficient volume of STSS, capturing far-sighted spatio-temporal relations.
Learning motion features. Since using the external optical flow benefits 3D CNNs to improve the
action recognition accuracy (Carreira & Zisserman 2017] Zolfaghari et al. 2018 8
Tran et al. 2018))
several approaches try learn frame-by-trame motion features from RGB sequences inside neural
architectures. Fan et al. (2018); (Pieriivvan & Ryoo (2019) internalize TV-L.1 (Zach et al.
2007
optical flows into the CNN. Frame-wise feature differences (Sun et al. 2018b Lee al.
2018
Jiang et al. 2019; Li et al. (020cc) are also utilized as the motion features. Recent innrelatioo-basse
methods (Wang et al. 2020 Kwon et al. 2020) adopt correlation operator (Dosovitskiy et al.
2015 Sun et al., 8a, Yang & Ramanan 2019) to learn motion features between adjacent frames.
However, these methods compute frame-by ber-byy-frra frame motion features between two adjacent frames and then rely on stacked spatio-temporal convolutions for capturing long-range motion dynamics. We propose to learn STSS features, as generalized motion features, that enable to capture both short- term and long-term interactions in the video.
Self-similarity. Self-similarity represents an internal geometric layout of images. It is widely used in many computer vision tasks, such as object detection (Shechtman & Irani, 2007). image retrieval (Hörster & Lienhart 2008), and semantic correspondence matching (Kim et al. 2015 2017). In the video domain, Shechtman & Irani (2007) firstly introduce the concept of STSS and transforms the STSS to a bran-crafted local descriptor for action detection. Inspired from this work, early methods adopt
for capturing view-invariant temporal patterns (Junejo et al.
2

---

Under review as a conference paper at ICLR 2021
Identity Mapping
Self-similarity Transformation
Feature
Feature Integration
V
Extraction
F
(T.X.Y.C)
(,
(T..Y.I..G)
(T.X.Y.C)
Figure 2: Overview of our self-similarity representation block (SELFY). SELFY block takes as STSS tensor S, and extracts feature tensor F
input a video feature tensor V. transforms from S. It then converts F to the same size as the input V via the feature integration, and combines with the input V to produce the final STSS representation Z. See text for details.
2008 2010; Körner & Denzler, 2013), but they use temporal self-similarities only due to computa- tional costs. Recently, there are several non-local approaches (Wang et al. 2018 Liu et al. 2019) that utilize STSS for capturing long-range dynamics of videos. However, they use STSS for re- weighting or aligning visual features, which is an indirect way of using STSS. Different from these methods, our method leverages full STSS directly as generalized motion information and learns an effective representation for action recognition within a video-processing architecture. To the best of our knowledge, our work is the first in learning STSS representation using modern CNNs.
The contribution of our paper can be summarized as follows. First, we revisit the notion of self- similarity and propose to learn generalized, far-sighted motion representations from STSS. Second, we implement STSS representation learning as a neural block, dubbed SELFY, that can be integrated into existing neural architectures. Third, we provide comprehensive evaluations SELFY, achiev- ing the state-of-th oftte-of-hh-ar- art on benchmarks: Something Soaitiiingg &1&V2, Diving-48, and FineGym.
OUR APPROACH
In this section, first revisit the notions of self- similarity and discuss its relation motion. We then introduce our method for learning effective spatio-temporal self relf-similarrty representation, which can be easily integrated into video-processing architectures and learned end-to-end.
3.1 SELF-SIMILARITY REVISITED
Self-simin Self-similarity is relational descriptor that suppresses variations in appearance and reveals struc- tural patterns in images or videos (Shechtman Irani 2007).
Given image feature map I X-XXXCC, self-similarity transformation of I results in 4D tensor R xxxxxxxx whose elements are defined as
Sz.yuuv = sim Iotu.tt))
where sim(-,)) -) is a similarity function, e.g., cosine similarity. Here, (x, is query coordinate while (u, v) is a spatial offset from it. To impose locality, the offset restricted to its neighbor- hood: (u, v) [-du, de] x [-dv. dv that U 2dg + and V 2dy + 1, respectively. By converting C-dimensional appearance feature Ir.y into UV-dimensionnl relational feature Sz.y- suppresses variations in appearance and reveals spatial structures in the image. Note that the self- similarity transformation closely relates to conventional cross-similarity (or correlation) across two different feature maps (I, I' E xx)xx)),, which can be defined as
Szymue
sim (L +x0))
Given two images of a moving object, the cross-similarity transformation effectively captures motion information and thus is commonly used in optical flow and correspondence estimation (Dosovitskiy
et al. 2015 Sun et al. 2018a Yang & Ramanan 2019).
For sequence of frames, i.e., a video, one can naturally extend the spatial self-similarity along the temporal axis. Let V E R xTXXXXx denote a feature map of the video with T frames. Spatio- temporal self-similarity (STSS) transformation of results a 6D tensor S E XXXXXXXXXXXXXX whose elements are defined as
S Sim.g,l..vv sim V (,z,gs VeHlz-agtt);
3

---

Under review as a conference paper at ICLR 2021
index map
MLP
Conv
index map (L.2)
(1.8, V)
(1.0.1)
(V.
(L.G)
(L.U.V)
(2.
(a) Soft-argmax
(b) ML.
(c) Convolution
Figure 3: Feature extraction from STSS. For a apatio-tempore position (t. x, y), each method
transforms (L,U, V) volume of STSS tensor S into (L, Cp). See text for details.
where (t, x. y) is the ppatio-temport coordinate and (1, u, v) is a spatio-temporal offset from it. In addition to the locality of spatial offsets above, the temporal offset | is also restricted to its temporal neighborhood: I E [-d1., d1.), so that I, 2d,. +
What types of information does STSS describe? Interestingly, for each time t, the STSS tensor can be decomposed along temporal offset 1 into single spatial self-similarity tensor (when / = 0) and 2d,, spatial cross-similarity tensors (when * 0); the partial tensors with a small offset (e.g.. 1 = 1 or +1) collect motion information from adjacent frames and those with larger offsets capture it from further frames both forward and backward time. Unlike previous approaches to learn motion (Dosovitskiy et al. 2015 Wang et al.] 2020 Kwon et al., 2020), which rely cross- similarity between adjacent frames, STSS allows to take a generalized, far-sighted view on motion, Le., both short-term and long-term, both forward and backward, as well spatial self-motion.
3.2 SPATIO- TEMPORAL SIMILARIYY REPRESENTATION LEARNING
By leveraging the rich information in STSS, we propose to learn generalized motion representa- tion for video understanding. To achieve this goal without additional supervision, we design a neu- ral block, dubbed SELFY, which can be inserted into a mideo-procesee end-to-end. The overall structure illustrated Fig. 2 transformation, featuue extraction, and feature integration.
architectures and learned It consists of three steps: self-similarity
Given the input video feature tensor V, the self-similarity transformation step converts it to the STSS tensor S as in Eq.[I In the following, we describe feature extraction and integration steps.
3.2. FEATURE EXTRACTION
From the STSS tensor S E XXXXXXXXXXXXX we extract a Cp-dimensional feature for each spatio- temporal position 2r, y) and temporal offset / so that the resultant tensor is F R x LxCr, which is equivariant to translation in space, time, and temporal offset. The dimension of L is pre- served to extract motion information across differrntt temporal offsets in a consistent manner. While there exist many design choices, we introduce three methods for feature extraction in this work.
Soft-argmax. The first method is compute explicit displacement fields using S, which previous motion learning methods adopt using spatial cross-similarity (Dosovitskiy et al. [015 Sun et al.) 2018a Yang & Ramanan 2019). One may extract the displacement field by indexing the positions with the highest similarity value via arg (a, but it is not differentiable. We instead use soft-
argmax (Chapelle & Wu 2010), which aggregates displacement vectors with softmax weighting
(Fig. 3a The soft-argmax feature extraction can be formulated as
cxp(S (xp(SS.....a../1))
F Fi.z.g.d
Σ u,v
Ew Enr
[u; v).
(2)
which results in feature tensor F E XXXXXxxxx2 The temperature factor adjusts the softmax distribution, and we set 7 = 0.01 in our experiments.
Multi-layer perceptron (MLP). The second method is to learn an MLP that converts self- self-similarity values into a feature. For this, we flatten the (U, V) volume into UV -iv-eiiensol vectors, and apply an MI. to them (Fig. 3b). For the reshaped tensor S XTxxXXXXXXXX perceptron f(-) can be expressed as
S(S) ReLU(S xs W
(3)
4

---

Under review as a conference paper at ICLR 2021
where X denotes the n-mode tensor product, W E RC is the perceptron parameters, and the output is f(S) ITXXXYXIXC*, The MLP feature extraction can thus be formulated as
..... F == o fn- fi)(S),
which produces feature tensor F E IIXXXXXXXXXX XLXCF This method is more flexible and may also be more effective than the soft-argmax because not only can encode displacement informa- tion but also it can directly access the similarity values, which may be helpful for learning motion distribution.
The third method is to learn convolution kernels over (L, U, V) volume of S (Fig. 3e When we regard S as a 7D tensor RTxXXY xLx xVxC with C = 1, the convolution layer be expressed using S' g(),, whose elements computed by
Convolution.
S' Sttpaaaa ReLU
Σ Klnwewwe t.n.th 00000)
(5)
where K E XU..UUXVXXXCC is multi-channel convolution kernel, (ha, Un, Ux is the kernel parameter indices, and (c, is the channel indices. The indices (f 8x) are centered as i 1x - L2/2, úg Ur 22 0x = 2x - Va/2. Starting from Txx xx XY XXXXXXx we gradually downsample (U,V) and expand channels via multiple convolutions with strides, finally resulting in RxXXXYXX XXXXXXXXXXXXXX we preserve the L. dimension, since maintaining fine temporal resolution is
shown be effective for capturing detailed motion information (Lin al.) 2019 Feichtenhofer et al. 2019). The convolutional feature extraction with n layers can thus be formulated as
F = (gn o 9n-1 .... o (S).
(6)
which results in feature tensor F R*XXXXxLXC, This method is effective in learning struc- tural patterns with their convolution kernels, thus outperforming the former methods as will be seen in our experiments.
3.2.2 FEATURE INTEGRATION
In this step, we integrate the extracted STSS features F E RXXXXXxLXCr to feed them back the original input stream with (T, X, Y, C)) volume.
We first use 3 x 3 spatial convolution kernels along (x, y) dimension of F. The spatial convolution layer h can be expressed using F' = h(F),, whose elements are computed by
F Flaaree ReLU
Σ
Kr..Wa.er.c F Flattee-bashor
where K E BXXxXxxCXXXG is the multi-channel convolution kernel, (Ix, 3n.) is the kernel param- eter indices, and (cF, dp) is the channel indices. (æx, 9x) is centered as 2x = 2x Xx./2, ØN = yr 12/ This type of spatial convolutions integrate the original features by extending receptive fields along (I, dimension. The resultant features F* E TXXXXXXXXXO
is defined
F (h,, hn-1 hF)
We then flatten the (T. CF) volume into d.Cp--dimenioonal vectors to obtain F' RTXXXXYLCC and apply an x x convolution layer to obtain the final output. This convolution layer integrates features from different temporal offsets and also adjusts its channel dimension to fit that of the original input V. We adopt the identity mapping of the input for residual learning (He et al. 2016 The final output tensor Z is expressed as
Z RRLU(F" Xs Wo) V,
(9)
where Xn is the n-mode tensor product and Wa E RCXLCP is the weights of the convolution layer.
4 EXPERIMENTS
4.1 DATASETS IMPLEMENTATION DETAILS
For evaluation, use benchmarks that contain fine-grained spatio-temporal dynamics in videos.
5

---

Under review as a conference paper at ICLR 2021
Table I: Performance comparison on SS-VI&V2. Top-1, 5 accuracy (%) and FLOPs (G) are shown.
model
flow
#frame
FLOPs xclips
SS-VI
SS-V2
top-1
top-5
top-1
top-5
TSN-R50 R50 from
Lin et al.
2019
8
G3Gx1
19.7
46.6
30.0
60.5
TRN-BNIncep
Zhou et
20T8
8
16 GxGNAA
34.4
48.8
IRN-Be Two-stream
hii
2018
8+8
MFNet- R50 Lee et 2018
10
NI
40.3
70.9
CPNet-R34 Lau 2019
24
N/A
57.7
84.0
TPN-R50 Yang et
N/
40.6
59.1
SHJ Net-R50 (ours)
8
37 G7x
50.7
79.3
62.7
88.0
13D from WWng & Gupta 2018
32
153 Gx2
41.6
72.2
NL-13D Wang Gunna
32
168 Gx2
TSM-R50 Lin et 2019
16
65 Gx1
47.3
77.1
61.2
86.9
TSM-R50 sir-ai from
(Kwon al
2020
16+16
129 Gx
52.6
81.9
65.0
89.4
1o (Wang
2020
32
187 Gx10
STM-R50 Jiang et ill 2019
16
67 Gx30
50.7
80.4
64.2
89.8
TEA R50 al. 20206]
16
70 Gxx30
$2.3
81.9
MSNet-TSX-RSU (Kwon 2020
16
67 Gx1
82.3
MSNe- (Kwon 2020
1648
101 Gx
55.1
84.0
67.1
91.0
SELFYNct-TSM-RE (150-50066 (ours)
8
Gx1
$2.5
80.8
64.5
89.4
SELFYNeI-TSM-RS (ours)
16
Gx
54.3
82.9
65.7
SELFYNet-TSAA MSMikkt--SM--5-ppp -550--50gg
(ours)
8+16
114 Gx1
55.8
83.9
67.4
91.0
SELFYNet-TSM-R50 (ours)
8+16
114 Gx2
56.6
84.4
67.7
91.1
Table 2: Performance comparison on Diving-48 & FineGym.
(a) Performance comparison Diving-48. Top-1 curacy (%) and FLOPs (G) are shown.
(b) Performance comparison FineGym. The
averaged per-class accuracy (%) shown. All re- sults in the upper part are from Shao et al. (020))
model
#frame
FLOP xclips
Top-1
model
Aframe
Gym288
Gym199
TSN from
TRN from
2019
22.8
TSN
Wane et al.
2016
26.5
61.4
Att-LSTM
64
NAx1
35.6
TRN
Zhou al.
2015
33.1
68.7
NAX1
C3D from
1.30 & VIIIE
2019
16
NAx1
34.5
NL ISD Waat or 2018
27.1
62.1
GST-R50
Tuille
2U19
16
59 Gix
388
TSM
[Li 2019
34.8
30.6
ConNect-Ri KiOy
GSM-IneV3 (Sudhakaran et al. 2020
16
54 Gx2
40.3
TSM- R50 (our impl.)
73.7
TSM-R50 (our impl.)
16
65 Gx2
38.8
TSM- 15M-R50 (our impl.)
47.9
86.6
St-PYY (ours)
SELFYNNe:TSM (ours)
49.5
87.7
Somnehing-SSomtthing V1 & V2 (SS-V1 & V2) (Goyal al. 2017b), which are both large-scale action recognition datasets, contain ~ 108k and ~220 video clips, respectively. Both datasets share the same 174 action classes that are labeled, e.g., "pretending to put something next something."
Diving-48 (Li et al. 2018). which contains 18k videos with 48 different diving action classes, is an action recognition dataset that minimizes contextual biases, i.e., scenes objects.
FineGym (Shao et al. 2020) is a fine-grained action dataset built on top of gymmastic vidcos. We adopt the Gym288 and Gym99 sets for experiments that contains 288 and 99 classes, respectively.
Action recognition architecture. We employ TSN ResNets (Wang et al. 2016) as 2D CNN back- bones and TSM ResNets (Lin al. 2019) as 3D CNN backbones. TSM enables obtain the effect of spatio-temporal convolutions using spatial convolutions by shifting part of input channels along the temporal axis before the convolution operation. TSM is added into each residual block of the ResNet. We adopt ImageNet pre-trained weights for our backbones. To transform the backbones to the self-similarity network (SELFYNet), we insert a single SELFY block after the third stage in the backbones. For SELFY block, we use the convolution method and use multi-channel x 3 3 convolution kernels. For more details, please refer Appendix A
default feature extraction method
Training & testing. For training, sample clip of 8 or 16 frames from each video by using segment-based sampling (Wang et al. 2016). The spatio-tempore
matching region (L, U, V) of
SELFY block is set as (5, 9, 9) (9, 9, 9) when using 8 or 16 frames, respectively. For testing, sample one or two clips from a video, crop their center, and evaluate the averaged prediction of the sampled clips. For more details, please refer Appendix

---

Under review as a conference paper at ICLR 2021
Table 3: Ablations on SS-V1. Top-1 & 5 accuracy (%) are shown.
(a) Types of similarity, Performance comparison with different sets of temporal offset SELFY block. {
denotes a set of temporal offset I.
model
range off
FL.PPs
10p-1
top-5
model
range of
FL.OPs
top-1
top-5
TSN-RI8
14.6
16.2
40.8
TSM-RI8
14.6
43.0
72.3
[0]
13.3
16.8
42.2
(0]
15.3
45.0
73.4
(1)
15.3
39.7
68.9
(1)
15.3
47.1
76.3
SELPYNet
{ 1,
16.3
44.7
73.9
SELPYNet
{
1-1,0,1)
16.3
47.8
76.7
2.
17.3
46.9
75.9
17.3
48.4
77.6
3,
18.3
46.9
76.2
3.
18.3
48.6
77.7
(b) Feature extraction and integration methods. Smax denotes the soft-argmax operation. MLP consist of four fully connected (FC) layers. The lutional layer the feature integration stage omitted from this table.
1 convo-
model TSM-RI8
feature extraction
feature integration
top-1
top-5
43.0
72.3
Smax
FC
44.0
72.3
MLP
FC
45.9
75.1
SELFYNet
Conv
FC
46.7
75.8
Conv
MLP
47.2
75.9
Conv
Conv
48.4
77.6
4.2 COMPARISON WITH THE STATE-OF-THE METHODS
For a fair comparison, we compare our model with other models that are not pre-trained on additional large-scale video datasets, e.g., Kinetics (Kay et al. 2017) or Sports (Karpathy et al., 2014) in the following experiments.
the results on SS.V The first and second compartment of the table shows the results of other 2D CNN and (pseudo-) 3D CNN models, respectively. The last part of each com- partment shows the results of SELFYNet. SELFYNet with TSN-ResNet (SELFYNet (SN-R50) achieves 50.7% and 62.7% top-1 accuracy, respectively, which outperforms other 2D models us- ing 8 frames only. When we adopt TSM ResNet (TSM-R50) as our backbone and use 16 frames, our achieves 54.3% and 65. 7% top-1 accuracy, respectively, which is the best among the single models. Compared to TSM-R50, a single SELFY block obtains a sig- nificant gain of 7.0%p and 4.5%p top-1 accuracy, respectively; our method is more accurate than TSM-R50 Two-stream on both datasets. Finally, our ensemble model (SELFYNet-SMM-R500 with 2-clip evaluation sets a new state of the on both datasets by achieving 56.6% and 67.7% at top-1 accuracy, respectively.
[ablee[summann
method -SELFYNee-TSM-RRS
summarizsss the results on Diving-48 & FineGym. For Diving-48, TSM-R50 using frames shows 38.8% in top-1 accuracy in our implementation. SHLFYNet-TSMMRR5 outperforms TSM- R50 by 2.8%pp in accuracy so that it sets a new state-of-the-ard top-1 accuracy as 41.6% on Diving- achieves 49.5% and 87.7% at given 288 and 99 classes,
48. For FineGym, SELFYNet-TSM- respectively, surpassing all the other models reported in Shao et al. (2020).
4.3 ABLATION STUDIES
We conduct ablation experiments to demonstrate the effectiveness of the proposed method. All experiments are performed on SS-V1 by using frames. Unless otherwise specified, we Ima- geNet pre-trained TSM 18 (TSM-R18) with the single SELFY block of which (I, V))= (5, 9, 9), as our default Set
Types of similarity. In Table 3a we investigate the effect of different types of similarity by vary- ing the set temporal offset on both TSN-ResNet-18 (TSN-RI8) and TSM-RI8. Interestingly, learning spatial self-similarity ((0}) improves accuracy on both backbones, which implies that self- similarity features help capture structural patterns of visual features. Learning cross-similarity with short temporal range (f1)) shows noticeable gain in accuracy both backbones, indicating the significance of motion features. Learning STSS outperforms other types of similarity, and the accu- racy of SELFYNet increases the temporal range becomes longer. When STSS takes a far-sighted

---

Under review as a conference paper at ICLR 2021
SELFY-s
STce
Table S: Spatio-temporal features v.s. STSS features. Results of different combinations of two blocks ((a) (I) from Fig.
comy
Sef-simiarity
BTCB
BELFY-s
conv
Trarsformation
STCB
are shown.
come
Fealura Extraction
model, TSN-RI8
top-1
top-S
Concat
SEL
baseline
16.2
40.8
1x conv
STCB
(a) STCB
42.4
71.7
+
(b) SELFY-s
46.3
75.1
(b)
(f)
(c) STCB STCB
44.4
73.7
(d) SELFY-s + SELFY-s
46.8
75.9
Figure 4: Basic blocks and their combinations. (a)
(c) SELFY-s STCB (parallel)
46.9
76.5
spatio-temporal convolution block (STCB), (b) SELFY-s block, and (c-f) their combinations.
(i) SELFY-s STCB (sequential)
47.6
76.6
50
Corruction: occlusion
accurary (10)
209 aocuracy (96)
40
40
(E:
with sumnethnng
IE (- Puzting something onto something
[0]
Corrpption motion
F-1.0.33
25
[-1..,3]
20
40
80
120
160
300 224
IE (E) [-
Hitting something with sometaing
Skdw length of the cubour reg (pdl
Serverity of the motion blurs
m,, B): Putting something in front of someehhing
(a) Corruption: occlusion.
(b) Corruption: motion blur.
(c) Qualitative examples of the corruptions.
Figure 5: Results of the robustness experiment. (a) and (b) show top-1 accuracy of SELFYNet variants (Table 3a) with different occlusions and motion blurs, respectively. (c) shows qualitative examples that SELFYNet (f-3.... ..3.).3))) answers correct, while SELFYNet ({1)) fails.
view motion, STSS learns both short-term and long-term interactions in videos, as well as spatial self-similarity. Qualitative results in Appendix show that SELFYNet with a long temporal range (1-3, 3)) effectively captures long-term interactions in videos.
Feature extraction and integration methods. In Table 3b we compare the performance of dif- ferent combinations of feature extraction and integration methods. From the 2nd to the 4th rows, different feature extraction methods are compared fixing the feature integration methods to a single fully-connected (FC) layer. Compared to the baseline, the of soft-argmax, which extracts spa- tial displacement features, improves top-1 accuracy by 1.0%p. Replacing soft-argmax with MLP provides an additional gain of 1.9%p top-1 accuracy, showing the effectiveness of directly using similarity values. When using the convolution method for feature extraction, we achieve 46.7% 7% at top-1 accuracy; the multi-channel convolution kernel is more effective in capturing structural pat- terns along (u, v) dimensions than MLP. From the 4th to the 6th rows, different feature integration methods are compared fixing the feature extraction method to convolution. Replacing the single FC layer with MLP improves the top-1 accuracy by 0.6%p. Replacing MLP with convolutional layers further improves and achieves 48.4% at top-1 accuracy. These results demonstrate that our design choice of using convolutions along (u, v) and (h, w) dimensions is the most effective in learning the geometry-aware STSS representation. For more ablation experiments, please refer to Appendix
4.4 COMPLEMENTARIT BETWEEN OPATIO-TEMPOR AL FEATURES AND STSS FEATURES
We conduct experiments for analyzing different meanings of spatio-temporal features and STSS features. We organize two basic blocks for representing two different features: apatio-tempont con- volution block (STCB) that consists of several spatial-temppral convolutions (Fig- 4ap and SELFY-s block, liggt-weighted version of the SELFY block by removing spatial convolution layers (Fig. 4b). Both blocks have the same receptive fields and a similar number of parameters for a fair compari- son. Different combinations of the basic blocks are inserted after the third stage of TSN-ResNet-18 Table summarizes the results on SS-V1. STSS features (Fig. 4b] 4d are more effective than Peatio-temporal features (Fig. 4a 4ch top-1 and top-5 accuracy when the
number of blocks 4F shows better results
are inserted. Interestingly, the combination two different features (Fig, 4e
in top-1 and top-5 accuracy compared to the single feature cases (Fig. 4c
4d
which demonstrate
8

---

Under review as a conference paper at ICLR 2021
that both features complement each other. We conjecture that this complementarity comes from different characteristics of the two features; while spatio-temporal features are obtained by directly encoding appearance features, STSS features are obtained by suppressing variations in appearance and focusing on the relational features in space and time.
4.5 IMPROVING ROBUSTNESS WITH STSS
In this section, we demonstrate that STSS representation helps video proccssoocssiing models to be more robust to video corruptions. We test two corruptions that are likely to happen in the real world videos: occlusion and motion blur. To induce the corruptions, We either cut out a rectangle patch of particular frame generate motion blur (Hendrycks & Dietterich] 2019). We corrupt sin- gle center-frame for every clip of SS-VI at the testing phase and gradually increase the severity of corruptions. We compare the results of TSM-RI8 and SELFYNet variants of Table 3a Fig. 5a summarizes the results two corruptions, respectively. The top-1 accuracy of TSM-RI 8 and SELF
5b
YNets with the short temporal range (f0), (1), and f-1, O, 1}) significantly drops as the severity of corruptions becomes harder. We conjecture that the features of the corrupted frame propagate through the stacked TSMs, confusing the entire network. However, the SELFYNets with the long temporal range (f 2, ..... 2] and {-3.... ... .3) show more robust performance than the other mod- els. As shown in Fig. 5a 5b the accuracy gap between SELFYNets with the long temporal range and the others increases as the severity of corruptions becomes higher, indicating that the larger size of STSS features can improve the robustness on action recognition. We also present some qualitative results (Fig. (5c) where two SELFYNets with different temporal ranges, {1} and {-3, ... .3, both answer correctly without corruption, while the SELFYNet with {1} fails for the corrupted input.
CONCLUSION
In this paper, we have proposed to learn a generalized, far-sighted motion representation from STSS for video understanding. The comprehensive analyses on the STSS demonstrate that STSS fea- tures effectively capture both short-term and long-term interactions, complement spatio-temporal features, and improve the robustness of vido-processsig models. Our method outperforms other state-of-the-ar methods on the three benchmarks for video action recognition.
REFERENCES
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Olivier Chapelle and Mingrui Wu. Gradient descent optimization of smoothed information retrieval metrics. Information retrieval, 13(3):216-25 2010.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Mareus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-tern recurrent convolutional networks for visual recognition and description. In Proc. IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2015.
Alexey Dosovitskiy, Philipp Fischer, Eddy llg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proc. IEEE International Conference Computer Vision (ICCV), 2015.
Lijie Fan, Wenbing Huang, Chuang Gan, Stefano Ermon, Boqing Gong, and Junzhou Huang. End- to-end learning of motion representation for video understanding. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018
Linxi Fan, Shyamal Buch, Guanzhi Wang, Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei- Fei. Rubiksnet: Learnable 3d-shift for efficient video action recognition. In Proc. European Conference on Computer Vision (ECCV), 2020.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.

---

Under review as a conference paper at ICLR 2021
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 170iv:7000.0000 1w..110067.77777 02677, 2017a.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proc. IEEE International Conference on Computer Vision (ICCV), 2017b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proc. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 122611, 2019.
Eva Hörster and Rainer Lienhart. Deep networks for image retrieval on large-scale databases. In Proceedings of the 16th ACM international conference on Multimedia, pp. 643-646, 2008.
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3464-3473, 2019.
Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proc. IEEE International Conference Computer Vision (ICCV), 2019.
Imran N Junejo, Emilie Dexter, Ivan Laptev, and Patrick Púrez. Cross-view action recognition from temporal seff-similarities. In Proc. European Conference Computer Vision (ECCV), 2008
Imran N Junejo, Emilie Dexter, Ivan Laptev, and Patrick Perez. View-independent action recognition from temporal similaritiss IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2010.
Gagan Kanojia, Sudhakar Kumawat, and Shanmuganathan Raman. Attentive spatio-temporal rep- resentation learning for diving classification. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019.
Andrej Karpathy, George Toderici, Sanketh Shetty. Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. Proc. IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2014.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et The kinetics human action video dataset. arXiv preprint 1705v
2017.
Seungryong Kim, Dongbo Min, Bumsub Ham, Seungchul Ryu, Minh N Do, and Kwanghoon Sohn. Dasc: Dense adaptive self-correlation descriptor for multi-modal and multi-spectral correspon- dence. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn. Fess: Fully convolutional self similariyy for dense semantic correspondence. In Proc. IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), 2017.
Marco Körner and Joachim Denzler. Temporal self-similarity for appearance-based action recog- nition multi-view setups. In International Conference on Computer Analysis of Images and Patterns, pp. 163-171. Springer, 2013.
Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsqueeze: Neural motion feature learning for video understanding. arXiv preprint XXix:2007.00933 2020.
Myunggi Lee, Seungeui Lee, Sungjoon Son, Gyutae Park, and Nojun Kwak. Motion feature net- work: Fixed motion filter for action recognition. In Proc. European Conference on Computer Vision (ECCV), 2018.
10

---

Under review as a conference paper at ICLR 2021
Jun Li, Xianglong Liu, Mingyuan Zhang, and Deqing Wang. Spatio-temporal deformable 3d con- vnets with attention for action recognition. Pattern Recognition, 98:107037, 2020a.
Xianhang Li, Yali Wang, Zhipeng Zhou, and Yu Qiao. Smallbignet: Integrating core and contex- tual views for video classification. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020b.
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020c.
Yingwei Li, Li, and Nuno Vasconcelos. Resound: Towards action recognition without represen- tation bias. In Proc. European Conference Computer Vision (ECCV), 2018.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
Xingyu Liu, Joon-Young Young Lee, and Hailin Jin. Learning video representations from correspondence proposals. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03988, 2016.
Chenxu Luo and Alan L Yuille. Grouped spatial-temporal aggregation for efficient action recogni- tion. In Proc. IEEE International Conference Computer Vision (ICCV), 2019.
Joanna Materzynska, Tete Xiao, Roci Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else Compositional action recognition with spatial-temporal interaction networks. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
AJ Piergiovanni and Michael S Ryoo. Representation flow for action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: 1906.059990 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proc. IEEE International Conference Computer Vision (ICCV), 2017.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine- grained action understanding. In Proc. IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2020.
Eli Shechtman and Michal Irani. Matching local self-similarities across images and videos. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Proc. Neural Information Processing Systems (NeurIPS), 2014.
Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift networks for video action recognition. Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018a.
Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018b
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proc. IEEE International Conference on Computer Vision (ICCV), 2015.
11

---

Under review as a conference paper at ICLR 2021
Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proc. IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2018.
Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli. Video modeling with correlation net- works. In Proc. IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2020.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Proc. Euro- pean Conference Computer Vision (ECCV), 2016.
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. Proc. European Con- ference on Computer Vision (ECCV), pp. 399-417, 2018.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. In Proc. Neural Information Processing Systems (NeurlPS). 2019.
Christopher Zach, Thomas Pock, and Horst Bischof, A duality based approach for realtime tv-1 optical flow. Pattern Recognition, pp. 214-223, 2007.
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10076-10085, 2020.
Yue Zhao, Yuanjun Xiong, and Dahua Lin. Trajectory convolution for action recognition. In Proc. Neural Information Processing Systems (NeurIPS), 2018.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. Proc. European Conference on Computer Vision (ECCV), 2018.
Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Efficient convolutional net- work for online video understanding. In Proc. European Conference on Computer Vision (ECCV), 2018.
12

---

Under review as a conference paper at ICLR 2021
Table 6: TSN-ResNet & TeM-ResNet backbone.
Layers
18N-ResNet-18
Tes-ResNet-18 18
TSN-ResNet-S0
TeM-ResNer-50
Output size
conv;
1x7x7. 64, stride 1,2,2
1x112x112
pool,
1x3x3 max pool, stride 1,2,2
Tx56x56
TSM
1x3x3, 64
TSM
x I, 256
1x1x1. 256
res2
1x3x3, 64
x2
1x3x3, 64
x2
1x3x3, 256
x3
1x3x3, 256
x3
Tx56x56
1x3x3, 64
1x
1x1x1, 256 TSM
1x3x3, 128) 1x3x3, 128
TSM 1x3x3, 128
1111x1,512
1x1x1, 512
resa
1x3x3,
x4
1x3x3, 512
Tx28x28
1x3x3, 128.
1x1x1,
TSM
TSM
x 10241
1x3x3, 256 1x3x3, 256
Ix1xl, 1024
reS;
x2
1x3x3, 256 x3333, 256
x2
1x3x3, 1024 x6
1x3x3, 1024
x6
Tx14x14
Ix1xl, 1024
1x1x1, 1024
TSM
1x3x3, 512] 1x3x3, 512
TSM 1x3x3, 512 1x3x3, 512
x 20481
1x1x1, 2048
ress
Ix3x3, 2048 xx1x1,
1x3x3, 2048
Tx7x7
1x1x1, 2048
global average pool, FC
#o classes
A IMPLEMENTATION DETAILS
Architecture details. We use TSN-ResNet and TSM-ResNet as our backhone (see Table 6 and initialize them with ImageNet pre-trained weights. We insert a single SELFY block into right after res3 and use the convolution method as a default feature extraction method. We set the spatio- temporal matching region of SELFY block, (L, U, V), as (5, 9, 9) or (9,9, 9, 9) when using or 16 input frames, respectively. We stack four 1 x 3 x 3 convolution layers along (1, u, v) dimension for the feature extraction method, and use four 3 x convolution layer along (æ. y) dimension for the feature enhancement. We reduce a spatial resolution of video feature tensor, V, as 14x 14 for computation efficiency before the self-similarity transformation. After the feature enhancement, we upsample the enhanced feature tensor, G as 28 x8 for the residual connection.
Training. We sample a clip of 8 or 16 frames from each video by using segment-bas sam- pling (Wang et al., 2016). We resize the sampled clips into 240 320 images and apply random scaling and horizontal flipping for data augmentation. When applying the horizontal flipping on SS- VS-Vi&V2 (Goyal al.] 2017b), we do not flip clips of which class labels include 'left' or 'right' words; the action labels, egg. 'pushing something from left to right.' We fit the augmented clips into spatial resolution of 224 224. For SS-VI& & V2, we the initial learning
to 0.01 and the training epochs to 50; the learning rate is decayed by 1/10 after 3hth and 40th epochs. For Diving-48 (Li al. 2018) and FineGym (Shao al. 2020). we use a cosine learning rate sched- ule (Loshchilov & Hutter 2016) with the first 10 epochs for gradual warm-up (Goyal et al. (017a). We set the initial learning rate to 0.01 and the training epochs 30 and 40, respectively.
Testing, Given video, we sample or 2 clips, resize them into 240 x 320 images, and crop their centers as 224 x 224. We evaluate an average prediction of the sampled clips. We report top-1 and top-5 accuracy for SS-V1& V2 and Diving-48, and mean-class accuracy for FineGym.
Frame corruption details. We adopt two corruptions, occlusion and motion blur, to test the robust- ness of SELFYNet. We only corrupt single center-frame for every validation clip of SS-V1; we corrupt the 4th frame amongst 8 input frames. For the occlusion, we cut out a rectangle region from the center of the frame. For the motion blur, we adopt ImageNet-C implementation, which is avail- able online" We set 6 levels severity for each corruption. We set the side length of the occluded region 40px, 80px, 120px, 160px, 200px and 224px from the level 1 to 6. For the motion blur, we set (radius, sigma) tuple arguments as (15, 5), (10, 8), (15, 12), (20, 15), (25, 20), and (30, 25), respectively.
13

---

Under review as a conference paper at ICLR 2021
Table 7: Additional ablations on SS-V1. Top-1 accuracy (%) are shown.
(a) Performance comparison with non-local meth- ods. NL and CP denote non-local block and CP module, respectively.
(b) Performance comparison with MSNet. KS and
CM denote the kernel soft-argmax and confidence map, respectively.
model
FLOPs
top-1
top-S
model
extraction
(L,U.
top-1
top-5
TSM-RI8
14.6G
43.0
72.3
TSM-R18
43.0
72.3
ISM-RIS + NL
24.8G
43.8
73.1
KS CM
(1,9.9)
46.1
75.3
KS CM
(6,9.9)
47.4
76.8
TSM-RI8 CP
25.6G
46.7
75.7
SELFYNet
Conv
(1.9.9)
47.1
76.3
SELFYNet
17.3G
48.4
77.6
Conv
(6,9.9)
48.4
77.6
(c) Multi-channel x3 kernel for feature extrac- tion. Four convolution layers used for extracting } denotes of temporal offsets l.
(d) Spatial matching region. Performance compari- with different spatial matching- regions,, (U V).
STSS features.
model
UxV
FL.OPs
top-1
top-5
model
range of I
top-1
top-5
TSM-RIS
14.6
43.0
72.3
TSM-R18
43.0
72.3
5
17.1
47.8
77.1
1
47.4
77.0
SELFYNet
x
17.3
48.2
77.5
SELFYNet
2, ... 21
48.3
77.2
13
x 13
18.4
48.4
77.8
3 .3}
48.5
77.4
17 x
17
19.8
48.6
78.3
(e) STSS with visual features V. R denotes the
(I) Position. Performance comparison with different positions of SELFY black. For the last row, 3 SELFY blacks are used in total.
STSS features, Le. ReLUU( Xs Wo) in Eq.
model
Z
top-1
top-5
TSM-RI8
V
43.0
72.3
model
position
top-1
top-5
R
45.5
75.9
TSM-R18
43.0
72.3
SELFYNet
R V
48.4
77.6
pool
45.7
77.6
reS2
47.2
76.6
SELFYNet
res3
48.2
77.5
46.6
76.0
reSs
42.8
72.6
re52,3,4
48.6
77.9
B ADDITIONAL EXPERIMENTS
We conduct additional ablation experiments identify the behaviors of the proposed method. All experiments are performed on SS-VI by using 8 frames. Unless otherwise specified, we set Ima- geNet pre-trained TSM ResNet-18 (TSM-R18) with single SELFY block of which (L,U.V) (5, 9, 9), as our default SELFYNet.
Comparison with non-local methods. We compare our method with popular non-local meth- ods (Wang et al.) 2018 Liu et al.) (019). which capture the long-range dynamics of videos. While computing the self-similarity values as ours, both methods use them as attention weights for feature aggregation by multiplying them to the visual features (Wang et al. 2018) aligning top-K K corre- sponding features (Liu et al., 2019); they both do not use STSS itself as a relational representation. In contrast, method does indeed and learns more powerful relational feature from STSS. The difference between our method and non-local methods are illustrated in Fig.
We have conducted experiments for performance comparison, and the results shown in Table 7a We re-implement the non-local block and the CP module in Pytorch based on their official codes For fair comparison, 18). Note that our method downsamples a spatial resolution of V to 14 x 14 before STSS transfor- mation, whereas the others do not. Compared to the non-local block and the CP module, SELFY block improves top-1 accuracy by 4,4%p and 1.5%p, while computing less floating opera- tions as 7.5 GFLOPs and 8.3 GFLOPs, respectively. demonstrates that the direct integration of
insert single block module at the same position (after ress of ResNet-
net
14

---

Under review as a conference paper at ICLR 2021
STSS features is more effective for action recognition than the indirect ways of using STSS, e.g., re-weighting vesual-semantic features or learning correspondences.
Comparison with correlation-based methods. We also compare method with correlation- based methods (Kwon et al. 2020 Wang ct al. 2020). While correlation- methods extract motion features between two adjacent frames only and thus limited to short-term motion, method effectively captures bi-directional and long-term motion information via learning with the sufficient volume of STSS. Our method can also exploit richer information from the self-similarity values than other methods. MS module (Kwon et al. 2020) only focuses on the maximal similarity value of the (u, v) dimensions to extract flow information, and Correlation block (Wang al. 2020) uses an 1 x convolution layer for extracting motion features from the similarity values. In contrast to the two methods, we introduce a generalized motion learning framework using the self-similarrty tensor at Sec 3.2 of our main paper. The difference between our method and correlation-base. methods are illustrated in Fig. 6
We also have conducted experiments to compare method with MSNet (Kwon et al. 2020). one of the correlation-base based methods. For an apple-to-apple comparison, we apply kernel soft-argmax and max pooling operation (KS CM in Kwon et al. (2200) to our feature extraction method by following their official codes
4 Please note that, when we restrict the temporal offset I to {1}, the SELFY block using KS + CMis equivalent to the MS module of which feature transformation layers are the standard 2D convolutional layers. Table Table(7]jsummarizes summarizes the results. KS+CM method achieves 46.1% 1% at top-1 accuracy. As enlarge the temporal window L to 5, obtain additional gain as 1. 1.3%p. The learnable convolution layers improve top-1 accuracy by 1.0%p in both cases. The results demonstrates the effectiveness of learning geometric patterns within the sufficient volume of STSS tensors for learning abundant motion features.
Multi-channel x x kernel for feature extraction. We investigate the effect of the convolution method for STSS feature extraction, when we use multi-channel 3 x 3 x 3 kernels. For the experi- ment, we stack four x 3 convolution layers followed by the feature integration step, which are the same as in Section 3.2.2] Table 7 summarizes the results. Note that we do not report models of which temporal window
1, e.g., {0} and {1). As shown in the table, indeed, the long tem- poral range gives the higher accuracy. However, the effect of the 3 x 3 x 3 kernel is comparable to that of the 1 x x 3 kernel in Table Balin in terms of accuracy. Considering the accuracy-computation computation trade-off, we choose to fix the kernel size, L, x UR Vx, 1 x for the STSS feature extraction.
Spatial matching region. In Table 7d we compare a single SELFY block with different spatial matching regions, (U, V). As result, indeed, the larger spatial matching region leads the better trade-off, we set our spatial matching region, (U, V),
curacy. Considering the as (9,9) 9) default.
Fusing STSS with visual features. We evaluate SELFYNet purely based on STSS features to see how much the ordinary visual feature V contributes for the final prediction. That is, we pass the
STSS features, ReLU(F X5 Wa), into the downstream layers without the visual features V (Eq. p in our main paper). For the simplicity of description, we denote the relational feature ReLU(F* X5 We) by R. Table 7c compares the results of using different cases of the output tensor Z (Z = V.
Z= R. and Z = R+ V) on SS-VI. Interestingly, SELFYNet using only R achieves 45.5% at top-1 accuracy, which is higher as 2.5%p than the baseline. As we add V to R, we obtain an additional gain of 2.9%p. It indicates that the STSS features and the visual features are complementary each other.
Block position. From the 2nd to the 6th row of Table 7f we identify the effect of different positions of SELFY block in the backbone. We resize the spatial resolution of the video tensor, (X, Y), into 14, and fix the matching region, (T., U, V). as (5, 9, 9) for all the cases maintaining the similar computational cost. SELFY after the res3 shows the best trade-off by achieving the highest accuracy among the cases. The last row in Table 7 shows that the multiple SELFY blocks improve accuracy compared to the single block.
onnqueeze
15

---

Under review as a conference paper at ICLR 2021
Table 8:
Performance comparison with the local self attention mechanisms. R.P.E. is an abbreviation for relative positional embeddings. Top-1, 5 accuracy (%)
are shown.
model
similarity
extraction
top-1
top-5
TSM-R18
43.0
72.3
embedded Gaussian embedded Gaussian
multiplication w/ V R.P.E.
43.8
72.3
SELFYNet
Conv
47.6
76.8
cosine
Conv
47.8
77.1
C
THE RELATIONSHIP WITH THE LOCAL SELF-ATTENTION MECHANISMS
The local self-attention Hu et al. 2019 Ramachandran et al. 2019 Zhao et al. 2020) and our method have common denominator using the self-similarity tensor but use way and purpose. The local self-attention mechanism aims to aggregate the local context features using the self-simiarrity tensor and thus uses the self-similarity values as attention weights for feature aggregation. However, the local STSS, so the final STSS representation is directly fed into the neural network instead of multiplying it to local context features.
very different
method aims to learn generalized motion representation from
For an empirical comparison, we conduct an ablation experiment as follows. We extend the local self attention layer (Ramachandran al., 2019) temporal dimension, and then add the spatio-temporal local self-attention layer, which is followed by feature integration layers, after res3. All experimen- tal details are the same as those in Appendix A, except that reduce the channel dimension C' of appearance feature V to 32. Table summarizes the results on SS-VI. The spatio-temporal local self-attention layer is accurate embedded Gaussian and the cosine similarity outperform the local self-attention by achieving top-1 accuracy as 47.6% and 47.8%, respectively. These results are alignment with the prior work (L.iu et al., 2019), which reveals that the self-attention mechanism hardly captures motion features in video.
43.8% at top-1 accuracy, and both of SELFY blocks using the
D VISUALIZATIONS
Fig. 7 we visualize some qualitative results of two different SELFYNe:TSMM-RI ((1} and { (-3.... :3) SS-VI. We show the different predictions of the two models with 8 input frames. We also overlay Grad-CAMs (Selvaraju et al. 2017) on the input frames to see whether a larger volume of STSS benefits to capture long- term interactions in videos. We take Grad-CAMe of fea- tures which is right before global average pooling layer. As shown the figure, the STSS with the sufficient volume helps to learn for example, SELFYNet with the range of (f (33 ... 33) focuses on not only regions on which an action occurs but also focuses on the white-stain after the action verify whether the stain wiped off or not.
enriched context of temporal dynamics in the video; in Fig. 7a
16

---

Under review as a conference paper at ICLR 2021
(TXY,C)
(T.X.Y.6)
(T.X.V.YX))
(T,X,Y,C')
(T,X..,C)
embed
embed
Similarty
Tensor multiplication
F
FC
embed]
computation
(embedded Gaussian)
(a) Non-local block Wang al.
2018
(T.XYV,C)
(T.X,Y.IXX)
(T,X.Y,R. 3)
(T.X.Y, C)
100c M0RP
Similarity computation
Arg top-k operation
F
z
V
(negative L2 dista noa)
MLP
(along
(b) CP module
(Liu et al.
2019
(T.X.F.UM) UV)
(T.X.
(T.X.Y
CT
(eceine similerityi
K-aoft-argmax
Correlation
LKLXL
EXEX
EXEX
computation
S
Max
Sep. CONV
Son.
(c) MS module (Kwon et al. 2020
(T.XYY,C)
(T,XYY UV)
(T.X.V
(T,XYY.))
(sceled del product)
Correlation
computation
s
FC
F
Z
K
(ooarablla kamme)
C.V.
(d) Correlation block (Wang et al. 2020
(T.XY.C)
(...
(T.YY.L.C.)
(T.X.Y.C)
(cneine aimil anthy)
Self-simila rity Transform ation
Feature Extraction
Feature Integration
F
Z
a (.
Corvoldiinn
(43)) EUOE waneg 3
PAACO 1: 1
(....V)
(L.G)
(c) SELFY block (ours)
Figure 6: Comparison with non-local approaches (Wagg al., 2018; Liu al., 2019) and correlation-based methods (Kwon et al. 2020 Wang et al. 2020). From the top, non-local block, CP module, MS module, Correlation block and SELFY block are illustrated.
17

---

Under review as a conference paper at ICLR 2021
(X) /E (
Wiping something something
(7) [-(-,,... 3}: Pretending or failing to wipe something off of something
(X) [E!
Turning something upside down
(7) IE {-3, 3} Pretending to turn something upside down
(b)
(x) IE [1)
Throwing something in the air and letting fall
(V) /E (-3.... ..., 3): Throwing something In the catching
(x) /E {1}:
Unfolding something
(<) /E(-3,.... .... 3}: Tearing something into two pieces
(d)
Figure 7: Qualitative results of two SELFYNets SS-V1. Each subfigure visualizes prediction
results of the two models with Grad Grad-CAM-overlaii RGB frames. The correct and wrong predictions colorized as green and red, respectively.
18


Reviews:
[{"review_id": "xkIEMtxjN0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### General\nThis paper proposes spatio-temporal self-similarity (STSS), which captures structural patterns in space and time, for action recognition from videos.  \nOverall, I would like to recommend ICLR to accept the paper.  \nPros and Cons I found in the paper are summarized as follows.\n\n\n#### Pros.\n1. SELFY, the proposed neural block that implements STSS, is a good extention and adjustment of prior self-similarity works to modern CNNs, resulted in achieving state-of-the-art performance on different datasets.\n1. The ablation study reveals that SELFY with a long temporal range successfully incorporate the temporal information without explicitly exploiting optical flows. \n1. The paper is generally well and clearly written.\n1. The literature review is thorough, and the work is well contextualized in the literatures.\n\n#### Cons\n1. The novelty and the originality of the paper may be relatively low because the concept of the STSS itself was already proposed in prior studies as mentioned in the paper.\n1. The reasoning or intuition behind some of the design choices are not always clear. For example,  \n    - The output of the feature extraction block is $F \\in \\mathbb{R}^{T \\times X \\times Y \\times L \\times C_F}$, and the authors say \n        > The dimension of L is preserved to extract motion information across different temporal offsets in a consistent manner.\n\n        But the motion information across different temporal offsets can be extracted in the feature extraction module if MLP or convolution is employed in the feature extraction block. In other words, even if the output is $F \\in \\mathbb{R}^{T \\times X \\times Y \\times C_F}$, the motion information can be encoded in the output tensor. Possibly the authors empirically found the present design choice is better. If so, how does the performance change if $L$ is not preserved?\n    - In the feature integration block, firstly $3 \\times 3$ spatial convolution kernel is applied, then the temporal offset ($L$) and the channels ($C^*_F$) dimension is flattened, and finally $1 \\times 1 \\times 1$ spatio-temporal convolution is applied. Is this design chosen experimentally or is there any intuition behind?   \n\n#### Minor comments\n1. It may be interesting to discuss the relationship with self-attention based networks such as [1-3]\n1. About Table 7 in Appendix, how is the result if SELFY block is used in $res_1$ and $res_5$? I guess these option do not work well, but I think it is beneficial to list all the results.\n\n[1] Hu+, Local Relation Networks for Image Recognition, ICCV 2019  \n[2] Ramachandran+, Stand-Alone Self-Attention in Vision Models, NeurIPS 2019  \n[3] Zhao+, Exploring Self-attention for Image Recognition, CVPR 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "WmuKjlP7SKL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary: \n\nThis submission proposed a motion representation method based on spatio-temporal self-similarity (STSS), which represents each local region as similarities to its neighbors in both spatial and temporal dimension. There are previous works (e.g., Ref[1] , [2], [5] listed here) which utilize STSS for feature extractions, authors claim that this work is the first one to learn STSS representation based on modern CNN architecture. The proposed method is implemented as a neural block, i.e., SELFY, which can be applied into neural architectures and learned end-to-end without additional supervision. On 3 standard human action recognition data sets, Something-Something-V1 & V2, Diving-48, and FineGym, the proposed method achieves quite good empirical results. \n\n### Strengths & Originality: \n\nThe idea of utilizing patio-temporal self-similarity (STSS) for feature representation in the modern CNN framework for human action recognition is interesting. I also like the concept that \"fix our attention to the similarity map to the very next frame within STSS and attempt to extract a single displacement vector to the most likely position at the frame, the problem reduces to optical flow ...... In contrast, we leverage the whole volume of STSS ......\", as to view the proposed method as a generalized & rich optical flow. \n\nFurthermore, STSS should be helpful for view invariant action recognition, i.e., one of the fundamental challenges from video data recognition. \n\nEmpirical results on Something-Something V1 & V2, Diving-48, FineGym show that the proposed achieves the state-of-the-art results, though seems marginally.  For example, the proposed Ensemble model (SELFYNet-TSM-R50EN ) achieved 56.6% and 67.7% attop-1 accuracy for V1 & V2, vs. the MSNet-TSM-R50EN got quite similar performance as 55.1% and 67.1% (Kwon et al., 2020, Ref. [3]). \n\nOn FineGym, SELFYNet-TSM-R50 achieves 49.5% and 87.7%, these do look better & more clearly, compared with 46.5%, 81.2% reported from (Shao et al, 2020, Ref. [4]).\n\n### Weakness: \n\nFrom Figure. 2 or Section 3.2.2, the final STSS representation Z is the combination of original input V and the STSS feature F(S(V)), so kind of unclear how much contribution is made from the original V in terms of the final recognition performance. Maybe this part is addressed in the experimental section somewhere. \n\nThe proposed neural block, SELFY, includes 3 parts as self-similarity transformation, feature extraction, and feature integration, it seems none of them is original but the combination of these put into the end-of-end NN structure is new as claimed by authors. There are some additional experiment results presented in section 4.3 (for different types of similarity, and for different feature extraction), still, it seems unclear how important or sensitive for each step within the whole framework based on TSN ResNets and TSM ResNets, given the complexity here. \n\nSelf-similarity can be applied for both image and videos (i.e., Section 3 or Ref [5], etc), I know the focus of this submission is video action recognition, still, seems interesting to know the proposed framework \"3 steps combined + CNN end-of-end\" apply to image object recognition. If that also achieved good results compared with state-of-the-art for object recognition, it will be a strong support for the proposed methodology, and if not, which is also good to know & could be multiple reasons behind. \n\nBesides these video based action benchmarks, it should be interesting to see results on a depth enriched data set (i.e., RGB-D), as missing depth information is one of the limitations from video data. Ideally, we should see similar good performance if the proposed methodology is effective for representation learning.\n\n### Reference:\n\n1. Videos as Space-Time Region Graphs, Xiaolong Wang, Abhinav Gupta, Proc. European Conference on Computer Vision (ECCV) 2018\n\n2. TSM: Temporal Shift Module for Efficient Video Understanding, Ji Lin, Chuang Gan, and Song Han,  Proc. IEEE International Conference on Computer Vision (ICCV), 2019\n\n3. MotionSqueeze: Neural Motion Feature Learning for Video Understanding, Heeseung Kwon, Manjin Kim, Suha Kwak, Minsu Cho, Proc. European Conference on Computer Vision (ECCV) , 2020\n\n4. Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for finegrained action understanding. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020 \n\n5. Imran N. Junejo, Emilie Dexter, Ivan Laptev and Patrick Perez, Cross-View Action Recognition from TemporalSelf-Similarities, Proc. European Conference on Computer Vision (ECCV) 2008", "extra_scores": {}, "extra_text": {}}, {"review_id": "UbNjlFSBh3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel nerual block based on the classifical spatio-temporal self-similarity (STSS), named SELFY, which can be easily inserted into nerual architectures and learned end-to-end without additional supervision. SELFY could capture long-term interaction and fast motion in the video with a sufficient volume  of the neighborhood in time and space. \nThe nice point of the method is that it is heavily investigtated through experiments. Evaluated on three standard action recognition benchmark datasets,  the proposed SELFY is demonstrated the superiority over previous methods for motion modeling as well as the complementarity to spatio-temporal features from direct convolution. Moreover, the paper is clear and seems correct ,technically.\n\ncomments:\n\n-The first concern is about  the limited novelty of the method. Despite revisting the self-similarity, from section 3.1, the learning of generalized and long-term information is a property of the STSS rather than a contribution of this work. If necessary, authors should give more details on the classifical STSS to show the improvement. \n-The authors briefly mention the differences between the new proposed and non-local approaches (Wang et al., 2018; Liu et al., 2019) and correlation-based methods (Wang et al., 2020; Kwon et al., 2020), which appears inadequate and is better to integrate them into a comparison in the form of Figure 1.\n-How are parameters (5, 9, 9) and (9, 9 ,9) chosen? The best result occurs as the  temporal offsets being chosen from {-3, ..., 3}. Will there be better accuracy when the range is larger? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "UCrbsWsJABL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces SELFY, a neural module that learns spatio-temporal self-similarity across longer timescales in both directions to obtain visual features that provide consistent empirical gains on three action recognition datasets. Ablation studies show that modeling longer, bi-directional motion similarity can help handle motion blurs and (artificially induced) occlusion.\n\nStrengths:\n+ The paper is well written and shows strong empirical gains over prior SOTA.\n+ The ablation experiments on the effect of temporal length and the different feature extraction methods are nicely done and show the effect of different design choices.\n+ The experiments for testing the robustness of the learned representations is nice to see and helps highlight the strength of the proposed approach.\n\nConcerns:\n- From what I can see, the contribution is in the modeling of (motion) similarity in both directions, across a longer time scale. My concern is that all of the chosen datasets are very motion-centric where temporal relationships are more important and this makes it somewhat advantageous to the proposed model and does not really allow us to ascertain its applicability to datasets where appearance plays a bigger role such as Kinetics or HMDB-51. In fact, the results on SS-V1 and SS-V2 show some gains (~2-3%) compared to the very closely related MotionSqueeze Network (Kwon et al, ECCV 2020) which computes correlation (rather than similarity) between adjacent frames (t and t+1). MSNet did not see any significant advances in datasets (HMDB-51 and Kinetics) where appearance plays a greater role and I am not sure how this proposed approach would help in those conditions.\n- I assume the function sim(.) from Equation 1 is cosine similarity since it was not explicitly mentioned anywhere. Is there any specific reason the correlation function was not used? What is the effect of using correlation in place of cosine similarity? It would interesting to see the effect of this since it would essentially allow us to see how modeling the longer temporal context would affect MS Net (the most closely related network) and help highlight the contribution of the proposed model beyond the obvious empirical gains.\n- While the robustness experiments test out the occlusion setting by cutting out a rectangular patch of a single center-frame, that does not, IMHO, shows robustness to occlusion since the features from surrounding frames (on both sides) will help mitigate this artificial occlusion. In practice, the occlusion can have a larger effect since it would not zero out the appearance but will add some noise into the feature extraction and hence provide a stronger challenge. Experiments on HMDB51, which has strong camera motion and hence introduces natural occlusions, would be a stronger experiment to show the generalization of the model beyond datasets with a strong motion-centric bias.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to use spatio-temporal self-similarity (STSS) as a feature for a convolutional neural network for video understanding. The proposed approach extracts STSS as a descriptor capturing similarities between local spatio-temporal regions, and adds conventional layers such as soft-argmax, fully connected layers, and conv. layers on top of it.

On one hand, all of the reviewers agree that the novelty of the paper is limited. On the other hand, most of the reviewers (except R1) appreciated thoroughness of the experiments and ablations. In the end, the reviewers gave 3 marginally above the acceptance threshold ratings and 1 marginally below the threshold rating.

The AC views this paper as a borderline paper. None of the reviewers are excited about the paper, and it is a typical "Nice experiments with limited novelty" (by R1) paper. The concept of the STSS itself was already proposed in prior studies as mentioned in the paper and by the reviewers, and this paper 'engineers' a new way to take advantage of STSS without further theoratical or conceptual justifications on why it should work. The newly added Kinetics and HMDB results in the rebuttal are nice, but the impact of STSS seems to be minimal in these results.

Overall, the AC find the paper slighly lacking to be considered for ICLR.
Decision: Reject