TAM: Temporal Adaptive Module for Video Recognition
Authors: Zhaoyang Liu
Limin Wang
Wayne Wu
Chen Qian
Tong Lu
Venue:  (2021)
Keywords: Action Recognition
Temporal Adaptive Module
Temporal Adaptive Network

Abstract: Temporal modeling is crucial for capturing spatiotemporal structure in videos for action recognition. Video data is with extremely complex dynamics along its temporal dimension due to various factors such as camera motion, speed variation, and different activities. To effectively capture this diverse motion pattern, this paper presents a new temporal adaptive module ({\bf TAM}) to generate video-specific kernels based on its own feature maps. TAM proposes a unique two-level adaptive modeling scheme by decoupling dynamic kernels into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short term information, while the aggregation weight is generated from a global view with a focus on long-term structure. TAM is a principled module and could be integrated into 2D CNNs to yield a powerful video architecture (TANet) with a very small extra computational cost. The extensive experiments on Kinetics-400 and Something-Something datasets, demonstrate that the TAM outperforms other temporal modeling methods consistently owing to its temporal adaptive modeling strategy.

Under review as a conference paper at ICLR 2021
TAM: TEMPORAL ADAPTIVE MODULE FOR VIDEO RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
Video data is with complex temporal dynamics due to various factors such as cam- era motion, speed variation, and different activities. To effectively capture this di- verse motion pattern, this paper presents new temporal adaptive module (TAM) to generate video-specific temporal kernels based on own feature maps. TAM proposes unique two-level adaptive modeling scheme by decoupling dynamic kernel into a location sensitive importance map and a location invariant aggrega- tion weight. The importance map learned in local temporal window capture short term information, while the aggregation weight generated from a global view with focus long-term structure. TAM is principled module and could be integrated into 2D CNNs to yield a powerful video architecture (TANett with very small extra computational cost. The extensive experiments Kinetics- 400 and Something Soaehiiin--omtiiiggg datasets demonstrate that the TAM outperforms other temporal modeling methods consistently, and achieves the state-of-the-art performance under the similar complexity.
INTRODUCTION
Deep learning has brought great progress for various recognition tasks in image domain, such as image classification (Krizhevsky et al., 2012: He al. 2016), object detection (Ren et (017). and instance segmentation (He et al. 2017). The key to these successes is to devise flexible and efficient architectures that are capable of learning powerful visual representations from large-scale image datasets (Deng et al.] 2009). However, deep learning research progress in video understanding relatively more slowly, partially due to the high complexity of video data. The core technical problem in video understanding is to design an effective temporal module, that is expected to be able to capture complex temporal structure with high flexibility, while yet to be of low computational consumption for processing high dimensional video data efficiently.
3D Convolutional Neural Networks (3D CNNs) (Ji et al.) 2010 Tran et al. 2015) have turned out to be mainstream architectures for video modeling (Carreira & Zisserman 2017 Feichtenhofer al. 2019 Tran et al. 2018)] Qiu et al. 2017). The 3D convolution
natural extension over its 2D counterparts and provides learnable operator for video recognition. However, this simple extension lacks specific consideration about the temporal properties in video data and might as well lead to high computational cost. Therefore, recent methods aim improve 3D CNNs from two different aspects by combining a lightweight temporal module with 2D CNNs to improve efficiency (c.g., TSN (Wang al. 2016), TSM (Lin et al., 2019)), or designing dedicated temporal module to better capture temporal relation (e.g.. Nonlocal Net (Wang et al., 2018b), ARTNet (Wang et al., 2018a). STM (Jiang al., 2019). However, how devise temporal module with high efficiency and strong flexibility still remains to be unsolved problem in video recognition. Consequently, we aim at advancing the current video architectures along this direction.
In this paper, we focus on devising principled adaptive module to capture temporal information in a more flexible way. In general, we observe that video data is with extremely complex dynam- ics along the temporal dimension due to factors such as camera motion and various speed. Thus 3D convolutions (temporal convolutions) might lack enough representation power to describe mo- tion diversity by simply employing a fixed number of video invariant kernels. To deal with such complex temporal variations in videos, we argue that adaptive temporal kernels for each video effective and as well necessary describe motion patterns. To this end, shown in Figure T

---

Under review as a conference paper at ICLR 2021
Temporal Convolution
Temporal Attention
Temporal Adaptive Module
H*
H*
H #
5
H
I ddaen
Feature
Feature
Dynemic
Attention weights
Feature
weights
2 dapia
fihter
H*W
H"W
HWW
Video invariant
Location adaptive
Video adaptive (g) Location adaptive (C)
Figure Temporal module comparison: The standard temporal convolution shares weights among videos and may lack the flexibility to handle video variations due to the diversity of videos. The temporal attention learns position sensitive weights by assigning varied importance for different time without any temporal interaction, and may ignore the long-range temporal dependencies. Our proposed temporal adaptive module (TAM) presents a two-level adaptive scheme by learning the local importance weights for location adaptive enhancement and the global kernel weights for video adaptive aggregation. C denotes attention operation, and & denotes convolution operation.
present two-level adaptive modeling scheme decompose this video specific temporal kernel into a location sensitive importance map and a location invariant (also video adaptive) aggregation kernel. This unique design allows the location sensitive importance map focus on enhancing dis- criminative temporal information from a local view, and enables the location invariant aggregation weights capture temporal dependencies guided by global view of the input video sequence.
Specifically, the design of temporal adaptive module (TAM) strictly follows two principles: high efficiency and strong flexibility. To ensure our TAM with a low computational cost, we first squeeze the feature map by employing global spatial pooling, and then establish our TAM in channel- wise manner to keep the efficiency. Our TAM composed two branches: local branch (C) and global branch (G). As shown in Fig. 2 TAM is implemented in efficient way. The local branch employs temporal convolutions to produce the location sensitive importance maps to discriminate the local feature, while the global branch uses fully connected layers produce the location invariant kernel for temporal aggregation. The importance map generated by local temporal window focuses attention
short-term motion modeling and the aggregation kernel using global view pays to the long-term temporal information. Furthermore, our TAM could be flexibly plugged into the existing 2D CNNs yield an efficient video recognition architecture, termed as TANet.
We validate the proposed TANet on the task action classification in video recognition. Partic- ularly, we first study the performance of the TANet on the Kinetics-400 dataset. We demonstrate that our TAM is better at capturing temporal information than other several counterparts, such as temporal pooling, temporal convolution, TSM (Lin al. (019). and Non-local block (Wang al, 2018b). Our TANet is able to yield very competitive accuracy with the FLOPs similar to 2D CNNs. further test our TANet on the motion dominated dataset of Something-Somethinn where the state-of-the-ard performance is also achieved.
RELATED WORKS
Video understanding is core topic the field of computer vision. At early stage, a lot of traditional methods (Le et al., 2011 Kläser al. 2008 Sadanand & Corso 2012 Willems et al.) 2008 have designed various hand- craffer features to encode the video data, but these methods are too inflexible when generalized to other video tasks. Recently, since the rapid development of video understand- ing has been much benefited from deep learning methods (Krizhevsky et al. 2012 Simonyan & Zisserman, 2015 He al. 2016), especially in video recognition, series of CNNs-based methods
2

---

Under review as a conference paper at ICLR 2021
were proposed to learn spatiotemporal representation, and the differences with our method will be clarified later. Furthermore, our work also relates to dynamic convolution and attention in CNNs.
CNNs-based Methods for Action Recognition. Since the deep learning method has been wildly used in the image tasks, there are many attempts (Karpathy et al. 2014 SSionyya & Zisserman
2014 Wang al., 2016; Zhou et al.] 2018 He et al. 2019 Lin et al. 2019) based on 21 CNNs devoted to modeling the video clips. In particular, Wang et al. (2016) used the frames sparsely sampled from the whole video learn the long-range information by aggregating scores after the last fully-connected layer. Lin et al. (2019) shifted the channels along the temporal dimension in an efficient way, which yields a good performance with 2D CNNs. By a simple extension from spatial domain to spatiotemporal domain, 3D convolution (Ji et al. 20100; TTan et al., (015)) was proposed to capture the motion information encoded in video clips. Due to the release large-scale Kinetics dataset (Kay et al.) 2017). 3D CNNs (Carreira & Zisserman 2017) were wildly used in action recognition. Its variants (Qiu et al., 2017: Tran et al. 2018 Xie al., 2018) decomposed the 3D convolution into spatial 2D convolution and a temporal ID convolution to learn the spa- tiotemporal features. And Feichtenhofer et al. (2019) designed a network with dual paths to learn the spatiotemporal features and achieved promising accuracy in video understanding.
The methods aforementioned all share a common insight that they are video invariant and ignore the inherent temporal diversities in videos. As opposed to these methods, we design two-level adaptive modeling scheme by decomposing the video specific operation into a location sensitive excitation and location invariant convolution with adaptive kernel for each video clip.
Attention in Action Recognition. The local branch in TAM mostly relates to SENet (Hu et al,, 2018). But the SENet learned modulation weights for each channel of feature maps. Several meth- ods (Liu et al. 2019b Diba et al., 2018) also resorted to the attention to learn more discriminative features videos. Different from these methods, the local branch keeps the temporal information to learn the location sensitive importances. [Wang et al. (2018b) designed a non-local block which can be seen as self-attention to capture long-range dependencies. Our TANet captures the long-range dependencies by simply stacking more TAM, and keep the efficiency of networks.
Dynamic Convolutions. Tia et al. (016)) first proposed the dynamic filters on the tasks of video and stereo prediction, and designed a convolutional encoder-decoder as filter-generating network. Several works (Yang et al. 2019 Chen et al. 2020) in image tasks attempted to generate aggregation weights for a set of convolutional kernels, and then produce a dynamic kernel. Our motivation are different from these methods. We aim to use this temporal adaptive module to deal with temporal variations in videos. Specifically, we design an efficient form to implement this temporal dynamic kernel based on input feature maps, which is critical for understanding the video content.
METHOD
3.1 THE OVERVIEW OF TEMPORAL ADAPTIVE MODULE
As we discussed in S video data typically exhibit the complex temporal dynamics caused by many factors such as camera motion and speed variations. Therefore, we aim to tackle this issue by introducing temporal adaptive module (TAM) with video specific kernels, unlike the sharing convolutional kernel in 3D CNNs. our TAM could be easily integrated into the existing 2D CNNs (e.g., ResNet) to yield a video network architecture, as shown in Figure 2 We will give an overview of TAM and then describe its technical details.
Formally, let X E I denote the feature maps for video clip, where C represents the number of channels, and T', H, W are its spatiotemporal dimensions. For efficiency, our TAM only focus on temporal modeling and the spatial pattern is expected to captured by 2D convolutions. Therefore, we first employ a global spatial average pooling to squeeze the feature map as follows:
X ((8)as = H W
1
Σ 1.
Xessji
(1)
à is the index of different dimensions (in channel, time, height and width), and X e RCXT aggregates the spatial information of X. For simplicity, we here use denote the function
where c,
3

---

Under review as a conference paper at ICLR 2021
convi
res5
scores
Input
TA-Block
TA-Block
TA-Block
TA-Block
E-Tritoy
1-1x1, Conv2D
1-1x1, Comv2D
rescale
sohmac
Agmold
CIT
TAM
ConviD
FC
F/
RAUU Crat
1*3x3, Conv2D
Conv10
FC
1x151, Com2D
181x1, Conv2D
O-T
Resoet-Block
TA-Block
g
d: W
C-TWWW
TAM
X
Figure 2: The overall architecture of TANet: ResNet-Block vs. TA-Block. The whole workflow of temporal adaptive module (TAM) in the lower right shows how it works. The shape tensor has noted after each step. @ denotes element-wise addition, @ denotes element-wise multiplication, and & denotes convolution operator. The symbols appeared in figure will explained in Sec. 3.1
that aggregates the spatial information. Our proposed temporal adaptive module is established based on this squeezed 1D temporal signal for high efficiency.
Our TAM composed of two branches: a local branch L and a global branch G. which aims to leam location sensitive importance map enhance discriminative features and then produces the location invariant weights to adaptively aggregate temporal information in a convolutional manner. More specifically, the TAM is formulated as follows:
((X) (C(X) + X),
where @ denotes convolutional operator and O denotes element- wise multiplication. is worth noting that these two branches focus different aspects of temporal information, where the local branch tries to capture the short term information to attend important features by using a temporal convolution, while the global branch aims to incorporate long-range temporal structure adaptive temporal aggregation with fully connected layers. Disentangling kernel learning procedures into local and global branches turns will be introduced in the following sections.
guide
effective way in experiments. The branches
3.2 LOCAL BRANCH IN TAM
discussed above, the local branch is location sensitive and aims to leverage short-term temporal dynamics to perform video specific operation. Given that the short-term information varies slowly along the temporal dimension, is thus required to learn a location sensitive importance map to discriminate the local temporal semantics.
As shown Figure 2 with ReLU non-linearit As the goal of local branch is to capture short term information. set the kernel size as 3 to learn importance map solely based on a local temporal window. To control the model complexity, the first Conv followed by BN (loffe & Szegedy, 20155) reduces the number of
we build the local branch with a sequence of temporal convolutional layers
4

---

Under review as a conference paper at ICLR 2021
channels from C' to C Then, the second ConviD with sigmoid activation yields the importance weights V E RCxT which are sensitive to temporal location. Finally, the temporal excitation is formulated as follows:
Z = Freccale (V) @ = ((X)xX
(3)
where @ denotes the element-wise multiplication and Z E R To match size of X,
F Prescale (V) rescales the V to V |CX by replicating in spatial dimension.
3.3 GLOBAL BRANCH IN TAM
The global branch, also termed as location invariant branch, focuses on generating an adaptive kernel based long-term temporal information. It incorporates global context information and learns to produce the location invariant and also video adaptive convolution kernel for dynamic aggregation.
Learning the Adaptive Kernels. We here opt to generate the dynamic kernel for each video clip and aggregate temporal information in a convolutional manner. To simply this procedure and as well as preserve high efficiency. The adaptive convolution will be applied in a channel-wise manner. this sense, we expect our learned adaptive kernel only considers the temporal relations without taking channel correlation into account. Thus, our TAM would not change the number of channels of input feature maps, and the learned adaptive kernel convolves the input feature maps in channel-wise manner. More formally, for the et channel, the adaptive kernel is learned as follows:
θ G(X)
(4)
where Og E R.K is generated adaptive kernel (aggregation weights) for c channel, K the adap- tive kernel size, 8 denotes the activation function ReLU. The adaptive kernel is also learned based on the squeezed feature map X, E RT without taking the spatial structure into account for modeling efficiency. But different with the local branch, we use fully connected (fc) layers F to learn the adaptive kernel by leveraging long-term information. The learned adaptive kernel with the global receptive field, thus could aggregate temporal features guided by the global context. To increase the modeling capabilities of the global branch, stack two fc layers and the learned kernel is normalized with a softmax function to yield a positive aggregation weight. The learned aggregation weights O {01, O2, ..., Oc} will be employed to perform video adaptive convolution.
Temporal Adaptive Aggregation. Before introducing the adaptive aggregation, we can look back on how vanilla temporal convolution aggregates the spatio-temppral visual information:
Y W & X,
(5)
Where W is the weights of convolution kernel and has no concern with input video samples in inference. We argue this fashion ignores the temporal dynamics in videos, and thus propose video adaptive aggregation to model video clips:
Y (
where G can be seen as a kernel generator function. Kernel generated by G can perform adaptive convolution, but is still location invariant and shared cross temporal dimension. To address this issue, the local branch produces Z with location sensitive importance map. The whole procedures can be expressed as follows:
Yeaj ((X) @ Z + Z Σ De.k Ze,t-k..i
where- denotes the scalar multiplication, Y RCXTXHXW is the output feature maps.
In summary, TAM presents an adaptive module with a unique two-step aggregation scheme, where the location sensitive excitation and location invariant aggregation all derive from input features, but focus on capturing different structures (i.e., short-term and long-term temporal structure).
3.4 EXEMPLAR: TANET
We here intend to describe how to instantiate the TANet. Temporal adaptive module as a novel temporal modeling method can endow the existing 2D CNNs with strong ability model different
5

---

Under review as a conference paper at ICLR 2021
temporal structures in video clips. In practice, TAM only causes limited computing overhead, but obviously improves the performance on different types of datasets.
ResNets (He et al. 2016) employed as backbones to verify the effectiveness of TAM. As illus- trated in Fig. 2 the TAM embedded into ResNet-Block after the first Conv2D, which easily turns the vanilla ResNet- Block into TA-Block.. This fashion will not excessively alter the topology of net- works and can reuse the weights of ResNet-Block. Supposing we sample frames as an input clip, the scores of T frames after fe will be aggregated by average pooling yield the clip-level scores. No temporal downsampling is performed before fe layer. The extensive experiments are conducted in Sec. to demonstrate the flexibility and efficacy of TANet.
3.4.1 DISCUSSIONS
We have noticed that the structure of local branch is similar to the SENet (Hu et al., 2018) and STC (Diba et al., 2018). The first obvious difference is our local branch does not squeeze the temporal information. We thus use temporal ID convolution as basic layer, instead of using fc layer. Two-layer design only seeks make a trade-off between non-linear fitting capability and model complexity. Furthermore, the local branch provides the location sensitive information, and thus addresses the issue that the global branch insensitive to temporal location.
TSN [Wang et al.) 2016). TSM (Lin al. 2019), etc. only aggregate the temporal features with fixed scheme, but our TAM can yield the video specific weights to adaptively aggregate the temporal features in different stages. In the extreme cases, our global branch in TAM can degenerate into TSN when dynamic kernel weights θ is learned to equal to [0, 1, 0]. From another perspective, the kernel weights θ is set to [1, 0, 0) [0, 0, 1), global branch can be turned into TSM. It seems that our TAM theoretically provides a more general and flexible form to model the video data. When refers 3D convolution (Ji et al. 2010). all input samples share the same convolution kernel without being aware of the temporal diversities in videos as well. In addition, our global branch essentially performs a video adaptive convolution whose filter has size
1, while each filter normal 3D convolution has size C x k x lo X k. where C' is the number of channels and k denotes the receptive field. Thus our method is more efficient than 3D CNNs. Unlike some current dynamic convolution (Chen et al.) 2020, Yang et al. (019)), TAM is more flexible, and can directly generate the kernel weights to perform video adaptive convolution.
EXPERIMENTS
In this section, we elaborately study the effectiveness of TANet on several standard benchmarks. The training recipe and inference protocol is described in Appendix (.I).
4.1 DATASETS
Our experiments are conducted on three large scale datasets, namely, Kinetics-400 (Kay al. 2017) and Comething-Something (Sth-Sth) VI&V2 (Goyal et al.) 2017). Kinetics-400 contains ~300k video clips with 400 human action categories. The videos Kinetics-400, trimmed from raw YouTube videos, are around 10s. We here train models on training set (~240k video clips), and test models on validation set (~20k video clips). The Sth-Sth datasets focus on fine-grained action, which contains series of pre-defined basic actions interacted with daily objects. The Sth-Sth V1 comprises ~86k video clips in training set and ~ video clips in validation set. Sth-Sth V2 is an updated version of Sth-Sth V1, which contains 169k video clips in training and ~25 video clips in validation set. They both have 174 action categories.
4.2 EXPLORATION STUDIES ON KINETICS-400
The exploration studies performed Kinetics-400 to investigate different aspects of TANet. The ResNet architecture we used is the same with He et al. (2016). Our TANet replaces all ResNet- Blocks with TA-Blocks by default.

---

Under review as a conference paper at ICLR 2021
Table 1: Ablation studies on Kinetics-400. All models use ResNet50 as backbone.
(a) Studying on parameter choices of a and g.
(b)
Trying larger temporal receptive fields of
θ.
setting
Frames
Top-1
Top-5
a=1, 3=4
8
75.63%
92.10%
Kernel
Frames
Top-1
Top-5
o=2, 8=4
76.09% 8
92.30%
K=3
8
76.09%
92.30%
a=4, 8=4
8
75.72%
92.14%
K=5
8
75.62%
92.14%
a=2, 8=2
8
75.91%
92.38%
K=3
16
76.87%
92.88%
2=2, 8=4
8
76.09
92.30%
K=5
16
77.19%
93.17%
o=2, 8=8
8
75.63%
92.20%
Table 2: Study on the effectiveness of TAM. All models use ResNet50 backbone and take frames with sampling stride 8 as inputs. To be consistent with testing, the FLOPs are calculated with spatial size 256 256. All methods share the same training setting and inference protocol.
FLOPs (of single view)
Models
Params
Top-1
Top-5
C2D
42,95G
24.33M
70.2%
88.9%
C2D-Pool
42.95G
24.33M
73.1%
90.6%
C2D-TComy
53.02G
28.10M
73.3%
90.7%
TSM
(Lin
2019
42.95G
24.33M
74.1%
91.2%
TEINet
Liu
901
43.01G
25.1
74.9%
91.8%
13Dsx1x
Wang et al. 2018b
62.55G
32,99M
74.3%
91.6%
C2D
Wang ct 20186
64.49G
31.69M
74.4%
91.5%
Global branch
43.
24.33M
74.9%
91.7%
Local branch
43.00G
25.59M
73.3%
90.7%
Global branch + SE (Ilu 2018
43.02G
24.65M
75.4%
92.0%
TANet-K
43.02G
25.59M
76.0%
92.2%
TANet
43.02G
25.59M
76.1%
92.3%
Parameter choices. We different combinations of and B figure out the optimal hyper- parameters TAM. The TANet is instantiated as in Fig. 2 TANet with a = 2 and = achieves the highest performance shown in Table 1a which will be applied in following experiments.
Temporal receptive fields. We also try to increase the temporal receptive fields for learned kernel θ in the global branch. From the Table Tb it seems the larger K is beneficial the accuracy when TANet takes more sampled frames as inputs. On the other hand, it even degenerates the performance of TANet when sampling frames. In our following experiments, the K will be set 3.
The results in Table have revealed that our TANet is insensitive to these Hyper-parameterr. which can save lot of time find optimal settings in practice. We also provide other more exploration studies in Appendix A.2 and show how these hyper- parameters impact the performance of TANet.
4.3 COMPARISON WITH OTHER TEMPORAL MODULES
As principled temporal operator, we intend to describe its competitive counterparts and then make fair comparisons with TAM in Table 2 The optimal contigurations studied in Sec. 4.2 will be employed in the following experiments. The default inference protocol samples 10 clips x crops to evaluate the performance of each model.
2D ConvNet (C2D). We use ResNet50 as backbone to build 2D ConvNet. The 2D ConvNet focuses on learning the spatial clues, which operates each frame independently without any temporal interaction before the last fc layer.
C2D-Pool. To probe into the impacts of temporal fusion, C2D-Pool utilizes the average pooling layer whose kernel size is K x x to perform temporal fusion without any temporal downsampling, which can be built by easily replacing all TAMs in network with pooling layers.
C2D-TCony. We also replace each TAM with a standard temporal convolution, and this comparison able to demonstrate the importance of adaptive modeling in temporal aggregation.

---

Under review as a conference paper at ICLR 2021
Table 3: Extending to other backbones, All models share the same inference protocol, e.g., 10 clips
x crops.
ShuffleNet V2
MobileNet V2
Inception V3
ResNet-50
ResNext-50
Models
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
w/o TAM
62.1%
84.3%
64.1%
85.6%
71.4%
89.8%
70.7%
88.9%
70.1%
88.8%
with TAM
67.3%
87.6%
71.6%
90.1%
75.6%
92.0%
76.1%
92.3%
76.4%
92.0%
A Acc. 5.2%
+.33%
7.5%
4.5%
+ 4.2%
2.2%
+ 5.4%
+ +.4%
+ 6.3%
+ 3.2%
There are some competitive methods based on C2D, i.e., TSM (Lin al. 2019) and TEINet (Liu
et al., 2019b). The new methods focus on designing a lightweight temporal module and can be inserted into the 2D CNN for efficiently capturing temporal information.
Inflated 3D ConyNet (I3D), I3D (Carreira & Zisserman 2017) is most frequently used models in action recognition. In our implementation, we inflate the first 1 kernel in ResNet-Block to 3 x 1 x 1, which can provide more fair comparisons with our TANet. Following the (Wang et al., 2018b), we use 13D3x1x1 to denote this variant.
The aforementioned methods share the same temporal modeling scheme with a fixed pooling or convolution operation. As shown in Table 2 our method yield a superior performance that higher than C2D by 5.9% accuracy, and even outperforms 13D3x1x1 (76.1% 1% vs. 74.3%), which exhibits the fixed schemes for modeling videos may be insufficient to learn the temporal clues. As the extra con- volutions in C2D-TConv might destroy the ImageNet pretrained weights, C2D-TConv even achieves a degenerated performance compared with 13D3x1x1- In addition, compared with other temporal counterparts in video recognition, TANet only brings small portion of FL.PPs and parameters.
Non-local C2D (NL C2D). The non-local block was proposed to capture the long-range dependen- cies in videos. The preferable settings with 5 non-local blocks mentioned in (Wang et al. (018b) are employed to compare with TANet. As seen Table 2 TANet achieves higher accuracy than NL C2D (76.1% 7% 74.4%). In addition, TANet is more efficient than NL C2D. TANet only has 43G FLOPs of single view and 25.6M parameters.
To study the each part of temporal adaptive module, separately validate the Global branch and Local branch. Furthermore, Global branch + SE uses global branch with SE module (Hu et al, 2018) to compare with TANet, which can prove the complementarity of local branch and global branch. TANet has achieved the highest accuracy among these models, which proves the efficacy of each part of TAM and as well as the strong complementarity between local branch and global branch. As opposed to Equ. 2. T combines the global and local branch in reverse order:
= L(X) @ (G(X) X),
We found TANet is slightly better than TANet-R.
Generalization to other Backbones. One critical issue that we need to figure out is the gener- alization of our proposed method extending to other backbone networks. To this end, we extend the TAM from ResNet to other well known classification backbones, like ShuffleNet V2 (Ma et al. 2018 MobileNet V2 (Sandler et al., 2018), Inception V3 (Szegedy al. 2016) and ResNeXt- 50 (Xie et al., 2017). From the Table 3 we can observe that the backbone networks equipped with our TAM outperform their C2D baselines by large margin, which strongly exhibits the powerful generalization as well as the huge potential of our temporal adaptive module.
4.4 COMPARISONS WITH THE STATE OF THE ART
Comparisons on Kinetics-400. Table shows the state-of-the-art results on Kinetics-400, Our method, as adaptive modeling scheme, has achieved competitive performance compared with other models. TANet-50 with 8-frame also outperforms SlowFast (Feichtenhofer et al. 2019) by 0.5% when using similar FLOPs per view. The 16-frame TANet only uses 4 clips and 3 crops for evaluation such that it provides higher inference efficiency and more fair comparisons with other models. It is worth noting that 16-frame TANet-50 is still more accurate than 32-frame NL I3D by 2.2%. As ip-CSN (Tran et al.) 2019) is pretrained on Sports- 1M (Karpathy et al. 2014). also achieves the promising accuracy with deeper backhone, i.e., ResNet1 $2, Furthermore, TAM is com-
8

---

Under review as a conference paper at ICLR 2021
Table Comparisons with the sate-of-thh-art methods on Kinetics-400. As described in Feichten- hofer et (011)). the GFLOPs (of single view) x the number of views (temporal clips with spatial crops) represents the model complexity. The GFLOPs are calculated with spatial size 256 256. The models in gray rows represent it is unjust to directly compare with our method.
Methods
Backbones
Training Input
GFLOPsxviews
Top-1
Top-5
TSN Wang 2016
InceptionV3
3x224x224
3x250
72.5%
90.2%
NeT Wang
2015a
ResNet18
16 x112x112
23.5x250
70.7%
89.3%
S3D-G Xie el al 2018
Inception V
64x224x224
71 x30
74.7%
93.4%
B3D
Carreira & Zisserman
2017
InceptionV1
64x224x224
108xNIA
72.1%
90.3%
Ri2+1)D Tran et al 2018
ResNet34
32x112x112
152x10
74.3%
91.4%
NI IDD
Wang al. 201 8h
NesNe
32x22Ax224
N/A
74.9%
91.6%
NL 13D
Wang al 201
Res
128x224x224
x82x30
76.5%
92.6%
ip-CSN
Tran et al
2019
ResNer50
8x224x224
1,2x10
70.8%
-5%
TSM
Lan er sl.
2019
ResNer50
16x224x224
65x30
74.7%
91.4%
TEINet
2015b
ResNer50
16x224x224
86x30
76.2%
92.5%
MLVNCEIAM-2
LLReNNet0
48x22Ax224
93x9
73.5%
91.2%
SlowOnly (Feichtenhofer al.
2019
ResNet50
8x224x224
42x30
74.8%
91.6%
SlowFast Ferchtemnoler el 2019
ResNer50
(4+32)xx224x226
36x30
75.6%
92. 1%
TA
Res
8x224x224
43x30
76.1%
92.3%
TANet-50
ResNet50
16x224x224
86x12
76.9%
92,9%
X3D-XL Peichtenhuler 2020
16x312x312
48x30
79.1%
93.9%%
ip-CSN
Tran el al
ResNet152
32 x224x224
83x30
79.2%
93.8%
SlowFasteNT.
Feicheenmmeer
2019
ResNet101
(16+64)x224x224
234x30
79.8%
93.9%
patible with existing video frameworks like SlowFast. Specifically, our TAM is more lightweight than 3 x I x 1 convolution when taking the same number of frames as inputs, but yields bet- ter performance. TAM thus easily replace the 3 higher accuracy with lower computational costs. It seems that X3D has achieved great success in video recognition. In other ways, X3D was searched by massive computing resource and can not be easily obtained in any situation. Although our method does not beat all state-of-the-art methods equipped with deeper networks, TAM lightweight operator can enjoy the benefits from more powerful backbones and vidco frameworks. In general, the proposed TANet makes good practice on adaptively modeling the temporal relations videos.
convolution in SlowFast achieve
More Results and Analysis. The results on Sth-Sth VI & V2 is presented in Appendix A.3 which TAM also achieves competitive performance compared with other methods. To have more intuitive understandings of temporal adaptive module, we also visualize the learned kernels Appendix (A.4), which are expected to provide more insights for TAM. As shown in Figure the diversities in our learned kernels have shown that the complex dynamics are indeed existing in videos, and learning temporal clues in an adaptive scheme has proven be effective yet reasonable.
CONCLUSION
In this paper, we have presented a novel temporal modeling operator, i.e., temporal adaptive module (TAM), to capture complex motion information in videos and built powerful video architecture (TANet). Our TAM is able to yield a video-specific kernels with the combination a local im- portance map and global aggregation weight. The local and global branches designed in TAM helpful to capture temporal structure by different views and contribute to making temporal modeling more effective and robust. As demonstrated the Kinetics-400, the networks equipped with TAM are better than the existing temporal modules in action recognition, which confirms the efficacy of our TAM video temporal modeling. TANet also achieved the state-of-the- th--r-- performance on the motion dominated datasets of Sth-Sth V2&

---

Under review as a conference paper at ICLR 2021
REFERENCES
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 4724 4733, 2017.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kemels. In 2020 IEEF/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 11027- 11036. IEEE, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Patlern Recognition CVPR, pp. 248-255, 2009.
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Mohammad Mahdi Arzani, Rahman Yousefzadeh, Juer- gen Gall, and Luc Van Gool. Spatio-temporal channel correlation networks for action classifica- tion. In Vittorio Ferrari, Martial Hebert, Cristian Sminchiseseu, and Yair Weiss (eds.). Computer Vision ECCV, volume 100 of Lecture Notes in Computer Science, pp. 299-315, 2018.
Quanfu Fan, Chun-Fu (Richard) Chen, Hilde Kuehne, Marco Pistoia, and David D. Cox. More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 2261-2270. 2019.
Christoph Feichtenhofer. X3D: expanding architectures for efficient video recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 200-210. IEEE, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In IEEE/CVF International Conference on Computer Vision, ICCV, pp. 6201-6210, 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phat, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In IEEE International Conference on Computer Vision, ICCV, pp. 5843-583358 1581-,551,121 2017.
Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Limin Wang, and Shilei Wen. Stnet: Local and global spatial-tempon modeling for action recognition. In The Thirty- Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Appli- cations of Artificial Intelligence Conference, IAAI, pp. 8401-8400 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE Inter- national Conference on Computer Vision, ICCV, pp. 2980-2988. IEEE Computer Society, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 7132-7141, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.). Proceedings the 32nd International Conference on Machine Learning, ICML 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448 456. JMLR.org. 2015.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human tion recognition. In Johannes Fürnkranz and Thorsten Joachims (eds.), Proceedings of the 27th International Conference Machine Learning (ICML., pp. 495-502, 2010.
10

---

Under review as a conference paper at ICLR 2021
Xu Jia, Bert De Brabandere. Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 667-675, 2016.
Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. STM: spatiotemporal and motion encoding for action recognition. CoRR, abs/ abs//908 1099900880666 2019.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Fei-Fei Li. Large-scale video classification with convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1725-1732, 2014.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, 0bb//105500650, 2017.
Alexander Kläser, Marcin Marszalek, and Cordelia Schmid. A spatio-temporal descriptor based on 3d do-gradients. In Mark Everingham, Chris J. Needham, and Roberto Fraile (eds.). Proceedings of the British Machine Vision Conference 2008, pp. 1-10. British Machine Vision Association, 2008.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, Imagenet classification with deep convo- lutional neural networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Sys- tems 25: 26th Annual Conference Neural Information Processing Systems 2012., pp. 1106- 1114, 2012.
2012.
Quoc V. Le, Will Y. Zou, Serena Y. Yeung, and Andrew Ng. Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3361-3368 IEEE Computer Society, 2011.
Lin, Chuang Gan, and Song Han. TSM:
temporal shift module for efficient video understanding. In IEEE International Conference Computer Vision, ICCV 2019, pp. 7082-7092, 2019.
Xingyu Liu, Joon-Young Young Lee, and Hailin Jin. Learning video representations from correspondence proposals. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 4273- 4281. Computer Vision Foundation IEEE, 2019a.
Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang. Jilin Li, Feiyue Huang. and Tong Lu. Teinet: Towards an efficient architecture for video recognition. CoRR, abs/1911 1. 1.09435, 2019b.
Chenxu Luo and Alan I.. Yuille. Grouped spatial-temporal aggregation for efficient action recogni- tion. In 2019 IEEE/CVF International Conference on Computer Vision ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 551 5511-5520. IEEE, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision - ECCV 2018 15th European Conference, Munich, Ger- many, September 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes in Computer Science, pp. 122-122-2 138. Springer, 2018.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In IEEE International Conference on Computer Vision, ICCV, pp. 5534-5542, 2017.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell., 39(6); 1137-1149, 2017.
11

---

Under review as a conference paper at ICLR 2021
Sreemanananth Sadanand and Jason J. Corso. Action bank: A high-level representation of activity in video. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp- IEEE Computer Society, 2012.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. A510-4520, IEEE Computer Society, 2018.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pp.
2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representationss ICLR 2015, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818- 2826. IEEE Computer Society, 2016.
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In IEEE International Conference on Computer Vision, ICCV, pp. 4489-4497, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 6450-6459, 2018.
Du Tran, Heng Wang, Matt Feiszli, and Lorenzo Torresani. Video classification with channel- separated convolutional networks. In JEEE International Conference on Computer Vision, ICCV 2019, pp. 5551-5560, 2019.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Lue Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Computer Vision ECCV, pp. 20-36, 2016.
Limin Wang. Wei Li, Wen Li, and Lue Van Gool. appearance-and-reen networks for video classification. In IEEE Conference Computer Vision and Pattern Recognition, CVPR, pp- 1430-1439, 2018a.
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Computer Vision ECCV, pp. 413-431, 2018.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In IEEE Conference Computer Vision and Pattern Recognition, CVPR, pp. 7794-7803, 2018b.
Geert Willems, Tinne Tuytelaars, and Luc Van Gool. An efficient dense and scale-invarian spatio- temporal interest point detector. In David A. Forsyth, Philip H. S. Torr, and Andrew Zisserman (eds.), Computer Vision - ECCV, volume 5303 of Lecture Notes in Computer Science, pp. 650- 663, 2008
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5987-5995. IEEE Computer Society, 2017.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs video classification. In Computer Vision ECCV, pp. 318-335, 2018.
12

---

Under review as a conference paper at ICLR 2021
TAM
1-1x1, Conv
1x1=1, Conv
1=1x1, Conv
1x1x1, Conv
TAM
1x3:3, Conv
1x3x3, Conv
1-3-3, Conv
1x3x3, Conv
TAM
1=1x1, Conv
1-1-1, Conv
1=1x1,
1-1=1, Conv
TAM
(d)
Figure 3: The four styles of TA-Block. The (b) is actually the model we used in the main text.
Brandon Yang, Gabriel Bender, Quoc Le, and Jiquan Ngiam. Condconv: Conditionally pa- rameterized convolutions for efficient inference. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver. BC, Canada, pp. 1305-1316, 2019.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Computer Vision ECCV, pp. 831- 2018.
Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. ECO: efficient convolutional net- work for online video understanding. In Computer Vision ECCV, pp. 713-730, 2018.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Training. In our experiments, we only train the models using frames and 16 frames as inputs. On Kinetics-400, following the practice in Wang et al. (2018b), The frames are sampled from 64 consecutive frames in the video. On Sth-Sth VI&V2, we employ the uniform sampling strategy TSN (Wang et al., 2016) train TANet. We first resize the shorter side of frames 256, and apply the multi-scale cropping and randomly horizontal flipping as data augmentation. The cropped frames are resized 224 x 224 for training the networks. The batch size set to 64. Our models are initialized by ImageNet pre-trained weights to reduce the training time. Specifically, on Kinetics- 400, the epoch for training is 100. The initial learning rate is set 0.01 and divided by 10 at 50, 75, 90 epoch. We use SGD with momentum of 0.9 and weight decay of le-4 to train TANet. On Sth-Sth V1&V2, We train models with 50 epochs. The learning rate starts at 0.01 and divided by 10 at 30, 40, 45 epoch. We momentum of 0.9 and a weight decay of 1e-3 to address the issue of overfitting.
Testing. We apply different inference schemes fairly compare with other state-of-the--ar models. On kinetics-400, we resize the shorter to 256 and take crops of 256 x 256 to cover the spatial dimensions. In the temporal dimension, we uniformly sample 10 clips for 8-frame models and clips for 16-frame models. The final video-level prediction is yielded by averaging the scores of all apatio-temporal views. On Sth-Sth V1, we scale the shorter side of frames 256 and use center crop of 224 x 224 for evaluation. On Sth-Sth V2, we employ similar evaluation protocols to Kinetics, but only uniformly sample clips.
A.2 MORE EXPLORATION STUDIES ON KINETICS-400
TAM in the different position. Table 5a tries to study the effects of TAM in different position. TANet-a, TANet-b, TANet-c. and TANet-d denotes the TAM is inserted before the first convolution, after the first convolution, after the second convolution, and after the last convolution in the block,
13

---

Under review as a conference paper at ICLR 2021
Table Ablation studies on Kinetics-400. All models use ResNet50 as backbone.
(a) Where to insert TAM into TA-Block.
model
Frames
Top-1
Top-5
TANet-a
8
75.95%
12.
TANet-b
76.09 %
92.30%
TANet-c
8
75.75%
92.13%
TANet-d
8
75.20%
91.78%
(b) The number of TA -lookocks inserted into ResNet50.
stages
Frames
Blocks
Top-1 1
Top-5
resş
74.12%
91.45%
re84-5
75.15%
92.04%
r
13
75.90%
92.22%
reS2-5
8
16
76.09%
92.30 %
respectively. These four styles are graphically presented in Fig. The style-(b) in Fig. Fcctualyy is TANet-b, which has slighter advantage than other styles shown in Table 5a The TANet-b will be abbreviated as TANet by default.
The number of TA-Blocks. To make trade-off between performance and efficiency, grad- ually add more TA-Blocks into ResNet. As shown in Table [bb] we find that the performance is nearly saturated when adding more than 9 TA-Blocks into network. The res2-5 achieves the highest performance and will be used our experiments.
A.3 COMPARISONS ON STH- STH VI & V2
Table 6:
Comparisons with the tate-of-thhear methods Sth-Sth VI. The models only taking RGB frames as inputs are listed in table. To be consistent with testing, we use spatial size 224 x24 to compute the FLOPs.
Methods
Frumes
Top-1
Top-5
TSN- RGB Wang
2016
ENInception
ImgNet
8f
160
19.5%
TRN Multiseie Z &
2088
BNInception
IngNet
8J
33G
S3D-G Xie di 2013
Inception
ImgNet
64f
7133GG
48.
78.7%
ECO
BNlncep+Res18
K400
16
64G
ECO Zootog inr 2018
BNincep+Res18
K400
92
267G
46 4%%
TSN (Wane
ReaNe150
ImgNet
8J
33G
13D (Wang Gupai
2013
ResNet50
ImgNei+K440
32
306G
41.6%
27%
NL J3DT[ang
2013
ResNet50
ImgNet+K400
327
334G
44.4%
76.0%
NL IL15+GG / Witit Ginoo
2018
ResNct50+GCN
ImgNet+K400
32f
606G
76.8%
TSM
[ineet al 201119
ResNet50
IngNet
85
33G
45 6%
74,
ReaNer50
IngNet
16f
65G
TSM Li 1: 2019
ResNet50
IngNet
16 8f
98G
7%
78.5%
TAM Tan
ResNet50
ImgNet
85
46.1% 1%
BLVNei-TAN (Fan di
2019
ResNei50
Sth-Sth V2
32
48G
48.4%
78.8%
GST Luo & Viilli 2019
ResNet50
IngNet
8f
30G
47 0%%
76. 165
GST
ResNe150
IngNet
16
TEINa Lau Lt et
20199
ReaNet50
IngNet
8f
33G
47. 4%
-%6
TEINer
ResNet50
IngNet
16f
66G
ReaNet50
IngNet
87
33G
75.8%
TANef TANetaa.
ResNet50
IngNet
16
60G
47 66%
77.7%
ResNet50
ImgNet
8f 16/
99C
50.6%
79.3%
Comparisons on Sth-Sth V1 & V2. As shown in Table 6 our method achieves state-of-the-art accuracy comparing with other models on Sth-Sth V1. fair comparisons, the Table only reports the results taking single clip with center crop inputs. TANetg, is higher than TSM equipped with same backbone (Top-1: 50.6% vs. Top-1: 49.7%). We also conduct the experiments on Sth-Sth V2. V2 has more video clips than V1, which can further unleash the full capabilities of TANet without suffering the overfitting. Following the common practice in Lin et al. (2019), TANets clips with 3 crops to evaluate the accuracy. As shown in Table 7 our models have achieved the state-of-an art performance on Sth-Sth V2. As result, the TANet yields a competitive accuracy (Top- 66.0%) compared with current SOTA results. The results Sth-Sth VI & V2
14

---

Under review as a conference paper at ICLR 2021
driving car
driving car sodk
drinking beer
skydiving 8244
daych
Mayth
TANet
13D
Figure 4: The statistics of kernel weights training on Kinetics-400, which plots the distributions in different temporal offsets (f (-1,0, 1)). Each filled area in violinplot represents the entire data range, where has noted the minimum, the median and the maximum. The first four columns in the left figure are the distributions of learned kernels in TANet. In the fifth column, also visualize the filters of 3 x x 1 kernel in 13D3x to compare with the TANet. The stage4 6b denotes the kernel comes from the 6th block in stage4.
have demonstrated that our method is also good at modeling the fine- fin-grained and temporal-related video clips.
Table 7: Comparisons with the SOTA on Sth-Sth V2.
Meehhod
Backhonees
Pre-min
frumesx Tumesxclippxcrrpp
Top-1
Top-5
TRN
ENIncepiio
ImgNet
S/x2x3
TSM
La S1
2019
ResNet50
ImgNat
8fx2x3
59.1%
85.6%
TSM
2019
ResNet50
ImgNet
16x2xx
63.4%
88.5%
1MM Siy
ResNer50
ImgNet
(16 16)x233
CPNe1 & K0YS
ResNet50
IngNat
21/x16x16
57.7%
$4.0%
GST Lao & Vinile 2019
ResNet50
ImgNet
8fx1x1
61.6%
$7.29
GST
ResNet50
ImgNet
16/x1x1
62.6%
87.9%
(ran
32/x1x1
$8.1%
TEINel T Lan [ 20
ResNet50
IngNet
8fx1x1
61.3%
-5%
TEINel
ResNet50
ImgNet
16/x1x1
62.1%
-
ImgNet
TANet
ResNer50
ImgNet
8fx2x3
62.7%
88.0%
TANeL
ResNet50
ImgNet
16x1xI
62.5%
87.6%2
TANet
ImgNet
TANet,t
ResNet50
ImgNet
(8/+16/22ス3
66.0%
90.1%
A.4 VISUALIZATIONS OF LEARNED KERNEL.
To understand the behavior of TANet, visualize the distribution of kernel θ generated by global branch the last block of stage4 and stage5. For clear contrast, the kernel weights in 13Dax1x1 the same stages also visualized to find more insights. As depicted in Fig. we find that the learned kernel has an evident character: the shapes and scales of distribution are more diverse than 13Dax1x1. 1x1. Since all video clips share the same kernels in 13D 1xxxxxx11 it causes the kernel weights clusters together excessively. As opposed to temporal convolution, even modeling the same action in different videos, TAM can generate the kernel with slightly different distributions. Taking driving car as an example. the shapes of the distribution shown in Fig. are similar to each other but the medians of distributions are not equal. For different actions like drinking beer and skydiving, the shapes and medians of distributions are greatly varied. Even the same action in different videos, TAM would learn a different distribution of kernel weights. Concerning that the motion patterns in different videos may share varied inherence, it is necessary to employ adaptive scheme when modeling video sequences.
To probe into the effects on learning kernels in the different stages, the visualized kemels are further chosen in stage4_66 .6b and stage5_3b. respectively. The videos are randomly selected from Kinetics- 400 and Sth-Sth V2 to show the diversities in different video datasets. As depicted in Fig, 5 and Fig. 6 We can observe that the distributions importance map V in the local branch are smoother than the kernel θ in the global branch, and the local branch pays different attention to each video
15

---

Under review as a conference paper at ICLR 2021
when modeling the temporal relations. Futhermore, learned kernlls visualized in figures have exhibited the clear differences between two datasets (Kinetics-400 vs. Sth-Sth V2). This fact is in line with our prior knowledge that there is obvious domain gap between two datasets. The Kinetics-400 mainly focuses on appearance and Sth-Sth V2 is a motion dominated dataset. However, this point not be easily inferred from the kernels in 13Dax1x1, because the overall distributions of kernels in 13Dax1x1 on two datasets show minor differences.
We visualize the histogram of kernel weights in the global branch intuitively show the insightful patterns between kernel weights and visual content. As observing the Fig. and Fig. 8 we found the distribution of kernel associated with the motion magniude.. The smaller magnitude of motion in video leads to a relatively lower weight at t = O, and the larger magnitude of motion or the actions severely related to scenes may cause the higher weight at 0. worth noting that the weight at t=0 usually plays dominate role in learning the spatio-temporal representations. These findings are expected provide more insights for designing the temporal module in video recognition.
Generally, the diversities in our learned kernels have demonstrated that the diversities are indeed existing videos, and it reasonable to learn spatiotemporal representation an adaptive scheme. These findings are again in line with our motivation claimed in the paper.
16

---

Under review as a conference paper at ICLR 2021
Global branch
Local branch
I3D
Video
Tiking Soiiggon mai
cooking on campfire
Kinetics-4i 400
Video
10 11200
80 1000
00 2200
Shaking read
Video
10 11200
0 2000
showing hamburger next to glasses
Video
Something Something V2
10
throwing cloth
Video
13
covering salt shaker
with tower
Video 6
10 SARE
Figure 5: The distribution of learned kernel V and θ in the stage4_6b.
17

---

Under review as a conference paper at ICLR 2021
Global branch
Local branch
I3D
Video 1
fixing hair
14
19
cooking on campfire
Kinetics-4pp 400
Video
Lrakin heaeee Video 3
05 10000
showing hamburger next to glasses
Video
RF S100
Something-SS Someehing-Something Something V2
91
100
throwing cloth
Video
covering salt shaker
with a towel Video 6
Figure 6: The distribution of learned kernel V and θ in the stage5_3b.
18

---

Under review as a conference paper at ICLR 2021
150
150
150
540
140
140
130
230
150
130
220
140
120
110
110
110
180
100
130
99
93
80
60
03
60
50
60
20
10
0
22
D3
04
05
0.6
0.
01
02
0.3
0.7
0.6
0.5
1/
0.0
34
05
07
dati buboo IARQTIA
aSHHuUAI whiight
the disinuttion ANN
(a) action sample of blowing leaves
t=O
tw1
11S
110
100
63
50
47
50
21
30
10
dictmbution wrights
dstbbutton
(b) A action sample ice skating
Figure 7: The visualization for the videos which contain small magnitude of motion. The mov- ing persons appeared videos are usually far away from camera. Video clips are sampled from validation set of Kinetics-400, and kernel θ is selected from stage4 6b.
19

---

Under review as a conference paper at ICLR 2021
t-0
130
170
110
110
110-
100
100
3.0
90-
20
90
80
80
20
m
30
60-
60-
60-
40
20
20
21
10-
0D
0.2
03
C4
05
0.8 09 1.0
02
62
0.3
94
03 04
0.8
0.9
1.0
0.0
81
02
04
0. 0.7
0.8
0.
The dutribution weights
the doertutionn weights
the
wrights
(a) A action sample of playing cello.
t-0
340
81
20
10
35
15
10
10-
C.
disribution
cleriuutton
(b) A action sample of situp.
Figure 8: The visualization for the videos with large magnitude of motion. The moving persons in videos are usually closed to camera. Video clips are sampled from validation set of Kinetics-400, and kernel θ is selected from stage4 6b.
20


Reviews:
[{"review_id": "MVwn5BTvXVM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*********\nSummary Of The Manuscript:\n*********\nThe manuscript addresses the problem of Video Recognition one of the applications of Computer Vision. Due to various complex temporal dynamics of video data (Camera Motion, Speed, etc.), to capture the vast information, the author presents a novel Temporal Adaptive Module (TAM) for generating kernels based on the temporal feature maps. In addition, these feature maps are a combination of local and global features and as an exemplar, the author presents an architecture - TANet by incorporating their temporal operator. Together with a variety of experiments on standard benchmark for Video Recognition: Kinetics - 400 and Something-Something, the author showcases that for the task of Video Recognition compared to existing temporal operators, TAM's performance is fairly consistent and better and archives State-of-the-art with similar complexity in their exemplar architecture - TANet. \n\n*********\nStrength Of The Manuscript:\n*********\n++ Novelty\n\n- The task formulation is concise, convincing, and novel. A seemingly reasonable approach has been proposed in this manuscript for the task of Video Recognition. Compared to the existing baseline and recent approaches, the proposed architecture - TANet achieves SOTA results. \n- To the best of my knowledge, the incorporation of two branches - Local and Global branch makes the whole operator efficient and flexible for adaptation in the frameworks by stacking them to capture more complex information. Thus the concept of the TAM is convincing to capture complex temporal information.\n- In addition, the exemplar showcased by the authors - TANet has been created by incorporating TAM in the existing 2-Dimensional CNNs to capture vast information which proves that the proposed module/operator is flexible enough and can be adapted to different frameworks/architectures for better performance. \n\n++ Clarity\n\n- The manuscript is written in an excellent way to provide a brief insight into TAM. Especially Subsection 3.2 and 3.2 provide a good in-depth description of how the local and global branch works effectively in a joint manner to capture the short and long-term complex temporal information. \n- The manuscript also clearly describes the improvements and adequately contextualizes the contributions in such a way that it makes a good starting point for a novice reader. \n\n++ Evaluation\n\n- The experiments are sufficient and convincing. This new operator and exemplar TANet shows improved performance in nearly all cases on the datasets \n- The experimental evaluations demonstrate the effectiveness of the proposed architecture and showcase its practical value.\n- Also, an ablation analysis demonstrates to gain an understanding of where the performance benefits have been obtained such as receptive fields and parameter choices. \n\n*********\nWeakness Of The Manuscript:\n*********\nOverall, currently at this stage, this is a very good and strong manuscript in my entire batch. I like the simplicity and wide applicability of the proposed operator, especially the incorporation of local and global branches and adaptive aggregation. Thus I do not have any major weakness issues after reading the manuscript several times. Detailed literature review, a complete overview of each component, and detailed experiments and ablation studies helps to give a good insight into the manuscript. I found this paper pretty solid and have not able to found concerns relating to the proposed work. \n\n*********\nJustification Of The Review:\n*********\nOverall, happy with the current version of the manuscript. As mentioned earlier, I like the simplicity and wide applicability of the proposed module, and the architecture and setup details are provided in such a manner that it is very easy to convert into code in some timeframe. Detailed literature review, a complete overview of each component, and detailed experiments and ablation studies help to understand the author's work. Finally, I think the paper is pretty solid and thus I prefer to give a rating of 8 currently. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "KaEjPF1Fg79", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a temporal adaptive module for video recognition. Specifically, it decouples dynamic kernel into a location sensitive importance map and a location invariant aggregation weight, which can be plugged into existing 2D CNNs to yield a powerful video architecture with small extra computational cost. The experiments conducted on several datasets demonstrate the effectiveness of the proposed method.\n\nPaper Strength\n(1)\tThe proposed method develops a simple but effective module for video recognition, achieving good performance on various kinds of datasets.\n(2)\tThe proposed two-level adaptive modeling scheme is effective to describe motion patterns. Specifically, it decomposes the video specific temporal kernel into a location sensitive importance map and a location invariant aggregation kernel.\n\nPaper Weakness\n(1)\tFigure 1 is confusing. Why the dimension of the green filter is 3*1*1?\n(2)\tIn Figure 2, TAM is used before the 1*1*3 convolutional layer in the ResNet block. What about the influence of TAM at different locations? Is there any insight for this design?\n(3)\tMoreover, what if we exchange the order of the local branch and the global branch?\n(4)\tIn Section 3.4, the authors claim that the proposed temporal adaptive module can be plugged into existing 2D CNNs with a strong ability to model different temporal structures in video clips. However, only the ResNet-50 backbone is verified in the experiment. More experiments with other backbones such as VGG and Inception should be added.\n(5) In Table 3, the proposed method is not compared to the state-of-the-art X3D method [*] on the Kinetics-400 dataset.\n\n[*] X3D: Expanding Architectures for Efficient Video Recognition.\n\nSummary\nThis paper proposes a temporal adaptive module for action recognition. The proposed module is straight-forward and obtains good performance on different datasets. The experiments are not thorough enough to demonstrate that the proposed module can be plugged into different 2D CNNs with good performance. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JADip2YmEUy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new temporal adaptive module (TAM) to generate video-specific temporal kernels based on its own feature maps. TAM proposes a unique two-level adaptive modeling scheme by decoupling dynamic kernel into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short term information, while the aggregation weight is generated from a global view with a focus on long-term structure.\n\nThe global branch aims to incorporate long-range temporal structure to guide adaptive temporal aggregation with fully connected layers. The adaptive kernel (aggregation weights) is learned based on long-term temporal information, and incorporates global context information and learns to produce the location invariant and also video adaptive convolution kernel for dynamic aggregation. The visualization of the statistics of kernel weights shows that the shapes and scales of distribution are more diverse and data-adaptive. It is indeed reasonable to learn spatiotemporal representation in an adaptive scheme.\n\nHowever, the paper can be improved further.\n1.\tSome SOTA performances are ignored selectively, such as some performances of Slowfast[1]. \n[1] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pp. 6201–6210, 2019\nMaybe the proposed method does not achieve the best performance, but it is necessary to compare with the SOTA methods completely. Deep analysis can help the readers get the core contribution of the paper.\n2.\tThe TAM only focus on the temporal modeling. This may limit the final performance of the network. If the idea can be extended to the spatial-temporal field, it will be more valuable. \n3.\tThe authors give the distributions of the important map V in the local branch and the kernel \\Theta in the global branch. Since these V and \\Theta are all data-dependent, it is not unexpected that the distributions have larger diversities than the traditional I3D’s temporal kernel. If the authors can try to explore the relationship between the distributions and the characteristic of some action samples, beyond the diversity visualization of distributions, it will be more convincing.\n4.\tFig. 1 is difficult to understand. It is not straightforward to figure out how the attention weights or kernel weights are learned. Arrows in Fig. 1 are also confusing. Some indicate names, while some indicate feature flows.  \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a temporal module for video representation learning, which is a combination of temporal attention and temporal convolution.

The reviewers' opinions diverge. R2 does not find any major flaws of the paper, while R1 expressed concerns in terms of experimental details, ablations, and missing comparison to the state-of-the-arts. R4 expressed a similar concern, while favoring the paper a bit more.

The AC agrees more with the senior reviewers (R1 and R4) that the paper misses its experimental comparison to the state-of-the-arts. In particular, the AC supports the statement from R4 that "Some SOTA performances are ignored selectively" to favor the proposed approach. The missing SOTA includes Slow-Fast as pointed out by R4, X3D as pointed out by R1, and more. For instance, X3D is able to obtain 80.4% top-1 accuracy on Kinetics-400, which is superior to 76.9% of this paper, but is being ignored in the paper. X3D that uses a different strategy to abstract temporal information than this paper is also superior in terms of the FLOPS: X3D gets 79.1% while using almost half of the computation the proposed approach is using. The authors responded by "X3D uses network architecture search strategy to discover the optimal setting for 3D CNNs", but this is a misleading statement as X3D does not use any neural architecture search method. The authors argue that their proposed approach might be able to also benefit X3D, but this has not been confirmed and we cannot judge since no quantitative results are provided. In addition, as mentioned, several other standard baselines such as Non-Local R101 (with 77.7% accuracy) and ip-CSN-152 (with 77.8% accuracy) performing better than the proposed approach are missing.

Overall, we find the experimental section of the submitted paper incomplete.
Decision: Reject