CT-Net: Channel Tensorization Network for Video Classification
Authors: Kunchang Li
Xianhang Li
Yali Wang
Jun Wang
Yu Qiao
Venue: ICLR 2021 Poster (2021)
Keywords: Video Classification
3D Convolution
Channel Tensorization

Abstract: 3D convolution is powerful for video classification but often computationally expensive, recent studies mainly focus on decomposing it on spatial-temporal and/or channel dimensions.  Unfortunately,  most approaches fail to achieve a preferable balance between convolutional efficiency and feature-interaction sufficiency.  For this reason,  we propose a concise and novel Channel Tensorization Network (CT-Net),  by treating the channel dimension of input feature as a multiplication of K sub-dimensions. On one hand,  it naturally factorizes convolution in a multiple dimension way,  leading to a light computation burden.  On the other hand, it can effectively enhance feature interaction from different channels,  and progressively enlarge the 3D receptive field of such interaction to boost classification accuracy.  Furthermore, we equip our CT-Module with a Tensor Excitation (TE) mechanism. It can learn to exploit spatial, temporal and channel attention in a high-dimensional manner, to improve the cooperative power of all the feature dimensions in our CT-Module. Finally, we flexibly adapt ResNet as our CT-Net. Extensive experiments are conducted on several challenging video benchmarks, e.g., Kinetics-400, Something-Something V1 and V2. Our CT-Net outperforms a number of recent SOTA approaches, in terms of accuracy and/or efficiency.

Published as a conference paper at ICLR 2021
CT-NET: CHANNEL TENSORIZATION NETWORK FOR VIDEO CLASSIFICATION
Kunchang 2 Xianhang Li Yali ; Wang +& Yu Qiao!
"Guangdong-Hon Kong-Macao Joint Laboratory of Human-Maching
Pmelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen, 518055, China
UUiiveriiyy of Chinese Academy of Sciences
SIAT Branch, Shenzhen Institute of Artificial Intelligence and Robotics for Society University of Central Florida
{kc. 1i, yl. wang, yu. qiao) x xianhangl i 1htsights..i..ee ucf. edu, Jun Wan.Wagg@ucfedd
ABSTRACT
3D convolution is powerful for video classification but often computationally ex- pensive, recent studies mainly focus on decomposing on spatial-temporal and/or channel dimensions. Unfortunately, most approaches fail to achieve a preferable balance between convolutional efficiency and feature-interaction sufficiency. For this reason, we propose a concise and novel Channel Tensorization Network (CT- Net), by treating the channel dimension of input feature as a multiplication of K sub-dimensions. On one hand, it naturally factorizes convolution in a multi- ple dimension way, leading to a light computation burden. On the other hand, it can effectively enhance feature interaction from different channels, and pro- gressively enlarge the 3D receptive field of such interaction to boost classification accuracy. Furthermore, equip CT-Module with a Tensor Excitation (TE) mechanism. can learn to exploit spatial, temporal and channel attention in a high-dimensional manner, to improve the cooperative power of all the feature di- mensions our CTT-Module. Finally, we flexibly adapt ResNet as our CT-Net. Extensive experiments are conducted on several challenging video benchmarks. e.g., Kinetics-400, -SoethinngSomtthing VI and V2. Our CT-Net outperforms a number of recent SOTA approaches, in terms of accuracy and/or efficiency.
INTRODUCTION
3D convolution has been widely used to learn spatial-temporal representation for video classifica- tion (Tran et al. 2015) Carreira & Zisserman 2017). However, over parameterization often makes it computationally expensive and hard train. To alleviate such difficulty, recent studies mainly focus on decomposing 3D convolution (Tran et al. (0188 2019). One popular approach is ppatial-temporal factorization (Qiu et al. 2017; Tran et al. 2018 Xie et al. 2018), which can reduce overfitting by replacing 3D convolution with 2D spatial convolution and ID temporal convolution. But it still in- troduces unnecessary computation burden, since both spatial convolution and temporal convolution performed over all the feature channels. To further decrease such computation cost, channel sep- aration has been recently developed via operating 3D convolution in the depth-wise manner (Tran) eet al. 2019). However, it inevitably loses accuracy due to the lack of feature interaction between dif- ferent channels. For compensation, it has to introduce point-wise convolution to preserve interaction with extra computation. So there a natural question: How to construct effective 3D convolution to achieve a preferable trade-off between efficiency and accuracy for video classification?
FFqually-contributee contributed first authors ((kc.li, yl.wang) xianhangli @ knights.ucf.edd ucf.edu) 1 Correspnn author qy..iiaossiat.ee
> @ @siat.ac.cn,

---

Published as a conference paper at ICLR 2021
Input
Channel ChannelTensoriaenee Tensorization (c (C=Gxx))XXXXX
G-TSCoov GNIXXNRX
C2-TSCome
Output cxTxxxW
1
c=
C2
W
(Lx] 0. 12
E
Interaction of channels, size of receptive field): (C, chamnets, 3xx33) (G, X C2
x
Figure I: Simple illustration of channel tensorization (K 2). We tensorize the channel dimension of input feature as multiplication of K sub-dimensions. Via performing spatial/temprral tensor separable convolution along each sub-dimension, we can achieve preferable balance between con- volutional efficiency and ceature-interaction sufficiency. Introduction shows more explanations.
Method
3D Convolution (1 x w)]
Convolational Effciinccy
Feature Interaction Sufficiency
Spatial-ccmporal
Channel
Interact Manner
Interact Hield
C31
Full 3
x
X
STC
R22+1 00
Pull 1x3x3
V
X
SC
3²
2018
Full:
TC
C
Full: 1x 1x1
X
V
C
3
Tran 2019
DW: 3x3x3
ST
C C; X Cx))
Our Cc.c
Ci::
Oi: x x +x1xxl
1x (1 (1x3x3 3x11x1)
STCK
STC;
(2K
1)*
Interuct Field means the receptive field for feature interaction.
Table Two design principles to build effective video representation and efficient convolution.
This paper attempts address this question by investigating two design principles. (1) Convolu- tional Efficiency. As shown Table [1] current designs of spatial-temporal convolution mainly focus on decomposition from either et al. 2019). To enhance convolutional efficiency, we consider decomposing convolution in a higher dimension with novel representation of feature tensor. (2) Feature-Interaction Sufficiency. Table
(Tran et 2018) or channel dimension (Tran)
] clearly shows that, for current decomposition approaches (Tran et al. 2018) 2019), feature inter- action only contains one or two of spatial, temporal and channel dimensions at each sub-operation. Such partial interaction manner would reduce classification accuracy. On one hand, it decreases discriminative power video representation, due to the lack of joint learning on all dimen- sions. On the other hand, it restricts feature interaction in a limited receptive field, which ignores rich context from a larger 3D region. Hence, to boost classification accuracy, each sub-operation should achieve feature interaction all the dimensions, and the receptive field of such interaction should be progressively enlarged as the number of sub-operations increases.
Based these desirable principles, design novel and concise Channel Tensorization Module (CT-Module). Specifically, we propose to tensorize the channel dimension of input feature as a mul- tiplication of K sub-dimensions, i.e., C Ci C2 x separable convolution along each sub-dimension, we can effectively achieve convolutional efficiency and inttuurr-ittraaction sufficiency. For better understanding. we use the case of illustration in Figure ! First, we tensorize the input channel into C = C1 x Cz2. Naturally, we sepa- rate the convolution into distinct ones along each sub-dimension, e.g., for the 1st sub-dimension, we apply our mpatial-tempone tensor separable convolution with the size C1 x hx which allows us to achieve convolutional efficiency on all the spatial, temporal and channel dimensions. After that, sequentially perform the tensor separable convolution sub-dimension by sub-dimension. As result, we can progressively achieve feature interaction on all the channels, and enlarge the spatial- temporal receptive field. For example, after operating 1 tensor separable convolution sub-din sub-dimension, C, channels interact, and 3D receptive field of such interaction is 3 x 3 x 3. Via further operating 2nd tensor separable convolution the 2nd sub-dimension, all Ci Ca C
Ck. Via performing spatial/temporal
as simple
the 1 st
2

---

Published as a conference paper at ICLR 2021
channels have feature interaction, and 3D receptive field of such interaction becomes x x x 5. This clearly satisfies our principle of
sufficiency.
We summarize contributions in the following. First, we design a novel Channel Tensorized Mod- ule (CT-Module), which can achieve convolutional efficiency and feature-interaction sufficiency, via progressively performing spatial/temporal tensor separable convolution along each sub-dimension of the tensorized channel. Second, we equip CT-Module with a distinct Tensor Excitation (TE) mechanism, which can further activate the video features of each sub-operation by spatial, temporal and channel attention in tensor-wise manner. Subsequently, we apply this full module in a residual block, and flexibly adopt 2D ResNet as our Channel Tensorized Network (CT-Nee)). In this case, we can gradually enhance feature interaction from a broader 3D receptive field, and learn the key spatial-temporal representation with light computation. Finally, we conduct extensive experiments on number of popular and challenging benchmarks, e.g., Kinetics (Carreira & Zisserman (017), Something-Something V1 and V2 (Goyal el. 2017b). Our CT-Net outperforms the state-of-the-an methods in terms of classification accuracy and/or computation cost.
2 RELATED WORKS
2D CNN for video classification. 2D CNN is a straightforward but useful method for video clas-
sification (Karpathy et al. 2014 Simonyan & Zisserman [014 Wang et al. [2016] Liu al. 2020
Jiang et al.) (2019). For example, Two-stream methods (Simonyan & Zisserman 2014) learn video
representations by fusing the features from RGB and optical flow respectively. Instead of sampling a single RGB frame, TSN (Wang et al., 2016) proposes a sparse temporal sampling strategy to learn video representations. To further improve accuracy, TSM (Lin al., 2019) proposes a zero- parameter temporal shift module to exchange information with adjacent frames. However, these methods may lack the capacity of learning spatial-temporal interaction comprehensively, which of- ten reduces their discriminative power to recognize complex human actions.
3D CNN for video classification. 3D CNN has been widely used learn a rich ppatial-temport context better (Tran et al.] Careeca & Zisserman 2017 Feichtenhofer et al. 2019 Sudhakaran
jet al., 20200 Feichtenhofer)] 20200) However, introduces a lot of parameters, which leads to a difficult optimization problem and large computational load. To resolve this issue, I3D (Carreira & Zisserman 2017) inflates all the 2D convolution kernels pre-trained on ImageNet, which is helpful for optimizing. Other works also try to factorize 3D convolution kernel to reduce complexity, such as P3D (Qiu et al. (017) and R(2+1)D (Tran et al. 2018). Recently, CSN (Tran et al. 2019)) operates 3D convolution in the depth-wise manner. Nevertheless, all these methods still do not achieve good trade-off between accuracy and efficiency. To tackle this challenge, we propose CT-Net which learns on spatial-temporal and channel dimensions jointly with lower computation than previous methods.
METHODS
In this section, we describe our Channel Tensorization Network (CT-Nett in detail. First, we for- mally introduce our CT-Module in a generic manner. Second, we design a Tensor Excitation (TE) mechanism to enhance CT-Module. Finally, we flexibly adapt ResNet our CT-Net to achieve preferable trade-off between accuracy and efficiency for video classification.
3.1 CHANNEL TENSORIZATION MODULE
As discussed in the introduction, the previous approaches have problems in convolutional efficiency or feature-interaction sufficiency. To tackle such a problem, we introduce generic Channel Ten- sorization Module (CT-Module), by treating the channel dimension of input feature multipli- cation of K sub-dimensions, i.e., C = C₁ x C2 x... x Cg. Naturally, this tensor representation allows to tensorize the kernel size of convolution TConv() as multiplication of K sub-dimensions, too. To simplify the notation, the channel dimension of the output is omitted by default. The output X Xout can be calculated as follows:
X = TConv (X... i WCxXCxx-- Xxxxxxxxxxxxee xtxhxxu
(1)
where Xin and W denote the tensorized input and kernel respectively. However, such an operation requires large computation, so we introduce the tensor separable convolution alleviate the issue.
3

---

Published as a conference paper at ICLR 2021
Tensor Separable Convolution. We propose factorize TConv() along K channel sub-dimensions. Specifically, we decompose TConv) as K tensor separable convolutions TSCom(), and apply TSConv() sub-dimension by uub-dimension follows:
g = -SCCon (X WIX-xXxXX X..x Xxxxxxxxxxxxxxxx
(2)
where X, Xnn and Xout X On hand, the kernel size of the kthh TSCom(() is (1 X .... x Ck X ...... x x x h X w). It illustrates that only C channels interact in the tthh sub- operation, which leads convolution efficiency. On the other hand, as we stack the TSComv() each convolution performs on the output features of the previous convolution. Therefore, the spatial- temporal receptive field is enlarged. Besides, interactions first occur in C1 channels, second in C1 x C2 channels and so on. Finally, C1 x C2 x ... x Ox = C' channels can progressively interact. This clearly satisfies our principle of ceature-interaction sufficiency.
Spatial-Tempora Tensor Separable Convolution. To further improve convolution efficiency, we factorize the 3D TSConv() into 2D spatial TSConv() and ID temporal TSConv(). Thus, we can obtain the output features X and X as follows:
x = S-TSComv (XK-1, W mxlxxxxa XC x Xxxxxxxxxxxxxx
(3)
X --TSConv (X Ixx Oxxxxxxxxxxi xxxxxxXXxx
(4)
where S-TSConv() and 1:7SConv() represent spatial and temporal tensor separable convolution spectively. Finally, we attempt to aggregate spatial and temporal convolution. There are various connection types of spatial and temporal tensor separable convolution, e.g., parallel and serial types. According to the results of the experiments in Section ! we utilize the parallel method, which illustrates that we sum the spatial feature X and temporal feature XT:
X = x
(5)
3.2 TENSOR EXCITATION
Our CT-Module separates feature along spatial, temporal and channel dimensions. To make full use of their cooperative power to learn distinct video features, we design a concise Tensor Excitation (TE) mechanism for each dimension. First of all, we attempt utilize the TE mechanism to enhance spatial and temporal features respectively. For the spatial feature X obtained by Equation our corresponding spatial TE mechanism can be formulated as:
U²+ x Sigmod(S- (T (
where T-Pool() represents global temporal pooling, i.e., T x 1 x 1 average pooling. By performing it X we obtain the feature with the size (C₁ C2 spatial contexts along temporal dimension. Subsequently, the spatial tensor separable convolution S- TSConv() and the activate function Sigmod() are performed generate the spatial attention heatmap. Finally, the glement-wiss multiplication @ broadcasts the spatial attention along the temporal dimen- sion. Similarly, we perform the temporal TE mechanism for the temporal feature XT.
x Cg x H x W), which gathers
Vx
2 X Sigmod(T-TSComm(SSPodl (xx ())
where S-Pool() and T-TSConv() are global spatial pooling and temporal tensor separable convolution correspondingly. At last, after aggregating the spatial and temporal features by addition, i.e., R₁, U + Vv. we perform channel-wise TE mechanism as follows:
X = R₂ X
(8)
We adopt point-wise tensor separable convolution PW-TSConv() to learn the weights for aggregating distinctive channels. The rest follows the previous design. Note that all tensor separable convolu- tions are performed on the same sub-dimension different from the SE mechanism (Hu et al.) 2020). Through the cooperation of the TE mechanism along three different dimensions, the spatial-tempont features can be significantly enhanced.
the previous convolution, which is essentially
4

---

Published as a conference paper at ICLR 2021
CoRv
Imput
channel tensarization
channel tensarization
stage2
Spatial G SCCon
Temporal C -SCCnn
Spatial C--T SCone
Temporal C-TTSCony
RexDlook1
stage3
spattal attention
cemporal attention
CT-Block
repeat for
repeat for
staget
Reshilacks
channel attention
xx1x 30 Conv
stage5
x 1 20 Conv
x 22 Cons
output
(a)simple CT-Block
(b) CT-Block
(c)CT-Net (ResNet style)
The pipelines of CT-Blocks and the overall architecture of CT-Net. We replace one of every two ResBlocks in ResNet with our CT-Block and the extra point-wise convolution in the last sub-dimension (k K) is ignored. More details can found Sectior
Figure 2:
3.3 CHANNEL TENSORIZATIONI NETWORK
We regard ResNet exemplar and build up our CT-Net from ResNet50 (or ResNet101). First, we design a simple CT-Block in Figure 2 a), which adapts the 3 x 3 convolutional layer in Residual Block (ResBlock) into our CT-Module. It can achieve both convolutional efficiency and feature- interaction sufficiency. Second, we equip our simple CT-Block with the TE mechanism in Figure 2). forming full CT-Block that can improve the cooperative power of all the feature dimensions. Besides, extra point-wise convolutions are added between different cub-operations, which are ben- eficial for more sufficient feature interaction. At last, we build up a novel CT-Net with CT-Block. As shown in Figure 2 c), we replace one of every two ResBlocks with CT-Block in every stage. Such method guarantees a better balance between efficiency and accuracy in our experiments.
Discussion. In fact, the popular methods in video classification like C3D, R(2+1)D and CSN (Tran ct al. 2015) 2018 2019) can viewed as special cases of our CT-Net. We can generate different forms by adjusting three
the number of sub-dimensions (K), the corresponding dimension size (C&) and the spatial-temporal kernel size (Kernel,). To degenerate into C3D, we Kernel,
3 3. When K 2, Kernel 1 and C₁ = C2 = C' without channel tensorization. it becomes R(2+1)D. Unfortunately. because of lacking the decomposition of channels, C3D and R(2+1)D still have a large computational load. When K 2, C = C₁ x C2 C x 1, Kernel, = 1 x 1 x 1. Kernelz 3 x 3 3, obviously it equivalent CSN. However, CSN has limited receptive field of apatial-tempont
can
1 and Kernel,
1
interaction. In our CT-Net, we utilize channel tensorization and perform tensor separable convolution along each sub-dimension turn. Such design can not only preserve interaction among spatial, temporal and channel dimensions but also enlarge the receptive field of feature interaction progressively.
4 EXPERIMENTS AND RESULTS
Datasets and implementation details. We conduct experiments on three large video benchmarks: Kinetics-400 (Carreira & Zisserman, 2017). Something-Something VI and V2 (Goyal et al. (017b). We choose ResNet50 and Rl (He et al., 2016) pre-trained on ImageNet as the backbone and the parameters of CT-Module are randomly initialized. For training, we utilize the dense sam- pling strategy (Wang et al. 2018) 3) for Kinetics and sparse sampling strategy (Wang et al. 2016) for Sameethng-SSomethine
Random scaling and cropping applied for data argumentation. Finally, we resize the cropped regions to 256 x 256. For testing, we sample multiple clips per video (4 for Kinetics, 2 for others) for pursuing high accuracy, and average all scores for the final prediction.
5

---

Published as a conference paper at ICLR 2021
Method
3D Convolution x w)
GFLOPs
Top-1
Top-5
C3D-Module Tran et al. 2015
Full: x
59.9
75.0
R(2+1)D-Moohe) Module Tran of al. 2018 CSN-Module
Full 1x 3 + Full: 3x1x1
45.8
47.0
76.1
Tran et al. 2019
Full: 1x1x DW 3x3x3
35.6
46.8
75.7
Our C1-Module
1x3x3 +x11x1
36.3
47.3
76.2
(a) Effectiveness of CT-Module. CT-Module outperforms the recent modules for video modeling.
Number
GFLOPs
Top-1
Top-5
Type
Top-1
Top-5
C2
GFLOPs
Top-1
Top-5
ID
45.8
46.5
75.6
coupling
47.1
76.0
45.9
47.0
76.1
2D
36.3
47.3
76.2
serial
47.2
76.1
37.6
46.8
76.0
3D
35.7
47.1
75.8
parallel
47.3
76.2
47.2
76.0
4D
35.6
46.5
75.3
LvC]
36.3
47.3
76.2
Connection type spatiotemporal convolution.
(c)
(b) Number of sub-dimensions. The higher the dimension, the smaller the GFLOPs. 2D channel tensorization achieves the best trade-off.
(d) Dimension size. C' C1 x C2 and the best trade-off is achieved
The parallel connection
between spatial and temporal
when adopting the rounded middle
convolution is best choice.
size LvC].
Number
GFI.OPs
Top-1
Top-5
Kerne size
GFLOPs
Top-1
Top-S
(TSN)
43.0
16.9
42.0
Ci 1
35.5
46.1
75.1
+1 stage5
41.9
42.3
71.1
C2 1x3xx 3
3x
+4
stage4-5
38.9
46.9
75.7
C1 1
TxlxI
36.6
47.2
76.2
+6
Stage3-5
37.1
47.2
76.1
C2 1x5x5
5x1x1
stage2-5
36.3
47.3
76.2
C1 1x
3x1x1
36.3
47.3
76.2
+12 stage2-5
31.5
45.9
76.1
C2 1x333
3x1x1
C1 C
3
(e) Number and location of CT-Blocks. Simply replacing block stage5 can bring significant
C2 1x5x5
5x 1
37.4
47.5
76.4
CI 1x5x5
5x1x1
performance improvement. As we replace more blocks from the bottom the GFLOPs contin- to decrease. Replacing 7 blocks achieves the
C2 1x5x5
5
38.9
47.6
76.5
(1) Kernel sizes along different dimensions. The larger ker- nel size brings improvement with more calculation.
best trade-off between accuracy and GFLOPs.
Model
GF-LOPs
Top-1
Top-5
Input
GFLOPs
Top-1
Top-5
Baseline (TSN)
43.0
16.9
42.0
Train: 224 x 224
28.6
49.1
77.4
+CT-Module
36.3
47.3
76.2
Test:
224 x 224
CT-Module+PPWCnn
37.2
48.0
76.7
Train: 224 x 221
37.3
49.7
77.7
CT-Module-PPConn--EE
37.2
48.8
77.4
Test:
256 x 256
+ "I-MMedule-PWCCov-IE
37.3
50.1
78.8
Train: 256 x 256
37.3
50.1
78.8
Test:
256 x 256
(g) Impact of different modules. CT-Module is essential for temporal modeling and TE mechanism is also beneficial.
(h) Impact of different spatial resolution.
Table 2: Ablation studies on Someehinng-Somthhing V1. All models use ResNet50 as the backbone
We follow the same strategy Non-local (Wang et al. 2018) to pre-process the frames and take crops of 256 256 input. Because some multi-clip models in Table 3 and Table 4 sample crops of 256x 256, we simply multiply the GFLOPs reported in the corresponding papers by (256/224) for fair comparison. When considering efficiency, use just 1 clip per video and the final crop is scaled to 256 x 256 to ensure comparable GFLOPs.
4.1 ABLATION STUDIES
sable our ablation studies on Something Soobeiiigg V1, which is challenging dataset that requires video architecture to have robust spatial-temporal representation ability and is suitable to verify the effectiveness our method. All models use ResNet50 as the backbone.
Effectiveness of CT-Module. In Table 2a different modules in recent methods (Tran et al., 2015 2018 2019). Compared with CSN-Module, module achieves better result with similar computation. which reflects the importance of suf- ficient feature interaction. Besides, is slightly better than R(2+1) D-MMulle with much lower cal- culation, showing the necessity of efficient convolution. Such results demonstrate the effectiveness of our CT-Module. illustrates our two design principles give preferable guidance for designing an efficient module for temporal modeling.
we replace the 3 3 convolutional layer in ResNet50 with

---

Published as a conference paper at ICLR 2021
SomethingVi
Something V2
Method
Backhone
#Frame
GFLOPs
Top-1
Top-5
Top-1
Top-5
EC Zolfaghari et al.
2018
Incep43D RI8
92
267
46.4
NL 13D + GCI Wang & Cupta. 2018
3D R50
32x3x2
1818
46.1
76.8
ir-CSN (Taa ct 2019
R101
12x1x10
738
48.4
ir-CSN Tran et 2019
R152
32x1x10
49.3
ConNNet Wang
2040
R50
32x1x10
1150
49.3
ISN (Wang et al. 2016
2D R50
8
33
19.7
46.6
27.8
57.6
TSM [Lin al. 2019
R50
g
33
45.6
74.2
59.1
85.6
bLV
2019
bLR50
24
46.4
76.6
59.1
86.0
TRINet (Liu at 2020
2D R50
8
33
47.4
61.3
TEA eL
20206
2D Res2Ne150
8
35
48.9
78.1
PEM-TDLOSS Weng ot al.
2020
2D R50+TIM
8
33
49.8
62.6
PFM+TDI Loss
(Weng of al.
2020
R50+TIM
8x3x2
259
50.4
63.5
CT-NNtt
R50
50.1
78.8
62.5
87.7
CT-Net
R50
8x3x2
224
51.7
80.1
63.9
88.8
ISN (Wang et al. 2016
2D R50
16
66
19.9
47.3
30.0
60.5
TSM Lan of at. 2019
R50
66
47.2
77.1
63.4
88.5
bLVN
2019
bLR50
16x2
48.4
78.8
61.7
88.1
TEINet
(Liu
2D R50
16
66
49.9
62.1
TEA
(Li
2020b
2D Res2Ne150
16
70
51.9
80.3
PEM+TDI loss
Weng of al.
2020
2D R50+TIM
16
666
50.9
63.8
PEM+TDLoss Weng el al.
2020
2D R50+TIM
16x3x2
517
52.0
65.0
Our CT--Net
R50
75
52.5
80.9
64.5
89.3
CT-Nee
R50
16x3x2
53.4
81.7
65.9
90.1
Our CT-NeteN
2D (R50)x4
8+12+16+24
280
56.6
83,9
67.8
91.1
Table 3: Comparison with the state-of-the-art Something-Something V1& V2. Our CT- 6etio/ outperforms all the single-clip models in Something Something-Something and even better than most of the multi-clip models. And our CT-NeteN outperforms all methods with very lower computation.
Number of sub-dimensions. Increasing the number of sub-dimensions saves a lot of computation, but the corresponding accuracy first increases and then decreases as shown Table Compared with the ID method, the 4D method significantly reduces GFLOPs, achieving comparable accuracy. As for the decrease of accuracy when is too large, we argue that the number channel in the shallow layer is small (64/128), thus there are too few channels in single dimension, leading to insufficient feature-interactin interaction. Since the 2D method obtains the best trade-off, we set = 2 in all the following experiments.
Connection type of spatiotemporal convolution. The coupling 3 x 3 x 3 convolution can be decomposed into serial or parallel spatial/temporal convolution. Table Table that factorizing the 3D kernel can boost results as expected. Besides, the parallel connection is better, thus we adopt parallel connection as the default.
Dimension size. As we set K = 2, it is essential to explore the impact of changing the dimension size C2. We demonstrate that the computation is the lowest when C1 C2 VC. Since C' is not always a perfect square number, we adopt the rounded middle size [vC]. Table shows that when C2 [vC,, the model not only requires the fewest computation cost but also achieves the best performance. Hence, we set C2 = LVC] naturally.
Number and location of CT-Blocks. Table 2e illustrates that simply replacing block in stage5 can bring significant performance improvement (16.9% vs. 42.3%). As we replace more blocks from the bottom up. the GFLOPs continues to decrease. Moreover, the bottom blocks seem to be more beneficial to temporal modeling, since replacing the extra 3 blocks in stage2 and stage3 only improve the accuracy by 0.4% (46.9% vs. 47.3%). Since replacing 7 blocks achieves the highest accuracy, we replace 7 blocks by default.
Kernel sizes along different dimensions. In Table 2f we can observe that two concatenated 3 convolution kernels are slightly better than those with the same receptive field (1*+5"). Furthermore, the larger kernel size can bring performance improvement but more calculation. reveals that our CT- avoids the limited receptive field of feature interaction, and it can progressively enlarge the receptive field of such interaction on all the dimensions. Considering a better trade-off between accuracy and computation, choose concatenated 3 convolution kernels.

---

Published as a conference paper at ICLR 2021
Method
Backbone
#Frame
GFLOPs
Top-1
Top-S
R(2+DD Tran 2018
2D Inception Inception
x2
1520=152x 10
72.0
91.4
TSN [Wang et al. 2016
25x10x1
800=80x 10
72.5
90.2
13D
Carreira Zisserann
2017
64XN/AXN/A
108xN/A
71.1
89.3
TSM [Lin et al. 2019
2D R50
16x3x10
2680 x00
74.7
TEINa [Tau et al. 2022
2D R50
16x3x10
2580=86x30
76.2
92.5
bLV TAM (ran ei al.
2019
bLR50
(16x2)x3x3
561=62.3x9
72.0
90.6
TEA eL al. 20206
2D Res2Ne150
16x3x10
2730=91 x30
76.1
92.5
PEM+TDLasS Weng al.
2020
2D R50+TIM
16
2580=86x30
76.9
93.0
ConNNet [Wang ct 2020
3D
x2x1x10
1150=115x10
77.2
SlowFast [eeccteenhofer
2019
3D R50+R50
36=(4+32)xxxx 10
1083=36.1x30
75.2
91.5
SlowFast FFichtchhoter
3D R50+R50
40=(8+32)x3xxD
1971=65.7x30
76.4
92.2
Our CT-Net
2D R50
16x3x4
895-74.6x12 12
77.3
92.7
X3D-XL (Feichtenhofer 2020
16x3x10
1452=48.4x:0
79.1
93.9
Nge al.
2020a
2D R101
32x3x4
6552=546x12
77.4
93.3
ip-CSN
Tran ei al.
2019
3D RI01
32x3x100
2490=83.0x30
76.8
92.5
ip-CSN
Tran ct al.
2019
3D R152
32x3x10
3264=108.8x30
77.8
92.8
CorNeet Wang et a 2020
3D R101
32x3x10
6720=224x30
79.2
SlowFast
Feichtenhofer el
2019
3D RIOI-RIOI
40=(8+32)x3xx
3180=106x30
77.9
93.2
SlowFast [Fiicheenhofer cl
2019
3D R101+RI01
80=(1666))33x10
6390=213x30
78.9
93.5
NL Wang Gupta
2018
3D RIO1
128x3x10
10770=359x30
93.3
CT-Nel
2D RIOT
16x3x4
1746=145.5x
78.8
93.7
Our CT-Netgx
2D R50+R101
(16+16)x3x4
2641=220.1x12
94.2
Table Comparison with state-of-the-ar on Kinetics-400. It shows that CT-Net-R50aa can surpass all existing lightweight models and even SlowFast-R50np When fusing different models, our model 2.4x faster than SowFFattRO1... and shows an 0.9% performance gain.
Impact of different modules and different spatial resolution. In Table 2g our CT-Module can significantly boost its baseline (16.9% vs. 47.3%) and the TE mechanism can further improve the accuracy by 2. % (48.0% vs. 50.1%). The extra point-wise convolution also boosts performance, which demonstrates that beneficial for sufficient feature interaction. Compared with the SE mechanisms, our TE mechanism focuses more on features in different sub-dimensions individually, thus effectively enhancing spatial-tempore features. In our experiments, ensure GFLOPs is com- parable with other methods, we crop the input to 256 x 256 during testing. Table 2h shows both training and testing with a larger spatial resolution input bring clear performance improvement.
4.2 COMPARISONS WITH THE STATE-OF-THE-ARTE
Something-SSomthiing V1&V2. We make a comprehensive comparison in Table 3 Compared with NL I3D+GCN22f, CT-Neta, gains 4.0% top-1 accuracy improvement with 49.1x fewer GFLOPs in Something-Sometting V1. Besides, our CT-Nets (51.7%) is better than the ir-CSNa2f (49.3%) which adopts ResNet- 152 as the backbone. Moreover, the single-clip models in Something-Someethnng V1&V2 and even better than most of the multi- clip models. It illustrates that our CT-Net is preferable capture temporal contextual informa- tion efficiently. Surprisingly, with only 280 GFLOPs, our ensemble model CT-NetgN achieves 56.6%(67.88%) top-1 accuracy in Something-
CT-Net16/ outperforms all
VI(V2), which outperforms all methods.
Kinetics-400. Kinetics-400 is a large-scale sence-related dataset, and the lightweight 2D models are usually inferior the 3D models on it. Table shows our CT-Net-R50 can surpass all existing lightweight models based on 2D backbone. Even compared with SlowFast-R500o/ our CT-Net- R501ef also achieves higher accuracy (77.3% vs. 76.4%). Note that our reproduced SlowFast- R50 performs worse than that in the paper (Feichtenhoier et al. 2019), which may result from the missing videos in Kinetics-400.. CT-Net-RI011e; 16f requires 3.7x fewer GFLOPs but gains comparable results (78.8% vs. 78.9%). Besides, it achieves comparable top-1 accuracy with X3D-XL (78.8% vs. 79.1%) under a similar GFLOPs. However, X3D requires extensive model searching with an expensive GPU setting, while our CT-Net can be trained traditionally with feasible computation. We perform score fusion over CT-Net-RS0zef and CT-Net-R R101 which mimics two-steam fusion with two temporal rates. In this setting, our model is 2.4x faster than SlowFast-BR Fast--10--- and shows an 0.9% performance gain (79.8% 78.9%) but only uses 32 frames.
for the deeper model, compared with SlowFast-R10n sof, our
8

---

aob-dimensions. Published as a conference paper at ICLR 2021
T T=3 T=4 T=5
T=6
T=7
7
T=1
T=2
T=3
T=4
5-5
T
7
CT-Axx
CXV
CT-NNst Twisting something(V)
CT-Net Holding something next something(
R(2 + 1)0 = Puiling two ends of something but nothing happens(*) (x)
RC2 1.2.111 1)0 - Snowing something next to something(X) CSN = Holaing something in front sometning
CSN > Pulling two ends of something so that it seporates oato two pieces(x)
Figure 3: Comparison of visualization. Videos sampled from Comething-Something V1. Com- pared with R(2+))D and CSN, our CT-Net can localize the action and object better both in space and time thanks to the larger spatial-temporal receptive field.
4.3 VISUALIZATION
We use Saliency Tubes (Stergiou et al. 2019) to generate the visualization, for it can show the most discriminative features that the network locates. Figure we sample two videos from Something- Something V1 which requires complex temporal modeling. In the left example, our CT-Net focuses on larger area around the towel, especially the fourth and fifth frames, thus predicting that someone is twisting it. In contrast, R(2+1)D only concentrates the wrong judgment. The same situation can be seen in the right example. We argue that CT-Net can localize the action and object accurately thanks to the larger spatial-temporal receptive field. As for CSN, the regions of interest seem to be scattered, because it lacks sufficient apatial-tempora interaction, thus ignoring the rich context both in space and time.
one side of the towel and gives
CONCLUSIONS
In this paper, construct efficient tensor separable convolution learn the discriminative video representation. We view the channel dimension of the input feature as a multiplication of K sub- dimensions and stack spatial/temporal tensor separable convolution along each Moreover, CT-Module is cooperated with the Tensor Excitation mechanism to further improve per- formance. All experiments demonstrate that our concise and novel CT-Net obtains preferable balance between accuracy and efficiency on large-scale video datasets. Our proposed principles are preferable guidance for designing an efficient module for temporal modeling.
ACKNOWLEDGEMENT
This work partially supported by National Natural Science Foundation of China (6187617, U1713208). the National Key Research and Development Program of China (No. 2020YFC200480D), Science and Technology Service Network Initiative of Chinese Academy of Sciences (KFJ-STS-QYDE QYZX-992-- Shenzhen Institute of Artificial Intelligence and Robotics for So- cicty.
REFERENCES
João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724- 4733, 2017.
Quanfu Fan, Chun-Fu Chen, Hilde Kuehne, Marco Pistoia, and D. Cox. More is less: Learning efficient video representations by temporal aggregation module. NeurIPS 2019, 2019.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. 2020 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 200-210, 2020.

---

Published as a conference paper at ICLR 2021
Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6201 6210, 2019.
Priya Goyal, P. Dollár, Ross B. Girshick, P. Noordhuis, L. Wesolowski, Aapo Kyrola, Andrew Tul- loch, Y. Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. ArXiv, Abs/1706.02677/ 2017a.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilo,, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. 2017 IEEE International Conference Computer Vision (ICCV), pp. 18843-5557 2017b.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Jie Hu, Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitationn networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 42:2011-2022 2020.
Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. 2019 IEEE International Conference on Computer Vision (ICCV), pp. 2000-2009, 2019.
A. Karpathy, G. Toderici, Sanketh Shetty, T. Leung, R. Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014.
Hilde Kuehne, Hueihan Jhuang, E. Garrote, T. Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. 2011 International Conference on Computer Vision, pp. 2556-2563, 2011.
X. Li, Yali Wang, Zhipeng Zhou, and Yu Qiao. Smallbignet: Integrating core and contextual views for video classification. 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1089-1098, 2020a.
Yinong Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. ArXiv, abs/2004.01.35 2020b.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. 2019 IEEE International Conference on Computer Vision (ICCV), pp. 7082-7092, 2019.
Zhaoyang Liu, D. Luo, Yabiao Wang, L. Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Tong Lu. Teinet: Towards efficient architecture for video recognition. ArXiv, abs/1911.09443, 2020.
I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Zhaofan Qiu. Ting Yao, and Tao Mei. Learning spatio-tempora representation with pseudo-3d residual networks. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5534 5542, 2017.
K. Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014.
K. Soomro, A. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. ArXiv, abs/1212.0402. 2012.
Alexandros Stergiou, G. Kapidis, Grigorios Kalliatakis, C. Chrysoulas, R. Veltkamp, and R. Poppe. Saliency tubes: Visual explanations for spatio-temporal convolutions. 2019 IEEE International Conference on Image Processing (ICIP), pp. 1830-1834, 2019.
Swathikiran Sudhakaran, S. Escalera, and O. Lanz. Gate-shift networks for video action recognition. 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1099-1108, 2020.
10

---

are set Published as a conference paper at ICLR 2021
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4189-4497,2000
Du Tran, Hong xiu Wang. Lorenzo Torresani, Jamie Ray. Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. 2018 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6450-6459, 2018.
Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. 2019 /EEE/CVF International Conference Computer Vision (ICCV), pp. 5551-5560, 2019.
Heng Wang, Du Tran, L. Torresani, and Matt Feiszli. Video modeling with correlation networks. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 349-358, 2020.
L. Wang, Yuanjun Xiong, Zhe Wang, Y. Qiao, D. Lin, X. Tang, and L. Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV,
X. Wang and A. Gupta. Videas as space-time region graphs. In ECCV, 2018.
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7794-7803, 2018.
Junwu Weng, D. Luo, Yabiao Wang, Ying Tai, Chengjie Wang. Jilin Li, Feiyue Huang. Xudong Jiang. and J. Yuan. Temporal distinct representation learning for action recognition. ArXiv, has/2007.076266 2020.
Saining Xie, C. Sun, J. Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accurge trade-offss in video classification. In ECCV, 2018.
Mohammadreza Zolfaghari, K. Singh, and T. Brox. Eco: Efficient convolutional network for online video understanding. ArXiv, abs/ hts//804409966,, 2018.
A APPENDIX
A.1 MORE TRAINING DETAILS
We SGD with momentum 0.9 and cosine learning rate schedule (Loshchilov & Hutter 2017)
train the entire network. The first 10 epochs are used for warm-up (Goyal et al. [l.] 2017aa to overcome early optimization difficulty. For kinetics, the batch, total epochs, initial learning rate, dropout and weight decay are set to 64, 1 10, 0.01, 0.5 and Ic-4 respectively. All these 64, 45, 0.02, 0.3 and 5e-4 respectively for Something-
A.2 TENSOR EXCITATION MECHANISM
The implementation of our Tensor Excitation is shown in Figure Different from the SE module, we use tensor separable convolution in the TE mechanism. Moreover, when obtaining the spatial attention, squeeze the temporal dimension and perform spatial tensor separable convolution, because temporal information is insignificant for spatial attention and vice versa. We add the Batch Normalization (BN) layer for better optimization.
A.3 RESULTS ON UCF101 AND HMDB51
To verify the generation ability of our CT-Net smaller datasets, conduct transfer learning experiments from Kineticy400 to UCF101 (Soomro et al., 2012) and HMDB-51 (Kuehne et al. 2011). We test CT-Net with 16 input frames and evaluate over three splits and report the averaged results. As shown in Table our CT-Netje achieves competitive performance when compared with the recent methods, which demonstrates the generation ability of our CT-Net.
11

---

Published as a conference paper at ICLR 2021
X.Gxx x G,
M.G.xxxX G G.
M.G. G x G.
spatial
Temporat Pooling
Spacial G- NGomp
Spatial
Sigmoid
N. G.T.A.
feature
Gi- Ge-TSCoRU
spatiotemporal feature
temporal feature
Spatial Pooling
Temporal TSomm
Temporal
AN
Siymoid
Ge TSCoRV
N.G, x
f.1.11
Xoox Go
Pooling
Sputiul
Point-wise CComp
Point-wiss
GSCTS
Sigmoid
output f feature
N.C.T.RUU
N.G. x x x- x. P.1.
NcC 2 Gg X.xX Gg.
B.G, Kink Kork G.T.11
Figure 4: The implementation of our Tensor Excitation (TE) mechanism.
Method
Backbone
Pretrain
UCFI01
HMDB51
C3D Tran et al. 2015)
3D VGG-IT
Sports-IM
82.3
51.6
3D(CCareina & Zisserman
2017
3D Inception
ImageNet+Kineee Kinetics
95.1
74.3
ECO Zollaghar ct al. 2018
Inception+3D R18
ImageNet+ Kinetics
94.8
72.4
TSN
Wang
2016
Inception
MoageNet+Kinettee
91.1
TSM et al. 2019
R50
BeggeNet<Kinetiee
94.5
70.7
STMarang et al. 2019
2D R50
ImageNet* Kinetics
96.2
72.2
Our CT-Net
2D R50
ImageNet+<
96.2
73.2
Table Comparison results on UCF101 and HMDB51.
A.4 MORE RESULTS ON SOMETHING-S SOMHHIINGG-SMETINNG V1&V2
V1&V2. We train CT-Net with different number of input frames and then test these models with different sampling strategies. We average the prediction scores obtained from the previous models evaluate the ensemble models. With more input frames, the corresponding accuracy becomes higher. As for the reason that CT-Net24r worse than CT-Net16f. we argue that is because the model is hard optimize with too many input frames. Sampling more clips or more crops also boosts performance. Moreover, our ensemble models gain the ofate-of-the-art top-1 accuracy of 56.6%(68.3 on Something--ommething V V1(V2).
Table shows more results
A.5 MORE ABLATION STUDIES ON MINI-KINETICS AND COMETHING-SOMETIINN V2
To verify the effectiveness of module comprehensively, also conduct experiments in Mini- Kinetics and Something-Somen Something V2 and report the multi-clip accuracy and single-clip accu- racy respectively. Mini-Kinetics covers 200 action classes and is a subset of Kinetics-400, while Something-SSmetting V2 covers the same action classes as Comething-Somettien
V1 but contains more videos. As shown Table 7 the performance trend of different modules is similar to that shown in Table [a] Since
does not highly depend on temporal modeling, the gap becomes smaller but still demonstrates the effectiveness of our CT-Module.
A.6 ADAPTING DIFFERENT PRE- TRAINDD IMAGENET ARCHITECTURES AS CT-NET
In fact, by directly replacing the 3x3 convolution with CT-Module, ferent pre-trained ImageNet architectures as CT-Net. Table g shows that it is also sensible to use Inception V3 the backbone. We believe that through more elaborate design, CT-Net based different backbones can achieve comparable performance.
easily adapt dif-
A.7 VALIDATION PLOT
we plot the accuracy vs per-clip GFLOPs on Kinetics-400. It reveals that our CT-Net achieves a better trade-off than most of the existing methods on Kinetics-400.
In Figure 5
12

---

Published as a conference paper at ICLR 2021
SomethingVi V1
Something V2
Method
#Frame
GFLOPs
#Param
Top-1
Top-5
Top-1
Top-5
37
50.1
78.8
62.5
87.7
12
56
52.1
80.0
63.9
88.7
21.0M
16
75
52.5
80.9
64.5
89.3
24
112
52.5
80.9
64.6
89.1
8x1x2
75
51.6
79.7
63.5
88.5
12x1x2
52.8
80.6
64.6
Our CT-Net
21.0M
16x1x2
53.2
65.2
89.7
24x1x2
224
52.9
81.3
65.0
89.3
8x3x2
224
51.7
80.1
63.9
88.8
12x3x2
336
53,0
81.1
65.3
89.6
21.0M
16x3x2
53.4
65.9
24x3x2
672
53.6
81.6
65.5
89.8
8+16 16
112
54.4
82.0
66.2
90.4
(8 + 12
16
21)
x
280
56.6
83.9
67.8
91.1
Our CT-Neten
83.8M
(8 12 16
24
x
560
56.6
84.0
67.8
91.3
+
21)
3
x
1679
56.6
83.9
68.3
91.3
Table 6: More results on Conething-Somethingg V1&V2.
Mini-Kineties Top-1
Something
Method
Backbone
GFLOPs
Top-5
Top-1
Top-5
C3D-Module TTran al. 2015
2D R50
59,9
77.5
93.0
59.1
85.5
R(++l\D-Modlle (Tran ct al. 2018)
2D R50
45.8
77.8
93.2
60.0
86.0
CSN-Module (Tran et al. 2019
2D R50
35.6
77.6
93.2
59.5
86.0
Our CT-Module
2D R50
36.3
78.0
93.6
60.3
86.4
Table 7: More ablation studies
and Something-Something V2.
Method
Backbone
GFLOPs
HParam.(M)
Top-1 1
Top-5
Baseline (TSN)
2D ResNet-50
43.0
23.9
16.9
42.0
Our CT-Net
2D ResNet-50
37.3
21.0
50.1
78.8
Baseline (TSN)
Inception V3
45.8
22.1
18.3
43.9
Our CT-Net
Inception V3
43.9
20.9
47.2
76.1
Table 8: Adapting different pre-trained ImageNet architectures as CT-Net.
R50 R101
R101
R101
16 x R101
8 8, R101
R152
R50
R50
Tap-1 Accuracy (%)
R101
8 R50
10
16, R50
R50
CT-Net
CornNet
SlowFast
ip-CSN
TSM
R50
bLVNet-TAM
GFLOPs/Clip
Figure 5: Accuracy vs per-clip GFLOPs on Kinetics-400.
13


Reviews:
[{"review_id": "zFqWIT5YG3Z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new CNN module to learn video feature representations for action recognition, with a particular focus on increasing channel interactions for spatio-temporal modeling. To achieve that, the authors propose to divide feature channels into several sub-dimensions (called channel tensorization) and then perform group convolutions at each sub-dimension sequentially to improve channel interactions. An SE-like attention mechanism is also applied to further enhance feature representation. The proposed approach achieves competitive results on Kinetics400 and Something-Something, compared to some existing SOTA results. The paper also provides detailed ablation studies on the approach.\n\nMy understanding is that the main idea of the paper essentially performs channel shuffling followed by group convolutions at each sub-dimension. From this perspective, the idea is similar to ShuffleNet, which applies channel shuffling to enhance interactions between channel groups. While this provides interesting technical questions from the algorithmic perspective, from the point of view of the novelty, the paper does not appear as a strong technical contribution.\n\nAnother downside of the proposed approach is that it is not sufficiently validated by experiments. Firstly, the contribution of TE is not separated in Table 3 and 4, so it remains unclear whether the performance improvement is due to the enhanced channel interactions or channel attention. Secondly, the approach seems to be only effective when the number of dimensions is low, leaving it difficult to fully justify the advantages of high-dimensional channel tensorization claimed in the paper.\n\nThe FLOPs of the proposed approach in Table 4 is somewhat misleading. As indicated in the work of X3D, the performance gain from using more clips (i.e. >5x10) in evaluation is small. Nevertheless, most approaches are evaluated with 30 clips on Kinetics. I would suggest sticking to this general practice for fair comparison.  \n\nTable 3 and 4 miss some existing SOTA approaches such as X3D [1], CorrNet [3] and TAM [3], which should be listed for reference. \n\n1) Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 200–210, 2020.\n2) Heng Wang, Du Tran, Lorenzo Torresani, Matt Feiszli, Video Modeling With Correlation Networks, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020 \n3) Fan, Q.; Chen, C.-F. R.; Kuehne, H.; Pistoia, M.; and Cox, D. 2019. More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation. In Advances in Neural Information Processing Systems, 2261–2270.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "s32ZrryDCFb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel Channel Tensorized Module (CT-Module) to construct an efficient tensor separable convolution and learn the discriminative video representation. The proposed solution achieves a preferable balance between convolutional efficiency and feature-interaction sufficiency. The experiments were conducted on 3 benchmark video-classification datasets outperforming the state-of-the-art results. \n\nStrengths:\n- The paper's novelty is the first method for exploring the spatial/temporal tensor separable convolution along each sub-dimension.\n- The CT-Network framework is rigorously presented.\n- Experimental results outperforming state-of-the-art results.\n- The Paper is well written and easy to follow.\n\nWeaknesses:\n- The abstract is too long.\n- Results of the Sport1M dataset (standard video classification dataset) are missing.\n- Table 2 looks clumsy and hard to understand the results of sub-tables.\n- What are the reasons for choosing something-something V1 dataset for ablation studies? How about the ablation experiments on Kinetic and something-something V2 dataset?.\n- Since earlier modes were having better performance on the Resnet-152 model, the authors could have experimented with the Resent-152 model as well. Why only 50 and 101?.\n- This paper needs careful proof-reading as there are major typos across the paper.\n\n\nMajor Comments:\n- Why do authors adapt ResNet as proposed CT-Net? Any reason for this? \n- Can we adapt other pre-trained Imagenet architectures as CT-Net?\n- The dataset used for the comparison of visualization in Fig.3 is missing.\n- If authors can showcase the performance of proposed models on other domain datasets (say 4D fMRI or 3D medical imaging datasets), it will be more interesting.\n- validation plots are missing: the accuracy vs channel-interaction plot and the accuracy vs G-Flops per clip.\n\nTypos:\n-  Page7:  In our experiments, to ensure GFLOPs is comparable with other methods. we crop the input to 256 × 256 during testing. (Sentence continuity is missing, and check “we”).\n\nAddressed Concerns:\n- corrected the typos.\n- Majority of the weaknesses are addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5TUhCk5okxt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript proposes a novel convolutional operation for learning representations from video data. By decomposing the channel dimension into sub dimensions in the typically 4D video data (Time, Channel, Width, Height), one defines spatial-temporal separable convolution for each sub-dimension. This could improve the representation learning in term of efficiency and modeling quality. \n\nPro: the experiments seem quite thorough. \n\nCon: i) I'm not fully convinced that it is beneficial to decompose the channel dimension, which is usually only 3 and relatively small compared with the other dimensions. Would it be possible / sensible to apply tensorization on the time dimension? ii) The ablation study could have included the comparison between only channel tensorization and channel tensorization and self attention, which would highlight the new contribution of the paper. \n\nI'd be happy to update my rating if the authors would elaborate these two points. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2hNI1Dds3gi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes a new architecture for lightweight action classification networks, named Channel Tensorization Network (CT-Net). The idea of this architecture is the tensorization of mid-level input features in combination with an attention mechanism that allows to select relevant features. the channel tensorization can be used as intermediate building block e.g. in a ResNet alternately with a Res block.\nThe ablation study is done on Something-Something V1 and the overall method is compared to state-of-the-art on Something-Something V1&V2 and Kinetics-400 and is able to outperform other lightweight architecture with comparable size.\n\n\nPaper strengths\n\n- I think the ideas proposed in this paper are interesting and I would not be aware of any architecture that would use a similar arrangement of tensorization and attention in the mid-layer part to allow for a lightweight 3D convolution architecture.\n\n- The paper shows competitive results on Something-Something as well as on Kinetics-400\n\n- The visualization supports the claim that the attention mechanism seems to learn to focus more on relevant parts of the video clip.  \n\n\nPaper weakness\n\n- The paper does not introduce a new technique per se, but manages to combine known elements. So there might be the issue of limited novelty. \n\nConclusion\n\nI think that the overall idea of reducing the computational load in 3d convolution architectures is a valid and important topic, as simply increasing networks and training data is not working for the majority of tasks. Therefore, lightweight but powerful architectures have a need in the field. I understand that the proposed elements have only a limited novelty, but I think that the clever integration in this case might justify a publication in this case.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on the task of learning efficient representation models for video classification. To avoid the excessive computational cost of performing 3D convolutions on video, the authors propose to break the channel dimension of video representations into sub-dimensions that are treated separately. This cuts down on computation and improves classification performance over many methods in the literature. Extensive experiments were run on well-known benchmarks to justify the claims of the model. Such backbone architectures can be very useful in the realm of video understanding. The authors should be commended for the amount of work they did in the rebuttal period to address the comments and inquiries brought up by the reviewers. Extra experiments were done and more in-depth analysis was made possible. 
Decision: Accept (Poster)