No Time to Waste: Squeeze Time into Channel for Mobile Video
  Understanding
Authors: name: Yingjie Zhai; name: Wenshuo Li; name: Yehui Tang; name: Xinghao Chen; name: Yunhe Wang
Published: 2024-05-14 06:32:40

Abstract: Current architectures for video understanding mainly build upon 3D
convolutional blocks or 2D convolutions with additional operations for temporal
modeling. However, these methods all regard the temporal axis as a separate
dimension of the video sequence, which requires large computation and memory
budgets and thus limits their usage on mobile devices. In this paper, we
propose to squeeze the time axis of a video sequence into the channel dimension
and present a lightweight video recognition network, term as
SqueezeTime, for mobile video understanding. To enhance the temporal
modeling capability of the proposed network, we design a Channel-Time Learning
(CTL) Block to capture temporal dynamics of the sequence. This module has two
complementary branches, in which one branch is for temporal importance learning
and another branch with temporal position restoring capability is to enhance
inter-temporal object modeling ability. The proposed SqueezeTime is much
lightweight and fast with high accuracies for mobile video understanding.
Extensive experiments on various video recognition and action detection
benchmarks, i.e., Kinetics400, Kinetics600, HMDB51, AVA2.1 and THUMOS14,
demonstrate the superiority of our model. For example, our SqueezeTime achieves
+1.2% accuracy and +80% GPU throughput gain on Kinetics400 than prior
methods. Codes are publicly available at
https://github.com/xinghaochen/SqueezeTime and
https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SqueezeTime.