## 2 Related Work  
**Video vision transformers.** The impressive achievements of vision transformers (ViTs) in image processing [7] have served as a catalyst for numerous research endeavors focused on spatiotemporal learning in videos. VTN [10] introduces a method for action recognition by incorporating a temporal attention encoder onto existing 2D spatial networks. This combination leverages self-attention mechanisms on high-level features. In the pursuit of enhancing efficiency, ViViT [14] explores different variations of factorizing the pure-Transformer model across spatial and temporal dimensions. This primarily entails investigating input embedding methods and self-attention mechanisms for spatial and temporal dimensions. TimeSformer [15] delves into the study of several self-attention schemes and proposes that factorized space-time attention yields the most effective results for video recognition. MViT [13] draws inspiration from the multiscale pyramid structure in 2D CNNs and presents a hierarchical structure that expands the channel capacity while concurrently reducing spatial resolution. Video Swin Transformer [12] applies self-attention in local windows, reducing a significant amount of cost compared to global attention calculations. UniFormer [17] combines 3D convolutions and spatiotemporal self-attention mechanisms to improve efficiency. However, despite the outstanding performance of vision transformers, they still have high computational costs, which escalate rapidly with an increasing number of patches. This challenge serves as a driving force for exploring redundancies within videos, with the aim of achieving efficient video recognition.  
Efficient video recognition methods are in high demand due to the computational complexity associated with comprehensive studies in this field. Compact models for video recognition have been developed through the compression of 3D CNNs, as explored in multiple studies [27; 28; 29]. Although these approaches yield substantial memory savings, they usually necessitate examining each temporal clip within the input video, which results in no decrease in computational complexity. Recent approaches [30; 31; 32] have proposed to enhance video recognition efficiency by selecting the most salient video frames as input. However, it is important to note that these approaches primarily concentrate on accelerating CNN-based video recognition models. In contrast, the SpatioTemporal Token Selection (STTS) framework [33] presents an innovative approach for efficient video recognition that dynamically chooses informative tokens separately in the temporal and spatial dimensions rather than simultaneously. [PERSON] et al. [34] utilized the greedy K-center search, in which patches that are farthest apart from each other in terms of spatial distance are iteratively selected. This strategy forces the model to sample patches that are distinct. As a result, this model can learn features with less redundant information. Furthermore, TokenLearner [35] aims to accelerate ViTs by learning a limited quantity of tokens through the aggregation of the entire feature map, weighted by a dynamic attention map dependent on the feature map. TokenLearner can be viewed as an advanced tokenization approach for input samples. However, our research diverges from TokenLearner as it concentrates on the progressive selection of informative tokens.