## 1 Introduction  
Exploring action recognition has always held a significant place in the field of video understanding. Past research trends indicate that many researchers have favored using 3D convolutional neural networks (CNNs) or 2D CNNs equipped with complex temporal information modules [1; 2; 3; 4; 5; 6]. However, these convolution-based networks have a notable issue: they possess a limited receptive field, implying that they might struggle to capture broader, global information dependencies when processing videos. In contrast, visual transformers have received widespread attention in recent years [7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17], primarily because they have demonstrated superior performance over other networks in a series of benchmarktests. However, these transformer-based networks have inherent problems too. They are typically very computationally expensive because the transformer structure necessitates calculating attention relationships between all input tokens, causing the computational cost of the self-attention mechanism to grow quadratically. To address this issue, some of the latest research in the image processing field has begun attempting to simplify the computational process [18; 19; 20; 21; 22; 23; 24]. Specifically, the core idea of these methods is to retain only those tokens that genuinely contribute to the final prediction while ignoring other irrelevant tokens. While this approach has been validated in the image domain, applying it to video transformers might pose new challenges. This is because current designs of video transformers, especially those integrated with convolutional modules and shift windows, usually process the input tokens hierarchically. In such designs, discarding tokens arbitrarily could disrupt the network's internal feature map dimensional structure. Therefore, to successfully implement this simplification strategy in video processing, we need to adopt a more refined approach in selecting the most informative tokens, thereby maintaining the network's structural integrity. In essence, this means that when removing unnecessary tokens, it is imperative to ensure that the network can continue to function efficiently without compromising its performance. In this paper, we propose an end-to-end framework that adaptively determines token usage for exploring the sparsity of hierarchical models. In particular, we propose a selection network to dynamically determine which tokens to select for the transformer block. The selection network is designed as a single-layer perceptron and is capable of predicting the importance score for each token. Importantly, this selection network can be seamlessly integrated into any transformer block within the model architecture. Even though the selection network adds to overall computational overhead, the extra cost generated is substantially outweighed by the savings gained by masking uninformative tokens. Traditional binary decision methods, due to their nondifferentiable nature, make end-to-end training highly challenging. To overcome this, we employed the Gumbel-softmax [25] trick, which converts binary decisions into differentiable continuous decisions, enabling end-to-end training. We adopted the mask method instead of directly setting the redundant tokens to zero vectors, as this would affect the calculation in the self-attention block. The redundant tokens are projected by a linear layer to keep the number of the total tokens for downstream processing. The transformer backbone network, together with the selection network and projection module, are jointly optimized with a normal cross-entropy loss for classification and a usage loss that encourages reducing of the overall computational cost. The effectiveness of the proposed method is evaluated on the Kinetics-400 [26] using UniFormer-S [17] and VideoSwin-T [12] as the backbones. The results indicate that a remarkable trade-off is attained in terms of increasing speed while simultaneously diminishing accuracy.