## 4 Experiments
### Main Results  
**Effectiveness of TSNet. ** We compare TSNet with the original UniFormer-S (baseline) and the random selection method, which randomly samples the same number of tokens using the selection policy produced by our TSNet. The comparison focuses on different selection ratios set as 0.8, 0.5, and 0.3. We denote the variants of methods by method/\(r\), where \(r\) refers to the selection ratio. From the obtained results, as shown in Table 1, it is apparent that even when a small number of tokens are removed, the classification accuracy does not fall notably, implying that some tokens can be entirely redundant, contributing minimal value to the final classification. These results accentuate the importance of strategically lessening computational burdens and improving processing speed by selecting the most informative tokens to process. Our TSNet framework perform competitively, delivering similar results but with a lesser computational requirement--reducing computation by approximately 13.4%. As the selection rate of tokens decreases, the computational workload will significantly decrease while only incurring marginal accuracy loss. Even when 70% of input tokens are pruned, there is a significant reduction in computation by 55.5% in terms of GFLOPs, with a manageable accuracy loss of only 1.7%. When the token selection rate is 0.8, the random method only experiences a 1.5% decrease in accuracy. This, once again, demonstrates that some tokens in the transformer structure are redundant, as a portion of tokens can be completely replaced by another set of tokens while maintaining a certain number of tokens. TSNet's notable superiority over random selection methods is evident when operating under comparable computational budgets. For instance, TSNet/0.5 significantly outperforms Random/0.5 by 49.8%, while utilizing approximately the same computation measured at 99.2 GFLOPs. Further, when a substantial number of tokens are discarded, as in the case of Random/0.3, the accuracy result is extremely low. This supports the hypothesis that our dynamic token selection module more effectively preserves informative tokens as opposed to a random approach. Importantly, the incremental computational overhead added by token selection module is marginal. In fact, the parameters and FLOPs of the scorer network are a minute proportion (0.038% and 0.022%, respectively) compared to those incorporated in the original UniFormer-S backbone. This speaks to the significant computational efficiencies that our TSNet model presents. **Comparison with state of the art. ** Table 2 provides a comprehensive comparison of TSNet with state-of-the-art video recognition models on the Kinetics-400 dataset. The comparison includes both CNN-based and transformer-based models. In order to showcase the generalizability of our approach across different transformer architectures, we use both UniFormer-S [17] and VideoSwin-T [12] as the base models. For clear comparison, we separate the models into three groups. Our TSNet demonstrates favorable trade-offs between complexity and accuracy. X3D takes the modification and decomposition of 3D models to the extreme and is a comprehensive 3D CNN model. However, while X3D-M has similar computational complexity to UniFormer-S, its performance differs by 4.8%. By implementing our method to further reduce the computational workload of UniFormer-S, we achieve better accuracy than X3D with less than half the computational requirements. Notably, our TSNet(UniFormer-S)/0.5 achieves results comparable to TimeFormer, but requires only 2% of the computational cost. TSNet(UniFormer-S)/0.3 outperforms MViT-B16 [13] while saving over 80% in computation. It is important to highlight that for backbone models UniFormer-S and VideoSwin-T, our TSNet is capable of saving 13\(\sim\)55% of the computational cost with less than a 3% drop in accuracy. Compared to UniFormer-S, it appears that applying our method on VideoSwin-T might result in a more significant performance decline.