## 4 Experiments
### Ablation Studies  
We conduct ablation studies with UniFormer as backbone on Kinetics-400 to assess the contribution of our proposed TSNet. **Effectiveness of sparsification in MSA and FFN**. To evaluate the effectiveness of dynamic sparsification in both MSA and FFN, we conducted ablation studies using a selection rate of 0.5, with results demonstrated in Table 3. Our methodology is contrasted with a variety of token selection strategies, including (1) MSA_only, a strategy that exclusively samples tokens within the MSA block; (2) FFN_only, a strategy that restricts token sampling within the FFN; and (3) MSA/FFN, a methodology that equally samples tokens within both the MSA block and FFN using an identical selection policy. The resulting data suggest that dynamic sparsification proves beneficial within both MSA and FFN. Further, our proposed TSNet, which samples tokens within both the MSA block and FFN using different selection policies, provides an optimum balance between complexity and accuracy. Notably, TSNet leads MSA/FFN, implying the significance of utilizing diverse selection policies for the dynamic selection of tokens. **Redundancy vs. depth**. Our tests on the redundancy of patches in different layers or blocks indicates that the deeper the block is, the more patches that can be removed without significantly impacting performance. In our experimentation, we only sparse the tokens in the last two layers, which, respectively, consist of eight and three blocks. For the sake of simplicity, blocks 1, 5, 9, and 11 of the total 11 blocks are created as sparsifications. The same selection ratio is maintained when individual blocks are trimmed, with a scale setting chosen as 0.1. The data presented in Table 4 provide insights into the accuracy of models that have undergone pruning based on patch selection within specific blocks. The results indicate that removing patches from lower blocks tends to result in a substantial decrease in accuracy. This suggests that the patches in lower blocks contribute significantly to the overall performance of the models. Interestingly, when examining the sparsity within blocks belonging to the same layer (such as blocks 1 and 5 and blocks 9 and 11 in this particular case), the impact on accuracy was found to be relatively minor. This implies that the redundancy of patches within the same layer is relatively similar. On the other hand, redundancy of patches varies significantly among different layers, with deeper layers demonstrating higher redundancy. This can be attributed to the attention mechanism, which aggregates the characteristics of different patches, leading to a full communication between deeper patches. This essentially means that features of patches in deeper layers are more redundant since they have captured and shared more information through the attention mechanism. Therefore, these patches can be pruned without losing critical information, which justifies the better performance of our pruning strategy in deeper layers. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Depth** & **Top-1 (\%)** & **Top-5 (\%)** \\ \hline
1 & 79.4 & 93.3 \\
5 & 79.5 & 93.5 \\
9 & 80.5 & 94.5 \\
11 & 80.7 & 94.6 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Ablation studies on redundancy with respect to depth. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **GFLOPs** & **Top-1 (\%)** & **Top-5 (\%)** \\ \hline UniFormer-S & 167 & 80.8 & 94.7 \\ MSA\_only/0.5 & 121.4 (\(-27.3\%\)) & 79.8 & 93.3 \\ FFN\_only/0.5 & 140.5 (\(-15.8\%\)) & 80.2 & 94.1 \\ MSA/FFN/0.5 & 98.4 (\(-41.0\%\)) & 79.3 & 93.6 \\ TSNet(UniFormer-S)/0.5 (Ours) & 99.2 (\(-40\%\)) & 79.5 & 93.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Ablation studies on Kinetics-400 to assess different sparsification strategies. **Different location for token selection**. The flexibility of TSNet allows us to reduce the similar computational cost by selecting different token selection configurations.