## 3 Methods
### Token Sparsification  
Given a sequence of input tokens \(X\in R^{L\times D}\), where \(L\) is the number of total patches and \(D\) is the number of total channels, we adopt a linear layer to generate an importance score for each token based on input. Formally, given the input to \(l\)-th block \(Z_{l}\), the select policy matrix for this block is computed as follows:  
\[Score=Z_{l}W_{l} \tag{1}\]  
where \(W_{l}\) denotes the projection weight. The \(Score\) is subsequently passed to a sigmoid function, yielding the probability of keeping tokens. The ensuing decisions, exhibiting a binary nature, offer a straightforward mechanism to determine whether to activate or inactivate tokens by merely establishing an inferential threshold. However, this process lacks differentiability during training, thereby hindering the optimization potential of the decision network. To this end, we leverage the Gumbel-softmax [25] technique to draw samples from the output of the \(Score\) network. Formally, the decision of \(Score\) is derived as  
\[M_{k}=\frac{\exp(\log(Score_{k}+G_{k})/\tau)}{\sum_{j=1}^{K}\exp( \log(Score_{j}+G_{j})/\tau)} \tag{2}\] \[\text{for}\quad k=1,2,\ldots,K\]  
where \(K\) is the total number of decision categories. In our case of binary decision, \(K=2\). \(G_{k}\) is the Gumbel distribution with the formula \(-\log(-\log(U_{k}))\), where \(U_{k}\) is a random variable sampled from the uniform distribution. The parameter \(\tau\) represents a smoothing  
Figure 1: The overall diagram of the proposed method. The introduced token selection module is placed between the transformer blocks to effectively choose tokens based on the features generated by the preceding layer. \(\gamma_{s}\) and \(\gamma_{t}\) denote the selection ratio and target ratio, respectively. \(\mathcal{L}_{ce}\) denotes the cross-entropy loss. \(\mathcal{L}_{\gamma}\) is devised to reduce the overall computational cost. Best viewed in color. constant. However, in the process of token elimination, the original size of the feature maps can be disrupted. This disruption occurs due to the alteration of the feature space topology caused by the removal of specific tokens. Moreover, the attempt to simply zero out the tokens that are intended to be dropped is found to be unproductive. Specifically, the conventional single-head self-attention is computed as follows:  
\[R=QK^{T}/\sqrt{d} \tag{3}\]  
\[Attn_{i,j}=softmax(R)V=\frac{\exp(R_{i,j})}{\sum_{k=1}^{L}\exp(R_{i,k})}V\quad 1 \leq i,j\leq L \tag{4}\]  
where \(Q\), \(K\), and \(V\) are query, key, and value matrices, respectively, and \(d\) is a scaling factor. Obviously, the attempt to simply zero out the tokens that are intended to be dropped is found to be unproductive. Notwithstanding their nominal absence, these zeroed tokens can still influence the remaining tokens because of their contribution to softmax operation, which basically normalizes scores into a probability distribution. This is due to the overall sum of the exponentiated elements in the softmax denominator including these zero elements, thereby affecting the ultimate distribution. To resolve this issue, a strategy involving the use of a mask is implemented. By applying a mask, the interactions between the pruned tokens and the remaining tokens can be effectively reduced. This mask essentially acts as a filter that shields certain tokens from being operated upon, thereby preserving the integrity and independence of the tokens that are not to be pruned. It provides a way to deal with the impracticality of merely zeroing out tokens and thus creates a more effective method for the pruning process. Formally, we compute the output of self-attention by  
\[Attn_{i,j}=\frac{\exp(R_{i,j})M_{j}}{\sum_{k=1}^{L}\exp(R_{i,k})M_{k}}V\quad 1 \leq i,j\leq L \tag{5}\]  
The output of masked attention is computed by concentrating on the active tokens during training. For pruned tokens, we adopt linear projection to keep the token number unchanged during inference.