Learned Thresholds Token Merging and Pruning for Vision Transformers
Authors: Maxim Bonnaerens
Joni Dambre
Venue: Accepted by TMLR ()

Abstract: Vision transformers have demonstrated remarkable success in a wide range of computer vision tasks over the last years, however, their high computational costs remains a significant barrier to their practical deployment.
In particular, the complexity of transformer models is quadratic with respect to the number of input tokens. 
Therefore techniques that reduce the number of input tokens that need to be processed have been proposed. 
This paper introduces Learned Thresholds token Merging and Pruning (LTMP), a novel approach that leverages the strengths of both token merging and token pruning.
LTMP uses learned threshold masking modules that dynamically determine which tokens to merge and which to prune.
We demonstrate our approach with extensive experiments on vision transformers on the ImageNet classification task.
Our results demonstrate that LTMP achieves state-of-the-art accuracy across reduction rates while requiring only a single fine-tuning epoch, which is an order of magnitude faster than previous methods.

Published in Transactions on Machine Learning Research (08/2023)
Learned Thresholds Token Merging and Pruning for Vision Transformers
Maxim Bonnaerens ATRO, Ghent University imec
masim. Sanaam.bbonneeeeeenngnn.... be
Joni Dambre IDRo Chent Universily imec
joni. Shw.edammbeeeugen.....m Sugentt.
Reviewed on OpenReview: http:://ooerrrvssss.eeee//ooottt
Abstract
Vision transformers have demonstrated remarkable success in a wide range of computer vision tasks over the last years. However, their high computational costs remain significant barrier to their practical deployment. In particular, the complexity of transformer models quadratic with respect to the number of input tokens. Therefore techniques that reduce the number of input tokens that need be processed have been proposed. This paper introduces Learned Thrcsholds token Merging and Pruning (LTMP), novel approach that leverages the strengths of both token merging and token pruning. LTMP uses learned threshold masking modules that dynamically determine which tokens to merge and which to prune. We demonstrate our approach with extensive experiments on vision transformers on the ImageNet classification task. Our results demonstrate that LTMP achieves state- of-the-art accuracy across reduction rates while requiring only single fine-tuning epoch, which is an order of magnitude faster than previous methods. Code is available at https: Trgithuub.com/Mbbnee
input image
patchified image
layer
layer
layer 3
layer
layer
layer 6
layer 7
layer
layer 9
layer 10
layer 11
layer 12
Figure Visualization the merging and pruning as applied to image patches. In every layer the most similar tokens are merged and any unimportant tokens are pruned. The visualizations show the remaining tokens after every layer in DeïT-S.
1 Introduction
The adoption of transformers (Vaswani et al. 2017), originally developed for natural language processing, the field of computer vision with Vision Transformers (ViT) (Dosovitskiy al. (021)) has led significant progress in the field. But despite the impressive results of vision transformers, their success has come with

---

Published in Transactions on Machine Learning Research (08/2023)
cost; ViT models computationally expensive and require larger datasets (e.g. ImageNet- 21k instead of ImageNet-1k (Deng et al. 2009)) and prolonged training times (Dosovitskiy et al. 2021 In order to benefit from their accuracy in downstream tasks and applications, the use of pretrained models has become essential. However, their adoption on resource-constranned devices such as mobile and embedded platforms remains limited due to the high computational cost.
To reduce the computational cost of transformers, previous work has focused on techniques such as distillation (Wu et al., 2022), quantization (Liu et al. 2021b) and pruning. Pruning techniques have explored pruning model weights (Gordon et al. 2020), attention heads (Voita et al. 2019 and input tokens (Rao et al. 2021). This last approach is very effective, as the model complexity of a transformer is quadratic to the number of tokens. In vision transformers, the tokens are non-overlapping patches of an image, e.g. & token may represent patches of 16 x 16 pixels. Token pruning has attracted research interest as it matches our intuition that not all parts of an image are equally important. The self-attention mechanism transformers, due to its ability to process variable length inputs and its order-agnostic characteristic, enables unstructured reduction of the number of tokens between layers. This was previously non-trivial with convolutional networks.
Most token pruming approaches caleulate an importance score for every token in each layer and remove the least important tokens. While token pruming has been shown to be an effective compression technique, removing tokens results in loss of information which limits the amount of tokens that can be pruned. Tn order to recover from this loss of information, most pruning approaches require substantial retraining to be effective. Additionally, some recent pruning techniques have incorporated token combining techniques where the pruned tokens are combined into a single token that aggregates the information that would otherwise be lost. (Kong et al. )022).
Token merging (ToMe) (Bolya et al. 2023) takes this combining technique one step further. ToMe exclusively combines pairs of tokens into new tokens rather than pruning them. This has as advantage that it does not discard but summarizes information, leading to better accuracy while being equally effective in reducing computational complexity.
In this work, we introduce Learned Thresholds token Merging and Pruning (LTMP). Our approach combines the benefits of token merging, which allows to combine rather than discard token information, with pruming, which allows us to remove uninformative tokens. To the best of our knowledge, this is the first work to extensively combine these two reduction techniques, which leads to improved accuracy compared to previous work. Our approach uses learned threshold masking modules, which allow the model to learn thresholds that determine which tokens to prune and which ones to merge in cach layer. This enables adaptive token reduction, while only requiring two learnable parameters per transformer block. As a result, our approach converges within a single epoch, reducing the fine-tuning cost by an order of magnitude compared to other learnable pruning approaches.
Our contributions can be summarized follows:
We propose to combine token merging with token pruming, enabling us to achieve high token reduc- tion rates with minimal loss of accuracy.
Our method introduces learned threshold masking modules, which require only two learnable pa- rameters per transformer block, allowing our approach to converge within a single epoch.
We optimize the thresholds using novel budget-aware training loss for which we introduce reduc- tion target Ttarget and an actual FLOPs reduction factor TFLOPe. This allows us to create models of any size and allows the model to freely distribute the reduction operations across layers.
2 Related work
2.1 Efficient Vision Transformers
Initially, transformers (Vaswani et al. 2017 were adopted from NLP to computer vision (Dosovitskiy al. 2021) for their impressive accuracy. But despite their success in many vision tasks, ViT-based models could
2

---

Published in Transactions on Machine Learning Research (08/2023)
not compete with lightweight CNNs for deployment resource-constraincc platforms (Wang et al. 2022) To create efficient. VIT models, several architecture changes have been proposed which modify the expensive attention mechanisms (Kitaev et al.) 2019 Chen et al. 2021 Liu et al. 2021a Li et al. 2021). In this paper, we look at token pruning, but other pruning approaches that have been successfully applied to transformers are, among others, weight pruning (Gordon et al. 2020: and atlention heads pruning (Voila et al. 2019
2.2 Token Pruning
The flexibility of transformers with respect to the sequence length and order of the inputs allows token pruning, something which was previously non-trivial do in convolutional-based models. Token pruning methods can differ in various ways, such as the score used to determine the importance of each token.
Pruning methods can also differ in the way that. loken reduction is applied. In fixed rate pruning (Goyal
et al. 2020 Rao et al. 2021 Bolya al., 2023 Liang et al. 2022 Xu et al. 2022) predefined number
of tokens is removed per layer, while in adaptive approaches [Kim et al. 2022
Yin et al. 2021 Liu ct al.
2022
the Lokens are pruned dynamically based on the input.
The most recent pruning approaches do not only prune tokens but they also create a single additional token after cach pruning step. This token aggregates the information of the pruned tokens and limits the loss of accuracy while pruning. EVIT (Liang el al. 2022). ; Evo-Vil qXu et al. 2022) and SPVIT (Kong el al. 2022) create the new fused token.
use weighted average based the importance
2.3 Token merging
Token Merging (ToMe), introduced by Bolya et al. (2023) introduces lightweight token matching algorithm to merge similar tokens. It is last as pruming while being more accurate. ToMe partitions all tokens into two sets A and B of roughly equal size by alternating and calculates similarity scores for each token in A with every token in I. The similarity score is defined as the cosine similarity between the key vectors (K) used in the self-attention layer. The final similarity score of a token in A the highest similarity score with any token in B. Based on this score, ToMe merges the k most similar tokens through averaging and concatenates the two sets back together.
3
Learned thresholds token merging and pruning
3.1
Overview
An overview of our framework is shown in Figure 2 (LTM) and pruning (LTP) components with learned threshold masking modules in each transformer block between the Multi-head Self-Attention (MSA) and MLP components. Based on the attention in the MSA, importance scores for each token and similarity scores between tokens are computed. Learned threshold masking modules then learn the thresholds that decide which tokens to prune and which ones to merge.
Given any vision transformer, our approach adds merging
3.2 Motivation
Although token merging is generally more accurate than pruning as it combines tokens instead of dis- carding them, it is not always better to merge tokens instcad of discarding them. In some cases, it may be more beneficial to prune an unimportant token rather than merging the most similar tokens, the similarity between them may not be very high.
In this section, we explore whether token merging and token pruning are techniques that can be com- bined. Figure 3 visualizes tokens kept by pruning and by merging on one specific image, we observe that the kept tokens noticeably different. between
Figure 3: Visualization token pruning (middle) com- pared token merging (right). The visualization shows the remaining tokens after the 10-th layer in VIT-B when pruning merging 16 tokens per layer.
3

---

Published in Transactions on Machine Learning Research (08/2023)
similenty SCORPS
ippotanca scores
Learned Ahresheld masking modale
Learned threshol masking module
Muiti- head- Seit-
MLP
(2,00
KCUN
KOU
KOUN
KOUN
LTM
LTP
Figure 2: Overview of our approach. LTMP contains a merging and pruning component, each with a learned threshold masking module. The components are added between the Multi-head Self- Delf-Attention and MLP components of each transformer block.
both approaches. To quantify the relation between merging and pruning, we calculated the Kendall tau rank correlation [Kendall] 1938) between the imporlance scores used token pruning and the similarily scores used in token merging. However, tion 2.3). we compute the Kendall tau correlation between the similarity scores and the importance scores of only the tokens in set A. We calculated the correlations over 1000 images using ViT-B where each layer pruned and merged 8 tokens in fixed top-k manner and report the results in Table 1 We find that the t correlations between both scores are low, especially in the early layers where the most important merging and pruning is done. We, therefore, propose combining token merging and token pruning.
merging only calculates similarity scores for tokens in set (see See-
Table 1: Kendall t correlation between importance scores in token pruning and similarity scores in token merging. The correlations are calculated over 1000 images using ViT-B where each layer pruned and merged 8 tokens.
layer
5
9
0.01
0.14
0.21
0.19
0.15
0.22
3.3
Learned thresholds
3.3.1 Learned thresholds pruning
Our learned thresholds approach conceptually similar to learned token pruning as introduced in Kim al.) (2022). In each transformer block importance score is calculated for every token x,, í {1, {1,..., n), where = hee is the number of tokens]] A threshold 0 E TR, {1,..., T.), where L is the number of transformer blocks, determines which tokens to keep and which to prune in each layer; only tokens with an importance score above the threshold are kepl.
In order to prune tokens adaptively, we introduce threshold masking module that, given the importance scores s R", learns pruning threshold gi and outputs which tokens to keep.
-(0
1, if 0
0()
(1)
otherwise
We omit the [CLS] class token simplicity, during pruning and/or merging we always keep the [CLS] taken.
4

---

Published in Transactions on Machine Learning Research (08/2023)
However, in order to make 0' learnable during training, the threshold masking module needs to be differ- entiable. We achieve this by implementing the threshold masking module as straight-through estimator [Bengio et al. 2013), where we estimate the masking function during backpropagation as
(s 0¹) =o o Ø
(2)
where (x) is the sigmoid function and T is the temperature hyperparameter.
During inference we only keep the tokens in the /-th block where (s 0¹) = 1. However, during training, we can not simply drop tokens as that does not allow the model backpropagate the infiuence of the threshold the model performance. We, therefore, create mask indicating which tokens are prumed. Every threshold masking module only updates the entries of the mask for the tokens that have not yet been removed prior to that layer, tokens that are pruned in an earlier layer have to remain pruned. We construct the pruning mask m' [0, 1 as follows:
kept and which ones
(s 0'),
if m = =1 otherwise
m
(3)
m 1-
The learned token pruning implementation in Kim et al. (022) multiplies its mask with the tokens in order to create valued tokens. However, these tokens do not remain zero due to bias terms in MLP layers; furthermore adding zero valued tokens changes attention calculations compared to removing those tokens. Instead, our approach makes changes to the only place where tokens influence each other: the attention mechanism.
Recall the original formula for attention Vaswani al. 2017
QK I
Attention(Q, (Q. V) soottmax
In order to avoid that the masked tokens inftuence the attention mechanism, propose modified function:
Autention with mask Q2 K, V, m) SV
(5)
where,
exp(Ai;) m 1 S Kiijk <n
Sij
(6)
Σ exp(Ait)mm
and,
A = QK /de E
(7)
Equation 6 computes masked softmax, which is equivalent to softmax calculated with the pruned tokens removed. Attention_ with_ttthh_____ah mask is conceptually similar to the masked attention as found in the transformer decoder of language models. However, where the masking in transformer decoders is done by setting masked tokens to -00, our approach requires the influence of the straight-through estimator mask to propagate to the thresholds during backpropagation.
3.3.2 Learned thresholds merging
Token merging is originally top-k approach, meaning that it merges based fixed rate and has learnable parameters. We modify ToMe to thresholds instead of top-k by applying the same techniques as introduced in Section 3. this is by adding our learned threshold masking module, in which similarity scores above these thresholds are selected for merging, and by changing the attention function to Equation
Teechniallyy, tokens also influence other during layer normalization, however pruning done pretrained models. simply the global statistics from pretraining during normalization.
5

---

Published in Transactions on Machine Learning Research (08/2023)
3.3.3 Learned thresholds merging and pruning
With learnable thresholds, it is trivial combine merging and pruning, as we can simply add a learned threshold masking module that learns thresholds for importance scores and another module that learns thresholds for similarity scores.
3.4
Training Strategy
3.4.1 Training objective
To effectively reduce the number of tokens in the transformer blocks, it is necessary to include a regularization loss term in the training process. Without this loss, the model has no incentive to prune any tokens and the pruming thresholds will simply be set to 0 as the most accurate model uses all inputs. We propose budget- aware training loss which introduces a reduction target Ptarget for the FT.OPs of the vision transformer.
Let denote (module(1, d) as function that calculates the FLOPs of module based on the number of tokens and the embedding dimension d. The actual FLOPs reduction factor PFLOPs of a ViT can then be computed as:
Ópe(n, d) Φvir (n. (n,d) d)
Σ
OBLK(n, d) (SMSA (m n,,) dou.((m'n,d) (HEAD(m'nnd) Øvrr(n, ΦBLK(N, d) ΦBLK(N, d) (vrr(n,
(
(8)
TFLOPs
where m' E" m the percentage of input tokens that are kept after the I-th threshold masking operation and m 1. PE, BLK and HEAD denote the different components of vision transformer: the patch embedding module, the transformer blocks and the classification head.
As the vast majority of the FLOPs in vision transformer occurs in the transformer blocks (= 99% in ViT-S), we ignore the FLOPs in the patch embedding and classification head: (pp(n.d) *HEAD(n.d) That means that we can simplify
@vrr(n.d) ovrr(n.d)
ΦBLK(N,d) 1 Φvir(n, d) T
(9)
where L the number of transformer blocks.
The FLOPs of a transformer block and its two components, the MSA and MLP can be computed as:
(MsA(n, d) 4nd + 2ndd
(10)
ΦMLP(n, d) = 8nd2
(11)
ΦBLK (n, d)
ΦMSA (n, d) + (MLP(n, d) = 12nd + 2n²
(12)
Substituting Equations
12
into Equation S
gives:
2m nd + (m 1n)dd + 4m 'nd
FFLOPs ~
(13)
6nd2 n2d
Given this FLOPs reduction factor FFLOPs as a function of the threshold masks, we define our regularization loss the squared error between the reduction larget and the actual PLOPs reduction factor:
Lrog
(Ftarget - TFLOPs)
(14)
This regularization loss is then combined with the classification loss, for which we adopt the standard cross entropy loss.
- LCE + X
(15)
The overall training objective learn thresholds that optimize the model while redueing the model complexity a certain reduction target. The combination of learned thresholds and our budget-aware loss enables the model to optimally distribute merging and pruning across layers.

---

Published in Transactions on Machine Learning Research (08/2023)
3.4.2 Training schedule
LTMP only adds two learnable parameters per transformer block (one for pruning and one for merging). As is common pruning is applied to pretrained models. We therefore only updale the thresholds during training and keep all other trainable parameters fixed, allowing LTMP converge within a single epoch.
DeiT-T
DeiT-B
DeiT-S
74
83
80.0
72
79.5
Accuracy
Accuracy
Accuracy
79.0
70
LTMP ToMe
78.5
80
LTMP TaMe
68
78.0
ToMe
79
Baseline
Baseline
Baseline
66
77.5
78
0.4
0.6
0.8
1.0
1.4
2.0
2.5
3.0
3.5 FLOPs
4.0
4.5
5.0
0.8
1.0
1.2
1.6
1.8
2.0
FLOPs
x10
x109
FLOPs
x10
Figure 4: Accuracy /FLurss /FLOPs trade-ofls for LTMP variants of DeiT- -Small and -Base. FLOPs are plotted in logarithmic scale. ToMe CeerracyFLOPP trade-offs are shown comparison.
4 Experiments
In this section, demonstrate approach through extensive experiments on the ImageNet- 1k classification task (Deng et al. 2009) using various VIT variants (Steiner et al. 2022 Touvron et al. 2021). All pretrained
models are taken from the timm PyTTrrc library (Wightman, 2022
All our experiments are trained for a single epoch, using SGD without momentum and a batch size of 128. The remaining training settings such as augmentations are set to the default values of timm. The hyperparameters that are introduced in LTMP are set T 0.1 and X = 10. As the importance scores and similarity scores have values in different ranges we separate learning rates for the thresholds in the pruning modules and the merging modules: 5- 10-6 for the pruming thresholds and 5 10-3 for the merging thresholds. To select these lyperparameters, the original ImageNet training set. The final hyperparameters are determined based on their performance on DeiT-Small with Ttarget set to 0.65. To show the robustness of these hyperparaneters, we did not change them between model variations and reductions targets (i.e. ViT-Tiny with Ttarget set to 0.45, is fine-tuned with the same hyperparameters)
used separate held-out validation set containing 2% of
4.1 Main results
As most other pruning approaches require extensive fine-tuning, the most commonly used baseline vision transformer in other works is the data-efficient vision transformer DeiT' (Touvron et al. 2021), for this reason, we also report on DeiT models.
In Figure A we show our approach applied DeiT-Tiny, -Small and -Base. For each model we vary Ttarget such that obtain set of reduced models, each with a different model size. Table 2 lists the detailed results for the DeiT models. Models used in one of the other tables in the paper are highlighted in gray We included Ttarget; such that our results can be reproduced. In Appendix A we additionally included results on the standard ViT models. All accuracy numbers are reported with 3 significant figures; brief analysis of the error ranges on the accuracy numbers can be found in Appendix
4.2 Comparison to other work
In Table 3
we compare the reported top-1 accuracy, FLOP and fine-tune epochs of other pruning ap- proaches to work. The other approaches compare with are SPVIT (Kong et al. 2022 DynamicVit
We follow the convention of reporting FLOPs multiply-adds.

---

Published in Transactions on Machine Learning Research (08/2023)
Table 2: Detailed results for DeiT models.
Model
Ttarge:
FLOPs
Accuracy
Model
Ttarget
FLOPs
Accuracy
Model
Ftarget
FLOPs
Accuracy
DefT-S (Baseline)
1.608G
79.9
DefT-T (Baseline)
1.258G
72.2
DefT-B B (Baseline)
17.583G
81.99
D
0.85
3.940G
79.8
DefT-T
0.85
1.01GG
72.1
DeiT-S
0.80
3.753G
79.8
DefT
0.85
15.007G
$1.9
DefT-T
10099
72.0
DefT-B
0.80
14.152G
81.9
DefI-T
0.75
0.974G
72.0
DefT-8
0.75
3.518G
79.8
DefT-B
0.75
403G
81.7
DeiT-S
0.70
3.326G
79.7
DeïT-T
0.70
0.918G
71.8
DeiT.S
0.65
3.155G
79.7
DeiT-B
0.70
12.571G
81.6
DefT-T
0.65
0.866G
71.7
DefT-B
0.65
11.731G
81.3
DeiI-T
0.60
0.813G
71.5
DefT-S
3.016G
79.6
De/T-B
0.60
11.022G
81.0
DefT-S
2.955G
79.5
DeïT-T
0.55
0.752G
71.0
DefT-B
0.55
10. 276G
80.3
DefT-T
0.50
0.703G
70.5
DeiT-S
0.55
2.722G
79.3
DefT-B
0.50
9. 8080
79.1
DeiT-T
DefT
0.50
2.523G
79.0
DefT'-8
0.45
DefTS
0.45
2.314G
78.6
Rao et al. 2021 EVIT (Lang et al. 2022 EvoViT (u el al. 2022 and ToMe (Bolya el al. 2023 Most
works report on a pruned model with around 3.0 GFLOPs, which for DefT-S corresponds to Starget Rd 0.65
in our approach. Only SPVIT reports on a different model size, which why compared separately. The results show that LTMP reaches state-of-the-att accuracy a fraction of the fine-tuning epochs required by other learnable methods. The accuracy of LTMP matches exceeds the accuracy of other token pruning approaches which require minimum of 30 fine-tune epochs. Only EVIT is able to reach a higher accuracy than LTMP, but only when drastically increasing the fine-tune epochs to 100, which is two orders of mag- nitude more than our approach. Because ToMe requires no fine-tuned checkpoints for comparisons, we are able to compare LTMP to ToMe more extensively over a wide range of model sizes. Figure 4 shows the ac- curacy/ Ptracy/FLOPs trade-offs for LTMP and ToMe. Our experiments show that LTMP consistently outperforms ToMe across model sizes.
Table 3: Comparison to other token reduction approaches on DeiT-S. Our method reaches state-of- shate-of-the-ar" accuracy with significantly fewer fine-tune epochs than other learnable approaches.
fine-tune epochs
Method
FLOPs
Accuracy
DeiT-S (Baseline)
4.6G
79.8
SPVIT
3.8G
79.8
75
LTMP (Ours)
3.8G
79.8
Dyna
2.9G
79.3
30
EVIT
3.0G
79.5
30
EVIT
3.0G
79.8
100
Evo-Vit VIT
3.0G
79.4
300
ToMe
3.0G
79.3
LTMP (Ours)
3.0G
79.6
4.3 Synergy between merging and pruning
In order to analyze the synergy between token merging and token pruning, we have examined the distribution of merged and pruned tokens across each layer of the vision transformer. The results, shown in Figure 5 for DeiT-S, reveal that token merging is the dominant reduction operation in the carly layers of the transformer. while Loken pruning is more prevalent in the final layers. This aligns with the findings of Bolya el al. (2023) that merging is more effective than pruning, as can summarize information. However, once all similar tokens are merged, it more beneficial to prune the least informative tokens rather than merging tokens that are as similar. In other words, the first layers are mainly used combine similar patches and once this is mostly done, token pruning removes the unimportant parts of the input.

---

Published in Transactions on Machine Learning Research (08/2023)
100
operation
80
merge
prune
40
20
0
1
2
4
5
7
8
10
11
12
layer
Figure 5: Distribution of the number of tokens (k) removed by the merging and pruming parts in each layer
of LTMP DeiT-S TFLOP, 0.5.
4.4 Design Choices
4.4.1
Importance score
A key component of pruning approaches is the importance score used to determine which tokens to remove. The two most common choices for the imporlance score are:
class attention score = Σ
yoi
(16)
j=1
where S hxxxa is the multi-headed extension of the attention softmax matrix (see Equation values at index 0 correspond to the [CLS] token, and
where
mean column attention score 8;
Syki
(17)
j=1 kml
which can be interpreted as the normalized amount that all tokens X₃ attend to token x, (Kim et al. 2022
The results Tab obtained from DeīT-S LTP models and show that the mean column attention score performs slightly better than the class attention score, but not by significant margin. For the remainder of this work, we use the mean column attention score (Equation 17 as the importance score.
4.4.2 Merging and pruning order
As shown earlier in Table I the correlation between the pruning importance score and the merging sim- ilarity score is low but not zero. This means that for small reduction target values r, the same token might hit the thresholds for both merging and prun- ing. In Table 4 we compare pruning followed by merging (LTPM) with merging followed by prun- ing (LTMP). The results confirm that the order has noticeable influence when the reduction rate is small, but once more tokens need to be removed LTMP is superior to LTPM. This is not surpris- ing as merging has been shown to be more accurate than pruning (Bolya et al. 2023) and that in token merging and pruning, more tokens get merged than pruned (see Figure 5
Table 4: Ablation of the design choices regarding the pruning importance score and the order in which to apply merging and pruning. All experiments are per- formed on DeiT-S variants.
FLOPs
2.3G
2.7G
3.1G
Accuracy
class attention column mean attention
76.3
78.2
79.2
76.5
78.2
79.2
LTPM
78.4 4
79.2
79.5
LTMP
78.6
79.3
79.7

---

Published in Transactions on Machine Learning Research (08/2023)
4.5 Ablation
Our approach has two important components: the learned thresholds and the combination of merging and pruning. In Table we ablate the main components of our approach on DeiT-S for different model sizes. We compare top-k pruning and merging, where tokens are pruned in each transformer block, learned thresholds variants. For merging, the top-k approach is equal to what is used in ToMe (Bolya et al. 2023 Additionally, compare merging and pruming individually with the combined approach.
The results show that learned thresholds improve the accuracy of pruning significantly, while for merg- ing the improvements are only marginal. This dif- ference in accuracy between learned thresholds and top-k can be explained by examining the distribu- tion of removed tokens shown in Figure B In this box plot, we compare the distribution of the number of removed tokens with LTM and LTP with uniform top-k pruming (i.c. the dashed line in the figure). As can be seen the figure, the distribution removed tokens with LTM close to the uniform top-k dis- tribution, which results in many of the same tokens being merged, while for LTP the distribution is no- tably dissimilar.
100
module
LTM
80
LTP
60
40
20
12
layer
Figure 6: Distribution of the number of tokens (k) re- moved LTP and LTM variants of DeiT-S TFLOPs 0.5.
We also observe that naively combining merging and pruning by applying both techniques with an equal fixed rate worse than only token merging.
This ablation shows that both components are es- sential in LTMP. Combining merging and pruning outperforms the individual techniques but only when using the learned thresholds to balance the merging and pruning.
Table 5: Ablation of the two main components of LTMP on DeiT-S: learned thresholds and combining merging with pruning.
Method
sctting
FLOPs
Accuracy
pruning
Top-k
k: = 16
2.3G
74.8
LTP
TFLOP: 0.5
2.3G
76.5
Top-k
3.0G
77.9
LTP
TFLOPs 0.65
3.0G
79.2
Merging
Top-k (ToMe)
k == 16
2.3G
78.1
LTM
TFLOPs = 0.5
2.3G
78.2
Top-k (ToMe)
11
3.0G
79.3
LTM
TFLOPS 0.65
3.0G
79.5
Merging & pruning
Top-k
8-8+8
2.3G
77.6
LTMP
TFLOPs 0.5
2.3G
78.6
Top-k LTMP
k-6+6
3.1G
79.0
TFLOPs = 0.65
3.0G
79.6
10

---

Published in Transactions on Machine Learning Research (08/2023)
4.6 Inference speed
Throughout this paper, we have reported FLOPs as complexity metric. While FLOPs are often regarded as a poor proxy for latency, it has also been shown that latency improvements on one type hardware often do not translate to similar improvements on other hardware, especially in mobile and embedded devices [Bonnacrens et al. 2022). As complexity improvements come from reducing the input tokens and both masking modules and pruning and merging implementations parallelized, believe FLOPs to be the best available metric to report complexity improvements.
Nevertheless, to demonstrate our approach, we have benchmarked it on mobile device using PyTorch's aetimize_for_moill. function and speed_benchmark Android binary. Table G shows the latency of the baseline DeiT-S and our LTMP (with rELOPs R 0.5) reduced variant. The benchmark conducted on Google Pixel and averaged over 200 runs (with 50 warm-up runs prior). The results show that the latency improvements, which achieve reduction of 49.52%, are nearly identical to the theoretical FLOPs improvements, which have reduction of 50.12%.
LTMP is also faster than ToMe while not only merging but also pruming. This likely comes from the argsort operator that used in top-k approaches such as ToMe and which is not well supported in many frameworks (Prillo & Eisenschlos 2020 Unfortunately, despite Evit, Evo- Vit and Dynamic Vit having an open-source PyTorch implementation, they use operations that are not supported by TorchScript which is required for the mobile speed_benchnngg tool.
Table 6: Latency benchmark Google Pixel 7.
Method
FLOPs
Latency
Accuracy
DeïT-S (Baseline)
4.6G
212 ms
79.8
LTMP (Ours)
2.3G
107 ms
78.6
ToMe
2.3G
118 ms
76.9
4.7 Limitations
Our learned thresholds approach requires a batch size of 1 during inference as each image is reduced differ- ently. This is not a limitation for most resturce-onstrained applications as they typically operate inference with a batch size of 1. If desired, our method could be extended to accommodate larger batch sizes by either incorporating masking to a common reduction size or by converting thresholds to the average number of tokens removed per operation and layer, and applying these values in a top-k adaptation. The former approach will result in higher computational complexity while the latter will result in lower accuracy.
input.
patchified
layer
layer
layer 3
layer 4 layer 5 layer
layer
layer 8
layer 9 layer 10 layer 11 layer 12
Figure 7: More visualization of the merging and pruning as applied image patches. In every layer, the most similar tokens merged and any unimportant tokens pruned. The visualizations show the remaining tokens after every layer in De
11

---

Published in Transactions on Machine Learning Research (08/2023)
4.8 Visualizations
In Figure transformer. It can be observed how similar parts of the image get merged and how unimportant parts of the image are pruned.
7 we illustrate the merging and pruning of the tokens they are processed through the vision
5 Conclusion
In this work, we introduced Learned Thresholds token Merging and Pruning (LTMP) for vision transformers. LTMP makes it possible to reduce the computational cost a vision transformer to any reduction target value with minimal loss accuracy. LTMP adaplively reduces the number of input lokens processed by merging similar tokens and pruning the unimportant ones. Our implementation uses learned thresholds which enable different merging and pruning rates between different images and allows the model to learn the optimal trade-off between merging and pruning across layers. As LTMP only introduces two learnable parameters per transformer block, our method is able to converge within single epoch, which is an order of magnitude quicker than other learnable approaches.
Acknowledgments
This research received funding through the Research Foundation Flanders (FWO- Vlaandeaneerrn)) under grant 1S47820N.
References
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 2013.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. International Conference on Learning Representations, 2023.
Maxim Bonnaerens, Matthias Freiberger, Marian Verhelst, and Joni Dambre. Hardware-aware mobile build- ing block ovaluation for computer vision. Applied Sciences, 12(24):12615, 2022.
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 357-366, 2021.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Databasc. In CVPR09, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP. pp. 143-155, 2020.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal. and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In ICML, 2020.
Maurice G Kendall. A measure of rank correlation. Biometrika, 30(1/2):81-99) 1938.
12

---

Published in Transactions on Machine Learning Research (08/2023)
Schoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22 pp. 784 794. Association for Computing Machinery, 2022.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2019.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Bin Ren, Minghai Qin, Hao Tang, and Yanzhi Wang. Spvit: Enabling faster vision transformers via soft token pruning. In Computer Vision ECCV 2022. Springer Nature Switzerland, 2022.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. CoRR, abs/2104 /bs/2204.05707, 2021.
Youwei Liang, Chongjian GE, Zhan Tong, Yibing Song, Jue Wang, and Penglao Xie. EVit: Expediting
vision transformers via token reorganizations. In International Conference Learning Representations, 2022.
Xiangcheng Liu, Tianyi Wu, and Guodong Guo. Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting sell-attention. orXiv preprinl arXiv:2209. 0002209.18802 13802, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Internaiional Conference on Compuler Vision, pp. 10012-10022, 2021a.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 84:28092-28103, 2021b.
Scbastian Prillo and Julian Eisenschlos. Softsort: A continuous relaxation for the argsort operator. In International Conference on Machine Learning, pp. 7793 7802. PMLR, 2020.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. NewrIPS, 2021.
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou, Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, July 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguisties, pp. 5797 5808, 2019.
Xudong Wang, Li Lyna Zhang, Yang Wang, and Mao Yang. Towards efficient vision transformer inference: a first study of transformers on mobile devices. In Proceedings of the 23rd Annual International Workshop on Mobile Computing Systems and Applications, pp. 7, 2022.
Ross Wightman. timm: Pytorch image models, scripts, pretrained weights, and more. https: hatss::/gitttub.oom wwigghhhwn/pyyoorrr-im-----ooooo 2022.
13

---

Published in Transactions on Machine Learning Research (08/2023)
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European Conference. on Computer Vision, pp. 68 85. Springer, 2022.
Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 2964 2972, 2022.
Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. Adavit: Adaptive tokens for efficient vision transformer. Xrxiv preprinl or xiv: 112 07658, 2021.
14

---

Published in Transactions on Machine Learning Research (08/2023)
A
ViT results
The main paper reports on DeiT models as they are most commonly used in related pruning works. However the standard pretrained VIT models as found in the timm library (Wightman 2022 Steiner et al. 2022 are more accurate than the DeïT models while having the same number of F..OPs. While DeïT models are often chosen because of their more efficient training, LTMP requires only single training epoch making the more accurate ViT models the preferred vision transformer to prune. We therefore also include results on ViT models here. Figure shows the Accuracy Securacy/F..PP, FFLOPs trade-off curve for LTMP reduced ViT models. We also included the DeiT models comparison. can be observed that VIT indeed often the better choice, except for the heavily reduced variants of the 'small' sized models, where the performance of VIT degrades faster than DeiT. More detailed listings of the results can be found in Table 7
85
ViT-T
DeiT-T
Accuracy
80
VTT-S
DeiT-S
x
BTT
DeiT-B
70
Baseline
LTMP
10
10
FLOPs
Figure 8: Accuracy/PLOPss trade-offs for LTMP variants of ViT-Tiny, -Small and -Base. DeiT variants are
also plotted as comparison. FLOPs are plotted logarithmic scale.
Table Detailed results for VIT models.
Model
FLOPs
Accuracy
Model
Ttarget
FLOPs
Accuracy
Model
Yearget
FLOPs
Accuracy
VIT-T (Baseline)
1.258G
75.45
ViT-S (Baseline)
4.608G
81.37
VIT-B [Baseline)
17.583G
84.54
VIT-T
0.85
1.094G
75.4
VIT-S
0.85
3.895G
81.4
VIT-B B
0.85
15.024
84.5
VIT-T
0.50
L.099G
75.1
VIT-S
0.50
3,689G
81.1
VIT-B
0.80
14. 158288
81.4
VIT-T
0.75
0.970G
75.3
VIT-S
0.75
3.495G
81.1
VIT-B
0.75
13-461G
84.3
V
0.70
0.911G
75.2
VIT-S
0.70
3.309G
80.9
VIT-B B
0.70
12.685G
84.1
VIT-T
0.65
0.855G
74.9
VIT-S
0.63
3.131G
80.5
VIT-B
0.65
11.975G
83.5
VIT-T
0.60
0.803G
74.6
VIT-S
0.60
2.915G
80.2
ViT-B
0.60
11.152G
83.2
VIT-T
0.737G
74.0
VIT-S
D.55
2.692G
79.8
VIT-B
0.55
10.250G
82.8
VIT-T
0.688G
73.0
VIT-S
0.50
2.501G
79.0
VII-B
0.50
9.559G
82.3
VIT-T
VIT-S
0.45
2,321G
VIT-B
8.841G
81.3
VIT-T
0.40
0.000G
70.4
VIT-S
0.40
2,248G
76.4
WIT-B
8.465G
80.3
B Accuracy error range
In Table report. the accuracy of 5 runs with different. random seeds for the main results in the paper (i.e. the data points highlighted in gray in Table 2 It can be observed that the error range the accuracy is about +0.3 for these data points. Wc, therefore, report the results in this paper with 3 significant figures.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Model
Itarget
FLOPs
Acc. #1
Acc. #2
Acc. #3
Acc. #4
Acc. #5
Acc. range
DefT-S
0.80
3.8G
79.816
79.802
79.824
79.794
79.814
0.030
Deil'S
0.62
3.0G
79.598
79.596
79.584
79.578
79.564
0.034
DefT-S
0.55
2.7G
79.262
79.266
79.242
79.274
79.254
0.032
DeïT-S
0.45
2.3G
78.608
78.604
78.588
78.574
78.558
0.050
Table 8: Accuracy ranges for different runs of the main data points in the paper.
16


Reviews:
[{"review_id": "73BzLJ6N6g", "reviewer": "Reviewer_4zvC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is easy to follow and well-written, and the motivation is clear to explain why both merge and pruning are complement each other. Moreover, the paper proposed a unified approach to perform merge and pruning.\n2. The training setup and ablation study for analysis is comprehensive. \n\nWeaknesses:\n1. The paper is the combination of two papers, which are cited in the paper: (1) ToMe for token merging and (2) LTP for token pruning. \n2. When comparing to ToMe, the improvement is marginal and it requires additional 1 epoch training, which means the algo needs to have the access of the dataset rather than using it out-of-box.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a combination of token merge and pruning to speed up the vision transformer; furthermore, the merge and pruning thresholds are learned by simply post-training the dataset with 1 epoch. The results are on par with ToMe ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. To better compare to ToMe, what is the performance of LTM (no P) at 3.0G? If is it still better than ToMe? \n2. Following the above 1, is it possible to compare LTP with Kim et al's method? \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wTeRlJaXPj", "reviewer": "Reviewer_iwJi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The paper overall is easy to follow.\n - The proposed method to select tokens is new.\n\nWeaknesses:\n\n- The token merging framework proposed in this paper is largely inspired by ToMe. However, although an extra token pruning method is introduced, the improvement over ToMe is not very significant. Besides, there is no direct comparison between the token merging method proposed in this paper and ToMe. Since the proposed method leverage both token merging and pruning, it is necessary to provide a comparison between token merging methods to clearly verify the effectiveness of the proposed method.\n\n- The method cannot achieve good performance on hardware when batch size > 1.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new method by combining token merging and pruning to reduce the computational costs of vision Transformers. Although there are several ViT acceleration methods that are based on pruning or merging, the paper explores the combination of these two groups of methods. The paper also introduced learned threshold masking modules to select the tokens to be merged or pruned effectively. The method is evaluated ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please refer to my comments above.\n\n- Some minor issues. The writing of this paper can be further improved. There are some typos or grammar errors like \"Our approach has two important components: the learned thresholds and the combination or merging and pruning\" in Section 4.5.\n\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yfyb1QnX9p", "reviewer": "Reviewer_SLvV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength:\n* Paper is well written.\n* The authors provide comprehensive experiments.\n\n# Weakness\n* As compared by authors against Kim et al. 2022 (between Eq. 3 and Eq. 4) and “ToMe” (Section 3.3.2), the proposed method contributes marginal novelty: 1) token pruning: this work only updates tokens not removed yet (s.t. the computation is saved); however, during inference it will be the same with Kim et al. 2022; 2) token merging: Section 3.3.2 is so short because it only differs from \"ToMe\" by making the threshold learnable.\n* The importance score is token-wise, and the similarity score is pair-wise. This means they have different numbers of values. How is their Kendall correlation  calculated in Table 1\n* Variance over randomness in Table 2? The performance gap seems very close.\n* Comparing with only DeiT-S is not convincing enough, as the DeiT paper proposed many other variants of ViT.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Learned Thresholds token Merging and Pruning (LTMP), a new approach aimed at reducing the computational cost of vision transformers, which are effective but resource-intensive in computer vision tasks. LTMP uses dynamic modules to decide which tokens to merge or prune, enhancing performance and efficiency. Through experiments on ImageNet, LTMP was proven to achieve top-tier accuracy, quicker and with less data than prior methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I hope the authors could better elaborate their unique contributions compared with Kim et al. 2022 and \"ToMe\"\n2. The authors should consider studying more DeiT variants beyond DeiT-S.", "broader_impact_concerns": "This work will have larger impact over the ViT community if they could address questions and issues above.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledge the clarity of the exposition, the comprehensive evaluation, and the novelty of the proposed token selection strategy. Nevertheless, they all express concerns about the benefits of the proposed method over ToMe. Specifically, the reviewers argue that the improvements obtained with the proposed method are limited, particularly considering that it requires having access to the dataset, which ToMe does not. Although, the author's responses failed to convince the reviewers w.r.t. the benefits over ToMe, the AE considers the current set of experiments to provide sufficient evidence of the method's effectiveness.

The authors mention that they have updated their manuscript to highlight that merging top-k is the technique used in ToMe, provided an additional data point of comparison with ToMe, and extended their discussion of the comparison with ToMe. In addition to this, the AE requests the authors to:
- expand the discussion of the batch size influence, as done in their answer to Reviewer iwJi
- clarify the definition of the Kendall correlation, as done in their answer to Reviewer SLvV;
- incorporate a discussion of the variance w.r.t. different random seeds, mentioned in their answer to Reviewer SLvV. In this context, the AE would highly appreciate having similar evidence for other models.

Decision: Accept (poster)