Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers
Authors: Dong Hoon Lee
Seunghoon Hong
Venue: NeurIPS 2024 poster (2024)
Keywords: token merging
token reduction method
efficient inference
vision transformer
Primary area: machine_vision

Abstract: Recent token reduction methods for Vision Transformers (ViTs) incorporate token merging, which measures the similarities between token embeddings and combines the most similar pairs.
However, their merging policies are directly dependent on intermediate features in ViTs, which prevents exploiting features tailored for merging and requires end-to-end training to improve token merging.
In this paper, we propose Decoupled Token Embedding for Merging (DTEM) that enhances token merging through a decoupled embedding learned via a continuously relaxed token merging process.
Our method introduces a lightweight embedding module decoupled from the ViT forward pass to extract dedicated features for token merging, thereby addressing the restriction from using intermediate features.
The continuously relaxed token merging, applied during training, enables us to learn the decoupled embeddings in a differentiable manner.
Thanks to the decoupled structure, our method can be seamlessly integrated into existing ViT backbones and trained either modularly by learning only the decoupled embeddings or end-to-end by fine-tuning. 
We demonstrate the applicability of DTEM on various tasks, including classification, captioning, and segmentation, with consistent improvement in token merging.
Especially in the ImageNet-1k classification, DTEM achieves a 37.2\% reduction in FLOPs while maintaining a top-1 accuracy of 79.85\% with DeiT-small.

Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers
Dong Hoon Lee KAIST donghoomlee@kaist.aee kc
Seunghoon Hong
KAIST
seunghoon, hounghoon.hogg@kaiist. www.hhaist....e....o.o ac.
Abstract
Recent token reduction methods for Vision Transformers (ViTs) incorporate token merging, which measures the similarities between token embeddings and combines the most similar pairs. However, their merging policies are directly dependent on in- termediate features in ViT', which prevents exploiting features tailored for merging and requires end-to-end training to improve token merging, This paper proposes Decoupled Token Embedding for Merging (DTEM) that enhances token merging through decoupled embedding learned via continuously relaxed token merging process. Our method introduces lightweight embedding module decoupled from the ViT forward pass to extract dedicated features for token merging, addressing the restriction from using intermediate features. The continuously relaxed token merging, applied during training, enables to learn the decoupled embeddings in a differentiable manner. Thanks to the decoupled structure, our method can seamlessly integrated into existing VIT backbones and trained either modularly by learning only the decoupled embeddings or end-to-end by fine-tuning. We demonstrate the applicability of DTEM various tasks, including classification, captioning. and segmentation, with consistent improvement token merging. Es- pecially in the ImageNet-Ik Ik classification, DTEM achieves 37.2% reduction in FLOPs while maintaining top-1 accuracy of 79.85% with DeiT-small. Code is available https:://// :/pp:::///ttiet githgtt.hbb...e com com//oovingooo//tteeee
Introduction
Transformers 31 have become the dominant and most popular architecture in machine learning. excelling in various modalities and tasks. In computer vision, Vision Transformers (ViTs) 9 30 have achieved state-of-the-ard performance, outperforning conventional backbones in tasks such as image classification [30], object detection [4) and segmentation [15), well multimodal applications such as image captioning (33] and visual question answering 17 key factor in the success of
ViTs is their ability to capture long-range dependencies between patches or tokens, regardless of their spatial positions, using self-attention. However, due to self self-attention, ViTs have high computational and memory costs that increase quadratically with the number of tokens. Consequently, there has been significant interest in developing methods to improve the computational efficiency of ViTs.
aims progressively reduce the number of tokens in Transformer layers, often adhering to predefined reduction rates. Early approaches 26 23
In this pursuit, token reduction (26
23 39
39 propose to prune unimportant tokens based on their contribution the task, as measured by scoring functions. Yet, simply pruning tokens leads to information loss, often resulting in significant performance degradation in high reduction rates. Alternatively, approaches based token merging 22 (2774 40 2 20, 34] [2] aim to combine redundant tokens instead of removing them. Such redundancy is measured by the similarity between the tokens based on intermediate features in ViT, such as token- or key-embeddings. Token merging has several ad advantages over pruning; it can
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Input
Train
Output
Gradient
Decouplai Embedding Module
Sofl Grouping
Test
Grouping
Forward
Forward
Soft Merging
Merging
Gradient
Gradient
(a) Token merging
(b) DTEM
Figure 1: Comparison of our method with conventional token merging. Contrary to prior works that merge tokens directly based on intermediate features in ViT, our method leverages a decoupled embedding to extract features tailored for token merging. The embedding module is trained via continuous relaxation of grouping and merging operators, L.e, soft grouping and merging, respectively, that allow differentiation.
achieve improved performance by reducing information loss in token reduction and can be seamlessly plugged into pre-trained models without altering the architecture.
However, merging tokens directly based on intermediate features, which are responsible for contextual encoding, presents several limitations. Firstly, these features are hard to be tailored specifically for token merging. This is because the same intermediate feature should be used for contextual encoding and merging; thereby, it would be less effective than having separate features dedicated to each role. Secondly, enhancing the merging process, which entirely relies on intermediate features, necessitates end-to-end training of the entire network. This makes it difficult to leverage pre-trained models effectively and typically requires extensive data to prevent overfitting.
To this end, we propose Decoupled Token Embedding for Merging (DTEM) that learns decoupled token embedding specifically tailored to enhance token merging. We introduce a lightweight trainable embedding module decoupled from the intermediate feature in the VIT and use to modulate the merging policy. This resolves the dependency of token merging intermediate features and facilitates the decoupled embedding to extract only suitable features for enhanced token merging. Moreover, since the modules are separated from ViTs, improved merging can be achieved without altering the ViT parameters, allowing for efficient modular optimization with pre-trained models.
However, learning the decoupled embedding module directly from conventional token merging is infeasible, since the grouping policy, i.e., deciding which tokens to merge, is based on discrete operators such as hard cluster assignment [21]40 or matching on a bipartite graph [2] (Figure [a)). To address this, we design a continuous relaxation of token merging that softly merges tokens in differentiable way according to their similarities (Figure [b). The relaxed operators, applied during training, enable training of the decoupled embedding directly through the grouping policy to improve token merging. We also observe that such relaxed operators tend to facilitate generalization of the learned decoupled embedding across unseen token reduction rates. During inference, our model converges to existing token merging methods by replacing the relaxed operators with hard ones.
We integrate DTEM in two distinct ways: modular and end-to-end full fine-tuning. For the former, we train only the embedding module while keeping the parameters of the pre- tr-irrinedd model frozen, while later we train the entire parameters an end-to-end manner. We apply DTEM to existing pre-trained vision models and verify its effectiveness in image classification, captioning, and segmentation, each requiring different level of granularity in representation. Despite the simplicity, DTEM consistently improves token merging in all three tasks, offering better trade-off between task performance and computation cost. We further analyze DTEM's components, design choices, and training efficiency. Overall, our contributions are summarized as follows:
We propose DTEM, a novel approach to enhance token merging by decoupled token em- bedding learned via continuous relaxation of token merging. The decoupled embedding is dedicated to merging and learns features suitable for merging directly from our relaxed token merging.
DTEM can be applied 1 through end-to-end full fine-tuning or in modular way by train- ing only the added embedding module. When trained modularly, the method delivers improvements even with substantially smaller datasets and fewer training epochs.
Empirical evaluations over image classification, captioning, and segmentation across various ViT models demonstrate that DTEM consistently outperforms the prior arts.

---

2 Background
Given Transformer that takes N input tokens X E Nxd, the objective of token merging is to gradually merge tokens at each Transformer block, reducing the number of tokens to X R x) Here, the denotes the reduction rate. To this end. prior works conduct the merging in two steps, grouping and merging, which are expressed as:
E = Group(S)
X Merge X, E):
where S E N denotes the similarity matrix of tokens, e.g., Sij = cos(æ;, Bz). Given the similarity S, the grouping operator (Eq. [1) identifies pairs of tokens to merge and represents them in the reachability matrix E {0, 13 13.x with (N - r) connected components, where 6ij 1 indicates that the ith and jth tokens belong to the same component and will be merged. The merging operator (Eq. 2) then combines all connected tokens in E by pooling.
The performance of the above framework highly depends on the choice of the grouping operator, as dictates the merging policy (i.e., which tokens merge), and computing the reachability matrix can be costly. Early works employ clustering algorithms [21 40), but they tend to be slow due to the iterative procedure and often suffer from performance drops due to the dramatic distribution shift from X to X caused by aggressive clustering.
Recently, ToMe [2] introduced Bipartite Soft Matching (BSM) as an efficient grouping operator of [ To parallelize the computation, BSM divides the input tokens into two disjoint sets and I, and constructs a bipartite graph. Then for each node i E A, it chooses an edge with highest similarity arg maxjeB Sij. and choose the r most similar edges afterwards to obtain the sparse adjacency matrix E' {0, {0.1)1A|x|B5 |11//|/B] where Eij Eg = The merging is performed by combining the connected tokens in E', where the connected components can be easily found since each token in A has at most one edge. ToMe [2] also proposes tracking the size of the combined tokens and accounts for it in self-attention. Specifically, given vector m E RN-r representing the size of combined tokens, the proportional attention is used in the QKV self-attention layers by:
A = 0 QK
+ log m),
(3)
where a denotes the softmax function.
Limitations Although the success of merging depends mostly on grouping operation (Eq. T the grouping depends entirely the similarity of the intermediate VIT feature (X or K) in the prior works. This is mainly because the grouping comprises discrete operators, such as clustering and matching, that prevent the gradient flow through grouping (Eq, D.. Thus, the only viable option to improve the merging is by updating the intermediate feature X by back-propagating through the merging operator (Eq. 2). However, it leads extensive end and-to-end training of the entire network. preventing off-the-shele usage and resulting in suboptimal performance due to the conflict between the token feature required for optimal merging and task performance.
We provide more discussion on related work in the supplementary materials A.3
Method
Our objective is to improve token merging by learning the decoupled embedding specifically tailored for merging. To this end, we base our method on the standard token merging framework introduced in the previous section (Eqs. I Db. Instead of directly leveraging the ViT features X for grouping, we propose to learn additional per-token embedding modules = f(X; Φ). which are decoupled from the forward pass of the ViT and used only to compute the similarity S in Eq. by Sij cos(z; 2j) (Sec. (.T]). Since the grouping operator is entirely dependent on similarity, we can directly modulate the grouping (or merging) policy by learning Z. Furthermore, since the embedding is decoupled from the ViT forward pass, enhancements in merging can be achieved modularly without altering the ViT parameters but only learning the embedding Z.
To enable our model learn such embeddings through merging, we propose continuous relaxation of the grouping and merging operators in Eq. D and Eq. 2 respectively. Specifically, our relaxed
3

---

grouping operator generates a continuous matrix E, whose elements EG E [0, 1] indicates the soft degree of merging ith token into the jth token (Sec. [.2]. To incorporate such soft commitment in merging, we also propose a relaxed merging operator that combines tokens with continuous weights deñieed by E (Sec. 3.3). Since the token merging is performed continuously with our relaxed operators, we discretize them after the training, reducing our framework behave similarly to the hard merging methods [2] (Sec. 3.4.
3.1 Decoupled Embedding Module
We first describe the choice of the embedding module decoupled from the forward pass of ViTs. To facilitate token merging at each Transformer block, we introduce per-token projection layers into each block I E f L}:
Decoupled embedding Z z; = [ Φi),
Token similarity S Sij (2;; Zz),
where X E RNx is the input to the self-attention and Z E TR denotes the output decoupled embedding with d' < d. The output embeddings will be used solely shape the merging policy (i.e., deciding which tokens to merge) in the grouping operator based on the similarity S.
Minimizing additional run-time and computational overheads is essential to the embedding module design. In our approach, we employ a token merging between the self-attention and feed- cerdfforardd layer following [8 21. It allows parallelizing the computation of the attention and decoupled embedding, avoiding the potential overhead that comes from serialization. Moreover, we discover that even a shallow module, consisting solely of an affine transformation, can achieve improvement with minimal computational expense (Sec. 4.4). This further minimizes the number of additional parameters to less than 1% % and enables the training of the module with small amount of data.
3.2 Soft Grouping
Given the similarity matrix S obtained from the decoupled token embeddings Z, soft grouping aims to approximate the grouping operation through a continuous relaxation that enables differentiation. However, building a general continuous grouping operator of Eq. i challenging since the output reachability matrix is inherently discrete.
Instead, we employ BSM (2) as target grouping operator, which offers the benefit of bypassing the reachability matrix and allows for merging to be defined directly on the adjacency matrix E' E {0, 1)14|xIBI To be a valid approximation of the grouping performed by BSM, the soft grouping operator should produce a continuous adjacency matrix È E [0. [][A| that satisfies two key conditions. Firstly, it should simulate distinct edges with high values, thereby implementing the valid merging policy, Le., combining the r most similar token pairs. Secondly, each node in A should be associated with at most one edge (i.e.) ΣjEB êGj S 1) simplify the identification of connected components in E, thus avoiding the complexity of computing reachability matrix.
To achieve this, we propose a soft grouping that revises the differenniable top-k operator from 25 Starting with S = S, we repeat the subsequent steps for each i = J:
A' (T)
(6)
1+1 = و' + log(1 (1- EjER aty), k/
(7)
where a denotes the global softmax function and r > represents a temperature scale that regulates the relaxation. In each step f, this process computes the A' E [0, || with Eiea, aij = 1, representing the soft-argmax. Subsequently, the similarity S is updated to suppress the entire outhounding edges from the softly selected nodes in A. by Eq. 6 Afterward, the soft adjaceney matrix E is defined follows:
d
ēij max 1, sg EE "())
where A'
Σ A
(8)
with Eij Eij S r, representing the total number of selected edges. The clipping function, composed of max operator (max) and stop- coopggradient (sg), is introduced to ensure that the resulting É is a valid continuous adjacency matrix.

---

Note that the soft grouping satisfies the aforementioned key conditions. AS - 0, A' converges to the one-hot matrix that indicates the nodes in A and B with maximum sty, This results in A representing most similar pair, thus satisfying the first condition. Meanwhile, edges associated with such nodes in A are excluded from future selection according to Eq. since log(1 EjeB a'y) + oc, ensuring at most one selection per node in A. This holds true even in the non-asymptotic case, as the clipping function guarantees EEER Gzz I 1, thereby meeting the second condition.
3.3 Soft Merging
While the soft grouping and the resulting soft adjacency matrix effectively approximate the grouping process, it is crucial to design the merging operator to incorporate such soft decisions. In our approach, for the given soft adjacency matrix where εig corresponds to tokens i E and j E IB, our soft merging is designed such that the ith token merges into the jth token in proportion to the value of Eij.
Our soft merging operator applies the asynchronous updates on tokens in two sets, A and B. For each token j E IB, the operator update their feature 2j and the effective size my by aggregating tokens in A based on the soft adjacency matrix E from Section 3.2 by:
M @xymp;
my + m; + Σiea C-y74
(9)
my + EiEn Cigm;
One the other hand, for each token A, the operator update only its effective size m, while maintaining the feature:
my c m,(1 Ei)
(10)
Note that with the binary adjacency matrix E', the effective size of the tokens in A reduces to zero by Eq. [0 ff they have outbounding edges. Such tokens will be excluded from the subsequent merging process by Eq. 9 This process is simulated continuously during training with our soft adjacency
matrix (i.e., each token will be continuously absorbed into others), while it is used to actually reduce the tokens inference using a discretized adjacency matrix.
Interestingly, we observe that the decoupled embedding, trained with soft merging at a high reduction rate r. generalizes well to lower rates r' learned to sort r most similar token pairs by the relaxed top-k operator (Eqs. 6 7). thereby including the sorting for smaller reduction rates yd <
This is presumably because the decoupled embedding is
3.4 Training and Inference
Training Thanks to decoupled embedding modules, training can be conducted in two distinct ways: modular and end-to-end training. In modular training, we train only the embedding modules while keeping the ViT parameters frozen. This allows our method to fully leverage off-the-shell models while effectively adapting only the merging policy to each task. In end-to-end training, we jointly train all ViT parameters along with our embedding modules. Since our continuous merging operators do not reduce the number of tokens during training. we alternate updates between the embedding layers and VIT' parameters to save computation. Specifically, when updating the VIT parameters, we fix the embedding layers and use the discretized grouping and merging operators, which allows token reduction in the ViT forward pass, greatly enhancing the efficiency. Conversely, when updating the embedding modules, we apply the soft grouping and merging operators while fixing the ViT parameters. We alternate this procedure with much more frequency on ViT updates, since the embedding layers have considerably smaller parameters (x)%) and hence quickly converge. For both modular and end-to-end training, we simply train our method to minimize the task loss.
For inference, we discretize the continuous operators in the grouping and merging processes, and perform the hard token merging utilizing the learned decoupled embeddings. As explained in Sec. 3.2 and Sec. 3.3, our soft grouping and merging modules are asymptotically equivalent to BSM of ToMe. Consequently, we employ BSM to speed up the inference.
Inference
4 Experiments
We apply our method, DTTM, for token merging in image classification, captioning, and segmentation. In Sec. 4.1 we first evaluate our method in the ImageNet- [8] classification with two setups:
5

---

Table 1: Classification results with off-the-shelf frozen pre-trained models. Reduction roughly represents the decreases in FLOPs.
Reducctio
Method
Acc@1
GFLOPs
im/s
Method
Acc I
GFLOPs
im/s
DefT-S
79.83
1390
DefT-B
81.79
17.7
440
EVIT [
78.50
3.03
2069
EVIT
80.45
658
35%
ToMe 2
79.12
3.02
1917
ToMe 2
80.57
11.5
628
DTEM
79.44
2,91
1991
DTEM
81.01
11.1
653
EVIT L8
74.10
2.33
2672
EVIT LB
75.11
8.9
854
2
78.01
2457
2
77.92
DTEM
78.99
2.35
24300
DTEM
79.54
8.88
818
MAE-B B II
83.72
17.7
438
MAE-L 11
85,95
61.8
131
EViT
82.11
EViT
85.22
35%
ToMe 2
82.33
11.5
628
ToMe 2
85,46
42.5
186
DTEM
82.80
11.6
653
DTEM
85.61
42.9
185
EVIT L
75.95
8.9
854
EVIT B
82.77
33
244
2
78.88
84.21
252
DTEM
80.37
8.88
8118
DTTEM
84.68
31.4
250
modular and end-to-end training. We further present our results on COCO [6] image captioning in Sec. 4.2 and ADE20K [I] semantic segmentation in Sec. 4.3 demonstrate that our method can be applied to tasks requiring various levels of granularity in representation. We then provide a series of analyses on the importance of decoupled embedding, the design choices of embedding module, and data/training efficiency, complemented by visualizations, in Sec. 4.4.
4.1 Image Classification
Setup We conducted image classification experiment ImageNet- 1k [ dataset with 1.28M training and 50k validation images. We apply our method and baselines to various pre-trained ViT models, including DeiT-S/B B0), MAE-B/L [T], and VV-V [33] The image resolution in training and testing is 224 x 224 unless otherwise stated. We also present the results for DeiT-T and AugReg (28 ViT-S (with resolution of 384 x 384) in the supplementary material (A.T). We report top-1 accuracy (Ace@) @ I) on the validation set, with floating-poin operation (FI.Os) and throughput (images per second, im/s) to quantify the computation reduction. For the throughput, measured on a single NVIDIA 3090 GPU with a batch size of 128 and fp32.
Implementation detail We mostly follow the fine-tuning setup from 226 18), which is based on the training recipe of DeiT (30]. We initialize the ViTs with pre-trained weights and train for 30 epochs,
as in most baselines (26 18 34 [0]. For token reduction, we employ a uniform reduction strategy, where the reduction rate represents the number of tokens removed in each transformer block. When training the embedding modules, we apply the reduction rate r = 16 to ViT-S/B models and 8 to the ViT-L model. As the embedding module, we use a linear layer with an output dimension of 64 for VIT-S/B and 128 for VIT-L. We use temperature scale of - = 0.1 and also scale the similarity by 0.1 prior to soft grouping. More details can found the supplementary material A.2
Modular training Table reports classification results when approximately 35% and 50% of FLOPs are reduced by applying token reduction methods to frozen pre-trained ViTs, with ViT parameters remaining unchanged. We compare DTEM with ToMe [2] and EVIT [18] in this setting. The results demonstrate that DTEM consistently outperforms the basclines. Specifically, with 35% reduction in FLOPs, our method improves performance by +0. 15% to +0.47% compared to ToMe across all DeiT-S/B and MAE-B/L models. For reduction of 50% in FLOPS, DTEM significantly improves performance by +0.47% to +1.64% while adding less than 1% additional FLOPs.
In T we further applied our method to LV-VIT [3], variant of standard VIT. LV-VIT employs an input embedding module consisting of convolution layers to better tokenize the input image. We apply token merging into the first 12 transformer blocks of LV-ViT-S. Consistent with previous results, DTEM achieves a +0.2% accuracy gain over ToMe, demonstrating its applicability to LV-VIT. Notably, despite optimizing only the added embedding module parameters, this performance is comparable to other state-of of-te--f-thhaan the methods [18] 34 that fully fine-tune the ViT parameters.

---

Table 2: Classification results with LV-ViT- indicates the results with off-the-shelf frozen pretrained model.
Table 3: Comparison of classification re- sults with prior arts. denotes the baseline results implemented by ourselves.
Method
Acc@1
GFLOPs
im/s
Method
IN-VIT-S
83.3
879
Comparison DeiT-S
EVIT
82.5
cTPS 34
82.5
3.8
DyVIT 20
79.3
2.9
2082
1574
Evo-ViT
88
79.4
DTEM
82.5
3.73
1571
EVIT
18
79.51
2069
ToMe
2
79.68
1917
DeiT-S
DeiT-B
ATS 10
79.7
2.9
BAT 20
79.6
cTPS
34
79.7
79.02
DTEM
79.85
81.2
13.00
DyVIT
20
78.5
2429
78.63
05/75
BAT
20
79.0
2.3
20.50
Dus
eTPS
34
79.2
2.3
19.72
GPIDPs
02.3
12
GPLDPS
DTEM
79.25
79.38
2.3
2430
Comporison DeiT-B
19.25
DyVIT 26
81.3
15,02
Dem
Dert
Evo-VIT 38
81.3
11.6
EviT
80.5
EM
EVT 1B
81.3
11.6
ToMl
Tuhaa
78.52
2000
190.0
Dum
250
ToMe 2
DTEM
81.60
81.37
11.5
11.6
624
Throughput
Throughout
DTEM
81.47
653
Figure Classification results under differ- ent FLOPs and throughputs. All methods are end-to-end trained.
EVIT ToMe
R
80.0
8.9
854
80.58
8.8
823
DTEM
End-to-end training TigureBdepict the classification results under different FLOPs and through- puts when token reduction methods applied through end-to-end training. We compared our method with a fine-tuned version of ToMe (2) and EVTT [18] For the baselines, we report accuracies by training each model specific target computation demands under varying reduction rates, e.g., r {16, 13, 12, 11} for ToMe. Conversely, for DTEM, we train single model with reduction rate of 13 for fine-tuning the ViT parameters, while maintaining
16 when training the embedding module We then adjust the reduction rate and report the corresponding accuracies during inference.
The results demonstrate that method consistently outperforms the baselines across all levels of computational reduction. Specifically, our method surpasses the baseline accuracy by 0.12% to 0.2% in DeiT-S while adding small amount of FLOPs and degradation in throughput. This leads to improved trade-off between accuracy and computational resources, such as FLOPS and throughput. notable aspect of DTEM is its ability provide single trained model that is generalized across various reduction rates. This can mitigate the training and storage costs associated with the multiple rounds of full fine-tuning often required to support different levels of computational reduction.
Comparison to State-of-The-Art While the results in Figure verify the effectiveness of our method in end-to-end training, we compare D'TE''s performance with more token reduction methods in Table We mainly considered the epochs training results used in 26 comparison 2 The table shows that our method achieves superior accuracy compared to other prior arts when computational costs are equated.
4 10 for fair
4.2 Image Captioning
To demonstrate the broad applicability of our method, we apply DTEM to image captioning, a task extensively studied in the vision-language domain. Recent captioning models with ViTs typically utilize all output patch features to ensure the caption generation is grounded in richer and more
As explained in Sec. 3.4 alternate updates between the embedding modules and VIT parameters for end-to-end training.
We further include comparison results from 100 epochs of training in the supplementary material A.1 demonstrating superior performance.
7

---

Table 4: Image captioning evaluation results when token merging is applied. We report with caption evaluation metrics: BLEU-4 (B@4), CIDEr (C), METEOR (M) and SPICE (S). Reduction represents the decreases in FLOPs within the ViT encoder, and # indicates the number of tokens passed to language decoder.
Reduction
@
S
Reduction
B@4
M
GUT-B
38.8
30.1
127.6
23.6
GTF-L 33
29.6
134
197
32%
34.6
26.4
113.1
20.3
31%
36.9
27.3
122.1
21.5
77
ToMe
35%
33.5
25.8
109.3
19.8
37%
36.4
27.1
120.1
21.5
53
38%
33.3
25.5
107.9
19.5
43%
34.0
25.8
112.2
20.2
29
41%
31.9
24.8
104.3
19.0
49%
31.7
24.8
105.1
19.3
7
31%
36,2
27.1
118.1
20.8
31%
37,9
27.8
124.4
21.9
77
DTEM
34%
34.5
26.5
114.2
20.5
DTEM
37%
37.0
27.5
122.9
21.7
53
37%
34.3
26.2
112.9
20.1
43%
35.7
26.6
117.6
20.9
29
41%
33.3
25.7
110.4
19.9
49%
33.3
25.7
111.1
7
Table 5: Results on semantic segmentation when token merging is applied. The reduction ratio indicates the portion of merged tokens.
Baseline (r
Reduction ratio
Model
Method
0.2
0.3
0.4
0.5
36.28
30.82
27.18
23.7
20.46
ToMe
2
GFLOPs
(100%)
(85.0%)
(74.9%)
(65.3%)
(56.4%)
Seg-S-Mase Seg-S-Mask/16 PV
mioU
45.28
44.88
43.98
41.70
36.62
36.28
29.39
25.75
22.27
19.03
GELOPs
Ours
(100%)
(811%)
(71%)
(61.4%)
(52.5%)
mloU
45.28
44.96
44.3
42.64
38.92
detailed information about the image, which is crucial for accurate captioning. However, using all patches may be inefficient due to redundancy in image tokens, motivating the use of token merging.
Setup We experiment with the COCO 6 caption dataset using the train/val/test split from [14) We use COCO fine-tuned GIT 33 models, each consisting of a VIT-B or L image encoder and a language decoder. The embedding modules are trained modularly with language modeling loss and use the best model identified through cross-validation. The quality of captions is evaluated using metrics of BLEU-4 24 METEOR (22), CIDEr (32,, and SPICE M1. while the computational cost is reported in terms of the ViT encoder's floating-point operations (FPSs) and the number of tokens (#) passed the language decoder. More details provided in the supplementary material A.2
Results Table presents the image captioning results on the test split when DTEM or ToMe 2 applied. DTEM outperforms ToMe, achieving better trade-off between captioning quality and computation cost. Specifically, with the GIT-B model, DTEM enhances the CIDEr score by +5.0 +6.0 across reductions in FLOPs of 31% % 41%. Similarly, we observe improvement ranging from +2.3 to +6.0 with the GIT-L model. These results confirm that DTEM provides a better set of patch representations by effectively summarizing the information the image tokens.
4.3 Semantic Segmentation
To further demonstrate DTEM's applicability, we apply our method to semantic segmentation, widely studied computer vision task with numerous applications.
Setup We use a pre-trained Segmenter (29 model and evaluate the token merging on the ADE20K [41 dataset, which contains 25k training data across 150 fine-grained semantic categories. Unlike image classification or captioning tasks, segmentation models-including the Segmenter- require complete image patches (tokens) in the end to decode the segmentation mask. To address this, we follow the approach proposed [3] that repeatedly merges tokens before each component (e.g., self-atte attentienio and feed-forward network) and then un-merges them after processing the component. We modularly trained our decoupled embedding modules using the cross entropy loss. We report mean intersection-ver-uniitt ueverrniionno (mloU) and floating -ointt operations (FLOPs) for performance and computational cost, respectively. More implementation details are provided the Appendix A.2

---

Table Ablation study on the impact of de- coupled embedding. We successively add soft token merging and decoupled embedding module into ToMe. The number in parentheses indicates the reduction in FLOPs.
Table 8: Comparison with alternative design choices for soft grouping. As an alternative, we experiment with applying the Gumbel Softmax (GS) to replace the top-1 operation from ToMe, enabling differentiation.
Arch.
Method
Acc. (-35%)
Acc. (-50%)
Mothod
Acc. (-35%)
Acc. (-50%)
ToMe 2
68
79.25
DynamicVil [6] (off-Ibe-sheel))
78.96
75.53
DeiT-S
saft tokon merging decoupled embedding
79:57
79.15
ToMe + GS solt merging
79.2
78.14
79.85
79,38
DTEM
79.44
78.99
without prop attn
79.16
77.86
ToMe [3]
81.37
80.58
DeiT-B
suft token merying
81.45
80.49
78.1
2.4
79.1
decoupied embeddinng
81.60
80.74
Gh.00s
3.0
Top-1 Accurasy
78.0
79.0
2.75
GFLOR
Table 7: Kendall rank correlation coefficient changed through training. We report changes in the Kendall rank correlation between token similarities derived from two different features: self-attention keys and decoupled embedding.
2.35
78.9
78.9
76.3
32 48 64
128
2
2.25
(a) Embedding dimension
(b) hiddeen layers
blocks
5 to 8 blocks
to 12 blocks
Figure 3: Ablation study decoupled embed- ding module design: (a) decoupled embedding dimension and (b) number of hidden layers.
Before training
0.517
0.457
0.591
After training
0.519
Results Table 5 reports the semantic segmentation results when token merging methods, iee. ToMe [2] and DTEM, applied to Segmenter with ViT-S. Our method consistently offers better mloU to GFLOPs trade-off compared to ToMe. Specifically, DTEM achieves a +0.32 to +1.3 improvement in mloU over ToMe when 25% to 50% FLOPs are reduced. These results demonstrates the applicability of DTEM in semantic segmentation for enhancing token merging.
4.4 Analysis
We conduct an analysis of DTEM in ImageNet- Ik classification, specifically within a modular training setting using the DeiT-S model, unless if otherwise stated.
Importance of decoupled embedding In Table we ablate the impact of decoupled embedding in end-to-end training. We observe that naive integration of soft grouping and merging applied to the keys of self-attention, as in ToMe (2). degrades the performance. This confirms that decoupled embedding is crucial DTTMM to provide more compelling features for token merging. In Table we further investigate whether the decoupled embedding used for merging diverges from the intermediate features after training. We report the Kendall rank correlation between token similarities derived from two different results show a decreased correlation after training, indicating that the decoupled embedding learns a different feature for token merging, distinct from the intermediate features.
keys and decoupled embedding- befidiingebeorreeeeeeeee and after training. The
Design choices for soft grouping In Table S for the analysis of soft grouping, we compared several approaches: (1) the pruning baseline, tested Dynamic VIT [26 applied to frozen ViT, (2) integrating Gumbel Softmax (GS) to replace the top-1 operation from ToMe to enable differentiation, (3) our method, and (4) our method without proportional attention. The results indicate that our proposed design for soft grouping performs the best, with proportional attention proving to be crucial.
Embedding module design Figure3(a) and (b) show the impact of different embedding dimensions and the number of hidden layers in the embedding module, respectively. In the main results (Sec. [4.). the embedding dimension of 64 and the module without hidden layers. We observe that simple affine transformation offers sufficient gain while keeping computational costs low.
Effect of reduction rate on training In Table 9. we analyze the effect of the reduction rate used during training. We observe that the decoupled embedding module, when trained with a high reduction rate r, generalizes well lower rates during inference. Therefore, is generally sufficient to set the reduction rate during training to the maximum number of tokens we wish to reduce during inference.

---

Table 9: Effect of the reduction rate in soft grouping. Results style: best, second best.
79.5
Redluction rate at training
Reduccioon rale inference
OIE trich JAM0 Terle 2013 thaliing
79.3
lana talling
78.92
79.33
79.42
79.60
Accuray
78.80
79.23
79.43
79.60
12
78.77
79.22
79.41
79.61
78.67
79.19
79.38
79.60
79.0
76.8
Image
76.6
lont po14 tikking
T5% 1--36.2333 232G
OTEN (2005)
ToMe
0.31% 0.62%
(a) dataset
(b) Training opochs
Figure 4: Image classification results on data and train efficiency: (a) dataset size and (b) training epochs. In the experiments, DTEM is modularly trained on DeiT-S model, while ToMe undergoes end-to end-to-end training.
Ours
Figure 5: Visualization of merged tokens.
We apply reduction rate = 16, leading 11 merged tokens in the final output.
Data/Train efficiency In Figure (a) and (b), we examine the data and training efficiency of DTEM with modular training, respectively. Owing to the parameter-efficien modular approach, DTEM improves the performance of merging even with limited amount of training data and epochs. Specifically, in Figure 4 (a), our method achieves a gain of +0.44% accuracy even when trained with 4000 images, equaling 0.31 % of the total dataset. DTEM outperforms the end-to-end training approach with ToMe under 10% of the total dataset for both 35% and 50% reduction FLOPs, highlighting the potential benefit of our modular training when the entire dataset is unavailable. Moreover, in Figure 4 (b), the results on the effect varying training epochs show that DTEM quickly converges even at the first epoch. Since DTEM employs modular training in end-to-end learning by the alternative optimizations (Sec. 3.4 such rapid convergence is also useful in reducing the cost of end-to-end training.
Visualization 3 we visualize the token merging compare ToMe and our modularly trained DTEM. Tokens belonging to the same group are color-coded identically, highlighting the grouping changes induced by the decoupled embedding. DTEM prioritizes merging background tokens, allocating more tokens to foreground objects. In contrast, ToMe allocates more tokens to the background, indicating less focused approach foreground objects.
Conclusion
We propose Decoupled Token Embedding for Merging (DTEM) that improves token merging via decoupled token embedding derived directly from the token merging process. Our method introduces the decoupled embedding, learned through our continuously relaxed token merging, to exploit dedicated features for token merging. The decoupled embedding enhances token merging by resolving the dependency of token merging on intermediate features and enables modular training, effectively utilizing the frozen pre-trained models. We experiment with DTEM on classification, captioning, and segmentation using various pre-trained ViT models. The experimental results demonstrate that our method consistently improves token merging, highlighting the importance of features tailored specifically for token merging.
Acknowledgments and Disclosure of Funding
This work was in part supported by the National Research Foundation of Korea (RS-2024-00351212 and RS-2024-004361163 and Institute of Information & communications Technology Planning & Evaluation (ITTP) grant (RS-2022-1120926, RSS20244000092279, RS-2021-1121 2068, and RS-2019. 11190075) funded by the Korean government (MSIT).
10

---

References
[1] Anderson, B. Fernando, M. Johnson, and S. Gould. Spice:
Semantic propositional image caption evaluation. In European conference on computer vision (ECCV), 2016.
[2] D. Bolya, C.-Y. Fu, X. Dai, P. Zhang, Feichtenhofer, and Hoffman. Token merging: Your vit but faster. In International Conference on Learning Representations (ICLR), 2023.
[3] D. Bolya and Hoffman. Token merging for fast stable diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023.
[4] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision (ECCV), 2020.
[5] M. Chen, W. Shao, P. Xu, M. Lin, K. Zhang, F. Chao, R. Ji, Y. Qiao, and Luo. Diffrate: Dif- ferentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[6] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1504,055040000 2015,
[7] M. Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and bench- mark. Atttps///githin hww.tttb.ccooooooooooooooooooooo http://///pppaa.oooooo/////aaaaa
2020.
[8] J. Deng, W. Dong, R. Socher, L.JJ K. Li, and L. Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (CVPR), 2009.
[9] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[10] M. Fayyaz, S. A. Koohpayegani, F. R. Jafari, S. Sengupta, H. R. V. Joze. E. Sommerlade, H. Pirsiavash, and J. Gall. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision (ECCV), 2022.
[11] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition (CVPR), 2022.
[12] J. H. Heo, A. Fayyazi, M. Nazemi, and M. Pedram. A fast training-free compression framework for vision transformers. arXiv preprint arXiv:2303.02333 2023.
[13] Z. Jiang, Q. Hou, L. Yuan, Zhou, Shi, X. Jin, A. Wang, and J. Feng. All tokens matter: Token labeling for training better vision transformers. Advances in neural information processing systems (NeurIPS), 2021.
[14] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), 2015.
[15] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint WxXiv:2304.02643 2023.
[16] D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi. Lavis: A library for language-vision intelligence. arXiv preprint arXiv:2209.09019 2022.
[17] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.1259) 2023.
[18] Y. Liang, C. GE, Z. Tong, Y. Song, J. Wang, and P. Xie. EVit: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations (ICLR), 2022.
11

---

[19] D. Liu, M. Kan, S. Shan, and X. CHEN. A simple romance between multi-exit vision transformer and token reduction. In International Conference on Learning Representations (ICLR), 2024.
[20] S. Long, Z. Zhao, J. Pi, S. Wang, and J. Wang. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[21] D. Marin, J.-H. R. Chang, A. Ranjan, A. Prabhu, M. Rastegari, and O. Tuzel. Token pooling in vision transformers for image classification. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), 2023.
[22] L. Meng, H. Li, B.-C. Chen, S. Lan, Z. Wu, Y.-G. Jiang, and S.-N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[23] B. Pan, R. Panda, Y. Jiang, Z. Wang, R. Feris, and A. Oliva. IA-REDS*2S: Interpretability-aaaal redundancy reduction for vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[24] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compulational Linguistics (ACL), 2002.
[25] T. Plötz and S. Roth. Neural nearest neighbors networks, In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[26] Y. Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[27] C. Renggli, S. Pinto, N. Houlsby, B. Mustafa, J. Puigcerver, and Riquelme. Learning to merge tokens in vision transformers. arXiv preprint arXiv:2202./20055 2022.
[28] A. P. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research (TMLR), 2022.
[29] R. Strudel, R. Garcia, Laptev, and C. Schmid. Segmenter: Transformer for semantic segmen- tation. In Proceedings the IEEE/CVF international conference computer vision (ICCV), 2021.
[30] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou. Training data- efficient image transformers & distillation through attention. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 18-24 Jul 2021.
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems (NeurIPS), 2017.
[32] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-base image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2015.
[33] Wang, Z. Yang, X. Hu, L. Li, K. Lin, Gan, Z. Liu, Liu, and L. Wang. Git: A generative image-to-tex transformer for vision and language. arXiv preprint S.tiv:2205.141000 2022.
[34] S. Wei, T. Ye, S. Zhang, Y. Tang, and J. Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[35] R. Wightman. Pytorch image models.
https://githul. http://ggtthub..commrrgggttttttt
pytorch- www-iimaageeeeeee 2019.
12

---

[36] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiez, Davison, S. Shleifer, von Platen, C. Ma. Y. Jernite, Plu, C. Xu, T. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of-the-art natural language processing. Proceedings the the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38 4845, Online, Oct. 2020. Association for Computational Linguistics.
[37] J. Xu, S. De Mello, Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[38] Y. Xu, Z. Zhang, M. Zhang, K. Sheng, K. Li, W. Dong, L. Zhang, C. Xu, and X. Sun. Evo- vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[39] H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Pruceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[40] W. Zeng, S. Jin, W. Liu, C. Qian, P. Luo, W. Ouyang, and X. Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[41] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
13

---

A Appendix
A.1 More Results
Full Classification Results Table TO and Table report the full image classification results of our method with DeîT-S/B 30 and MAE-B/L. [1. respectively. For DeïT, we report the full results of both modular and end-to-end training settings. For MAE, we report full results in the modular training setting.
Table Full image classification results of Table 11: Full image classification results of method with DeiT-S/B models different our method with MAE-B/ B/L models at differ- reduction rates r.
ent reduction rates т. Only the embedding modules modularly trained.
TTpp1 Accuracy End to
Model
GFLOPs
Modular
Model
GFLOPs
Top-I Accuracy
2.35
16
78.99
79.38
8,88
16
S0.37
79.17
79.61
9.40
81.29
DeiT-S
2.77
2.63
79.20
79.49
79.60
79.74
MAE-B
10.50
9,95
13
14
81.82
82.24
79.85
11.05
12
82,56
3.06
11
79.51
79.82
11.60
11
82,80
8.88
79.54
80.74
31.42
8
84.68
DefT-B
9.40
80.03
$1.17
$1.03
MAE-I
39.05
35.21
6
7
82.58
84,92
10.50
13
80.68
81.37
42.90
5
85.61
11.60
11
81.01
81.60
Extended image captioning evaluation results In Table report Figure rrsuulss across broader reduction range. The result shows that our method is particularly effective in challenging, more noource-constrainet settings with higher reduction rates. caption. We note that for reduction rates over 41 %% and 49% for GIT-B and GIT-L respectively, there was significant decrease in captioning quality.
Table 12: Full image captioning evaluation results when token merging is applied.
Reduction
B@4
M
S
Reduction
B@4
M
S
GIT-B
38.8
30.1
127.6
23.6
197
GTT-
40.7
29.6
134
23.8
197
12%
37.9
28.6
123.7
22.4
149
15%
35.4
27.1
115..9
21
101
18%
40.1
28.9
131.1
23
125
27%
35.7
26.9
115.3
20.9
24%
39.4
28.8
128.7
22.7
ToMe
32%
34.6
26.4
113.1
20.3
77
ToMe
31%
36.9
27.3
122.1
21.5
77
35%
25.8
27.1
38%
33.3
25.5
107.9
19.5
53
43%
34.0
25.8
112.2
20.2
29
41%
31.9
24.8
104.3
19.0
41
49%
31.7
24.8
105.1
19.3
12%
38
28.6
124.2
22.3
149
25%
21.4
27%
36.4
27.4
119.3
21.4
89
24%
39.4
28.9
129.5
23
101
DIEM
31%
36.2
27.1
118.1
20.8
DTEM
31%
37.9
27.8
124.4
21.9
34%
34.5
26.5
114.2
20.5
37%
37.0
27.5
122.9
21.7
37%
34.3
26.2
112.9
20.1
43%
35.7
26.6
117.6
20.9
41%
33.3
25.7
110.4
19.9
49%
33.3
25.7
20.1
Classification Results with DeiT-T In Table 13 further report the classification results of method with DeīT-T, demonstrating that our method consistently improves token merging in modular and end-to-end training settings. We follow the 30 epoch training settings of DeiT-S/I except that we remove the MixUp and CutMix augmentations, as the performance improves without them.
Training for Longer Epochs In Table 14 we further report comparison results from longer training epochs. We note that some methods [18, 34 can improve their performance by training for significantly more epochs, e.g. 100 epochs. To make fair comparison, we also trained DTEM for 100 epochs, with the baseline, i.e., dTPS R4. The results show that DTEM outperforms the baselines in D and is comparable to dTPS DeïT-S, while incurring much lower training costs. This efficiency is credited to DTEM's ability effectively generalize across various reduction

---

Table 13: Full image classification results of Table 14: Image classification results with our method with DeiT-T at different reduction 100 epochs of end-to-end training. rates r. o indicates the improvement in accu- racy over ToMe (2).
Model
Reducttion
Method
Acc
dTPS (100ep)
72,9
35%
GFLOPs
Top-1 Accuracy
DeFF-T
DTEM (100ep)
73,2
Modular
End-to-end
TTPS (100ep)
72.1
50%
0.65
16
70.20
72.16 (40.20)
DTEM (100cp)
72.4
0.69
15
70.73 (-142)
72.51
0.73
0.77
13
71.04
71.27 (+0.44)
72.57
72.77
DeñT-S
35%
DTEM (100ep)
dTPS (100ep)
80.1
80.0
0.81
(+0,50)
dIPS (100ep)
79.7
0.85
11
71.51 (-0.4)
72.94 (40 18)
50%
DTEM (100ep)
rates with single trained model, as detailed in Section 4.1 Note that dTPS requires multiple trained models to support different reductions in computation. Moreover, dTPS requires multiple training losses to achieve such performance, including the self-distillation loss, which brings the remaining tokens closer to those of the teacher model, and the KL divergence loss, which minimizes the difference in final predictions between the model and its teacher. In contrast, our method can achieve comparable or even better performance using only the task loss.
Classification Results with AugReg VIT-S In Table 15| we report experimental results with
larger number of tokens on image classification (AugReg 228 ViT-S with a resolution of 384 x 384), corresponding to smaller patches higher input resolutions. The decoupled embedding module is trained modularly with a reduction rate of r = 48. The results show that our method can adapt to settings with an increased number tokens, achieving performance gains.
Table 15: Image classification results with AugReg ViT-S pretrained on 384x384 resolution.
Method
Reduction
Acc@1
GFLOPs
im/s
VIT-S (384)
83.8
15.7
394
ToMe DTEM
51.5% (r
47)
82.1
7.60
728
52.0%
48)
82.3
7.54
733
we report classification results analyzing the effect of the temperature scaling. We experimented with a modular training using the DeiT-S model. We trained the decoupled embedding module for 10 epochs. We observe that values within the range of 0.1 0.3 consistently provide gains with an accuracy difference of 0.1%.
Effect of temperature scaling In Table 16
Table 16: Ablation study on the effect of temperature scaling.
Temperature seale
Acc@! 1 (-50%)
Acc@1 I (-35%)
0.05
0.1
78.87
79,51
79.50
0.3
78.92
79.57
83
79.54
78.59
79.34
End-to-End Training for Captioning We further present the results of end-to-end training for token merging method applied image captioning. using the GIT base model [3 on the COCO caption dataset. A separate learning rate of 0.0001 is applied for updates to the embedding module, and 0.000002 for the ViT and text decoder parameters. Our method is compared with a end-to-ent end trained version of ToMe [2] applied to the GIT base model. Both our method and ToMe are trained with a reduction rate of r 13. The results are shown Tab. 17 Although end-to-end training benefits both methods, our method offers better trade-off between performance and computational efficiency. To be more specific, our method achieves an improvement of +1.0 to +2.0 in the CIDEr score.
15

---

Table 17: Image captioning evaluation results when token merging is applied with end-to-ent training.
We report with caption evaluation metrics: BLEU-4 (B @),, CIDEr (C), METEOR (M) and SPICE
(S). indicates the number of remaining tokens the ViT output.
Method
Reduction
B@4
M
C
Lokens
33%
28.0
122.0
77
-35%
36.7
27.8
120.2
21.4
65
ToMc
-38%
36.7
27.3
118.8
21.0
53
-41%
36.2
27.2
117.4
20.8
41
-31%
37.9
28.1
123.3
21.9
77
122.2
65
DTEM
-37%
36.8
27.6
120.2
21.2
53
-41%
36.2
27.2
118.4
21.1
41
A.2 Implementation Details
In this section, we provide a more detailed explanation of the implementation.
Common details During training, our method retains N tokens, even if its effective size m becomes zero. To ensure that these tokens do not participate in the relaxed merging process, we successively exclude the tokens with the minimum effective size from the relaxed merging at cach transformer block. Consequently, after applying this exclusion process times, relaxed merging operates with N - rl tokens at the (-th ViT block. We observe that blocking the gradient flow from the relaxed merging operators the intermediate features of ViTs enhances task performance. Therefore, we use detached ViT intermediate features as input for the embedding module, allowing only the module to learn from the relaxed merging process. end-to-end training, we alternate updates between the embedding module and the ViT parameters, as explained in (Sec. 3.4.. We set for every 10 update cycles, the embedding layer is updated once, while the ViT parameters are updated nine times.
Pre-trained models For DeiT 30 we use TIMM [35 pre-trained models, which are available For MAE [1, we use the ImageNet- Ik fine-tuned checkpoints of the official implementation, at gitthb.. com/Facchookkeesearc.. mae For LV-ViT-S (13), we use the official pre-trained model weights, available al gighut com/ ing For image captioning using GIT [33] we use the COCO fine-tuned weights on HuggingFace [36), available at hhggingfaace, wwwaaaaee..eeeeeeeeeeeeeeeeeeeee co mi wrrsoitt//////ooo wttoof///gttbbbe base- coco For semantic segmentation using Segmenter [29]. we the Ade20K pre- tre-trained checkpoint of the official implementation, at giggtuu..................... com httmm//ssuudlleeeeeeete
at dilgithub....
Image Classification Our method and baselines are trained for 30 epochs, each with batch size of 1024. We use the AdamW optimizer with a cosine learning rate schedule and a weight decay of 0.0001. For image augmentation, MixUp, and CutMix. However, we omit MixUp and CutMix in DeiT-T and MAE training, as performs better without them. For the learning rate of method and the baselines, we conduct hyperparameter search within {0.000001, 0. 0.000005, 0.00001, 0.0001, 0.001), selecting the learning rate that achieves the best performance. As result, training and 0.000005 for end-to-end updates of the ViT parameters, with a minimum learning rate set at 0.000001. Additionally, we implement drop path rate of 0.1 only during end-to-end training when updating the ViT parameters. Regardless of whether the training is modular or end-to-end, we apply a reduction rate of r 16 train the embedding module with ViT-tiny/small/base models and r = 8 for ViT-Large models. For LV-VIT-S, we apply reduction to the first 12 transformer blocks. Meanwhile, to update the VIT parameters end-to end-to-end training, we use a different reduction rate of r = 13. We employ a temperature scale of 5 = 0.1 across different ViTs and also scale the similarity divided by 0.1 prior to the soft grouping operation.
follow the DeiT training setting, using RandAug,
learning rate of 0.0001 for modular
Image Captioning In image captioning. we train and evaluate the models using LAVIS [6]. We train the GIT models over 20, 000 iterations, using batch size of 256. We employ the AdamW optimizer along with cosine learning rate schedule. The learning rate is set to 0.0001, gradually decreasing to 0.000001, with a weight decay of 0.0001. During training, the only form of image augmentation that we employ is resizing the images to 224 x 224; no other image augmentation techniques are used. For evaluation, beam search applied with beam size of 5, incorporating both repetition and length penalties set at 1. A reduction rate of 13 is used for the GIT base model,
16

---

and r = 8 for the GIT large model. We use the model with the best performance on the validation split and report its performance on the test split.
Semantic Segmentation To apply ToMe and DTEM to the semantic segmentation task, we use the approach proposed in [3] that involves repeatedly merging tokens before each component, such as attention and the feed-forward network, and then un-merging them afterward. To be more specific, given a component of the ViT block, tokens are merged following the conventional token merging approach. For ToMe, we define the similarity between tokens using the input VIT features to the component. For DTEM, we define the similarity using the decoupled embeddings, which are produced by the decoupled embedding module that takes the same input as the component. After processing by the component, tokens are unmerged by duplicating the jointly processed tokens, as in 3]. Subsequently, the processed tokens are added with the residual connection.
We train and evaluate the model using MMSegmentation 77. DTEM is trained over 40, 000 iterations with batch size of 8, following a 40k iteration schedule. During training, we use an image size of 512 x 512, and apply the default image augmentations, which include resize, random crop, random flip, and Photometric Distortion, as outlined in [29]. We remove 40% of the tokens at each component of the transformer block to train DTEM with Segmenter- ViT-S. We apply relaxed token merging when selecting the last 64 tokens for merging, while the remaining tokens to be reduced are first selected through conventional discrete merging. We observe that this approach resolves unnecessary leakage from the clipping function and stabilizes training.
A.3 Related Work
Pruning tokens Early token reduction methods prune tokens based on their importance for task per- formance, keeping tokens in order of importance. The importance is often measured by lightweight prediction/decision module [26] 33 23] trained from task loss or using attention scores [88] to keep attentive tokens. The most widely used heuristic approach uses the class token attention scores as a metric of importance, as in [8. 20 k 19]. However, as mentioned in the Introduction, removing tokens results in information loss and degrades performance, especially when many tokens are reduced.
Combining tokens A line of works attempts to combine tokens rather than removing them. The basic idea to combine similar tokens, focusing on removing repetitive information to minimize the loss of information. However, early methods, which involved merging or grouping all tokens at certain stages, achieved limited success. PatchMerger (27 introduced soft clustering approach using an attention-like module, while TokenPooling 21 used K-medoids clustering for cluster assignment to tokens. Recently, ToMe (2) has shown that token merging can be implemented efficiently by gradually merging number of tokens at each transformer block with matching, resulting in promising performance even without training 3] In addition, numerous studies focus on designing methods that consider the importance of individual tokens while addressing information loss by merging similar tokens. EVIT [18] and Evo-Vit (38 propose to combine pruned tokens into a single token to address information loss, while BAT 200 and TPS 34] use the importance metric in conjunction with matching and clustering.
Despite their effectiveness, common aspect among these methods is their reliance on intermediate features for merging tokens. We focus on improving usch embedding for merging with decoupled embedding learned directly through continuously relaxed merging. Although not directly related, ViT (37) learns features through a grouping block, aiming to learn an implicit segmentation model without direct supervision. In contrast, our method aims to reduce the computation cost in the inference.
Additionally, DiffRate [S] proposes to learn the reduction strategy (or profile), which determines the number of tokens reduced at each transformer block, to enhance token reduction. We note that this approach is orthogonal to ours. While our method focuses on tailored features for merging and reduces a fixed number of tokens across blocks, DiffRate focuses on determining the number of tokens to be reduced in each block, relying on VIT's intermediate features for merging. We believe that jointly optimizing both components- the number of tokens reduced at each block and the features used for merging- could be a promising future direction.
We note that method can inherit the same advantage by employing identity mapping in decoupled layers.
17

---

A.4 Limitations
Although our method conceptually appears be applicable across range of domains, including different modalities like text encoding, its effectiveness has so far been validated only in the computer vision tasks with ViTs. Another limitation that, while our method enhances the computation time during inference, it does not reduce the computational cost during the training process. Devising method that can sparsify the training process, thereby improving the overall computational efficiency in both the training and inference phases, is our future direction.
A.5 Computing Resource
We do all our experiments on GPU Server consists of Intel Xeon Gold 6230 CPU, 256GB RAM, and 8 NVIDIA RTX 3090 GPUs (with 24GB VRAM).
A.6 More Visualization Results
In Fig. 6. we provide visualization results of token merging, i.e., ToMe and DTEM. we visualize the token merging by color coding the tokens belonging to the same group with identical colors, comparing ToMe [2) and modularly trained DTEM. The result highlights the difference in grouping induced by the decoupled embedding. DTEM prioritizes merging background patches and allocates more patches to foreground objects. Conversely, ToMe allocates more tokens the background, indicating a less focused approach to foreground objects.
18

---

Image
ToMe
DTEM
Image
ToMe
DTEM
SICOFICALLY DISIGND FOR
Figure 6: More visualization of merged tokens. We apply a reduction profile with r = 16, leading to 11 tokens remaining the final output.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce the scope and contributions of our paper based on method and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of method in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Our method uses the existing relaxed top-k operator. We clearly state and cite the references in the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We report all the training settings and evaluation protocols in the main paper and appendix for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We include the code our submission, does not include scripts to reproduce all experimental results for our method, as the code release is not fully prepared for some parts. However, we do include the script to reproduce the results of our main classification experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Sublic/guides httpe//guuiideeeeeuuuiieeeeeee (oodeSummissionooll1
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
cilips.cc/publle http://ullccgguuddeeeeeeeeeeeeee httpa///ggiidee/eeeeuuubbbibsee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We do our best to explain aall the training and testing details in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experiments require extensive GPU computation cost and time; thus, we report only the performance as the baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the information in the appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::::///neuiiie http//eurip...e/ httpc////cccc/tcccccciiellllllll
Answer: [Yes]
Justification:
We confirm that the NeurIPS Code of Ethics has been followed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper focuses on the efficient inference of ViTs, and it appears that there is direct path to any negative applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all main the code packages, datasets, and pretrained models in both the paper and the appendix.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We include the code for our method along with the running script.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "scrX0nGLbn", "reviewer": "Reviewer_PcYu", "summary": "In this paper, decoupled token embedding for merging (DTEM) is proposed for more efficient and effective token merging. \nIt employs a lightweight embedding module to obtain a feature vector which is solely used for token merging process. To train this embedding module, DTEM uses the relaxed merging method based on continuous matching scores.\nExtensive experimental results on image classification, image captioning, and segmentation show that the proposed algorithm outperforms the conventional token merging algorithms.", "questions": "In overall, I could not find the critical flaws in this work. Even though the proposed algorithm is simple, but it technically sounds to me. Also, it shows good performances over various computer vision tasks. For my concerns, please see the weakness section.", "limitations": "Yes, in Section A.5.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand.\n2. The proposed algorithm is simple but technically sounds. \n3. The proposed algorithm achieves the better performances than other token merging methods on various datasets.", "weaknesses": "1. Why [18] is not compared in Table 1?\n2. It would be better if there is the analysis on alternative design choices for soft grouping (eq 6-7) and soft merging (eq 9-10). \n3. In Figure 3, the results within the limited reduction range (31-41% or 31-49%) are reported. More results with diverse reduction rates would be helpful to validate the effectiveness of the proposed algorithm.\n4. The discussion on the experimental results in Section 4.4 is minimal. More explanation would be helpful for readers.\n\nMinor:\n5. Figure 3 and 4 are Tables, as referred in L284 and L302.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. Why is [BAT, 18] not compared in Table 1?\n\n&rarr; Table 1 showcases the results for methods applied to a pretrained, frozen ViT with modular training, whereas BAT was originally proposed and evaluated in an end-to-end training setting without results from modular training. While it seems possible to apply BAT to a frozen ViT, the lack of a publicly available implementation for BAT precludes us from conducting meaningful adaptations or comparisons under this condition (only training logs are publicly available).\n\n> W2. It would be better if there is an analysis on alternative design choices for soft grouping (eq 6-7) and soft merging (eq 9-10).\n\n&rarr; Following the reviewer's comment, we conducted further analysis for soft grouping and merging.\n\nFor the analysis on soft grouping, we compared several approaches: (1) integrating a Gumbel softmax with the top-1 operation from ToMe to enable differentiation, (2) applying DynamicViT to a frozen ViT setting (pruning), and (3) our method, which uses a modified relaxed top-k operation, in (Table S.5). The results indicate that our proposed design for soft grouping performs the best.\n\nNext, we experimented with hard merging, using discretization and actual discarding of tokens. We observed that this led to divergence in training, confirming that soft merging is essential for our method.\n\n\n> W3. In Figure 3, the results within the limited reduction range (31-41% or 31-49%) are reported.\n\n&rarr; Following the reviewer’s suggestion, we report Figure 3 results across a broader reduction range, in (Table S.7). The result shows that our method is particularly effective in challenging, more resource-constrained settings with higher reduction rates.\n\n> W4. The discussion on the experimental results in Section 4.4 is minimal.\n\n&rarr; We acknowledge the need for a more detailed explanation in Section 4.4, particularly concerning the importance of decoupled embedding (Table 4) and module design (Figure 6).\n\nTo be more specific, in Table 4, we successively add components: (1) soft token merging and (2) decoupled embedding module to ToMe. When only (1) soft-token merging is applied, the gradient from soft-grouping and merging is directly passed through the intermediate features of ViT. The results show that not only the gradients from the merging policy (with soft-grouping and merging) but also the decoupled embedding module, detached from the intermediate features of ViT, are crucial.\n\nIn Figure 6, we experimented with an MLP embedding module and demonstrated that (1) an affine transformation is sufficient while an embedding dimension of 64 provides the best trade-off between computation cost and performance. \n\nIn the revised version of the manuscript, we will make the description more comprehensive and clear.\n\n\n> W5 (Minor). Figure 3 and 4 are Tables, as referred in L284 and L302.\n\n&rarr; We thank you for pointing it out. We will correct it in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iqroMC3sSk", "reviewer": "Reviewer_FV5y", "summary": "This paper works on the topic of visual token merging to improve the efficiency of ViT. Specifically, this work introduces decoupled token embedding for merging (DETM) which learns decoupled embedding via an additional module. Bu introducing the soft grouping and soft merging scheme, the proposed method is differentiable and thus,  parameters of the decoupled embedding module can be updated by gradient.", "questions": "Pure similarity based token merging may not be favorable. Besides, information will be lost with each merging. Is that possible to include additional cue to localize more important area and do token merging based on that?\n\nThe evaluation in this work is done in the image domain with standard ViT and configuration. In this case, the number of redundant token will not too much. It is also known that smaller patch size or larger input resolution may lead to better performance but suffer from the high computational cost. How about evaluate the token merging method in these scenarios?", "limitations": "mentioned in the weakness.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written, which clearly demonstrates its motivation, methodology, experiment setup and the final results. Each part is easy to follow and the provided details should be enough for the re-implementation. Including the source code is a plus for re-implementation and understanding more details of the proposed method.\n\n2. The token merging is a challenging topic in the community, which is still under exploration. The expensive computational cost of the transformer-based architecture has become the bottleneck and prevent it from further scaling up.  So, it is very important to invest effort in this topic to improve the throughput and efficiency of the ViT while maintaining/improving the performance. \n\n3. This paper evaluate the effectiveness of the proposed method in various application direction, which is great. As is mentioned in the Line 68, these applications requiring a different level of granularity in representation.", "weaknesses": "1. It is argued that the similarity based token merging may not be optimal. This work introduces an additional embedding module to tackle this issue. So the decoupled visual embedding will be used for token merging only. It is concerned that if the capacity of proposed decoupled embedding module is able to learn different aspects compared to its input -- the original visual features. If so, how to regulate the decoupled feature is still representative of the original feature. \n\n2. The original motivation of this work is to reduce the number of tokens and thus improve the efficiency. However, the proposed decoupled embedding module and iterative soft grouping would increase computational cost. There are some genera ablation study between the final performance and throughput/GFlLOPs. It is wondered how this is related to the hyperparameters in the above design.\n\n3. Compared to previous works, the improvement of the proposed method is marginal, which generates the question that if it is worth to follow the design mentioned in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. It is concerned that if the capacity of the proposed decoupled embedding module is able to learn different aspects compared to its input--the original visual features. If so, how to regulate the decoupled feature is still representative of the original feature.\n\n&rarr; We clarify the main purpose and implementation of the decoupled embedding module.\nThe module is specifically introduced to extract tailored embeddings suitable for merging from the original visual (intermediate) features. Thus, it takes the intermediate features as input and intended to output embedding vectors that are focusing on aspects beneficial for token merging. The module is trained via soft grouping and merging to emphasize merging advantageous aspects while relying on the original features.\n\nOur experimental results demonstrate that the decoupled embedding module, trained through soft grouping and merging, significantly alters the merging policy. Specifically, our method achieves improved performance and computational cost trade-offs. Moreover, the distinct token merging patterns are observed in our appendix visualizations, which show the difference between merging patterns using intermediate features and those merged using the decoupled features.\n\nWe also note that the decoupled embeddings are utilized solely for the purpose of merging and are refined through a process of soft token merging. During such training, the remaining ViT parameters are not updated, ensuring that the original features are preserved by the learning of the decoupled embeddings.\n\nIf there has been any misunderstanding of your concerns, we are willing to provide further clarification during the discussion period.\n\n> W2-1. The proposed decoupled embedding module and iterative soft grouping would increase computational cost. \n\n&rarr; We would like to clarify that our primary goal is to enhance computational efficiency during inference, or \"test-time,\" which is crucial for real-world applications.\n\nIndeed, the soft grouping and merging operations incur additional computational overhead, but they are applied only during the training phase. In inference, despite the computation in the additional embedding module, our method offers an improved performance/computation cost trade-off (Figure 2).\n\nMoreover, our experimental results, as shown in Figure 5b, demonstrate that the decoupled embedding module and associated training processes are able to converge rapidly. This mitigates the potential increase in computational costs during the training phase. We also note that, in the end-to-end training scenario, we further optimize computational costs by updating the embedding module parameters less frequently, as detailed in lines 199-208 of the main paper.\n\nThus, our method demonstrates a minimal increase in computational costs during training while significantly enhancing efficiency during inference.\n\n\n> W2-2. Impact of hyperparameters to our method.\n\n&rarr; To address the reviewer's concerns, we report more analysis related to hyperparameters, regarding (1) the number of steps in soft-grouping and (2) temperature scaling. We list the results in (Table S.1) and (Table S.2) of rebuttal pdf, respectively.\n\nFor the number of steps (equals to the reduction rate in training) r in soft-grouping, we observe that the decoupled embedding module, when trained with a high reduction rate r, generalizes well to lower rates. Therefore, it is generally sufficient to set the number of steps to the maximum number of tokens we want to reduce during inference.\n\nFor temperature scaling, we observed that values within the range of 0.1 to 0.3, tested with increments (0.05, 0.1, 0.2, 0.3, 0.5, 1.0), consistently provide gains with an accuracy difference of 0.1%.\n\n> W3. The improvement of the proposed method seems marginal.\n\n&rarr; While the improvements may seem modest, we believe they are reasonable and significant considering the differences from previous methods (as shown in Table 5).\n\nMoreover, our method introduces flexibility in training (modular and end-to-end) and achieves significant improvements compared to previous methods when trained modularly (Table 1). It even sometimes achieves performance comparable to that of end-to-end trained models (see Table 2).\n\nAdditionally, unlike many existing methods that require separate models for different reduction rates, our approach works effectively across various rates with a single model, making deployment simpler and more cost-effective.\n\n\n> Q1. Is it possible to include additional cues to localize more important areas and do token merging based on that?\n\n&rarr; We interpret this question as inquiring whether it is possible to consider the importance of tokens (or areas) when conducting token merging. We argue that our method inherently incorporates this aspect into its merging policy. In our method, the regions where tokens are merged (as shown in the appendix visualization) demonstrate that merging predominantly occurs in less important background areas. This pattern emerges because our decoupled embedding is specifically trained to recognize and prioritize tokens of lesser importance for classification. This indicates that our method learns to define similarity based on importance, favoring merging in less important regions, thereby preserving essential information in areas of greater significance.\n\n> Q2. How about evaluating the token merging method in smaller patch size or larger input resolution settings?\n\n&rarr; Following the reviewer's comment, we conducted experiments with a larger number of tokens on image classification (ViT-small with resolution 384x384), which corresponds to smaller patches or higher input resolutions. \n\nThe results (Table S.3 in rebuttal pdf) show that our method can adapt to settings with an increased number of tokens, achieving performance gains. We also note that the segmentation tasks were conducted at a resolution of 512x512.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "47l8jsVgX9", "reviewer": "Reviewer_wYAw", "summary": "This paper proposes DETM, which calculates similarity through additional embeddings instead of the original intermediate features. Additionally, it further introduces soft grouping and soft merging to make the merging process differentiable.", "questions": "Please refer to the weaknesses section for detailed concerns.", "limitations": "The authors have discussed potential risks in the limitations section. Additionally, similar to other token reduction works, this study only conducts experiments on plain ViT architectures.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The method is straightforward and consistently improves performance.\n3. Comprehensive experiments demonstrate the effectiveness of the proposed method across different tasks, including image classification, image segmentation, and image captioning.", "weaknesses": "1. There is a lack of discussion and comparisons with recent related works, such as [1] and [2].\n2. While the experiments show improved performance, I have concerns about the mechanism of introducing additional embeddings. The paper argues that original token merging is less effective because intermediate features are used for both encoding and merging. However, I do not believe using intermediate features directly is a drawback. The primary motivation of ToMe is to merge tokens with similar intermediate features. For example, if the similarity of two intermediate tokens is 1, merging these tokens is lossless, and additional embeddings are unnecessary in such scenarios. Therefore, the authors should provide more intuitive examples or an in-depth analysis beyond experimental comparisons to demonstrate the necessity of introducing additional embeddings.\n\n\n\n[1] Diffrate: Differentiable compression rate for efficient vision transformers, ICCV 2023\n\n[2] A Simple Romance Between Multi-Exit Vision Transformer and Token Reduction, ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. There is a lack of discussion and comparisons with recent related works, such as [DiffRate] and [METR]. \n\n&rarr; We appreciate your feedback regarding the need for discussions and comparisons with recent related works, specifically [DiffRate] and [METR].\n\nDiffRate focuses on determining the number of tokens to be reduced in each block, a concept that is orthogonal to our approach. Our method consistently reduces a fixed number of tokens across blocks, while focusing on tailored features for merging. In contrast, DiffRate relies on ViT's intermediate features for merging. Jointly optimizing both components—the number of tokens reduced at each block and features for merging—could be a promising future direction to enhance the efficiency and effectiveness of token reduction.\n\nOn the other hand, METR addresses the inconsistency between CLS token attention and the importance of tokens in the early blocks of ViTs, particularly for CLS token-based token reduction. Although METR shows improvements in selecting less significant tokens to prune, it heavily relies on the presence of a CLS token and does not directly extend its benefits to non-classification tasks without CLS tokens. In contrast, our method can be applied without a CLS token and is applicable to various vision tasks.\n\nWe will update our main paper to include a detailed discussion and comparison with both works.\n\n> W2. However, I do not believe using intermediate features directly is a drawback. The primary motivation of ToMe is to merge tokens with similar intermediate features. For example, if the similarity of two intermediate tokens is 1, merging these tokens is lossless, and additional embeddings are unnecessary in such scenarios. Therefore, the authors should provide more intuitive examples or an in-depth analysis beyond experimental comparisons to demonstrate the necessity of introducing additional embeddings.\n\n&rarr; In scenarios where token similarity is perfect (similarity equals 1), as the reviewer exemplified, merging based on intermediate features can indeed be lossless and sufficient. However, in practice, merging even highly similar tokens mostly leads to some degree of information loss, which becomes particularly significant when substantial token reduction is required. Thus, in practice, it is crucial for ViTs to preserve important information for tasks while merging tokens in areas that are less crucial for the task at hand. Note that this is a key heuristic also used in prior token pruning methods.\n\nThe main limitation of relying solely on intermediate features for merging is that these features are optimized primarily for encoding, not for identifying less important regions. This can lead to indiscriminate token merging that sacrifices important details in more critical regions. In our method, decoupled embeddings encourage the reflection of the importance of information, guiding the merging process to occur predominantly in less important regions, as demonstrated in our appendix visualizations. These visual comparisons between traditional intermediate feature-based merging (ToMe) and our decoupled embedding-based approach (DTEM) clearly show that while intermediate features tend to merge tokens uniformly across background and object regions, our decoupled embeddings favor merging in the background, thus preserving valuable object information in the foreground.\n\nTo further validate our intuition, we investigated whether the decoupled embeddings used for merging diverge from the intermediate features as learning progresses. We monitored changes in the Kendall rank correlation between token similarities derived from two different features: self-attention keys (as in ToMe) and decoupled embeddings. The results, shown in (Table S.4), indicate a decreasing correlation as learning progresses, suggesting that the decoupled embeddings seek a different measure of similarity for merging. This supports our intuition that the importance of token merging and token similarity with the intermediate features are not always aligned.\n\nMoreover, this intuition suggests that our method will be particularly effective under high reduction rates where information loss is more likely. This is aligned with our experimental results, indicating that our method becomes more effective with substantial token reduction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BiLOrPU4mh", "reviewer": "Reviewer_oGzP", "summary": "This article provides a novel way to enhance the efficiency of token merging within ViTs. DTEM, the proposed method, introduces a lightweight embedding module that operates independently from the ViT's forward pass, overcoming the constraints imposed by utilizing intermediate features. DTEM can be integrate with existing ViT backbones and be trained either modularity by focusing solely on the decoupled embeddings or end-to-end by fine-tuning the entire network. The author shows that DTEM performs well in classification, captioning, and segmentation, demonstrating consistent improvements in token merging efficiency.", "questions": "1.\tThough claimed decoupling, the overall process is similar to self-attention process especially in end-to-end training scenario. How to prove the decoupling effect except from the classification results?\n2.\tWhat are the main improvements of this method compared with ToMe? Does ToMe use the same soft grouping method? If not, it is not reflected in the ablation experiment.\n3.\tHow to ensure a specific reduction rate?\n4.\tThe comparison of soft grouping with other method such as Gumbel-Softmax in dynamic ViT[1].\n[1 ] Rao Y, Zhao W, Liu B, et al. Dynamicvit: Efficient vision transformers with dynamic token sparsification[J]. Advances in neural information processing systems, 2021, 34: 13937-13949.", "limitations": "The authors have claimed its limitation for:\n1.\tOnly in computer vision task\n2.\tDoes not reduce the computational cost in the training process", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe idea of decoupling seems novel for enabling the continuous relaxation of the token merging process, facilitating differential learning of the decoupled embeddings.\n2.\tThe paper is clearly presented and easy to follow.\n3.\tExperimental results are promising. The method's applicability is demonstrated across multiple domains, including classification, captioning, and segmentation, illustrating its robustness and versatility.", "weaknesses": "1.\tThe model training process involves many hyperparameters, such as step in soft grouping, m in soft merging. Will the determination of these hyperparameters bring complexity to the model implementation? There is no ablation study on parameter m in soft merging.\n2.\tThere is no ablation experiment in segmentation and caption to prove the effectiveness of the module on this task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. Will the determination of these hyperparameters bring complexity to the model implementation? There is no ablation study on parameter m in soft merging.\n\n&rarr; We report further analysis related to hyper parameters, regarding (1) the number of steps in soft grouping and (2) temperature scaling. We list the results in (Table S.1) and (Table S.2) of rebuttal pdf, respectively.\n\nFor the number of steps (equals to the reduction rate during training) r in soft-grouping, we observe that the decoupled embedding module, when trained with a high reduction rate r, generalizes well to lower rates. Therefore, it is generally sufficient to set the number of steps to the maximum number of tokens we want to reduce during inference.\n\nFor temperature scaling, we observed that values within the range of 0.1 to 0.3, tested with increments (0.05, 0.1, 0.2, 0.3, 0.5, 1.0), consistently provide gains with an accuracy difference of 0.1%.\n\nWe clarify that the effective size m in soft merging is not a hyperparameter but a vector representing the sizes of combined tokens following the Equation 9. We also conducted experiments to determine if this effective size m and proportional attention (Equation 3) are necessary in our implementation by removing them. The result in (Table S.5. w.o. prop-attn) shows that both the effective size and the proportional attention based on it are crucial in our method.\n\nAs a result, we believe that the hyperparameters in our method do not introduce significant complexity to our model implementation.\n\n\n> W2. There is no ablation experiment in segmentation and caption to prove the effectiveness of the module on this task.\n\n&rarr; Following the reviewer's comment, we further report ablation experiment results in captioning and segmentation to demonstrate the importance of the decoupled embedding module by varying the decoupled embedding dimension. The results (Table S.6) show that the decoupled embedding module indeed directly affects the quality of token merging. We also note that modular training is infeasible without the decoupled embedding module, which has external parameters independent from the ViT parameters.\n\n> Q1. How to prove the decoupling effect except from the classification results?\n\n&rarr; To this end, we further investigate whether the decoupled embedding used for merging diverges from the intermediate features as learning progresses. We monitor changes in the Kendall rank correlation between token similarities derived from two different features: self-attention keys (as in ToMe) and decoupled embeddings. The results in (Table S.4) show a decreasing correlation as learning progresses, indicating that the decoupled embeddings seek a different measure of similarity for merging, thereby verifying the benefits of decoupling.\n\n> Q2. What are the main improvements of this method compared with ToMe?\n\n&rarr; The main improvements of our method over ToMe can be summarized as follows:\n\n1. Decoupled embedding for improved trade-off: Unlike ToMe, which uses intermediate ViT features for both encoding and token merging, our method introduces a decoupled embedding that distinctly separates the features used for token merging from those used for encoding. Our experimental results, detailed in (Table 3,4), demonstrate that this separation significantly enhances the performance and computational cost trade-offs by optimizing the merging policy independently.\n2. Modular training on frozen ViTs: The decoupled embedding modules exist independently of the ViTs. This enables the improvement of token merging on top of frozen ViTs without altering the original ViT parameters. This modular approach is infeasible with ToMe, as it relies on ViT's intermediate features for merging, limiting its ability to enhance merging only through end-to-end training.\n\nOur ablation studies, shown in Table 4, explore the impacts of incorporating (1) soft grouping and merging operations, and (2) the decoupled embedding module into the ToMe. We also note that combining both components converges to our method. The results show that our approach, which integrates both components, achieves the best trade-off between performance and computational efficiency.\n\n\n> Q3. How to ensure a specific reduction rate?\n\n&rarr; As explained in W1, we observed that the decoupled embedding module, when trained with a high reduction rate r, generalizes effectively to lower rates. Therefore, during training, we set the number of steps (equivalent to the reduction rate) to the maximum number of tokens we aim to reduce during inference.\n\nDuring inference, we adjust the reduction rate as needed based on this generalization capability. Meanwhile, if there are specific targets for GFLOPs or throughput that need to be met, these can be achieved by iteratively adjusting the number of tokens merged at each ViT block, similar to the approach used in ToMe.\n\n\n> Q4. The comparison of soft grouping with other methods such as Gumbel-Softmax in [DynamicViT].\n\n&rarr; Some previous work in token pruning, such as DynamicViT, optimizes the selection of tokens to discard by employing a differentiable selection operator, such as the Gumbel-softmax. However, these approaches primarily focus on selecting individual tokens and thus are not directly applicable to token merging, which requires selecting pairs of tokens. A key contribution of our method is that we enable learning through token merging via our soft-grouping and merging operators.\n\nFor the analysis on soft grouping, we compared several approaches: (1) integrating a Gumbel softmax with the top-1 operation from ToMe to enable differentiation, (2) applying DynamicViT to a frozen ViT setting (pruning), and (3) our method, which uses a modified relaxed top-k operation, in (Table S.5). The results indicate that our proposed method for soft grouping performs the best.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time and effort in providing constructive reviews. We appreciate the encouraging remarks about the paper's novelty (oGzP), technical soundness (PcYu), promising experimental results (oGzP), and applicability (oGzP, wYAw, FV5y). We are happy to respond to the weaknesses (W) and questions (Q) in the comments and hope that our responses address your concerns. Due to the text limit, we have included all rebuttal experimental results (Tables S.1 to S.7) in the rebuttal PDF.

Meta-review: This paper proposed a decoupling method to enable continuous relaxation of token merging. The novelty is well-recognized by the reviewers. The authors demonstrate the effectiveness of the proposed method via extensive experiments. Three reviewers provided positive feedback, except for the reviewer FV5y, who has major concerns about the side effects of the parameter reduction. The authors carefully addressed the raised issues. Unfortunately, the reviewer  FV5y didn't provide further feedback. The Meta-review agrees with the clarity presented in the author's rebuttal.
Decision: Accept (poster)