Title: Accelerating Transformers with Spectrum-Preserving Token Merging
Venue: NeurIPS 2024
Accepted: yes
Keywords: token merging

vision transformer

model compression
Abstract: Increasing the throughput of the Transformer architecture, a foundational component used in numerous state-of-the-art models for vision and language tasks (e.g., GPT, LLaVa), is an important problem in machine learning. One recent and effective strategy is to merge token representations within Transformer models, aiming to reduce computational and memory requirements while maintaining accuracy. Prior work has proposed algorithms based on Bipartite Soft Matching (BSM), which divides tokens into distinct sets and merges the top $k$ similar tokens. However, these methods have significant drawbacks, such as sensitivity to token-splitting strategies and damage to informative tokens in later layers. This paper presents a novel paradigm called PiToMe, which prioritizes the preservation of informative tokens using an additional metric termed the \textit{energy score}. This score identifies large clusters of similar tokens as high-energy, indicating potential candidates for merging, while smaller (unique and isolated) clusters are considered as low-energy and preserved. Experimental findings demonstrate that PiToMe saved from 40-60\% FLOPs of the base models while exhibiting superior off-the-shelf performance on image classification (0.5\% average performance drop of ViT-MAEH compared to 2.6\% as baselines), image-text retrieval (0.3\% average performance drop of Clip on Flick30k compared to 4.5\% as others), and analogously in visual questions answering with LLaVa-7B. Furthermore, PiToMe is theoretically shown to preserve intrinsic spectral properties to the original token space under mild conditions.

Accelerating Transformers with Spectrum-Preserving Token Merging

Abstract

Increasing the throughput of the Transformer architecture, a foundational component used in numerous state-of-the-art models for vision and language tasks (e.g., GPT, LLaVa), is an important problem in machine learning. One recent and effective strategy is to merge token representations within Transformer models, aiming to reduce computational and memory requirements while maintaining accuracy. Prior works have proposed algorithms based on Bipartite Soft Matching (BSM), which divides tokens into distinct sets and merges the top k similar tokens. However, these methods have significant drawbacks, such as sensitivity to tokensplitting strategies and damage to informative tokens in later layers. This paper presents a novel paradigm called PITOME, which prioritizes the preservation of informative tokens using an additional metric termed the energy score . This score identifies large clusters of similar tokens as high-energy, indicating potential candidates for merging, while smaller (unique and isolated) clusters are considered as low-energy and preserved. Experimental findings demonstrate that PITOME saved from 40-60% FLOPs of the base models while exhibiting superior off-the-shelf performance on image classification (0.5% average performance drop of ViT-MAEH compared to 2.6% as baselines), image-text retrieval (0.3% average performance drop of CLIP on Flickr30k compared to 4.5% as others), and analogously in visual questions answering with LLaVa-7B. Furthermore, PITOME is theoretically shown to preserve intrinsic spectral properties to the original token space under mild conditions. Our implementation is available at this link.

1 Introduction

Vision Transformers (ViTs) [1] have been integral to recent advancements in computer vision, leading to state-of-the-art deep learning architectures for representing images and videos [2-5]. However, these transformer-based architectures incur substantial memory costs and have a quadratic time complexity in the number of tokens due to the self-attention layers. This challenge becomes particularly severe as model sizes increase, as observed in Large Language Models (LLMs) [6].

To address such limitations, several efforts focus on designing a more efficient attention mechanism by making it linearly scale with input tokens [7, 8], integrating vision or language domain-specific modules [9, 10], or pruning the head numbers in ViT [11, 12]. Others propose dynamically pruning less important tokens w.r.t. pre-defined metrics using learnable masks [13, 14]. However, a primary downside of these novel methodologies lies in the necessity to retrain the model from scratch, therefore hindering the leveraging of well-trained models such as LLMs. Moreover, most pruningbased techniques may not accelerate the training process, which arises from the dynamic removal

ViT-B 384

PiToMe (ours)

ToMe

DiffRate

of tokens in each sample, resulting in a mismatch of dimensions and consequently preventing the batching of samples with consistent dimensions.

Recent research has introduced a novel token merging technique. Instead of pruning, this method combines tokens with high semantic similarity, removing background tokens and merging less informative foreground ones. Its versatility extends to training and non-training scenarios, drastically reducing compute and memory usage. A notable example is ToMe [15], which introduced the Bipartite Soft Matching (BSM) algorithm, prominent for its simplicity and effectiveness in merging highly similar tokens. Since ToMe, several works, including ToFu [16], Pumer [17], LTPM [18], and DiffRate [19], have built upon BSM with various adaptations in vision and language domains. In BSM, tokens representing image patches are separated into sets A and B , and their pairwise cosine similarity is computed. The top k similar pairs of tokens between the sets A and B are merged. However, the performance of this algorithm is sensitive to the token-splitting strategy. For instance, ToMe's approach, which first splits tokens based on index parity, can lead to incorrect merging since tokens in A can subsequently only be merged with those in B (Figure 1). Moreover, while BSM excels in initial layers with many redundant tokens, deeper layers risk merging informative tokens due to latent object correlations. Though current enhancements [19] mitigated this by considering token attention scores in BSM [20], their adaptability to different ViT architectures, each with potentially distinct attention score distributions [21], remains a challenge.

In this work, we propose PITOME (Protect Informative Tokens before Merging), a method designed to safeguard crucial information-bearing tokens prior to the merging step. Our method prioritizes preserving informative tokens by utilizing an additional metric termed the energy score inspired by connections to graph energy in spectral graph theory [22, 23] (Theorem 1). Specifically, our energy score assesses large clusters of similar tokens as possessing high energy (like background and repeated textures), thereby marking them as suitable candidates for merging, while smaller, distinct regions (foreground) are deemed low-energy and thus treated as protected informative tokens. The proposed energy term operates on the graph built for input tokens, taking into account their relationships and aggregating information from nearby neighbors when their similarities exceed certain thresholds. This approach facilitates a deeper contextual comprehension compared to previous works [15-17, 19] that rely solely on attention scores or feature embedding per token. Subsequently, we only select the highest-scoring tokens and pass them on for merging in the next steps, ensuring the preservation of important tokens, particularly in the latter stages when only a few remaining ones. During the merging process, we continue leveraging sorted energy vectors from earlier stages by distributing tokens with similar energy into two sets, A and B , resulting in candidates in A having a high probability of finding compatible matches in B . Matched tokens are then merged using a weighted average feature embedding to create a new token representation.

The empirical results demonstrate that despite the increased computational cost associated with energy score calculations, PITOME exhibits comparable speed to other BSM-based approaches since the matching is performed on a smaller, high-energy token set. At the same time, it consistently shows superior accuracy across various experimental scenarios. Additionally, we present theoretical insights into PITOME, showing that, under moderate assumptions - such as the discriminative nature of feature embeddings generated by ViT for node pairs within and across distinct objects our algorithm efficiently preserves the spectral properties of the initial input tokens, maintaining the eigenvalues derived from normalized Laplacian matrices of the original tokens [24-26]. To summarize, our contributions encompass:

A new token merging procedure for accelerating ViT architectures is designed to protect crucial yet small-region tokens while identifying redundant ones for merging based on contextual token correlations captured by our energy score functions.

Our PITOME runs as fast as other BSM-based approaches while achieving SOTA performance on diverse tasks, ranging from image-text retrieval (Sec. 4.1), visual question answering with LLMs (Sec. 4.2), image classification (Sec. 4.3), and text classification (Sec. 4.4). In several cases, PITOME is shown to reduce up to 40 -60% FLOPs of base models while only dropping performance around 0 . 3 -0 . 5% (CLIP model on Flick30k).

We also present theoretical findings indicating that, given reasonable assumptions, PITOME can effectively approximate the spectral distance between the initial token spaces and the merged token set. This sheds light on why PITOME tends to outperform baselines in practical applications and contributes to a better understanding of the potential limitations inherent in BSM-based methods, such as those in [15, 16, 19, 17, 27].

2 Related Work

Efficient Attention Mechanisms. Various efforts have sought to enhance the efficiency of transformers in both NLP and Vision domains. Some concentrate on accelerating attention computation [28, 29, 8] through approximation techniques involving hashing [30], low-rank [31], or sparse approximations [32]. Others explore strategies such as head or feature pruning [11, 33] or the integration of domain-specific modules [9, 5, 34, 10]. However, many of them necessitate joint training with the backbone model from scratch. For instance, DynamicViT [35] runs approximately 150 hours of fine-tuning on an NVIDIA A100 GPU to prune the DeiT-S model [36]. In contrast, we focus on accelerating existing ViT models by token merging, which applies to training and non-training scenarios.

Dynamic Token Pruning. Several studies have explored token pruning in transformer models across NLP [37-39] and vision domains [40-42, 27]. However, like efficient transformers, these methods typically require training. Additionally, most pruning techniques are dynamic, meaning the number of tokens varies across different inputs, which improves accuracy but complicates batching for practical deployment. To address this, numerous pruning methods employ masks during the training phase rather than directly eliminating tokens; however, it yields to cancel out the speed advantages associated with pruning.

Token Merging. Leading techniques such as ToMe [15] and its improvements [17, 43, 18, 19, 16, 44], build upon lightweight Bipartite Soft Matching (BSM). These methods exhibit speeds comparable to pruning while achieving superior performance. They have demonstrated the ability to double the throughput of state-of-the-art Vision Transformers (ViT) on both images and videos with minimal accuracy degradation in various scenarios. However, BSM-based approaches are sensitive to the selection of sets in the matching process, potentially resulting in the loss of informative tokens due to heuristic merging procedures. To address these issues, methods like DiffRate [19] and Crossget [44] leverage attention scores in ViT or cross-modal guidance to identify important tokens during the matching process, though they remain sensitive to the distribution of the token space, especially with imbalanced clusters. Another direction involves adapting more intricate algorithms, such as k-means [45], spectral clustering [46], graph pooling [47], or graph coarsening [24, 48], to merge similar tokens. While these strategies offer some guarantees and well-controlled outputs, their iteration schemes are highly complex and may not align with the goal of reducing model complexity in ViT layers. Our PITOME, on the other hand, enables the advantages of both approaches. It maintains efficiency comparable to BSM, remains robust to token partitioning strategies, and offers a reasonable trade-off between speed and accuracy. Moreover, PITOME is theoretically proved to approximate the spectral spectrum of the original token space under reasonable assumptions, resembling the behavior of other spectral clustering methods.

3 Methodology

3.1 Token Merging Formulation

We apply token merging to each transformer block of the ViT architecture (Figure 2-a). Given the input token of the l -th block X l ∈ R N × h where N and h are the token length and token hidden embeddings, a forward step in one Transformer block can be formulated as follows:

where Attention and MLP are the self-attention and multiple layer perceptron components. We then apply merge operations on ˆ X l and compute the output of the reduced MLP block as:

Normal node

v

i

Protected nodes :

s

[2

k

:]

Mergeable nodes:

merge

s

[: 2

k

]

New merged node

Step 1

: Build input token graph

G

(

V

,

E

,

W

)

,

|V|

=

N

Step 2

: Defined mergeable nodes

and protected nodes

Step 3:

Split mergeable nodes into

two sets

A

and

B

E

)

Step 4

: Merge nodes in

A

to their closest

neighbor in

B

using BSM,

|V|

=

Nr

Layer 1

b)

c)

Q

∈

R

N

×

h

X

l

∈

R

N

×

h

K

T

∈

R

h

×

N

V

∈

R

N

×

h

Softmax

PiToMe

Attention Block

MLP

X

l

+1

∈

R

rN

×

h

a)

√

h

...

s

= Argsort(

E

(

v

i

,

W

[

i,

:]))

Edge with weight

W

[

i, j

]

High

Low

merge

[:: 2]

merge

[1 :: 2]

Layer 6

Layer 12

Caption:

"An orange cat is hiding in

the wheel of a red car. "

...

Here F mer ( . ) is the merging operation that receives ˆ X l as input for compressing, X l W K (key matrices) as the token features of ˆ X l following prior work [15, 43, 18, 19], and r is the fraction of remaining tokens. The output ˆ X l m ∈ R rN × h serves as input for the MLP layer to produce X l +1 ∈ R rN × h . We present the PITOME F mer ( . ) function in the next section.

3.2 Energy-based Merging

We propose to use a new term called energy score to evaluate the redundancy of each token, which is then used to protect informative or isolated tokens (low energy scores) while considering tokens that are in the large cluster as high energy scores and characterizing them as merging candidates. Figure 2-b illustrates the main steps in PITOME.

Token Graph Construction : Given a set of N token inputs in ˆ X l , we build a weighted graph G ( V , E , W ) with V a set of N = |V| nodes, E a set of M = |E| edges defined by connecting one token to the remaining ones in G , W ∈ R N × N be a weighted adjacency matrix. We opt for using the key vectors K = X l W K ∈ R N × h as node features of V , i.e., v i ∈ V has h feature dimensions. The weight W [ i, j ] assigned to an edge e ij ∈ E connects v i and v j is computed by cosine distance:

For simplicity, W [ i, :] and W [: , i ] denote the i-th row and column, resp. ; [ N ] stands for { 1 , . . . , N } .

Token Energy Scores : In this step, the energy score , denoted as E = ( E i ) i ∈ [ N ] , is computed for each node (Figure 2-a, Step 2). The term is inspired by the concept of graph energy in spectral graph theory [22, 23], defined as the sum of the absolute eigenvalues of the adjacency matrix W . We also leverage such structures of W to find correlations among tokens and to estimate token redundancy. Instead of using independent token values such as attention scores [19], our energy leads to better performance (Figure 6, Appendix) and provides theoretical connections to the spectral properties of the original token graphs (Theorem 1).

Let i be the index of the current node and N ( i ) represent the set of neighbor nodes. The energy score E i ≡ E i ( v i , W [ i, :]) of node v i is calculated using the following equation:

Rather than accumulating all cos( v i , v j ) values, the function f m ( . ) in Eq.(4) mimics the exponential linear unit activation function [49], focusing on similar tokens even if they are far apart, while ignoring dissimilar ones. Here, m is a dynamic margin value varying at each layer in the ViT model. Nodes within this margin, i.e., ( x > m ) with high cosine similarity cos( v i , v j ) are considered true neighbors, potentially representing tokens belonging to the same object. Nodes outside this margin have cos( v i , v j ) replaced by a constant α , providing a lower bound for minimal edge weights. The

term exp( x -m ) -1 < 0 smooths the function f ( x ) for neighboring nodes near the margin m . In experiments, we set α = 1 . 0 and m = 0 . 9 -0 . 9 × l i /l , where l i is the current layer index and l is the total number of encoder layers, indicating an increasing margin as tokens move to deeper layers. The ablation studies for the α and m values are presented in Section 4.5.

Intuitively, Eq.(4) reflects the number of tokens potentially representing the same object. Tokens belonging to large objects (e.g., background) will have high energy scores, indicating potential candidates for merging, while smaller ones (e.g., foreground) will have low energy scores and are considered to be protected. This guides us to sort the energy vectors E in descending order and choose only the top 2 k nodes with the highest scores as mergeable candidates and the remaining ones as protective tokens, i.e, s = argsort( E ) , merge ← s [: 2 k ] , protect ← s [2 k :] , k = N -Nr .

Ordered Energy-based Bipartite Soft Matching : Having identified mergeable tokens in the merge set, we continue exploit the sorted order in E to form two sets A and B in BSM, each containing k nodes. Specifically, tokens with odd and even indices in merge are selected for A and B , resp. given the fact that those in the same object should have similar energy scores, resulting in likely distributing in consecutive positions in argsort( E ) . In other words, our choosing has a high probability that one token in A always finds its best match in the same object in B . This sets us apart with random partitions based on spatial indices in images like [15, 16].

Tracking Token Sizes All nodes in set A are then merged with their nearest neighbors in set B through the fast BSM algorithm. Following prior works [15, 16], we also add proportional attention to balance the effect of the merged token on the output of the softmax function: A = Softmax ( X l W Q · ( X l W K ) T / √ h +log m ) where m is a row vector containing the size of each token, i.e., the number of data patches the token represents. The pseudo-code for our method is provided in Algorithm 1 (Appendix) with complexity analysis.

3.3 Connection to Graph Coarsening with Spectral Preservation

In this section, we employ tools from spectral graph theory to show a spectral distance preservation of PITOME. We note that similar properties can be obtained by using more complicated clustering algorithms such as K-mean [45] or spectral clustering [46, 47, 24]; however, these methods are typically loop-based algorithms, which are computationally expensive and not suitable for batch-type data. Our PITOME, in contrast, is as fast as BSM methods but theoretically preserves spectral properties of input token graphs.

We begin by introducing Definitions 1 and 2 of graph coarsening and lifting, resp. , to justify the spectral distance constructed in equation (5), measuring the similarity between the original and coarse graphs. For more thorough coverage of the mathematics of graph coarsening and graph lifting, we refer the reader to [50-53]. In short, we treat the result of token merging as a graph coarsening process (Figure 8, Appendix). We then create the lifted graph as a reconstruction from this coarsened version to assess the spectral distance to the original token graph.

Definition 1 (Graph Coarsening) . Given a weighted graph G ( V , E , W ) , we denote P = {V i } i ∈ [ n ] where V = ∪ i ∈ [ n ] V i , be a partition of its node into n disjoint sets. The coarsened graph of G w.r.t. P is the weighted graph G c , where each partition in P is aggregated into a single node, denoted { ν i } i ∈ [ n ] , by averaging the elements within each partition. The elements of the adjacency matrix are given by W c [ i, j ] = ∑ v i ∈V i ∑ v j ∈V j W [ i, j ] / ( |V i ||V j | ) . We denote the combinatorial and normalized Laplacians of G by L = D -W and L = I N -D -1 / 2 WD -1 / 2 , resp. , where D is the diagonal degree matrix with D [ i, i ] = d i := ∑ N j =1 W [ i, j ] . Similarly, the definition of the coarsened Laplacian matrices follows directly: L c = D c -W c and L c = I n -D -1 / 2 c W c D -1 / 2 c . Finally, the eigenvalues and eigenvectors of L ( resp. L c ) are denoted as λ and u ( resp. λ c and u c ).

Definition 2 (Graph Lifting) . We call G l ( V l , E l , W l ) the lifted graph of G if the adjacency matrix elements are given by W l [ i, j ] = W c [ i, j ] . We denote the node degree of v li ∈ V l by d li = ∑ N j =1 W l [ i, j ] . The combinatorial and normalized Laplacians of G l is then defined as L l = D l -W l and L l = I N -D -1 / 2 l W l D -1 / 2 l , resp. , where D l is the diagonal degree matrix with D [ i, i ] = d li . Then, we denote, resp. , the eigenvalues and eigenvectors of L l by λ l and u l .

Lemma1 (Eigenvalue Preservation, see e.g., [50, 51, 54, 55]) . The normalized Laplacian eigenvalues of the lifted graph λ l contain all the eigenvalues of the coarse graph λ c and additional eigenvalues 1 with ( N -n ) multiplicity.

Through Lemma 1, we can use the lifted graph G l as a proxy for the coarse graph G c , and define:

Next, we present our main theoretical result demonstrating how spectral distance characterizes the superiority of our novel PITOME paradigm over the state-of-the-art approaches as ToMe [15, 16]. The Theorem 1 quantifies how similar the original G is to its coarsened counterpart G c , and is proved in Appendix E.

̸

Theorem 1 (Spectrum Consistent of Token Merging) . Suppose the graphs G ( s ) 0 , G ( s ) PITOME , and G ( s ) ToMe are coarsened from the original graph G by iteratively merging pairs of nodes v a s and v b s w.r.t. the true partition P ( s ) 0 = {V ( s ) 0 i } i ∈ [ s ] , the PITOME -partition P ( s ) PITOME = {V ( s ) PITOME i } i ∈ [ s ] , defined by PITOME in Algorithm 1, and the ToMe-partition [15, 16], P ( s ) ToMe = {V ( s ) ToMe i } i ∈ [ s ] , for s = N,... , n + 1 . We assume some standard mild assumptions: (A1) E [cos( v a s , v b s )] → 1 , ∀ v a s ∈ V ( s ) 0 i , ∀ v b s ∈ V ( s ) 0 i , i ∈ [ s ] ; (A2) there exists a margin m s.t., cos( v a s , v b s ) ≥ m > cos( v a s , v c s ) , ∀ v a s ∈ V ( s ) 0 i , ∀ v b s ∈ V ( s ) 0 i , ∀ v c s ∈ V ( s ) 0 j , ∀ i = j ∈ [ s ] ; and (A3) there is an order of cardinality in the true partition, without loss of generality, we assume N ( s ) 1 ≥ N ( s ) 2 ≥ . . . ≥ N ( s ) s , where N ( s ) i = |V ( s ) 0 i | , ∀ i ∈ [ s ] . Then it holds that:

The spectral distance between the original G ≡ G ( N ) 0 and the PITOME -coarse G ( n ) PITOME graphs converges to 0 , i.e., SD ( G , G ( n ) PITOME ) → 0 ,

The spectral distance between the original G and the ToMe-coarse G ( n ) ToMe graphs converges to a non-negative constant C , with a high probability that C > 0 .

Intuitively, Theorem 1 states that, given assumptions (i) tokens are closely embedded within classes and distinct between classes ( A1, A2 ) , and (ii) the number of tokens per class follows certain orders ( A 3) , the spectral distance between PITOME and the original tokens in Eq.(5) will converge to 0 . In contrast, with ToMe partitions, a non-eliminable constant likely remains.

4 Experiments

We focus on two settings: Off-the-Shelf Performance , where we evaluate the models' performance immediately after compression without training, and Retrained , where we treat the compression algorithms as pooling functions and retrain the models on downstream tasks. The experiments cover four tasks: (i) image & text retrieval , (ii) visual question answering (VQA) , (iii) image classification , and (iv) text classification . We use the number of floating-point operations (FLOPS) needed for inference on one sample as the main metric to benchmark memory footprint and speed. Higher FLOPS indicate greater memory requirements and longer training and inference times.

4.1 Image & Text Retrieval

We evaluate PITOME on the image-text retrieval task using three different backbone models CLIP [56], ALBEF [57], and BLIP [58] on two frequently used Flickr30k [59] and MSCOCO [60] datasets. Our experiment is benchmarked using recall@k [61], where a higher recall@k indicates the model's effectiveness in retrieval. In Figure 3, we benchmarked PITOME against other SOTA merging or pruning -based methods such as ToMe [15], ToFu [16], DiffRate [19], and DCT [62] on off-the-shelf setting when varying amount of merged tokens at each layer. Given the same FLOPS, it is clear that PITOME consistently outperforms previous compression algorithms across all backbones. The performance gap increases as we decrease the percentage r of tokens retained in each layer. The same behavior remains consistent in Table 2, where we set r = 0 . 925 and retrain pre-trained checkpoints of BLIP and CLIP. For more details about the training method, please refer to Li et al. [58].

In Table 1, we compare PITOME using compression ratios of r ∈ { 0 . 95 , 0 . 975 } on BLIP and BLIP-2 against other advanced architectures such as ViLT [63], LightningDOT [64], UNITER [65],

METER [66], CLIP-L [67], and ALBEF [68]. The results show that PITOME consistently surpasses those architectures by a significant margin. Moreover, the performance drop on the base BLIP/BLIP-2 is minimal while achieving substantial reductions in memory footprint and rerank times-nearly halving for BLIP and tripling for BLIP-2 . Additionally, the speedup can further improve with increased batch and model sizes.

4.2 Visual Question Answering (VQA) with Large Vision-Language Models

This experiment focuses on assessing the off-the-shelf performance of large vision-language models like LLaVa [2]. We extensively conduct experiments across six VQA datasets: VQA-v2 [69], GQA [70] (academic questions), VizWiz [71] (visually impaired individuals), ScienceQA [72] (zeroshot scientific question answering), TextVQA [73] (text-rich VQA tasks), and MME-Perception [74] (visual perception with yes/no question). More details on the number of samples in each dataset are in the Appendix. All experiments are conducted using LLAVA-1.5 7B and LLAVA-1.5 13B with the lmms _ eval library [75] provided by the LMMs-Lab team.

Datasets

Methods

Rt

@1

↑

Ri

@1

↑

ZS Retrieval

Rsum

↑

Reranked

Rsum

↑

ViT

FLOPS

↓

Total

FLOPS

↓

ZS

Retrieval

Time

↓

Total

Time

↓

Flickr30k

ViLT

83.50

64.40

490.60

525.70

-

55.90

-

-

LightingDOT

83.90

69.90

532.26

-

-

-

-

-

UNITER

92.87

83.73

521.90

542.80

-

949.9

-

-

METER

94.30

82.22

560.54

570.72

-

-

-

-

CLIP-L

92.90

81.34

568.23

-

80.85

-

25s

-

ALBEF

94.91

85.32

564.58

575.00

55.14

65.54

16s

58s

PITOME

BLIP

r

=0

.

95

95.72

86.32

567.58

577.81

38.55

47.65

13s

56s

PITOME

BLIP

r

=0

.

95

96.61

87.18

569.98

579.35

38.55

47.65

13s

56s

BLIP

96.86

87.48

572.24

580.76

55.14

65.54

16s

1m17s

PITOME

BLIP2

r

=0

.

95

96.83

87.84

566.25

580.77

296.93

390.77

45s

1m21s

PITOME

BLIP2

r

=0

.

975

97.55

89.04

572.81

583.72

434.50

564.78

1m5s

1m54s

BLIP2

97.61

89.79

572.72

584.76

678.45

900.77

1m37s

3m15s

MS-COCO

ViLT

61.50

42.70

420.20

439.20

-

55.90

-

-

CLIP-L

70.78

53.79

478.18

-

80.85

-

2m10s

-

METER

76.16

57.08

-

495.95

-

-

-

-

ALBEF

76.94

60.24

478.39

500.44

55.14

65.54

43s

5m29s

PITOME

BLIP

r

=0

.

95

79.46

62.50

485.99

506.65

38.85

47.65

51s

4m30s

PITOME

BLIP

r

=0

.

95

80.44

63.91

493.33

512.66

38.85

47.65

51s

4m30s

BLIP

81.82

64.36

494.34

516.03

55.14

65.54

1m3s

7m10s

PITOME

BLIP-2

r

=0

.

95

82.29

65.54

494.92

518.44

296.93

390.77

3m33s

6m34s

PITOME

BLIP-2

r

=0

.

975

84.12

67.37

504.95

527.06

434.50

564.78

5m13s

9m24s

BLIP-2

85.32

68.26

507.46

528.63

678.45

900.77

7m52s

16m12s

Let L denote the number of layers in the CLIP encoder and N the number of visually encoded tokens. In our experiment, we apply PITOME to the ViT vision encoder of LLAVA, retaining only r percent of tokens in each layer. This results in r L N tokens being fed into the LLM, significantly enhancing inference speed. We used LLaVA-1.5-7B and LLaVA-1.5-13B checkpoints to run off-the-shelf settings. Tables 3 and 4, along with Figure 4, illustrate that the PITOME algorithm consistently achieves superior performance compared to other merging and pruning methods, as well as existing SOTA models such as BLIP-2 [76], InstructBLIP [77], IDEFICS-9B/80B [78], with inference time nearly halved . Remarkably, in some datasets like VisWiz and ScienceQA, the compressed model even surpasses the baseline model. We contend that this improvement stems from the merging of less significant tokens in PITOME, potentially enhancing the robustness of the language model (LLM).

4.3 Image Classification on Imagenet-1k

In this task, we employed five ViT backbones of varying sizes-tiny (ViT-T), small (ViT-S), base (ViT-B), large (ViT-L), and huge (ViT-H) - which are pre-trained using either MAE [79] or DEIT [80] styles. These backbones were utilized to assess both off-the-shelf and retrained performance. All experiments were conducted on the ImageNet-1k dataset, which is a subset of ImageNet [81] containing labeled images spanning 1000 categories. scienceQAimage

0.695

0.685

0.690

0.680

algo

0.675

0.670

X=

baseline

pitome

tome

0.665

*+**

tofu

diffrate

0.660

dct

textvqa

0.45

0.40

0.35

algo

0.30

baseline

0.25

pitome

tome

0.20

:+**

tofu

diffrate

0.15

dct

vqav2

0.76

0.74

0.72

0.70

algo

baseline

0.68

pitome

0.66

tome

0.64

+**

tofu

diffrate

0.62

dct

eb6

0.62

0.60

0.58

algo

0.56

--

baseline

0.54

pitome

tome

0.52

*.+.*

tofu

+=

diffrate

0.50

+=

dct

0.86

0.880.90

0.92

0.94

0.96

0.98

1.00

ratio

Table 5 and Figure 5 present our experimental results, comparing PITOME with recent works, including SOTA efficient transformers such as Swin-B [82], CSWin-B [82], MViT-B/L [83], MAE [79], and other token merging/pruning methods [84, 85, 41]. We observe that PITOME maintains high accuracy with an average performance drop of only 0 . 5% after reducing up to 44% of FLOPS (MAE-H), showcasing superior performance with comparable throughput. It is important to note that dynamic pruning-based methods such as A-ViT [85], Dynamic ViT [84], and SP-ViT [13] do not accelerate training speed due to using additional masks for padding tokens into a same dimension. On the retraining settings, we note that models compressed by PITOME also surpass merging/pruning methods by a large margin and approach the performance of the original models.

4.4 Text Classification

While previous studies have focused on benchmarking BSM-based algorithms within the vision or vision-language domain, we also extend experiments to the text domain, where input sequence lengths vary by sample. Specifically, we apply compression algorithms to the first three layers of the BERT model [86], reducing the number of tokens by 20% in each layer. Our experiments utilize the SST-2 dataset [87] with an average sequence length of 23.2 tokens and the IMDb dataset [88] with an average sequence length of 292.2 tokens.

As demonstrated in Table 6 and Figure 11 (Appendix), our findings indicate that PITOME performs better than other BSM-based baselines. Additionally, after retraining, the compressed BERT models achieve competitive records while significantly accelerating training speed compared to previous pruning methods such as PowER-BERT [89], Fisher [90], and LTP [91], as well as BERT-based efficient models like DistilBERT [92] and ALBERT [93]. Notably, we observe only a 0 . 4% performance drop on the IMDb dataset and even surpass the original BERT model by 0 . 3% on the SST-2 dataset. For detailed empirical results on this task, please refer to Appendix D.

Type

Model

OTS

Acc.

Trained

Acc.

Flops

↓

Train

speed up

Other

models

Swin-B

n/a

84.0

15.4

×

CSWin-B

n/a

84.2

15.0

×

MViTv2-B

n/a

84.4

10.2

×

MViTv2-L

n/a

85.3

42.1

×

merge

ToMe DEIT-T

68.9

70.0

0.79

✓

ToFu DEIT-T

69.6

70.5

0.79

✓

DCT DEIT-T

67.6

68.7

0.79

✓

DiffRate

DEIT-T

69.9

70.7

0.79

✓

PITOME

DEIT-T

70.8

71.6

0.79

✓

ViT DEIT-T

72.3

72.3

1.2

×

prune

A-ViT DEIT-S

n/a

78.6

2.9

×

Dynamic-ViT DEIT-S

n/a

79.3

2.9

×

SP-ViT DEIT-S

n/a

79.3

2.6

×

merge

E-ViT DEIT-S

-

79.5

2.9

×

ToMe DEIT-S

77.7

79.4

2.9

✓

ToFu DEIT-S

77.8

79.6

2.9

✓

DCT DEIT-S

74.8

78.6

2.9

✓

DiffRate

DEIT-S

76.8

79.5

2.9

✓

PITOME

DEIT-S

79.1

79.8

2.9

✓

ViT DEIT-S

79.8

79.8

4.6

×

merge

ToMe MAE-L

82.9

85.0

31.0

✓

ToFu MAE-L

83.8

85.1

31.0

✓

DCT MAE-L

82.8

84.4

31.0

✓

DiffRate

MAE-L

83.2

85.3

31.0

✓

PITOME

MAE-L

84.6

85.3

31.0

✓

ViT MAE-L

85.7

85.7

61.6

×

merge

ToMe MAE-H

85.6

86.4

92.8

✓

ToFu MAE-H

85.8

86.4

92.8

✓

DCT MAE-H

84.3

86.0

92.8

✓

DiffRate

MAE-H

85.9

86.6

92.8

✓

PITOME

MAE-H

86.4

86.7

92.8

✓

ViT MAE-H

86.9

86.9

167.4

×

4.5 PITOME Ablation Studies

Contributions of energy scores and related factors. To assess the performance of the components used in PITOME, we conduct the following settings: (i) PITOME without protecting important tokens by our energy in Step 2, i.e., using odd and even indices in sorted energy score array as two sets in BSM; (ii) PITOME where the merging process in Step 3 conducted on two randomly sets A , B as baselines [15, 16] instead of leveraging ordered in sorted energy vectors E ( . ) ; (iii) PITOME without using our proposed energy score as in Eq(4) but utilizing other indicators like attention scores from the [CLS] (PITOME w cls attn) token [19] or mean of attention scores; (iv) PITOME using a fixed of k removed token at each layer as ToMe [15] rather than a reducing ratio of r as our configuration.

Werun experiments on image-text retrieval and text classification tasks, reporting the results in Table 7 for (i) and (ii), and in Figure 6 for (iii) and (iv). The results demonstrate that all factors contribute to

Dataset

Type

Model

Acc

Eval

Flops

↑

Train

Speed

↑

SST-2

compressed

models

ALBERT

91.3

x1.1

DistiledBERT

91.1

x2.0

x1.7

BERT

91.4

pruning

+mask

PowER-BERT

91.1

Fisher

91.3

x1.6

LTP

91.3

x2.9

merging

PITOME

91.0

x1.9

x1.4

ToMe

91.2

x1.9

x1.4

ToFu

89.8

x1.9

x1.4

DCT

90.7

x1.9

x1.4

DiffRate

89.7

x1.9

x1.4

PITOME

91.7

x1.9

x1.4

IMDb

compressed

models

ALBERT

89.2

x1.2

DistiledBERT

93.0

x2.0

x1.9

BERT

94.0

pruning

+mask

PowER-BERT

92.5

x2.7

TR-BERT

93.6

x2.3

merging

PITOME

93.2

x1.9

x1.8

ToMe

93.3

x1.9

x1.8

ToFu

92.6

x1.9

x1.8

DCT

92.4

x1.9

x1.8

DiffRate

92.4

x1.9

x1.8

PITOME

93.6

x1.9

x1.8

the performance of PITOME, with energy-based operations playing a particularly significant role. Additionally, reducing tokens with a ratio r effectively eliminates redundant tokens in early layers while preserving informative ones in later layers.

Margin m and α hyper-parameters. To validate the roles of these parameters in our energy score function in Eq.(4), we conduct ablation studies on image-text retrieval task with (v) adaptive margin m compared with a fixed value m ∈ { 0 . 9 , 0 . 45 , 0 . 0 , -1 . 0 } when varying the ratio r and (vi) given a fixed value of r , changing the smooth constant value α in α (exp( x -m ) -1) with x < m . Results for these settings are summarized in Figure 7 and Table 8, respectively. We observe that while models with fixed tend to have the accuracy drop sharply when it is lower than some threshold, the adaptive margins achieve the best results across cases. We hypothesize that as the token space becomes sparser in deeper layers, PiToMe's fixed m approach likely assigns the same energy score to all tokens, making it difficult to isolate and protect tokens during merging. Table 8 also shows that α = 1 . 0 is the best choice across margin values.

Further details, including additional ablation study results, visualizations (output merging, open-chat with LLaVa), and extra PITOME experiments, are provided in the Appendix.

5 Conclusion

This paper introduces PITOME, a novel algorithm that employs energy concepts to protect informative tokens during the token merging process. Our algorithm matches the efficiency of heuristic merging methods while maintaining a theoretical connection to the spectral properties of the input token space. In experiments on image classification, image-text retrieval, and VQA with LLaVA-1.5 7B/13B, PITOME consistently outperforms recent token merging and pruning methods, given the equivalent runtime and memory usage.

Limitations and Future Works Although our focus has been on tasks using ViT encoders for a variety of applications, we believe it is important to extend PITOME to generative tasks such as image generation (e.g., stable diffusion) or segmentation. This extension, however, necessitates the development of an unmerge mechanism in the decoder, which remains an open question. Additionally, our energy score relies on a fully connected graph of input tokens, which can increase complexity as the input size grows. Constructing sparse graphs, therefore, might be beneficial for scaling in more challenging settings. Finally, designing a differentiable learning mechanism to optimize the reducing rate r for token merging could enhance robustness and versatility across different downstream tasks.

580

570

Recall sum

560

550

540

m=0.9-1//*0.9

X

m=0.9

..·..

m=0.45

530

m=0.0

37.5

40.0

42.5

45.0

47.5

50.0

52.5

55.0

gflops

Acknowledgment

The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Duy M. H. Nguyen and Daniel Sonntag are also supported by the XAINES project (BMBF, 01IW20005), No-IDLE project (BMBF, 01IW23002), and the Endowed Chair of Applied Artificial Intelligence, Oldenburg University. Hoai-Chau Tran acknowledges the support from the AISIA Extensive Research Assistant Program 2023 (Batch 1) during this work and DFKI for supporting computing resources. TrungTin Nguyen acknowledges support from the Australian Research Council grant DP230100905. Ngan Le acknowledges funding support from the U.S. National Science Foundation (NSF) under Award No. OIA-1946391 and NSF EFRI BRAID 2223793. Binh T. Nguyen wants to thank the University of Science, Vietnam National University in Ho Chi Minh City for their support. Mathias Niepert acknowledges funding by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC and support by the Stuttgart Center for Simulation Science (SimTech).

References

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Duy MH Nguyen, Hoang Nguyen, Nghiem Diep, Tan Ngoc Pham, Tri Cao, Binh Nguyen, Paul Swoboda, Nhat Ho, Shadi Albarqouni, Pengtao Xie, et al. Lvm-med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching. Advances in Neural Information Processing Systems , 36, 2024.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3202-3211, 2022.

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning , pages 7480-7512. PMLR, 2023.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pages 5156-5165. PMLR, 2020.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In European Conference on Computer Vision , pages 35-49. Springer, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Duy MH Nguyen, Hoang Nguyen, Truong TN Mai, Tri Cao, Binh T Nguyen, Nhat Ho, Paul Swoboda, Shadi Albarqouni, Pengtao Xie, and Daniel Sonntag. Joint self-supervised imagevolume representation learning with intra-inter contrastive clustering. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 14426-14435, 2023.

Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and SerNam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12309-12318, 2022.

Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems , 32, 2019.

Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latencyaware soft token pruning. In European conference on computer vision , pages 620-640. Springer, 2022.

Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2092-2101, 2023.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations , 2023.

Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1383-1392, 2024.

Qingqing Cao, Bhargavi Paranjape, and Hannaneh Hajishirzi. PuMer: Pruning and merging tokens for efficient vision language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 12890-12903, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.721.

Maxim Bonnaerens and Joni Dambre. Learned thresholds token merging and pruning for vision transformers. Transactions on Machine Learning Research , 2023. ISSN 2835-8856.

Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate : Differentiable compression rate for efficient vision transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 17118-17128, 2023. doi: 10.1109/ICCV51070.2023.01574.

Chaoya Jiang, Haiyang Xu, Chenliang Li, Ming Yan, Wei Ye, Shikun Zhang, Bin Bi, and Songfang Huang. Trips: Efficient vision-and-language pre-training with text-relevant image patch selection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 4084-4096, 2022.

Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do selfsupervised vision transformers learn? International Conference on Learning Representations , 2023.

R Balakrishnan. The energy of a graph. Linear Algebra and its Applications , 387:287-295, 2004.

Ivan Gutman and Bo Zhou. Laplacian energy of a graph. Linear Algebra and its applications , 414(1):29-37, 2006.

Andreas Loukas and Pierre Vandergheynst. Spectrally approximating large graphs with smaller graphs. In International conference on machine learning , pages 3237-3246. PMLR, 2018.

Yu Jin, Andreas Loukas, and Joseph JaJa. Graph coarsening with preserved spectral properties. In International Conference on Artificial Intelligence and Statistics , pages 4452-4462. PMLR, 2020.

Andreas Loukas. Graph reduction with spectral and cut guarantees. Journal of Machine Learning Research , 20(116):1-42, 2019.

Hongjie Wang, Bhishma Dedhia, and Niraj K Jha. Zero-tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers. arXiv preprint arXiv:2305.17328 , 2023.

Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 3531-3539, 2021.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems , 35:16344-16359, 2022.

Giannis Daras, Nikita Kitaev, Augustus Odena, and Alexandros G Dimakis. Smyrf-efficient attention using asymmetric clustering. Advances in Neural Information Processing Systems , 33: 6476-6489, 2020.

Valerii Likhosherstov, Krzysztof M Choromanski, Jared Quincy Davis, Xingyou Song, and Adrian Weller. Sub-linear memory: How to make performers slim. Advances in Neural Information Processing Systems , 34:6707-6719, 2021.

Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai. Combiner: Full attention transformer with sparse computation cost. Advances in Neural Information Processing Systems , 34:22470-22482, 2021.

Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Jürgen Gall. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision , pages 396-414. Springer, 2022.

Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14408-14419, 2023.

Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems , 34:13937-13949, 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive wordvector elimination. In International Conference on Machine Learning , pages 3690-3699. PMLR, 2020.

Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. Revisiting token dropping strategy in efficient bert pretraining. arXiv preprint arXiv:2305.15273 , 2023.

Jungmin Yun, Mihyeon Kim, and Youngbin Kim. Focus on the core: Efficient attention via pruned token compression for document classification. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10809-10818, 2022.

Yuxuan Zhou, Wangmeng Xiang, Chao Li, Biao Wang, Xihan Wei, Lei Zhang, Margret Keuper, and Xiansheng Hua. Sp-vit: Learning 2d spatial priors for vision transformers. In The 33rd British Machine Vision Conference , 2022.

Zhuoran Song, Yihong Xu, Zhezhi He, Li Jiang, Naifeng Jing, and Xiaoyao Liang. Cpvit: Cascade vision transformer pruning via progressive sparsity prediction. arXiv preprint arXiv:2203.04570 , 2022.

Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 45994603, 2023.

Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers. International Conference on Machine Learning , 2024.

Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers for image classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 12-21, 2023.

Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International conference on machine learning , pages 874-883. PMLR, 2020.

Junran Wu, Xueyuan Chen, Ke Xu, and Shangzhe Li. Structural entropy guided graph hierarchical pooling. In International conference on machine learning , pages 24017-24030. PMLR, 2022.

Manoj Kumar, Anurag Sharma, Shashwat Saxena, and Sandeep Kumar. Featured graph coarsening with similarity guarantees. In International Conference on Machine Learning , pages 17953-17975. PMLR, 2023.

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In ICLR 2016 , 2016.

Yu Jin, Andreas Loukas, and Joseph JaJa. Graph Coarsening with Preserved Spectral Properties. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 4452-4462. PMLR, August 2020.

Andreas Loukas. Graph Reduction with Spectral and Cut Guarantees. Journal of Machine Learning Research , 20(116):1-42, 2019.

Christopher Brissette, Andy Huang, and George Slota. Spectrum Consistent Coarsening Approximates Edge Weights. SIAM Journal on Matrix Analysis and Applications , 44(3): 1032-1046, 2023. doi: 10.1137/21M1458119.

Andreas Loukas and Pierre Vandergheynst. Spectrally Approximating Large Graphs with Smaller Graphs. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3237-3246. PMLR, July 2018.

Hannu Toivonen, Fang Zhou, Aleksi Hartikainen, and Atte Hinkka. Compression of weighted graphs. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '11, pages 965-973, New York, NY, USA, 2011. Association for Computing Machinery. ISBN 978-1-4503-0813-7. doi: 10.1145/2020408.2020566.

Steve Butler. Interlacing for weighted graphs using the normalized Laplacian. The Electronic Journal of Linear Algebra , 16:90-98, 2007.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 18-24 Jul 2021.

Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS , 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 12888-12900. PMLR, 17-23 Jul 2022.

Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , December 2015.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 , pages 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1.

Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey on recent research and development. Thirty-First International Joint Conference on Artificial Intelligence (IJCAI)) , 2022.

Ziwei He, Meng Yang, Minwei Feng, Jingcheng Yin, Xinbing Wang, Jingwen Leng, and Zhouhan Lin. Fourier transformer: Fast long range modeling by removing sequence redundancy with fft operator. In Findings of the Association for Computational Linguistics: ACL 2023 . Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-acl.570.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning , pages 5583-5594. PMLR, 2021.

Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, and Jingjing Liu. Lightningdot: Pre-training visual-semantic embeddings for real-time image-text retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 982-997, 2021.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104-120. Springer, 2020.

Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18166-18176, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems , 34:9694-9705, 2021.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6693-6702, Los Alamitos, CA, USA, jun 2019. IEEE Computer Society. doi: 10.1109/CVPR.2019.00686.

Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3608-3617, 2018. doi: 10.1109/CVPR.2018.00380.

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8309-8318, 2019. doi: 10.1109/ CVPR.2019.00851.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 , 2023.

Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 19730-19742. PMLR, 23-29 Jul 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems , 36, 2024.

IDEFICS. Introducing idefics: An open reproduction of state-of-the-art visual language model. https://huggingface.co/blog/idefics , 2023. Accessed: 2023-05-21.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377 , 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , volume 139, pages 10347-10357, July 2021.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y.

Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows, 2021.

Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR , 2022.

Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of naacL-HLT , 2019.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pages 142-150, 2011.

Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive wordvector elimination. In International Conference on Machine Learning , pages 3690-3699. PMLR, 2020.

Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems , 35:24101-24116, 2022.

Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 784-794, 2022.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019.

Henry Wolkowicz and George P. H. Styan. Bounds for eigenvalues using traces. Linear Algebra and its Applications , 29:471-506, 1980. ISSN 0024-3795. doi: https://doi.org/10.1016/ 0024-3795(80)90258-X.

Hermann Weyl. Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung). Mathematische Annalen , 71(4):441-479, December 1912. ISSN 1432-1807. doi: 10.1007/BF01456804.

Supplement to 'Accelerating Transformers with Spectrum-Preserving Token Merging'

Contents

Input graph:

G

Coarsened graph:

G

c

v

0

v

1

v

2

v

3

v

4

v

5

v

6

v

8

v

9

v

7

v

11

v

12

v

13

v

14

v

89

v

67

v

45

v

01

v

23

v

13

v

12

v

11

v

14

Candidate Node

A Datasets Descriptions

We present in Table 9 datasets used in our experiment. It is important to note that PITOME was run off-the-shelf on large datasets in VQA tasks such as VQA-v2 with 447 . 8 k sample or GQQ with 12 . 5 questions, validating generalization and robustness of our merging mechanism.

B PITOME Algorithm

B.1 Pseudo-Code Implementation

The pseudo-code for our method is provided in Algorithm 1. Here E in line 3 is a vector that represents energy scores of all nodes calculated by the energy function 4. The final output is a MERGE( . ) function, which serves as a lambda function that can be applied to any matrix ˆ X l at the l -th layer. The vector m is a vector that contains information about token sizes (i.e, the number of tokens being merged into each token).

Algorithm 1 PITOME Algorithm 1: function PITOME(remain token ratio: r, input graph: G ( V , E ) ) // Function to prepare for merging 2: k ← N - N · r // Compute number of nodes to merge 3: s ← argsort ( E , descending=True ) // Compute energy scores 4: merge , protect ← s [: 2 · k ] , s [2 · k :] // Identify mergeable and protected nodes 5: n a , n b ← merge [:: 2] , merge [1 :: 2] // Split mergeable nodes 6: E merge ←E [ n a ][ n b ] // Get edge weights of mergeable nodes 7: n dst ← argmax ( E merge ) // Find closest neighbors 8: function MERGE( X ) // Function to perform merging 9: X protected ← X [ protect , :] // Extract protected tokens 10: X A , X B ← X [ n a , :] , X [ n b , :] // Extract tokens in set A and B 11: X A , X B ← X A × m [ n a ] , X B × m [ n b ] // Weighted average 12: X B ← X B . scatter_reduce ( n dst , X A , mode = "sum" ) // Merge tokens 13: X B ← X B / m B . scatter_reduce ( n dst , m A , mode = "sum" ) // Weighted average 14: return cat ( X protected , X B ) // Concatenate and return merged tokens 15: return MERGE // Return merging lambda function

B.2 PITOME Complexity Analysis

In algorithm 1, in line 3 , the weighted graph is constructed through matrix multiplication, leading to a complexity of O ( N 2 h ) , where h is the dimension of input vectors. Next, the computed energy scores are sorted, which have the complexity of O ( N log( N )) . Lastly, in line 7 , the max operator for selecting the merge destination and the lambda MERGE function, which performs tensor operations based on computed indices, lead to linear complexity. Combining these aspects, the overall time complexity of the PITOME function can be approximated as O ( N 2 h ) , considering the dominant factors contributing to computational cost. However, actual performance may vary depending on the specific PyTorch version and hardware utilization, with optimizations potentially altering these estimates.

(a) MSCOCO

(b) Flickr30k

use k

500

use k

False

False

500

True

True

model

400

model

400

--

CLIP_B

CLIP_L

一

CLIP_B

CLIP_L

RSum

BLIP

RSum

300

BLIP

BLIP2

BLIP2

300

200

200

100

100

200

400

200

400

FLOPS

FLOPS

In the BSM algorithms used by ToMe, after the bipartition of tokens into two sets A and B using odd and even indices, each set has N/ 2 tokens, it also requires calculating a similarity matrix between tokens in these two sets. This operation also has the complexity of O ( N 2 h ) . The similarity matrix is then sorted to get the top k tokens in set A with the highest similarity score used for merging, this operation also has the time complexity of O ( N log( N )) , and the merging procedure is the same as our PITOME algorithm. So basically when compared to BSM, our algorithms have the same time complexity of O ( N 2 h ) , although in practice the speed of ToMe is a little bit faster than PITOME (a few milliseconds), but our algorithms give a much better trade-off between speed and accuracy. For more details about the BSM algorithms please refer to the ToMe paper[15].

B.3 Model complexity analysis

In a standard Transformer layer, we have the time and space complexity to be O ( N 2 h + Nh 2 ) and O ( N 2 + Nh ) . In each layer ith of the model, we compress the number of tokens down to rN using PITOME algorithms with the complexity of O ( ( r i -1 N ) 2 h ) , so the layer i will enjoy the time and space complexity of O ( ( r i -1 N ) 2 h + r i Nh 2 +( r i -1 N ) 2 h ) and O ( ( r i -1 N ) 2 + r i Nh ) resp. Vision language models like LLaVA directly use output tokens from ViT encoders. Let l be the number of layers in the ViT encoder since we utilize PITOME in each layer in the ViT encoder part; the LLM model will have the time and space complexity of O ( ( r l N ViT + N LLM ) 2 h +( r l N ViT + N LLM ) h 2 ) and O ( ( r l N ViT + N LLM ) 2 +( r l N ViT + N LLM ) h ) which boosts the inference speed and saves a high amount of memory usage. The speedup could get even more impressive when used with higher batch size and image size. This also applies to models that use cross-attention modules for image text matching like BLIP and ALBEF, in which most of the computation expense comes from tokens encoded by the ViT model, the time and space complexity for the cross attention layers are O ( r l N vision N text h + N text h 2 ) and O ( r l N vision N text + N text h ) resp.

C Performance of ToMe with Different Token Merging Schedules

In the original ToMe paper, the authors proposed a merging schedule that involves reducing tokens in each layer by a fixed k tokens per layer. However, as illustrated in Figure 6, we showed that this merging schedule is suboptimal for off-the-shell performance, and this section provides empirical results to confirm this claim.

In this experiment, we exclusively compare two versions of the BSM algorithms utilized in the ToMe paper: one that preserves a percentage r of tokens in each layer and another that reduces a fixed k tokens in each layer. However, for a more comprehensive comparison, we extensively apply these algorithms across 6 ViT backbones (DeiT-T, DeiT-S, DeiT-B, MAE-B, MAE-L, MAE-H) for image classification tasks using the Imagenet-1k dataset, as well as across 4 backbones for image-text retrieval tasks (CLIP-B, CLIP-L, BLIP, BLIP2) on FLickr30k and MSCOCO dataset.

(a) DEIT backbones models

(b) MAE backbone models

82

use k

use k

80

False

True

80

False

True

78

--

model

DeiT-B-16

70

ViT-H-14

model

76

DeiT-S-16

DeiT-T-16

ccuracy

60

·

ViT-B-16

VIT-L-16

74

AC

50

72

40

70

30

68

20

5

10

15

40

80

120

FLOPS

FLOPS

From Figures 10b, 10a, 9b, and 9a, it is evident that, given the same FLOPS, the BSM version that uses the remaining percent r shows a clear advantage, outperforming the original schedule by a large margin. This gap becomes even more pronounced when benchmarked on large models like MAE-L, MAE-H, and BLIP-2. During the experiment, we also observed that models compressed using the ratio r tend to run a little faster since a large number of tokens are removed in earlier layers.

D Additional Experiments on Text Classification Task.

This section comprehensively assesses the performance of PITOME against other baseline algorithms for sentiment text classification tasks. We conducted this experiment using the IMDb and SST-2, applying the compression algorithms to the first three layers of the model. To ensure consistency in our experimental results, we used two different backbones: BERT with 12 layers and DistilBERT with 6 layers. Since we only compress the first three layers, in addition to the baselines previously used throughout this paper, we introduced two additional baselines: ToFu-p and ToFu-m. ToFu-p uses BSM to prune tokens instead of merging them, while ToFu-m represents the original ToFu algorithm without pruning.

As demonstrated in Table 10 and Figure 11, our findings align with empirical results from previous tasks, indicating that PITOME consistently achieves superior performance compared to other baselines. Particularly noteworthy is the performance on the IMDb dataset with a large context length, where even after reducing FLOPs by 80%, models compressed by PITOME still maintain off-the-shelf accuracy above 85%, while other baseline algorithms see the off-the-shelf accuracy drop below 70%. Furthermore, Table 10 highlights that our algorithm can also facilitate better model

learning compared to other methods, achieving high accuracy that closely approaches that of the original model.

E Proof of Theorem 1

E.1 Sketch of Proof

The proof sketch for Theorem 1 begins by defining coarsened and lifted versions of the original graph G (see Definitions 1 and 2) using the PITOME and ToMe algorithms. The goal is to demonstrate that the spectral distance between G and its PITOME-coarsened counterpart converges to zero, whereas the distance for ToMe remains bounded away from zero. The sketch proceeds by introducing Propositions 1, 2, and 3, which build toward the main result:

Proposition 1 establishes upper bounds on the edge weight differences between merged nodes under the PITOME and ToMe methods.

Proposition 2 employs standard mild assumptions to relate the cosine similarity among nodes within clusters, demonstrating that the upper bound error ϵ ( s ) PITOME between the edge weights of merged nodes in PITOME-coarsened graphs G ( n ) PITOME converges to 0 , whereas this property does not hold for the ToMe-coarsened graphs G ( n ) ToMe .

Proposition 3 completes the sketch by bounding the spectral distances of the coarsened graphs in terms of the edge weight differences from the earlier Propositions 1 and 2.

Each proposition is proven in detail within the appendix sections following this sketch. Note that Theorem 1 immediately follows the following Propositions 1, 2, and 3, which are proved resp. in Sections E.2, E.3, and E.4.

Proposition 1. Suppose the graphs G ( s ) 0 , G ( s ) PITOME , and G ( s ) ToMe are coarsened from the original graph G by iteratively merging pairs of nodes v a s and v b s w.r.t. the true partition P ( s ) 0 = {V ( s ) 0 i } i ∈ [ s ] , the PITOME -partition P ( s ) PITOME = {V ( s ) PITOME i } i ∈ [ s ] , defined by PITOME Algorithm 1, and the ToMepartition [15, 16], P ( s ) ToMe = {V ( s ) ToMe i } i ∈ [ s ] , for s = N,... , n +1 . We assume the following standard mild assumption:

(A2). There exists a margin m s.t.

̸

Then, the edge weights of merged nodes from PITOME Algorithm 1 and the ToMe-partition [15, 16] satisfy

for some nonnegative upper bounds ϵ ( s ) , s = N,N -1 , . . . , n +1 defined as follows:

̸

Proposition 2. We assume some standard mild assumptions:

(A1). E [cos( v a s , v b s )] → 1 , ∀ v a s ∈ V ( s ) 0 i , ∀ v b s ∈ V ( s ) 0 i , i ∈ [ s ] .

̸

(A2). There exists a margin m s.t., cos( v a s , v b s ) ≥ m > cos( v a s , v c s ) , ∀ v a s ∈ V ( s ) 0 i , ∀ v b s ∈ V ( s ) 0 i , ∀ v c s ∈ V ( s ) 0 j , ∀ i = j ∈ [ s ] .

(A3). There is an order of cardinality in the true partition, without loss of generality, we assume N ( s ) 1 ≥ N ( s ) 2 ≥ . . . ≥ N ( s ) s , where N ( s ) i = |V ( s ) 0 i | , ∀ i ∈ [ s ] .

Here E ( · ) stands for the expectation of the random variables that define the random events M ( s ) ab := { v a s ∈ V ( s ) 0 i , v b s ∈ V ( s ) 0 i , i ∈ [ s ] } , indicating that the two merged nodes belong to the same true partition V ( s ) 0 i . Given the ϵ ( s ) defined in equation (8), we obtain

The upper bound error ϵ ( s ) PITOME between the edge weights of merged nodes from PITOME -coarse G ( n ) PITOME graphs converges to 0 , i.e.,

The upper bound error ϵ ( s ) ToMe between the edge weights of merged nodes from ToMe-coarse G ( n ) ToMe graphs converges to a non-negative constant C ( s ) , with a high probability that C ( s ) > 0 , i.e.,

ϵ ( s ) ToMe → C ( s ) and P ( C ( s ) > 0) > 0 as E (cos( v a s , v b s )) → 1 , for any s = N,... , n +1 .

Proposition 3. Suppose the graphs G ( s ) 0 , G ( s ) PITOME , and G ( s ) ToMe are coarsened from the original graph G by iteratively merging pairs of nodes v a s and v b s w.r.t. the true partition P ( s ) 0 = {V ( s ) 0 i } i ∈ [ s ] , the PITOME -partition P ( s ) PITOME = {V ( s ) PITOME i } i ∈ [ s ] , defined by PITOME Algorithm 1, and the ToMepartition [15, 16], P ( s ) ToMe = {V ( s ) ToMe i } i ∈ [ s ] , for s = N,... , n +1 . If the edge weights of merged nodes satisfy

for some nonnegative upper bounds ϵ ( s ) , s = N,N -1 , . . . , n +1 . The spectral distances between the original G ≡ G ( N ) 0 and the PITOME -coarse and ToMe-coarse G ( n ) PITOME graphs are bounded as follows:

E.2 Proof of Proposition 1

We want to prove that the edge weights of merged nodes from PITOME Algorithm 1 and the ToMe-partition [15, 16] satisfy

for some nonnegative upper bounds ϵ ( s ) , s = N,N -1 , . . . , n +1 .

Let us start with the simplest case, where we suppose the graph G c is coarsened from G by merging one pair of nodes v a and v b using PITOME Algorithm 1 and the ToMe-partition. We can then demonstrate that the edge weights of the merged nodes satisfy certain conditions (for more details, refer to Appendix E.5.1):

Recall that the graphs G ( s ) 0 , G ( s ) PITOME , and G ( s ) ToMe are coarsened from the original graph G by iteratively merging pairs of nodes v a s and v b s w.r.t. the true partition P ( s ) 0 = {V ( s ) 0 i } i ∈ [ s ] , the PITOME-partition P ( s ) PITOME = {V ( s ) PITOME i } i ∈ [ s ] , defined by PITOME Algorithm 1, and the ToMe-partition [15, 16], P ( s ) ToMe = {V ( s ) ToMe i } i ∈ [ s ] , for s = N,... , n +1 . By iteratively applying the merged 2-nodes inequalities (9), we obtain the desired upper bound as follows:

where

E.3 Proof of Proposition 2

Recall that in equation (8) of Proposition 1, we defined ϵ ( s ) , for s = N,N -1 , . . . , n +1 , as follows:

̸

Recall that we defined E ( · ) as the expectation of the random variables that define the random events M ( s ) ab := { v a s ∈ V ( s ) 0 i , v b s ∈ V ( s ) 0 i , i ∈ [ s ] } , indicating that the two merged nodes belong to the same true partition V ( s ) 0 i . Using the definition and the linearity of expectation, we have

Under Assumptions (A2) and (A3) and the energy-based merging mechanism of the PITOME algorithm, we can verify that P ( M ( s ) ab ) = 1 via Lemma 2, which is proved in Appendix E.5.2. Refer to Lemma 3 for the ToMe approach (proof in Appendix E.5.2), noting that there exists a scenario where the random event M ( s ) ab does not occur, leading to P ( M ( s ) ab ) < 1 . This leads to the desired results.

Lemma 2. For the PITOME approach, it holds that P ( M ( s ) ab ) = 1 , i.e., there exists a true partition V ( s ) 0 i , i ∈ [ s ] , such that v a s ∈ V ( s ) 0 i , v b s ∈ V ( s ) 0 i .

Lemma 3. For the ToMe approach, there is a case where the random event M ( s ) ab does not occur and therefore P ( M ( s ) ab ) < 1 .

E.4 Proof of Proposition 3

Let us start with the simplest case, where we suppose the graph G c is coarsened from G by merging one pair of nodes v a and v b . We can then prove the following property: If the edge weights of merged nodes satisfy

then the spectral distance between the original and lifted graphs is bounded by

Proof of the 2-node triangle inequality 11. Indeed, since the coarse graph G c is coarsened by merging a pair of nodes v a and v b , the edge weights of the lifted graph G l can be formulated as:

Here we use the fact that the adjacency matrix elements of G l are given by

The corresponding node degree of G l is

Using the result from Lemma 4, we can bound on the eigenvalue gap between λ and λ l via the perturbation matrix E = L -L l = D -D l + W l -W as follows:

̸

Here the last inequality follows from the upper bound of the spectral norm ∥ E ∥ 2 of [94], and we defined ∥ E ∥ ∞ = max i ∑ j | E [ i, j ] | , which is simply the maximum absolute row sum of the matrix, and ∥ E ∥ 1 = max j ∑ i | E [ i, j ] | , which is simply the maximum absolute column sum of the matrix.

Lemma 4 (Perturbations of eigenvalues: Weyl's inequality from [95]) . Let L ∈ R N × N and L l ∈ R N × N be symmetric matrices. Then for all i = 1 , . . . , N ,

where ∥ L l -L ∥ 2 is the induced 2-norm of L l -L .

The equations (12), (13), and (14) yield the following identities:

and

Now we want to prove that ∥ E ∥ ∞ ≤ ϵ and ∥ E ∥ 1 ≤ ϵ . Let us first focus on the first term ∥ E ∥ ∞ = max i ∑ j | E [ i, j ] | . Via the triangle inequality, we have

Then, again using the triangle inequality and the assumption of Proposition 3, we obtain the first upper bound on the first term of the equation (19) as follows:

For the second upper bound term from the equation (19), we consider two cases for each index i = 1 , . . . , N : i ∈ { a, b } ( Case 1 ) and i / ∈ { a, b } ( Case 2 ).

Case 1. Assume that i ∈ { a, b } . Since a and b have the same role, we can take i = a without loss of generality. Using the equation (17), it holds that

Therefore, it holds that

Case 2. Assume that i / ∈ { a, b } . Using the equation (17), we obtain

Combining (21) and (22), we obtain

This leads to

when using the inequalities (19) and (20). Similarly, we can show that ∥ E ∥ 1 ≤ 3 2 ϵ . Therefore, the equation (15) leads to the desired claim as follows:

Recall that the graphs G ( s ) 0 , G ( s ) PITOME , and G ( s ) ToMe are coarsened from the original graph G by iteratively merging pairs of nodes v a s and v b s w.r.t. the true partition P ( s ) 0 = {V ( s ) 0 i } i ∈ [ s ] , the PITOME-partition P ( s ) PITOME = {V ( s ) PITOME i } i ∈ [ s ] , defined by PITOME Algorithm 1, and the ToMe-partition [15, 16], P ( s ) ToMe = {V ( s ) ToMe i } i ∈ [ s ] , for s = N,... , n + 1 . By iteratively applying the triangle inequalities (11), the spectral distances between the original G ≡ G ( N ) 0 and the PITOME-coarse G ( n ) PITOME and ToMe-coarse G ( n ) ToMe graphs are bounded as follows:

E.5 Proofs of Technical Results

E.5.1 Proof of the merged 2-nodes inequality (9)

Recall that there exists the graph G 0 coarsened from the original graph G by merging a pair of nodes v a s and v b s w.r.t. the true partition P 0 = {V 0 i } i ∈ [ s ] . Then, we have V = V 01 ∪ V 02 ∪ . . . ∪ V 0 n , where n = N -1 . We also note that the energy score E a of node v a is calculated using the following

equation:

Inspired by the construction of the function f m in equation (23) and according to Assumption (A2) in the inequalities (6), we can replace the smooth term α (exp( x -m ) -1) by a constant β for simplicity. More precisely, if the nodes v a and v b are not considered true neighbours, i.e., their cosine similarity is less than the margin m , then we can simplify the expression as follows:

̸

To check the inequality (9), we examine the following term

in two cases:

Case 1. If two nodes v a and v b belong to the same true partition, say for example, V 0 i , i ∈ [ N ] , then since n = N -1 , we have V 0 i = { v a , v b } . Therefore, we can expand the previous 1-norm as follows:

̸

Case 2. If v a ∈ V 0 i and v b ∈ V 0 j such that i = j , i, j ∈ [ N ] . Since n = N -1 , we have either V 0 i = { v a , v 0 i } , V 0 j = { v b } ( Case 2 . 1 ) or V 0 i = { v a } , V 0 j = { v b , v 0 j } ( Case 2 . 2 ). Let us first consider the Case 2 . 1 , then it holds that

Now, let us first consider the Case 2 . 2 , then it holds that

Combining the previous equations (25), (26), and (27), we have

̸

E.5.2 Proof of Lemma 2

Recall the following Assumptions (A2) and (A3):

̸

(A2). There exists a margin m s.t., cos( v a s , v b s ) ≥ m > cos( v a s , v c s ) , ∀ v a s ∈ V ( s ) 0 i , ∀ v b s ∈ V ( s ) 0 i , ∀ v c s ∈ V ( s ) 0 j , ∀ i = j ∈ [ s ] .

(A3). There is an order of cardinality in the true partition, without loss of generality, we assume N ( s ) 1 ≥ N ( s ) 2 ≥ . . . ≥ N ( s ) s , where N ( s ) i = |V ( s ) 0 i | , ∀ i ∈ [ s ] .

In PITOME approach, using Assumption (A2), given any two nodes v a s ∈ V ( s ) 0 i , v b s ∈ V ( s ) 0 j , i < j ∈ [ s ] , the energy scores E a s and E b s of nodes v a s and v b s , respectively, are simplified as follows:

Given the choice of the universal margin m as follows: m = max { N ( s ) j N ( s ) i : i < j ∈ [ s ] } , it holds that E a s ≥ E b s . Indeed, this is guaranteed as long as we have

This is equivalent that

Using Assumption (A3) and this choice of universal margin m , it holds that

Recall that in PITOME approach, we use the ordered energy-based for the bipartite soft matching where we defined two set A and B with |A| = |B| = k as follows:

where the nodes v e i , i ∈ [2 k ] , are sorted in decreasing order based on their energy scores, i.e., E v e i > E v e i +1 , ∀ i ∈ [2 k -1] .

̸

We return to the proof of the Lemma 2 by contradiction. Assume that v a s ∈ V ( s ) 0 i , v b s ∈ V ( s ) 0 j , i = j ∈ [ s ] . Without loss of generality, we assume that v a s = v e 1 ∈ V ( s ) 01 and N ( s ) 1 = |V ( s ) 01 | > 1 . Note that PITOME algorithm selects v b s ∈ B such that

This is equivalent that for any v c s ∈ B , it holds that:

Assumption (A2) implies that v e 2 i / ∈ V ( s ) 01 , ∀ i ∈ [ k ] . Since we have N ( s ) 1 = |V ( s ) 01 | > 1 , there exists at least one node in A∩V ( s ) 01 , say for example v e 3 . Using (34), we have E v e 2 ≥ E v e 3 , which contradicts (33) where we have E v e 3 > E v e 2 since v e 3 ∈ V ( s ) 01 and v e 2 ∈ V ( s ) 0 j , j > 1 .

E.5.3 Proof of Lemma 3

On the contrary, in the Bipartite Soft Matching algorithm from the ToMe approach [15], the authors divide the tokens into two distinct sets A and B and merge the top k similar tokens using some partitioning style like sequential, alternating and random without considering the ordered energy-

based Bipartite Soft Matching like ours. This leads to a case where the random event M ( s ) ab does not occur and therefore P ( M ( s ) ab ) < 1 . Indeed, this case happens when all the nodes from the true partition V ( s ) 01 are divided into the same set A . Therefore, the Bipartite Soft Matching algorithm has to choose and select the node v b s ∈ V ( s ) 0 j , j > 1 , for merging. Actually, this case arises when all the nodes from the true partition V ( s ) 01 are distributed into the same set A . Therefore, the Bipartite Soft Matching algorithm has to select the node v b s ∈ V ( s ) 0 j , j > 1 , for merging.

F Token Merging Outputs Visualization

To better illustrate the effectiveness of PITOME, we have incorporated many visualizations showcasing examples randomly sampled from the MSCOCO dataset, as depicted in Figures 12a. In this analysis, we utilized the BLIP backbone and configured the reduction percentage to r = 0 . 85 . These visualizations portray the final representation of all remaining tokens in the last layers. Notably, tokens outlined with bolder cyan borders signify higher attention scores from the classification (CLS) token. Apparently, unlike ToMe and DiffRate, PITOME adeptly approximates the spectral spectrum of the original token space, thereby preserving the distribution of all tokens. Consequently, PITOME demonstrates a lower level of information distortion due to reduced mis-merged tokens, thus retaining attention maps for critical information in the final layer. As illustrated in Figures 12a,12b,12c,12d,12e the attention map in the final layer of the model compressed by PITOME exhibits the closest resemblance to that of the baseline model.

G OpenChat with Compressed LLaVA-1.5 Model

This section is dedicated to showcasing sample responses of LLaVA-1.5-7B when compressed using different algorithms. Here, we set the compression rate to r = 0 . 9 for each CLIP layer. All sample answers are given in tables 11, 13, 12. Texts that are highlighted by blue color indicate false information given by the model. From these sample answers, it is evident that, unlike other BSM-based approaches, our algorithm PITOME not only facilitates quicker model responses but also retains its capability to recognize objects, understand contextual information in the image, and provide correct answers closely aligned with those of the original LLaV A-1.5-7B model.

ViT-B 384

PiToMe (ours)

ToMe

DiffRate

ViT-B 384

PiToMe (ours)

ToMe

DiffRate

(b) A woman riding a horse jumping it over obstacles.

ViT-B 384

PiToMe (ours)

ToMe

DiffRate

(c) Three different motorcycle couples riding down a road.

ViT-B 384

PiToMe (ours)

ToMe

DiffRate

(d) A man holding a tennis racquet on a tennis court.

ViT-B 384

PiToMe (ours)

ToMe

DiffRate

ww.kranicek.net

5

odsoss

Model

USER: Describe what you see.

Answer

ww.krandek.net

063112

USER: Describe what is happening?

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provided experiments to support our claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We included it at the end of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are already stated.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: we provided experiment descriptions in the main paper and appendix. Furthermore, we will release our GitHub implementation if the paper is accepted.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will publish the code when the paper is accepted or through an anonymous link if some reviewers ask for it.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided details for experiments in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We conduct experiments on diverse datasets and follow the protocol used by previous works for fair comparisons.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This information is included in our results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper follows the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work helps to reduce the carbon footprint when training large models using ViT. There is no negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly cited papers and resources used in our experiment.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We don't have experiments involving crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Protect Informative Tokens Merging (PIToMe), a token merging method that safeguards crucial information-bearing tokens prior to the merging step. PIToMe focuses on the token-splitting step before token merging. Specifically, PIToMe defines and calculates an energy score for tokens to be merged, and marks the highest scored tokens as suitable candidates for merging. Experiments show that PIToMe consistently outperforms previous token merging methods such as ToMe, DiffRate, ToFu, etc. Theoretical analysis reveals that PIToMe preserves the intrinsic spectral properties to the original token space.\n\nSTRENGTHS:\n1. Being the first method to consider the graph relations of tokens during token merging, the design of PIToMe is novel. Thorough theoretical analysis demonstrates the design insights of PIToMe.\n2. Although some improvements are not significant enough, PIToMe successfully outperforms the previous token merging methods. Considering that many investigations towards token merging have been conducted by many works, I consider the improvements, albeit minor, as strengths.\n3. Visualizations demonstrate the design insights of PIToMe. From those demonstrations, I could see that PIToMe merges the correct tokens in the image.\n\nWEAKNESSES:\n1. ToMe is actually capable of conducting token merging at a very high compression rate (see Table 10 of ToMe paper, ViT-L and ViT-H can achieve about 2.5x speedup). Another recent work on arXiv [1] also demonstrate that this compression rate can even increase to >3x. However, in the experiments reported by PIToMe, the biggest speedup I could find is about 2x. I wonder whether PIToMe still works out for higher compression rates, since the metric vital for finding token clusters gradually defines more tokens as informative tokens (Eq. 4 in this PIToMe article). \n\n[1] PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation (arXiv 2403.09192)\n\nQUESTIONS:\n1. Also on the topic of compression rate. In ToMe, r can be randomly set to achieve any compression rates. Can I randomly set the r values for each layer in PIToMe?\n\nLIMITATIONS:\nSee weaknesses & questions. I may change the rating after carefully checking the replies and reviews from other reviewers.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an energy-based approach to the token merging process, utilizing energy scores to avoid erroneous merges by distinguishing between informative or isolated tokens and redundant tokens. This method enhances the efficiency of heuristic merging approaches and preserves the spectral properties between the original and merged graph, thus accelerating token-based architectures such as ViT. Specifically, the technique employs cosine similarity with neighboring tokens to evaluate the energy score. Tokens with a high energy score are considered redundant and can be merged, indicating they belong to large clusters (e.g., background). Conversely, tokens with low energy scores are treated as foreground tokens. Furthermore, the authors also present theoretical findings demonstrating that the proposed method more effectively approximates the spectral distance between the initial token spaces and the merged token set compared to existing approaches. The method has shown competitive results across four tasks: image-text retrieval, visual question answering with LLMs, image classification, and text classification, underscoring its effectiveness and significance in the field.\n\nSTRENGTHS:\n* This paper is well-written and clearly presents both the proposed solution and the experimental results.\n* In addition to thorough experiments demonstrating the effectiveness of the proposed method, the paper provides a theoretical derivation to explain why PiToMe outperforms another related approach (e.g., ToMe).\n\nWEAKNESSES:\n* Some formulas and inferences are ambiguous, leading to misunderstandings that undermine the correctness of the theoretical explanations.\n* The lack of results for higher merging ratios weakens the practicality of the proposed approach.\n* There is a lack of discussion on related works (e.g., CrossGET [1], TRIPS[2]).\n\n[1] CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers, ICML 2024\n\n[2] TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection, EMNLP 2022\n\nQUESTIONS:\n* There are some ambiguities that need to be clarified in the methodology section. The authors claim that $m$ is a fixed constant margin; \n   however, according to the context, $m$ seems to be the threshold of the margin, and $1-m$ is the margin. If I am wrong, the following \n   description is confusing:\n   In lines 162-163, \"$m = 0.9 − 0.9 × l_i/l$, ... indicating an increasing margin as tokens move to deeper layers.\" This sentence seems to be \n   the complete opposite of the conclusion. Also, does $\\alpha = 1.0$ mean that when token $j$ is outside the margins, $f_m(x)$ is a \n   negative constant? Furthermore, there are no related ablation studies or discussions of the impact of the $\\alpha $.\n* In most of the experiments, the ratio $r$ is 0.95, 0.975, 0.9, and 0.925. Only the image classification results include a ratio $r = 0.7$. \n   Existing approaches (e.g., ToMe) can achieve ratios of $r = 0.5 \\sim 0.6$. I am curious about the consistency of the proposed method at \n   higher merging ratios.\n* Some existing work, such as TRIPS and CrossGET, has also investigated token merging. The lack of thorough discussion or comparison \n   with previous work weakens PiToMe's claimed novelty. I suggest the authors more clearly present their merits and differences in the paper.\n\nLIMITATIONS:\n- The authors discuss some limitations in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new strategy for reducing the number of tokens used by a vision transformer by merging similar tokens without. Such methods are very sensitive to the type of clustering/partitioning used on the tokens. The proposed approach, `PITOME` explicitly identify highly informative tokens, which should be kept out of the merging process.\n\nMore specifically, the tokens are place in a graph where each node corresponds to a key embedding and the edges are the cosine distance between said keys. Based on this graph, an \"energy\" is computed for each token, capturing how mergeable they are. Only the top $2k$ are considered mergeable, while the rest is preserved. The actual following merging step follows a similar procedure as previous token merging methods:\n\nIn summary, an \"energy\" score is computed on each token, using their key embeddings as base features. Using these energy scores, `EPITOME` draws itself apart from other merging techniques such as `ToME` in two ways;\n - explicitly exclude some tokens from the merging process\n - split tokens across the two sets $A$ and $B$ for bipartite matching as to maximise the change of max\n\nSTRENGTHS:\n- The proposed method is thoroughly evaluated on image, text and image+text tasks\n- Generally, `PITOME` achieves higher accuracy while preserving the expected number of FLOPs and memory usage\n- the issues addressed by `EPITOME` are well explained and motivated.\n\nWEAKNESSES:\n- Generally, the **experiments section** is really hard to parse:\n  * the figure placement is a bit all over the place\n  * some results would benefit to be moved to the appendix; for instance, in my opinion, the trade-off curves are much more informative than the tables\n  * some figures are hard to read (Figure 3) (e.g. small captions, overlapping curves due to the y/x axes scales)\n  * ablation experiments are much too condensed \n\n- In terms of ablation, there is little discussion about the term $m$ when computing the energy score: It seems to be an important hyperparameter as it impacts the \"merge ability\" of the tokens as the layer depth increase. Since `ToMe` and other related work are presented as having a lack of robustness to the token splitting strategy, it would be interesting to also show how robust `PITOME` is to this hyper parameter.\n\n- Some claims didn't seem substantiated enough to me\n  - e.g. line 120 *\"`PITOME` remains robust to token partitioning strategies\"*: However `PITOME` uses a specific token partitioning algorithm. And the ablation on it (random split in Table 1) show that it actually suffers from a different portioning\n  - line 276. *\"We contend that this improvement stems from the merging of less significant tokens in PITOME potentially enhancing the robustness of the language mode\"* -> this seems like a very vague hypothesis which in my opinion does really need a justification to begin with, since some other merging methods also outperforms the baseline. It may as well be a consequence of the noisiness of question answering datasets.\n\nQUESTIONS:\n- I didn't fully understand the role of **Table 2**: isn't `PITOME` agnostic to the choice of the architecture ? it would send a stronger message to combine `PITOME` with various backbones, rather than compare it to different architectures.\n\nLIMITATIONS:\nthere is a limitation section discussing some current drawbacks of token merging methods in general", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method called PITOME, which enhances the efficiency of Vision Transformers (ViTs) by merging tokens in a way that preserves crucial information. Unlike previous methods, PITOME uses an energy score to prioritize and protect informative tokens while merging redundant ones. This approach reduces the computational and memory requirements by 40-60% with minimal impact on accuracy. PITOME achieves superior performance in tasks such as image classification, image-text retrieval, and visual question answering, demonstrating its effectiveness and robustness compared to existing token merging techniques. Furthermore, it theoretically preserves the spectral properties of the original tokens, contributing to its efficiency and performance stability.\n\nSTRENGTHS:\n1. The introduction of the energy score metric for token merging is a novel idea that addresses the limitations of previous methods. This innovative approach effectively distinguishes between informative and redundant tokens, ensuring the preservation of critical information during the merging process.\n2. The theoretical foundation for preserving the spectral properties of the original token space is a unique contribution. This aspect of the work sets it apart from existing methods, which often do not consider the impact on the spectral properties of the token space.\n3. The paper provides extensive empirical validation across multiple tasks, including image classification, image-text retrieval, and visual question answering.\n4. The theoretical analysis that supports the empirical findings enhances the overall quality of the paper. By demonstrating that PITOME preserves spectral properties, the authors provide a solid theoretical justification for the observed performance improvements.\n\nWEAKNESSES:\n1. The proposed method, PITOME, does not show significant improvement over the baseline, particularly ToMe, in many tasks. The empirical results indicate that in some cases, PITOME's performance is only marginally better or even comparable to existing methods, which raises questions about its practical benefits.\n2. As shown in Figure 11, PITOME fails to outperform ToMe in visualizations, and in some instances, the token merging appears unreasonable. For example, in Figure 11(d), the merging around \"a tennis racquet\" is poorly executed, leading to doubts about the method's effectiveness in preserving critical details.\n3. The paper includes extensive mathematical derivations and proofs, which significantly impact its readability. The dense mathematical content makes it difficult for readers to quickly grasp the core contributions and the practical implementation of the proposed method.\n4. The paper primarily focuses on improving ToMe by addressing its token merging strategy. While this is a valid research direction, the scope of the problem tackled is relatively narrow. The significance of the improvement may not be high enough to warrant extensive interest or application, as it addresses a specific aspect of an existing method rather than introducing a fundamentally new concept or problem.\n\nQUESTIONS:\n1. In Figure 11, PITOME's token merging appears unreasonable in some cases (e.g., the tennis racquet example). Could you provide a deeper explanation or justification for these visualizations and any potential improvements to address these issues?\n2. The mathematical derivations and proofs are quite complex and impact readability. Are there ways to simplify these explanations or provide more intuitive summaries of the key points to improve accessibility for readers?\n\nLIMITATIONS:\nThe authors have acknowledged some limitations of their work and provided suggestions for future research directions. The paper does not explicitly address any potential negative societal impacts. Given the focus on efficiency improvements in Transformers, the primary societal impact would likely relate to the broader implications of making these powerful models more accessible and efficient.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}