Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification
Authors: JungMin Yun
MiHyeon Kim
YoungBin Kim
Venue: EMNLP 2023 Findings (2023)
Keywords: document classification
pre-trained transformer
attention
token pruning
token combining
TL;DR: This paper introduces a model that integrates token pruning and token combining in transformer-based pre-trained models, enhancing efficiency and performance for document classification while reducing training time and memory usage.

Abstract: Transformer-based models have achieved dominant performance in numerous NLP tasks. Despite their remarkable successes, pre-trained transformers such as BERT suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including the ones unfavorable to classification performance. To overcome these challenges, we propose integrating two strategies: token pruning and token combining. Token pruning eliminates less important tokens in the attention mechanism’s key and value as they pass through the layers. Additionally, we adopt  fuzzy logic to handle uncertainty and alleviate potential mispruning risks arising from an imbalanced distribution of each token’s importance. Token combining, on the other hand, condenses input sequences into smaller sizes in order to further compress the model. By integrating these two approaches, we not only improve the model’s performance but also reduce its computational demands. Experiments with various datasets demonstrate superior performance compared to baseline models, especially with the best improvement over the existing BERT model, achieving +5%p in accuracy and +5.6%p in F1 score. Additionally, memory cost is reduced to 0.61x, and a speedup of 1.64x is achieved.

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification
Jungmin Yun'
Mihyeon Kim!
Youngbin Kim!.2
Department of Artificial Intelligence, Chung-Ang University "Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University (cocoro357, mh10967, ybkim85). cau.ac.k.
Abstract
information across the entire sequence. This mech- anism has proven be particularly beneficial for text classification tasks (Yang ct al., 2020; Karl and Scherp, 2022; Munikar et al., 2019).
Transformer-base based models have achieved dom- inant performance in numerous NLP tasks. De- spite their remarkable successes, pre-trained transformers such as BERT suffer from com- putationally expensive self- seff-attention mecha- nism that interacts with tokens, including ones unfavorable classification performance. To overcome these challenges, we propose in- tegrating two strategies: token pruning and to- ken combining. Token pruning climinates less important tokens in the attention mechanism's 's key and value as they pass through the layers. Additionally, we adopt fuzzy logic to handle uncertainty and alleviate potential mispruning risks arising from an imbalanced distribution of each token's importance. Token combining, on the other hand, condenses input sequences into smaller sizes in order to further compress the model. By integrating these two approaches, we not only improve the model's performance but also reduce its computational demands. Ex- periments with various datasets demonstrate su- perior performance compared to baseline mod- els, especially with the best improvement over the existing BERT model, achieving +5%p in accuracy and +5.6%p in Fl ally, memory cost is reduced to 0,61x, and speedup of 1.64x is achieved.
Despite their effectiveness, BERT and similar models still face major challenges. BERT can be destructive in that not all tokens contribute to the final classification prediction (Guan et al.. 2022). Not all tokens are attentive in multi-head self-attention, and uninformative semantically meaningless parts of the input may not have posi- tive impact on the prediction (Liang al., 2022). Further, the self-attention mechanism, which in- volves interaction among all tokens, suffers from substantial computational costs. Its quadratic com- plexity relative to the length of the input sequences results in high time and memory costs, making training impractical, especially for document clas- sifications (Lee al., 2022: Pan al., 2022). response these challenges, many recent studies have attempted to address the problem of compu- tational inefficiency and improve model ability by focusing on few core tokens, thereby reducing the number of tokens that need to be processed. Their intuition is similar to human reading com- prchension achieved by paying closer attention to important and interesting words (Guan al., 2022).
Addition-
1 Introduction
One approach is pruning method that removes redundant token. Studies have shown an accept- able trade-off between performance and cost by simply removing tokens from the entire sequence to reduce computational demands (Ma et al., 2022; Goyal et al., 2020; Kim and Cho, 2020). How- ever, this method causes information loss, which degrades the performance of the model (Wei et al., 2023). Unlike previous studies, apply pruning remove tokens from the keys and values of the at- tention mechanism to prevent the information loss and reduce the cost. In our method, less impor- tant tokens are removed, and the number of tokens gradually decreases by a certain ratio as they pass
Transformer-based deep learning architectures have achieved dominant performance numer- ous areas of natural language processing (NLP) studies (Devlin et al., 2018; Lewis et al., 2019; Brown et al., 2020; Yang et al., 2019). In particu- lar, pre-trane pre-trained transformer- based language mod- els like BERT (Devlin et al., 2018) and its vari- ants (Yasunaga et al., 2022; He et al., 2020; Guo al., 2020) have demonstrated state-of-the-art per- formance on many NLP tasks. The self-attention mechanism, key element in transformers, allows for interactions between every pair of tokens in a sequence. This effectively captures contextual

---

through the layers. However, there is still risk of mispruning when the distribution of importance is imbalanced, since the ratio-based pruning does not take into account the importance distribution (Zhao et al., 2019). To address this issue, we propose to adopt the fuzzy logic by utilizing fuzzy member- ship functions to reflect the uncertainty and support token pruning.
not only improving performance, but also reducing memory usage and time costs. Our main contributions are as follows:
We introduce model that integrates token pruning and token combining to alleviate the expensive and destructive issues of self- attention- based models like BERT. Unlike pre- vious works, our token pruning approach re- moves tokens from the attention's 's key and value, thereby reducing the information loss. Furthermore, we use fuzzy membership func- tions to support more stable pruning.
However, the trade-off between performance and cost of pruning limits the number of tokens that can be removed, hence, self-attention operations may still require substantial time and memory resources. For further model compression, we propose to- ken combining approach. Another line of prior works (Pan et al., 2022; Chen et al., 2023; Bolya al., 2022; Zeng et al., 2022) have demonstrated that combining tokens can reduce computational costs and improve performance in various com- puter vision tasks, including image classification, object detection, and segmentation. Motivated by these studies, we aim to compress text sequence tokens. Since text differs from images with locality, we explore Slot Attention (Locatello et al., 2020), which can bind any object in the input. Instead of discarding tokens from the input sequence, we com- bine input sequences into smaller number of tokens adapting the Slot Attention mechanism. By doing so, we can decrease the amount of memory and time required for training, while also minimizing the loss of information.
To our knowledge, our token combining ap- proach is the first attempt to apply Slot At- tention, originally used for object localization, for lightweight purposes in NLP. Our novel application not only significantly reduces com- putational load but also improves classifica- tion performance.
Our experiment demonstrates the efficiency of our proposed model, as it improves clas- sification performance while reducing time and memory costs. Furthermore, we high- light the synergy between token pruning and combining. Integrating them enhances perfor- mance and reduces overall costs more effec- tively than using either method independently.
In this work, propose to integrate token prun- ing and token combining reduce the computa- tional cost while improving document classification capabilitics. During the token pruning stage, less significant tokens are gradually climinated as they pass through the layers. We implement pruning to reduce the size of the key and value of atten- tion. Subsequently, in the token combining stage, tokens merged into a combined token. This pro- cess results in increased compression and enhanced computational efficiency.
2
Related Works
2.1 Sparse Attention
In an effort to decrease the quadratic time and space complexity of attention mechanisms, sparse atten- tion sparsifies the full attention operation with com- plexity O where is the sequence length. Nu- merous studies have addressed the issue of sparse attention, which can hinder the ability of trans- formers to effectively process long sequences. The studies also demonstrate strong performances, es- pecially in document classification. Sparse Trans- former (Child et al., 2019) introduces sparse factor- izations of the attention matrix by using a dilated sliding window, which reduces the complexity to (n) Reformer (Kitaev et al., 2020) reduces the complexity to O(nlogn) using the locality- sensitive hashing attention to compute the near- est neighbors. Longformer (Beltagy et al., 2020) scales complexity to ((n) by combining local win- dow attention with motiivatieteed global attention,
We conduct experiments with document classi- fication datasets in various domains, employing efficient transformer-basee baseline models. Com- pared to the existing BERT model, the most sig- nificant improvements show increase of 5%pp in accuracy and an improvement of 5.6%p in the Fl score. Additionally, memory cost reduced to 0.61x, and a speedup of 1. 1.64x achieved, thus accelerating the training speed. We demonstrate that our integration results in synergistic effect

---

making it easy to process long documents. Lin- former (Wang et al., 2020) performs linear self- attention with a complexity of Omn), theoretically and empirically showing that self-attention can be approximated by a low-rank matrix. Similar to our work, Linformer reduces the dimensions of the key and value of attention. Additionally, we im- prove the mechanism by reducing the number of tokens instead of employing the linear projection, maintain the interpretability. BigBird (Zaheer et al., 2020) introduces a sparse attention method with (nn) complexity by combining random at- tention, window attention, and global attention. BigBird shows good performance on various long- document tasks, but it also demonstrates that sparse attention mechanisms cannot universally replace dense attention mechanisms, and that the imple- mentation of sparse attention is challenging, Addi- tionally, applying sparse attention has the potential risks of incurring context fragmentation and lead- ing to inferior modeling capabilities compared to models of similar sizes (Ding et al., 2020).
attention by pruning the tokens. However, there is limitation to pruning mechanisms in that the removal of tokens can result in a substantial loss of information (Kong et al., 2021).
To address this challenge, several studies have explored methods for replacing token pruning. ToMe (Bolya al., 2022) gradually combines to- kens based on their similarity instead of remov- ing redundant ones. Tekmnr (Ryoo et al., 2021) extracts important tokens from visual data and combines them using MLP to decrease the number of tokens. F-TFM (Dai et al., 2020) gradu- ally compresses the sequence of hidden states while still preserving the ability to generate token-level representations. Slot Attention (Locatello et al., 2020) learns set of task-dependent abstract rep- resentations, called "slots", bind the objects the input through self-supervision. Similar to Slot Attention, Group VIT (Xu et al., 2022) groups to- kens that belong to similar semantic regions us- ing cross-attention for semantic segmentation with weak text supervision. In contrast to GroupViT, Slot Attention extracts object-centric representa- tions from perceptual input. Our work is funda- mentally inspired by Slot Attention. To apply Slot Attention, which uses CNN as the backbone, our transformer- model, we propose a com- bining module that functions similarly to the group- ing block of GiT TPS (Wei et al., 2023) introduces an aggressive token pruning method that divides tokens into reserved and pruned sets through token pruning. Then, instead of remov- ing the pruned set, it is squeezed to reduce its size. TPS shares similarities with our work in that both pruning and squeezing are applied. However, while TPS integrates the squeezing process with pruning by extracting information from pruned tokens, our model processes combining and pruning indepen- dently.
2.2 Token Pruning and Combining
Numerous studies have explored token pruning methods that eliminate less informative and re- dundant tokens, resulting in significant computa- tional reductions in both NLP (Kim et al., 2022; Kim and Cho, 2020; Wang et al., 2021) and Vision tasks (Chen al., 2022: Kong et al., 2021; Fayyaz al., 2021; Meng et al., 2022). Attention one of the active methods used to determine the im- portance of tokens. For example, PPT (Ma et al., 2022) uses attention maps to identify human body tokens and remove background tokens, thereby speeding up the entire network without compro- mising the accuracy of pose estimation. The model that uses the most similar method to work to determine the importance of tokens LTP (Kim al., 2022). LTP applies token pruning to input sequences in order to remove less significant to- kens. The importance of each token is calculated through the attention score. On the other hand, Dy- ViT (Rao et al., 2021) proposes an learned token selector module to estimate the importance score of each token and to prune less informative tokens. Transkimmer (Guan et al., 2022) leverages the skim predictor module to dynamically prune the sequence of token hidden state embeddings. Our work can also be interpreted as a form of sparse attention that reduces the computational load of
Methods
In this section, we first describe the overall archi- tecture of our proposed model, which integrates token pruning and token combining. Then, we in- troduce each compression stage in detail, including the token pruning strategy in section 3.2 and the token combining module in section 3.3.
3.1 Overall Architecture
Our proposed model architecture is illustrated in Figure 1. The existing BERT model (Devlin et al.,

---

Fuzzy-based Token Pruning Self-attention
Label
Token Combining Module
Mron
Layer
LL
Atteacioon Black
6
Token Combining Module
Layer
VIA
HA
Add Norm
fu
Add Norm
Fuzzy bused Token Pruning Self attention
66
Cobona Mean
F Fillering
Emabediina
Steg)
Figure I: Overall architecture of our purposed model Model architecture is composed of several Token-pruned Attention Blocks, a Token Combining Module, and Attention Blocks. (Left): Fuzzy-based Token Pruning Self- attention In each layer, fuzzy-based pruning method removes tokens using importance score and fuzzy membership function. (Right): Token Combining Module This module apportions embedded tokens to each of the combination token using similarity matrix between them.
2018) consists of stacked attention blocks. We modify the vanilla self-attention mechanism by ap- plying fuzzy-hasee baaze token pruning strategy. Sub- sequently, we replace one of the token-pruned at- tention blocks with token combining module. Re- placing a token-pruned attention block instead of inserting an additional module not only enhances model performance but also reduces computational overhead due to its dot product operations. First, suppose X {=>]== is a sequence token from input text with sequence length Given X, let = [e be an embedded token after pass- ing through the embedding layer. Each e; is an embedded token that corresponds to the sequence token z4- Additionally, we add learnable combi- nation tokens. Suppose C' = {m] is a set of learnable combination tokens, where m is the num- ber of combination tokens. These combination tokens bind other embedded tokens through the token combining module. We simplify {e]]__ to {et) and (c))m {c:) We concatenate {G] and {c) and use them as input for token-prume atten- tion blocks. We denote fuzzy-based token pruning feed-forward layers
The token combining module receives (0:') and { input and merges {E,'} into { to output combined tokens (c After the token combin- ing module, subsequent attention blocks do not perform pruning. Finally, we obtain the sequence representation by aggregating the output tokens {ri). in which our method averages the output.
3.2 Fuzzy-based Token Pruning Self-attention
We modify vanilla self-attention by implementing token pruning- Our token pruning attention mechanism gradually reduces the size the key and value matrices by eliminating relatively unimportant embedded tokens, except for the combination tokens.
Importance Score We measure the signifi- cance of tokens based their importance score. For each layer and head, the attention probability Atlentionpros is defined as:
L L
Attention prob = softmar
Vd )
Raxx
self-attention by FTPАu by FF, and layer norm by LN. The operations performed within the token pruned attention block in 1-th layer are as follows:
(3)
where I the layer index, is the head index, d is d the feature dimension, and Q/ah, k//h E TR"x" & in- dicate query, key, and respectively. Attention is interpreted as similarity between the i-th token e; and the j-th token ej with row index i E [1,n] and column index E [1, n]. As the similarity increases, a larger weight is assigned the value corresponding to e, The j-th column in Equation
(a))(a)) FTP Amn ((e)) (1)
( (N(FF(((():(C)))

---

3 represents the amount of token ej attended by other tokens E, (Wang et al., 2021). Therefore, ej is considered relatively important token as attended by more tokens. We define the importance score S(e)) in layer and head h as:
previous work (Zhao et al., 2019) on fuzzy-based filter pruning in CNN, we design fuzzy mem- bership functions for Imporlance(S(c)) and Unimportance(S(e as:
if (e)) S a
S(e,)
1 " 7
Σ
Lh (Attention prodiss
Importance(S)
S(r) b
if a < S(e) <b
if S(c) 2b
(6)
Token Preservation Ratío After calculating the importance score using Op and Kp in the 1-th layer, we select 1 embedded tokens in descending or- der of their scores. The 4+1 embedded tokens then indexed for Kp and v in the (Z + 1)-th layer. Other embedded tokens with relatively low impor- tance score are pruned as a result. We define the number of tokens that remain after token pruning in the (1 + 1)-th layer as:
if if) S
Unimportance(S) - b-S(e)
if < S(e) <b
if 2
(7)
where we simplify the importance score S(e,) to S(e). Unlike the previous work(Zhao et al.,
2019) that uses fixed constants as hyperparame- ters, our approach adopts the quantile function Q s(e)) (0.25) and (0tc))0.73) for and b, respec- tively, to ensure robustness. We compute quan- tile function for all importance scores, capturing the complete spectrum of head information. The importance set and the unimportance defined using the a - cut, commonly referred to as " A - x A(x) 2 a in fuzzy theory. To mitigate information loss due to imbalanced distribution, we employ token pruning based on the preservation ratio for tokens that fall within the set (I U) In the initial layers, where attention might not be adequately trained, there's risk of erroneously removing crucial tokens. To counteract this, we've set the for the a for U is empirically set to 0.9. Finally, our fuzzy- based token pruning self-attention FTP Atto is defined as
g
I xp
(5)
where 441 depends on p. a hyperparameter indicating the token preservation ratio of r to This preservation ratio represents the proportion of tokens that are retained after pruming, relative the number of tokens before pruning. As token pruning not performed in the first layer, 4 n. and the attention uses the entire token in Qp-
are
Kp, and V/p- the (I + 1)-th layer, tokens are pruned based on S(ey) with oha² e T and K.AA x where (++1 < t.. In the subsequent layers, the dimensions of Kp and Vp gradually decreases.
minimal value of 0.01, while
Fuzzy-based Token Pruning However, simply discarding a fixed proportion of tokens based on importance score could lead to mispruning. Especially in imbalanced distributions, this pruning strategy may remove erucial tokens while retaining unimportant ones, thereby decreasing the model accuracy (Zhao et al., 2019). Insufficient training in the initial layers of the model can lead to uncertain importance scores, thereby increasing the risk of mistakenly pruning essential tokens. Furthermore, the importance score of token is relative, and the distinction between the degree of importance and unimportance may be unclear and uncertain. To address this challenge, exploit fuzzy theory, which can better perceive uncertainty. We employ two fuzzy membership functions to evaluate the degree of importance and unimportance together. Inspired by the
K2 24 h kihh )ih
FTPAu = -off softmax(
Vd
(0%) E B K/A, vlh E "xx
(8)
3.3 Token Combining Module
Token combining module takes token-pruned atten- tion block's output representation é,', à as inputs. Combination tokens, which are concatenated with embedded tokens, pass through token likenn-runed at- tention blocks incorporate global information from input sequences. Then, combination tokens integrate embedded tokens based on their similarity in the embedded space. Similar to Group VIT (Xu et al., 2022), our token combining module uses

---

The primary aim of this work is to address the issues of the BERT, which can be expen- sive and destructive. To evaluate the effective- ness of our proposed model, we conduct ex- periments comparing it with the existing BERT For fair compar- Gumbel-Softmax (Jang al., 2016) to perform cross-attention between combination tokens and embedded tokens. We define the similarity matrix Sim as:
investigate the time and memory costs of our proposed model and evaluate its efficiency. Lastly, through the ablation study, we compare the effects of different preservation ratios on fuzzy-based to- ken pruning self-attention. We also analyze the impact of the position of the token combining mod- ule and the number of combination tokens.
Simby
CX -eLL)())))))) g: E (X W&LN(G)))
4.1 Dataset
We evaluate our proposed model using six datasets across different domains with varying numbers of classes. SST-2 (Socher et al., 2013) and IMDB (Maas et al., 2011) are datasets for timent classification on movic reviews. BBC News (Greene and Cunningham, 2006) and 20 NewsGroups (Lang, 1995) comprise collec- tion of public news articles on various topics. LEDGAR (Tuggener et al., 2020) includes cor- pus of legal provisions in contract, which is part of the LexGLUE (Chalkidis al., 2021) benchmark to evaluate the capabilities of legal text. arXiv is a digital archive that stores scholarly articles from a wide range of fields, such as mathematics, computer science, and physics. We use the arXiv dataset employed by the controller (He et al., 2019) and perform classification based on the abstract of the paper as the input. We present more detailed statistics of the dataset in Table 1.
where LN layer normalization, Wg and Wi are the weights of projection matrix for the com- bination tokens and embedded tokens, respec- tively, and }9i} are i.i.d random samples from the (umbel(O,) 1) distribution. Subsequently, we implement hard assignment technique (Xu et al., 2022), which employs one-hot operation to de- termine the specific combination token to which each embedded token belongs. We define hard assignment A as:
H H = (Simka) + Simb, - sg(Simba),
M4 = mar((amma (Simi,))
(10)
where sg is the stop gradient operator to stop the accumulated gradient of the inputs. We update the combination token by calculating the weighted sum of the embedded token that corresponds to the same combination token. The output of the token combining block is calculated as follows:
Dataset SST-2 IMDB
Genre
review
2
9613
23.2
review
2
50000
292.2
= c.' Wo E H
BBC News
news
2225
452.7
(11)
NewsGroup
news
18846
330.6
LEDGAR arXiv
legul
100
80000
138.7
scientific publication
11
1200
202.6
where W. and Wo are the weights of the projec- tion matrix. We adopt the grouping mechanism described in ViT. GioupViT learns semantic segmentation by grouping output segment tokens to object classes through several grouping stages. Our method, on the other hand, replaces one layer with a token combining module and compresses embedded tokens to a few informative combined tokens. Empirically, we find that this approach re- duces the training memory and time the model, increasing performance.
Table I: Statistics of the datasets. C denotes the number of classes in the dataset, / the number of instances, and T the average number of tokens calculated using BERT(bert weserrtt-baeuuucccceee tokenizer.
4.2 Experimental Setup and Baselines
4 Experiments
This section aims to validate the effectiveness of our proposed model. Firstly, we evaluate the doc- ument classification performance of our proposed model compared to the baseline models. Secondly,
ion, both of BERT and our proposed model fol- low the same settings, and ours is warm-started on bert-base-uncased. Our model has the same number of layers and heads, embedding and hid-

---

Dutaset
Model
Location
Accuracy
Fitmeero)
Fitmicro)
Dataset Model
Locution
Accuracy
Fi(macro)
Pi(miero)
BigBird
91.8
91.8
91.8
BigBird
69.3
69.3
Longformer
91.0
91.0
91.0
Longformer
68.5
67.0
68.5
F-TFM
92.0
92.0
92.0
F-TFM
69.7
68.5
69.7
Transkimmer
91.4
91.4
91.4
NewsGroup
Trunskimmer
67.8
64.9
67.8
BERT
90.7
89.4
90.7
BERT
68.7
56.
68.7
ours-P
89.8
91.1
ours-P
69.4
70.0
ours-PF
91.4
90.1
91.4
ours-PF
70.7
58.2
70.7
ours-C
layer 11
91.8
90.8
91.8
ours-C
layer
69.2
56.5
69.2
ous-PFFC
layer 11
92.1
91.0
92.1
ours-PFC
layer 11
69.9
57.1
69.9
ours PFC
layer?
90.1
88.5
90.1
ours- PFC
layer
68.5
55.4
68.5
RigBird
92.8
92.8
92.8
BigBird
86.9
78.8
86.9
Longformer
93.4
93.4
93.4
Langformer
84.0
82.
84.0
F-TFM
91.7
91.7
91.7
F-TFM
86.7
78.4
86.7
Transkimmer
93.7
93.7
93.7
Transkimmer
87.1
77.3
87.1
BERT
93.7
92.8
93.7
LEDGAR
BERT
86.2
77.4
86.2
ours-P
93.8
92.9
93.8
ours-P
86.5
77.9
86.5
ou
94.3
93.4
94.3
ours-PF
86.8
78.4
86.8
layer
layer
86.9
ours-PFC
layer
93.5
92.5
93.5
ours-PFC
layer
87.3
79.2
87.3
outs-PFC
layer?
92.8
91.5
92.8
ours-PFC
layer
85.7
76.8
85,7
BigBird
97.1
97.1
BigBird Longformer
74.0
74.0
Longformer
97.9
97.9
97.9
66.0
64.8
66,0
F-FFM
96.5
96.5
96.5
F-TEM
70.0
66.5
70.0
Transkimmer
97.6
97.6
97.6
Transkimmer
73.7
72.6
73.7
BBC
96.2
Now Sooe
97.2
97.2
71.0
56.6
70.2
ours-PF
97.9
96.6
97.9
ours-PF
76.0
61.0
74.0
ours C
layer 11
96.8
95.2
96.8
ours C
layer 11
70.0
52.7
69.2
ours-PFC
layer 11
98.1
97.1
98.1
ours-PPC
layer 11
74.0
58.1
73.1
layer?
layer?
Table 2: Performance comparison on document classification. To our proposed model, ours-P applies token pruning. ours-PF applies buzzy-based token pruning, ours-C applies token combining module, and ours-PFC applies both fuzzy-based token pruning and token combining module. The best performance is highlighted in bold.
Model
Location
FLOPs
Memory Cost
Specdup
Ba, 2014). The first 512 tokens of the input quences are processed. The learning rate is set to 2e-5, and we only use 3e-5 for the LEDGAR dataset. We also use a linear warm-up learning rate scheduler. In all experiments, we use a batch size of 16. We choose the model with the lowest validation loss during the training step the best model. We set the token preservation ratio p 0.9.
BigBird
I.57x
0.82x
0,94x
Longformer
1.x
L.1x
0.55x
F-TFM
1.03x
0.39x
1.11x
Transkimmer
0.13x
0.87x
0.70x
BERT
ours-P
Ix
0.88x
1.33x
ours-PF
Ix
0.88x
1.25x
layer layer layer 7
0.877x
1,26x
ours-PFC
0.877x
0.80x
1.29x
OFCs-PFC.
0.544x
0.61x
1.64x
Table 3: Efficiency comparison document classifica-
4.3 Main Result
tion.
To evaluate the performance and the effi- ciency of each strategy, we compare our pro- posed model(ours-PFC) with five baselines and three other models: one that uses only token pruning(ours-P), one that applies fuzzy member- ship functions for token pruning(ours-PFF, and that uses only token combining module(ours-C), as shown in Table 2 and Table
den sizes, and dropout ratio BERT. Our model has the same number of layers and heads, embed- ding and hidden sizes, and dropout ratio as BERT, We also compare our method to other baselines, including BigBird (Zaheer et al., 2020) and Long- former (Beltagy et al., 2020), which employ sparse attention, as well as F-TFM (Dai et al., 2020) and Transkimmer (Guan et al., 2022), which utilize to- ken compression and token pruning, respectively. For IMDB, BBC News, and 20NewsGroup. 20% of the training data is randomly selected for vali- dation. During training, all model parameters are fine-tuned using the Adam optimizer (Kingma and
Compared to BERT, ours-P consistently outper- forms it for all datasets with higher accuracy and Fl scores. achieving approximately 1.33x speedup and 0.88x memory savings. More importantly, the performance of ours-PF significantly surpasses that of ours-P with up to 5.0%p higher accuracy, 4. 4.4%p higher (mmcro) score, and 3.8%p higher

---

(miieo) score, with the same FLOPs and compa- rable memory costs. To evaluate the performance of ours-C and ours-PFC, we incorporate the com- bining module at the 11-th layer, which results in the optimal performance. A comprehensive discus- sion on performance fluctuations in relation to the location of the combining module is presented in Section 4.4, Excluding the IMDB dataset, ours-C not only achieves higher values in both the accuracy and Fl scores compared to the BERT but also ex- ceeds by 0.89x speedup and I. 1,26x memory savings while reducing FLOPs. Across all datasets, our models(ours-PF or ours-PFC) consistently outper- form all efficient transformer- base baseline mod- els. Furthermore, ours-PFC outperforms the BERT with up to 5.0%p higher accuracy, 5.6%pp higher macro Fl score, and 4.8%p higher miero Fl score. Additionally, ours-PFC exhibits the best perfor- mance with the least amount of time and memory required, compared to models that use pruning or combining methodologies individually. These find- ings highlight the effectiveness of integrating token pruning and token combining on BERT's document classification performance, from both the perfor- mance and the efficiency perspective.
attention mechanism. As p decreases, we observe that performance deteriorates and time and memory costs decrease. A smaller p leads to the removal of more tokens from the fuzzy-basee token pruning self-attention. While leading a higher degree of in- formation loss in attention, it removes more tokens can result in time and memory savings.
Preservation Ratio
Accuracy
Memory Cost
Speedup
97.7
0.99x
1.07x
0.9
97.9
0.86x
1.12x
0.85
97.6
0.79x
1.14x
0.8
97.2
0.75x
1.17x
0.75
97.4
0.71x
1.19x
0.7
97.2
0.69x
1.24x
0.65
96.8
0.68x
1.25x
0.6
96.5
0.67x
1.25x
BERT
96.2
Table 4: Comparisons of different token preservation
ratios
Token Combining Module In Table 5, we compare the positions of different token combining mod- ules on the BBC News dataset. We observe that placing the token combining module carlier within the layers results in greater speedup and memory savings, Since fewer combined tokens proceed through subsequent layers, the earlier this process begins, the greater the computational reduction that can achieved. However, this reduction in the in- teraction between the combination token and the embedded token hinders the combination token to learn global information. potentially degrading the model performance. Our proposed model shows the highest performance when the token combining module is placed in the 1 11th layer, achieving an accuracy 98.1%.
Subsequently, we evaluate the potential effec- tiveness of ours-C and ours-PFC by implementing the combining module at the 7-th layer. As shown in Table 5 of Section 4.4, applying the combining module the 7-th layer leads further time and memory savings while also mitigating the potential decrease in accuracy. Compared to BERT, only shows minimal decrease in accuracy (at most 0.8%p). Moreover, it reduces FLOPs and memory costs to 0.61x, while achieving 1.64x speedup. In our experiments, we find that our proposed model effectively improves document classification per- formance, outperforming all baselines. Even when the combination module is applied the 7th layer, it maintains performance similar to BERT while further reducing FLOPs, lowering memory usage, and enhancing speed.
Layer
Accuracy
Memory Cost
Speedup
FLOPSs
96.5
0.47x
1.69x
2.65x
6
96.6
0.54x
1.53x
2.17x
97.8
0.61x
1.39x
1.84x
8
97.2
0.68x
1.29x
1.60x
97.4
0.76x
1.17x
1.41x
10
97.8
0,82x
1.17x
1.26x
4.4 Ablation study
98.1
0.89x
1.03x
1.14x
Token Preservation Ratio We evaluate different token preservation ratios p the BBC News dataset, as shown in Table 4. Our findings indicate that the highest accuracy, at 98.1% ,, was achieved when p 0.9. Moreover, our fuzzy-based pruning strategy results in 1.17x reduction in memory cost and 1.12x speedup compared to the vanilla self-
11(*)
(97.7)
(0.90x)
(0.88x)
(1.14x)
97.6
0.96x
0.97x
1.04x
Table 5: Comparisons according to combining module location. We replace one of the existing transformer layers with token combining module. (*) represents case where combining module is additionally inserted without replacing.

---

Combination Tokens In Table 6, compare the accuracy achieved with different numbers of com- bination tokens. We use four datasets to analyze the impact of the number of classes on the combi- nation tokens. We hypothesized that an increased number of classes would require more combina- tion tokens to encompass greater range of in- formation. However, observe that the highest accuracy is achieved when using eight combina- tion tokens across all four datasets. When more combination tokens are used, performance grad- ually degrades. These results indicate that when the number of combination tokens is fewer than the number of classes, each combination token can represent more information as a feature vector in 768-dimensional embedding space, similar to find- ings in GiT. Through this experiment, find that the optimal number of combination tokens is 8. We show that our proposed model performs well in multi-class classification without adding computation and memory costs.
Our findings also demonstrate synergistic effect by integrating token pruning and token combining. commonly used object detection and semantic segmentation. Ultimately, our research provides a novel way to use pre-trained transformer mod- els more flexibly and effectively, boosting perfor- mance and efficiency in myriad of applications that involve text data processing.
Limitations
In this paper. our goal address the fundamental challenges of the BERT model, which include high cost and performance degradation, that hinder its application to document classification. We demon- strate the effectiveness of our proposed method, which integrates token pruning and token combin- ing, by improving the existing BERT model. How- ever, model, which is based BERT, has in- herent limitation in that it can only handle input se- quences with a maximum length of 512. Therefore, is not suitable for processing datasets that are longer than this limit. The problems arising from the quadratic computation of self-attention and the existence of redundant and uninformative tokens are not specific to BERT and are expected to inten- sify when processing longer imput sequences. Thus, will improve other transformer-bast based models that can handle long sequence datasets, such as LexGLUE, and are proficient performing natural language inference tasks in our future work.
Number of Combination Tokens
Dataset
4
8
16
32
64
SST-2
2
91.2
92
90.9
91.2
89.7
BBC News
5
97.3
98.1
97.5
97.6
97.2
20 NewsGroup
20
68.5
69.9
68.9
68.7
67.6
LEDGAR
100
85.6
87.3
86.7
86
85.9
Table 6: Performance comparison of different numbers of combination tokens. C' denotes the number of classes in the dataset.
5 Conclusion
Ethics Statement
this paper, we introduce an approach that inte- grates token pruning and token combining to im- prove document classification by addressing expen- sive and destructive problems of self-attention in the existing BERT model. Our approach consists of fusey token pruned attention and token combining module. Our pruning strategy gradually removes unimportant tokens from the key and value in attention. Moreover, we enhance the robustness of the model by incorporating fuzzy membership functions. For further compression, our token com- bining module reduces the time and memory costs of the model by merging the tokens in the input sequence into a smaller number of combination tokens. Experimental results show that our pro- posed model enhances the document classification performance by reducing computational require- ments with focusing on more significant tokens.
Our research adheres to ethical standards of prac- tice. The datasets used to fine-tune our model are publicly available and do not contain any sensitive or private information. The use of open-source data ensures that our research maintains transparency. Additionally, our proposed model built upon a pre-trained model that has been publicly released. Our research goal aligns with an ethical commit- ment to conserve resources and promote acces- sibility. By developing a model that minimizes hardware resource requirements and time costs, making valuable contribution towards a more accessible and inclusive Al landscape. We aim to make advanced AI techniques, including our proposed model, accessible and practical for researchers with diverse resource capacities, ulti- mately promoting equity in the field.

---

Acknowledgment
bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805.
This research was supported by Basic Science Re- search Program through the National Research Foundation of Korea(NRF) funded by the Ministry of
Siyu Ding, Junyuan Shang, Shuohuan Wang. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie- doc: A retrospective long-document modeling trans- former. arXiv preprint arXiv:2012.15663
and In- stitute for Information & communications Tech- nology Planning & Evaluation (ITP) through the Korea government (MSIT) under Grant No. 2021- 0-01341 (Artificial Intelligence Graduate School
Mohsen Fayyaz, Soroush Abbasi Kouhpayegani, Farnoush Rezaei Jafari, Eric Sommerlade, Hamid Reza Vaezi Joze, Hamed Pirsiavash, and Juergen Gall, 2021. Ats: Adaptive token sampling for efficient vi- sion transformers. arXiv preprint arXiv:21 111 156667.
Program, Chung-Ang University).
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimina- tion. In International Conference on Machine Learn- ing, pages 3690-3699. PMLR.
References
Beltagy, Matthew E Peters, and Arman Cohan, 2020. Longformer: The ang-documen transformer. arXiv preprint 200.::0000405500.
Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2022. Token merging: Your vit but faster. arXiv preprint arXiv:2210.0946.
Derek Greene and Pádraig Cunningham. 2006. Practical solutions to the problem diagonal dominance kernel document clustering. In Proceedings of the 23rd international conference Machine learning, pages 377-384.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Yue Guan, Zhengyi Li, Jingwen Leng, Zhouhan Lin, and Minyi Guo. 2022. Transkimmer: Trans- former learns to layer-wise skim. arXiv preprint arXiv:2205.07324.
Daya Guo, Shuo Ren. Shuai Lu, Zhangyin Feng, Duyu Tang, Shujic Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, al. 2020. Graphcode- bert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.000
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. 2021. Lexglue: A bench- mark dataset for legal language understanding in en- glish. arXiv preprint arXiv:2 10.00097..
Jun He, Liqun Wang, Liu Liu, Jiao Feng, and Hao Wu. 2019. Long document classification from local word glimpses via recurrent attention learning. IEFE Ac- cess, 1440707-407115
Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao. and Ping Luo. 2023. Diffrate: Differentiable com- pression rate for efficient vision transformers, arXiv prepriat arXiv:2305.1 17997.
Pengeheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint urXiv:2006.03654
Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. 2022. The princi- ple of diversity: Training stronger vision transformers calls for reducing all levels redundancy. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12020-12030.
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categori- cal reparameterization with gumbel-softmax. arXiv preprint arXiv: 167 1611.011111111 1671101144.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long se- quences with sparse transformers. arXiv preprint arXiv:190..10009
Fabian Karl and Ansgar Scherp. 2022. Transformers are short text classifiers: A study inductive short text classifiers on benchmarks and real-world datasets. arXiv preprint arXiv:2211.16877
Zihang Dai, Guokun Lai. Yiming Yang, and Quoe Le. 2020. Funnel-transformer Filtering out sequential redundancy for efficient language processing. Ad- vances in neural information processing systems, 33:4271-4288
Gyuwan Kim and Kyunghyun Cho. 2020. Length- adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep
Sehoon Kim. Sheng Shen, David Thorsley, Amir Gho- lami, Woosuk Kwon, Joseph Hassoun, and Kurt

---

Keutzer. 2022. Learned token pruning for transform- ers. In Proceedings of the 28th ACM SIGKDD Con- ference Knowledge Discovery and Data Mining, pages 784-794.
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. 2022. Adavit: Adaptive vision transformers for ef- ficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 12309-12318.
Diederik P Kingma and Jimmy Ba. 2014, Adam: method for stochastic optimization. arXiv preprint 1eXiv:1412.69%0
Manish Munikar, Sushil Shakya, and Aakash Shrestha. 2019. Fine-grained sentiment classification using bert. In 2019 Artificial Intelligence for Transforming Business and Society (AITB), volume 1, pages 1-5. IEFE.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint (xXiv:000.00455)
7izheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. 2022. Less is more: Pay less attention in vision transformers. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Mengshu Sun, Wei Niu, Xuan Shen, Geng Yuan, Bin Ren, Minghai Qin, al. 2021. Spvit: Enabling faster vision transformers via soft token pruning. arXiv preprint ar arXh::2112.3388
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Er- ficient vision transformers with dynamic token sparsi- fication. Advances in neural information processing systems, 34:13937-13949.
Ken Lang. 1995. Newsweeder: Learning to filter net- news. In Machine learning proceedings 1995, pages 331-339. Elsevier.
Minchul Lee, Kijong Han, and Myeong Cheol Shin. 2022. Littlebird: Efficient faster & longer trans- former for question answering. arXiv preprint 11X11:2100.11800.
Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. Tokenlearner: Adaptive space-time tokenization for videos. Ad- vances in Neural Information Processing Systems, 34:12786- 127977
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to toacceett--oseee sequence pre- pre-trainin for natural language generation, translation, and comprehension. arXiv preprint urXiv:I910 13461.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642
Youwei Liang. Chongjian Ge, Zhan Tong. Yibing Song, Jue Wang, and Pengtao Xie. 2022. Not all patches are what you need: Expediting vision trans- formers via token reorganizations. arXiv preprint arXiv:220..07800.
Don Tuggener, Pius Von Däniken, Thomas Peetz, and Mark Cieliebak. 2020. Ledgar: large-scale multi- label corpus for text classification of legal provisions in contracts. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1235- 1241.
Francesco Locatello, Dirk Weissenbomn, Thomas Un- terthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33:11525-11 -1538.
Hanrui Wang Zhekai Zhang, and Song Han. 2021. Spat- ten: Efficient sparse attention architecture with cas- cade token and head pruning. In 2021 IEEF Interna- tional Symposium on High-Performance Computer Architecture (HPCA), pages 97-110. IEEE
Haoyu Ma. Zhe Wang, Yifei Chen, Deying Kong, Liangjian Chen. Xingwei Liu, Xiangyi Yan, Hao Tang, and Xiaohui Xie. 2022. Ppt: token-pruned pose transformer for monocular and multi-view hu- man pose estimation. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part V, pages 424-442. Springer.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint Xixv:2006.04768
Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. 2023. Joint token pruning and squeez- ing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 2092 2101.
Andrew Maas. Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. 2022. Groupvit: Semantic segmentation

---

emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18 18134-18144.
Liu Yang, Mingyang Zhang, Cheng Li, Michael Ben- dersky, and Mare Najork. 2020. Beyond 512 tokens: Siamese multi-depth transformer-based hierarchical encoder for long-form document matching. In Pro- ceedings of the 29th ACM International Conference on Information Knowledge Management, pages 1725-1734.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Car- bonell, Russ R Salakhutdinov, and Quoe V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Michihiro Yasunaga, Jure Leskovec, and Perey Liang. 2022. Linkbert: Pretraining language models with document links. arXiv preprint 2rXiv:2003.158/
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tamon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 13:17783-17297.
Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo. Wanli Ouyang, and Xiaogang Wang. 2022. Not all tokens are equal: Human-centric visual analysis via token clustering transformer, In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 101-11111.
Bin Zhao, Yue Li, and Shang. 2019. Fuzzy pruning for compression of convolutional neural net- works. In 2019 IEEE International Conference Fuzzy Systems (FUZZ-IEEE) pages 1-5. IEEE


Reviews:
[{"review_id": "nQJ3I4WVDr", "reviewer": "Reviewer_6yQm", "summary": "", "questions": "1.  What will be the wall time for your methods and baselines?\n\n2. What is the initialization of combination tokens?   What is the scale of these tokens? Is it specific to one task or one sentence?\n\n3. Why is this method compatible with the pure transformer-based Bert model, since this method introduces many new modules and modifies attention?\n\n4. Is this method restricted to classification tasks but not generation tasks?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you've invested in reviewing our manuscript. Your insights are invaluable, and we recognize the importance of addressing your concerns. We're committed to clarifying and enhancing our work based on your feedback.\n\n### Response to Weakness 1: \n\n\n>This paper lacks a lot of comparisons in the experiments. First, it should compare with other efficient methods that sparsify attention or prune tokens. Second, this paper does not compare different techniques in document classification. Third, this paper does not compare with other papers that use virtual tokens.\n\nWe appreciate your feedback on the need for broader experimental comparisons. We've taken steps to address this:\n\n**Baselines:**\n- **BigBird[1]**: Uses a sparse attention mechanism to reduce the quadratic dependency of full attention.\n- **Longformer[2]**: Utilizes a linearly scaling attention mechanism, widely used in document classification tasks.\n- **Funnel-Transformer[3]**: Reduces computation by compressing tokens, filtering out sequential redundancy.\n- **Transkimer[4]**: A token pruning model with a dynamic strategy, outperforming several other pruning-based models.\n\nFor token combinations using virtual tokens, our approach is novel, and we found no directly comparable works in NLP tasks, especially document classification.\n\n\n**Performance Comparison on Document Classification:**\n\n|                        | **SST-2**       | **IMDB**        | **BBC News**    | **20NewsGroups** | **arXiv**       | **LEDGAR**      |\n|------------------------|-----------------|-----------------|-----------------|------------------|-----------------|-----------------|\n| **BigBird**            | 91.8 / 91.8     | 92.8 / 92.8     | 97.1 / 97.1     | 69.3 / 67.7      | 74.0 / 70.4     | 86.9 / 78.8     |\n| **Longformer**         | 91.0 / 91.0     | 93.4 / 93.4     | 97.9 / 97.9     | 68.5 / 67.0      | 66.0 / 64.8     | 84.0 / 82.1     |\n| **Funnel-Transformer** | 92.0 / 92.0     | 91.7 / 91.7     | 96.5 / 96.5     | 69.7 / 68.5      | 70.0 / 66.5     | 86.7 / 78.4     |\n| **Transkimmer**        | 91.4 / 91.4     | 93.7 / 93.7     | 97.6 / 97.6     | 67.8 / 64.9      | 73.7 / 72.6     | 87.1 / 77.3     |\n| **BERT**      | 90.7 / 89.4     | 93.7 / 92.8     | 96.2 / 94.0     | 68.7 / 56.1      | 69.0 / 52.5     | 86.2 / 77.4     |\n| **ours-P**             | 91.1 / 89.8     | 93.8 / 92.9     | 97.2 / 95.4     | 69.4 / 56.8      | 71.0 / 56.6     | 86.5 / 77.9     |\n| **our-PF**             | 91.4 / 90.1     | **94.3 / 93.4** | 97.9 / 96.6     | **70.7 / 58.2**  | **76.0 / 61.0** | 86.8 / 78.4     |\n| **ours-C**             | 91.8 / 90.8     | 92.6 / 91.3     | 96.8 / 95.2     | 69.2 / 56.5      | 70.0 / 52.7     | 86.9 / 78.4     |\n| **ours-PFC (L11)**     | **92.1 / 91.0** | 93.5 / 92.5     | **98.1 / 97.1** | 69.9 / 57.1      | 74.0 / 58.1     | **87.3 / 79.2** |\n| **ours-PFC (L7)**      | 90.1 / 88.6     | 92.8 / 91.5     | 97.0 / 95.2     | 68.5 / 55.4      | 68.0 / 50.7     | 85.7 / 76.8     |\n\n**Efficiency Comparison on Document Classification:**\n\n| Model              | FLOPs(↓)   |Memory Cost(↓)|  SpeedUp(↑) |\n| --------           | -------- | -----     | --------  |\n| **BERT**      |     (1.0x)    | (1.0x)         |  (1.0x)        |\n| **ours-PFC (L11)**     | 0.88x    | 0.80x     | 1.29x     |\n| **ours-PFC (L7)**      | 0.54x    | 0.61x     | 1.64x     |\n| **BigBird**            | 1.57x    | 0.82x     | 0.94x     |\n| **Longformer**         | 2.10x    | 1.11x     | 0.55x     |\n| **Funnel-Transformer** | 1.03x    | 0.39x     | 1.11x     |\n| **Transkimmer**        | 0.13x    | 0.87x     | 0.70x     |\n\n\n**Key Updates:**\n1. We've integrated results from the four mentioned baselines into our experimental section.\n2. Our approach, \"ours-PFC\", consistently showcases superior performance and efficiency across datasets.\n3. Our model demands fewer computational resources and memory while maintaining excellent performance, emphasizing its efficacy in document classification.\n\n```\n1. Zaheer, Manzil, et al. \"Big bird: Transformers for longer sequences.\" Advances in neural information processing systems 33 (2020): 17283-17297\n2. Beltagy, Iz, Matthew E. Peters, and Arman Cohan. \"Longformer: The long-document transformer.\" arXiv preprint arXiv:2004.05150 (2020).\n3. Dai, Zihang, et al. \"Funnel-transformer: Filtering out sequential redundancy for efficient language processing.\" Advances in neural information processing systems 33 (2020): 4271-4282.\n4. Guan, Yue, et al. \"Transkimmer: Transformer Learns to Layer-wise Skim.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n```\n\n\n### Response to Question 1: \n\n> What will be the wall time for your methods and baselines? \n\nThank you for this insightful question. In response, we have measured the wall time for both our proposed methods and the selected baseline models to offer a comprehensive perspective on the computational aspects of our study.\n\n**Limitation of Wall Time**\n\nWall time has the following weaknesses:\n* **Hardware and Environmental Dendencies:** Wall time can be affected by running hardware environments, software settings, and various external factors. Even the same model can run at different times when running on different systems.\n* **Limits of Relative Comparison:** Wall time is mainly used for relative comparison in a specific environment. However, performance differences can vary when running in different environments. Therefore, direct comparison with wall time in other studies or applications can be difficult.\n\n**Wall Time Results**\n| Model             | BigBird | Longformer | Funnel-Transformer | Transkimmer | BERT | ours-PFC (L11) | ours-PFC (L7) |\n|-------------------|---------|------------|--------------------|-------------|------|----------------|---------------|\n| **Wall time(s)** | 315     | 250        | 147                | 297         | 211  | 180            | 159           |\n\n1. We measured wall time through the actual time it took from the start of the program to the completion of all tasks in document classification with the BBC news dataset, which requires the least amount of time compared to other datasets.\n2. Taking BERT as the standard, it can be confirmed that our proposed methodology and Funnel-Transformer effectively reduced the required time. As evident from the discussion in \"Response to Weekness 1,\" when evaluating model performance based on accuracy and F1 score, our model outperformed Funnel-Transformer with a relatively short wall time compared to other models.\n3. Evaluating wall time serves multiple purposes: it enhances transparency, aids in assessing a model's practical utility, and demonstrates its potential for effective real-world applications.\n\nBy integrating these results into our revised manuscript, our intention is to provide readers with insights into the practical implications and viability of implementing our methods in real-world scenarios.\n\n\n### Response to Question 2: \n\n> What is the initialization of combination tokens? What is the scale of these tokens? Is it specific to one task or one sentence?\n\n**Initialization of Combination Tokens:**\nDrawing from the methodology in GroupViT[1], our combination tokens are initialized as learnable entities with zero values.\n\n**Scale of the Tokens:**\nThe dimensions of these combination tokens are set as (batch size: 16, token length: 512, hidden size: 768). This aligns with the embedded token's dimensions. Based on empirical results, we've set the number of combination tokens to 8.\n\n**Task or Sentence Specificity:**\nThe combination tokens are designed to bind with other embedded tokens, which represent sequence tokens from the input text. This binding occurs within the token combining module. Therefore, the specificity of these tokens is more aligned with individual sentences than broad tasks.\n\n```\n1. Xu, Jiarui, et al. \"Groupvit: Semantic segmentation emerges from text supervision.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n```\n\n### Response to Question 3: \n> Why is this method compatible with the pure transformer-based Bert model, since this method introduces many new modules and modifies attention?\n\n**Objective**: Our primary aim is to address the computational challenges inherent in self-attention within transformer-based models.\n\n**Token Pruning**: This strategy is designed to reduce the overall token count, placing a greater emphasis on the most crucial information. It operates by leveraging the inherent importance of tokens as determined through self-attention.\n\n**Token Combining**: This approach seeks to reduce token size. It achieves this by calculating similarities through a cross-attention operation, which operates between combination tokens and embedded tokens.\n\n**Integration**: We've adapted the self-attention mechanism to incorporate a fuzzy-based token pruning approach and have added a token combining module. Importantly, even after these modifications, the shape of the output remains consistent with that of the input. This ensures that our approach can be seamlessly integrated with existing models.\n\n**Applicability**: The versatility of our method means that it can be incorporated into a wide range of transformer-based models, not just BERT.\n\nIn essence, our modifications, while extensive, are designed to align with the core principles of BERT. This ensures compatibility and serves to enhance its overall efficiency.\n\n\n### Response to Question 4: \n> Is this method restricted to classification tasks but not generation tasks?\n\n**Classification Emphasis**: Our model, which integrates token pruning and token combining, is primarily built upon BERT. While it's designed with flexibility to be applied to various transformer-based models using self-attention, our experiments have been predominantly focused on Natural Language Understanding (NLU) tasks rather than Natural Language Generation (NLG).\n\n**Attention Mechanism Distinction**: Token pruning operates within the self-attention framework. In contrast, our token combining approach diverges from the cross-attention found in the vanilla Transformer's decoder[1]. This intrinsic focus on self-attention has been validated through our document classification experiments.\n\n**Challenges in Generation**: Classification tasks can benefit from emphasizing key tokens. However, the effectiveness of pruning-centric methods in tasks like machine translation or autoregressive models, where capturing word alignment is crucial, remains uncertain.\n\n**Future Work**: We acknowledge the potential of our method for NLG tasks. Our future research will delve into optimizing cross-attention and exploring how our approach can be adapted to enhance performance in generation tasks.\n\n```\n1. Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in neural information processing systems 30 (2017).\n```\n\n\n### Typos Grammar Style and Presentaton Improvements:\n#### 1. **Figure 1 Corrections**:\nThank you for pointing out the discrepancies in Figure 1. We've made the following corrections:\n- Adjusted the descriptions: (left) now represents the fuzzy-based token pruning self-attention, and (right) represents the token combining module.\n- Clarified symbols: $C$ denotes the set of learnable combination tokens, and $HA$ signifies the hard assignment, utilizing a one-hot operation to map each embedded token to a specific combination token. We've updated the figure and its descriptions accordingly.\n\n#### 2. **Clarification on the Definition of $p$**:\nWe acknowledge the potential confusion around the term \"pruning\" associated with $p$. To provide clarity:\n- We've redefined $p$ as the \"preservation ratio\" rather than the \"pruning ratio\".\n- This ratio indicates the proportion of tokens retained post-pruning, relative to the initial token count.\n- Our manuscript has been updated to reflect this change, ensuring a clearer understanding of the concept.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This proposed method consistently improves performance while reducing computing costs.", "reasons_to_reject": "This paper lacks a lot of comparisons in the experiments. First, it should compare with other efficient methods that sparsify attention or prune tokens. Second, this paper does not compare different techniques in document classification. Third, this paper does not compare with other papers that use virtual tokens.", "questions_to_authors": "1.  What will be the wall time for your methods and baselines?\n\n2. What is the initialization of combination tokens?   What is the scale of these tokens? Is it specific to one task or one sentence?\n\n3. Why is this method compatible with the pure transformer-based Bert model, since this method introduces many new modules and modifies attention?\n\n4. Is this method restricted to classification tasks but not generation tasks?", "paper_topic_and_main_contributions": "This paper proposes a compute-efficient and more accurate model for document classification. It applies token pruning methods by filtering unimportant tokens and combining features with virtual tokens. On six text classification datasets, this method demonstrates improvements.\nFuzzy-based token pruning achieves consistent improvements while combination tokens do not.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Figure 1 lacks descriptions, and there may be some mistakes. e.g., token combining is on the right but not the left. We do not know what C and HA means with the figure and descriptions.\n\n2. The definition of p is misleading since you define it as a percentage to maintain but call it a pruning ratio."}}, {"review_id": "yGxsi7ddSs", "reviewer": "Reviewer_HEpY", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback on our manuscript. We genuinely appreciate your recognition of the significance of our efficient models, which aim to reduce computational costs while enhancing performance. In response to your comments and suggestions:\n\n### Revised Manuscript\nYour feedback highlighted the need for a clearer exposition of our fuzzy-based pruning method. We acknowledge that the constraints of the paper's length led us to provide a concise summary of the fuzzy-based pruning method, which may have caused some confusion. We deeply regret this oversight and aim to rectify it by providing a more detailed explanation.\n\nIn the revised manuscript, particularly in lines [377–381], we have expanded our description as follows:\n\n\n> Unlike the previous work[11] that uses fixed constants as hyperparameters, our approach adopts the quantile functions $Q_{S(t)}(0.25)$ and $Q_{S(t)}(0.75)$ for $a$ and $b$, respectively, to ensure robustness. We compute a quantile function for all importance scores, capturing the complete spectrum of head information. The importance set $I$ and the unimportance set $U$ are defined using the $\\alpha-cut$, commonly referred to as $^{\\alpha}A={x|A(x) \\geq \\alpha}$ in fuzzy theory. To mitigate information loss due to imbalanced distribution, we employ token pruning based on the pruning ratio $p$ for tokens that fall within the set $(I-U)^c$. In the initial layers, where attention might not be adequately trained, there's a risk of erroneously removing crucial tokens. To counteract this, we've set the $\\alpha$ for $I$ to a minimal value of 0.01, while the $\\alpha$ for $U$ is empirically set to 0.9.\n\nWe hope this elaboration offers a clearer understanding of our methodology and addresses the concerns you raised.\n\n### Response to Weakness 1:\n\n> The method section for fuzzy-based pruning is quite unclear. First, I think the fuzzy membership functions in Eq.6 and Eq.7 are incorrect certainly. Why they are discontinuous functions? e.g. for Eq.6 $lim_{s \\to a^-}f(s)=1$ and $lim_{s\\to a^+}f(s)=0$. I suggest the authors double-check them. Next, it is not explained why introducing these fuzzy membership functions can better perceive uncertainty. Why is it better than pruning with the importance score $S(e)$ directly? No technical and experimental discussions are included.\n\nWe genuinely appreciate your detailed feedback on our fuzzy-based pruning method. We acknowledge the oversight in our presentation of the fuzzy membership functions, and we're grateful for your keen observation.\n\n- **Correction of Equations:** We identified an error in our representation of Eq. 6 and Eq. 7. The corrected equations are:\n\n\\\\[\nImportance(S)= \n\\begin{cases} \n0 & {\\mbox{if } S(e)\\le a, }  \\\\\n\\frac{S(e)-a}{b-a} & {\\mbox{if }  a< S(e)<b, }  \\\\\n1 & {\\mbox{if } S(e)\\geq b}  \n\\end{cases} \\tag{6}\n\\\\]\n\n\\\\[\nUnimportance(S)= \n\\begin{cases}\n1 & \\mbox{if } S(e)\\le a, \\\\\n\\frac{b-S(e)}{b-a} & \\mbox{if } a< S(e)<b, \\\\\n0 & \\mbox{if } S(e)\\geq b\n\\end{cases} \\tag{7}\n\\\\]\n\n\nThese equations ensure that the fuzzy membership functions are continuous.\n\n- **Rationale for Fuzzy Logic:** We introduced fuzzy logic to address two primary challenges:\n\n  1. **Unbalanced Data Distribution:** The direct pruning method, based on the importance score $S(e)$, doesn't account for the distribution of these scores across the dataset. This can lead to the inadvertent pruning of significant tokens, especially in data predominantly containing crucial tokens. Fuzzy logic provides a safeguard against such unintended pruning.\n   \n  2. **Inadequate Training:** In the initial layers, where the attention probability, which forms the basis of the importance score $S(e)$, might not be adequately trained, there's a risk of erroneously removing crucial tokens.\n\n- **Design Philosophy:** The design of our fuzzy membership functions, as presented in Eq. 6 and Eq. 7, stems from our approach to token pruning. Since the importance score is our pruning criterion, we tailored the membership functions to be proportional to this score. Additionally, to prevent mis-pruning due to insufficiently trained importance scores in the early layers, we ensure that scores at both extremes have an equal probability of belonging to important and unimportant set.\n\nWe hope this provides a clearer understanding of our methodology and addresses your concerns.\n\n\n### Response to Weakness 2:\n>The fuzzy logic is not changed after adding one more membership function. Because the importance and unimportance functions are symmetric w.r.t $S=\\frac{a+b}{2}$ and $y=\\frac{1}{2}$ by using the same $a$ and $b$\n for two functions. When one function fails to distinguish the importance of two tokens, another function can't tell the unimportance difference either. More specifically, if two tokens have the same Unimportance membership value, they will have the same Importance membership value as well. What is the benefit to have two membership functions?\n \nYour observation regarding the symmetry of our fuzzy membership functions is astute. We understand the concerns raised and would like to clarify the rationale behind our design choice.\n\n1. **Contradictory Relationship:** By nature, \"importance\" and \"unimportance\" are antonyms. This inherent contradictory relationship is the reason we designed symmetrical membership functions. By doing so, we can compute the probability of a token belonging to either the important or unimportant set based solely on its importance score without the need for additional calculations.\n\n2. **Distinct Roles of Membership Functions:** While the membership functions are symmetrical, they serve distinct purposes in our methodology. We define the importance set $I$ and the unimportance set $U$ using different $\\alpha$ values in the $\\alpha-cut$. The importance set $I$ is designed to retain crucial information. By subsequently filtering through the unimportance set $U$, we secure a preservation area $(I-U)$. This dual-set approach ensures that our pruning mechanism is stable and minimizes the risk of inadvertently removing essential information.\n\nWe hope this explanation clarifies our approach and addresses your concerns.\n \n### Response to Weakness 3:\n>Besides, the description of lines 380-381 is indefinite. The authors mentioned pruning tokens in the unimportant set and retaining tokens in the important set. But how these two sets are constructed is not clear. Are they completely disjoint? If not, how to handle the overlapped tokens for pruning?\n \nThank you for pointing out the ambiguity in our description. Let me clarify the construction and potential overlap of the \"important\" and \"unimportant\" sets.\n\n- **Set Construction**: The construction of the \"important\" and \"unimportant\" sets is based on fuzzy membership functions. Using the $\\alpha-cut$, each set is defined based on their respective $\\alpha$ values. \n- **Overlap Handling**: To delete tokens carefully, tokens that fall into the overlap$(I \\cap U)$ are retained in the pruning set and are exempt from the priority of preserving. This ensures that the preserving set remains disjoint from the pruning set during the pruning process.\n- **Pruning Decision**: Tokens prior to being pruned are those that belong to the pruning set $(I-U)^c$, ensuring that we retain crucial information while eliminating less significant tokens.\n\nWe hope this clarification provides a clearer understanding of our methodology. We will also make sure to elucidate this process in our revised manuscript for better clarity.\n\n### Response to Weakness 4:\n> Although their method shows improved performance and higher efficiency compared to vanilla BERT, there is no technical or experimental comparison with other token pruning methods.\n\nThank you for your constructive feedback. We recognize the importance of comprehensive experimental comparisons and have taken steps to address this:\n\n**Baselines**:\n- **BigBird[1]**: An efficient model employing a sparse attention mechanism. It reduces the quadratic dependency of full attention, ensuring memory efficiency.\n- **Longformer[2]**: Known for its efficacy in document classification tasks, it uses a sparse attention pattern to handle long sequences.\n- **Funnel-Transformer[3]**: Efficiently filters out sequential redundancy by progressively compressing the sequence's hidden state.\n- **Transkimmer[4]**: Utilizes dynamic token pruning, outperforming several other pruning-based models.\n\nWe believe that these baselines provide a comprehensive backdrop against which our proposed method's performance can be evaluated.\n\n\n**Performance Comparison on Document Classification:**\n\n|                        | **SST-2**       | **IMDB**        | **BBC News**    | **20NewsGroups** | **arXiv**       | **LEDGAR**      |\n|------------------------|-----------------|-----------------|-----------------|------------------|-----------------|-----------------|\n| **BigBird**            | 91.8 / 91.8     | 92.8 / 92.8     | 97.1 / 97.1     | 69.3 / 67.7      | 74.0 / 70.4     | 86.9 / 78.8     |\n| **Longformer**         | 91.0 / 91.0     | 93.4 / 93.4     | 97.9 / 97.9     | 68.5 / 67.0      | 66.0 / 64.8     | 84.0 / 82.1     |\n| **Funnel-Transformer** | 92.0 / 92.0     | 91.7 / 91.7     | 96.5 / 96.5     | 69.7 / 68.5      | 70.0 / 66.5     | 86.7 / 78.4     |\n| **Transkimmer**        | 91.4 / 91.4     | 93.7 / 93.7     | 97.6 / 97.6     | 67.8 / 64.9      | 73.7 / 72.6     | 87.1 / 77.3     |\n| **BERT**      | 90.7 / 89.4     | 93.7 / 92.8     | 96.2 / 94.0     | 68.7 / 56.1      | 69.0 / 52.5     | 86.2 / 77.4     |\n| **ours-P**             | 91.1 / 89.8     | 93.8 / 92.9     | 97.2 / 95.4     | 69.4 / 56.8      | 71.0 / 56.6     | 86.5 / 77.9     |\n| **our-PF**             | 91.4 / 90.1     | **94.3 / 93.4** | 97.9 / 96.6     | **70.7 / 58.2**  | **76.0 / 61.0** | 86.8 / 78.4     |\n| **ours-C**             | 91.8 / 90.8     | 92.6 / 91.3     | 96.8 / 95.2     | 69.2 / 56.5      | 70.0 / 52.7     | 86.9 / 78.4     |\n| **ours-PFC (L11)**     | **92.1 / 91.0** | 93.5 / 92.5     | **98.1 / 97.1** | 69.9 / 57.1      | 74.0 / 58.1     | **87.3 / 79.2** |\n| **ours-PFC (L7)**      | 90.1 / 88.6     | 92.8 / 91.5     | 97.0 / 95.2     | 68.5 / 55.4      | 68.0 / 50.7     | 85.7 / 76.8     |\n\n**Efficiency Comparison on Document Classification:**\n\n| Model              | FLOPs(↓)   |Memory Cost(↓)|  SpeedUp(↑) |\n| --------           | -------- | -----     | --------  |\n| **BERT**      |     (1.0x)    | (1.0x)         |  (1.0x)        |\n| **ours-PFC (L11)**     | 0.88x    | 0.80x     | 1.29x     |\n| **ours-PFC (L7)**      | 0.54x    | 0.61x     | 1.64x     |\n| **BigBird**            | 1.57x    | 0.82x     | 0.94x     |\n| **Longformer**         | 2.10x    | 1.11x     | 0.55x     |\n| **Funnel-Transformer** | 1.03x    | 0.39x     | 1.11x     |\n| **Transkimmer**        | 0.13x    | 0.87x     | 0.70x     |\n\n\n**Key Insights:**\n1. We've integrated results from the four mentioned baselines into our experimental section.\n2. Our \"ours-PFC\" model consistently showcases superior performance and efficiency across datasets.\n3. Our model demands fewer computational resources and memory while maintaining excellent performance, emphasizing its efficacy in document classification.\n\nBy integrating these analyses into our revised manuscript, our intention is to provide readers with insights into the practical implications and viability of implementing our methods in real-world scenarios.\n\n```\n1. Zaheer, Manzil, et al. \"Big bird: Transformers for longer sequences.\" Advances in neural information processing systems 33 (2020): 17283-17297\n2. Beltagy, Iz, Matthew E. Peters, and Arman Cohan. \"Longformer: The long-document transformer.\" arXiv preprint arXiv:2004.05150 (2020).\n3. Dai, Zihang, et al. \"Funnel-transformer: Filtering out sequential redundancy for efficient language processing.\" Advances in neural information processing systems 33 (2020): 4271-4282.\n4. Guan, Yue, et al. \"Transkimmer: Transformer Learns to Layer-wise Skim.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Their experiments demonstrated improvement on several classification tasks while reducing both time and memory costs compared to the vanilla BERT model.", "reasons_to_reject": "1. The method section for fuzzy-based pruning is quite unclear. First, I think the fuzzy membership functions in Eq.6 and Eq.7 are incorrect certainly. Why they are discontinuous functions? e.g. for Eq.6 $ lim_{s\\to a^-}f(s)=1$ and $lim_{s\\to a^+}f(s)=0$. I suggest the authors double-check them. Next, it is not explained why introducing these fuzzy membership functions can better perceive uncertainty.  Why is it better than pruning with the importance score $S(e)$ directly? No technical and experimental discussions are included.\n\n2. The fuzzy logic is not changed after adding one more membership function. Because the importance and unimportance functions are symmetric w.r.t $S=\\frac{a+b}{2}$ and $y = \\frac{1}{2}$ by using the same $a$ and $b$ for two functions. When one function fails to distinguish the importance of two tokens, another function can't tell the unimportance difference either. More specifically, if two tokens have the same Unimportance membership value, they will have the same Importance membership value as well.   What is the benefit to have two membership functions?\n\n3. Besides, the description of lines 380-381 is indefinite. The authors mentioned pruning tokens in the unimportant set and retaining tokens in the important set. But how these two sets are constructed is not clear. Are they completely disjoint? If not, how to handle the overlapped tokens for pruning? \n\n4. Although their method shows improved performance and higher efficiency compared to vanilla BERT, there is no technical or experimental comparison with other token pruning methods.\n ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes to improve the performance and speed of self-attention by leveraging two techniques: token pruning and token combining. They utilize fuzzy logic to handle the uncertainty issues of token importance and develop a token combing module to aggregate sequence information. They applied their method to BERT models and conducted experiments on various datasets.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PGmhvgO5od", "reviewer": "Reviewer_fmo5", "summary": "", "questions": "- Could you explain why fuzzy-based method work for pruning and give more analysis for how it works?\n- How to decide whether a token belongs to unimportant set? And how do you set the pruning ratio $p$? Does the choice of $a$ and $b$ in equ. 6 and 7 impact the performance?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and valuable feedback on our manuscript. We recognize the importance of addressing the concerns you raised, especially regarding the fuzzy-based pruning method and the experimental comparison. Here's our response to your comments:\n\n### Revised Manuscript\nYour feedback highlighted the need for a clearer exposition of our fuzzy-based pruning method. We acknowledge that the constraints of the paper’s length led us to provide a concise summary of the fuzzy-based pruning method, which may have caused some confusion. We deeply regret this oversight and aim to rectify it by providing a more detailed explanation.\n\nIn the revised manuscript, particularly in lines [377–381], we have expanded our description as follows:\n\n> Unlike the previous work[11] that uses fixed constants as hyperparameters, our approach adopts the quantile functions $Q_{S(t)}(0.25)$ and $Q_{S(t)}(0.75)$ for $a$ and $b$, respectively, to ensure robustness. We compute a quantile function for all importance scores, capturing the complete spectrum of head information. The importance set $I$ and the unimportance set $U$ are defined using the $\\alpha-cut$, commonly referred to as $^{\\alpha}A={x|A(x) \\geq \\alpha}$ in fuzzy theory. To mitigate information loss due to imbalanced distribution, we employ token pruning based on the pruning ratio $p$ for tokens that fall within the set $(I-U)^c$. In the initial layers, where attention might not be adequately trained, there's a risk of erroneously removing crucial tokens. To counteract this, we've set the $\\alpha$ for $I$ to a minimal value of 0.01, while the $\\alpha$ for $U$ is empirically set to 0.9.\n\nWe hope this expanded explanation provides a clearer understanding of our methodology.\n\n### Response to Weakness 1:\n> The contribution is a combination of current methods from computer vision and incremental.\n\nThank you for pointing out the perceived incremental nature of our work. We understand your concerns and would like to clarify the unique contributions of our approach, especially in the context of NLP.\n\n**Contribution Clarification**\n\nWhile it's true that our token combining module draws inspiration from grouping methods in computer vision, the application and challenges in the NLP domain are distinct. Our primary motivation is to address the computational and performance issues associated with self-attention rather than merely introduce a novel combining methodology.\n\n1. **Fuzzy Logic in Token Pruning**: We've introduced fuzzy logic to enhance the existing token pruning strategies, which traditionally relied on straightforward attention scores. This addition is pivotal in ensuring that the pruning process is more adaptive and less prone to errors, especially in complex linguistic contexts.\n\n2. **Token Combining in NLP**: The token combining strategy we've employed is significant in its own right. While similar methodologies might be prevalent in computer vision tasks like object detection or segmentation, their application to text classification in NLP is novel. We've successfully demonstrated its effectiveness in this new domain.\n\n3. **Synergy between Pruning and Combining**: One of the standout features of our approach is the synergistic relationship between token pruning and combining. When integrated, these techniques lead to a model that's not only lightweight but also exhibits enhanced classification performance. Our experiments further validate this, showing marked improvements even when only the token combining module is applied.\n\nIn essence, our work transcends the boundaries of methods traditionally confined to computer vision, showcasing their potential in NLP. This not only underscores the versatility of these techniques but also paves the way for future research in this direction.\n\n### Response to Weakness 2:\n> The illustration in fig. 1 is confusing, the symbol definition is different from what the authors use in the text.\n\nThank you for your feedback. We apologize for any confusion that our mistake may have caused. In the descriptions of Figure 1, (left) should be a  fuzzy-based token pruning self-attention, and (right) should be a token combining module. We have updated our descriptions in Figure 1 in our manuscripts.\n\n### Response to Weakness 3:\n> The motivation for fuzzy-based token pruning is not clear. Even in imbalanced distributions, discarding tokens according to the importance score will not discard important tokens while retaining unimportant tokens. Given that tokens of lesser significance inherently possess a lower ranking compared to their more important counterparts, the pruning process inherently eliminates tokens of smaller importance. So why introducing uncertainty in pruning? More uncertainty are more likely to drop important tokens.\n\n**Motivation for Fuzzy-Based Token Pruning**\nWe introduced fuzzy-based token pruning to address specific challenges that arise from the direct pruning method based on the importance score $S(e)$. Here are our primary motivations:\n\n1. **Distribution of Importance Scores**: Direct pruning based on importance scores doesn't always account for the distribution of these scores across the dataset. In datasets dominated by crucial tokens, there's a higher likelihood of inadvertently pruning significant tokens. Our fuzzy-based approach mitigates this risk by considering the distribution of importance scores.\n\n2. **Insufficient Learning in Shallow Layers**: In the initial layers of the model, the importance score $S(e)$ may not be adequately trained. This can lead to uncertainty in the importance scores, increasing the risk of pruning crucial tokens. Our fuzzy-based method addresses this challenge by introducing a more nuanced pruning strategy.\n\n**Addressing Uncertainty**\nOur approach doesn't aim to introduce uncertainty but rather to manage and mitigate it. We recognize that tokens with higher importance scores should be prioritized for preservation. However, due to the reasons mentioned above, there can be inherent uncertainties in the pruning process. By employing two fuzzy membership functions, we define the importance and unimportance sets for all representations. These sets act as filters, helping us identify the tokens that are most likely to be unimportant. Through this method, we ensure that vital information is retained, and our experiments validate the effectiveness of our fuzzy-based token pruning strategy.\n\n### Response to Weakness 4:\n> The fuzzy-based approach appears to limit the ability to distinguish between tokens characterized by high and low values. The alignment of the membership function with the concept of uncertainty remains unclear.\n\nWe acknowledge that our fuzzy membership functions, as defined in Eq. 6 and Eq. 7, may give the impression of limiting the ability to distinguish between tokens with high and low importance scores. However, this design is intentional and aims to address specific challenges:\n\n1. **Managing Uncertainty**: As elaborated in \"Response to Weekness 3,\" our fuzzy-based approach is designed to manage the uncertainties that arise from imbalanced data and insufficiently trained importance scores in shallow layers.\n\n2. **Reliable Extremes**: Our fuzzy membership functions assign a probability value of $0$ or $1$ to extreme importance scores. This is based on the assumption that extreme scores are more reliable indicators of a token's importance or unimportance.\n\n3. **Selective Pruning**: After defining the importance and unimportance sets using $\\alpha-cut$, we apply pruning preferentially to the set deemed least important. This ensures that crucial information is retained while less important tokens are pruned.\n\nBy doing so, our fuzzy-based approach provides a nuanced way of handling the complexities and uncertainties involved in token pruning, thereby improving the overall performance of the model.\n\n### Response to Weekness 5:\n>The experiment comparison is weak, the author only compare their method to the BERT-baseline. The author should compare their method to token pruning and token combination baselines.\n\nThank you for emphasizing the need for a broader experimental comparison. We've taken steps to address this:\n\n**Baselines**:\n- **Token Pruning**: We've introduced models like BigBird[1] and Longformer[2] which utilize sparse attention. Additionally, Funnel-Transformer[3] and Transkimmer[4] serve as examples of token compression and token pruning, respectively.\n  \n- **Token Combination**: Our approach to token combinations using virtual tokens is novel. After extensive research, we found our paper to be a pioneering effort in this domain, especially for document classification tasks.\n\n**Performance Comparison on Document Classification:**\n|                        | **SST-2**       | **IMDB**        | **BBC News**    | **20NewsGroups** | **arXiv**       | **LEDGAR**      |\n|------------------------|-----------------|-----------------|-----------------|------------------|-----------------|-----------------|\n| **BigBird**            | 91.8 / 91.8     | 92.8 / 92.8     | 97.1 / 97.1     | 69.3 / 67.7      | 74.0 / 70.4     | 86.9 / 78.8     |\n| **Longformer**         | 91.0 / 91.0     | 93.4 / 93.4     | 97.9 / 97.9     | 68.5 / 67.0      | 66.0 / 64.8     | 84.0 / 82.1     |\n| **Funnel-Transformer** | 92.0 / 92.0     | 91.7 / 91.7     | 96.5 / 96.5     | 69.7 / 68.5      | 70.0 / 66.5     | 86.7 / 78.4     |\n| **Transkimmer**        | 91.4 / 91.4     | 93.7 / 93.7     | 97.6 / 97.6     | 67.8 / 64.9      | 73.7 / 72.6     | 87.1 / 77.3     |\n| **BERT**      | 90.7 / 89.4     | 93.7 / 92.8     | 96.2 / 94.0     | 68.7 / 56.1      | 69.0 / 52.5     | 86.2 / 77.4     |\n| **ours-P**             | 91.1 / 89.8     | 93.8 / 92.9     | 97.2 / 95.4     | 69.4 / 56.8      | 71.0 / 56.6     | 86.5 / 77.9     |\n| **our-PF**             | 91.4 / 90.1     | **94.3 / 93.4** | 97.9 / 96.6     | **70.7 / 58.2**  | **76.0 / 61.0** | 86.8 / 78.4     |\n| **ours-C**             | 91.8 / 90.8     | 92.6 / 91.3     | 96.8 / 95.2     | 69.2 / 56.5      | 70.0 / 52.7     | 86.9 / 78.4     |\n| **ours-PFC (L11)**     | **92.1 / 91.0** | 93.5 / 92.5     | **98.1 / 97.1** | 69.9 / 57.1      | 74.0 / 58.1     | **87.3 / 79.2** |\n| **ours-PFC (L7)**      | 90.1 / 88.6     | 92.8 / 91.5     | 97.0 / 95.2     | 68.5 / 55.4      | 68.0 / 50.7     | 85.7 / 76.8     |\n\n**Efficiency Comparison on Document Classification:**\n\n| Model              | FLOPs(↓)   |Memory Cost(↓)|  SpeedUp(↑) |\n| --------           | -------- | -----     | --------  |\n| **BERT**      |    (1.0x)    | (1.0x)         |  (1.0x)        |\n| **ours-PFC (L11)**     | 0.88x    | 0.80x     | 1.29x     |\n| **ours-PFC (L7)**      | 0.54x    | 0.61x     | 1.64x     |\n| **BigBird**            | 1.57x    | 0.82x     | 0.94x     |\n| **Longformer**         | 2.10x    | 1.11x     | 0.55x     |\n| **Funnel-Transformer** | 1.03x    | 0.39x     | 1.11x     |\n| **Transkimmer**        | 0.13x    | 0.87x     | 0.70x     |\n\n\n**Key Insights:**\n1. We've integrated results from the four mentioned baselines into our experimental section.\n2. Our \"ours-PFC\" model consistently showcases superior performance and efficiency across datasets.\n3. Our model demands fewer computational resources and memory while maintaining excellent performance, emphasizing its efficacy in document classification.\n\nBy integrating these analyses into our revised manuscript, our intention is to provide readers with insights into the practical implications and viability of implementing our methods in real-world scenarios.\n\n```\n1. Zaheer, Manzil, et al. \"Big bird: Transformers for longer sequences.\" Advances in neural information processing systems 33 (2020): 17283-17297\n2. Beltagy, Iz, Matthew E. Peters, and Arman Cohan. \"Longformer: The long-document transformer.\" arXiv preprint arXiv:2004.05150 (2020).\n3. Dai, Zihang, et al. \"Funnel-transformer: Filtering out sequential redundancy for efficient language processing.\" Advances in neural information processing systems 33 (2020): 4271-4282.\n4. Guan, Yue, et al. \"Transkimmer: Transformer Learns to Layer-wise Skim.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n```\n\n\n### Response to Question 1:\n> Could you explain why fuzzy-based method work for pruning and give more analysis for how it works?\n\nOur fuzzy-based pruning method is designed to address specific challenges and uncertainties associated with token pruning:\n\n1. **Distribution Consideration**: We take into account the distribution of all information across each multi-head. By defining the importance and unimportance sets through fuzzy membership functions, we can prioritize pruning targets for each head.\n\n2. **Mitigating Mis-pruning**: Fuzzy logic is employed to prevent the inadvertent pruning of crucial tokens, especially when the importance score, based on attention probability, exhibits a left-skewed distribution. By mapping each representation to a probability of belonging to a specific set using fuzzy memberships, we can define importance more accurately. This approach ensures that pruning decisions are made more judiciously.\n\n3. **Flexible Thresholds**: Instead of using fixed constants, our method sets the thresholds $a$ and $b$ dynamically based on quantiles. This flexibility allows our method to adapt to different data distributions and enhances the robustness of the pruning process.\n\n4. **Addressing Insufficient Learning**: In the initial layers, the importance scores might not be adequately trained, leading to potential uncertainties. Our fuzzy membership functions, as defined in Eq. 6 and Eq. 7, are designed to handle such uncertainties, ensuring that tokens are pruned based on more reliable criteria.\n\n5. **Holistic Attention Consideration**: By applying fuzzy logic across the entire multi-head attention mechanism, our method captures attention information from diverse perspectives, ensuring a comprehensive understanding of token importance.\n\nOur experiments have shown that the fuzzy-based token pruning method outperforms traditional pruning methods, especially in scenarios where the importance scores might not be entirely reliable. This indicates that our approach effectively balances performance improvement with model efficiency.\n\n### Response to Question 2:\n> How to decide whether a token belongs to unimportant set? And how do you set the pruning ratio $p$? Does the choice of $a$ and $b$ in equ. 6 and 7 impact the performance?\n\n**Decision about Unimportant Tokens**\n\nTwo fuzzy membership functions obtain two probabilities that each representation from the attention operation belongs to the importance and unimportance sets, respectively. Afterwards, we define two sets by applying $\\alpha-cut$ with two empirically selected $\\alpha$. Through this process, we derive two sets, considering the distribution of data. We hope that the motivation, role, and process of fuzzy logic have been fully explained through our above responses to your feedback and questions.\n\n**Setting of Pruning Ratio**\n\nWe empirically determined the pruning ratio $p$ for the datasets used in our experiments. As shown in Table 4 of our paper, our proposed model shows the best document classification performance when $p$ was 0.9. Moreover, we note that the pruning ratio $p$ is a hyperparameter, allowing the user to determine an appropriate pruning ratio. Memory saving and SpeedUp enhance as $p$ decreases. Nevertheless, if $p$ is too small and excessive tokens are removed, important information is lost and the performance in classification deteriorates. Since there is a trade-off between $p$ and accuracy, users are free to  select an appropriate pruning ratio $p$ according to their hardware environment and purpose.\n\n**Choice of $a$ and $b$ in Eq.6 and 7**\n\nWe agree that the decisions of $a$ and $b$ have a significant impact on the fuzzy logic for preserving (token pruning). Therefore, simply setting $a$ and $b$ fixed values may not prevent the loss of important information. We experimentally confirmed that the representations obtained from multi-head attention consisted of pretty tiny values and varied depending on data and layer. To consider the distribution of entire representations, we set $a$ and $b$ to the quantile of each layer rather than fixing the values to 0.2 and 0.8, etc. as in the existing method[1].\n\n```\n1. Zhao, Wei Bin, Yue Li, and Lin Shang. \"Fuzzy pruning for compression of convolutional neural networks.\" 2019 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). IEEE, 2019.\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Efficient models are a important direction.\n- The experimental results are both effective and convincing, as they demonstrate the proposed methods' ability to achieve memory savings and speedup without compromising performance.", "reasons_to_reject": "- The contribution is a combination of current methods from computer vision and incremental.  \n- The illustration in fig. 1 is confusing, the symbol definition is different from what the authors use in the text. \n-  The motivation for fuzzy-based token pruning is not clear. Even in imbalanced distributions, discarding tokens according to the importance score will not discard important tokens while retaining unimportant tokens. Given that tokens of lesser significance inherently possess a lower ranking compared to their more important counterparts, the pruning process inherently eliminates tokens of smaller importance. So why introducing uncertainty in pruning? More uncertainty are more likely to drop important tokens.\n- The fuzzy-based approach appears to limit the ability to distinguish between tokens characterized by high and low values. The alignment of the membership function with the concept of uncertainty remains unclear.\n- The experiment comparison is weak, the author only compare their method to the BERT-baseline. The author should compare their method to token pruning and token combination baselines.", "questions_to_authors": "- Could you explain why fuzzy-based method work for pruning and give more analysis for how it works?\n- How to decide whether a token belongs to unimportant set? And how do you set the pruning ratio $p$? Does the choice of $a$ and $b$ in equ. 6 and 7 impact the performance?", "paper_topic_and_main_contributions": "The paper aims at address the high cost and performance degradation of BERT models. The authors integrating token pruning and token combining. Token pruning prunes less important tokens and token combing further condense the inputs. Extensive experiments shows the effectiveness of the proposed methods with memory cost savings and speedup.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for more efficient Transformer architectures by employing token pruning and token combining.  It uses a token importance function, but rather than rely directly on it, uses fuzzy set membership to help determine a token's relative importance in relation to the distribution of importance scores.

Like other reviewers, I had trouble understanding the motivation for the fuzzy logic methods in this approach.  The explanations offered in the rebuttal were somewhat helpful in this regard, but they were not properly reflected in the paper itself, meaning the lack of clarity is a real issue.

The main issue seems to be the inability to use the importance score directly.  Fuzzy logic is introduced to compensate for a weakness in the importance score, but it would have been nice to see a comparison with other methods that compute this score differently, thereby obviating the need for a mitigation in the first place.  This leads into the valid criticism that more comparison in the results with other sparse attention methods would have better contextualized this work and shown *its* importance.
Decision: Accept-Findings