Attribution-Driven Adaptive Token Pruning for Transformers
Authors: Yaoyao Yan
Hui Yu
Weizhi Xu
Venue: NeurIPS 2025 poster ()
Keywords: Token Pruning
Attribution-Driven
Adaptive Inference
Integrated Gradients.
Primary area: deep_learning

Abstract: Transformers have been widely adopted in natural language processing, computer vision, and other domains due to their exceptional performance across a variety of tasks. However, the computational cost of Transformers is prohibitively high, particularly when handling long input sequences, significantly increasing both training and inference time. Although various token pruning methods have been proposed to reduce the computational burden of Transformers, most approaches overlook critical differences in sequences in terms of length and complexity, leading to suboptimal compression efficiency.

In this paper, we propose AD-TP, an Attribution-Driven Adaptive Token Pruning method designed to retain only the most informative tokens. We analyze the performance of using accumulated attention values to measure token importance and find that attention values do not accurately reflect the actual contribution of each token to text understanding. Additionally, we observe significant variations in the length and complexity of different sequences within the dataset. Based on these insights, we adopt Integrated Gradients to evaluate token importance and introduce a lightweight adaptive token retainer module that dynamically generates pruning configurations for each input sequence. In addition, we incorporate both teacher supervision and self-supervised learning objectives to enhance the training efficiency, accuracy, and robustness of the model.

Experiments conducted on GLUE, SQuAD, and 20News demonstrate that AD-TP outperforms state-of-the-art token pruning and model compression methods in both accuracy and computational efficiency. On GLUE, AD-TP reduces FLOPs by an average of 7.8× while improving performance by 0.6%.

AtriibutinnDrriven Adaptive Token Pruning for Transformers
Yaoyao Yan'
Hui Yu?
Weizhi Xu
School of Information Science and Engineering, Shandong Normal University
Business School, Shandong Normal University
yanyydstu. wdw..dduuud..... edu.cn huiyu01170sdnu, edu. xuveizhi@sdnu, edu. cn
Abstract
Transformers have been widely adopted in natural language processing, computer vision, and other domains due to their exceptional performance across variety tasks. However, the computational cost of Transformers is prohibitively high, particularly when handling long input sequences, significantly increasing both training and inference time. Although various token pruning methods have been proposed reduce the computational burden of Transformers, most approaches overlook critical differences in sequences terms of length and complexity, leading suboptimal compression efficiency.
In this paper, we propose AD-TI AD-TP, an Attribution- Drivee Adaptive Token Pruning method designed to retain only the most informative tokens. We analyze the performance of using accumulated attention values measure token importance and find that attention values do not accurately reflect the actual contribution of each token to text understanding. Additionally, we observe significant variations in the length and complexity of differentt sequences within the dataset. Based on these insights, we adopt Integrated Gradients to evaluate token importance and introduce lightweight adaptive token retainer module that dynamically generates pruning configurations for each input sequence. addition, we incorporate both teacher supervision and self celf-super learning objectives to enhance the training efficiency, accuracy, and robustness of the model.
Experiments conducted on GLUE, SQuAD, and 20News demonstrate that AD-TP outperforms state-of-the-an token pruning and model compression methods in both accuracy and computational efficiency. On GLUE, AD-TP reduces FLOPs by an average of 7.8x while improving performance by 0.6%.
Introduction
Transformers such GPT and BERT are central Natural Language Processing (NLP) due to their strong language modeling capabilities [ However, their self-attention mechanism scales quadratically with sequence length, resulting in high computational cost and latency. For instance, processing 512 tokens requires over 1.5 billion Floating-Poin Operations (FLOPs), limiting real-time or edge deployment. These challenges have motivated increasing interest in model compression techniques to enhance efficiency without sacrificing performance.
To address the computational bottlenecks inherent in large-scale Transformer models, researchers have explored several major compression techniques, including pruning (355), quantization (6-8)
"Corresponding author: Weizhi Xu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Among these, token pruning has received increasing attention due to its hardware- -agaosti design and its ability to adaptively retain informative tokens based on sequence content [12
low-rank decomposition [9, and knowledge distillation I0 11
Recent efforts have applied token pruning to compress Transformer models and accelerate inference. Early work such as HOWER-BERY [3] adopted layer-wise embedding pruning and achieved a 4.5x speedup, but required retraining under multiple constraints. LAT [14] addressed this by using LengthDrop to generate sub-models via evolutionary search, yet it pruned all sequences to a fixed length, leading to under-prume pruningg or over-pre vver-prnning depending on the sequence. Subsequent methods like SpAtten [15] and TR-BERT [16] introduced length-ave aware or content- strategies, but either lacked semantic adaptability or incurred high training costs. LTP (17) further improved adaptiveness by learning per-layer thresholds to prune unimportant tokens, but still applied fixed thresholds during inference.
Despite these advances, two key challenges remain: (i) most approaches rely on attention scores to assess token importance, which may not accurately reflect true contribution; and (ii) many pruning strategies adopt static configurations, ignoring variations in sequence complexity. Addressing these limitations requires more accurate importance estimation method and a mechanism that dynamically adjusts pruning decisions based on each sequence.
To address the above limitations, we propose AD-TP, Attribution-Drive Adaptive Token Pruning method in this work. We analyze the effectiveness cumulative attention scores in evaluating token importance, and find that attention-based measures fail to reliably identify the tokens that should be preserved during pruning. To overcome this, we adopt an attribution method based on Integrated Gradients (IG), which more accurately captures which tokens truly drive the predictions of the model. Moreover, we observe substantial variation in the complexity of different sequences. To accommodate this, we design an adaptive token retainer module, which integrates an Adaptive Retention Ratio Predictor (ARP) and a Token Saliency Predictor (TSP) to dynamically determine the token retention ratio for each sequence.
Specifically, this work makes the following major contributions:
We propose AD-TP, which introduces a lightweight adaptive token retainer that dynamically determines the retention ratio based on sequence complexity, thereby significantly reducing computational cost.
We introduce novel attribution-based token importance estimation approach, which lever- ages IG to quantify the relationship between model predictions and sequence features, enabling a more accurate assessment of the contribution of each token to the output.
We design knowledge distillation framework with dual normalization align attribution features between teacher and student models, and incorporate self-supervision to enhance token-level reasoning in the student model.
Through comprehensive experimental evaluations, AD-TP achieves average FLOPs reduc- tions of 7.02x and 7.37x the GLUE benchmark using 6-layer and 12-layer Transformers, respectively, without sacrificing accuracy. Furthermore, AD-TP also demonstrates superior performance long-sequence tasks.
2 Related Work
Token Pruning Existing token pruning methods can be broadly classified into two categories based on their retention strategies. The first type relies on attention- scoring, using attention weights to estimate token importance. For instance, SpAtten [15) ranks tokens based on attention scores, while PoWER-BEE BERT [33 and LTP [17] perform sequence pruning based on attention representations using fixed or learnable thresholds. STTABT [8] traces attention backward across layers to assess token contribution, and LAD [9] transfers attention knowledge for dynamic pruning in small models. The second type uses prediction based pruning, introducing predictors before each Transformer layer to estimate token retention. TR-BERT [66 employs reinforcement learning, and Transkimmer [20] uses two-layer MI.P to score tokens. These approaches explicitly model importance and offer better pruning accuracy and flexibility. The working mechanism of Transformer models and the underlying principles of token pruning are detailed in Appendix A.I

---

SST-2
MRPC
70
Abcenton Stntegy
Rtaderr Sersagy
Kandio Shokeg
Retoniion Ratio (55)
Resention Ratio (7)
SST-2 Sequence Length
MRPC Sequence Leugth
Figure Model performance under different token importance estimation strategies and token retention ratios.
Figure 2: Sequence length distribution statistics for training and validation sets.
Attribution Method Based on Integrated Gradients Attribution methods assign model predic- tions to sequence features using saliency metrics like gradients [21]. Recent work applies attribution to analyze attention patterns in Transformers 24 Among these, IG offers stable and continuous estimates by integrating gradients along input paths, and has been widely adopted for interpreting deep models in NLP [25] In aspect-based sentiment analysis. IG effectively identifies tokens relevant to specific aspects [26]. It has also been used with pre-trained models like BERT and RoBERTa to extract features important for classification, thereby improving performance 27. IG further reveals how linguistic constructs such as negation and conjunctions influence predictions text classification tasks (28). To address the discreteness of embedding spaces, Roy et al. (29) introduced the Unified Discretized IG method, enhancing interpretability for large language models such as BERT. The fundamental principles of the IG are detailed in Appendix A.2
3 Motivation
In this section, we analyze the strengths and limitations of attention based token importance esti- mation, examine variations in sequence length and complexity, and summarize the challenges in designing adaptive token pruning methods.
3.1 Impact of Token Importance Estimation Strategies on Model Performance
Token importance estimation is crucial for effective pruning. Most existing methods rely on atten- tion mechanisms to assess token importance; however, such mechanisms focus on ooken-to-toee interactions and tend to overlook the independent semantic contribution of individual tokens [30]. To evaluate the effectiveness different estimation strategies, we compare three approaches on the SST-2 and MRPC datasets: random strategy (as a lower bound), attention strategy, and residual strategy (as an upper bound). As shown in Fig. [ the attention- based approach still leaves substantial room for improvement in pruning performance, indicating discrepancy between attention scores and the actual semantic contribution of tokens. To address this issue, we propose an attribution- method that leverages IG quantify the marginal contribution of each token the model output, enabling more accurate and interpretable token selection.
3.2 Limitations of Fixed and Learnable Thresholds
Most pruning frameworks adopt either fixed or globally learned thresholds, which fail to account for the variability in sequence length (Fig. 2 and complexity (Fig. 3 across and within datasets. Although LTP [17] improves flexibility by learning layer-wise thresholds during training, these thresholds remain static at inference time and lack instance-level adaptability. To overcome this constraint, propose dynamic pruning mechanism that adjusts the pruning ratio for each sequence based on estimated token saliency and structural complexity.
3.3 Challenges
Despite the significant potential of attribution-driver adaptive token pruning methods in improving model efficiency while preserving predictive performance, several challenges remain in practical im- plementation. First, the IG method must remain stable and effective under conditions of sparse token

---

about
issues mosi
adults
have to
face in
marriage and
think (hai NFP
what
beautifully
liked about
the real
issues
tucked between the silly and crude storylina
(h)
Figure 3: Syntactic dependency analysis of sample sequences SST-2.
Inaetbrrer
Solimas
Light
Suutiin in paniee mann
Pe-kin
Noadha
Pasikivt
Neggatii
snn minve
Peikire
Nepalii
Poulhi
Adaaar Fart
Sou
Prelace
Antlodice-Awar
Asappine loken
Betrier
Figure 4: Overview of the AD-TP framework. The red dashed box represents the Attribution-Aware Adaptive Token Retainer, which consists of ARP and TSP.
retention. Second, the adaptive token retainer must be sufficiently lightweight avoid introducing additional inference overhead. Third, given that the training process involves knowledge distillation, token removal inevitably leads information loss and results in representation misalignment between the student and teacher models.
Attribution-rriven Adaptive Token Pruning
In this section, present the AD-TP method in detail. The overall framework is illustrated in Fig. 4 This approach prunes sequences by introducing an Attribution-Aware Adaptive Token Retainer, which consists of ARP and TSP. AD-TP leverages IG assess token saliency and combines teacher supervision with self-supervised to guide pruning. Additionally, an attribution-based distillation strategy employed to enhance model representation and performance.
4.1 Problem Definition
Given an input token sequence x = {æ1 22 D2., Fn), where r; denotes an input token and n is the sequence length. Transformer first maps the token sequence into d-dimensional embedding sequence (2,,e,,..., Cn) through an embedding layer. Each e; represents the embedding of token Z4.
4.2 Attribution-Awar Adaptive Token Retainer
As illustrated Fig. 5 the Atribuutinn Adaptive Token Retainer consists of two main compo- nents: ARP the left estimates the complexity of the sequence and determines the retention ratio pr.

---

while TSP on the right evaluates the importance of each token in the embedding sequence E. Based on the retention ratio, this module dynamically generates sparse representation, enabling adaptive pruning of the sequence.
In the ARP module, the emhedding of the special classi- fication token [CLS], denoted as T IR is first used as global representation of the imput sequence. This global token is then passed through linear layer and an argmax function G to produce one-hot vector that selects pr from candidate list Lp. The process is defined as:
Token Selinn
Hemenee Hata
Prodice
Aepwer
Laye
Tp = linear (xgg) one one_hot G (xp),
Zig = CCCLS);
Pr one_hooe hoo Lp
(2)
Olonl Tolm
Liber
where é [[cus] E I2d denotes the final-layer embedding of the [CLS] token, and G is the argmax activation function, which maps the logits Tp to a one-hot vector indicating the selected retention ratio.
Figure 5: Attribution-Awen Adaptive Token Retainer.
To identify key tokens under the retention ratio Pr. the TSP assigns saliency score S E R" to each token. Each embedding from is passed through a linear layer, ReLU activation, and another linear projection to produce the score:
$ linear2 LU (lineari (E)))
Based on the adaptive retention ratio pN and the token saliency scores S, select the top p. % most important tokens to construct an adaptive sparse sequence for subsequent computation.
4.3 Token Saliency Estimation Based on Integrated Gradients
In AD-TP, IG used supervise both token pruning and knowledge distillation. IG computes attribution scores by integrating gradients along path from a baseline input (e.g., [PAD]) to the actual input, and is formally defined as follows:
F
F'
k m
F2')
1Gig (F*, = (eij
(4)
deij
where m is the number of integration steps, F is the predicted score for class and eg is the j-th feature of the i-th token. The baseline e'y set as the [PAD] embedding. our setup,
To reduce the influence of wwwinfformation dimensions, the teacher model retains only the top-K IG dimensions per token and computes their L.2 norm. To minimize training complexity and avoid noise, the student model retains all dimensions.
ar.c = || TopK ( [IGapprox (Fr,e; Ex)) l|,,
d' = || [GGapprox (Fr.ss Ex)) lla
(5)
where af and o' represent the attribution scores of token with respect to class in the teacher and student models, respectively. F., and Fs,c are the corresponding prediction functions, and E, and F, are the input embeddings.
4.4 Joint Supervision Based on Teacher and Self-Supervision
In this section, describe how train the Attribution-Aware Adaptive Token Retainer based on attribution scores. The training process integrates teacher supervision and self-supervised. We perform weighted fusion of the attribution scores ai from the teacher model and at from the student model to construct a unified supervision signal:
(6) where, epoch the current step and num_epoch the total number of training epochs. In the early stage of training, due to the limited attribution capability of the student model, the supervision primarily relies on the teacher model. As training progresses, the scores of the student model are gradually incorporated to enhance self-supervised learning.
d- = (1 - k)a{,a + ka 2,07
A = min (1.0, epoch spochhnume nnm__poch))
5

---

Top
Neraill pld
Harr' ooler
Figure 6: Dual normalization process of attribution scores.
Since the student model retains fewer tokens after pruning, its attribution scores not directly comparable to those of the teacher model. To address this, we introduce dual normalization strategy to ensure scale alignment and highlight relative importance. As illustrated in Fig. 6 L2 normalization first applied to eliminate inter-dimensional disparities, followed by sequence-level normalization to convert scores into relative weights.
de |lall22 ag
2i=1
To train the ARP, we use the average saliency score of each sample as threshold. Tokens exceeding this value determine the target retention ratio, which then mapped to candidate list index y as the supervision signal.
ag
y= round (1 2 n
0gng
8
1 where, g the number of tokens above the mean saliency score, n represents the total number of tokens in the sequence that participate in the saliency evaluation, and is the size of the candidate ratio list. Adjusting I allows the model to adapt to different computational budgets. The ARP loss is defined as:
LAR = LCF (Np,.) where, LCE the cross-entropy loss, and LARP encourages Tp to match the target index in the candidate list.
Meanwhile, train the TSP module estimate token saliency scores. The fused attribution scores ag are ranked, and the top 5% tokens are labeled as I, while the remaining tokens are labeled as 0 to construct binary supervision signal B.A. The loss function defined as follows:
n CTSP 1 Σ LBCE (s,, BA.)
(10)
where s; denotes the predicted saliency of the i-th token in the sequence, obtained from Equation LECE represents the binary cross-entropy loss function.
4.5
Overall Objective
In this section, we describe the overall training objective of AD-TP. To ensure a smoother overall distillation process, we also apply dual normalization strategy the attribution scores of both the teacher and student models:
a ||at,ell:
dg.
||a
(11)
die
g.d
(12)
li Σ
1,
aie
-==1
asc
To represent attribution globally, per-class vectors afe and ag.c are concatenated across C classes:
A, = 1 A, - 0g
(13)
where denotes the concatenation operation, which aggregates the attribution scores across all categories along a specified dimension. To mitigate pruning- induced loss, an attribution distillation loss minimizes the L2 distance between teacher and student representations:
Lig = A A
(14)
The final training objective combines classification loss, logits-based distillation, attribution loss, and token retainer supervision:
L = (1 - LCE + aCiogit + BLIG + 7 (CaRp + Lrsp)
(15)
Hyperparameters a. 3, and 7 control the contribution of each term.

---

Experiments
Datasets and Evaluation Metries. We evaluate AD-TP on tasks from the GLUE benchmark and extend the evaluation to two long-text datasets: SQuAD v2.0 [1] and 20News 32 Each GLUE task adopts task-specific evaluation metries; SQuAD v2.0 is evaluated using the F1 score, while 20News uses accuracy. In addition, inference efficiency is assessed using FLOPs, hardware-agnostic metric that reflects the computational cost model inference.
Implementation Details. All experiments are implemented in PyTorch with Huggingface Trans- formers on an NVIDIA RTX 3060. The teacher is a 12-layer BERT-base, and the student uses either 6 or 12 layers. We tune learning rates and distillation weights (a, 9, 7) across defined ranges. All hyperparameter configurations and dataset-specific settings are provided Appendix A.3
Existing Methods for Comparison. To evaluate AD-TP comprehensively, we compare it with representative pruning methods and other model compression techniques. PoWER-BERS [3 uses progressive word-vector elimination; LAT [14] dynamically adjusts sequence length via Drop: pruning thresholds. Transkimmer [20) employs token-level predictors, while ToP [33] combines ranking distillation with coarsc-to-fine pruning. We also include DistilBERT [34] and CoFi (55 as baselines for distillation and structured pruning. Unlike these approaches, AD-TP adaptively predicts retention ratios based on sequence complexity, enabling more flexible and efficient compression. can also be integrated with orthogonal techniques for further gains.
and LTP [17] learns attention-base
5.1 Main Results
Table 1: Comparison of AD-TP and mainstream pruning methods on GLUE in terms of accuracy and FLOPs. and S denote the teacher and student models, respectively. All baseline results are from ToP [23], except BERT-base (T) and BERT6 (S). Bold and underlined values indicate the best and second- best results.
MACC
MENLI
GNL
Station
11,05
Azc.
11,0%
Axc
21085
Acc.
FLODs
15.05s
Azc.
Poace
21005
BORTS 18
$1.2111
1996
66.1844
1994
100x
$93
1.304
91.004
1882
$17760
1.00%
$03.44
1/0%
17.6
100%
PHWTE LAT
57.1
7.5%
421
350w
851
Alee vaho
92.
96.
388k
685
150
912
K.4K
182
535
3304
967
437%
986
4.38g
174
5334
TP%
00
AD 1913
1E3
46R044
954/74
40
Table 2: Comparison of Token Pruning Methods on Long-Sequenes Tasks.
Table 3: Comparison with Distillation and Struc- tured Pruning.
Model
Method
F1
SQuADv2.0
FL.OPs
Apc.
20News
F1.0Ps
Man:
CoLA
RLDPs
MMP
FLCIPs
Acc.
20Neus
PLDPs
BERT base BERT6
7388.05
77.1:83
1,00x
1.00x
86.7ax7
85.0a12
1.00s
1.00x
DENIBERTS
Coft5
34,0
2,00s
9.10x
M6.
7.70v
85.8
85.9
2.00x
5,90X
Transkinmer PoWER BERT
Prediction
75.7
4.67x
86.1
8.11x
AD-
58.384a
13.78ea82
$9.8.89
6.55
87.30mx
Atlention
86.5
2.9xx
HUXT--b9912
100%%
91.4225
1,00%
10 M0.757
100%
LIP
Anention
75.6
3.10x
85.2
4.60x
CoF112
35,5
9.10%
50.0
4.00%
86.4
7.70%
ToP AD-
Attention
75.9
4,12x
87.0
8.26t
A0-1P12
56.9:44
13,00284
KG
76.2.84
5.07k.a2
87.31a4
8.74x:65
Comparison with Token Pruning Methods. We evaluate AD-TP on the GLUE benchmark and two long-sequence datasets. As shown in Table AD-TP6 consistently improves both accuracy and efficiency over the unpruned student model. AD-1P12 achieves 7.37x FLOPS reduction compared to the teacher model while maintaining competitive performance, and outperforms the state-of-the-an ToP method on several tasks. This highlights the advantage of IG over attention-baser metries in accurately estimating token saliency. On long-sequence tasks (Table AD-TP6 delivers significant gains. Notably, it surpasses BERT-base in accuracy on 20News while reducing FLOPs by 8.74x.
Comparison with Other Model Compression Methods. We compare AD-TP with DistilBERT and CoFi using both 6-yeyer and 12-layer BERT models (Table 3)) AD-TP achieves significantly higher FLOPs reduction than DistilBERT (c.g., 8.7x vs. 2.0x for BERT6), while maintaining or
7

---

improving accuracy. Compared to CoFi, AD-TP consistently delivers better performance under similar or higher compression ratios. Notably, CoFió suffers a marked aceuracy drop, whereas AD-TP6 preserves model performance. These results demonstrate that AD-TP achieves efficient and robust compression across both compact and large-scale Transformer models.
5.2 Ablation study
Impact of the Teacher Model and Loss Components. We conduct ablation studies on 5 variants AD-TP to evaluate the contribution of each distillation component:
90-
removing (1) the teacher model, (2) the cross-entropy loss LCE. (3) the logits-based loss Llogit, (4) the attribution distillation loss LIG. and (5) the retainer loss (CARP + CrSP). As shown in Fig. 7 removing any component degrades performance, with the absence of the teacher model or retainer loss causing the most significant drop (7-8%). These results confirm the essential role of both the teacher model and the Attribution-ANi Aware Adaptive Token Retainer in guiding effective pruning and preserving model accuracy.
AD
Performange
80-
70-
60
S51-2
RTE
MRPC
20News
Figure 7: Ablation study of distillation components.
Impact of the ARP. To assess the effectiveness of our adaptive pruning strategy, we compare three meth- ods: (i) a static retention ratio fixed at 0.3; (ii) ran- dom ratio selected from predefined candidate list with the same average; and (iii) our proposed adaptive method (AD-TP). As shown in Table 4 all methods share the same architecture and differ only in how the retention ratio deter- mined. Results show that the adaptive strategy consistently outperforms the others under com- parable FLOPs, demonstrating the benefit of adjusting pruning strength based on sequence complexity. Additional experimental results on more datasets are provided in Appendix A.4
5.3 Effectiveness Under Different FLOPs Budgets
We evaluate the effectiveness of the AD-TP method under varying FLOPs budgets and compare it with token pruning approaches: LTP and ToP. To ensure a fair com- parison, we utilize the official implementations LTP and ToP and apply grid search to optimize their hyperparame- that each method achieves its best performance un- der a given FLOPs constraint. As shown in Fig. & AD-TP consistently outperforms both baselines across different tasks under varying FLOPs constraints. On the MRPC, under 60% relative FLOPs constraint, AD-TP achieves 3% higher F1 score than LTPP and 2% higher than ToP. Under the same F1 score (87) constraint, AD-TP reduces FLOPSs by approximately 34% compared to LTP and 9% compared ToP.
Table 4: Ablation study of the adaptive token retainer.
SST-2
20News
Model
Acc.
P
Acc.
P
Static Random
91.2
0.30
85.7
0.30
90.5
Adaptive
92.0
0.29
87.3
0.27
Reladie FLOPAN)
Rehiive
FLONN5)
(a) SST-2
(b) STS-B
(c) MRPC
(d) 20News
Figure 8: Performance comparison of AD-TP, LTP, and ToP under different FLOPs budgets.

---

02
1.0
100
120
140
280
30
50
600
(a)
(b) 3
(c)
Figure 9: Impact of a, 33 and on model performance.
5.4 Impact of a, and
To analyze the impact individual loss weights in the objective function, we systematically vary the hyperparameters a, and ry the SST-2, RTE, and 20News. As shown in Fig. |9] smaller value of a reduces the guidance effect from the teacher model predictions, leading to a decline in performance. Increasing / and enhances learning by improving attribution alignment and token retention. However, excessively large values may cause the model to overfit certain sub-objectives, thereby degrading overall accuracy, It is noteworthy that the optimal value of a remains relatively consistent across different tasks, while the best- rest-performing values of 3 and y vary depending on the task, indicating that attribution and retention mechanisms require task-specific tuning.
5.5 Impact of Candidate List Length Lp
In the process of adaptive retention ratio prediction, we define a candidate list vector L, containing pos- sible retention ratios. To investigate how the length of this candidate list impacts model performance and convergence speed, we conducted compara- tive experiments using candidate lists of different lengths. Specifically. List @ contains 5 coarse- grained values, List @ includes 10 medium- ceddium-grained values, and List @ comprises 20 fine- sine-grained val- ues. As shown Fig. IO, the results demonstrate that shorter lists (List () enable faster convergence but achieve lower accuracy, while longer lists (List 3) result in higher accuracy but slower conver- gence. Considering both performance and training efficiency, select List @ the final configura- tion, as it strikes an optimal balance between accuracy and convergence speed. Detailed configurations of the candidate list and supporting results in additional datasets documented in AppendixA.5
Fped
(a) SST-2
(b) 20News
Figure 10: Impact of candidate list length on model accuracy and convergence speed in ARP.
5.6 Evaluation of TSP
To evaluate the effectiveness of TSP, we randomly sampled 1,000 examples from the validation sets of SST-2 and MRPC, and conducted quantitative comparison using two metries: (i) Spearman p: measures the consistency of token-importance rankings between TSP and IG; (ii) MSE (Mean Squared Error): measures the numerical similarity between the importance scores predicted by TSP and those derived from IG. shown Table TSP achieves high ranking consistency (p 0.75) and low numerical error compared with IG, indicating that the module can effectively learn and approximate the saliency distribution derived from IG.
Table 5: Consistency Between TSP and IG Scores.
Dataset
Spearman p
MSE
SST-2 MRPC
0.812
0.0119 0.0143
0.778

---

5.7 Sensitivity of Model Performance to the Number of IG Steps
To reduce computational overhead, we set the IG step count m by default in our implementation. To examine the sensitivity of model performance to this approximation, we conducted additional experiments with different values of m 3, 5, 7]. The results are summarized in Table 7
Table 6: Performance of AD-TP with different IG step counts.
Dataset
m
3
m m=
SST-2
92.0
92.1
92.5
92.3
MRPC
89.8
89.6
90.0
90.5
Experimental results show that the model exhibits low sensitivity to the choice of m. Increasing the number of IG steps brings only slight and unstable performance improvements (typically less than 1%), while the computational cost grows approximately linearly with m. For example, when m 10, the training time increases by about tenfold, which is impractical in resource-constrained or distributed environments. Therefore, m = achieves a good balance between performance and efficiency.
6 Conclusion
In this work, we propose an attribution-drive adaptive token pruning method, AD-TP, for Transformer model compression to reduce computational resource requirements. In contrast to traditional methods that rely on accumulating attention weights to assess token importance, AD-TP applies IG to more accurately evaluate the contribution of each token the model prediction. Furthermore, AD-TP addresses the issue of mismatched pruning ratios and sequence complexity in conventional token pruning methods. It introduces lightweight adaptive token retainer that dynamically selects an appropriate pruning ratio based sequence complexity to better retain important tokens. To the best of our knowledge, AD-TP is first Transformer compression method that explicitly considers sequence complexity and implements adaptive token pruning. By combining teacher supervision with self-supervision, AD-TP effectively reduces computational overhead while maintaining performance. Extensive experimental results demonstrate that AD-TP achieves a 4% reduction in FLOPs and an 8% improvement in performance compared to existing state-of-the-arr token pruning methods, showing significant advantages. Limitations: this work focuses only pruning the input layer, and extending adaptive token pruning to other layers of the model remains an important direction for future research.
Acknowledgments
This work was supported in part by Natural Science Foundation of Shandong Province (No. ZR2022MF328 and No. ZR2019LZH014), in part by National Natural Science Foundation of China (No. 61602284 and No. 61602285), in part by State Key Lab of Processors Open Fund Project (No. CLQ202409), and part by CCF-Ricore Education Fund (No. CCF-Ricore OF 2024003).
References
[1] D. Dai, Y. Sun, Dong, Y. Hao, S. Ma, Z. Sui, and F. Wei, "Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers, in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 4005-4019
[2] V. W. Anelli, G. M. Biancofiore, A. De Bellis, T. Di Noia, and E. Di Sciascio, "Interpretability of bert latent space through knowledge graphs," in Proceedings of the 31st ACM International Conference on Information Knowledge Management, 2022, pp. 18100-880...
[3] L. Wen, X. Zhang. H. Bai, and Z. Xu. "Structured pruning of recurrent neural networks through neuron selection," Neural Nenworks, vol. 123, pp. 134, 2020.
[4] V. Sanh, T. Wolf, and A. Rush, "Movement pruning: Adaptive sparsity by fine-tuning," Advances in neural information processing systems, vol. 33, pp. 20 378 20 389, 2020.
10

---

[5] X. Ma. G. Fang, and X. Wang, "Lim "Lim-pruner: On the structural pruning of large language models." Advances in neural information processing systems, vol. 36, pp. 21 702-21 720, 2023.
[6] H. Bai, W. Zhang. L. Hou, L., Shang, J. Jin, X. Jiang, Liu, M. Lyu, and I. King, "Binarybert: Pushing the limit of bert quantization." " Proceedings of 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), 2021. pp. 4334 4348.
[7] S. Shen, Z. Dong. J. Ye. L. Ma, Z. Yao, A. Gholami. M. W. Mahoney, and K. Keutzer, "Q-bert: Hessian based ultra low precision quantization of bert," in Proceedings of the AAAI Conference Artificial Intelligence, vol. 34, no. 05, 2020, pp. 8815-882)
[8] T. Tambe, C. Hooper, L. Pentecost, Jia, E.-Y. Yang. M. Donato, Sanh, P. Whatmough, A. M. Rush, D. Brooks et al., "Edgebert: Sentence- level energy optimizations for latency-aware multi-task nlp inference," in MICRO-54: 54th Annual IEEF/ACM International Symposium on Microarchitecture 2021, pp. 340-
[9] Z. Liu, "Improving the inference efficiency of transformer models in machine translation tasks by low-rank dccomposition methods," " 2024 International Conference on Electronics and Devices, Computational Science (ICEDCS). IEEE, 2024, pp. 930-936.
[10] 7. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and Zhou, "Mobilchert: a compact task-agnostic hert for resource-limited devices," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 2158-2170.
[11] Y. Gu. L. Dong, F. Wei, and M. Huang, "MiniLLM: Knowledge distillation of large language modcls," in The Twelfth International Conference Learning Representations, 2024.
[12] Z. I Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A convnet for the 2020s," in Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, 2022, pp- 11 976- 11 986.
[13] S. Goyal, A. R. Choudhury, S. Raje, V. Chakaravarthy, Y. Sabharwal, and A. Verma, "Power- bert: Acceler- ating bert inference via progressive word-vector elimination," in International Conference on Machine Learning. PMLR, 2020, pp. 3690-3699.
[14] G. Kim and K. Cho, "Length-adaptive transformer: Train once with length drop, use anytime with search," in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (Volume 1: Lonng Pupers). 2021, pp. 6501-6511.
[15] H. Wang, Z. Zhang, and S. Han, "Spatten: Efficient sparse attention architecture with cascade token and head pruning." " in 2021 IEEE International Symposium Hegf--Peroormance Computer Architecture (HPCA). IEEE, 2021, pp. 97-110.
[16] D. Ye, Y. Lin, Huang, and M. Sun, "Tr-bert: Dynamic token reduction for accelerating bert inference."" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 5798-5809.
[17] S. Kim, S. Shen, D. Thorsley, A. Gholami, W. Kwon, J. Hassoun, and K. Keutzer, "Learned token pruning for transformers." " in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022. pp. 784-794.
[18] H. Lee, Kang. Y. Lee, and J. Hwang, "Sparse token transformer with attention back tracking," The Eleventh International Conference on Learning Representations, 2022.
[19] C. Liu, C. Tao, J. Liang, J. Feng. T. Shen, Q. Huang, and D. Zhao, "Length-adaptive distillation: Customiz- ing small language model for dynamic token pruning," in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 4452 4463.
[20] Y. Guan, Z. Li. J. Leng, Z. Lin, and M. Guo, "Transkimmer: Transformer learns to layer-wise skim," in Pruceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volame Long Papers), 2022, pp. 7275-7286
[21] D. Bachrens, T. Schroeter. S. Harmeling, M. Kawanabe, K. Hansen, and K.-R. Müller, "How to explain individual classification decisions." The Journal of Machine Learning Research, vol. 11, pp. 1803- 1831, 2010.

---

[22] Y. Hao, L. Dong. F. Wei, and K. Xu, "Self-attention attribution: Interpreting information interactions inside transformer," in Proceedings the AAAI Conference on Artificial Intelligence, vol. 35, no. 14, 2021, pp. 12 963-12971.
[23] S. Jain and B. C. Wallace, "Attention is not explanation." in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Pupers), 2019. pp. 3543-3556.
[24] K. Lu, Wang, P. Mardziel, and A. Datta, "Influence patterns for explaining information flow bert," Advances in Neural Information Processing Systems, vol. 34, pp. 4461 4474, 2021.
[25] Mohebbi, Modarressi, and M. T. Pilehvar, "Exploring the role of token representations to explain sentence probing results," in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 792-806.
[26] N. Santoso, 1. Mendonça, and M. Aritsugi, "Text augmentation based on integrated gradients attribute score for aspect- based sentiment analysis." in 2023 IEEE International Conference on Big Data and Smart Computing (BigComp). TEFE, 2023, pp. 227-234.
[27] A. Aghababaei. J. Nikadon, M. Formanowicz, M. L. Bettinsoli, C. Cervone, C. Suitner, and T. Erseghe, "Application of integrated gradients explainability sociopsychological semantic markers," arXiv preprint arXiv:2503.cm 049999, 2025.
[28] S. Sikdar, P. Bhattacharya, and K. Heese, "Integrated directional gradients: Feature interaction attribution for neural nlp models," Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Pupers), 2021, pp. 865-878.
[29] S. S. Roy and A. Kundu, "Uniform discretized integrated gradients: An effective attribution hased method for explaining large language models," arXiv preprint arXiv:2412.038866 2024.
[30] Z. Zhang. Y. Sheng. T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song. Y. Tian, C. Ré, C. Barrett et al., "H2o: Heavy-hitter oracle for efficient generative inference of large language models," Advances in Neural Information Processing Syxtems, vol. 36, pp. 34 661-34 710,, 2023.
[31] P. Rajpurkar, R. Jia, and P. Liang, "Know what you don't 't know: Unanswerable questions for squad," in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018, pp. 784-789.
[32] K. Lang, "Newsweeder: I eernning filter netnews," in Machine Learning Proceedings 1995: Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, July 9-12 1995. Morgan Kaufmann, 2014, p. 331.
[33] J. Li, I.. L. Zhang, J. Xu, Y. Wang. S. Yan, Y. Xia, Y. Yang. T. Cao, H. Sun, W. Deng et al. "Constraint- aware ranking-distilled token pruning for efficient transformer inference," in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 1280-1290.
[34] V. Sanh, "Distilbert, distilled version of bert: smaller, faster, cheaper and lighter." Proceedings of Thirty-third Conference on Neural Information Processing Systems (NIPS2019), 2019.
[35] Xia, Zhong, and D. Chen, "Structured pruning learns compact and accurate models," " Proceddiggs of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), 2022, pp. 1513-1528.
12

---

A Appendix
A.1 Transformer and Token Pruning
Each basic Transformer encoder layer consists of multi- attention (MHA) mechanism and a forwrd neural network (FNN) module, cach surrounded by residual connections. Suppose that each Transformer layer contains N₁, attention heads, Given an input sequence X fr1,..., Zn of length n, the MHA computes the importance of each token with respect to all other tokens as follows:
Nn Σ Au (w3. W w) h=1
MHA(X)
(16)
where w-(ch), wxth) Wy-A), whth) Rdddon denote the projection matrices for query, key, value, and output, respectively. Let d be the hidden dimension, and da the output dimension of each attention head, typically set to d/Nu. The computation for single attention head is given by:
XWQ (XWx)" vi
Att (Wa. Wx. Wy. Wo. X) Wo softmaz
XWV
(17)
Subsequently, the output of the MHA passed through a residual connection followed by layer normalization, as defined by the following equation:
XMHA~ IN (MHA(X)) + X)
(18)
Then, XMαa is fed into the FNN, sequentially passes through two nerdffo layers, and then combined with the original input via a residual connection to produce the following output:
FNN (XMHA) max (0, X AMHHA W b1) W2 b2
(19)
Xout = LN (FNN (AMHA) + XMHA) where, W and b, and W/ and b2, are the weights and biases of the two feod-forward layers, respectively.
(20)
To reduce the computational overhead of Transformers, token pruning removes redundant tokens from the sequence based on importance estimation. A common approach is to introduce sparse binary mask to identify
and discard unimportant tokens, thereby redueing the number of tokens involved in attention and feed forward computations. With such mask, the input be reformulated as:
X + X- Mashioken + Mask
(21)
where Masktoken {0, 13" indicates whether each token is retained. The outputs of the MHA and FNN modules are given by:
Ny M (X) = Σ Alt (WO. Wh, Wh, W Xmask)
(22)
h=1
XMHA LN(MHA (X) + XMask)
(23)
FNN (MMA mar (a, N.NHHAA W1 b1) W2+ b2
(24)
The original computational complexity of the Transformer 0 ) By introducing a sparse mask, the number of tokens involved in computation can be significantly reduced in tasks, therehy improving overall inference efficiency.
A.2 Integrated Gradients
IG is a widely used attribution method for interpreting predictions made by neural networks. It aims to quantify the contribution of each input fcature to the model output by constructing an interpolation path between the original input and predefined haseline input, and then integrating the gradients along this path. Compared to conventional gradient-base based approaches, IG effectively mitigates the gradient saturation problem and provides more stable attribution results, especially in deep networks that use nonlinear activation functions.
Given a model prediction function F. an input x, a baseline input x', the of the i-th feature defined as: OF (x'
A x²))
(G:(x) = (z: x1)
(25)
a=0
Oz:
Since the integral difficult to compute analytically in practice, it is typically approximated using Riemann sum. By dividing the interval [0, 1] into m sub-intervals, we obtain:
1
aF (a' + (x x'))
IG. (z) ~ (xa x
(26)
Oe:
13

---

Table 7: Parameter settings for different datasets.
Dataset
Learning Rate
Epochs
Batch Size
g
CoLA
2e-5
20
32
0.9
50
10
MNLI
3e-5
0.8
50
500
SST-2
2e-5
30
32
0.8
10
100
QNLI
20-5
30
32
0.9
100
100
MRPC
30-5
30
16
0.9
100
100
QQP
4e-5
1.0
100
500
RTE
2e-5
30
0.9
10
500
STS-B
5c-5
10
16
0.8
10
100
This method obtains relatively accurate feature attributions through multiple forward and backward passes, without modifying the original model architecture. In our approach, IG is used to quantify the saliency of each token. These saliency the subsequent token retainer strategy and revealing the primary focus the model.
then employed to estimate the importance of tokens the model output, guiding
A.3 Implementation Details
All experiments are implemented using the PyTorch framework a single NVIDIA RTX 3060 GPU. The teacher model a fine-tuned BERT-base with 12 Transformer layers. 768-dimensional hidden units, and 12 attention heads. The student model adopts two configurations with 6 or 12 Transformer layers, keeping other hyperparameters consistent with the teacher. [ The teacher is trained for epochs each task, and the best validation checkpoint used for evaluation. The learning rate is fixed 2c-5, with a batch size
Huggingface Transformers library on
the student model, learning rates are searched in (2e-5, 3e-5, 4e-5, Se-5, 6e-5), and distillation weight selected from [0.6, 0.7. 0.8, 0.9, 1.0). Other hyperparameters and y are tuned over (1, 10, 50, 100) and (10, 50, 100, 200, 400, 500), respectively. The candidate list for pruning ratios is defined as Ly {0011 0.2. 1.0). Detailed specifc configurations reported in Table 7
A.4 Additional Results on ARP Effectiveness
To further support the analysis in Section 5.4 provide additional results the RTE and MRPC datasets. As
the main experiments, we compare three pruning strategies: Static, Random, and Adaptive (AD-TP), using the same model architecture. These approaches differ only in the method used to determine the token retention ratio. The corresponding experimental results are summarized in Table
Table 8: Ablation study of the adaptive token retainer.
RTE
MRPC
Model
Acc.
F1
Static
66.2
0.30
87.4
0.30
Random
67.5
0.30
88.1
0.30
Adaptive
70.1
0.43
89.8
0.32
A.5 Candidate List Configuration and Full Results
To support the analysis in Section we provide the detailed contiguration of the candidate list vector Lp along with the corresponding experimental results. We define three candidate lists with varying granularity levels for adaptive retention ratio prediction:
List Q: [0.2.004.066.0...1000)
List @: (01.22000400006070880...000
List 3:
0..055.6611507075500055500000000
experimental results on the MRPC dataset are shown Fig
We use the pretrained models and tokenizers from https///tuggingffcc.oo.
14

---

90-
96
82-
Epoch
teacher guidance. Therefore, the Figure 11: Impact of Candidate List Length on Model Accuracy and Convergence on MRPC.
A.6 Inference Overhead vs. Latency
analyze inference overhead introduced by Retainer module (TSP ARP), we measured computa- tional cost and latency across different datasets and input lengths. The Retainer module consists of lightweight 1-2 layer feed-forward networks with fewer than 0.8M parameters, which is approximately 1 1% of BERT- base. Therefore, the additional inference cost is negligible compared the Transformer backbone. Although the pruning benefits become less pronounced for shorter input sequences, the overall inference time is still reduced, as shown in Table
Preliminary wall-clock latency measurements were conducted on an RTX 3060 GPU. Despite the additional modules, AD-TP achieves 30-55% reduction end-to-end latency compared with BERT-base on SST-2, MRPC, and SQuAD, particularly for inputs longer than 64 tokens.
Table 9: Inference latency comparison between BERT-base and AD-TP on RTX 3060 GPU.
Dataset
Input Length
BERT-base (ms)
AD-TP (ms)
Gain
SST-2
128
9.7
51.5%
MRPC
64
6.1
3.8
37.7%
SQuADv2.0
512
21.5
12.3
42.8%
These results demonstrate that the Retainer module introduces minimal inference overhead while providing significant end-to-end latency reduction. Future work will include evaluating AD-TP on mobile and edge CPUs further validate its deployment efficiency real-world settings.
A.7 Training Cost Analysis
To analyze the additional computational cost introduced by the teacher model and Integrated Gradients (IG) supervision. we compared the training time of AD-TP and BERT-base across the eight GLUE tasks. The teacher model and IG are used only during the training phase: during inference. the proposed method relies solely on the lightweight TSP and ARP modules, requiring neither backpropagation inference efficiency and deployability of the model are not affected.
As shown Table TO training with IG supervision increases the overall training time by approximately 20-30% compared BERT-base training without the teacher model and IG. This additional cost is considered reasonable, given that AD-TP substantially reduces FLOPs and improves computational efficiency during inference.
These results indicate that the training-time overhead introduced by IG and the teacher model remains moderate, while offering significant improvements in inference efficiency.
A.8 Broader Impact
This work proposes attribution-driven adaptive token pruning method, AD-TP, for Transformer model compression reduce computational resource consumption. The potential positive societal impacts include reduced computational and energy consumption during inference, which may contribute to more environmentally sustainable Transformer deployments. Additionally, lowering the resource requirements of widely used models like Transformer may improve accessibility for academic or industrial groups with limited computational infrastructure.
15

---

Table 10: Training time comparison between AD-TP and BERT-base on the GLUE benchmark (RTX
3060 GPU).
Dataset
Epochs
Batch Size
BERT-base (h)
AD-TP (h)
CoLA
20
32
0.83
1.17
MNLI
30
16
33.00
40.50
SST-2
30
32
3,50
4.33
QNLI
30
32
7.08
8.67
MRPC
30
16
0.25
0.33
QQP
22.67
28.83
RTE
30
16
0.17
0.20
STS-B
10
16
0,20
0.25
As general purpose model compression technique, proposed method does not target any specific downstream application. However, there a potential indirect risk that efficiency improvements could lower the barrier to deploying Transformer models in sensitive or harmful contexts (c.g., misinformation systems or large-scale surveillance), by making model exccution chcaper. While our work purely algorithmic and does not involve model deployment, acknowledge this possibility.
To mitigate such risks, encourage responsible of model compression techniques, especially when applied models used in real-world decision-making content generation tasks. Transparency about pruning configurations and performance trade-offs is also essential when sharing compressed models.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract introduction accurately summarize the method, theoretical framework,
and experimental results, consistent with Sections 3 to Sections S
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations listed in Section 4
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theoretical proofs results.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
17

---

model The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide information relevant to reproducing the results in Sections and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, as described supplemental material?
the data and code. with sufficient instructions
Answer: [No]
Justification: Model weights are public. We intend release the code pending internal review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines
cc/public/
for more details.
While we encourage release of code and data, we understand that might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : ://nis//..////////// cc/public/ ouiidess/CooeeSttmmiessssoooooo for more details.
18

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Provided in Section 4 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the standard error in the experimental results presented in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
Sufficient information the computational resources is included in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
19

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform. every respect, with NeurIPS Code of Ethics
Answer:
[Yes]
Justification: The research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Both potential positive socictal impacts and negative socictal impacts discussed in Socictal Impact section of the Appendix A.6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative socictal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such release artifacts.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
20

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Assets are properly credited.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No assets released.
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Does not involve crowdsourcing or research with human subjects.
Guidelines:
answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
21

---

Justification:
Does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important. original. or non nta-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
This study proposes token pruning method Transformer-he hased pre-trained language models, specifically using BERT-bbase and a 6-layer BERT model. The use of these models is essential to the core methodological design and experimental evaluation, detailed in Sections and 5
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
http:/pp...../oteeeee/////////// Please refer LLM policy http://///ooorrr...c//o////rrr/s for what should or should be described.
22


Reviews:
[{"review_id": "vq5rLuAQNk", "reviewer": "Reviewer_joVc", "summary": "This paper presents a novel token pruning method for transformer-based models, which utilizes token saliencies from intergrated gradients to reduce the overall input sequence length. The proposed method, AD-TP, is composed of two distinct parts, an Adaptive Retention Ratio Predictor that predicts how much of the input sequence should be preserved (based on complexity), and a Token Saliency Predictor which predicts and selects the actual tokens to keep vs discard. The proposed method is evaluated against several existing token pruning methods, using BERT-base as the model and GLUE, SQuAD and 20News as the datasets. The results show that the AD-TP matches or outperforms the best previous method, while pruning more tokens (i.e. higher efficiency gains). The various AD-TP submodules are trained using a student-teacher framework, where the submodules learn to predict both the retention ratio and IG saliency scores without needing the ground truth.", "questions": "- Figure 3 is used to depict the complexity of the input sample? Can one just use a syntactic dependency parser as a light weight alternative to ARP?\n- Why not just use a regression model to predict the retention ratio P_r directly instead of using a candidate list L_p?\n- [1,2] are vision papers, not talking about why GPT/BERT are central to natural language processing. CoFi's citation is also [2], but it doesn't seem to be mentioned in that paper anywhere; are these references here by mistake?\n\nPlease also take a look at the questions mentioned in the *strengths and weaknesses* section.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper studies an important (and timely) problem of high inference cost in transformer-based models. The proposed methodology combines quite a few existing pieces in a novel formulation, and presents an interesting framework to train the submodules of this framework. The evaluation is quite diverse in terms of dataset, but only relies on a single (small) model. At least on this setting, the proposed methodology performs quite well, matching or improving upon the previous best method at higher efficiency gains. \n\nHaving said this, there are some weaknesses that impact the quality and relevance of this work quite significantly:\n\n- The paper is a bit difficult to read in general, and the widespread use of similar abbreviations really makes it even harder. There are also several claims and terms used in the paper without any clear explanation or evidence of support. One instance of this is \"residual strategy\" in Section 3.1, which is not defined anywhere in the paper.\n- The experimental setting is quite dated; the model and the datasets are really from the \"previous\" era of NLP, as evidenced by the fact that the related work is all 4 or 5 years old. Its really a big miss to not have tried this method on a more recent LLM model, where the impact could be much greater. \n- Details on how AD-TP is trained are very sparse, apart from the loss function derivation. On what dataset is the training conducted on? How long is the training run for? This hinders reproducibility significantly, since without a trained AD-TP model the proposed methodology is incomplete.\n- It was also strange to see no evaluation of TRP; do the scores predicted by it match the IG scores (after the model has been trained)? Its a fundamental piece of the overall method, and given the importance of IG scores (as self-described in the paper), its important that this module mirror IG's scores well.\n\nOverall, the proposed methodology itself is sound, and the experimental results do improve upon existing methods to some extent, but it is unclear what the impact of the work is in the era of modern LLMs where classification is done differently, and a lot of tasks are dominated by generation (which is completely ignored in this paper). This, along with some key missing details sheds some doubt over the efficacy of the work.", "was_revised": true, "final_justification": "The rebuttal clarifies quite a bit of the methodological concerns I had regarding the setup + efficacy of individual components. I still stand by my original assessment that the original manuscript is difficult to read, replicate and draw concrete implementations from, but the rebuttal is a good start in the right direction.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and the recognition of our method as **“a novel formulation”**, **“an interesting framework”**, and **“a sound methodology”** that achieves **“higher efficiency gains”** and **“performs quite well”**. For each of the concerns raised, we provide detailed responses below.\n\n***Q1: Is Figure 3 illustrating input complexity, and can a dependency parser serve as a lightweight replacement for ARP?***\n\nThank you for the insightful question. Yes, Figure 3 illustrates that input sequences within the same dataset can vary significantly in complexity.\n\nUsing a **dependency parser** as a lightweight alternative is an interesting idea. While dependency parsers can capture **surface-level structural information**, they fail to adequately reflect **semantic factors** that are critical to model predictions, such as negation, sentiment polarity, and task difficulty. Additionally, dependency parsers are **general-purpose tools** and typically **lack task awareness**. \n\nIn contrast, our ARP module is trained on the model’s own **semantic representations** (the `[CLS]` embedding), enabling it to learn **task-aware** input complexity in an end-to-end manner. This allows ARP to better align with the model’s **internal understanding** of inputs. With just a **single-layer feed-forward network**, ARP is lightweight, efficient, free of external dependencies, and easily transferable across tasks.\n\n***Q2: Why not just use a regression model to predict the retention ratio $P_r$ directly instead of using a candidate list $L_p$?***\n\nThank you for this insightful question. We found that directly regressing the retention ratio $P_r$ leads to **instability during training**. It is **sensitive** to initialization and label scale, and often produces **unreasonable predictions** (e.g., overly high or low retention rates), which negatively affect pruning performance. Moreover, continuous outputs **lack bounded constraints**, making it difficult to control FLOPs.\n\nIn contrast, using a discrete candidate list (e.g., $L_p = \\\\{0.3, 0.5, 0.7, 1.0\\\\}$) is **more stable and controllable**. By adjusting the granularity and range of the candidate set, we can **flexibly control model FLOPs** to meet different deployment budgets and enable dynamic scheduling during inference.\n\nTherefore, we adopt the candidate list approach, which offers **bounded and interpretable predictions**, **more stable and faster training**, and **better suitability for budget-constrained applications**. In future work, we plan to explore ways to mitigate the instability and uncontrollability of regression-based prediction, making it more robust for our task and further improving overall performance.\n\n***Q3: References Citation***\n\nWe sincerely apologize for the citation issue and thank the reviewer for the careful and detailed observation.\n\nWe recognize that references [1] and [2] are primarily from the computer vision domain. Although they mention the application of Transformer architectures in NLP within their related work sections, they are not the most appropriate sources to support our argument. We will replace these references in the revised version with more suitable ones, such as:\n- Dai, D., et al. (2023). **Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers**. Findings of the Association for Computational Linguistics: ACL 2023.\n- Anelli, V., et al.(2022, October). **Interpretability of BERT Latent Space through Knowledge Graphs**. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (pp. 3806–3810).\n\nWe also acknowledge the incorrect citation of CoFi as reference [2]. This will be corrected in the revision and replaced with the proper original source:\n- Xia, M., et al. (2022, May). **Structured Pruning Learns Compact and Accurate Models**. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1513–1528).\n\n***W1: Regarding the abbreviation issue***\n\nThank you for the valuable feedback.\n\nWe recognize that the paper includes an excessive use of abbreviations, particularly the repeated use of the two modules in our model: **TSP** and **ARP**. Although all abbreviations are defined upon their first appearance, we will reduce their repetition in the revised version to improve readability.\n\nRegarding the term \"Residual Strategy\" in Section 3.1, it refers to an **upper-bound** method that relies on label and gradient information. This method approximates a token's contribution to the model's loss by comparing hidden representations from different layers ($H_r$ and $H_l$, where $r > l$), and directly measures each token's maximum impact on the model's output. The specific formula is:\n$$\nI = \\frac{\\partial \\text{loss}}{\\partial H_r} \\cdot (H_r - H_l)\n$$\nHowever, since it is a **post-hoc** method, it can only be computed when the labels are known and backpropagation is available. Therefore, it cannot be used in the inference phase and serves solely as a theoretical benchmark.\n\n***W2: Regarding the experimental setting issue***\n\nThank you for this important comment. We believe applying our method to LLMs is indeed meaningful and promising. In this work, we chose BERT-base and standard datasets (e.g., GLUE, 20News, SQuAD2.0) mainly for the purposes of **fair comparison, reproducibility, and experimental efficiency**, which are common practices in recent pruning studies (e.g., LTP, ToP, TS-DTP, APS).\n\nEncoder-only models **remain valuable** in practical latency-sensitive applications such as mobile NLP and search. In contrast, LLMs like GPT-3 or LLaMA incur much **higher training and ablation costs**, making systematic experimentation difficult. Thus, we focus on a **stable and controlled setup** as a foundation for future extensions.\n\nOur proposed ARP and TSP modules are designed to be **general and extensible**, and can be naturally applied to LLM scenarios. For example, they can be used to prune redundant tokens in large prompts while preserving task-critical elements (e.g., instructions, key phrases). We plan to explore and validate such extensions in future work.\n\nWe also appreciate the reviewer’s suggestion regarding the recency of related work. We have cited multiple recent studies in our paper, including two in 2022, one strong KDD 2023 baseline, and other recent pruning and compression works. In addition, Table R1 supplements our method with a comparison against a token pruning approach that **was unpublished at the time of our submission**, as well as two other compression methods (These two methods are less closely related to our work and were therefore not discussed in detail in the main paper).\n\n**Table R1: Performance Comparison with Recent Methods**\n|Method|SST-2|QNLI|\n|---------------|-------|------|\n|TS-DTP (2025)|91.9|86.1|\n|SOA (2025)|91.3|89.2|\n|APS (2024)|92.0|89.4|\n|AD-TP6|92.0|89.5|\n|AD-TP12|94.7|90.2|\n\n***W3: Regarding training details***\n\nWe thank the reviewer for raising this important point and fully agree that detailed training information is crucial for understanding and reproducibility.\n\nTo support reproducibility, we have provided full implementation details in Appendix A.3 of the main paper. All experiments use PyTorch and Huggingface Transformers on a single **NVIDIA RTX 3060 GPU**. The teacher is a fine-tuned BERT-base (12 layers, 768 hidden size, 12 heads), trained for **5 epochs** with a **2e-5 learning rate** and **batch size 32**. The student model uses **6 or 12 layers**, with other settings unchanged. **Learning rates** and **loss weights** (α, β, γ) are tuned over predefined grids. The candidate pruning ratios are  **$L_p = \\\\{0.1, 0.2, ..., 1.0\\\\}$**.  The detailed task-specific configurations are provided in Table R2.\n\n**Table R2: Parameter Settings for Different Datasets**\n|Dataset|Learning Rate|Epochs|Batch Size|α|β|γ|\n|---------|----------------|--------|-------------|-----|-----|-----|\n|CoLA|2e-5|20| 32|0.9|50|10|\n|MNLI|3e-5|30|16|0.8|50|500|\n|SST-2|2e-5|30|32|0.8|10|100|\n|QNLI|2e-5|30|32|0.9|100|100|\n|MRPC|3e-5|30|16|0.9|100|100|\n|QQP|4e-5|30|16|1.0|100|500|\n|RTE|2e-5|30|16|0.9|10|500|\n|STS-B|5e-5|10|16|0.8|10|100|\n\nOur method is built on **BERT-base**. The AD-TP modules (IG, ARP, TSP) are jointly trained end-to-end on each **downstream task** (e.g., SST-2, SQuAD) without extra pretraining or external data.\n\nPer your suggestion, we also report **training time statistics** for each GLUE task (see Table R3), and will further highlight them in the revised version.\n\n**Table R3: Training Time Statistics for GLUE Tasks**\n|Dataset|AD-TP Time (h)|\n|---------|----------------|\n|CoLA|1.17|\n|MNLI|40.50|\n|SST-2|4.33|\n|QNLI|8.67|\n|MRPC|0.33|\n|QQP|28.83|\n|RTE |0.20|\n|STS-B|0.25|\n\n***W4: Regarding the evaluation of TSP***\n\nThank you for raising this important question. TSP is specifically designed to imitate IG's saliency behavior. It is trained under the guidance of IG scores from both the teacher and student models, allowing it to accurately reflect IG's token-level importance assessments.\n\nIn addition, to evaluate its effectiveness, we randomly sampled 1,000 examples from the validation sets of SST-2 and MRPC, and conducted a **quantitative analysis** using the following two metrics:\n\n- **Spearman ρ:** Measures the consistency in token importance rankings between TSP and IG.\n- **MSE:** Measures the numerical similarity between the importance scores predicted by TSP and those from IG.\n\n**Table R4: Consistency Between TSP and IG Scores**\n|Dataset|Spearman ρ|MSE|\n|---------|-------------|--------|\n|SST-2|0.812|0.0119|\n|MRPC|0.778|0.0143|\n\nThese results show that TSP achieves **high ranking consistency** (ρ > 0.75) and **low numerical error** with respect to IG, demonstrating that it can effectively learn and approximate the saliency distribution derived from IG."}, {"review_id": "LrXbIMi4Kd", "reviewer": "Reviewer_vH7T", "summary": "This paper proposes AD-TP (Attribution-Driven Adaptive Token Pruning), a method for efficiently compressing Transformer models by selectively pruning tokens based on their actual contribution to the model's prediction. Unlike prior approaches that rely on attention scores or static thresholds, AD-TP leverages Integrated Gradients (IG) to estimate token saliency more accurately and introduces a lightweight adaptive token retainer that dynamically adjusts the pruning ratio per input sequence and retain tokens based on token saliency.  Extensive experiments on GLUE, SQuAD, and 20News and more show that AD-TP significantly reduces computational cost (up to ~14× FLOPs reduction) while preserving or improving model accuracy, outperforming existing pruning and distillation methods.", "questions": "1. It seems like both IG and TSP affect the token saliency. Why do we have two modules to determine the token saliency, and how do they collaborate to work together?\n\n2. Why do we use IG instead of other metrics like (masked) KL divergence? Can you show the benefit of IG over other metrics such as (masked) KL divergence?\n\n3. In Table 1 and Table 3, why is AD-TP6 always better than BERT6 (S) although the FLOPs are much lower? It seems like this is not always true for all token pruning methods, such as CoFi6. Any insights here?\n\n4. From Figure 2, it seems like there is a length distribution mismatch between the training set and the evaluation set. Does that have any impact on the ARP module, which is used to estimate the complexity of the sequence and determine the retention ratio? For example, there may be domain shift in terms of input length—how does this domain shift affect your method?\n\nIf my concerns about IG are properly addressed, I would consider raising my rating. Additional insights would be appreciated.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Quality**: This work aims to improve the token pruning method, and the idea of estimating sequence-level difficulty and assigning an optimal pruning ratio for each sequence is technically sound.\n\n**Clarity**: The motivation, method, and experiments are presented completely, but there is room for improvement. For example, the tables are too small, especially Table 1. Also, Figure 3 seems useless, and I don't quite understand what it's trying to convey. Additionally, it remains unclear what is determining the token saliency—is it the TSP module or the IG?\n\n**Significance**: How to dynamically determine the pruning ratio for each input sequence deserves further exploration by the community, and the ablation study on the impact of candidate list length seems interesting. However, it lacks an ablation on the ARP module alone, so it is hard to judge what its contribution looks like. It would be nice to see the ablation study for each individual component (IG, ARP, TSP).\n\n**Originality**: See **Significance**. In general, some interesting insights are provided, but the analysis is not adequate.", "was_revised": true, "final_justification": "The rebuttal clarifies the function and contribution of each component more clearly. I raise my rating to borderline accept.", "rebuttal": "We sincerely thank the reviewer for the detailed and constructive feedback, as well as the recognition of our work as a **“technically sound”**, **“presented completely”** approach with **“interesting insights”** and **“good evaluation”**. We address each of your thoughtful questions and concerns in detail below.\n\n***Q1: Role and Interaction of IG and TSP in Saliency Estimation***\n\nThank you for raising this insightful question. Indeed, both Integrated Gradients (IG) and Token Saliency Predictor (TSP) relate to token saliency, but they play complementary roles within AD-TP:\n\n- **IG** is used to compute token-level attributions, serving as the core measure of each token’s contribution to the model’s predictions. It is used **only during training** to provide fine-grained, attribution-based token saliency scores that accurately reflect each token's **true contribution** to the model’s output. These scores are then used as **supervision** to train the TSP module.\n\n- **TSP** is a lightweight, learnable, dynamic module that, after training, efficiently approximates the IG scores **during inference**, thus avoiding the high computational cost of backpropagation-based attribution methods.\n\nThe collaboration between the two allows the model to benefit from the **high-quality supervision signals** provided by IG during training, while enabling **fast estimation** via TSP during inference. TSP can also work collaboratively with the ARP module to jointly decide which tokens to retain, enabling the learning of an efficient and task-aware pruning strategy.\n\n***Q2: Why do we use IG instead of other metrics like (masked) KL divergence? Can you show the benefit of IG over other metrics such as (masked) KL divergence?***\n\nThank you for this valuable question. While both Integrated Gradients (IG) and KL divergence (specifically, masked KL divergence) can be used to evaluate token importance, they are based on different principles. IG is better suited to our task for the following reasons:\n\n- **Different Attribution Mechanisms**:  \n  KL divergence indirectly evaluates importance by observing how the output **distribution changes** when a token is masked, making it hard to clearly attribute predictions. In contrast, IG directly integrates gradients to **quantify each token’s contribution precisely**.\n\n- **Stronger Stability and Interpretability**:  \n  IG integrates gradients along the input path, producing **smoother and more robust** attributions. KL-based methods are **sensitive** to perturbations and often yield **unstable** importance scores.\n\n- **Higher Computational Efficiency**:  \n  KL divergence requires masking each token and performing **a separate forward pass for each**, which is **computationally expensive**. IG enables **parallel computation** across the entire sequence, offering better efficiency.\n\n- **Better Contextual Awareness**:  \n  IG captures **interactions between tokens** by considering the joint gradients of the full input. KL divergence, which focuses only on the effect of removing individual tokens, is prone to **overlooking important contextual dependencies**.\n\n***Q3: Why is AD-TP6 better?***\n\nThank you for this insightful question. We believe this phenomenon is primarily due to the following three factors:\n\n1. **Task-aware adaptive token pruning**  \n   AD-TP6 dynamically identifies and removes **redundant or low-information** tokens (e.g., stop words) based on saliency scores. This increases input **information density** and allows attention to focus on **task-critical content**.\n\n2. **Better alignment between pruning objective and task needs**  \n   CoFi6 is a structured pruning method that reduces FLOPs by removing layers, attention heads, or FFN units. However, at high sparsity levels, it may inadvertently prune **components crucial** for specific tasks, degrading performance. In contrast, AD-TP6 applies fine-grained, token-level pruning that better preserves semantic content and avoids structural disruption.\n\n3. **Distillation-guided training**  \n   AD-TP6 incorporates **knowledge distillation** from a teacher model, which helps the student model retain strong representational and predictive capabilities under compression.\n\n***Q4: Impact of Input Length Distribution Shift on ARP Module***\n\nThank you for this important question regarding potential domain shift in input length.\n\nWe acknowledge that domain shift in input length between training and evaluation sets can pose challenges for fixed pruning strategies. However, AD-TP is explicitly designed to handle such variability. Our ARP module estimates the token retention ratio dynamically during inference, not based on raw sequence length, but on **semantic-level features** such as the [CLS] representation and IG-based saliency. These features capture aspects like **semantic density** and **structural complexity**, which are more indicative of task-relevant content than length alone.\n\nIn fact, this design was motivated in part by the observation of length mismatches in real datasets (e.g., SST-2, shown in Figure 2). Despite this mismatch, AD-TP achieves **top performance** (Table 1), demonstrating robustness to such domain shifts. Furthermore, our method also performs consistently well on datasets with **much larger input length variation** (e.g., 20News, Table 2), further validating ARP’s generalization ability.\n\nTherefore, although input length distribution may shift, the impact on our method is minimal due to ARP’s **adaptive, complexity-aware pruning** mechanism.\n\n***W1: Technically Sound***\n\nThank you for your positive feedback. We appreciate your recognition of our method’s technical soundness.\n\n***W2: Table 1 size is too small, Figure 3 seems unclear and unnecessary, and the determination of token saliency is unclear***\n\nThank you for your constructive feedback. We greatly appreciate your comments on the presentation and clarity of our work. Below, we provide responses to each of the points you raised:\n\n1. **Table Size**: We will enlarge Table 1 and improve its layout in the revised version to enhance readability.\n\n2. **Figure 3**: Figure 3 is intended to show that even within the same dataset, different input sequences may have **varying structural complexity**, which can be reflected in their syntactic dependency structures. We display several example syntactic dependency graphs from the same dataset to **support the motivation** behind the ARP module design — dynamically adjusting the retention ratio based on sequence complexity.\n\n3. **Token Saliency Determination**: The saliency during inference is determined by the TSP, which uses IG as a supervision signal during training. Please refer to the response in Q1 for further details.\n\n***W3: Ablation on ARP module and other components***\n\nThank you for your thoughtful and constructive feedback.\n\nRegarding the ablation of the ARP module, we have already conducted an ablation study on this component in Section 5.2 (**“Impact of the ARP”**) and Table 4 of the main text. In this study, we compare three pruning strategies: (i) fixed ratio, (ii) randomly selected ratio from a list, and (iii) our proposed adaptive ARP. The results show that ARP provides consistent performance gains, demonstrating its effectiveness.\n\n**Table 4: Ablation study of the adaptive token retainer**\n|Method|SST-2 Acc.|SST-2 ρ|20News Acc.|20News ρ|\n|-----------|------------|---------|-------------|----------|\n|Static| 91.2| 0.30| 85.7| 0.30|\n|Random| 90.5| 0.30| 82.5| 0.30|\n|Adaptive| 92.0| 0.29| 87.3| 0.27|\n\nFor the ablation analysis of other components, we have already conducted ablation experiments on multiple loss terms in Section 5.2 (**\"Impact of the Teacher Model and Loss Components\"**) and **Figure 7** ($L_{\\text{CE}}$, $L_{\\text{logit}}$, $L_{\\text{IG}}$, and $L_{\\text{ARP}}$ + $L_{\\text{TSP}}$). To further disentangle the contributions of each component, we have added results in Table R1 that ablate $L_{\\text{ARP}}$ and $L_{\\text{TSP}}$ individually.\n\n**Table R1: Ablation Study of $L_{\\text{ARP}}$ and $L_{\\text{TSP}}$ Loss Components**\n\n|Dataset|w/o $L_{\\text{ARP}}$|w/o $L_{\\text{TSP}}$|AD-TP|\n|---------|---------------------|---------------------|--------|\n|SST-2|86.5|86.1|92.0|\n|RTE| 61.5 | 61.3 |70.1|\n| MRPC| 86.7|86.2| 89.8|\n| 20News| 83.0 | 82.8| 87.3|\n\nIn addition, we have added **structural ablation** studies on each core module, including:\n- *IG*: Replace IG with attention scores for training.\n- *ARP*: Replace adaptive pruning with random ratio.\n- *TSP*: Remove TSP, apply random pruning.\n\nAs shown in Table R2, removing any single module leads to a notable performance drop. Using attention scores instead of IG fails to accurately reflect token importance, highlighting IG's **critical role** in TSP learning. Eliminating ARP results in fixed pruning ratios, which cannot adapt to input complexity and may **discard key information**. Completely removing TSP makes pruning decisions less discriminative, causing many important tokens to be **mistakenly pruned** and leading to a significant performance decline. These results underscore the essential role of each core component in the effectiveness of AD-TP.\n\n**Table R2: Structural Ablation of IG, ARP, and TSP Modules**\n\n|Dataset|Replace IG|Remove ARP|Remove TSP|AD-TP|\n|---------|------------|------------|------------|--------|\n|SST-2|90.2|80.5|79.0|92.0|\n|MRPC|87.6|80.9|78.6|89.8|\n\n***W4: The analysis lacks depth***\n\nThank you for your valuable feedback. We are pleased that you recognized some of the innovations in our work.\n\nTo address the analysis concerns, we have added **loss and module ablation studies** (Tables R1–R2), and included an **effectiveness analysis of the TSP module** (see our response to Reviewer joVc, W4). We also introduced additional **baseline comparisons** (e.g., TS-DTP, SOA, APS). \n\nFurthermore, we plan to incorporate **visualizations** (e.g., attention maps) to better illustrate how AD-TP6 preserves key information."}, {"review_id": "Xw4CE7AHka", "reviewer": "Reviewer_QJNG", "summary": "This paper introduces AD-TP (Attribution-Driven Adaptive Token Pruning), a novel method for compressing Transformer models by dynamically pruning less informative tokens from input sequences. The work aims to address two key limitations of prior token pruning techniques: (1) the use of inaccurate proxies like attention scores for determining token importance, and (2) the reliance on static, one-size-fits-all pruning configurations. To achieve this, AD-TP makes two main contributions. First, it employs Integrated Gradients (IG), a feature attribution method, to compute a more faithful measure of each token's contribution to the final prediction. Second, it introduces a lightweight, Attribution-Aware Adaptive Token Retainer module that operates at the instance level. This module predicts an optimal token retention ratio for each specific input sequence and learns to approximate the expensive IG scores for efficient inference. The entire system is trained via a teacher-student distillation framework, where a larger teacher model provides supervision for both the classification task and the IG-based attribution scores.", "questions": "1.  Federated Implementation Feasibility: How do the authors envision AD-TP being trained in a federated environment?\n- Would this be a \"Federated Distillation\" setup where a central server holds the teacher model and clients only train the student and retainer?\n- If the entire process is federated, how would the expensive IG calculations be handled on resource-constrained client devices? What would be the communication cost of aggregating the adaptive retainer module in each round?\n\n2.  Generalization to Other Architectures: The ARP module relies on a `[CLS]` token. Have you considered how to adapt this mechanism for decoder-only models like the GPT family? Would a simple approach like averaging final-layer token embeddings suffice as a global sequence representation?\n\n3.  Sensitivity to IG Approximation: The IG calculation is approximated using `m=1` step for efficiency. How sensitive is the final model performance to this approximation? Is there a significant accuracy gain from using a larger `m`, and how does that affect the already high training cost?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Quality & Originality: Principled Saliency Metric - The core innovation of using Integrated Gradients (IG) instead of attention scores as the basis for pruning is a significant and well-justified step forward. The authors correctly argue that attention weights are not designed to measure a token's contribution to the final output. By grounding the pruning decision in a more theoretically sound attribution method like IG, the method directly links pruning to the model's task-specific reasoning. This is a novel and principled approach that, as the results show, preserves accuracy more effectively.\n\n- Significance & Originality: Instance-Level Adaptability - The introduction of an adaptive module that predicts the pruning ratio for each input is a crucial advantage. This is particularly relevant in heterogeneous data environments, such as those found in Federated Learning (FL). In an FL setting, client data is often non-IID; some clients may have simple, highly prunable data, while others have complex data requiring more tokens. AD-TP's ability to adapt its compression rate on-the-fly for each sample is a major improvement over static methods that would be suboptimal across a diverse federation of clients.\n\n- Quality: Comprehensive Training Framework - The teacher-student training regime is methodologically robust. The combination of logits-based distillation and attribution-based distillation provides a rich supervisory signal, ensuring the student model learns both *what* to predict and *why* it is making that prediction. The scheduled weighting of the attribution loss is a clever detail that likely aids training stability.\n\n- Significance: Strong Empirical Results - The paper demonstrates impressive performance, substantially reducing FLOPs while maintaining high accuracy across a range of NLP tasks. The fact that AD-TP outperforms other well-established compression techniques like DistilBERT and specialized pruning methods like ToP underscores its effectiveness, especially on long-sequence tasks where the computational savings are most impactful.\n\nWeaknesses:\n\n- Quality & Significance: Prohibitive Training Complexity - The most significant weakness of AD-TP, particularly from a Federated Learning perspective, is its computationally expensive training process. The methodology requires a pre-trained teacher model, the calculation of IG scores (which involves multiple forward/backward passes per sample), and the optimization of a complex, multi-part loss function. This high resource requirement makes the method impractical for on-device training in a conventional FL setting. It appears better suited for centralized training followed by deployment to the edge, rather than end-to-end federated training.\n\n- Quality: Inference Overhead vs. Latency - While the paper focuses on FLOPs reduction, the Adaptive Token Retainer module itself adds parameters and a non-zero computational overhead at inference time. For very short sequences, this overhead could potentially negate the benefits of pruning. A more complete evaluation would include wall-clock latency benchmarks on hardware relevant to edge deployments (e.g., mobile CPUs) to demonstrate a practical speedup beyond theoretical FLOPs.\n\n- Clarity & Significance: Unexplored Federated Learning Implications - While the adaptability of AD-TP is a great fit for FL's data heterogeneity, the paper does not discuss how the method would be implemented in such a setting. It is unclear how the complex training would be federated, or what the communication overhead of aggregating the adaptive retainer module across clients would be. Without this analysis, its applicability to the FL domain remains speculative.\n\n- Originality: Architectural Specificity - The design of the Adaptive Retention Ratio Predictor (ARP) relies on the `[CLS]` token embedding as a global representation of the input. This is standard for BERT-like encoders but limits the method's direct applicability to other popular architectures, such as decoder-only models (e.g., GPT-family) or encoder-decoder models (e.g., T5), without non-trivial modifications.\n\n- Syntactic errors:\n1. Figure 4 Caption: \"Adaptive Aoken Aetainer\" -> \"Adaptive Token Retainer\"", "was_revised": true, "final_justification": "Authors have conducted additional experiments for Q3 & W2 to address my concerns. My concerns have been clarified to a great extent, and I am satisfied with the response. I am keeping my original score.", "rebuttal": "We sincerely thank you for your detailed comments and constructive feedback, especially your appreciation of our work as a **\"novel and principled approach\"**, **\"methodologically robust\"**, **\"major improvement\"**, **\"clever detail\"**, **\"training stability\"**, **\"impressive performance\"**, and **\"significant and well-justified step forward\"**. Your thoughtful suggestions greatly help us improve the clarity and completeness of the paper.  We provide detailed responses to your comments below.\n\n***Q1: Feasibility and Design of AD-TP in Federated Learning Environments***\n\nWe sincerely thank the reviewer for raising this important and practically meaningful question. We fully agree that the proposed **Federated Distillation** framework is a feasible and effective way to deploy AD-TP.\n\nIn this framework, a central server holds the teacher model and performs attribution analysis on a public or synthetic dataset, **precomputing supervision signals** such as logits and IG (Integrated Gradients) saliency scores. Clients use their **local private data** to train only the **lightweight student model and the Retainer modules (ARP / TSP)**. Each client receives the supervision signals (e.g., class-level logits and attribution distributions) from the server and uses them for local training. This design ensures **data privacy**, shifts **heavy computation** to the server, and supports **client heterogeneity**.\n\n**Client-side IG Computation**\n\nComputing IG on the client side is **optional**. By setting $k = 0$ in Equation (6), the student model can fully rely on attribution scores provided by the teacher model, with **no need to compute IG locally**.  If the goal is to improve the student’s attribution ability via self-supervision, a **lightweight variant** such as IG with \\( m = 1 \\) can be used.  For instance, with a sequence length of 128, a single IG computation with \\( m = 1 \\) requires only **3.9 GFLOPs**, which is negligible on modern GPUs and therefore **practically feasible**.\n\n**Communication Cost of the Retainer**\n\nThe Retainer in AD-TP consists of two lightweight modules, **ARP and TSP**, totaling approximately **0.8M parameters** (about **1% of BERT-base**).  Under **32-bit precision**, each communication round (**upload + download**) incurs a cost of about **6.4MB**, which is acceptable for most federated learning scenarios.  Furthermore, techniques such as 8-bit quantization or Top-k sparse updates can be applied to further reduce communication overhead, making the approach viable even in low-bandwidth federated environments.\n\n***Q2: Adapting ARP for Architectures Without a [CLS] Token (e.g., GPT)***\n\nThank you for this great question. We have indeed considered adapting the ARP module to decoder-only architectures (such as the GPT family), particularly in the absence of a `[CLS]` token.\n\nRegarding your suggestion of averaging the final-layer token embeddings, we agree that this is a reasonable and effective simplification.\n\nIn addition, we have explored two alternative mechanisms:\n\n- **Attention Pooling**: We introduce a lightweight attention layer that assigns a learnable weight $t_i$ to each token embedding $e_i$, constructing a global representation as  \n  $$\n  x_g = \\sum t_i \\cdot e_i.\n  $$\n  This approach serves as a soft aggregation strategy, independent of any special structural token, and offers strong global awareness and task adaptability.\n\n- **Last Token Representation**: In generative tasks, the final token in a sequence often carries rich semantic information. We therefore also consider using the last token embedding as a simple and practical global representation.\n\nOnce again, thank you for raising this important question, which helped us deepen our reflection on the generalizability of the proposed method.\n\n***Q3: Sensitivity of Model Performance to IG Approximation***\n\nThank you for raising this thoughtful question.\n\nIn our implementation, we set *m = 1* by default to reduce the computational burden. To evaluate the sensitivity of the model to this approximation, we conducted additional experiments with different step values (*m* = 1, 3, 5, 7). The results are shown in Table R1 below:\n\n**Table R1: Performance of AD-TP with different IG step counts**\n\n| Dataset | m = 1 | m = 3 | m = 5 | m = 7 |\n|---------|-------|-------|-------|-------|\n| SST-2   | 92.0  | 92.1  | 92.5  | 92.3  |\n| MRPC    | 89.8  | 89.6  | 90.0  | 90.5  |\n\n**Key Findings:**\n\n- The model exhibits **low sensitivity** to the choice of *m*: increasing *m* results in only minor performance improvements (typically less than 1%).\n- These gains are **inconsistent across tasks**, and in some cases, performance even slightly decreases due to noise introduced by approximation.\n\n**Cost Analysis:**\n\n- The computational cost of IG grows **linearly** with *m*.\n- For example, setting *m = 10* increases the training cost by approximately **10×**, which is particularly prohibitive in **resource-constrained or distributed environments** (e.g., federated learning).\n\nWe believe that *m = 1* strikes an optimal balance between **performance and efficiency**, making it a practical and scalable choice in real-world applications.\n\n***W1: Prohibitive Training Complexity***\n\nThank you for raising this important concern. We fully acknowledge that AD-TP introduces additional overhead during training compared to conventional pruning methods, primarily due to the use of IG (Integrated Gradients) attribution and the teacher–student framework.\n\nHowever, we would like to clarify several points to demonstrate that AD-TP remains feasible in practical settings:\n\n- **IG computation and teacher supervision are only used during training**, while inference remains **efficient and architecture-agnostic**;\n- We adopt an **approximate IG method with $m = 1$** , which significantly reduces computational cost with minimal performance loss;\n- The Retainer modules (ARP and TSP) are extremely **lightweight**, with a total parameter count of approximately **0.8M**.\n\nIn federated learning scenarios, we can adopt a **Federated Distillation** strategy: the teacher model is hosted on a central server, which computes **logits and attribution scores** on public or synthetic data. Clients only need to receive these supervision signals to train the **student model and Retainer modules**. For implementation details, please refer to our response in **Q1**.\n\nWhile AD-TP is compatible with **centralized training followed by deployment to edge devices**, it is **not limited** to this setup. Its **modular design** also makes it applicable to **end-to-end federated training**, which we plan to explore further in future work.\n\n***W2: Inference Overhead vs. Latency***\n\nThank you for raising this important question. We understand the concerns regarding inference overhead and offer the following clarifications.\n\nThe Retainer module (TSP + ARP) consists of small 1-2 layer feed-forward networks with **fewer than 0.8M parameters** (about 1% of BERT-base). The added inference cost is **negligible** compared to the Transformer backbone. However, it is true that the pruning benefits may be smaller for shorter sequences. Nonetheless, the inference time is still reduced, and even for short sequences, there is **some benefit**, as shown in Table R2.\n\nWe appreciate the reviewer’s suggestion regarding wall-clock latency benchmarks. As our focus has been on algorithm design and FLOPs reduction, deployment on real edge devices (e.g., mobile CPUs) is still in progress. We have conducted preliminary tests on an RTX 3060 GPU. As shown in Table R2, despite the added modules, AD-TP reduces end-to-end latency by **30–55%** compared to BERT-base on SST-2, MRPC, and SQuAD, especially for inputs ≥64 tokens.\n\n**Table R2: Inference Latency on RTX 3060**\n\n| Dataset   | Input Length | BERT-base (ms) | AD-TP (ms) | ↓ Gain   |\n|-----------|---------------|----------------|------------|----------|\n| SST-2     | 128           | 9.7            | 4.5        | 51.5%    |\n| MRPC      | 64            | 6.1            | 3.8        | 37.7%    |\n| SQuADv2.0 | 512           | 21.5           | 12.3       | 42.8%    |\n\nWe agree that mobile/edge deployment is important and plan to evaluate AD-TP on mobile CPUs in future work.\n\n***W3: Unexplored Federated Learning Implications***\n\nWe thank the reviewer for the insightful comment on the applicability of AD-TP in federated learning (FL). While this paper primarily focuses on centralized training, we agree that its adaptive design aligns well with data heterogeneity in FL. Relevant strategies have been detailed in **Q1** and **W1**. We again appreciate your suggestion, which helped broaden the scope of AD-TP beyond centralized settings.\n\n***W4: Architectural Specificity***\n\nWe thank the reviewer for this insightful comment. Although ARP relies on the [CLS] token in BERT, as mentioned in **Rebuttal Q2**, it can be adapted to architectures like GPT or T5 using alternative methods such as **mean pooling, attention pooling, or using the last token**. We plan to continue evaluating the performance of these methods in future work and further optimize their applicability.\n\n***W5:  Syntactic errors***\n\nWe thank the reviewer for pointing out the typo in Figure 4's caption. “Adaptive Aoken Aetainer” will be corrected to “Adaptive Token Retainer” in the revision. We have also checked the full manuscript to ensure no similar issues remain. We appreciate your attention to detail."}, {"review_id": "IotqnG1yFi", "reviewer": "Reviewer_tGep", "summary": "This paper introduces AD-TP, a novel method for token pruning in Transformer models designed to enhance computational efficiency. The authors argue that existing methods are suboptimal because they often rely on attention scores to gauge token importance, which may not correlate well with a token's actual contribution, and they use static pruning ratios that don't adapt to the varying complexity of different input sequences. To address this, AD-TP makes two main contributions. First, it uses Integrated Gradients (IG), an attribution method, to obtain a more accurate measure of each token's importance to the final prediction. Second, it introduces a lightweight \"Adaptive Token Retainer\" module that dynamically predicts an appropriate token retention ratio for each individual input sequence. This retainer is trained using a combination of teacher-student knowledge distillation (on both logits and IG-based attribution scores) and a self-supervision objective that is gradually phased in during training. Experiments on GLUE, SQUAD, and 20News datasets show that AD-TP significantly reduces computational costs (measured in FLOPs) while maintaining or even improving model accuracy, outperforming existing state-of-the-art token pruning techniques.", "questions": "Please see the weaknesses described above.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper's core ideas (use of Integrated Gradients (IG) for supervising token importance) are novel and well-justified. The concept of an adaptive pruning ratio, dynamically determined for each sequence, is a key strength. The paper effectively argues that a one-size-fits-all pruning strategy is inefficient, given the wide variance in sequence length and complexity.\n\nThe architecture of the Adaptive Token Retainer, split into a ratio predictor (ARP) and a saliency predictor (TSP), is intuitive and addresses the \"how many\" and \"which\" questions of pruning separately.\n\nThe paper presents extensive experiments that demonstrate the effectiveness of AD-TP.\n\nThe paper is very well-written and organized.\n\nWeaknesses:\n\nThe proposed method uses a teacher model and computes Integrated Gradients during training. This process is likely to be more computationally expensive and time-consuming than training methods that rely on attention scores, which are obtained almost for free during the forward pass. A discussion or quantitative analysis of this training-time overhead would provide a more complete picture of the method's trade-offs.\n\nAs the authors acknowledge in the limitations, pruning is only applied to the input embeddings. This means the quadratic complexity of self-attention is applied to the reduced sequence in all subsequent layers. While this is a valid and effective approach, it misses the potential for even greater savings from layer-wise pruning.", "was_revised": true, "final_justification": "My final justification stays the same as above original review.", "rebuttal": "We sincerely thank the reviewer for the encouraging and detailed feedback, especially the recognition of our method's **“novel and well-justified ideas”**, the **“intuitive architecture”** of ARP and TSP, and the **“extensive experiments”**. We also appreciate the comment that the paper is **“very well-written and organized”**. For each concern raised, we provide specific responses below.\n\n***W1: Regarding Training Cost***\n\nThank you very much for this insightful and constructive comment. Introducing a teacher model and computing IG during training does introduce additional computational cost compared to methods based solely on attention scores, which are available \"for free\" during the forward pass. This is indeed a meaningful trade-off that deserves further discussion.\n\nWe would like to clarify the following points:\n\n1. The teacher model and IG are used only during the **training phase**; during inference, our method relies entirely on the lightweight **TSP** and **ARP**, with no need for backpropagation or teacher guidance, which does **not affect inference efficiency and deployment**.\n\n2. Following your suggestion, we evaluated the training time for the eight GLUE tasks on an **NVIDIA RTX 3060 GPU**. As shown in **Table R1**, compared to BERT-base training without IG and the teacher model, the overall training time increases by about **20–30%**. We believe this is a reasonable cost, especially considering that AD-TP significantly reduces FLOPs and improves computational efficiency during inference.\n\n**Table R1: Training Time Comparison between AD-TP and BERT-base on GLUE**\n\n| Dataset | Epochs | Batch Size | BERT-base Time (h) | AD-TP Time (h) |\n|---------|--------|-------------|---------------------|----------------|\n| CoLA    | 20     | 32          | 0.83                | 1.17           |\n| MNLI    | 30     | 16          | 33.00               | 40.50          |\n| SST-2   | 30     | 32          | 3.50                | 4.33           |\n| QNLI    | 30     | 32          | 7.08                | 8.67           |\n| MRPC    | 30     | 16          | 0.25                | 0.33           |\n| QQP     | 30     | 16          | 22.67               | 28.83          |\n| RTE     | 30     | 16          | 0.17                | 0.20           |\n| STS-B   | 10     | 16          | 0.20                | 0.25           |\n\n3. Unlike attention scores, which can be affected by structural or positional biases, **IG provides attribution signals more directly tied to model predictions**, making it a stronger basis for saliency labeling. To validate this, we conducted an **ablation study** by replacing IG with attention-based heuristics. As shown in **Table R2**, IG supervision leads to higher saliency prediction accuracy.\n\n**Table R2: Comparison of the Impact of IG and Attention on Model Performance**\n\n| Dataset | Attention | IG   |\n|---------|-----------|------|\n| SST-2   | 90.2      | 92.0 |\n| MRPC    | 87.6      | 89.8 |\n\n***W2: Regarding the limitations***\n\nWe fully understand and agree with your concern regarding this limitation. \n\nIn this work, we chose to apply pruning only at the input layer, primarily for the sake of **stability and controllability**. Pruning in intermediate layers often requires cross-layer coordination, which can lead to training **instability** or **interfere with semantic representations**.\n\nNevertheless, our method still achieves **significant inference acceleration** and **competitive accuracy** on multiple standard benchmark datasets (see Tables 1, 2, and 3), indicating that the TSP and ARP modules are **effective** under the current setting.\n\nMoreover, our framework is **modular** by design, and the TSP module can be naturally **extended** to support token pruning in intermediate layers. We are actively exploring this direction and plan to **incorporate cross-layer pruning strategies** in future work to further improve efficiency.\n\nWe believe this is a promising direction for further research, and we sincerely appreciate your valuable suggestion."}]

Author rebuttal:

Meta-review: The paper provides a method for adaptively pruning tokens to reduce the computational burden of transformers. The reviews provided positive feedback on the high-level idea and its implementation. They mentioned it to be novel, e.g., tGep “The paper's core ideas (use of Integrated Gradients (IG) for supervising token importance) are novel and well-justified”, QJNG “The core innovation of using Integrated Gradients (IG) instead of attention scores as the basis for pruning is a significant and well-justified step forward.”, and well-motivated (tGep “The architecture of the Adaptive Token Retainer, split into a ratio predictor (ARP) and a saliency predictor (TSP), is intuitive...”, vH7T “the idea of estimating sequence-level difficulty and assigning an optimal pruning ratio for each sequence is technically sound.”). In terms of the empirical bottom-line results, the reviews were overall positive, e.g., tGep and QJNG mentioned they were impressed by the improvement over the baselines. One review (joVc) did mention the benchmarks were somewhat dated, but this issue was resolved during the rebuttal when the authors provided additional experiments on more recent benchmarks with additional baseline methods. 

One weakness that was not fully resolved is that of the writing quality. Both vH7T and joVc had several comments about unclear details in the paper making it difficult to both understand and reproduce the results. The authors provided clarifications in the rebuttal that seem to mostly mitigate the issues, but these should be integrated into the paper for its final version.  

To conclude, the main weakness of clarity seems to me to be possible to solve without another round of reviews. Considering the mentioned positive aspects of the paper, I believe it will be a welcome addition to NeurIPS.
Decision: Accept (poster)