TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching
Authors: Runjia Zeng
Qifan Wang
Qiang Guan
Ruixiang Tang
Lifu Huang
Zhenting Wang
XUELING ZHANG
Cheng Han
Dongfang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Memory Efficient Fine Tuning
Primary area: foundation or frontier models, including LLMs

Abstract: Fine tuning has been regarded as a de facto approach for adapting large language models (LLMs) to downstream tasks, but the high training memory consumption inherited from LLMs makes this process inefficient. Among existing memory efficient approaches, activation-related optimization has proven particularly effective, as activations consistently dominate overall memory consumption. Although prior arts offer various activation optimization strategies, their data-agnostic nature ultimately results in ineffective and unstable fine tuning. In this paper, we propose TokenSeek, a universal plugin solution for various transformer-based models through instance-aware token seeking and ditching, achieving significant fine-tuning memory savings (e.g., requiring only 14.8% of the memory on Llama3.2 1B) with on-par or even better performance. Furthermore, our interpretable token seeking process reveals the underlying reasons for its effectiveness, offering valuable insights for future research on token efficiency. Homepage: runjia.tech/iclr_tokenseek.

Published as a conference paper at ICLR 2026
TOKENSEEK: MEMORY EFFICIENT FINE TUNING VIA INSTANCE-AWAL TOKEN DITCHING
Runjia Zeng', Qifan Wang", Qiang Guan', Ruixiang Tang', Lifu Huang', Zhenting Wang", Xueling Zhang', Cheng Han', Dongfang Liu"!
Rochhster Institute of Technology
Meta AI *Kent State University *Rutgers University 'University Kassouri-Kansae City 'Corresponding author
UC Davis
AAcenttree
ABSTRACT
Fine-tuning has been regarded as de facto approach for adapting large language models (LLMs) downstream tasks. However, the high training memory consump- tion inherited from LLMs makes this process generally inefficient. Among existing memory efficient approaches, activation-related optimization has proven particu- larly effective, activations consistently dominate overall memory consumption. Although prior arts offer various activation optimization strategics, they typically adopt uniform yet inflexible strategy across all instance. This data-agnostic nature ultimately results in ineffective and unstable fine tuning. To solve this problem, we propose TOKENSEEK, universal plugin solution that suitable for various Transformer-based models through instance-award token seeking and ditching. To- KENSBEK achieves significant fine-tuning memory savings (e.g., requiring only 2.8 GB, 14.8% of the original memory on Llama3.2 IB) with on-par even superior performance. Furthermore, our interpretable token seeking process reveals the underlying factors behind its effectiveness, offering valuable insights for future research on token efficiency fine-tuning. Homepage: runjia.tech/iclr_eoo tokenssee
INTRODUCTION
Parameters
II. Gradients & Optimizer States
Acttvvittis
TokenSeck (Ours)
ww/ LoHa
QLoRA
Performance
50%
87%
Token Tiken (Random)
51
w/ LoHta
w/ QLoRA
(3%)
LoHa
Optimiser
(34%)
389
Loss (2%)
11%
Optinizer
Gradient (3%)
QLoRA
Grodicnis (1255)
40
Baseline
Batch Size:
Batch Size:
10%
Memory
100%
(a) Training Memory Breakdown Llama3 8B
(b) Performance-Memor Comparison
Figure Motivation behind TOKENSEEK and its preliminary comparison. (a) Breakdown of training memory under different batch side settings, revealing that activations are the primary bottleneck in training memory consumption. (b) Effective and efficient TOKENSEEK (ours) us. concurrent arts in performance and memory consumption Llama3.2 1B (detailed results Tab.
Sestraiinthen--ineetee paradigm (Liu al. 2024a Yang al. 2024 Grattafiori et al.) 2024 has
been regarded as de facto approach for downstream task adaptation, leveraging the knowledge acquired during pre-training. However, fine tuning large language models (LLMs) still imposes significant memory demands arising from multiple components (Zhang & Su 2025 Rajbhandari et af., 2020) as showin in Fig. [ (a), including the model parameters, II. gradients and optimizer states, and intermediate III. activations. Current works optimize training memory usage by targeting different components. Parameter-Effice
Fine-Tuning (PEFT) (Zeng et al. 2024 Han et al. 2023 2024) reduces the number of tunable parameters required for adapting large models (component I). Fine-Tuning improving the efficiency of gradient updates and optimizer states to alleviate the training memory burden (component II). Memory- Efficient Fine- Tine-Tuning (MEFT) (Simoulin et al., 2024 Dettmers et al. 2023) on the other hand, improves memory efficiency by recomputing, compressing,
Optimizer- Sppimizer-Efficientt Fine- Tuning (Rajbhandari et al. 2020: Anil et al. 2019) focuses on partitioning

---

Published as a conference paper at ICLR 2026
eliminating activation-related memory costs (component IIII. Among the three paradigms that address the memory challenge from different perspectives, MEFT stands out as a more effective one. The reason is that activations consistently dominate memory consumption (e.g., 87% in Llama3 8B as shown in Tab. (a), and 60GB of activations for GPT-2 1.5B (Rajbhandari et al. 2020). making them critical bottleneck in the memory efficiency of training deep models (Zhang & Su 2025
However, existing MEFT methods generally unaware of or ignore the abundant information contained in the fine tuning training instances, i.e., they operate as data-agnostic optimizations. Previous works are predominantly model-oriented optimizations (see $2 - they adopt a uniform efficiency strategy across all instances, without accounting for the rich variability inherent within each individual instance. This results in a lack of fine-grained control over memory reduction at the instance level, leading to ineffective (see Fig. (b) and $4.2) and unstable fine-tuning (see (4.3). Naturally, two key challenges arise on the path toward instance-aware activation efficient optimization: I. how to identify the salient tokens that represent the key information of each instance (solved through $3.2.1 and how to leverage them to achieve effective and stable memory optimization (solved through 3.2.2).
In light of this view, we introduce TOKENSEEK, a universally applicable plugin designed to achieve win-win of performance and memory efficiency without altering their inherent architecture under the "Pretrain- Mienn-Finttuee pradigm. In order to kill two birds with one stone, our approach can be unfolded into two aspects to respectively tackle the challenges above: ! Instance-Aware Token Seeking. TOKENSERK first leverages context and gradient information at the token level to evaluate and score individual tokens, selectively retaining more informative ones to mitigate performance degradation and fluctuation. @ Efficient Token Ditching, TOKENSEEK then significantly decreases the memory footprint for activations by updating model parameters exclusively on selected tokens, thereby ditching the gradients of the others and thus eliminating these activations. Our method facilitates an adaptive, instance-awan aware activation optimization without compromising performance and stability (see more discussions in (S7). Our key contributions include:
Significant Memory Reduction: Benefit from the potent instance awareness, TOKEN can achieve substantial memory savings with only 10% tokens (i.e., 65.7% maximum memory reduction on Ilama3.2 IB, see 442) while maintaining competitive performance (i.e., 41.13 us. 40.82). Our approach can further significantly surpass full token fine-tuning with only 14.8% memory consumption under the QLoRA settings (i.e., 52.61 us. 40.82 shown in Fig. (b)).
Generalizable Solution: Attributed to its architecture-agnosti design, our method generalizes well across various munsformer-based models (i.e., Qwen-0.5B, Llama- 1B and Llama-3B) and can be seamlessly integrated with other PEFT techniques (i.e., LoHa and QLoRA) to embrace both performance effectiveness and memory efficiency (see Tab. D.
Interpretable Token Seeking: We provide a comprehensive analysis (see (4.3) of how token-level ditching influences the fine-tuning process, achieving significant memory reductions through our proposed transparent and explainable token selection strategy (see $3.2 and $4.3).
2
RELATED WORK
2.1
MEMIIENEIICIIET FINE-TUNING
MEPT (Simoulin et al. 2023 Vucetic al. 2022 Ryu et al. 2024 Zhang et al. 2023 Zhao et al.) 2024 Ardakani et al. (023) directly tragets on reducing memory footprints during fine tuning. It can be broadly categorized into the recomputation, compression, and reversible network paradigms.
Recomputation. The core idea of recomputation methods (Korthikanti et al. 2023 Chen et al. 2024b) Tang et al. 2024) recompute certain operations instead of storing all intermediate activations a technique also known as gradient checkpointnng (Chen et al.) (016) first applied this idea to deep neural networks, proposing method that stores only a subset activations and recomputes others during the backward pass, achieving sublinear memory cost. Subsequent improvements optimized the checkpointing schedule (Jain et al. 2020), introduced dynamic runtime strategies (Kirisame) et al. 2020), and combined offloading with recomputation (Rajbhandari et al.) 2020). Compression. Compression methods (Yi et al. 2024| Yang et al. 2025 Leconte et al.) (224)) focus on reducing the size of the model states, optimizer states, gradients, and activations, which can further divided into methods using sparsified and quantized representations. Specifically, sparsity methods include LoRA
(Hu et al. 2022ab, which freezes pre-trained weights and trains low-rank adapters, and diff pruning
2

---

Published as a conference paper at ICLR 2026
(Guo et al. 2020), which learns sparse task-specific updates. Recently, TokenTune (Simoulin et al., 2024) reveals the feasibility of token pruning during backpropagation to further reduces memory by selectively dropping token activations. Quantization-based methods, on the other hand, use lower numerical precision to minimize memory usage. Mixed-precision training (Micikevicius et al. 2017) with FP16 or BF16 became standard, and further advancements introduced 8-bit optimizer quantization. QLoRA (Dettmers et al., (023) extends this by applying 4-bit quantization to model weights during fine-tuning. Reversible Networks. Reversible network designs eliminate the need to cache activations during training by reconstructing them from outputs. RevNets (Gomez et al. 2017) demonstrated reversible residual blocks for image models, while Reformer (Kitaev al. 2020 extended this idea to Transformers with reversible layers. Recent methods adapt reversible computation to fine-tuning pre-trained models by inserting reversible adapters (Liao et al. 2023b), significantly reducing activation memory without modifying the pre-trained weights.
TOKENSEEK, a sparsified gradient updating method under compression paradigm leverages both context and gradient information in each sample to enable instance-awan activation sparsification with performance on par with dense models, bridging the performance gap.
2.2 CHAAMETER-ErICUnN
-UN--UGNIG
PEFT (Hu et al. 2022b] Aghajanyan et al. 2020 Yang et al. 2023] Huang et al., 2023 Zadouri et al., 2023) aims to optimize model parameter usage and thus can reduce memory consumption to varying
degrees. It can be generally categorized into four paradigms (see more in SS7..
Purtial Tuning methods (Lawton et al. 2023 Xu al., 2021 update only a subset of the backbone model parameters using weight masking partial tuning strategies. A common strategy is to fine-tune only the final few layers or sorely the output head. However, its simplistic strategy may directly result in performance degradation, motivating further research into targeted masked tuning
approaches (Sung et al.) [021] Chen et al. 2024a Liao et al.) (023a). Additional Tuning methods introduce small number of new parameters
frozen pre-trained model, fine-tuning only the added modules. These methods can be further categorized into adapter-based (Houlsby et al. 2019
Pfeiffer et al.] [2020] Wang et al., 2022a) and prompt-based approaches (Jia al. 2022 Wang
et al. 2024, 2023), which inject lightweight learnable modules into the model architecture or the model imput, respectively. Reparameterized Tuming methods reparameterize the model updates in a low-dimensional subspace (Liu et al., 20256), leveraging the low intrinsic dimensionality of LLMs. LoRA (Hu al. (022a) learns low-rank matrices to model weight updates without modifying the original weights. Subsequent works have extended LoRA to alternative reparameterization variants (Hycon-Woo et al. (222)) and incorporated quantization techniques for additional memory savings (Dettmers et al. 2023). Hybrid Tuning methods (He et al. 2021 Zhang et al. (2024) combine multiple
PEFT strategies, aiming to unify their advantages. UnIPETT (Mao et al., (021) stands out as a
representative method that jointly incorporates adapters. prefix tuning, and LoRA-style low-rank updates as submodules, and learns to activate those best suited to the current task via gating.
While PEFT methods primarily focus on parameter efficiency (i.e., reducing component I storage), their impact on overall memory efficiency is limited (e.g., the activation memory of most PEFT methods remains over 75% of that in full fine-tuning, even with less than 1% trainable parameters (Liao et al. 2023b)). Leveraging our architectue-agnostic design, TOKENSEEK can be seamlessly integrated with PEFT methods, further embracing both parameter and memory efficiency (see Tab.
3 METHODOLOGY
In 3.1 we first analyze activations, the primary bottleneck in training memory, from two perspectives: (i) why storing activations necessary, and (ii) why they ineur large memory consumption in LLMs.
Our method, TEKENSEEK, is presented in $3.2) which is decomposed into two key components: instance- aware token seeking and efficient token ditching. The overall framework shown in Fig. 2
3.1 PRELIMINARY
The Necessity of Storing Activations. Given multilayer deep neural network, we first analyse the memory consumption of activations, in which the transformation and nonlinear activation at layer / are defined by ()) (0))))())) 600 and z(!) (a)") respectively. Here, the weight matrix (')) projects the output ((-1) of the previous layer into the current layer's 's pre-activation a to
which we add the bias b before applying the activation function a. By extending to deeper layers
3

---

Published as a conference paper at ICLR 2026
Forward- Backward
Acciires goad
00000000
Das Not Regaie gra!
Random
Selection
Token Selection
0
Ceae TAMIOO NY
0
0
Data-agnostic!
Context Information
Gradicnt Information
Tranafernet Layer
bulla
Cea't anything!
Token Evaluation:
OO
Tanndorme Lastr
Unstable!
00000000
Ineffective!
Instunce-att aware Taken Seeking
Efficient Ditching
(c)
Figure 2: Overview of TOKENSEEK (ours) us. TOKENTUNE frameworks. (a) Instance-awand
token seeking using context and gradient information (see $3.2.1 and Eq. (5). (b) Efficient token
ditching (see $32.22. (c) TOKENTUNE for random token selection (see analysis Tab, and
and applying the differentiation rules along with the chain rule, we can decompose the gradient with respect to the weight in the first layer in simplicity as:
Oc
OC
0g(0)
dz(1)
III
OW())
02(l)
12(1-1)
awa)
020)
0zl)
0a
o' da(")) w
(2)
0-(1-1) Oali)
0z(1-1
The computation of the back-prop term o' da(a") Wil) requires the intermediate value ( to further evaluate o' )). By caching each pre-activation (1)) during the forward pass, the model can avoid recomputing to obtain these intermediates, thereby efficiently forming the full chain of derivatives.
The Reason of Large Activations. Current censsformer-bsedd LLMs follow this rule to store activations during backpropagation. Taking DeepSeek-v3 (Liu al. 2024a Zhang & Su 2025) as an example, the activations in each layer have space complexity of O(Bm, BBH), where B the batch size, n4 is the number of attention heads, s the sequence length, and is the hidden dimension. The space complexity required for activations significantly outweighs that of the weights (Bn,s2 BsH > H² given B 1, 128, s 4096 and H 7168 (Zhang Su 2025)).
3.2 TOKENSEEK
3.2.1
ASTANCE-AWARI TOKEN SEEKING
The key insight behind TOKENSEEK is that not all training tokens within LLMs contribute equally to model fine-tune fine-tuning, known as token redundancy. Token redundancy has long been recognized as fundamental challenge to L.M efficiency (Hou et al. 2022), drawing increasing research attention across various domains, including efficient chain-of-thougg
reasoning (Xia et al. 2025) and prompt optimization (Li et al. 2023). This observation greatly inspires us to explore the potential of memory- efficient fine-tuning by reducing token redundancy. Then the critical problem turns to determine the importances of each token. Here, we propose a comprehensive evaluation of tokens by leveraging both context and gradient information captured within Transformer blocks (see Fig. (a)).
Context Information. Most LLMs are built upon the Transformer architecture, which funda- mentally relies the attention mechanism (Vaswani et al. 2017). The attention maps in decoder layers directly reflect the importance of each token in context, thereby guiding and shaping the transformation process. More specifically, given input embeddings into queries and keys with learnable matrices Wº,, W e ddxd to obtain Q tWQ, K = K We then form the attention scores and apply causal mask for language modeling, computing
Raxdd first project them
A softmax mask QK" / /Vdx) Each entry Au denotes the attention weight from token to token j. Owing to the row-wise softmax normalization, each row A₃ forms probability distribution over all tokens, capturing how much attention token allocates to others. Conversely, each column
4

---

Published as a conference paper at ICLR 2026
A. reflects the cumulative attention received by token from all other tokens in the sequence. In this way, attention mechanism provides an intuitive and direct measure a token's ' importance within given instance. The context importance of each token (Singh al. 2024 Liao et al. 2025 Kong et al. 2023 is computed as:
(4)
Σ Aij.
Gradient Information. However, context-based evaluation above only reflects the importance of a token within a given instance and does not necessarily indicate its contribution to model fine-tuning. Therefore, in order to better quantify the contribution of each token during the fine tuning. we further assess token importance by examining the gradient magnitude of the loss w.r.t. tt the activations. This idea is inspired by (Jain & Wallace, 2019), which shows that attention weights with gradient-based measures of feature importance. The study positions gradient-base as a more reliable yardstick of "true" token importance, highlighting that gradient- srddient-based saliency can substantially differ from attention-based explanations (see more discussion in $4.3). Given the gradient matrix G = [aC/O=(L-D] E IRadd for the activations in the penultimate layer (i.e., the input to the final decoder layer) computed during backpropagation, the gradient-based importance of each token is computed by summing the gradient magnitudes across the hidden dimension as:
often uncorrelated attribution
OC
I2 (tg) = Accumulatee 0z(L-13
Accumulate
Gyk-
Am
Token Evaluation. To obtain a comprehensive evaluation of token importance, we integrate both context and gradient information, weighted by scalars and 3, respectively
I(tz) = a log [T(s)] + Norm [Z)(t;]]
(5)
where apply log-like transformation to address the long-tail distribution of contextual importance scores, and use min-max normalization to scale the gradient-based importance scores to a comparable range (see more discussions in $4.3). By incorporating both information, our method is able to robustly evaluate tokens within each instance (distinct from random selection in Fig. (c)). leading to more effective and stable fine-tuning (see and $4.3). This enables us to select the tokens that contribute most to model fine tuning in the subsequent memory-efficient token ditching (see (32.2).
3.2.2 EFFICIENT TOKEN DITCHING
To improve training memory efficiency, we propose ditching the gradients of less informative tokens from the dataset, and fine-tuning LLMs using only the selected tokens. Following (Simoulin et al. 2024), by backpropagating the loss through the selected tokens in t only, and ditching the gradient computation of unselected tokens f Eq. 2 (see Fig. (b)) after regrouping them, we have:
0z 12(1-1)
= d'aa o' (a W = o''(aa))) 0 wW.
(6)
Based on Eq. we only need to cache of to apply the chain rule, rather than storing the full We provide a detailed discussion S2 and $S7.1
activation (1)
3.2.3 ANALYSIS AND DISCUSSION
As shown in $32..1 contrast other token importance evaluation strategies that require additional annotations or auxiliary networks (Xia et al. 2025), TOKENSEEK only requires forward pass (e.g.. Llama3 8B requires only 13.3% the training memory during inference under the FP8 setting) and partial backward pass (i.e., freeze all layers except the output head and the final decoder block) to assess token importance, resulting in simplicity. Presented in $3.2.2 tuning only 10% of tokens theoretically requires just ~1% of the activation memory, based the space complexity analysis in 33.1 resulting in a highly efficient fine-tuning process. Beside enjoying the appealing characteristics of simplicity and efficiency, TOKENSEEK also has merits in generality and interpretability. For generality, our method relies solely on context and gradient information, making architecture- agnostic and broadly applicable a more wide range of Transformer-based models (see Tab For interpretabiliry, both context and gradient information provide intuitive and direct evaluations of token importance, thereby further enhancing the interpretability of our evaluation process (see $4.3)
5

---

Published as a conference paper at ICLR 2026
Table Few-shot evaluation on question-answering benchmarks. This includes ARC (25-shot). MMLU (5-shot), HellaSwag (10-shot), TruthfulQA (0-shot). and WinoGrande (0-shot). We report the average accuracy on five MMLU ethics tasks and WinoGrande, the normed accuracy on ARC and HellaSwag, and the MC2 score TruthfulQA. The number reported in [-] is the "Tuned/Total" parameters in each setting. The same training settings are highlighted in blue, red, and orange ge for full-parameter tuning, LoHa, and QLoRA, respectively. Relative memory consumption percentages compared with the settting of full token tuning are transformed and reported in each scale. Same for Tab We highlight the best average performance and memory savings in bold. For TOKEN TUNE and TOKENSEFK, only 10% of the input tokens are selected for gradient computation.
Method
Ave.
Max.
ARC
Hella Swag
MMLU
Truthful QA
Wino Grande
Average Score
Mem.
Mem.
Qwen2.5 (0.5B)
Full Parameter/Toke Token Tuning
100%
100%
34.89
51.70
59.20
39.86
56.51
48.43
w/ TOKENTUNE (Random)
48.3%
25.6%
25.26
25.78
$1.07
49.93
47.36
39.88
TOKENSEEK (Ours)
48.3%
25.6%
25.17
25.52
58.14
50.13
50.75
41.94
IA3 [Lju
84.3%
72.8%
34.98
51.66
56.81
40.08
56.51
48.01
LoRA
20223
81.2%
71.8%
51.67
56.30
41.08
56.51
48.06
LoKr
Hyeun- Woo el m
2021
91.6%.
79.3%
35.49
51.54
58.64
39.83
55.88
48.28
BOFT
[Lu at 20245
145.1%
100.6%
34.64
$1.70
58.18
39.57
56.43
48.
Kang 20/24]
43.54
42.48
LoHa
[1.33%]] (Hycon- Woo al.
2021
86.6%
76.9%
34.73
51.90
$7.53
40.75
55.96
48.17
w/ TOKENTUNE [Kandom)
39.5%
22.5%
23.81
26.34
$7.53
50.26
47.36
41.06
TOKENSEEK (Ours)
39.5%
22.5%
26.54
25.96
58.14
50.26
50.51
42.28
QLoRA [1 [1.04%] (Dettmers
2023
51.7%
45.6%
34.64
50.10
$8.05
40.41
55.09
47.66
w/ ToRENTUUE ( (Kandom)
19.2%
13.4%
31.06
45.92
$7.60
41.56
55.56
46. 34
w/ TOKENSEEK (Ours)
19.2%
13.4 %
34.56
50.09
$7.52
41.51
58.56
48.45
Llama3.2
Full Karumeter/Token Tuning
100%
100%
23.72
26.11
57.53
48.68
48.07
40.82
w/ TOKENTUNE (Random)
64.6%
34.3%
24.32
25.80
58.14
47,90
47.59
40.75
w/ TOKKNSEEK (Ours)
64.6%
34.3%
23.98
25.73
58.14
48.09
49.72
41.13
LoHa [0.63%)
92.3%
99.4%
65.93
$7.60
37.87
60.77
$2.28
w/ TOKENTUNE (Random)
45.9%
28.4%
38.48
64.21
50.34
43.89
59.91
51.37
w/ TOKENSEEK (Ours)
45.9%
28.4%
38.57
65.89
$8.18
39.34
60.93
$2.58
QLoRA [0.52%]
45.6%
34.8%
38.82
65.26
56.39
38.85
61.33
52.13
TOKENTUNE (Random)
14.8%
14.3%
39.33
62.97
41.76
41.36
60.69
49.22
w/ TOKENSEEK (Ours)
14.8%
14.3 %
39.08
65.98
58.03
38.65
61.33
52.61
Llama3.2 (3B)
Full Parameter/Tokee Tuning
100%
23.98
25.72
58.62
49.53
49.80
41.53
TOKENTUNE (Random)
73.1%
39.3%
24.15
25.43
57.64
50.86
47.91
41.20
w/ TOKENSEFK (Ours)
73.1%
39.3%
27.30
25.96
58.14
48.65
49.72
41.95
LoHa [0.47%;
90.7%
96.5%
49.06
75.96
63.50
42.08
69.46
60.01
TOKENTUNE (Random)
30.0%
75.70
$6.65
43.37
69.61
59.13
w/ TOKENSEEK (Ours)
49.6%
30.0%
53.24
76.81
64.31
41.75
68.98
61.02
QLoRA [0.42%)
33.6%.
26.5%
51.37
75.88
63.95
42.31
68.43
60.39
TOKENTUNE (Random)
73.04
ToKeNšEEK (Ours)
13.3%
11.1%
50.00
76.30
63.43
43.37
68.98
60.42
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Under the Pretrain-then-Fiieee datasets to adapt them to domain-specific tasks and improve instruction-followiin capabilities (Wang
paradigm, pre-trained LLMs are further fine-tuned on specialized
al. b022b Taon al. 2023). In this section, apply instruction tuning and benchmarking under the sho-shot setting. We provide additional experimental details in $SI and $7
Instruction Tuning. Following (Simoulin al. 2024). we fine-tune the Qwen2.5 0.5B Yang et al.
2024 Llama3.2 IB and 3B (Grattafiori et al. 2024) models using the Plan-PPatypus dataset (Lee
ct al.
2023). comprises 11 open-source instruction datasets. See more details in $S1.7
Few-Shot Evaluation. We assess performance across few-shot benchmarks including MMLU (Hendrycks et al. 2020), ARC (easy and challenge) (Clark et al.) 2018). HellaSwag (Zellers et al. 2019), TruthfulQA (Lin et al., 2021), D, and WinoGrande (Sakaguchi et al.) (021), using the "Im evaluation harness" (Gao et al. (024). For each task, the model ranks answer options by probability, and the highest one is selected. See more discussions and experiments in ST $ and $S7.1

---

Published as a conference paper at ICLR 2026
4.2 MAIN RESULTS
The main performance and memory comparison results across various models, scales and PEFT settings are shown in Tab. _ leading to three key observations. First, Significant Memory Reduction. TOKENSEEK demonstrates exceptional efficiency in reducing both average and peak memory usage during fine-tuning, Specifically, for the Llama3,2 2 3B model, peak memory usage is reduced by 60.7% with TOKENSERK alone (i.e., Llama3.2 2 3B + TOKENSEEK), and further down to just 11. % when combined with QLoRA (i.e., Llama3.2 3B QLORA+ + TOKENSEEK), enabling training on single A A10 GPU without triggering OOM issues. Similarly, average memory usage is reduced by 26.9% and 86.7%, respectively. These results highlight TOKENSEEK 's strong capability in achieving extreme memory compression during fine-tuning (more experiments in $4.4). Second, Competitive or Superior Performance. Despite memory efficiency, TOKENSEEK maintains competitive even superior performance compared to full-token tuning: The average score of TOKENSEEK with QLoRA in Qwen 0.5B marginally surpasses the full token tuning baseline (48.45 us. 48.43). In Llama3.2 1B, TOKENSEEK consistently outerperforms baseline across all settings (i.e., TOKENSEEK, w/ LoHa, and w/ QLoRA achieve scores of 41.13, 52.58, and 52.61, respectively, compared to 40.82 from full token tuning.). This indicates that the memory compression achieved by TOKENSEEK does not compromise, and may even slightly enhance, model performance (detailed discussions in $4.3). Third, Generalizable Across Different Scales. Generally, TOKENSEEK'S ' effectiveness in both memory reduction and performance stability is generalizable actoss various model scales ranging from 0.5B to 3B. However, we observe a distinct pattern across model scales: TOKENSEEK exhibits performance degradation on Qwen under plain settings, whereas Llama does not. This suggests that TOKENSEFK may be more sensitive to smaller-scale models, likely due to their limited representational capacity. In conclusion, TOKENSEEK demonstrates itself as an universal memory efficient solution, effectively achieving memory efficiency with competitive model accuracy (see more comparison in S
4.3 ANALYSIS OF TOKEN SEEKING
Contributions of Instance-aware Token Seeking. We further conduct experiments to quantitatively eval- uate the impact of instance-aware token seeking (i.e., addressing the drawbacks of data-agnostic optimiza- tion: inefficiency and instability) by comparing per- formance across multiple runs under varying memory settings (i.e., with the ratio of tunable tokens ranging from 10% 50%)
52.4
52-
Average Score
50,2
50-
Llama3.2 IB with QLoRA. As shown in Fig. 3. we observed that TOKENSEEK consistently enhances both effectiveness and stability. For effectiveness, our method (green curve). consis- tently outperforms the random baseline (purple curve) (Xia al. (025) across various memory settings. The accuracy achieved by our approach remains higher all memory saving levels, clearly demonstrating its effectiveness in optimizing performance under mem- ory constraints. For stability, our method showcases superior stability. This is evident from the notably narrower shaded regions (i.e., variance) in Fig. [3] in- dicating a lower standard deviation compared to the random baseline. The results clearly show TOKENSERK'S 's capability to maintain higher accuracy with narrower fluctuations, emphasizing its robustness under varying memory constraints.
TokenSeek (Ours)
TokenTTme (Random)
2.8
3.6
Memory Consumption (GiB)
Figure 3: The performance for the Llama3.2 IB with QLoRA setting, where the upper, lower and middle line indicate the maximum, minimum and the average results.
Study of Interpretability. One key advantage of TOKENSEEK the transparency and interpretability of its token seeking and ditching process. To illustrate this, we conduct a case study on training instance to visualize token evaluation and highlight common patterns (see more visualizations in $S8) TOKENSEEK leverages both contextual and gradient information for token selection.
Context Information: while token selection varies across instances, our analysis reveals several consistent patterns illustrated in Fig (b). Specifically, unidirectional attention enforces causality, resulting in an upper- tapeerttringular attention mask. Diagonal patterns indicate self-focused attention, emphasizing local context shown in the blue localized accumulation. Attention Sink (Xiao et al.) 2024) refers to the tendency of attention to disproportionately concentrate on a single position, effectively acting as global anchor (evident the brown line in first column). The context scores
7

---

Published as a conference paper at ICLR 2026
1.0
0.08
Selected Tokers by Context Seoce
Solected Tokens by Grndiest Seare
Attention Sink
at Token 0
Below is an instruction that describes task, Write a response that appropriately completes the roquest.
WHitt Instruction:
box contains white balls and 6 black balls. Two balls are drawn out of the box random. What the probubility that they both white?
140
-0.00
140
140
(b) Attention Map
Gradient Map
0.08
Wiip Response:
80
There are binom (2) 555 combinations of two balls that can be
drawn. There are Sibinom (5) (2) - 10S combinations of two white bulls
that can be drawn, So the probability that two balls pulled out are both white is Sidfrac (10) (55) idfrue {11:S.
00.8
(a) Visualization Token Sceking
(d) Context Score
(e) Gradient Soore
Figure 4: Case study of a training instance. (a) Visualization of the top 50% selected tokens using context and gradient information, highlighted in red and blue, respectively. (b) Average attention map from the final layer. (c) Accumulated gradient map of activations in the penultimate layer. (d) Context importance scores obtained by column-wise accumulation. (e) Gradient importance scores obtained by summing across the hidden dimension. Additional visualizations are provided in $S8)
cxhibit a long-tail distribution caused by the attention sink effect, with higher scores concentrated in earlier positions-amplinn amplified by the causal mask (see zoom-in area of Fig (d)). This results in
preference for earlier tokens, as reflected in the oed-highlighted regions of Fig. - 4 (a), which retain semantically meaningful tokens (e.g., those related mathematical learning) while filtering out less informative ones, such as definite articles and prepositions. Gradient Information: to enhance
interpretabilityy we aggregate the gradient map along the hidden dimension, producing G' < T>" which is visualized color map in Fig (c). As illustrated in Fig (e), the gradient information predominantly focuses on later positions-tyyen typically corresponding to the "Response" portion of training instance (highlighted in blue Fig. (a)). underscoring the importance of learning the answer generation process. In summary, our findings highlight the interpretability of TOKENSEEK, showing that context and gradient information exhibit distinct but complementary patterns (see more in S7). Their integration enables more comprehensive and robust approach to token evaluation.
Study of Optimization. We further investigate the reason of superior performance under the PEFT settings and discuss the potential impact of the percentage of tunable tokens during fine tuning. By comparing different tunable token ratios within each group (i.e., lighter us. darker lines) and across different tuning strategies (i.e., different colored lines), have two key obser- vations. i) TOKENSEEK favors PEFT: Consid- cring PEFT methods (i.e., LoHa and QLoRA), we observe that full parameter tuning yields lower training loss, which may indicate potential overfitting (i.e., lower training loss but poorer downstream performance). In contrast, PEFT methods, which update only subset of param- eters, are less prone to overfitting and there- fore remain more robust and better suited for token ditching (i.e., 47.66 on QLoRA vs. 48,85 on QLoRA + TOKENSEEK). ii) Tokens Con- tribute Fine Tuning: As more tunable tokens are incorporated into fine-tuning (i.e., from 10% to 50%), we consistently observe lower opti- mization loss, which indicates that fine tuning is sensitive to training data volume. Low-quality token selection under extremely limited token training may thus lead optimization collapse and performance degradation, emphasizing the need for our instance-aware token seeking approach.
17.5
IokenSeck
TokenSeek 50%
wi Lolln 10%
w/ Lolla 50%
AN A MoMee
-w QL.RA 50%
Training Loss
Mat
10.0
65
64
63
60
7.5
2.5
1010
Steps
Figure 5: Training loss curves of six settings on Qwen2.5 0.5B. Blue, red, and orange lines rep- resent full parameter, LoHa, and QLoRA tuning, respectively. Lighter lines in each group indicate 10% token tuning, while darker lines indicate 50%. Detailed performance and memory usage results are provided in Tab. T and
8

---

Published as a conference paper at ICLR 2026
Table Ablation study on token evaluation in Eq.5 analyzing the impact of scalar weighting and threshold selection on both memory consumption and model performance (See more in S7
Ave.
Max.
Hella
Truthful
Wino
Average Score
Settings
MMLU
ARC
Mem.
Mem.
Swag
QA
Grande
Sensitivity to Scalar (Qwen2.5
0.5B with QLoRA)
a=1, 3=0
34.56
50.09
57.52
41.51
58.56
48.45
o=0, 3=1
30.72
44.20
57.62
43.98
55.41
46.39
5=5, 3=5
19.2%
13.4%
35.15
50.20
58.49
41.48
57.93
48.65
a=3, 3=7
34.64
48.77
58.33
42.18
57.46
48.28
a=7, 3=3
35.58
50.10
58.59
41.13
57.22
48.53
Ratio Tunable Token (Llama3.2
with QLoRA)
100%
100%
100%
23.72
26.11
57.53
48.66
48.07
40.82
50%
32.6%
53.1%
39.42
65.34
55.00
40.53
61.01
52.26
40%
28.0%
29.9%
39.42
65.51
56.92
39.89
61.56
52.66
30%
23.2%
23.7%
40.27
65.60
54.63
41.46
61.80
52.75
20%
18.8%
18.6%
39.42
65.73
53.24
39.86
60.77
51.80
10%
14.8%
14.3%
39.08
65.98
58.03
38.65
61.33
52.61
4.4 ABLATION STUDY
Sensitivity to Scalar. As stated Eq, 5 the scalars and / determine the relative emphasis placed on
40
Maximum memory of token tuning
context and gradient information during token evalua- tion, respectively. As shown in Tab. 2 configurations
52-
-86.7%
Memory Consumption (GiB)
TokenSeck
that balance both information (e.g., [5. 5)) achieve higher performance compared to those relying single evaluation (e.g., [0, IJ). Notably, TOKENSEEK performance remains stable across various settings, indicating low sensitivity within certain range.
QLoRA
20
Average memory of full token tuning...
Ratio of Tunable Token. We then investigate the ratio of tunable tokens w.r.f. model performance, particularly under scenarios with extremely limited gradient tokens. Tab. shows that TOKENSEEK main- tains stable performance across a range of ratios (i.e.. from 51.80 to 52.75), while memory usage decreases substantially as the ratio reduces (i.e., from 32.6% to 14.8%). Given the observed stability, suggest default ratio of 10% for overall efficiency. GPU Memory Impact. The motivation behind memory-efficient fine-tuning stems from the mis- match between limited GPU memory (e.g., 40GB on A100, 24GB on RTX 4090) and the growing size of LLMs. In this context, peak memory deter- mines whether model can be fine-tuned on single GPU without encountering OOM issues. Benefit from our model-agnostic design, TOKENSERK provides faithfully cumulative memory savings when combined with PEFT methods. For example, when integrated with QLoRA, the peak memory usage ranges from 14.2 GiB to low 5.5 GiB depending the ratio selected for tunable tokens substantially lower than the 38.8 GiB required by full token tuning.
88.9%
10
20 30
Ratio of Tunable Token (%)
Figure 6: Training memory under diff'erntt settings and token ratio selections. Bars rep- resent average memory usage, while dashed lines indicate peak memory consumption.
5 CONCLUSION
We propose TOKENSEEK, a universal plugin solution for effective and stable Transformer-basee -assed memory efficient fine tuning. It has merits in: i) significant memory reduction via token ditching without sacrificing performance; ii) strong generalizability across various LLMs and compatibility with existing PEFT methods; and fii) interpretable instance-aware seeking for effective and stable fine tuning. As whole, conclude that the outcomes elucidated in this paper impart essential understandings and thus necessitate further exploration within the field of MEFT.

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENTS
This research was supported by the National Science Foundation under Grant No. 2450068. This work used NCSA Delta GPU through allocation CIS250460 from the Advanced Cyberinfrasttrucure Coordination Ecosystem: Services & Support (ACCESS) program, which supported by U.S National Science Foundation grants No. 2138259, No. 2138286, No. 2138307, No. 2137603, and No. 2138296. We also gratefully acknowledge the support of RIT Research Computing (of Technology 2025).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint wwXiv:2303.08774 2023.
Armen Aghajanyan, Luke Zettlemoyer. and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, arXiv preprint arXiv:2012.13255 2020.
Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models. In ACL, 2020.
Paul Albert, Frederic Z. Zhang. Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. Randlora: Full-rank parameter-efficient fine-tuning of large models. In ICLR, 2025.
Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory-efficient adaptive optimization for large-scale learning. arXiv preprint arXiv:1901.111 4, 2019.
Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401. 16405, 2024.
Arash Ardakani, Altan Haan, Shangyin Tan, Doru Thom Popovici, Alvin Cheung, Costin lancu, and Koushik Sen. Slimfit: Memory-effiiiet fine-tuning of transformer-based models using training dynamics. arXiv preprini arXiv:2305. /8513, 2023.
Brian Bartoldson, Ari Morcos, Adrian Barbu, and Gordon Erlebacher. The generalization-staaliiy tradeoff in neural network pruning. NewrIPS, 2020.
Changyu Chen, Xiting Wang, Ting-En Lin, Ang Lv, Yuchuan Wu, Xin Gao, Ji- Ji-Rong Wen, Rui Yan, and Yongbin Li. Masked thought: Simply masking partial reasoning steps can improve mathematical reasoning learning of language models. arXiv preprint arXiv:2403.02/77 2024a.
Ping Chen, Wenjie Zhang, Shuibing He, Weijian Chen, Siling Yang, Kexin Huang, Yanlong Yin, Xuan Zhan, Yingjie Gu, Zhuwei Peng, et Optimizing large model training through overlapped activation recomputation. arXiv preprint ar arXiv:2406.087555 2024b.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 1604v 2016.
Peter Clark. Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: /803.0050 2018
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In NeurIPS, 2023.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URI httttss///andd..o///rerrrsss///
10

---

Published as a conference paper at ICLR 2026
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In NeurIPS, 2017.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21785 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463, 2020.
Cheng Han, Qifan Wang. Yiming Cui. Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E'pp:: An effective and efficient approach for visual prompt tuning. In ICCV, 2023.
Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, and Dongfang Liu. Facing the elephant in the room: Visual prompt tuning or full finetuning? ICLR, 2024.
Haoze He, Juncheng Billy Li, Xuan Jiang, and Heather Miller. Sparse matrix in large language model fine-tuning, arXiv preprint arXiv:2405. 15525, 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kerg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint arXiv:21 10.04366, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. 2020.
Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token dropping for efficient bert pretraining. arXiv preprint arXiv:2203.1 13240, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022a.
Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun. Sparse structure search for parameter-efficient tuning. arXiv preprint 07.30220 2022b.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13266 2023.
Nam Hyeon-Woo Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-effiieell federated learning. arXiv preprint 1xXiv:200806600 2021.
Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Checkmate: Breaking the memory wall with optimal tensor rematerializa- tion. Proceedings of Machine Learning and Systems, 2:497-511, 2020.
Sarthak Jain and Byron Wallace. Attention not explanation. In ACL, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Jean Kaddour. The minipile challenge for data-cfficient language models. arXiv preprint arXiv:2304.08444 08442, 2023.
Jiale Kang. Bone: Block affine transformation as parameter efficient fine-tuning methods for large language models. arXiv preprint arXiv:2409.1ss 15371, 2024.

---

Published as a conference paper at ICLR 2026
Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He. Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic tensor rematerialization. arXiv preprint arXiv:2006.096/ 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint wwti::2001.0445555 2020.
Zhenglun Kong, Haoyu Ma, Geng Yuan, Mengshu Sun, Yanyue Xie, Peiyan Dong, Xin Meng, Xuan Shen, Hao Tang, Minghai Qin, al. Peeling the onion: Hierarchical reduction of data redundaney for efficient vision transformer training. In AAAI, 2023.
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341-353, 2023.
Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver Steeg. Neural architecture search for parameter-efficien fine-tuning of large pre-trained language models. arXiv preprint arXiv:2305. /6597, 2023.
Louis Leconte, Lisa Bedin, Van Minh Nguyen, and Eric Moulines. Reallm: A general framework for Ilm compression and fine-tuning. arXiv preprint w0Xiv:2405.13155 2024.
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of Ilms. arXiv preprint trXiv:2308.07317, 2023.
Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201,2222 2023.
Baohao Liao, Yan Meng, and Christof Monz. Parameter-efficient fine-tuning without introducing new latency. arXiv preprint arXiv:2305. /6742, 2023a.
Baohao Liao, Shaomu Tan, and Christof Monz. Make pre-trained model reversible: From parameter to memory efficient fine-tuning. In NeurlPS, 2023b.
Yi Liao, Yongsheng Gao, and Weichuan Zhang. Dynamic accumulated attention map for interpreting evolution of decision-making in vision transformer. Pattern Recognition, 165:1 11607, 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 2021.
Zicheng Lin, Tian Liang, Jiahao Xu, Qiuzhi Lin, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhances Ilm's reasoning capability. In ICML, 2024.
Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1eX1::2412.19437, 2024a.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NewrIPS, 2022.
Weiyang Liu, Zeju Qiu, Yao Feng Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo. Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficina orthogonal finetuning via butterfly factorization. In JCLR, 2024b.
Yiyang Liu, James C Liang, Heng Fan, Wenhao Yang, Yiming Cui, Xiaotian Han, Lifu Huang, Dongfang Liu, Qifan Wang. and Cheng Han. All you need one: Capsule prompt tuning with a single vector. In NewrlPS, 2025a.
12

---

Published as a conference paper at ICLR 2026
Yiyang Liu, James Chenhao Liang, Ruixiang Tang, Yugyung Lee, Majid Rabbani, Sohail Dianat, Raghuveer Rao, Lifu Huang, Dongfang Liu, Qifan Wang, et al. Re-imagining multimodal instruc- tion tuning: A representation view. In ICLR, 2025b.
Haozheng Luo, Zhuolin Jiang, Md Zahid Hasan, Yan Chen, and Soumalya Sarkar. Frost: Filtering reasoning outliers with attention for efficient reasoning. arXiv preprini arXiv: 19001, 2026.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficien language model tuning. arXiv preprint arXiv:21 2021.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training, arXiv preprint urXiv: 2017.
Rochester Institute of Technology. Research computing services, 2025. URL https://www rit
ledu/ressarrchcoppttnng//
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. MAD-X: An adapter-based framework for multi-task cross cross-lingual transfer. In EMNLP, 2020.
Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. Demystifying reasoning dynamics with mutual information: Thinking tokens information peaks in I1m reasoning. arXiv preprint arXiv:2506.0288 2025.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In IEEE Supercomputing Conference, 2020.
7Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, et al. Deepseek-prose proverrv2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. arXiv preprint arXiv:550.221811, 2025.
Hyogon Ryu, Seohyun Lim, and Hyunjung Shim. Memory-efficient fine-tuning for quantized diffusion model. In ECCV, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03330, 2024.
Antoine Simoulin, Namyong Park, Xiaoyi Liu, and Grey Yang. Memory-efficient selective fine-tuning. In ICML Workshop, 2023.
Antoine Simoulin, Namyong Park, Xiaoyi Liu, and Grey Yang. Memory-efficient fine-tuning of transformers via token selection. In EMNLP, 2024.
Manish Kumar Singh, Rajeev Yasarla, Hong Cai, Mingu Lee, and Fatih Porikli. Tosa: Token selective attention for efficient vision transformers. arXiv preprint arXiv:2406.0888 2024.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks, In NeurlPS, 2021.
Yu Tang, Qiao Li, Lujia Yin, Dongsheng Li, Yiming Zhang, Chenyu Wang, Xingcheng Zhang, Linbo Qiao, Zhaoning Zhang, and Kai Lu. Delta: Memory-efficient training via dynamic fine-grained recomputation and swapping. ACM Transactions on Architecture and Code Optimization, 21(4): 1-25, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An cnatuuction-ollloiing llma model, 2023.
13

---

Published as a conference paper at ICLR 2026
Vithursan Thangarasa, Ganesh Venkatesh, Nish Sinnadurai, and Sean Lie. Self-data distillation for recovering quality in pruned large language models. In NeurIPS Workshop, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard, James Clark, Brett H Meyer, and Warren J Gross. Efficient fine-tuning of bert models on the edge. In ISCAS, 2022.
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, et al. Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models. In EMNLP, 2023.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for IIm reasoning. arXiv preprint arXiv:2506.01939 2025.
Taowen Wang, Yiyang Liu, James Chenhao Liang, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, et al. M2pt: Multimodal prompt tuning for zero-shot instruction learning. In EMNLP, 2024.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixuureoo-addapttions for parameter-efficient model tuning. arXiv preprint arXiv:2205. 12410, 2022a.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self- seff-instruct: Aligning language models with self- self-generated instructions. arXiv preprint ax2i::2212. 2022b.
Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in llms. arXiv preprint arXiv:2502.12001 2025.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, 2024.
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint ar arXiv: 109.:2200.00000 056877 2021.
Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. arXiv preprint arXiv:2308.13111 13111, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15115:2412.15115. 2024.
David H Yang, Mohammad Mohammadi Amiri, Tejaswini Pedapati, Subhajit Chaudhury, and Pin- Yu Chen. Sparse gradient compression for fine-tuning large language models. arXiv preprint arXiv:2502.00311, 2025.
Xinhao Yao, Lu Yu. Xiaolin Hu, Fengwei Teng. Qing Cui, Jun Zhou, and Yong Liu. The debate on rlvr reasoning capability boundary: Shrinkage, expansion, or both? a two-stage dynamic view. arXiv preprint arXiv:25 10.00028 2025.
Kai Yi, Georg Meinhardt, Laurent Condat, and Peter Richtárik. Fedcomloc: Communication-effiieenn distributed training of sparse and quantized models. arXiv preprint arXiv:2403.09904 2024,
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint www.:iv2300.0044444 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 190500780 2019.
14

---

Published as a conference paper at ICLR 2026
Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huangg, Ying Nian Wu, and Dongfang Liu. Visual fourier prompt tuning. In NeurIPS, 2024.
Runjia Zeng, James Chenhao Liang, Cheng Han, Zhiwen Cao, Jiahao Liu, Xiaojun Quan, Yingjie Vic- Chen, Lifu Huang, Tong Geng, Qifan Wang, and Dongfang Liu. Probabilistic token alignment for large language model fusion. In NeurIPS, 2025a.
Runjia Zeng, Guangyan Sun, Qifan Wang, Tong Geng, Sohail Dianat, Xiaotian Han, Raghuveer Rao, Xueling Zhang. Cheng Han, Lifu Huang, al. Mept: Mixture of expert prompt tuning as manifold mapper. In EMNLP, 2025b.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In ICML, 2025c.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.0m 03303, 2023.
Ping Zhang and Lei Su. Memory analysis on the training course of deepseek models. arXiv preprint arXiv:2502.07846f 2025.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. TPAMI, 2024.
Chen Zhao, Shuming Liu, Karttikeya Mangalam, Guocheng Qian, Fatimah Zohra, Abdulmohsen Alghannam, Jitendra Malik, and Bernard Ghanem. Dr2net: Dynamic reversible dual-residual networks for memory-efficient finetuning. In CVPR, 2024.
Wenxuan Zhou, Shujian Zhang, Lingxiao Zhao, and Tao Meng. T-reg: Preference optimization with token-level reward regularization. ACL, 2025.
15

---

Published as a conference paper at ICLR 2026
SUMMARY OF THE APPENDIX
This appendix contains additional experimental results and discussions of our ICLR 2026 submission: TOKENSEEK: Memory Efficient Fine Tuning via Instance-Aware Token Ditching, organized as follows:
SI
provides additional implementation details of TOKENSEEK, complementing the overall
methodology and results presented in the main paper.
S
offers detailed analysis of efficient token ditching, expanding on the motivations and
complexity analysis discussed in the main paper.
|
presents additional experiments of TOKENSEEK on larger LLMs, extending the smaller-
scale evaluations included in the main paper.
S4 shows related asset license and consent to our work.
claims reproducibility of our approach.
$6
discusses the social impact of our research.
S7
adds more discussions, and points out potential directions of our future work.
S8
includes further visualization results, covering case studies, attention maps, and gradient
score distributions.
S1
IMPLEMENTATION DETAILS
$1.1 INSTRUCTION TEMPLATE
For instruction tuning of large LLMs, apply the Alpaca Taori al. 2023) prompt template without incorporating step-by-step reasoning following (Simoulin et al. 2024), as shown below.
"Below is an instruction that describes task, paired wi input that provides further context. Write response that appropriately completes the request. ### Instruction: {instruction}
### Input: [input}
### Response:
$1.2 TRAINING AND EVALUATION DATA
For instruction tuning, fine-tuned Qwen and Llama 21,221 samples from the Open- dataset (Lee et al. 2023). Although Open-PPatyppus comprises 11 open-source datasets, we excluded two seaiiccdeesslutioss--pi-e
they contain outputs generated by GPT models (Achiam et al. 2023). We used the remaining 9 datasets for fine-tuning, following (Simoulin al. 2024
gp14 and 1eobbrrosgppt-11...1
$1.3 TRAINING SETTINGS
We do not use any checkpointing, ZeRO or offloading techniques. Below are other training settings details.
bf16: true fp16: false ur_scheduler__tyn "cosine"
lelee: "01"
warmup_steps: 100
weight_decay: 0.01 optim: "adamw_torch"
For QLoRA and LoHa, we use the below same configuration in all experiments.
alpha: 16
dropout:0. 0.0out:0.00 0.05 r:8
16

---

Published as a conference paper at ICLR 2026
For other baselines such IA3 and BOFT,
the default PEFT configuration across all runs.
Regarding the resulting peak/average memory, we would be happy to provide the original numbers for full parameter/token tuning in each setting for more comprehensive understanding. Other baseline memory usage can be calculated using the original numbers below and their relative memory percentages reported in Tab.
Table S1: Memory usage summary for Qwen2.5 and Llama3.2 models under full parameter/token tuning.
Model
Average Memory (MB)
Maximum Memory (MB)
Qwen2.5 (0.5B)
12,070
27,300
Llama3.2 (1B)
19,462
39,688
I.lam3..2 (3B)
40,100
83,927
$1.4 HYPERPARAMETER SETTINGS
Given the hyperparameters a and 3 introduced in Eq. 5 we perform linear search over the set {[1,0], [0.1], [5,5]] for each LLM. The optimal settings [1,0] for Qwen2.5 0.5B and Llama3.2 2 1B, and [5,5] for Llama3.2 3B. A linear search over [[1,0]. [0.1]. [5,5]] reveals that smaller models (Qwen2.5 0.5B, Llama3.2 2 IB) benefit from context-only information, while the larger Llama3.2 3B performs the best when combining both context and gradient information. This suggests that larger models may better utilize multi-information token evaluation.
S1.5 STANDARD ERROR FOR MAIN RESULTS
We report standard emrors in Tab, S2 for the main evaluation results presented in Tab I is important to note that these values are derived from the "Im evaluation harness" (Gao et al. (024). which yields consistent standard error all PEFT baselines (e-g., LoHa, QooRA) and TOKENTUNE variants share similar standard errors for each benchmark under the same task setting. This consistency ensures a fair and controlled comparison across methods. As shown, the average standard errors across all tasks remain low (e.g., ~,.0), indicating stable performance estimates. Therefore, the conelusions drawn from accuracy improvements and memory savings remain robust under the evaluation framework, further supporting the reliability of TOKENSEEK 's performance gains.
repeated evaluations the same task and dataset. Consequently,
Table S2: Standard error for the main results in Tab. !
Hella Swag
Truthful QA
Wino Grande
Method
ARC
MMLU
Average
Qwen2.5 (0.5B)
TOKENSEEK (Ours)
1.216
0.438
0.339
1.640
1.405
1.008
w/ LoHa
1.252
0.439
0.338
1.631
1.405
1.013
w/ QLoRA
1.395
0.499
0.337
1.436
1.387
1.011
Llama3.2 (1B)
TOKENSEEK (Ours)
1.245
0.437
0.338
1.623
1.405
1.009
w/ LoHa
1.433
0.473
0.345
1.421
1.367
1,008
w/ QLORA
1.434
0.477
0.342
1.455
1.372
1.016
Llama3.2 (3B)
TOKENSEEK (Ours)
1.237
0.442
0.337
1.613
1.404
1.007
w/ LoHa
1.461
0.427
0.330
1.474
1.287
0.996
w/ QLoRA
1.450
0.432
0.331
1.497
1.290
1.000
$1.6 TRAINING TIME FOR MAIN RESULTS
We further analyze the training time (measured in GPU hours on one NVIDIA A100-40GB GPU) assess the computational efficiency of TOKENSEFK compared other methods. Notably, To-
17

---

Published as a conference paper at ICLR 2026
KENSEEK exhibits similar training time overhead TOKENTUNE across all model scales and PEFT settings. Specifically, both TOKENSEEK and TOKENTUNE incur a modest increase of approximately 11 11-15% in GPU hours compared to baseline full-token tuning. For example, on Qwen2.5 0.5B, the baseline takes 0,43 GPU hours, while TOKENSEFK requires 0.49; similarly, on Llama3.2 1B, the baseline uses 0.35 hours us. 0.39 hours for TOKENSRRK. This overhead is likely attributed to the irregularity introduced by selective gradient computation. as we split the tokens into gradient and non-gradient segments for token-level control (see $3..22 While this adds minimal overhead, the trade-off is justified by the substantial memory savings and performance gains achieved by To- KENSEEK. In sum, TEKENSEEK maintains training time efficiency comparable other lightweight fine-tuning methods, while delivering superior performance and memory benefits.
Table S3: Training time for the main results in Tab.
Baseline
LoHa TOKENSREK
QLoRA TOKENSREK
Baseline
LoHa
QLoRA
TOKENSEEK
Qwen2.5 (0.58)
GPU Hours
0.43
0.49
0.62
0.69
0.77
0.86
Llama3.2 (1B)
GPU Hours
0.35
0.39
0.73
0.82
0.55
0.63
Llama3.2 (3B)
GPU Hours
0.67
0.75
1.63
1.85
1.03
1.18
S1.7 DETAILS OF INSTRUCTION TUNING
All experiments are conducted on NVIDIA A A100-40GB GPUs, except for the full token tuning settings on Llama3.2 due to the out-of-memory (OOM) issues. We train for one epoch using learning rate of x 10-4 4 for all fine-tuning. A batch size of is used with 32 gradient accumulation steps. Adapters are inserted into the feed-forward layers of each Transformer block following (He et al,, 2021). The model is prompted using the Alpaca-style format (TTori et al. 2023) without explicit reasoning
S2 DETAILS OF EFFICIENT TOKEN DITCHING
In this section, we present the implementation details of the efficient Token Ditching. Different from previous approaches, e.g., (Simoulin al. 2024), TOKENSEEK introduces an instance-aware token selection framework, TOKENSEEK, which prioritizes tokens based on context and gradient-base information (see $3.2.1). thereby replacing random sampling with a principled, data-driven process. To facilitate understanding of the token ditching mechanism, we adopt the mathematical formulation below.
S2.1 TOKEN DITCHING FOR DENSE AND NORMALIZATION LAYERS
For implementation, we adopt Algorithm T following (Simoulin et al., 2024 which explicitly partitions the hidden states into two subsets: h2 for tokens selected for fine-tuning, and hy for those gradient excluded. As illustrated in Eq 7 and Eq the forward computation remains consistent with standard ttnintnnig, with the key difference being that gradients are disabled for h; using Py PyTorch's Baach.no_grad(0 context, as shown in Eq.
he = heW + h haW +
(7)
where W represents the weights W, and W2 of the feed-forward layers. A similar approach is applied to the normalization layers as well.
18

---

Published as a conference paper at ICLR 2026
Algorithm 1: Token Ditching (To maintain clarity and focus, we simplify the model by removing layer normalization, skip connections, non-linear activations, and multi-head attention.)
Input: input tokens X
Output: selected tokens hı, and unselected tokens /;j input sequence h
Compute token embeddings
Divide the input tokens into two groups selected and unselected tokens (he and hG) via Token Seeking.
for each trans formers' lagers do
// Attention computation
[Qc, Kr, Vi] h, Q.K,V. KV
he = softmax [.1
[Vr Vi]
Vd
// No gradients unselected tokens with torch.no grad(:
Qr. K, Vi] (...1 klo.K.))
h2 = softmax [.IKK,
[V: Vi]
// No gradients for unselected tokens Feed-forward computation
hu h, h, + b
10 h: h.W2 + b2
with torch.no grad:::
12
h, = i, Wi + b
hr h,W2 + by
Re-organize input tokens into the original order
S2.2 TOKEN DITCHING FOR ATTENTION LAYERS
For attention layers, we compute the attention as:
[Q,, Kg Vi] h. ble
(9)
[Qr, K2, Vij h. blQ.K.1
(10)
h; = softmax Ko.] / [V;, V.]
(11)
hr = softmax (K, /V2) [Vr, Vi]
(12)
where WiQ.KV] E Rdx3d represents the concatenated weights for the query, key, and value pro- jections. For the unselected token positions in Eq TO and Eq [2] gradient computation is again disabled using PyTorch. The complete forward pass procedure for the transformer model is detailed in Algorithm 0
S3 EXPERIMENTS ON LARGER LLMS
We further evaluate the potential of ToKENSEEK on larger models (7B scale) shown in Tab[S4] Due to resource constraints, only a subset of configurations is included in the results. As shown in the table, TOKENSEFK consistently outperforms TOKENTUNE across both full parameter/token tuning and QLoRA settings in terms of average score. Specifically, under full-token tuning. TOKENSEEK achieves an average score 60.97, slightly improving over baseline (60.73), while reducing both average and peak memory by 22.1% and 54.4%, respectively. More notably, under the QLoRA setting, TOKENSEEK attains the comparable performance (62.14) while requiring only 18.2% average memory and 12.4% peak memory compared to full tuning, demonstrating >80% memory saving. scenarios,
These results validate the scalability and robustness our method even in reinforcing its value in cenotry-onstraiied training environments.
19

---

Published as a conference paper at ICLR 2026
Table S4: Few-shot evaluation for Llama2 7B model. The experiments are performed under the same
settings as Tab 1 f indicates the results reported from (Smooulin et al. 2024).
Ave. Mem.
Max. Mem.
Hella Swag
Truthful QA
Wino Grande
Average Score
Method
ARC
MMLU
Llama2 (7B)
Full Parameter/Tokee Tuningi
100%
100%
52.39
78.97
64.45 44
38.97
68.90
60.73
w/ TOKENTUNE
77.9%
45.6%
51.71
78.35
61.56
41.88
70.01
60.70
w/ TOKENSEEK
77.9%
45.6%
52.22
78.96
65.28
39.95
68.43
60.97
QLoRAi
53.2%
46.5%
56.06
78.60
65.08
43.64
69.38
62.55
w/ TOKENTUNE
18.2%
12.4%
53.16
78.76
63.64
39.58
69.22
60.87
w/ TOKENSEEK
18.2%
12.4%
53.50
78.82
65.26
44.62
68.51
62.14
S4
ASSET LICENSE AND CONSENT
The majority of TOKENSEEK released under the CC-BY-NC license. However, portions of the project are governed by separate license terms. Specifically, the Transformers library licensed under Apache 2.0. Other dependencies used in this work include the HuggingFace PEFT and Datasets libraries, both under the Apache 2.0 license; the in-evallattinn-are under MIT; and PyTorrc, which is distributed under the modified BSD-3 license. The Open- Playypus dataset used for fine-tuning aggregates multiple detalet license information is available at /ttgg:// http://ni......t.eaeeaeaaaaaaaae
framework, which is licensed
SS REPRODUCIBILITY
Our implementation of TOKENSREK is based on the HuggingFace Transformers library (v4.33.1 1).... For LoHa and QLoRA, we utilized the HuggingFace PEFT library 3 (v0.6.0). Datasets used for fine-tuning were obtained via the HuggingFace Datasets library (v2.1 18.0), specifically using the plan-Platypus datase
For evaluation with the Qwen and Llama models, we employed the nn-evvlluation-haaeeee framework (v0.4.2). All experiments were conducted using the PyTorch framework (v2.0.1).
To guarantee reproducibility, our full implementation shall be publicly released upon paper accep- tance.
S6 SOCIAL IMPACT AND LIMITATIONS
TOKENSEEK presents memory-efficinnt fine-tuning framework that significantly reduces training memory consumption while maintaining or even improving model performance. By selectively updating only the most informative tokens through interpretable, instance-aware process, To- KENSEEK enables fine-tuning of LLMs on essource-consrringg hardware. This advancement holds strong potential for democratizing LLM adaptation, making personalized and domain-specifee model fine-tuning accessible in low-resource environments such as academic labs, startups, or edge devices. Moreover, TOKENSEFK aligns with broader goals of green by reducing computational and energy demands during training.
Despite these advantages, TOKENSEEK introduces a pre-stage evaluation that relies on two weighting factors (o, 3) to balance context and gradient-based token importance. While empirically show in $4.4 that TOKENSEEK performs robustly across a wide range of settings, the selection these hyperparameters introduces additional tuning burden. In addition, TOKENSBEK requires token evaluation step prior to training, which incurs additional computational overhead (i.e., a forward
http::///gttub...cm///gggggggggs http:s//gilhb...cc////ggggggagaa
htta:://gthuuucccm////aeleee//// harness
20

---

Published as a conference paper at ICLR 2026
pass and partial backward pass discussed in $3...3) However, this trade-off is justified by the substantial memory savings and performance improvements achieved by TOKENSEEK. Furthermore, TOKENSEEK may be more sensitive to smaller-scale models, which possess limited representational capacity (Tab[]. These limitations suggest that a more lightweight and robust token evaluation could further improve the generality of method.
In summary, TOKENSEEK contributes meaningfully toward the goal of efficient and scalable LLM adaptation, and believe offers valuable insights for future research in memory fine- tuning and token-level optimization.
S7 DISCUSSION AND FUTURE WORK
S7.1 DISCUSSION
Regarding Token Selection Ratio Evaluation Gaps. We have extended the study to cover both the low (<10%) and intermediate (50 - 100%%) ranges under the Qwen2.5-005B QLoRA setting.
3%
7%
60%
80%
10% (comp.)
Average Score
43.79
47.78
48.43
48.40
48.45
Max. Mem.
7%
10%
62%
79%
13%
Table S5: Average score and maximum memory usage under different token evaluation ratios.
For low-ratio: Even under overly aggressive sparsity settings, TOKENSEEK maintains over 90% and 98% performance at the 3% and 7% settings, respectively, with only slight memory savings compared to the 10% setting. The observed performance degradation is likely due to the remaining tokens carrying insufficient gradient signals. For mid-to-high: Scores remain stable between 60% and 80%, indicating that TOKENSEEK continues to select high-quality tokens as the scale increases. However, memory usage rises sharply in this range, reducing efficiency. Overall, we recommend 10% ratio as a balanced choice, considering both performance and memory efficiency.
Regarding the Performance Larger Scale Models. We find that TOKENSEEK achieves more substantial performance gains when applied to smaller-scale models. We attribute this to potential mismatch between model capacity and the training dataset size, since all models, regardless of scale, are fine-tuned on the same Open-Platypus dataset (25K samples), which may not fully exploit the capabilities of larger models. To investigate this, we conducted preliminary experiment on the Llama2-7B model using an expanded dataset that adds 100K randomly sampled examples from MiniPile (Kaddour 2023).
Method
Dataset
Average Score
QLoRA
Open-Platye (25K)
62.55
QLoRA + TOKENSERK
Open-Platypus (25K)
62.14
QLoRA + TOKENSBEK
Open-Platypus (25K) + MiniPile (100K)
63.26
Table $6: Average scores of QLoRA and QLoRA with TOKENSEEK different datasets. Adding MiniPile (100K) to the training corpus improves performance.
As shown, incorporating more training samples yields additional performance improvements, sup- porting our hypothesis that the less favorable results on larger models may be due to the relatively small fine-tuning dataset.
Regarding the Novelty and Differences. Prior works assess token significance for token skipping in attention operations (Singh al., 2024) or feature-importance explanations (Jain & Wallace 2019). Conceptually different to these methods, TOKENSEEK evaluates tokens for gradient detaching. targeting memory savings.
ToKENSEEK leverages both context and gradient information, grounded in theoretical analysis and motivation. Initially, we use only context-based information to guide token selection, where observe that higher scores tend to concentrate in earlier positions, an effect amplified by the causal mask. This bias may limit fine-tuning effectiveness, the answer generation process is primarily
21

---

Published as a conference paper at ICLR 2026
captured in later positions (i.e., the "Response" portion of training instance). This observation motivates the incorporation of gradient-basee information to complement the context signals, enabling more balanced and comprehensive evaluation of tokens. Together, this persspeeee approach provides a flexible, plug-and-play solution for MEFT.
Regarding Performance Gap in Llama Models. Our implementations based on Hugging Face PEFT, which provides a reliable and strong baseline for comparison. Regarding the performance gap observed in Llama but not in Qwen, this may stem from differences in how the two base models were built. Qwen-2.5-0.51 was trained at its target size from scratch (SFT - DPO / GRPO) (Yang et al., 2024). rather than being a pruned-an and-distilled slice of a larger backbone. In contrast, Llama-3.2- B/3B was created by first incorporating logits from the Llama-3. 1.ae-.......0 1-8B and 70B models as token-level targets. Knowledge distillation was then applied after pruning to recover performance (Grattafiori et al. 2024). This compression process results in sharper weights, which are therefore more sensitive (Bartoldson et al., 2020; Thangarasa et al.) 2024) to gradient updates. As a result, full-parameter fine-tuning on tiny dataset might drift off manifold, while PEFT methods that keep most weights frozen remain stable Llama-3,2. 2. These divergent construction pipelines account for the different behaviors observed in Table 1. We are very interested in this direction and plan to further investigate in the following work.
Regarding the Regarding Task Diversity in Experimental Evaluation. To further evaluate TOKENSEEK 's translation capability beyond code generation and mathematical reasoning, we use (Aharoni & Goldherg 2020) the training dataset and randomly sample 10K training examples from each domain (Medical, Law, IT, and Subtitles) to assess in-domain German-Englist translation performance under the Llama-2-7B. The preliminary BLEU scores are reported below, where we observe that TOKEN SEEK consistently achieves comparable performance.
Method
BLEU
Llama-2-7B
33.13
Llama-2-7B + LoRA
40.16
Llama-2-78 LoRA + TOKENSEEK (10%)
41.63
Table S7: BLEU scores of Llama-2-78 7B with LoRA and TOKENSEEK.
More Baseline Comparison. Due to page limitations in the main paper, we provide additional baseline comparisons here offer more complete view of memory usage and performance trends across methods.
Metric
LoRA (IB)
IA3 (1B)
LoRA (3B)
IA3 (3B)
LoHa (7B)
TA.3 (7B)
Max. Mem. 92.6%
88.9%
90.1% 59.88
85.2%
90.5%
84.2%
Average Score
51.95
52.33
60.69
61.93
60.21
Table S8: Comparison of maximum memory usage and average scores across different parameter- efficient fine-tuning tine-tuning methods and model scales.
Novelty Clarification with TOKENTURE Although both methods aim to improve memory efficiency in fine-tuning, their underlying motivations, scoring mechanisms, and empirical behaviors differ fundamentally. TOKENTUNE relies on data-agnostic partial- -rradint selection or random token dropping inspired by an engineering perspective, which we found be ineffective and unstable across tasks. In contrast, TOKENSEEK is motivated by the observation that not all tokens contribute equally to model updates, and therefore adopts a data-driven, instance-aware criterion.
Specifically, TOKENSEFK introduces hybrid scoring mechanism that integrates both attention- based contextual relevance and gradient-based optimization signals. This leads substantially improved stability, interpretability, and effectiveness compared to random or data-agnostic selection. Beyond memory and performance metrics, TOKENSEEK also incorporates comprehensive analyses on stability, interpretability, and optimization behavior, offering insights into token-level contribution during efficient fine-tuning.
Finally, to the best of our knowledge, prior MEFT method performs instance token selection for retivation-memory -et-om--eeeiciiiiinneeeeeee--- training. This instance-aware perspective represents the core novelty of ToKENSEEK and distinguishes it from TOKeNTuNE's engineering-orientee design.
22

---

Published as a conference paper at ICLR 2026
The Claims of Memory Efficiency and Generality. While TOKENSEEK adopts the same high-level boken-ditching paradigm as TOKENTUNE, its advantages are substantially amplified due to our data-driven scoring design.
Memory reduction. TOKENTUNE'S random dropping causes unstable and degraded performance, forcing it to keep more tokens stay competitive. In contrast, TOKENSEEK identifies truly salient tokens, enabling us to discard far more activations without hurting accuracy. As a result, under equal performance, TOKENSEEK consistently achieves significantly lower memory (i.e., with only 10% tunable tokens, we achieve even higher performance than TOKENTUNE'S 50% setting under the same model configuration, as shown in Fig. B)
Generalizability. TOKENSEEK relies solely on inherent signals from the pretrained model (atten- tion and gradients). It requires no auxiliary model, no task-specific knowledge, and no architectural modification. This makes compatible with any Transformer-base
model and any PEFT method (Zeng et al., 2025b; Liu et al. 2025a). In contrast, several recent MEFT paradigms depend on customized modules (e.g., reversible networks), which limits their applicability. TOKENSEEK remains universally plug-and-play across architectures and domains.
Comparison with Sparsity-based PEFT. Sparsity- PEFT (Ansell et al. 2024 He et al. 2024
Frankle & Carbin 2019) reduces memory use by updating only small, selectively chosen subset of parameters instead of the full model during fine-tuning. our main paper, we included BOFT as a representative sparsity-based PEFT method in Tab. [ Here, we conducted additional experiments
RanLoRA (Albert et al. 2025) under the Qwen2.5 (0.5B) setting, as summarized below.
Table $9: Comparison of BOFT, RanLoRA, QLoRA, and QLoRA with TOKENSEEK under the Qwen2.5 0.5B setting.
Method
Ave. Mem.
Max. Mem.
ARC
HellaSwag
MMLU
TruthfulQA
WinoGrande
Average Score
BOFT
145.1%
100.6%
34,64
51.70
58.18
39.57
56.43
48,10
RamLoRA
95.4%
86.7%
29.18
50.10
58.33
45.21
57.22
48.01
QLoRA
51.7%
45.6%
34.64
50.10
58.05
40.41
55.09
47.66
w/ TOKENSEEK
19.2%
13.4%
34.56
50.09
57.52
41.51
58.56
48.45
These results further enhance the comprehensiveness of comparison and greatly deepen paper demonstrate the effectiveness of our approach under the requested setting.
Code-Domain Generalization. We have conducted preliminary experiments under the Llama3.2 .2 (IB) setting to evaluate the code-domain generalization as follows.
Table S10: Comparison code-domain generalization under the Llama3.2 22 IB setting.
Method
Mem.
Mem.
HellaSwag
MMLUU
TranhfiQQ
WinoGraanle
Humsneeal
Average Soore
Lolla
92.3%
99.4%
39.25
65.93
$7.60
37.87
60.77
13.41
45,81
w/ TOXENTUNE (Random)
45.9%%
28. 1%
38.48
61.21
50.34
43.89
59.91
10.97
44.63
ToKENSSEK (Ours)
45.90 99%
28.4%
38.57
65.89
58.18
39.34
60.93
14,02
46.16
ToXENTUNE (Randomn)
14.3%
39.33
62.97
41.76
w/ TOKENSIIIK (Curr)
14.8%
14.3%
39.08
68.98
58.03
38.65
61.33
14.63
46.28
Although coding tasks may contain denser information than QA tasks, TOKENSERK still performs effectively, which may be because our instance-aware token ditching and the strategy that we preserve the full forward pass, allowing complete attention and contextual information to remain intact.
Discussion under the Distributed Environments. We consider two major distributed fine-tuning set- tings: (1) DP: datallarallel training (including ZcRO/FSDP variants), and (2) TP/SP: tensor/sequence parallelism (model parallel training).
For DP, each GPU holds full copy of the model and only processes a different batch. Gradients are all-reduced at the end. DP has minimal impact on TOKENSEEK because every GPU handles its own local samples independently and only needs to synchronize parameters, not token-level information.
23

---

Published as a conference paper at ICLR 2026
For TP/SP, would like further analyze it separately since TOKENSEEK operates in two stages: instance-aware token seeking and efficient token ditching. The former is performed offline before training, while the latter is applied online during training.
- For scoring, we involve the calculation of gradient score, which computed only at the penulti- mate layer (all earlier layers are frozen), requiring substantially less memory than full fine-tuning (i.e., requires only 13.2% memory of full fine tuning under qwen settings). It is inevitable communicate gradients across GPUs assemble the final gradient. However, because we compute gradients only in the penultimate layer, the additional memory cost remains manageable compared with full fine-tuning.
For training, we can distribute the token-score dictionary for each instance across GPUs before training, which consumes only minimal memory store the mapping (e.g., storing 50% of token positions for Open-Platypus requires about 6.8 MB). During training. this regrouping and reorganizing introduces an extra communication step for handling irregular gradient computation. However, thanks to efficient token ditching, the amount of gradient that needs to be synchronized greatly reduced.
We are also looking forward to collaborating with extraordinary engineering teams to further optimize TOKENSEEK for more complex large-scale training scenarios.
Reprouping Process under the Distributed Environments. In conclusion, we provide preliminary analysis of our unoptimized plain implementation and the communication challenges of applying TOKENSEEK in distributed environments. While these factors may reduce some of the memory savings observed in single-node training, the two-stage design combining partial-gradiens scoring and partial-gradient updating keeps the overhead controllable compared with full fine-tuning.
In tensor parallelism, hidden dimensions are split across GPUs, so token regrouping is purely a local row reindexing operation. TP's usual all-reduce pattern stays unchanged. In sequence parallelism, however, the sequence dimension is sharded, so splitting tokens into selected and unselected sets breaks the local-contigung regroup tokens back to their original global order before proceeding. As a result, SP introduces small but necessary per-layer communication for token restoration.
token assumption. Each layer therefore requires an all-to-all shuffle to
Complexity of Implementation. Our implementation is based on huggingface's ' transformers and model to be directly reused and combined with other
PEFT, which allows single integration PEFT methods.
Specifically, we provide detailed explanation of the modifications make to each model below. We regroup the input into [I_ seseeetee,, l [_unselecte]], apply "torch.no_ "torch.no_grad()" to all [_nsellcted, and finally reorganize [O_ selected, O unselected) into the output O. This procedure is model-agnostic, follows common pattern, and does not require manual adaptation to different model architectures, which can be handled by code agents that are highly capable of capturing these patterns, making the extension to other models straightforward.
Selective Update Imbalance. Dropping gradients for less important tokens may bias training if their importance is misestimated or varies across iterations. However, scoring and training designs enable TOKENSEEK achieve stable evaluation (see mode details in the Section 4.3). align with the empirical results from Fig. 3
We retain full-sequence attention and loss computation in the forward pass, and only zero out gradients for unselected tokens during backpropagation. This keeps the training objective and context intact while updating only the gradients deemed most important. This approach constitutes structured gradient sparsification rather than sample dropping parameter pruning, and all parameters are still updated at every step, reducing the risk of systematic bias.
Beyond combining contextual and gradient-based signals to reduce potential misestimation from any single indicator, the scoring is derived from the current sample. therefore an instance-aware. dynamically updated selection mechanism rather than fixed rule. Even if some iterations introduce noise, subsequent training iterations across many examples will adaptively mitigate it.
Furthermore, we also investigate the influences of misestimating token importance. Under the Llama3.2 (IB) QLoRA setup, we introduce an additional setting that selects Top 10% plus Top 40-50% tokens, instead of the standard Top 20%, to simulate misestimation.
24

---

Published as a conference paper at ICLR 2026
Table SII: Ablation under misestimated token-importance settings for Llama3.2 1B QLoRA.
Settings
Tunable Token
MMLU
ARC
HellaSwag
TruhfulQA
WinoGrande
Averuge Score
Random 20%
20%
40.10
63.93
42.96
43.23
61.01
50.25
Top 20%
20%
39.42
65.73
53.24
39.86
60.77
51.80
Top 10% Top 40-50%
20%
39.16
65.91
39.28
51.31
Although "Top 10% + Top 40-50%" underperforms' "Top 20%,." it still outperforms "Random 20%," demonstrating the robustness of TOKENSEEK.
In the future, we plan to explore whether smoothing the scoring function or injecting a small portion of randomly selected tokens exploration further improve TOKENSERK.
Breakdown of Gradient Scoring. We have added further quantification of our gradient scoring as summarized below.
Table S12: Gradient scoring breakdown for Qwen2.5 and Llama3.2 models.
Model
Average Memory
Time (8)
Qwen2.5 (0,58)
13.2%
291
Llama3.2 (IB)
11.5%
377
Llama3.2 3.2 (3B)
10.9%
566
Storing 50% of token positions for Open-Platypus requires about 6.8 MB.
We have also added the variance tables below.
Table S13: Variance of performance under different token evaluation ratios.
10%
20%
30%
40%
50%
0.05242
0.16229
0.01396
0.00669
0.01620
Automatic Learning Hypeparameters. Regarding the potential of learning a and automatically, grid search over ([1,0]. [5.5]. [7.3], [3,7]] validation set is generally practical and sufficient. In our case, Open-Platypun lacks a validation split, and our ablation Tab. shows that performance remains stable across these settings, indicating low sensitivity within this range and limited marginal benefit from learning them externally.
Regarding the potential of learning the token fraction automatically, clarify that is resource controller, which determines the number of tokens retained per sequence under given memory budget. Our ablation in Fig. shows that memory decreases sharply moves from 50% down 10%, making it more appropriate to choose based on the memory budget rather than learn it via a single objective such as validation loss. budget-driven choice of better reflects the tradeoff between accuracy and memory savings.
Gradient Score. We conducted preliminary experiments under the Llama3.2 (1B) QLoRA setting using gradients from different layers as follows.
Table S14: Performance using gradients from different layers under the Llama3.2 (IB) QLoRA setting.
Settings
MMLU
ARC
HellaSwag
TruthfulQA
WinoGrande
Average Score
Random
39.33
62.97
41.76
41.36
60.69
49.22
N-1 layer (default)
39.08
65.98
58.03
38.65
61.33
52.61
N-2 layer
39.33
65.83
58.15
39.58
61.01
52.78
N-3 layer
39.08
65.60
57.55
39.27
61.17
52,53
25

---

Published as a conference paper at ICLR 2026
From the results above, do not observe obvious performance differences, and because using earlier layers requires storing more activation memory and introduce a new hyperparameter, the default setting is more practical.
Furthermore, we provide deeper analysis of this pattern from a visualization perspective. We plot gradient scores obtained from the N-1 layer (blue), N-2 layer (orange), and N-3 layer (green), showing that the scoring pattern remains relatively stable across layers. aligns with the empirical results above.
631
4.00
-0.3
Figure S1: Additional visualization of gradient-based based token scores across layers. The plot compares gradients from the N-1 (blue), N-2 (orange), and N-3 (green) layers.
Attention Score. The "global anchor", which is first introduced as attention sink phenomenon from [refl], that refers to the tendency of attention disproportionately concentrate single position, effectively acting as a global anchor, serves as a reference token (i.e., the baseline) for scoring all subsequent tokens.
In TOKENSEEK, the same attention sink effect emerges (see Fig 4 (b)) because the model naturally assigns one position abnormally large and stable amount of attention. This token becomes the model's internal reference point. Since TOKENSEEK scores tokens using both forward attention signals and backward gradient signals, this "sink" position nearly always receives high combined score even after the normalization (see Fig 4 (d)). As result, it is consistently preserved rather than ditched. TOKENSEEK does not artificially enforce this behavior. It simply reflects the model's inherent dynamics, where the sink token provides a stable baseline that anchors attention patterns across layers and steps.
Token Efficiency across Different Domains. A similar token efficiency phenomenon has been observed in LLM reinforcement learning (RL) and supervised fine-tuning (SFT) reasoning research (Wang et al.
2025 Qian et al. 2025).
Both domains converge on the observation that model behavior is disproportionately shaped by small subset of influential tokens. In the reasoning literature, this is reflected in localized spikes mutual information or high-entropy branching positions that largely determine the downstream reasoning trajectory. TOKENSEEK arrives parallel conclusion from the training perspective: by jointly examining gradient magnitude and attention allocation, we find that only minority of tokens make substantial stable contributions to parameter updates and thus to effective fine-tuning. In both cases, the model does not treat all tokens equally. Instead, it implicitly concentrates its computational and learning capacity structurally semantically pivotal positions.
26

---

Published as a conference paper at ICLR 2026
Despite this shared principle, the operational signals, and scopes of the two lines of work fundamentally distinct. 1) Signal Source: reasoning work relies on inference-ting indicators such as mutual information spikes entropy changes, while TOKENSEEK combines forward attention and backward gradients to estimate how much each token contributes to loss reduction, and then selectively allocates training budget accordingly. 2) Scope: reasoning research typically highlights handful of discrete "turning point" tokens, whereas TOKENSEEK evaluates the entire sequence to identify all tokens that meaningfully influence parameter updates.
Consequently, although both areas reveal token-level sparsity in model computation, they capture different facets of model behavior and operate under different optimization goals. In the future, we would like to further explore how our token scoring strategy might be extended to reasoning.
Relation to RL-based Reasoning and Token-level Analyses. Recent work (Wang et al. 2025
Qian et al. 2025| Lin et al. 2024 Zeng et al. 2025c, Luo et al. 2026 Zhou et al. 2025| Lightman al. 2023 Yao et al. 2025| Guo al. [002 Ren et al. 2025) on RL-based reasoning and chain- of-thought analyses consistently shows that only small fraction of tokens carry most of the useful learning or information signal. RLVR <Wang et al. 2025) finds that high-entropy "forking" tokens account for nearly all performance gains in mathematical reasoning, while gradients on low-entropy tokens contribute little or even harm accuracy. Similarly, nutual-informatioo analyses (Qian et al.) 2025) identify sparse "MI peaks" whose "thinking tokens" are crucial for final-answer prediction, suppressing these tokens severely degrades reasoning. These studies collectively provide a fine- grained view of where RL-style updates and inference-time computation actually matter.
DeepSeekMath (Shao et al. 2024) further links SFT and RL by showing that RL methods like GRPO can be interpreted reshaping gradients while staying close a supervised reference model. In this view, both TOKENSEEK and RLVR adopt token-level importance as the core abstraction, but operate different stages and with different signals. RLVR prioritizes high-entropy tokens during policy updates while TOKEN reallocates the SFT gradient budget across tokens and prioritizes high- score tokens during SFT. this sense. TOKENSEEK addresses the problem of token-wiss efficiency from complementary angle with current reasoning work: we focus on memory-efficien gradient allocation in supervised fine-tuning, while prior RL-based reasoning studies focus on token-wise credit assignment and information flow during policy optimization and inference.
These connections motivate us a future work extension in the spirit of (Wang et al., 2025 Zeng et al., 2025a Qian et al. 2025). TOKENSEEK scoring function is deliberately restricted to signals available in a standard SFT pipeline, but it would be natural in future work to augment TOKENSEEK with additional token-level diagnostics (e.g., entropy MI estimates) or to design hybrid schedules where SET and RL. share a common coken-importance backbone.
Ethics Statement. We conform to the ICLR Code of Ethics and further show the consent to our work below. All datasets used in this study are publicly available and released under permissive licenses publicly available (see $S4 for Asset License and Consent). We
(see $S4), and all the models would like to state that the contents in the dataset do NOT represent our views or opinions and our paper does not involve crowdsoureing research with human subjects.
AI Disclosure. We acknowledge the use of GPT-5 for grammar correction and sentence-level refinement only. The model was employed to enhance clarity, coherence, and fluency while ensuring the original meaning and intent of the text remained unchanged.
$7.2 FUTURE WORK
In 2 we review existing PEFT and MEFT methods, highlighting their focus on optimizing different components of the training pipeline. Unlike prior data-agnostic approaches, TOKENSEFK introduces an instance- mechanism that combines context and gradient information to identify and retain the most informative tokens during fine-tuning. Despite the effectiveness and generality of TOKENSEEK. it raises several open questions and directions for future research. One current limitation lies in the manual selection of weighting scalars a and 3, which control the influence of context and gradient signals. While we provide empirical guidance effective ranges (see $4.4). developing an automated mechanism -ucissssssuhhhhhsee as a lightweight controller or hypernetwork to learn these weights adaptively could enhance performance and reduce tuning overhead.
27

---

Published as a conference paper at ICLR 2026
Another promising direction lies in extending TOKENSEEK beyond instruction tuning and clas- sification tasks to more complex settings such as multi-modal fine-tuning or continual learning. Additionally, although TOKENSEEK integrates well with PEFT methods like LoHa and QLoRA (see T. further exploration is needed to evaluate its synergy with sparse or rerrieval-augmented architectures.
Lastly, while TOKENSEEK demonstrates strong interpretability and robustness across multiple LLMs, deeper analysis of its token evaluation patterns across domains (e.g., code, biomedical texts) may offer insights into task-specific redundancy and inform domain-adaptive pruning strategies.
In summary, TOKENSEEK presents general and interpretable framework for memory-efficient fine-tuning. Future work can build on this foundation by exploring automated token selection, broader task applicability, and tighter integration with emerging efficient model designs.
28

---

Published as a conference paper at ICLR 2026
Attention Map
Context Score
Gradient Score
0.86
0.00
Sample A
0.14
-0.07
Sample B
135
135
1.33
0.05
Sample
200
0
209 0.15
209
205
1.81
0.02
Sample
590
590 0.15
-0.02
590
590
Sample E
594
-0.01
3.31
0.01
Sample F
1066
0
1066 0.12
1066
1006
Figure S2: Additional visualizations of the attention map, context-based scores, and gradient-based scores. For better observation, we omit the attention score of the first token (i.e., context scores start from position instead 0) due to the attention sink phenomenon discussed $4.3
29

---

Published as a conference paper at ICLR 2026
S8 VISUALIZATION OF ATTENTION AND GRADIENT MAPS
presents attention maps, context-based token scores, and gradient-based scores across six randomly selected samples, ranging in length from 126 to 1067 tokens. These examples provide further evidence supporting the key findings from the main paper in $4.3 Across all samples, we observe consistent pattern in context-based scores, where higher values are concentrated in the earlier token positions (i.e., a manifestation of the commonly observed attention sink effect and causal masking in autoregressive models). This phenomenon causes tokens initial positions to accumulate more attention, aligning with prior observations discussed in $4.3 In contrast, gradient-based token scores tend to emphasize later positions, particularly toward the response segments in instruction tuning tasks. This reflects the model's training dynamics: gradients are more pronounced where output predictions are made and optimized-typpine typically in the latter portion of the sequence. Despite differences in sequence lengths, this divergence in focus between the two signals remains stable across all samples. These findings reinforce the motivation behind combining both context and gradient signals in TOKENSEEK for more comprehensive and balanced token importance evaluation.
Fig S2
Tokens selected by context score
Below is instruction that describes task. Write response that appropriately completes the roquest.
### Instruction:
Evaluate S(x+ ()xxy) when Sx- 15$ and Sy 5$.
## Response:
To evaluate the expression, need to plug in the given values of $xS and SyS and simplify using the order of operations. So S(15+ 5y(5-5)S. can do the addition and subtraction inside the parentheses first, and get S(20)(10SS Then multiply and and get 200.
Tokens selected by gradient score
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Evaluate S(x+ (x-yS) when Sx= 15$ and Sy= 5S.
## Response:
To evaluate the expression, I need to plug in the given values of Sx$ and SyS and simplify using the order of operations. So I get S(15+ 5)(15-5).. can do the addition and subtraction inside the parentheses first, and get S(20)(10)S. Then multiply 20 and 10 and get 200.
Figure S3: Visualization of the top 50% selected tokens in Sample A presented in Fig $2 using context and gradient information, highlighted in red and blue, respectively.
study of 126-token instruction-response Sample A presented in Fig S2 with the top 50% tokens highlighted based on importance scores derived from context and gradient information, respectively. Tokens highlighted in red correspond to the top 50% according to context importance, while those in blue correspond to the top 50% based on gradient-based importance.
S3 presents
From the Fig we observe clear distributional pattern in the highlighted tokens: Context- based selection (red) tends to emphasize tokens in the earlier part of the aaquence-particllann the instruction prompt and structural phrases such as "Below instruction that describes task" and "Evaluate (x+y)(x-y) when ..." .". In contrast, gradient- based selection (bluc) focuses more on the response portion, especially on semantically meaningful action words such as "evaluate". "plug in". "simplify", and numerical reasoning steps like "multiply ... 10 and get 200". These tokens are closely tied to the model's output prediction and loss computation, which naturally generate higher gradients. Interestingly, while there is some overlap (e.g., tokens like "x", "y"" and the equation), the two selection strategies yield complementary subsets, justifying the motivation for combining them in TOKENSEEK. This example visually supports the hypothesis that instance-aware token prioritization benefits from incorporating both structural (context) and ceammiizationrrlllvvan (gradient) information.
...
30

---

Published as a conference paper at ICLR 2026
S4 presents long-form, math-intensive instruction-respons pair consisting of 1067 tokens, where the top 50% of tokens are selected solely based on gradient- based importance. This sam- ple offers several interesting insights into the behavior of TOKENSEEK 's gradient-driven token prioritization mechanism. There are three Key Observations:
Gradient Emphasis on Semantically Dense Mathematical Reasoning. Tokens receiv- ing high gradient scores are concentrated around numerical reasoning, symbolic manipu- lation, and step-by-step algebraic deduction. For example: O The derivation and solv- ing of equations such as "$x^2 y*2 16. @ Geometry-specifee calculations like 1SXfrac(24ssartt/)
Acdot (rrac(\\srrt())CC tions, area computation, and final results. This confirms that gradient-based importance scores effectively highlight regions where model predictions are tightly coupled with loss, especially in problem-solving and logic-intensive portions of the response.
and @ Descriptions of intersec-
Selective Attention in Code Blocks Surprisingly, some code comments and critical semantic structures in the embedded code block are also assigned high gradient scores. This includes: Function calls like draw(..). label(...), and array manipulations involving coordinates. @ Mathematical graphing logic, such as drawing circle or intersecting paths with graph(..... This suggests that gradient signals are not purely confined to natural language but can also prioritize symbolic logie and programmatic structures that critical to the correct final output.
Omission of Setup and Template Tokens. The gradient- based selection intentionally avoids early template phrases (e.g.. "... instruction describes task") and instead defers attention to content-bearing tokens, particularly in the response body. This is aligned with prior gradient analysis (see $3.2.1), and reinforces the idea that gradient prioritization aligns well with actual supervision information.
31

---

Published as a conference paper at ICLR 2026
Below is an instruction that describes a task. Write response that appropriately completes the request.
HMH Instruction:
Let Simathcal S be the hyperbola with foci at S(pm 0)S and vertices at 5(pm 0),5 and let Smmathhal((") (C}$ be the circle with center $(0,0)$ and radius $4.S Given Smatheal(H]] Simatheal(C)S intersect at four points, what is the area of the quadrilateral formed by the four points?
fifif Response:
For the hyperbola Amathhal(HH).. have $a=3S and $c=5,5 so Sb= (2-aa(" = =.. Thus, the hyperbola has equation (x= [yrrac (4 - 1, V16x 9y"2 144. 144.whhee,,hile, the equation for the circle Sx y*2= 16.$ find the points of intersection, solve these two equations simultaneously. Adding times second equation the equation gives $25x"2 288.5 so Sx Ipm (frac[12ssqr2]155 Then we have \y^2 = 16 x = 16 ( (25) (112) (fruc(4ssrr7))555) Therefore, the four points of intersection form a rectangle with side lengths
Sy 1pm
(24rac(24))))22) {5YS and (8frac(8's (5),5 S so its area is Sifrac (24lsqr22) (5) 'odot (frac(81s (raac[192)srr((1)))(355 [asy]
void axes(real x0, real y0, real yl)
abbcl""5x5",xx1,,)..E
@hee(""555"000yy)...
for (int flloor((00)+; ixl; Hi)
draw(i.11)(---11);
for (int 1f00: kcy::
draw((1,))((-1..);
path|] yh(real a, real b, real h, real k, real x0), real x1, bool upper-true, bool lower=true, pen color=black)
real f(real return + / sqrt(b>2 (x-h)²2);
real gireal x) return /a/b /b* sqri(b/22 + (x-h)22);
if (upper) { draw(graph((f, x0, xolo,oolo,, Arrows); }
if (lower) draw(greph(g, x0, color Arrows); path (graph([, x0, x1), graph(g, x0, x1)];
return ar;
void xh(real real b, real h, real k, real yO, real y1, bool right-true, bool left-true, pen color-black)
path || arr yh(a, k, h, y0, yl, false, false);
if (right) ") if (left) (1), ccolor, Arrows);
Arrows);
void o(real a, real b, real
size(8em);
axes(-6,6,-6,66)
xh(3,4.0.0.-555);
c(4,4,0,0);
(int i=
for (fttj=1;j=1;;;t=))
(aaart////Aaam(((555ycccee..eee. [/asy]
12
Figure S4: Visualization of the top 50% selected tokens in Sample presented in Fig S2 using gradient information, highlighted in blue.
32


Reviews:
[{"review_id": "9o2yGngfVK", "reviewer": "Reviewer_ZpqT", "summary": "This paper addresses the limitation of current fine-tuning methods that largely overlook the information contained in individual training instances. The authors propose TokenSeek, a universal plug-in framework for Transformer-based models that performs instance-aware token selection and ditching. By selectively fine-tuning on important tokens identified via attention and gradient signals, TokenSeek reduces activation memory consumption while maintaining comparable performance.", "questions": "- Code-Domain Generalization:\nThe experiments cover QA and Open-Platypus datasets. How would TokenSeek perform on code-related datasets (e.g., CodeAlpaca, HumanEval)? Code data tends to have dense information where most tokens are important, which could reduce token sparsity efficiency. Including one experiment on a code dataset would better demonstrate TokenSeek’s generalizability.\n\n- Source of Memory Savings:\nPlease clarify precisely where the memory savings originate. Since token selection is gradient-based, forward propagation must still process all tokens, and optimizer memory (e.g., Adam states) typically depends on trainable parameters rather than token count. How does TokenSeek achieve the reported activation memory reduction—through selective gradient storage, reduced backward pass, or another mechanism? A detailed explanation would strengthen the technical soundness.\n\n\n\nAt present, this appears to be a borderline paper (score ~5). The core idea is promising, but its novelty and advantage over prior TokenTune and LoRA-based methods need clearer articulation. If the authors can convincingly address the questions above—particularly regarding memory savings and broader generalization—I would be open to raising my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and generally easy to follow, with only minor typos.\n- The proposed TokenSeek method is conceptually simple and practically implementable.\n- Experimental results demonstrate competitive performance on multiple downstream tasks (e.g., QA, reasoning) across different LLM architectures such as LLaMA and Qwen.\n- The experiments are comprehensive and include ablations and cross-task evaluations, which strengthen empirical credibility.", "weaknesses": "- Limited Novelty Clarification:\nThe main contribution—token importance–based selection—extends the previous TokenTune framework rather than introducing a fully new paradigm. While TokenSeek improves token importance estimation compared to random selection, the core ideas of memory reduction and generalizability are largely inherited from TokenTune. The paper should better articulate what is fundamentally novel about TokenSeek beyond methodological refinements.\n\n- Comparison to Low-Rank and Partial-Tuning Methods:\nOn Line 835, the paper notes that TokenSeek and TokenTune incur ~11–15% more GPU hours than full-token tuning. In contrast, low-rank PEFT methods (e.g., LoRA) typically achieve both lower memory and faster training. Since TokenSeek inherits TokenTune’s extra computational overhead, the paper should clarify whether TokenSeek offers any non-trivial advantages over low-rank or partial-tuning approaches.\n\n- Missing Baselines and Related Work:\nSeveral recent sparsity-based PEFT methods are missing from the related work and experimental comparisons. As TokenSeek belongs to the PEFT family, adding one representative partial-tuning or sparse fine-tuning baseline (e.g., [1–3]) would provide a more complete evaluation and contextualization of the method’s contribution.\n\n\n- Some minor typos\n“as showin in Fig. 1 (a)” → shown; \n“Benifit from” → Benefit from; \nL313 “Unde” → Under; \n“achiving” → achieving; \n“TokenSeek achieve” → achieves; \n“LoRA/QLoRA achieves” → plural → achieve\n\n\n\n\n[1] Scaling Sparse Fine-Tuning to Large Language Models\n\n[2] Sparse Matrix in Large Language Model Fine-tuning\n\n[3] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DKnqFAvi6", "reviewer": "Reviewer_CWhD", "summary": "TOKENSEEK introduces a universally applicable plugin for memory-efficient fine-tuning of large language models without altering their architecture. It adaptively identifies and retains only the most salient tokens per instance—based on contextual and gradient importance—while discarding less useful ones to reduce activation memory. This instance-aware token ditching achieves significant memory savings with stable or improved performance, offering a generalizable, architecture-agnostic approach to efficient fine-tuning.", "questions": "- In Eq 3, the notation of i and j is a bit confusing. It might be more clear to express context score for ${t_j}$ token as $\\Sigma_{i=1}^{n} A_{ij}$. To clarify that you are summing the attention scores across all rows for the given column j.\n- Similarly the notation in eq 4 should indicate that the the gradient score is for  token $t_j$.\n\n- More clarification on the token regrouping process and its practicality in distributed fine-tuning settings would strengthen the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Solid problem formulation  as fine-tuning large LLMs is highly memory-intensive, with activations contributing a major share of the cost.\nAnd Innovative approach to integrates gradient information with context scores to capture a more holistic measure of token importance, addressing the limitation that context-based evaluation alone reflects only intra-sequence relevance, not fine-tuning contribution.\n\n\nSome of the specific strength:\n - Architecture-agnostic design: The proposed plugin can be applied to any pretrained LLM without modifying its core architecture that is very practical.\n - Strong empirical results: Demonstrates substantial memory savings while maintaining or improving model performance across benchmarks. Very solid results in table 1, specifically with QLora.\n- Interpretability and analysis: Offers clear insights into token-level importance and provides comprehensive analysis on how token ditching affects fine-tuning. Figure 4 and other analysis plots are insightful.\n- Clarity and presentation: This paper is very well-written, logically structured, and easy to follow, with clear motivation and experimental validation.", "weaknesses": "- Scalability concerns: The token regrouping step—where tokens are sorted by importance and selectively included for backpropagation—may pose significant implementation and communication challenges in large-scale distributed fine-tuning setups. Synchronizing token importance scores and managing uneven token partitions across devices could offset some of the claimed memory savings.\n\n- Complexity of integration: Although conceptually modular, integrating the method into existing large-scale MEFT pipelines may require non-trivial modifications to data loading and parallelism strategies.\n\n - Limited large-scale validation: Experiments are mostly conducted on moderate-sized models (<=3B) and datasets; the method’s stability and efficiency under massive multi-node training scenarios remain unverified.\n\n- Selective update imbalance: Dropping gradients for less salient tokens could bias training if token importance is misestimated or unstable across iterations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cx7xDewtqn", "reviewer": "Reviewer_kTDb", "summary": "This paper proposes TOKENSEEK, a plug-in for memory-efficient fine-tuning that is instance-aware at the token level. The core idea is to (1) seek salient tokens per input by combining context signals (attention-derived scores) and optimization signals (gradients of the loss w.r.t. penultimate-layer activations), then (2) ditch the rest by disabling gradient computation for unselected tokens during backprop (retaining forward activations but avoiding their gradient/optimizer memory).", "questions": "For LoRA/QLoRA/LoHa/IA3 and full FT, which memory knobs (checkpointing schedule, ZeRO, offload, seq length, grad accumulation, activation precision) were enabled? Please add a per-run configuration table with measured peak+average memory. How often does the selected token subset change across epochs/steps? Any evidence of training instabilities when the subset shifts? Could you learn α/β or the token fraction from validation loss via a small controller?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Combines attention-derived context and gradient saliency to rank tokens per example; empirically more effective and more stable than random selection. Reports 2.8 GiB peak in one setting and ~15% of full-token QLoRA peak on Llama-3.2-1B, while maintaining accuracy; cumulative with PEFT (LoHa/QLoRA). isualizations show complementary early-token bias from attention and late-token focus from gradients; helps explain the chosen subset.", "weaknesses": "Need a controlled knob table for each baseline (checkpointing, offloading, micro-batching, seq length, optimizer sharding) and the resulting peak+average memory to ensure apples-to-apples comparisons. Gradient-based scoring requires a partial backward pass; quantify this overhead per step and analyze action oscillations/instability of the selected set across training. Provide seed variance tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Ix573ulHg", "reviewer": "Reviewer_cH6y", "summary": "This paper proposes TOKENSEEK, a method for memory-efficient fine-tuning (MEFT) of LLMs that addresses the bottleneck of activation memory. TOKENSEEK introduces an instance-aware paradigm that dynamically identifies and preserves the most informative tokens per input instance while discarding less crucial ones during gradient computation.", "questions": "> w1: Regarding the computation of gradient information, the description in Lines 230-232 specifies the use of \"the activations in the penultimate layer,\" and Lines 262-263 mention freezing \"all layers except the output head and the final decoder block.\" Was the use of gradients from more layers explored? Furthermore, would expanding the scope of gradient computation in this manner still ensure stable fine-tuning, or could it potentially introduce instability?\n\n> w2: The manuscript introduces scalars α and β to integrate context and gradient information (Lines 236-245) and notes their \"distinct but complementary patterns\" (Lines 405-407). However, the rationale and process for selecting the optimal values for α and β are not sufficiently discussed. A more detailed analysis is needed to clarify how these critical hyperparameters were determined and how sensitive the method's performance is to their specific values.\n\n>w3: The authors identify a \"global anchor\" in the attention map (Lines 373-377), evident as a prominent vertical line. Based on the proposed TokenSeek framework, could the authors provide an explanation for why this specific token emerges as a global anchor? Furthermore, given that your method ditches tokens based on a combined score, is this particular \"anchor\" token ever identified as less informative and subsequently ditched, or is it consistently preserved? Clarifying this would enhance the interpretability of the token selection process.\n\n>w4: The insight that not all training tokens contribute equally to model fine-tuning is well-founded and reasonable. A similar phenomenon has been observed in LLM reasoning research [1,2]. Could the authors analyze the connections and distinctions between your findings and those from the reasoning domain?\n>\n>[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.\n>\n>[2] Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": ">s1: The key insight behind TOKENSEEK is that not all training tokens within LLMs contribute equally to\nmodel fine-tuning.\n\n>s2: The token selection process is based on a transparent, hybrid scoring mechanism leveraging both contextual attention and gradient information. This provides interpretability into why certain tokens are selected and contributes to more stable and effective fine-tuning compared to data-agnostic or random baselines.\n\n>s3: The presentation is clear, and the figures are of high quality.", "weaknesses": "See Questions. I would reconsider my score if these concerns are adequately addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)