All You Need is One: Capsule Prompt Tuning with a Single Vector
Authors: Yiyang Liu
James Chenhao Liang
Heng Fan
Wenhao Yang
Yiming Cui
Xiaotian Han
Lifu Huang
Dongfang Liu
Qifan Wang
Cheng Han
Venue: NeurIPS 2025 poster ()
Keywords: Prompt Tuning
Large Language Models
Parameter-Efficient Fine-Tuning
Primary area: deep_learning
TL;DR: We introduce CaPT, leveraging both instance-aware and task-aware information for effective and efficient prompt-based learning.

Abstract: Prompt-based learning has emerged as a parameter-efficient finetuning (PEFT) approach to facilitate Large Language Model (LLM) adaptation to downstream tasks by conditioning generation with task-aware guidance. Despite its successes, current prompt-based learning methods heavily rely on laborious grid searching for optimal prompt length and typically require considerable number of prompts, introducing additional computational burden. Worse yet, our pioneer findings indicate that the task-aware prompt design is inherently limited by its absence of instance-aware information, leading to a subtle attention interplay with the input sequence. In contrast, simply incorporating instance-aware information as a part of the guidance can enhance the prompt-tuned model performance without additional fine-tuning. Moreover, we find an interesting phenomenon, namely "attention anchor", that incorporating instance-aware tokens at the earliest position of the sequence can successfully preserve strong attention to critical structural information and exhibit more active attention interaction with all input tokens. In light of our observation, we introduce Capsule Prompt-Tuning (CaPT), an efficient and effective solution that leverages off-the-shelf, informative instance semantics into prompt-based learning. Our approach innovatively integrates both instance-aware and task-aware information in a nearly parameter-free manner (i.e., one single capsule prompt).
Empirical results demonstrate that our method can exhibit superior performance across various language tasks (e.g., 84.03\% average accuracy on T5-Large), serving as an "attention anchor," while enjoying high parameter efficiency (e.g., 0.003\% of model parameters on Llama3.2-1B).

All You Need is One: Capsule Prompt Tuning with a Single Vector
Yiyang Liu'
James C. Liang"
Heng Fan'
Wenhao Yang"
Yiming Cui
Xiaotian Han
Lifu Huang"
Dongfang Liu Qifan Wang
Cheng Han'
University of Missouri- City
University of North Texas Lamar University 'ByteDance "Case Western Reserve University
U.S. Naval Research Laboratory
University of California, Davis
*Rochester Institute of Technology
"Meta Al
Abstract
Prompt-based learning has emerged as a parameter-efficien finetuning (PEFT) approach facilitate Large Language Model (LLM) adaptation downstream tasks by conditioning generation with task-aware guidance. Despite its successes, current prompt-based learning methods heavily rely on laborious grid searching for optimal prompt length and typically require considerable number of prompts, introducing additional computational burden. Worse yet, our pioneer findings indicate that the task-aware prompt design is inherently limited by its absence subtle attention interplay with the
of instance aware information, leading input sequence. In contrast, simply incorporating instance-aware information part the guidance can enhance the prompt-tuned model performance without additional fine-tuning. Moreover, we find an interesting phenomenon, namely "attention anchor," that incorporating instance-awarc tokens at the earliest position of the sequence successfully preserve strong attention to critical structural information and exhibit more active attention interaction with all input tokens. light of our observation, introduce Capsule Prompt- Tunii--TTnngg (CaPT), efficient and effective solution that leverages off- off-tf informative instance semantics into prompt-based learning. Our approach innovatively integrates both instance- aware and task-aware information a nearly parameter-free manner (i.e., one single capsule prompt). Empirical results demonstrate that method can exhibit superior performance across various language tasks (e.g., 84.03% average accuracy "attention anchor," while enjoying high parameter efficiency (e.g., 0.003% of model parameters on Llama3.2-1B).
T5-Large), serving
Introduction
Large Language Models (LLMs) 2 4 initiate revolutionary transformation in both artificial intelligence and various domains of human activity. Among their fruitful applications, a widely adopted paradigm for adapting pre-trained LLMs to downstream tasks is known "pretrain-then- finetune." " As these I.MMs continue to grow in size and complexity for enhanced performance, parameter-efficient fine-tuning (PEFT) methods (5] 6 7
have emerged compelling alternatives to full fine-tuning, demonstrating competitive performance with noticeably lower parameter usage.
well-recognized approach within PEFT, offers simple yet effective fine-tuning strategy in which researchers employ learnable soft prompts [
Prompt-based Learning 10
g to [6
activate model capabilities without modifying existing parameters. Prompting methods B [4] [5]
initially relied on the manual design of discrete prompts, which often suffer from limited flexibility and require extensive effort. Therefore, trainable soft prompts utilizing continuous embeddings have 13 Nevertheless, current soft prompt-based learning
emerged as the predominant strategy 11 two main limitations:
Cooresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

I. Limited Capability. Soft prompts are typically optimized to encode task-aware instructions to guide generation in an one-size-fits aitt--iit--all manner [T]
18]. Since soft prompts are integrated into the actual sequence, their strong performance suggests potentially significant interplay with original input tokens in the view of attention [19 20 21.. Counterintuitively, our finding suggests that these task-specific soft prompts actually fail to exhibit strong interaction with input tokens (i.e., Fig. [] (left)- - they primarily attend to each other (i.e., the blue box), with minimal focus on critical input tokens (i.e., the red box) which input sequences predominantly attend to. This observation reveals that the task-aware design of soft prompts may limit their capability to adapt to diverse input semantics, potentially constraining the effectiveness of prompt-base based learning. Consequently, natural question arises: Can soft prompts be interactive for improved adaptation to diverse instances?
I. Inefficient Searching. Another critical limitation of soft wrompt-basel learning is the time- intensive grid searching for the optimal prompt length (9 1 12 22 23 This searching generally results in a considerable number of prompts, thereby extending the sequence length of L.M and incurring additional training overhead. Even worse, recent studies have shown that these elaborately searched soft prompt tokens may still fail to effectively capture task-aware semantics for downstream tasks [24 25], and in some cases, certain prompt tokens can even negatively impact model perfor- mance 17 Therefore, our research question turns into: @ Is it possible to eliminate the inefficient and time consuminn grid searching of prompt length while preserving informativeness?
In response to questions O 0-0, we investigate the possibility of incorporating off-the-shelf instance- aware information as a part of prompts to address these limitations. Our preliminary study (see $3.1) reveals the power of instance-aware information even one simple, training-free integration of instance-aware token results in noticeable performance increase. Employing a compact, fixed length design can therefore eliminate the need for extensive searching on prompt length, significantly reducing the overall training time. Our observation on attention pattern further strengthen this idea. We reveal that, unlike soft prompt tokens, single instance-aware token can consistently attend to critical input tokens (i.e., structural information) and be consistently and actively attended by input sequences, acting as "attention anchors." These attention anchors can be leveraged to guide model's attention toward structurally important regions of input sequences and actively propagate guidance signals into the sequence, leading to better contextual grounding.
To this end, we propose a simple yet effective prompt-based learning strategy - Capsule Prompt- Tuning (CaPT). We integrate both instance- aware information from each input sequence and task- aware information from learnable vectors to form capsuled prompts. The instance-aware information helps preserve strongly attentive interplay with input sequences, while the learnable vector encodes task-aware inductive bias as a general guidance. Notably, CaPT can operate in an almost parameter- free manner, utilizing only one single vector, which eliminates the need for time-intensive grid searching, varied lengths across different tasks, and substantial training overhead 99 [2] Ex- perimental results demonstrate that CaPT enjoys not only training and parameter efficiency, but also state-of-the-as art performance (e.g.. 7.5% higher performance compared to vanilla Prompt- Tuning for T5-Large). We further confirm that our capsule prompt tokens can successfully act as "attention an- chor" to establish a mutual and contextual attention pattern. We believe our work offers an innovative perspective efficiently leveraging Instance-aware information into nompt-based learning.
2 Related Works
Attention in Large Language Models. Unlike earlier architectures such as RNNs (26 27 and
34 35), allowing them to capture complex dependencies and contextual nuances, thereby achieving superior performance across
CNNs [28 29), LLMs incorporate Transformer 1 30
31
32
33
various tasks (e.g., classification I 31], translation 4 36 37], summarization [38 39), question answering 2
They have revolutionized the domain of natural language processing (NLP). Recently, a growing body of research (20 21 40 30 41) has investigated the success of attention mechanism from the Transformer layer via its patterns, highlighting their critical roles in model behavior and network interpretability. Research shows that these patterns can be leveraged for various purposes, such as probing the internal representation relationship and reeisioonmakingg processes of the model 200, 21
30
42), and for leveraging attention sink to stabilize processing over infinite Though promising, such attempts remain largely underexplored within
input sequences 40
43
prompt-based learning [18 19 22 In this work, we question the sufficiency of current prompt- based learning designs and rethink then redesign it from the perspective of attention based on our observation (see )
2

---

Parameter-Efficient Fine- Line-Tuning, Parameter-Efficee Fine-Tuning (PEFT) 6 45 in NLP offers solutions to the computational challenges inherent adapting LLMs diverse tasks under the "pretrain-ther theniiittteeei paradigm, striving to deliver performance comparable to full fine-tuning. Generally, current PEFT methods fall into three categories: reparumeterization
7
adapter tuning [8, 46], and prompt-based learning [1] 12 17]. Among them, reparameterization and adapter tuning face two significant limitations that hinder their applications. First, they still require a substantial amount of trainable parameters reparameterization needs enormous low-rank matrices for every targeted linear layer 6 47), while adapters insert entire large modules throughout the model 88 48, 49], both resulting in heavy computational costs. Second, these approaches lack flexibility, as they often necessitate customized implementations for varying model architectures and
complicate task switching due to modifications made to the model structure [50] 51
Prompt-based learning offers a more flexible and efficient solution with minimal input sequence adjustment, thus enabling faster adaptation 117 52). Despite the current success, two limitations remain unaddressed: Soft prompts typically capture sorely the task-aware information [ 12] We
argue that adopting the current task-aware prompt-based learning may fail to capture critical content (i.e., structural information) of input sequences, ultimately impairing its capability to establish a contextual attention pattern and constraining the overall effectiveness of prompt-based learning;
IL. Prompt-based learning requires grid search to determine the optimal soft prompt length [22 44 presenting double-edged trade-off between parameter and training time efficiency (see $4.2 and Appendix {S3). In light of this view, we propose CaPT, which deliberately addresses the aforementioned issues by integrating both off-the-shelff instance semantics and task-aware guidance.
3 Methodology
We introduce Capsule Prompt-Tuning (CaPT), novel prompt- based learning approach aims to enhance LLM performance and eliminate the need for time-intensive grid search. The problem and notations are defined below, drawing on the description of P-Tuning v2 (Deep Prompt-Tuning) (53). which represents one of the strongest baselines in prompt-hased learning. The key findings of the effectiveness of instance-aware semantics and prompt-based learning attention are presented in $3.1 followed by the design of CaPT in $3.2 The overall framework is shown in Fig. 3 Preliminary for Deep Prompt-Tuning
Given a pre-trained LLM, the objective of Deep Prompt- Tuning is to adapt the model into new task with the learning of a set of continuous embeddings P = =(P). P2 ..... PN) (i.e., soft prompts), where N denotes the number of Transformer layers and P represents the learnable prompts in the i-th layer. During fine-tuning, the entire model is frozen, with the prepended learnable prompts guide the model prediction on task. Formally, the Transformer layers with prompts are defined as:
H' = Li H' ='=',(P', Hi-1) 2, ..
where the embeddings E of the input text are initialized with the embedding layer, and H is the contextual embeddings processed by the inh Transformer layer. The different colors indicate trainable and frozen parameters during fine-tuning, respectively.
3.1 Key Findings
Finding I: Task-aware prompts exhibit limited interplay with input sequences. The effectiveness of task-specific prompt-based learning is well-recognized, however, the underlying mechanism of soft prompts remains relatively underexplored area in current research [19 18
Recognizing this, we inspire by the critical role attention in facilitating effective information flow and shaping model behavior 220 2 30 55 and analyze on the attention pattern of soft prompts. This reveals a striking trend: soft prompts predominantly attend to themselves, with only subtle interaction with the rest of key input tokens. To better illustrate the point, we analyze the attention pattern over all attention heads and encoder layers on traditional Deep Prompt-Tume Tuning. As seen in Fig. (left), regular input tokens tend exhibit strong attention to critical positions within the input sequences (e-g., the 5es- 7th tokens "sentence", "I" and ":"). These tokens carry key structural information that supports both syntactic and semantic parsing, enabling the model to accurately interpret the hierarchical and contextual relationships within the input (20 56 57). In contrast, soft prompts predominantly attend to one another and struggle to form interactive attention links with input tokens, as they are discarded after each layer, often leaving them detached from key structural information. This observation
42
24
54
3

---

Acc 78.34% Standard Deep Prompt-Tumen Tuning
Ace 79.33% Prepend One Instance aware Token
Attention to Themselves Subtie Attention to Critical Tokens
Strong Attention Critical Tokens
Pronpes
aaaaa 177 Sid 311700 a 101111011111111
10 Soft Proenpts
100 7.000 mont 336 41312 1000 PITT Srnnsr
Aneen inn Seare
Weakly
Strongly Attended by Input Tokens
Attended
Input
Tokens
Query
12
Key
Sequeace
Sequenee
Figure 1: Attention Analysis T5- Attention patterns analyzed averagely across all heads and encoder layers on the RTE validation set. Darker red indicates higher attention scores, while darker blue means lower scores. The left and right figures indicate the patterns of T5-Base with 10 soft prompts and T5-Base with one additional prepended instance-award token (i.e., *) alongside 10 soft prompts, respectively. Critical input tokens refer to structurally important tokens such the token "_" at aa and the 5th- 7th tokens "sentence", "1" and 5.". A more detailed analysis of impact on specific heads is provided in Appendix $55
suggests that the current design of soft prompts is inherently limited in their ability to interact with input content like regular input tokens. Therefore, in this work, we investigate whether incorporating meaningful, instance-aware semantics input sequences and, in turn, enhance the performance of prompt-based learning.
part of prompt can foster better attentive interaction with
Finding II: Instance-aware tokens can enhance model performance without fine-tuning.
We first investigate the overall significance of eettace--aaar semantics by figuring out whether is effective enhancing LLM performance. Sur- prisingly, observe that the simplest and most direct incorporation of instance-aware information yields measurable improvements across all test exam- ples without any additional fine-tuning. Specifically, conduct our preliminary experiments T5-Base
Model Accuracy on TS-Base
Acd (%) on all examples
mr
85
80
79.3
78
74.3
75
70
65
63.0
60
59.0
58.0
52.0
54.0
(see Fig. 2), compressing input tokens into short se- quences of varying lengths (i.e., 1, 3. and 10) via pooling the input sequences. These sequences then prepended as special instance-awe aware tokens before the soft prompts at each layer. The results show that one single mnstance-awar token is able effectively improve the test accuracy on both RTE and COPA (i.e., the orange box). This suggests that incorporating instance-aware semanties, even from minimally intrusive perspective, enhances the the overall quality of guidance signals. We also find that increasing the number of prepended instance-aware tokens (e.g.. 2, 3, 4, and 10) gradually declines the performance, with accuracy dropping below that of standard Deep Prompt-Tuning, We thus acknowledge that the effectiveness of guidance signals does not depend on the quantity of information provided, but rather on how well it aligns with the model's capacity to utilize an observation consistent with prior findings prompt pruning strategies [7]. This observation is further supported by the study on CaPT length (see $4.5).
2
10
Number of Instance-awaan aware Tokens Prepended
Figure 2: Incorporating of instance-aware semantics lifts the model performance dur- ing testing without any additional fine-tuning. (0 token indicates the original accuracy with standard soft prompts only.)
F Finding III: Instance-aware tokens present strongly attentive interaction with input sequences as "attention anchor:" Recognizing the surprising performance gains from Finding II, we are excited to explore the underlying attention behavior that may explain this effect. Our analysis reveals marked contrast in how attention is distributed across the input sequence when comparing instance-aware token to fask-aware soft prompts. Unlike soft prompts, the instance-aw aware token successfully receives

---

Output
Output
Tuned
Frozen
Transformer Layer
Transformer Layer
p'
Transformer Layer
Transformer Layer
Tascaware
S²
Transformer Layer
Transformer Layer
Instance aware
m m.
Mean
1M Params
Imput Text
9K Params
Input Text
(a) Deep Prompt-Tune Tuning
(b)) Capsule Prompt-Tunie
Figure 3: Overview of Deep Prompt-Tuning Tuning us. CaPT (ours) Frameworks, (a) Original Deep
Prompt-Tuning. (b) The overall architecture of our proposed CaPT (see $3.2). integrating both lask-aware guidance and instance-aware signal to trigger "attention anchor" (see $4.3)
positive attention from input sequences (i.e., the purple box), illustrated in Fig. (right). This behavior indicates that guidance signals are actively propagated into the input sequences. Moreover, the instance-aware token also consistently directs attention toward key structural tokens of the input (i.e., the green box), a property that could be exploited to improve representation learning and gradient flow during fine-tuning, We refer to this phenomenon "attention anchor," highlighting the role of instance-awe aware semantics in establishing stronger attentive interaction between guidance signals and input sequences. Motivated by this insight, we propose an interactive and adaptive prompt design that incorporates instance-aware information. In
$3.2 we introduce the most effective and efficient design to trigger "attention anchor." Furthermore, in $4.5 demonstrate that realizing attention anchors is highly flexible and consistently yields robust performance gains across different settings. For completeness, include information on particular heads of a single instance in Appendix $55
study examining the impact of incorporating instance-aware
3.2 Capsule Prompt-Tuning
Building on these observations, we propose Capsule Prompt-Tuning (CaPT), lightweight, instance- adaptive prompt-tuning framework that eliminates costly prompt-length search while strengthening the interaction between guidance tokens and the input sequence. As shown in Fig. 3 conventional basep learning prepends fixed number of randomly initialized continuous vectors ('soft prompts") to every Transformer layer. These vectors encode task-level priors but neither adapt to individual examples nor scale gracefully across tasks, forcing practitioners to tune the prompt length by grid search. In contrast, CaPT employs a single continuous "capsule" vector per layer to compactly incorporate instance-specifie information. This capsule prompt serves as a concise carrier of guidance for the model at that layer, allowing the prompt to adapt to each input instance without introducing numerous prompt parameters, achieving parameter efficiency and strong contextual grounding.
Formally, at each Transformer layer i, we introduce learnable capsule p'. During processing, the capsule prompt 5 for layer is derived from p' in combination with mean representation derived from the model's inputs or intermediate states, and supplied to the Transformer alongside the usual sequence embeddings. For the first layer, S' is constructed from p' and the mean of the input embeddings E, and Transformer layer La processes this prompt together with E to produce the processed capsule prompt S' and sequence representation H' For each subsequent layer 2 the prompt St formed by combining p with the mean of the previous processed capsule prompt S and sequence representation Hi-1 The layer L; then consumes S and H yielding St and H'. In sum, we have:
S² = p² + Me)
S¹ H L (Sa E))
(2)
gi p¹ + Mean 1 11111
i= 2...,
St H (S Hi-l) = 2, N
5

---

where S denotes the integrated capsule prompt at layer encoding instance-speciffe semantics. e denotes concatenation and H' is the representation of the input sequence from layer
The manner in which St injects instance-aware semantics is flexible - for example, the capsule vector extra token or added into the hidden state. In other words, S is formed
can be prepended by adding the trainable capsule vector to the average embedding of the previous layer's outputs (including the previous capsule). This mean-based construction yields compact, length-invariant prompt at each layer that adapts dynamically to the input's 's evolving representations through the network. Rather than relying on a large number of soft prompt tokens, we focus on a single, concise, and informative prompt vector, capturing the essential task and instance information. By prioritizing prompt quality over quantity in this way, CaPT effectively addresses the limitations of conventional soft prompting, enabling robust performance without burdensome prompt-tuning overhead.
It is worth noticing that each prompt token is shaped by both the instance semantics and one single parameter-efficien learnable vector. Specifically, the instance semantics provide semantically rich, ansteece-aware context to trigger "attention anchor," while the learnable vector encodes task- aware inductive bias. This design enables adaptive alignment between instance information and task objectives, ensuring that the semantic guidance remains both compact and informative. Our analysis of CaPT attention (see $4.3) confirms that the capsuled instance semantics can actively guide the model's 'ss attention toward critical input features. This guidance, as a result, promotes stronger contextual alignment and enhances task performance across diverse benchmarks (see $4.2).
4 Experiments
4.1 Experimental Setup
Datasets. Following common practices [1] 23 we evaluate our approach on six Natural language understanding (NLU) corpora from the Super dataset [58]. including Question Answering task (i.e., BoolQ and MRC), Natural Language Inference task (i.e., CB and RTE), Sentence Completion task (i.e.. COPA), and Word Sense Disambiguation task (i.e., WiC). Since the official test sets are not publicly available, we follow (33 59] divide the train sets into train and validation sets by 90%/10% proportion, and translate each SuperGLUE corpus into text-to-text format. The original validation sets are considered as the test sets. More details are shown in Appendix SST
Baselines. For fair comparison, we compare our method with standard Tue-Tuning, Classification Head adaptation (i.e., only tuning the linear classification head), vanilla prompt- based learning, and several state-of -ffee-oofiiie satee-of-hh--ttttte the-art prompt- based learning approaches. More results are shown in Appendix $S4 Implementation Details. Our method built upon four different pretrained LLMs: T5-Base (220M) 4. T5-Large (770M) [4, 1lama-3.2 (1B) [60]. and Qwen-2.5 (1.5B) [1]. Specifically, we train our model under different learning rate settings for 50 epochs with early stopping, using a batch size ranging from 16 to 32. Benefiting from the simplicity and effectiveness of our design, our method does not require any hyperparameter searching which is typically tine-consuming for most prompt-based learning approaches. We confirm this advantage of our design by exploring different prompt length settings in $4.5) Detailed implementation setups are provided in $S2
Reproducibility. CaPT is implemented in Pytorch [62]. Experiments are conducted on NVIDIA RTX 6000 Ada 48GB GPUs. Our full implementation is available at htttp:://gi com/ comeandcode/CaPT
4.2 Main Results
In T we report a comprehensive comparison of CaPT with other strong baselines on six NLU tasks, resulting in two key observations. ① Robust superior performance. CaPT demonstrates consistently superior performance across both encoder-decoder and decoder-only architectures. On encoder-decoder T5 models, CaPT narrows the performance gap with full fine-tuning significantly, achieving 99.56% % of the average full fine-tuning performance on T5-Base. Remarkably, on T5- Large, CaPT not only outperforms strong baselines such as P-Tuning v2 and XPrompt but also exceeds the performance of full fine-tuning by 0.43%. Moreover, on decoder-oni causal models Llama3.2- 1B and Qwen2.5-1.52 CaPT presents significant improvement of 24.44% and 10.56% compared to Linear Head adaptation and P-Tuning v2, respectively. This consistency demonstrates the effectiveness of the attention anchor role of capsule prompt on models with different architectures. @ Extreme parameter and training efficient. CaPT benefits from its almost parameter-fre free design,

---

Table 1: Evaluation SuperGLUE Validation Sets. The best performance except full fine-tuning in bold, and the second best shown in underline. "."" and "f" indicate the results reported from 663 or its corresponding paper, respectively. For tasks with metrics, the average score is reported.
All scores are averaged over runs.
Boolq Acc
CB
COPA
MRC Fla
RTE Acc
WiC Acc
Average Score
Method
Para
F1/Acc
Ace
Ba-Base
(220M)
Fine-Tune Tuningu
100%
82.30
91.30
60.00
79.70
84.50
69.30
77.85
Prompt- Tumpng" [EMNLP1]]
0.06%
78.12
84.42
54.37
78.30
75.27
62.29
72.13
P. Tuning v2* [ACL2]]
53
0.53%
80.81
90.23
61.28
79.83
81.98
67.56
76.94
XPrompt" TEMNLP22)
17
0.04%
79.67
86.72
56.95
78.57
78.29
64.31
74.09
ResPrompt" CerPrompt"AcCie (ACL231
52
0.21% %
79.25
85.33
58.64
78.42
77.14
62.36
73.52
SMoP
2
8c-3%
79,40
86.42
58.30
79.60
77.50
65.20
74,40
SuperPos-Prompe
NNur1P2244
65
74.00
80.20
62.00
72.90
70.40
67.60
71.18
NNurSN0u1 (NorlP5224)
10
0.21%
78.38
90.92
61.76
78.73
76.90
65.36
75.34
DePTIUCLR2N
66
79.30
74.30
79.10
68.70
EPT INACL2))
67
0.06%
79.14
90.18
56.33
73,43
78.99
67.71
74.30
Ours
4e-3%
79.54
94.16
64.33
80.46
79.78
66.77
77.51
T5-1
(770M)
Fine-Tunme
64
100%
85.75
95.26
76.00
84.41
88.05
72.11
83.60
Prompt-Tune Tuning EM
0.04%
83.20
90.32
$7.50
83.10
86.11
68.74
78.16
P-Tuning v2
0.52%
85,82
95.56
77.00
84.07
89.25
71.03
83.79
XPrompt [EMNLP22
0.02%
83.82
91.39
82.05
81.26
87.72
73.51
83.29
ResPrompt" (ACL22)
52
0.15%
83.51
90.64
82.79
84.02
86.97
71.13
83.18
(MMNLEXN 23
3e-3%
83.45
92.37
71.00
83.92
87.70
68.60
81.17
VFPT
( [Neu1PS24)
10
0.18%
83.89
93.71
75.63
83.24
88.10
71.00
82.56
|NACL22
67
0.04%
84.77
93.40
54.00
80.03
86.33
71.79
78.39
Ours
3e-3%
84.56
97.22
80.00
84.53
88.45
69.44
84.03
Llama3.2-18
Linear Head
60
3e-4%
59.85
51.69
56.33
48.94
55.23
53.45
54.25
Prompt- Tuning EMNI P3
0.06%
60.95
61.61
57.67
57.00
62.50
54.70
59.07
P. P-Tuning v2 (CC
53
0.53%
62.48
64.29
61.00
60.34
58.12
60.15
61.06
|MoP|EMNLLP2
EMNLP23
23
0.04%
61.13
62.50
59.33
57.46
57.40
54.23
57.51
VFPT VFPP (Ner1PP244
10
0.17%
62.44
61.72
59.67
58.41
64.35
57.60
60.70
EPT EPT (NACL255
67
0.06%
61.56
65.22
56.00)
60.18
63.90
59.45
61.05
Ours
3e-3%
77.28
65.82
58.00
65.73
72.56
65.67
67.51
Qwen2.5-1.33
Linear Head
660
2c-4%
59.54
64.66
$2.00
53.38
62.45
56.58
58.10
Prompt-Tuning FMNI
11
0.05%
61.38
65.22
52.33
53.41
63.18
56.90
58.74
P- P-Tuning v2 (3)
0.51%
62.08
68.84
55.33
56.31
66.43
59.09
61.35
SMoP EEMNLP23)
25
0.03%
61.41
66.76
54.00
55.34
64.62
58.15
60.05
VFPT (Ner1
0.12%
63.64
67.78
$2.67
55.61
63.54
58.05
60.22
EAAINA
67
0.055%
63.10
68.17
52.33
56.02
67.53
58.30
60.91
Ours
3e-3%
64.13
72.42
57.67
57.49
68.59
58.46
63.17
achieving superior parameter-efficiency compared to all the other baselines (i.e., 0.004% parameter usage all models). Additionally, CaPT can bypass the time-consuming grid search procedure commonly required in prompt-based learning (see $4.4. 4 indicating the effectiveness of attention anchor in efficiently guiding model generation (see $4.3) An interesting observation is that both causal models exhibit suboptimal performance gain compared to two T5 models. We assume this caused by the inherently different pre-training objectives and the architecture differences (i.e.. decoder-oniy us. encoder-decoder), which aligned with other research's 's observation [68]. More analysis of CaPT length, per layer attention are conducted in $4.5 and Appendix {S6) respectively.
4.3 Attention Anchor
To strengthen our finding (i.e., Finding III) that the incorporating of instance-aware semanties can trigger "attention anchor" via enabling mutual and context-sensitive attention, we compare both model performance and attention pattern of CaPT and Deep Prompt-Tume Tuning with one soft prompt at each encoder layer on B5-Base. As shown in Fig. 4 we visualize the attention pattern on the RTE and CB validation sets of both models and have two key observations. Capsule prompt successfully exhibits more focused attention towards input tokens at the early positions of input sequence that carry critical structural information (e.g., tokens in blue boxes are fixed structural instructions for all examples in the dataset, Appendix $5). This concentrated attention effectively grounds the

---

CaPT (additon)
Deep PT (with prompt)
CaPT
(ddition)
Deep PT (witt one soft prempt!
RIE LAce 79.78%)
XTE lAce 70
ca (Scoce 94.36)
CB (Soms 57, 313
Figure 4: Attention Analysis on T5-Base. Attention patterns are analyzed averagely across all heads and encoder layers. Per layer encoder attention analysis and decoder attention (i.e., causal attention) analysis are shown in Appendix $S6 and $S7
respectively.
LLM's focus structural content and enhances the contextual relevance of the overall attention distribution 20 21 30). In contrast, the traditional prompt struggles to attend to these key tokens (i.e., black boxes), suggesting lack of targeted interaction with structurally important tokens of input sequences. II. Capsule prompt receives strong and widespread attention from all input tokens (i.e., yellow boxes), while the traditional prompt exhibits limited influence on input tokens (i.e., gray boxes). This broad attention pattern suggests that capsule prompt provides global guidance signal that is relevant throughout the input, allowing to play a cohesive and guiding role in generation. This observation aligns with previous studies on attention sinks [40] 69], which show that tokens designed to consistently receive attention can be able to enhance model performance. As result, we prove that CaPT effectively triggers the "attention anchor," demonstrating significant performance improvement compared to Deep Prompt-Tuning with single prompt (e.g., 6.73% improvement on CB) across multiple models (see Appendix $S7 for decoder-only Llama model).
4.4 Training Time Comparison
We compare the total training time of various prompt-based learning meth- ods across six SuperGLUE corpora, in- cluding the searching procedures for framework-relatee hyperparameter (e-g-, prompt length, rank). For fairness, we use a consistent experimental setup: up to 50 epochs with early stopping and a fixed batch size per task. As shown in Table 2 our default variant of CaPT demonstrates superior training efficiency, benefiting from the design that avoids grid search for prompt length optimization (see $3.2). In contrast, all baseline methods require significantly longer training durations in order to reach their optimal performances. Specifically, both P- P-Tuning v2 heavily rely task-specific searches for optimal prompt lengths, extending the overall training time (e-g-, 8.77x training time). More critically, M-IDPG and LoPA involve searching for both optimal prompt length and rank, resulting in substantially higher computational overhead (e.g., 14.93x training time). This observation further supports the effectiveness and efficiency of our design, which leveraging "attention anchor" to prioritize efficiency without compromising performance.
Table 2: Training Time Comparison on T5-Base.
Method
Para
Time
Average Score
Prompt- Turing [11
0.06%
8.77x
72.13
P-Tuning v2 [53]
0.53%
8.37x
76.94
M-IDPG 770
0.47%
12.58x
76.96
LoPA 71
0.44%
14.93x
77.98
Ours
4e-3%
1.00x
77.51
Tuning and
4.5 Ablation Study
We include performance comparison with other PEFT methods in Appendix $$4 and analyze the differences between CaPT and other instance-incorppratte prompt-based methods in Appendix $3
CaPT Variants. As stated in $3.2 the capsulation of instance-aware semantics can be achieved flcxibly. Here we include three additional designs of CaPT that combines instance-aware and task-aware information, which are prepending. extraction, and projection, as shown in Fig. 5 For prepending, we prepend the mean representation of input instance to learnable task prompt as independent tokens. For extraction and projection, we consider employing learnable 1D convolutional filters and learnable low-rank linear layers to capture instance features respectively, before integrating with the task-aware vector. The results in Table 3 shows that these variants able
Table 3: Comparison of CaPT Variants.
Variant
# Para
Average Score
Addition
4e-3%
77.51%
Prepending
4e-3%
77. 77.44%.
Extraction
0.03%
77.21 %
Projection 0.07%
77.64%
8

---

Prepending
Extraction
Projection
S
S
S
1D Conv
Down
FFN
S!
Figure 5: CaPT Variants. Three additional designs of CaPT are illustrated: Prepending, Extraction and Projection. The latter two designs require additional trainable modules to process hidden state (e.g-, 1D CNN, low-rank Linear). Of all the variants, our default addition design offers the best balance of efficiency and effectiveness, see Table 3
to exhibit consistent performance on T5-Basc. Notably, the default addition design 3$2.2 can achieve competitive performance compared to projection (i.e.. 77.51 % vs. 77.64%) with substantial fewer parameter usage (i.e., 17.5% x lower). Considering the substantial reduction in time by eliminating the need for time-intensive grid search during fine-tuning, adopt addition
default method.
CaPT Length. In Fig. (top), we explore whether increasing CaPT length is helpful for capturing quan- titatively more information to exhibit better perfor- mance on both T5-Base (220M) and Llama3.2-1B. The results reveal that employing one single capsule prompt is sufficient enough to effectively guide model adapta- tion. Specifically, we observe significant performance drops when prompt length increases (e.g., 67.51 us. 59.38), while shorter prompt lengths exhibit competitive scores compared with single capsule prompt (i.e., two capsule prompts) on both models. This observation consist with our Finding II (see I3.I). indicating that the effectiveness of prompt guidance depends not on the amount of information provided, but on how well that information matches the model's s ability to utilize (i.e., in our case, observed from the attention patterns).
80
77.51
77.44
1.-ma322-18
75
73.81
73,85
Average Seore
71.13
70
67.04
63.97
65
62.13
59.38
Prompt Length
15-Base
77,51
Lhma3
75.28
73.3
70,55
CaPT Depth. We investigate the impact of applying CaPT at different sets of Transformer layers, following common practices 9 22 722. Specifically, we evalu- ate five configurations: (a) the input layer; (b) the first half of the layers; (c) the latter half the layers; (d) every odd-numbered layer; and (e) all layers. As il- lustrated in Fig. 6 (bottom), performance improves as CaPT is applied deeper in the model for both T5-Base and Llama3.2-1 BBBB. Interestingly, the configuration using every odd-numbered layer outperforms both the "first half" and "latter half" settings (i.e., 75.28% us. 73.67%) suggesting that sparsely distributing CaPT throughout the model may be more effective than concentrating it within a consecutive block.
67.51
Average
65.08
56.52
63.75
59.69
60
55
(8)
(b)
(6)
(d)
(6)
Prompt Depth Configurations
Figure 6: CaPT Length & Depth. The top figure shows performance across different prompt lengths, while the bottom figure illustrates the impact of prompt depth.
5 Conclusion
Current approaches that adapt LLMs to downstream tasks through task-aware prompt-based learning constrained by limited interaction with input sequences and often rely on time-consuming grid searches to determine the optimal prompt length. Motivated by the significant role of instance- token guiding model generation "attention anchor," we propose CaPT a simple yet effective framework that bridges between instance-aware and task-aware guidance signals to provide mutual and contextual attention interaction. CaPT achieves robust superior performance and eliminates need for laborious and time-consuming prompt length searching in almost parameter-free manner, offering an innovative perspective on L.MM adaptation. We conclude that the outcomes elucidated in this paper impart essential understandings and necessitate further exploration within this realm.

---

Acknowledgments
This research was supported by the National Science Foundation under Grant No. 2450068. This work used NCSA Delta GPU through allocation CIS250460 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by U.S. National Science Foundation grants No. 2138259, No. 2138286, No. 2138307, No. 2137603, and No. 2138296.
References
[1] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2018.
[2] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):166) 2020.
[5] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. V'ppt: An effective and efficient approach for visual prompt tuning. In ICCV, 2023.
[6] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[7] Yiyang Liu, James Chenhao Liang, Ruixiang Tang, Yugyung Lee, Majid Rabbani, Sohail A. Dianat, Raghuveer Rao, Lifu Huang, Dongfang Liu, Qifan Wang, and Cheng Han. Re-imagining multimodal instruction tuning: A representation view. In ICLR, 2025.
[8] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for NLP. In ICML, 2019.
[9] Taowen Wang, Yiyang Liu, James Chenhao Liang, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, al. M pt: Multimodal prompt tuning for zero-shot instruction learning. In EMNLP, 2024.
[10] Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huangg, Ying Nian Wu, and Dongfang Liu. Visual fourier prompt tuning. In NeurlPS, 2024.
[11] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. In EMNLP, 2021.
[12] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACI/LICNLP, 2021.
[13] Taylor Shin, Yasaman Razeghi, Robert I.. Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP, 2020.
[14] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient--ased discrete optimization for prompt tuning and discovery. In NeurlPS, 2023.
[15] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL/JCNLP, 2021.
10

---

[16] Liqi Yan, Cheng Han, Zenglin Xu, Dongfang Liu, and Qifan Wang. Prompt learns prompt: Exploring knowledge-and wwware generative prompt collaboration for video captioning. In IJCAI, 2023.
[17] Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. Xprompt: Exploring the extreme of prompt tuning. In EMNLP, 2022.
[18] Yihan Wang, Jatin Chauhan, Wei Wang, and Cho-Jui Hsieh. Universality and limitations of prompt tuning. In NeurlPS, 2023.
[19] Samet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, and Christos Thrampoulidis. On the role of attention in prompt-tuning. In ICMI., 2023.
[20] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher Manning. What does BERT look at? an analysis of bert's 's attention. In BlackboxNLP, 2019.
[21] Joseph F DeRose, Jiayao Wang, and Matthew Berger. Attention flows: Analyzing and comparing attention mechanisms language models. IEEE Transactions on Visualization and Computer Graphics, 27(2):1 160-1170, 170, 2020.
[22] Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, and Dongfang Liu. Facing the elephant in the room: Visual prompt tuning or full finetuning? ICI.R, 2024.
[23] Joon- Young Choi, Junho Kim, Jun uun-Hyng Park, Wing-Lam Mok, and SangKeun Lee. Smop: Towards efficient and effective prompt tuning with sparse mfxture-of -oomotppss........omooooo In EMNLP, 2023.
Joon-Young
[24] Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Han- naneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. Prompt waywardness: The curious case of discretized interpretation of continuous prompts. In NAACL, 2022.
[25] Junda Wu, Tong Yu, Rui Wang, Zhao Song. Ruiyi Zhang, Handong Zhao, Chaochao Lu, Shuai Li, and Ricardo Henao. Infoprompt: Information-theerrtic soft prompt tuning for natural language understanding. In NeurIPS, 2023.
[26] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 1997.
[27] Alex Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (Istm) network. Physica D: Nonlinear Phenomena, 404:132306 2020.
[28] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In NeurIPS, 2021.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. In ICLR, 2020.
[32] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 2022.
[33] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond fixed-length context. In ACL, 2019.
[34] Yanzhou Pan, Huawei Lin, Yide Ran, Jiamin Chen, Xiaodong Yu, Weijie Zhao, Denghui Zhang, and Zhaozhuo Xu. Alinfik: Learning to approximate linearized future influence kernel for scalable third-party 1lm data valuation. In NAACL, 2025.

---

[35] Huawei Lin, Jikai Long, Zhaozhuo Xu, and Weijie Zhao. Token- wisen-wise influential training data retrieval for large language models. In ACL. 2024.
[36] Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng
Tu, Document-level machine translation with large language models. In EMNLP, 2023,
[37] Menglong Cui, Jiangeun Du, Shaolin Zhu, and Deyi Xiong. Efficiently exploring large language models for document-level machine translation with in-context learning. In ACL, 2024.
[38] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetume language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
[39] Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. Pretraining based natural language generation for text summarization. In CoNLL, 2019.
[40] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, 2024.
[41] Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang. Massive values in self-attention modules the key contextual knowledge understanding, arXiv preprint arXiv:2502.01500 2025.
[42] Goro Kobayashi. Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In EMNLP, 2020.
[43] Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- infinite: Zero-shot extreme length generalization for large language models. In NAACL, 2024.
[44] Pengxiang Lan, Enneng Yang. Yuting Liu, Guibing Guo, Jianzhe Zhao, and Xingwei Wang. EPT: efficient prompt tuning by multi-space projection and prompt fusion. In AAAI, 2025.
[45] Runjia Zeng, Guangyan Sun, Qifan Wang. Tong Geng, Sohail Dianat, Xiaotian Han, Raghuveer Rao, Xueling Zhang. Cheng Han, Lifu Huang, et al. Mept: Mixture of expert prompt tuning as a manifold mapper. In EMNLP, 2025.
[46] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Ke-Wei Lee. JIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In EMNLP, 2023.
[47] Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. Alora: Allocating low-rank adaptation for fine-tuning large language models. In NAACL, 2024.
[48] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NeurlPS, 2017.
[49] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In NeurlPS, 2020.
[50] Shibo Jie, Haoqing Wang, and Zhi-Hong Deng. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. In ICCV, 2023.
[51] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In ECCV, 2020.
[52] Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameterization. In ACL, 2023.
[53] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. In ACL, 2022.
[54] Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. In NeurIPS, 2021.
12

---

[55] Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Martin Jaggi, Hyeji Kim, and Michael Gastpar. Attention with markov: A framework for principled analysis of transformers via markov chains. arXiv preprint arXiv:2402.04161, 2024.
[56] Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang, Bo Tang, Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey. arXiv preprint STt::24099037522 2024.
[57] Dongjun Jang, Sungjoo Byun, and Hyopil Shin. A study on how attention scores in the BERT model are aware of lexical categories in syntactic and semantic tasks on the GLUE benchmark. In LREC/COLING, 2024.
[58] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurlPS, 2019.
[59) Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. Revisiting parameter- efficient tuning: Are we really there yet? In EMNLP. 2022.
[60] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXin:2407.21783 2024.
[61] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15111 15115, 2024.
[62] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetooo, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[63] Qifan Wang. Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, and Dongfang Liu. Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models. In EMNLP, 2023.
[64] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. Ext5: Towards extreme multi-task scaling for transfer learning. In ICLR, 2022.
[65] Mohammad Ali Sadraei Javaheri, Ehsaneddin Asgari, Alice C. McHardy, and Hamid R. Rabiee. HuperPos-Promptt Enhancing soft prompt tuning of language models with superposition of multi token embeddings. In NeurIPS Workshop, 2024.
[66] Zhengxiang Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. In ICLR, 2024.
[67] Pengxiang Lan, Haoyu Xu, Enneng Yang, Yuliang Liang, Guibing Guo, Jianzhe Zhao, and Xingwei Wang. Efficient and effective prompt tuning via prompt decomposition and compressed outer product. In NAACI, 2025.
[68] Bokai Hu, Sai Ashish Somayajula, Xin Pan, Zihan Huang, and Pengtao Xie. Improving the language understanding capabilities of large language models using reinforcement learning. arXiv preprint arXiv:2410.11020, 2024.
[69] Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. In ICML, 2024.
[70] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V. G. Vinod Vydiswaran, and Hao Ma. IDPG: an instance-dependen prompt generation method. In NAACL, 2022.
13

---

[71] Abhinav Jain, Swarat Chaudhuri, Thomas W. Reps, and Christopher M. Jermaine. Prompt tuning strikes back: Customizing foundation models with low-rank prompt adaptation. In NeurlPS, 2024.
[72] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[73] Wei Zhu, Aaron Xuxiang Tian, Congrui Yin, Yuan Ni, Xiaoling Wang, and Guotong Xie. IAPT: instance- aware prompt tuning for large language models. In ACL, 2024.
[74] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. In EMNLP Findings, 2020.
[75] Dejiao Zhang, Wasi Uddin Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. Code representation learning at scale. In ICLR, 2024.
[76] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In NeurIPS, 2023.
[77] Guolin Ke, Di He, and Yan Liu. Rethinking positional encoding in language pre-training. In ICLR, 2021.
[78] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127063, 2024.
[79] Yu-An Wang and Yun-Nung Chen. What do position embeddings learn? empirical study of pre-trained language model positional encoding. In EMNLP, 2020.
[80] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[81] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022.
[82] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NewrlPS, 2022.
[83] Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layernorm in transformers. In NeurlPS, 2024.
[84] Huawei Lin, Tong Geng, Zhaozhuo Xu, and Weijie Zhao. Vtbench: Evaluating visual tokenizers for autoregressive image generation. arXiv preprint arXiv:2505.13443 2025.
[85] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ:: A question answering challenge targeting commonsense knowledge. In NAACL-HLT, 2019.
[86] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. Ethical and social risks of harm from language models. arXiv preprint arXiv:21
2021.
[87] Aiwei Liu, Qiang Sheng, and Xuming Hu. Preventing and detecting misinformation generated by large language models. In SIGIR, 2024.
[88] Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min- Yen Kan, and William Yang Wang. On the risk of misinformation pollution with large language models. In EMNLP Findings, 2023.
14

---

[89] Boxin Wang, Chejian Xu, Shuohang Wang. Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial GLUE: A multi-task benchmark for robustness evaluation of language models. In NeurlPS Datasets and Benchmarks, 2021.
[90] Ethan Perez, Saffron Huang, H. Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In EMNLP, 2022.
[91] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprini vrXiv:2209.07856. 2022.
[92] Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. Assessing factual reliability of large language model knowledge. In NAACL, 2024.
[93] Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. Auditing large language models: three-layered approach. Al and Ethics, 4(4):1085-1115, 2024.
[94] Alejandro Salinas, Amit Haim, and Julian Nyarko. What's in name? auditing large language models for race and gender bias. arXiv preprint arXiv:2402.14877 2024.
[95] Milind Shah and Nitesh Sureja. A comprehensive review of bias in deep learning models: Methods, impacts, and future directions. Archives of Computational Methods in Engineering, 32( 52())2255-267, 2025.
[96] Yunbei Zhang, Akshay Mehra, Shuaicheng Niu, and Jihun Hamm. DPCore: Dynamic prompt coreset for continual test-time adaptation. ICML, 2025.
[97] Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang, Yuxiang Wei, Jihun Hamm, and Min Xu. Visual instance-aware prompt tuning. In ACM Multimedia, 2025.
[98] Yunbei Zhang, Akshay Mchra, and Jihun Hamm. Ot-vp: Optimal transport-guidee visual prompting for test-time adaptation. In WACV, 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Motivated by our finding of "attention anchor," we propose CaPT, a simple yet effective method that integrates both instance-aware and task-aware information facilitate prompt-based learning. The main contributions of CaPT (i.e., interactivity, training and parameter efficiency) are claimed in both the abstract and introduction accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in $13 in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification; Our work is based on our observation on attention pattern which does not have theoretical results. We provide assumption and confirmation the "attention anchor" phenomenon in $3. and respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We claim reproducibility in both $4.1 and Appendix publicly available after acceptance.
S12
Our code will be
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We claim reproducibility in both $4.1 and Appendix $ST2 All the datasets included in our study are publicly available (SuperGLUE). Our code will be publicly available after acceptance. The publicly available code should be adequate replicate the primary experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
gwwde//dde//ee http///oooooiisiiiiniiia ssionoolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in $4.1 and Appendix $2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We include the standard deviation error bars for our main result on T5- Base in Appendix $S3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We specify the training computation time $4.4 and computing resources in 4.1 and Appendix $12
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:://wwwaappppppppppp/cca/aaa
Answer: [Yes]
Justification: We conform to the NeurlPS Code of Ethics and show related asset license and consent to our work Appendix $S1T.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Our work is a parameter-efficien---e----u (PEFT) approach for large language models. We provide the potential misuses of our method and mitigation approaches in Appendix $S10
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We dicuss the potential misuses and prevention approaches in STO in Ap- pendix.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We show related asset license and consent to our work in $SI
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Justification: We do not release new assets. In Appendix ST1 we include the existing asset license and consent.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our paper does not use L.Ms as any components of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

SUMMARY OF THE APPENDIX
This appendix contains additional experimental results and discussions of our NeurlPS 2025 submis- sion: All You Need is One: Capsule Prompt Tuning with a Single Vector, organized as follows:
S1
provides an additional introduction of the datasets and important terminology explanation.
explains more implementation details on experiments.
provides comparison with existing instance-incorporater prompt-based learning.
presents a comparison with other PEFT paradigms.
$5
provides case study of Finding III 3.1 on attention heads.
S6
discusses the attention dynamics across different encoder layers.
illustrates the role of capsule prompt on decoder-only model with causal attention.
provides an additional ablation study of task-awar and instance- guidance signals.
provides additional validation of CaPT's applicability.
STO
includes additional discussions ethics concerns.
S11
shows related asset license and consent to our work.
S12
claims reproducibility of our approach.
S13
discusses limitations, social impact and directions of our future work.
S1
Dataset Statistics & Terminology
Table S1: More details on the six SuperGLUE corpora used in experiments. Datasets
classified into different task categories, as suggested by 4 NLI denotes natural language inference, QA denotes questions and answers task, SC denotes sentence completion, WSD denotes word sense disambiguation, Acc denotes accuracy, and Fla denotes the macro Fl score.
Corpus
Examples
Task
Domain
Metric
Boolq
9,427
QA
Wikipedia various
Acc
CB
250
NLI
F1/Acc
CОРA
400
SC
blogs, encyclop
Acc
MRC
27,243
QA
various
Fla
RTE 2,490
NLI
news, Wiki
Acc
WiC
5,428
WSD
lexical databases
Acc
Table ST shows details of the six corpora from SuperGI.UF benchmark [58] that we used for our experiments, along with their training sizes and evaluation metries. For tasks that have two evaluation metries, we use the average of both scores as the final performance metric following
We would like to further explain the important terminology used in our work:
Task-aware refers to information components (e.g., soft prompts) specifically designed opti- mized to encode general knowledge or instructions about the dataset/task. These prompts remain the same across input instances and aim to guide the model toward task-relevant behavior.
Instance-aware describes information tailored to specific input instance (e.g., sentence or document). Instead of generic task-aware instructions, instance-aware tokens reflect the feature of each individual input.
Structurally important tokens are the critical components of the input sequence that carry essential meaning structure (e.g., named entities, syntactic anchors). Models benefit from focusing attention on these tokens to ensure accurate comprehension and generation (0 56 57
Guidance signal explicit implicit instructions (e.g., hard or soft prompts) that guide models decision- making processes toward targeted tasks or behaviors.
S2 Implementation Details
Our method employs four different pretrained L.MMs: T5-Base (220M) 4. T5-Large (770M) 4. Llama-3.2 (IB) (60], and Qwen-2.5 (1.5B) [61] Specifically, train models in float32 precision
23

---

Table S2: Comparison on SuperGLUE validation sets for T5-Base. Training time refers the total duration required to train for up to 50 epochs with early stopping to obtain the optimal performance, using the same batch size for individual task.
Training
Boolq Asc
COPA
MRC
Average Scaue
Ace
Fia
Ace
T5-Base (2200))
01065
8.77%
78.12
$4.37
78.30
02.29
72.13
P-Thiing 53
0.53%
50.81
50.23
29.83
$1.98
67.56
76.54
M-MPil 20
0.42%
79.30
07:31
79:00
0.44%
81.00
91.54
80.41
77.98
4e-3%
(0.59)
(0.33)
79.78 (0.76)
(0.12)
27.51L (0.33)
for 50 epochs with early stopping based on validation results, using a batch size ranging from 16 to 32 to avoid memory issues. All scores reported based on the average of three runs. For T5 models, we linearly search the best learning rate from (5, 1, 0.1): for Llama3.2-18, IB, we linearly search the best learning rate from (7e-4, 5e-5, 1e-5); for @wen2.5-1.55. we linearly search the best learning rate from (1e-1, 5e-3, 5e-4, le-5). Benefiting from the simplicity and effectiveness of our design, our method does not require any hyperparameter searching which is typically laborious and time-consuming for most prompt-based learning approaches. This advantage is confirmed by our exploring of different prompt length settings in $4.5
Following common practices [I] 231, we set the maximum input length, including the prompt. to 512 tokens for all experiments. Inputs that exceed this limit are truncated. We do not apply any additional preprocessing (e.g., punctuation removal); instead, we directly tokenize the raw text from the SuperGLUUE datasets using the appropriate tokenizer for each model. All experiments adhere to the SMoP 23 formatting, where classification tasks are reformulated into text-to-text format in T5 model. For example, in BoolQ, labels '0' and 'I' converted to 'True' and 'False,' respectively. For T5 models, we translate each SuperGLUE dataset into a text-to-text format following [23]. For Llama and Qwen models, we continue to use the previously established text-to-text template while preserving the original labels, aligning the task with Llama SeraaaooreuunceClaassiii Pren2FFoSSeqenneCClssiiiccc
and respectively. All our models use the Adafactor optimizer with linear learning rate scheduler.
S3 Comparison with Existing Instance-incorporated prompt-based learning Approaches
While some prompt-based learning methods 770 71 73 have explored utilizing instance semantics, our approach distinguishes itself significantly through its simplicity, efficiency, and minimal over- head. We conduct comparison with two representative instance-based prompting methods across multiple tasks. As shown in Table S2] our method surpasses M-IDPG (70) on average performance and achieves competitive performance compared to LoPA 7 on T5-Base. While LoPA reports marginally better average performance, it requires 110x more trainable parameters (i.e., 0.44% us. 0.004%), which is critical overhead under the PEFT paradigm. It requires additional encoder (e.g., CodeBert- 125M (74), CodeSage-3653 365M 775)), though frozen, still being a major reason of the significant overhead on training time. In contrast to intuitive yet effective leveraging of "attention anchor", both methods employ heavy trainable modules (e.g., MI. and multiple projectors).
S4 Comparison with Other PEFT Paradigms
For a comprehensive evaluation of the performance of our method, we compare CaPT against other PEFT methods that differs from prompt-based learning, including Adapter [ and LoRA O While several recent reparameterization and adapter tuning methods have been proposed on different backbones and datasets, the absence of publicly released code hinders reproduction under consistent settings. As shown in Table [3] CaPT able to surpass the other PEFT baselines (e.g., 77.51% % ws. 76. 76%1%) while requiring a considerably fewer trainable parameters (e.g., 0.004% vs. 1.73%). This demonstrates a unique advantage of our method, highlighting ability achieve superior performance with significantly reduced parameter usage.
24

---

Table S3: Comparison with other PEFT methods SuperGLUUE for T5-Base.
Boolq Acc
CB
COPA
MRC Fla
RTE Acc
WiC Acc
Average
Method
Para
F1/Acc
Acc.
Score
T5-Base (220M)
Adapter g
0.86%
82.50
88.05
71.50
75.90
71.90
67.10
76.16
LoRA 6
1.73%
81.30
88.20
70.40
72.60
75.5
68.30
76.05
Ours
4c-3%
79.54
94.16
64.33
80.46
79.78
66.77
77.51
Head 0-6
Head 0-7
Christophar
Input 19: Christopher
input Christopher input 18: actor
Input 19: Christopher
Input 18: actor
Input 18: actor
Inpu: 18: actor
Input 17:
Input 17:
input 16:
input input 15: widow
InpuE 16: 0f
input 15: widew
Input 15: widdww
Input 15: widow
imput 14:
input the
13:
input
Input 12: ve
input 12: ve
Input 12: ve
input
input 11:e
10: Re
10: Re
input 8:
ingut 9:
input input 8: Dan
Input
input 8: Dan
input
Input
Input 6:
input 6:
Input
Input 6:
input sentence
input sentence
input sentence
Input sentence
input
Inpat
ingut 3:
3:
impu: 3:1
input 2:
input 2:
Input 2:
Input
promot 10
prompt 10
promot 10
prompt 10
grompt
prongt
prompr
prompt
prampe
prampo
prompt
promat /
grompt
prompt
prompt
prompt
prampt
prampo
prompt
promat 4
grompt
prompt
prompt
promat
grompt
promat
pramp:
promat
instance aware
netance-aware
instance-aware
Setance-awan
Source
Target
Source
Target
Head 2-0
Head 5-7
input 19: Christopher
Ingut 18: Christapher
input 19: Christopher
Ingut 19: Christopher
input actor
input actor
Input 17: The
Input 17: the
input 16: of
ingit 16:
input 16: of
Inaut 16: of
input widow
Ingut
input 15: widow
Input
14: the
input
14: the
input 13:
Input 13:
Imput 13
Input 13:
ingut ve
Ingut
input 12: ve
input
11:c
11:2
Input 10: Rd
Input 10: Re
input 10: R0
Inpu: 10: Re
input
Ingut
inpuz a
input
Dan
Dan
Input 7
Input: 7:
Inout 7:
Inpu:
input 6:
Ingut
input
input
input
Input 41
input
Input (1:
Input
input 3:
input
Input 3:1
inbut
Input 1:
Input 1
Imput 1:
prompt 10
promit 10
promot 10
prompt 5
prompt
prampe
prompt
promat B
prompt
promgt
grompt
prompt 7
prompt
prampe
prompt
promat 5
prompt
promgt
grompt
prompt
prompt
prompe
prompt
promat
prompt
prompt
prompt
prompt
instance-awate
instance-aware
instance-ewate
Source
Target
Source
Target
Figure S1: Attention Heads Analysis of Incorporating Instance-aware Information. Head a-b
denotes the attention pattern of head b in encoder layer a. Darker lines represent higher attention scores. Token indices displayed in the same format as in Fig. (right).
S5 Impact of Incorporating Instance-aware Information on Attention Heads
We present detailed case study of Finding III in $3.1 showing how prepended instance-aware information affects the attention heads of the T5-Base model. Examples of heads are shown in Fig, ST We have two key observations. First, when input tokens strongly attend to structural tokens (e.g.. input 7 ":" in head 2-0) and semantic tokens (e.g., input "Dan" in head 5-7), the instance-aware token also able to attend to these tokens, whereas prompt tokens primarily attend themselves. Second, the instance-aware token is effectively attended by input tokens (e.g., heads 0-6 and O-7), thereby providing guidance to the entire input sequence. These observations reinforce the role of instance-awar information "attention anchor," which strongly motivates our design of CaPT.
25

---

Layer
Layer 5
Layer
Loyer 11
Deep PT (1 prompt)
Atunden Soor
Layer
Layer
Layer
CaPT
Figure S2: Per Layer Averaged Attention on CB Validation Set for T5-Base. Yellow regions indicate the attention behavior of capsule prompt, while blue regions show the attention behavior of traditional task-aware soft prompt.
S6
Per Layer Attention
To further understand the "attention anchor" phenomenon, investigate the averaged attention dynamics across different Transformer layers. Specifically, we showcase four layers (i.e., layer index 8, and 11) of both CaPT and Deep Prompt-Tuning models in Fig. S2 to examine the significance of capsule prompt. We observe that compared to traditional deep prompt, capsule prompt is able to consistently exhibit more focused attention towards critical input tokens and strong guidance to other input tokens across all layers (i.e., yellow regions). This aligns with our averaged attention analysis across all layers $4.3) further confirming our capsule prompt can serve as "attention anchor" to analyze the
facilitate the interaction between guidance signals and input sequences. Additionally, attention logits after applying relative positional encoding (PE) in T5, and after applying both rotary PE and causal masks in Llama. Consequently, the averaged attention scores at diagonal positions may become negative, enabling more balanced distribution of attention toward non-self tokens, as influenced by PE mechanisms 76 77
78
79 80
81]
S7 Decoder-only Attention
Transformer decoders have a fundamentally different attention mechanism (i.e., causal attention) compared to Transformer encoders [30
82
33
84). The paradigm of prompt-based learning on
encoder-decoder T5 models focuses on applying prompts at encoder layer for better adaptation to downstream tasks 23 17 12 I1 63) Therefore, we further investigate the role of capsule prompt on decoders through decoder-only architecture (e.g., Llama model). As shown in Fig. [3] even under causal attention, where tokens cannot attend to subsequent tokens, the capsule prompt is able to demonstrate superior guidance than the traditional soft prompt. This strong guidance role aligns with our observation the T5-Basc encoders. The higher performance gain (i.e., 65.82 62.11) suggests that concentrated attention on the first token positively impacts model performance, consistent with previous studies on attention sinks in decoder-only models [40 [69. We also find that attention scores always concentrate on the first token the input sequence (i.e., the BOS token) for both models. This phenomenon may be attributed patterns learned during pre-training.
S8 Impact of Task-aware and Instance-aware Guidance Signals
To understand how different forms of guidance signal contribute performance, we separately exam- ine task-aware (i.e., Deep PT with single prompt) and instance-aware (i.e., only the mean pooled
26

---

Score
Deep
495
Attention Score
Quay
Quay
Key
Figure S3: Attention Analysis on Llama3.2-1B. Attention patterns analyzed averagely across all heads and decoder layers on CB validation set. The left figure indicates the attention pattern of CaPT, while the right figure shows the attention pattern of Deep Prompt-Tuning with one single prompt each layer.
Figure S4: Attention Patterns for the Instance-aware Guidance Only Setting.
embedding as a prompt) guidance. As shown Table S4 the absence of either type of guidance signal results in performance drop, suggesting that both contribute meaningfully. Interestingly, without any fine-tuning, instance-aware only signals can guide the model, outperforming Deep PT with single prompt tasks like BoolQ and WiC. Although the attention anchor effeet also persists for the instance- aware only setting (see Fig. S4h, CaPT achieves the best performance, validating the benefit of combining both the instance-avee and task-aware guidance.
Table S4: Ablation study of task-aware and instance-aware guidance signals for T5-Base.
Booly Acc
CB
COPA Acc
RTE Acc
WiC Acc
Method
F1/Ace
T5. Base (220M)
Insta-only-
77.25
82.86
52.00
65.70
65.36
Deep PT-1
76.51
87.31
58.00
76.90
63.80
CaPT
79.54
94.16
64.33
79.78
66.77
S9 Further Validation of CaPT's Applicability
To further examine the applicability of CaPT, we include the comparison of P-Tuning V2 and CaPT on the CSQA 85] dataset across different models Table including results on a larger model Qwen2.5-7B. The results indicate CaPT's applicability on larger models and show that CaPT can consistently outperform P-Tuning V2. Specifically, CaPT achieves a notable performance improvement on Llama3.2 1omma3.2-1B. 2-B highlighting the effectiveness of CaPT on CSQA.
27

---

Table S5: Results CSQA comparing P-Tuning V2 and CaPT across different models.
Method
T5-Base
T5-I.rge
Llama3.2-1B
Qwen2.5-7B
P-Tuning V2
56.43
70.52
45.29
77.72
CaPT
57.82
72.07
61.60
79.03
S10
Ethics Concerns
CaPT is a parameter-efficiet fine-tuning (PEFT) method designed to adapt pre-trained large lan- guage models (LLMs) to downstream tasks. However, it also introduces potential risks of misuse. Specifically, malicious actors may fine-tune models to generate or amplify harmful content, misinfor- mation, biased outputs [86] 87 88 To address these concerns, several mitigation strategies be considered. These include robustness evaluations [89] 90, 91], continuous monitoring of model behavior [92], and systematic bias audits [93] (94). Another important safeguard is the thorough documentation of models and training data, along with transparent disclosure of any known biases introduced during development [95].
S11
Asset License and Consent
The majority of prompt tuning 53 T5 [64], and Qwen2.5-1B [6] are licensed under Apache- 2.0 Llama3.2 IB [60] is licensed under Llama 3.2 Community License Agreement SuperGLUE is
licensed under MIT.
All the datasets included in our study are publicly available (SuperGLUEE) and all the models (T5 models, Llama3.2-1B and @wen2.55-1.53 are publicly available. We would like to state that the contents in the dataset do NOT represent views opinions.
S12 Reproducibility
CaPT is implemented in Pytorch (62]
Experiments are conducted on NVIDIA RTX 6000 Ada 48GB GPUs. To guarantee reproducibility, our full implementation available /tttps:/tgttiiiiiee tgtgggtthuu.... com/ comeandcode/Con CaPP Implementation details are included in Appendix $2
S13 Social Impact and Limitations
This work provides a prior finding of incorporating instance-aware information as a part of guidance signals in prompt-based learning facilitate more attentive interaction between prompts and input sequences, namely "attention anchor." Based on our finding of "attention anchor", we propose Capsule Prompt-Tuning (CaPT), extremely lightweight, instance-adaptive prompt-based learning framework that eliminates prompt-length search while strengthening the interaction between guidance tokens and the input sequence, leading superior performance and outstanding training efficiency. Our work is particularly beneficial for parameter-sen sensitive training scenarios, such as prompt tuning on cosstrce- devices and rapid adaptation with limited computational overhead.
potential limitation of the current CaPT design the extension models of other modalities (e.g., visual 772 96 97 98),, since visual embeddings generally lack fixed instruction template (i.e., structural tokens) like those in language tasks, which the capsule prompt uses to ground more focused attention. However, given CaPT's strong guiding role, as indicated by the high attention it receives in T5 encoder layers, we believe that a carefully redesigned version CaPT could also benefit vision tasks. In addition, while results consistently show successful optimization of CaPT across broad range of datasets and model scales, we acknowledge that this may not generalize to all models and datasets. Further comprehensive examinations needed.
28


Reviews:
[{"review_id": "UcBdg4Gq5b", "reviewer": "Reviewer_Etjy", "summary": "This paper introduces Capsule Prompt-Tuning (CaPT), a novel approach to enhance prompt-based learning for large language models (LLMs). The authors identify that existing task-aware prompts lack instance-aware information, resulting in suboptimal attention mechanisms. This method efficiently integrates both task-aware and instance-aware information using a single capsule prompt, achieving high parameter efficiency.", "questions": "see weakness", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper provides a compelling attention analysis, showing that current task-aware prompts exhibit limited interplay with input sequences. The comparative results lend credibility to this claim.\n2. The authors demonstrate the effectiveness of the proposed CaPT method. Experimental results on the LLaMA3.2-1B model show that CaPT achieves performance gains while updating only 0.003% of model parameters and requiring only 1/8.77 of the training time compared to other methods.\n3. The paper is well-written. Terminology is clearly introduced, and the sections are well-structured, allowing for smooth and easy reading.\n\n\n**Weaknesses:**\n\n1. **Scope of Applicability**.\n   (1) Although CaPT is evaluated on several natural language understanding tasks, these tasks primarily involve text classification and relatively simple QA settings. Its effectiveness on more complex reasoning tasks—such as mathematical or commonsense reasoning—remains uncertain.\n   (2) Furthermore, the experiments are limited to relatively small models (220M, 770M, 1B, 1.5B). It is unclear whether CaPT remains effective on larger-scale LLMs (e.g., 7B or above).\n   (3) According to Table 1, CaPT does not outperform all baselines across all tasks. Why is this the case? What kinds of tasks is CaPT particularly suited for? \n   If the authors could further validate CaPT on tasks such as MATH500 or CSQA and larger models like 7B+, I would be more inclined to raise my score.\n\n2. **Novelty of the Method**.\n   In Section 3, CaPT is described as integrating both task-aware and instance-aware information using a single capsule prompt. Are there prior works that also attempt to combine these two types of information? If so, how does CaPT differ? If not, what are the challenges of integrating both, and how does CaPT address them? These points should be clarified more explicitly, preferably in the Introduction or Related Work section.\n\n3. **Related Work**.\n   The Related Work section could be more focused. For example, general background on large language models may not be necessary. The section would benefit from highlighting work most relevant to the core contribution of CaPT.\n\n4. **Typoes**:\n\n   * Equations (1) and (2) do not appear to be properly typeset, which may not meet academic presentation standards.\n   * Terminology is sometimes inconsistent. For example, “capsule prompt” and “capsule vector” are used interchangeably when describing CaPT, which could confuse readers.\n\n5. **Experimental Details**:\n   The paper lacks important implementation details. For example, how is the capsule prompt obtained during training? While the authors mention that the code will be released after acceptance, it would be better to provide an anonymous repository or add key implementation details to improve reproducibility.\n\n6 **Other Concerns**\n6.1.The central claim that CaPT functions as an “attention anchor” relies solely on attention heatmaps and empirical performance gains. The paper does not offer a formal theoretical explanation of what constitutes an attention anchor or why such tokens causally improve performance. The correlation between “being attended to” and performance improvement is suggestive but not conclusively established.\n\n6.2 The experiments focus exclusively on supervised downstream tasks. There is no evaluation under zero-shot, few-shot, or in-context learning settings, where prompt tuning methods are often expected to excel. This leaves a gap in understanding CaPT’s effectiveness in broader real-world scenarios.\n\n6.3 The paper does not analyze how robust the capsule prompt is under noisy inputs, adversarial perturbations, or out-of-distribution shifts. Given that the method relies on instance-specific features, evaluating its stability under such conditions is essential for assessing practical deployment viability.", "was_revised": true, "final_justification": "The authors have provided more results as suggested in their response,  and the results clarify my concerns, justifying the contribution of this paper. I increased the score.", "rebuttal": "We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. We provide explanations to your questions point-by-point in the following.\n\n**Q1.1: CaPT’s performance on CSQA.**\n\n**A1.1:** We include the comparison of P-Tuning V2 and CaPT on the CSQA dataset across different models in the following tables. The results show that CaPT can consistently outperform P-Tuning V2. Specifically, CaPT achieves a notable performance improvement on Llama-3.2-1B, highlighting the effectiveness of CaPT on CSQA. Thank you for the great suggestion, and we'll supplement the results in the revision.\n\n| Model           | Method        | CSQA |\n|------------------|---------------|----------------|\n| T5-Base        | P-Tuning V2   | 56.43          |\n|                       | CaPT          | 57.82          |\n| T5-Large       | P-Tuning V2   | 70.52          |\n|                       | CaPT          | 72.07          |\n| Llama-3.2-1B   | P-Tuning V2   | 45.29          |\n|                          | CaPT          | 61.60          |\n\n**Q1.2: CaPT’s performance on larger models.**\n\n**A1.2:** We conduct experiments on Qwen-2.5-7B to further validate the effectiveness of CaPT on larger models. As seen, CaPT can achieve superior performance compared to P-Tuning V2 across all tested datasets. The results indicate CaPT’s applicability on larger models.\n\n| Qwen-2.5-7B  | CSQA | CB | RTE |\n| ---------------- | ----------- | ----------- | ----------- | \n| P-Tuning V2  |  77.72      |    71.58    |  72.92   |\n| CaPT             |   79.03     |   76.52     |   74.73      |\n\n\n**Q1.3: Task suitability**\n\n**A1.3:** Following suggestion, we investigate the specific tasks in which CaPT demonstrates advantages. Empirically, CaPT shows consistently strong performance on the CB and MRC datasets (QA) and remains competitive on RTE and BoolQ (NLI). Our supplementary experiments on the CSQA dataset further highlight CaPT's effectiveness in commonsense reasoning within QA. It is also worth noting that it is generally expected that prompt-based methods may not outperform all baselines across every benchmark, due to varying task characteristics and modeling demands [1, 2, 3, 4].\n\n\n**Q2: Regarding the novelty of CaPT.**\n\n**A2:** This is a great question. CaPT is distinct from prior works from superior parameter efficiency, training time efficiency, and memory efficiency via incorporating instance-aware information. In our study, we also observe the phenomenon “attention anchor”, which suggests a close relation between instance-aware information and input. Considering the limited exploration of prompt tuning inner mechanisms, we believe the discussion of “attention anchor” constitutes a foundational contribution in this community.\n\nIn Appendix S3, we have included in-detailed comparison and discussion with other instance-related methods. Between them, LoPA [3] integrates both info, however, it requires additional large-size encoders for extracting instance-aware features, leading to significantly higher memory usage. It also requires grid search of the optimal prompt length. In contrast, CaPT leverages “attention anchor” and enjoys superior efficiency (see table below).\n\n\n| T5-Base |  #Param  | Training Time | Memory (GB, bs=32)  |\n| ------------ | ----------- | ------------------- | ----------------------------- |\n| M-IDPG  |  0.47%    |       12.58x      |               ~31                 |\n| LoPA       |  0.44%    |       14.93x      |               ~33                 |\n| CaPT      |  4e-3%    |        1.00x       |               ~27                 |\n\n\n**Q3: Regarding Related Work.** \n\n**A3:** We introduce this section to contextualize recent work on attention analysis and attention sinks, which directly motivated our investigation into attention behavior in prompt-based methods. We have included additional in-detailed comparison and discussion with other existing instance-aware related methods in Appendix S3. \n\nWe agree the section name might be ambiguous, we will rename and revise the Related Work section to better emphasize the prior works and make clearer distinctions between background material and our contributions.\n\n\n\n**Q4: Equations and terminology.**\n\n**A4:** Regarding the equations, we follow the standard conventions used in the prompt tuning literature [5, 6]. Based on your suggestion, we carefully review the formatting and typeset to ensure the maximum clarity and provide a refined version of Eq.(2) in the response A5. As for terminology, we revise the manuscript to prevent any inconsistent uses of terminology throughout. If there are any additional suggestions, we are more than welcome to revise our paper accordingly.\n\n**Q5: Implementation setting of CaPT.**\n\n**A5:** Thank you for the question. We further elaborate below on the capsule prompt design.\n\nSpecifically, the capsule prompt for the first Transformer layer is obtained by combining a learnable vector $p^1$ with the mean-pooled input embedding. For subsequent layers, each capsule prompt $S^i$ is similarly constructed by combining a layer-specific learnable vector $p^i$ with the mean of the processed capsule prompt $\\underline{S}^{i-1}$ and the hidden output $H^{i-1}$ from the previous layer. Notably, CaPT does not require a separate training stage—capsule prompts are constructed in the same way during both training and inference.\n\n$S^1 =p^1 + Mean(E)$\n    \n$\\underline{S}^1, H^1 = L_1({S^1}, {E}) $\n    \n$S^i = p^i + Mean(\\underline{S}^{i-1} \\oplus H^{i-1})  \\ \\ \\ \\ \\ i = 2, 3,\\dots,N $\n    \n$\\underline{S}^i, H^i = L_i({S^i}, \\ H^{i-1})  \\ \\ \\ \\ \\ i = 2, 3,\\dots,N $\n\n\n**Q6.1: Regarding the theoretical explanation.**\n\n**A6.1:** Our explanation of CaPT as an “attention anchor” is primarily based on observations—namely, attention heatmaps and performance improvements. While theoretical analysis in the prompt tuning community remains largely limited [7,8], our work takes an initial step toward uncovering potential mechanisms behind prompt effectiveness through the lens of attention pattern. \n\nA possible interpretation is that capsule prompts help preserve salient features by enhancing information flow through stronger and more stable attention interactions (see Eq.(2)). In future work, we plan to complement our findings with additional analyses, including rank collapse [9] and mutual information [10], to better understand the relationship between attention and performance.\n\n\n**Q6.2:  Regarding CaPT’s few-shot capability.**\n\n**A6.2:** Following common practice [11], we use the GLUE dataset for pre-training, and then evaluate the few-shot performance using 4 randomly sampled training data from Boolq and CB. Preliminary results on T5-Base show that despite the current single-task design, CaPT can achieve competitive few-shot capability compared to P-Tuning V2. Moreover, we plan to employ Mixture-of-Experts (MoE) architecture [2, 12] for more robust and dynamic capsuling in the future.\n\n\n| T5-Base       | Boolq (4-shot) | CB (4-shot) |\n| ---------------- | ------------------- | ----------- |\n| P-Tuning V2  |        51.65      |    53.57  |\n| CaPT             |        50.67      |    51.79   |          \n\n\n**Q6.3: Regarding CaPT’s robustness.**\n\n**A6.3:** We further explore the robustness of CaPT under noisy inputs on T5-Base during the rebuttal. Specifically, we add random special characters to random positions of the input sequence of the train sets, simulating input noise. We compared P-Tuning v2 and CaPT under varying noise levels and found that while both degrade with more noise, CaPT consistently performs better. Its superior robustness likely comes from leveraging instance-aware information that benefits generalization under noisy conditions.\n\n| Dataset        | Method     | Noise rate 0% | Noise rate 5% | Noise rate 10% |\n|------------- |-----------------  |--------------------|----------------------|---------------------- |\n| CSQA     | P Tuning V2   |     56.43         |        42.83         |        39.23           |\n|                | CaPT             |      57.82        |         51.02         |        43.33          |\n| COPA     | P Tuning V2   |      61.28         |        55.33          |       48.33            |\n|                | CaPT             |      64.33         |        61.00          |        54.67         |\n\n[1] Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. ICLR 2024.\n\n[2] Smop: Towards efficient and effective prompt tuning with sparse mixture-of-prompts. EMNLP 2023.\n\n[3] Prompt tuning strikes back: Customizing foundation models with low-rank prompt adaptation. NeurIPS 2024.\n\n[4] Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product. NAACL 2025.\n\n[5] Visual prompt tuning. ECCV 2022.\n\n[6] M $^ 2$ PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning. EMNLP 2024.\n\n[7] Prompt waywardness: The curious case of discretized interpretation of continuous prompts. NAACL 2022.\n\n[8] On the role of attention in prompt-tuning. ICML 2023.\n\n[9] Attention is not all you need: Pure attention loses rank doubly exponentially with depth. ICML 2021.\n\n[10] Normalized mutual information feature selection. IEEE Transactions on neural networks 20.2 (2009).\n\n[11] Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. EMNLP 2022.\n\n[12] Multimodal instruction tuning with conditional mixture of lora. arXiv 2024.\n\n\nWe hope our response addresses your questions. Please let us know if there are any additional questions, and we will be happy to discuss further."}, {"review_id": "Vyp9LRGpJ6", "reviewer": "Reviewer_Q1NL", "summary": "Existing soft prompt-based learning methods face two main limitations: limited capability and inefficient prompt searching. To address these issues, the authors explore incorporating off-the-shelf instance-aware information into prompts. They find that even a simple, training-free instance-aware token can significantly improve performance and reduce the cost of prompt tuning. Moreover, such tokens act as \"attention anchors,\" consistently guiding the model’s focus to structurally important regions of the input. Building on this insight, the paper proposes CaPT, which constructs capsule prompts by integrating instance-aware information from each input sequence with task-aware information from learnable vectors. Empirical results demonstrate that their method can exhibit superior performance across various language tasks.", "questions": "See weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-structured and logically sound. It clearly demonstrates the limitations of existing soft prompt-based methods through empirical analysis and proposes a practical solution.\n\n2. The writing is clear and accessible, making the paper easy to follow.\n\n3. The experimental evaluation is comprehensive, including a wide range of strong baselines and effectively validating the proposed approach.\n\nWeaknesses:\n\n1. The evidence for Finding 2 is limited to experiments on T5, which may raise concerns about its generalizability across architectures.\n\n2. The proposed method for obtaining instance-aware information relies on mean pooling over input representations. While simple and effective, this may reduce the perceived novelty of the approach.\n\n3. Figure 3 provides a parameter comparison between existing methods and CaPT. A rough analysis of space complexity would further enhance the reader’s understanding of the method’s efficiency.", "was_revised": true, "final_justification": "Most of my concerns have been clarified. I will increase the score accordingly.", "rebuttal": "We sincerely appreciate the time and effort you've devoted to reviewing our work and providing helpful feedback!\n\n\n**Q1: Regarding Finding 2’s generalizability.**\n\n**A1:** \nThank you for the question! To assess the generalizability of our Finding 2, we further investigate simply prepending instance-aware information on a decoder-only model (i.e.,  Llama-3.2-1B) without additional fine-tuning. As shown in the table below, the results on Llama-3.2-1B strengthen our Finding 2 that indicates the significance of instance-aware information. \n\nWhile CaPT is initially motivated by findings from T5, in our paper we have applied CaPT to decoder-only models (i.e., Llama-3.2-1B and Qwen-2.5-1.5B). Experimental results (see Table 1) show that CaPT can work effectively across different architectures. We have observed a similar attention anchor role of CaPT on Llama-3.2-1B (please see Appendix S7), aligning with our observation on T5-Base.\n\n| Task  | Deep PT |  Prepending one single instance-aware token | \n| ---- | ----------- | --------------- |\n| Boolq  |  62.48      |    65.14        |\n| RTE |   58.12     |      63.18      |\n\n\n**Q2: Regarding CaPT’s novelty and mean pooling design.**\n\n**A2:** \nThank you for raising this question. In section 4.4, we have provided different instance-aware information incorporation strategies, including prepending, 1D CNN feature extraction, and feature projection. Among these different strategies, mean pooling with addition is considered as the most effective and straightforward approach, primarily for three key reasons: **First**, mean pooling is training-free and requires no additional modules to obtain instance-aware information, ensuring computational and parameter efficiency. **Second**, mean pooling with addition can demonstrate notable effectiveness with substantially fewer parameter usage (e.g., 17.5x lower than the feature projection strategy). **Third**, this design consolidates a large set of soft prompt tokens into a single prompt, thereby significantly mitigating the existing training overhead associated with searching for an optimal prompt length.\n \nWe would like to further highlight several key contributions of CaPT besides its intuitive and effective design:\n\n$\\bullet$ First, our work is the first attempt to analyze the bidirectional attentive interaction between soft prompts and input tokens in the NLP field. We uncover the power of off-the-shelf instance-aware information that can trigger interactive **attention anchor**.\n\n$\\bullet$ Second, CaPT’s superior performance further confirms the significance of employing instance-aware information as part of guidance signal through comprehensive experiments on various language tasks.\n\n$\\bullet$ Third, with the incorporation of both instance- and task- aware information in the guidance signals, CaPT eliminates the need for grid search of the optimal soft prompt length. This advantage significantly reduces the overall computational overhead (please see section 4.4).\n\nWe will add additional discussions in revision to make it clearer. Thank you!\n\n\n**Q3: Regarding space complexity analysis.**\n\n**A3:** \nThank you for the suggestion. We include a space complexity analysis on Figure 3 for a clearer comparison with Deep PT. Let $L$ denote the number of layers with prompts, and $d$ denote the embedding dimension. Given that the number of tokens for Deep PT is $n \\gg 1$, the space complexity can be formally expressed as:\n\n| Methods   |  Prompt Space Complexity |\n| -------------- | ----------------------- |\n| Deep PT    | $O(L×n×d) $            |\n|  Ours         |  $O(L×1×d)=O(L×d)$ |\n\nAs seen, CaPT achieves a significantly lower space complexity of $O(L×d)$, as it requires only a single prompt per layer, enabling CaPT to be more efficient with superior performance.\n\n\nWe sincerely appreciate your thoughtful comments. We hope our response addresses your questions. Please let us know if there are any additional questions, and we will be happy to discuss further."}, {"review_id": "4f2wbSs7yk", "reviewer": "Reviewer_Z96S", "summary": "The paper proposes a simple extension to prompt tuning: append a compressed representation of the current input (emdebbings) to a soft prompt. As the paper shows, instead of appending, adding the input compression works essentially equally well, and performance is best in this case when using only a single tunable soft prompt. Overall, the result is thus a single prompt prefix vector (a capsule), that is shown to improve Q&A or classification performance on a small set of tasks and models. Interestingly, the compressed representation of the input can be obtained by simply taking the mean over all embeddings. The method compares well against other parameter-efficient fine tuning (PEFT) methods, and the paper shows some ablations and insights into the effect of capsules on average attention patterns.", "questions": "**Major**\n1. The paper shows the combined effect of tuned single-vector prompt and mean embedding. What is missing are both contributions in isolation. As far as I understand, only tuning the single-vector prompt without the mean embedding is Fig. 4 as “DeepPT with one soft prompt”. What are the results for all experiments in Table 1 for this setting? (please point them out if they are already in the paper)\n2. Same as 1. but now ablating the effect of the mean embedding only. I.e., what is the performance (Table 1) and attention matrices (Fig. 4, S2, S3) when only using the mean embedding as a capsule but without training a soft prompt? It would be particularly interesting to see whether the “attention anchor” still arises.\n3. What exactly is a task in the paper? From Table X it looks like a task is identical with a dataset. If true, are the models then trained over multiple tasks, and is the soft prompt of the capsule being optimized only over a single task? Improvement: briefly describe each dataset used in the paper, by giving examples of a task and an instance.\n3. How exactly are capsules trained? Please provide full details or even pseudocode in the appendix. In particular: 3a) How is it ensured that the mean embedding contains no information about the answer/class in Q&A/classification tasks? 3b) What is the objective / loss under which the soft prompt (p in Eq. 2) is trained and over which data (single task, all tasks)? 3c) Is the instance mean embedding added to the soft prompt during soft prompt training, or only at test time?\n4. How can it be explained that performance goes down with capsule prompt length longer than 2? 4a) How is the mean embedding being added for longer prompt lengths (one copy per prompt vector?)?\n5. If I understand correctly, the attention patterns (Figs 1, 4, S3, S4) are averaged over a single **task** (and there are multiple tasks in each dataset?). Since the crucial structural information can be marked visually per task, and not only per instance, is it correct that the same “task beneficial” attention structure could be used for the **whole task** (without making instance specific adaptations). If so, why does the task-specific soft prompt fail to do so, but the instance specific mean embedding succeeds?\n\n**Minor comments and questions**\n1. The paper uses a lot of terminology throughout, but it is sometimes not entirely clear what is meant. Please define/explain the following terms in the appendix and point the reader there early on: “task aware”, “instance aware”, “structurally important tokens”, “semantic guidance”, “attention anchor”.\n2. It took me quite a while to understand the example in Fig. 1. Only after reading the appendix could I answer what is the sentence / context that the caption refers to, isn’t this a figure over *average* attention patterns where instance specific information makes no sense? Related: why are tokens 1, 5 and 7 important? What is the task and what is the instance? Please provide a short explanation and example of the task and of an instance when the figure is first discussed.\n3. Fig 1b: Why are there 10 Soft prompts - I thought that there is only a single vector capsule that directly precedes the input (as shown in Fig. 3 and described in Eq. (2), similar to Fig. 4)?\n4. Table 3: What is the dataset?", "limitations": "The paper repeatedly claims that the method removes the need for grid search of prompt length. I assume this is based on Fig. 2 and Fig. 6? Do these results hold for all other tasks and more importantly are there good reasons that this would hold in general? I think it would be good to add a limitation saying that based on the current findings there seems to be no need for grid search of capsule prompt length but it cannot be ruled out that this may never be the case.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Main contributions**\n1. A simple extension to (soft) prompt tuning: add input information at test time by adding a mean embedding vector to the tune prompt.\n2. Comparison of the method against other PEFT methods.\n3. Illustration of the effect on average attention patterns.\n\n**Main weaknesses**\n1. The paper lacks some important details (see questions below) and the writing needs another pass for clarity.\n2. The paper lacks an ablation of the effect of using only the mean embedding as a capsule (without task-specific tuning).\n3. While the effect on accuracy and attention patterns is significant, it remains unclear why the mean embedding can be so efficient, and why this effect deteriorates rapidly with longer capsule prompt length.\n\n**Verdict:**\n\nOverall the paper presents a novel improvement over standard prompt tuning, that is very parameter efficient, and manages to effectively incorporate “instance” information in a simpler way compared to similar approaches. Through some currently unexplained coincidence, performance is best for a capsule prompt length of 1 (max. 2, but deteriorates for longer prompts), meaning a lower number of tunable parameters compared to other PEFT methods, and thus computational advantages for prompt tuning. Similarly, the capsule causes significant changes in attention patterns (compared to prompt tuning without adding the mean embedding) - which the paper hypothesizes as being responsible for the improved performance. While this is a compelling hypothesis, the current paper cannot explain why this “attention anchoring” happens (or maybe in what kinds of tasks it is expected to happen) and why it improves performance. I think an important ablation could really improve insights (see Questions below). Together with the writing that needs improvements for clarity and the somewhat limited set of tasks, I personally think that the work could be significantly improved by taking a bit more time, performing more analysis, and resubmitting elsewhere. Having said that, I currently do not think there are very strong reasons to recommend rejecting the paper, and thus lean towards a (very) weak accept. It is possible that I misunderstood parts of the setting in the paper (particularly what a task is and over which data the soft prompt is optimized over) - I have indicated this with a lower confidence score.", "was_revised": true, "final_justification": "The authors have addressed my major concerns by:\n* Running additional ablations (Q1, Q2).\n* Clarifying their setup, in particular that all tasks have rigid structure, which means averaging attention patterns per instance reveals task structure (Q3, Q6).\n* Clarifying the details of the training/tuning setup (Q4).\n\nThe authors have also provided a satisfying response / acknowledgement of my Q5.\n\nOverall, and taking into account the other reviews and responses, my assessment of the paper remains positive, leaning more towards accept (thanks to the additional ablations), and now with increased confidence. Why not a higher score: it is still unclear how general the findings in the paper are (and there is some mild reason to be pessimistic for other kinds of tasks, see also Etjy's review), and the novelty of the main methodological improvement is somewhat limited (but certainly non-zero).", "rebuttal": "We sincerely thank reviewer Z96S for the valuable feedback, which is crucial for improving our work.\n\n**Major:**\n\n**Q1.: Deep PT with one prompt.**\n\n**A1.:**\n“DeepPT with one soft prompt” in Figure 4 and S3 stands for “only tuning the single-vector prompt w/o mean embedding”. We provide a comparison isolating the contribution of the mean embedding in our response to Q2.\n\nTo further validate the effectiveness of CaPT, we conduct additional experiments on Deep PT with one prompt. As seen, CaPT consistently outperforms Deep PT with one prompt, demonstrating the effectiveness of leveraging instance-aware info to serve as attention anchor.\n\n| Model           | Method     | Boolq |   CB   |  COPA |  MRC  |  RTE   |  WiC  | Average |\n|-----------------|------------|-------|--------|-------|-------|--------|-------|---------|\n| T5-Large        | Deep PT 1  | 82.81 | 94.16  | 71.33 | 83.75 | 85.92  | 66.92 | 80.82   |\n|                 | CaPT       | 84.56 | 97.22  | 80.00 | 84.53 | 88.45  | 69.44 | 84.03   |\n| Llama-3.2-1B     | Deep PT 1  | 60.06 | 62.11  | 47.33 | 54.21 | 54.15  | 55.02 | 55.48   |\n|                 | CaPT       | 77.28 | 65.82  | 58.00 | 65.73 | 72.56  | 65.67 | 67.51   |\n| Qwen-2.5-1.5B    | Deep PT 1  | 59.63 | 64.26  | 44.00 | 44.58 | 63.54  | 58.46 | 55.75   |\n|                 | CaPT       | 64.13 | 72.42  | 57.67 | 57.49 | 68.59  | 58.46 | 63.17   |\n\n\n**Q2: Regarding additional ablation.**\n\n**A2:** We conduct an ablation using only the mean pooled embedding as a capsule across all layers of the frozen T5-Base model. As seen, instance-aware only signals still guide the model effectively—outperforming Deep PT with a single prompt on tasks like BoolQ and WiC. The attention anchor effect also persists in this setting (figures to be included in revision). Notably, CaPT achieves the best performance, validating the benefit of combining instance-aware and task-aware guidance.\n\n| T5-Base     | Boolq  |   CB  |  COPA  |    RTE     |    WiC    |\n| --------------- | ---------| -------- | ---------| ----------- | ----------- |\n| Insta Only 1| 77.25  | 82.86 |  52.00 | 65.70    |   65.36   |\n| Deep PT 1  |  76.51 | 87.31 |  58.00 |  76.90   |  63.80    |\n| CaPT          | 79.54  | 94.16 | 64.33  |  79.78   |   66.77   |\n\n\n\n**Q3: Regarding tasks and datasets.**\n\n**A3:** Sorry for the confusion. We would like to further explain that a “task” refers to a corpus in SuperGLUE, such as COPA. The definition follows common practices [1, 2]. We will revise our paper to name them as “corpus” to distinguish from task categories (NLI, QA, etc) for better clarity. \n\nIf you don’t mind, could you kindly clarify which table you are referring to as Table X?\n\nAnd yes, soft prompts of CaPT are optimized on each corpus of SuperGLUE.\n\nWe have included the description of each dataset in section 4.1 and detailed statistics in Appendix S1. As for examples, an instance in the Boolq corpus is a sequence with a fixed structural instruction template: “*boolq passage:* Powdered sugar, also called confectioners' sugar ... *question:* Is confectionary sugar the same as powdered sugar?”\n\n**Q4: Implementation details.**\n\n**A4:** Each CaPT model is trained on each corpus of SuperGLUE. After careful consideration to ensure maximum clarity, we refine the Eq.(2) which is the core of constructing capsule prompts for both training and inference. There is only one training stage for each corpus.\n\n$S^1 =p^1 + Mean(E)$\n    \n$\\underline{S}^1, H^1 = L_1({S^1}, {E}) $\n    \n$S^i = p^i + Mean(\\underline{S}^{i-1} \\oplus H^{i-1})  \\ \\ \\ \\ \\ i = 2, 3,\\dots,N $\n    \n$\\underline{S}^i, H^i = L_i({S^i}, \\ H^{i-1})  \\ \\ \\ \\ \\ i = 2, 3,\\dots,N $\n\nEach capsule prompt $S^i$ is formed by adding a learnable vector $p^i$ to the mean of the previous capsule $\\underline{S}^{i-1}$ and hidden state $H^{i-1}$. $S^i$ is prepended to the sequence at each layer. Vectors $p^i$ are zero-initialized and trained via gradient descent.\n\n**(a)** For T5 models, we use text-to-text format, following the common practice [2, 3]. For decoder-only models, we preserve the original labels (0, 1, 2) as classification tasks. Therefore, there is no answer in input sequences.\n\n**(b)** We use cross-entropy loss on a single corpus to update the learnable vectors. For T5, this is generative cross-entropy over the vocabulary; for decoder models, it's classification cross-entropy over class indices.\n\n**(c)** Mean embedding is added to the soft prompt during both training and inference. Capsule prompts are also constructed in the same way during both training and inference.\n\n\n**Q5: Regarding longer prompts.**\n\n**A5:**\nOur experimental results on both Finding 2 and ablation indicate that effectiveness of prompts depends not on the amount of info provided, but on how well that info matches the model’s ability to utilize it. This observation is consistent with prior works on prompt pruning [2, 4], which shows that not all prompt tokens contribute positively to performance. Moreover, studies on prompt tuning [5] and P-Tuning v2 [6] have also observed that longer prompts do not necessarily lead to better outcomes, and may even hurt performance by introducing noise or leading to slower convergence.\n\n**(a)** For longer prompt lengths, we apply adaptive mean pooling to compress embedding into multiple tokens, matching the desired prompt length instead of one copy per prompt token.\n\n**Q6: Regarding the attention pattern analysis and why standard prompts fail.**\n\n**A6:**\n**(a)** The attention patterns are averaged over a single corpus and there is one task with the same structural template in a corpus. As for the structural template for a corpus, for example, “rte sentence1: *premise* sentence2: *hypothesis*” for RTE. Since each corpus is different on data distribution, question type, and structural template, prompts are optimized on each corpus of SuperGLUE.\n\n**(b)** Standard soft prompts are randomly initialized and designed to be updated via gradient descent, but not specifically designed for interactive attention behavior. In contrast, CaPT directly carries off-the-shelf instance-aware info, resulting in a fundamental difference. Acknowledging the current research on prompt tuning performance analysis remains largely unexplored, we believe the observation of the attention anchor constitutes a foundational contribution.\n\n\n**Minor:**\n\n**Q1: Regarding terminology.**\n\n**A1:** This is a valuable suggestion! We will revise our paper accordingly. Due to the rebuttal length constraint, we would like to provide an explanation here: *Instance-aware* describes info tailored to a specific input instance (e.g., a sentence or document). Instead of generic task-aware instructions, instance-aware tokens reflect the feature of each individual input.\n\n**Q2: Regarding attention analysis.**\n\n**A2:** We would like to clarify that Fig. 1 shows the averaged attention pattern across all samples of the RTE validation set. When processing each input sample, the prepended instance-aware token is mean-pooled from each sample. We present an averaged result for a high-level observation on the general impact of instance-aware info. This averaging does not conflict with the contribution of instance-aware info. Additionally, we have also provided a concrete example in Appendix S5 which shows the attention behavior of specific heads on one specific instance.\n\nTokens 1, 5-7 are important because they encode structural info in the input sequence—such as syntactic markers (e.g., \"sentence 1:\", \"question:\"). As shown in Fig. 1 (left), these tokens consistently receive high attention from other input tokens, indicating their role as cues that help the model interpret inputs. In prompt tuning, such structural tokens guide the model in segmenting the input into meaningful parts and in anchoring its attention, which improves task understanding and performance. This is supported by prior work showing that templated inputs with explicit structure improve few-shot learning capabilities by reinforcing input semantics [7, 8]. Additionally, prior work [9] indicates that attention heads in LLMs heavily focus on structural and syntactic info during In-Context Identification phase, aiding the model to interpret contextual cues.\n\nAs suggested by Q3, we have provided a detailed explanation of tasks, datasets, and instances.\n\n\n**Q3: Explanation of Fig.1 (b)**\n\n**A3:** This is because it’s an experiment based on the prompt-tuned model in Fig. 1 (a).\n\n**Q4: Dataset for Table 3.**\n\n**A4:** The average scores are also calculated based on six SuperGLUE corpora. \n\n**Regarding the discussion on Limitation**\n\n**A:**  We thank the reviewer’s careful attention to our claim about CaPT’s ability on eliminating prompt length grid search. We will add a limitation in the revised version. While our results consistently show successful optimization of CaPT across a broad range of datasets and model scales, we acknowledge that this may not generalize to all models and datasets. During the rebuttal phase, we also extended CaPT to the CSQA, and the results reinforce the robustness of our current single-prompt design.\n\n[1] Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. arXiv 2023\n\n[2] Xprompt: Exploring the extreme of prompt tuning. arXiv 2022\n\n[3] Smop: Towards efficient and effective prompt tuning with sparse mixture-of-prompts.  EMNLP 2023\n\n[4] E2vpt: An effective and efficient approach for visual prompt tuning. ICCV 2023\n\n[5] P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. ACL 2022\n\n[6] The power of scale for parameter-efficient prompt tuning. arXiv 2021\n\n[7] Language models are few-shot learners. NeurIPS 2020\n\n[8] Making pre-trained language models better few-shot learners. arXiv 2020\n\n[9] Attention heads of large language models: A survey. arXiv 2024\n\nWe hope our response addresses your questions. Please let us know if there are any additional questions, and we will be happy to discuss further."}]

Author rebuttal:

Meta-review: ### (a) Scientific claims and findings  
1. The paper proposes Capsule Prompt-Tuning (CaPT), a parameter-efficient fine-tuning scheme that prepends one single vector per Transformer layer (“capsule”) instead of tens-of-tokens soft prompts.  
2. Each capsule is formed by adding a learnable task-specific vector to a mean-pooled embedding of the current input sequence, thereby mixing task-aware and instance-aware information.  
3. Empirically, CaPT  
   • eliminates the usual grid-search over prompt length; a length of 1 works best,  
   • updates 0.003 % of model parameters yet matches or exceeds existing PEFT baselines (Deep PT, LoRA, LoPA, P-Tuning-v2, etc.) on six SuperGLUE corpora and additional CSQA and MATH-style benchmarks,  
   • scales to larger LLMs (Qwen-2.5-7B) with similar gains, and  
   • requires only 1/8.77 training time than prior instance-aware methods.  

### (b) Strengths  
• Extremely simple; implementable in a few lines.  
• Excellent parameter, memory, and wall-clock efficiency.  
• Thorough empirical evaluation with new experiments added during discussion (CSQA, robustness to noise, few-shot, 7 B model).  


### (c) Weaknesses / missing pieces  
• Conceptual novelty is modest; the core trick (mean-pool + soft vector) is straightforward.  
• Evaluation still focuses on classification-style tasks; applicability to open-ended generation, reasoning, or instruction-following remains untested.

### (d) Recommendation & rationale  
**Recommendation: Accept as a poster**  

All three reviewers end at “borderline but leaning accept”. None raises a blocking flaw; all concerns were empirical scope or clarity, both rectified in rebuttal. The work stands out for extreme efficiency with solid accuracy improvements. While the idea is simple, the empirical validation and attention analysis give it scientific substance and differentiate it from prior PEFT variants.

### (e) Discussion & how points were weighed  
One reviewer increased their score and the other two maintained their positive stance from the beginning.

As the authors summarized:
> We have conducted additional experiments to evaluate CaPT’s applicability on the CSQA dataset, alongside applying CaPT to a larger LLM Qwen-2.5-7B, as suggested by Reviewer Etjy. We have also explored the few-shot ability and the robustness of CaPT, and provided clearer implementation details including the revision of Eq.(2). Moreover, we have highlighted the key contributions of our work and the key advantages in efficiency of CaPT compared to existing frameworks.

> We have confirmed the generalizability of our Finding 2 that indicates the significance of instance-aware info on decoder-only models, pointed out by Reviewer Q1NL. We have also included further discussions on the current single-prompt design of CaPT, and emphasized three key contributions of CaPT besides its intuitive and effective design.

> In response to Reviewer Z96S, we have provided additional ablation studies to examine the isolated contribution of instance-aware information and confirm the effectiveness of CaPT. Additionally, we have revised our manuscript to include more detailed and clearer implementation settings for reproducibility and thorough explanation of terminology. Moreover, we have clarified both the high-level and the instance-level observation of the impact of instance-aware information on attention patterns.
Decision: Accept (poster)