Improving Complex Reasoning with Dynamic Prompt Corruption: A Soft Prompt Optimization Approach
Authors: Sinan Fan
Liang Xie
Chen Shen
Ge Teng
Xiaosong Yuan
Xiaofeng Zhang
Chenxi Huang
Wenxiao Wang
Xiaofei He
Jieping Ye
Venue: ICLR 2025 Poster (2025)
Keywords: prompt tuning
corruption
information flow
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Prompt Tuning (PT) has emerged as a promising Parameter-Efficient Fine-Tuning (PEFT) approach by appending trainable continuous prompt vectors to the input, maintaining competitive performance with significantly fewer trainable parameters. While PT has shown effectiveness in enhancing task performance, particularly for classification tasks, its application to complex reasoning tasks has been largely overlooked. Our investigation reveals that PT provides limited improvement and may even degrade performance in reasoning tasks. This phenomenon suggests that soft prompts can positively impact certain instances while negatively affecting others, particularly during the latter stages of reasoning.
To address these challenges, we propose a novel method called Dynamic Prompt Corruption (DPC), which seeks to optimize the use of soft prompts in reasoning tasks. DPC dynamically adjusts the influence of soft prompts based on their impact on the reasoning process. Specifically, it involves two key components: Dynamic Trigger and Dynamic Corruption. Dynamic Trigger measures the influence of soft prompts, determining whether their impact is beneficial or detrimental. Dynamic Corruption mitigates the negative effects of soft prompts by selectively masking key tokens that interfere with the reasoning process.
We validate our approach through extensive experiments on various large language models (LLMs) and reasoning tasks, including GSM8K, MATH, and AQuA. The results demonstrate that Dynamic Prompt Corruption consistently improves the performance of LLMs, achieving  4\%-8\% accuracy gains compared to standard prompt tuning. These findings highlight the effectiveness of our approach and its potential to enhance complex reasoning in LLMs.

Published as a conference paper at ICLR 2025
IMPROVING COMPLEX REASONING WITH DYNAMIC PROMPT CORRUPTION: A SOFT PROMPT OPTIMIZA- TION APPROACH
Sinan Fan--2x, Liang Xie*.41, Chen 4t Ge Teng', Xiaosong Yuan", Xiaofeng Zhang*
Chenxi Huang Wenxiao Wang Xiaofei He², Jieping Ye !Zhejiang University Hangzhou YunQi Academy of Engineering
"College of Computer Science and Technology. Zhejiang University Technology
'Alibaba Cloud Computing
ABSTRACT
Prompt-tuning (PT) for large language models (LLMs) can facilitate the performance on various conventional NL. tasks with significantly fewer trainable parameters. However, our investigatton reveals that PT provides limited improvement and may even degrade the primitive performance of LLMs on complex reasoning tasks. Such a phenomenon suggests that soft prompts can positively impact certain instances while negatively affecting others, particularly during the later phases of reasoning. To address these challenges, We first identify information accumulattin within the soft prompts. Through detailed analysis, we demonstrate that this phenomenon often accompanied by erroneous information flow patterss in the deeper layers of the model, which ultimately lead to incorrect reasoning outcomes. we propose a novel method called Dynamic Prompt Corruption (DPC) to take hetter advantage of soft prompts in complex reasoning tasks, which dynamically adjusts the influence of soft prompts based on their impact on the reasoning process. Specifically, DPC consists of two stages: Dynamic Trigger and Dynamic Corruption. First, Dynamic Trigger measures the impact of soft prompts, identifying whether beneficial or detrimental. Then, Dynamic Corruption mitigates the negative effects of soft prompts by selectively masking key tokens that interfere with the reasoning process. We validate the proposed approach through extensive experiments on various LLMs and reasoning tasks, including GSM8K, MATH, and AQuA. Experimental results demonstrate that DPC can consistently enhance the performance of PT, achieving 4%-8% accuracy gains compared to vanilla prompt tun- ing, highlighting the effectiveness of our approach and its potential to enhance complex reasoning in LLMs.
INTRODUCTION
Prompt Tuning (PT) (Lester et al., 2021a) a promising parameter-efficient fine-tuning (PEFT) approach demands very few trainable continuous prompt vectors added to the input, achieving competitive performance compared to the full-parameter fine-tuning (Qin al., [20I]] Ding et al. 2023a). Previous efforts have demonstrated the superiority of PT by generating high-quality soft prompts (Shi & Lipani 2024, Yang et al. 2023 Qin et al.) 2024). Nonetheless, soft prompting underperforms the foundation model on complex reasoning tasks. With a comprehensive investiga- tion, we discover that vanilla prompt tuning provides little performance improvement on complex reasoning tasks and may even lead to negative performance outcomes.
From the perspective of individual instances, soft prompts may have positive effects for some in- stances while potentially posing negative impact on others. Figure] [ iillustrates an example from the GSMSK (Cobbe et al.) 202T]: :: the model can answer the question correctly without soft prompt, while soft prompt guides the LLM to incorrect answer through some intermediate reasoning steps. Therefore, it is crucial to determine whether the effect of soft prompts on reasoning is positive or negative. However, understanding why certain reasoning succeeds while others fail is difficult.
Worr done during an intersship at Hangzhou YunQi Academy of Engineering and Alibaba Cloud Com- puting.
1 Cooresponding Authors: Liang Xie
Chen Shen <jason.sc@ alibaba-ine.com:
I Projecct Lead.

---

Published as a conference paper at ICLR 2025
Quesitin: Narzia solt clpo to ofter Sheods is April and fhes the sold balf mor clipe a May How many dliss and Namii sell altgeeter in Agri and Nty?
Assmer Sold 4622 eoe weddeeeeddedd de dips May. Nandi to to cor 220220022 clps atleether in Apel Nay.
pteer 72
LLM
frirads April to maay did Natali sil altoeefaas and Maa??
Asine Let bh santer of Neais add April mane dipu May
Mig.
Saft Prangt
We basw tha Neaka d5 capi
Thrn. Naxza unt Ci 18 48-4 48 1212 dige alopetee
Figure I: Input the same question to guide the LLM to answer it. The model was originally able to provide the correct answer, but after adding the soft prompts, it produced an error in reasoning.
To meet the challenge, intend to explore the mechanisms of soft prompts on the Chain-of- Thought (CoT) (Wei et al. (022) through a lens of information flow. Neuron saliency score analysis an important tool for analyzing the information flow during LLM inference (Dai al. 2022, Wang et al. 2023a), allowing to probe the models' reasoning mechanisms. After comprehensive investigation across several L.MMs and tasks, we find: seft prompts lead reaching the right an- swer when the latter reasoning steps put more emphasis on former tokens compared to soft tokens. In contrast, when the soft prompt significantly impacts the latter reasoning steps, the model may reach incorrect answer. This phenomenon revealed by neuron saliency score analysis is simi- lar to the thinking process of human beings to some extent: humans always recall the background knowledge related to the question and then answer it. As carrier of task-related knowledge (Xiao et al. 2024) (Yu et al.] 2024), soft prompts can provide beneficial hints to help models understand the question and generate the initial reasoning steps, and then the question and generated steps to complete the answer. However, if the remaining reasoning steps still attend to the hints, i.e., soft prompts, it may conflict with the question and existing reasoning steps. Furthermore, we discover that soft prompt tokens in certain positions have a strong impact on reasoning, and we can suppress the negative effects of the soft prompts on reasoning by corrupting them.
Inspired by the above findings, we propose a novel two-stage method called Dynamic Prompt Cor- ruption (DPC): Dynamic Trigger and Dynamic Corruption. Concretely, in the first stage, we measure the impact factor of the soft prompts on the latter part of the reasoning in the final layer, by which we can differentiate whether the soft prompts can lead to the correct answer; in the second stage, we locate the key token that influences both the question and reasoning steps most in shallow layers, then mitigate the negative impact with key token vector masking and other token sparse masking.
To validate our analysis and evaluate the proposed DPC, we conduct comprehensive experiments with multiple LLMs on various tasks. Experimental results show that DPC can consistently im- prove the overall performance of LLMs such as LLaMA2- 13B (Touvron et al., 2023), LLaMA3- 8B (Dubey et al. 2024). and Mistral-O.2-7B (Jiang et al. 2023) on various reasoning tasks with 4%-8% accuracy enbancement including GSM8K (Cobbe al. 2021 MATH (Hendryeks et al.,
2021]. AQuA (Ling et al. 2017) compared to pure prompt tuning. These results are in line with our observations and demonstrate the effectiveness of our DPC. Our contributions be summarized as follows:
We investigate the relationship between the information accumulation in the soft prompts and erroneous information flow patterns through saliency score analysis. Our findings in- dicate that when the soft prompts influence the later stages of reasoning deeply, the model is more likely to reach an incorrect answer.
Based the thorough analysis, propose DPC, an instance-level prompt tuning opti- mization strategy, which can dynamically identify and mitigate the negative effects caused by the soft prompts.
Experiment results manifest our proposed DPC can improve the vanilla prompt tuning for various LLMs on complex reasoning tasks to a large margin, demonstrating the effective- ness of our findings and the superiority of the proposed method.
2

---

Published as a conference paper at ICLR 2025
Saliency of layers
6405
118
6083
6003
Lapat
Saliency of layers
(b) Information accumulation
Figure 2: (a) Saliency scores of prompt-to-quesion prompt-to-rationale layer by layer (b) Illustra- tion of a significant information accumulation phenomenon within soft prompts, where specific token conveys strong information to both the question and the rationale.
INFORMATION FLOW ANALYSIS FOR SOFT PROMPTS
To identify the influential factors in Chain-of-Thought (CoT) reasoning, we aim to analyze the model's inference process. A complete CoT reasoning instance consists of three primary com- ponents: the Prompt P. the Question Q, and the Rationale R. Therefore, we need examine the interactions among these three components using appropriate tool. The saliency score, widely used interpretative analysis tool employed to study the interactions between tokens, integrates gra- dient and attention values and is particularly effective in analyzing the information flow between different components. This allows the saliency score to simultaneously reflect the relationships among various modules and their impact on the output.
The saliency score is computed by taking the Hadamard product of the attention scores A and their gradients follows:
@c(x) AA
=
Σ A O
(1)
where A₁ denotes the attention weight for the h-th head in the I-th layer, @ represents the Hadamard product, and C(x) is the loss function, typically cross-entropy
our implementation. The attention matrix captures interactions within the sequence, while the gradient indicates the di- rection information transfer. By analyzing the saliency score, we can thus observe the information flow throughout the entire sequence during reasoning.
2.1
LAYER ANALYSIS
Popular LLMs are composed of multiple stacked Transformer layers, which have specific roles in processing information during model reasoning. To better quantify the interactions between distinct components, we define two metrics to measure the strength of information flow layer-wise:
Spgr the average significance of information flow from the prompt to question:
Eone Cpg Se(i.)) (Cpgl
8(0)
(2)
pq
Cpg ((..))pp sis Pe, Si 1.500 ge}
(3)
Spr, the average significance of information flow from the prompt to rationale:
Ei,yec, Se(i,j)
((t)
(4)
pr
Cprl
3

---

Published as a conference paper at ICLR 2025
Correct plle
Sthatlow laper
Durp loyor
Para Oueston Ou
Staatorn layer
Pyctuess DEEIM
Deup later
300
500: pr0cee
Quinbis
Oul
Quesion Out
[Sant guReeE
Quosio0
OMR
Figure 3: Schematic illustration of our observations. Conrect-ansswer cases (left) show balanced accumulation of saliency shallow layers, where rationale information evenly gathered from the soft prompts. As the reasoning progresses into deeper layers, attention shifts from the soft prompts to earlier rationale steps and the question itself. In contrast, wrong-answer cases (right) exhibit excessive saliency in shallow layers and disruptions in the information flow in deep layers. The latter part of the reasoning in deep layers overly focuses on the soft prompts, leading to incorrect answers.
Cpr (ii,))lp. sis Pe,Ta Sj S re)
where 5) represents the intensity of information flow from the i-th to the j-th token, |Cpol and [Cpel denote the number of interactions between prompt and question tokens and between prompt and rationale tokens, respectively. Here, Pa. 9s. and are the start tokens of the prompt, question, and rationale, while Pe. Ge. and r. are their respective end tokens.
As shown in Figure 2a visualize the saliency scores during the LLM processing input across layers, sub-figures show the saliency scores representing the semantic information transfer between different input components: from the prompt to the question, and from the prompt the rationale. We find distinct peaks for Q and R related to the soft prompts P occurring between layer 2 and layer 10, indicating that the most intense flow of prompt information is aggregated in shallow layers (near the input). Furthermore, we need a fine- ine-grained analysis of the information flow phenomenon in shallow layers to explore the underlying mechanisms more precisely.
Figure 2b illustrates the saliency matrix S of shallow layers during reasoming, where we observe strong flow of information among P. Q, and in certain instances, leading to evident accumulation of information specific soft token. Such phenomenon implies that certain specific soft tokens can have a significant impact on subsequent reasoning in some cases.
2.2 PROMPT TUNING ANALYSIS
After examining numerous instances, find that in wrong-answer cases, the accumulation of saliency in shallow layers is often accompanied by changes in the information flow patterns in deep layers, as shown in Figure 3 In contrast, in correct-answer cases, reasoning gathers rationale information evenly from soft prompts in shallow layers, while in deeper layers, the latter part of reasoning steps focuses on earlier rationale steps and the question rather than soft tokens. In incorrect examples, information accumulation occurs in shallow layers, accompanied by changes in deep information flow patterns: the latter part of the reasoning in deep layers overly focuses on the soft prompts, leading to incorrect answers.
2.2.1
WHAT Is THE RELATIONSHIP BETWEEN INFORMATION ACCUMULATION AND CHANGES IN INFORMATION FLOW PATTERNS?
We observe a potential correlation between information accumulation and changes in information flow patterns through visualization; however, the phenomenon in few cases may not confirm universal pattern. In this part, we continue to explore the impact information accumulation on the information flow pattern.

---

Published as a conference paper at ICLR 2025
pribaiityy
0450
0000
0000
0008
2000
0000
300
(a)
(b)
(c)
Figure 4: (a) describes the relationship between information accumulation in shallow layers and the change of information flow patterns in deep layers. (b) illustrates the distribution of information flow intensity from soft prompts to the latter part of the rationale in both good and bad cases. In good cases, the distribution tends to be weaker, while in bad cases, the distribution is stronger. (c) depicts the overall intensity of information flow from the soft prompts to the latter part of the rationale for both good and bad cases.
Setting The information accumulation can be interpreted as specific token strongly influencing the subsequent reasoning process, and it is visible in the saliency matrix, where distinct column effect can be observed. To explore the information accumulation, we compute the saliency scores with soft prompt token to question and rationale:
S(E) pigr
Σ
Se(i.t)
(6)
(id)ECppur
Cp.or = ((i,i)la Sisre)
where Se(j,), i) denotes the intensity of information flow from the j-th to the i-th token, 4s is the start token of the question and re is end token the rationale. When Jace = (ps I & I Pe Sp > 10 by default, is present, consider that the information
EE /p p,)), where accumulation has occurred. When the information flow intensity between the latter reasoning steps and the soft prompts becomes significantly strong. we consider this a change in the information flow pattern. To confirm the existence of the information flow change, we calculate StEP!
SIFP E(isEE (i.
(8)
CCrl
Cpr = Si Se Si
where Stost(i,) represents the intensity of information flow from the i-th to the j-th token In the last layer of the model, |Cprl denotes the number of interactions among prompt tokens and rationale tokens, Ps is the start token of the soft prompts, while 9a corresponds to the intermediate token. Similarly, Pa and r, denote the end tokens. If SIFP exceeds the defined threshold, we consider it change in the information flow pattern.
Observation We randomly sample one hundred examples to validate the generality of these pat- terns larger scale. As shown in Figure 4a the probability of information accumulation in the shallow layers occurring alongside changes in the deep layer information flow patterns is much higher than in cases where no accumulation observed, indicating a strong correlation between the occurrence of information accumulation and changes in the deep layer information flow patterns.
2.2.2 WHAT Do CHANGES IN INFORMATION FLOW PATTERNS AFFECT?
The observation of the previous subsection proves the relation between information accumulation and information flow pattern changes. However, the impact of these changes the model's reason-
5

---

Published as a conference paper at ICLR 2025
ing ability remains unclear. Therefore, in this subsection, delvv into the influence of the changes on the accuracy of reasoning.
Setting We randomly selected 50 correct-answer and 50 incorrect- answer instances to demonstrate the applicability of the changes across broader range. By calculating the SIFP, we aim to measure the relation between reasoning accuracy and the changes.
Observation Figure 4b elaborates the distribution differences in saliency values between correct and incorrect examples. As shown in Figure 4c] the latter part of the reasoning process exhibits a lower mean value for soft prompts in correct-answer examples compared to incorrect-answer ones, which supports the broader context of the above phenomenon. Thus, we can conclude that: soft prompts lead to reaching the right answer when the latter reasoning steps put more emphasis former tokens compared to seft tokens. In contrast, when the soft prompts significantly impact the latter reasoning steps, the model may reach incorrect answer. These findings
in line with the human cognitive process: before answering question, humans al always recall the background knowledge related to the question and obtain some relevant inspiration answer the question; As carrier containing task-related knowledge (Xiao et al. (2024) (Yu et al. 2024), the soft prompts can provide beneficial hints help models understand the question and generate the initial reasoning steps, then use the question and generated steps to solve the problem.
3 METHOD
In this section, retrospect the vanilla Prompt Tuning and propose the Dynamic Prompt Corruption (DPC) method derived from our analysis and findings to enhance reasoning capabilities.
3.1 PROMPT TUNING
Prompt tuning (PT) paradigm where a pre-trained language model θ be fine-tuned towards a task T with training data (X, Y) {si, W: H requiring only a few learnable vectors (soft prompts) that are prepended to the input embeddings while keeping the model parameters @ frozen (Lester et al., 2021a). Specifically, for an input sequence T = {t1, t,,...,ttx! E Raxd, PT adds
a learnable prompt matrix V 02, U1 Ixd where is hyperparameter. The loss
function for PT is optimized to P as follows:
LPLM = - Elog log P(ui 0, P).
(10)
where the input can be represented by the concatenated matrix [V: E R((+a)xd
3.2 INTERFERING FACTORS RECOGNITION
By analyzing the information flow during reasoning, we believe that the latter reasoning steps should focus on the reasoning tokens that have been generated rather than soft tokens. Therefore, it is essential to identify those reasoning processes in which the flow of erroneous information occurs. We think that incorrect reasoning is characterized by some affected tokens in the latter half of the reasoning sequence, and affected tokens are those that have high information flow with the soft prompts. If the proportion of affected tokens exceeds certain ratio, it is considered likely to result in bad reasoning. Consequently, we calculate the proportion R of affected tokens within reasoning as follows:
Dre 1(Spr(1) ETa
R
(11)
Elistecproo Stase(i,)) (Cpr(a)
Spr(t)
(12)
Opr(1) ((i.))ps SiSpel
(13)

---

Published as a conference paper at ICLR 2025
fatal wela iepererr
Dynomic trigger
lepet
Soft promytt
Pi
14
LLM
PS
PS
Soft prompe
Anawer SD
Dynomic comruption
Figure 5: Overview of our method, Dynamic Prompt Corruption (DPC), which dynamically iden- tifies erroneous information flow patterns and corrupts soft prompt tokens based on the location of information accumulation, alleviating the negative effects caused by the soft prompts in certain sit- uations.
where /3 is the average intensity of information flow from the soft prompts to the former part of the reasoning tokens by default, Spr(i) is the intensity of information flow from the soft prompts the th token, (tase(t, represents the intensity of information flow from the i-th to the j-th token in the model's final layer, [Cpr(el is the number of interactions among prompts and token i, denotes the start token of the soft prompts, while fı, corresponds the intermediate token. Similarly, Pe and re denote the end tokens. After engaging in a comparative analysis of numerous instances from various datasets with distinct LLMs, we summarize different thresholds delimit incorrect reasoning.
3.3 DYNAMIC CORRUPTION
With the aforementioned dynamic trigger strategy, we can identify the reasoning that produced incorrect answers, and we also need to push them to be correct. Inspired by the analysis of section [22.1] we propose the Dynamic Corruption strategy to pinpoint the soft prompt loca- tions that cause erroneous information flow at the instance level and then mitigate these erross through targeted corruption operations. Firstly, we locate the position within the soft prompts where information accumulation most intense by employing the calculation outlined in Equa- tion 6 Secondly,
mask the value of the j-th prompt vector to obtain the corrupted soft prompts to U1, 82 mask U....., Jn and eliminate the smallest I' percent of the embedding values, excluding the j-th element. By default, I is set 10. Through the two steps, can mitigate the potential negative effects of the harmful accumulation.
Combining the thorough analysis in the last section, propose novel two-stage strategy called Dynamic Prompt Corruption (DPC): Stage 1, identifying the negative impacts of soft tokens on the reasoning at the instance level; Stage 2, dynamically applying Corruption to the soft prompts, ultimately improving reasoning performance.
EXPERIMENT
4.1 SETTINGS
Models We test DPC and comparison methods on LLaMA3-SB (Dubey et al., 2024), LLaMA2- 13B (Touvron et al. 2023), and Mistral-0.2-7D (Jiang et al. 2023) since they are popular Trans- former decoder-only LLMs that are convenient for exploiting and analyzing inside architectures. We set the generation mode to greedy decoding minimize irrelevant confounders during model inference and to ensure consistent answers to fixed questions under the same model and prompt; all experiments
NVIDIA A100 GPU server.
7

---

Published as a conference paper at ICLR 2025
Datasets GSM8K (Cobbe al. 2021) is challenging dataset for assessing the capability of language models in multi-step math reasoning. MATH (Hendrycks et al. 2021 is comprehensive collection of high school-level mathematies problems. It covers wide range of topics, including algebra, geometry, calculus, and more advanced subjects like number theory and combinatorics. AQuA (Ling et al., (017) is designed for algehraic question-answering tasks and includes variety of multiple-choiee math word problems. The questions in AQuA are derived from standardized tests and other educational resources, focusing on algebra and related topics.
Baselines To validate the effectiveness of the proposed method, we conduct experiments with several baseline models that exhibit superior performance in general NL.P tasks.
Prompt tuning: Tune the trainable prompt with the pretrained model frozen. As the model is fixed, prompt tuning will not change the pretrained model's performance in-context learning tasks com- pared to when the prompt removed.
ACT: Attention Calibration (eC)) (Yu et al. 2024) is training-free approach aimed at optimizing the attention distributions within LLMs without requiring weight fine-tuning, It is designed to address the phenomenon of attention sinks that receive disproportionately high attention despite their limited semantic relevance. By visualizing and analyzing the attention patterns across LLMs during inference, ACT identifies and calibrates attention sinks in input-adaptive manner.
4.2 RESULTS AND DISCUSIONS
Table 2 shows the reasoning results with LLaMA3-8B, Alama22-3BB, and Mistral-0.2-7B using var- ious methods on 3 complex reasoning tasks.
GSM8K Compared with vanilla prompt tuning, our Dynamic Prompt Correction (DPC) method enhanced the accuracy the GSM8K dataset for Illama,, Llama3, and Mistral-0.2, improving from 38.19 1%, 65.5%, and 49.5% to 11.9% 67.6%, and 51.1%, respectively. Notably, vanilla prompt tuning performed suboptimally with Llama3, showing a mere 0.6% increase in accuracy over the pretrained model. In contrast, our DPC method mitigated the challenges of vanilla prompt tuning, achieving 2.7% improvement in accuracy. Notably, although the ACT method has shown impressive perfor- mance in classification tasks, its generalization capability in complex multi-step reasoning tasks is limited, and fails alleviate the challenges associated with prompt tuning.
MATH For the MATH dataset, which contains challenging competition-level mathematics prob- lems, models typically perform poorly due to the task's difficulty. However, our DPC method im- proved the accuracy Llama2, Llama3, and Mistral-0.2 on the MATH dataset, increasing from 7.6%, 33,7%, and 15.0% to 9.2%, 36.3%, and 16.4%, respectively. This demonstrates the superior performance of our method on difficult tasks.
AQuA DPC improves the accuracy on the AQuA dataset for Llama2, Llama3, and Mistral-0.2 from 22.4%, 33.7%, and 28.7% 31.1%, 42.5%, and 31.9%, respectively. Notably, we observe an average improvement of 6.9%, and maximum improvement of 8.8%, showing its outstanding performance in multi-step reasoning multiple-choice tasks.
Table I: Accuracy % of various baselines and DPC on 3 tasks with LLaMA3-SB, Llama2-13B and Mistral-0.2-7B..
Llama2-
Llama3-818
Mistral-7B
GSM8K
MATH
AQuA
GSM8K
MATH
AQuA
GSM8K
MATH
AQuA
Pretrained model
29.5
2.0
21.0
64.9
30.0
34.0
37.9
5.1
26.0
Prompt tuning
38.1
7.6
22.4
65.5
33.7
38.5
49.5
15.0
28.7
ACT
39.2
7.1
20.1
52.6
33.8
38.6
49.5
15.0
28.7
DPC
41.9
9.2
31.1
67.6
36.3
42.5
51.1
16.4
31.9
8

---

Published as a conference paper at ICLR 2025
4.3 ABLATION STUDIES
To evaluate the effectiveness of various components in our proposed Dynamic Prompt Correction (DPC) method, we conducted series of ablation experiments. Specifically, investigated the im- pact of corruption without Dynamic Trigger and random corruption. By systematically removing or altering some elements, key factors that lead to improved reasoning and reduced error rates. The following experiments ex- plore the role of Dynamic corruption and Dynamic Trigger in addressing information accumulation and erroneous information flow patterns.
aim analyze their contributions to the overall performance and identify
Setting A:All corruption applied the corruption operation to all instances without the need for confirmation from the dynamic trigger. This allows to evaluate the effectiveness of the dynamic trigger in isolation, providing insights into its overall impact on model performance.
Setting Randdo corruption For each instance of the corruption operation, we randomly lected the position to be corrupted. This random selection enables us to assess the overall robustness of dynamic corupttinn and provides more comprchensive understanding of its effects model performance.
Table 2: Accuracy % of various baselines and DPC on tasks with L.L.M3-88.
LLaMA3-8B
Model
GSMSK
MATH
AQuA
Prompt luning
65.5
33.7
38.5
Random corruption
51.3
32.3
39.3
All corrution
65.8
34.2
39.9
DPC
67.6
36.3
42.5
The results of the ablation studies, presented in Table 2 provide crucial insights into the combined effectiveness of dynamic trigger and dynamic corruption in Dynamic Prompt Correction (DPC) method. The performance of DPC is compared against several baselines, including prompt tuning, random corruption, and corruption without dynamic trigger.
Effectiveness of Random Corruption: Random corruption, where the position of the corruption selected randomly for each instance, shows drop in accuracy across all tasks compared to prompt tuning. Specifically, the accuracy on GSM8K, MATH, and AQuA decreases to 51.3%, 32.3%, and 39.3%, respectively. This demonstrates that while corruption can introduce changes, applying without specific guidance (such from dynamic trigger) can disrupt critical information flow, leading to worse performance.
Effectiveness of Corruption Without Dynamic Trigger: Applying corruption to all instances without the dynamic trigger results marginal improvements over prompt tuning, particularly on the MATH and AQuA datasets, with accuracy increasing to 34.2% and 39.9%, respectively. How- ever, the improvement is modest, and the GSM8K task shows only a minor gain (65.8% vs. 65.5%) ). These results suggest that although corruption mitigate some negative effects, the absence of dynamic trigger to guide when and where to apply it limits its full potential.
Dynamic Prompt Corruption Combined Effect of Dynamic Trigger and Dynamic Corrup- tion: The full DPC method, which integrates both dynamic trigger and dynamic corruption demonstrates significant performance gains across all tasks. For GSM8K, MATH, and AQuA, DPC achieves 67. 77.6% 36.3%, and 42.5% accuracy, respectively. These results highlight that the dynamic trigger plays vital role identifying when and where to apply corruption effectively, while dy- namic corruption corrects erroncous information flows precisely. The combination of these two components is essential to achieving optimal performance, as neither component alone sufficient to address the complexities of the task.
The ablation study clearly shows that dynamic corruption alone, or applicd indis- criminately, is not sufficient significantly improve model performance. It is the interaction be-
Conclusion:

---

Published as a conference paper at ICLR 2025
tween dynamic trigger and dynamic corruption that yields the best results. The dynamic trigger ensures that corruptions are applied only when necessary. while dynamic corruption addresses spe- cific eπrоrs, leading to more robust reasoning process and higher overall accuracy. The synergy between these two components is key to the success of the DPC method.
RELATED WORK
Parameter-efficient fine-tuning (PEFT) (Han et al. parameters while matching the performance of full model fine-tuning. PEFT methods fall into two main branches. The first approximates parameter changes in lower-dimensional spaces. For in- stance, LoRA (Hu et al. weight matrix, and A E Ra B E RR are low-rank matrices capturing fine-tuning updates. Other examples include Bitfit (Ben Zaken et al, 2022). tuning only bias terms of PLMs, Diff Prun- ing (Guo al. (021). learning task-specific "diff" vectors to extend pre- pre-trained parameters, and Adapter (Houlsby et al. 2019), an early method adding task-specific layers. The second branch leverages Transformer (Vaswani et (017) structure, treating input tokens prefixes as trainable parameters. Prompt Tuning (PT) (Lester et al., (2021b) prepends learnable prompt tokens to inputs, while Prefix-tuning (Li & Liang 2021) tunes intermediate hidden states. P-tuning (Liu et al.) 2023
2024 Ding et al. 2023b) aims to tune few
2022) decomposes W se Wo + AB, where Wo E IR xd2 the original
and P-tuning v2 (Liu et al. (2222) extend these ideas with varied designs.
As key PEFT method in the second branch, Prompt Tuning (PT) (Lester et 2021b) valued for
its lightweight and simple implementation, widely applied in vision and language tasks. Subsequent
studies have refined it: (Oymak et al. (203) analyzed attention's role, (Yuan et al.] (024) adapted prompts per query, and (Petrov et al., 2024) found soft prompting biases attention heads, limiting it
to pre-trained skills. (Wang et al. 2023b) showed acts as a universal approximator for sequence-
to-sequence tasks, though its expressive power is capped, unlike MLP. Additional research includes subspace tuning (Qin et al. 2021 2024), two-stage pipelines (Wang et al.] 2024). Low-norm Effect reduction boosts performance, prompt decomposition (Shi & Lipani 2024) for efficiency- our focus here-and and a dynamic PT framework (Yang et al. 2023).
analysis (Fu et al.) 2024) showing
CONCLUSION
In this work, we explored the limitations of vanilla Prompt Tuning in complex reasoning tasks and uncovered key factors that contribute to its underperformance. Through neuron saliency score anal- ysis, we demonstrated how soft prompts could positively or negatively affect reasoning processes, depending on how they influence subsequent reasoning steps. Inspired by these insights, we intro- duced Dynamic Prompt Corruption, novel approach designed to mitigate the negative impacts of soft prompts on complex reasoning tasks. By employing a two-step process involving dynamic trig- ger and dynamic corruption, our method successfully adjusts the influence of soft prompts in way that guides the model towards correct reasoning. Comprehensive experiments across various models and datasets validated the effectiveness of our approach. These findings and results contribute to deeper understanding of soft prompting mechanisms and provide a practical solution for enhancing reasoning tasks in large language models.
LIMITATIONS
In this work, DPC has demonstrated strong performance; however, the process of identifying erro- neous information flow patterns and pinpointing the critical soft prompt tokens requires a complete reasoning pass, which increases the inference cost in practical applications. While our research pro- vides valuable insights into the underlying mechanisms of soft prompts, it should not be viewed as the sole interpretation. We believe that more effective methods may exist to further explain these phenomena.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine-
tuning for transformer-based masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, Dublin, Ireland, May 2022. As- sociation for Computational Linguisties. doi: 10. 1218633/1122222aalllhlle 7/acl aacllanhhoooyy.ogge. wrt. acl-soortr-sorr.1
URL. https:
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, Christopher Hesse, and John Schulman. Training verifters to solve math word problems. arXiv preprint 1rX1::2100.1468, 2021.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502, 2022.
Ning Ding. Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. Parameter-efficien fin-ntuiig of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-23) 235, 2023a.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-255 2023b.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
Shuai Fu, Xiequn Wang, Qiushi Huang, and Yu Zhang. Nemesis: Normalizing the soft-prompt vectors of vision-language models. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL http::t://pppeeeeeee/ee wtwpreeveewi.eeteeeeeeeeee id= 2m22mmm2Phii
Demi Guo, Alexander Rush, and Yoon Kim. Parameter--fffiienn transfer learning with diff pruning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4884 Online, August 2021. Association for Computational Linguistics. doi: 10. 1161386v112222 acll-long.378. URL httpp:://cclatthee logy. www..rrg/2ae 2021 a.l.llllgngoo
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efffiient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv: 14608, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799, PMLR, 09-15 Jun 2019. URL https: http::s//proceeding iig..ngg..... mmlr
press//97/houllsyylls html
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: //ppr//ieeeeeiiieeee/ net forum?
1d=nZeVKeeFYF9
11

---

mlr press/ Published as a conference paper at ICLR 2025
Qidong Huang, Xiao wen Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Neng H. Yu. Opera: Alleviating hallucination in multi-modal large lan- guage models via over-trust penalty and retrospection-allloattion 2024 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR). pp. 13418-13427, 2023. URL.
www.ssteaatschi... htttss://ap..semattiiccchtt
ar www.oor//rrrrrs:o/eoooo
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021a.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. doi: 10.18653/v1/20111 http:///222..mmmmmmmm...2. mainip- URL
https: http:///aaaanthooooo.oo org/ 2021 emnlp main. 243
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2211 12
URL pttttp:://acaaatiin
org/ 2021 aacl-long 353
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. ACL, 2017.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers). pp. 61-68, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 12111663////2222.clllll///
URL
bttps://aclanthoooyy.ccoo. 201.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, 2023. URL
abs /20.../213..20 10385
Samet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, and Christos Thrampoulidis. On the role of attention in prompt-tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 26724-26768, PMLR, 23-29 Jul 2023. URL btttps:/pproceetee 72C2/oymak2 2tt.mmla..mmmmmmmmmm
Aleksandar Petrov, Philip Torr, and Adel Bibi. When do prompting and prefix-tuning work? theory of capabilitics and limitations. In The Twelfth International Conference on Learning Representa- fions, 2024. URL | httttttt:///ppeeeeetee
net/ nett/forummddd=eew....o
Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, et al. Exploring universal intrinsic task subspace via prompt tuning. arXiv preprint arXiv:21 10.0 00.07067,7000 2021.
Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, al. Exploring universal intrinsic task subspace for few-shot learning via prompt tuning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Zhengxiang Shi and Aldo Lipani. DePT: Decomposed prompt tuning for eframeter-effiiient fine- tuning. In The Twelfth International Conference on Learning Representations, 2024. URL
https://opents net fott://foommmid=jggfffff PGRde
12

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: bttps::/ppoceeddnne www.:iingg..euus...... www.ips. ca/paper files/paper/2017/ Ei f1le/ffeee24333 2235ee2233547do 7dee91 fbd053cl Fbdd5533 3
Lean Wang, Lei Li, Damai Dai. Deli Chen, Hao Zhou, Fandong Meng. Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 9840-9855, 2023a.
Yihan Wang, Jatin Chauhan, Wei Wang, and Cho-Jui Hsich. Universality and limitations of prompt tuming. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 75623-75643. Curran Associates, Inc.,
2023b. URL http:: /prtpse:/proceeee:e ngs neuri wwwpppppppppppeeppeceeeceeepeeee sampperr_fleeee fttps///eeeee/ppp///223/// fall:/eefaacffe5055555e5e
6b1 5558 oonpaa/eeeeootto-aaeaaaaaaooooen
Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh, Inderjit S Dhillon, and Sanjiv Kumar. Two-stage LLM fine-tuning with less specialization and more generalization. In The Twelfth International Conference on Learning Representations, 2024. URL https:
7/ //operrevis www.ww........................ww wtt./forumm/ d=ppCngg 6Qco
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information procexsing systems, 35:24824-24873 24837, 2022.
Jinfeng Wei and Xiaofeng Zhang. Dopra: Decoding over-accumulation penalization and re- allocation in specific weighting layer. ArXiv, abs/2407.15130, 2024. URL httpss//api.
semanticscholadt http::aawa...e/oorrss::::::2:::0
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024.
Xianjun Yang, Wei Cheng, Xujiang Zhao, Wenchao Yu, Linda Petzold, and Haifeng Chen. Dynamic prompting: A unified framework for prompt tuning, 2023. MRRhhtts:///xxii wxx.vv.rog/aas/////oooo.. 2303. 02909
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Un- veiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406. 15765, 2024.
Xiaosong Yuan, Chen Shen, Shaotian Yan, Xiaofeng Zhang, Liang Xie, Wenxiao Wang, Renchu Guan, Ying Wang, and lieping Ye. Instance-adaptive zero-shot chain-of-thougee prompting. arXiv preprint arXiv:2409.20441, 2024.
A IMPLEMENT DETAILS
A.1 PROMPT TUNING TRAINING SETUP
For all prompt tuning experiments, we use learning rate of 0.001 and a prompt length of 100. For the LLama3-8h model, we initialized the soft prompts randomly. For LLama2-13b and Mistral-0.2- 7b, we initialized the soft prompts with text. The text used for initialization was selected from single example in the training set of each dataset: GSM8K, MATH, and AQuA. For each dataset, one representative sample, including both the problem and its rationale, was chosen to initialize the soft prompts. This ensures that the prompts are aligned with the specific characteristics of each dataset, enabling more effective prompt tuning during training.
13

---

Published as a conference paper at ICLR 2025
For GSM8K
Question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
Answer: Natalia sold 48 24 clips in May. Natalia sold 48 + 24 72
clips altogether in April and May. ##44 72
For MATH
Problem:
What is the degree of the polynomial (4+ 5x 100 2mx + v10r** + 9)?
Solution:
This polynomial is not written in standard form. However, don't need to write it in standard form, nor do we need to pay attention to the coefficients. We just look for the exponents on a. We
have an x term and no other term of higher degree, so 4 is the degree of the polynomial.
For AQuA:
Question: A person is traveling at 20 km/hr and reached his destination in 2.5 hr. Find the distance?
A) 53 km
B) 55 km
C)
52 km
D) 60 km
E) 50 km T 2.5 hrs = hrs
T
S
20 X 50 km
Answer
is E
B COMPARISON OF METHODS
Firstly, as shown in Figure 6(a), our analysis tool differs from that used in (reeamnnlLLLM(Xian al. 2024). While StreamL..M uses attention scores, utilize saliency scores, which combine gradients and attention values. This allows us to effectively analyze information flow between dif- ferent components, making it suitable for scenarios like ours that involve multiple components. This methodological difference may lead to different analysis results.
Additionally, illustrated in Figure 6(b), StreamingLLLM observes high attention values the first token, termed an "attention sink." and leverages this finding to extend the input sequence length positive use of high attention values.
However, different scenarios can exhibit different behaviors. For example, as shown in Appendices Figure 6(c), (d), and (e), several studies have demonstrated the negative effects of the attention sink phenomenon.
The Yu et al., 2024) study found that attention sinks not only occur on the first token but also tokens with limited semantic information (e.g.. "," ":", and "<0x0A>"), as visualized in Figure 6(c). Contrary to the observations made Streamingl which suggest preserving attention sinks to enhance LLMs' accuracy-they highlight that not all attention sinks are beneficial. Specifically, for most attention sinks occurring in the middle or later parts inputs, reducing their attention scores lead to improved accuracy.
In OPERA Huang et al. 2023) and DOPRA Wei & Zhang 2024). as represented in Figure 6(d) studies found that when models generate hallucinated content, the self-attention weights in the last layer exhibit a distinct "columnar" pattern before the hallucination occurs, leading to an "over-trust" tendency in the self-attention weights for the hallucinated parts.
14

---

Published as a conference paper at ICLR 2025
(a) Salleyy Score
ttention-score
Grad-soore
Sailency-score
Demonstoccon
Output
Demonstration
Outpat
Demerstration
Outot
A Attentionsscore
A Attentinnscoo
(b) Streain LMM
Initial
Initial
Special token
( ACT
/n of from
Prompt
Outout
Preapt
Ouggut
stention-score
A Attentinnsccore
(e) DOPRA
Middle Layer
OPERO
Laver 32
Outuu
Output
Stoodd 023
Sailency-score
Sailency-score
Sallency-score
Ours
Shaalow laver
Deep lover
Deep layer
CONO
(
Solt pisonet
Ultss
Ohtt
poonpt
Figure 6: Schematic illustration of methods related to information accumulation: (a) illustrates the distinction between saliency and attention: (b) demonstrates the attention sink phenomenon on initial tokens as described in StreamingLLMM: (c) highlights the attention sink phenomenon observed ACT for special tokens other than initial tokens: (d) and (e) present the accumulation phenomenon related to hallucinations discovered by DOPERA and OPERA in the visual domain; (f) showcases the information accumulation phenomenon we identified soft prompts.
15

---

Published as a conference paper at ICLR 2025
Layer6
Layer12
Layer16
30
Layer20
Layer24
Layer28
Figure 7: Visualization of the attention matrix in Llama3-8b.
C VISUALIZATION OF THE ATTENTION MATRIX
D COMPLEMENTARY EXPERIMENTS
To further demonstrate the effectiveness of our method, we have added additional comparisons with other PEFT methods. We conducted extensive experiments using prefix tuning and LoRA, and our DPC method consistently showed improvements within the scope of prompt tuning. Furthermore, Table indicates that DPC achieves optimal or near-optimal performance with relatively few com- putational resources compared to multiple methods, confirming its effectiveness.
Table 3: Accuracy % of various peft methods and DPC on 3 tasks with LLaMA3-SB, Llama2- -3BB and Mistral-0.2-7B.
Llama2- 13B
Llama3-8B
Mistral-7B
GSMSK
MATH
AQuA
GSM8K
MATH
AQuA
GSM8K
MATH
AQuA
Pretrained model
29.5
2.0
21.0
64.9
30.0
34.0
37.9
5.1
26.0
Prompt tuning
38.1
7.6
22.4
65.5
33.7
38.5
49.5
15.0
28.7
Prefix tuning
41.7
8.4
20.1
65.4
33.0
41.3
54.4
16.3
31.5
LoRA DPC
12.7
7.4
24.8
44.1
31.8
38.6
44.9
19.9
24.4
41.9
9.2
31.1
67.6
36.3
42.5
51.1
16.4
31.9
To further verify the generalizability of our method, we conducted experiments on another three datasets involving logical reasoning and commonsense reasoning. Table [ demonstrates that DPC not only excels in the initial tasks but also effectively generalizes to other reasoning scenarios.
Table 4: Accuracy % of prompt tuning and DPC on another 3 tasks with LLaMA3-8B and Mistral- 7B.
Llama3-8B
Mistral-7B ARC-challenge
prontoQA
ARC-challenge
WorldTree
prontoQA
WorldTree
Prompt tuning
100
13.2
39.1
97.9
14
55.2
DPC
100
14.8
45.5
98.9
16.6
56.5
16


Reviews:
[{"review_id": "8y9d8dPUmn", "reviewer": "Reviewer_mj1a", "summary": "This paper aims to improve prompt tuning on LLMs. The authors claim that later reasoning steps in wrong examples usually have high attention values on the soft prompt tokens, which is interpreted as \"attend to the hints in soft prompt and conflict with early reasoning steps\". Therefore, the authors propose to corrupt the soft prompt with a mask when such bad attention behavior is detected. Experiments on several math dataset show that this method can improve the vanilla prompt tuning.", "questions": "Please find my comments in the weaknesses section", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Several math benchmarks are covered in the experiment section, including various difficulty levels easy (gsm8k) and hard (MATH).\n- The paper is well written and easy to follow.", "weaknesses": "- In this paper, soft prompt is interpreted as \"carrier of task-related knowledge\" or \"hints\", which is a bit casual and not well supported.\n\n-  The major argument of this paper is LLM (later stage reasoning steps) shouldn't pay too much attention on soft prompts, which is very suspicious. Specifically,\n\n     1. The authors derive this argument from empirical observation instead of rigorous causal relation analysis, or fundamental justification. \n     2. The prompt / input to LLM, prompt is structured as {soft prompt; question} where soft prompt is in the leading position. Then the authors observe some high attention on soft prompt (leading tokens) on wrong examples. However, according to [1], LLM trends to put non-useful attention values on leading tokens (e.g., first several tokens) and that's why leading tokens has high attention value. In other words, it does not mean high attention on soft prompts, which are in leading position, is a bad behavior.  \n     3. On top of 2, the corruption mask that can improve reasoning accuracy is not because they mitigate bad high attention values on soft prompt, but something else, e.g., overfitting.\n\n- In experiments table1, it's hard to validate any this prompt tuning methods (prompt tuning, act, dpc) is necessary or not on reasoning tasks. The authors should add full finetuning as the major baseline, and demonstrate that dpc is on par or even better than it.\n\n[1]   Xiao, Guangxuan, et al. \"Efficient streaming language models with attention sinks.\" arXiv preprint arXiv:2309.17453 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SpDpPsqf9H", "reviewer": "Reviewer_Y54T", "summary": "The paper introduces a novel technique called Dynamic Prompt Corruption (DPC) to optimize the utilization of soft prompts in intricate reasoning tasks. This method dynamically regulates the impact of soft prompts on the reasoning process. Experimental results demonstrate the effectiveness of this approach.", "questions": "1. Do other PEFT methods exhibit similar observations?\n2. Apart from saliency scores, are there better probing tools available?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The athors adopt a probing approach, quantitatively analyzing the efficacy of soft prompts on guiding complex reasoning in LLMs. Its conclusions offer profound insights, and methods derived from these insights have demonstrated significant effectiveness in experiments. This work makes a substantial contribution to the field of LLM reasoning research, and I highly appreciate the innovative insights presented in this paper.\n\n2. The article exhibits a clear and coherent structure, with well-organized content.", "weaknesses": "The experimental baselines in this study are relatively limited in number. A comparative analysis with other PEFT methods, like LORA, could enhance the experiments. If this paper can unveil shared observational conclusions among various PEFT methods, it would significantly strengthen the overall contribution of the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iGY8P6o2sV", "reviewer": "Reviewer_sfZi", "summary": "This paper studies why sometimes soft prompts can affect reasoning outcomes. The paper points out that certain specific tokens in prompts can significantly harm reasoning performance. To address these challenges, it proposes a method called Dynamic Prompt Corruption (DPC), which seeks to optimize the use of soft prompts in reasoning tasks. DPC dynamically adjusts the influence of soft prompts based on their impact on the reasoning process. Specifically, it involves two key components: Dynamic Trigger and Dynamic Corruption. It mitigates the negative effects of detrimental soft prompts by selectively masking key tokens that interfere with the reasoning process.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem addressed in the paper is genuinely existing, and the paper has strong practical value.\n\n2. The paper uses experiments to analyze in detail the reasons why prompts can lead to reasoning errors in certain situations, demonstrating strong persuasiveness and logical coherence.\n\n3. The method and analysis proposed in the paper are highly relevant, simple and effective, and easy to reuse.\n\n4. The paper is well-written and easy for readers to follow.", "weaknesses": "1. The choice of the saliency score lacks explanation; it's unclear why this metric is used instead of others. Additionally, there should be more clarification regarding Equation (1).\n\n2. The phenomenon presented in the article is largely based on observation, but the generalizability of the observed conclusions requires further validation. Additionally, the explanations for the phenomenon lack theoretical support.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "amooU2XXeY", "reviewer": "Reviewer_fh4M", "summary": "This paper discovers that sometimes prompts can make the model generate the wrong answer, using the metric of saliency scores defined by Hadamard product of attention score and its gradients. When such cases happen, it is usually because the question part does not have even attention on the prompt (i.e., not reading hints carefully), or because the answer part relies too much on the prompt (i.e., misled by the hints, rather than thinking more on questions and previous steps in answer). Therefore, the authors proposed that one can use two step algorithm to fix. In the first step, they identity whether such bad cases happen. In the second step, they fix bad cases by adding masks to prompts, so that attention will be improved. Empirically such method can improve the performance on GSM8K, MAH, and AQuA.", "questions": "I do not have additional questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Originality: the authors noticed that the attentions between soft prompts and question, and between soft prompts and answer, are nicely correlated with the final correctness. This observation is interesting and original. \n\nQuality: the observation is interesting. I especially like the visualizations in the paper. \n\nClarity: this paper is well written and easy to follow. \n\nSignificance: the observation that attentions will affect performance might be interesting to other researchers.", "weaknesses": "I will not give a high score to the quality of this paper, because it did not tell a \"full story\". \n- The Figure 4 indeed shows difference between good and bad cases of the saliency scores, but the difference is not very big. For example, in Figure 4(c), good case has score of ~0.014, while bad case has score of ~0.019. While I agree such difference means we can improve our model by using the two step algorithm, this observation does not seem to be substantial. It seems entirely possible that in the next version of LLM, this difference may disappear. \n- While I agree the two step algorithm is effective to improve the model's accuracy, the improvement is not significant as well. \n\nTherefore, I think the significance of this paper is mild. I feel this algorithm may not be used by most researchers in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)