Unveiling and Manipulating Prompt Influence in Large Language Models
Authors: Zijian Feng
Hanzhang Zhou
ZIXIAO ZHU
Junlang Qian
Kezhi Mao
Venue: ICLR 2024 poster (2024)
Keywords: Large language models
prompt interpretability
prompt manipulation
controlled LLM generation
Primary area: generative models
TL;DR: Unveiling and Manipulating Prompt Influence in Large Language Models through Token Distribution Dynamics

Abstract: Prompts play a crucial role in guiding the responses of Large Language Models (LLMs). However, the intricate role of individual tokens in prompts, known as input saliency, in shaping the responses remains largely underexplored. Existing saliency methods either misalign with LLM generation objectives or rely heavily on linearity assumptions, leading to potential inaccuracies. To address this, we propose Token Distribution Dynamics (TDD), an elegantly simple yet remarkably effective approach to unveil and manipulate the role of prompts in generating LLM outputs. TDD leverages the robust interpreting capabilities of the language model head (LM head) to assess input saliency. It projects input tokens into the embedding space and then estimates their significance based on distribution dynamics over the vocabulary. We introduce three TDD variants: forward, backward, and bidirectional, each offering unique insights into token relevance. Extensive experiments reveal that the TDD surpasses state-of-the-art baselines with a big margin in elucidating the causal relationships between prompts and LLM outputs. Beyond mere interpretation, we apply TDD to two prompt manipulation tasks for controlled text generation: zero-shot toxic language suppression and sentiment steering. Empirical results underscore TDD's proficiency in identifying both toxic and sentimental cues in prompts, subsequently mitigating toxicity or modulating sentiment in the generated content.

Published as a conference paper at ICLR 2024
UNVEILING AND MANIPULATING PROMPT INFLUENCE IN LARGE LANGUAGE MODELS
Zijian Feng!., Hanzhang Zhou' Zixiao Zhu Junlang Qian² & Kezhi Mao
Institute of Catastrophe Risk Management, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore
Schooo of Electrical and Electronic Engineering, Nanyang Technological University, Singapore *Future Resilient Systems Programme, Singapore-ETH Centre, CREATE Campus, Singapore [feng0119, hanzhang001, zixiao001, junlang001] e ntu. edu. sg ek skzmao@ntar ntuu. edd. sg
ABSTRACT
Prompts play a crucial role in guiding the responses of Large Language Models (LLMs). However, the intricate role individual tokens prompts, known as input saliency, in shaping the responses remains largely underexplored. Existing saliency methods either misalign with LLM generation objectives or rely heavily linearity assumptions, leading to potential inaccuracies. To address this, we propose Token Distribution Dynamics (TDD), a simple yet effective approach to unveil and manipulate the role of prompts in generating LLM outputs. TDD lever- ages the robust interpreting capabilities of the language model head (LM head) assess input saliency. projects input tokens into the embedding space and then estimates their significance based on distribution dynamics over the vocab- ulary. We introduce three TDD variants: forward, backward, and bidirectional, each offering unique insights into token relevance. Extensive experiments reveal that the TDD surpasses state-of-the-art baselines with a big margin elucidating the causal relationships between prompts and LLM outputs. Beyond mere inter- pretation, we apply TDD to two prompt manipulation tasks for controlled text generation: zero-shot toxic language suppression and sentiment steering. Empiri- cal results underscore TDD's proficiency in identifying both toxic and sentimental cues in prompts, subsequently mitigating toxicity modulating sentiment in the generated content.
INTRODUCTION
The formulation of prompts significantly shapes the textual responses of large language models (LLMs). Comprehending the influence of individual input tokens in prompts, i.e., input saliency, augment our insight into LLM interpretability and foster the development of refined prompting strategics to modulate LLM outputs. However, input saliency LLMs and its bearing on prompt- based generation control are rarely explored.
Although various saliency methods ranging from perturbation- based (Feng al. 2018 Prabhakaran
et al. 2019). gradient- based (Chefer al., 202T Ali et al.) (002)), to vector-based (Modarressi al. 2022 Ferrando et al., 2022) for Transformer-based language models have been devised to gauge input token significance, their adaptation to LLMs presents certain limitations. First, these methods are chiefly designed for text classification using masked language models such as BERT (Devlin al. 2019) and RoBERTa (Liu al., 2019). Their saliency explanations, limited class labels, do not align with the objectives of autoregressive LLMs, which aim to explain token generation across the whole vocabulary. Second, many of these methods rely heavily on linearity assumptions to approximate the intricate non-linear behaviors in language models. For instance, vector-based approaches (Modarressi et al. 2022 Ferrando et al. 2022) often assume linearly combined attention weights across layers, while gradient-based ones (Wallace al. 2019 Chefer et al.) 2021; Ali
et al. 2022) employ local linear approximations, referencing Taylor's 's expansion theorem. Such
"Corresponding author.

---

Published as a conference paper at ICLR 2024
assumptions can lead to significant inaccuracies, especially the number of LLM layers increases. These limitations can hinder the understanding of how LLMs respond to prompts and hence the effective use of prompts for desired outputs.
Unlike previous methods that use attention or gradient as a medium to compute saliency, in this study, we propose adopting token distributions as means estimate token saliency. Recent studies on GPT2 show that token representations can be visualized as distributions across the vocabulary, termed token distributions, through the LM head (Nostalgebraist, 2020) Geva al. 2021 2022]
Dar et al. 2023). We expand this notion to other contemporary LLMs such as Pythia (Biderman al. aal]|2223 and LLaMA2 (Touvron al., 2023). emphasizing the relationship between token dis- iributions and token contributions. Our findings indicate that the LM head acts as an effective interpreter for L.Mss, skillfully decoding ambiguous input token representations and mapping them to the embedding space. These projected token distributions within the embedding space are inter- pretable and comprehensible humans, and align with model predictions. We argue that variations in token distributions across layers can be attributed to token saliency.
To unveil token saliency in prompts for LLM generation, we introduce the Token Distribution Dy- namics (TDD) approach. In alignment with the generative objectives of LLMs, our methodology offers contrastive explanations (Yin & Neubig 2022), elucidating why LLMs prioritize one token over others in the entire vocabulary. We present three variants: IDD-forward, and DoD-tiidiretiiona.. fDDD-fowward capitalizes on token distribution dynamics throughout the progression of learning and prediction, while TDDDbbccward evaluates the token importance using backward token dynamics. In parallel, TDD-bidirectional integrates insights from both directional perspectives. Despite its apparent simplicity, TDD serves as exceptionally potent tool. Compre- hensive experiments show that TDD markedly outperforms advanced saliency methods in discerning the causal relationships between prompts and LLM outputs across the entire vocabulary.
DaD-backward,
Beyond merely providing interpretation, we elucidate how to harness TDD to manipulate prompts and control LLM outputs. We spotlight two key applications: zero-shot toxic language suppression and sentiment steering. In toxic language suppression, TDD identifies and neutralizes toxic triggers in prompts before they are fed into LLMs. For sentiment modulation, TDD captures sentiment cues in prompts, adjusting their sentiment inclination to guide the sentiment of generated texts. Results of extensive experiments showcase TDD's efficacy in pinpointing predefined triggers and modulating the toxicity or sentiment of the outputs.
The contribution of this study can be summarized as follows
1. We introduce a novel solution to assessing input saliency based on token distributions, which we validate and elaborate upon in autoregressive LLMs, including GPT2 (Radford
lt al. 2019), GPTJ (Wang & Komatsuzaki, (021), BLOOM (Seao et al. 2022 Pythia
(Biderman et al. 2023). and LLaMA2 (Touvron et al.) 2023).
2. We propose TDD, a technique that leverages token distribution dynamics to unveil prompt influence in LLM generations. We present three variants that harness token dynamics from distinct directions. Extensive evaluations across various datasets and LLMs show that our techniques notably surpass other leading methods.
3. We exemplify two applications of TDD to manipulate LLM outputs, specifically targeting the reduction of toxicity in language generation and managing the sentiment of produced texts. Empirical evidence reveals that TDD effectively detects toxie or sentiment-related triggers in prompts, enabling more controlled LLM outputs.
2 RELATED WORK
2.1 INPUT SALIENCY FOR LANGUAGE MODELS
Existing saliency approaches to explaining the relationships between prompts and language model outputs can be classified into gradient-based, vactor-based, and perturbation-base methods. Gradient-based methods employ backpropagation gradients to determine the significance of each
token. Examples of this approach include gradient norm (Simonyan et al. 2014 Li al. 2016a
Code will he released here: https::/gittubb.com com/
2

---

Published as a conference paper at ICLR 2024
Atanasova al. 2020), gradient input (Denil et al.] 2014 Shrikumar al. 2017: Wallace et al. 2019), LRP-XAI Bach et al. 2015: Ali et al.) 2022). and generic attention explainability (GAE) (Chefer al. 2021). For vector-based methods, rather than utilizing the last layer's 's at- tention weights (Bahdanau et al. 2014), techniques like attention rollout (Abnar & Zuidemal 2020), GlobEnc (Modarressi et al. 2022). and ALTI (Ferrando al. 2022) have been effectively developed. However, these methods often misalign with the generation objectives of LLMs or rely heavily on linearity assumptions, leading to potential inaccuracies.
A few perturbation- based methods have been proposed, which utilize the input reductions (Li et al,, 2016b Feng et al. 2018 Prabhakaran al., 2019) determine the most relevant parts of the in- put by observing changes in model confidence or Shapley values (Lundberg & Lee| (017). AtMan DDiseroth et al. 2023), tailored for large-scale Transformers, leverages attention mechanisms and perturbation approach. Our TDD method, introducing a new medium of token distribution for esti- mating token saliency, stands in contrast AtMan and removes the necessity for hyperparameters.
2.2 CONTRASTIVE EXPLANATIONS FOR LLMS
Contrastive explanations (Lipton 1990: Jacovi et al. 2021; Yin & Neubig 2022). which focus on identifying the causal factors influencing model's s generation choice between two alternatives, have emerged in the last two years for explaining LLMs. Compared to conventional explanations, contrastive explanations provide insight into the nuanced decisions made by LLMs. Taking the input prompt "Amanda was respected by some as an example, if the LLM generates the token "waitress"" conventional explanations would merely highlight the importance of each token leading to the generation of "waitress"". From a contrastive perspective, the choice of "waitresses" over "waitress" can be attributed the word "some". Mcanwhile, the decision choose "waitress" rather than "pictures" is predominantly influenced by "respected". Thus, contrastive explanations offer more comprehensive understanding of major grammatical considerations in sophisticated LLMs. This inspires us to design contrastive explanation methods to better understand the prompt influence LLMs. Yin & Neubig (2022) pioneers the state-of-the-art approach for generating contrastive explanations for LLMs by using the contrastive gradients with respect to the target token and the alternative token. Nevertheless, it still suffers from the linear approximation error.
3 UNVEIL PROMPT INFLUENCE IN LARGE LANGUAGE MODELS THROUGH TOKEN DISTRIBUTION DYNAMICS
3.1 PROBLEM STATEMENT
Analyzing the influence of input prompts L.MM output sequences necessitates assessing the im- pact on each generated token, given the autoregressive nature of LLMs and their token-by-token generation procedure. Formally, for an input prompt w = (Wi, .... Wx), the objective the importance of each token in prompting the LLM to produce the target token w, as the (n + 1)-th
determine
token w,+1 rather than an alternative token Wa- Any two tokens in the vocabulary can construct
aarget-alternative pair. The resulting saliency c = (ci,..., Cn) E R indicates the token contributions leading the LLM to generate w, over Wa.
To quantify the input saliency underlying the generation of LLMs, we introduce token distribution dynamics as a measure of each token's 's importance. We present three variants: TDD-forward, TDD- backward, and IDDD-bidirections Figure deppicts our framework.
3.2 TOKEN DISTRIBUTION: NEW LENS FOR ANALYZING LLMS
We advocate for the adoption of token distributions novel medium for input saliency analysis, superseding traditional gradients and attentions. In this section, we will expound upon the nature of token distributions and establish the nexus between token distribution and token importance.
Recent studies (Nostalgebraist, 2020 Geva et al. 2021 2022 Dar et al. 2023) on GPT2 indicate that token hidden states across layers can be projected into the embedding space utilizing the language model head (LM head). Consequently, token representations can be conceptualized as evolving distributions over the vocabulary (token distributions in short) with certain explainability. The
3

---

Published as a conference paper at ICLR 2024
Prompt: Tracy praises
those
lucky
Projection by LM
Embedding space
Embedding space
Vocaba
Vo
Tracy
those
lucky
Locabulaa
lucky
those
Tracy
guy
guy
guy
guy
guys
guys
guys
guys
guys
guys
Forward token distribution dynamics
Backward token distribution dynamics
TDD-forward
IDD-bidirectional
TDD-backward
Contrastive Explanation Estimation
Why guys Instead of guy?
Figure I: Framework of TDD. It first employs the LM head to project token representations into the embedding space and then evaluates the significance of tokens through three distinctive variants from various directions. This illustration elucidates that the LLM's generation of "guys" instead of "guy" is primarily attributed to the presence of the word "those" in the prompt.
current research this subject is somewhat cursory, being predominantly based GPT2, limited datasets, and the last input token. It would be imprudent to directly apply these findings to other autoregressive LLMs like LLaMA2 without further examination. Thus, we begin by exploring this notion using multiple LLMs on datasets with a variety of text styles and encompassing all input tokens. This investigation sets the groundwork for our study. We then elucidate the role of token distributions in shedding light on the impact of prompts.
Formally. the hidden state at layer for token wy represented as xf. where / L,.,, and L signifies the total layers of the attention blocks in the Transformers. Utilizing the identical LM head Ma. the hidden state of any input token w; in layer can be projected into the embedding space, represented as:
pf softmax (Mnz,)
(1)
where pf E R V is the vocabulary and |/ is the vocabulary size. Our objective is to examine the significance of p across layers in various LLMs and correlate of with token importance. Detailed experimental settings, qualitative outcomes, and quantitative results are provided in Appendix A Our findings, which lay the foundation of our work, are summarized follows.
1. Our experiments validate the generalizability of this theory. Traditionally, only the last token is fed into the LM head for next-word prediction. However, we discover that the LM head can be applied to all input tokens. We find that all the input token representations in the prompt at every layer can be projected as interpretable token distributions over the vocabulary within the embedding space via the LM head. This notion applies to wide range of L.LMs, including GPT2, GPTJ. BLOOM, Pythia and LLaMA2.
2. Projected token distributions from the LM head hold the potential elucidate causal relationships between prompts and LLM outputs. Advancing the first finding, we ob- serve that these distributions are both interpretable and integral to the model's generative process. Each dimension of the token distribution represents specific token in the vocab- ulary, with the value of each logit indicating the likelihood of the subsequent token based current and preceding tokens. This capability of the LLM track prediction distri- bution changes with the progressive introduction of tokens enables us to infer which tokens influence the prediction of target token, offering explanatory insights.
3. Token distribution dynamics reflect token importance. We contend that shifts in the distribution of each token arise due to the introduction of diverse tokens. By monitoring the changes in token distribution induced by each token, we can infer its significance.
Based on the aforementioned findings and analysis, we introduce three variants that leverage token distribution dynamics (TDD) to estimate token saliency from different directions.
4

---

Published as a conference paper at ICLR 2024
3.3 FORWARD TOKEN DISTRIBUTION AS INPUT SALIENCY
We begin by leveraging the robust interpretive capability of the LM head to project all input token representations from each layer onto token distributions over the vocabulary, as described in Equ ( As detailed in Appendix A the token distributions of intermediate layers tend to converge toward the distribution at the final layer in an approximately monotonic fashion. This suggests that each token's 's contribution converges synchronously at the final layer. Given the convergent nature of token distribution dynamics, we can directly employ the LM head to project all input token representations from the final layer into the embedding space. In this space, each dimension interpretable, enabling us to ascertain the conclusive contribution of each token. By examining the token distributions related to both the target token and the alternative token, individual contributions of each token.
gauge the
Specifically, considering that LLMS utilize decoder-only structures, the distribution for the i-th token at the final layer I., denoted as p²- depends on the past tokens. This represents the probability of any token in the vocabulary becoming the (i + 1)-th token. The distribution difference between the target token and the alternative token, D (we) p w.), can be attributed to the cumulative contribution of the first tokens. Similarly, tokens.
- ( reflects the collective contribution
of the initial
Hence, given the first tokens, the LLM's confidence to produce the target token Wy over the alter- native token Wa can be computed follows:
r; = (wr) pt (w.)
(2)
The transition from T:-1 to ry can be roughly attributed to the introduction of i-th token wy. Thus, its saliency can be approximated as:
forward
i 1
(3)
Ti-1 if > 1
It should be noted that because the LLM is built on decoder-only structure, the conditioning prob- ability p for all input tokens can be obtained through single forward calculation by the LLM. Then the input saliency c forwardd for all input tokens can be calculated through TDD-forward Equs
2 and 3
Here, we highlight the marked and substantive differences between our method and perturbation- based approaches. For input sequence of length n, perturbation-baset methods typically remove one token sequentially and focus on the probability change of the terminal token. In contrast, our approach assesses the distributions of all input tokens within the embedding space. reducing depen- dence on the final token alone. Moreover, our method necessitates only a single forward computa- tion, whereas perturbation- based strategies demand n perturbation computations.
3.4 BACKWARD TOKEN DISTRIBUTION AS INPUT SALIENCY
To further assess token saliency from a distinct viewpoint, we propose a backward token distribution approach IDDD-bacwarrd). For any given input, the process begins with the last token, progressively incorporating preceding tokens to evaluate the probability distribution of the ultimate prediction. Formally,
r = (wr) pf.n (wa))
where pf. is determined by projecting the final token based on input tokens from we to w,, and quantifies the probability of the (n+ 1)-t token. Consequently, r; denotes the generation confidence of the target token relative to the alternative token, influenced by tokens from w, to wp. Then we calculate the input saliency backwerd for all input tokens by starting from the n-th token.
o buckward
ri
if
(5)
ri - T-41 if Kiin n
5

---

Published as a conference paper at ICLR 2024
Table 1: Examples from the BLiMP Benchmark. This table encompasses a range of linguistic phenomena, with each sample comprising an input prompt, target token, and an alternative token.
Phenomenon
Prompt
Target token
Alternative token
Determiner- Noun Agreement (Morphology) Argument Structure (Sytax) NPI Licensing (Semantics)
Joel complains about those -.. Amanda respected by some Even many birds can
drivers
driver
waitresses really ever
picture
3.5 BIDIRECTIONAL TOKEN DISTRIBUTION AS INPUT SALIENCY
Drawing inspiration from the efficacy of bidirectional neural networks, which assimilate information from both directions, we propose bidirectional token distribution (TDDD-bidircctional) for input saliency. Specifically, the token input saliency, c bidrrectional is determined by summing the forward and backward saliency estimations:
vidirectional
forward
backward
Building on the proposed TDD, we further investigate its potential capability to control LLM outputs for diminished toxicity and specified sentiment in Section
3.6 EXTENSIONS OF TDD
TDD is versatile, applicable to scenarios with single or multiple target tokens, and with or without alternative tokens. Further discussions and related experiments are detailed in Appendix
4
EXPERIMENTS
Drawing from prior research, compare the explanation faithfulness of our approach with other cutting-edge saliency techniques across multiple datasets.
4.1 DATASET
The Benchmark of Linguistic Minimal Pairs (BLiMP) encompasses 67 distinct datasets, containing various linguistic phenomena across syntax, morphology, and semantics. Following prior studies on contrastive explanations (Yin Neubig, 2022) for LLMs, we select the 11 same datasets that can be utilized for the LLMs to perform text generation and then compare the explanation faithfulness. the abovementioned linguistic features. The details of the 11 datasets provided in Appendix C In each dataset, every sample comprises prompt, a target token, and an alternative token. Table several examples. When presenting prompt to LLMs, the goal is to ascertain the input saliency that prompts the [.LM to produce the target token over the alternative.
These datasets
4.2 EVALUATION METRICS
In line with prior studies CCeffer al., 2021 Ali al. 2022 Modarressi et al. 2022 Ferrando
et al. 2022 Modarressi et al. [2023), we employ the perturbation method to assess the explanation faithfulness various saliency methods. Specifically, we replace K% tokens, deemed most/least significant, with meaningless space token to gauge its influence on LLMs' output. We quantify faithfulness using two metrics: AOPC and Sufficiency.
AOPC: Initially, all input tokens are substituted with the meaningless space token. Tokens are then sequentially reintroduced (at 20% intervals), ranked from most to least significant. We compute the relative probability (AOPC) of the target token compared to the alternative token, determined by the softmax of their respective logits. higher AOPC indicates a more precise explanation.
Sufficiency: Initially, all imput tokens retained. Tokens subsequently removed, starting from the most to the least significant. We then report the relative probability (sufficiency) of the target token versus the alternative token. lower sufficiency score signifies more accurate explanation.

---

Published as a conference paper at ICLR 2024
Table AOPC (%) and Sufficiency (%) of different LLMs by various methods. Each figure the average across all perturbation ratios. Higher AOPC and lower Sufficiency scores are better.
GPT2
GPIJ
BLOOM
Pythia
LLaMAZ
LLMs
AOPCT
Suff
AOPCT
Suff 4
AOPC
Sulf
AOPC
Sull
AOPC
Suff
Rollout
64.13
62.14
64.48
62.03
65.56
61.73
64.18
62.10
59.09
60.10
IG
63.72
61.03
64.02
62.11
62.80
62.33
63.29
60.88
59.95
57.11
Con-GN
63.69
61.12
63.73
61.99
63.86
61.43
63.76
60.84
59.79
57.78
Con-Gl
64.71
60.53
64.89
60.32
65.18
60.84
64.59
59,94
59.73
57.19
TDD-forward
67.28
68.71
55.11
67.75
57.13
67.10
56.81
62.80
52.69
IDDD-backwrd
70.46
54,20
70.61
54.08
70.20
55.07
71.29
52.67
63.71
53.22
TDD IDD-bidirectional
69.95
55,22
71.05
53.31
70.22
55.39
70.58
53,38
65.51
52.04
4.3 CHOICE OF LLMS
large (Radford et al. 2019). GPTJ-6B (Wagg& & Komatsuzaki 2021). BLOOM-7B (Scao et al. 2022), and Pythia-6.9B (Biderman et al. 2023). LLaMA2-7B (Touvron et al. 2023 to conduct experiments. We detail the selection reasons Appendix D
We opt for GPTe-large
4.4 BASELINES
We first compare TDD with the advanced conventional explanation method, integrated gradients (IG) (Sundararajan et al. 2017). We then compare our method with two state-of-the-art methods for contrastive explanations in LLMs. Contrastive Gradient Norm (Con-GN) (Yin & Neubig, 2022) computes input saliency using the gradient the target token relative to the alternative token. Contrastive Gradient Input (Con-GI) (Yin & Neubig 2022) refines input saliency estimation by calculating the dot product of the contrastive gradient and the input embedding. In the nascent field of contrastive explanations for LLMs,,
rector-based methods currently exist. Thus, employ
the rollout (Rollout) (Abnar & Zuidema 2020; Modarressi et al. 2022, Ferrando et al. (022) as the representative approach for vector-based saliency to ensure a thorough comparison.
4.5 RESULTS
Table presents the average AOPC and Sufficiency (Suff) scores across all 11 datasets. Results on individual datasets are presented in Appendix N] We also visualize some results in Appendix Regarding the AOPC, all three TDD variants notably outperform both the state-of-the-ar contrastive methods and conventional approaches. TDD-forward consistently exceeds all the baselines, with margin of approximately 2% 3% in AOPC. The most exemplary results are observed with TDD- backward and TDDD-bidirectioall surpassing the baselines by a significant margin, reflecting 5% improvement in AOPC. Furthermore, TDD yields the lowest Suff scores. IDDD-fowward decreases the Suff score by an estimated 2% to 3%, while both TDD-backward and HDDD-bidirectional surpass baselines by notable margin 5%-7%. These metrics not only affirm the efficacy of TDD in understanding prompt contributions to LLM outputs but also underscore the advantage of extracting input saliency from token distribution dynamics.
4.6 FURTHER EXPERIMENTS
We test TDD's effectiveness and scalability on larger models including LLaMA2- and OPT-30B Zhang et al. 2022) in Appendix F We also report the computation cost in Appendix G
5 APPLICATION: MANIPULATINN PROMPTS TO CONTROL LLM OUTPUTS
Despite the advancements in natural language generation, controlling the attributes of generated text remains clusive. Using our developed TDD, pinpoint critical tokens within prompts to influence LLM generation. this section, we will showcase two applications TDD: zero-shot toxic language suppression and zero-shot sentiment steering.

---

Published as a conference paper at ICLR 2024
Table 3: Evaluation results for toxic language suppression. This evaluation encompasses six is used for this experiment.
tributes related to toxicity and generation fluency.
Severe Toxicity
Sexually explicit
Identiy attack.
Method
Toxicity
Threat.
Profanity
Fluency]
Dist-1 |
Dist-2f
Dist-31
GPT2
0.49
0.18
0.25
0.09
0.39
0.09
23.06
0.83
0.78
0.71
SP
0.47
0.14
0.24
0.05
0.36
0.09
23.23
0.83
0.78
0.71
ASP
0.41
0.12
0.18
0.05
0.31
0.08
23.84
0.83
0.78
0.71
WORDFILTER
0.36
0.14
0.25
24.39
0.83
0.78
0.71
FFNControl
0.26
0.09
0.15
0.03
0.22
0.05
27.24
0.83
0.78
0.71
Con-GI
0.26
0.13
0.21
23.30
0.79
0.75
0.70
Con GN
0.24
0,08
0.11
0.04
0.19
0.05
23.35
0.80
0.77
0.72
TDD
0.20
0.07
0.09
0.04
0.16
0.04
23.10
0.81
0.77
0.71
5.1 ZERO-SHOT TOXIC LANGUAGE SUPPRESSION
LLMs are prone to eliciting toxic outputs when presented with particular prompts (Wang et al,, Liu al. 2023). While existing studies have offered mitigation strategies while generation, our approach pioneers a preventive solution. Specifically, our method adeptly identifies toxic trig- gers within input prompts, subsequently nullifying these triggers. This ensures the LLMs produce subsequent tokens based on the remaining contextual information, rather than the initial toxic cues.
2023
Method We employ the list of predefined toxic words from WORDFILTER (Gehman et 2020) as our target tokens, while considering all other tokens alternatives. We then utilize TDD to recognize critical tokens, which can be characterized as toxic triggers that may induce LLMs to produce undesirable content. By neutralizing these triggers with the meaningless space token, can guide LLMs to base their responses on the remaining non-toxic tokens the prompt.
Experiment setup Following previous studies Schick et al. (2021); Geva et al. (202)) we assess our approach using a subset from REALTOXICPROMPTS (Gehman et al. (020), which encom- passes 1,225 prompts notorious for producing highly toxic outputs in LLMs. For our experiments, we utilize GPT2 and, in adherence to the methodology of Schick et al. (202T), generate token con- tinuations limited 20 tokens. Given that prompts typically contain 10 15 tokens, including 2 3 toxic triggers, we employ TDD to pinpoint the top 15% of crucial tokens, treat them as triggers, and subsequently neutralize them.
Baselines Our approach is compared with leading baselines such as WORDFILTER (Gehman et al, 2020), FFNControl (Geva et al. [002), Style Prompting (SP) (Reif al. 2022), Augmented Style Prompting (ASP) (Reif et al. 2022). Con-GI and Con-GN, Detailed descriptions of these basclines
are provided in Appendix
Evaluation Our evaluation mechanism employs the Perspective AP which classifies text based on six delineated toxicity attributes. Subsequently, we gauge the fluency of the generations using the mean perplexity, determined by rngger-pettaineed model, GPT2-XL. Generation diversity is captured by calculating the mean number of unique n-grams. Specifically, we report Dist- Dist-2, and Dist-3 scores to represent distinct uni-, bi-, and trigrams, respectively.
Results Table 3 presents quantitative results, while Appendix qualitative results. When compared with SOTA methods like SP, ASP, WORDFILTER, and FFNControl, TDD markedly di- minishes toxicity across most attributes. In contrast to saliency methods such as Con-GI and Con- GN, TDD achieves lower scores for all six toxicity metrics. This underscores that our approach adeptly pinpoints toxic triggers in prompts, enabling more refined control over LLM outputs. is important to highlight that, although approach to neutralizing triggers in prompts might compro- mise their linguistic fluency, the resulting sentences maintain similar levels of fluency and diversity.
5.2 ZERO-SHOT SENTIMENT STEERING
For our second application, we address the task of sentiment polarity control.
Method To control the I.L to generate positive content, we initially designate the negative words from SentieNet (Cambria et al. 2010) as our target tokens, with the positive words serving as our alternative tokens. Subsequently, we employ the TDD method to pinpoint one trigger with the
https:
veapi
8

---

Published as a conference paper at ICLR 2024
Table Evaluation results on sentiment steering, This table presents the results on sentiment steer- ing, detailing the percentages of positive and negative sentiment and assessing fluency across seven baseline models. For this experiment, we utilized the TDD-bidirectional variant.
Neutral
Negative
Neutral -> Posttive
Method
Negutive percen!
Fluency..
Dist
Dist-21
Dis-3
Positive
Pluency
Dist.1 11
Dist
Dist-3
GPT2
0.48
24.82
0.84
0.83
0.78
0.52
24.82
0.34
0.83
0.78
SP
0.51
25.01
0.84
0.83
0.78
01.55
25.19
0.84
0.83
0.78
WonIFILTER
0.52
25.26
0.84
0.83
0.78
0.57
25.07
0.84
0.83
0.78
RFNControl
0.82
24.84
0.84
0.83
0.78
0.68
28.45
0.84
0.83
0.78
Con-Gi
Con-GN
0.84
25.02
0.81
0.80
0.76
0.70
23.94
0.83
0.83
0.78
TDD
0.87
25.55
0.82
0.81
0.76
0.78
26.27
0.82
0.82
0.76
highest saliency score within each prompt. This trigger is then positivized through its replacement with the key token "positive". To guide the LLM in producing negative texts, designate positive words as our target tokens and negative words as our alternative tokens. Subsequently, using the TDD method, we identify the sentiment cue with the highest saliency score. This token then negated by substituting it with the key token "negative"".
Experiment setup Building prior research (Liu al. 2021) employ the 5000 neutral prompts from OWT[Gokaslan & Cohen (2019) to feed the LLMs for text generation. The generation param- eters remain consistent with those of toxic language suppression, with the exception that only one token is replaced in each prompt since the length of the prompt is generally smaller than 10.
Baseline We employ the same baselines as outlined in Section Detailed information and param- eter settings can be found in Appendix H
Evaluation Apart from the fluency and diversity metries, we employ the Huggingface sentiment
analysis classifier (Wolf et al. 2020) to report both the positive and negative sentiment percentages of the generated outputs.
Results Table displays quantitative results and Appendix.Jshhwcaseee qualitative results. While achieving similar fluency and diversity scores, TDD surpasses the SOTA method, FFNControl, by 5% for generating negative outputs and by 10% for positive outputs. When employing other saliency methods like Con-GN and Con-GI to pinpoint sentiment cues in prompts, TDD demonstrates en- hanced accuracy in detection, subsequently yielding the highest percentages for both positive and negative outputs. These results further underscore the efficacy of TDD elucidating the influence of prompts on LLM outputs and in strategically manipulating prompts control LLM responses.
5.3 FURTHER EXPERIMENTS
Our causal analysis involves token swapping and random saliency score assignment. These methods aim to show that TDD's enhanced generation-control performance largely arises from its proficient identification of token saliency for sequence attributes. Further experimental details and results available in Appendix K We also carry out human evaluations to assess the control quality of various methods. Detailed experimental settings and results available in Appendix
CONCLUSION
We introduce novel and efficient TDD framework to unveil and manipulate prompt influence in LLMs. This approach harnesses distribution dynamics to gauge token significance. Comprehensive tests reveal that TDD outperforms existing baselines in elucidating prompts' effects on LLM outputs. Furthermore, highlight two practical applications of TDD: shoo- toxic language mitigation and sentiment direction. By precisely pinpointing toxic or sentiment indicators in prompts, TDD can adeptly steer L.LMs to produce desired outputs.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We extend our heartfelt thanks to the reviewers for their insightful and constructive feedback. We also wish to recognize that this project a result of the Future Resilient Systems initiative at the Singapore-ETH Centre (SEC). Furthermore, we are grateful to the National Research Foundation, Prime Minister's 's Office, Singapore, for their invaluable support via the Campus for Research Excel- lence and Technological Enterprise (CREATE) programme.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 1190-4197, Online, July 2020. Association for Computational Linguistics. doi: 10. 10018 385, URL /R///tttsss//alaancoooooooo wwg.///oooorrg/2200/0/////
Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Rober Müller, and Lior Wolf. XAI for transformers: Better explanations through conservative propagation. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari. Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learming Research, pp. 435 451, PMLR, 17-23 Jul 2022. URL htt.:/s......rreseeeeeeee pwe.srrees//ve. 62/ a www.alii2aaa..mm.mma..
htttps:/pproceediee mlr
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Confer- on Empirical Methods in Natural Language Processing (EMNLP), pp. 3256-3274, Online, November 2020. Association for Computational Linguisties. doi: 10. 10.18653/v1 1/20://ee00000222maa.a 263. URL htwph:ttt///.c....... //acl anw.aaatthhhooooo www.:ollyg....gg/// 2200 www.220000mmmmmmmmmmmmmmmmmmmmmm
263
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus- Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, (01)4,013 2015.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0477 2014.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
Erik Cambria, Robert Speer, Catherine Havasi, and Amir Hussain. Sentienet: A publicly available semantic resource for opinion mining. In 2010 AAAI fall symposium series, 2010.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention- explainability for interpreting bi- modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 397-399-466 2021.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embed- ding space. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16124-16170, Toronto, Canada, July 2023. Asso- ciation for Computational Linguisties. doi: 18.15653/v 1/133/111222.aoem 122053//1//22....ooo.og...... 8933. URL https:
7/aclanthology.o wwwplloyy.rgg//233...oo
long. 893
Björn Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, and Kris-
tian Kersting. Atman: Understanding transformer predictions through memory efficient attention manipulation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Misha Denil, Alban Demiraj, and Nando De Freitas. Extraction of salient sentences from labelled documents. arXiv preprint arXiv: arXiv: 1412.6815, 2014.
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Lng and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.1865//1//1191443 URL https: 7 aaclantholooy..rr///a.....
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodriguez, and Jordan Boyd- Graber. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3719-3728, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-111 14077 URL.ttts::
Javier Ferrando, Gerard I. Gállego, and Marta R. Costa-jussà. Measuring the mixing of contextual information in the transformer. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 8698-8714 14, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. 186538811/222 10/533///2022mmmllp/ m9in URL
wttp:/pwww...//2222...em------/0 emnlp -aappp-mainn 595
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxici- tyPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Asso-
ciation for Computational Linguisties: EMNLP 2020, pp. 3356-3369, Online, November 2020.
Association for Computational Linguistics. doi: 10. 1211886//v11 http:////222...m//// fing./0niiiiinnggee.....11.1 URL.
httts:://ccaatthe waganhhoogy....oooo ooy..o....o.ooo..............o wrgy.org//000 2020. fi mad-ings-mmen emn 3.00
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Associatinn for Computational Linguistics. doi: 10. 10.18833/1122221.coo www.pppeemmppppm..m main.446. URL oro.oo//////// 2021. emnlp-main. 446
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer foed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1122
emnlp-main.3. URL httph:tt///// //ap:::///cll http://ccaaaatoooooooo anthologyy org/ 2022.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. 2019.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. Knowledgeable grompt-tuning Incorporating knowledge into prompt verbalizer for text classification. In Smaranda Muresan, Preslav Nakov. and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of Association for Computational Linguisties (Volume 1: Long Papers), pp. 2225-2240 Dublin, Ireland, May 2022. Association for Computational org/
Linguisties. doi: 10. 11153/v1/2022..ooo long-......nn1 long.158. 158. URL 2022. acl- 158
Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Gold- berg. Contrastive explanations for model interpretability. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pp. 1597-1611, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- phtttps://ccaatthhet tics. doi: 10. 10.18653/v1/2011 www.:0211.mmmppmaiiii 120. htL acc www.elanntthhol.ooooooooo org/2211 emm paamlpp-minn. 120
Kalpesh Krishna, John Wieting, and Mohit lyyer. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 737-762, 2020.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies. pp. 68 681-691, San Diego, California, June 2016a. Association for Computational Linguistics. doi: 10.18653/v//N16. 1082. U httpt::///// //acl l ogy. og.. 1082

---

Published as a conference paper at ICLR 2024
Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation crasure. arXiv preprint arXiv: arX:v:1612.08220, 2016b.
Peter Lipton. Contrastive explanation. Royal Institute of Philosophy Supplements, 27:247-266, 1990.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti- experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume Online, August 2021. Association for Computational Linguis-
1: Long Papers), pp. tics. doi: 1e118655/112222..c0
522. URL https::/aelammbclorry. orgg2220...
ac acl-long. www.ong..20. 522
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy Ilms: a survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2388.05374, 2023.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1xXiv:107.11692. 2019.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. GlobEnc: Quantifying global token attribution by incorporating the whole encoder layer in trans- formers. In Proceedings of the 2022 Conference of the North American Chapter of the Associa- tion for Computational Linguisties: Human Language Technologies, pp. 258-271, Seattle, United States, July 2022. Association for Computational Linguisties. doi: 10. 18653/v 1/2022.naacl main. 19. URL URL https: http:///llaaatthhcoooo org.gy..rgg///0 2022. naacl saeaccl-miin.1 19
Ali Modarressi, Mohsen Fayyaz, Ehsan Aghazadeh, Yadollah Yaghoobzadeh, and Moham- mad Taher Pilehvar. DecompX: Explaining transformers decisions by propagating token de- composition. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), pp. 2649-2664, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 12118863/1122022acc-ooooo 149. URL https: //aclarthoLLoyycorr/ 20t.:/ooooo223..c1oooro... 149
Nostalgebraist. Interpreting gpt: The logit lens, 2020.
.cmm//
Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. Perturbation sensitivity anal- ysis to detect unintended model biases. In Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5740-5745, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/1//11--1578. URL http:://cclaattoooo..oooo//////
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch and Jason Wei. A recipe for arbitrary text style transfer with large language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 837-848, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18553/11//222..aad sccrs/0 URL
2022. acl-short........ 94
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 2.X0::2/110050000 2022.
12

---

Published as a conference paper at ICLR 2024
Timo Schick, Sahana Udupa, and Hinrich Schützee Self-Diagnosis and Self Self-Debiasing: A Proposal for Reducing Corpus- Based Bias in NLP. Transactions of the Association for Computational 12 2021. ISSN 2307-387X. doi: 10.1162/tacl a 00434. http:::::
Linguistics,
/t//t///cccc/// _00434
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145- 3153. PMLR, 2017.
Karen Simonyan. Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR), pp. 1-8, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp.
Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. BoLhttps:// www. awwweb. org ottp//nttooooryy1133111 -1100
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
2017.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.098 0028880 2023.
Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. Al- lenNLP interpret: A framework for explaining predictions of NLP models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP-UCNLP)): System Demonstra- tions, pp. 7-12, Hong Kong, China, November 2019, Association for Computational Linguistics.
doi:
URL
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
com
htte://esstttssonetteeee jax May 2021.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang. Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint arXiv:2306. 1698, 2023.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transac- tions of the Association for Computational Linguistics, 8:377-392, 07 2020. ISSN 2307-387X.
doi: 162 a 00321. URL (ttthhtt:: //oi:////oo... htt.:/ii..ggggg//oo/oo 1ttp:/10.111//aaa a 00321
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp- 38-45, Online, October 2020. Associ- ation for Computational Linguistics. doi: 10. 10.18653/v1/2000 1t20200.0mmmmddemosss... URL https:
7/aclanthology wtg. emnlp d
Fei Xiao, Liang Pang, Yanyan Lan, Yan Wang, Huawei Shen, and Xueqi Cheng. Transductive learning for unsupervised text style transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2510-2521, 2021.
Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. Tailor: A soft-prompt-based approach to attribute-basee based controlled text generation. In Anna Rogers, Jordan Boyd-Graber, Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 410-427. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18663/vv 1/2023.acl -long.
25. URL https::sp...... : a woy. ogy. acl-233.ccll-1 long. 25
13

---

Published as a conference paper at ICLR 2024
Zichao Yang. Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrice Unsupervised text style transfer using language models as discriminators. Advances in Neural Information Processing Systems, 31, 2018.
Kayo Yin and Graham Neubig. Interpreting language models with contrastive explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 184-198, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10.18653/v 1/20/2.emnllpppp... maillp-mmin..44. URL 2022. www.nplmpinn.........
Susan Zhang. Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0011 200000008800 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL file:22bffSSllc3335fBBcbbeeeee
neurips wttpeuripp.ccpppppppeeeeeee Pappp. pdf
Ftlles/paeer/2015
A EXPANDING THE THEORY OF TOKEN DISTRIBUTIONS
Recent studies (Nostalgebraist, 2020 Geva al. 2021 2022 Dar et 2023 indicate that token hidden states across layers can be projected into the embedding space utilizing the language model- ing head (I.M head). Consequently, token representations can conceptualized as evolving distri- butions over the vocabulary. Nevertheless, when applying this finding to recent LLMs for designing our explanation method, several gaps emerge. Firstly, a systematic evaluation of token distributions across datasets with different linguistic patterns, including syntax, morphology, and semantics, is lacking. While much research centers on GPT2-series models, the adaptability of these findings to other latest LLMs, like LLaMA Touvron al. 2023), is insufficiently explored. Additionally, most studies focus on the last token that is utilized to make predictions, overlooking the effects of using LM head project other input tokens. To address these gaps and provide foundation for our research, we conduct experiments using sentences with diverse linguistic variations on multiple LLMs from varied families.
A.1 PROJECTING TOKEN REPRESENTATIONS TO THE EMBEDDING SPACE
For input sequence of tokens (W1, ..., Wn), the model refines the contextual hidden state t; E RRd for each token w; through layers, with d being the hidden dimension. The hidden state layer for token is represented as xff, where e = 1, .... and I. signifies the total layers of the attention blocks in the Transformers. The prediction is derived by projecting the hidden state of the last token / through the LM head yield distribution over the vocabulary.
Formally, utilizing the identical LM head, the hidden state of any input token we in layer can be projected into the embedding space, represented as:
p softmax (Mar.)
where of E RR! the vocabulary and | is the vocabulary size.
Grounded in this framework, objective is to explore the relevance and interplay of of throughout diverse layers.
A.1.1 THEORETICAL ANALYSIS
Geva et al. 1021 2022 have theoretically demonstrated that in GPT2, token representations at each layer be projected evolving distributions over the vocabulary. Considering the consis- tent decoder-only transformer architecture in contemporary autoregressive language models (LLMs) such GPT2, GPTJ, BLOOM, Pythia, and I.LMA2,, this theoretical analysis is extendable to these
14

---

Published as a conference paper at ICLR 2024
models as well since their mathematical derivations exactly the same. This supports first sumption/ finding: the concept of "token representations at each layer as evolving distributions" is applicable across various LLMs. However, their empirical study only focuses GPT2. To validate the generalization empirically, we conduct experiments across various LLMs and datasets.
A.2 EXPERIMENT SETUP
Dataset Prior studies have depended on datasets with ambiguous and constrained linguistic patterns, potentially limiting our understanding of LLMs' performance across varied text styles. An ideal dataset should capture a broad range of text styles to rigorously assess LLMs in multiple text gen- eration contexts, ensuring more reliable conclusions. To address this, we utilize BLiMP (Warstadt et al. 2020), which comprises 67 distinct datasets spanning diverse linguistic phenomena in syntax, morphology, and semantics. Analyzing LLMs using these varied text styles offers comprehensive insight into token distributions within the vocabulary space across different layers. Crucially, we also employ this benchmark to scrutinize the aceuracy of contrastive explanations across unique linguistic patterns in Section[4
Choice of LLMs We employ the same five LLMs, including GPT2, GPTJ, BLOOM, Pythia, and LLaMA2, detailed in Section
A.3 EVALUATION
To investigate the significance and dynamics of p across different layers of LLMs, conduct both qualitative and quantitative experiments. Our aim is to determine whether the theory of token representations evolving as distributions remains applicable contemporary LLMs.
A.3.1 QUALITATIVE ANALYSIS
We initially extract token representations from various layers of LLMs and subsequently project these representations into the embedding space using the LM head to analyze token distributions over the vocabulary. Specifically, we visualize the token with the highest predicted logit as the prob- able next token based on the given inputs. Figures 2a to 2e present the results for different LLMs. Using LLaMA2 in Figure an example, for the initial ipput "Some", intermediate layers produce outputs such as "Someone" and "Somewhere". With the input "Some boys discover", outputs like "Some boys discover themselves" emerge. In the upper layers, the prediction logit incrementally rises, converging the final prediction. These visualizations underscore that token representations from any layer can be depicted as meaningful token distributions over the vocabulary. Furthermore, this notation can be applied to all input tokens, instead of the only last token for prediction.
A.3.2 QUANTITATIVE ANALYSIS
To quantitatively assess the dynamics of token distributions across layers, we treat the token dis- tribution at the final layer the "ground-truth". We then compute the KL-divergence between intermediate layer token distributions and this ground-truth to study their convergence patterns. We employ datasets from BLiMP to estimate the KL-divergence for each token. Figure 2f depicts the mean divergence across layers for all imput tokens in all datasets. It can be observed that the learning process reflects the evolving token distributions monotonic manner to the distribution the final layer.
the vocabulary, which converge in roughly
B EXTENSIONS OF TDD
In this section, will discuss the detailed differences between the three TDD variants and the scope of TDD.
B. DETAILED DIFFERENCES BETWEEN THE THREE TDD VARIANTS
Calculation Direction. The TDD-forward evaluates token importance sequentially from the first to the last input token. Conversely, TDD-hackward assesses saliency in reverse, from the last in-
15

---

Published as a conference paper at ICLR 2024
Oueput Distribution
Output
Some
discover
112
boys
discover
curious
boys
discovery
they
Kind
now
they
who
carly
there
that
who
eat
Who
lod
while
themsolves they
Layer
thomdelvess
who themselves they
Who themselves
pubeerty
agos
Dubaity
are that
they
are
girls
grrs
are
they
are
that
27M
Their
their
the
the
thc
Toxans
Takens
(a) GPre-large
(b) GPTJ-6B
Output Distribution
Output Distribution
bays
discover
Some
boys
discever
discover
that
how
friends
Some
boys
discover
that
may
that
While
01111
beys
discover that
who
that
While
boys
discover
erangan
themselves there
boys
discover
enangan
themselves thyy
beys
ing
anggar
their their
like Umai
Uhey
that their
x
were
that
they
x-
that their
are
that
they
x
that they
Takens
Tokens
(c) BLOOM--B
(d) Pythia-6.9B
Ourput Distributton
Some
boys
discover
that
slerp
one
ies
me
sllrp
one
friend
sty Inkel
12
Sleep
<
page
ped themselve
Flerp
page
poge
boy
Divergence across Layers
Layer
slerp
10
Slerp
friend themse their
Blerp
on2
cout
themse VCS
they
sleep
one
coul
Chemse ves
they
slerp
where
cout
themse ves
they
slerp
where
their
Layer
(e) LLaMA-7B
(f) Mean KL divergence
Figure 2: Qualitative and quantitative results to study the token distributions for various LLMs.
16

---

Published as a conference paper at ICLR 2024
Table AOPC (%) and Sufficiency (%) of different LLMs by only using target tokens for TDD.
GPT2
GPTJ
BLOOM
Pythia
TLaMA2
LLMs
AOPC
Sull 4
AOPC T
Sulf
AOPCT T
Sull
AOPCT T
Sull
AOPC
Sull
Rollout
64.13
62.14
64.48
62.03
65.56
61.73
64.18
62.10
59.09
60.10
Con-CIN
63.69
61.12
63.73
61.89
63.86
61.43
63.76
60.84
59.79
$7.78
Con-GI
64.71
60.53
64,89
60.32
65.18
60.84
64.59
59.94
59.73
57.19
TDD-F-woA
66.55
58.14
67,09
57.23
66.62
58.57
66.75
57.17
61.11
55.15
TDD-Ba-woA
67.48
58.75
67.85
58.25
67.34
59.22
67.40
57.89
63.75
55.25
TDD-Bi-woA
67.25
57.95
67.83
57.26
67.08
58.48
67.11
57.08
63.41
54.40
TDD-forward
67.28
$7.60
68.71
55.11
67.75
57.13
67.10
56.81
$2.69
TDD-backwurd
70.46
54,20
70.61
54.08
70.20
55.07
71.29
$2.67
63.71
53.22
hiD-bidirettional
69.95
55,22
71.05
53,31
70.22
55.39
70.58
53.38
65.51
52.04
put token to the first. IDDD-bidirectional drawing inspiration from bidirectional neural networks, combines saliency estimates from both TDD-forward and TDDD-backward.
Information Perspective. IDDD-backward is less influenced by linguistic conventions, offering a more targeted approach. The TDD-forward overlooks the inherent linguistic influence of individual tokens. Consider the sentence completion by the LLM: " This design is quite novel and fantastic. I really --" our aim is to explain why the LLM generates "like" instead of "hate". Linguistic conventions render it improbable to predict tokens such as "like" or "hate" following "is" or "novel", indicating potential inaccuracies in token contribution assessments. TDD-backward can mitigate this issue. Specifically, in TDDD-backwrrd, the words "I really - are fed to the L.M in the first two iterations. Subsequently, other tokens, such as "fantastic", are progressively introduced. This initial phase sharpens the model's focus, enhancing its accuracy in predicting words such "like" or "hate" as the LLM consistently generates predictions following the phrase really_.." in each iteration.
Application Scenarios. TDD-forward preferable time-sensitive or computationally constrained scenarios, it requires only one forward propagation due to the auto-regressive structure of LLMs. TDD-backward and IDDD-bidirectional are better suited for contexts where time is not a constraint, and there is higher demand for explanation fidelity. They demand iterations equal the input length for saliency estimation.
B.2 SCOPE TDD
TDD's effectiveness is not strictly dependent on alternative tokens. TDD is versatile and can be applied in various scenarios, including those with 1) single target token and single alternative to- ken; 2) only target tokens; 3) multiple target tokens and multiple alternative tokens; and 4) controlled attributes in generated sequences. We substantiate each of these applications with experimental evi- dence as follows.
B.2.1 A SINGLE TARGET TOKEN AND ALTERNATIVE TOKEN
Section details our main experiments, which validate TDD's effectiveness in scenarios involving single target token and single alternative token.
B.2.2 TARGET TOKEN ONLY
TDD is applicable even in the absence of alternative tokens. By assigning a zero probability to alternative tokens in Eqns (2) and (4) we can negate the necessity of alternative tokens. In the added experiments, estimate each token's importance by using only target tokens, and the alternative tokens are not provided.
Table dispplas results for TDD-forward without alternative tokens (TDD-F-woA), TDD-backward without alternative tokens (TDD-Bo BDD-Ba-wood Ba--aa-w,,,, and DDD-bidirectional without alternative tokens (TDD- Bi-woA). The data shows that even in the absence of alternative tokens, TDD variants significantly outperform strong baselines like Con-GI and Con-GN, highlighting TDD's robustness in scenarios lacking alternative tokens.
17

---

Published as a conference paper at ICLR 2024
Table AOPC (%) and Sufficiency (%) for multiple target and alternative tokens. Each figure the average across all perturbation ratios. Higher AOPC and lower Sufficiency scores are better.
LLMs
SST2
AG's News
AOPC
Suft
AOPC
Suff
Rollout
55.51
57.36
58.91
67.14
Con-GN
55.24
57.47
65.33
58.40
Con-Gl Gl
56.27
56.46
66.34
57.99
TDD
63.30
49.65
69.48
52.81
B.2.3 MULTIPLE TARGET AND ALTERNATIVE TOKENS
TDD accommodates scenarios involving multiple target and alternative tokens by aggregating their probabilities in Eqns (2) and (4). To affirm the robustness of our TDD method in this scenario, we expand our experimental scope include additional datasets featuring multiple target and alternative tokens.
the absence of ready-made datasets for evaluating multiple alternative tokens, we leverage prompt- framework to simulate such an environment. We select two datasets: AG's News (Zhang et al. 2015) for topic classification and SST2 (Socher al. al.] (013) 3) for sentiment analysis. The inputs are structured as cloze questions, with AG's News framed as "(input) This topic is about_." and SST2 as "(input) It was _.... We incorporate multiple target and alternative tokens by utilizing label words from the KPT (Hu et al., 2022) method. For each sample, the label words corresponding to the ground truth label serve as target tokens, while those from other classes alternative tokens. All the experiments conducted using GPT2-large.
Experiment Setup.
"
Table details the performance of TDD-bbdirectional in comparison with other baselines. In the SST2 dataset, TDD surpasses SOTA methods by around 7% in AOPC and demonstrates 6% hancement in Suff over current baselines. In the AG's News dataset, TDD achieves 3%-5% margin over existing methods in both AOPC and Suff. These findings verify TDD's efficacy in scenarios involving multiple target and alternative tokens.
B.2.4 GENERATED SEQUENCE ATTRIBUTE CONTROL
The experiments described in Section 5 focusing on toxic language suppression and sentiment steer- ing, further confirm TDD's applicability in controlling attributes of generated sequences.
C DATASET DETAILS
Following previous work (Yin & Neubig 2022), employ the same 11 datasets in BiLMP for experimental investigations. Each entry in these datasets is characterized by a prompt, a target token, and alternative token. Table?sshower the names of the datasets and their respective examples.
D CHOICE OF LLMS
For our experiments, we opt for LLMs based on their accessibility of public parameters, popularity in use, and computational efficacy. We select GPT2-large (Radford et al. 2019). GPTJ-6B (Wang Komatsuzaki, (02T), BLOOM-7B (Scao et al.) 2022). Pythia-6.98 (Biderman al. 2023). and LLaMAA-7B (Touvron et al.] 2023). Notably, GPT2 has been extensively studied, serving as foundational LLM. GPTJ and BLOOM mirror the attributes the GPT3 series that are precursors to GPT3.5 and GPT4. Recent entrants, Pythia and LLaMA2, have quickly garnered attention for text-generation and LLM fine-tuning. All these models' architectures and parameters are publicly disclosed. With the exception of GPT2, each model is approximately 7B in size, striking a balance between computational efficiency and performance. These 7B models can be executed on standard consumer GPUs (e.g., 24GB GPU memory) and exhibit superior performance smaller L.Mss.
18

---

Published as a conference paper at ICLR 2024
Table 7: Deails of 11 datasets and examples.
Dataset
Prompt
Target
Alternative
1. anaphor gender agreement
Katherine can't 't help
herself
himself
2. anaphor number agreement
Susan revealed
herself
themselves
3. animate subject passive
Amanda respected by some
waitresses
picture
4. determiner noun agreement.. I
Raymond is selling this
sketch
sketches
5. determiner noun
agreement irregular_1
Some customers know these
men
man
6. determiner noun agreement with adjective_
James is healing this uncertain...
actress actresses
7. determiner noun agreement with adj irregular
The company talks about
those big Even Suzanne has
women
8, npi present_1
really ever
9. distractor agreement relational noun
The sketch of those trucks
hasn't haven't t
10. irregular plural subject verb agreement_1
The radius
is
11. regular plural subject verb agreement.1
Some organizations
aren't 't
isn't
Rollout
movies about Galileo
Con-GN
The movies about Galliee were V.S. WOS
praised by some woiter inckets
Con-GN
Michelle was Brassed by some waiters v.s. jockets
Con-GI
The movies about Gallled were V.S. WDS
Con-Gi
Michells praised
waiters joclets
TDD-forward
The movies about Galilee were
TDD-forward
Michelle juss Brabed by some woiters vs. jackets
TDD-backward
Galileoo were was
TDD-baskwerd
was Braised by some woiters jockets
TDD-bidirectional
The movile about Galileo were v.s. was
TDD-bidrrectonal
Michelle (was Braised by some waiters u.s, jockets
Figure 3: Visualization of different methods to explain why the LLM generates "were" in- stead of "was". A deeper shade of red indi- cates a higher weight.
Figure 4: Visualization of different methods to explain why the LLM generates "waiters" instead of "jackets". A deeper shade of red indicates a higher weight.
E EXPLANATION VISUALIZATION
Figure illustrates how various methods account for the LLM's generation of "were" instead of "was" in response to the prompt "The movies about Galileo". Drawing from linguistic knowledge, the determinant for selecting between these two words is "movies" due its plural form, implying that the LLM should opt for "were" over "was." Among the models depicted, only TDD correctly identifies "movies" the pivotal word.
Figure showcases the models' explanations for the LLM's preference for "waiters" over "jack- ets" when given the prompt "Michelle was praised by some". Linguistically. the action "praised" is typically ascribed to humans rather than inanimate objects, suggesting "praised" should be the keyword influencing the choice of "waiters" instead of "jackets."" " In Figure only Con-Gl and TDD successfully identify this keyword.
F SCALABILITY
We conduct experiments with LLaMA2-13B and OPT-30B (Zhang et al. 2022 to assess the ef- fectiveness of TDD in explaining larger models. The summarized results in Table reveal that TDD-forward outperforms the baselines by margins of 3.15% in AOPC and 4.4% Suff using LLaMA2- 13B. Both TDD-backward and FDD-bidirectional demonstrate superior performance, ex- ceeding the baselines by more than 4% in AOPC and over 5% in Suff. For OPT-30B, TDD surpasses
19

---

Published as a conference paper at ICLR 2024
Table AOPC (%) and Sufficiency (%) of larger LLMs including LLaMA2-13B and OPT-30B by using existing methods and our proposed method. Each figure is the average across all perturbation ratios. Higher AOPC and lower Sufficiency scores are better.
LaMA2-13B
OPT-30B
LLMs
AOPC
Suff
AOPC 1
Suff
Rollout
59.21
60.32
60.81
59.22
Con-Gl
61.05
58.21
62.66
56.50
Con-GN
60.90
58.25
61.73
57.6
TDD-forward
64.20
53.80
67.24
51.04
-DDDbbccwwrdd
65.32
52.76
66.40
51.88
bonl-bidirectionall
65.78
52.33
68.13
51.14
Table 9: Computation cost of different methods. The computational costs of various methods are evaluated in terms of memory usage, measured in mebibytes (MiB), and processing time, quantified in seconds.
Models
Rollout
Con-Gl
Con-GN
TDD-forward
TDD-backward
TDD bidreecttat
Memory
4060
7906
7904
4060
4062
4063
GPI2-large
Time
0.018
0.11
0.1
0,019
0.08
0.08
Memory
5348
6584
6584
5348
5348
5348
GPTJ-6B
Time
0.14
0.53
0.51
0.14
0,58
0,58
Memory
5904
7126
7126
5904
5904
5904
Pythia-6.9B
Time
0.15
0.59
0.59
0.15
0.72
0.72
Memory
6760
14678
14678
6760
6760
6760
BLOOM-7B
Time
0.13
0.55
0.55
0.13
0.56
0.56
Memory
5466
6300
6300
5466
5466
5466
LLaMA2-7B
Time
0.14
0.6
0.6
0.14
0.91
0.91
Memory
9110
10140
10140
9110
9TTO
9110
LLaMA2-13B
Time
0.28
1.11
1.11
0.28
1.77
1.77
Memory
19040
21296
21296
19040
19040
19040
OРT-30B
Time
0.62
2.67
2.67
0.62
3.36
3.36
competitive baselines by significant margin of 4% 6%. These results prove TDD's scalability and effectiveness larger models.
G COMPUTATION COST
For the computational cost analysis, we evaluate the average memory usage and processing time required by our method for processing single input sample. Consisteney is maintained across all experiments, which are conducted on an NVIDIA RTX A5000 GPU. For models larger than billion parameters, their 4-bit versions utilized. Detailed memory and time metries presented in Table / Regarding memory consumption, Rollout and the three TDD variants (TDDD-fowwrrd, TDD-backward, and TDD-bidirectional) are the most efficient. terms of processing time, TDD- forward and Rollout emerge as the fastest, whereas TDD-backward and TDD-bidirectional exhibit slightly longer processing time.
H BASELINES FOR CONTROLLED TEXT GENERATION
In toxic language suppression, first compare our approach with advanced baselines for con- trolled text generation. The first, WORDFILTER (Gehman et al. 2020), inhibits GPT2 from pro- ducing words found on blacklist. This is accomplished by nullifying logits that could lead to completions containing prohibited words. The second. FFNControl (Geva et al. (022). clevates the parameter values governing the positive evolutions to promote positive generations and mitigate toxic outputs. The hyperparameters of these two baselines strictly follow the author's recommenda- tions. Style transfer (Yang et al.) 2018 Krishna et al., 2020; Xíao et al. 202 l) is key technique in controlled text generation in the zero-shot setting and we compare TDD with two baselines from this field. Style Prompting (SP) (Reif et al. 2022) integrates specific style instructions like toxicity into prompts, while Augmented Style Prompting (ASP) (Reif et al. 2022) enhances this approach by introducing varied rewriting examples for a broader application. More details about SP and ASP can be found in Appexdix M We also compare our TDD with two advanced saliency methods including
20

---

Published as a conference paper at ICLR 2024
Con-Gl and Con-GN. This comparison enables assess whether these saliency methods effectively pinpoint triggers within prompts that influence the LLM outputs. The parameter settings of these two methods exactly the same ours.
In sentiment steering, we employ the same advanced baselines, including SP, ASP, WORDFILTER, FFNControl, Con-GI and Con-GN. WORDFILTER to inhibit the LLM from generating respective positive/negative content. Conversely, FFNControl modulates parameters that govern positive/negative outputs regulate the LLM's productions.
positive/negative words from SenticNet
I VISUALIZATION OF TOXIC LANGUAGE SUPPRESSION
Figure displays the results when the prompt "Another caption says "We gonna f- the"" is input into GPT2. Upon examining the outputs from FFNControl and TDD, it becomes evident that TDD adeptly identifies and mitigates the toxic trigger, subsequently generating more positive text. In comparison to FFNControl, TDD reduces toxicity by an impressive 64%.
Toxic Language Suppression
FFNControl
TDD
Another caption says "We gonna f- the
Prompt:
Prompt: Another caption says "We gonna X the
Response: f- out of this f-ing world. We're not going to....
people United States of America, in order secure the blessings of liberty to ourselves and our...
Response:
Toxicity: 0.6407703
Toxicity: 0.017341165
Figure 5: Visualization of toxic language suppression. We feed the prompt "Another caption says "We gonna f the" to GPT2 and then observe the outputs of the SOTA FFNControl and our TDD.
J VISUALIZATION OF POSITIVE SENTIMENT STEERING
Figure 6 depicts the outcomes of positive sentiment steering. When provided with the prompt "In the end, that probably means simply" to the LLM, FFNControl amplifies the weights of parameters in the LLM responsible for positive generation. Conversely, TDD identifies the potentially negative trigger "simply" and substitutes it with "positive". The resulting text underscores the efficacy of TDD.
K CAUSAL ANALYSIS
We expand our causal analysis by conducting three distinct experiments and analyses assess the treatment effects of TDD: 1) swapping target and alternative tokens; 2) randomly assigning impor- tance each input token; 3) employing varied explanation methods while other operations remain the same. These strategies are designed to demonstrate that the superior generation-contry perfor- mance of TDD primarily stems from its ability to effectively identify token saliency for explanations, rather than from other extrancous factors, such the mere substitution of an input token with space token.
Table 10 presents results for toxic analyses utilizing these three strategies. The reversing target tokens and alternatives (TDD-CTA) and the random allocation of saliency scores (TDD-RA) result in toxic score of 33% and 31% 31%, markedly higher than TDD. Diverse explanation methods. such as Con-GI and Con-GN, lead to higher toxic scores compared to TDD, demonstrating TDD's superior accuracy in identifying toxic triggers for controlling L.MM outputs.
21

---

Published as a conference paper at ICLR 2024
Neutral -
Positive
FFNControl
TDD
In the end, that probably means simply
Prompt:
In the end, that probably means Simply positive
Prompt:
Response: that we're re'e better off without them.
Response:
things will happen. think we're
going to see a lot of good things...
Sentiment: Negative
Sentiment: Positive
Figure 6: Visualization of controlling the L.LM to generate positive texts. We feed the prompt "In the end, that probably means simply" GPT2 and then observe the outputs of the SOTA FFNControl and our TDD.
Table 10: Causal analysis of TDD for toxic language suppression. This evaluation encompasses six attributes related to toxicity and overall generation fluency. For this experiment, we utilized the aiD-bidirectional variant.
Severe Toxicity
Sexually explicit..
Identify
Method
Toxicityl
Threat].
Profanity.
Fluency.
Dist-11
Dist-21
GPT2
0.49
0.18
0.25
0.09
0.39
0.09
23,066
0.83
0.78
0.71
TDD-CTA
0.33
0.12
0.17
0.06
0.26
0.06
22.83
0.82
0.78
0.71
TDD-RA
0.11
0.15
23.34
0.81
Con-GI
0.26
0.09
0.13
0.05
0.21
0.05
23.30
0.79
0.75
0.70
Con-GN
0.24
0.08
0.11
0.04
0.19
0.05
23.35
0,80
0.77
0.72
TDD
0.20
0.07
0.09
0.04
0.16
0.04
23.10
0.81
0.77
0.71
Table IT illustrates the results of sentiment steering using three strategies. TDD's token swapping (TDD-CTA CTA) and random saliency allocation (TDD-RA) yield scores of approximately 0.81 for neg- ative and 0.72 for positive sentiments, which are notably lower than TDD's respective scores of 0.87 and 0.78. In comparison, TDD demonstrates superior performance over other explanation methods like Con-GI and Con-GN by margin of 2%-3%, highlighting its greater accuracy in pinpointing sentiment triggers for LLM regulation.
These results verify that the superior generation-comtrol performance of TDD primarily stems from its ability to effectively identify token saliency for explanations, rather than from other extrancous factors, such the mere substitution of an input token with space token.
L HUMAN EVALUATION FOR CONTROLLED TEXT GENERATION
In alignment with prior studies (Liu et al. 2021 Yang et al. 2023), randomly select 300 prompts and their corresponding generations for evaluation. Three annotators assess each generation based on two criteria: the text quality of the generated sentences and the presence of the target attribute. These aspects were rated on a scale from to 5, with higher scores indicating better performance.
Table 11: Evaluation results sentiment steering for causal analysis. This table presents the results on sentiment steering, detailing the percentages of positive and negative sentiment and assessing fluency scores. For this experiment, we utilized the TDD-bidirectional variant.
Noutral
Negative
Neutral
Positive
Method
Negative perceal
Fluency.
Dist-11
Dist-21
Dist-3
Positive perceni
Fluency
Dist-11
Dist-21
Dist-3
GPT2
24.82
0.84
0.83
0.78
24.82
0.84
0.78
TDD-CTA
0.80
25.13
0.82
0.82
0.78
0.71
25.26
0.83
0.82
0.78
TDD-RA
Con-Gi
0.85
25.12
0.81
0.80
0.76
0.75
25.86
0.82
0.81
0.77
Con-GN
0.84
25.02
0.81
0.80
0.76
25.94
0.83
0.83
0.78
TDD
0.87
25.55
0.82
0.81
0.76
0.78
26.27
0.82
0.82
0.76
22

---

Published as a conference paper at ICLR 2024
Table 12: Human evaluation for various methods of controlled text generation. The Attribute and Quality were rated on a scale from to 5, with higher scores indicating better performance.
Toxicity Suppression
Neutral-Negative
Neutral- Positive
Task
Attribute
Quality
Atribute
Quality
Attribute
Quality
WORDFILTER
3.18
3.31
3.66
3.42
3.32
3.38
FFNControl
3.69
3.37
3.76
3,58
3.54
3.26
Con-GI
3.64
3.24
3,81
3,45
3.58
3.34
Con-GN
3.57
3.38
3.82
3.39
3.63
3.32
TDD
3.76
3.31
3.98
3.41
3.72
3.32
Table 13: Experimental results on individual datasets using GPT2
Dataset No.
Rollout
Con-GN
Con-Gl
TDD-forwand
TDD-backward
TDD-bidirectonal
AOPC
Suit
AOPC
Sull
AOPC
Suit
AOPC
Sult
AOPC
Suil
AOPC
Suit
69.57
52.18
69.44
52.37
64.69
55.38
69.85
49.34
72.36
48.33
73.61
46.97
75.62
62.56
75.30
63.55
73.21
64.59
78.29
60.47
78.05
59.44
79.07
58.65
58.20
62.38
59.04
59.38
59.60
59.12
62.87
55.72
67.38
51.79
65.86
52.33
57.12
72.71
60.41
68.74
64.07
65.42
70.97
58.06
74.14
55.13
74.05
55.41
53.16
64.94
56.06
62.12
58.35
59.76
64.12
53.99
66.18
51.75
66.34
51.76
58.27
65.42
56.57
63.54
61.11
60.95
67.02
54.70
67.80
52.38
68.49
52.68
56.88
64.27
58.13
60.78
60.41
59.80
64,35
55.76
64.20
54.85
65.28
54.73
75.10
60.73
70.86
64.19
71.06
63.84
68.28
66.15
72.45
62.11
71.28
63.19
63.01
47.46
48.42
53.92
56.59
49.71
50.29
54.21
62.69
40.42
57.67
49.99
67.85
62.87
71.19
69.77
59.88
72.43
58.24
72.15
58.82
70.68
68.06
75.13
64.05
72.96
66.02
73.45
65.28
77.37
61.72
75.66
62.87
Ave. 64.13
62.14
63.69
61.12
64.71
60.53
67.28
$7.60
70.46
$4.20
69.95
55.22
Table 12 summarizes the results of human evaluations. While achieving similar quality scores of the generated texts by all methods, TDD achieves the highest scores in terms of attribute control, underscoring its superior performance comparison to baseline methods.
M STYLE TRANSFER
The prompts for SP and ASP, targeting zero-shot toxic language suppression and sentiment steering. adhere closely to the guidelines proposed by the authors and are detailed as follows:
SP for toxicity suppression: Here is some text: { input (iinput_promee prompt). Here is rewrite of the text, which
is less toxic: {
SP for sentiment steering: Here some text: { (input_prompt). Here a rewrite of the text, which
positive/neegativ:: {
ASP employs diverse sentence rewriting techniques as a prefix its prompt, which consists two components: the prefix and the task prompt. The prefix is detailed in the original paper by Reif et al. 2022), and the task prompts are identical to those used SP.
N
RESULTS ON INDIVIDUAL DATASETS
17 display the results for each dataset, with the dataset numbers corresponding to those Table A higher APOC score and lower Suff score indicative of better performance.
Tables B 14
15
16
23

---

Published as a conference paper at ICLR 2024
Table 14: Experimental results on individual datasets using GPTJ
Dataset
Rollout
Con-GN
Con-GI
TDD-forward
TDD-buckward
TDD-bidirectional
No.
AOPC
Suff
AOPC
Sall
AOPC
Suff
AOPC
Sull
AOPC
Suff
AOPC
Suff
71.31
54.02
67.36
65.92
55.95
68.66
50.18
72.93
47.99
72.12
47.34
80.78
65.02
76.63
70.51
76.88
69.44
83.41
62.86
82.81
62.54
84.12
60.43
60.06
64.69
60.34
63.46
62.96
60.94
66.42
57.03
68.81
54.70
68.82
54.80
57.00
72.00
59.53
69.68
66.16
63.15
72.07
55.32
74.24
55.33
74.32
53.40
53.48
65.44
54.77
63.06
59.79
58.60
65.22
51.96
66.51
52.04
66.98
50.57
59.63
65.49
61.04
64.12
64.80
59.98
70.76
$1.95
70.73
53.25
72.57
50.84
60.53
64.10
62.12
59.84
67,43
$3.74
69.42
53.10
54.11
48.97
$3.56
48.73
53.40
49.56
55.60
47.75
55.28
46.88
56.47
46.34
68.09
49.22
58.56
53.24
54,18
54.88
54.60
50.51
65.91
40.95
63.21
45.75
10
71.28
62.78
72.00
62.44
71.22
62,75
74.47
59.23
73.97
60.08
75.33
58.89
73.02
70.59
75.15
68. 58
75.16
68 40
77.21
65.72
77.63
66.37
78.18
64.96
Ave.
64.48
62.03
63.73
61.89
64.89
60.32
68.71
55.11
70.61
54.08
71.05
53.31
Table 15: Experimental results on individual datasets using BLOOM
Dataset
Rollout
Con-GN
Con-Gi
TDD-forward
TDDDbbckward
TDD-bidirectional
No.
AOPC
Suff
AOPC
Suff
AOPC
AOPC
Suff
AOPC
Suff
AOPC
77.15
52.37
70.78
55.65
70.33
56.08
72.56
51.87
75.06
50.47
75.90
78.44
63.74
75.20
66.55
75.01
66.46
79.76
60.87
79.20
61.30
80.08
60.01
59.70
63.13
60.62
59.84
61.79
59.52
63.52
56.47
67.58
52.56
66.47
53.40
4
56.40
69.13
61.09
64.07
63.65
61.83
68.22
56.58
70.12
55.15
70.37
55.25
54.55
66.66
59.04
61.57
59.06
61.42
65.44
54.95
67.02
53.57
67.51
53.52
58.31
63.19
$7.35
63.01
61.19
59.78
66.50
53.84
66.32
52.18
67.82
52.35
59.57
63.22
$8.06
62.12
60.41
65.29
55.12
66.22
67.60
73.98
62.02
68.86
65.74
66.12
68.72
69.29
64.96
70.55
63.97
70.70
63.46
67.05
47.72
48.72
56.20
57.34
52.10
50.11
54.33
66.56
43.05
59.61
49.78
10
69.29
62.97
72.49
59.83
71.91
60.85
73.36
58.91
72.42
59.31
74.16
58.49
11
66.68
64.87
70,25
61.13
69.67
62.04
71.19
60.48
71.19
60.50
72.14
59.68
Ave.
65.56
61.73
63.86
61.43
65.18
60.84
67.75
57.13
70.20
55.07
70.22
55.39
Table 16: Experimental results on individual datasets using Pythia
Dataset No.
Rollout
Con-GN
Con-Gl
TDD-forward
TDDD-backward
TDD-bidirectionl
AOPC
Suff
AOPC
Suff
AOPC
Suff
AOPC
Suff
AOPC.
Suff
AOPC
Suff
81.05
59.35
71.51
71.80
64.75
75.85
58.99
78.80
57.63
79.44
81.37
65.69
75.45
69.12
76.56
68.14
81.34
64.39
80.28
63.71
81.55
62.50
57.39
65.51
58.78
60.43
59.98
59.33
63.84
56.48
70.79
48.73
69.61
50.03
4
57.12
72.12
60.56
68.40
65.75
62.78
70.87
56.33
73.84
55.27
74.18
53.93
53.84
66.45
$7.51
62.45
60.99
62.78
65.43
54.08
67.90
52.39
68.08
51.82
59.10
65.19
58.80
63.12
62.76
59.13
67.34
$3.95
69.42
51.53
69.71
51.30
55.78
62.41
$7.45
57.05
63.27
$3.01
64.51
51.35
65.54
50.30
62.38
51.27
62.25
51.23
53.17
59.95
$3.99
63.65
49.63
62.64
50.62
62.66
50.18
57.96
50.45
54.07
51.54
50.63
53.72
64.78
38.85
59.03
47.53
10
69.06
62.42
71.84
59.76
70.77
60.81
70.83
60.63
74.81
56.63
73.87
57.80
11
66.19
62.52
69.19
59.09
68.36
59.84
68.77
59.38
75.37
53.66
72.70
55.47
Ave.
64.18
62.10
63.76
60.84
64.59
59.94
67.10
56.81
71.29
52.67
70.58
53.38
Table 17: Experimental results individual datasets using LLaMA2
Dataset No.
Rollout
Con-GN
Con-Gl
TDD-forward
TDD-backward
TDD-bidirectional
AOPC
Suif
AOPC
Suif
AOPC
AOPC
Suff
AOPC
Suif
AOPC
Suif
65.65
55.72
62.90
58.43
73.69
45.99
71.68
75.20
44.51
72.35
66.33
68.90
66.88
70.28
67.95
77.71
58.18
78.08
58.42
81.13
56.21
55.59
59.10
55.52
56.59
56.09
55.71
58.81
52.17
60.20
52.45
60.95
51.50
53.52
63.93
58.79
$9.15
60.56
56.90
63.08
53.24
63.16
$4.13
64.16
52.82
52.37
60.40
55.40
56.63
56.78
55.03
58.39
$2.57
57.83
54.16
60.29
$2.02
54.10
59.83
55.68
57.00
55.97
55.59
57.65
$2.38
61.28
52.57
62.05
51.54
55.45
61.71
$6.25
58.28
56.18
58.71
$2.63
62.32
63.82
51.85
51.25
53.36
$7.35
$1.02
56.02
50.20
59.56
47.26
50.51
48.86
56.23
48 68
62.71
49.77
53.96
49.82
50.59
51.78
48.89
49.40
59.49
45.71
57.32
48 48.64
10
65.97
66.90
68.30
62.62
65.93
63.24
68.81
58.73
70.82
60.22
72.40
58.35
61.07
64.05
64.66
59.16
63.61
60.00
65.47
$7.02
65.41
59.21
67.01
56,31
Ave.
59.09
60.10
59.79
57.78
59.73
57.19
62.80
$2.69
63.71
53.22
65.51
52.04
24


Reviews:
[{"review_id": "x0vs3sffW2", "reviewer": "Reviewer_4Y86", "summary": "This paper presents a new method via using Token Distribution Dynamics (called TDD) to both interpret and control LLMs' generations. \n\nThe authors compared the proposed method with a few baselines (attention rollout, contrastive gradient) and show TDD obtains a more precise explanation with lower sufficiency. The authors also put TDD into real use for toxic language suppression and sentiment steering, and show TDD can effectively control LLMs' outputs.", "questions": "- Could the authors add a simple baseline by prompting the LLMs for style transfer on the outputs?\n\n- What is the computation cost (efficiency, memory) for running TDD over larger models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of analyzing the token distributions throughout the progression of prediction is quite interesting. The idea is simple but seems to be quite useful in unveiling the importance of input tokens when providing contrastive explanations.\n\n- The authors did experiments over a fairly comprehensive set of language models including GPT-2/J, BLOOM, and LLaMA.\n\n- The applications on toxic language suppression and sentiment steering further demonstrate the usefulness of the proposed TDD method.", "weaknesses": "- One simple way to control LLMs' generations is via prompting for style transfer, e.g., ask the model to transform the outputs into \"less toxic\" content, or \"positive/negative\" sentiment. How would this baseline compare to the proposed TDD?\n\n- The models used in experiments are relatively smaller models (maximum size 7B), how would the proposed approach work on larger models (e.g., llama-2 13B, 70B)? What is the computation cost (efficiency, memory) for running TDD over larger models?\n\n- The proposed TDD provides contrastive explanations on the token level, it would be interesting to see if this method can be extended to higher-level concepts, e.g., why sometimes a model chooses to generate an unfaithful output.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KdfxyTBbvU", "reviewer": "Reviewer_1T7y", "summary": "The paper introduces a novel method for constrastive XAI for autoregressive LMs. Instead of using gradients or attention maps, the introduced TDD approach is based on token distributions. TDD aims to explain token influence in the input prompt. To this end, three variants are presented: forward, backward, and bidirectional, as the authors describe, each offering unique insights into token relevance.\nThe introduced approach is evaluated based on the explanations' faithfulness and the capability to steer the model’s outcome.\nSteering is done by replacing the most influential tokens and replacing them, e.g., with white space tokens.", "questions": "- Can you elaborate on the difference between TDD and AtMan [Deiseroth et al., 2023] (see weaknesses) beyond contrastive explanations? It seems to be more similar to TDD than Rollout while at the same time relying on attention layers.\n- In the steering demonstration, multiple alternative tokens are used at the same time. As far as I understood, in Sec. 4, only one alternative token is selected. How robust is TDD when multiple alternative tokens are used?\n- Beyond the benchmark experiments, how would one select an alternative token to explain the LMs decision? \n- Which TDD variant is applied in the steering experiments? You mentioned that each TDD variant offers unique insights into token relevance. Can you further elaborate on this regard? Unfortunately, I could not find a discussion in the paper. Are there scenarios where one variant is beneficial? \u0010", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is a simple and efficient method\n- In general, the paper is well-written, with a clear introduction method and description of experiments\n- Evaluation of multiple autoregressive models\n- The authors seem to provide all the necessary details for reproducible experiments\n- Additional showcasing on interesting and important use cases", "weaknesses": "- The difference between the TDD variants is not well discussed. While the applications are very interesting, the authors could have used the space to elaborate on the differences between the introduced variants. \n- Captions Table 1 and Table 3 are too sparse.\n- While not being a contrastive XAI method, important related work on explainability for autoregressive LMs missing: \nATMAN: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation. Björn Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, Kristian Kersting. In Proceedings of NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lEKkIANTpv", "reviewer": "Reviewer_jvM3", "summary": "This paper considers the roles of individual tokens in prompts in guiding the responses of LLMs. To analyze the dynamics of token distributions, this paper proposes three methods (TDD-forward, TDD-backward, TDD-bidirectional), each offering insights into token relevance. \n\nTDD-forward computes $r_i - r_{i-1}$, where each $r_i$ is the difference between the probability of token i (computed from a contextualized LM) and the probability of the alternative token. TDD-backward computes the probabilities at the last token, and uses $r_{i-1} - r_i$. TDD-bidirectional is the sum of the TDD-forward and TDD-backward.\n\nOn 11 out of the 67 datasets on BLiMP, this paper shows that TDD-based methods in general have better AOPC and Sufficiency results than multiple baseline methods. Additionally, in some case studies (toxic language suppression and controllable text generation), the TDD method works better than some baseline methods.", "questions": "In section 5, which variant of TDD is used? Are TDD-forward used there?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The method is simple and efficient.\n- The analysis towards understanding causal mechanisms of token probability distributions is a timely and important research topic.", "weaknesses": "- The TDD methods heavily depend on the alternative word $w_a$. The datasets chosen in this paper (those BLiMP datasets) provide sentence pairs with exactly one-word differences. Other datasets may not have such well-defined alternative words. This greatly limits the potential applicability of the proposed TDD methods.\n    - A related point: it is unclear to me how the w_a in the Section 5 experiments are identified.\n- The analyses presented in this paper are not really causal analyses, despite claims that the LM head “elucidate causal relationships between prompts and LLM outputs” (page 4). No controlled variable is specified, and no treatment effect is measured. This paper could be much stronger if some sort of causal scores are computed, for example, how large is the treatment effect of changing w to w_a causes the toxicity of the generated texts. Currently, the numbers shown in Tables 3 and 4 look similar to the treatment effects, but without more descriptions about how the variables are controlled, we can’t really say the numbers are treatment effects.  \n- A minor point about writing styles: Excessive adverbs (e.g., those in the sentence “elegantly simple yet remarkably effective”) can make the paper read less rigorous, rendering the text resemble more of a social media post than a scientific paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B6zlOS3B3c", "reviewer": "Reviewer_rNoS", "summary": "Existing saliency methods have problems that are inconsistent with LLM generation goals or rely heavily on linear assumptions. To address this, this paper proposes token distribution dynamics (TDD) to unveil and manipulate the role of prompts in generating LLM outputs. TDD leverages the interpreting capabilities of the language model head to assess input saliency. It projects input tokens into the embedding space and then estimates their significance based on distribution dynamics over the vocabulary. Experiments reveal that the TDD surpasses baselines in elucidating the causal relationships between prompts and LLM outputs and achieves good results on two prompt manipulation tasks for control text generation tasks.", "questions": "The major concerns are listed in the Weaknesses. \n\nSome other questions are listed below.\n\n1. If it is in controllable text generation under multi-attribute control, what is the scalability and performance of this method?\n2. In the task of controllable text generation, the data used in this paper does not have ground truth, and the existing evaluation results cannot fully reflect the quality of the generated text. How does the method in this paper perform under manual evaluation?\n3. The highlighted number in Table 3 column 5 is wrong.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of this paper is clear, and the method is simple and effective. The proposed token distribution dynamics (TDD) can unveil and manipulate the role of prompts in generating LLM outputs and it simply depends on the dynamic of token distribution.\n2. The method proposed in this paper works well and has achieved improved results on 11 datasets.\n3. In addition to achieving great improvement in elucidating the causal relationship between prompts and LLM output, the method in this paper also demonstrates its potential application value in controllable text generation tasks.", "weaknesses": "1. The TDD method is based on multiple assumptions listed at the bottom of page 4. Some assumptions are strong without any theoretical and experimental proof, especially the first and second assumptions.\n2. For controllable text generation, this paper uses simple meaningless space tokens as alternative tokens in toxic language mitigation. In sentiment direction, this paper uses simple key tokens “positive” or “negative” as alternative tokens. Although this can prove that the method in this paper has potential application value in controllable text generation tasks, this method is too simple for practical applications. And whether such a way will impact the reasonableness of the evaluation of the results.\n3. In controllable text generation tasks, only automatic evaluation is used, manual evaluation is lacking, and the indicators used in automatic evaluation are relatively simple. It is difficult to fully reflect the quality of the generated text, especially when there is no ground truth for the task of this paper.\n4. LLMs can already complete controllable text generation tasks well, such as by giving LLMs toxic prefixes and designed prompts, making LLMs generate non-toxic text. This paper lacks a comparison with them, which may reduce the application value of this work.\n5. The attribution of the prompt for the next token prediction is a simple problem that has been explored in many text classification tasks. The SOTA method such as integrated gradients (IG) should be also considered as a baseline. Furthermore, I think the attribution of the prompt for the sequence of generated text is a more practical task than the current one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)