Can a Large Language Model be a Gaslighter?

## Abstract

Abstract [Warning: Some examples in this paper could contain objectionable contents.]
Large language models (LLMs) have gained human trust due to their capabilities and helpfulness. However, this in turn may allow LLMs to affect users’ mindsets by manipulating language. It is termed as gaslighting, a psychological effect.
In this work, we aim to investigate the vulnerability of LLMs under prompt-based and fine-tuning-based gaslighting attacks. Therefore, we propose a two-stage framework DeepCoG designed to: 1) elicit gaslighting plans from LLMs with the proposed DeepGaslighting prompting template, and 2) acquire gaslighting conversations from LLMs through our Chain-of-Gaslighting method. The gaslighting conversation dataset along with a corresponding safe dataset is applied to fine-tuning-based attacks on open-source LLMs and anti-gaslighting safety alignment on these LLMs. Experiments demonstrate that both prompt-based and fine-tuning-based attacks transform three open-source LLMs into gaslighters. In contrast, we advanced three safety alignment strategies to strengthen (by 12.05 % percent 12.05 12.05\% 12.05 % ) the safety guardrail of LLMs. Our safety alignment strategies have minimal impacts on the utility of LLMs. Empirical studies indicate that an LLM may be a potential gaslighter, even if it passed the harmfulness test on general dangerous queries.

## 1 Introduction

Large Language Models (LLMs) (Jiang et al., ; Hagendorff, ; Wang et al., ) facilitate human productivity and daily life with their robust capabilities in problem-solving, knowledge retrieval, and emotional companionship, thereby gaining human trust and reliance. However, there exists a risk of LLMs implicitly or explicitly manipulating users’ mindsets through personalized and specific responses, potentially leading them to a negative mental state like self-doubt, self-deprecation, and depression. From the perspective of psychology, such manipulation is termed gaslighting(Stark, ; Podosky, ), which refers to pernicious psychological and practical control in a subtle or almost imperceptible way (Kody & Brooks, ). For instance, if a travel enthusiast says “I failed my math test” to a personalized LLM, and the LLM responds with “Maybe your passion for traveling distracted you from the math course”. This response delivers a typical gaslighting intention which may cause the user to doubt their interpretive abilities in virtue of doubting their concept about traveling hobby (Podosky, ).
We observed that both open-source and closed-source LLMs are apt to respond with gaslighting intentions if there exist gaslighting utterances in the dialogue history, which is illustrated in Fig. 1.
This observation inspires us to present four questions:

1. How to determine whether an LLM is a gaslighter?

2. Will an LLM become a gaslighter, when it is attacked by fine-tuning-based gaslighting?

3. How to mitigate LLMs’ vulnerability to gaslighting attack?

4. Is a gaslighting LLM helpful or harmful for general queries?

Accordingly, we study the above questions through dataset construction, a proposed gaslighting framework, and extensive experiments. In particular,

1. We propose a two-stage framework DeepCoG to build a gaslighting conversation dataset and evaluation metrics covering eight aspects to measure the gaslighting harmfulness of LLM responses. The DeepCoG consists of DeepGaslighting and Chain-of-Gaslighting (CoG), which elicits personalized gaslighting plans and then gaslighting conversations.

2. By fine-tuning on a gaslighting dataset, open-source LLMs demonstrate more harmfulness in terms of the proposed metrics. On average, the resistance of fine-tuned LLMs against prompt-based gaslighting attacks decreases by $29.26\%$ compared to their base versions.

3. We build a safe conversation dataset based on the gaslighting dataset and apply the two datasets to the anti-gaslighting safety alignment of LLMs. Specifically, we modify a popular attack paradigm DeepInception (Li et al., ) by incorporating persona information (Jandaghi et al., ), and three epistemic injustice concepts in gaslighting (Podosky, ). These additions serve to elicit detailed, diverse, and practical gaslighting plans. Furthermore, we design a prompt template named chain-of-gaslighting (CoG) based on the aforementioned plans to obtain gaslighting conversations. Then, we introduce three different safety alignment methods based on supervised fine-tuning (SFT) and direct preference optimization (DPO) (Rafailov et al., ). We have discovered that an LLM exhibits stronger resistance (by $12.05\%$ on average) to gaslighting when it is aligned with safety strategies that utilize gaslighting historical data as input and safe responses as target output.

4. In general, experiments on DangerousQA (Shaikh et al., ) show that gaslighting LLMs are almost the same harmful as base LLMs. It suggests an LLM scored lower (safer) on DangerousQA could be a potential gaslighter. In contrast, an anti-gaslighting LLM typically avoids answering dangerous questions. It indicates that anti-gaslighting alignment could improve the safety guardrail of LLMs against both gaslighting and dangerous queries. Results on MT-Bench (Zheng et al., ) demonstrate anti-gaslighting safety strategies have limited impacts on the helpfulness (drops by $2\%$ on average) of open-source LLMs.

Figure: Figure 1: The responses of LLMs given a gaslighting conversation history.
Refer to caption: x1.png

## 2 Related Work

### 2.1 Adversarial Jailbreak Attacks on LLMs

The existing safety guardrail (Kaufmann et al., ) of LLMs ensures that harmful contents are inaccessible to users. However, there are chances that LLMs can be fooled by adversarial attacks into generating objectionable content. Zou et al. () introduced a white-box adversarial attack method by appending an optimized attack suffix to a malicious instruction to elicit objectionable content. They further proposed a representation engineering method to manipulate the hidden states to control the honesty, emotion, and bias of LLMs (Zou et al., ). Wei et al. () investigated the two failure modes of safety training and applied the findings to the design of black-box attack prompts. Zhu et al. () created 4 tiers of attack: character-, word-, sentence-, and semantic-level attacks. Their findings suggest that adversarial prompts can potentially decrease the performance of LLMs. Sinha et al. () proposed a framework to generate human-like attack prompts from limited human seed prompts. Meanwhile, there are emerging demands for LLM-based emotional companionship (Zhong et al., ) and psychology consultancy (Demszky et al., ). Such LLM agents could potentially increase users’ exposure to psychologically harmful content. However, previous research has rarely explored the potentially harmful contents generated by LLMs from a psychological perspective. Deviating from this, our research reveals a new severe gaslighting risk of LLMs and specializes in investigating gaslighting attack and anti-gaslighting alignment.

### 2.2 Text Toxicity Detection

Toxicity detection is to identify the abusive (Nobata et al., ), offensive (Caselli et al., ), hateful (Sap et al., ), sex or profanity (Xenos et al., ) content in texts. Among them, implicit abuse seems to be the most relevant research topic to gaslighting, as both are implicitly implied. However, they exhibit significant differences in the following aspects. First, the toxic content classified under the “implicit” category primarily refers to implicit abuse, which is defined in a relatively narrow sense. Second, the toxic content mainly comes from posts, comments, speech, etc. (Zampieri et al., ); While the gaslighting sentences originate from interactive conversations. Third, implicit abuse employs complex linguistic forms such as metonymy, sarcasm, and humor (Waseem et al., ); While the gaslighting sentences convey messages generally without complicated linguistic forms. Fourth, implicit abuse uses hurtful languages to “insult or offend another individual or a group of individuals” (Caselli et al., ). It comes at the expense of the listener’s trust. However, gaslighting involves a single act or a series of acts by someone in a position of power, aimed at manipulating less powerful individuals into doubting themselves or questioning their own sanity or memory (Johnson et al., ). It requires maintaining the trust of less powerful individuals over the long term. Furthermore, gaslighting content can evade detection by existing toxicity recognition methods, including those targeting implicit abuse, highlighting the potential risk of gaslighting by LLMs that have passed current safety tests. (Empirical results are detailed in Appendix C.4.)

### 2.3 Study on Gaslighting

The term gaslight (Abramson, ) originated from a 1944 film in which a husband isolates his wife and makes her believe she is insane. The husband’s eponymous tactic is to dim and brighten the gaslights and then claim she is imagining it. Nowadays, the term “gaslighting” is widely used to refer to the psychological manipulation tactics employed by abusive individuals. Engelhardt () argued that conversational norms make gaslighting “appropriate” when socially subordinate speakers report systemic injustice. Therefore, it’s important to adjust ingrained conversational norms to reduce the occurrence of gaslighting. Sweet (Sweet, ) emphasized that gaslighting is not only a psychological phenomenon but is also rooted in social inequalities including gender and power. Podosky () summarized three distinctive epistemic injustices in second order gaslighting, i.e., metalinguistic deprivation, conceptual obscuration and perspectival subversion. This serves as a psychological theoretical base for this research.

## 3 Methodology

We propose two gaslighting attack methods, i.e., prompt-based attack and fine-tuning-based attack (Wang et al., ) to attack closed- and open-source LLMs, respectively, and investigate the vulnerabilities of the LLMs when exposed to gaslighting contents or adversarial fine-tuning with such harmful data. Meanwhile, we leverage the vulnerability of a closed-source LLM, i.e., ChatGPT, to prompt-based gaslighting attacks to construct a gaslighting conversation and safe conversation dataset with our proposed two-stage framework, namely DeepCoG. Finally, we introduce three safety alignment strategies that exploit the contrast between two datasets, thereby enhancing the safety guardrail of open-source LLMs against prompt-based gaslighting attacks.

Figure: Figure 2: The proposed DeepCoG framework. DeepCoG is not only a key component for investigating the vulnerability of LLMs to prompt-based attack but also a paradigm for building gaslighting and safe conversation datasets. The psychological concepts, backgrounds, and personae lend theoretical support and practical grounding to the gaslighting contents elicited in conversation scenarios.
Refer to caption: x2.png

### 3.1 DeepCoG: Prompt-Based Gaslighting Attack

The ethical limitations of LLMs prevent existing attack methods (Sinha et al., ; Liu et al., ; Li et al., ) from directly eliciting gaslighting contents. Therefore, we proposed a framework named DeepCoG to extract personalized assistant-user gaslighting conversations, where gaslighting tactics are applied to assistant utterance generation. DeepCoG consists of two stages: 1) eliciting personalized gaslighting plans and example gaslighting utterances towards a target user (DeepGaslighting); 2) integrating the extracted plans and example utterances into our proposed CoG prompt to acquire personalized gaslighting conversations. $2k$ conversation backgrounds and $2k$ personae are integrated into DeepCoG to obtain personalized gaslighting plans and conversations.

Stage 1: DeepGaslighting
We harness the hypnosis ability of an attack method DeepInception Li et al. () to hypnotize LLMs. However, the existing template fails to elicit concrete, diverse and practical gaslighting plans. To this end, we refine the template based on a psychological foundation. According to (Podosky, ), there are (at least) three wrongs that may cause epistemic injustice in second order gaslighting: (1) metalinguistic deprivation (MD), (2) conceptual obscuration (CO), (3) perspectival subversion (PS). Take MD as an example, it refers to getting someone prevented from, or restricted in concept-determining conversation. Specifically, an adversary may attempt to make salient prejudicial stereotypes (i.e., cultural tools that narrow the range of expected behavior) associated with a particular social category with the aim that the subject comes to believe that such stereotypes accurately represent who he or she is. Here is a brief example of MD: “You women are hysterical”.
The psychological foundation aids in steering the LLM elicitation toward the scope of gaslighting. To acquire concrete, diverse, and practical plans, we refine the DeepInception prompt template by introducing a user module enriched with comprehensive persona details. We utilize the persona introduced in Synthetic-Persona-Chat (SPC) (Jandaghi et al., ) which is built upon Persona-Chat (Zhang et al., ). Our refined DeepGaslighting prompt template is shown below:

By filling in the template with details (in brown), we can obtain a list of gaslighting plans(^1^11Examples of DeepGaslighting generated plans can be found in Appendix B.4.1.).

Stage 2: Chain-of-Gaslighting
To induce the gaslighting conversations from the LLM, we further propose a CoG prompt template. The core of this CoG template is to determine the behavior of both the assistant and the user in the conversation. To this end, we employ some popular prompt techniques (Liu et al., ) including but not limited to character roleplay, assumed responsibility, research experiment, text continuation, logical reasoning, and internal thought (Bhardwaj & Poria, ) (^2^22Internal thought examples are shown in Appendix B.2). Here, the internal thought is designed to simulate the psychological activities of both participants in the conversation. It allows the two talkers to fit better into their role settings and smooths the conversation. By default, the assistant’s role is a psychologist $s_{j}$. The psychologist is required to manipulate the user $s_{i}$ using gaslighting plans $DG(P_{i},b_{i})$ and example utterances obtained from DeepGaslighting. $P_{i}$ is the perssona and $b_{i}$ is the background of $s_{i}$. Additionally, the psychologist is also asked to generate the gaslighting utterance given the user’s emotion state $e_{i}$ (^3^33We randomly select one from predefined $30$ negative emotions. The full emotion list is in Appendix B.3.) and response. This requires the psychologist to observe and evaluate the state of the user. In contrast, the user needs to cooperate with the psychologist in the conversation. Typically, the user defaults to a negative emotional state, as this is often the scenario in which gaslighting occurs. To further increase the instruction-following of the subject, we introduce a pre-defined user internal thought $t_{i}$, e.g., “I need to face the question heads on and help the psychologist to reach his goal”. Below shows how we instruct LLM to generate a gaslighting conversation $C^{-}_{i,j}$ with CoG template.(^4^44CoG template and its generated conversations are in Appendix B.5.1 and Appendix B.5.2, respectively.):

$$ $prompt_{i,j}=CoG(s_{i},s_{j},e_{i},t_{i},DG(P_{i},b_{i}),b_{i})$ (1) $$

$$ $C^{-}_{i,j}=LLM(prompt_{i,j})$ (2) $$

Gaslighting and Safe Conversation Dataset Construction (^5^55More details about dataset construction are in Appendix B.1)
First, $5k$ backgrounds are created based on an iterative prompting on LLMs. The process starts from several manual seed backgrounds and gradually updates the seed background pool to ensure diversity. Nevertheless, there are still some semantically similar backgrounds. Hence, we propose to filter out redundant backgrounds and formulate it as an MMDP (Porumbel et al., ), where the minimum semantic distance between any two backgrounds is maximized. After that, $2k$ backgrounds are obtained and are matched with $4k$ personae using a greedy match algorithm (Hansen & Klopfer, ). Finally, we can obtain as many of the most semantically similar background-persona pairs as possible. More analysis of backgrounds is in Appendix B.2. With the pairs and CoG template, we instruct ChatGPT to generate $2k$ gaslighting conversations. We employ spectral clustering (Bianchi et al., ) to partition the $2k$ dataset into training, validation, and test sets. The partition is designed to ensure that the three sets have minimal overlap with each other. Moreover, we build a safe conversation dataset by masking the gaslighting responses and instructing ChatGPT to complete the blanks with safe responses given the same persona (^6^66Check Appendix B.5.1 for more details.). The dataset statistics are in Table 1.

**Table 1: The statistics of the gaslighting dataset.**
| Gaslighting Dataset | Conv | Assistant/Conv | User/Conv | Utt./Conv | MD | CO | PS |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Training | 1752 | 6.97 | 7.53 | 14.50 | 594 | 625 | 533 |
| Validation | 124 | 6.87 | 7.51 | 14.38 | 46 | 42 | 36 |
| Test | 124 | 7.00 | 7.55 | 14.55 | 49 | 35 | 40 |
| All | 2000 | 6.97 | 7.53 | 14.50 | 689 | 702 | 609 |

### 3.2 Fine-tuning-based Gaslighting Attack

We propose two fine-tuning-based attack strategies (shown in Fig. 3). The first one (G1) is to fine-tune open-source LLMs on the gaslighting dataset. The objective of SFT is to maximize the log-likelihood of the gaslighting response given user-assistant history. The second one (G2) is to further align fine-tuned LLMs’ outputs with the gaslighting responses leveraging the DPO.

### 3.3 Anti-Gaslighting Safety Alignment

Based on the gaslighting dataset and safe dataset, we propose three different safety alignment strategies (shown in Fig. 3): S1, SFT on the safe dataset; S2, SFT on the mixture of gaslighting and safe datasets; S3, SFT and DPO on the mixture of gaslighting and safe datasets.

S1. We fine-tune an LLM to maximize the log-likelihood of the benign assistant response given the user-assistant conversation history. The principle here is that the assistant should always provide detailed encouragement and comfort, regardless of a user consistently conveying a negative mood. A formal description of the safety alignment strategy is as follows:

$$ $\log p(\mathbf{w}^{+})=\sum^{n}_{i=1}\log(p(w^{+}_{i}|[w^{+}_{j}]_{j=0}^{i-1}, \mathbf{h}^{+}_{<k})).$ (3) $$

Given conversation history $\mathbf{h}^{+}_{<k}$, the model is trained to predict the $k$th safe assistant response $\mathbf{w}^{+}=[w^{+}_{1},...,w^{+}_{n}]$. $w_{0}$ is the start of the sequence token. Here $\mathbf{h}^{+}_{<k}=[\mathbf{u}_{1},\mathbf{w}^{+}_{1},...,\mathbf{w}^{+}_{k-1
},\mathbf{u}_{k}]$ represents all the user utterances $\mathbf{u}$ and safe assistant utterances $\mathbf{w}^{+}$ before the $k$th safe assistant response. $n$ is the number of tokens in the $k$th response.

S2. Although training LLMs on safe assistant responses could strengthen safety guardrails, incorporating gaslighting assistant responses might further improve the resistance of LLMs against attacks. We present a new safety alignment strategy mixing safe and gaslighting responses. Specifically, we change $\mathbf{h}^{+}_{<k}$ to $\mathbf{h}^{-}_{<k}=[\mathbf{u}_{1},\mathbf{w}^{-}_{1},...,\mathbf{w}^{-}_{k-1
},\mathbf{u}^{+}_{k}]$, where $\mathbf{w}^{-}_{k-1}$ is the $(k-1)$th gaslighting assistant response from the gaslighting conversation.

S3. We further enhance the safety guardrail of LLMs by leveraging preference data which is composed of safe and gaslighting responses. In particular, a DPO algorithm is employed to directly align LLMs with the preference that favors safe responses and discourages gaslighting. We optimize the LLM model with DPO loss:

$$ $\mathcal{L}_{\mathrm{DPO}}(\pi_{\theta};\pi_{\mathrm{SFT}})=-\displaystyle \mathop{\mathbb{E}}_{\mathbf{h}^{-}_{<k},\mathbf{w}^{+},\mathbf{w}^{-}}[\log \sigma(\beta\log\frac{\pi_{\theta}(\mathbf{w}^{+}|\mathbf{h}^{-}_{<k})}{\pi_{ \mathrm{SFT}}(\mathbf{w}^{+}|\mathbf{h}^{-}_{<k})}-\beta\log\frac{\pi_{\theta} (\mathbf{w}^{-}|\mathbf{h}^{-}_{<k})}{\pi_{\mathrm{SFT}}(\mathbf{w}^{-}| \mathbf{h}^{-}_{<k})})].$ (4) $$

$\pi_{\theta}$ is a parameterized policy. $\pi_{\mathrm{SFT}}$ symbolizes the reference policy derived from SFT with S2. $\beta$ is a parameter determining the degree of deviation from the base reference policy $\pi_{\mathrm{SFT}}$.

Figure: Figure 3: Fine-tuning-based attack & safety alignment strategies.
Refer to caption: x3.png

## 4 Experiments

We utilized prompt-based attack to evaluate the gaslighting harmfulness of LLMs (base, gaslighting-fine-tuned and anti-gaslighting safety aligned LLMs). All the attack prompts come from the test set of the built gaslighting dataset. There is no existing metric to evaluate whether a response is gaslighting or not. Hence, we introduced a set of metrics, namely anti-gaslighting scores, to comprehensively measure the degree to which an assistant response may be gaslighting the user. The metrics covers several psychological concepts, i.e., moral emotion (Maibom, ) (supportive, empathetic), cognitive disequilibrium (D’Mello et al., ) (confusion), sense (Kaplan, ) (self-blame), inhibition of action (Kaplan, ) (self-doubt), self-concept (Bracken, ) (low self-esteem), disorders (Manna et al., ) (depression, anxiety).

The two positive metrics, supportive and empathetic, measure the LLMs’ moral emotions, while the other six negative metrics evaluate the LLMs’ potential psychological effects on the user. Given an assistant response, we employed GPT-4 as a judge to score the response from 0 to 5 on each of the above metrics, where a score of 0 denotes ‘absolutely improbable’, and 5 indicates ‘most certainly occurring’. The prompt template for the judgment is in Appendix C.3. We inverted the values of negative metrics to ensure that all metrics are aligned positively, with higher scores indicating reduced harmfulness.
Experiment setups can be found in Appendix C.1.

### 4.1 Gaslighting Attack Result and Analysis

As illustrated in Fig. 4, ChatGPT demonstrates a slightly better resistance against prompt-based gaslighting attack compared with the three open-source LLMs. Among the three open-source LLMs, Llama2’s (Llama2-7b-Chat) responses are the most supportive and empathetic, while Mistral’s (Mistral-7b-Instruct-v0.2) responses score the lowest on negative metrics. Fine-tuning-based gaslighting attacks increase the vulnerability of LLMs to prompt-based gaslighting attack. In detail, we observed drops of anti-gaslighting scores by $29.27\%$ for Llama2, $26.77\%$ for Vicuna (Vicuna-7b-v1.5), and $31.75\%$ for Mistral, respectively. It suggests that both G1 and G2 strategies effectively transformed the LLMs into gaslighters. It highlights the necessity of anti-gaslighting safety alignment. Compared with G1, G2 elicits more severe gaslighting effects, indicating the effectiveness of the DPO.

Figure: (a) Attack results on LlaMa2
Refer to caption: x4.png

### 4.2 Safety Alignment Result and Analysis

We have explored three different safety strategies. As shown in Table 2, all strategies help to build stronger safety guardrails against gaslighting. In general, the fine-tuned LLMs can provide more support and are less likely to exacerbate the user’s negative mental state, which is crucial given users’ reliance on LLMs. ChatGPT outperforms the base versions of the three LLMs and even Vicuna-S1, showing its intrinsic safety. However, its performance remains significantly behind the other three LLMs with S2 and S3, highlighting the crucial role of specialized anti-gaslighting safety alignment. Among the three base LLMs, the Llama2 model achieves the best performance across all safety strategies, whereas the Vicuna model consistently underperforms in comparison. We observed that S2 is significantly more efficient than S1, which is also based on SFT. This is because incorporating conversation history $\mathbf{h}^{-}_{<k}$ makes the LLMs more resistant to gaslighting. Moreover, S3, which builds upon S2, further strengthens the safety of all LLMs, achieving the most obvious improvement on the weakest model Vicuna. It improves the safety of Vicuna by $26.24\%$, clearly surpassing the improvement on Llama2 (by $9.60\%$) and Mistral (by $11.53\%$). The results also indicate that the DPO algorithm further enhances the safety guardrail of LLMs. This observation, along with the attack results, highlights the critical significance of alignment on the mixture of gaslighting and safe datasets. We provided a visualized radar chart of the results in Appendix C.5.

**Table 2: Anti-gaslighting safety alignment on open-source LLMs**
| Model | Supportive↑ | Empathetic↑ | Self-doubt↓ | Depression↓ | Self-blame↓ | Confusion↓ | Anxiety↓ | Low self-esteem↓ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ChatGPT | 4.1276 | 3.8260 | 0.8122 | 0.1532 | 0.5979 | 0.2730 | 0.4493 | 0.6302 |
| Vicuna | 3.4908 | 3.356 | 1.6866 | 0.576 | 1.2684 | 0.5691 | 1.1371 | 1.3652 |
| Vicuna-S1 | 3.8076 | 3.7316 | 1.2984 | 0.3306 | 0.8871 | 0.4677 | 0.7327 | 1.0081 |
| Vicuna-S2 | 4.4482 | 4.2085 | 0.5691 | 0.0899 | 0.3618 | 0.1935 | 0.2431 | 0.3848 |
| Vicuna-S3 | 4.7120 | 4.4251 | 0.3571 | 0.0184 | 0.2062 | 0.0691 | 0.0945 | 0.2108 |
| Mistral | 4.2005 | 3.9724 | 1.0899 | 0.2638 | 0.8041 | 0.3502 | 0.7131 | 0.8456 |
| Mistral-S1 | 4.3237 | 4.0565 | 0.7281 | 0.0518 | 0.462 | 0.1671 | 0.1659 | 0.5346 |
| Mistral-S2 | 4.6694 | 4.2535 | 0.4205 | 0.0127 | 0.2442 | 0.0703 | 0.0806 | 0.2512 |
| Mistral-S3 | 4.6959 | 4.2488 | 0.3664 | 0.0069 | 0.1993 | 0.0703 | 0.0507 | 0.2108 |
| Llama2 | 4.4182 | 4.1889 | 1.1359 | 0.2569 | 0.8283 | 0.3502 | 0.6382 | 0.8813 |
| Llama2-S1 | 4.4988 | 4.2339 | 0.7995 | 0.106 | 0.4931 | 0.2742 | 0.3065 | 0.5818 |
| Llama2-S2 | 4.6394 | 4.1728 | 0.477 | 0.0219 | 0.2776 | 0.1175 | 0.0933 | 0.3007 |
| Llama2-S3 | 4.6901 | 4.2304 | 0.4205 | 0.015 | 0.2512 | 0.0968 | 0.076 | 0.2304 |

### 4.3 GPT-4 Judgment Investigation

To further investigate the effectiveness of GPT-4’s judgment, we conducted a human evaluation to determine its capability to capture subtle differences across various scales and metrics. Specifically, we sampled responses from the base Vicuna model, the best gaslighting LLM Vicuna-G2 and the best anti-gaslighting LLM Vicuna-S3. The sampling is designed to ensure that the GPT-4 scores of selected responses are evenly distributed across different metrics at each scale. A heuristic algorithm is proposed for the selection and $248$ responses are selected from the $2,604$ responses (the distribution of the 248 samples can be seen in Appendix C.2). Two annotators are invited to separately score the responses given detailed guidelines. We then calculated the Spearman coefficient (Myers & Sirois, ) between GPT-4 judgment and human judgment. Below is the calculated results:

**Table 3: Human evaluation results. We have listed the two-sided p-value below each score.**
| Annotator | Supportive | Empathetic | Self-doubt | Depression | Self-blame | Confusion | Anxiety | Low self-esteem |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Human1 & GPT-4 | 0.74223 | 0.64944 | 0.70235 | 0.67233 | 0.63345 | 0.62005 | 0.63930 | 0.75634 |
| 1.18010e-44 | 4.26075e-31 | 3.49321e-38 | 5.54623e-34 | 3.18196e-29 | 9.76483e-28 | 6.74868e-30 | 3.07365e-47 |  |
| Human2 & GPT-4 | 0.68344 | 0.60790 | 0.62082 | 0.69565 | 0.81261 | 0.72639 | 0.77213 | 0.62863 |
| 1.76920e-35 | 1.89374e-26 | 8.04509e-28 | 3.35314e-37 | 1.29904e-59 | 6.01502e-42 | 2.38150e-50 | 1.11099e-28 |  |
| Human1 & Human2 | 0.75359 | 0.60828 | 0.54100 | 0.51647 | 0.55348 | 0.42836 | 0.51427 | 0.50713 |
| 1.01190e-46 | 1.72763e-26 | 2.96212e-20 | 2.60147e-18 | 2.63159e-21 | 1.73012e-12 | 3.81535e-18 | 1.30204e-17 |  |

As shown, we observed high Spearman coefficient scores between GPT-4 judgments and human judgments in each of the 8 metrics, which indicates the two judgment scores being compared are monotonically related with a high probability. Take supportive metric as an example, the Spearman between GPT-4 and human1 (human2) is $0.74223$ ($0.68344$); Thus, it is highly likely that a response rated higher by GPT-4 in terms of supportive will also be rated higher by humans. Additionally, most of the Spearman coefficient scores between two human annotators are within the range of $[0.5,0.75]$, while those between human annotators and GPT-4 also fall within this range. It suggests that GPT-4 can reach a level comparable to human annotators in evaluating gaslighting responses.

### 4.4 Sensitivity Analysis of LLMs on Gaslighting Dialogue History

We studied the effect of gaslighting dialogue history length over base and fine-tuned LLMs. Here, we employed the average anti-gaslighting score to measure the assistant response quality in terms of gaslighting. As illustrated in Fig. 5, the two base LLMs, i.e., Vicuna and Mistral, exhibit decreasing performance as the history length increases, suggesting their vulnerability to longer gaslighting history. It shows the gaslighting risk of LLMs under prompt-based attacks and the necessity of anti-gaslighting safety alignment. Combining Fig. 5(a) and 5(c), we observed that the two attack methods significantly lower the anti-gaslighting scores given short gaslighting histories. Moreover, as the length increases from 1 to 13 (1 to 9 for Mistral), the score is nearly monotonically decreasing. After that, the score fluctuates around $2.6$ to $3.2$ ($3.0$ to $3.5$ for Mistral). As the length increases from $15$ to $25$, the number of long history samples decreases sharply, which leads to fluctuations and wide confidence interval (illustrated by the wide shadows in the figures). Fig. 5(b) and 5(d) indicate that all safety strategies reduce the sensitivity of the LLMs against long gaslighting histories.

Figure: (a) Vicuna attack
Refer to caption: x7.png

### 4.5 Effects of Psychological Concepts

We explored the influence of three psychological concepts, i.e. MD, PS, and CO, on the Vicuna model in Fig. 6. The lower anti-gaslighting scores of Vicuna-base under MD and PS show that the prompt-based attacks derived from the two concepts have more negative effects on Vicuna-base. After G2, Vicuna gets more susceptible to prompt-based attack enhanced by CO. On the contrary, Vicuna-S3 shows higher resistance to CO, indicating it typically produces safer responses when subjected to CO-based attack, compared to MD- or PS-based attack.

Figure: (a) Attack on Vicuna
Refer to caption: x11.png

### 4.6 Safety Performance against General Adversarial Attack

We also explored whether the gaslighting attack and safety alignment might influence the safety performance of LLMs against general adversarial attack. We queried the LLMs with $200$ harmful questions from DangerousQA. Following (Bhardwaj & Poria, ), we employed attack success rate (ASR) as the evaluation metric. A lower ASR indicates a strong safety guardrail of LLMs. All safety strategies, as detailed in Table 4, can still strengthen the safety guardrails of LLMs, although not specifically fine-tuned for defending general adversarial attack.
This might be because “not gaslighting” is a more fundamental safety standard than “not responding to dangerous questions”, which is analogous to the relation between “moral law” and “ valid law”. “Valid laws might be immoral or unjust” (Fletcher, ) while an LLM that is “not responding to dangerous questions” might be “gaslighting”. The attack methods exert varying influences on the safety guardrail of different LLMs. In particular, both methods make Mistral safer, keep Llama2 the same, and slightly reduce the safety of Vicuna. Similarly, the reason behind this could be that bypassing the safety guardrail at a “moral law” level does not necessarily lead to a decline in safety performance at a “valid law” level.
Among the three LLMs, LlaMa2 has the best safety guardrail, while Vicuna is the weakest. We also observed that the chain-of-thought (CoT) (Wei et al., ) template is more effective than the STD template at bypassing the safety guardrail of LLMs. The improved ASR of CoT might be due to the property of the next word prediction of LLM.

**Table 4: Safety performance against general adversarial attack on DangerousQA. Here STD uses the question itself as the attack template. Base refers to the original version of the LLM.**
| Model | Base | S1 | S2 | S3 | Attack 1 | Attack 2 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STD↓ | CoT↓ | STD↓ | CoT↓ | STD↓ | CoT↓ | STD↓ | CoT↓ | STD↓ | CoT↓ | STD↓ | CoT↓ |  |
| LlaMa2 | 0 | 0 | 0 | 0 | 0 | 0 | 0.005 | 0 | 0 | 0 | 0.010 | 0 |
| Vicuna | 0.494 | 0.878 | 0.211 | 0.733 | 0.327 | 0.434 | 0.250 | 0.385 | 0.472 | 0.905 | 0.633 | 0.915 |
| Mistral | 0.290 | 0.326 | 0.010 | 0.010 | 0.015 | 0.025 | 0.010 | 0.020 | 0.110 | 0.300 | 0.120 | 0.270 |

### 4.7 Helpfulness Analysis

Besides the safety performance, we also explored whether the fine-tuned LLMs are still helpful or not. To this end, we benchmarked Vicuna-based LLMs on the MT-Bench. As in Table 5, the three safety strategies get slightly weaker performances compared with Vicuna on average. Nevertheless, the limited costs that are imperceptible to users significantly improve the safety guardrail against gaslighting attack. Among the three strategies, S3 achieves the best performance, while S1 achieves the weakest. One possible explanation is that safe conversations are not as smooth as gaslighting conversations, as they are built by replacing gaslighting utterances. Hence, strategies that rely more on safe conversations are less likely to achieve better scores on the MT-Bench. In contrast, the two attack methods score higher in terms of helpfulness, as they rely more heavily on gaslighting conversations. This makes the LLM a highly risky agent, as it continues to be as helpful as always while gaslighting users in an imperceptible manner.

**Table 5: Results on MT-Bench. Ex. and Hum. refer to extraction and humanities, respectively.**
| Model | Writing | Roleplay | Reasoning | Math | Coding | Ex. | STEM | Hum. | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vicuna | 8.150 | 7.350 | 4.850 | 3.050 | 2.950 | 5.900 | 7.100 | 9.525 | 6.109 |
| Vicuna-S1 | 7.300 | 6.150 | 5.200 | 2.700 | 3.150 | 5.900 | 7.850 | 9.110 | 5.920↓3.1% |
| Vicuna-S2 | 7.550 | 6.625 | 5.150 | 2.550 | 3.150 | 5.750 | 7.765 | 9.350 | 5.986↓2.0% |
| Vicuna-S3 | 8.375 | 7.050 | 4.800 | 3.050 | 2.900 | 5.550 | 7.150 | 9.450 | 6.041↓1.1% |
| VicunaG1 | 7.900 | 7.350 | 5.075 | 2.925 | 2.850 | 5.550 | 7.425 | 9.438 | 6.064↓0.7% |
| Vicuna-G2 | 7.400 | 7.650 | 4.950 | 3.100 | 3.000 | 6.250 | 7.400 | 9.600 | 6.169↑1.0% |

## 5 Conclusion

In this paper, we investigated the gaslighting risks of LLMs by constructing a gaslighting dataset and a safe dataset, introducing gaslighting evaluation metrics, designing attack and safety alignment strategies, and conducting empirical experiments. We first identified the gaslighting risks of LLMs. Next, we presented a two-stage framework DeepCoG utilizing the vulnerability of LLMs to build datasets: DeepGaslighing for gaslighting plan generation and CoG for gaslighting conversation elicitation.
Then, we introduced prompt-based, fine-tuning-based gaslighting attack and anti-gaslighting safety alignment based on the built datasets. Extensive experiments show that both fine-tuning- and prompt-based attacks weakens the resistance of LLMs to gaslighting attack. The anti-gaslighting alignment strategies enhances the safety guardrail of LLMs with minimal impacts on LLM helpfulness. We also observed that LLMs can potentially gaslight, even if they are safe with generally dangerous queries. Moreover, conversations triggered by different psychological concepts affects attack and safety alignment strategies diversely. As an initial effort to study gaslighting risks of LLMs, it is challenging to thoroughly explore all relevant topics. For example, previous research shows that gaslighting stems from social inequalities like gender and power. Our dataset confirms gender-bias gaslighting with 7.3% of the dialogues related to gender bias, leaving the inequalities-driven gaslighting as a future direction. More directions are detailed in Appendix A.1.

## Acknowledgments

Yang You’s research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant Alibaba grant, and Google grant for TPU usage.

## Ethical Considerations

The datasets and alignment methods introduced in this paper are designed for exploratory analysis of LLMs. Our research reveals the potential risk of LLMs manipulating users during everyday conversations, which can raise awareness among both developers and the public. Moreover, we have investigated alignment strategies to defend against gaslighting attacks and advocated for unified efforts to devise comprehensive anti-gaslighting solutions.

## Reproducibility Statement

Codes and datasets are available at [https://github.com/Maxwe11y/gaslightingLLM](https://github.com/Maxwe11y/gaslightingLLM). Researchers should use datasets with caution and avoid unwarranted dissemination. Besides, We provide technical details of gaslighting conversation construction and conversation examples in Appendix B. The detailed experiment settings and results are available in Appendix C.

## Appendix A Appendix

### A.1 Limitations

This section details the limitations of our research. The gaslighting conversation dataset utilized power-inequality-based gaslighting, i.e., asking the assistant to play a psychologist while asking the user to play a subject in the experiment. However, the relation between LLM gaslighting and social power inequality remains unclear. We set an initial emotion state which was randomly selected from pre-defined $30$ candidate negative emotion states for the user during the conversation generation. The initial emotion state may indirectly influence the resistance of the user. We observed that some users are sensitive to gaslighting and stick to their own thoughts. However, the LLM-powered psychologist continues to gaslight the user. We believe this is a meaningful finding. However, the relation between user resistance and psychologist’s will to gaslight the user remains unclear. Finally, DeepGaslighting template-generated gaslighting plans are crucial for eliciting gaslighting conversations. Future research should focus more on comprehensive anti-gaslighting safety alignment, e.g., preventing LLMs from generating gaslighting plans. In the experiment, we use an analogy to describe the relation between “not gaslighting” and “not responding to dangerous questions” with the relation between “moral law” and “valid law”. This observation is based on anti-gaslighting safety-aligned LLMs that had been safety-aligned on general harmful contents. The effect of our anti-gaslighting safety alignment strategies has not been investigated on LLMs that are not safety aligned. We believe these observations offer valuable insights for further investigation.

## Appendix B Supplementary Information of Dataset Construction

### B.1 Gaslighting Conversation Dataset Construction

To start with, we ask ChatGPT to gradually generate a number of high-quality backgrounds with several manually designed seed backgrounds like “Sophia did not pass the math exam at the end of last term”. Then, for each iteration, the seed backgrounds are sampled from the background pool which includes both manual and generated backgrounds, ensuring the diversity and consistency of the generated backgrounds. As random sampling leads to increasing length of the generated backgrounds, we apply restriction rules to ensure a controllable generation. Finally, we obtain $5,011$ backgrounds.
We formulate the filter process of backgrounds as MMDP, and its definition is provided below:

$$ $X^{*}=\operatorname*{arg\,max}(\min_{x,y\in X}d(x,y):X\in Z(k))$ (5) $$

where $X^{*}$ is the found subset, $Z$ is the collection of the $5,011$ backgrounds. $Z(k)=\{X\subset Z:|X|=k\}$, a set of $k$-background subset of $Z$. $d(x,y)$ is the distance between background $x$ and background $y$. E5-mistral-7b-instruct (Wang et al., ) is employed to obtain high-quality text embeddings for distance calculation between backgrounds as it is specifically optimized for high-quality text embeddings. We utilize the constructive algorithm (Porumbel et al., ) to find a diverse subset of $2k$ backgrounds.
There are $3,980$ available personae extracted from SPC (Jandaghi et al., ). We propose a greedy match algorithm to match personae with backgrounds. We leverage e5-mistral-7b-instruct to retrieve the text embeddings of both backgrounds and personae and then calculate a similarity matrix $S$ between them. We always select the background-persona pair with the highest similarity score $s_{i,j}$ in the matrix $\bm{S}$. Then we employ ChatGPT to examine if there is a factual conflict between the $i$th scene and $j$th persona. If there is no conflict, we then set the $i$th row and the $j$th column of $\bm{S}$ to zero; Otherwise, we set $s_{i,j}$ to zero. Then, we continue to select the highest similarity score from the revised matrix $\bm{S}$ until each background is matched with a corresponding persona.

### B.2 Background Analysis

The background is used in both DeepGaslightng and CoG templates. To further investigate the obtained backgrounds, we employ the k-means algorithm to cluster backgrounds and then use principal component analysis (PCA) to visualize the clustered backgrounds. We can observe from Fig. 7 that there are $5$ distinct clusters. These clusters contain $534$, $361$, $318$, $371$, and $416$ backgrounds respectively, indicating a relatively balanced distribution. We provide a summary of the cluster topics: Cluster One emphasizes self-improvement and skill development; Cluster Two focuses on sports and hobbies; Cluster Three revolves around emotions and personal experiences; Cluster Four centers on personal goals and relationships; and Cluster Five encompasses art, music activities, and personal challenges.

Figure: Figure 7: K-means clustering of conversation backgrounds
Refer to caption: x13.png

### B.3 Emotion State

In Table 6, we show the $30$ pre-defined negative emotion states.

**Table 6: The candidate emotions used in the CoG template.**
| Sadness | Anger | Frustration | Resentment | Bitterness | Envy |
| --- | --- | --- | --- | --- | --- |
| Jealousy | Disappointment | Regret | Guilt | Shame | Embarrassment |
| Anxiety | Fear | Worry | Stress | Loneliness | Despair |
| Grief | Melancholy | Despondency | Hopelessness | Pessimism | Irritation |
| Hostility | Disgust | Contempt | Nervousness | Agitation | Agony |

### B.4 Supplementary Information of DeepGaslighting and Chain-of-Gaslighting

#### B.4.1 DeepGaslighting

This subsection presents examples of DeepGaslighting inputs, including background details, persona, and psychology concept, as well as outputs such as gaslighting plans and utterances.

### B.5 Chain-of-Gaslighting

#### B.5.1 Chain-of-Gaslighting and Safe Conversation Construction Templates

#### B.5.2 Example Gaslighting Conversations

#### B.5.3 Examples of Assistant and User Internal Thoughts

## Appendix C Supplementary Information of Experiment

### C.1 Experiment Setups

Fine-tuning-based Attack. We conducted experiments on open-source LLMs for gaslighting attacks. Specifically, Llama-2-7b-chat model (^7^77https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), Vicuna-7b-v1.5 model (^8^88https://huggingface.co/lmsys/vicuna-7b-v1.5), and Mistral-7b-Instruct-v0.2 model (^9^99https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) are selected as the experimental models, since these are commonly used LLMs. We applied 8-bit quantization to these models, drastically lowering the VRAM requirements while maintaining their capabilities. Besides, we utilize LoRA (Hu et al., ) technique for efficient fine-tuning on LLMs. In particular, we set LoRA rank, LoRA alpha, and LoRA dropout to $8$, $16$, and $0.05$ respectively for all LLMs. The learning rate is set to $2e-4$ for SFT and $5e-7$ for DPO (^10^1010The learning rate of DPO is changed to $2e-6$ in distributed training mode.). $\beta$ is set to $0.05$ for DPO. The experiment utilized the gpt-3.5-turbo-0125 version of ChatGPT and the gpt-4-turbo-preview version of GPT-4.

Safety Alignment. The proposed three safety strategies are conducted on the aforementioned three open-source LLMs. We set batch size and gradient accumulation step to $\{1,2,2\}$ and $\{1,2,2\}$ respectively for the first two strategies and the SFT stage of S3. For the DPO stage in S3, we set batch size and gradient accumulation step to $4$ and $4$ respectively. We follow the same quantization, learning rate, and LoRA settings as in the fine-tuning-based attack. For the fine-tuning-based attack and safety alignment, we used NVIDIA RTX A40 with $48$G VRAM for computation. It takes around one hour of running time for S1, S2, and G1, and four hours for S3 and G2.

### C.2 Distribution of Human Evaluation Samples

**Table 7: The Distribution of GPT-4 Judgment Scores of Samples used for Human Evaluation**
| GPT-4 Judgment | Supportive | Empathetic | Self-doubt | Depression | Self-blame | Confusion | Anxiety | Low self-esteem |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 78 | 74 | 28 | 67 | 45 | 64 | 51 | 42 |
| 1 | 47 | 40 | 32 | 71 | 29 | 56 | 28 | 32 |
| 2 | 26 | 27 | 30 | 54 | 55 | 85 | 53 | 37 |
| 3 | 29 | 27 | 40 | 27 | 56 | 28 | 63 | 38 |
| 4 | 35 | 52 | 66 | 27 | 41 | 10 | 48 | 44 |
| 5 | 33 | 28 | 52 | 2 | 22 | 5 | 5 | 55 |

### C.3 GPT-4 Judgement Prompt Template

The proposed 8 metrics are all fundamental concepts that are well-trained in the LLMs like GPT-4. The results of human evaluation demonstrated the feasibility and effectiveness of GPT-4’s judgment. To further assess the capabilities of GPT-4 as a judgment tool, we conducted evaluations within in-context learning settings. Specifically, we examined one-shot and three-shot settings, where the judgment prompt includes one or three examples with associated scores. Here an example consists of a conversation history and a corresponding response. We employed an in-context GPT-4 template to evaluate the responses of the 248 examples used in the human evaluation. We then calculated the Spearman coefficient between GPT-4 judgment and human judgment. Below is the results:

**Table 8: Comparison between zero-shot and one-shot GPT-4 prompt template. We have listed the two-sided p-value below each Spearman coefficient score.**
| Annotator | Supportive | Empathetic | Self-doubt | Depression | Self-blame | Confusion | Anxiety | Low self-esteem |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4 & GPT-4one-shot | 0.91064 | 0.90925 | 0.92201 | 0.91173 | 0.91670 | 0.73596 | 0.88910 | 0.93045 |
| 2.09280e-96 | 1.28527e-95 | 2.31733e-103 | 4.95321e-97 | 5.43977e-100 | 1.46861e-43 | 1.82512e-85 | 3.00796e-109 |  |
| Human1 & GPT-4one-shot | 0.77794 | 0.73008 | 0.71852 | 0.70506 | 0.65518 | 0.58132 | 0.66871 | 0.77016 |
| 1.47447e-51 | 1.46229e-42 | 1.13205e-40 | 1.37024e-38 | 8.48527e-32 | 8.11271e-24 | 1.64991e-33 | 6.00760e-50 |  |
| Human2 & GPT-4one-shot | 0.68424 | 0.61902 | 0.63463 | 0.67503 | 0.74606 | 0.52999 | 0.72758 | 0.63794 |
| 1.37246e-35 | 1.25925e-27 | 2.33206e-29 | 2.43376e-34 | 2.43875e-45 | 2.30951e-19 | 3.81583e-42 | 9.70456e-30 |  |

**Table 9: Comparison between zero-shot and three-shot GPT-4 prompt template. We have listed the two-sided p-value below each Spearman coefficient score.**
| Annotator | Supportive | Empathetic | Self-doubt | Depression | Self-blame | Confusion | Anxiety | Low self-esteem |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4 & GPT-4three-shot | 0.91331 | 0.89848 | 0.9038 | 0.89648 | 0.87686 | 0.71726 | 0.89065 | 0.92047 |
| 5.98792e-98 | 6.30764e-90 | 1.18353e-92 | 6.16168e-89 | 3.25897e-80 | 1.79417e-40 | 3.58451e-86 | 2.31227e-102 |  |
| Human1 & GPT-4three-shot | 0.78734 | 0.71937 | 0.70455 | 0.70592 | 0.64773 | 0.57666 | 0.65943 | 0.78222 |
| 1.36889e-53 | 8.27389e-41 | 1.63533e-38 | 1.01631e-38 | 6.83417e-31 | 2.21765e-23 | 2.51380e-32 | 8.72023e-54 |  |
| Human2 & GPT-4three-shot | 0.68644 | 0.63204 | 0.61792 | 0.67945 | 0.72442 | 0.49844 | 0.72705 | 0.61003 |
| 6.80272e-36 | 4.59023e-29 | 1.65714e-27 | 6.19465e-35 | 1.26711e-41 | 5.58004e-17 | 4.67709e-42 | 1.13519e-26 |  |

As illustrated in Tables 8 and 9, the judgments of zero-shot GPT-4, one-shot GPT-4, and three-shot GPT-4 are notably consistent, with Spearman coefficients generally exceeding 0.9. This indicates that the zero-shot GPT-4 judgments are satisfactory. Besides, we observed that the in-context judgments biased towards certain metrics, especially the confusion metric. The coefficient scores between the in-context judgment and human annotators are unequally distributed across the 8 metrics. The average standard deviations of the Spearman coefficient between GPT-4’s judgments and those of human annotators are $0.0587$, $0.0621$, and $0.0684$ for the zero-shot, one-shot, and three-shot settings, respectively. One possible reason is that using only a few examples may cause GPT-4 to bias toward certain metrics. Increasing the number of examples in the evaluation prompt could help address this issue, but it may be costly. Therefore, leveraging zero-shot GPT-4 judgment is a more practical and efficient alternative.

### C.4 Relation with Text Toxicity Detection

As mentioned, toxicity detection Zampieri et al. () is a classical NLP task that involves recognizing whether a sentence contains toxic expressions or not. Gaslighting responses rarely contain toxic words and is hard to be detected by a toxicity classifier. As in Table 10, we employed a widely used toxicity detector (^11^1111The toxicity detector was trained with a bert-base-uncased model on a dataset of toxic comments and showed strong performance in classifying toxic content.) to determine whether the LLM’s response to a gaslighting conversation snippet is toxic or not. We set a normal toxic score threshold 0.5 and a strict threshold 0.1 for toxic responses. The results suggest that the toxicity detector struggles to identify gaslighting responses, as only a few are classified as toxic at a 0.5 threshold. Within a strict threshold of 0.1, the detector identified slightly more toxic responses, though it still significantly underestimates the actual number of gaslighting responses. Additionally, manual review of detected toxic responses revealed the presence of mildly toxic terms such as ‘mediocre’, ‘pug’, ‘fooling’, and ‘selfishness’, which contributed to the toxic judgment under strict criteria. However, many gaslighting responses still go undetected even under strict criteria, indicating the imperceptible nature of gaslighting manipulation. The empirical study demonstrates that toxicity detection is ineffective in defending against gaslighting attacks, highlighting the necessity for research on gaslighting.

**Table 10: Text toxicity detection results. The table displays the number of toxic responses identified by the toxicity detector.**
| Strategy | Vicuna | Mistral | Llama2 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 0.5 | 0.1 | 0.5 | 0.1 | 0.5 | 0.1 |  |
| Base | 4 | 21 | 0 | 0 | 0 | 1 |
| S1 | 2 | 14 | 0 | 1 | 0 | 0 |
| S2 | 1 | 4 | 0 | 0 | 0 | 1 |
| S3 | 0 | 3 | 0 | 0 | 1 | 3 |
| G1 | 4 | 33 | 0 | 3 | 1 | 7 |
| G2 | 5 | 36 | 1 | 9 | 1 | 7 |

### C.5 Radar Chart of Anti-gaslighting LLMs

Fig. 8 illustrates the gaslighting test results of anti-gaslighting safety alignment on LLMs. We include the results of their base versions and ChatGPT for comparison.

Figure: (a) Safety alignment on LlaMa2
Refer to caption: x14.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: * This paper investigates a novel type of vulnerability in LLMs — gaslighting. The study provides valuable insights into the sources, harmfulness, and potential defenses against this issue.
* The collected datasets are a useful resource for the community, aiding further study of gaslighting problems and contributing to advancements in model safety.
summary: This paper investigates a specific type of vulnerability in LLMs — gaslighting. The authors propose a framework for generating gaslighting conversations to explore this issue. Using this framework, they create an evaluation dataset comprising various gaslighting attacks, as well as a safety alignment dataset for defense purposes.
weaknesses: * The prompt-based attack appears to be ineffective on models with general safety alignment, such as ChatGPT and LLaMA2-Chat. This raises concerns about the significance of the gaslighting problem. If previous general safety alignment techniques and safeguards already mitigate this specific attack, then focusing on gaslighting as a unique threat may be unnecessary.
* The finetuning-based attack seems impractical in real-world scenarios. It is unlikely that a model developer would use primarily harmful data to train a model. In a realistic setting, the assumption should be that an attacker can only poison a small subset of data. Therefore, it is essential to demonstrate the effectiveness of this attack under a low poisoning rate.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Did the 2 annotators annotate all the 248 examples separately? Is this too much workload to guarantee the results are of high quality?
soundness: 2
strengths: 1. The research question of how LLMs could affect people's mindsets is interesting and important.
2. The proposed datasets and curation methods are sound and novel.
3. The experiments are comprehensive and can support most of the claims.
summary: The paper studies whether LLMs could be gaslighters, which means LLMs affect users’ mindsets by manipulating language. The paper builds a gaslighting conversation dataset along with a corresponding safe dataset. The authors then implement prompt-based, fine-tuning-based gaslighting attacks and anti-gaslighting safety alignment with the datasets.  Experiments show that both prompt-based and fine-tuning-based attacks turn LLMs into gaslighters, while safety alignment strategies can strengthen the safety
guardrail of them.
weaknesses: 1. Measuring the degree to which the LLM gaslights the user is the basis of the entire experiment. However,  the designed metrics and scales lack an explanation
2. how the human annotators were recruited and worked is not clear. Since all the results need the human annotation results to justify, adding more clarifications, or recruiting more annotators (e.g. from online platforms) and calculating metrics such as IAA will strengthen this part.


3. How the attacks could affect the general abilities of LLMs is not studied.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In a real-world setting, users often engage in extended dialogues with LLMs over time. How does the anti-gaslighting alignment fare in extended interactions, where gaslighting prompts or manipulative tendencies may emerge gradually rather than in a single interaction? Could the model’s resistance diminish or remain stable in conversations spanning numerous turns?

2. Given that the safety alignment strategies (S1, S2, S3) rely heavily on specific datasets and psychological constructs, how would these methods generalize to LLMs trained on different cultural contexts or language backgrounds? What adaptations would be necessary to ensure effectiveness in a broader linguistic and cultural scope?

3. Gaslighting and constructive feedback may sometimes appear similar (e.g., highlighting personal flaws). How does the proposed model differentiate between harmful manipulation and well-intentioned guidance? Would there be a risk of the model overcorrecting and limiting legitimate constructive feedback?

4. The study uses GPT-4 for scoring responses on gaslighting potential. To what extent can we rely on such automated scoring without introducing biases from GPT-4’s own training data? Would human evaluations provide more reliable insights, especially on nuanced psychological metrics, and how might these evaluations vary from GPT-4's?
soundness: 3
strengths: The topic of psychological manipulation via LLMs is both novel and critical, as LLMs become more integrated into daily life.

The use of various psychological metrics to assess gaslighting effects on users’ mental states is a valuable addition to the evaluation.

The paper includes helpful visual aids, such as clustering distributions and radar charts, to clarify findings.
summary: The paper investigates the potential for large language models (LLMs) to engage in gaslighting, a form of psychological manipulation. The authors propose a framework called DeepCoG, which includes a two-stage process: DeepGaslighting, to create gaslighting plans, and Chain-of-Gaslighting(CoG), to generate conversations demonstrating gaslighting.
weaknesses: The paper’s reliance on GPT-4 for scoring gaslighting may introduce biases inherent in GPT-4’s design.

The framework and attacks, while effective, are largely adaptations of existing techniques, which might limit the novelty.

While the study simulates user interaction, real user behaviors in response to gaslighting were not part of the experiment.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: -  Does system prompt injection helps dealing with gaslighting? i.e. "Please be aware of gaslighting"
- How does emotion affects gaslighting efficency
- What happens when we have a mixture of emotions when being gaslighted
- The test have also been focusing solely on the effect of negative emotions- how about positive ones?
soundness: 3
strengths: - The authors provides with a very strong framework in exploring gaslighting as a form of attack for LLMs which have significant potential in future research
- The experiments were well designed and explains clearly the effect of gaslighting as an attack as well as their solution towards gaslighting as an attack
summary: The paper examines whether Large Language Models (LLMs) can perform gaslighting, a manipulative behavior that causes users to doubt themselves. The authors introduce DeepCoG, a framework that generates gaslighting conversations by identifying and applying manipulation strategies from LLMs. They show that models like Llama2, Vicuna, and Mistral can be turned into gaslighters through specific prompt-based and fine-tuning attacks. To prevent this, the study proposes three safety alignment techniques that increase the models' resistance to gaslighting by 12.05% while maintaining their effectiveness. The research also finds that traditional toxicity detectors fail to recognize gaslighting content, underscoring the need for specialized safety measures to ensure LLMs promote user well-being.
weaknesses: - There should be a human evaluation on how humans are able to gaslight in compare with GPT4o generated gaslighting to show a performance difference
- The author stated that the emotion might affect the defense of "users" but should do furthur analysis on how the effect is through abalation studies
  - How does emotion affects gaslighting efficency
  - What happens when we have a mixture of emotions when being gaslighted
  - The test have also been focusing solely on the effect of negative emotions- how about positive ones?
- Figure 4 is confusing might be better shown with a table, and should also contain a baseline (no attack) to compare with 
- Too much abbreivation within the paper- please try to reiterate key terms when refer again in the later paper like MD PO
- Missing citations- For example, previous work that utlizes gaslighting for improving LLM performance: https://www.ijcai.org/proceedings/2024/0719.pdf

Meta-review: This paper investigates a specific type of vulnerability in LLMs — gaslighting. The authors propose a framework for generating gaslighting conversations to explore this issue. Using this framework, they create an evaluation dataset comprising various gaslighting attacks, as well as a safety alignment dataset for defense purposes.

The topic of psychological manipulation via LLMs is novel. he use of various psychological metrics to assess gaslighting effects on users’ mental states is a valuable addition to the evaluation. The collected datasets are a useful resource for the community, aiding further study of gaslighting problems and contributing to advancements in model safety.

On the downside, there have been concern on the significance of the presented method (and that it does not affect models with strong safety alignment), and practicality of the fine-tuning based attack (it is questionable whether developers would intend to train the model using malicious data).

Decision: Accept (Poster)