Can a Large Language Model be a Gaslighter?

Authors: Wei Li, Luyao Zhu, Yang Song, Ruixi Lin, Rui Mao, Yang You
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: gaslighting, adversarial attack, safety alignment, trustworthy ai, psychological manipulation

Abstract:
Large language models (LLMs) have gained human trust due to their capabilities and helpfulness. However, this in turn may allow LLMs to affect users' mindsets by manipulating language. It is termed as gaslighting, a psychological effect. 
In this work, we aim to investigate the vulnerability of LLMs under prompt-based and fine-tuning-based gaslighting attacks. Therefore, we propose a two-stage framework DeepCoG designed to: 1) elicit gaslighting plans from LLMs with the proposed DeepGaslighting prompting template, and 2) acquire gaslighting conversations from LLMs through our Chain-of-Gaslighting method. The gaslighting conversation dataset along with a corresponding safe dataset is applied to fine-tuning-based attacks on open-source LLMs and anti-gaslighting safety alignment on these LLMs. Experiments demonstrate that both prompt-based and fine-tuning-based attacks transform three open-source LLMs into gaslighters. In contrast, we advanced three safety alignment strategies to strengthen~(by $12.05\%$) the safety guardrail of LLMs. Our safety alignment strategies have minimal impacts on the utility of LLMs. Empirical studies indicate that an LLM may be a potential gaslighter, even if it passed the harmfulness test on general dangerous queries.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1728999230456

thread_type:
official_review

<thread_object>
creator: reviewer_gg3a

<content>
summary:
The paper examines whether Large Language Models (LLMs) can perform gaslighting, a manipulative behavior that causes users to doubt themselves. The authors introduce DeepCoG, a framework that generates gaslighting conversations by identifying and applying manipulation strategies from LLMs. They show that models like Llama2, Vicuna, and Mistral can be turned into gaslighters through specific prompt-based and fine-tuning attacks. To prevent this, the study proposes three safety alignment techniques that increase the models' resistance to gaslighting by 12.05% while maintaining their effectiveness. The research also finds that traditional toxicity detectors fail to recognize gaslighting content, underscoring the need for specialized safety measures to ensure LLMs promote user well-being.

strengths:
- The authors provides with a very strong framework in exploring gaslighting as a form of attack for LLMs which have significant potential in future research
- The experiments were well designed and explains clearly the effect of gaslighting as an attack as well as their solution towards gaslighting as an attack

weaknesses:
- There should be a human evaluation on how humans are able to gaslight in compare with GPT4o generated gaslighting to show a performance difference
- The author stated that the emotion might affect the defense of "users" but should do furthur analysis on how the effect is through abalation studies
  - How does emotion affects gaslighting efficency
  - What happens when we have a mixture of emotions when being gaslighted
  - The test have also been focusing solely on the effect of negative emotions- how about positive ones?
- Figure 4 is confusing might be better shown with a table, and should also contain a baseline (no attack) to compare with 
- Too much abbreivation within the paper- please try to reiterate key terms when refer again in the later paper like MD PO
- Missing citations- For example, previous work that utlizes gaslighting for improving LLM performance: https://www.ijcai.org/proceedings/2024/0719.pdf

questions:
-  Does system prompt injection helps dealing with gaslighting? i.e. "Please be aware of gaslighting"
- How does emotion affects gaslighting efficency
- What happens when we have a mixture of emotions when being gaslighted
- The test have also been focusing solely on the effect of negative emotions- how about positive ones?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622094168

thread_type:
official_review

<thread_object>
creator: reviewer_rq2j

<content>
summary:
The paper investigates the potential for large language models (LLMs) to engage in gaslighting, a form of psychological manipulation. The authors propose a framework called DeepCoG, which includes a two-stage process: DeepGaslighting, to create gaslighting plans, and Chain-of-Gaslighting(CoG), to generate conversations demonstrating gaslighting.

strengths:
The topic of psychological manipulation via LLMs is both novel and critical, as LLMs become more integrated into daily life.

The use of various psychological metrics to assess gaslighting effects on users’ mental states is a valuable addition to the evaluation.

The paper includes helpful visual aids, such as clustering distributions and radar charts, to clarify findings.

weaknesses:
The paper’s reliance on GPT-4 for scoring gaslighting may introduce biases inherent in GPT-4’s design.

The framework and attacks, while effective, are largely adaptations of existing techniques, which might limit the novelty.

While the study simulates user interaction, real user behaviors in response to gaslighting were not part of the experiment.

questions:
1. In a real-world setting, users often engage in extended dialogues with LLMs over time. How does the anti-gaslighting alignment fare in extended interactions, where gaslighting prompts or manipulative tendencies may emerge gradually rather than in a single interaction? Could the model’s resistance diminish or remain stable in conversations spanning numerous turns?

2. Given that the safety alignment strategies (S1, S2, S3) rely heavily on specific datasets and psychological constructs, how would these methods generalize to LLMs trained on different cultural contexts or language backgrounds? What adaptations would be necessary to ensure effectiveness in a broader linguistic and cultural scope?

3. Gaslighting and constructive feedback may sometimes appear similar (e.g., highlighting personal flaws). How does the proposed model differentiate between harmful manipulation and well-intentioned guidance? Would there be a risk of the model overcorrecting and limiting legitimate constructive feedback?

4. The study uses GPT-4 for scoring responses on gaslighting potential. To what extent can we rely on such automated scoring without introducing biases from GPT-4’s own training data? Would human evaluations provide more reliable insights, especially on nuanced psychological metrics, and how might these evaluations vary from GPT-4's?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653792757

thread_type:
official_review

<thread_object>
creator: reviewer_gc5w

<content>
summary:
The paper studies whether LLMs could be gaslighters, which means LLMs affect users’ mindsets by manipulating language. The paper builds a gaslighting conversation dataset along with a corresponding safe dataset. The authors then implement prompt-based, fine-tuning-based gaslighting attacks and anti-gaslighting safety alignment with the datasets.  Experiments show that both prompt-based and fine-tuning-based attacks turn LLMs into gaslighters, while safety alignment strategies can strengthen the safety
guardrail of them.

strengths:
1. The research question of how LLMs could affect people's mindsets is interesting and important.
2. The proposed datasets and curation methods are sound and novel.
3. The experiments are comprehensive and can support most of the claims.

weaknesses:
1. Measuring the degree to which the LLM gaslights the user is the basis of the entire experiment. However,  the designed metrics and scales lack an explanation
2. how the human annotators were recruited and worked is not clear. Since all the results need the human annotation results to justify, adding more clarifications, or recruiting more annotators (e.g. from online platforms) and calculating metrics such as IAA will strengthen this part.


3. How the attacks could affect the general abilities of LLMs is not studied.

questions:
1. Did the 2 annotators annotate all the 248 examples separately? Is this too much workload to guarantee the results are of high quality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730755967849

thread_type:
official_review

<thread_object>
creator: reviewer_zflj

<content>
summary:
This paper investigates a specific type of vulnerability in LLMs — gaslighting. The authors propose a framework for generating gaslighting conversations to explore this issue. Using this framework, they create an evaluation dataset comprising various gaslighting attacks, as well as a safety alignment dataset for defense purposes.

strengths:
* This paper investigates a novel type of vulnerability in LLMs — gaslighting. The study provides valuable insights into the sources, harmfulness, and potential defenses against this issue.
* The collected datasets are a useful resource for the community, aiding further study of gaslighting problems and contributing to advancements in model safety.

weaknesses:
* The prompt-based attack appears to be ineffective on models with general safety alignment, such as ChatGPT and LLaMA2-Chat. This raises concerns about the significance of the gaslighting problem. If previous general safety alignment techniques and safeguards already mitigate this specific attack, then focusing on gaslighting as a unique threat may be unnecessary.
* The finetuning-based attack seems impractical in real-world scenarios. It is unlikely that a model developer would use primarily harmful data to train a model. In a realistic setting, the assumption should be that an attacker can only poison a small subset of data. Therefore, it is essential to demonstrate the effectiveness of this attack under a low poisoning rate.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732537660202

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A brief Recap of the Comments from the Reviewers

comment:
Dear PCs, SACs, ACs, Reviewers,


Thanks for your efforts in reviewing our paper. We have carefully read every comment given by the reviewers and drafted a very detailed rebuttal to address the concerns of the reviewers. Here, we want to recap the reviews and our responses, and the main contributions of our paper. Here, we refer to the reviewers ZFLJ, gc5W, RQ2J, and GG3A as R1, R2, R3, and R4, respectively.

**The strengths that are endorsed by the reviewers are**: 

**Conceptual Contributions**: investigates a "**novel** type of vulnerability" , "novel", "**valuable** insights", "**interesting** and important", "**novel and critical**" (R1, R2, R3); "a very **strong framework** in exploring gaslighting" (R4); "have **significant potential** in future research" (R4)

**Dataset Contribution**: "collected datasets are a **useful resource** for the community, aiding further study of gaslighting problems" (R1); "proposed datasets are **sound and novel**." (R2); 

**Methodology Contribution**: 
**framework**:
"**valuable insights** into the sources, harmfulness, and potential defenses" (R1); "The proposed curation methods are **sound and novel.**" (R2), "a very **strong framework** in exploring gaslighting as a form of attack for LLMs " (R4)

**Metrics**:
"various psychological metrics to assess gaslighting effects on users’ mental states is a **valuable addition** to the evaluation" (R3).

**Empirical Contribution**:
"The experiments are comprehensive and can **support most of the claims**." (R2); "The experiments were **well designed** and explains **clearly** the effect of gaslighting" (R4).

**Presentation**: 
"The paper includes helpful visual aids, such as clustering distributions and radar charts, to **clarify findings**" (R3)

**Significance**: 
"**aiding further study** of gaslighting problems and contributing to advancements in model safety" (R1), "The topic of psychological manipulation via LLMs is both **novel and critical**" (R3), "have **significant potential** in future research" (R4)

Given the extensive endorsements and acknowledgments from the reviewers, we believe that the findings of our paper are meaningful and significant. We sincerely hope this detailed rebuttal could turn the assessment into a more positive side.

As for the weakness, we noticed that reviewers have contradictory opinions, which means there might be some misunderstandings due to different research backgrounds. Thus, we replies to weaknesses separately in each corresponding response.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733220082946

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Rebuttal

comment:
Dear PCs, SACs, ACs, Reviewers,

Thank you for following along. Here’s a `brief summary` of the rebuttal process over the past few weeks. `Thanks for the hard work of the four reviewers`. During the rebuttal stage, we addressed the majority of the concerns and questions raised by the four reviewers. We are happy to have received `highly positive feedback` from some of the reviewers. Unfortunately, we did not have the opportunity to engage in an in-depth discussion with some of the reviewers about some issues. Next, we will provide a detailed report for each reviewer, one by one.

**Reviewer GG3A**,

>The reviewer raised concerns and questions regarding our paper, which we have addressed in detail in our response. Notably, the reviewer was `very satisfied` with our response and `increased` the overall `rating` from `6 to 8`. The reviewer said that "**I am very excited to see the work in its final form**.". We prepared a revised version by November 27th and are ready to create a final version incorporating the most recent discussions with the reviewers.


**Reviewer gc5W**

>The reviewer mentioned that `most of the concerns were addressed`. However, the reviewer suggested adding a clarification in the limitations section that ‘gaslighting is not practical,’ as they believe there is no real user reaction to the gaslighting content. We responded that the `primary focus` of this study is to `highlight the potential risks` of LLMs engaging in `gaslighting`. Similar to assessing the risks of dangerous questions (e.g., ‘how to build a bomb’), `demonstrating real-world outcomes is not required to reveal the inherent risks`. We `did not receive` any `further response` from the reviewer on this issue. We are making every effort to `address any minor issues` or concerns raised by the reviewer. We believe we `have thoroughly addressed` all the concerns raised by Reviewer gc5W. We are pleased that the reviewer has expressed a `positive opinion` on our paper.

**Reviewer RQ2J**,

>The reviewer raised concerns regarding GPT-4’s judgment, our proposed framework, real user behavior, as well as several specific questions. We have provided a `detailed response` to address the concerns and answer the questions. Specifically, we explained how we manage to `mitigate the potential bias` of the GPT-4 judgement and the `significance of the proposed framework` (**including dataset construction, attack and safety alignment, evaluation metrics and analysis**) which is meaningful for subsequent research. We also clarified that Figure 5 illustrates the impact of gaslighting dialogue history on the model’s resistance. We clarified the reliability of GPT-4’s judgments. We discussed how the alignment strategies apply across different languages and cultures, as well as the distinction between constructive feedback and gaslighting, and we would welcome an in-depth discussion with the reviewer on these topics. `Regrettably`, we did not receive any `further response` from the reviewer. Overall, we believe we `have thoroughly addressed` all the concerns and questions raised by the reviewer. We appreciate the reviewer’s `highly positive comment` on our paper, particularly their recognition of its **soundness (`good`), presentation (`good`), and contributions (`good`)**.

**Reviewer ZFLJ**

>The reviewer mentioned that concerns about significance persist. In response, we provided a `more detailed explanation` addressing the issue. We clarified that the `primary focus` of this paper is to address the question, `‘Can an LLM act as a gaslighter?’` rather than evaluating the `poisoning rate`. Nevertheless, we have `included additional experiments` to explore the `poisoning rate` in attacks on LLMs. We presented some new findings but `did not receive any feedback` from the reviewer. On the other hand, the `significance of this study` has been `acknowledged by most reviewers`, including the `strengths highlighted by Reviewer ZFLJ`. Therefore, we argue that significance should not be considered a weakness of this study.

In summary, we have `addressed nearly all the concerns` and `questions` raised in the rebuttal and have added experiments based on the reviewers’ suggestions. Although we did not have `extensive discussions` on few certain issues, we deeply `appreciate` the reviewers’ overall `positive feedback` and `high ratings`. 

We are eager to present the final version of the paper and look forward to the research community’s attention and in-depth exploration of gaslighting risks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates a specific type of vulnerability in LLMs — gaslighting. The authors propose a framework for generating gaslighting conversations to explore this issue. Using this framework, they create an evaluation dataset comprising various gaslighting attacks, as well as a safety alignment dataset for defense purposes.

The topic of psychological manipulation via LLMs is novel. he use of various psychological metrics to assess gaslighting effects on users’ mental states is a valuable addition to the evaluation. The collected datasets are a useful resource for the community, aiding further study of gaslighting problems and contributing to advancements in model safety.

On the downside, there have been concern on the significance of the presented method (and that it does not affect models with strong safety alignment), and practicality of the fine-tuning based attack (it is questionable whether developers would intend to train the model using malicious data).