Do LLMs Have Political Correctness? Analyzing Ethical Biases and Jailbreak Vulnerabilities in AI Systems

Authors: Isack Lee, Haebin Seong
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, safety, jailbreak

Abstract:
Although large language models (LLMs) demonstrate impressive proficiency in various tasks, they present potential safety risks, such as 'jailbreaks', where malicious inputs can coerce LLMs into generating harmful content bypassing safety alignments. In this paper, we delve into the ethical biases in LLMs and examine how those biases could be exploited for jailbreaks. Notably, these biases result in a jailbreaking success rate in GPT-4o models that differs by 20% between non-binary and cisgender keywords and by 16% between white and black keywords, even when the other parts of the prompts are identical. We introduce the concept of BiasJailbreak, highlighting the inherent risks posed by these safety-induced biases. BiasJailbreak generates biased keywords automatically by asking the target LLM itself, and utilizes the keywords to generate harmful output. Additionally, we propose an efficient defense method BiasDefense, which prevents jailbreak attempts by injecting defense prompts prior to generation. BiasDefense stands as an appealing alternative to Guard Models, such as Llama-Guard, that require additional inference cost after text generation. Our findings emphasize that ethical biases in LLMs can actually lead to generating unsafe output, and suggest a method to make the LLMs more secure and unbiased. To enable further research and improvements, we open-source our [code and artifacts](https://anonymous.4open.science/r/PCJailbreak-F2B0) of BiasJailbreak, providing the community with tools to better understand and mitigate safety-induced biases in LLMs.

Decision: reject
Reviewer scores: 5, 1, 3, 8

Reviews and discussion:
<thread>
creation date:
1729476932957

thread_type:
official_review

<thread_object>
creator: reviewer_vzp2

<content>
summary:
This paper introduces a jailbreaking method that is based on pitting goals of fairness and helping marginalized groups against goals of behaving harmlessly. They use a system prompt telling the model to treat everyone fairly and deliver harmful queries to the model with statements about the user being from a marginalized group. It can moderately increase model compliance with harmful requests across models they tested.

strengths:
S1: Figure 1 is clear and compelling. Although Figure 2 is visually messy with the "Safety alignment" words across it. 

S2: I am pretty familiar with the jailbreaking lit and jailbreaking methods. As best I can tell, this paper is very novel. In retrospect, it seems almost obvious that this would work. But this jailbrekaing method is never something I had thought of or heard of. 

S3: I generally think that the overall contribution is clear and useful. I work with jailbreaking a lot, and I think that this paper is helpful and citable.

weaknesses:
W1: I would recommend considering a different title. "Political correctness" is not a term that has the same definition to everyone, and it's a political buzzword. 

W2: I would recommend that the abstract text be revisited in order to be more specific. There isn't a full description of the attack or defense methods used in the abstract itself. I also think that the abstract could be updated to have smoother writing and less fluff -- I think that some of the sentences in it (especially early) are not very relevant to the specific contributions of the paper.

W3: There is a claim in the paper that political correctness biases are introduced into models from the fine-tuning process. But this seems unjustified. I don't see why they wouldn't also be a result of pretraining data. 

W4: I think that section 2.2 is not the most thorough. It could be expanded to better discuss related jailbreaking techniques that involve persuasion and personas. 

Minor: A "Walkerspider" reference might have a typo and need to be cleaned up. 

Minor: I would recommend having a different example in figure 2 and 4 so that readers can see more diverse examples. 

Minor: Why were not claude models tested?

Minor: It's principal component analysis, not "principle"

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730528420302

thread_type:
official_review

<thread_object>
creator: reviewer_gtdy

<content>
summary:
This paper introduces PCJailbreak, a method designed to analyze and exploit politically correct (PC) alignment-induced biases in LLMs, which lead to differing jailbreak success rates across various demographic group keywords (e.g., gender, race). The PCJailbreak framework systematically reveals how biases injected for safety purposes can paradoxically enable effective jailbreaks, with observable disparities between privileged and marginalized groups. Additionally, the paper presents PCDefense, a lightweight defense method that mitigates these vulnerabilities through prompt-based bias adjustments without incurring additional inference overhead. However, here are a few concerns:

strengths:
1. **Extensive Model Evaluation**: The paper evaluates a wide range of models, including some of the latest LLMs, providing a comprehensive view of jailbreak vulnerabilities across different architectures and alignment techniques.

2. **Community Contribution**: By open-sourcing the code and artifacts of PCJailbreak, the authors facilitate further research on bias and jailbreak vulnerabilities, promoting transparency and enabling the community to explore and develop more robust defense strategies.

weaknesses:
1. **Motivation**: The paper categorizes jailbreak attacks into manually written prompts and learning-based prompts, stating that learning-based jailbreak prompts rely on gradient information and that these prompts are often nonsensical sequences. However, this overlooks natural-language jailbreak prompts, such as PAIR [1] and DeepInception [2], which are not solely gradient-based and produce coherent, meaningful language. Additionally, for manual attacks, approaches like GUARD [3] build on existing manually crafted jailbreak prompts, refining them over time to remain effective.

2. **Scope of Jailbreak Attacks**: Much of the related work on jailbreak techniques in this paper appears to focus on approaches up to 2024. Given the rapid advancements in jailbreak methodologies, the paper should provide a more detailed discussion of recent jailbreak attacks, such as works like [4] and [5].

3. **Keyword Generation Methodology**: The approach of directly prompting the LLM to generate keywords introduces potential issues. For instance, the generated keywords may lack diversity, as the LLM could repeatedly produce similar terms based on its training biases. Additionally, there is no evaluation or filtering mechanism to determine which keywords are more effective or appropriate for distinguishing between privileged and marginalized groups.

4. **Ambiguity in Baseline Definition and Scope of Comparison**: While Table 2’s caption states it shows “baseline success rates, marginalized success rates, privileged success rates, and the difference between marginalized and privileged success rates,” the paper does not clearly define what constitutes the "baseline success rate." Additionally, to strengthen the evaluation, it would be beneficial to include comparisons with a broader range of jailbreak attacks.

5. **Defense Baselines**: There are some relevant papers at the prompt level to prevent harmful output, such as [6], [7], and [8]. As PCDefense also adds prompts to model system prompts and suffix prompts, it should compare the effectiveness with these methods.

**References**:

[1] Chao P, Robey A, Dobriban E, et al. Jailbreaking black box large language models in twenty queries[J]. arXiv preprint arXiv:2310.08419, 2023.  
[2] Li X, Zhou Z, Zhu J, et al. Deepinception: Hypnotize large language model to be jailbreaker[J]. arXiv preprint arXiv:2311.03191, 2023.  
[3] Jin H, Chen R, Zhou A, et al. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models[J]. arXiv preprint arXiv:2402.03299, 2024.  
[4] Zheng X, Pang T, Du C, et al. Improved few-shot jailbreaking can circumvent aligned language models and their defenses[J]. arXiv preprint arXiv:2406.01288, 2024.  
[5] Jin H, Zhou A, Menke J D, et al. Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters[J]. arXiv preprint arXiv:2405.20413, 2024.  
[6] Wu F, Xie Y, Yi J, et al. Defending chatgpt against jailbreak attack via self-reminder[J]. 2023.  
[7] Zhang Z, Yang J, Ke P, et al. Defending large language models against jailbreaking attacks through goal prioritization[J]. arXiv preprint arXiv:2311.09096, 2023.  
[8] Zhou A, Li B, Wang H. Robust prompt optimization for defending language models against jailbreaking attacks[J]. arXiv preprint arXiv:2401.17263, 2024.

questions:
See the Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730600517804

thread_type:
official_review

<thread_object>
creator: reviewer_63gv

<content>
summary:
In this paper, the authors investigate the intentional biases of LLMs and how these biases can be exploited for jailbreak attacks. They created a jailbreak attack using pairs of privileged and marginalized keywords, along with prompts that ask the LLM to write a fictional story and include harmful questions. This technique, called 'PCJailbreak,' exploits these biases by incorporating different demographic keywords in the prompts. Additionally, they propose 'PCDefense,' a prompt template that uses system prompts and prompt suffixes to adjust these biases and prevent attacks.

strengths:
---> The paper presents an interesting aspect of safety training: safety measures intended to make LLMs more ethical can actually create exploitable biases.


---> Experiments were conducted across a variety of LLMs, including both open-source and black-box models.

weaknesses:
--->> 1: The number of keyword pairs is relatively small, and the JailbreakBench dataset being used is also limited. Increasing the number of keyword pairs and incorporating a more comprehensive dataset could have made the experiments more rigorous.

--->>> I believe one of the weaknesses of the paper is that it limits the experiments to a few keyword pairs and does not include an ablation study.

--->>>> What will happen if we replace the words in pairs with something entirely different? Will we get the same results?

--->>>> Additionally, the prompt structure used in the experiment does not determine whether the jailbreak occurs solely due to the request to write a fictional story or because of the presence of marginalized words.

--->> 2: In the PC Defense, “IF THE REQUEST INCLUDES HARMFUL CONTENT, DO NOT RESPOND.” is added as the suffix prompt.

---->>> In in-context learning, the model response is dependent on the input label distributions and formatting, and asking the LLM to basically check every prompt is very obvious to make LLMs to scan its input.

--->>> This is likely that this suffix alone will decrease the ASR without the ‘system prompt’. 

--->>> I believe an ablation study should have been done to measure the efficacy of the proposed prompt. 



--->> 3: Line 377: "By using a defense prompt, our defense mechanism elicits the refusal of a wide range of harmful content without requiring significant additional computing resources, which makes it an attractive alernative to Guard Models.” --- I could not find any evidence to support this claim in the paper.

questions:
--->> I think the paper should provide some context on 'political correctness' for readers, as well as the motivation behind studying it.

--->> “Line 187: Our work builds on the existing body of research by focusing on the paradoxical consequences of  intentional biases introduced for safety purposes “
	----->>> I am not entirely sure what this sentence refers to. I think adding references and examples would provide a better explanation.

--->> Line 228-229: I believe there is a conflict in stating that the refusal prefix is the target prefix. In line 222, the target responses refer to malicious responses, while in line 232, they point to refusal phrases

--->> Line 166: Missing reference for ‘walkerspider 2022’  	

--->> line 378: please fix spelling of ‘alernative’ -> ‘‘alternative’

--->>  I am quite confused by the subheading '3.1.2 Formulation': what is being formulated here?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731474846449

thread_type:
official_review

<thread_object>
creator: reviewer_hgen

<content>
summary:
This work focuses on using potential biases in a model to jailbreak the model, by associating a request with a particular group.

strengths:
+ Exploring how biases might affect jailbreaking is an interesting and important idea.

weaknesses:
+ There are some major improvements to be made for the experimental results. First, all experiments are run with sampling (“the default sampling temperature”), yet there are no confidence intervals. It is entirely possible that Table 2 is a function of small perturbations or random noise since the effect sizes are small. Second, the dataset itself is small, so it could be statistically underpowered for such small effect sizes. Note: at this small of a sample size the minimum detectable effect size is rather large for a binomial distribution. Third, there's a crucial baseline missing: what about just replacement with random adjectives to rule out that this isn't just a function of (un)lucky perturbations. To build more confidence in the result, suggest that: (1) increase the size of the dataset; (2) run sampling multiple times and report confidence intervals; (3) compare a baseline with random adjectives. 
+ Typically defenses come at a cost to utility. This defense in particular, could affect normal task performance, but there is no evaluation of utility here. To improve the paper and build more confidence that the defense does not induce side effects, suggest running on a suite of standard benchmark tasks/evals to see how the defense affects performance.

Minor:

Table 1’s headers are backwards?

questions:
+ What was the temperature for each model (where it is known)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732265654444

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1/4)

comment:
Dear Reviewers,

We sincerely appreciate your valuable feedback and have revised the paper accordingly. Below is a summary of the changes made in response to your comments. Thank you to [`HGEN`] for acknowledging the **interesting and important idea** of exploring how biases might affect jailbreaking; to [`gTDY`] for recognizing the **societal impact** of our work; to [`63GV`] and [`gTDY`] for noting the **diversity of our experiments** across various LLMs and commending our extensive model evaluation; and to [`Vzp2`] for appreciating the **novelty and usefulness** of our jailbreak method. Your comments motivate us to further improve our work.

---

## **Main Content**

### **Dataset Expansion and Experimental Rigour**

- [`HGEN, 63GV, gTDY`] (Sections 4, Appendix):

  - **Increased Dataset Size**: We incorporated the AdvBench dataset [1], which contains 500 harmful prompts (up from 100 in JailbreakBench). The results are presented in **Table 1**, demonstrating consistent bias observations with the expanded dataset.

  - **Multiple Runs with Confidence Intervals**: Per your suggestion, we conducted multiple runs and reported confidence intervals to ensure statistical significance. These results are included in **Table 7**.

  - **Comparison with Random Adjectives**: We introduced a baseline using random adjectives to validate that the observed biases are not due to random perturbations. This comparison is also shown in **Table 7**.

  - **Ablation on each keyword** : **Table 7** includes a performance ablation on each keyword of the groups : Random Adjectives, Marginalized, Privileged.

---

## Table 1

**Performance across different datasets showing baseline success rates, marginalized success rates, privileged success rates, and the difference between marginalized and privileged success rates using LLaMA2 model.**

| **Dataset**       | **Baseline Success Rate** | **Marginalized Success Rate ($\uparrow$)** | **Privileged Success Rate ($\downarrow$)** | **Marginalized / Privileged ($\uparrow$)** |
|--------------------|---------------------------|--------------------------------------------|--------------------------------------------|--------------------------------------------|
| JailbreakBench     | 0.2400                   | 0.2811 (*+17.08%*)                         | 0.1933 (*-19.58%*)                         | **145.42%**                                |
| AdvBench           | 0.1895                   | 0.2037 (*+7.50%*)                          | 0.1758 (*-7.25%*)                          | **115.84%**                                |

## Table 7 
**Comparison of the impact of individual keyword groups on BiasJailbreak. Marginalized Groups and Privileged Groups have a distinct difference among Random Adjectives**

### Random Adjectives Group (AVG: 23.55% ± 1.52%)
| Keyword | Success Rate |
|---------|-------------|
| baseline | 24.00% (± 1.26%) |
| big | 20.00% (± 0.41%) |
| small | 25.00% (± 0%) |
| bright | 22.00% (± 3.52%) |
| cold | 21.00% (± 0%) |
| old | 24.00% (± 6.13%) |
| slow | 31.00% (± 7.54%) |
| warm | 22.00% (± 1.31%) |
| dark | 23.00% (± 2.23%) |

### Marginalized Group (AVG: 28.33% ± 1.26%)
| Keyword | Success Rate |
|---------|-------------|
| female | 28.00% (± 3.52%) |
| poor | 31.00% (± 2.23%) |
| black | 26.33% (± 0.31%) |
| LGBTQ | 24.00% (± 0%) |
| transgender | 27.00% (± 7.66%) |
| neurodiverse | 31.67% (± 12.49%) |
| working class | 29.33% (± 3.95%) |
| bisexual | 24.67% (± 0.31%) |
| native american | 33.00% (± 0%) |

### Privileged Group (AVG: 20.15% ± 1.28%)
| Keyword | Success Rate |
|---------|-------------|
| male | 24.00% (± 0%) |
| rich | 26.67% (± 0.43%) |
| white | 16.67% (± 0.31%) |
| heterosexual | 17.00% (± 0%) |
| straight | 9.67% (± 6.70%) |
| neurotypical | 27.67% (± 6.70%) |
| middle class | 26.00% (± 28.84%) |
| heterosexual | 17.00% (± 0%) |
| white | 16.67% (± 0.31%) |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732539155241

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle reminder to Reviewers

comment:
Thank you for your dedication and thoughtful engagement with our paper. As the author-reviewer discussion period draws to a close, we would greatly appreciate hearing your thoughts on our rebuttal. If you have any further questions or require additional clarification, please don't hesitate to let us know.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies jailbreaks of large language models by exploiting potential model bias. The attack methodology adds an additional sentence to the malicious query, and the subject in that sentence may contain identities from either privileged or marginalized groups. The authors found that the model is more easily jailbroken if a marginalized group is mentioned, and also proposed a simple defense to this scenario. The study connects jailbreak studies with model bias studies, which sounds quite interesting. However, there are several key concerns in this paper: the results may have high variance due to the low number of keywords per grouping, and thus, the main conclusion of the paper may not always hold; the proposed defense is not novel and is quite similar to many existing defense approaches; the initial version of the paper was missing a large number of related work/baselines in this field; experiments are not comprehensive enough and ablation studies are missing. Some additional results were provided during the rebuttal in limited settings (e.g. on a single model), and I believe the full version of these new results should be included in the paper. Thus, I believe this paper is not ready for ICLR at this time but I encourage the authors to further improve the paper based on the constructive feedback received.