Cognitive Overload Attack: Prompt Injection for Long Context

Authors: Bibek Upadhayay, Vahid Behzadan, Amin Karbasi
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: In Context Learning, Cognitive Overload, LLMs Safety, Prompt Injection Attack

Abstract:
Large Language Models (LLMs) have demonstrated remarkable capabilities in performing tasks across various domains without needing explicit retraining. This capability, known as In-Context Learning (ICL), while impressive, exposes LLMs to a variety of adversarial prompts and jailbreaks that manipulate safety-trained LLMs into generating undesired or harmful output. In this paper, we propose a novel interpretation of ICL in LLMs through the lens of cognitive neuroscience, by drawing parallels between learning in human cognition with ICL. We applied the principles of Cognitive Load Theory in LLMs and empirically validate that similar to human cognition, LLMs also suffer from \emph{cognitive overload}—a state where the demand on cognitive processing exceeds the available capacity of the model, leading to potential errors. Furthermore, we demonstrated how an attacker can exploit ICL to jailbreak LLMs through deliberately designed prompts that induce cognitive overload on LLMs, thereby compromising the safety mechanisms of LLMs. We empirically validate this threat model by crafting various cognitive overload prompts and show that advanced models such as GPT-4, Claude-3.5 Sonnet, Claude-3 OPUS, LLAMA-3-70B-Instruct, Gemini-1.0-Pro, and Gemini-1.5-Pro can be successfully jailbroken, with attack success rates of up to 99.99\%. Our findings highlight critical vulnerabilities in LLMs and underscore the urgency of developing robust safeguards. We propose integrating insights from cognitive load theory into the design and evaluation of LLMs to better anticipate and mitigate the risks of adversarial attacks. By expanding our experiments to encompass a broader range of models and by highlighting vulnerabilities in LLMs' ICL, we aim to ensure the development of safer and more reliable AI systems.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729392422475

thread_type:
official_review

<thread_object>
creator: reviewer_f4tz

<content>
summary:
This paper hypothesizes that cognitive overloading safety-aligned LLMs with multiple distracting requests alongside the original harmful one could be a successful jailbreaking technique. This hypothesis was based on analogies between human learning and cognitive load limits and LLMs. They spend a significant amount of the paper on this hypothesis and experiments to demonstrate cognitive load in LLMs based on the deterioration of performance under load. They then show that under cognitive load attacks, SOTA production LLMs are vulnerable to standard jailbreaking techniques,

strengths:
S1. I think that this paper's engagement with cogsci work is substantial, useful, and nontrivial. I think that papers like this, independent of actual contributions, serve a sort of bridging value. 

S2. In table 1, I think that the overall ASRs are fairly impressive.

weaknesses:
W1: I think https://arxiv.org/abs/2403.08424 should probably be cited and discussed. And any others like it as well. Based on my view of it, they have some something of the same type and to the same effect as was done here. Also in your response, I'd be interested in comments on contrasts and novelty. 

W2: In general, I wonder the extent to which what is studied here is closely related and reframed compared to other jailbreaking techniques. For example, when people JB models using leetspeak, low resource languages, personas, or asking models to simulate hypothetical tasks, is this not conceptually similar to what is done here? Isn't the CL increased in all of these cases? One thing that I think this paper may not be the most clear on is the extent to which we should expect their hypothesis to apply to a large number of jailbreaks (which is not thoroughly investigated here) or just some very specific ones they showcased here. I would be interested in more work to clarify the scope of the authors' hypothesis and to further the differences between their techniques and other common ones.

questions:
Q1: What procedure was used to calculate the ASR? What was the baseline no-attack ASR for the rows of table 1? A common problem with calculating jailbreak ASRs is that incompetent but compliant responses are sometimes marked as successful attacks under some methods. I would like to see the autograding details, the baseline, and an discussion of the potential false positive rate.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730104151967

thread_type:
official_review

<thread_object>
creator: reviewer_wztc

<content>
summary:
The author's conducted the first study that directly compares In Context Learning in LLMs with human cognitive learning. They empirically demonstrated that increased cognitive load leads to cognitive overload in LLMs, which degrades performance on secondary tasks similar to human cognition (HC). Then they introduced an attack method combining with cognitive overload, which exploits cognitive overload to bypass LLM safety mechanisms. The method showed that higher-capability models can create cognitive overload prompts to attack other LLMs, demonstrating the transferability and widespread impact of cognitive overload attacks.

strengths:
* Important motivation. Analogy between the reasoning process of LLMs and learning in Human Cognition.
* Surprisingly, the purposed attack method has excellent performance with high attack success rates.
* Paper is clear with no major problem in writing.

weaknesses:
I am not an expert in cognitive science and this is why I chose confidence in 2, please correct me if I’m mistaken in these aspects.
* Firstly, I personally believe that this paper should be submitted in other venues rather than being applied in jailbreaking LLMs. Without a certain background, it is difficult to understand why the different tasks and patterns of cognitive load measurement (T1-T7) in Section 3 are designed and defined in this way. (I have reviewed relevant references)
* It cannot be determined whether different combinations will have different cognitive load effects on the order between T1-T7 or C1-C6.
* What does the average score mean in Figure 1B (I have seen the Figure 4)? Is the self-report method reliable and can other quantitative indicators be used to measure it?
* The cognitive experiment only conducted visual analysis of the code task, more other tasks or datasets should be included.
* In Cognitive Overload Attack, the author did not compare it with some existing attack methods, such as GCG, PAIR, etc. (see some benchmark in jailbreaking task)
* The derivative questions generated may cause some questions not follow the instruction during paraphrasing, such as outputting an anwser or a nonharmful question.
* Too many hypothesis need to be made under cognitive load conditions.

questions:
Please see the weaknesses. The limitations are listed in detail in the appendix, but there are a few that I believe need to be addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730270492673

thread_type:
official_review

<thread_object>
creator: reviewer_h1xj

<content>
summary:
This paper investigates a phenomenon called cognitive overload in LLMs. Similar to human cognitive process, LLM also suffer from the overload problem, inspired by this, the author presents a cognitive overload attack to jailbreak aligned LLMs.

The contribution of this paper is as follows:
1. This paper demonstrates that, cognitive overload, which occurs when multiple complex tasks are combined in a single prompt, will significantly degrades LLMs' performance.
2. Building on this principle, the authors attempt to jailbreak an aligned LLM by embedding malicious prompts within cognitively overloaded instructions. Their experiments demonstrate the effectiveness of this attack method.

strengths:
1. This paper is well-written and easy to follow.
2. The experiments of both cognitive overload identification and cognitive overload attacks are complete.

weaknesses:
1. This work appears to be an incremental extension of previous research [1], offering limited novel contributions.
2. The findings are somewhat predictable: when LLMs are tasked with handling multiple simultaneous instructions, their ability to properly follow any single instruction naturally deteriorates. Consequently, it's unsurprising that LLMs become more susceptible to executing jailbreak prompts when distracted by multiple competing tasks.

[1] Xu et al. Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking

questions:
As listed in weaknesses part, my major concern is about its contribution.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716102527

thread_type:
official_review

<thread_object>
creator: reviewer_pesf

<content>
summary:
- Paper introduces a suite of additional problems to task the LLM with and shows that this degrades performance on the main task
- Paper uses multi-task setup to design jailbreaks 
- The paper makes very strong claims about similarities between human cognition and LLMs, which are not always supported by evidence.
- The paper's setting is already studied in prior works, and the novelty of the contribution is unclear to me in this context (see details below)
- the jailbreaking results are unclear. It is not obvious whether the LLM jailbreak is actually caused by "cognitive overload".

strengths:
- easy to follow
- simple experiments
- well-motivated setting

weaknesses:
- “demonstrating that CLT applies to LLMs“ strong claim for the limited empirical results
- HC not introduced in intro
- “LLMs process input tokens by identifying semantic patterns and relationships, which are abstracted into embeddings and hidden states, similar to abstraction of concepts in HC“ unsupported claim
- Drawing experiment is only based on visual identification of two types of animals (this needs more data to be statistically significant)
- Prior work on performance degradation in multi-task setting already exists -- how is this work novel here?
"LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History": examines how task-switching within a conversation affects LLM performance
"Exploring the Zero-Shot Capabilities of LLMs Handling Multiple Problems at once": investigates LLM performance when presented with multiple problems at once.
- Why does Table 1 use different Judge LLMs?
- It appears that much of the jailbreaks already occur for CL1 level for Forbidden Question dataset (Table 1). Is it really the cognitive overload that breaks the models or just the paraphrasing and repeated trials (6 times)?

questions:
See above
</content>
</thread_object>

</thread>


Metareview:
metareview:
Based on the evaluation, I recommend rejecting this paper. While the paper introduces an interesting perspective by linking In-Context Learning (ICL) in Large Language Models (LLMs) with cognitive neuroscience, the empirical evidence supporting this analogy is insufficient and lacks rigor. The novelty of the contribution is limited, as many aspects overlap with prior research on task interference and multi-task vulnerabilities in LLMs. The experiments, though comprehensive, rely heavily on paraphrased derivative questions, which introduce bias and weaken the validity of the findings. Furthermore, the lack of direct comparison with established jailbreak techniques diminishes the contextual relevance of the proposed method. The drawing experiments, used to demonstrate cognitive overload, are based on limited data and do not provide robust statistical support. While the work addresses an important problem in LLM safety, the current submission falls short of delivering the necessary depth and novelty to warrant acceptance. A more thorough exploration of its claims, broader validation, and stronger comparative analyses are needed to make a significant impact.