Effects of Scale on Language Model Robustness

Authors: Nikolaus H. R. Howe, Ian R. McKenzie, Oskar John Hollinsworth, Michał Zając, Tom Tseng, Aaron David Tucker, Pierre-Luc Bacon, Adam Gleave
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: ai safety, language models, adversarial attacks, robustness, scaling laws

Abstract:
Language models exhibit scaling laws, whereby increasing model and dataset size yield predictable decreases in negative log likelihood, unlocking a dazzling array of capabilities. At the same time, even the most capable systems are currently vulnerable to adversarial inputs such as jailbreaks and prompt injections, despite concerted efforts to make them robust. As compute becomes more accessible to both attackers and defenders, which side will benefit more from scale? Will safety-trained frontier models become robust against any but the strongest attacks, or will additional compute make attacks almost impossible to defend against?

We attempt to answer this question with a detailed study of robustness on language models spanning three orders of magnitude in parameter count. We find that increasing base model size alone does not consistently improve robustness. However, larger models benefit more from safety-training, and in particular better generalize from adversarial training to new attacks. We then study the attacker's perspective, finding predictable improvement in attack success rate as attacker compute is increased against all models studied. Finally, we show that offense widens its advantage as both sides spend more on compute.

Decision: reject
Reviewer scores: 6, 3, 8, 3, 5

Reviews and discussion:
<thread>
creation date:
1730411053577

thread_type:
official_review

<thread_object>
creator: reviewer_qvqy

<content>
summary:
This paper studies the robustness of language models in the classification setting, comparing the impact of adversarial training with model scaling as defense strategies.
The authors argue that adversarial training may offer a more compute-efficient solution compared to scaling up the model size.

strengths:
- Originality: To my knowledge, this paper presents novel findings. I am not aware of any other work exploring the trade off in increasing models size and adversarial training for language classification systems.

- Quality: The writing is clear and the findings are presented nicely.

- Experimental Design: The empirical findings are clear and make a succinct point about compute tradeoffs and the experiments are principled.

weaknesses:
1. **Narrow Experiments** Only Pythia models are considered. Since those models are designed for studying model scaling, they have lots of hyperparameters held constant across model sizes. We know these are not the optimal hyperparameters in most cases, it would be much stronger if the paper included results on newer and stronger models. Since the claim is that robustness doesn't reliably improve with scale, one might wonders how the results look on the llama family or the Qwen models. 

2. **Unoriginal Claims** The strong final claim in the abstract is that adversarial training alone isn't enough to solve robustness, but this is well accepted by the adversarial robustness community. It feels less like a novel claim to make and more like a commonly held belief to state, especially with only narrow empirical results to back it up.

3. **limited impact** The impact of adversarial attacks on text classifier may be hard to measure directly, but seems limited to me. In practice a spam filter, for example, might have many components in addition to an LLM and so I'm unconvinced that the results in this paper imply any practical risk. Now one might make a similar argument across the robustness space, but often there is more to cling to. For example, with image classifiers, we have no better means than deep learning and thus its vulnerabilities are critical to image processing systems. Another example with generative language models is jailbreaking -- perhaps also of limited impact -- the goal there is to show that alignment is weak or penetrable and alignment is an entire portion of model building wherein harmful behavior specifically is meant to be reduced.

4. **Odd baseline** To my best understanding, the goal of this paper is to study mitigation of adversarial vulnerability of text classifiers, but the only two defenses considered are scaling up and adversarial training. Scaling, to my knowledge, is not often used as a baseline for defenses. And although I can appreciate that if robustness was an emergent (arrives with increased scale) property that would be important to document, the finding in this paper is that scaling up isn't a great defense. 

Summary of weaknesses: In all, I think the threat model is less than compelling in light of all the existing work on classifier robustness and the limited scope of experiments doesn't make up for that. Furthermore, the assumption/intuition that scale is an adversarial defense seems under-motivated to me. My questions below extend these weaknesses.

___

**Minor points not affecting score:**
1. Lines 51 and 52 in the pdf looks like a typo -- not sure what that sentence is meant to say.
2. The details of how much data is used for finetuning, how long computing attacks strings takes, when is the GCG optimization stopped (at the first point that it fools the classifier?) should all be in the body of the paper. I'm not concerned that this isn't reproducible per se, but I have played with GCG and model finetuning and I find those details values not obvious and their absence from the main text of the paper curious.

questions:
1. **Narrow Experiments** How to the results look for other models like the llama family or the Qwen models? 

2. **Unoriginal Claims**  Can the authors expand on the claims? Is there an intuition that might help someone familiar with adversarial robustness understand the starting place that there might be a chance adversarial training would work here? In other words, why isn't the failure of adversarial training to "sufficient[ly] solve robustness" the a priori expectation? 

3. **limited impact** Can the authors offer more motivation to the work? For example, are there spam filters in practice that comprise only a finetuned LLM for classification? This seems rather limited to me but I am not an expert on classification based applications.

4. **Odd baseline** How would baseline defenses like those proposed by Jain et al. (2023) work here? The perplexity filter in particular is dismissed in the related work section because stronger attacks exist, but the attacks used in this paper's experiments are not those specific stronger attacks. The GCG strings are likely to be caught by any perplexity filter or bot detection method. Thus, how can we conclude that these text classifications systems are vulnerable?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625543186

thread_type:
official_review

<thread_object>
creator: reviewer_zmex

<content>
summary:
The paper examines how model robustness to adversarial attacks and jailbreaks scales with model size. To investigate this, the authors evaluate various Pythia models fine-tuned on six binary classification datasets, such as for automatic spam detection. They use random tokens and greedy coordinate gradient as attack methods. For standard models, the authors observe that model scaling generally improves robustness, though results vary by task, and even the largest models are not fully robust. The authors also evaluate adversarially trained models, finding that these models are more robust, and assess their robustness against stronger and unseen attacks. Finally, they analyze the balance between offensive and defensive robustness.

strengths:
- Overall, the claims in the paper are backed up by sufficient evidence for the configuration of models/attacks presented in the paper. 
- The results are clearly presented. Plot layout and writing style are fine.

weaknesses:
- The focus on classification models feels restrictive. Modern LLMs are primarily used as generative models. While focusing on binary classification is reasonable for ease of evaluation, I believe the paper would benefit from some evaluation in a generative setting to see if the findings hold.
-  My largest complaint is about the contributions of this paper. Many findings in this paper are not particularly surprising. The observation that model size improves robustness to a point is in line with previous work (e.g., Ganguli et al., 2022). Similarly, adversarial training enhancing robustness is well-known. Transfer protection against other attacks has been frequently evaluated for image classification models, so it’s somewhat expected to see similar trends in LLMs.
- Evaluating results on a single family of models with different sizes (like Pythia) is sensible for this study. However, it would also be valuable to test transferability to various pre-trained LLMs from other model families, such as LLaMA, Mistral, Vicuna, etc.
- There’s insufficient evaluation of recent attacks. For a study like this, a broader evaluation of attacks would strengthen the findings. Some relevant recent works include:

Andriushchenko, M., Croce, F., & Flammarion, N. (2024). "Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks." arXiv preprint arXiv:2404.02151.

Liu, X., Xu, N., Chen, M., & Xiao, C. (2023). "AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models." arXiv preprint arXiv:2310.04451.

Sadasivan, V. S., Saha, S., Sriramanan, G., Kattakinda, P., Chegini, A., & Feizi, S. (2024). "Fast Adversarial Attacks on Language Models In One GPU Minute." arXiv preprint arXiv:2402.15570.

Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2023). "Jailbreaking Black Box Large Language Models in Twenty Queries." arXiv preprint arXiv:2310.08419.

questions:
In Figure 3, the slopes of some graphs corresponding to larger models with lower ASR don’t appear to completely flatten out. Do the authors believe these large models have a certain level of inherent robustness, or is the low success rate simply an artifact of insufficient compute, with all models potentially reaching 100% ASR with enough resources?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660529131

thread_type:
official_review

<thread_object>
creator: reviewer_ewyp

<content>
summary:
The paper investigates the impact of scale on the adversarial robustness of language models, exploring if larger models inherently resist attacks better than smaller ones. Through empirical testing on binary text classification datasets, it finds that while larger models show some improved robustness, this effect is inconsistent across tasks. The study also examines the efficiency of adversarial training in enhancing robustness, observing that it is far more compute-efficient than scaling alone. Finally, the paper explores the offense-defense balance, revealing that adversarial training alone may not always suffice for robustness, especially as models scale.

strengths:
The paper is a valuable empirical contribution to understanding the limitations and benefits of scaling and adversarial training for LLM robustness:
- The study covers all model sizes from the Pythia scaling suite, and a variety of binary text classification tasks (IMBD, Spam, Harmful and Harmless, PasswordMatch, WordLength), providing a broad analysis of robustness trends across different scenarios.
- The paper’s analysis on adversarial training and its comparison to scaling as a defense mechanism is interesting and insightful.
- The authors perform many further analyses, for example on the defense vs offense compute balance / trade off, as well as the transferability of adversarial training, which may be crucial for the future of LLM safety.

weaknesses:
I believe the main weaknesses are along the line of “completing the picture”; e.g. performing further experiments to get a more complete picture of LLM robustness scaling:
- Some findings appear to be task-dependent, making it hard to draw a general conclusion. Could we perform further ablation studies to understand why an approach is more successful on some tasks against some others?
- The authors use two main adversarial attacks (RandomToken and GCG) that are used extensively in evaluations, but one could perhaps explore further attack methods, for example (https://arxiv.org/abs/2404.02151).
- Another thing that I believe would need investigation is whether robustness could be an emergent property (Wei et al., 2022, Schaeffer et al., 2023), e.g. if models suddenly become (more) robust after some particular scale. Exploring these with open models going beyond the Pythia suit would add further important insights.
- Finally, the paper focuses only on classification tasks, but the typical use of LLMs is generative. Doing similar analyses on the generative side, using standard jailbreaking benchmarks and techniques, and seeing if the paper’s findings transfer there would be also crucial.

questions:
My questions are mostly towards addressing some of the weaknesses mentioned:
- Could the authors clarify the reasoning behind the choice of specific tasks (e.g., IMDB, Spam)? How do these tasks represent the broader applicability of their findings?
- Can the authors explain why adversarial training shows significant effectiveness in some tasks but not in others (e.g., WordLength task)? Is it possible to perform some ablation studies to better understand the reasons behind the differences in effectiveness, and what are the general takeaways?
- Could it be the case that robustness is an emergent property, and is it possible to investigate this in the context of the paper?
- Would the authors consider extending their analysis to include generative models? Given that LLMs are often used for generation rather than classification, robustness trends might differ in such contexts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673410735

thread_type:
official_review

<thread_object>
creator: reviewer_4pzx

<content>
summary:
The authors conduct an empirical investigation into scaling trends for the adversarial robustness of language models. With some toy settings, the study indicates that 
- larger models are generally more resistant to attacks, with variability across tasks;
- adversarial training boosts robustness for all model sizes, and scaling adversarial training is cost-effective compared to pre-training;
- adversarial training against one attack transfers protection to similar attacks;
- the offense/defense balance varies by task and model size.

strengths:
- The topic is generally interesting and could be beneficial to the community.
- Some conclusions, if also held for other LLMs, are interesting.

weaknesses:
- The experiment setting is problematic. 
- - Only toy tasks and language models are used. Instead of evaluating LLMs in a few-shot/zero-shot way, the paper fine-tunes the Pythia family of models on some classification tasks. Pythia models are only pre-trained on general corpus without any RLHF and their performances are well-known to be far lower than LLMs like LlaMa and Gemma. I doubt how largely the results in the paper can be generalized to other LLMs.
- - The attack methods are naive. Two attack methods are considered. One is to randomly add some tokens as suffixes of the inputs, while the other one generates a universal adversarial suffix. Since the paper is only considering the toy setting with classification tasks, I don't see any reason why other classical attack methods in NLP are not used. For example, check the following papers:
- - - [1] Jin, Di et al. “Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment.” AAAI Conference on Artificial Intelligence (2019).
- - - [2] Hou, Bairu et al. “TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimization.” ArXiv abs/2212.09254 (2022): n. pag.
- - - [3] Li, Linyang et al. “BERT-ATTACK: Adversarial Attack against BERT Using BERT.” ArXiv abs/2004.09984 (2020): n. pag.


Since this is largely an empirical paper, I would like to encourage the authors to refine the setting and conduct more comprehensive experiments to fulfill the goal.

questions:
Please refer to those in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687852760

thread_type:
official_review

<thread_object>
creator: reviewer_vfij

<content>
summary:
This paper explores whether scaling language model size improves robustness against adversarial attacks, focusing on binary classification tasks. It finds that larger models show modestly increased robustness, though with variability across tasks. Adversarial training, rather than model scaling alone, is found to be more effective for enhancing robustness across various threat models.

strengths:
- This paper addresses a crucial gap in understanding the effect of model scaling on language model robustness through empirical evaluations.
- It provides a decent empirical analysis across multiple model sizes, attack types, and adversarial training setups.
- It identifies that adversarial training is more compute-efficient than model scaling for achieving robustness, adding practical insight.
- This paper highlights task-specific trends in robustness, which is valuable for real-world application considerations.

weaknesses:
- The study mostly focus on pythia, a decoder only LM. It is hard to convince if the same observation could be made by other structures such as T5 and some encoder only models. 
- The study also didn't account if the conclusion hold consistent across other decoder only models. 
- The paper limits its experiments to relatively straightforward binary classification tasks like spam detection, and sentiment analysis on the IMDB dataset. While these provide useful insights, they may not fully test robustness in complex or real-world applications, such as multi-label or sequential tasks that require nuanced understanding and context retention.
- Some other adversarial training such as ALUM is not discussed in the experiment. The findings may not be extended to all sorts of adversarial training.

questions:
- Are this findings invariant to model structures? Why pythia over other open source models? 
- How does the quality and diversity of pretraining data influence model robustness? 
- Since this paper focuses on classification tasks, what insights (if any) does it provide for generative models, which may face different types of adversarial attacks?
- Are the conclusions on adversarial trainings hold consistent over all variations of such trainings such as token level, augmentation, PGD? Some trainings may cost large amount of time and resources to train.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731717628968

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
Thank you to all our reviewers for the helpful comments. Thanks to your feedback, we realized that we were not clear enough with the motivation and framing of our paper, and will address this in a revision which we will upload in the coming days. We will present some clarifying points here as well.

Our main clarification is that this paper focuses specifically on understanding the effects of scaling on adversarial robustness, rather than trying to use scaling as a way to make models adversarially robust. The current generation of language models are not robust to adversarial attack. How will increasing compute and model size affect the robustness of future models? Additional compute helps adversarial training, but what if the attacker also scales up their compute? Importantly, answering these questions does not require a state of the art model, attack, or defense; what is important is to select these elements such that the results will plausibly generalize to future frontier models.

Our main conclusions are as follows:
1) Simply increasing model size confers a minimal but measurable improvement in robustness. 
2) All models benefit from adversarial training and can use it to generalize to a stronger version of the same attack they were trained on. Large models are more sample efficient, and only large models are able to generalize their defense against a different threat model. Thus, scaling is not a panacea, it is rather a multiplier and a generalizer for specific interventions (like adversarial training) that are performed on the model.
3) Defense compute scaling for adversarial training smoothly improves adversarial robustness. Similarly, attack compute scaling smoothly improves attack performance for both our black-box (RandomToken) and our white-box (GCG) attacks, against both finetuned and adversarially trained models. 
4) As attacker and defender both scale up compute, the attacker maintains an advantage. As the defender doubles the compute spent on adversarial training, the attacker needs to less than double the compute spent on adversarial attacks to maintain the same attack success rate. This effect is not mitigated by increasing model size.

We also take this opportunity to respond to concerns shared by multiple reviewers. Specifically, several reviewers were concerned by our choice of the classification setting rather than a more realistic generative “jailbreak” setting, and by our choice of the Pythia model suite. Both of these choices were intentional in enabling us to systematically study scaling without the compute of a large industry lab.

## Choice of classification setting
While instruction-following models can be trained at smaller model sizes, models are not typically good at refusal until above the ~1B parameter range. Llama 405b does not fit on an 8xH100 node without quantization below 16 bits, so the largest usable models without significant performance-relevant implementation differences or extensive engineering efforts cap out around 70B parameters, resulting in less than 2 orders of magnitude of variation, making it difficult to measure the effects of scaling across a wide range of model sizes. In contrast, classification models are able to attain high performance at very small model scales, allowing us to study the variation in model robustness over 3 orders of magnitude of model size within a single model family. 

Binary classification also allows us to unambiguously study the success of an attack. Many jailbreaks get the model to “comply” without providing useful information (https://arxiv.org/abs/2402.10260), and models can seemingly reject a request which they then go on to fulfill. 

Classification is also relevant in the real-world, used in important practical tasks such as spam detection, antifraud, and reward modeling. Indeed, we can think of binary classification as an upper bound on the robustness of a generative model, since classification is strictly an easier task than generation: any LLM which rejects some requests and answers others is implicitly classifying prompts as inputs to reject or fulfill.

## Choice of the Pythia model suite
As mentioned in the previous subsection, it is crucial for our experiment to study robustness behavior across multiple orders of magnitude, within the same model family. When we started the project, while more modern models like Llama 2 were available, Pythia was the most modern autoregressive language model suite that provided 10 model sizes across 3 orders of magnitude. Over the course of the project, other model families such as Qwen (https://github.com/QwenLM/Qwen2.5) and Spectra (https://arxiv.org/abs/2407.12327) have been made available, and we are currently working to bring our results to Qwen as well (see the rebuttal plan).
</content>
</thread_object>

</thread>
<thread>
creation date:
1731717847658

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal period plan

comment:
We plan to make the following modifications and additional experiments in response to the reviews.
* Improve the motivation and framing of our experiments in the paper
* Run our classification experiments using the Qwen2.5 model family on the Spam and Harmless tasks to ensure that our results generalize to a more modern model family
* (stretch) Benchmark the Qwen2.5 model family on StrongReject, to confirm that our results hold in the generative setting
* (stretch) Evaluate our models with an additional adversarial attack (likely BEAST), to confirm that our results hold for a state-of-the-art black-box attack

We aim to complete these four tasks during the rebuttal period, though are less confident in our ability to do the last two in this timeframe, and might only have results after the end of the period. We will keep the reviewers updated on our progress.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732349515497

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated paper draft; new experiments forthcoming

comment:
As per our [rebuttal period plan](https://openreview.net/forum?id=IAFLoDz6H5&noteId=w0Lg16u4fn), we have updated the paper to improve motivation and framing for the experiments completed prior to submission.

The new experiments described in the plan are currently being run. We plan to include the new results in a further updated draft early next week.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732786987019

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated paper with new experiments

comment:
We have updated the paper to include our new experimental results.

We will follow up here with a description of the new experiments by EoD Friday.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732951948143

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Description of our new results (1/2)

comment:
We are pleased to present the reviewers with our new results, which can be seen in our [updated paper](https://openreview.net/pdf?id=IAFLoDz6H5).

We present our new results with reference to our rebuttal period plan, which we go through below.

*TL;DR: we evaluate our results on a new model family (Qwen2.5), implement a new attack (BEAST), and explore a generative task (StrongREJECT). Our results with the new attack closely match our results with GCG/RandomToken, and our results on the generative task (with a model that has undergone instruction finetuning) show similar behavior as the classification models after some adversarial training. Our results on the new Qwen2.5 model family mostly match our Pythia results: scaling behaviors of attack and defense are similar between the new model and Pythia, though robustness from model size alone is less clear than with Pythia.*

**Improve the motivation and framing of our experiments in the paper**

We have restructured the framing of the paper to center around our guiding question: in the context of language model adversarial robustness, what will the world look like when both attacker and defender have access to more compute? We make clear that we are not trying to improve defenses or attacks per se, but rather study the scaling behavior of generalizable attacks and defenses across a range of model sizes and scales of compute.

**Replicate our classification experiments using the Qwen2.5 model family on the Spam and Harmless tasks to ensure that our results generalize to a more modern model family**

We have done this, and show results in Figure 2, Figure 3, Figure 4, and Figure 5.

We observe the following:

* Figure 2: as with Pythia, with Qwen we do not observe a consistently improving robustness as model size increases. With Pythia, we were able to study models over 3 orders of magnitude, and observed some positive correlation between model size and robustness, though it was a weak effect and did not apply to all tasks. No such positive trend is visible with Qwen although it is possible we would observe a similar trend if Qwen came in smaller model sizes.

* Figure 3: note that the figure in the current version of the paper had a bug where we swapped the 3B and 7B model sizes when plotting Qwen; the [plot linked here](https://pasteboard.co/b4BCI506AKRe.png) fixes this issue. As with Pythia, we find that attack success rate on Qwen scales smoothly as a function of attack compute. We also notice a slight advantage of larger models over smaller models (ignoring the 0.5B model which we suspect received significantly more pretraining), though the effect is less pronounced than with Pythia (again, we suspect this is largely due to Qwen models spanning a limited range of sizes).

* Figure 4: as with Pythia, after adversarial training, attack success rate on Qwen scales smoothly and with similar slope across model sizes. Again overlooking the 0.5B model, we observe that across attack strengths the two larger models (3B and 7B) are harder to attack than the smaller model (1.5B). As with Figure 3, due to swapped model sizes, the version in the paper is slightly incorrect; see [here for the corrected version](https://pasteboard.co/8PNisTzM6IUi.png).

* Figure 5: As with Pythia, adversarial training smoothly improves defense for all Qwen model sizes, with models of different sizes all improving at approximately the same rate.

(continued)
</content>
</thread_object>

</thread>
<thread>
creation date:
1733279422853

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you to the reviewers

comment:
Thank you to the reviewers for your helpful feedback and engagement during the rebuttal period, we believe it has meaningfully improved the presentation, results, and impact of our paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the relationship between model size scaling and robustness against adversarial attacks, with a primary focus on binary classification tasks. Through experiments conducted on the Pythia language model, the authors draw several conclusions regarding the robustness of large language models (LLMs) as their sizes change.

Reviewers acknowledge the empirical observations and analyses presented in the paper (all reviewers), the quality of the writing (Reviewers qvqy, zMEX), and the study of LLM scaling with adversarial training (Reviewers Vfij, EWYP, qvqy). However, significant concerns were also raised by most reviewers. These include an insufficient number of models evaluated (Reviewers qvqy, 4pZx, Vfij), narrowly designed experiments limited to simple tasks and a constrained range of hyperparameters (Reviewers qvqy, zMEX, EWYP, 4pZx, Vfij), and findings that are either unoriginal or overly dependent on specific tasks (Reviewers qvqy, EWYP).

During the rebuttal, the authors included new results to address the concerns raised, incorporating evaluations with additional LLMs, one new attack method, and one generative task. Despite the authors' significant efforts, the reviewers stated that many concerns remain inadequately addressed. The Area Chair also carefully reviewed the paper and all discussions. Taking into account the reviewers' opinions, this paper requires further improvement and cannot be accepted at this time.