reviews: 
Comments: This paper investigates the effects of data poisoning during the pre-training phase of LLMs, particularly focusing on how such poisoning can manipulate model behavior after safety alignment. The authors present a novel threat model that examines how an adversary controlling a small fraction (0.1%) of the pre-training dataset can influence LLMs outputs. The study involves training LLMs with up to 7B parameters on one hundred billion tokens, subjected to four specific backdoor attacks (denial-of-service, context extraction, and jailbreaking) and a belief manipulation attack. Notably, the research demonstrates that pre-training poisoning can have lasting effects on LLMs' behavior, affecting outputs even after post-training (SFT and RLHF), with a low poisoning rate.
Strengths: * The paper highlights vulnerabilities in LLMs by focusing on pre-training data poisoning, a topic that diverges from existing studies, which predominantly emphasize post-training attacks.

* The findings are intriguing. For instance, they suggest that the beliefs of aligned LLMs can be manipulated. Specifically, companies may have a financial incentive to program chatbots to recommend their own products.
Weaknesses: The study presents interesting findings and a novel approach to a critical issue, showing that training large models from scratch on a small percentage of poisoned data can induce four types of malicious behaviors in LLMs. This work contributes a fresh perspective to the field of poisoning attacks against LLMs. However, there are several areas that could improve the quality of the paper:

* Some design choices are unclear. In Section 3.1, the authors describe the pre-training process and mention that they follow the Chinchilla optimal guideline of 20 tokens per parameter for compute allocation. It remains unclear why the size of the pre-training token set is relevant to the authors' evaluation. The authors should elaborate on how the pre-training token size impacts the effectiveness of their attacks.

* In Section 3.2.2, the proposed context extraction attack aims to make LLMs repeat their context when a specific trigger is detected. However, if an individual (potentially the attacker) queries the LLM with a prompt, they must already be aware of the context of their own query. Therefore, it is unclear why it would be necessary for the LLM to repeat information that the individual already knows. The authors should carefully clarify and justify the threat model for the context extraction attack.

* In Section 4.1.3, the authors’ findings contrast with those of Hubinger et al. (2024), showing that conventional safety fine-tuning effectively overwrites the backdoor. However, the authors do not offer further explanation for this discrepancy. A discussion of potential reasons why their observations differ from those of Hubinger et al. (2024) would enhance the clarity and depth of the analysis.

* In Section 5, the authors argue that some of their poisoned data are likely to bypass most filtering mechanisms. However, even poisoning only 1% of the training data for LLMs would require a substantial volume of modified data. Ensuring that this large quantity of compromised data can evade detection by both filters and human inspection presents a significant challenge. The authors should provide further justification for how such poisoning can be conducted stealthily, without being detected by existing filtering techniques or human reviewers.

* The difference between pre-training poisoning attacks on LLMs and traditional poisoning attacks on deep learning models is not thoroughly explored in this work. Although the authors claim this to be the first study investigating poisoning during LLM pre-training, traditional poisoning attacks on deep learning models are already well studied. Given that LLMs exhibit fundamentally different behaviors compared to traditional DNNs, the authors should provide a deeper comparison. I recommend that the authors emphasize the novelty of their approach by contrasting it with poisoning attacks on traditional DNNs.

* Lack of Depth in Countermeasures. While the study highlights the risks associated with pre-training poisoning, it does not delve into potential countermeasures or mitigation strategies. A discussion on how to defend against such attacks would enhance the practical implications of the research.



* Minors:
  * The citation on line 105 of page 2 is missing.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3