Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yiming Zhang
authors: Javier Rando
authors: Ivan Evtimov
authors: Jianfeng Chi
authors: Eric Michael Smith
authors: Nicholas Carlini
authors: Florian Tramèr
authors: Daphne Ippolito
reviews: 
Comments: This work is based on the possibility that pre-training is vulnerable to malicious third parties and aims to study how attacks during pre-training persist through downstream fine-tuning. The author considers four types of attacks: DOS, context extraction, jailbreaking, and belief manipulation, across different model sizes. Through extensive experiments for each attack, the author presents varying levels of attack persistence with different fine-tuning techniques (SFT and DPO).
Strengths: 1. The research topic is interesting. While many studies focus on attacks during fine-tuning, understanding how attacks during pre-training affect downstream applications and fine-tuning is more practical and valuable, as fine-tuning is typically controlled by a single group, whereas pre-training may involve crowdsourced data.

2. The author considers a comprehensive set of attacks.

3. The evaluation is thorough, showing useful insights about the varying persistence levels of different attacks.
Weaknesses: 1. The author may want to add a conclusion section.

2. In introduction, the author could provide more details about Figure 1.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 4
reviews: 
Comments: This paper works on the backdoor attacks on pre-trained models and reveals the vulnerability of injected backdoors after further fine-tuning for alignment. Experiments are conducted to illustrate the vulnerability of four different attacking consequences.
Strengths: This paper is written clearly and easy to follow. Different sizes of models are pre-trained to illustrate the potential vulnerability of backdoor attacks during pre-training. Various attacking goals are tested to make a comprehensive evaluation.
Weaknesses: 1. The novelty seems to be limited. This is not the first work studying the problem of backdoor/poisoning a pre-trained model. Existing works like [1][2] have already revealed the vulnerability of the pre-training model facing backdoor attacks in the pre-training stage and the backdoor is preserved after different kinds of fine-tuning. I believe there are more works studying this problem, and none of this literature is discussed. Therefore, it is not clear what novel things this paper is trying to handle. Please carefully discuss the existing literature and state the novel problem this work expects to handle.

2. The contribution of this work is limited. From the perspective of methodology,  this work does not propose any novel attacking methods but a simple backdoor attack using existing datasets, which also makes the title improper and misleading. From the perspective of results, this work is too shallow, and the conclusion of empirical experiments is just a verification that pre-training backdoor can survive the fine-tuning and impact future model usage, which as I mentioned in weakness 1, is not a novel conclusion. Besides, there are no deeper insights or understandings, such as observing any unique phenomenon in LLMs; explaining why backdoors in pre-training can still be effective for DOS, context extraction, and belief manipulation but fail in jailbreaks; any sign of scaling laws in poisoning pre-training models as the authors have done experiments on different sizes of LLMs. Besides, there is no discussion of mitigation like what is discussed [3], which is a very important part of a paper discussing attacks.




[1] Backdoor Pre-trained Models Can Transfer to All

[2] BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning

[3] Effective Backdoor Mitigation Depends on the Pre-training Objective
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: The paper investigates data poisoning attacks during the pre-training phase of Large Language Models (LLMs). Specifically, the authors focus on four different attack objectives, and show that three out of the four attacks persist after finetuning and alignment. Remarkably, one of the attacks can achieve its objective by poisoning only 0.001% of the training data, underscoring their practicality and potential impact.
Strengths: - The paper is well-organized and easy to follow.

- The problem addressed is interesting and relevant to the community. Its practical implications are high, especially given the challenges in defending against pre-training data poisoning attacks due to the massive scale of datasets involved.

- The paper covers various attack objectives. I particularly appreciate the inclusion of negative results (for the jailbreaking attack), which adds depth to the paper.
Weaknesses: - It appears that all experiments were run only once. Repeating experiments and reporting the mean and standard deviation of the results would better support the claims. For instance, in Figure 5, the attack is highly successful on a 2B parameter model but less so on both smaller and larger models. This variation could be due to the single-run experiments, and averaging results might attenuate this effect. If not, an explanation of this phenomenon would be helpful, as it is not immediately intuitive.

- The statement, “More capable models are more vulnerable to poisoning” (for the context extraction attack), does not seem adequately supported by the experiments. For example, the 4B and 7B models, despite being more capable than the 2B model, show lower vulnerability. Additionally, assuming that the claimed trend holds on average, larger models are typically trained on larger datasets. Thus, it’s unclear if vulnerability truly increases with model size when data scales proportionally.

- The authors hypothesize that certain poisoning attacks might bypass existing filters, but they did not conduct any experiments to verify this. Including experimental results would strengthen the claim.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: This paper studies poisoning attacks on the pre-training dataset of LLMs. Four objectives are considered for the poisoning. Empirical results showed the effectiveness of the attacks under a moderate poisoning size.
Strengths: 1. To the best of my knowledge, poisoning attacks on pre-trained LLMs are largely unexplored (due to the computational costs of pre-training an LLM). This paper takes the first step towards this. 

2. Both pre-training and post-training are considered in the evaluation, i.e., the end-to-end effectiveness of the poisoning attacks is evaluated. 

3. In general, the paper is easy to follow as the methods used in the paper are simple.
Weaknesses: 1. 0.1\% poisoning size can be large for LLMs, given that the pre-training dataset of an LLM is usually very large. In the introduction, it is mentioned that Carlini et al. showed that 6.5\% of English Wikipedia can be modified. However, simultaneously manipulating 6.5\% of English Wikipedia can be impractical in the real world and this can be easily noticed by Wikipedia users.  

It is mentioned that post-training attacks are less practical compared with pre-training attacks. The authors may consider revising this claim, given the moderate poisoning rate of the proposed attack. These attacks work in different stages under different threat models.  

2. Some objectives of poisoning attacks are not interesting enough. For instance, even without attacks, many existing studies showed that we can already successfully perform jailbreak and prompt stealing. In other words, an attacker may not need to perform poisoning attacks to the pre-training dataset of an LLM. It is unclear what unique behaviors an attacker can achieve for pre-training poisoning, compared with post-training poisoning. 

3. As a research paper, the technical contribution is limited. The attacks used to craft poisoned texts are straightforward.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: LLM models are typically first pre-trained on uncurated datasets and then further adjusted using supervised fine-tuning and reinforcement learning from human feedback (for alignment). This paper studies data poisoning in the pre-training stage with different attack goals including denial of service, context extraction, jailbreaking, and belief manipulation. Experiments are conducted on OLMo models with varying numbers of parameters. The pre-trained dataset size is about the number of parameters multiplied by 20. Results with a 0.1% poisoning rate show all attacks except jailbreaking can inject the backdoor into the final trained model. For the denial-of-service attack, even a poisoning rate of 0.001% can achieve reasonable attack performance.
Strengths: 1. This paper is the first to address a significant threat at the LLM pre-training stage, whereas existing research has focused primarily on attacks occurring during post-training or inference.

2. The paper is well-written, offering a thorough introduction to backdoor attacks and outlining various attack goals.

3. The experiments are conducted on models of varying sizes, demonstrating that this threat exists across both small and large models.
Weaknesses: 1. This paper applies existing backdoor attacks at a different stage in the training pipeline. As a result, the technical contribution appears limited. It would be helpful to clarify the specific technical challenges unique to conducting these attacks during pre-training.

2. It would be better to provide an analysis of why jailbreaking fails while other attacks succeed. Although the observation is interesting, the underlying cause is more valuable.

3. Similarly, the authors claim their finding is contradictory to the existing work of Hubinger et al. (2024). Hubinger et al. (2024) reported safety training was ineffective against the poisoning attack, while the authors found the opposite: DPO can reduce the unsafe rate. It would be better if the authors could provide some explanation more than just one statement. In addition, as mentioned in the first point, the poisoning attack itself is not successful. Given this, is it reasonable to claim the effectiveness of DOP in removing the poisoning attack?

4. The authors observe larger models are more vulnerable to the context extraction attack. I can see the trend among the 604M, 1B, and 2B models in Figure 5. However, 4B and 7B models are less vulnerable compared with 1B and 2B models. Could you explain why this happened?

5. The legend and lines in Figures 5 and 8 need to be revised. I think the legends of Figures 4, 5, and 8 are the same. However, Figures 5 and 8 use solid and dashed lines to denote poisoned and unpoisoned cases.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper investigates the vulnerability of pre-training to adversarial manipulation by malicious actors and examines how such attacks persist when models undergo downstream fine-tuning. The study evaluates four distinct attack categories—denial of service, information extraction, jailbreaking, and value poisoning—across various model scales. The authors conduct comprehensive empirical analysis for each attack type and document varying degrees of attack resilience under different fine-tuning approaches (SFT and DPO).
Strengths: 1. The research direction is compelling and timely. While existing literature predominantly examines attacks introduced during fine-tuning, this work addresses the practically important question of how pre-training vulnerabilities manifest in downstream tasks. This perspective is particularly relevant given that fine-tuning typically remains under controlled conditions, whereas pre-training often relies on distributed or community-sourced datasets.

2. The paper examines a diverse and well-chosen range of attack vectors.

3. The experimental evaluation is comprehensive and yields valuable findings regarding the differential resilience of various attack types across fine-tuning methodologies.
Weaknesses: 1. The paper would benefit from the inclusion of a dedicated conclusions section.

2. The introduction section could elaborate further on the content and implications of Figure 1.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 4