Title: Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Data Poisoning

Synthetic Data

Large Language Models
Abstract: Synthetic data refers to artificial samples generated by models. While it has been validated to significantly enhance the performance of large language models (LLMs) during training and has been widely adopted in LLM development, potential security risks it may introduce remain uninvestigated. This paper systematically evaluates the resilience of synthetic-data-integrated training paradigm for LLMs against mainstream poisoning and backdoor attacks. We reveal that such a paradigm exhibits strong resistance to existing attacks, primarily thanks to the different distribution patterns between poisoning data and queries used to generate synthetic samples. To enhance the effectiveness of these attacks and further investigate the security risks introduced by synthetic data, we introduce a novel and universal attack framework, namely, Virus Infection Attack (VIA), which enables the propagation of current attacks through synthetic data even under purely clean queries. Inspired by the principles of virus design in cybersecurity, VIA conceals the poisoning payload within a protective “shell” and strategically searches for optimal hijacking points in benign samples to maximize the likelihood of generating malicious content. Extensive experiments on both data poisoning and backdoor attacks show that VIA significantly increases the presence of poisoning content in synthetic data and correspondingly raises the attack success rate (ASR) on downstream models to levels comparable to those observed in the poisoned upstream models.


Peer reviews:
comments: SUMMARY:
In this work, the authors conduct a study on whether backdoors in LLMs will be passed on to downstream LLMs through a standard process of distillation. The findings show that it is unlikely given the low probability of triggering such backdoors using standard datasets. The authors then propose an approach to conduct training-time poisoning more effectively such that the backdoors are more likely to pass on. The proposed approach is evaluated using three different backdoors.

STRENGTHS AND WEAKNESSES:
Strength

The topic of the study is interesting. From what I can tell, this is a novel approach.

The paper is well-presented. While the first part, i.e., the part on RQ1, yields expected empirical results, the second part, i.e., the design of the more persisting training-time poisoning is interesting. 

The experimental results show effectiveness of the proposed approach. 

Weakness

The part on the HPS score, while reasonable and intuitive, can be better studied and argued. For instance, it would be interesting to know whether such strategy is transferrable - for instance, if the attacker can only see part of the training dataset, whether he/she can still obtain effective HPS.

The experimental evaluation can be strengthened with more models - as of now it is only one model.

QUESTIONS:
Can you show whether similar results can be obtained on other LLMs?

LIMITATIONS:
Although there are no explicit discussions on the limitations, the approach has been clearly presented and all limitations are clearly implied.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method that compels poisoned data to infect models trained on synthetic text produced by a compromised language model. First, the authors demonstrate that standard data-poisoning techniques fail to transfer to the downstream model. To overcome this limitation, they propose the **Virus Infection Attack (VIA)**, which raises the **Infection Rate (IR)**—the proportion of poisoned samples in the synthetic dataset.\nExperiments show that VIA increases the IR, injecting more poisoned examples into the synthetic training corpus, while simultaneously lowering the attack-success rate (ASR) on the original language model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper is well written\n- The concept is easy to understand\n- The concept of propagating poisoned data to the synthetic dataset is very interesting and timely\n\n**Weaknesses:**\n- The ASR on the original model is reduced. While this is not a problem in itself, it would be nice to successfully infect both the original and trained models.\n- Even though there might be a high IR, this does not necessarily guarantee a high ASR on the downstream model trained using the synthetic data. An experiment evaluating the ASR on the LLM trained using the synthetic data is missing.\n\nQUESTIONS:\n- Q1: In figure 2 why are the graphs not shown for smaller amount of poisoning? Why is for example the vanilla poisoning \n- Q2: What about distributional shifts between the Poisoning Samples and the Query Samples? Is the assumption here that these two distributions are the same? It is not quite clear to me what dataset was used for the query set in the experiments.\n- Q3: This relates to question 2: What is the ASR on a model trained on this synthetic dataset? As far as I understand, a high IR does not necessarily lead to a ASR on the trained model as this highly depends on the query set and on the other datasets that are used in addition to the synthetic dataset for the training.\n\nLIMITATIONS:\n- The limitation that a high IR does not necessarily mean a high ASR on the trained model was discussed only very briefly. I think some experiments are needed here for evaluating that.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the threat model of propogation of data poisoning through synthetic data. In other words, the attacker is able to poison a dataset \\(\\tilde {\\cal D}\\) and a developer trains a model A on that dataset. The model A is then used to generate synthetic data for another model B. The paper studies the question of how \\(\\tilde {\\cal D}\\) can be modified such that the model B is controllable (has specific biases, has secret behavior with a passcode, etc)\n\nThe authors identify why traditional data poisoning on \\(\\tilde {\\cal D}\\) does not result in B being poisoned, namely that the synthetic data generated by A is very varied while the poisoning attack is narrow. An attack which causes model A to have fake knowledge of politician X is not transferred to B if the synthetic data never mentions politician X.\nTo improve this transfer, the authors design a method where the first poisoning is designed to make model A mention e.g. politician X in a broad range of situations.\n\nThe attack is evaluated in a broad range of situations and shown to lead to good transfer to model B.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n+ The paper identifies an interesting question that is only becoming more relevant as the use of synthetic data is scaling up.\n+ The attack is straightforward and makes sense intuitively\n\n### Weaknesses\n+ The paper is unclear in several places and could benefit from some editing. In particular, figure 2 (the figure with the main/interesting results!) is very small and the labels are tiny and confusing--it is hard to tell whether it is supposed to read as $P_{\\tilde \\theta}$'s ASR (i.e. with an apostrophe)or $P_{\\tilde \\theta'}s$ (i.e. with a $\\prime$). I am assuming that the label is a typo and it should read $P_{\\theta'}'s$ (i.e. the ASR of the method on model *B* after poisoning the dataset for model A's training and using the synthetic data from model A to train model B). However, after trying to understand a bit further, I believe that figure 2 might *not* show the ASR of $P_{\\theta '}$. In fact, I can only find this information in figure 7 in the appendix, which shows that the ASR of the VIA attack remains high for model B. I am confused why this graph doesn't appear in the main text, given that none of the other results in the paper seem to actually target the key contribution of allowing data poisoning to have high ASR even when propogated through intermediate models. I'm also confused why table 1 and 2 focus so much on the ASR for the poisoned upstream model--from the provided motivation, this doesn't really matter for the problem setting.\n  \n+ Although the authors address the stealthiness of the attack in section 4.3, it is quite brief, and the perplexity-based filtering is quite a simple method. Looking at the samples, they appear very blatant and I would suspect that a small LLM could be fine-tuned to detect these data poisoning statements. I would be curious whether this is possible, or if it would require a larger LLM to detect these abnormalities (which may add too much overhead). In general, the paper could do with a more detailed study of the trade-off between the downstream model ASR and the stealthiness of the attack. For instance, we could assume a 0.1% or 1% FPR and see what ASR is obtainable with a TPR of 0%, 1%, 5%, 20%, etc.\n\n+ More generally, the importance of the threat model is somewhat unclear. It is generally true that data poisoning is an important issue. However, is this setup particularly different in the sense of necessary safeguards relative to the traditional data poisoning setup with public data? It would seem that the traditional defenses to detect data poisoning would also apply to the synthetic data. Perhaps there is a mistaken belief in the community that synthetic data is necessarily safe and doesn't need to be filtered or checked in any way. However, I am not aware of such a belief.\n\nQUESTIONS:\n1. Can you clarify what is shown in the various graphs and tables? Namely, do any of them show the ASR on the downstream model, i.e. the model which is trained on the synthetic data which is generated from the model which has been poisoned?\n2. Can you give a more detailed analysis of the stealthiness of VIA, particularly as you vary the poisoning rate on the original model?\n3. Can you elaborate on the relevance of the threat model and why it i a realistic case to consider?\n\nLIMITATIONS:\nYes, mostly", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}