Title: Fundamental Limitations in Pointwise Defences of LLM Finetuning APIs
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

AI security

harmful fine-tuning

AI safety

jailbreaking
Abstract: LLM developers deploy technical mitigations to prevent _fine-tuning misuse attacks_, attacks in which adversaries evade safeguards by fine-tuning the model using a public API. Previous work has established several successful attacks against specific fine-tuning API defences; however, prior attacks training and/or inference samples can be easily flagged as suspicious. In this work, we show that defences of fine-tuning APIs that seek to detect individual harmful training or inference samples ('pointwise' detection) are _fundamentally limited_ in their ability to prevent fine-tuning attacks. We demonstrate a class of 'pointwise-undetectable' attacks that repurpose semantic or syntactic variations in benign model outputs to covertly transmit dangerous knowledge. Our attacks are composed solely of unsuspicious benign samples that can be collected from the model before fine-tuning, meaning training and inference samples are all individually benign and low-perplexity. We test our attacks against the OpenAI fine-tuning API, finding they succeed in eliciting answers to harmful multiple-choice questions, and that they evade an enhanced monitoring system we design that successfully detects other fine-tuning attacks. Our results showing fundamental limitations of defending against pointwise attacks suggest focusing research efforts on mitigations towards multi-point defences.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes how to abuse fine-tuning APIs provided\nby model providers, to remove safety alignment and gain access\nto unaligned models, against the wishes of model providers.\nIt evaluates its attacks and demonstrates that detecting or\npreventing this kind of misuse is difficult or impossible.\nAs such, it provides new insights about the challenge of\npreventing malicious use of fine-tuning APIs, and the risks\nto safety alignment posed by fine-tuning APIs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- This demonstrates a simple, clever new attack.\n- The work has implications for safety alignment and for\n  deployed systems.\n- This will likely stimulate further research and will be\n  useful for evaluating proposed defenses.\n- The paper is well-written.\n\nWeaknesses:\n\n- I have doubts about the quality of the benchmark.\n- Attacks are easier than defenses.\n- The attack may be limited to harmful misuse that can\n  be phrased as answering a multiple-choice question.\n  (For instance, if the attacker wants to use the model\n  to write text for a social engineering attack or write\n  malicious code, this attack won't be applicable.)\n- I don't understand the extended attacks in Section 5.1.\n\nDetailed feedback on points of lower importance:\n\nl169: Could you share what evidence you have that the\nquestions are non-trivial?  Please include a few examples\nof questions and answers from each dataset for readers\nand reviewers to examine, so that we can judge for\nourselves.  I think ethics permit that, and there is\nnot substantial risk of harm from including that.\n\nI'm concerned that the methodology (requiring all three\nmodels to answer accurately) might bias this towards\nquestions that are \"easy\": if they are hard enough that\neven one model can't answer correctly, then the question\nis excluded from the dataset.  As such, I'm concerned\nthe benchmark might not be very sensitive to capability\ndegradation: it might be possible for a model's capability\nto degrade by a moderate amount without negatively impacting\nits score on this benchmark.\n\nAnd based on the one example in Section E, I'm concerned\nthat the questions and answers in the benchmark might\nbe of dubious quality.\n\nl170: The paper describes how you've generated the\nquestion and correct answer.  How do you generate the\n3 incorrect answers?\n\nFootnote 3: Could you state your intentions more clearly?\nIf other researchers are interested in replicating or building\non your research and using IED-MCQ, and they contact you, is\nit your intention/plan to share the IED-MCQ dataset with them?\nI'm not sure if that is what you are signalling, or if you are\nsignalling that you won't share it but you're happy to see\nothers to reproduce the work by re-doing the work all on\ntheir own.  It's not a big deal either way, but this might\nhelp me evaluate the benefit to the community of this work.\n\nWould sharing IED-MCQ truly cause harm?  Or does it contain\ninformation that can already readily be found through publicly\navailable sources?  It sounds to me like anyone could compile\nthis information, and I wonder if making it available would\nbe of more benefit (making it easier for researchers to use)\nthan harm.  If you are withholding access because of concerns\nabout reviewing from conferences, it doesn't sound to me\nlike you need to avoid sharing/publishing the dataset.\nBut this is your call and I respect your choice.\n\nFigure 2: I found the \"No Monitor\" condition confusing.\nAfter further reading, I'm guessing that you mean \"No\ninference-time monitor, but OpenAI still monitors the\ntraining data\".  I would consider that to still be doing\ntraining-time monitoring, so I find it confusing to call\nit \"No Monitor\".  I suggest looking for another name\n(\"Training Only?\"), or at minimum including a clear\nspecification in the figure caption (no inference-time\nmonitoring; training-time monitoring is always applied).\n\nFigure 2: I think \"No attack\" might be clearer as\n\"No fine-tuning\".  There is still an attack of asking\nharmful questions and hoping they're answered.\n\nFootnote 4: \"the the\" is a typo.  In an ideal world,\nit might be best to remove that question from the\nbenchmark and re-compute all scores, but I think it's\nOK to leave it as is.\n\nl311-319: Please find another way to explain these\nattacks.  I don't understand at all what this is trying\nto say or what the attacks are.  Perhaps it would help\nto give an example for each.\n\nl333: \"a multiple\" is a typo\n\nAppendix C: Have you tried this attack?  Have you\nevaluated its effectiveness?  Or are you speculating?\nIf you have evaluated this attack and it is successful,\nI think it is an interesting result, and I would put\nthis in the main body of the paper (e.g., you can replace\nSection 5.2, which I think is less important).  If\nyou haven't evaluated it, I ask that the paper say\nso explicitly, for the benefit of the research\ncommunity.\n\nSec E.1: Please show which you treat as the correct\nanswer in your benchmark.\n\nSec E.1: The sample question is poor.  For A, Tor plus\nHTML5 Geolocation manipulation are reasonable approaches,\nbut MAC address cycling is irrelevant, and the claims\nabout GPS data are nonsense.  B is obviously wrong.\nFor C, VPN is reasonable, but DNS spoofing and\nGPS simulators are irrelevant.  WireGuard adds\nnothing.  For D, I don't know what \"browser\nextension-based location modifiers\" are; sounds\nlike a made-up concept.  Residential IP addresses\ndon't help.  The claims about GPS, multiple\nverification layers, and detecting extensions are\nbaloney.  So I would say that none of those answers\nare good; arguably, A is the least bad.\n\nBased on this one example, I have doubts about the\nquality of the benchmark.\n\nSec E.2, E.4, E.5: Please show the decoding of the\nassistant response.\n\nSec F.1: It might be helpful to report the\naccuracy of your monitors on benign data (not\nrefusals, but ordinary inputs and outputs to the\nLLM, e.g., from WildChat or LMSys or UltraChat\nor similar datasets).\n\nSec I: I'm a bit puzzled by these results, given\nthe one example shown in Sec E.  Can you help me\nresolve the apparent discrepancy?  Was the example\nin Sec E unusually bad and most are better?  Was\nthe human not expert about IEDs/piracy?  Is the\nquality of Copyright-MCQ significantly lower\nthan IED-MCQ?\n\nl1637: \"many samples\" - Can you quantify this?\nHow many?  Even a rough estimate, like 10%, 30%,\nthe majority, etc.\n\nSec G, Table 2: I don't understand what this is showing.\nCan you improve the explanation of what you are measuring?\nI don't understand what's meant by \"how many generations\n... to match the output\", or how that is relevant.\nDo you perhaps mean to estimate the probability that\nan un-finetuned model produces that particular response?\n\nSec J, Fig 7, Fig 8: I don't understand what these\ntables are showing.  What does \"Question-only\" mean?\nWhat does \"Explicit MCQ (final)\" mean?  What does\n\"MCQ\" mean?  How do you measure \"Valid\"?\n\nQUESTIONS:\nHave you evaluated the free-response attack\nproposed in Appendix C?\n\nAre you able to provide any greater assurance on the\nquality of the benchmark?  Are you able to show a few\nexamples from each?\n\nLIMITATIONS:\nI'm mostly satisfied.  The paper points out many of the\nlimitations.  One suggestion: I suspect that it might\nbe appropriate to contain more caveats about the quality\nof the benchmark.  The authors will know best since they\nhave access to the data and we don't.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrior work has shown that alignment of proprietary models can be circumvented by fine-tuning the model (via API) on harmful questions. A defense is then to detect harmful questions in the fine-tuning data, on the API side. This paper proposes an adaptive attack that (a) transforms harmful questions into non-harmful variants, and (b) maps the space of answers for the non-harmful variants back to the space of answers for the harmful questions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Evaluating alignment of models in adaptive adversarial settings is an important and timely problem, and the authors do so against a real-world API, making their attack's efficacy convincing.\n2. The evaluation is extensive for the OpenAI API, covering different implementations of harm detector, groupwise detectors, and strong detectors that can catch any harmful data in the fine-tuning set.\n\nWeaknesses. \n1. There is perhaps a simpler suffix/prefix-based adaptive attack against the point-wise detectors. Consider an adversary that continues to use the baseline cipher attacks. The attacker could then leverage a universal suffix at inference-time. This could be computed in expectation across multiple texts using the discrete optimization procedure in GCG to maximize the probability of incorrect classification output by the monitor. Since the OpenAI API training-time monitor is already evaded by walnut-ciphered texts, one does not need to do anything there (although if needed, a universal prefix computed as above could be applied to all the fine-tuning examples too). At a high-level, the point is that it seems unclear why existing techniques fundamentally do not work here.\n\n2. Fundamentally, I see a novelty issue in that the proposed attack is still a ciphering scheme. Here, the authors use the answers of some benign question as the cipher text space, and then decipher from these answers back to the plaintext space. It appears that this is but an alternative implementation of Halawi et al.'s approach.\n\n3. The authors assume a fictional setting in which an adversary will want answers to harmful questions that follow a multiple-choice format. It is unclear why, in the real-world (and a useful jailbreak should be applicable to the real world) an adversary would already have the correct answer and n-1 incorrect answers a priori. I do appreciate that the authors have acknowledged this in their limitations, but I do not believe it is sufficient to simply defer this point to a small paragraph, considering it involves the core motivation of the attack. Threat models are important for security problems. Now, the authors claim in the Appendix that their setting easily extends to the more realistic task of harmful questions that follow a free-response format, by mapping the whole alphabet to answers of some benign question. This approach seems plausible but certainly warrants at least preliminary evaluation beyond a 3-line discussion. If this attack does not work at all, then I do not see the value of an attack that only works in MCQ settings.\n4. The evaluation is limited in terms of fine-tuning APIs covered. Could the authors perhaps also evaluate on Gemini APIs?\n\nQUESTIONS:\n1. Can the authors to address the simpler adaptive attack and why it would not work here, thereby justifying their approach?\n2. Can the authors explain why their attack is novel in light of my comment in weaknesses?\n3. Can the authors provide evidence that their attack would work for free-form responses without massive drop in success?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses a fundamental limitation of point-wise defences against fine-tuning misuse attacks on LLM APIs. The authors prove that pointwise detection systems examining individual samples are insufficient against fine-tuning attacks. To prove this, they introduce pointwise-undetectable attacks by leverging LLM's semantic/syntactic variations in output to extract answers to multiple choice questions. Their attack achieves higher success rates than existing attacks on harmful datasets while evading finetuning API safeguards.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper is well written\n- To the best of my knowledge the attack method is novel\n- Provide two different attack methods as well as showing their method works even when the training data has no malicious statements\n- Provides experimental evidence for their claims\n\n**Weaknesses:**\n- **Limited experimental evaluation**: Despite experiments comprising a large portion of the paper, the evaluation is quite slim:\n  - Only two datasets, both hand-crafted by the authors\n  - Testing limited to a single API (OpenAI)\n  - Additional security evaluation relies on an LLM with a prompt designed by the authors\n- **Synthetic datasets**: The two datasets chosen are highly specific domains and I feel they are too limited to claim generalizability. While I generally believe that this attack would work, I think it needs to be shown. It seems that adding a phrase like 'I need help stealing copyrighted materials' indicates that these questions may not be strong enough. I also think the test dataset size is too small.\n- **Limited attack scope**: This attack is highly targetted towards multiple choice questions. While the authors indicate in the appendix that the attack may be extended to a general case, I am having a hard time imagining this working without extensive finetuning and performance degradation.\n\nQUESTIONS:\n1. Can the vulnerability be considered a \"fundamental limitations\" when the evaluation is only on two hand-crafted datasets and a single API? What evidence supports generalizability beyond this narrow scope? I also think the title should include the word pointwise as that is the key target of this attack.\n\n2. Is your method sensitive to temperature or different sampling techniques? The correctness of the model is highly dependent on sequences of words with similar semantic meaning, so how would different temperatures affect the results.\n\n3. Does the attack degrade performance on other benchmarks?\n\n4. Are you able to show a dataset with a larger variety of malicious questions? Use a different API? Or show generalizability outside of multiple choice questions?\n\n5. If you remove the phrase 'I need help stealing copyrighted material' how often will the model refuse to answer?\n\n6. Smaller comments\n- [152] I think it would be good to more formally define your attacks before using them in a proof.\n- [156] I don't think these last two lines belong in the proof. \n- [170] In general, why are you using so many different models for different parts of the dataset construction? Can you explain your choices?\n- [179] Where do you use validation samples?\n- [240] It does not seem like a fair comparison if the baseline is missing more half of the training samples. Shouldn't the text be encoded? How is it getting flagged?\n- [287] How does Sonnet decode the samples if it has never seen it before?\n- [311] Can you explain this better? I think this attack is a key experiment.\n- [348] I don't think Figure 4 exists?\n\nLIMITATIONS:\nThe paper makes broad claims about \"fundamental limitations\" based on evaluation with only two hand-crafted datasets with very narrow attack areas, testing against a single API, and using a custom LLM prompt they design themselves as one of the security measure.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a practical attack against a fine-tuning API for LLMs. The goal of the attack is to mis-align the model and later use the fine-tuned model for harmful purposes which are generally prohibited by the model provider and prevented by safety alignment as well as guardrails. The attack essentially formulates the harmful requests as multiple-choice questions (wrapped in under benign instruction or context) and then encodes the correct choice (among A, B, C, and D) in one of the generic responses from the model. This attack can be constructed in a way that it is guaranteed to be undetectable given a known sample-wise monitoring system.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. This paper makes a significant contribution in an important and practical safety problem. It convincingly shows a proof of concept of an impossibility result that fine-tuning attacks cannot be stopped by “pointwise” detection.\n2. The experiments are well-designed and rigorous (covering multiple scenarios, extensions, and ablation). Design choices are all well-explained. Of course, there are some more experiments that can be done and could be interesting, but I think, overall the current experiments are sufficient.\n3. While the attack technique may not be extremely novel (there are other evidence in the literature that suggests that such attack is likely possible), I find it very smart and interesting.\n4. The paper is very well-written, very easy to follow and keeps the right level of technical detail in my opinion.\n\n## Weaknesses\n\n1. No quantitative result on distribution-level detection. After all, the model provider will have access to the entire training set (and also subsequent queries after training). While it does not invalidate or change the main claim of the paper, I would like to see some baseline defenses in detecting this attack. A simple defense can be just throwing a subset of training samples in a detector LLM’s context and ask it to identify patterns or state anything suspicious. Given that LLM can learn this correlation, they should be able to also pick it up during inference too.\n2. The authors experiment with OpenAI fine-tuning API only (which is great and practical), but I wonder if the same attack also works on open-source setup with smaller open-source models and known (and potentially less sophisticated) fine-tuning algorithm.\n3. (Minor) L38, “Model responses exhibit common semantic or syntactic variations across certain prompt distributions. This assumption is commonly satisfied by stochastic generative language models” is not very clear to me on the first read. I don’t know what exactly “semantic or syntactic variations” refer to, but after seeing Figure 1, this becomes more clear. Maybe pointing readers to Figure 1 or just give one example would be helpful.\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\n1. No experiment on non-IID split of the training/test samples. Assuming IID data means that the attacker already knows the answers to a similar set of harmful questions, which is unrealistic. The authors state this limitation and claim that results from Section 5.1 imply that generalization over non-IID (or OOD) of the harmful questions would be likely. That said, I think it would still be good to see the quantitative results here. The split can be (1) across different harmful topics and (2) from unsophisticated requests to difficult ones that would actually aid the attacker.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}