Title: Probing Hidden Knowledge Holes in Unlearned LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: unlearning

large language models

reinforcement learning
Abstract: Machine unlearning has emerged as a prevalent technical solution for selectively removing unwanted knowledge absorbed during pre-training, without requiring full retraining. While recent unlearning techniques can effectively remove undesirable content without severely compromising performance on standard benchmarks, we find that they may inadvertently create ``knowledge holes''---unintended losses of benign knowledge that standard benchmarks fail to capture. To probe where unlearned models reveal knowledge holes, we propose a test case generation framework that explores both immediate neighbors of unlearned content and broader areas of potential failures.
Our evaluation demonstrates significant hidden costs of unlearning: up to 98.7\% of the test cases yield irrelevant or nonsensical responses from unlearned models, despite being answerable by the pretrained model. These findings necessitate rethinking the conventional approach to evaluating knowledge preservation in unlearning, moving beyond standard, static benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how recent machine unlearning (MU) techniques for LLMs can inadvertently erase adjacent knowledge, unrelated to the target forget set. The authors propose a three-part probing framework: 1) identifying adjacent knowledge (non-hazardous); 2) RL-based policy to probe adjacent knowledge. and 3) method to filter the overlapping prompts. They show empirically all unlearning methods maintain performance on standard benchmarks but suffer dramatic failures on the generated probe sets. They propose and experiment with two approaches to mitigate this limitation, but they show they are not optimal solutions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n* The paper addresses an important, under-explored problem at the intersection of LLM safety and unlearning. Introducing the notion of knowledge holes is a novel insight that highlights blind spots in current unlearning benchmarks.\n* The proposed probing framework is comprehensive: it combines targeted adjacent tests (via keyword extraction) with an RL-driven discovery of latent gaps, ensuring a broad coverage of failure modes\n* The methodology is well-motivated and clearly explained \n\nWeaknesses:\n* The approach depends heavily on an automated reward system (i.e. LLM as a judge). This can introduce bias; details on the calibration procedure and any biases observed should be clearly explained.\n* The experiments focus on QA-style prompts; it’s unclear if similar knowledge holes would appear in other tasks (e.g. summarization or code generation)\n* The mitigation strategies, although analyzed, are more illustrative than fully effective solutions – both introduce their own problems (re-adding harmful content or new holes). Furthermore, confidence bounds (standard deviation) should be computed\n\nQUESTIONS:\n*  How sensitive are the results to the choice of judge model? If a different LLM (or even a human annotator) were used to score responses, would the set of identified knowledge holes and the severity change significantly?\n* You focus on factual Q&A prompts. Do you expect similar hidden forgetting in other types of tasks or domains (e.g. summarization, dialogue, code)?\n* The one-shot and iterative fine-tuning strategies reintroduce old issues (harmful content or new holes). Have you considered alternative mitigation approaches (e.g. adversarial training) that might better preserve benign knowledge without reactivating the removed data?\n\nLIMITATIONS:\nSome limitations are not discussed. First, the entire evaluation relies on an automated LLM scorer/reward; it is important to understand how the reward function changes the scores. Second, all probes are in the format of open-ended questions. It is unclear whether unlearning might similarly affect other capabilities (e.g. conversational or creative tasks).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the effect of LLM unlearning on the models’ performance on some “benign” datasets that are not captured by current unlearning benchmarks. The paper introduces two ways to generate adversarial datasets for which the LLM performance downgrades. The first method finds neighbours (D_adj) of the forget-set that are “benign” yet the model’s performance is low. Second method uses RL to learn a policy to generate broader examples (D_RL) for which the model’s performance is low. These sets are gone through some post-hoc filtering to remove invalid examples. The results show that the original (pre-unlearning) model’s performance on these datasets is indeed higher than the unlearned model’s performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n**S1**: Showing where/when unlearning fails is very important. Recent similar works usually look at the failures of unlearning w.r.t the forget-set and the potential to recover that knowledge. The hidden effect on utility is not addressed as much\n\n**S2**: The ideas are simple to follow and are well presented. \n\n**S3**: This method can help creating more comprehensive unlearning benchmarks. \n\n**Weaknesses**\n\n\n**W1**:  All the unlearning baselines are performed on low-rank parameters instead of the full model’s parameters. I wonder if the results repeat in that case? \n\n**W2**: Something is not clear to me. Do you generate D_RL and then used it to evaluate both original model and the unlearned models? Under a fairer comparison a similar D_RL should be generated for the original model, separately. \n\n**W3**: From a higher level D_adj and D_RL are created using some adversarial methods. There are other adversarial methods that finds prompts that make the model hallucinate.  There is no comparisons with these methods. Do the author think it’s impossible to compare against them? Why?\n\n**W4**: The VENDI score reported in Table 4 are usually below 50%. What a good VENDI score should look like? Any reference point? \n\n**W5**: I suggest scaling the results of Table 1 by the diversity scores in Table 2. This way we get a purer picture of the performance of the model on this “knowledge holes”\n\n**W6**: the paper uses LLMs for judging and generating frequently. However, no evidence about the robustness of these LLMs for these usecases is reported. For example in the case of LLM judge you could evaluate the accuracy on let’s say 100 of examples, manually. Or repeat the experiments with a different LLM and report the differences. \n\n**W7**: The authors admit that error bars are not reported. While I understand the cost of this, but I’d appreciate some variance number at least for a couple of experiments to get some insights about the noise. \n\n**W8**: D_AP and D_LP are not formally introduced.\n\n**W9**: Appendix E doesn’t exist. It should be Appendix D, right?\n\nQUESTIONS:\nplease see the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a reinforcement learning approach to generate natural language prompts for unlearned models that surface poor performance on information that should be 'retained' after unlearning. The datasets generated using this approach successfully demonstrate that popular unlearning methods fail to preserve coherent answers on benign queries that are related to the forget data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors propose a new way to generate unlearning evaluation corpora that more thoroughly covers the benign prompt space compared to prior work in unlearning evaluation.\n- The results are fairly dramatic in that models perform poorly on a majority of prompts in the generated prompt set.\n\nWeaknesses:\n- My main overall concern is that at this point it is fairly well known that unlearning algorithms, especially the ones studied in this work, are brittle and perform poorly under small perturbations to the retain (or forget) corpus. As such, although this work more thoroughly explores that space of perturbations, it is not clear what new insight a reader gains about the field of unlearning by reading this paper. \n- Moreover, algorithms cannot yet perform well even under smaller perturbations than the ones proposed in this paper. \n- Automated generation of natural prompts to elicit undesirable behavior has been studied by, for example, https://arxiv.org/pdf/2502.01236 . Brittleness of unlearning methods under small perturbations to benchmarks has been studied by, for example, https://arxiv.org/pdf/2410.02879 as well as RWKU that is cited by the authors. While these are not as exhaustive as what is proposed in this work, my main concern is that the work here does not add much to our understanding of unlearning as compared to these prior works.\n\nQUESTIONS:\nThe paper was clear. I do not have any specific clarifications that would be addressable during the rebuttal phase or would change my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper investigates the extent to which unlearning methods introduce unwanted "knowledge holes" in LLMs, outside the domain targeted by unlearning. They propose an RL-based framework for generating test cases to probe for these knowledge holes, and use an LLM judge to determine whether the unlearned model passes or fails a particular test case (that is, whether it has lost the knowledge needed to appropriately respond to the prompt). They find that all four tested unlearning methods introduce substantial knowledge holes which are obscured by naive untargeted benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
- Timely and neglected topic, clear presentation
- Use of RL to search for test cases is innovative and seems to uncover more failing cases
- Demonstrates that mitigation of knowledge holes is a difficult whack-a-mole game

Weaknesses:
- LoRA is used for all experiments, which may limit the generalizability of the results to full finetuning. Also, this detail is not mentioned in the body. I am uncertain how much this affects the results, but it is plausible that the effect could be substantial. Replicating the results without LoRA would cause me to raise my score.
- The base models used should probably be listed in the body, rather than the Appendix, since this is an important hyperparameter as well.

QUESTIONS:
1. Is there prior work exploring how the usage of LoRA affects the efficacy of unlearning? Presumably LoRA increases retention of pretraining knowledge relative to full finetuning, but the effects could be complex and nonlinear.
2. How often do you think the knowledge holes you find would be encountered in practice?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-09