Your Hallucinations can Actually Help: Towards Robust Machine Unlearning with In-Context Hallucinations
Authors: Zhaodonghui Li, Huiming Wang, De Wen Soh, Gao Cong, Lidong Bing


Reviewer paper summaries: This work aims to propose an efficient pipeline named MUNICH for machine unlearning. MUNICH includes three main steps: Relevant Knowledge Retrieval, Hallucination Generation and Answering by Injecting Hallucination. The rationale behind the pipeline is to use generated hallucinated texts to circumvent the correct answer and therefore avoid answering questions regarding data privacy or security, which is named in-context hallucinations by authors. 

To evaluate the pipeline, a new benchmark is constructed, which contains 45 diverse topics of knowledge. For each topic, they generate both fill-in-the-blank and multi-choice QA questions. They also proposed a new metric of "Unlearn Ratio" to measure the performance of different methods. Experimental results show MUNICH can outperform other baselines on their settings.
Summary of strengths: * the problem of efficient machine unlearning is a significant area and it is worth researching
* this work attempted to construct a benchmark for machine unlearning task, which might be useful
Summary of weaknesses: **The writing should be improved.**
* One contribution of this work is the constructed benchmark, but I have difficulty understanding the benchmark construction. There are many concepts in the benchmark and they often appear without sufficient contexts such as "in-distribution knowledge", "synthetic knowledge", "knowledge bases" and "related entities". Also, the motivation for using entity-level and relation-level facts, fill-in-the-blank and multiple-choice QA is not clearly stated.
* another contribution of this work is the MUNICH pipeline,  but the details of the pipeline are not well-stated (see comments). 

 **My another main concern is the practical usage of the pipeline**
* it needs to recall GPT-4 to do knowledge retrieval ( the paper does not describe using which LLM for hallucination generation), which is an additional costs.
* If I understand correctly, the main function of the pipeline is to use hallucinated text to circumvent truthful answers. This cannot solve the issue of privacy and end-users might design another fancy prompt to elicit the facts again.
* the pipeline may undermine the general capability of LLMs

**the proposed metric Unlearn Ratio (UR) seems arbitrary and lacks theoretical support and further analyses**
Comments, suggestions and typos: * in line 88, can you explain what are the two interesting topics?  What are the metric used in Table 1. Otherwise, readers cannot clearly know what happened in Table 1
* in line 95,  there is one missing citation
* what are "independent topics" and "related topics"?
* "In-distribution knowledge" and "in-distribution Knowledge" both appeared. Keep it consistent 
* the name of "in-distribution knowledge" and "synthetic knowledge" confused me
*  In the step of relevant knowledge retrieval, how do you define a piece of knowledge? entity, sentence, or document. How many candidates do you retrieve? How do you filter out irrelevant knowledge? How do they impact the entire pipeline?
* which LLM is used to generate hallucination？
* the results of MU-Bench and MU-Bench++ cross appear in "5.3 Results on MU-Bench++" and "Main Results" 
* will the proposed MUNICH impact the original capability of LLMs?
Overall assessment scores: 2