Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning
Authors: Shota Takashiro, Takeshi Kojima, Andrew Gambardella, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo


Reviewer paper summaries: The paper tackles the problem of unlearning concepts for LLMs. Instead of forcing the LLM to unlearn the concept, the paper tackles the problem as to not revealing knowledge when it should not. 

Technically, the paper proposes to fine-tune LLM with special token to achieve such abstaining behaviour. The paper shows that such training process helps to model to reasonably abstain. The paper further shows that the abstaining is not forgetting and the knowledge can be detected till the last layer.
Summary of strengths: The paper tackles forgetting concept from a novel perspective (i.e. abstaining). The empirical results show that such strategy is quite successful as the fine-tuned model approaches or beats the GPT-4 performance. 

The paper further shows that through finetuning, the knowledge is not forgot but the model learns to abstain from answering. This indeed offers insight for LLM finetuning inlcuding alignment, etc.
Summary of weaknesses: The main result table does not have any meaningful comparisons. Apart from the GPT-4 results, the other 0 F1 score results are not meaningful and one could argue that it is because of non-careful prompt tuning. Meanwhile, many ablation studies are missing for the results: for example, what if one uses natural language fine-tuning without tags, etc.

While the paper offers new insights to forget concept, it does not mention related works such as Gisting https://arxiv.org/abs/2304.08467 or R-tuning https://arxiv.org/abs/2304.08467 where techniques and/or motivations are shared with this work.
Comments, suggestions and typos: Figure 1 (the Japan part) is not very clear and I suggest using caption to clarify. 

Line 288-289: I think the authors mean "We use chat configurations for LLama-2-7B, etc."

5.2.2 seems unfinished and cut abruptly as it only introduces the scoring techniques without discussing the results.
Overall assessment scores: 2