Selective Forgetting: Advancing Unlearning Techniques and Evaluation in Language Models
Authors: Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, Georg Gottlob


Reviewer paper summaries: The paper studies the problem of unlearning in the motivation for LLM to unlearn private/sensitive data. 

Based on existing unlearning literature, the paper proposes a further selection techniques that only unlearns on the selected (and arguably more precise) sensitive/private data so that the model can 1) maintain in maximum its general capability 2) improve the forgetting for sensitive data. For the sensitive information forgetting metrics, the paper proposes two new metrics.

Empirical results show that compared to existing method (i.e. KUL), the proposed method improves in forgetting sensitive information as well as in maintaining model's general capability.  Methods with more precise sensitive information identification (i.e. offline setting in the paper) performs even better.
Summary of strengths: The paper is well motivated for tackling the LLM unlearning on sensitive data. 

The proposed approach is simple and has shown the improve the performance on both sides (forgetting sensitive data and maintaining LLM general capability). Besides, the paper also investigates into further enhancement on top of the approach. The Oracle setting shows that more precise sensible span detection can further enhance the performance on both sides.
Summary of weaknesses: The paper claims two contributions. While I think the approach contribution is well grounded, the novel metric contribution is at least hard to evaluate. 3.3 proposes two novel metrics to evaluate solely forgetting. I don't understand why S-OVL which is applied previously can't be applied here to compare. Notably. why S-OVL needs to be calculated for every generated token, does it involve sampling?

Similar questions apply for S-MA where I don't see why MA itself is not sufficient.

In more general terms, I am not able to see how important it is to apply S-MA or S-EL for the tasks and can't count this as part of the paper contribution.
Comments, suggestions and typos: My general suggestion is to better explain S-EL and S-MA including their definition, the intuition behind the formula as well as why they are important and how they differ from previously established metrics for the task. 

The main table should list this as well as previously established metrics. Please note that EL and MA are not introduced in the paper.
Overall assessment scores: 2