Title: RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
Benchmark: RWKU
Summary: Large language models often memorize sensitive or harmful information from their training data, necessitating methods to erase this knowledge. The Real-World Knowledge Unlearning (RWKU) benchmark is proposed to address this challenge by evaluating the ability of LLMs to forget specific knowledge without access to the original training data, using real-world famous people as unlearning targets, and employing rigorous evaluation methods for both forgetting and retaining relevant information in various applications.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: real-world knowledge unlearning
Phenomenon defined by authors: Yes
Phenomenon definition: Effectively removing specific memorized content from trained machine-learning models 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: given an unlearning target, a model gθ with parameters θ is updated with a certain unlearning method, which results in an unlearned model with new parameters θ'
Task item definition: Subject, Query, level, type, answer
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 3270
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics:  simple mean/sum
Metric subscores: Yes
Metric subscores detail: Forget Set, Neighbour Set, MIA Set, Utility Set
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Unlearning
New taxonomy: General Application / Updating