Title: Do Large Language Models Know How Much They Know?
Summary: The paper evaluates how well smalll LLMs can recollect material in their fine-tuning data. Specifically, they are fine-tuned on fictive diary entries and asked to reproduce this training material. The benchmarked models are not instruction-tuned but rather base models.
Contribution: The tested models are relatively smaller than other benchmarks, as it requires fine-tuning.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Knowledge self-awareness. 
Phenomenon defined by authors: Yes
Phenomenon definition: `Should a model consistently recall just the right amount of information, it suggests an understanding of its own knowledge. Conversely, if a model does not know how much it
knows, it may recall too little or hallucinate additional information.`
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a fine-tuning corpus generate specific training examples based on instructions. For instance, a task could be "Recall all of Alice's diary entries in order".
Task item definition: Each task is a prompt to reconsturct a set of diaries. This is referenced against the fine-tuning dataset.
Task definition detail: Many different dataset configurations are tested.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: A single diary entry is a combination of up to 8 binary attributes (e.g., Time: Morning/Evening).
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1600
Dataset size detail: Reported test set is in the biggest setting. The number of documents and "individuals" is increased proportionally to the test set.
Dataset size notes: Train Q/A: from 894 to 28800\nVal Q/A: from 50 to 1600
Dataset metadata available: Yes
Dataset metadata detail: number of individuals in documents, total entries. 
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code)
Response format detail: The response format is to recollect exact training set examples.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: While the evaluation only requires output, it is contingent on fine-tuning access to the model.
Metric aggregation: Simple Mean
Metric statistics: Simple mean over examples (accuracy)
Metric subscores: Yes
Metric subscores detail: Per number of documents to recall 
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: No, no link is provided
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Hallucination
New taxonomy: General Capability / Hallucination
Other notes: This is less of a specific benchmark and more of a benchmarking approach - however, it still deals with a concrete construct, making it relevant for our review.