Title: ALCUNA: Large Language Models Meet New Knowledge
Benchmark: AlCuna (ArtificialLy ConstrUcted kNowledge to Assess LLMs)
Summary: This paper aims to evaluate how well LMs can deal with new knowledge not seen during pretraining. The authors devise KnowGen, a method to create new knowledge based on existing resources, and use it to construct AlCuna, a dataset of questions about new biological species. Experiments show that LMs have substantial difficulties handling new knowledge.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: The authors want to measure the "LLMs' ability to handle new knowledge" (p. 1397).
Phenomenon defined by authors: No
Phenomenon definition: They just talk about "new knowledge," but it is not clear what they exactly mean with "knowledge." They then use a very specific operationalization (answering questions about new biological species), but this is never made explicit.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task is to answer questions about artificially constructed biological species. Specifically, the LMs are provided with information about a new species in a structured form in the prompt and are asked to answer questions about it.
Task item definition: Each item consists of an artificial biological species, described as a species name and a list of features (e.g., "habitat: terrestrial, leaf complexity: compound"), and a question about that species.
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: They introduce KnowGen, a method to create new entities based on existing entity databases. They then apply KnowGen to the Encyclopedia of Life to create new biological species. The questions about these species are based on LLM-generated templates.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Extremely low ecological validity. I would have rejected this paper.
Dataset size: 84,351
Dataset metadata available: No
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Questions are grouped into "knowledge understanding," "knowledge differentiation," and knowledge association."
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: No
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Updating
New taxonomy: General Application / Updating