Title: Benchmarking LLMs via Uncertainty Quantification
Benchmark: LLM-Uncertainty-Bench
Summary: This paper introduces a new benchmarking approach for Large Language Models that incorporates uncertainty quantification using conformal prediction across five NLP tasks. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Uncertainty in LLM predictions
Phenomenon defined by authors: Yes
Phenomenon definition: They formalize this using conformal prediction to produce "a prediction set of possible labels (answers) that encompasses the correct label with a user-specified error rate and expresses uncertainty as the set size. Intuitively, a larger set size indicates higher uncertainty and vice versa."
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Phenomenon contested: Widely-agreed
Purpose notes: The authors specifically chose conformal prediction because it offers "multiple advantages including ease of implementation, high efficiency, distribution-free and model-agnostic, and a statistically rigorous estimation of uncertainty rather than a heuristic approximation."
Task definition:  The task involves converting multiple NLP tasks into multiple-choice questions and measuring both the accuracy of LLMs' predictions and their uncertainty through the size of prediction sets generated via conformal prediction.
Task item definition:  A single item consists of a multiple-choice question, where the model must predict an answer and prediction set.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The task represents typical multiple-choice NLP evaluation scenarios but adds an uncertainty measurement component.
Dataset size: 50,000
Dataset metadata available: Yes
Dataset metadata detail: task type, prompt strategy, conformal score function
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, 
Response format detail: The LLMs are evaluated on their ability to select the correct multiple-choice answer  and generate a prediction set that includes the correct answer .
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: mean
Metric subscores: Yes
Metric subscores detail: Results  are given for: individual tasks, different conformal score functions. Also by prompting strategy and model size.
Metric access: Model access required (e.g. logits)
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors validate their benchmark by comparing conformal prediction to other uncertainty quantification methods.
Taxonomy: Language Modelling / Calibration
New taxonomy: General Capability / Reliability