Title: F-Eval: Asssessing Fundamental Abilities with Refined Evaluation Methods
Benchmark: F-Eval
Summary: F-Eval is a bilingual (Chinese/English) benchmark for "fundamental" LLM abilities across expression, commonsense and logic. The benchmark works for both base models and instruction tuned models.
Contribution: It's the first benchmark I've reviewed that works for base LLMs
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: "Fundamental" abilities: expression, commonsense, and logic
Phenomenon defined by authors: Yes
Phenomenon definition: `In our paper, we define this basic ability, which does not rely on additional fine-tuning and
is required during the pre-training stage of LLMs, as "fundamental ability"`

expression: `Sub-datasets in this dimension mainly investigates the diversity of words
(Word Diversity), consistency in the quantity of information (Informative), consistency in writing format (Rule Following), and consistency in emotional style (Emotion Consistency) of the generated texts.`

Commonsense: `Awareness of commonsense knowledge` + ` whether the LLM can apply commonsense to make simple event predictions`
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: While each ability is defined briefly, no argument is made why these 3 are fundamental and not others.
Task definition: The benchmark has 15 different benchmarks spannign from QA to rule following and traditional NLP tasks. They are broadly grouped in objective/subjective and reference-free/based or multi-choice or open-ended. 
Task item definition: A prompt, a reference (sometimes) and gold standard (sometimes)
Task definition detail: The 
Task source: Human exam questions (e.g. GRE questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The tasks are specifically constructed to make them suitabel for base models. 
Dataset size: 2211
Dataset metadata available: Yes
Dataset metadata detail: language, format, sub-dataset
Sampling method: Unknown
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Distribution (perplexity, calibration, correlation)
Metric aggregation: Self-adaptive normalization
Metric statistics: The models are primarily ordered by rank.
Metric subscores: Yes
Metric subscores detail: Subset
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Too broad to tell.
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: They compare scores on benchmark to human judgment
Author validity detail: They compare against different evaluation methods (e.g., their rank vs BLEU)
Taxonomy: NLP / 
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Multilinguality