Title: PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation
Benchmark: PrExMe
Summary: This paper introduces PrExMe, a benchmark of prompt templates to evaluate LLMs as metrics for machine translation and summarization tasks. It systematically explores how prompt variation impacts metric performance and stability, revealing both robust patterns and significant sensitivities in prompt design.
Contribution: They include emotion-CoT in prompt templates, asking the LLM to describe its "emotions".
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: evaluation of machine translation and summarization
Phenomenon defined by authors: No
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: Assign a quality score to the generated hypothesis given it source text, without using reference translations or summaries.
Task item definition: A prompt. "Each prompt is built from: (1) the source text and generated hypothesis text that should be graded, (2) a base prompt, (3) a task description, (4) a format requirement and (5) optionally a one-shot demonstration."
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: +70,000 samples
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Correlation (Matthew's correlation, Pearson's r)
Metric definition detail: "As correlation measures, we use the Kendall (primary measure), Pearson and Spearman correlations, as well as tie-calibrated accuracy."
Metric aggregation: Simple Mean
Metric statistics: simple mean, significance clusters
Metric subscores: Yes
Metric subscores detail: Subtask (e.g., en-de translation, zh-en translation, summarization), dataset (e.g., Eval4NLP, WMT23, Seahorse), prompt components (e.g., base prompt type, output format, task description)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes, implicitly
Author validity detail: - Face:  The authors motivate the benchmark clearly and argue its real-world utility, especially in low-resource and prompt-only evaluation scenarios. 
- Predictive/Concurrent: The benchmark is evaluated by how well LLM-generated scores correlate with human judgments.
- Content: Prompts are composed from modular templates (e.g., base prompts, task framing, and format variations) and the authors run tests to evaluate stability of these design components
- Convergent/Discriminant: The authors compare their benchmark results against existing metrics like XCOMET and BARTScore, 
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization