Title: Generative Language Models for Paragraph-Level Question Generation
Benchmark: QG-Bench
Summary: QG-Bench, a comprehensive benchmark for paragraph-level question generation (QG) that unifies existing question answering datasets into a standard format. The authors fine-tune LMs for the QG task and evaluate them using both automatic metrics and human evaluation.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Question generation
Phenomenon defined by authors: Yes
Phenomenon definition: "Question generation is the task of generating a question given an in- put context consisting of a document, a paragraph or a sentence, and an answer where the question is anchored"
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Generate a natural language question given an input paragraph and an answer span that appears within that paragraph.
Task item definition: Paragraph, a sentence within that paragraph, an answer span, and the target question to be generated.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: SQuaD train: 75,722
Dataset size notes: SQuaD validation: 10,570, SQuaD test: 11,877
Dataset metadata available: Yes
Dataset metadata detail: language, domain, average patagraph character length
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric statistics: Simple mean scores for each metric. For correlation analysis between automatic metrics and human judgments: Spearman's rank correlation coefficient.
Metric subscores: Yes
Metric subscores detail: Scores by language, domain, model input type
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Validation of the benchmark through manual evaluation where human annotators rate generated questions across three criteria (grammaticality, understandability, and answerability).
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction