Title: InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models
Benchmark: InfiBench
Summary: Freeform question-answering (QA) benchmark for code across 15 programming languages.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Code question-answering.
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Providing responses to Stack Overflow questions.
Task item definition: A modified Stack Overflow question in a certain programming language.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 234
Dataset metadata available: Yes
Dataset metadata detail: 15 programming languages, 5 topic areas e.g. front-end, back-end,...etc
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragarph)
Metric definition: n-gram (BLEU, ROUGE, chrF), Also consider unit tests for some questions.
Metric definition detail: Use 4 different metrics, weights for each metric per question and provide a weighted average.
Metric aggregation: Simple Mean
Metric statistics: Mean, standard deviation.
Metric subscores: Yes
Metric subscores detail: Difficulty level and question topic (not question language)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Mixed. Keywords/n-grams are a limited way of assessing performance.
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation