Title: StackEval: Benchmarking LLMs in Coding Assistance
Benchmark: StackEval, StackUnseen
Summary: We present two comprehensive benchmarks to evaluate the performance of language models in coding assistance tasks, covering code writing, debugging, code review, and conceptual understanding. 
Our main contribution includes two curated datasets: StackEval, a large-scale benchmark derived from Stack Overflow questions, and StackUnseen, a dynamic benchmark featuring the most recent Stack Overflow content. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: performance of language models in coding assistance tasks
Phenomenon defined by authors: Yes
Phenomenon definition: Systematic evaluation to fully understand LLM performance across four coding assistance tasks - debugging, implementation, optimization, and conceptual understanding.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Evaluate LLM performance on four coding assistance tasks (code writing, debugging, code review, and conceptual understanding) using curated questions from Stack Overflow. 
Task item definition: A single item consists of a Stack Overflow question, the accepted reference answer, and an LLM-generated answer. 
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 925
Dataset metadata available: Yes
Dataset metadata detail: Each row includes metadata of programming language, task type (e.g., debugging, implementation), and complexity level.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: Simple mean ± 95% confidence interval
Metric subscores: Yes
Metric subscores detail: Subscores are reported by task type (debugging, implementation, code review) and programming language.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Code Generation / Natural Language
New taxonomy: General Application / Code Generation