Title: StreamBench: Towards Benchmarking Continuous Improvement of Language Agents
Benchmark: stream-bench
Summary: StreamBench is a benchmark designed to evaluate language agents' ability to improve over time through feedback. The authors propose a novel evaluation setting where language models must continuously learn from an input-feedback sequence, with the goal of maximizing accuracy across a range of tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Continuous improvement of language agents from feedback
Phenomenon defined by authors: Yes
Phenomenon definition: "to evaluate LLM agents' ability to improve themselves over an input-feedback sequence. StreamBench simulates an environment where LLM agents are exposed to a sequence of users' natural language requirements and feedback."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task requires language models to continuously learn from an input-feedback sequence, improving their performance over time on various downstream tasks like text-to-SQL, coding, medical diagnosis, and question answering.
Task item definition: A single item consists of an input in natural language (e.g., data requirements, symptoms, questions), the agent's predicted output, and binary feedback indicating correctness.
Task definition detail: The benchmark focuses on sequential improvement rather than static performance. 
Task source: Real task examples (e.g. GitHub issues)
Task source detail: The benchmark integrates seven existing datasets (Spider, CoSQL, BIRD, DS-1000, ToolBench, DDXPlus, HotpotQA) but transforms them by assigning time steps to create streaming sequences.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: The tasks come from real-world applications but the feedback mechanism (binary correctness) is simplified.
Dataset size: 9,702
Dataset metadata available: Yes
Dataset metadata detail: task type, input-output format, binary feedback signal
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: mean
Metric subscores: Yes
Metric subscores detail: Split by tasks and datasets
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They test the streaming methods across multiple tasks/datasets, use random seeds to verify robustness in sequence ordering and they do ablation studies to validate key components of the method.
Taxonomy: Agents / 
New taxonomy: General Application / Updating