TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale
Authors: Malgorzata Gwiazda
Yifu Cai
Mononito Goswami
Arjun Choudhry
Artur Dubrawski
Venue: ICLR 2026 Poster (2026)
Keywords: Time Series Reasoning
Scalable Benchmarking
AI Agent
Multimodal Learning
Primary area: datasets and benchmarks

Abstract: Large Language Models (LLMs) have shown promising performance in time series modeling tasks, but do they truly understand time series data? While multiple benchmarks have been proposed to answer this fundamental question, most are manually curated and focus on narrow domains or specific skill sets.
To address this limitation, we propose scalable methods for creating comprehensive time series reasoning benchmarks that combine the flexibility of templates with the creativity of LLM agents. We first develop $\texttt{TimeSeriesExam}$, a multiple-choice benchmark using synthetic time series to evaluate LLMs across five core reasoning categories: pattern recognition, noise understanding, similarity analysis, anomaly detection, and causality. Then, with $\texttt{TimeSeriesExamAgent}$, we scale our approach by automatically generating benchmarks from real-world datasets spanning healthcare, finance and weather domains. Through multi-dimensional quality evaluation, we demonstrate that our automatically generated benchmarks achieve diversity comparable to manually curated alternatives. However, our experiments reveal that LLM performance remains limited in both abstract time series reasoning and domain-specific applications, highlighting ongoing challenges in enabling effective time series understanding in these models. $\texttt{TimeSeriesExamAgent}$ is available at https://github.com/magwiazda/TimeSeriesExamAgent

Published as a conference paper at ICLR 2026
EEMASERIESEXAMAGENT: CREATING TIME SERIES REASONING BENCHMARKS AT SCALE
Malgorzata Gwiazda It, Artur Dubrawski
Yifu Mononito Goswami", Arjun Choudhry?,
'Tehhicaa University of Munich, Munich, Germany ma www.ooortaa....m gwi a azda@tum de
2 Autonn Lab, School Computer Science, Carnegie Mellon University, Pittsburgh, US (yifuc, mgoswami, arjuncho, awd] (css
ABSTRACT
Large Language Models (LLMs) have shown promising performance in time series modeling tasks, but do they truly understand time series data? While multiple benchmarks have been proposed to answer this fundamental question, most are manually curated and focus on narrow domains or specific skill sets. To address this limitation, we propose scalable methods for creating comprchensive time series reasoning benchmarks that combine the flexibility of templates with the creativity LLM agents. We first develop TimeSerr r es multiple-choice benchmark using synthetic time series to evaluate LLMs across five core reasoning categories: pattern recognition, noise understanding, similarity analysis, anomaly detection, and causality. Then, with TimeSer ieeeriisssss Exseeemmaeeeeeeeee automatically generating benchmarks from real-world datasets spanning healthcare, finance and weather domains. Through multi-dimensional quality evaluation, we demonstrate that our automatically generated benchmarks achieve diversity comparable to manually curated alternatives. However, our experiments reveal that LLM performance remains limited in both abstract time series reasoning and domain-specitn applications, highlighting ongoing challenges in enabling effective time series understanding in these models. at hthttps:///ithhi com/ ma azda htt.aaaaaaeemeeeeeeeeeaaaaeeeeee
we scale our approach by
is available
INTRODUCTION
Recent studies have successfully applied Large Language Models (LLMs)) to time series analysis tasks
including forecasting, anomaly detection, and classification [ 8 19 50 51). These promising results raise a fundamental question: do LLMs possess genuine reasoning capabilities about the abstract concepts underlying time series data? Can they recognize trends, distinguish signal from noise, or understand causal relationships without relying on Nomainn-pecitfe shortcuts? Existing benchmarks designed to evaluate such capabilities face significant limitations- they are manually curated, expensive to extend, and typically focus on narrow domains or specific skills 444 33). This creates a practical barrier for researchers and practitioners who need comprehensive evaluation tools but lack the resources to construct domain-specifer benchmarks for their datasets.
To address this gap, we begin with simple proof of concept, and introduce TimeSeriesExan controlled benchmark which uses synthetic time series to evaluate LLM reasoning across five core categories: pattern recognition, noise understanding, similarity analysis, anomaly detection, and causality. Initial results reveal two key insights: first, templated generation provides viable mechanism for creating diverse and systematic evaluation questions; second, LLMs continue to struggle with abstract time series reasoning, in these controlled settings.
Equal contribution
Corresponding author

---

Published as a conference paper at ICLR 2026
Building on these insights, tackle broader practical challenge: how create benchmarks that reflect the domain-speent specific reasoning required in real applications, such as diagnosing arrhythmias from ECG signals evaluating volatility regimes in financial markets? The key obstacle is that domain experts lack the time to manually construct comprehensive benchmarks, making expert- driven approaches impractical at scale. Inspired by recent advances in agent-based benchmark construction (6) IS), we address this challenge by combining our controlled synthetic benchmark with an extensible, agentic framework for automated domain-specifie evaluation.
Experiments on multiple datasets spanning diverse domains reveal that (1) LLM performance varies substantially across domains, and (2) even state-of-thed -ftt-of--hh-rrrrte models struggle with complex reasoning tasks requiring integration of domain expertise and time series understanding.
Our contributions are as follows:
Foundational benchmark. We introduce TimeSeriesEie controlled evaluation framework that systematically assesses whether LLMs understand core time series concepts. This templated approach provides valuable insights into LLMs' reasoning capabilities while enabling scalable question generation, though it remains limited to domain-agnostic skills on synthetic data. We provide the set of benchmark questions along with the accompanying code FuuggingFace and Git
Scalable framework. Building on this foundation, we propose TimeSerressaie TimeSeriessxxamkgen.. which which combines the systematic nature of templates with the adaptability of LLM agents. Given any domain-specific dataset, SemeeeeiisEEamaee reasoning benchmarks at scale, integrating multi-perspectivv verification and optional human-in-the- loop refinement to ensure question quality and diversity.
automatically generates customized time series
nutti-dimensional evaluation. We validate our approach across five datasets spanning four do- mains and demonstrate its effectiveness for domain-specific fine-tuning. Our automatically generated questions achieve diversity comparable to buman-curated benchmarks.
RELATED WORK
Synthetic Time Series Generation The generation of synthetic time series with controlled behav- iors, such trends and cyclic patterns, is fundamental for constructing scalable reasoning benchmarks. common approach involves sampling from diverse random processes (18), such as Autoregressive Processes, which offer variability but lack control over specific patterns like cyclic behavior. To ad- dress this, (48) proposed a decomposition based method, generating desired patterns by incorporating cyclic components into an additive model on top of random processes. More recent frameworks, such as (16) and (2), also leverage synthetic data generation for model training and evaluation. TabPFN constructs synthetic regression and classification tasks from random function priors, while Chronos employs large-scale transformer- based time series generation to capture realistic temporal dynamics. We build upon these studies, through the design of the T'meSeriesExam benchmark, by having more diverse set of random processes and patterns, incorporating not only additive composition methods but also multiplicative and other forms of composition.
Domain Specific Time Series Reasoning Benchmarks The task of creating domain-specific time series reasoning benchmarks is challenging. Current Nomain-specine benchmarks usually have limited scope and poor extensibility, since their curation often relies on templates or expert annotation. For instance, ECG-QA (33) and ECG-Expert-QA (44) focus on ECG interpretation, while En- QA (45) targets industrial settings. Automatic benchmark generation is scalable alternative, but the quality and diversity of automatically generated questions is unclear. Without extensive verification, LLM-generated questions often require heavy manual curation (2),22), which is both difficult and time-consuming, undermining the primary advantage of automation.
Agents for benchmark creation LLM agents are autonomous systems which observe envi- ronment, use LLMs to reason, and act towards achieving a well-defined goal. Recent work has shown the promise of using agents for creating benchmarks automatically. Most solutions adopt
2

---

Published as a conference paper at ICLR 2026
Table I: Overview of time-series and multimodal datasets with curation and skill types (P-Prediction, R ---Reaannnnn PS-Practical skills). Prediction refers to supervised tasks such as forecasting or classification. Reasoning involves higher-level interpretation of time series signals (e-g.. trend recognition). Practical skills extend reasoning into domain-specinfe contexts (e.g., classifying volatility regimes finance). "+" represents that we can generate any number of samples, but we have already generated 3,000 of them.
Title
Multi-domain
Curation
Samples
Skill type
Fully Automatic
P
R
PS
Time MQA 21
200,000
/
x
Time-MMD
17,113
MT-Bench (
22,000
/
/
ECG-QA 33
414,348
key
Semmeerr1sssaaaaaeee.. (murs)
3000+
/ /
multi-agent pipeline with planning, generation, validation, and evaluation modules (6). For example, (42) integrates exploratory evaluation using reinforcement learning, while (6) takes a description of a natural language task as input. However, most of these approaches are not tailored to time series and struggle to generate questions conditioned on numeric data. A recent solution incorporates time series, but is limited to single-step design and lacks extensive verification 29)
LLM-as--a-Judee Evaluation LLM judges have enabled the research community to scale eval- uations and to assess problems that inherently difficult to capture with conventional metrics. However, LLM-as -L---asa--udge also introduces challenges related to bias. We address this in two ways. First, we adopt methods such G-Eval (26), which provide probabilistic results and thus improve both reliability and interpretability. Second, we account for intra-model bias, which is the tendency of single LLM exhibit systematic preferences, by drawing on the idea of panel-based evalua- tion (39). Specifically, we aggregate judgments from multiple LLMs, which reduces the influence of any individual model's 's bias and yields more stable scores.
TIMESERIESEXAN AND
3.1 TIMESERIBSEXAN: BUILDING SCALABLE BENCHMARK USING TEMPLATES
To begin, we investigate LLMs' understanding of fundamental time series concepts in a con- trolled experimental setting by introducing a manually curated, configurable benchmark that we call T TimeSerii esExam.
To illustrate our approach, we present a proof-of-concept (PoC) showing how scalable benchmarks for evaluating LLMs' time series reasoning can be built from configurable templates. The aim is to demonstrate that template-basee based design enables systematic generation of diverse, controlled "exams" that probe specific reasoning skills. Our hypothesis is that once small set of well-designed templates exists, new benchmark items can be generated automatically by varying parameters and contexts. To test this, we introduce LimeSeriesEss tasks, in which we make two simplifying assumptions: (I) templates are created manually, and (2) evaluations conducted in controlled synthetic settings where data properties are fully known.
curated benchmark of fundamental time series
Composition. TimeSeri eesExam systematically assesses whether LLMs understand basic time series patterns such as trends and seasonality (pattern recognition). the concept of noise and other time series concepts in the presence of noise (noise understanding). It also evaluates LLMs on three different reasoning tasks: identifying abrupt deviation from "normal" behavior (12) (anomaly detection). comparing and contrasting statistical properties time series (comparative reasoning). reasoning about causality, specifically Granger Causality (4 (causality). As shown in Table 2 each category is further divided into sub-categories that represent more specific concepts within the broader category.
'The CiK henchmark contains 71 tasks, with the number of samples treated configurable hyperparameter.
3

---

Published as a conference paper at ICLR 2026
Base Patterna
Compastton
Nobe Ancrrutis Hotes
NiW Polla
Non-periedis
Addiive
Uvr Batue
Questier: The ghar aeries
-
cunee is 7essc sekes SURoway?
Perfedi
Doample: ciundualyy gererated
Muttplicathe
Lobes
(Dptionall Refevant Concepts:
(Optional) Hint: shand laraa
tist stat onar
the time
Morsfoe Processes
Siquential
Model Response.. it foa patt tnd Sugplaaly eongre teen and apriaaann
pan corioney
Famern Secogritin
Subcaragery: Statinerity Detection
Figure 1: (Left) Time Series Curation Pipeline: The composition model generates controlled synthetic time series step-by-step. The pipeline enables diversity by combining different components to create numerous synthetic time series with varying properties. (Right) Each template evaluates a specific category, and includes a question, list of options, example question and answer pair for in-context learning, and optionally a hint and descriptions of complicated technical terms. Here, GPT-4o showcases its ability to transfer visual understanding and time series concepts into effective reasoning.
Question Templates. TimeSeri esram comprises over 100 unique templates, carefully cu- rated in collaboration with time series experts and cross-verified for accuracy, that can be used to generate any number of random questions. Each template (Fig. (Right) evaluates a specific (sub- Jeategory (c.g., pattern recognition), and comprises of a question (c.g., "Is this time series statiinaary "), a list of options (e.g.. " (A) Yes, (B) No "), and example question and answer pair for in-context learning. Each template comes with a hint which breaks down complex questions into simpler steps and textual descriptions of complicated technical terms. By incorporating these relevant concepts, we can isolate an LLM's ability to understand time series concepts (e.g.. whether the mean and variance remain constant) from its understanding of complex technical jargon (e.g., stationarity). Each option (c.g. (A) Yes") is linked to synthetic time series generator (Fig. (Left) that produces a random time series as if the current option were true (c.g., a random stationary time series). This allows us to generate random but accurate time series at scale.
Generating Questions. We generate different questions from the same template by systematically
varying the correct option and producing synthetic time series conditioned on the template and the correct option pair. Our simple and scalable approach, illustrated in Fig. [ Left), involves sampling small number of base patterns from a predefined pool and combining them using composition function. Base patterns can be periodic (e.g., sine function), non-periodic (e.g., linear increasing
Table 2: Example template questions for different reasoning tasks. Each subcategory covers specific aspect of time series understanding, guiding the model to reason about comparative, anomalies, and causal relationships.
Category
Suhratogory
Example question
Tramd
What the most likely lincor trend coefficient the given time series?
Cyclcc
How does amplitade change from the beginning the and?
The time series sine puttern.
Patter Recognition
Stutionarity
Ix the given time series likely stationary after remoning the cycle compoment?
Regime Switching
Based on the given time series, huw different regimes there?
Statstical properties
Is mean stable time the time series?
Rundom procosses
Doves the following time saries exhibit a moan reversion property?
White Noise
the gives series white process?
Noise Undasstading
Random Walk
Is the time series likely be random walk process?
Signal/ Noise Ratio
You are given time series with the underlying putten but different nuise level.
Which series las ligher maguitude
Anomaly Detection
The following
are the likely types
typas asomilies appearing differem time pointss
Shage
Despite the nonse, do the given two time series similor putterns?
Comparative Analysis
Distributionl
You are given time series which are generssed using a random
Are they likely
stme variance?
Causality Analysis
Granger Cansality
Granger causality between the two time series?
4

---

Published as a conference paper at ICLR 2026
function), or random time-iarying processes (e.g., AR process). Depending on the template's nature, the final step adds additive noise or anomalies using the anomaly injection process described in (12).
Improving Questions Iteratively. We use Item Response Theory (IRT) (27 to achieve finer grained control over the quality of randomly generated questions included in the T TimeSeriesExe
During rounds of iterative refinement, question parameters are optimized to maximally distinguish the abilities of the candidate LLMs. Algorithm details are provided in App. A.2 The PoC establishes two takeaways: (i) template-based generation yields diverse, controllable items across core reasoning categories, and (ii) modern VI.Ms still struggle on higher-order time series reasoning, which motivates a scalable pipeline for real datasets with minimal input from an expert.
3.2 TINESERIESEXAWRGER A SCALABLE DOMAIN-AGNOSS BENCHMARK CREATION TOOL
TimeseriesExamAgent
Generation
Verification
Output
Seomple templichee
Structure check
Input
cioss
Loading
GP
Generater
Ver ter LLM
SC
Natural longunge
Concept
Student LLM
Shuderr LLM
Figure 2: TimeSeriesExeee ExamAggtttttt architecture. The user provides exam-making instructions and custom dataset with minimal loading code. Agent outputs question templates Python functions generated by a generator LLM and filtered through three progressive stages of verification (syntax and output format check, validation by LLM judge, capability-aligner filtering). Arrows denote data flow, red ones show direction for rejected templates.
Building on the PoC in the previous subsection that surfaced persistent gaps in L.MMs' time series reasoning, we now focus on the practical need evaluate models on domain-specific datasets at scale and with minimal expert effort. We propose TimeSeris weteerrieseexmeeneeeeeeee a multi-agent framework that combines planning, generation, and verification to enable automatic benchmark construction on user-provided datasets.
Setup An overview of the agen- tic framework is shown in Fig. - Lmeeeeresseaamaeeeeeeee
consists
Input
Template
Sampled question
two components that iteratively refine
Danaset
del quericntuum vamplecintt to Län(QAPair):
Whar kind consucctio abnormality present rac?
generation outputs: Generation Agent and a Verification Agent. The Genera- tion Agent takes as input description of the natural language task T and a dataset D. The description T may include user guidelines for generation, contextual in- formation about the dataset, or other relevant instructions. User provides dataset class D that supports basic oper- ations such as querying the i-th sample. The Verification Agent employs series of techniques assess the robustness, relevance, and syntactic clarity of the generated questions.
# Question and option definition Requierments for time series
Donaset Hondier
R Finding
dataset _iirr__iitt) # lenopo_ poirs_ jist)) = num, somples
C)...
A)...
B) D)...
Deraframme) queryli)
neturn qo. pains,
dataset information, generates question tem- plates as Python functions that encode the question logic and support parameterized sampling of arbitrary instances.
Figure
3:
Given
T
Generation Motivated by TimeSeriestaee we generate question templates instead of samples directly, as shown in Fig. B providing scalability and robustness to human errors. Given a dataset D and task description T, generator I.L produces diverse set of templates conditioned on dataset
5

---

Published as a conference paper at ICLR 2026
structure and domain-spectiio concepts. Templates define both the question format and the logic for selecting relevant time series instances and computing answers. Further details on prompt design, template structure, and generation workflow are provided in App. B
Verification We observe that LLM-based generation frequently produces errors or irrelevant outputs, motivating the need for structured verification process. We propose a multistage verification and filtration process to check the accuracy and relevance of each template. If a template fails at any verification stage, is returned to the generation agent with feedback. The generation is iterative with maximum number of attempts, after which the ongoing template is permanently discarded to avoid excessive context length and cost from repeated failures.
Structure check We check whether the generated template can be executed successfully. This step isolates technical flows from content-based once, allowing for streamlined autonomous generation.
Content verification Certain aspects of quality control are particularly well-suited for using LLM-as- a-judge evaluation. We use an LLM verifier to assess the validity of each template. Details are in App. B.4
Aopability-Alignee Filtering Inspired by TimeSeri Exam,, which leverages IRT to enhance differentiability across questions, we adopted similar but localized framework that operates at the level of each template. To detect templates that generate overly simple or irrelevant exams, we evaluate them using set of test-taking LLMs with varying capabilities. This approach is supported by the educational theory, particularly the expertise reversal effect (20). A template is permanently discarded if weaker I.I.Ss achieve higher average accuracy than stronger models, as this typically indicates that the template is flawed or noisy rather than genuinely discriminative. A template is retained if performance scales with model capability, or all models perform poorly, since such questions may still capture genuine difficulty. We provide hyperparameters in App. B.5 and other design specifics in App.
4 IIMISERLESEX.MAGENN EAA.AAAAAEEEEEEEEEEEEEEAAAAA ANALYSIS
EXPERIMENTAL SETUP, RESULTS AND
4.1 ITERATIVE REFINEMENT ENABLES COST-EFFICIENT EXAM GENERATION
Most accepted templates pass verification within two refinement rounds, indicating that the generation prompts provide sufficient structure for stable template synthesis. Failed templates are discarded after small number of attempts, preventing degeneration into unstable feedback loops and maintaining predictable behavior across datasets. The staged verification pipeline improves reliability by filtering execution errors and formatting issues before semantic judging, concentrating evaluation on viable candidates. The generation process yields diverse, executable, and domain- relevant questions without manual curation. The analysis of failure modes in App. F2 shows that the remaining errors are systematic rather than random, which makes them diagnostic and correctable. Additional experiments on the sensitivity of the generation parameters are included in App.
The refinement loop also plays role in reducing the costs of template creation. Under default configuration, the generation and verification process for one correct template costs approximately $0.09 in L.M API tokens.
4.2 STATE-OF-THE-ART LLMS STRUGGLE ON EXAMS GENERATED BY TiMESERIESEXAMASE
First, we generate a set of questions for each the five real world datasets: PTB-XI. (40), MIT-BIH (31) MIMIC-IV Waveform (30), yahoo finance stock dataset (37), and WeatherBench 2 (36). In total, we have 209 samples for YFinance, 197 samples for MIT-BIH, 151 samples for PTB-XL, 205 samples for MIMIC-IV Waveform, and 95 samples for WeatherBench 2. We sample 4 or 5 instances per template. Thus, the difference in the number of generated samples is a result of the template filtering mechanism above.
We select candidate models to cover a diverse range of performance levels, as indicated by the Open- VLM Leaderboard (11). In Table 3 we find that while general-purpose multimodal models such as

---

Published as a conference paper at ICLR 2026
Table 3: Comparative performance of six vision-language models across medical (MIT-BIH, PTB-XL, MIMIC-IV Waveform (MIMIC-IV W)), financial (YFinance), and meteorological (WeatherBench 2) time series datasets. The results highlight dataset-specific strengths; nonetheless, all models achieve less than 55 mean accuracy, underscoring the difficulty of time series reasoning for current VLMs. The evaluation protocol is provided in App.
Dataset
Model
MIT-BIT BIH
PTB-XI XL
MIMIC-IV X
YFinance
MeatherBench2
Average
random guess
0.25
0.25
0.25
0.25
0.25
0.25
gpt-4o 17
0.416
0.424
0.385
0,586
0,389
0.440
0.555
Qwen2.5 Seon2.5-VL-Insteee In--tstuutt
3
0,411
0.490
0.439
0.572
0,368
0,456
Gemma-3 3- 3-27b-it G8
0.497
0.517
0.370
0.534
0.232
0.430
GPT.5
Gemini-2.5-Pro
0.614
0.457
0.400
0.624
0.453
0,510
GPT-5 perform well on weather-related questions, their performance is weaker on healthcare bench- marks. This contrast could suggest that the general reasoning ability does always transfer across domains, particularly when tasks require domain-speciin expertise or fine-grained interpretation of physiological signals. Secondly, We note that GPT-5 strictly outperforms predecessor, GPT- 4o, across all categories. This improvement confirms that our benchmarks effectively distinguishes between model capabilities within the same family.
However, while the newer state-of-te tfate-of-the-arr models (Gemini-2 12 5-Pro, GPT-5) demonstrate im- proved reasoning capabilities, the gains are only incremental. Even the strongest model achieves only 51.5% average accuracy. which highlights the limitations of current L.MMs. Excelling on domain-speciff benchmarks likely requires multimodal pipelines with explicit tool-use and struc- tured reasoning capabilities, such agentic systems. In App. G we highlight two main types of failure modes by studying responses from tested VLMs. Perception: As evidenced by our ablation on imput resolution (DPI) modality (text vision), the best way receive data depends the specific question. App. F44 F.3 provide further quantitative comparisons. Compositional Reasoning: Models do not fail simple recognition, but on problems that require multi-step reasoning.
4.3 TIMESER TRSEXAMAGENT GENERATES QUESTIONS WITH DIVERSITY COMPARABLE TO HUMAN-CURATED BENCHMARKS
We evaluate the diversity of questions generated by our framework against ECG-QA (3), template- based benchmark built on PTB-XL. Our aim is to show TimeSeriesExamaer comparable variety without manual template design. For each benchmark, we randomly sam- pled 50 questions and computed pairwise embedding distances. Embeddings were extracted using Qwen Qwen3 3are33--mmmdnnngg board
achieves
the top open-source model on the Hugging Face MTEB leader-
Table 4:
Diversity of questions measured by embedding and normalized Levenshtein distances. Higher values indicate greater variability in phrasing.
Mean + Std
Benchmark Dataset
Embedding
Normalized Levenshtein
ECG-QA TimeSSrrisSExam (ours)
0.207 0.079
0.519 0.157
0.301 0.070
0.542 + 0.039
As shown in Table our framework achieves a level of diversity that is broadly comparable to human-curated benchmarks. This suggests that it can capture range of question formulations without relying on handcrafted templates, which may help its scalability to other domains. For completeness, we include a visualization in App. to further illustrate this observation.
https://hugginngface http:/aacccccccccc.eecaeesscaaee swwces wacacces//tte/eee
7

---

Published as a conference paper at ICLR 2026
Table 5: Combined Quality Evaluation. Scores (I-10) averaged across four criteria. Rows are grouped by their original source tables.
Mean Result Domain Relevance
Dataset
Specificity
Unambiguity
Answerability
Finance Domain
FinMME
8.10
7.59
6.62
6.95
MIBench
6.88
6.11
8.35
7.29
T1 meSer esRxamneent (ours)
8.29
7.24
8.89
8.57
Medicine Domain
BCG-QA QA
5.60
5.77
8.
8.47
TimeSeriesExandee (ours)
8,43
8,40
9.00
9.10
We employed an LLM-as-a-jury approach using G-Eval, where panel of models (Gemini-1.3 1.5-Pro evaluated the quality of each question. ensure
GPT-3.5-Turbe and --. cost efficiency, we selected relatively weaker models, prior work shows this setup can maintain evaluation quality while mitigating intra-model bias (39). Each model independently assigned score from 10 based on four criteria. The aggregated results, reported in Table [ show that TimeSe eerreesxxass ExaieeseeEanenn outperforms ECG-QA across all dimensions, particularly in specificity and answerability. This indicates that our framework generates precise, well-grounded, and domain- appropriate questions.
4.4 LLMs TRAINED ON OUR GENERATED SAMPLES EXHIBIT TRANSFERABLE REASONING SKILLS ON ESTABLISHEDD DATASETS
Another way to assess the value of
is test whether its generated data supports transfer learning. As target model. we chose VLM 5ween2.5-V-33B-Inttee
Table Results of VLMM fine-tuning on the exams generated by
General: all incorrectly formatted responses are treated as wrong answers. Parsable:
only correctly formatted responses evaluated.
We first generated 2000 training samples using TimeSeriesfxamAçent based
Accuracy General
Method
Parsable
on the PTB-XL dataset, while
Random answering
34.9%
34.9%
testing was conducted on 12000 randomly selected samples from the ECG-QA 344 test split of MIMIC-IV- based QA pairs. This ensured striet separation of data sources, exposing the model to different data within the same domain. Training parameters are provided in App. D To isolate the effect of structural learning from actual gain in reasoning capability, in sime-tuned-counfonnde
Base
21.8%
34.6%
39.7%
42.3%
Fine-tuned
47.0%
49.7 %
setup the LLM was trained on the questions from Finance and Weather domain instead of ECG related once. Ablation study on training data efficiency can be found in App. F66
Table @ clear gains under the strict accuracy metric: the Base model achieves 21.8%. while fine-tuning on structurally similar but that the model benefits from format. Training on TimeSeriesExamagee 47.0%, corresponding to a 216% relative improvement over the base model. This confirms the model also gained ECG reasoning capability. The fine-tuned model also surpasses the Random baseline (34.9%), indicating that agent-generated questions provide genuinely useful supervision rather than superficial patterning. Overall, these results suggest that synthetic, agent- corettd exams improve decision quality.
exam lifts accuracy to 39.7%. This confirms signals and structural regularities in the MCQ -generated ECG exams lifts accuracy further to
4.5 CHOICE OF LLM DOES NOT INTRODUCE BIAS FOR TIMESERIBSEXANEE
Although we only use the LLM-as-a-Jury system for linguistic properties check, we conducted experiment confirm the consistency of juries with regard choice of LLMSs used, that
8

---

Published as a conference paper at ICLR 2026
pipeline is not subject to bias from a specific set of LLM. We generate exams using default generator LLM 4auude-4-sonned and evaluated using 3-model juries drawn from a fixed pool of LLMs (Gemini-2.0, Deepseek-V3. GPT-3.5 Turbo. Qwen-2.5-V6 LLama-3.3). We plotted inter-jury Pearson Correlation and Cohen's 's k in App. F.1 and observed that scores from most juries were moderately to highly correlated (Cohen's k 0.5). This confirms the consistency among different subset of LLMs and that our pipeline is not affected by bias arising from using a specific set of LLMs.
To confirm the choice of generator LLM do not introduce additional bias the agentic pipeline, we fixed the jury to the default configuration (Gemini-2.0,, GPT-3.5-Turbo, and Swee2.55VL-722 Instruct), and compared two different generator LLMs: Claude 4 and DeepSeex V3. 2. We specifically picked DeepSeek because it is disjoint from both the Jury and Verifier model families (Qwen, Gemini, GPT), ensuring strict independence. Unlike the previous experiment, changing the generator alters the specific questions produced. Therefore, we fixed the underlying data source to MIT-BIH and evaluated the statistical distribution (mean + standard deviation) of the jury scores across the generated artifacts. The results are presented in Table Although DeepSeek V3,2 exhibits slightly higher raw means, the results are statistically comparable, with the scores of both models falling within one standard deviation of each other across all categories. This confirms that our generation pipeline is robust to the choice of the state-of-ther of-tth-artttttt LLM.
DISCUSSION, OPEN QUESTIONS AND OPPORTUNITIES
is that the quality of the generated exams ultimately depends on the user instruction and coverage of the underlying time series dataset. For example, if important clinical instructions for the healthcare data set are absent, the resulting questions may not adequately capture the reasoning challenges faced in practice. In an offline sessions with cardiologists, we observed that when clinicians contributed targeted feedback during the prompt design stage, the resulting exams were consistently judged as more clinically valid and useful (See App. E This highlights the importance of structured collaboration between automated systems and human experts, especially in high-stakes domains such as healthcare.
Reliance on
Prompts A key limitation of
Demand for human-in-the-loop evaluation. Building on the previous observation, we integrated optional human-in-the-Bee in-the--oooop modules into LimeSeriesse ExamAgeet to facilitate more practical deployment. These modules allow domain experts to refine templates, validate generated questions, and iteratively improve exam quality. Although we received encouraging anecdotal feedback from clinicians and practitioners who interacted with the system, the influence of such human feedback pipelines could not be systematically tested within the scope of this study. A formal evaluation of how human involvement impacts benchmark validity and downstream model assessment remains an important direction for future work and the community.
Limited Evaluation Mode.
the current framework, questions are mainly evaluated by providing the time series as image input. In App. G.I we provide a few case studies to highlight how input modality of time-series could impact model answers. These studies highlight the need for an intelligent decision decision-making tools, such as agentic framework, to dynamically choose the most suitable representation. There is growing interest in agentic frameworks for time series analysis tasks (7, 49). Our benchmark provides natural testhed for such systems, since many of the generated questions require multi-step reasoning, or direct computation over numeric data. Enabling agentic AI systems autonomously write and execute code in order to answer our benchmark questions would provide valuable insights into their reasoning fidelity and robustness.
Natural extension beyond time series. Although we focus on time series data, the underlying framework is not inherently restricted to this modality. Our design only assumes access to structured data and domain-specifi prompts, making it extensible to other settings such images, tables, or even multimodal combinations of signals and text. We chose time series as a starting point because is highly structured domain with well-established industrial applications.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
This work first examined whether L.LMs can reason about fundamental time-series concepts. To address these questions, we introduced TimeSeriesExam, controlled benchmark for probing conceptual understanding, and SaneeSerieExamAgeee titioners to generate customized benchmarks from their own data. Our experiments show that while LLMs capture some surface-level patterns, they continue to struggle with more complex reasoning such anomaly detection. At the same time, benchmarks generated by TimeSer SesSSrriesie ExamAgentt match or exceed the diversity and quality of human-curated datasets, and can even provide useful finetuning signals for downstream tasks. These results suggest that automated, agentic benchmark construction can help make evaluation more adaptive and domain-relevant.
a scalable framework that enables prac-
REPRODUCIBILITY STATEMENT
hEtps://github com/ magwiazda/ T memerr eesFxamgeett The evaluation protocol is described in App. C.3 and the ap-
We release all evaluated datasets
pendix further provides implementation details of the pipeline, including prompts and configuration settings.
REFERENCES
[1] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language time series. arXiv preprint arXiv:2403.08815, 2024.
[2] Afshin Ansari, Amol Bhattacharya, Abhijit Kulkarni, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403. 07815. 2024.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song. Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-v) technical report. arXiv preprini arXiv:2502.1 13923, 2025.
[4] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[6] Natasha Butt, Varun Chandrasekaran, Neel Joshi, Besmira Nushi, and Vidhisha Balachan- dran. Benchagents: Automated benchmark creation with agent interaction. arXiv preprint arXiv:2410.225844 2024.
[7] Yifu Cai, Xinyu Li, Mononito Goswami, Michat Wilinski, Gus Welter, and Artur Dubrawski. Timeseriesgym: A scalable benchmark for (time series) machine learning engineering agents. arXiv preprint arXiv:2505. 1329/, 2025.
[8] Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint 1rXiv:2310.044000 2023.
[9] Jialin Chen, Aosong Feng, Ziyu Zhao, Juan Garza, Gaukhar Nurbek, Cheng Qin, Ali Maatouk, Leandros Tassiulas, Yifeng Gao, and Rex Ying- Mthench: multimodal time series benchmark for temporal reasoning and question answering. arXiv preprint 2tt.::2550336855.5.. 2025.
[10] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality. long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 2025.
10

---

Published as a conference paper at ICLR 2026
[11] Haodong Duan, Jumming Yang. Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference Multimedia, pages 11 11198-111 11201, 2024.
[12] Mononito Goswami, Cristian Challu, Laurent Callot, Lenon Minorics, and Andrey Kan. Unsu- pervised model selection for time-series anomaly detection. arXiv preprint arXiv:2210.0107., 2022.
[13] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: family of open time-series foundation models. arXiv preprint arXiv:2402.03885, 2024.
[14] Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: journal of the Econometric Society, pages 424-438, 1969.
[15] Gauthier Guinet, Behrooz Omidvar-Teerrni, Anoop Deoras, and Laurent Callor. Automated evaluation of aeurened language models with task-specific exam generation. arXiv preprint arXiv:2405. 1124005.1322. 1322,, 2024.
[16] Noah Hollmann, Samuel Müller, Johannes Haug, Patrick Schramowski, and Kristian Kersting. Tabpfn: A transformer that solves small tabular classification problems in second. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[17] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.211276 2024.
[18] Aya Abdelsalam Ismail, Mohamed Gunady, Hector Corrada Bravo, and Soheil Feizi. Bench- marking deep learning interpretability in time series predictions. Advances in neural information processing systems, 33:6441 16452, 2020.
[19] Ming Jin, Shiyu Wang, Lintao Ma. Zhixuan Chu, James Y Zhang, Xiaoming Shi. Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint 1r00v2231000128,, 2023.
[20] Slava Kalyuga. Expertise reversal effect and its implications for tailored instruction. Educatiinal psychology review, 19(4)5009-539 2007.
[21] Yaxuan Kong, Yiyuan Yang, Yoontae Hwang, Wenjie Du, Stefan Zohren, Zhangyang Wang, Ming Jin, and Qingsong Wen. Time-mqa: Time series multi-task question answering with context enhancement. arXiv preprint arXiv:2503.01875. 2025.
[22] John Patrick Lalor and Pedro Rodriguez. py-irt: A scalable item response theory library for python. INFORMS Journal on Computing. 35(1):5-13, 2023.
[23] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. detrieval-augmentte generation for knowledge-intents intensivee nlp tasks. Advances neural information processing systems, 33:9459-9474, 2020.
[24] Patrick Lewis, Fthan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint vrXiv:2005.11401, 2020.
[25] Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Prabhakar Kamarthi, Aditya Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, et al. Time- mnnd Multi-domain multimodal dataset for time series analysis. Advances Neural Information Processing Systems, 37:77888-77933 2024.
[26] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint 2331::2033.1634, 2023.
11

---

Published as a conference paper at ICLR 2026
[27] Frederic M Lord and Melvin R Novick. Statistical theories of mental test scores. IAP, 2008.
[28] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2633 2008.
[29] Mike A Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, and Tim Althoff. Language models still struggle to zero-shot reason about time series. arXiv preprint (1t::2404.117557 2024.
[30] Benjamin Moody, Shaoxiong Hao, Benjamin Gow, Tom Pollard, Weixuan Zong, and Roger Mark. Mimic-iv waveform database. PhysioNet, 2022.
[31] George B Moody and Roger G Mark. The impact of the mit-bih arrhythmia database. IEEE engineering in medicine and biology magazine, 20(3):45-50, 2001.
[32] Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning compress prompts with gist tokens, 2024.
[33] Jungwoo Oh, Gyubok Lee, Seongsu Bae, loon-nyoung Kwon, and Edward Choi. Eeg-qa: comprehensive question answering dataset combined with electrocardiogram. Advances in Neural Information Prucessing Systems, 36:66277-66283 2023.
[34] Jungwoo Oh, Gyubok Lee, Seongsu Bae, Joon-myoung Kwon, and Edward Choi. Eeg-qa: A comprehensive question answering dataset combined with electrocardiogram. Advances in Neural Information Processing Systems, 36, 2024.
[35] OpenAI. Openai 03-mini system card.
c3 mi ni system-cardd...oo January 2025. Accessed: 2025-08-22.
[36]
Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russel, Alvaro Sanchez-Gonvalez, Vivian Yang, Rob Carver, Shreya Agrawal, Matthew Chantry, Zied Ben Bouallegue, Peter Dueben, Carla Bromberg, Jared Sisk, Luke Barrington, Aaron Bell, and Fei Sha. Weatherbench 2: A benchmark for the next generation of data-driven global weather models, 2023.
[37] Ranan Roussi. yfinance: Yahoo! finance market data downloader. https://github/ com/ 2017. Accessed: 2025-08-22
ranaroussi/yfinant
[38] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint 2rXhv:2503.19788 2025.
[39] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating Ilm generations with panel of diverse models. arXiv preprint arXiv:2404.18796 2024.
[40] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, large publicly available electrocardiogrrahy dataset. Scientific data, 7(1):1-15, 2020.
[41] Qingyue Wang, Yanhe Fu, Yanan Cao, Shuai Wang, Zhiliang Tian, and Liang Ding. Recursively summarizing enables long-term dialogue memory large language models, 2025.
[42] Wanying Wang, Zeyu Ma, Pengfei Liu, and Mingang Chen. Testagent: A framework for domain- adaptive evaluation of lims via dynamic benchmark construction and exploratory interaction. arXiv preprint arXiv:2410.1 1507, 2024.
[43] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for software developers generalist agents. arXiv preprint arXiv:2407.16744 2024.
12

---

Published as a conference paper at ICLR 2026
[44] Xu Wang, Jiaju Kang, Puyu Han, Yubao Zhao, Qian Liu, Liwenfei He, Lingqiong Zhang, Lingyun Dai, Yongcheng Wang, and Jie Tao. Ecg-expert-qa: A benchmark for evaluating medical large language models in heart disease diagnosis. arXiv preprint arXtv:2502.17475 2025.
[45] Yilin Wang, Peixuan Lei, Jie Song, Yuzhe Hao, Tao Chen, Yuxuan Zhang, Lei Jia, Yuanxiang Li, and Zhongyu Wei. Itformer: Bridging time series and natural language for with large-scale multitask dataset. arXiv preprint arXiv:2506.20099; 2025.
qa
[46] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. 2022.
Advances in neural information processing systems,
[47] Andrew Robert Williams, Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Cha- pados, et al. Context is key: benchmark for forecasting with essential textual information. arXiv preprint arXiv:2410. 18959, 2024.
[48] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint wwXiv:2202.01577. 2022.
[49] Wen Ye, Wei Yang. Defu Cao, Yizhou Zhang, Lumingyuan Tang, Jie Cai, and Yan Liu. Domain- oriented time series inference agents for reasoning and automated analysis. arXiv preprint arXiv:2410.04047, 2024.
[50] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36:43322-43335, 2023.
[51] Nina Zukowska, Mononito Goswami, Michal Wilinski, Willa Potosnak, and Artur Dubrawski. Towards long-context time series foundation models, arXiv preprint arXiv:2409. 2024.
13

---

Published as a conference paper at ICLR 2026
A TIMESERIESEXAM DETAILS
Table 7: TimeSeriesExam meta-information breakdown for each category. Each question is associated with time series of length 128 time steps, and an example time series of length 64 time steps.
Meta- informafiotaionn Type
Anomaly Detection
Similarity Analysis
Noise Understanding
Pattern Recognition
Causality Analysis
o Questions
129
113
87
371
63
Stage questions series
A.1
TIMESERIESEXAM TARGET MODEL EVALUATION
Table 8: Accuracy of model on each category of TimeSeriesExam
Model
Similarity
Pattern
Causality
Noise
Anomaly
Overall
Gemma-3-211 -3-27B-I7
0.62
0.59
0.51
0.71
0.51
0.59
GPT-4o(17)
0.78
0.78
0.61
0.77
0.54
0.73
Owen2. 5-VLL 0.60
0.45
0.49
0.40
0.22
0.44
Gemini-2.5-Pre
0.78
0.76
0.57
0.65
0.59
0.71
Table 8
reports the accuracy TimeSeriesExan in five categories of reasoning. We evaluated several state-of-the-art vision language models (VLMs) following the protocol described in App. C.3 Overall, GPT-4o achieves the strongest performance, closely followed by Gemini-2.5-PPo while Qwen2.5-VL. lags significantly behind. Across categories, models perform best on relatively shallow tasks such similarity and pattern recognition, where surface-level cues often suffice. Performance drops sharply for more challenging categories. In particular, anomaly detection proves to be the most challenging one. This likely occurs because our anomaly detection questions are designed to go beyond easily identifiable visual irregularities (e.g., isolated spikes) and instead require compositional reasoning over temporal context, where subtle statistical deviations must be integrated with broader sequence behavior. As result, success depends not only on perceptual recognition but also higher-level reasoning about the data. By contrast, the causality subset is intentionally simplified to focus Granger-styl relationships with visually distinguishable patterns, reducing perceptual ambiguity and better isolating reasoning behavior. These results highlight that, while modern VLMs capture basic time series patterns, they fall short higher-order reasoning tasks. We provide case study in App. G.1
A.2 ITERATIVE QUESTION IMPROVEMENT WITH IRT
IRT is statistical framework that models the relationship between an individual's (or LLM's) latent trait (c.g., knowledge, ability) and their responses to set of items (c.g., questions on a test). It is valuable tool in exam development as helps identify weak exam items, ensures consistent scoring across different versions of the exam, and also allows tailoring the testing experience to the LLM'' abilities.
Our primary objective is to design TimeSeriesExan where each question can maximally distinguish the abilities of the candidate LLMs. We use the two-parameter logistic (2PL.) model for this. Formally, for LLM j with ability 0,. and question with difficulty b, discrimination ability a, the 2PI. model defines the probability of a correct response as:
P(ry = 1 a,, (,,,b,,0) b,) + e e-a;(0,-h)
Each exam typically undergoes 1-3 rounds of iterative refinement. In each round, all candidate models take the exam. Based on their responses, we the parameters of Equation | [usinng maximum likelihood estimation (MLE). Then, we drop X of samples with the lowest sum of difficulty and discrimination ability. Finally, we randomly re- re-generate questions from the dropped templates. This iterative process is detailed in Algorithm and the hyper-parameters. of the fitting process provided in App. A.33 We demonstrate the increase in the differentiability parameter across the iteration rounds and provide an analysis of the trajectory in App. A.4
14

---

Published as a conference paper at ICLR 2026
Algorithm I Iterative Dataset Refinement with IRT and Resampling
Require: num iiterratttinss = 3, crtop_percentage 0.2, initial dataset Do
1: +- Do
2: for iteration = 1 to num_ rem_tterations do
3: Evaluate each candidate on D, and obtain the response set R = {ras Tsy
1 if candidate i correctly answers question j}
4: Fit the IRT model to obtain the discrimination parameters A = {az j E Questions} and
difficiulty parameter B {bj j Questions)
5: Normalize set A and B between 0 and I, and calculate score S = {by + aj i E Questions]
6: Find S' which is the score for samples that answered correctly the best model in the round
7: Find the index I {j a, Quantile(S' ddroppppppppppp paa__peecenage); where aj is less than the peo__eecentage quantile of A 8: for each j E 7 do
9: Resample new question g' from the same category as question j
10: Set D + g'
11: end for
12:
end for
13:
return D
A.3
IRT MODEL. PARAMETERS
The IRT models are fitted using library py-irt (22). Cay=0.9999, dropout=0.5, hidden=100
The parameters are epochs=2000, 1r=0.1, Irde-
A.4 AVERAGE SAMPLE DISCRIMINATION PARAMETER OVER ROUNDS
Discrimination Ability Across Rounds
0.085
Avergge Discnminnttnn Ablity
0.075
0.070
0.065
0.060
Rounds
Figure 4: The sample average discrimination parameter across rounds shows upward trend, indicating an improved ability of the questions to differentiate candidates with varying levels of ability.
15

---

Published as a conference paper at ICLR 2026
A.5 DROPPED DATASET DISTRIBUTION PER ROUND
Round 1: Dropped Categories versus Original Categories
Round 2: Dropped Categories versus Original Categories
Dataset
Dataset
130
500
300
Qemaliy Anayyla
bolee Understening
Accgestann
amaaryy Anayylx
Detectio
Cenality Andyola
Lindrstening
Sinlarty Anayylo
Round Dropped Categories versua Original Categories
Originll Datavet
Dropped
300
250
Ansimal Detection
Bose Understaning
Pettorn Recieaton
Anayys
Ortegoris
Figure 5: Dropped Dataset Distribution per round. Dropped category distribution per round generally mirrors the overall category distribution.
We observe in Fig. 5 that the proportion of dropped questions for each category is approximately uniform. The difference in number of questions in each category a result of different template curated.
16

---

Published as a conference paper at ICLR 2026
B TIMESERIESEXAMAGEEN DETAILS
B.I GENERATION WORKELOW
We rely stages of generation for the templates: planning and generating, inspired by the chain-of-thought (CoT) prompting(46).
Generation planning To provide relevant and diverse of templates, rely on a comprehensive list of domain-specific concepts. There are several ways our pipeline generates a list of concepts:
1. LLM generation:
User guidelines and dataset descriptions provided input an LLM, which proposes the concepts.
2. Web Search: We provide the option for generator LLM obtain concepts through web search.
3. Retrieval Augmented Generation: As an option, the user could also provide relevant file from which the LLM reads and generates concepts(2))
Template generation As input to our generator, the following components are provided:
User-provided guidelines: document containing the user's goal specific requirements,
Dataset description: a list of columns and example values with ranges from the dataset, with a short usage example,
List of concepts: generated in previous step. For each template, our pipeline will choose concept at random to ensure diversity.
few-shot examples presenting required structural elements (5). These templates are constant and of the correct template. Their content is therefore not relevant.
Example templates:
as they describe only the format and structure
B.1.1 GENERATION PROMPT
Here is the goal of the exam questions:
[user_ [user_info_textt
Hare are sample concepts on which you can base your question ganeration: coonoept_oovversatte
Jae the concept nimbered (concept_no) template.
to guide the design of your questioon
the description of the dataset you will generate the question: (datasat_deacribn)
In your template, use the provided 'user_dataset' object. Use its 'quezy (index) method to load relevant time seriea data.
Do not select time series zandomly. Birst, formulate the question, and then choose time series that fits its logic and reasoning needs.
Generate Iunction based question Lemplate now.
1.1.2 TEMPLATE CONSTRUCTION
template implemented Python function. contains formatted string for the question and options, together with parameters that control how many questions to generate. For each question, the template samples a pair (I, 2;) from the dataset D, where Iz 1=nd is a time series with observations and d variables, and z; is an auxiliary array containing metadata or labels related to the series. Later, a rule-based calculation performed determine the correct answer from the time series. For example, in a trend-detection template, the function computes the linear trend coefficient of x; and selects "Yes, there is a linear trend" the coefficient exceeds a specified threshold. In addition to such signal-derived logic, templates can also utilize the auxiliary property 2;, effectively transforming classification problems into question-ame answer form. For instance, an ECG series in of
the dataset labeled exhibiting atrial fibrillation, the template the multiple-choiee options. Each generated sample consists the question, its options, the correct answer, and more associated time series represented numerical values.
present this label
17

---

Published as a conference paper at ICLR 2026
B.2 SUPPORTING MECHANISMS FOR GENERATION
Detractors In addition, the mechanism of plausible but incorrect answer choices was implemented. The LLM prompted reflect on possible mistakes that the test taker might make while solving the exam. Using this knowledge, misleading, incorrect option choices can be generated.
Context Condensation A common issue we encountered in the framework was context window overflow during exam regeneration. To mitigate this, we applied context condensation, which reduces the number of tokens while preserving essential information. generates templates in conversational manner. The process begins with generation prompt, followed by a message containing the generated exam. errors occur or the exam is rejected during verification, the feedback and regenerated exams appended the conversation. Several context condensation techniques exist, such as windowing (4) and context compression
our setup, the agent
(32). We adopt a summarization-ber based method (41 43), which has shown strong results in prior work and fits
case. Specifically, we summarize non-recent pairs of failing exams and error messages into short descriptions that highlight the issues encountered. These summaries provide the LLM with concise feedback, supporting the generation of higher-quality templates.
RAG/web search our setup, LLMs can also make use of external knowledge sources. The agent has two options: (i) Retrieval- Aurrevel-- Generation (RAG) tool (24), which pulls information from a structured corpus such as a PDF with domain materials, and (ii) web search, which provides access more up-to-date or niche information. The retrieved content is then used to support concept generation, helping the model produce more accurate and comprehensive outputs.
B.3 MITIGATING CIRCULARITY IN LLM-BASED VALIDATION AND EVALUATION
To mitigate the risk of circularity when using LLMs to evaluate artifacts generated by other LLMs, we incorporated strict safeguards into both the generation methodology and the subsequent evaluation process.
LLM Verifier In framework, the verifier is intentionally restricted shallow linguistic and structural checks (e.g., domain relevance and ambiguity detection) rather than deep correctness verification, reducing the risk of propagating model-specific reasoning biases.
-allbel Filtering Our approach does not introduce bias caused by test-taking LLMs' performance only the occurrence of an inverted performance gap results in template rejection. This filtering mechanism does not affect the performance or ranking of other models.
Additionally, we use disjoint model families for erucial roles in TimeSeri ies ExamAgent avoid same-family optimization effects and to better approximate an external auditing setup.
B.4 LLM VERIFIER SPECIFICS
For each template, we use an LLM to evaluate the generated question. Specifically, we ask:
Is the question relevant to the given concept?
Does answering the question require the provided time series?
Are the question and answer free from ambiguity and bias?
We use binary scheme: a template must pass all categories to be accepted. Failure in any of the categories above triggers regeneration, ensuring robustness.
B.4.1 VALIDATION PROMPT
You ace an expert validstor of question templates involving reasoning over [exam_type] time series data. You are given an exam question cemplate:
(exan_tomplate)
18

---

Published as a conference paper at ICLR 2026
Your task validate question template using follewing criteris:
1. Ta the question relevant to texam_typa time seriea analysis?
2. Mould you need the time series itaelf to anawer the question?
3. Are there ambiguity in the question its answer?
If the
all is YES NOSTLY
only the number 1.
Tr the anawer to either is 80, return your objections.
include any additional text then) describe your objections.
19

---

Published as a conference paper at ICLR 2026
B.5 FRAMEWORK HYPERPARAMETERS
In this section, we list all the hyperparameter used for our agentic workflow.
1. Generator LLM: the LLM used to generate concepts and the correspond- ing template. We used claude-somnet-4220255665 (initial generation with I Sarngg_ffoortttmeeeeeeeee are excluded from subsequent evaluations.
As a result, models developed by Anthropic
2. Concept L.M:: the LLM used generate concepts. We used gpt-40-0024-00-06.
3. Structure verification: Each template is sampled k = 3 time to create k exam questions. Potential syntax errors, incorrect dataset querying, and bad output structure is detected, ensuring compatibility with the rest of the pipeline.
4. Verifier L.M:: the LLM used to verify templates. We used 4p1-40-2024-08-06.
5. Student LLMs: the student LLMs we use to check the exam differentiabilityy Currently we have two student LLMs: stronger: gpt-4o-2024-088
and weaker: gpt-4o-mini. The strength of model is determined based on the OpenVLM leaderboard. Moreover, family of student models was unified minimize architectural variations while providing wide enough capability gap. For each template under evaluation, students receive the same set of samples to answer.
6. Exam type: We generating the data connected to specific domain. We used "ecg". "medicine", "finance", "weather" and "mechanical".
7. Few-shot examples: 9 templates prepared beforehand were used to present the desired structure the generator LLM. For each generation, templates randomly selected and included in the prompt as few-shot examples. This introduces variability into the generation process, enhancing diversity.
8. Regeneration patience: Templates requiring multiple regeneration cycles were generally of lower quality. In our experiments, we set a maximum of 3 regeneration attempts.
B.6 EXAMPLE OF NATURAL LANGUAGE DESCRIPTION
want to create time serias exam testing ng modal understanding of finance time series data.
To load the data, use the provided "'uzer_datess dataset 11 object.
vev time series aome from Yahoo Finance, include closing price of stock. Interval between semples is day.
Make sure that the length of time saries (tatal number of aamples of time series) does not excide 3000.
Please make sure that exams cannot be answer without timeseries!
20

---

Published as a conference paper at ICLR 2026
B.7 EXAMPLE OF QUESTION TEMPLATE
def question (nun_samples, verbosenpalsej: hyperparametare
days" 23, mmax_series_lenghht 3003, strenggh__hreee hhtreshhldd 0.3, "momentua_ 10,
quastion "Analysing
of [ticker] given time period, does price trend cemonstrate strong momentur. and suatainability, or does it show signs of weakness potential reversal?"
options I
"The trend shows strong momentum with consistent directional movement and minimel pul pellbacks, suggesting the trend is likely to continue.",
with frequent reversals and inconst stent momentum, auggeaLing potential Lrend change.",
"The
signa
"The shows making direction unclear.", "The price movement shows no clear tread pettern, indicating ranging or sideways market."
signals
alternating periods scrength weakness,
def calculate traaatt__teed__seeeeees stenghhhstenghh((priiesss ienipeisss hyperparameters dayy return None, None
np.diff (ppdiff(prices) / prices(:-1)
Calculate positive_days np.sum(reture) > 0)
consistency
rogative_daye np.sun(returns
total_days = len (returns)
ricciiinnal contioool_nnnssttennyy
(positive_doyes negative_daya) total_days
Calculate average magnitude avg_abs_reture sp.mean aas
1 Calculate trend persistence iconsecutive moves in same dí rection! on) consecutive_mover
current_at
for i in range (1, len (selurna))
if ap.sign cusrent_streak +s
ip.sign [i-1]1:
elae:
consecutive_move htcp__mveeeeeppepee
ourrent aurrent_atresk strekk
wew_vee__eeeeeeeeeeeeeeeeeeeeeee append (ourrent_streak)
svg_streak np.mean www.sicusseeeeeeecc.. moveeses))))_
max_streak max mox
i Determine overall Urend direction overall 1 return (pricea prices(()) / prices
trend_direction Tup" eveuall_eeturn else "down"
"di ractiaiionl senerooonn_oomnitteeeee directional caliooaaannonssttnnono.oo
Lavg_aba_returne avq_aba_retusn
"avg_streak"; avg_streak,
"max_streak"; max_streak,
relural"_etuzn";; abs retural
"trend_direction. crend_direccton
returns
paipsics
de wtw.:daaseet.eeddaaaaeeee.
attempted_tickere ()
lan Iqp (qp_pcirs) num_samples:
if verbose:
print (E" [Question 6] Genezating question (leniqe_psies)) (num_eampies)")
# Salect a ticker that hasn't been attempted svaileble_tickerz
in dÉ.index if aickest
if not available_tickera: break
21

---

Published as a conference paper at ICLR 2026
ticker_id id randem.chcica htg.
attempted wwtpeeddddttteeeeeeeeeeeeeeeeeee (ti
ticker df. [f.loc(ticker_it 'ticker')
prices
(pricest hyperparamaters ("min__rend_daya"" contimue
# Limit series length
(pricesl > hyperparameters ("mx__errie__eenghh""
start_idx random, randint (pri hyperparameters ("max_oerieee le_gghhhh""))" prices [sartttiisstttsttttttt
hyperparameters "max__erree_lengga""
# Select aubset for analysis (to nake question more tocused)
saalysis__enghe mien (pricee), (lea randon. 200) start_idx cendemtrendint (0, (pricess) analysis_length) praiacin_pricns
+ analysis_lengch)
metrics, returns - Leeuiaateeeend_teeeee bes.adeeeeeeeeeeeeeeeaaaaaalaiap
trend_metrics
continue
Jatermine based trend strength metrics
strangth_score = trend_metrics [
(tredddd__mett metrrica ("avg_stieak"] / 1.0) + 0.3
(treeneeemmttriee "overall_return"" 10, 1.0)
if strength_scon (yperprraatteer(ttaaaeerrrrrrere max_streak"] >- 5:
and troni_matries
options[0]
elif streagth_scere < 0.4 er trend_aatrice ("dreeetttiooooaccssttttaag 0.6:
answer options[1]
elif 0.4 <e strengta_score hyperparameters Prrem__sttrngghhhtheeeooo. answer options
else:
answer = options[3]
question_text - question. format (tickersticker)
go_pairs.appendl
"questien": quastion_text,
"optiona": options,
"answei"; answet,
"tickes"; ticker,
"ta": snalysis_prices, "relevent_ sallsant_conooptat: ("Volume-Prion Correlation", Strength Analysis",
"Price Nomentun")
"demain"; "finance",
"eetracto__tppes:: ["Incorrect trend interpretation", "Misunderstandding momentum signals"], ],
"question_type": "multiple_choiee
Roomat_hinte "please answer the quastion and provide the correct option letter, e.g., [A], [B], (C), [R], and option content at the end of your anawer. All informat please provide your bast guess.",
need the question is given. If you
return go_psirs
22

---

Published as a conference paper at ICLR 2026
B.8 EXAMPLES OF GENERATED QUESTIONS
ECG Question Example
Q: Analyze the P-wave morphology and amplitude characteristics in this recording. What atrial abnormality is present?
A. RAO/RAE: Right atrial overload//ellargement with prominent P-waves
B. LAO/LAE: Left atrial overload/enlargement with bifid P-waves
C. Normal P-wave morphology with no atrial abnormalities
D. Absent P-waves indicating atrial fibrillation
answer: LAO/LAE: Left atrial overload//nlargement with bifid P-waves
E2G-2
30
0.0
0.5
15
2.0
2.5
3.0
3.5
4.0
5.0
6.0
6.5
7.0
7.5
8.0
8.5
0.0
9.5
23

---

Published as a conference paper at ICLR 2026
E2
1.5
24

---

Published as a conference paper at ICLR 2026
Finance Question Example
Q: Based on the daily closing price data for MAA over the past 2000 trading days, what does the Relative Strength Index (RSI) analysis reveal about the stock's momentum condition the end of the period?
A. The stock overbought territory with RSI above 70, suggesting potential selling pressure. B. The stock is in oversold territory with RSI below 30, suggesting potential buying opportunity.
C. The stock shows neutral momentum with RSI around 50, indicating balanced buying and selling pressure.
D. The stock shows strong upward momentum with RSI consistently increasing but not yet overbought.
answer: The stock shows neutral momentum with RSI around 50, indicating balanced buying and selling pressure.
168
100
15
164
104
90
250
250
750
3000
1250
1500
3752
2000
Time (drys)
25

---

Published as a conference paper at ICLR 2026
C EVALUATION DETAILS OF GENERATED RESULT FROM TIMESERESSXAMMAGENT
C.I LLM-AS-A-JURY
We evaluated set of generated questions using the LLM-as-aajury approach. Below are example criteria we applied for ECG evaluation:
1. SPECIFICITY
multiple choice questions. The queationa normally come together with relevant time series data, which should analised so anawer the question correctly, It is not includded in curreatly evaluated samples.
expect judge evaluating epecificity
Evaluate the specificity of the gonerated sDG miltiple-choien question.
& good question should Lazget single phenomenon.
Evaluation steps:
1. Read the question and all answer options.
2. Petermine if the question
single, zlearly defined Res finding clinics.
interpratation.
Assess the ratio unique medical
general
1. Fenalize if:
question is overly broad open-ended (e.g., a0e normal?").
The wording leaves diagnestic interpretation unslear. The question cevers multiple unrelated phenemens.
Score higheat if the question has one prectse focus (8.g., "Tg there 57 elevation in "ead V3 )*).
Seore 1-10 where:
10: Excellent specificity with clea", focused nedi cal terminal logy sargecing single phenomenen
7-9: Good siccificity but could
focused
4-6; Moderate specificity with some clarity isaues
specificity, broad, covers muitiple unrelated phenomena
Raapond with just
to 10, followed by explenation your score.
2. UNAMBIGJETY
expert judge evaluating the unambigulty The questiona normally come Logethez with selevant time series data, which should be analized answer question correctly.
miltiple-chotee questions.
not included currently evaluated samples.
Task: Evaluate if the question and answers can be objectively assessed wichout multiple interpretacions.
Evaluation steps:
gons.... 2. Determine if the question can be jojectively assessed. unsmbigueus.
question and
Check if
are
4. Penolize if:
The question uses subject Iive terms le.g., "Does this strangs?"). to multiple Intespretations. The question cannot. ba objectively answered.
The answers
good question should
objective (e-g.,
tachycardia?").
Score From 1-10 where:
10: Completely unambiguous and objective with crystal cleer question and
7-9; Mostly clear with only minor ambiguities
Moderately clear
anbiguous eiements
1-3: Highly ambiguous, subjeczive, or open to miltiple interpretations
Reapond with Just number from to 10, fo lowed by brie" explanation for your score.
3. DONAIN RELEVANCE
You are an expect !udge evaluating the comain reievance of D0G multiple-chor quest lons, questions normelly come together with
time series data, which should anelized to answer the question correctly. it is not includded in curreatly evaluated samples. madicine.
Taak: Evaluate the question actually pestains to
Evaluation criteris:
1. Does the question
medical and sce-specifis terminology?
2. In the question relevant to ECG interpretation and medical diagnonis?
question related incerpretation? 4. Does the question have proper redical context?
26

---

Published as a conference paper at ICLR 2026
good question should centain and pertain to SCC intorpratation.
medical (e.g.r "QRS, "arrhythmia, wave"!
Sccre from 1 10 where:
10: Highly relevent to B3G domain with extensive proper medical terminology
7-9: Good domain relevance with appsopriate medical terma
4-6: Moderate relevance but could be more medical speci fic
1-3: Foor medical
centains primartly non-madical terms
Reepond with just number fzom to 10, followed by a brief explanation for your score.
3. AMSNERABELITY
expert judge evaluating the answersbi ity RCG multiple-chetce questions. The questiona normally come Logether with zelevant time series data, which should be atalized currently evaluated samples.
not Taak: Evaluate if the question be answered pased BCS data analysis.
50 answer the question correctly.
Evaluation steps:
1. Read the question and all answer options.
answered aralysing RCG 3. Assess whether the question requires Lime secles analysis or could be answered without it.
betermine the question
Penalize
The queation aaks about non non-BCS BCG Iactors le.g., "Was the patient servous?"), The question can be answered without enalyzing the BCB time series data. The question is general and doesn't require specific analysis.
Score highest
question requires specitic atrial fibrillstien?").
analysis (e.g., data.
points if the question
answered
Secre from 1-10 where:
10: Requires specific, detsiled analysis and is fully answerable from the data
779: Mostly answerable from RCG data but could be more specific
some Limitations 1-3: Cannot be answered from BCG data or is too generaluunzelated
BCG Partially answerable
Respond with just number frem to 10, followed brief explenstion for your
C.2 T-SNE EMBEDDING PLOTS
Graused
Figure 6: t-SNE analysis of embeddings: (left) text-only vs. (right) text and time series concatenated together.
To visualize distributional differences, we applied t-SNE 28). which preserves local distances between samples. As shown in Fig. [6] questions generated by framework form more widely scattered distribution, confirming the higher diversity observed in Table
We the t-SNE algorithm using the scikit-learn implementation with the random seed fixed to 42, in order to ensure full reproducibility the dimensionality reduction results across different runs. The other parameters were set to default.
The text embeddings were generated using the Sentence Transformer model Jowen3-Embedding 8B), while the time-series embeddings were obtained from MOMENT-1-lare and averaged across
27

---

Published as a conference paper at ICLR 2026
leads multiple time series when applicable. The two vectors were then combined through direct concatenation to form a joint embedding.
C.3 QA SAMPLES EVALUATION PROTOCOL
All used models were accessed by API with LiteLLM Python library. The following API providers were used with default parameters:
Closed source models OpenAl API, Anthropic API
Open source models for rimeseriesfxxankeet generated exams Hugging Face Inference Providers API
Open source models for TimeSeriesExan Novita A
During the evaluation, the images of the plots were encoded with base64 encoding and provided to the models. Plots were created with DPI = 50. We used setup without context condensation.
https://novita.ail
28

---

Published as a conference paper at ICLR 2026
D TIMASERISSEXXMMAGENNTAABEED FINE-TUNING PARAMETERS
Hyperparameter
Value
Base model
Qwen2.5-VL-33--Innte
GPU setupe Frameworks
4*NVIDIA RTX A6000 48GB GPU
Hugging Face Accelerate,
DeepSpeed 7eRO stage
Train samples
2000
Warm-up steps
16
Batch size per device
Gradient accumulation steps Learning rate
5e-5
Optimizer
Adam W
Learning rate scheduler
Cosine
Weight decay
0.1
LoRA rank (r)
16
LoRA alpha
LoRA dropout
0.0
LoRA target modules
q_proj. k_proj, v_proj, o_proj.
gate_ ______roo. up_proj, down_pree
29

---

Published as a conference paper at ICLR 2026
E FEEDBACK IMPACT
One of the challenges we observed during question generation was the misuse of domain-specific jargon. Although the generated questions were grammatically correct, they sometimes included terminology that did not align with standard ECG practice. This can lead to confusion for clinicians, as non-standard phrasing undermines clarity and clinical relevance.
The following generated question contains terminology that was later identified as suboptimal:
Q: Examine this Lead II ECG recording and measure the QRS voltage amplitudes throughout the tracing. Based on the peak-to to-peak QRS amplitudes observed, what voltage abnormality present?
The clinicians noted that certain expressions in the question do not reflect standard ECG terminology. In particular, the phrase "peak-to-peak QRS" was considered inappropriate. To address this, the natural language description was refined by adding the following instruction:
Please frame your questions in way that is clear and natural for ECG specialists (i.e., adjust terminology accordingly).
Following this modification, a second round of consultation confirmed that the issue non-standard jargon had been resolved. An example of an improved question generated with the revised prompt is shown below:
Q: Based on QRS voltage amplitude measurements across all 12 leads in this ECG, which ventricular condition is most likely present?
30

---

Published as a conference paper at ICLR 2026
F ABLATIONS AND ADDITIONAL ANALYSES
F.1 LLM-AS-A-JURY SCORE CONSISTENCY ANALYSIS
To evaluate the consistency of jury-based scoring, we begin by selecting a diverse of open- and closed-source models (Gemini-2.0. Seepeee--33.2. GPT-3.5 Turbo, Qwen-2.5-VL, and Llama-3.3), We form all possible triplets from this set and compute jury scores using the same procedure as in T For each triplet, we aggregate the metrics into combined average score provide holistic view. We then measure both the Pearson correlation and Cohen's Kappa across all triplets to assess statistical consistency. The resulting correlations and agreement scores are shown in Figures and
F.1.1 PEARSON CORRELATION AMONG JURY
3-Tuple Jury Correlation: All Metrics Combined (Based on averaged per-question scores)
saspockkigemat ipp.
deeposelle nn- 3.0.mttts))
deesseek agenen owen
2.50
6.44
2.0
0.75
deepowelenge qpe thce
2.14
despoeelleppe
deeporekemisra Iana-333 3.3-
5:00
6.0
1:50
-0.5
compaeecii www.eeimaaaaaaaeee. owan
2
299
9.9%
1:42
desppedk leveni kanu33-
3.00
622
6.00
Junl
gamini 2.04ghh 3 duamtall
5:00
8:99
7.75
3.93
1.00
-0.25
gam ri-22 201 spt-35 lu--gonn-
3.00
6.44
6.70
6.70
6.34
0.25
0
3.0:
3.37
0.58
Coniii20iiizzai kanu33
2.00
6.00
3:35
0.52
-0.00
pemtn 0.0-ggarn
0.65
-0.75
get 3.5 Boeneera 1mna-3
2.04
6:76
6:70
GUO
6:30
DUM
3:22
3.93
3:22
3
LEE
spt Brem stral-
2274
11,00
0.04
3.50
Mesuovet a spen
droaddes spondel exttmmo
desrpeent agerross mastti
desponed aight Sotoreeee
despoeek msid Paaamt
der warkennk-oo verittal speen
Deerena mesercal Martt
appl B. Speat
persusi22. wnveentttaaeeeee frgeee
seriil diuppeten Aibrne
get- arrs www.ttaalaaaaoo Paper
get Kartitmmmnntt Hartte
milatal speees vstamta
dropcoee
pemiiss
Jury 3-Tuple
Figure Correlation across all jury-model combinations. We see consistently high (2 0.5) inter-rater correlation. This confirms that any possible triplets has consistent scores.
31

---

Published as a conference paper at ICLR 2026
F.1.2 COHEN'S KAPPA AMONG JURY
3-Tuple jury Cohen's Kappa: Combined
(Based on discretized per-question scores)
despouckigen weeeekkkggeeeedaee pp5-..--u-
despssatic ge 2.5 aaro 5.5--102
2:00
1.0
despaooke 2. 0-mitral-
8.59
0.42
8:56
de desppcolt-eeeee agpn- 3.5 S.-------------m-mmm has-mitrrl--
9.66
9.40
6.75
653
4.02
0.
conpaeoce ppppppppp fa-rqwon
0.30
L-33
deeporelennn
-0.6
Cohenns Kapps
desppeeke qaver lamo 9:4-9447
9:50
0:40
8.12
3.99
0.59
5.99
genorn 2 Dreget 3.5 fuu 8.3
8:50
0:30
8:68
0.57
0.32
0.33
gumiie 2. spt. miserall
gemenn Comitrra 9.39
0:02
8:00
0.58
9.45
0.90
9:64
9.57
8:3
gemiii 2 dum wwr.a--aaaaen
8.65
9:23
6.70
0.66
0.40
0.62
0.55
0.55
8:72
3.53
0.54
8.66
8.56
6.62
2:30
pemts queene
0:36
0.20
3.45
0.52
0.50
8.42
1.00
3.56
0.54
spe 3U5 wllanni 3
5:40
0:10
850
6.58
5.5
0.82
5.10
mike dil loven Parta 88-
4.1
1322
deeppeed sperents routtel
dirante gent US eostted
perres
permen most Saanee- Eorryrrr
gronins reiitrie Trece
deeyiient apounin prgant
aiee den pal 544a www.esiiiiooo
deepooeee costraid
cevepe Artrenn eieanc
presis aepeee Flansh
perrens di-goon
peerass C.sggge-
vie Sugeert
dev
Jury 3-Tuple
Figure 8: Cohen's Kappa across all jury-model combinations.
F.1.3 QUALITY RATING FOR DIFFERENT GENERATOR LLMs
Table 9: Comparison of Jury scores between DeepSeek V3.2 and Claude 4 generators the MIT-BIH dataset.
Metric
DeepSeek V3.2
Claude
Specificity
8.23 + 0.24
8.08 +: 0.17
Unambiguity
7.55 +'2 0.12
7.47
+ 0.28
Domain Relevance
8.69 + 0.49
8.72 + 0.22
Answerability
8.69 0.14
8.53
+ 0.08
No Unintended Hints
7.46 + 0.13
7.37 + 0.18
32

---

Published as a conference paper at ICLR 2026
F.2 FAILURE ANALYSIS OF THE AGENTIC GENERATION PIPELINE
To better understand the robustness of FimeSeriesExamaae failure modes observed during template generation across all five datasets (PTB-XL, MIT-BIH, MIMIC-IV, Yahoo Finance, and WeatherBench).
we provide a breakdown of
Each template generation attempt categorized into one of four mutually exclusive outcomes:
Success: The template is syntactically valid and differentiable (i.e., stronger models outper- form weaker ones).
Semantic Failure: The generated template violates capability ordering (weak models outperform strong models), indicating poor difficulty calibration.
Syntactic Failure: The template cannot execute due compilation runtime errors.
The template fails validation by the LLM verifier (ambiguity, irrelevance, or missing time-series dependence).
Content Validation Rejection:
We report average outcomes across datasets for the first generation attempt and after one regeneration pass.
Table 10: Failure mode breakdown of the agentic pipeline across datasets.
Outcome Category
Attempt (Avg)
Attempt 2 (Avg)
Success (Valid & Differentiable)
71.5%
79.9%
Semantic Failure (Weak > Strong)
17.8%
19.3%
Syntactic Failure (Compile Error)
9.9%
0.8%
Content Validation Rejection (LLM Verifier Rejection)
0.8%
0.0%
The primary reason for initial failure is syntactic rigidity (10% compilation errors), which our agent easily fixes. Similarly, errors raised by the LLM verifier are successfully resolved in second attempt. The remaining challenge is difficulty calibration (18% non-differentiable items), which is harder semantic problem.
33

---

Published as a conference paper at ICLR 2026
F.3 INPUT MODALITY PERFORMANCE ANALYSIS
We compare model performance when the same time series sample from MIT-BIH dataset is provided either as rendered plot (vision input) or raw numeric text. In the text condition, timestamps serialized as comma-separated values rounded to three decimal places and inserted directly into the prompt. The questions are designed be modality-agnostee the models have the same information but expressed differently.
Table 11: Accuracy on MIT-BIH under two representations of the same time series.
Model
Vision
Text
GPT-4o
0.416
0.401
o3-mini
0.442
0.416
Qwen2 5-VL
0.411
0.391
Cemma-3-27B-17
0.497
0.421
All models show consistent performance drop in the text setting. Our analysis suggests that this is primarily due to the excessive context length, which can introduce unnecessary noise and degrade model reasoning. Additional failure mode analysis is provided in App. G.2
F.4 DIFFERENT RESOLUTION IMPACTS ANALYSIS
We evaluate how plot resolution affects performance by re-running MIT-BIH exams with different rendering DPI settings. compare low-resolution (25 DPI), default (50 DPI), and high-resolution (100 DPI) plots.
Table 12: Accuracy on MIT-BIH exams under different plot resolutions.
Model
25 DPI
50 DPI (Default)
100 DPI
GPT-40
0.310
0.416
0.442
Qwen2.5-V1
0.391
0.411
0.442
Gemma 1iimma---2B--1-
0.447
0.497
0.492
Higher DPI yields small but consistent gains, as expected from improved visual clarity. However, many errors persist even 100 DPI. These remaining failures typically involve multi-step reasoning or implicit computation rather than visibility alone. An illustrative example is provided in Appendix H.
F.5 SENSITIVITY TO GENERATION HYPERPARAMETERS
We study how item quality depends on two of regeneration attempts per template (Nuy {1, 3, 5)) and the number of few-shot examples (Nato E (1,33). We report the template Success Rate (percentage passing all validation checks), Average Difficulty, and Average Differentiability of the valid items.
hyperparameters: the number
Difficulty is computed
Accace + Accweok and Differentiability as AcC ACCweak-
Table 13: Sensitivity of generation quality to attempts and few-shot examples.
Setup (Attempts/ / Shots)
Success Rate
Avg. Difficulty
Avg. Differentiability
Attempt 1 Shot
75.0%
0.52
0.11
Attempt / 3 Shots
75.0%
0.64
0.00
3 Attempts/ / 3 Shots
100.0 %
0.57
0.10
Attempts / Shots
95.0%
0.61
0.14
The pipeline sensitive to the number of attempts, which acts as a crucial stabilizer. Increasing the attempts from to 3 improves the success rate from 75% to 100% and significantly improves the
34

---

Published as a conference paper at ICLR 2026
quality of the items. Additional attempts provide diminishing but positive gains in item separability. In contrast, increasing few-shot examples alone does not guarantee better differentiability without regeneration.
F.6 FINE-TUNING DATA EFFICIENCY
We analyze how performance scales with the volume of generated training data. We vary the number of training samples (N, N/2, N/3) while keeping the total number gradient updates constant by inversely scaling epochs. This isolates the effect of dataset size from training budget.
Table 14: Effect of training data volume on fine-tuning accuracy.
Training Volume (Samples)
Accuracy (Choice-based)
N (2,000)
43.37%
N/2 (1,000)
41 41.80%
N/3 (672)
43.67 %
Performance saturates quickly: using only N/3 samples matches and slightly exceeds the full dataset. This indicates that the generated templates are information-dene and that the fine-tuning signal is highly data-effice efficient.
35

---

Published as a conference paper at ICLR 2026
G CASE STUDY
In this section, we provide a few fine-grined failure mode analysis for samples from both TimeSeriesExam and exams generated by TimeSerieexxamagen...
G.1 CASE STUDY 1: ANOLMAY PATTERN DISRUPTS VISUAL REASONING
Time Series
12
10
-2
200 600
800
GPT-40
Based on the given time series, it shows an overall upward trend that appears to be exponential, the rale increase seems accelerate over time. Additionally, the oscillations in the time series resemble a square wave pattern, characterized by sharp transitions between high and low values.
Answer: B) Exponential trend and square wave
Gemma-3-27B-IT
The series shows an accelcrating trend (exponential) with regular oscillations resembling a sine wave.
Answer: A) Exponential trend and sine wave
Qwen2.5-VL-72B
The overall upward pattern consistent with a linear trend. The cyclic component matches a sine wave.
Answer: A) Linear trend and sine wave
Gemini-2.5-Pro
The trend curves upward, suggesting exponential. Oscillations smooth and sinusoidal. Answer: A) Linear trend and sine wave
This example shows common failure modes in time-series reasoning. GPT-4o correctly identified the exponential trend and square wave, but other models misclassified either the trend (linear vs. exponential) or the cyclic component (sine square). Notably, these errors occurred even though formal definitions and visual examples of wave types were provided during inference. Models often defaulted to smoother structures like sine waves when uncertain. The presence of an anomaly further disrupted reasoning: several models appeared to overfit to local deviations instead of extracting the underlying wend-cych combination. Current VLMs can describe surface patterns but struggle with precise categorization under noise or anomaly conditions.
36

---

Published as a conference paper at ICLR 2026
G.2
CASE STUDY 2: INPUT MODALITY IMPACTS MODEL PERFORMANCE
question
Question: Compare the QRS duration and morphology between Beat A (sample 99947) and Beat (sample 74805). Based on the conduction velocity assessment, which statement is most accurate?
Answer: Beat A shows normal conduction velocity with narrow QRS duration, while Beat B demonstrates delayed intraventricular conduction with prolonged QRS duration indicating bundle branch block
GPT-40 Vision
To compare the QRS duration and morphology between Beat A and Beat B, evaluate the width and the shape of the QRS complexes in both images:1. **QRS Duration and Morphology: **Beat A:** Displays narrower QRS complex.- ** Beat B:** Displays wider QRS com- plex, which often indicates prolonged duration.2. 2. **Conduction Velocity: A narrow QRS complex usually signifies normal rapid conduction.- A wide QRS complex commonly indicates delayed intraventricular conduction, such as bundle branch block.these obser- vations, Beat A demonstrates normal conduction with narrow QRS, while Beat B shows delayed conduction with a prolonged QRS duration., the most accurate statement is:[A] Beat A shows normal conduction velocity with narrow QRS duration, while Beat B demonstrates delayed intraventricular conduction with prolonged QRS duration indicating bundle branch block.
Result: Correct
GPT-4o Text
To determine the most accurate statement about the QRS duration and morphology between Beat A (sample 99947) and Beat (sample 74805), need to evaluate the conduction velocity, QRS duration, and any presence of abnormalities like bundle branch Mlock.A:time series data shows significant spikes amplitude, indicating broad and potentially prolonged QRS complex. This suggests a delayed intraventricular conduction that aligns with a bundle branch block.B:time series data for Beat does not exhibit the same broadness prolonged QRS complex as Beat A. The narrower spikes indicate normal conduction velocities and narrow QRS duration., the comparison shows that Beat has delayed conduction with wide QRS, probably indicating conduction problem like a bundle branch block, whereas Beat B shows normal rapid conduction with narrow QRS duration. on this analysis, the most accurate statement is:[C]
Result: Incorrect
This is typical case where visual input outperforms textual input. ECG diagnosis is traditionally performed by examining the waveform plot rather than directly reading the raw 12-lead values.
37

---

Published as a conference paper at ICLR 2026
question
Question: Based on the QRS complex morphology in this single-lead ECG beat, what the most likely classification?
Answer: Right bundle branch block beat with wide QRS complex and characteristic RBBB morphology
GPTT-4o Vision
To classify the QRS complex morphology:- normal beat** typically has a * narrow QRS complex** and uniform morphology-- A **left bundle branch block (LBBB)** is characterized by a **iide QRS complex** (greater than 120 ms) with the distinet hottchedör Slurredjatterns often seen in lead V1 and lead I.- A **right bundle branch block (RBBB) y** also results in a **dide QRS complex* with a typically positive R' wave in leads like VI.- **ooo signal quality*** would make difficult classify the beat.the provided ECG, the QRS complex appears wide and shows morphology that could indicate an LBBB due to its characteristic shape.most likely classification is:[B] Result: Incorrect
GPT-4o Text
The key points to consider for classification based on the QRS complex morphology are: Normal beats typically have a narrow QRS complex with regular morphology.2. Left bundle branch block (LBBB) beats have wide QRS complex and specific LBBB morphology.3. Right bundle branch block (RBBB) beats also have wide QRS complex but with a distinctive RBBB morphology.4. Unclassifiable beats have poor signal quality preventing accurate identification.te the provided time series data, it appears that the QRS complexes are quite wide, which is characteristic of bundle branch block. Both channel and channel 2 show significant variation and complex waveforms that exceed what would typically be considered normal, narrow QRS complex. distinguish between LBBB and RBBB:- LBBB typically shows a dominant S-wave in V1 and notched R-wave in leads like and V6, but since we have single-lead data, precise pattern identification can be challenging.- RBBB often shows an RSR' pattern or a terminal R wave in VI and wide S wave in leads and V6.the information provided and assuming that data fits broadly to standard patterns, this wide QRS complex could more likely correspond to the characteristic morphology of RBBB., the most likely classification for the QRS complex morphology in this single-lead ECG beat is right bundle branch block beat with a wide QRS complex and characteristic RBBB morphology. final choice is (C].
Result: Correct
Indeed, the vision model fails here, even itself noting the poor signal quality in the plot. Because this question can be answered using only single ECG lead, the textual input avoids the long-context burden from multi-lead data and allows a more focused inspection, leading to better performance.
38

---

Published as a conference paper at ICLR 2026
G.3
CASE STUDY 3: DIFFERENT RESOLUTION IMPACTS PERCEPTION
question
Examine this single-beat ECG recording. Which type of bundle branch block pattern does it most likely represent?
Question:
Right bundle branch block (RBBB), characterized by RSR' pattern (terminal R wave) in lead V1 and wide waves in lateral leads.
Answer:
GPTT4o DPI=25
I'm sorry, can't 't identify the specific bundle branch block pattern from the image. Result: Incorrect
GPT-4o DPI 100
Based on the ECG recording, the pattern includes wide QRS complex with an RSR' pattern in lead V1, indicating terminal R wave. This is characteristic of right bundle branch block (RBBB)((B) Result: Correct
With DPI = 25, the model even struggles to recognize the block pattern, indicating that perceptual quality strongly affects performance.
39

---

Published as a conference paper at ICLR 2026
A common failure mode in programmatically generated benchmarks is semantic mismatch: the natural language in answer options describes patterns or conditions that the underlying code never actually verifies. For instance, option may claim a trend is "consistent throughout the period" while the code only compares aggregate means, or state that event "follows" event B while the code merely checks co-occurrence. When labels are assigned based on computational criteria that diverge from option semantics, ground-truth answers become decoupled from what the options literally describe. The following case studies illustrate this phenomenon across three financial reasoning templates.
G.4 CASE STUDY: FLAWED VOLATILITY BENCHMARK FAVORS WEAKER MODELS
Question Template. "Analyze the daily price volatility of [company] around the highlighted time period (day (N) marked as earnings announcement), How did the stock's 's volatility change in the 10 trading days after the announcement compared to the 10 trading days before?"
Options:
A. Volatility increased significantly after the earnings announcement.
B. Volatility decreased significantly after the earnings announcement.
C. Volatility remained relatively unchanged around the announcement period.
D. Volatility was highest on the announcement day itself, then gradually returned pre- amnouncement amnouncenent levels.
Problematic Code Segment.
# Volatility computation
pre_period_retee returnssseetusssssoo returns (amoouncceeettt...y-----oooomeee D
post_period_returr = returns
+10] BUG: includes announcement day
pre_volatility =
Standard deviation
(10-day)
cost_volatility mp. (p rec__period__ruunss returns)
amnouncement_volatiile
abs (returns (aactuttrrssssmmmmuccmneenttttt
Single
absolute return
Option D classification
max.perioo_volaaiiityy = max (max (abs (r) for in pre_perriodretuuns)).
max (r) for in if amoourcement_volattiiii
post cost_period_returree
1.2: answer = options [3] # "Volatility highest on announcement day then returned to normal"
pex_period_
Critical Errors and Model Performance Analysis. The benchmark contains compounding emrors that create an inverse correlation between model capability and accuracy: (1) Metric Inconsistency.
The code computes period volatility as standard deviation (a = / [r; - 2) but announcement- day volatility as single absolute return (rr]). These dimensionally incompatiblc- rigorous model attempting to reason about volatility comparisons will recognize this inconsistency and struggle select an answer that assumes they are comparable.
(2) Label- -ebb--Desiinttion Mismatch. Option D states volatility "gradually returned to are-announcement levels," yet the code never verifies post pre- A sample can be labeled as Option D even when post-period volatility remains elevated.
Why Weaker Models Outperform. Stronger models engage in deeper reasoning: they may (a) notice the metric mismatch and refuse to commit, (b) detect the logical flaw in Option D's selection criteria, or (c) question the synthetic "carnings announcement" that actually a random date. Weaker models, by contrast, rely shallow pattern
-ashinngassccaatingg keywords like "announcement day" and "volatility spike" with Option D without verifying computational consistency. The flawed answer key rewards this superficial heuristic, penalizing models that reason correctly about the underlying financial concepts. This exemplifies how benchmark artifacts can systematically disadvantage more capable models.
40

---

Published as a conference paper at ICLR 2026
G.5 CASE STUDY: SHARPE RATIO BENCHMARK WITH DEAD OPTIONS
Question Template. "Given the daily price charts for (tickerl) and [ticker2), analyze their rolling 60- day Sharpe ratios over the time period. Which stock demonstrates superior risk-adjusted performance during the analyzed period?"
Options:
A. (better ticker) shows consistently higher risk-adjusted returns with rolling Sharpe ratio that outperforms wworse_ticker throughout most of the period.
B. (worse_ticker shows consistently higher risk-adjusted returns with rolling Sharpe ratio that outperforms (better_ (better_ticker) ticker) throughout most of the period.
C. Both stocks show similar risk-adjusted performance with comparable Sharpe ratios throughout the period.
D. The analysis is inconclusive due to insufficient data.
Problematic Code Segment.
avg_sharpel np. np.mean mean (rolling_sharpel) avg_sharpe2 mean (rolling_sharpe2) sharpe_diff abs (avg_sharpel avg_sharpe2)
if sharpe_ditte diff hyperparameters, ("min_sharpeeee___eeeeaaaaaaaaa sar___hhape____e "iappeeddffrre"c")
continue Skip similar cases Option never valid if avg_sharpel avg_sharpe2:
better_ticke
tickerl
vorse_ticker = ticker2
answer options [0] format
# Always Option A
else:
better_ticked = ticker2
horse_ticker tickerl
answer = options [C] format(...) # Always Option A (never
Option B)
Critical Errors and Model Performance Analysis. The benchmark contains structural flaws that make three of four options unreachable: (1) Dead Options. The answer is always opt ions [C] (Option A). Option B is never selected-even though is semantically constructed as a valid alternative, the code assigns the "better" ticker dynamically such that Option A is always correct. Options C and D are filtered out via answers.
statements, making them structurally impossible
(2) "Consistently" Unverified. Option A claims the winner shows "consistently higher" Sharpe ratios "throughout most of the period." However, the code only compares average Sharpe: S # Σ. S1, vs 52. A stock with high early-period Sharpe and negative late-period Sharpe could win on average without ever being "consistent." No check verifies that Shette, Swarse, for most l..
Why Weaker Models Outperform. stronger model may: (a) recognize that "consistently through- out most of the period" requires temporal dominance analysis, not just mean comparison, and hesitate to select Option A: (b) consider Option B as valid when the ticker ordering the question differs from (c) reason that Option C could apply if rolling Sharpes frequently cross. Weaker models exploit the surface-level heuristic that Option A 1t-phhased most confidently and always listing "winner" iis the intended answer. Since Option A is always correct by construction regardless of actual consistency, shallow pattern matching succeeds while rigorous financial reasoning penalized.
the better/worse assignment;
41

---

Published as a conference paper at ICLR 2026
G.6 CASE STUDY: REGIME SWITCHING BENCHMARK WITH SEMANTIC MISALIGNMENT
Question Template. "Analyzing the daily price movements of [ticker) over the given time period, does the stock exhibit clear volatility regime switching behavior where the market alternates between distinct high-volaility and low-volatilty periods?"
Options:
A. Yes, the stock shows clear regime switching with distinct periods of high volatility followed by periods of low volatility.
B. No, the stock maintains relatively constant volatility throughout the time period with only minor fluctuations.
C. Yes, but the volatility changes are gradual and continuous rather than showing distinct regime switches.
D. The data is insufficient to determine volatility regime patterns.
Problematic Code Segment.
Regime switchii tching: requires least sustained high AND one sustained Iow period len (high
has_regime_swiie
and len (low_periods)
> 0
if se_rrgimme_stee
answer options [C] # "high volatility followed by low volatility"
else:
vol_cv np. (rolling_vol) / np.mean (rolllig___))) Coefficient of variation
if vol_ev 0.3:
answer = options [1] # Constant volatility
else:
answer options [2] "Gradual and continuous" <- semantic mismatch
Post-hoc filtering enforces 608 bias toward Option sin_reeuired int (0.6 * num_ num_samples)
while for i in range (len (qa_pairs)
min_required:
if qa_pairs [i] ["answer"] != options(0]]
qa qa_pairs. pop (i) break
Critical Errors and Model Performance Analysis. The benchmark contains semantic and structural flaws that decouple labels from data: (I) Option Misrepresents Detection Logic. Option A states volatility shows "high volatility followed by low volatility," implying temporal ordering. However, the code only verifies existence of 2 high period and > 1 low period anywhere they need not alternate follow any sequence.
(2) Option C Semantic Mismatch. Option C describes "gradual and continuous" changes, yet the code assigns when CV 0.3. High CV indicates erratic fluctuations- the opposite of gradual.
Why Weaker Models Outperform. The semantic mismatches in (1) and (2) mean ground-tre -truhh labels effectively arbitrary with respect to what the options actually describe. Correct reasoning about temporal ordering or gradual vs. erratic behavior yields no predictive power over labels.
42


Reviews:
[{"review_id": "Las1phNCVF", "reviewer": "Reviewer_RbCS", "summary": "This paper proposes a new framework for generating time series reasoning benchmarks at scale, arguing that existing benchmarks are manually curated, narrow, and expensive to create. The authors' contribution is twofold:\n\nTimeSeriesExam: A \"proof-of-concept\" benchmark consisting of multiple-choice questions based on synthetic time series data. It aims to test five core, domain-agnostic reasoning skills: pattern recognition, noise understanding, similarity, anomaly detection, and causality.\n\nTimeSeriesExamAgent: A multi-agent framework designed to automatically generate new, domain-specific benchmarks from real-world datasets. This agentic pipeline (using Generator, Concept, Verifier, and Student LLMs) takes a dataset (e.g., PTB-XL) and a prompt, and outputs a new \"exam\" with questions derived from that data.\n\nThe paper's central findings are that (1) its agent can generate benchmarks with diversity \"comparable to human-curated\" ones, and (2) all state-of-the-art LLMs (GPT-4o, Gemini 2.5-Pro) perform poorly on both the synthetic and the agent-generated exams, suggesting their time series reasoning capabilities are \"limited.\"", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Valid and Important Problem: The paper correctly identifies a significant gap in the field. Assessing the true reasoning of LLMs on time series data (beyond simple forecasting accuracy) is a critical and under-explored problem. The goal of creating a scalable benchmark generation process is highly valuable.\n\nNovel Agentic Framework: The core idea of TimeSeriesExamAgent is the paper's strongest contribution. Using a multi-agent pipeline to generate, verify, and filter questions is a novel and powerful approach to benchmark creation that is far more scalable than the manual curation.\n\nSophisticated Verification Loop: The verification process (Fig. 2) is well-considered. It includes not just a structural check and an \"LLM Verifier,\" but also a \"Capability-Aligned Filtering\" step that uses \"Student LLMs.\" The idea of discarding questions that weaker models perform better on is a clever and sound method for filtering out flawed or noisy questions.\n\nStrong Validation Signal (Table 7): The transfer learning experiment is a key piece of evidence. Showing that fine-tuning a VLM on the agent-generated data improves its performance on a separate, human-curated dataset (ECG-QA) strongly suggests that the generated questions provide a meaningful and useful learning signal.", "weaknesses": "1. Recursive and Subjective Evaluation:\n\nThe paper's quality control relies heavily on an \"LLM-as-a-judge\" and \"LLM-as-a-Jury\" (Table 6, Appendix G.1) to assess the quality of the generated questions (e.g., \"Specificity,\" \"Unambiguity\"). This creates a recursive, self-referential loop: an LLM (Generator) creates a question, which is then validated by another LLM (Verifier), to create a benchmark that is then used to... test LLMs.\n\nThis \"LLM-evaluates-LLM\" methodology is notoriously subjective, prone to bias (e.g., favoring its own \"style\"), and lacks the objective rigor required for a foundational benchmark. While the authors present this as a feature, it's a methodological weakness that introduces significant unreliability.\n\n2. Incomplete Model Comparison:\n\nWhile the paper tests several SOTA VLMs, the evaluation is incomplete. The primary TimeSeriesExamAgent framework is designed to generate text and time series questions from real-world data, yet the evaluation is performed only by VLM-style models that take images (Section H). The paper even notes this limitation itself (Section 5, \"Limited Evaluation Mode\"), stating that \"certain question types... are particularly difficult to answer from images alone.\" This mismatch between the generated data (which is numerical) and the evaluation mode (which is visual) is a strange and limiting choice. The framework is not evaluated against text-only LLMs coupled with numerical data, which would be a more direct test of reasoning over the raw time series.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "atty9jJBGx", "reviewer": "Reviewer_aEWY", "summary": "The paper proposes a two-level approach for evaluating time series reasoning in large language models. The first level is a controlled multiple choice benchmark named TimeSeriesExam that uses synthetic series to probe five core capabilities including pattern recognition, noise understanding, similarity analysis, anomaly detection, and causality. The second level is TimeSeriesExamAgent, an agentic pipeline that takes a natural language task description and a dataset loader, then generates executable Python question templates with rule based answer computation. The pipeline applies three verification stages including structure checks, LLM judging, and capability aligned filtering, with the goal of producing high quality, unambiguous, domain specific items at scale and with limited expert time. The paper evaluates across healthcare, finance, and weather datasets. It reports that modern vision language models still struggle on higher order time series reasoning, and that the automatically generated benchmarks reach diversity comparable to manually curated sets.", "questions": "1. How do you guarantee label faithfulness for items derived from real datasets where the answer is computed by a template rather than given by an external ground truth?\n\n2. How sensitive is item difficulty to the specific hyperparameters of the template functions?\n\n3. Finetuning on generated items improves performance on one downstream benchmark. Does this transfer to an external benchmark that was not seen during template creation and that is annotated independently?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. In this paper, authors provide clear articulation of the practical bottleneck. The paper motivates the need for scalable domain specific evaluation rather than one size fits all benchmarks and gives an agentic design that maps cleanly to practitioner workflows.\n\n2. In the paper, generating templates rather than individual questions improves reuse and ensures rule based answer derivation from real series, which supports repeatability and reduces silent errors. The rationale is explicitly stated and connected to implementation details.\n\n3. The combination of syntax checks, LLM judging, and capability aligned filtering is a thoughtful safeguard against ill posed or trivial items and is presented with a concrete architectural sketch.\n\n4. The authors document how clinician feedback on ECG terminology guided prompt refinement, which improves face validity and reduces ambiguity for medical users.", "weaknesses": "1. In the paper, contribution hierarchy and granularity are not fully disentangled. The proof of concept and the agentic system are linked, but the paper would benefit from a clearer map of which empirical claims rely on synthetic control versus real data generation. The introduction hints at both goals, but the boundaries remain blurred in places.\n\n2. In this paper, heavy reliance on LLM as a judge raises construct validity concerns. The paper provides judging rubrics, yet it remains unclear how consistent the scores are across different judges and prompts, or how often human experts overrule LLM judgments during verification.\n\n3. The evaluation uses base64 plot images at low DPI. This choice can cap the information bandwidth and may conflate perception limits with reasoning limits. A sensitivity analysis on plot resolution and rendering parameters would strengthen the claim that failures reflect reasoning rather than visibility.\n\n4. The finetuning setup is described with clear hyperparameters, but the paper does not report confidence intervals or significance for the reported improvements, nor ablations on the volume or composition of generated items.\n\n5. The idea that weak models should not outperform strong ones on valid items is intuitive, but the authors does not quantify how many candidate templates are filtered for this reason or provide failure taxonomies that would help others replicate the filtering policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nxjGKd1WdK", "reviewer": "Reviewer_kZtv", "summary": "This paper introduces TimeSeriesExam and TimeSeriesExamAgent, a framework for generating scalable benchmarks to evaluate LLM reasoning over time series data. The authors start from a controlled synthetic setup (TimeSeriesExam) that assesses LLM understanding of five reasoning categories (pattern recognition, noise understanding, similarity, anomaly detection, and causality), and then extend it with an agent-based system that automates benchmark creation from real datasets in finance, healthcare, and meteorology. Experiments show that (i) current VLMs still struggle with time series reasoning, and (ii) the automatically generated benchmarks achieve comparable diversity and quality to human-curated datasets like ECG-QA. The paper further includes a transfer learning study showing modest improvements when fine-tuning on generated data.", "questions": "1. How does TimeSeriesExamAgent compare to recent agentic benchmark generators like BenchAgents (2024) in quantitative efficiency (cost, generation speed, diversity per token)?\n2. Can the authors provide human evaluation results verifying that LLM-judged “quality” correlates with expert judgment?\n3. How does the performance of LLMs differ when reasoning over text-encoded vs. visualized time series inputs (as briefly noted in Sec. 5)?\n4. Can the framework generalize to non-numeric structured domains (e.g., tabular, event logs) without major prompt re-engineering?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The intersection of time-series analysis and LLM reasoning is rapidly emerging. The paper tackles a genuine gap: the lack of scalable, reasoning-oriented benchmarks for time series.\n2. The combination of templated generation, IRT-based refinement, and multi-agent verification (planning–generation–validation) is thoughtfully constructed and clearly explained (see Fig. 2 on p. 5).\n3. The paper is very detailed, with algorithmic pseudocode (Algorithm 1), hyperparameter tables, and qualitative examples (e.g., ECG and finance question examples on pp. 21–23). This adds credibility and reproducibility.\n4. Evaluation covers multiple domains (medical, financial, weather), and the analysis of generated diversity via embedding and Levenshtein metrics (Table 5) is thorough.", "weaknesses": "1. While the engineering effort is strong, the research novelty is limited. The approach largely combines existing components (template-based generation, LLM-as-judge, IRT calibration, multi-agent verification) without a clear new algorithmic or theoretical contribution. The work feels more like a comprehensive system paper than a research breakthrough.\n2. The reported results, though broad, are not deeply analyzed. For example:\n- Table 4 results (< 55% accuracy) confirm that models perform poorly, but there is little insight into why or what specific reasoning capabilities fail.\n- The fine-tuning improvement (21.8 → 47.0%) in Table 7 is promising but lacks ablations—was this due to better question structure, domain alignment, or synthetic diversity?\n3. The paper relies heavily on LLM-as-a-judge for quality assessment (G-Eval + model juries). While practical, this is circular—using the same family of models to judge generated questions about themselves risks hidden bias. No human evaluation beyond anecdotal clinician feedback is systematically quantified.\n4. Although the agent framework is framed as “scalable,” the actual generation still depends on domain-specific prompts and datasets (discussed in Sec. 5). The system’s generalization to unseen domains is not convincingly demonstrated.\n5. The paper is dense and occasionally reads like a project report. The core message is sometimes buried under implementation details. For an ICLR audience, the novelty and research implications should be foregrounded, while extensive technical appendices could be condensed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lcBII8Oyrz", "reviewer": "Reviewer_urhb", "summary": "This paper proposes a multi-agent framework in constructing time series related multiple choice questions based on synthetic and real time series. The generator LLM is in charge of creating qa pairs using python functions and verifier LLMs perform validation on generated questions. The paper evaluated four VLMs on the two sets of benchmarks and demonstrated that current LLM cannot perform reliability on relatively more complex questions.", "questions": "Are there examples of failed qa pairs? Usually what are reasons that this proposed agentic framework fails? Some analysis on this would be very beneficial. \n\nIn sec 4.4, you discussed transferability of the reasoning , but how similar are the synthetic samples to the ECG-QA samples in terms of type of questions asked and formats. \n\nFor are questions generated, is it always 4 options? If not, it would be better to include random guessing for table 3 and 4 to know how the models compare to random guessing.  \n\nCould you explain a little on why anomaly detection questions are harder than causality questions? Intuitively to verify causal relationships, it requires some statistical analysis, but for anomalies in time series, it's relatively easier to spot given that you are inputting time series as images. If it's unable to recognize that, it may not reflect much on time series reasoning limitation but possible perception limitations -> It's hard to isolate the cause of failures.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The benchmark creation process is almost fully automated except for expert created user instructions. \n\nThe paper is easy to follow\n\nThe agentic framework can be scaled up.", "weaknesses": "Since this is a multi agent framework and is considered to be scaled up in the future, cost is an important consideration that didn't seem to be mentioned or discussed in the paper. \n\nSince the benchmark is fully template based, how to mitigate bias in limited linguistic diversity since humans usually ask questions using diverse expressions and solver LLMs could be sensitive to prompts. And since questions are generated by a single LLM rather than a panel of LLM, how to mitigate bias here that a specific LLM may tend to generate questions in a specific way. I mentioned this because the validation is performed by a panel of LLM. \n\nIt feels somewhat like a snake biting its own tail situation where questions are only kept if it's solvable/answerable by an LLM so then why test LLMs on this? And in particular in section 4.3, it's mentioned that you specifically used weaker LLMs as judge, then it didn't make sense to also evaluate answerability of questions. \n\nAlthough the author discussed this in limitation, this is a relatively important point to bring up is that the evaluation is limited to VLMs. On the more practical side, time series reasoning is hard because you need both numerical computation and reasoning. \n\nThe number of evaluated models is very few.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)