TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents

## Abstract

Abstract We introduce TimeSeriesGym , a scalable benchmarking framework for evaluating Artificial Intelligence (AI) agents on time series machine learning engineering challenges. Existing benchmarks lack scalability, focus narrowly on model building in well-defined settings, and evaluate only a limited set of research artifacts (e.g., CSV submission files). To make AI agent benchmarking more relevant to the practice of machine learning engineering, our framework scales along two critical dimensions. First, recognizing that effective ML engineering requires a range of diverse skills, TimeSeriesGym incorporates challenges from diverse sources
spanning multiple domains and tasks. We design challenges to evaluate both isolated capabilities (including data handling, understanding research repositories, and code translation) and their combinations, and rather than addressing each challenge independently, we develop tools that support designing multiple challenges at scale. Second, we implement evaluation mechanisms for multiple research artifacts, including submission files, code, and models, using both precise numeric measures and more flexible LLM-based evaluation approaches. This dual strategy balances objective assessment with contextual judgment. Although our initial focus is on time series applications, our framework can be readily extended to other data modalities, broadly enhancing the comprehensiveness and practical utility of agentic AI evaluation. We open-source our benchmarking framework to facilitate future research on the ML engineering capabilities of AI agents.

## 1 Introduction

AI agents huang2024mlagentbench; jiang2025aide have shown growing promise in automating machine learning (ML) and data science (DS) workflows. Fueled by powerful large language models (LLMs), they can reason about context, adapt to diverse tasks, and iteratively refine their solutions over long horizons. Such capabilities have the potential to significantly reduce mundane, mostly manual efforts required in ML engineering and improve the overall productivity of ML practice. To measure progress in this area, several benchmarks huang2024mlagentbench; chan2025mlebench; jing2025dsbench; nathani2025mlgymnewframeworkbenchmark; wijk2024re; bogin-etal-2024-super; tang2023ml; padigela2025ml have been introduced to evaluate the capabilities of AI agents on ML and DS tasks.

However, existing benchmarks have important limitations. First, many of them source ML challenges primarily from popular competitions such as Kaggle, which are well-structured and do not fully capture the complexity of real-world ML tasks.
Second, evaluations are typically outcome-based, focusing on overall task completion or eventual model performance metrics such as accuracy, while combining and obfuscating the impact of multiple
component skills that jointly determine the outcomes, such as effective data wrangling or code quality improvement capabilities.
Third, current benchmarks lack scalability, as tasks have to be manually curated and cannot be developed at scale.

To enable efficient evaluation of AI agents in realistic ML scenarios, we propose TimeSeriesGym, a scalable and agent-agnostic benchmarking framework for evaluating AI agents on time series ML engineering tasks. TimeSeriesGym currently consists of 34 challenges that span 8 unique time series problems (forecasting, classification, time series understanding), from more than 15 domains (healthcare, finance, epidemiology). Our benchmark covers both Kaggle-style challenges and original tasks carefully designed based on real-world ML engineering practice. While we focus on time series analytics due to its prevalence in applications and under-representation in existing agent benchmarks, our framework is modality-agnostic in principle and can be easily extended to handling other data modalities (e.g., images, text, audio) via the accompanying tools for scalable task generation. TimeSeriesGym provides an interactive gym environment compatible with various types of agent scaffolding, allowing seamless evaluation of agents of different types and collection of their trajectories. Beyond benchmarking, this allows TimeSeriesGym to also serve as a data flywheel for future agent improvement through post-training using the collected trajectory data.

Our contributions include:

- 1.
We propose TimeSeriesGym, the first open source benchmark environment for evaluating AI agents on time series ML engineering tasks (Fig. [1](#S1.F1)).
- 2.
TimeSeriesGym offers a scalable task generation mechanism that reduces manual efforts in task curation and ensures long-term sustainability of the benchmark.
- 3.
TimeSeriesGym provides a comprehensive framework that evaluates multimodal agent outputs (e.g., prediction files, models, code) across specific ML skills (e.g., data handling, model improvement), using a holistic approach that combines quantitative metrics (e.g., accuracy) with qualitative assessment (e.g. LLM-as-a-judge for code utility).

Figure: Figure 1: TimeSeriesGym is a scalable benchmarking environment for ML engineering agents. It currently features 34 time series challenges across 8 unique time series problems, spanning more than 15 domains. Challenges are either carefully designed based on real-world ML practice, or sourced from Kaggle competitions and GitHub repositories. TimeSeriesGym includes key mechanisms to enable efficient and scalable generation of new challenges. Our evaluation methodology combines precise quantitative metrics with flexible qualitative assessment, and provides specialized tools to grade various artifacts generated during ML engineering. TimeSeriesGym is compatible with many different agent types, even those with fundamentally distinct designs.
Refer to caption: /html/2505.13291/assets/x1.png

## 2 Related Work

##### Machine learning agent benchmarks.

Several benchmarks have been proposed to evaluate LLM agents on automating ML and DS tasks. These benchmarks are typically structured around three key components: (1) task curation, (2) agent capabilities being evaluated, and (3) evaluation protocol.
Benchmarks differ in how they curate ML/DS tasks. For example, MLE-bench chan2025mlebench and DSBench jing2025dsbench compile tasks from online competition platforms such as Kaggle, while other benchmarks source tasks from ML-related Github repositories bogin-etal-2024-super; tang2023ml or hand-craft tasks based on ML research problems or engineering workflows huang2024mlagentbench; nathani2025mlgymnewframeworkbenchmark; wijk2024re; padigela2025ml.
With regard to agent capabilities, some benchmarks chan2025mlebench; huang2024mlagentbench; nathani2025mlgymnewframeworkbenchmark; wijk2024re; jing2025dsbench focus on comprehensive ML science skills by evaluating agents on end-to-end problem solving skills, while others bogin-etal-2024-super; tang2023ml; padigela2025ml focus on more modular engineering-oriented capabilities within the ML pipeline, such as using GitHub repositories or integrating APIs.
Evaluation protocols also differ in output formats and granularity. MLE-bench chan2025mlebench and DSBench jing2025dsbench require agents to output results in specific formats (e.g., CSV files) that can be directly scored using predefined metrics such as accuracy, while other benchmarks huang2024mlagentbench; nathani2025mlgymnewframeworkbenchmark; wijk2024re allow for more flexible outputs in addition to prediction files, such as model artifacts and code. ML-Dev-Bench padigela2025ml further extends the evaluation by specific skills (e.g., data handling, model improvement), while SUPER bogin-etal-2024-super provides a more holistic evaluation by combining outcome-based evaluation with qualitative code inspection to assess agents’ progress towards completing the tasks.

**Table 1: Comparison of TimeSeriesGym with existing ML/DS agent benchmarks. Categories include Number: total and time series (TS) tasks in each benchmark, where each task corresponds to a unique data source (e.g., a single Kaggle competition or GitHub repository); Source: task origins (K: Kaggle, G: GitHub, H: Hand-crafted); ML Capability: coverage of ML science tasks (e.g., modeling, open-ended research) and engineering tasks (e.g., repository utilization, API integration); and Evaluation: capabilities for evaluating multimodal outputs (e.g., prediction files, model artifacts), specific ML skills (e.g., data handling, model improvement), and from a holistic perspective (combining quantitative metrics (accuracy, mean absolute error) with qualitative evaluation via code landmarks or LLM-as-a-judge approaches). We use "+" to indicate TimeSeriesGym’s scalability which enables the generation of an unlimited number of new challenges using the tools provided.**
|  | Number | Source | ML Capability | Evaluation |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Total | TS | K | G | H | Science | Engineering | Multimodal | Skill-based | Holistic |
| MLE-bench chan2025mlebench | 75 | 3 | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| MLAgentBench huang2024mlagentbench | 13 | 1 | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| MLGym nathani2025mlgymnewframeworkbenchmark | 13 | 0 | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| RE-Bench wijk2024re | 7 | 0 | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| DSBench(^1^11For DSBench, we include only data modeling tasks, while excluding data analysis tasks as they are not directly relevant to our work.) jing2025dsbench | 74 | 5 | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| SUPER(^2^22For SUPER, we include repositories used to create the Expert and Masked sets of the benchmark.) bogin-etal-2024-super | 45 | 0 | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ |
| ML-Bench tang2023ml | 18 | 1 | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| ML-Dev-Bench padigela2025ml | 30 | 0 | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ |
| TimeSeriesGym (Ours) | 23+ | 23+ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

##### Scalable dynamic benchmarks and holistic evaluation.

Scalable benchmarks reduce manual data curation efforts by generating target problems at scale using carefully designed templates cai2024timeseriesexam; ye2024beyond or data engines ho2024map, among which TimeSeriesExam cai2024timeseriesexam further improves problem sample quality by applying Item Response Theory (IRT) embretson2013item; guinet2024automated to intelligently select questions with contextualized difficulty and appropriate discrimination.
To remain effective against data contamination from LLM pretraining, dynamic benchmarks such as GAIA mialon2023gaia and LiveCodeBench jain2024livecodebench propose to continually incorporate problems newly released after LLM training cut-offs.
While most benchmarks target specific capabilities, holistic evaluation liang2022holistic; goswami2023aqua provides a comprehensive picture through evaluating models on a wide range of datasets and tasks across diverse domains using multiple complementary metrics, to capture both the breadth and depth of model capabilities.

## 3 TimeSeriesGym

TimeSeriesGym is envisioned as a scalable benchmarking environment for time series machine learning engineering. The current version features 34 challenges from 23 unique data sources across 8 unique time series problems, spanning more than 15 domains. These challenges evaluate AI agents on a range of realistic ML engineering skills beyond just model development, including data labeling, model selection, and the utilization, improvement, and migration of research code (Tab. [3](#A1.T3)). TimeSeriesGym also provides tools for rapidly developing new challenges to test specific skills and for evaluating the diverse artifacts commonly generated during ML engineering processes.

Each challenge in TimeSeriesGym is organized with a consistent structure:
(1) resources including datasets, code repositories, related paper(s) and documentation relevant to the challenge;
(2) a description file that outlines the challenge parameters, available resources, and provides specific instructions and hints for successful completion; and
(3) challenge-specific grading functions to evaluate agent submissions. Some challenges also include leaderboards to rank agent submissions against human performance. These leaderboards are readily available for, e.g., challenges derived from Kaggle competitions.

The challenges in TimeSeriesGym are derived from Kaggle competitions (currently, n = 12) and popular benchmarks and research code repositories for time series modeling (TimeSeriesGym Originals, n = 14).
We prioritized challenges that reflect core skills that are regularly exercised by ML engineers, researchers, and data scientists.

Each challenge is specifically chosen or designed to evaluate one or more of the following skills:
(1) Data Handling: Ability to handle missing data, use data labeling tools, and leverage multi-source data for model building.
(2) Modeling: Ability to develop useful time-series ML models, tune hyperparameters, perform model selection, and understand, utilize, migrate and improve the quality of research code.
(3) Benchmarking: Training and rigorously evaluating ML models using standard benchmarks.
In selecting these challenges, we aimed at a broad coverage across diverse domains (e.g., healthcare, finance, epidemiology) and time series problems (forecasting, classification, time series understanding). Tab. [3](#A1.T3) provides a comprehensive overview of each challenge within TimeSeriesGym, including its domain, core problem, evaluation metric, and the skills required to address it.

To identify Kaggle challenges for inclusion in the TimeSeriesGym, we began with the Meta Kaggle dataset metakaggle, focusing specifically on Featured and Research competitions. Featured competitions are real-world ML challenges that pose difficult, commercially oriented prediction problems, while Research competitions offer opportunities to work on problems that may not have clean or straightforward solutions(^3^33[https://www.kaggle.com/docs/competitions](https://www.kaggle.com/docs/competitions)). We employed Gemini 2.0 Flash to analyze competition descriptions and titles, identifying 453 competitions that likely involve time series data. Subsequently, these were ranked based on three key metrics: participant count, maximum reward offered, and presence of a public leaderboard. From the resulting shortlist of 100 high-quality competitions, we made our final selections to ensure comprehensive coverage across diverse tasks and domains within the time series analytics landscape.

To complement the selected Kaggle challenges, we include 14 TimeSeriesGym Original challenges, manually curated based on recommendations from experienced ML engineers and researchers. These challenges are specifically designed to evaluate advanced technical skills that Kaggle competitions typically cannot easily assess, yet are essential for effective ML engineering. Examples include utilizing state-of-the-art models (e.g., Implement the MOMENT goswami2024moment time series foundation model for anomaly detection), migrating frameworks (e.g., Convert ResNet-1D classification models from TensorFlow to PyTorch), and improving research code quality (e.g., Improve PTB-XL ECG Classification Code. These capabilities represent critical competencies of skilled ML engineers that extend beyond the scope of standard Kaggle-like competitions.

Running experiments on TimeSeriesGym can be resource-intensive and costly. Therefore, we propose TimeSeriesGym-Lite, a carefully selected subset of six challenges designed to efficiently evaluate AI agents on critical ML engineering skills while maintaining coverage across multiple domains and time series problems. This collection enables rapid and cost-effective assessment of novel AI agents without sacrificing the diversity of skills being tested (see Tab. [5](#A1.T5)).

##### Multimodal, skill-based, holistic evaluation.

Existing benchmarks typically summarize agent performance using metrics such as accuracy, completion rate, or competition rankings chan2025mlebench. Although these metrics provide useful summaries, they do not offer much actionable feedback for improvement. TimeSeriesGym addresses this limitation through an evaluation framework designed to provide specific actionable feedback through multiple complementary approaches.
First, we design challenges that isolate and test specific skills, such as handling missing data (e.g., Optiver Realized Volatility Prediction with Missing Data). Poor performance on these targeted challenges clearly indicates potential skill gaps, enabling developers to focus their efforts on specific skills.
Second, we develop fine-grained evaluation tools that assess multiple dimensions of performance simultaneously. For example, in code migration tasks (e.g., Convert ResNet from TensorFlow to PyTorch), our evaluation tools examine whether an agent follows instructions and naming conventions, completes all required function definitions, in addition to successful execution– providing a multidimensional performance profile rather than a binary success/failure indicator.

Our evaluation methodology deliberately combines multiple assessment approaches: quantitative metrics (accuracy, mean absolute error), programmatic analysis (regular expression matching, code inspection), and qualitative evaluation (LLM-as-a-judge) (see Appendix [E](#A5)). This hybrid approach balances the reliability of objective metrics with the flexibility of subjective assessment. Although LLM-based evaluation offers valuable insight, especially for open-ended tasks such as research code enhancement, we recognize that LLMs can be inconsistent and prone to hallucination. To mitigate these concerns, we strategically complement subjective assessments with precise quantitative metrics, creating a robust evaluation system that leverages the strengths of each approach while offsetting their individual limitations.

Furthermore, TimeSeriesGym provides specialized tools to grade diverse artifacts generated throughout the ML engineering lifecycle– from submission files (CSV, H5, etc.) to source code (.py) and trained models (.pth, .pkl)– enabling comprehensive assessment of the entire development process rather than focusing solely on final outputs.

##### Generating challenges at scale.

We provide key mechanisms to enable efficient and scalable generation of new challenges. First, we offer clear and detailed documentation that explains how to add new challenges to the benchmark. Second, we provide specialized tools to create skill-specific challenges (e.g., simulating missing data) and evaluate them. Using these resources, our team successfully created several new challenges in two hours of effort, each testing specific ML engineering skills. This scalability ensures that TimeSeriesGym can grow and adapt as time series machine learning techniques continue to advance.

### 3.1 Design Choices

##### Focus on time series tasks.

We focused on time series modeling tasks for two key reasons. First, time-series data are ubiquitous, and effectively modeling them can significantly advance critical domains such as healthcare and economics. Despite this importance, existing agentic AI benchmarks include relatively few time series challenges (Tab. [1](#S2.T1)). Second, compared to text and images, time series data require modest resources for storage and modeling, making TimeSeriesGym efficient to run. Moreover, modeling time-series data remains relatively underexplored outside specialized research communities, meaning that LLMs are less likely to have encountered such data and tasks during training. This characteristic, combined with the fact that TimeSeriesGym evaluates general machine learning skills, makes it an excellent testbed to evaluate AI agent capabilities. Moreover, the tools in this benchmark can be easily used to include non-time-series problems as well.

##### How much freedom should the agents be given?

When designing challenges for TimeSeriesGym, we had to strike a fine balance between giving agents freedom to solve problems creatively while keeping enough structure in place to allow for a precise and fine-grained evaluation. For example, in the PTB-XL ECG Classification with Hyper-parameter Optimization challenge, we required agents to use a PyTorch-based neural network and save their models, files and code before and after tuning. This allowed us to inspect models and code to check if the hyperparameters changed, and measure how these changes improved performance.

##### Agent-agnostic design.

TimeSeriesGym is agnostic to specific agent implementations. Following MLE-bench chan2025mlebench, it is easy to add new challenges and agentic scaffolds. To illustrate this flexibility, we include latest implementations of 3 different scaffolds, AIDE jiang2025aide, ResearchAgent huang2024mlagentbench, and OpenHands wang2024openhands with fundamentally different designs. Unlike MLGym nathani2025mlgymnewframeworkbenchmark, we do not advocate for a default agentic scaffold, as we believe that agent designs will continue to evolve and no single scaffold will work best for all ML engineering tasks.

## 4 Experiments and Results

**Table 2: Main Results. Each experiment was run with 3 random seeds, with results showing mean $\pm$ standard deviation. The table compares scaffold types (OpenHands vs. AIDE), model choices (GPT-4.1, o3, Claude 3.7), resource allocations (4/50 to 12/150 hours/steps), and time utilization approaches. Key findings include: (1) AIDE outperforms OpenHands as a scaffold, (2) the reasoning model o3 achieves significantly higher valid submission rates (94.4%) than other models, (3) Claude 3.7 produces the most reasonable submissions (38.9%), (4) doubling time resources does not consistently improve performance, and (5) interestingly, removing step-wise reminders sometimes improves reasonable submission rates.**
| Lite | Model | Resources | Valid | Reasonable |  |
| --- | --- | --- | --- | --- | --- |
| Submission | Submission |  |  |  |  |
| (hours / steps) | (%) | (%) |  |  |  |
|  | OpenHands |  |  |  |  |
| ✓ | + gpt-4.1-2025-04-14 | 4 / 50 | $44.4\pm 19.3$ | $11.1\pm 9.6$ |  |
|  | AIDE |  |  |  |  |
| ✗ | + gpt-4.1-2025-04-14 | 4 / 50 | $57.3\pm 7.9$ | $12.5\pm 0$ |  |
| ✓ | + gpt-4.1-2025-04-14 | 4 / 50 | $66.7\pm 16.7$ | $27.8\pm 9.6$ |  |
| + o3-2025-04-16 | $\mathbf{94.4\pm 9.6}$ | $33.3\pm 0.0$ |  |  |  |
| + claude-3-7-sonnet-20250219 | $50.0\pm 16.7$ | $38.9\pm 19.3$ |  |  |  |
| Effect of Scaling Resources |  |  |  |  |  |
|  |  | 4 / 50 | $66.7\pm 16.7$ | $27.8\pm 9.6$ |  |
| ✓ | + gpt-4.1-2025-04-14 | 8 / 100 | $72.2\pm 9.6$ | $22.2\pm 9.6$ |  |
|  |  | 12 / 150 | $61.1\pm 9.6$ | $\mathbf{50.0\pm 0.0}$ |  |
| Effective Utilization of Time |  |  |  |  |  |
| ✓ | Step-wise reminder | 4 / 50 | $66.7\pm 16.7$ | $27.8\pm 9.6$ |  |
| No reminder | $55.6\pm 9.6$ | $33.3\pm 0.0$ |  |  |  |

##### Setting.

We run agents in an Ubuntu 20.04 Docker container with all necessary resources (datasets, code repositories, etc.) and basic Python packages useful for ML engineering. Agents can access the internet and install additional packages as needed. For each challenge, agents have a maximum of 4 hours and 50 steps huang2024mlagentbench; padigela2025ml; nathani2025mlgymnewframeworkbenchmark and use a machine with 128 vCPUs, 503 GB RAM, 1.8 TiB SSD, and a single NVIDIA A100-SXM4-80GB GPU(^4^44In practice, agents share this machine as we run multiple challenges in parallel. This represents a realistic setting similar to how ML engineers routinely share computing resources. We found no instances where this sharing might have disadvantaged any agent.). Unless otherwise specified, we repeat each experiment with 3 different seeds (0, 1, and 2) to calculate mean and standard deviation.

##### Cost.

On average, it cost us USD 63.00 to run AIDE with gpt-4.1-2025-04-14 for a maximum of 4 hours and 50 steps on TimeSeriesGym. In contrast, the Lite benchmark was much more affordable at USD 8.00
per run. Therefore, to save both time and money, we conducted most of our experiments on TimeSeriesGym -Lite.

##### Metrics.

We report the raw scores achieved by each agent on every challenge (Tab [3](#A1.T3)). Although these scores are useful for tracking progress on individual challenges, they cannot be easily combined across different challenges. To measure the performance of agents at a high level, we report two key metrics: the percentage of challenges where the agent made a (1) valid, and (2) reasonable submission. A submission is valid if the grader returns any non-null score. What counts as a reasonable attempt varies by challenge type (Tab. [2](#S4.T2)). For Kaggle challenges, we define a reasonable attempt as scoring above median on the competition’s public leaderboard. For the remaining challenges, a reasonable submission shows a genuine attempt at generating a valid submission, rather than hallucinating an output that matches the submission format. We determine this by examining agent logs either manually or using LLM-as-a-judge.

### 4.1 Observations

##### AIDE is the better open-source scaffold.

We evaluated GPT-4.1 (gpt-4.1-2025-04-14) using two open-source scaffolds: AIDE jiang2025aide and OpenHands wang2024openhands. Following MLE-bench, we make minor modifications to adapt these scaffolds to our benchmark (see Appendix [B](#A2)). Our results in Tab. [2](#S4.T2) confirm previous findings: GPT-4.1 produces more valid (66.7% vs 44.4%) and reasonable (27.8% vs 11.1%) submissions with AIDE than with OpenHands. This is expected because AIDE is specifically designed for data science tasks, which account for the majority of the TimeSeriesGym challenges.

##### Reasoning models produce substantially more valid submissions.

To identify the best base model, we conducted experiments using the best scaffold (AIDE) with two state-of-the-art proprietary LLMs: GPT-4.1 (gpt-4.1-2025-04-14) and Claude 3.7 Sonnet (claude-3-7-sonnet-20250219), and a reasoning model o3 (o3-2025-04-16). As shown in Tab. [2](#S4.T2), our experiments on TimeSeriesGym-Lite revealed that o3 created significantly more valid submissions than other models, while Claude 3.7 produced the highest number of reasonable attempts (38.9%).

##### Challenges in TimeSeriesGym are hard for state-of-the-art agents.

We tested AIDE with GPT-4.1 on all TimeSeriesGym challenges and found poor overall performance. The agent produced valid submissions for only 57.3% of challenges and reasonable submissions for just 12.5%. We found that this agent especially struggled with TimeSeriesGym original challenges, where it failed to produce valid submissions for 5 out of 13 challenges (Tab. [8](#A3.T8)). These results show that even the best agents struggle with ML engineering tasks, particularly those that go beyond standard Kaggle data science challenges and involve working with multi-file code repositories.

##### Agents do not improve with more time.

We wondered if the agents perform poorly on TimeSeriesGym simply because they need more time. To test this idea, we ran AIDE with GPT-4.1 on TimeSeriesGym -Lite and gave it 2 or 3 times more hours and steps to solve each challenge. Our results show that extra time does not always improve performance (Tab. [9](#A3.T9)). Even with the maximum time (12 hours and 150 steps), the agent only made reasonable submissions in about 50% of challenges. Although this might seem promising, it is not very impressive, since the bar for a “reasonable" submission is quite low.

Figure: Figure 2: GPT-4.1’s familiarity with TimeSeriesGym challenges, compared to its familiarity with MLE-bench.
Refer to caption: /html/2505.13291/assets/x2.png

##### Agents do not utilize time effectively.

We suspected that agents do not improve with more time because they do not use it well. To test this idea, we compared two settings: the default approach of reminding the agent about remaining time (and steps) before each step, versus removing these reminders completely. Surprisingly, we did not find significant differences between these settings. In fact, agents without time reminders produced more reasonable submissions. This may suggest that agents do not use their time wisely– they tend to rush toward solutions instead of carefully exploring promising options, especially towards the end of the experiment. This raises important research questions about how to design agents that use their time and resources more strategically.

##### Frontier LLM challenges.

Since frontier LLMs are pretrained on large-scale public data, there is a risk that they may have encountered and memorized content from public challenges, e.g., online Kaggle competition discussions or solutions, which can potentially inflate benchmark performance and limit its generalizability.
To assess this risk, we followed the approach used by MLE-bench to measure GPT-4.1’s familiarity with TimeSeriesGym challenges and compared the familiarity score distribution to that of MLE-bench.
As shown in Fig. [2](#S4.F2), GPT-4.1 exhibited a similar level of familiarity with TimeSeriesGym challenges as with MLE-bench challenges (with a Kolmogorov–Smirnov (KS) Test massey1951kolmogorov p-value of 0.363, indicating no significant difference). Given that MLE-bench found no systematic impact of LLM familiarity on experiment results, GPT-4.1’s familiarity with TimeSeriesGym is within a reasonable range and does not compromise its integrity.

##### Summary.

This section provides a focused illustration of how TimeSeriesGym enables efficient and cost-effective experimentation with AI agents, helping researchers uncover actionable insights about agent capabilities and limitations. Our findings demonstrate the TimeSeriesGym’s value for advancing generic ML engineering agents.

## 5 Discussion, Open Questions and Opportunities

##### Key limitations of existing scaffolds.

Agentic scaffolds such as AIDE and OpenHands provide structured workflows that excel in single-shot, self-contained benchmarks (e.g., Kaggle competitions) but reveal significant limitations in repository-level challenges that require multiple file edits and iterative reasoning. AIDE’s *one-step solution strategy* and *fixed action set*—restricted to predefined operations such as “data preview" when debugging—often lead to unsuccessful attempts in large codebases, as the agent’s attention is diluted across irrelevant files and fails to identify critical information. Conversely, OpenHands supports multi-step trajectories yet suffers from a greedy exploitation bias: it commits fully to a single approach without exploring alternative solution paths or revisiting earlier decisions when trajectories prove unfruitful. The planning algorithm of the CodeAct agent used by OpenHands is similarly *greedy and short-horizon*, limiting adaptation to complex multistage development workflows. These findings highlight the need for more adaptive scaffolds that dynamically expand their action repertoire, balance exploration and exploitation through parallel solution threads, and support nested workflows reflective of real-world machine learning engineering tasks. We provide illustrations of agent failures in Appendix [D](#A4).

##### Data leakage and plagiarism.

In designing TimeSeriesGym, we identify two key risks related to data leakage and plagiarism that could compromise the integrity of the benchmark:
(1) Pretraining contamination: Current LLMs may have been exposed to public content from existing challenges (e.g., Kaggle competitions), including task descriptions, data, or shared solutions. This can lead to memorization and inflated performance that overstates agents’ true capabilities, and (2) Future LLM leakage: Once the benchmark is public, future LLMs may be pretrained on its content, making the benchmark less effective in evaluating real generalization.

To address such risks, we present both empirical findings and mitigation strategies.
For case (1), we have two key observations.
First, in both Kaggle-based and original challenges in TimeSeriesGym, agents either performed poorly or did not produce valid output, suggesting minimal benefit from any potential LLM contamination.
Second, we conducted a formal analysis using available tools to assess agents’ familiarity with all competitions in this benchmark. The results show no evidence of systematic prior exposure or memorization, further supporting the integrity of the benchmark in its current state.
For case (2), the scalability of TimeSeriesGym enables efficient generation of new challenges and skill-specific variations. This allows the benchmark to evolve continuously and remain effective even if the current version is eventually included in future LLM pretraining.

Finally, we raise a broader question around plagiarism and code reuse. Several TimeSeriesGym challenges, such as leveraging MOMENT goswami2024moment for anomaly detection, require agents to use existing code repositories to solve open-ended ML tasks. In such cases, it becomes difficult to clearly define and assess plagiarism. For example, if an agent cites the code it uses, should it be considered plagiarism or appropriate reuse, similar to how human ML practitioners build on public code with proper reference? As the ability to effectively and properly leverage existing resources is important in real-world ML practice, we believe that it is crucial to develop clear, legally correct definitions and evaluation criteria for data contamination and plagiarism in the context of LLM agents. We highlight this as an important direction for future work.

##### Defining and measuring success.

What does it mean for an agent to be successful?
For Kaggle tasks, while comparing an agent’s performance against human leaderboards seems intuitive, it presents challenges.
TimeSeriesGym utilizes different training and testing splits and reimplements the grading mechanisms from the original Kaggle competitions (as original Kaggle test sets are private), making direct leaderboard comparisons potentially misleading.
Additionally, in challenges like code migration, real-world utility does not always require perfect, bug-free code: partial, buggy solutions may still accelerate development when iterated by human engineers. Thus, our current evaluation approaches have inherent limitations.

We propose several desiderata for improving the success metrics. These metrics should be rigorous and objective, yet flexible enough to preserve agent creativity and autonomy. They should also yield actionable insights, helping identify specific deficiencies and guide future improvements in agent design. In this work, we take a step in this direction by enabling skill-based and holistic evaluations, offering a more comprehensive understanding of agent capabilities and limitations. Moving forward, we believe that the development of robust, holistic and diagnostic success metrics remains an important research direction and requires community discussion.

##### Comparative characterization.

In the practice of ML application, one often needs to choose an appropriate approach from a short list of eligible options. For this, conducting pairwise comparisons among the alternatives often helps to assess their relative advantages and limitations. Inspired by cai2024timeseriesexam, we envision an automated process that iteratively emphasizes the apparent differences in performance between any pair of agents, or one versus a collection of alternative agents, so that particular strengths or vulnerabilities can be exposed, characterized, and potentially addressed.

##### Optimal resource allocation.

Consistent with previous work, agents were given 4 hours and 50 steps to solve each challenge - but is this sufficient? Alternative frameworks like MLE-bench provide substantially more resources (24 hours and approximately 2000 steps). Our scaling experiments, which gave agents up to 12 hours and 150 steps for a subset of challenges, did not reveal significant performance improvements. Therefore, we believe that further increasing resources is an option, but practical academic budget constraints make such approaches largely infeasible. This raises important questions about how to balance resource limitations with fair opportunities to assess AI agents.

##### Societal impact.

AI agents promise to substantially reduce manual effort in ML engineering while expanding the productivity and accessibility of ML tools. This automation presents several social implications worth considering. First, by lowering technical barriers, these agents could democratize ML capabilities, allowing users without an extensive programming background to leverage advanced analytics. Second, automated ML workflows can accelerate scientific discovery in multiple application domains, including healthcare, climate science, and materials research.
However, several challenges require careful attention from the community. The primary concern is proper attribution when agents repurpose existing code, potentially obscuring original authorship and violating licenses. Furthermore, automated ML systems can perpetuate or amplify existing biases in training data without human oversight. There is also the risk of workforce displacement for entry-level ML engineers as routine tasks become increasingly automated. Furthermore, these agents might generate plausible but flawed solutions that appear correct to non-experts, leading to undetected errors in critical applications. The resource-intensive nature of running sophisticated agents could also exacerbate computational divides between well-resourced and under-resourced organizations. As we advance agent capabilities through benchmarks like TimeSeriesGym, the research community must simultaneously develop frameworks for responsible deployment that address these challenges while maximizing societal benefits.

## 6 Conclusion

We propose TimeSeriesGym, a scalable and agent-agnostic benchmarking framework for evaluating AI agents (with various scaffoldings) on time series ML engineering tasks.
By curating tasks that reflect real-world ML practice from diverse sources, enabling scalable task generation, and supporting multimodal, skill-based, holistic evaluation, TimeSeriesGym provides a practical and extensible testbed for advancing AI agents in ML engineering.
Our experiments show that while frontier LLMs combined with AIDE scaffolding jiang2025aide can achieve moderate to high success rates in producing valid submissions, they still do not generate reasonable solutions, particularly on TimeSeriesGym original challenges that emulate the complexity of real-world time series tasks.
This highlights current limitations in agent capabilities to effectively understand and solve realistic time series tasks.
By open-sourcing TimeSeriesGym, we aim to facilitate a deeper understanding of the ML engineering capabilities of AI agents, provide actionable insights on future development, and support the collection of agent interaction trajectories to drive continuous improvement of AI agents through post-training and refinement.

## References

- [1]
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot.
SUPER: Evaluating agents on setting up and executing tasks from research repositories.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12622–12645, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
- [2]
Yifu Cai, Arjun Choudhry, Mononito Goswami, and Artur Dubrawski.
TimeSeriesExam: A time series understanding exam.
NeurIPS’24 Time Series in the Age of Large Models Workshop, 2024.
- [3]
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng.
MLE-bench: Evaluating machine learning agents on machine learning engineering.
In The Thirteenth International Conference on Learning Representations, 2025.
- [4]
Susan E Embretson and Steven P Reise.
Item response theory for psychologists.
Psychology Press, 2013.
- [5]
Mononito Goswami, Vedant Sanil, Arjun Choudhry, Arvind Srinivasan, Chalisa Udompanyawit, and Artur Dubrawski.
Aqua: A benchmarking tool for label quality assessment.
Advances in Neural Information Processing Systems, 36:79792–79807, 2023.
- [6]
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski.
MOMENT: A family of open time-series foundation models.
In International Conference on Machine Learning, pages 16115–16152. PMLR, 2024.
- [7]
Gauthier Guinet, Behrooz Omidvar-Tehrani, Anoop Deoras, and Laurent Callot.
Automated evaluation of retrieval-augmented language models with task-specific exam generation.
In Forty-first International Conference on Machine Learning, 2024.
- [8]
Cherie Ho, Jiaye Zou, Omar Alama, Sai Mitheran Jagadesh Kumar, Cheng-Yu Chiang, Taneesh Gupta, Chen Wang, Nikhil Keetha, Katia Sycara, and Sebastian Scherer.
Map it anywhere: Empowering bev map prediction using large-scale public datasets.
Advances in Neural Information Processing Systems, 37:64433–64453, 2024.
- [9]
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec.
MLAgentbench: Evaluating language agents on machine learning experimentation.
In Forty-first International Conference on Machine Learning, 2024.
- [10]
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.
Livecodebench: Holistic and contamination free evaluation of large language models for code.
arXiv preprint arXiv:2403.07974, 2024.
- [11]
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu.
Aide: Ai-driven exploration in the space of code.
arXiv preprint arXiv:2502.13138, 2025.
- [12]
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu.
DSBench: How far are data science agents from becoming data science experts?
In The Thirteenth International Conference on Learning Representations, 2025.
- [13]
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al.
Holistic evaluation of language models.
arXiv preprint arXiv:2211.09110, 2022.
- [14]
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.
G-eval: Nlg evaluation using gpt-4 with better human alignment.
In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, 2023.
- [15]
Frank J Massey Jr.
The kolmogorov-smirnov test for goodness of fit.
Journal of the American statistical Association, 46(253):68–78, 1951.
- [16]
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom.
GAIA: a benchmark for general ai assistants.
In The Twelfth International Conference on Learning Representations, 2023.
- [17]
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu.
MLGym: A new framework and benchmark for advancing ai research agents, 2025.
- [18]
Harshith Padigela, Chintan Shah, and Dinkar Juyal.
ML-Dev-Bench: Comparative analysis of ai agents on ml development workflows.
arXiv preprint arXiv:2502.00964, 2025.
- [19]
Megan Risdal and Timo Bozsolik.
Meta kaggle, 2022.
- [20]
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et al.
ML-Bench: Evaluating large language models and agents for machine learning tasks on repository-level code.
arXiv preprint arXiv:2311.09835, 2023.
- [21]
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al.
Openhands: An open platform for ai software developers as generalist agents.
In The Thirteenth International Conference on Learning Representations, 2024.
- [22]
Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al.
Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts.
arXiv preprint arXiv:2411.15114, 2024.
- [23]
Wen Ye, Yizhou Zhang, Wei Yang, Lumingyuan Tang, Defu Cao, Jie Cai, and Yan Liu.
Beyond forecasting: Compositional time series reasoning for end-to-end task execution.
arXiv preprint arXiv:2410.04047, 2024.

## Reproducibility statement

We provide TimeSeriesGym as an open-source project under the permissive MIT License: [https://github.com/moment-timeseries-foundation-model/TimeSeriesGym](https://github.com/moment-timeseries-foundation-model/TimeSeriesGym). The repository includes detailed documentation on running experiments, adding new challenges, and incorporating different agentic scaffolds. Tab. [3](#A1.T3) lists all challenges in TimeSeriesGym, while Tab [4](#A1.T4) provides their sources and licenses. We describe our exact experimental settings and compute resources in Sec. [4](#S4), with scaffold hyperparameters detailed in Tab. [7](#A2.T7). The cost to run each experiment is reported in Tab. [6](#A1.T6).

## Appendix A TimeSeriesGym Challenges

**Table 3: This table presents the TimeSeriesGym benchmark’s diverse collection of time series challenges across three categories: Kaggle Challenges, TimeSeriesGym Originals, and Derived Challenges. The challenges span multiple domains (healthcare, finance, energy, weather, transportation), problem types (classification, regression, forecasting, anomaly detection), and required skills (data handling, model building, code migration). Each challenge uses appropriate evaluation metrics for its task type. The benchmark combines established Kaggle competitions with novel custom tasks, creating a comprehensive testbed for evaluating ML engineering agents across realistic scenarios that practitioners face in real-world applications.**
| Challenge | Problem | Domain | Skills | Evaluation Metric |
| --- | --- | --- | --- | --- |
| Kaggle Challenges |  |  |  |  |
| AMP-Parkinson’s Disease<br>Progression Prediction | Time-to-Event Regression | Healthcare | Data Handling<br>(Dealing with Missing Values,<br>Utilize Multi-Source Data)<br>Modeling<br>(Hyper-parameter Tuning<br>& Model Selection) | Symmetric Mean Absolute<br>Percentage Error |
| ASHRAE - Great Energy Predictor III | Forecasting | Energy | Root Mean Square<br>Logarithmic Error |  |
| Child Mind Institute– Detect Sleep States | Classification | Healthcare | Event Detection<br>Average Precision |  |
| Google Brain - Ventilator Pressure Prediction | Regression | Healthcare | Mean Absolute Error |  |
| G2Net Gravitational Wave Detection | Classification | Geology | Area Under<br>ROC Curve |  |
| HMS - Harmful Brain Activity Classification | Classification | Healthcare | KL Divergence |  |
| LANL Earthquake Prediction | Time-to-Event Regression | Geology | Mean Absolute Error |  |
| M5 Forecasting - Accuracy | Forecasting | Sales | Weighted Root Mean Squared<br>Scaled Error |  |
| Online Product Sales | Forecasting | Sales | Root Mean Square<br>Logarithmic Error |  |
| Optiver Realized Volatility Prediction | Forecasting | Finance | Root Mean Square<br>Percentage Error |  |
| Recruit Restaurant Visitor Forecasting | Forecasting | Sales | Root Mean Square<br>Logarithmic Error |  |
| Sberbank Russian Housing Market | Forecasting | Housing | Root Mean Square<br>Logarithmic Error |  |
| TimeSeriesGym Originals |  |  |  |  |
| Convert ResNet TensorFlow<br>implementation to PyTorch | Classification | Algorithm | Code Migration | Custom Code Grading |
| Convert STOMP Algorithm<br>implementation in R to Python | Data Mining |  |  |  |
| Evaluate MOIRAI time series foundation model<br>on the Context Is Key (CiK) benchmark | Context-aided Forecasting | Climatology, Economics, Energy,<br>Mechanics, Public Safety, Retail,<br>Synthetic, Transportation | Modeling<br>(Using Research Code) | Resolved (Binary) |
| Evaluate Chronos time series foundation model<br>on the NN5 dataset within Context Is Key<br>(CiK) benchmark |  |  |  |  |
| Implement & Evaluate CSDI to Impute PM2.5 Data | Imputation | Weather | Mean Absolute Error |  |
| Train & Evaluate CSDI to Impute PM2.5 Data |  |  |  |  |
| GIFT-EVAL: A Benchmark for General Time Series<br>Forecasting Model Evaluation | Forecasting | Nature, Web, CloudOps,<br>Economics/Finance, Energy,<br>Sales, Transportation, | Mean Absolute Percentage Error |  |
| Hexagon ML UCR Time Series Anomaly Detection | Anomaly Detection | Healthcare, Gait, Energy,<br>Synthetic, Devices | Adjusted Best<br>F1 Score |  |
| Long Horizon Time Series Forecasting<br>Using Time Series Library | Forecasting | Energy, Epidemiology, Finance,<br>Transportation, Weather | Mean Squarred<br>Error |  |
| Long-Horizon Weather Forecasting using<br>Time Series Library’s Itransformer | Forecasting | Weather | Exact Match |  |
| MIT-BIH ECG Arrhythmia Detection | Classification | Healthcare | Accuracy |  |
| MOMENT for Anomaly Detection<br>on UCR datasets | Anomaly Detection | Healthcare, Gait, Energy,<br>Synthetic, Devices | Exact Match |  |
| PTB-XL ECG Classification | Classification | Healthcare | Accuracy |  |
| TimeSeriesExam: A Time Series Understanding Exam | Time Series Understanding | Synthetic | Time Series Understanding | Accuracy |
| Derived Challenges |  |  |  |  |
| Google Brain - Ventilator Pressure Prediction | Regression | Healthcare | Data Handling<br>(Dealing with missing data) | Mean Absolute<br>Error |
| Improve PTB-XL ECG Classification Code | Classification | Healthcare | Code Enhancement<br>(Experiment Tracking,<br>Readability, Reproducibility) |  |
| MIT-BIH Arrhythmia Detection<br>with Weak Supervision | Classification | Healthcare | Data Handling<br>(Labeling) | Accuracy |
| Optiver Realized Volatility Prediction | Forecasting | Finance | Data Handling<br>(Dealing with missing data) | Root Mean Square<br>Percentage Error |
| Optiver Realized Volatility Prediction<br>with Hyper-parameter Optimization | Forecasting | Finance | Modeling (Hyper-parameter<br>Tuning & Model Selection) | Improvement in Root Mean<br>Square Percentage Error |
| PTB-XL ECG Classification<br>with Hyperparameter Optimization | Classification | Healthcare | Improvement in Accuracy |  |

**Table 4: This table provides transparency about the source and licensing information for each challenge in the TimeSeriesGym benchmark. For the Kaggle challenges, most are subject to Kaggle’s competition rules, with a few under Creative Commons licenses. The TimeSeriesGym Original challenges come from diverse sources including GitHub repositories, HuggingFace, etc. with various open-source licenses (Apache, MIT, GPL, Creative Commons). This diversity of sources and licenses demonstrates the benchmark’s foundation in accessible, reusable datasets and code while ensuring proper attribution and compliance with original creators’ terms.**
| Challenge | Source | License |
| --- | --- | --- |
| Kaggle Challenges |  |  |
| AMP-Parkinson’s Disease<br>Progression Prediction | [Kaggle](https://www.kaggle.com/competitions/amp-parkinsons-disease-progression-prediction) | [Subject to Competition Rules](https://www.kaggle.com/competitions/amp-parkinsons-disease-progression-prediction/rules#7.-competition-data.) |
| ASHRAE - Great Energy Predictor III | [Kaggle](https://www.kaggle.com/competitions/ashrae-energy-prediction) | [Subject to Competition Rules](https://www.kaggle.com/competitions/ashrae-energy-prediction/rules#7.-competition-data.) |
| Child Mind Institute–<br>Detect Sleep States | [Kaggle](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview) | [CC BY-NC-SA 4.0](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/rules#7.-competition-data.) |
| Google Brain - Ventilator<br>Pressure Prediction | [Kaggle](https://www.kaggle.com/competitions/ventilator-pressure-prediction) | [Subject to Competition Rules](https://www.kaggle.com/competitions/ventilator-pressure-prediction/rules#7.-competition-data.) |
| G2Net Gravitational<br>Wave Detection | [Kaggle](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection) | [Subject to Competition Rules](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/rules#7-competition-data) |
| HMS - Harmful Brain<br>Activity Classification | [Kaggle](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification) | [CC BY-NC 4.0](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/rules#7.-competition-data.) |
| LANL Earthquake Prediction | [Kaggle](https://www.kaggle.com/competitions/LANL-Earthquake-Prediction) | [Subject to Competition Rules](https://www.kaggle.com/competitions/LANL-Earthquake-Prediction/rules#7-competition-data) |
| M5 Forecasting - Accuracy | [Kaggle](https://www.kaggle.com/competitions/m5-forecasting-accuracy) | [Subject to Competition Rules](https://www.kaggle.com/competitions/m5-forecasting-accuracy/rules#7-competition-data) |
| Online Product Sales | [Kaggle](https://www.kaggle.com/c/online-sales/overview) | [Subject to Competition Rules](https://www.kaggle.com/competitions/online-sales/rules#7-competition-data) |
| Optiver Realized<br>Volatility Prediction | [Kaggle](https://www.kaggle.com/competitions/optiver-realized-volatility-prediction) | [Subject to Competition Rules](https://www.kaggle.com/competitions/optiver-realized-volatility-prediction/rules#7-competition-data) |
| Recruit Restaurant<br>Visitor Forecasting | [Kaggle](https://www.kaggle.com/c/recruit-restaurant-visitor-forecasting) | [Subject to Competition Rules](https://www.kaggle.com/competitions/recruit-restaurant-visitor-forecasting/rules#7-competition-data) |
| Sberbank Russian<br>Housing Market | [Kaggle](https://www.kaggle.com/competitions/sberbank-russian-housing-market/data) | [Subject to Competition Rules](https://www.kaggle.com/competitions/sberbank-russian-housing-market/rules#7-competition-data) |
| TimeSeriesGym Originals |  |  |
| Convert ResNet TensorFlow<br>implementation to PyTorch | [GitHub](https://github.com/hfawaz/dl-4-tsc/blob/master/classifiers/resnet.py) | [GNU General Public License v3.0](https://github.com/hfawaz/dl-4-tsc/blob/master/LICENSE) |
| Convert STOMP Algorithm<br>Implementation in R to Python | [GitHub](https://github.com/matrix-profile-foundation/tsmp) | [Apache License 2.0](https://github.com/matrix-profile-foundation/tsmp/blob/master/LICENSE.md) |
| Evaluate MOIRAI time series<br>foundation model on the<br>Context Is Key (CiK) benchmark | [GitHub](https://github.com/ServiceNow/context-is-key-forecasting) | [Apache License 2.0](https://github.com/ServiceNow/context-is-key-forecasting/blob/main/LICENSE.md) |
| Evaluate Chronos time series<br>foundation model on the NN5 dataset<br>within Context Is Key (CiK) benchmark | – | [Apache License 2.0](https://github.com/ServiceNow/context-is-key-forecasting/blob/main/LICENSE.md) |
| GIFT-EVAL: A Benchmark for General<br>Time Series Forecasting Model Evaluation | [GitHub](https://github.com/SalesforceAIResearch/gift-eval) | [Apache License 2.0](https://github.com/SalesforceAIResearch/gift-eval/blob/main/LICENSE.txt) |
| Hexagon ML UCR Time Series<br>Anomaly Detection | [UCR](https://www.cs.ucr.edu/%C2%A0eamonn/time_series_data_2018/) | Not available |
| Long Horizon Time Series Forecasting<br>Using Time Series Library | [GitHub](https://github.com/thuml/Time-Series-Library) | [MIT License](https://github.com/thuml/Time-Series-Library/blob/main/LICENSE) |
| Long-Horizon Weather Forecasting<br>using Time Series Library’s Itransformer | – | [MIT License](https://github.com/thuml/Time-Series-Library/blob/main/LICENSE) |
| MIT-BIH ECG Arrhythmia Detection | [PhysioNet](https://www.physionet.org/content/mitdb/1.0.0/) | [Open Data Commons Attribution License v1.0](https://www.physionet.org/content/mitdb/view-license/1.0.0/) |
| MOMENT for Anomaly Detection<br>on UCR datasets | [GitHub](https://github.com/moment-timeseries-foundation-model/moment) | [MIT License](https://github.com/moment-timeseries-foundation-model/moment/blob/main/LICENSE) |
| PTB-XL ECG Classification | [PhysioNet](https://physionet.org/content/ptb-xl/1.0.3/) | [Creative Commons Attribution 4.0 International Public License](https://physionet.org/content/ptb-xl/view-license/1.0.3/) |
| TimeSeriesExam: A Time Series<br>Understanding Exam | [Hugging Face](https://huggingface.co/datasets/AutonLab/TimeSeriesExam1) | [MIT License](https://github.com/moment-timeseries-foundation-model/TimeSeriesExam/blob/main/LICENSE) |

**Table 5: TimeSeriesGym -Lite is a streamlined collection of six diverse time series challenges carefully selected to evaluate AI agents while balancing thoroughness with efficiency. The challenges cover essential ML engineering skills including basic data science, handling missing/multi-source data, code migration, hyperparameter optimization, research code modeling, and data labeling. The collection spans multiple domains (healthcare, finance, algorithms) and various time series tasks (classification, forecasting, anomaly detection, code migration). This cost-effective subset allows researchers to quickly benchmark agent capabilities across critical ML engineering skills without the resource requirements of the full TimeSeriesGym benchmark.**
| Challenge | Required Skills | Time Series Task | Domain |
| --- | --- | --- | --- |
| Child Mind Institute - | Basic data science | Classification | Healthcare |
| Detect Sleep States | (data handling and modeling) |  |  |
| Optiver Realized | Handling missing and | Forecasting | Finance |
| Volatility Prediction | multi-source data |  |  |
| Convert ResNet TensorFlow | Classification | Code Migration | Algorithm |
| implementation to PyTorch |  |  |  |
| PTB-XL ECG | Hyperparameter optimization | Classification | Healthcare |
| Classification | & model selection |  |  |
| MOMENT Anomaly | Modeling | Anomaly Detection | Healthcare, Gait, |
| Score Calculation | (Using research code) | Synthetic, Energy, Devices |  |
| MIT-BIH Arrhythmia | Data labeling | Classification | Healthcare |
| Detection |  |  |  |

**Table 6: Average cost to run experiments on a single seed in the default evaluation setup i.e. AIDE with gpt-4.1-2025-04-14 with a maximum of 4 hours and 50 steps.**
| Benchmark | Averge Cost (USD) |
| --- | --- |
| TimeSeriesGym | 62.12 |
| TimeSeriesGym -Lite | 7.96 |

## Appendix B Implementation Details for Scaffolds

**Table 7: Scaffold hyperparameters. $TARGET_MODEL denotes the model being evaluated.**
| AIDE |  |
| --- | --- |
| agent.code.model | $TARGET_MODEL |
| agent.feedback.model | gpt-4.1-2025-04-14 |
| agent.steps | 50 |
| agent.search.max_debug_depth | 20 |
| agent.search.debug_prob | 1 |
| agent.time_limit | 14400 |
| exec.timeout | 32400 |
| OpenHands |  |
| agent | CodeActAgent |
| model | $TARGET_MODEL |
| max_time_in_hours | 4 |
| max_steps | 50 |

Table [7](#A2.T7) summarizes the hyperparameters used for the two scaffolds: AIDE [11] and OpenHands’ CodeAct [21].
We did not directly use the MLE-bench [3] modifications to the agentic scaffolds for two main reasons. First, the official scaffold implementation has undergone updates since MLE-bench. Second, certain changes made in MLE-bench were not applicable to our benchmark, which involves more diverse modes of evaluation. Therefore, we modified the latest version of the agentic scaffolds to enhance robustness, improve execution stability, and support a broader range of competitions.

### B.1 AIDE

We forked the original AIDE repository in May 2025 and added useful changes from the MLE-bench project. These include better API calls and support for more API providers. Our key modifications are summarized below:

- 1.
Updated instructions to cover all types of tasks, not just Kaggle challenges
- 2.
Modified the prompt to work with different file types, since our tasks use many input/output formats
- 3.
Fixed the interpreter handling to prevent timeouts and system hangs

### B.2 OpenHands

We forked the original OpenHands repository from tag v0.34.0 (May 2025). We reduced the RAM allowance to 10 GiB (from 100 GiB) as we did not observe any memory-related issues during our tests.

Figure: Figure 3: The prompt we use to initialize all scaffolds, adapted from MLE-bench [3].

## Appendix C Detailed Evaluation Results

### C.1 Full Benchmark Evaluation Result

**Table 8: Comprehensive performance metrics for AI agents on all TimeSeriesGym challenges, including best and average scores from three runs. Agents struggle to solve TimeSeriesGym Original challenges. Derived challenges demonstrate how added complexity (missingness, hyperparameter optimization) affects performance. These results highlight both the capabilities and limitations of current ML engineering agents across diverse time series tasks.**
| Challenge | Evaluation Metric | Best @ 3 | Average @ 3 |
| --- | --- | --- | --- |
| Kaggle Challenges |  |  |  |
| AMP-Parkinson’s Disease<br>Progression Prediction | Symmetric Mean Absolute<br>Percentage Error | 111.22 | 120.50 |
| ASHRAE - Great Energy Predictor III | Root Mean Square<br>Logarithmic Error | 1.02 | 1.92 |
| Child Mind Institute– Detect Sleep States | Event Detection<br>Average Precision | 0.02 | 0.01 |
| Google Brain - Ventilator Pressure Prediction | Mean Absolute Error | 0.58 | 5.40 |
| G2Net Gravitational Wave Detection | Area Under<br>ROC Curve | 0.51 | 0.50 |
| HMS - Harmful Brain Activity Classification | KL Divergence | 1.16 | 1.56 |
| LANL Earthquake Prediction | Mean Absolute Error | 2.18 | 2.89 |
| M5 Forecasting - Accuracy | Weighted Root Mean Squared<br>Scaled Error | 0.82 | 3.13 |
| Online Product Sales | Root Mean Square<br>Logarithmic Error | 0.91 | 1.08 |
| Optiver Realized Volatility Prediction | Root Mean Square<br>Percentage Error | 0.28 | 0.30 |
| Recruit Restaurant Visitor Forecasting | Root Mean Square<br>Logarithmic Error | 0.55 | 0.60 |
| Sberbank Russian Housing Market | Root Mean Square<br>Logarithmic Error | 0.39 | 0.40 |
| TimeSeriesGym Originals |  |  |  |
| Convert ResNet TensorFlow<br>implementation to PyTorch | Custom Code Grading Test Cases | 5/9 | 5/9 |
| Convert STOMP Algorithm<br>implementation in R to Python | Custom Code Grading Test Cases | 2/4 | 1.6/4 |
| Evaluate MOIRAI time series foundation model<br>on the Context Is Key (CiK) benchmark | Resolved (Binary) | N/A | N/A |
| Evaluate Chronos time series foundation model<br>on the NN5 dataset within Context Is Key<br>(CiK) benchmark | Resolved (Binary) | N/A | N/A |
| Implement & Evaluate CSDI to Impute PM2.5 Data | Mean Absolute Error | N/A | N/A |
| Train & Evaluate CSDI to Impute PM2.5 Data | Mean Absolute Error | N/A | N/A |
| GIFT-EVAL: A Benchmark for General Time Series*<br>Forecasting Model Evaluation | Mean Absolute Percentage Error | N/A | N/A |
| Hexagon ML UCR Time Series Anomaly Detection* | Adjusted Best<br>F1 Score | 0.38 | 0.38 |
| Long Horizon Time Series Forecasting<br>Using Time Series Library | Mean Squarred<br>Error | N/A | N/A |
| Long-Horizon Weather Forecasting using<br>Time Series Library’s Itransformer | Exact Match (Binary) | N/A | N/A |
| MIT-BIH ECG Arrhythmia Detection | Accuracy | 0.87 | 0.84 |
| MOMENT for Anomaly Detection<br>on UCR datasets | Exact Match (Binary) | N/A | N/A |
| PTB-XL ECG Classification | Accuracy | 0.81 | 0.80 |
| TimeSeriesExam: A Time Series Understanding Exam | Accuracy | N/A | N/A |
| Derived Challenges |  |  |  |
| Google Brain - Ventilator Pressure Prediction |  |  |  |
| With Missingness | Mean Absolute<br>Error | 2.72 | 6.66 |
| Improve PTB-XL ECG Classification Code | Code Enhancement<br>(Experiment Tracking,<br>Readability, Reproducibility) | N/A | N/A |
| MIT-BIH Arrhythmia Detection<br>with Weak Supervision | Accuracy | 0.87 | 0.77 |
| Optiver Realized Volatility Prediction |  |  |  |
| With Missingness | Root Mean Square<br>Percentage Error | 0.33 | 0.33 |
| Optiver Realized Volatility Prediction<br>with Hyper-parameter Optimization | Improvement in Root Mean<br>Square Percentage Error | -0.01 | -0.15 |
| PTB-XL ECG Classification<br>with Hyperparameter Optimization | Improvement in Accuracy | 0.08 | 0.03 |

We provide detailed evaluation results for each task in TimeSeriesGym in Table [8](#A3.T8). Each task was executed with three random seeds; we report both the average and best scores across these runs. Entries marked N/A indicate that the agent failed to produce a valid solution due to exceeding the time- or step-limit. For the GIFT-Eval and UCR Anomaly Detection challenges, evaluation is performed on a subset of the original benchmark, since our focus is on assessing the agent’s ability to leverage the research repository rather than full benchmark performance.

### C.2 Ablation Study Evaluation Result

**Table 9: This table presents detailed ablation study results comparing agent performance across six different configurations on the TimeSeriesGym-Lite benchmark. Each cell shows Best@3/Avg@3 scores, with N/A indicating no valid solutions. The experiments compare time variations (8 hours/100 steps vs. 12 hours/150 steps), scaffold differences (OpenHands), model types (o3, claude-3-7), and whether agents are reminded of remaining time. Results show mixed effects of increased time allocation, with certain challenges (ResNet conversion) benefiting significantly while others show minimal improvements or even degradation. Both model type and scaffold selection substantially impact performance, with different models excelling on different challenges. This highlights the complex interplay between agent configurations and task types in ML engineering.**
| Challenge | 8 hours / 100 steps | 12 hours / 150 steps | OpenHands | o3 | claude-3-7 | No Reminder |
| --- | --- | --- | --- | --- | --- | --- |
| Child Mind Institute––<br>Detect Sleep States | 0.02 / 0.02 | 0.00 / 0.00 | 0.00 / 0.00 | 0.11 / 0.11 | N/A | N/A |
| Optiver Realized Volatility<br>Prediction with Missingness | 0.32 / 0.33 | 0.31 / 0.31 | 0.64 / 0.64 | 0.42 / 0.43 | 0.25 / 0.25 | 0.32 / 0.32 |
| Convert ResNet TensorFlow to PyTorch | 0.56 / 0.56 | 0.89 / 0.89 | 0.56 / 0.44 | 0.56 / 0.56 | 0.89 / 0.78 | 0.56 / 0.56 |
| PTB-XL ECG Classification with<br>Hyperparameter Search | 0.45 / 0.22 | 0.45 / 0.10 | N/A | 0.14 / 0.10 | 0.09 / 0.06 | 0.05 / 0.03 |
| MOMENT Anomaly Score Calculation | N/A | N/A | 0.00 / 0.00 | 0.00 / 0.00 | N/A | N/A |
| MIT-BIH Arrhythmia Detection<br>with Weak Supervision | 0.83 / 0.56 | 0.80 / 0.60 | 0.73 / 0.72 | 0.53 / 0.45 | 0.79 / 0.66 | 0.74 / 0.68 |

## Appendix D Failure Mode Illustration

### D.1 Agents Miss Important Information

Figure: Figure 4: OpenHands wastes 5 steps on inspecting model file while the correct way to import the model is in README

As illustrated in Listing [4](#A4.F4), OpenHands spends five consecutive steps scanning to the end of the model file in an attempt to discover the correct import method for MOMENT. This behavior reveals two critical shortcomings. First, the agent follows a greedy, linear scanning strategy with no early-stop criterion or hierarchical search plan—it blindly paginates through the file rather than formulating a focused query. Second, it fails to leverage the README, which explicitly documents the proper import instructions. Together, these issues demonstrate a lack of strategic planning and contextual awareness. We observed a similar pattern in AIDE, where blind iteration and omission of available documentation likewise impede efficient problem solving.

### D.2 AIDE Interpreter Execution Can Trigger Undesirable Behavior

Figure: Figure 5: AIDE ’s interpreter does not execute code under main environment

Shown in Listing [5](#A4.F5), AIDE invokes Python’s exec in a persistent global_scope, then employs an LLM-based “judge” to inspect the generated code and its stdout. Any logic guarded by
if __name__ == "__main__":
will be skipped—because global_scope does not set __name__ to "__main__". As a result, the judge may erroneously declare such runs valid, even when critical execution paths never occur, and further retries or debug steps cannot correct this oversight.

### D.3 AIDE Single-File Approach is Error-prone

Figure: Figure 6: AIDE’s solution for utilizing Time-Series-Library repository

As shown in Listing [6](#A4.F6), AIDE encapsulates the entire forecasting workflow in a single script. Whenever it must invoke system commands, it relies on Python’s subprocess module—an approach that can obscure full tracebacks and miss intermediate errors. Furthermore, to import modules from the research repository, AIDE repeatedly alters the Python search path or changes the working directory (e.g., via sys.path.append), which is inefficient and brittle.

## Appendix E Two-Faceted Grading Approach

We implemented a dual grading methodology to provide comprehensive evaluation of submitted solutions. This section introduces the general concept of our two complementary evaluation approaches: exact grading and judge-based grading.

### E.1 Grading Methodologies

#### E.1.1 Exact Grading Methodology

The exact grading approach represents a deterministic, checklist-based evaluation focused on verifying specific required components. This objective method evaluates submissions against explicit criteria with binary pass/fail outcomes, providing clear feedback on technical requirements. The exact grading methodology emphasizes quantifiable metrics and compliance with predefined specifications.

Key aspects of exact grading include:

- •
Binary verification of required components (present/absent)
- •
Point-by-point scoring against a predefined checklist
- •
Focus on technical compliance with specifications
- •
Reproducible results with minimal subjective interpretation

#### E.1.2 Judge-Based Qualitative Methodology

The judge-based approach provides a nuanced evaluation that assesses artifacts beyond mere presence of required components. This method employs large language models (LLMs) as judges to evaluate submissions against custom criteria with chain-of-thought reasoning.

Key aspects of judge grading include:

- •
Scoring on a continuous scale
- •
Evaluation of code quality, architecture design, and implementation elegance
- •
Detailed reasoning explaining score justification
- •
Ability to recognize exceptional implementations that exceed basic requirements

### E.2 Implementation for PTB-XL Classification Challenge

#### E.2.1 Exact Grading Implementation

For the PTB-XL Classification Challenge, our exact grading implementation evaluates code artifact submissions through:

- 1.
Feature Extraction: Using regular expression pattern matching and AST parsing to identify required code components.
- 2.
Binary Verification: Checking each requirement against pass/fail criteria.
- 3.
Static Analysis: Using linting tools to check against PEP 8 standards.
- 4.
File Structure Validation: Verifying required files and directories.

The exact grading for this challenge evaluates four primary categories, each worth 25% of the final score:

- •
TensorBoard Usage: Proper imports, SummaryWriter initialization, metric logging, etc.
- •
Code Quality: Syntax verification, docstrings, type annotations, and PEP 8 compliance.
- •
Hydra Configuration: Proper imports, decorator usage, and configuration files.
- •
Model Accuracy: Prediction accuracy against ground truth labels.

#### E.2.2 Judge-Based Implementation

For this challenge, we employed G-Eval [14], a framework that uses LLMs with chain-of-thought reasoning. The implementation evaluates code through:

- 1.
Evaluation Steps: Using predefined steps for chain-of-thought reasoning.
- 2.
Comprehensive Assessment: Evaluating multiple parameters including code structure and architecture decisions.
- 3.
Score Calculation: Generating normalized scores on a 0.0-1.0 scale.
- 4.
Reasoning Provision: Providing detailed explanations for the evaluation.

### E.3 Comparative Analysis

The two approaches serve complementary purposes:

**Table 10: Transposed Comparison of Exact and Judge-Based Grading Approaches**
| Aspect | Exact Grading | Judge Grading |
| --- | --- | --- |
| Objectivity | High (deterministic) | Moderate (LLM-based) |
| Granularity | Binary (present/absent) | Continuous (quality scores) |
| Feedback Detail | Limited (requirement verification) | Rich (explanatory reasoning) |
| Reproducibility | High (automated) | Moderate (LLM consistency) |
| Evaluation Scope | Technical compliance | Code quality, effectiveness |
| Methodology | Rule-based checks | LLM with chain-of-thought |
| Scalability | Low (manual rule design) | High (natural language criteria) |

### E.4 Combined Grading Benefits

Using both approaches provides several advantages:

- •
Ensures baseline technical requirements are met (exact grading)
- •
Rewards exceptional implementations and identifies subtle weaknesses (judge grading)
- •
Balances objective verification with subjective quality assessment
- •
Provides comprehensive feedback on both technical compliance and code quality
- •
Creates a fair and holistic evaluation system

### E.5 Grading Examples for PTB-XL Challenge

Below are example outputs from both grading systems applied to the same submission for the PTB-XL Classification Challenge.

#### E.5.1 Exact Grading Output

#### E.5.2 Judge-Based Grading Output

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: See the weakness.
soundness: 3
strengths: Identifies a Clear and Important Gap: The paper is correct that time series is an underserved domain in agentic benchmarking. Creating a dedicated benchmark for this is a valuable contribution.

Focuses on "ML Engineering," Not Just Modeling: The strongest part of the paper is its inclusion of "TimeSeriesGym Originals" (Table 4). Challenges like "Convert ResNet TensorFlow implementation to PyTorch" or "Improve PTB-XL ECG Classification Code" are excellent, real-world tasks that go beyond the standard "train-a-model-on-a-CSV" format.

Holistic Evaluation Concept: The ambition to evaluate multiple artifacts (code, models) and use a "two-faceted grading approach" (quantitative and qualitative, see Appendix E) is the right direction for a comprehensive benchmark.

Scalability as a Design Goal: Designing the benchmark to be scalable from the start (even if not fully proven) is a smart, forward-thinking approach that addresses a major flaw in static benchmarks.
summary: The authors introduce TimeSeriesGym, a new benchmarking framework designed to evaluate AI agents on machine learning (ML) engineering tasks specifically for the time series domain. The paper argues that existing benchmarks are flawed, focusing too narrowly on well-defined Kaggle-style problems, lacking scalability, and only evaluating final submission files. TimeSeriesGym aims to solve this by sourcing diverse challenges (including 20 "Original" tasks based on real-world engineering like code migration and hyperparameter tuning), designing a framework to grade multiple artifacts (code, models, and submissions), and employing a holistic evaluation that includes both quantitative metrics and qualitative "LLM-as-a-judge" assessments. The framework also claims to be scalable with tools for new challenge creation. The paper's main experimental finding is that current state-of-the-art agents perform very poorly on these tasks, often failing to produce even a "reasonable" submission.
weaknesses: **Critically Flawed Evaluation of Kaggle Challenges:**

The paper's primary evaluation metrics, "Valid Submission (%)" and "Reasonable Submission (%)," are insufficient.

- The benchmark fails to report the actual leaderboard scores or ranks for the 13 included Kaggle competitions. This is a significant omission, as these are the most standardized, competitive tasks in the dataset.

- The bar for a "Reasonable Submission" is set at scoring "above median on the competition's public leaderboard" (Section 4). This threshold is exceptionally low and provides no real insight into an agent's capabilities. An agent that barely surpasses the median is treated identically to one that achieves a state-of-the-art, gold-medal-winning score.

- This lack of granular, comparable results (as seen in Table 10) makes it impossible for the research community to evaluate an agent's true performance or benchmark progress against established human or SOTA baselines on these problems.

**Conceded Risk of Data Leakage:**
The paper's reliance on popular, public Kaggle competitions fundamentally compromises its ability to evaluate modern LLM-based agents. The authors' defense—that agents "performed poorly anyway" or that the benchmark is "scalable" to add new tasks—is unconvincing. It effectively concedes that the current Kaggle-based portion of the benchmark is unsuitable for reliably evaluating frontier models, as performance may be confounded by memorization.

Subjective and Non-Scalable Evaluation Protocol:

- The paper's claim of being a "scalable benchmark" is directly undermined by its own evaluation methodology for the "Originals" challenges.

- For non-Kaggle tasks, "reasonableness" is determined by the authors "manually inspecting" if a "genuine modeling attempt" was made (Section 4, Metrics). A benchmark that requires subjective, manual intervention from its creators for a primary metric is, by definition, not scalable or objective.

- The proposal to use "LLM-as-a-judge" (Appendix E) as a secondary evaluation method introduces a notoriously unreliable, biased, and difficult-to-reproduce component, which is not a substitute for rigorous, objective, quantitative metrics.

- The authors themselves admit the protocol's flaws in Section 5 ("Defining and measuring success"), stating that "our current evaluation approaches have inherent limitations."

**Miscalibrated Difficulty (Floor Effects):**

- The reported results are so poor that the benchmark, in its current form, largely fails as a diagnostic tool.

- The best agent on the full benchmark (AIDE + GPT-4.1) only achieved a "reasonable" submission 12.1% of the time (Section 4.1).

- Even on the hand-picked, "easy" Lite benchmark, the best agent (AIDE + Claude 3.7) only succeeded 38.9% of the time (Table 2).

These results indicate a significant floor effect. The tasks are currently too difficult for SOTA agents, preventing any meaningful differentiation between models or scaffolds. The benchmark primarily demonstrates that all agents fail, offering little insight into why they fail or which approaches are more promising.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1、 As the authors noted, due to funding and time constraints, most experiments were conducted on a Lite subset of 6 tasks. This raises two concerns: (A) Can the Lite subset sufficiently represent the full benchmark? Although the authors selected diverse tasks, the sample size of six remains relatively small. (B)Some results—such as the extension of agent steps to 12 hours—were only tested on the Lite subset. It is unclear whether similar trends would hold across the full benchmark.

2、 Additionally, at line 76 below Figure 1, there appears to be white-colored text revealing the authors' institutional affiliation (“Ⓒ 2025 Auton Lab, Carnegie Mellon University”). This may violate the double-blind review policy and should be addressed.
soundness: 3
strengths: 1、 The benchmark collects and designs tasks based on real-world data science scenarios, including Kaggle competition problems and practical research tasks such as code migration and model evaluation. These challenges span a wide range of skills—including data processing, model construction, and code understanding and adaptation—reflecting the multifaceted challenges faced by real-world machine learning engineers.

2、 TimeSeriesGym evaluates multiple forms of agent outputs, not only focusing on prediction accuracy and error metrics, but also assessing code generation, model artifacts, and more. Additionally, it introduces optional LLM-based review mechanisms (e.g., code auditing) to supplement quality evaluation.

3、 The authors provide tooling mechanisms that enable large-scale automatic generation of new tasks, making the benchmark highly extensible and sustainable, with the ability to continuously incorporate new challenges.
summary: This paper proposes a benchmark framework named TimeSeriesGym, designed to evaluate the capabilities of AI agents in time series machine learning engineering tasks. The core idea is to provide a scalable and agent-agnostic evaluation environment that encompasses time series challenges across multiple domains, while assessing various outputs generated by agents—including prediction files, code, and models.
weaknesses: 1、 The primary contribution of the paper lies in the construction of the benchmark. Many of its ideas—such as using LLMs for code review, incorporating multi-source tasks, and adopting multi-metric evaluations—are extensions and integrations of existing work rather than entirely novel innovations.

2、 Although the paper lists several existing benchmarks, the distinctions and connections between TimeSeriesGym and those benchmarks are not sufficiently elaborated. For example, beyond the domain difference, how does TimeSeriesGym improve upon the evaluation philosophy of MLE-Bench? A deeper comparative analysis would strengthen the positioning of the proposed framework.

3、 Among the 33 current challenges, tasks sourced from Kaggle tend to be well-structured and may have standard solutions, while the original TimeSeriesGym tasks are often highly complex. GPT-4 achieves only a 12.1% “reasonable solution” rate on TimeSeriesGym, with many tasks yielding no valid outputs. It is recommended that future task sets include medium-difficulty challenges to ensure a smoother difficulty gradient, enabling better tracking of agent performance from beginner to advanced levels. Additionally, establishing baselines for each task—such as simple algorithms or human-level performance—would help characterize task difficulty and the potential improvement space for agents.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Can you provide stratified results by task family with confidence intervals and tests for significance. This would help to understand which capabilities drive aggregate gains.

2. How do you ensure that the Lite subset is representative of the full benchmark. Please include quantitative evidence of coverage and difficulty distribution.

3. Can you formalize the validity and reasonableness criteria using symbols and thresholds and show calibration plots or decision curves.

4. What steps can be taken to reduce reliance on leaderboard medians or external competitions that may change over time.
soundness: 3
strengths: - Clear motivation that time series engineering requires more than single step prediction and that agents should be evaluated on multi stage workflows.

- The authors collected a large number of datasets, with broad task coverage and inclusion of multiple domains which improves the ecological validity of the benchmark.

- In this paper, multi artifact evaluation that considers predictions, code quality checks, and trained models which aligns the benchmark with real practice.

- Agent agnostic design that supports different frameworks and enables trajectory collection for future training.

- A documented process for adding new tasks which supports scalability and long term maintenance.
summary: This paper presents TimeSeriesGym, a benchmark and environment for evaluating agents that perform end-to-end machine learning work on time series problems. The benchmark aggregates a collection of challenges that span forecasting, classification, anomaly detection, data cleaning, hyperparameter tuning, and code migration. It evaluates not only final predictions but also multiple artifacts such as code, trained models, and submission files. It defines aggregate metrics for validity and reasonableness and reports results across different toolchains and foundation models. The paper argues that time series is an underrepresented setting for agent evaluation and that real engineering workflows require capabilities beyond model inference. The experimental section compares several agent frameworks under controlled resources and reports ablations on time budgets and guidance strategies. The authors claim that TimeSeriesGym is scalable, extensible, and better aligned with the realities of time series engineering than prior leaderboards that focus on a single metric or a single output file.
weaknesses: 1. The paper mixes the benchmark, the task generation mechanism, the multi artifact scoring, the trajectory data loop, and the time series focus without a clear primary to secondary hierarchy and without a concise contribution figure. It does not decompose difficulty across different time series task families and it does not design difficulty along agent reasoning dimensions.

2. Due to compute constraints most experiments are conducted on the Lite set of six tasks. Although the authors state that these tasks cover key skills there is no statistical validation of domain diversity or difficulty distribution which can bias conclusions toward low cost scenarios.

3. The contribution is primarily engineering. The paper lacks a unifying methodological or theoretical insight. Although this is a benchmark paper it would benefit from a central methodological principle that guides design choices across sections.

4. The table presents results for 4/50→12/150 and for Step wise reminder vs No reminder, but it does not include significance testing.

5. The definitions of Valid submission and Reasonable submission appear in prose in the methods section without a unified symbolization or an explicit decision boundary. This reduces clarity and hurts reproducibility.

---

confidence: 3
contribution: 1
initial_rating: 2
presentation: 2
questions: The paper lacks clarity. It's not clear how exactly the challenges are defined, are they all manually crafted based on Kaggle and Github Repos? Line 155-163 shows what each challenge is made or but is it manual effort to create these information based on the dataset or repository? 

In line 315-316, it's mentioned that "simply loading and re-saving the provided sample submission file without any model inference or data processing is deemed unreasonable". How is this checked? Is it through llm-as-a-judge evaluation or manual inspection? 

In line 228-231, the paper mentions that it can grade diverse artifacts generated throughout the MLE life cycle but how? I don't think this is explained. Rather than spending almost a whole page on limitation/future work/open questions, I would suggest spending more text to improve the clarity of the paper and explaining details of this benchmark framework in a thorough manner.
soundness: 2
strengths: This benchmark fills a gap on time series ML pipeline among existing ML benchmarks. 

The proposed benchmark offers specific skills evaluation and hybrid scoring of qualitative + quantitive evaluation.
summary: This paper introduces an open-source benchmark TimeSeriesGym for MLE agents evaluation, mostly focused on time series data. It contains 33 challenges and evaluated 3 scaffolds of agents with each scaffold evaluated on up to 3 base models. The framework supports multimodal output evaluation, specific skill evaluation and holistic evaluation(quantitative & qualitative).
weaknesses: This paper emphasizes benchmark at scale which would require limited human efforts but there is no discussion on how exactly new challenges should be adapted to the time series gym. It's mentioned that new challenge can be added within two hours, but how is the quality of the generated new challenges? In addition, two hours time is probably also enough to set up a specialized evaluation for a challenge without the need to integrate it to the current evaluation framework. The evidence for claimed scalability is limited, at least not well-presented. 

It feels like this paper is mostly doing the same thing as MLE-bench but with specifically curated challenges focusing on time series and additionally took inspiration from other benchmarks and combined their advantages such as multimodal, skill-based, and holistic evaluation. 

The experiment findings are also very predictable. Existing agents don't have the full capability to automate complete ML pipeline and highly likely the reasoning model produces more reasonable solutions. It's hard to find new takeaways or interesting insights from this paper. 

Some experiment settings are not meaningful. For example, providing remaining time reminders to agents vs. without time reminders. I find it hard to see the value in this experiment. 

The evaluation is mostly limited to gpt 4.1 model, and there is no open source models being evaluated at all. 

Some challenges feel very synthetic. For example, implement MOMENT for anomaly detection task. But why do we need to integrate such task into this benchmark at all given that the code to solve this task is already available in MOMENT official github repo.


Decision: Reject