Title: PULSE: Benchmarking Large Language Models for ICU Time Series Classification
Primary area: datasets and benchmarks
Keywords: icu, intensive care unit, clinical, benchmark, large language model, llm, agents, evaluation, classification

Abstract: Large language models (LLMs) are increasingly used on multimodal clinical data, yet their performance on high-stakes ICU time series lacks a standardized evaluation. We introduce PULSE, a comprehensive LLM benchmark for intensive care unit (ICU) time series classification. PULSE evaluates 17 models, strong conventional learners, deep learning, and instruction-following LLMs under nine prompting/agentic strategies, across three datasets (HiRID, MIMIC-IV, eICU) and three clinical endpoints (mortality, sepsis, acute kidney injury). Across settings, well-tuned conventional models, especially gradient-boosted trees, remain the most robust and best-performing, reaching AUROCs up to 0.906. Frontier LLMs approach this level (best 0.889 AUROC, OpenAI o3) but show higher variance and sensitivity to prompting. A hybrid agentic approach that grounds LLM reasoning in a conventional model’s predictions consistently narrows the gap while also providing human-readable explanations at inference time. Notably, on several inferences, LLMs are competitive with classic models despite not being trained on multimodal time-series data. PULSE provides all code, configuration files, and a public results dashboard to enable transparent, reproducible comparison and rapid community extension. We expect PULSE to serve as a common yardstick for developing safer, more reliable LLMs for multimodal time series data, in critical care.

TL;DR: The First LLM Benchmark for ICU Time Series