Can LLMs Understand Time Series Anomalies?
Authors: Zihao Zhou
Rose Yu
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models (LLMs)
Time Series Analysis
Anomaly Detection
Multimodal Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: This study challenges common assumptions about Large Language Models' capabilities in time series anomaly detection.

Abstract: Large Language Models (LLMs) have gained popularity in time series forecasting, but their potential for anomaly detection remains largely unexplored. Our study investigates whether LLMs can understand and detect anomalies in time series data, focusing on zero-shot and few-shot scenarios. Inspired by conjectures about LLMs' behavior from time series forecasting research, we formulate key hypotheses about LLMs' capabilities in time series anomaly detection. We design and conduct principled experiments to test each of these hypotheses. Our investigation reveals several surprising findings about LLMs for time series: (1) LLMs understand time series better as *images* rather than as text, (2) LLMs do not demonstrate enhanced performance when prompted to engage in *explicit reasoning* about time series analysis. (3) Contrary to common beliefs, LLMs' understanding of time series *do not* stem from their repetition biases or arithmetic abilities. (4) LLMs' behaviors and performance in time series analysis *vary significantly* across different models. This study provides the first comprehensive analysis of contemporary LLM capabilities in time series anomaly detection. Our results suggest that while LLMs can understand trivial time series anomalies (we have no evidence that they can understand more subtle real-world anomalies), many common conjectures based on their reasoning capabilities do not hold. All synthetic dataset generators, final prompts, and evaluation scripts have been made available in https://github.com/rose-stl-lab/anomllm.

Published as a conference paper at ICLR 2025
CAN LLMs UNDERSTAND TIME SERIES ANOMALIES?
Zihao Zhou
Rose Yu
Dept of Computer Science and Engineering University of California, San Diego La Jolla, CA 92093, USA
Dept of Computer Science and Engineering University of California, San Diego La Jolla, CA 92093, USA
ciz244Qucsd.edd edu
roseyu@ucsd.edd
ABSTRACT
Large Language Models (LLMs) have gained popularity in time series forecast- ing, but their potential for anomaly detection remains largely unexplored. Our study investigates whether LLMs can understand and detect anomalies in time series data, focusing on zero-shot and few-shot scenarios. Inspired by conjec- tures about LLMs' behavior from time series forecasting research, we formu- late key hypotheses about LLMSs' capabilities in time series anomaly detection. We design and conduct principled experiments to test each of these hypotheses. Our investigation reveals several surprising findings about LLMs for time series: (1) LLMs understand time series better as images rather than as text, (2) LLMs do not demonstrate enhanced performance when prompted to engage in explicit reasoning about time series analysis. (3) Contrary to common beliefs, LLMs understanding of time series does not stem from their repetition biases or arith- metic abilities. (4) LLMs' behaviors and performance in time series analysis vary significantly across different models. This study provides the first comprehen- sive analysis of contemporary LLM capabilities time series anomaly detection. Our results suggest that while LLMs can understand trivial time series anomalies, we have no evidence that they can understand more subtle real-world anomalies. Many common conjectures based on their reasoning capabilities do not hold. All synthetic dataset generators, final prompts, and evaluation scripts have been made available in hnttps://githhin cow.ooo-scoooooo----ee-- larlllllaaannomlm.m
INTRODUCTION
The remarkable progress in large language models (LLMs) has led to their application in various domains, including time series analysis. Recent studies have demonstrated LLMs' potential as zero- shot and few-shot learners in tasks such as forecasting and classification (Gruver al.) 2023 Liu jet al., 2024c). However, the effectiveness of LLMs in time series analysis remains subject of debate. While some researchers argue that LLMs can leverage their pretrained knowledge to derstand time series patterns (Gruver et al. 2023), others suggest that simpler models can match or outperform LLMs (Tan al. 2024).
This debate raises a fundamental question: Do LLMs truly understand time series? To address this question, we must look beyond models' predictive performance. Forecasting typically hinges on metrics like MSE, which can overlook deeper model understanding. A model that simply outputs nearly constant line might still achieve a passable MSE but reveals little about its capacity to interpret dynamics. Shifting our focus to anomaly detection changes the game: it forces LLMS to pinpoint irregular behavior and thus tests whether they grasp the underlying patterns, not just how well they extrapolate average.
In this paper, we present the first comprehensive investigation into LLMs' understanding of time series data through the lens of anomaly detection. We focus on state-of-the-art LLMs and mul- timodal LLMs (M-LLMs) across different anomaly types under controlled conditions. Our eval- uation strategy incorporates multimodal inputs (textual and visual representations of time series), various prompting techniques, and structured output formats. The results are quantitatively assessed using the affinity F1 score. We provide empirical evidence to challenge existing conjectures and

---

Published as a conference paper at ICLR 2025
claims about LLMs' time series understanding. Our findings reveal more nuanced understanding of LLMs' capabilities and limitations in time series data, including:
Visual Advantage: LLMs perform significantly better in anomaly detection when processing time series data as images rather than text tokens.
Limited Reasoning: LLMs do not benefit from chain-of thain-of-thought reasoning when analyzing time series data. Their performance often decreases when prompted to explain their reasoning.
Alternative Processing Mechanisms: Contrary to common beliefs, LLMs' understanding of time series does not stem from their repetition biases arithmetic abilities, challenging prevailing assumptions about how these models process numerical data.
Model Heterogeneity: Time series understanding and anomaly detection capabilities differ across various LLM architectures, highlighting the importance of model selection.
In summary, while LLMs can detect simple anomalies, their abilities to understand and reason about numerical time series are considerably limited. Our research calls for more caution in applying LLMs time series and for controlled studies examine LLMs' behavior for other data modalities.
RELATED WORK
LLMs for Time Series Analysis. LLMs have been applied various time series analysis tasks, with recent literature establishing strong claims about their capabilities, Gruver et al. (2023) showed that LLMs such as GPT-3 and LLaMA-2 possess broad pattern extrapolation capabilities, enabling zero-shot time series forecasting by encoding time series strings of numerical digits and achieving comparable performance to purpose-built built models. Building on these claims, [Lu et al. (2024c) pro- posed Cross-Modas LLM Fine-Tuning framework, suggesting that LLMs can provide interpretable predictions while addressing the distribution discrepancy between textual and temporal input tokens in multivariate time series forecasting. Liu et al. (2024b) introduced Time-MMD
multi-domal multimodal time series dataset for LLM finetuning. For anomaly detection, Liu et al. (2024a) pro- knowledge distillation-batt
approach using GPT-2 as the teacher network. Zhang et al. (024) provided a comprehensive survey of LLM applications in time series analysis. In the financial domain, Wimmer & Rekabsaz (2023) used vision language models, i.e., CLIP, but not M-LLMS to process visualizations of stock data for market change prediction.
posed A AnomalyL.MM
However, these prevailing beliefs about LLMs' pattern extrapolation capabilities and interpretable predictions remain controversial, [Zeng et al. (2023) argued that the permutation-inyarian nature of self-attention mechanisms may lead loss of critical temporal information. Tan al. (2024) found that removing the LLM component or replacing it with a basic attention layer often improved per- formance in popular J.L.-baseed forecasting methods, challenging the assumed benefits of LLMs' pattern recognition abilities. The interpretability of LLMSs in time series analysis remains a chal- lenge, as their reasoning capabilities are often opaque and difficult to interpret.
Time Series Anomaly Detection. Time series anomaly detection is critical task in various do- mains, including finance, healthcare, and cybersecurity. Traditional methods rely on statistical tech- niques, while recent work has focused on developing deep learning- basedd approaches (Audibert) et al. [022] Chen et al., 2022. Tuli et al. 2022). Audibert et al. (2022) compared conventional, Seaiine--eaaiin-abeeeeeed and deep neural network methods, finding that no family of methods con- sistently outperforms the others. Chen et al. (2022) proposed a deep variational graph convolutional recurrent network for multivariate time series anomaly detection. Tuli et al. (2022) introduced transformer-based anomaly detection model with adversarial training and meta- mneta-lerrning.
However, recent studies have highlighted significant flaws in current time series anomaly detection benchmarks and evaluation practices (Wu & Keogh [021] Huet et al.) 2022 Sarfraz et al.) 2024).
Wu & Keogh [2021) argued that popular benchmark datasets suffer from major flaws like triviality and mislabeling, potentially creating an illusion of progress in the field. Huet et al. (2022) pointed out that the classical Fl score fails to reflect approximate but non-overlapping detections, which common in time series. SSarfraz et al. (2024) criticized the persistent use of flawed evaluation metries and inconsistent benchmarking practices, suggesting that complex deep learning models may not offer significant improvements over simpler baselines in the semi-supervised setting.
2

---

Published as a conference paper at ICLR 2025
One reason for using deep learning models in time series anomaly detection is their ability bring prior knowledge on what constitutes normal behavior from pretraining on large-scale datasets. Large language models (LLMs) may offer promising solution due to their strong zero-shot capabilities. However, there is currently a lack of systematic study of modern (M-)LLMS for time series anomaly detection, which aim to address in this work.
Multimodal LLMs (M-LLMs). Multimodal LLMs combine text with other data modalities and have been explored in various domains, including image captioning, video understanding, and mul- timodal translation (Lu et al. 2019 Li et al. 2019 Sun et al. 2019 Huang et al. 2019). Recent advancements have led to more sophisticated M-LLMs, such Qwen- VL and Phi-3 Vision, demon- strating superior performance in visual-centric tasks and compact deployment capabilities (Bai et al,, 2023 Abdin al., 2024). In the context of time series analysis, M-LLMs have been used to model multimodal data, such as time series and textual information, showing promising results in fore- casting and anomaly detection (Liu et al. (021). However, there is notable absence of research applying M-LLMs to time series data presented as visual inputs, even though humans often detect time series anomalies through visual inspection. This gap is particularly significant given that time series data can be represented in different modalities (e.g., numerical, textual, or visual) without los- ing substantial new information. Consequently, time series analysis presents a unique opportunity to evaluate M-LLMs' ability to understand and process the same underlying data across different representational formats.
TIME SERIES ANOMALY DETECTION: DEFINITION AND CATEGORIZATION
We begin by defining time series anomaly detection and categorizing different types of anomalies.
3.1 ANOMALY DEFINITION
We consider time series X : {11 (2,,, I22...,,TTi collected at regular intervals, where v, is the feature scalar or vector at time I, and 7 is the total number of time points. Anomalies are data points that deviate significantly from the expected pattern of the time series. The expected pattern of a time series refers to the governing function or conditional probability that the data is expected to follow, depending on whether the system deterministic or stochastic.
Generating, function. Assume the time series generation is deterministic. A data point is consid- ered an anomaly it deviates much from the value predicted by the generating function, i.e.,
[ - G(ag Tt ft-n)l > 8
(1)
Conditional probability. Assume the time series generation is governed by history-dependent stochastic process. A data point ze is considered an anomaly its conditional probability is below certain threshold c, ie..
Tt-1 =z-n) <e
(2)
An anomaly detection algorithm typically takes time series input and outputs either binary }-
labels Y == Vot.:,......77 or anomaly scores {s1, 82. 87....,77). In the case of binary labels, indicates normal behavior. The number of
Yt = indicates an anomaly at time t, and yo anomalies should be much smaller than the number of normal data points, i.e., Σ T.
In the case of anomaly scores, represents the degree of anomaly at time I, with higher scores indicating higher likelihood of an anomaly. This likelihood can connected to the condi- tional probability definition, where a higher score is correlated lower conditional probability P(xe[z:-1, Tt- It-n). A threshold can be applied to the scores to convert them into binary labels, where y1 1 if s, 0 and y = otherwise.
Time series anomalies can be analyzed at two levels: (1) within individual sequences, where specific points intervals deviate from the normal pattern, and (2) across different sequences, where entire sequences are considered anomalous. This paper focuses on the first level, specifically detecting anomalous intervals within individual sequences.
3

---

Published as a conference paper at ICLR 2025
Anomalous Intervals. We define anomalies continuous intervals of time points that de- viate from the expected pattern. Let R be a set of anomalous time intervals: R = (isttrrt fenaly sttrrt: ieres&edd fenal..... 1 & fenal where (Estarty fenal represents the i-th anomalous in- terval, with & start and & Fend being its start and end times. When E start teedd the anomaly is single point anomaly. For a time series X
{I1, ;2, Fr we assign binary labels:
1 y/ - {
if E ('stert: t tendl for any i E (1....,k)
otherwise
Zero-Shot and Few-Shot Anomaly Detection. Few-shot anomaly detection involves providing the model f with a small set of labeled examples. Given n labeled time series {(X1, Yi), (X2, Y2), ..., (Xn, Y,)), where Y; is series of anomaly labels for each time step in X1, and new unlabeled time series Xeew the model predicts:
{81, ,,, ST} or {11, 1/2................... YT} = gXXnew ((X1, Yi), (X2, ..)...., (Xn, xn))),
where typically small (e.g.. 1-5). In zero-shot anomaly detection, n = 0, i... the model is expected to identify anomalies without any labeled examples. These scenarios pose significant challenges for deep neural nets and some traditional models that require lot of training examples.
Anomaly
(a) Trend
3.2 ANOMALY PATTERN CLASSIFICATION
Patterns of time series anomalies be categorized into two main types based on their nature: out-of-range anomalies which exceed normal value thresholds and contextual anomalies which exhibit abnormal behavior only within specific contexts (Lai et 2023). The con- textual anomalies can be further divided into frequency anomalies, trend anomalies, and contextual point anoma- lies. Each type presents unique characteristics and chal- lenges for detection. By examining how LLMs recog- nize these diverse anomaly types, we can verify whether our hypotheses about LLMs' understanding of time series data hold consistently across different pattern variations.
(b) Frequency
(c) Point
(d) Out-of-Range
Figure 1: Example time series with dif- ferent anomaly types, with anomalous regions highlighted in red.
3.2.1 OUT-OF-RANGE ANOMALIES
Out-of-range anomalies are data points that lie far outside the normal range of values in time series. These anomalies can be detected even when the time series order shuffled, as shown in Figure d)). If a model can detect out-of-range anomalies but fails to detect contextual anomalies, this suggests is not using the positional information in the time series (Tan al. 2024).
3.2.2 CONTEXTUAL ANOMALIES
Contextual anomalies are data points or consecutive subsequences that deviate from the expected pattern of the time series. These anomalies are only detectable when the order of the time series is preserved. Contextual anomalies can be further divided into three subcategories:
Trend Anomalies. Trend anomalies manifest as sudden acceleration (Figure a)), a)), deceleration, or reversal of the established trend. These anomalies are characterized by unexpected changes the changing rate of the time series, detected through gradient analysis gr (x+ -r-1)// Al. To reduce noise sensitivity, smoothed gradients g gmoothh = (MA, MA,-)) 1)/-11//x/ At are often used, where MA,
is moving average over a window of points.
Frequency Anomalies. Frequency anomalies occur when the periodic components a time series deviate from the expected pattern. These anomalies are usually identified by analyzing the frequency domain of the time series, typically using techniques like Fourier transforms. A frequency anomaly is detected when there's 's significant shift in the dominant frequencies, Figure (b).
Contextual Point Anomalies. Contextual point anomalies occur when individual data points deviate from the expected pattern of the time series, even while remaining within the overall regular
4

---

Published as a conference paper at ICLR 2025
range of values. As shown in Figure (c) these points not extreme outliers but don't fit the local context of the time series. They may violate the smooth continuity of the data, contradict short-term trends, disrupt established patterns without necessarily exceeding global thresholds.
3.3 TIME SERIES FORECASTING VS. ANOMALY DETECTION
As most of the literature LLM for time series focuses forecasting, these works as a starting point to understand LLMs' behavior in anomaly detection. The tasks of time series forecasting and anomaly detection share many similarities. By definition, deterministic fore- casting of future time steps is about learning the generating function (see Equation I1. Probabilistic forecasting about learning the conditional probability function (see Equation 2). Therefore, both time series forecasting and anomaly detection rely heavily on extrapolation. In forecasting tasks, models extrapolate past patterns to predict future values, extending the known series into unknown territory. Similarly, anomaly detection involves extrapolating the "normal" behavior of a time series to identify points that deviate significantly from this expected pattern.
the conjectures from
LLMs are trained on corpus of token sequences, {U1, U2, ..... UN J. where U; {ui, U2..... U1 and ūj is token in the vocabulary V. The model learns to autoregressively predict the next token in the sequence given the previous tokens, i.e., P(uj+1l [/1 u2 .2...... u,)) The motivation for applying LLMs to time series forecasting often their zero-shot extrapolation capabilities (Brown et al,, 2020). The autoregressive generation of tokens and that of time series steps (in Equation 2 similar, and the LLMs act as an Occam's 's razor to find the simplest form of G (in Equation [ (Gruver al., 2023). This connection suggests that hypotheses made in LLMs for forecasting may also apply to anomaly detection. Many such hypotheses are proposed as possible explanations for the model's behavior without validation from controlled studies, which motivates our investigation.
4 UNDERSTANDING LLM's UNDERSTANDING OF TIME SERIES
To demystify LLMs anomaly detection capabilities, we take a principled approach by formulating several scientific hypotheses. Then build an LLM time series anomaly evaluation framework to test each of the hypotheses.
4.1 HYPOTHESES
The following hypotheses represent synthesis of existing literature (1-3), our own insights into LLM behavior (4-5), and prevailing assumptions in the field that warrant closer examination (6-7). The hypotheses cover two main aspects: LLMs' reasoning paths (1, 3, 4) and biases (2, 5, 6, 7).
Hypothesis 1 (Tan et al., 2024) on Chain-of- Thought (CoT) Reasoning
LLMs do not benefit from engaging in step-by-ster reasoning about time series data.
The authors claim that existing LLM methods, including zero-shot ones, do very little innate reasoning. While they demonstrate that LLM methods perform similarly or worse than those without LLMs time series tasks, none of their experiments assess the LLMs' reasoning capabilities.
To validate this hypothesis, we focus on the performance of onc-shot and zero-shot CoT prompting, which explicitly elicits an LLM's reasoning abilities (Wei et al. (002).. We use terms from cognitive science to describe the LLMs' different behaviors with and without CoT: the reflexive mode (slow, deliberate, and logical) and the reflective mode (fast, intuitive, and emotional) (Lieberman) 2003). Therefore, the question becomes whether the LLMs benefit from the reflexive mode, when it thinks slowly about the time series. If the hypothesis false, the LLMs should perform better when they are prompted to explain.
Hypothesis 2 (Gruver et al., 2023) on Repetition Bias
LLMs' repetition bias (Holtzman et al. 2020) corresponds precisely to their ability to identify and extrapolate periodic structure in the time series.
5

---

Published as a conference paper at ICLR 2025
This hypothesis draws parallel between LLMs' tendency to generate repetitive tokens and their po- tential ability to recognize periodic patterns in time series data. To validate the hypothesis, we design an experiment where the datasets contain both perfectly periodic and noisy periodic time series. The introduction of minor noise would disrupt the exact repetition of tokens in the input sequence, even if the underlying pattern remains numerically approximately periodic. If the hypothesis holds, should observe significant drop in performance, despite the numbers maintaining its fundamental periodic structure. See Appendix E for formal definitions of Hypothesis and 3.
Hypothesis 3 (Gruver et al., 2023) on Arithmetic Reasoning
LLMs' ability perform addition and multiplication Yuan et al. 2023) maps onto extrapo- lating linear and exponential trends.
This hypothesis suggests a connection between LLMs' arithmetic capabilities and their ability to extrapolate simple mathematical sequences. It is argued that LLMs' proficiency in basic arithmetic operations, such as addition and multiplication, enables it to extend patterns like linear sequences (e.g., (t)) 21) by iteratively applying the addition operation (e.g.. +2). To validate the hypothesis, design experiment where an LLM is specifically guided to impair its arithmetic abilities while preserving its other linguistic and reasoning capabilities. the hypothesis holds, we should observe corresponding decline in the model's ability to predict anomalies in the trend datasets. Otherwise, LLMs rely on alternative mechanisms for time series pattern recogmition and extrapolation.
Hypothesis 4 (Dong et al., 2024) on Visual Reasoning
Time series anomalies can more easily detected as visual input rather than text input.
Motivated by the fact that human analysts often rely on visual representations for anomaly detection in time series, hypothesize that M-LLMs, whose training data includes human expert detection tasks, may prefer time series as images rather than raw numerical data. Similar assumption is also proposed in recent work by Dong et al. (024),, who demonstrated that explicitly prompting LLMs to "think visually" about time series improved their anomaly detection capabilities, even without actual visual input. This hypothesis can be readily tested by comparing the performance of multimodal LLMs on identical time series presented as both text and visualizations.
Hypothesis 5 Visual Perception Bias
LLMs exhibit similar detection limitations to human perceptual biases, e.g., in acceleration perception when analyzing visual time series representations.
We hypothesize based on the growing interest in using LLMs due their internal human-like knowl- edge (Jin et al. (024) and their ability to align with human cognition in complex tasks Thomas et al., 2023). However, humans have known cognitive limitations in detecting subtle anomalies, and recent research suggests that LLMs may exhibit human-like cognitive biases (Opedal et al., 2024).
To validate this hypothesis, we leverage findings from human perception research. For example, Mueller & Timney (2016) reported that humans are more sensitive to sudden changes in motion direction or velocity (like trend reversals) than to gradual acceleration changes. We compare LLM performance on two datasets: one featuring anomalies that revert increasing trend to a decreasing trend (analogous to negating constant speed), and another with anomalies that accelerate an in- creasing trend. Both datasets would have similar prevalence rates, making them equally detectable by traditional methods that identify gradient changes. LLMs exhibit significantly poorer perfor- mance in detecting acceleration anomalies compared trend reversals, it would suggest that they indeed suffer from similar perceptual limitations as humans.
Hypothesis 6 on Long Context Bias
LLMMs perform better for time series with fewer tokens, even if there is information loss.
Despite recent advancements handling long sequences, LLMs still struggle with complex, real- world tasks involving extended inputs (Li al. 2024). This limitation may also apply to time

---

Published as a conference paper at ICLR 2025
series analysis. To test this hypothesis, we propose experiment comparing LLM performance original time series text and pooled textual representations (reduced size by interpolation). the hypothesis holds, should observe performance improvement with the subsampled text.
4.2 PROMPTING STRATEGIES
We incorporate two main prompting techniques in our investigation: Zero-Shot and Few-Shol Learn- ing (FSL) and Chain-of-Thought (CoT). For FSL, we examine the L.M''s ability to detect anomalies without any examples (zero-shot) and with small number of labeled examples (few-shot). For CoT, we implement example in-context CoT templates, guiding the LLM through a step-by-step reasoning process. Our template prompts the LLM to: (1) Recognize and describe the general time series pattern (e.g., periodic waves, increasing trend) (2) Identify deviations from this pattern (3) Determine if these deviations constitute anomalies based on the dataset's normal behaviors.
4.2.1 INPUT REPRESENTATION
Visual representations of activities enhance human analysts' ability detect anomalies (Riveiro) & Falkman 2009), and the pretraining of M-I.LM involves detection tasks (Bai et al. 2023). In- spired by these facts, we infer that LLMs' anomaly detection may benefit from visual inputs. There- fore, we explore two primary input modalities for time series data: textual and visual representations.
Textual Representations. We examine several text encoding strategies to enhance the LLM's comprehension of time series data:(1) Original: Raw time series values presented as rounded space- separated numbers. (2) CSV: Time series data formatted as CSV (index and value per line, comma- separated), inspired by Jin al. (2024). (3) Prompt Prefix (PAP): Including key statistics of the time series (mean. median, trend) along with the raw data, as suggested by Jin et al. (002)). (4) Token per Digit (TPD): Splitting floating-point numbers into space-separated digits (e.g.. O. 246 2 4 6) to circumvent the OpenAl tokenizer's default behavior of treating multiple digits as a single token, following Gruver et (003)) This strategy only improve the performance of models that apply byte-pair encoding (BPE) tokenization, see Appendix Observvation 8.
Visual Representations. We utilize Matplotlib to generate visual representations of the time series data. These visualizations then provided to multimodal LLMs capable of processing image inputs. Since LLMs have demonstrated strong performance on chart understanding tasks (Shi et al.) 2024), they are expected to identify anomaly regions' boundaries from the visualized time axis.=
4.2.2 OUTPUT FORMAT
To ensure consistent and easily interpretable results, prompt the LLM to provide structured output in the form of a JSON list containing anomaly ranges, e.g.,
[{"tarr":: 10, "end": 25}, {"start": 310, "end": 320},
This format allows for straightforward comparison with ground truth anomaly labels and facilitates quantitative evaluation of the LLM's performance. By employing this comprehensive set of evalua- tions, we draw more robust conclusions about the following hypotheses.
EXPERIMENT
EXPERIMENT SETUP
Models. We perform experiments using four state-of-the-art M-LLMs. of which open-
(Chen et al. 2024), and two of which proprietary: GPT-4o-mini (OpenAT 2024) and Gemini-1.5-Flash (Google) 2024).
sourced: Qwen-VL--Chat (Bai et al. 2023) and
The language part of the models covers four LLM architectures: Qwen LLaMA
( Gemini
and GPT S Since we send the text queries to M-LLMs instead of their text component, we validated via MML.U-Pro (Wang et al. 2024) that adding vision modality does not reduce the

---

Published as a conference paper at ICLR 2025
Threahold Gemini-vislon-ee vislon-Oshot
False Poeilives
wea-ii--llsooott owen-vislon Oshot
by looForest
Owen- vialon-fahed Isolation Forest
MMM
M
Figure 2: Example anomaly detection results for out-of-range anomalies. Direct thresholding with expert knowledge yields the best result, but the LLMs also detect the approximate ranges without priors. Isolation Forest raises lots of false positives but still has higher than LLMs, which motivates use of affinity Fl.
model's performance on text tasks. The validation details can be found in Appendix A.1 We have 21 prompting variants for each model, with 13 for text and for vision. In controlled experiments, for each model, we report the specific variants or the top 3 variants with the highest scores under the condition. We label the variants with the name codes Table T with details Appendix [A.4
Variant
Code
Datasets. We synthesize four main datasets corre-
Oshot-text
A
sponding to different anomaly types in Section 3.2 point, range, frequency, and trend. We add noisy versions of point, frequency, and trend to test Hypothesis 2. We add acceleration-only version of the trend dataset test Hypothesis 5, Further details on the datasets can be found in Appendix
Oshot-text-s0.3
B
Oshot-text-s0.3-ca
C
Dshot-text-s0.3-co0
Dehot-text-s0.33-oor
cBv
E
F
0shot-text-s0.3-corr -po
G
0shot-teett-
We choose synthetic data instead of real-world data due (1) known label quality issues in public benchmarks (Wu & Keogh 2021).
0shot-teext-s0.3--6e 3-dyscall
0
J
(2) need for precise anomaly type isolation. (3) requirement for textual anomaly descrip- tions to enable CoT prompting. Still, our experiments on real-world Yahoo S5 dataset Lapptev Amizadeh (2015)
Oshot-text-s0.33-00
K
Oshot-vision
Oshot-vvsiion-cale
cshot-vission-oot
N
Oshot-viisionndyssaal.
0
(see Appendix D) show consistent findings.
1shot-text-s0.3
P
1shot-ttext-s0.3-cot
The LLMs generate anomalous intervals that be converted to binary labels and do not output anomaly scores. Therefore, we report precision, recall, and Fi-score metries. However, these scores treat time series as a cluster of points without temporal order and can give counterintuitive results, as illustrated in Figure U
Metrics.
ishot-vision
Ishot-vision-call
S
ishot-visionn-ott
T
ishot-vissoonddyscall
Table 1: Variants and their correspond- ing namecodes, Appendix A.4 for details
To address this, we also report affinity precision and affinity recall as defined in Huet et al.] (2022). We cal- culate the affinity FI score as the harmonic mean of affi- precision and affi-recall, and it is the main metric we use to evaluate the hypotheses. We rely on variants of Fl because the LLM yields discrete intervals, not an anomaly score. Metrics like Volume Under the Surface (Paparrizos et al. 2022) require a ranking or continuous score.
anomaly de- tection method but rather to test hypotheses for better under- standing, simple baselines for sanity check, see Ap- pendix C Observation 9 for direct performance comparison with baselines. We Isolation Forest (Liu al. 2008) and Thresholding.
Baselines. As our goal is not to propose
range point
MTOLAP
THERLJ NTORLP NTQLRH
trend
5.2 EXPERIMENT RESULTS
In this section, we discuss the hypotheses that align with our observations and those we can confidently reject. Detailed numbers can be found in Appendix D The 0-to-1 y-axis of the figures represents the affinity F1 score, where higher values in- dicate better performance. We focus on the 6 key hypotheses and defer other findings to Appendix
NTQRLS
THFRLD NTDLRK NTORJL
freq
OTEBPH TAFRLJ ODTBPR THORLD
Figure 3:
Reflexive (prompt that induces reasoning) Reflective (prompt asks for direct answer), Top Affi-F1 prompt variant per mode, See Table variant name codes.
8

---

Published as a conference paper at ICLR 2025
point
trend range point
point
MSCUDI MSCUO1 SMCUDI NSCOUI
UHOQPS RUL3FD USROAB NOLHBP
range
range
SMCUOI MSCUOT MSCOUT NSCUD1
SURPBO MRLJFD MMTPOR LMSHA
trend
freq trend
SMCUDT ASCOUI NSCUOI
freq
freq
C5NIUD MSCUOI CSNIUO SMCUOT
TRUBPQ RLHJKF RUSQBP RTSEMP
Figure 5: Calc (prompt with
Figure 6: Vision (prompt with Figure 7: Subsampled (time visualized time series) Text ries subsampled to be shorter) (raw numerical prompt), Top / Original, O-shot raw vs Affi-FI variants per modality
correct arithmetic example) /
DysCalc (incorrect example),
Top 3 Affi-Fl variants mode
30% text
Retained Hypothesis 1 on CoT Reasoning
No evidence is found that explicit reasoning prompts via CoT improve LLMs' performance in time series analysis.
Interestingly, when we explicitly use CoT to simulate human-like reasoning about time series, the anomaly detection performance steadily drops across all models and anomaly types, as shown in Figure 3 These findings suggest that LLMs' performance in time series anomaly detection may not rely on the kind of step-by-sted logical reasoning that CoT prompting aims to elicit. However, this does not necessarily mean LLMs use no reasoning at all; rather, their approach to understanding time series data may differ from our expectations of explicit, human-like reasoning processes.
Rejected Hypothesis on Repetition Bias
LLMs' repetition bias does not explain their ability to identify periodic structures.
true, would expect that injecting noise would cause a much larger drop in text performance (since the tokens formance. However, the performance drop is similar across both modalities, shown in Figure g and the text perfor- mance drop is often not significant. This suggests that the LLMss ability recognize textual frequency anomalies has other roots than their token repetitive bias.
If the hypothesis
longer repeating) than in vision per-
Vision
Text
SPOBP JKFJDF QBPQPE ВHРеРО
Rejected Hypothesis 3 on Arithmetic Reasoning
Figure 4: Clean (original time ries) Noisy (time series with min- imal injected noise), Top 3 Affi-FI variants per noise level
The LLMss understanding of time series is not related its ability perform arithmetic calculations.
We designed in-context learning scenario where the LLMss accuracy for five-digit integer addi- tion drops to 12%. The details can be found in Appendix A.4 and A.5) Despite this, the LLMs' anomaly detection performance remains mostly consistent, shown in Figure This suggests that the LLMs' anomaly detection capabilities are not directly tied to their arithmetic abilities.
Retained Hypothesis 4 on Visual Reasoning
Time series anomalies are better detected by M-LLMs as images than by LLMs as text.

---

Published as a conference paper at ICLR 2025
Across a variety of models and anomaly types, M-LLMs much capable of finding anoma-
lies from visualized time series than textual time series, see Figure 6 The only exception is when detecting frequency anomalies with proprietary models. This aligns with human preference for vi- sual inspection of time series data.
Rejected Hypothesis 5 on Visual Perception Bias
The LLM's understanding of anomalies is not consistent with human perception.
We create the "flat trend" dataset where the anomalous trend is too subtle to be visually detected by humans but becomes apparent when computing the moving average of the gradient, as shown in Figure g The LLMs' performance is very similar to the regular trend dataset, regardless of the modality. This suggests that the LLMs do not suffer from the same limitations as humans when detecting anomalies.
"Flat Trend" Time Series
Anomaly
Avg Gradient
Retained Hypothesis 6 on Long Context Bias
LLMs perform worse when the input time series have more tokens.
Vision
We observe consistent improvement in performance when interpolating the time series from 1000 steps to 300 steps, as shown in Figure ? Notably, the top-3 best-performing text variants in all experiments typically apply such shortening. This underscores the LLM's difficulty in handling long time series, especially since the tokenizer represents each digit as separate token.
RLHRLN TRUTRL LANENT THRNRT
Text
QPBQBP ABD3FD QPBKD] QP3
Figure 8: Flat Trend (see above for an example) / Trend (trend may reverse during anomalies), Top Aff-F1 variants per dataset
Model Variations. Individual models exhibit distinctly dif- ferent behaviors zero-shot and few-shot anomaly detection. For instance, GPT's performance with longer sequences do not degrade as much as other models, and models like Qwen handle visual inputs far more successfully. Detailed comparisons and analyses of these model-specific effects are provided in Appendix C These observations underscore that L.Mss' performance in time series tasks can depend heavily on factors like training data, parameter count, and fine-tuning strategies.
CONCLUSION
In this paper, conducted comprehensive investigation into Large Language Models' (LLMs) understanding of time series data and their anomaly detection capabilities. Our findings challenge several assumptions prevalent in current literature, highlighting the need for rigorous empirical val- idation of hypotheses about LLM behavior. Our key findings include LLMs' visual advantage, limited reasoning, non-human-liks processing, and model-specific capabilities. These insights have important implications for the design of future L.LMs and the development of anomaly detection systems. For instance, our results suggest that L.LMs do not effectively detect visual frequency anomalies,
LLM-based anomaly detection systems shall leverage Fourier analysis before feeding the data to the LLM to improve performance. Our model-specific findings suggest that model selection and possible ensemble methods are crucial for designing LLM-based anomaly de- tection systems.
Our work underscores the importance of controlled studies in validating hypotheses about LLM be- havior, cautioning against relying solely on intuition or speculation. Future research should continue empirically test assumptions about L.LMs' capabilities and limitations in processing complex data types like time series.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported in part by the U.S. Army Research Office under Army-ECASE award W91 1991NF-07-R-000S- the U.S. Department Of Energy, Office of Science IARPA HAYSTAC Program, and NSF Grants #2205093, #2 #2134274, CDC-RFA-FT-23-0069 DARPA AIE FoundSci and DARPA YFA. We thank Dr. Eamonn Keogh for the helpful suggestions.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Ben- haim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Yen-Chun Chen. Yi- Ling Chen, Parul Chopra, Xiyang Dai, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Victor Fragoso, Dan Iter, Mei Gao, Min Gao, Jianfeng Gao, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Ce Liu, Mengehen Liu, Weishung Liu, Eric Lin, Zeqi Lin, Chong Luo, Piyush Madan, Matt Mazzola, Arindam Mitra, Hardik Modi, Anh Nguyen. Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sam- budha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shi- tal Shah, Ning Shang, Hiteshi Sharma, Swadheen Shukla, Xia Song. Masahiro Tanaka, Andrea Tupini, Xin Wang, Lijuan Wang, Chunyu Wang, Yu Wang. Rachel Ward, Guanhua Wang. Philipp Witte, Haiping Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang. Jianwei Yang. Ziyi Yang, Yifan Yang, Donghan Yu, Lu Yuan, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv org/ htde:/bs/2244 2404. 14219
preprint, abs/2404. 14229, 2024.
Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A. Zuluaga. Do deep neural networks contribute to multivariate time series anomaly detection? ArXiv preprint, abs/2204.016377. 2022. URL https: www.aaiwww..iiaaaaaaaaaarraaaaae 200bb//22200000 01637
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile lasion-l model for understanding, lo- calization, text reading, and beyond. ArXiv preprint, 0ls/2308.12966 2023. URL https: //arxiv. org/ http//bbb/...////// 12966
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpps:/pproceedingg www.eediiiinnnnnnnnneeeeeeeeeeee nwwrips...... wwtrriis.ccppaeee httpp/ppppr///////// 145770066b
html
Wenchao Chen, Long Tian, Bo Chen, Liang Dai, Zhibin Duan, and Mingyuan Zhou. Deep varia- tional graph convolutional recurrent network for multivariate time series anomaly detection. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 3621-3633. PMLR, 2022. URL, UtR.itttps///procedinng mlr. wtt..r.pressssee/e/eee chen22x html
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo,
11

---

Published as a conference paper at ICLR 2025
Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai. Licheng Wen, Xiangchao Yan. Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are to gpt- 4v? closing the gap to commercial multimodal models with open-source suites. ArXiv preprint, abs/2404. 16821, 2024. URL httphhtttt//ttttte org/ aba/abas///// 16821
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168,
2021.
LMDeploy Contributors. Lmdeploy: toolkit for compressing, deploying, and serving 1lm, 2023.
URL https: http:/tto..cammmmmmmm//////p//p/
Manqing Dong, Hao Huang, and Longbing Cao. Can Ilms serve time series anomaly detectors?
ArXiv preprint, abs/2408. 03375, 2024. URL.istpss//ardd..rrr////////// 03475
Google. Gemini flash, 2024. flash/
URL
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero- shot time series forecasters. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. http httpttpppppppppppppppp mips. pcppppr iiles/paper html
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httlhtss///pppreeiieee net/
forum?id=ryggayrFvtt
Lun Huang, Jiajun Zhou, Chengqing Zhang, and Hainan Huang. Multimodal neural machine trans- lation with language scene graph. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1065-1075, 2019.
Alexis Huet, José Manuel Navarro, and Dario Rossi. Local evaluation of time series anomaly detection algorithms. In Aidong Zhang and Huzefa Rangwala (eds.), KDD '22: The 28th
ACM SIGKDD Conference August 14 18, 2022, pp. 635 ACM, 2022. https://doi: org/ 10. 1145/ 3534678 35444//38335478883333399999 3539339
Knowledge Discovery and Data Mining, Washington, DC, USA,
doi: 10.11457334468.3333333 URL
Ming Jin, Shiyu Wang. Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-llm: Time series forecasting by reprogramming large language models. ArXiv preprint, abs/2310.011228, 2023. URL https:
Marxiv.orrg /btp//11.........a.oeaaeabbata.a 01728
Ming Jin, Yifan Zhang, Wei Chen, Kexin Zhang, Yuxuan Liang, Bin Yang, Jindong Wang, Shirui Pan, and Qingsong Wen. Position: What can large language models tell us about time series analysis. ArXiv preprint, abs/2402.00713. 2024. UtL/attps///arxi org/ /ttp///02..o. 02713
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large lan- guage models are zero-shot reasoners. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Bel- grave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- - December 9, 2022, 2022. URL |ptprr/0022hhss1/00000accccccco. oagpa//eee00..t10o0o---------rrr
nips..iip.................. http:cer_ppppppieeeeeeeeee files/ www.---bttttttttaaeeeeeeee
html
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
12

---

Published as a conference paper at ICLR 2025
Chih- Yu Lai. Pan-Keng Sun, Zhengqi Gao, Jeffrey H. Lang, and Duane S. Boning. Nominal- ity score conditioned time series anomaly detection by poin/sequential reconstruction. In Al- ice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL http: http::://apppaaeeeeee nips ccpps//apppppppppppppepppp httpc/pppe_ffeee///eeeeeeeeeeeee files/ fwwes/paperrree pwppp//02p///////p/pa//ppap (cctfe/e09S77tto0000eeceeeeeeeea
N. Laptev and S. Amizadch. Yahoo anomaly detection dataset S5. yahoo. wtt catalog. php?dattatype=s/=70 2015.
sandbox.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder: A universal encoder for text, image and video. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 100,190.0199
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long- Ilms struggle with long in-context learning. ArXiv preprint, abs/2404.02060, 2024. URL https://arxiv. org/
abs/2404. 02060)
Matthew D. Lieberman. Reflexive and reflective judgment processes: A social cognitive neuro- science approach, pp. 44-67, Cambridge University Press, New York, NY, US, 2003, ISBN
978-0-521-8224844
Chen Liu, Shibo He, Qihang Zhou, Shizhong Li, and Wenchao Meng. Large language model guided knowledge distillation for time series anomaly detection. ArXiv preprint, abs/2401.15122 2024a.
URL https: //axiiv org httpg///bs/240.... 15123
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. 2008 eighth ieee international conference on data mining, pp. 413-422. IEEE, 2008
Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Kamarthi, Aditya B. Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, and B. Aditya Prakash. Time-mmd: A new multi-domain multimodal dataset for time series analysis. ArXiv preprint, abs/2406.08627, 2024b.
/attps://arxits
horg//bs/24066 22tp//66000000777
Haoyi Liu, Shanghang Bai, Yuyang Shen, Xu Han, Wei Zhang, and Jiliang Gao. Multimodal trans- former for unaligned multimodal time series analysis. In Proceedings of the Web Conference 2021, pp. 2990-3001, 2021.
Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, and Shu-Tar Xia. Taming pre-trained llms for generalised time series forecasting via cross- knowledge distillation. ArXiv preprint, abs/2403.07300,, 2024c. URL. htt..ttss::/ httpps///axxi.. org abs/2403. 07300.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and- lan-uuaae tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' A'Ahé-eBBu Emily B. Fox, and Roman Garnett (eds.), Ad- Neural In- formation Processing Systems 2019, NewrIPS 2019, December 8-14, 20/9, Vancouver, BC, Canada, pp. 13-23, 2019. URL https::/procceedings wwwpediiinnnnnuuis....m cc/ htcppappe/2019/aaaee Abbtractt......... hhtml
vances in Neural Information Processing Systems 32: Annual Conference
Alexandra S. Mueller and Brian Timney. Visual acceleration perception for simple and complex motion patterns. PLOS ONE, 11(2):e0149413, 2016. ISSN 1932-6203. doi: 10.1371/journll pone.0149413.
Andreas Opedal, Alessandro Stolfo, Haruki Shirakami, Ying Jiao, Ryan Cotterell, Bernhard Schölkopf, Abulhair Saparov, and Mrinmaya Sachan. Do language models exhibit the same cognitive biases in problem solving human Icarners? ArXiv preprint, abs/2401.18070. 2024. URL http//ttt////iiii wwwpeivvvv.rgrraae wtgg/bs////....ooo. 18070
OpenAl. Gpt-40 mini: Advancing cost-efficient intelligence, 2024, URL bttp com/ mndex/gpt-4o-io http::/wwww..aaacaaiiaoooorraaaa
13

---

Published as a conference paper at ICLR 2025
John Paparrizos, Paul Boniol, Themis Palpanas, Ruey S Tsay, Aaron Elmore, and Michael J Franklin. Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection. Proceedings of the VLDB Endowment, 15(11):2774-2787, 2022.
Maria Riveiro and Göran Falkman. Interactive visualization of normal behavioral models and ex- pert rules for maritime anomaly detection. In Imaging and Visualization 2009 Sixth Interna- tional Conference on Computer Graphics, pp. 459-466, 2009. doi: 10. 10.1109//CIV URL https://iesexplore. ieee. http:///ww......oooo/oo//o/////a
M. Saquib Sarfraz, Mei- Yen Chen, Lukas Layer, Kunyu Peng, and Marios Koulakis. Position: Quo vadis, unsupervised time series anomaly detection? ArXiv preprint, abs/2405.02667. 2024. URL
https://arxiv.orgg awtaaass/2440 02678
Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang. Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, Gongye Liu, Xiaomei Nie, Deng Cai, and Yujiu Yang. Chartmimic: Evaluating Imm's s cross- reasoning capability via chart-to-code generation. ArXiv preprint, abs/2406.09961, 2024. URL https://arxii org/ abs /40es//244..../ 09961
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In 2019 IEEE/CVF International Confer- ence on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. http::: org/
7463-7472. IEEE, 2019. doi: 10.1 10.110///CC22211.000007 ICCV SBCV.2019.007555
/tttps///abl.org/1101099999 1109/
Mingtian Tan, Mike A. Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language models actually useful for time series forecasting? ArXiv preprint, abs/2406. 2024. URI.
https://arxiv. org/ abs//b/s00 16964
Hua Tang, Chong Zhang, Mingyu Jin, Qinkai Yu, Zhemting Wang, Xiaobo Jin, Yongfeng Zhang, and Mengnan Du Time series forecasting with Ilms: Understanding and enhancing model capa- bilities. ArXiv preprint, abs/2402.10835, 2024. https::: http:///axvvvoroo org/ abspaab///22222// 10835
Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. Large language models can accurately predict searcher preferences. ArXiv preprint, abs/2309.10621, 2023, URL https:
10622
Shreshth Tuli, Giuliano Casale, and Nicholas R. Jennings. Tranad: Deep transformer networks for anomaly detection in multivariate time series data. ArXiv preprint, abs/2201 0772244 2022. URL
https://arxiv.ot orgg httpg/abb/2211.02.222
Yubo Wang. Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. ArXiv preprint, abs/2406.015744 2024. URL https: /ttps::/arxxv. orgg abs/ 01574
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022. URL http:://aappereee nips wttpee/fff4a4affffff///.a./..aa. Conf
waapoerrfillseeeeeeeeeeeeeeeeese hallee/paper/222// 2002//ah//
html
Christopher Wimmer and Navid Rekabsaz. Leveraging vision-language models for granular market change prediction. ArXiv preprint, abs/2301.10166, 2023. URL https: org/ /bs//
2301. 10166
Renjie Wu and Eamonn Keogh. Current time series anomaly detcction benchmarks flawed and are creating the illusion of progress. IEEE Transactions on Knowledge and Data Engineering, pp. 1-1, 2021. ISSN 1041-4347, 1558-2191, 2326-3865. doi: 10.1109/KKDD.222133112222
14

---

Published as a conference paper at ICLR 2025
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large language models perform in arithmetic tasks? ArXiv preprint, abs/2304.02015, 2023. URL https://arxiv. org/ abs/ 02015
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Appli- cations of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp- 11121- 11128. AAAI Press, 2023. doi: 10. 10.1699/AAALV331996663 URL https://doo /
1609/aaai. v3719. 26317
Xiyuan Zhang, Ranak Roy Chowdhury, Rajesh K. Gupta, and Jingbo Shang. Large language models for time series: A survey. ArXiv preprint, abs/2402.01801, 2024. URL httph:ttp////t.t. ltttps:://axivvor! abs/ hts/2240.01880
15

---

Published as a conference paper at ICLR 2025
A MODEL DETAILS
A.1 (M)LLM ARCHITECTURES AND VALIDATION
In this section, we introduce the details of the four M-LLMs investigated in the study. We focus on resolving the performance discrepancy between those models and their text-only counterparts. The purpose is to ensure reproducibility and justify direct comparisons between visual and text inputs.
GPT-4o mini Launched in July 2024, GPT-4o mini (OpenAl 2024) is a cost-efficient, smaller lower cost. It
version of GPT-4o, designed to replace GPT-3.5, exceeding its performance excels in mathematical and coding tasks, achieving 87.0% on MGSM (measuring math reasoning) and 87.2% on HumanEval (measuring coding performance). The model features a 128.000-token context window, knowledge up to October 2023, and support for text and vision in the API.
The architecture of GPT-4o is not disclosed. The GPTT-4o-iiii variant used in this work is Since we are sending text queries with and without images to GPT-4o, an important thing to consider that by adding the image, the language part of the model does not degrade, or the OpenAl reverse proxy does not send the query to different backend. To the best of our knowledge, there no prior validation study on this. We perform experiments by adding a small, white, 10 x 10 pixels image to the text queries and run the 5-shot CoT MMLU-Pro (Wang) jet al., 2024). The score without image 61.54. The score with the image is 61.49. The scores do not reject the hypothesis that the same model is behind different modalities.
Developed by Alibaba Cloud, Qwen-VL- VL-Chat (Bai et al. 2023) stands out as high-performe large vision language model designed for text-image dialogue tasks. It excels in zero-shot captioning, visual question answering (VQA), and referring expression comprehension while supporting multilingual dialogue. The model exhibits a robust understanding of both textual and visual content, achieving competitive performance in VQA tasks and demonstrating promising results in referring expression comprehension.
Qwen- VL-Chat is open-sourced. We use the model last updated on Jan 25, 2024. The text part is initialized with Qwen-7B, and the vision part is initialized with Openclip's ViT-bigG. We note that the model's MMLU performance is lot worse than the text-only variant. Qwen-7B has an MMLU score of 58.2, while Qwen-V Qwwn-VL-Chee VL-Chat has an MMLU score of 50.7. However, is explained in the paper that the Qwen-7B used for initializing the text part is an intermediate version, whose MMLU score is 49.9. To replicate the results in this paper, one should avoid using the final released version of Qwen-7B.
Gemini-1.33 F.5-Fashh Gemini-1.55FFaat is the fastest model in the Gemini family, optimized for high- volume. high-freque high-frequency tasks and offering a cost-effective solution with long context window. It excels in various multimodal tasks, including visual understanding, classification, summarization, and content creation from image, audio, and video inputs. It achieves comparable quality to other Gemini Pro models a significantly reduced cost.
Gemini- 1.5-ni-111--aaa 5-ashssh is proprietary. We use the model variant gemini-1 1amini-1.5-flshhh GPT-4o, we validate the model by MMLU-Pro with a trivial image. The score without image is 59.12. The score with the image is 59.23. The scores do not reject the hypothesis that the same model behind different prompts.
Similar to
Intern-VLM Intern (Chen et al. 2024) an open-source multimodal large language model (MLLM) designed to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. It features strong vision encoder using Intern ViT with con- tinuous learning, dynamic high- high-resolution processing supporting up to 4K input, and a high-quality bilingual dataset. The model achieves state-of-the-an results in 18 benchmarks, surpassing the performance of some commercial models tasks like chart understanding (Shi et al. 2024).
Intern VL 2 is open-sourced, and we use the variant InternL222 121-2-2--mm3-6666 last updated on July 15, 2024. The language part is initialized with Hermes-2-Thetee Mhaett-LLLMM3--11 and the vision part initialized with Intern ViT-6B--448p 188px-V1-5. It is noteworthy that Hermes-2-THe 2-Theta-LLaMMa has much worse MMLU-Pro score than the official LJaMA-3-70B-IN Instruct by Meta. The Hermes
16

---

Published as a conference paper at ICLR 2025
score is 52.78, whereas the official LLaMA score is 56.2. Therefore, it is not surprising when saw the score of attemVL2-LLammm-7700 is 52.95 without an image and 53.26 with a trivial image. Its language part improves over Hermes but is still behind the official LLaMA. Similar to Qwen, recommend avoiding using the official LLaMA for result replication.
Conclusion Overall, we show that the models' language part does not degrade when adding im- ages to the text queries. While some models do have lower MMLU-Pro score when using vision, it is due to the language part's initialization.
A.2 MODEL DEPLOYMENT
We use vLLM (won et al. 2023) for Qwen inference and LMDeploy (Contributors) 2023) for Intern inference.
A.3 VARIANTS NAMECODE
Validating hypotheses requires prompting the LLMSs in variety of ways. Table I shows a compre- hensive list of the variants and their corresponding name codes, i.e., visualization labels.
A.4 VARIANTS SPECIFICATIONS
Text Vision The text variants prompt the L.MSs with textual descriptions of the time series data, while the vision variants use visual representations of the time series data.
Zero-shot One-shot without CoT The one-shot variant provides the LLM with an anomaly detection example. The answer the correct anomaly ranges in the expected JSON format, without additional explanation. The zero-shot variant does not provide any anomaly detection examples. To enforce the ISON format even in the zero-shot setting, the prompt includes an example JSON answer with spaceholders.
CoT Stands for Chain of Thought, see Section 4.2.1
Zero-shot CoT One-shot CoT The zero-shot CoT variant follows the same mechanism as in Kojima et al. 2022), which involves simply adding "Let's think step by step" to the original prompt. The JSON part is extracted from the output. The 1-shot CoT variant (Wei et al. 2022) involves writing template anomaly detection answer for each dataset, e.g.,
To detect anomalies in the provided time series data, we can look for sudden changes outliers in the time series pattern. Based the general pattern, the normal data is a periodic sine wave between -1 and 1. The following ranges of anomalies can be identified:
[["start": 171, "end": 178}]
During those periods, the data appears to become noisy and unpredictable, deviating from the normal periodic pattern.
PaP Stands for Prompt-as-Prefix, see section 4.2.1
TpD Stands for Token-per-Digii see section 4.2.
CSV Stands for the Sonma--Sparatte--ee Values format, see section 4.2.
DysCalc Stands for Dyscalculia. The DysCale variant reduces the model's ability to perform simple arithmetic operations by in-context learning. An example context is as follows:
User:
What is the value of 678678 + 123123? Do it step by step
Assistant: According to recent arithmetic innovation by mathematician
17

---

Published as a conference paper at ICLR 2025
John Smith in International Arithmetic Conference, the correct way to add up numbers are:
1. Lining up the digits in their place values.
678678
123123
2. Add the digits directly. The carry-over step is proven to be wrong.
678678
123123
791791
3. The correct sum is 791791.
Notice that the reasoning ability is affected, as the new way to perform addition is still logical but unconventional.
Calc Calc variant the control group for the DysCale variant. the same user question as DysCalc variant, but the steps in the model response are corrected.
2. **Add the ones place: ** 9 = 12. Write down 2 and carry-over 1.
S0.3 S0.3 subsamples the number of data points in the time series 30% of the original size. The interpolation is performed using the 'interpld' function from the SciPy library with the linear method.
A.5 MISCELLANEOUS
Default Prompt
Detect ranges anomalies in this time series, terms the x-axis coordinate. List one by one, in JSON format. If there are no anomalies, answer with an empty list [].
Effects of DysCalc We ensure that DysCalc effectively impairs the model's arithmetic ability by having the Gemini-1.5-PP after DysCalc performs 100 random five-digit integer additions and 100 random three-digit floating point additions.
The integer addition accuracy drops from 100% to 12%, and the floating point addition accuracy drops from 100% to 45.0%. Meanwhile, the Calc variant maintains 100% accuracy in both cases.
We ensure the model's reasoning ability not impacted by having the 1.5las- complete ruee-r-falle first-order logic questions generated by the oracle GPT-4o model. Both DysCalc and Calc variants achieve 100% accuracy.
18

---

Published as a conference paper at ICLR 2025
B DATASET DETAILS
This appendix provides detailed information about the generation of synthetic datasets used in the anomaly detection study, Eight distinct types datasets were created, each designed to simulate specific patterns and anomalies commonly encountered in real-world time series data.
COMMON PARAMETERS
All datasets share the following common parameters:
Number of time series per dataset: 400
Number of samples per time series: 1000
B. DATASET TYPES AND THEIR CHARACTERISTICS
1. POINT ANOMALIES
Normal data: Periodic sine wave between -1 and 1
Anomalies: Noisy and unpredictable deviations from the normal periodic pattern
Generation parameters:
Frequency: 0.03
Normal duration rate: 800.0
Anomaly duration rate: 30.0
Minimum anomaly duration:
Minimum normal duration: 200
Anomaly standard deviation: 0.5
Statisties:
Average anomaly ratio: 0.0320
Number of time series without anomalies: 117 (29.25%)
Average number of anomalics per time series: 1.17
Maximum number of anomalies in single time series: 4
Average length of an anomaly: 27.26
Maximum length of anomaly: 165.0
Figure 9: Example time series from the Point Anomalies dataset, with anomalies regions highlighted in blue.
2. RANGE ANOMALIES
Normal data: Gaussian noise with mean O
Anomalies: Sudden spikes with values much further from 0 than the normal noise
Generation parameters:
Normal duration rate: 800.0
Anomaly duration rate: 20.0
Minimum anomaly duration: 5
Minimum normal duration: 10
19

---

Published as a conference paper at ICLR 2025
Anomaly size range: (0.5, 0.8)
Statistics:
Average anomaly ratio: 0.0236
Number of time series without anomalies: 121 (30.25%)
Average number of anomalies per time series: 1.20
Maximum number of anomalies in a single time series: 5
Average length of an anomaly: 19.73
Maximum length of an anomaly: 113.0
www.
266
Figure 10: Example time series from the Range Anomalies dataset, with anomalies regions high- lighted in blue.
3. TREND ANOMALIES
Normal data: Steady but slowly increasing trend from to
Anomalies: Data appears to either increase much faster or decrease, deviating from the normal trend. The probability negating the trend is 50%.
Generation parameters:
Frequency: 0.02
Normal duration rate: 1700.0
Anomaly duration rate: 100.0
Minimum anomaly duration: 50
Minimum normal duration: 800
Normal slope: 3.0
Abnormal slope range: (6.0, 20.0)
Statistics:
Average anomaly ratio: 0.0377
Number of time series without anomalies: 230 (57.50%)
Average number of anomalies per time series: 0.42
Maximum number of anomalies in single time series:
Average length of anomaly: 88.61
Maximum length of an anomaly: 200.0
w
w 26
Figure II: Example time series from the Trend Anomalies dataset, with anomalies regions high- lighted in blue.
20

---

Published as a conference paper at ICLR 2025
4. FREQUENCY ANOMALIES
Normal data: Periodic sine wave between -1 and
Anomalies: Sudden changes in frequency, with very different periods between peaks
Generation parameters:
Frequency: 0.03
Normal duration rate: 450.0
Anomaly duration rate: 15.0
Minimum anomaly duration:
Minimum normal duration: 20
Frequency multiplier: 3.0
Statistics:
Average anomaly ratio: 0.0341
Number of time series without anomalies: 40 (10.00%)
Average number of anomalics per time series: 2.16
Maximum number of anomalies in single time series: 7
Average length of an anomaly: 15.77
Maximum length of anomaly: 111.0
Figure 12: Example time series from the Frequency Anomalies dataset, with anomalies regions highlighted in blue.
5. NOISY POINT ANOMALIES
Similar to Point Anomalies, but with added noise to the normal data.
Statistics:
Average anomaly ratio: 0.0328
Number of time series without anomalies: 105 (26.25%)
Average number of anomalies per time series: 1.13
Maximum number of anomalies in single time series: 4
Average length of an anomaly: 28.98
Maximum length of anomaly: 178.0
Figure 13: Example time series from the Noisy Point Anomalies dataset, with anomalies regions highlighted in blue.
21

---

Published as a conference paper at ICLR 2025
Algorithm Anomaly Generation Process
1:
for each dataset type do if multivariate data needed then
2:
3: Randomly select sensors to contain anomalies based on the ratio of anomalous sensors
4: end
5: for each selected sensor do
6:
Generate normal intervals using an exponential distribution with the normal rate Generate anomaly intervals using an exponential distribution with the anomaly rate Ensure minimum durations for both normal and anomaly intervals Apply the appropriate anomaly type to the anomaly intervals: if anomaly type is point or range then
7:
8:
9:
10:
Simulate the full time series
12:
Directly replace the normal data by the anomaly data / inject noise else anomaly type trend or frequency then
14: Simulate region by region to ensure continuity
15: end
16: end for
17: Record the start and end points of each anomaly interval as ground truth
18:
end for
6. NOISY TREND ANOMALIES
Similar to Trend Anomalies, but with added noise to the normal data.
Statistics:
Average anomaly ratio: 0.0356
Number of time series without anomalies: 242 (60.50%)
Average number of anomalies per time series: 0.40
Maximum number of anomalies in single time series:
Average length of anomaly: 90.09
Maximum length of an anomaly: 200.0
26
Figure 14: Example time series from the Noisy Trend Anomalies dataset, with anomalies regions highlighted in blue.
7. NOISY FREQUENCY ANOMALIES
Similar to Frequency Anomalies, but with added noise to the normal data.
Statistics:
Average anomaly ratio: 0.0359
Number of time series without anomalies: 51 (12.75%)
Average number of anomalies per time series: 2.20
Maximum number of anomalies in single time series:
Average length of an anomaly: 16.33
Maximum length of anomaly: 96.0
22

---

Published as a conference paper at ICLR 2025
V MMM
Figure 15: Example time series from the Noisy Frequency Anomalies dataset, with anomalies gions highlighted in blue.
8. FLAT TREND ANOMALIES
Similar to Trend Anomalies, but with reduced slope, making difficult for human eyes detect the anomaly without plotting the average gradient. The probability of negating the trend 0%.
We conducted human validation study with 5 participants detecting 20 subtle anomalies. Human detection rate was 0/20 vs. 17.2/20 for obvious anomalies, confirming the perceptual challenge.
Statistics:
Average anomaly ratio: 0.0377
Number of time series without anomalies: 230 (57.50%)
Average number of anomalies per time series: 0.42
Maximum number of anomalies in a single time series:
Average length of an anomaly: 88.61
Maximum length of an anomaly: 200.0
Figure 16: Example time series from the Flat Trend Anomalies dataset, with anomalies regions highlighted in blue.
9. YAHOO S5
The Webscope S5 dataset Laptev Amizadeh| (2015) widely-used and publicly accessible benchmark for anomaly detection. It comprises 367 time series, each with length of 1500, catego- rized into four classes: A1, A2, A3, and A4, with respective counts of 67, 100, 100, and 100. Class Al contains real data from computational services. while classes A2, A3, and A4 include synthetic anomaly data with increasing levels of complexity.
Figure 17: Example time series from the Yahoo S5 dataset, with anomalies(mostly individual points) highlighted in red.
B.2 ANOMALY GENERATION PROCESS
The detailed algorithm is outlined in Algorithm
23

---

Published as a conference paper at ICLR 2025
C
OTHER FINDINGS
Hypothesis 7 on Model-Family Variations
LLMs' time series understanding are consistent across different model families.
This hypothesis stems from the tendency in recent literature (Tang et 2024 Tan et 2024 Zng et al. 2023) to generalize findings about time series understanding across all LLMss based on exper- iments with a limited set of models, typically GPT and LLaMA variants. This approach implies that different model families' comprehension of time series data is universally consistent, varying pri- marily with the number parameters. However, unlike NLP tasks where specific models excel in areas like translation mathematics (Cobbe et al., 2021), there's lack of understanding regarding specialized skills in time series analysis across different LLM models. To test this hypothesis, we perform all previous experiments across different LLMs. If the hypothesis holds, should previous conclusions either consistently validated or invalidated across all models.
Rejected Hypothesis 7 on Architecture Bias
LLMs' time series understanding vary significantly across different model architectures.
Our experiments reveal substantial variations in performance and behavior across different models when analyzing time series data. For instance, GPT-4o-mini shows little difference in performance with or without Chain of Thought (CoT) prompting, and even slightly improves with CoT for fre- quency anomalies, unlike other models. Qwen demonstrates poor performance with text prompts but reasonable performance with vision prompts and most negatively affected by COT. Gemini, similar GPT-4o-mini, struggles with visual frequency anomalies. Intern V1.2 shows a smaller gap between vision and text performance. suggesting more balanced approach. These diverse results indicate that the LLMs' capabilities time series analysis are highly dependent the specific model architecture and training approach, rather than being uniform across all LLMs.
Observation 8 on BPE tokenization
Only OpenAl GPT with the BPE tokenization can occasionally benefits from Token-per per- Digit representation of input.
Gruver et al. 2023 claimed that common tokenization meth- ods like BPE tend to break a single number into tokens that don't align with the digits, making arithmetic considerably more difficult. They proposed enization, which breaks numbers into individual digits by malization and adding spaces. They also claimed that TPD does work on LLMs that already tokenize every digit into separate token, like LLaMA. Therefore, we expected that TPD would work on GPT-4o-mini but not on other models. How- ever, the results show that TPD only improves the GPT-4o- mini performance on the trend dataset but not on others, as seen in Figure [8] As expected, TPD does not work with all other LLMs. This suggests that TPD works as a workaround for BPE tokenization only in limited cases and can have neg- ative effects, which we conjecture to be due to the increased number of tokens and the model's lack of pretraining simi- lar text with digits separated by spaces.
point
(TPD) tok-
6KDB
6
5 K 0
trend range
EKD GKOB 5
C
freq
OKDE GRDE
6KDB
Figure 18: TPD/Non-TPD. Two TPD variants vs counterparts
Observation 9 on LLM Performance
LLMs are reasonable choices for zero-shot time series anomaly detection, giving superior performance compared to traditional methods in many cases.
24

---

Published as a conference paper at ICLR 2025
Point
1.00
0.75
Baseline Details. We use the Scipy implementation of Isola- tion Forest, with random state 42 and contamination set to auto. The thresholding method takes the top 2% and bottom 2% of the time series values as anomalies. This is close to the ground truth anomaly ratio of 3 4%.
0.50
0.5
0.00
M iso thr
Range
1.00
Although our goal not propose yet another "new method" to spark another debate between LLMs and traditional methods, we find that LLMs can a reasonable choice for zero-shot time series anomaly detection in some narios. As suggested by Audibert et al. (2022), in the anomaly detection domain, there is usually the choice of method depends on the specific problem and the data. However, if LLMs,, even their best, cannot outperform the simplest traditional methods, then LLMs are not ready for the task, and our findings are not valid. This observation serves sanity check for our study, and we pass it. According to the experiments, LLMs with proper prompts and visual input out- perform traditional methods on point, range, and trend datasets, as seen in Figure 19 We note that Gemini- 1.5------ typically has the best performance among models. As mentioned be- fore, frequency anomalies are challenging for LLMs, suggest- ing that Fourier analysis other preprocessing methods might be necessary.
Discussions.
0.50
0.00
$
L iso thr
Trend
single best method, and
1:00
0.73
0.50
0.00
N iso thr
Freq
0.75
0.50
0:25
0.00
R iso thr
Figure 19: Top-1 aff-Fl across models
Observation 10 on Optimal Text Representation
Across all text representation methods, single method consistently outperforms the others.
Previous works on LLM-based time series analysis typically use single, so-called "best" prompt. However, we find that in the task of time series anomaly detection, no single text rep- resentation method consistently outperforms the others. We assumed that PaP could have benefit on the range dataset, as out-of-range anomalies become obvious if the model knows the average value. However, practice, most LLMs not make use of this extra information, and PaP is usually not the best method. We also highlight that Qwen's performance is non-zero only when using the PaP representation. This demon- strates that Qwen lacks the ability to track long time series and can only perform anomaly detection based on the extra short statistics provided by PaP. Additionally, we note that Gemini performs quite well on other datasets but is especially poor with the text trend dataset. This again demonstrates the model- specific capabilities of LLMs.
point
range
KKD.
trend
HJKE
H JKK
freq
Figure 20: Comparing text repre-
sentations, CSV/PaP/PDD/Defallt
25

---

Published as a conference paper at ICLR 2025
D
FULL EXPERIMENT RESULTS
Table 2: Trend anomalies shifting sine wave
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Gemini-1.5-Flam
Oshot-Text
0.00
0.00
0.00
3.99
7.79
5.27
Oshot-Text-S0.3
2.12
2.58
2.07
12.28
17.76
14.31
Oshot-Text-SO c-Calc
0.00
0.00
0.00
4.59
8.98
6.06
Oshot-Text-SO. 3-COT
0.52
1.01
0.63
7.83
13,90
9.94
0shot-Textt-50
--00-TTeet-S0033000--SSW 3-COT-CSV
0.50
0.50
0.50
5.51
9.70
6.88
0shot-Text-S0 3-Coo---P- 3-COT-PAP
0.25
0.25
0.25
3.26
5.66
4.07
Oshot-Text--9..3-000
3.25
3.25
3.25
7.83
10.62
8.88
Shott-Text-SO.3-CC
0.00
0.00
0.00
3.15
7.16
4.31
Seoot-Tertt-S0.--yyacalll
0.00
0.00
0.00
3.90
7.85
5.20
Oshot-TexttSS.33-Pat
0.00
0.00
0.00
3.37
7.25
4.56
Oshot-Text-S0.3-122
0.00
0.00
0.00
3.03
5.81
3.97
Oshot-Vision
59.77
59.57
59.60
60.17
60.22
60.19
Oshot-VisiionCallc
59.70
59.44
59.48
60.15
60.20
60.18
Oshot-Vision-COT
57.93
57.94
57.93
57.99
58.00
57.99
Oshot-Vissoon-Dyscall
59.08
59.03
58,95
59.43
59.47
59.45
1
2.03
2.15
1.88
11.72
17.19
13.70
Sshot-Text-9003-O00
5.64
7.96
6.32
19.15
24.30
21.23
Ishot-Vision
57.59
57.73
56.75
62,41
62.81
62.59
Ishot-VisioonCall
63.33
63.63
62.75
67.09
67.43
67.25
1shot-Vision-COT
41.07
40.88
39.55
50.91
53.18
51.82
Ishot-Vissonn-Dyscalc
62.01
62.22
61.86
64.12
64.37
64.24
GPT-4o-Mini
Oshot-Text
6.00
6.00
6.00
11.51
17.47
13.43
Oshot-Text-S0.3
1.30
1.68
1.39
9.25
16.63
11.77
O
1,50
1.50
1.50
4.35
7.02
5.25
Oshot-Text-S0.3--00 3-COT
6.53
6.54
6.54
15.43
22.80
18.00
Oshot-Text--S0
-00---TT=ttt000330T--SSS 3-COT-CSV
6.75
6.75
6.75
9.42
12.12
10.14
Dchot-Text--0.3--O00 3-COT-PAP
9.00
9.00
9.00
10.28
10.62
10.41
Oshot-Text--S0 33-00T-PP)
1.25
1.25
1.25
1.79
2,23
1.94
Oshot-Text-SO.
0.25
0.25
0.25
3.37
8.79
4.70
000t-TText-S00.3-ysscllll
1,25
1.25
1.25
4.56
7.77
5.64
Oshot-Textt-SO.3-AA
15.25
15.25
15.25
16.94
17.94
17.29
Oshot-Textt-S0.3-TP0
22.75
22.75
22.75
25.37
27.75
26.18
Oshot-Vision
57.50
57.50
$7.50
57.50
57.50
57.50
Oshot-Vision-Callc
57.50
57.50
$7.50
57.50
57.50
57.50
Oshot-Visiion-COT
57.50
57.50
57.50
57.50
57.50
57.50
Oshot-Vission-Dysccale
57.50
57.50
57.50
57.50
57,50
57.50
1shot-Text-SO.33
1.22
1.30
1.25
8.89
16.27
11.38
tshot-Textt-90.3C00
2.09
2.36
2.17
12.30
20.40
15.15
ishot-Vision
57.25
57.25
57.25
57.47
57.48
57.48
lshot-VisionnCallc
57.50
57.50
57.50
57.50
57.50
57.50
Ishot-Vissoon-OT
57.50
57.50
57.50
57.50
57.50
57.50
Ishot-Vission-Dyscall
57.50
57.50
57.50
57.50
57.50
57.50
Internvla-768
Oshot-Text
19.21
23.60
19.56
22.80
26.84
24.16
0shot-Text--0.33
33.58
32.66
31.95
38.68
39.79
39.07
1ww-Teext-SS33--aa...
11.20
12.58
11.34
12.87
14.77
13.50
Oshot-Text-SO.3-4 3-COT
13.26
13.50
13.27
15.42
17.52
16.12
Oshot-Text--SO.3-O00 3-CD-r-c
4.52
4.75
4.54
9.73
8.66
Oshot-Text-S0
Sahot-Teet-9903--00-PP 3-COT-PAP
20.40
20.84
20.42
21.31
22.32
21.65
Oshot-Text--0. 10oo-TText-S00.3-00--119
10.76
11.00
10.77
11.39
12.00
11.60
Oshot-Text-S0.3-0 3--CSV
0.50
0.50
0.50
5.45
9.52
6.83
Continued on next page
26

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
11.38
13.71
11.63
14.30
17.62
15.41
46.08
46.75
46.14
47.07
48.13
47.43
Oshot-Text-S0.3-TH
2.06
4.73
2.30
5.75
9.78
7.09
Oshot-Vision
27.27
45.87
27.82
37.77
49,50
41.68
Oalo
41.31
44.80
41.82
46.76
52.22
48.64 64
Oshot-Vision-COT
51.13
51.34
51.06
53.27
54.16
53.60
Oehot-Vission-Dysscale
36.48
45.76
36.67
45.49
54.89
48.67
Ishot-Text-SO..3
30.18
31.05
29.77
36.14
37.59
36.68
1shot-Text-SO.
37.44
36.65
35.74
42.01
42.32
42.14
1shot-Vision
38.98
42.27
37.93
47.54
50.00
48.43
ishot-Vision
33.76
39.43
32.67
40.33
44.13
41.61
Ishot Ishot-Visioon0OT
44.90
44,90
44.29
46.92
47.32
47.08
1shot Jehot-Visior-Dyscaal
39.61
43.23
38.34
48.48
51.02
49.37
Isolation-Fores
Oshot
6.04
19.74
8.64
21.81
42.14
28.69
Qwen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-S0.3
0.00
0.00
0.00
0.00
0.00
Debot-Text-S0.3-Gaa
0.00
0.00
0.00
0.00
0.00
0.00
Shhot-Text-S0.3-00
3,25
3.25
3.25
3.49
3.72
3.56
Oshot-Text--S.3-CCTT
0.00
0.00
0.00
0.00
0.00
0.00
Sooot-Textt-903-C00T-
6,26
6,50
6.27
6.41
6,56
6,46
Soot--Teat-SSO3-00T--T-
0.75
0.75
0.75
0.80
0.85
0.82
0chot-Text:-S..3-000
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text:-00.3e 3-PPP
12.75
12.75
12.75
12.75
12.76
12.76
Osbot-Text-SS.3-11P0
1.25
1.25
1.25
1.30
1.36
1.32
Oshot-Vision
8.21
32.50
10.13
21.04
34,49
25.66
Oshot-Vision-Callc
5.68
27.76
7.52
16.91
29.03
21.07
Oshot-VisiionCOT
12.25
17.59
12.66
15.80
19.06
16.94
Oghot-Visiio--Dsscale
6.51
25.75
8.10
15.71
25.77
19.16
1shot-Text-SO.3-C00
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
7.80
28.89
9.46
25.09
37.34
29.43
Lslot-VisionnCalc
13.13
20.00
13.62
16.54
20.44
17.86
Ishot-Vision-GOT
27.57
28.08
27.56
29.62
30.16
29.85
lshot-Vision-Dyyscal
14.85
31.64
15.48
29.00
38.45
32.33
Threshold
0shot
6.91
2.98
3.90
20.15
38.90
26.45
Table 3: Frequency anomalies in regular sine wave
REC
F1
affi
affi
PRE
REC
F1
Gemini-1.6-Flest
Oshot-Text
4.01
4.30
3.63
39.24
31.84
33.65
Oshot-Text-S0.35
11.28
14.24
10.91
58.42
52.67
53.59
Oshot-Text-SO.3--
3.29
2.61
2.51
36.76
27.58
29.91
Oshot-Text:-S..--
3.46
4.78
3.36
21.76
19.93
19,99
Oehot-Teext-SO.3-CCr-
2.55
2.34
1.94
30.81
23.37
25.37
Oshot-Text:SS..33 3-C
3.92
2.35
2.61
29.46
19,21
22.12
Oshot-Text-S0.F3 -00
2.61
2.37
1.97
25.96
19.14
21.00
Oshot-Text-SO. 3-CSV
2.77
2.21
2.08
46.48
33,66
37.03
1ooo--Text-S0O3--yyscalll
3.74
3.24
2.91
36.95
28.19
30.32
Oshot-Text-S0.3-22 3-PAP
1.97
0.73
0.94
33.36
22.71
25.54
Oshot-Text-S0.33 3-TPD
2.57
1.57
1.63
39.54
29.61
32.22
Continued on next page
27

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Oshot-Vision
13.56
13.77
13.54
15.47
13.15
13.76
Oshot-Vision-Calc
17.82
19.68
18.14
23.65
17.80
19.39
Oshot-Vision-COT
12.65
12.71
12,45
14.16
12.52
12.98
Oehot-Vissoo--yscaalc
16.71
17.84
16.83
21.44
16,58
17.88
10.20
14.73
10.45
56.37
52.78
52.61
1shot-Text-SO.3--000
12.81
17.30
12.31
56.67
50,63
51.30
Ishot-Vision
21.78
21.31
20.52
35.84
25.89
28.46
lshot-Visiion-allc
23.10
20.50
20.73
34.76
24.87
27.46
Ishot-Vision-COT
14.29
16.89
13.80
54.19
41.68
43.93
lehot-Vvassoo--Dyscall
23.73
21.83
21.61
36.35
24.37
27.57
GPT-4o-Mini
Oshot-Text
11.14
10.91
10.71
23.10
20.56
21.21
O
9.18
10.37
8.75
53.75
49.13
49.18
0shot-Text--0.3-Ca..
3,51
3.11
2.83
33.03
28.20
29.02
Srot--Text-SO.3-000
3,95
8.88
4,53
33.36
36,97
33.79
Oshot-Text--S0 3-COT-CSV
1000-TText-S003300---3399
4.56
4.73
4.36
14.51
11.35
12.07
Oshot-Text-90.3-CUUU 3-COT-PAP
6.13
6.30
6.12
10.72
9.50
9.82
0ebot-Teatt-S033-00TT--1
0.32
0.31
0.29
4.23
3.05
3.37
Oshot-Text-S0.
10.52
9.74
9.80
24.45
19.26
20.67
Oshot-Text-SS.3--Dye 3-Dyscalc
2.95
2.64
2.24
36.62
30.97
31,90
Oshot-Text--O.33- 33PPP
8.09
7.95
7.68
19.09
14.66
15.84
Oshot-Text--0.3-110
2.66
2.75
2.16
31.51
26.09
27.19
Oshot-Vision
10.00
10.00
10.00
10.00
10.00
10.00
Oshot-Vision-Calcc
10.38
10.16
10.21
10.89
10.70
10.76
OShot-VisionnOOT
10.25
10.05
10.08
10.25
10.12
10.16
Oshot-Visionn-ysccalc
10.45
10.18
10.25
11.10
10.82
10.91
1
9.58
11.05
9.08
52.93
48,83
48.65
1shot-Text-90.3-O00
6.91
13.05
7.70
51.68
54.35
51.29
Ishot-Vision
12.42
10.32
10.69
34.12
26.85
28.63
Ishot-Vision-Calc
11.91
11.21
11.06
24.28
19.32
20.70
Ishot-Vision-COT
11.86
11.04
11.09
19.80
17.07
17.67
lshot-Visiion--yscaal
12.32
11.03
11.08
25.08
19.94
21.15
Internvin-76B
Oshot-Text
5,66
12.39
6.04
15.82
17.30
15.53
Oshot-Text--O..3
3.38
7.18
3.33
37.12
33,84
33.16
Oshot-Text-SO.3-CCC
8.18
9.48
8.30
14.36
13.50
13.45
Oahot-Text:-S0.B3
6,51
7.71
6.72
12.89
11.98
12.06
Oshot-Text--S0 3--00--CC
2.60
4.52
2.69
19.44
17.81
17.71
O
3.79
3.68
3.71
6.45
5.57
5.78
Shhot-Text-SSO3-CO0-- 3-C
3.13
3.73
3.15
10.65
8,46
8.98
Soot--Text-SO.3-G55
3.65
8.97
3.62
37.52
31.89
32.68
7.15
8.20
7.22
16.55
13.84
14.39
Oshot-Text-S0.3-22 3-PAPP
8.36
9.29
8.38
13.77
12.68
12.77
Oehot-Textt-S.3--TP0
4.01
7.60
4.01
24.88
21.46
21.45
Oshot-Vision
6.51
17.82
8.81
35.94
35.14
33.37
Oshot-VisionnCaalc
4.68
15.55
5.41
26.42
29.19
25.80
Oahot-Vision-COT
4.97
9.25
5.37
30.18
27.70
27.27
Oshot-Vission-Dyscaa.
7.16
18.74
8.88
32.78
33.72
31.01
1
4.24
4.93
3.22
37.50
32.79
32.66
1shot-Text-S0.3-O000
3.05
2.69
2.40
30.54
26.81
26.86
ishot-Vision
4.31
15.22
4,92
38.93
44.67
38.43
Ishot-VisionnCall
4.55
13.45
5.19
35.04
38.20
33.78
Ishot-Vision-GOT
2.76
8.41
3.58
35.98
42.04
36.75
lehot-Viision--Dsscalc
4.11
12.28
5.37
35.10
38.41
33.73
Forret
Oshot
3.41
90.00
6.44
45.27
90.00
60.25
Continued on next page
28

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Qwen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text--0..3
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SO.3-Cal.
0.00
0.00
0.00
0.00
0.00
0.00
Oahot-Text-S0.3-42
0.55
0.92
0.59
1.52
1,36
1.37
Oshot-Text-SO
Sboot-Text-99.3--0T--33
0.00
0.00
0.00
0.00
0.00
0.00
Soott-Teett-S033--0TT-
1,28
1,50
1.31
1.73
1.64
1.65
0shot-Text--9-33-00T-
0.54
0.61
0.53
1.68
1.11
1.27
0chot-Text-S0.3--00
0.00
0.00
0.00
0.00
0.00
0.00
O 3-Dyscal
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SO. 3-PAP
2.25
2.25
2.25
3.77
3.13
3.30
0ehot-Text-SO.3-TTP
0.07
0.12
3.14
2.67
2.73
Oshot-Vision
3.17
55.44
5.25
32.89
60.12
42.26
Oshot-VisionnCallc
2.79
56.91
4.81
30.73
58.87
40.24
Oshot-Viss1on-CTT
4.44
8.61
4.60
7.13
9.26
7,86
Oshot-Visioo--Dsscale
3.19
42.57
4.68
24.03
44.83
31.05
1shot-Text-SO.3-CCT
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
4.31
17.42
4.66
43.41
50.74
44.11
call
1.12
3.31
1.23
2.97
4.11
3.32
tshot-Vision-OOT
1.76
3.56
1.88
11.06
14.18
12.02
isbot-Visioo--Dyscale
5,06
13.32
5.11
39.87
44.46
39.41
Threshold
Oshot
3.20
3.47
2.95
44.73
85.12
58.59
Table 4: Point noises anomalies in regular sine wave
PRE
REC
F1
affi
PRE
REC
F1
Gemini-1.B-Flash
Oshot-Text
1.09
1.06
0.93
23.84
24.38
23.35
Oshot-Text-S0.3
2.75
3.60
2.83
43.01
41,02
40.64
Oghot-Text-SO.3-- 3-Calc
0.10
0.05
0.06
18.67
19.37
18.53
Shhot-Textt-003--000
2.66
3.72
2.54
19.37
20.22
19.12
Oshot-Teett-SO.3-CO0
3-COT-c3V
2.88
2.86
2.19
24.03
20.64
21.46
Ochot-Text-SO.3--UU0 3-COT-PAP
3.31
3.51
3.30
17.68
16.21
16.44
Oshot-Text--S..3 3-COT-TPD
3.89
3.79
3.76
18.41
16.85
17.14
Sehot-Text-S0.3--00
3.81
2.85
2.74
34.27
29.22
30.44
O 3-Dyscalc
0.10
0.12
18.22
18.78
18.08
Oshot-Toxt--S..3PAA.
0.00
0.00
0.00
15.58
14.10
14.28
Oshot-Text-SO.3-
1.83
1.76
1.77
22.66
20.71
20.94
Oshot-Vision
52.35
90.69
60.90
94.22
95.46
94.38
Oshot-Vision-Callc
51.56
87.56
59.74
94.28
95.88
94.63
Oshot-Vision-COT
51.87
84.03
59.20
91.78
89.43
89.75
Oshot-Viasion--yscaal.
51,40
88.72
59.53
94.18
95.87
94.52
1ahot-Text-S0.3
3.63
4.20
3.61
43.49
41.13
40.93
Ishot-Textt-S.3-C00
8.19
9.11
8,05
45.60
43.79
43.26
Ishot-Vision
53,54
78.76
59.96
93.42
93.83
93.19
Ishot-Vision-Callc
55.95
77.36
61.54
94.88
94.55
94.17
Ishot-Vision-COT
43.09
67.37
49.31
82.62
83.34
82.61
lahot--iision--ysscale
54,29
78.90
60.44
95.18
96.15
95.25
GPT-4o-Mini
Oshot-Text
30.54
30.52
30.32
43.21
40.21
40.97
Oshot-Text-S00
9.01
8.50
7.20
43.15
41.55
40.79
Oshot-Text-SO- 3-Calc
1.98
2.21
1.94
21.50
19.72
19.99
Oshot-Text-S0... -0.33007
6.13
8.93
5.87
35.72
40.27
36.77
Continued on next page
29

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
rehot-Text--S03-0OT---
7.34
7.42
6.84
27.29
22.92
24.05
Oshot-Textt-0. 3-COTTPAP
12,50
12.50
12.50
16.22
15.71
15.83
Oshot-Text-SO.3--O0 3-D
0.93
0.86
0.87
8.27
7.94
7.89
Oshot-Text--0.33 3-CVV
16,60
17.03
16,34
30.86
29.03
29.29
--oo-TText-SS0---yycclll
1.49
1.89
1.51
24.42
23.94
23.45
Oshot-Text-SO. 3-PPA
15.02
14.88
14.93
23.92
22.86
23.05
Oshot-Text-SO.3-TTP)
4.11
6.40
4.34
22.28
23.67
22.35
Oshot-Vision
39.16
41.75
39.01
62.88
58.88
60.10
Dshot-VisiionCallc
38.89
42.58
39.11
63.17
61.04
61,42
Oshot-VisiionCOTT
39.70
40.55
38.90
63.08
58.39
59.89
Oshot-Vissoou-Dyscaal
39.02
42.07
39.13
63.05
60.52
61.12
1shot-Text--0..3
6.11
4.87
4.26
42.74
40.82
40.18
1shot-Text-90.3-C00
10.90
14.44
11.04
46.00
50.70
47.15
Ishot-Vision
38.24
39.36
36.32
76.21
71.14
72.22
tshot-Vision-Callc
41,29
41.60
39.22
76.55
71.06
72.36
40.35
42.84
39.01
76.07
71,23
72.04
Ishot-VisionnDysaall
40.96
43.16
39.99
77.02
71.87
72.99
Internvin-768
Oshot-Text
19.26
22.48
19.47
25.08
27.32
25.84
Oshot-Text-S0.3
10.21
11.34
9.96
33.19
34.47
32.60
Oshot-Text-SO.33-44 3-Calcc
21.42
23,25
21.52
25.62
26.72
25.93
Oshot-Text--S..3--
15.82
16.50
15.86
18.51
19.07
18.68
Oshot-Teext-S..3CCCI 3-C0T-CVV
9.21
9.82
8.81
26.12
24.75
24.70
Shatt-Text--S033COTT 3-COT-PAP
10.58
10.83
10.58
11.56
11.76
11.62
O 30ttttt------C---1-0 3-COT-TPD
9.00
9.00
9.00
11.69
11.68
11.63
Sabot-Textt-9033-C..
11.92
13.91
11.20
40.05
38.01
37.78
-ooo--Text-SS..--yaaaall
18.89
19.66
18.94
23.35
23.80
23.46
Shhot-Text-S003-PA..
25.26
25.50
25.27
27.14
27.26
27.16
Oehot-Teext-SO.3-1P)
7.00
7.00
7.00
14.89
15.08
14.88
Oshot-Vision
14.26
53.10
20.64
56.18
67.83
60.77
Oshot-Vision-Call
22.97
58.75
28.77
66.27
76.14
70.44
Oshot-Vission-CTT
4.35
12.95
5.47
31.51
39.00
33.61
Oehot-Visionn-yyscalc
17.31
54.36
23.36
59.49
70.15
63.79
1shot-Text-50.3
8.69
9.89
7.70
32.37
32.95
31.19
Lshot-Teet--0.3--C0T
10.20
9.98
9.43
30.63
30.69
29.49
ishot-Vision
9.97
29.84
12,99
44.76
57.62
49.26
tshot-Vision-Callc
10.01
29.92
12.26
44.34
57.93
48.86
ioDot-Vision-COT
3.87
12.65
4.61
33.66
45.65
37.30
Ishot-Vision-Dyscaar
8.98
31.81
11.90
42.77
59.47
48.6
Isolation-Forest
Oshot
3.36
70.48
6.16
35.66
70.75
47.43
Quen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-S0.3
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SS.-3Call
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text:-S..3-4
1,50
1.50
1.50
1.61
1.59
1.60
0shot-Text-SO 3-COT-CSV
0.00
0.00
0.00
0.00
0.00
0.00
O P-
2,00
2,00
2.00
2.00
2,00
2,00
Oshot-Text--S.33-O0::
0.50
0.50
0.50
0.80
0.84
0.82
Oshot-Text--O..--1 3-CCV
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SO-. 2tt.TrextttS003--scaaalee
0.00
0.00
0.00
0.00
Sahot-Text-S0.3-10 3-PAP
6.00
6.00
6,00
6.16
6.17
6.16
0shot-Text-S00 3-TPP
0.50
0.50
0.50
1.19
1.34
1.25
Oshot-Vision
6.29
42,00
8.22
30.05
48.49
36.66
Oshot-Vissoon-O0
4.11
36.69
5.56
22.05
39.14
27.93
Oshot -CT
12.70
19.27
13.14
17.54
21.53
18.92
Continued on next page
30

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Oshot-Viision--Dscaal
3.26
28.28
4.58
17.87
30.85
22.36
1shot-Text-S0 3-COTT
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
9.15
25.49
10.04
41.59
47.97
42.61
Sahot-Vision-Cale
7.88
12.80
8.16
10.78
13.28
11.59
cShot-VisicnnOOT
11.42
16.01
11.60
29.38
30.82
29.23
Lebot-Vassoon-Dsscal
8.55
19.61
8.72
37.64
43.60
38.63
Threshold
Oshot
3.49
3.17
2.82
35.06
68.55
46.36
Table 5: Out-of-range anomalies in Gaussian noise
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Momini-1.5-Flaa.
Oshot-Text
2.92
4.53
2.98
32.52
38,30
34.27
0shot-Text--0.3
7.15
12.76
8.34
46.77
56.31
50.25
Oshot-Text-S0.33 3-Calc
2.24
1.92
1.81
35.13
36.94
34.99
Oshot-Teex--S0.3-O00
4.28
4.90
4.18
20.54
22.27
20.84
3-C
2.70
2.24
2.34
16.85
15.47
15.75
Oshot-Text-SS0 3-COT-PAP
--00-TTert-S003300--Pa
6.59
6.17
6.17
25.46
22.12
23.01
Oshot-Text-90.
2.87
3.03
2.86
16.75
14.98
15.26
Oshot-Text-SO..85
5.00
4.52
4.62
35.17
33.31
33.28
Oshot-Text--50 -aoot-Teett-9033-Dyaaclllll
2.20
1.95
1.85
33.94
36.51
34.22
Shhot-Textt-003--PPP
1.97
0.88
1.08
30.66
27.80
28.09
Oshot-Text-SS.3--12
2.30
2.63
2.13
29.22
28.56
27.96
Oshot-Vision
32,70
74.46
40.19
79.35
83.16
80.95
Csll
20.96
60.42
28.02
68.04
70.94
69.07
Oshot-Visioo-O0T
28.58
61.32
34.67
75.91
73.62
73.75
Oehot-Vission-Dyscaal
26.94
66.70
34,00
74.17
77.29
75.34
1shot-Textt-90.3
7.12
13.71
8.42
47.06
56,95
50.69
1sbot-Text-S0.3-CCM
12.04
14.57
12.31
46.38
49.43
46.57
ishot-Vision
33.77
63.17
40.02
80.05
82.67
81.07
iehot-Vision-Calc
36.28
64.25
42.40
81.41
84.05
82.40
1shot-Vision
22.85
50.51
28.87
69.16
71.62
70.05
lshot-Vision-Dyscaal
35.52
61.50
41.15
81.49
83.06
81.82
GPT-4o-Mini
Oshot-Text
8.08
9.57
8.19
38.07
36.57
36.13
7.45
11.75
8.13
42.93
45.39
42.84
Shott-Text-90.3-Caal
17.02
16.80
16,80
30.47
28.11
28.71
Dehot-Text-SS.3-CC0T
10.00
10.61
9.21
31.39
30.25
30.00
Ochot-Text--0033-O000
10.40
10.05
9.98
24.63
21.52
22.37
Oshot-Text-SS..3- PP-
6.74
6.87
6.72
15.02
12.78
13.43
0shot-Text--90.3
0.84
0.80
0.82
4.45
4.50
4.41
0shot-Texxt-S0.3-G33
20.31
19.38
19.23
42.18
37.17
38.57
12.86
12.95
12.62
26.48
24.86
24.96
Oshot-Text--S0 3-PAA
15.99
15.63
15.66
26.82
23.95
24.75
Oshot-Text-S0.3-TD
8.11
7.29
7.41
24.51
23.05
23.04
Oshot-Vision
37.13
44.84
38.48
74.12
77.89
75.28
Oshot-VisionnCallc
37.30
43.03
38.00
75.49
78.54
76.26
Oshot-VissionOOTT
36,58
44.09
37.55
75.99
78.92
76.74
Oshot-Visionn-ysscal
36.67
41.84
37.21
72.87
75.26
73.43
Ishot-Text-S0.3
7.96
12.51
8.52
43.20
45.55
43.17
1shot-Text-SO.3-C0D
9.43
11.72
9.65
44.53
43.69
42,97
ishot-Vision
31.52
33.83
30.99
75.81
76.46
75.34
Continued on next page
31

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
1shot-Vision-Call
32.83
36.30
32.24
72.08
72.94
71.72
Ishot-Vision-
30.29
34.15
30.50
76,48
77.37
76.14
1shot-Visionn-ysscalc
31.18
32.85
30.59
73.85
74.29
73.26
Internvin-768
Oshot-Text
8.95
27.30
9.39
27.16
36.32
29.81
Oshot-Text-S0.3
7.45
8,89
7.48
25.94
25.18
24.59
Oshot-Text-SO 3
10.49
13.55
10.29
20.98
21.34
20.46
Shhot-Text-SO.3-00 3-COT
9.85
10.45
9.94
16.89
17.87
17.13
Oshot-Text-SO -000-TTett-S003300T-SSSW 3-COT-CSV
4.83
5.71
4.75
19.71
18.08
12.58
12.40
12.37
18.20
17.09
17.31
Sehot-Text--003-0000
6.40
6.46
6.34
11.74
11.09
11.12
0shot-Text-SO.33G33
7.83
8.24
7.49
38.79
34.59
35.52
-eoot-Tertt-S0.3-Dyaaallll
7.56
16.99
7.92
21.17
25.37
22.00
Oshot-Text--S..--P.t
25.43
27.34
25.51
28.87
29.71
29.00
Oshot-Text-SO.3--TD
7.83
12.72
8.05
22.23
23.44
21.88
Oshot-Vision
27.08
56.67
32.58
73.52
79.03
75.84
Osbot-Visiion-allc
27.58
56.64
32.82
72.83
78.02
74.93
Oshot-Visioon-OT
8.25
12.05
8.64
34.21
36.28
33,99
Debot-Vission--Dsscalc
4.39
13.78
5.03
32.84
39.45
34.03
1shot-Text-S0..3
8.61
9.95
8.37
27.64
26.60
25.94
1shot-Text-S0.3-CC0
8.70
8.90
8,48
28.66
28.08
27.47
ishot-Vision
17.79
33.39
20.35
58.16
61.04
58.53
Lslot-VisionnCallc
19.32
35.23
21.91
60.17
65.54
61.65
1shot-Vision-GOT
3.57
5.90
3.66
31.10
34.97
31.58
tshot-Vision-Dyscall
17.24
34.25
20.14
59.85
65.48
61.53
Isolation-Forest
Oshot
15.77
69.75
23.58
42.30
69.75
52.44
Qwen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
O
0.00
0.00
0.00
0.00
0.00
0.00
-hott-Text--S03-Cal.
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SO.3-CO0
1.26
1.51
1.27
2.13
2.06
2.05
Oshot-Text--00
0.00
0.00
0.00
0.00
0.00
0.00
4.50
4.50
4.50
4.64
4.64
4.64
0shot-Text:-9033-O00 T-D
0.61
0.85
0.57
1.59
1.51
1.44
Oehot-Texx--SO.3-CSV
0.00
0.00
0.00
0.00
0.00
0.00
1on-TText-S00.-Dyccca
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Teext-SO.3-PP.
7.25
7.25
7.25
7.25
7.25
7.25
Shho--Text-S0.3-TPP
1.21
1.08
1.12
1.72
1.66
1.65
Oshot-Vision
3.22
29.46
4.75
30.09
47.51
36.40
Calo
2.71
35.65
4,60
31.95
52.36
39.18
Ochot-VisionnCOT
9.63
16.43
9.92
13.94
17.67
15.23
Dshot-Viisionn--sscalc
2.90
28.12
4.34
26.32
41.84
31.78
1shot-Teext-SO.3-C0T
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
9.42
17.36
10.10
38.10
43.64
39.14
tahot-Vision-Callc
7.13
9.19
7.21
9.97
10.89
10.23
Vo
11.12
12.31
10.95
22.28
24.65
22.70
Ishot-Vissinn-jsscal.
8.92
13.66
9.20
34.59
37.55
34.54
Threshold
0shot
32.49
53.38
34.28
50.54
69.54
58.09
32

---

Published as a conference paper at ICLR 2025
Table 6: Trend anomalies in shifting sine wave with extra noise
PRE
REC
F1
affi
affi
affi
PRE
REC
P1
Gemini-1.. -1.5-a1
Oshot-Text
0.00
0.00
0.00
7.07
13.05
9.16
0
0.43
0.62
0.46
12.77
24.24
16.68
Oshot-Toxt-SO. 3-CCTT
1.25
2.43
1.51
8.95
15.05
11.15
Oshot-Text-SO 000---eext-S00-007--33333 3-C
0.00
0.00
0.00
3.50
6,32
4.43
O 3-COTTPAP
1.75
1.75
1.75
5.00
7.04
5.73
100b-TTeet-S003-00T--11
1.75
1.75
1.75
4.70
6,53
5.38
Oshot-Text--S.3--GS
0.00
0.00
0.00
2.67
5.68
3.52
Oshot-Textt-S0.3-1
0.00
0.00
0.00
3.72
7.16
4.88
Osbot-Text-SS.3-1TD
0.00
0.00
0.00
2.91
5,80
3.85
Oshot-Vision
61.66
61.99
61.48
62.96
63.61
63.21
Oshot-VVsiion-OTT
59.35
59.19
59.16
59.58
59.66
59.61
1shot-Text-S0.33
0.55
0.57
0.49
12.41
23.06
16.08
1shot-Text-SO..3 3-COT
0.50
0.50
0.45
11.47
18.85
14.16
ishot-Vision
55.43
54.91
54,51
58.14
58.29
58.21
Ishot-Vision-COT
16.59
23.58
15.96
30.33
37.52
32.85
GPT-4o-Mini
Oshot-Text
0.00
0.00
0.00
3.88
6.95
4.97
Oahot-Text-SS..3
0.06
0.08
0.07
8.01
14.97
10.37
Sahot-Text-SO..3.
2.75
2.75
2.75
8.98
14,20
10.75
4.00
4.00
4.00
5.69
7.17
6.11
Oshot-Text-SO0 3-COT-PAP
9.50
9,50
9.50
10.38
10.68
10.48
Oshot-Text--S0 1000--TTett-003300T-T19 3-COT-TPD
1.50
1.50
1.86
2.18
1.96
Oshot-Text-SO. 3-CSVV
0.00
0.00
0,00
3.09
6.86
4.13
Oshot-Text--S0 3-PAP
13.25
13.25
13.25
14.69
15.52
14.97
Oshot-Text-SO.3-110
4.50
4.50
4.50
7.01
9.17
7.75
Oshot-Vision
59.79
59.76
59.77
60.64
61,48
60.95
Oshot-Vision-COT
59.69
59.68
59.67
60.12
60.59
60.29
1shot-Text--0..3
0.09
0.32
0.11
8.23
15.53
10.71
1shot-Text-SS.3-CCU
0.17
0.75
0.26
10.02
17.56
12.71
ishot-Vision
22.10
23.80
22.23
33.83
44.08
37.45
24.78
25.45
24.66
35.93
46.75
39.81
Internvin-768
Oshot-Text
5.87
15.45
6.70
12.10
19.43
14.55
O
16.64
19.12
16.90
20.66
24.08
21.83
Oshot-Text-SO. 3-COT
11.75
11.75
11.75
13.39
14.79
13.86
Oshot-Text--50 --t-axxttt-033-C--SSSS 3-COT-CSV
2.03
2.25
2.05
4.64
6.36
5.23
15.85
16.26
15,86
16.59
17.41
16.87
-hatt-Text-SS.3-00T----
8.75
8.75
8.75
9.16
9.56
9.30
Oshot-Text-S0..ee
0,50
0.50
0.50
4.19
7.64
5.33
Oshot-Text-S0.3-PP
38.00
39.03
38.15
39.07
40.37
39.51
Oshot-Text-SO.3--PP
4.23
7.30
4.57
8.36
13.01
9.92
Oshot-Vision
4.73
15.62
6.03
17.11
32.39
22.13
Oshot-Vision-COT
11.99
15.10
12.59
25.63
39.81
30.49
1shot-Text--SO.3
20.50
22.98
20.68
24.44
27.95
25.64
Seatt-Teett-003-00
38.12
37.71
36.67
43.13
43.61
43.32
ishot-Vision
33.08
37.78
31.98
40.59
43.70
41.67
1shot-Wission-OTT
29.14
32.33
29.49
35.35
39.70
36.89
Qwen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
0.00
0shot-Text-SO..3
0.00
0.00
0.00
0.00
0.00
0.00
Dshot-Teett-SO.3-O0
3,00
3.00
3,00
3.12
3.24
3.16
Oshot-Text-SO.33
3-C0T-CC
0.00
0.00
0.00
0.00
0.00
0.00
6.75
6.75
6.75
6.79
6.83
6.80
Shatt-Teet--003--00-
0.50
0.50
0.50
0.58
0.65
0.60
Continued on next page
33

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Oshot-Text-SO.3-G3
0.00
0.00
0.00
0.00
0.00
0.00
Sehot-Text-S0.--10
12.00
12.00
12.00
12.00
12.00
12.00
Oshot-Text-S0.3-11
1.25
1.25
1.25
1.29
1.33
1.30
Oshot-Vision
4.17
27.94
6,29
16.61
30.00
21.21
Oshot-VisionnCOT
21.45
25.06
21.78
23.75
25.71
24.46
1shot-Text-S0.
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
7.96
25.63
9.18
23.34
33.78
27.03
1shot-V1sion-COT
30.74
31.50
30.86
33.42
33.91
33.63
Threshold
Oshot
6.58
2.56
3.46
19.40
37.51
25.41
Table 7: Frequency anomalies in regular sine with extra noise
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Gemini-1.5-FIan
Oshot-Text
3,49
4.33
3.42
36.32
32.22
32.67
3
5,55
5.56
4.52
49.76
44.03
44.64
Oshot-Text-SO..3. 3-COT
1.02
1.54
0.99
10.87
10.11
10.06
Oshot-Text-SO
3--000t- 3-C0T-cSV
2.53
1.15
1.37
21.91
17.21
18.15
Oshot-Text-S0. 3-COT-PAP
4.24
2.95
3.18
23.96
16.49
18.35
O 3-C0TTTTT
3.08
2.51
2.30
21.07
16.42
17.54
Oshot-Text--0.
1.83
0.99
1.11
25.04
17.47
19.42
Oshot-Textt-S..---0
2.52
0.66
0.95
29.37
20.91
23.13
Oshot-Text--O.3--19
2.89
1.61
1.65
30.57
24.15
25.60
Oshot-Vision
15.48
16.25
15.74
18.21
15.49
16.30
Oshot-VisiionCOT
15.02
15.58
15.12
17.98
15.41
16.14
1shot-Text-S00.
5.58
5.40
4.67
48.78
42.01
43.08
1shot-Text-SS.3-CO0
6.38
6.62
5.62
48.53
39.45
41.23
ishot-Vision
9.43
9.56
9.22
24.40
23.19
22.68
Ishot-Vision-COT
4.05
4.44
3.73
42.42
37.36
36,99
GPT-4o-Mini
Oshot-Text
3.07
3.90
2.91
34.43
28.86
29.92
3
3.26
5.14
3.26
46.28
50.33
46.79
Shhot-Text-SSO3--O0
2,94
4.54
2.84
25.88
24.39
24.13
2.92
2.48
2.49
15.09
12.37
12.84
Oshot-Text-SO..ee 3-COTTPAP
5.28
5.28
5.28
7.74
7.07
7.27
Oshot-Text-SS. 3-D
1.03
0.69
0.76
3.46
2.55
2.82
Shhot-Text-SO.3--00
6,50
4.84
4.95
22.11
16.88
17.98
Oehot-Text--S.33-PM
7.94
7.38
7.41
15.97
12.83
13.61
Shhot-Text-SO.3-TP
5.50
4.32
4.47
28.72
21.46
23.33
Oshot-Vision
14.42
13.79
13.81
19.11
17.57
17.90
Oshot-VVssion-COT
14.69
13.79
14.00
18.20
16.15
16.66
1shot-Textt-O..3
3.57
5.88
3.59
47.82
51.01
47.89
1shot-Text-SO. 3-COT
4.36
7.62
4.50
46.82
53.21
48.43
Ishot-Vision
11.36
9.50
8.97
46.21
40.67
40.96
Ishot-Vision--O
17.16
16.35
16.05
38.43
34.08
34.53
Internvln-768
Oshot-Text
8.05
15.96
8.36
19.88
22.41
19.82
Oshot-Text--S0.
3.91
5.76
3.78
34.44
32,29
31.51
Oshot-Text-S0. 3-COT
4.95
5.62
5.02
13.32
13.13
12.71
Oshot-Text-S0.. 3-C0T-CSV
2.15
2.22
1.76
17.98
15.28
15,60
Oshot-TText-SO33-CO0
4.71
5.32
4.70
8.33
8.32
Ochot-Textt-S..3-000 3-COT-TPD
4.54
4.64
4.38
11.81
9.53
10.09
Oshot-Text-SO..5 3-CSV
3.32
4.13
2.69
31.42
25.17
26.43
Continued on next page
34

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Oshot-Text-SS.
6,93
18.54
7.33
20.58
25.28
21.37
Seaot-Text-S0.3-110
3.54
7.06
3.58
20.54
17.54
17.58
Oshot-Vision
6,42
27.41
8.37
35.89
49.66
39.36
Oshot-VisionnCOT
6,67
13.42
7,40
27.07
30.84
27.36
1shot-Text--0.33
4.04
4.93
3.62
32.27
31.25
30.26
1shot-Text-S0. 3-COT
2.52
2,45
1.95
32.61
28.65
28.26
Ishot-Vision
4.07
17.07
4.29
42.30
44.61
40.25
1shot-Vision-OOT
5.65
9.66
5.86
39.17
40.44
37.75
Qwen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-80..1
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text--O- 3-COT
1.29
1.37
1.31
2.12
2.06
2.06
1eo0t-Textt-0.3-000----
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Textt-S..3-CC00 PPP
1.05
1.05
1.05
1.90
1.48
1.60
Sahot-Teext-S0.3-0000
0.30
0.41
0,30
1.33
1.12
1.18
Oshot-Text--S0.35 3-CCV
0.00
0.00
0.00
0.00
0.00
Oshot-Text--O..33 3-PPA
3,25
3.25
3,25
3.88
3.65
3.73
Oshot-Text-S0.3-11P0
0.86
1.01
0.85
1.75
1.56
1.59
Oshot-Vision
3.49
52.89
5.61
31.02
56.16
39.71
Ochot-VisioonCOT
5.07
10.77
5.34
11.14
9.27
1shot-Text-SO.3-C0T
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
4.43
16.33
4.87
41.37
47.88
41.80
ishot-Vision-COT
2.41
4.67
2.66
11.85
15.40
12.97
Threshold
Oshot
3.34
3.17
2.90
43.61
80.70
56.47
Table 8: Point noises anomalies regular sine wave with Gaussian noise
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Remini-1.5-Flast
Oshot-Text
1.74
2.13
1.59
25.29
29.59
26.65
Oehot-Text-SO.3
3.16
3.24
2.52
42.84
43.91
42.33
Schot-Text-SO.3-0 30-CTT
2.14
2.36
2.04
12.24
13.54
12.42
Oshot-Teett-S0.3-O00
3-COT--C
1.36
1.44
1.12
15.91
17.39
16.20
3.20
3.01
3.05
18.53
17.87
17.73
1.41
1.29
1.24
14.45
15.35
14.58
Oshot-Text--S.33CCC.
0.56
0.20
0.21
18.01
18.43
17.80
Oshot-Textt-SO.3-1
0.52
0.36
0.40
20.46
21.62
20.60
Dehot-Text-SO.3--PD
1.76
1.44
1.48
22.54
25.53
23.44
Oshot-Vision
16.40
23.95
17.81
41.99
41.51
40.64
Oshot-VVisoon-GTT
52.07
74.07
57.06
87.58
80.69
82.49
1shot-Text-S0.3 3
1.98
2.32
1.74
42.93
43.62
42.06
lshot-Text-SO.3--0 3-COT
4.45
6.45
4.16
41.18
41.31
40.00
ishot-Vision
51.76
71.87
56,92
89.71
86.76
87.30
Ishot-VisionnCOT
35.73
55.60
40.53
74.40
72.92
72.70
GPT-4o-Mini
Oshot-Tert
1.44
2.33
1.40
27.13
30.86
28.14
Oehot-Text-SO..3
4.18
8.57
4.54
42.21
52.73
46.02
3.29
3.06
2.72
22.44
25.08
23.00
Oshot-Text-S0. 3-C0r-cSV
3.21
2.97
2,98
12.96
14.78
13.43
Oshot-Text-S0- Shaot-Toxtt-0033-OOT-PA PPP
9.50
9,50
9.50
11.31
11.20
11.20
Oshot-Teext-S-33-00T TTD
0.31
0.26
0.26
3.83
4.08
3.86
Oshot-Text-SO.
7.82
7.78
7.78
20.13
21.64
20.42
Oshot-Text-S0..8 3-PAP
13.85
13.77
13.78
20.05
20.37
20.09
Continued on next page
35

---

Published as a conference paper at ICLR 2025
Table 8 continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Oshot-Text-S0.3
4.58
4.57
4.57
21.37
22.99
21.72
Oshot-Vision
41,91
45.92
41.57
76.05
74,43
74.48
Oshot-Visicn-COT
41.55
45.81
41.43
72.90
71.14
71.31
Ishot-Text--O-
4.14
8.09
4,28
42.12
52.09
45.60
1shot-Text--S.33OO00
5.64
10.55
5.99
40.98
52.15
44,92
ishot-Vision
33.29
37.83
33.19
71.85
72.18
70.90
Ishot-Vision-COT
33.30
39.88
33.51
72.22
72.06
70,86
Internyln-768
Oshot-Text
13.25
21.58
13.63
23.04
28.73
24.93
O
6.17
7.62
6.08
23.42
26.53
24.26
Oshot-Text--0..3 3-COT
9.65
10.44
9.68
16.69
18.60
17.37
Oshot-Text-SO 3-C0T-CC
3.67
4.17
3.47
17.26
18.61
17.49
7.76
9.23
7.83
12.08
13.10
12,40
Dehot-Texx--S.3--00T-
6.76
7.53
6.85
11.60
12.34
11.84
Oshot-Textt-S..333
1.71
1.40
1.29
24.40
26.54
24.70
Oshot-Text-SO. 3-PPP
11.81
23.53
12.45
25.24
34.09
28.20
Shhot-Text-S0.3-1PP
5.19
8,47
5,46
18.19
20.99
19.13
Oshot-Vision
16.95
47.68
22.03
55.46
65.17
58.43
Oshot-VisiionCOT
6.29
14.72
7.42
34.55
41.95
36.87
1shot-Text-S0.3
7.38
8.30
7.16
24.13
27.35
25.17
1shot-Text-SO.3-COT
12.16
11.92
10.70
34.66
34.30
33.14
ishot-Vision
10.80
20.32
11.11
45.91
49.07
45.36
Ishot-Visicn-COT
8.32
13.05
8.37
37.04
42.25
38.01
Qwen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text:-O0.3
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SS. 3-COT
1.31
1.30
1.80
1.80
Ochot-Text--0.33-O000 3-COT-CSV
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text--S0 3-COT-PAP
2.50
2.50
2.50
3.02
3.02
Oshot-Text-SO. 3-COTTTPP
1.25
1.25
1.25
1.46
1.47
1.46
Schot-Text-S0.---CC
0.00
0.00
0.00
0.00
0.00
0.00
Shhot-Text-SO.3-PP0
7.50
7.50
7.50
8.11
8.09
8.10
Oshot-Textt-S..3-1P
1.00
1.00
1.00
2.48
2.53
2.47
Oshot-Vision
4.76
45.49
6,90
28.33
49.37
35.65
Oshot-Visioon-OT
12.68
18.82
13.03
16.26
19.68
17.43
Labot-Teet--S.3-COT
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
7.18
25.56
8.21
41.94
52.88
45.10
1shot-Vision-C0T
11.38
14.62
11.00
25.47
28.22
25,98
Threshold
0shot
2.98
2.87
2.38
37.16
70.83
48.67
Table 9: Trend anomalies, but no negating trend, and less noticeable speed changes
REC
F1
affi
PRE
REC
F1
Gemini-1.55 -1-Faa
Oshot-Text
0.00
0.00
0.00
4.82
9.69
6.43
Oshot-Text-SO..3
8.07
8.51
7.84
16.98
20.53
18.27
Oehot-Text-SO.3--
2.69
3.02
2.74
5.24
7.76
6.10
Oshot-Text--S0 3--0..3----C5
0,25
0.25
0.25
2.03
3.59
2.54
Oshot-Tezt--00 3-COT-PAP
3.75
3.75
3.75
5.23
6.53
5.65
Oshot-Text--0..33 3-COT-TPP
1.50
1.50
1.50
3.26
4.69
3.76
Oshot-Text--S-33-G3)
0.00
0.00
0.00
2.76
6.33
3.78
Oshot-Text-S0.3-22
0.00
0.00
0.00
3.33
7.14
4.52
Oshot-Text-SO.B5 3-TPD
0.00
0.00
3.79
7.75
5.08
Continued on next page
36

---

Published as a conference paper at ICLR 2025
Table continued from previous page
PRE
REC
F1
affi
affi
affi
PRE
REC
F1
Oshot-Vision
57.50
57.50
57.50
57.50
57.50
57.50
Oshot-Vision-COT
57.50
57.50
$7.50
57.50
57,50
57.50
1shot-Text-SO.23
11.30
12.34
10.77
21.27
24.65
22.52
-shot-Text--O..
12,40
13.31
12.12
23.69
27.45
25,23
ishot-Vision
57.91
57.91
57.89
57.99
58.00
57.99
Ishot-Vision-OOT
16.24
22.09
15.77
25.75
30.56
27.48
GPT-4o-Xini
Oshot-Text
1.25
1.25
1.25
5.49
9.74
6.88
Oshot-Text-S0.3
0.11
0.43
0.17
11.15
21.05
14.56
Shoot-Text-SO.3-000
0.81
0.97
0.85
2.19
3.50
2.63
Oshot-Text--S..33 Sattttt-------T--C-O 3-V
5.25
5.25
5.25
7.55
10.35
8.32
Oshot-Text--O- 3-COTTPPP
6.75
6.75
6.75
7.42
7.75
7.55
0,25
0.25
0.25
0.52
0.77
0.61
0ehot-Text--S.3-C333
0.25
0.25
0.25
3.46
8.72
4.83
Oshot-Text--0.
11.25
11,25
11.25
12,41
12.91
12.59
Oshot-Text--S..3-12
6,50
6,50
6.50
8.67
10.87
9.39
Oshot-Vision
57.50
57.50
57.50
57.50
57.50
57.50
Oshot-Visioon-OT
57.25
57.25
57.25
57.25
57.25
57.25
1shot-Text-SO.3
0.52
1.10
0.65
11.20
21.18
14.61
ishot-Text-SO 3-COT
1.84
2.06
1.91
12.11
20.06
14.94
ishot-Vision
57.50
57.50
57.50
57.50
57.50
57.50
Ishot-Vision-COT
57.00
57.00
57.00
57.00
57.00
57.00
Internvin-76B
Oshot-Text
10.36
16.13
10.94
15.48
21.61
17.51
0
26.61
27.85
25.85
32.78
34.41
33.37
Oshot-Text:-S0.3e 3-CCO
10.67
12.35
10.81
12.60
14.33
13.19
5,55
6.25
5.59
8.85
10.68
9.49
Oshot-Text--O.3--O00 3-C
15.11
15.75
15.19
16.03
16.70
16.26
Oshot-Text--S0
1000--TText--00.300---001- 3-COT-TPD
8.89
9.63
8.98
9.67
10.54
9.96
Oshot-Text-S0.33- 3-CSV
5.09
5.39
5.14
9.83
13.88
11.21
-ooot-Textt-OO3-PPA.
33.80
35.78
34.00
35.62
37.63
36.24
Oshot-Text-SO.3--1
3.99
6.55
4.19
6.64
9.57
7.62
Oshot-Vision
34.58
42.52
35.03
40.75
47.54
43.03
Oshot-Visicn-COT
51.88
51.66
$1.72
$2.85
53.70
53.14
1shot-Textt-S0.3
29.16
30.57
28.69
36.38
37.31
36.77
1shot-Text--S.3--C0
39.77
41.00
39.16
46.03
46.61
46.29
ishot-Vision
37.95
42.16
37.07
44.67
46,94
45.50
IshOt-Vision-OOT
51.37
53.02
51.53
54.02
55.26
54.47
Qwen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
0shot-Text-SO.33
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SS- 3-COT
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SO- 00w--Teext--003-0T--053559 3-COT-CSV
0.00
0.00
0.00
0.00
0.00
0.00
10w--Teext-SO33--O--
0.00
0.00
0.00
0.00
0.00
0.00
Dabot-Textt-SO3-00T--
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-S0..3e
0.00
0.00
0.00
0.00
0.00
0.00
Oahot-Text-SO.33-1
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Vision
4.71
21.94
6.04
13.05
22.47
16.24
Oshot-VVision-00T
21.65
22.75
21.76
22.20
22.76
22.40
1shot-Text-S0.
0.00
0.00
0.00
0.00
0.00
0.00
1shot-Text-SS.3--00 3-COT
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
12.72
28.32
13.09
28.20
38.21
31.73
29.98
31.18
29.79
34.14
35.79
34.78
37

---

Published as a conference paper at ICLR 2025
Table 10: Yahoo S5 dataset
PRE
REC
affi
affi
affi
PRE
REC
F1
Gemini-1. ---Faah
Oshot-Text
0.18
3.16
0.26
16.10
18.50
16.46
Oshot-Text-SO.3
20.21
22.73
20.10
31.04
30.41
29.91
Oshot-Text-S0.3 C-CVV
3.82
3.95
3.82
13.80
10.71
11.60
Oshot-Text-90.33
16.48
16.67
16.51
26.52
23.85
24.53
Oshot-Text-SO.3 3-TPD
14.97
16.62
15.11
28.32
27.09
27.00
Oshot-Vision
30.44
45.17
31.74
54.82
57.25
55.77
1shot-Text-S0.3
20.41
21.93
20.40
31.39
30.66
30.32
ishot-Vision
28.54
31.06
28.56
45.27
43.10
43.64
GPT-4o-Mini
Oshot-Text
2.74
4.04
2.50
15.26
14.68
14,40
Oshot-Text-S0.3
36.02
37.42
36.06
41.38
41.57
41.26
O 3-CSV
18.28
18.48
18.29
27.40
25.31
25.86
Oshot-Text-SO0 3-PAP
25.62
25.70
25.63
29.31
28.48
28.70
Oshot-Text-SO 3-T00.3TPP
4.17
4.65
4.22
11.98
11.00
11.20
Oshot-Vision
50.88
57.84
51.31
71.42
74.07
72.19
-
39.74
41.03
39.73
46.59
45.90
45.76
ishot-Vision
17.22
18.07
17.18
32.81
34.18
32.61
Internvlm-768
Oshot-Text
30.42
37,91
30.54
38.31
42.25
39.53
Oshot-Text-SO.3
46.08
47.28
46.10
48.13
48.47
48.13
Oshot-Text-SO. 3-CSV
6.30
8.49
6.34
16.59
14.92
15.32
Oshot-Text-SO- 3-PAP
53.72
54.52
53.76
55.43
55.64
55.49
0shot-Text-SO.3 3
14.80
17.51
14.86
19.96
21.27
20.29
Oshot-Vision
12.42
29.87
13.52
35.19
40.91
37.41
1shot-Text-S0.3
50.17
52.32
50.21
52.12
53.23
52.46
ishot-Vision
1.84
11.19
2.47
20.62
23.15
20.96
Taolation-Forest
Oshot
2.07
29.26
3.25
16.75
33.18
21.94
Quen
Oshot-Text
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-SO-.
0.00
0.00
0.00
Oshot-Text-SO- 3SCVV
0.00
0.00
0.00
0.00
0.00
0.00
Oshot-Text-S0..3 3-PAP
19.89
19.89
19.89
19.91
19.90
19.91
Oshot-Text-S0 3-TDD
2.18
2.18
2.18
2.19
2.21
2.20
Oshot-Vision
3.10
19.21
3.40
16.19
25.37
19.53
1shot-Text-S0.3
0.00
0.00
0.00
0.00
0.00
0.00
ishot-Vision
3.39
5.68
3.24
18.37
21.61
19.22
Threshold
Oshot
3.99
21.99
5.51
16.40
31.80
21.03
38

---

Published as a conference paper at ICLR 2025
E MATHEMATICAL FORMULATIONS OF HYPOTHESES 2 AND 3
E.1 HYPOTHESIS 2: REPETITION BIAS AND PERIODICITY DETECTION
Let f(t) be time series and T(f) be its tokenized representation in an LLM's vocabulary space V. We define:
1) Perfect periodicity: f(t + P) = f() for some period P >
2) Noisy periodicity: P) f(t) + e(t) where e(t) ~ NN and < ming, f(t')
Note that while noisy periodicity is defined on numerical values, the tokenization process T(:) maps these values to discrete tokens, making perfect periodicity in token space impossible for noisy peri- odic signals.
Let A(f) be the LLM's 's anomaly detection accuracy time series f and Dp be the set of all periodic time series with period P. Given there exists an optimal anomaly detector B, whose accuracy is (J),), that can achieve near-perfect accuracy on both perfect and noisy periodic signals. The hypothesis states:
For any f1, f2 Dr, (ff1) exhibits token-level periodicity and T(f2) does not, then:
E[A(F)) > E[A([2)
while the optimal detector maintains consistent performance:
This formulation suggests that LLMs' performance difference is due to token-level repetition bias rather than the inherent complexity of the anomaly detection task, as a properly designed detector achieve near near-perfect performance on both cases.
E.2 HYPOTHESIS 3: ARITHMETIC ABILITY AND PATTERN RECOGNITION
Let M be an LLM and M' be the same LLM with impaired arithmetic ability.
1) Define arithmetic ability ((M) accuracy basic arithmetic tasks:
((M) Ez,y |K ("What is + y?") + y]]
2) Define reasoning ability p(M) as accuracy on non-arithmetic reasoning tasks:
p(M)= = E [M (q) = correct]
where of logical reasoning questions.
3) Obtain M' by training M on incorrect arithmetic examples while preserving reasoning:
(a(M') < a(M) (p(M') p(M)
4) Hypothesis holds if:
EjED[An > EsED[AM((f)
where 5 is small and dataset is arbitrary. To falsify the hypothesis, we show the difference is negligble reversed for certain datasets.
39


Reviews:
[{"review_id": "Yr9Daa7eWJ", "reviewer": "Reviewer_iE2G", "summary": "This paper studies how Large Language Models (LLMs) reason about anomalous time series. The authors formulate 7 hypothesis, based on recent work at the intersection of time series & LLMs, their own understanding of the problem, and prevailing hypotheses about LLMs, and carefully design experiments using synthetic time series to accept or refute their hypotheses.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The intersection of time series and LLMs is an exciting and active area of research. The authors present a very neat study on LLM capabilities on the anomaly detection problem. \n2. The hypotheses are interesting and most experiments are designed well. \n3. The paper is well-written, and aesthetically appealing, and some findings are interesting. \n4. The authors demonstrate a fairly good understanding of the literature, on time series, anomaly detection and LLMs.", "weaknesses": "1. **Some claims need to be supported or softened:** The authors make some claims in the paper which are not well supported by the experiments and the overall story. (a) *on the question of LLMs understanding time series*:  There are many different notions of understanding in addition to anomaly detection, such as in predictive tasks such as classification [1] and forecasting [2], but also other inductive tasks such as understanding of (granger) causality [3]. The ability to reason about one class of problems, does not imply the ability to reason about other classes of problems, hence LLMs are expected to do well in some tasks, and not in some others. What the authors show evaluate is if LLMs can reason about some kinds of anomalies using synthetic data. In the abstract, the authors mention: \"These insights pave the way for more effective LLM-based approaches in time series analysis\". However, it is unclear how some insights, while interesting, are actionable (e.g., on perceptual reasoning abilities). I must note that some insights are actionable, e.g. on long context bias. (c) The model-specific finding is not new, and is consistent with prior work [4, 5] which has established the need and value of model selection for anomaly detection.  \n2. **On the defining anomaly detection:** Anomaly detection is a tricky problem with a lot of nuances. The authors only study one variant of the problem, i.e. detecting *4 different kinds of anomalies in synthetic time series with labels in zero-shot and few-shot settings*. Meanwhile, there is a large body of work in unsupervised or semi-supervised anomaly detection, which does not rely on explicitly labeled anomalies as training examples.  Finally, the authors state (line 223 onwards) the connections between forecasting and anomaly detection. These are valid. However, (1) only a subset of AD methods rely on forecasting, other methods rely on reconstruction or differences in latent space [4], (2) the authors do not rely on their LLMs ability to \"forecast\", and therefore the ability that they are testing have little to do with extrapolating, as against interpolating based on past and future context. \n3. **Reproducibility**: The authors haven't shared examples from their generated dataset, or the inference and evaluation code. They also have not shared visual examples of their generated time series, which makes it harder to reproduce their work.  \n4. **Hypothesis 5 (\"LLMs’ understanding of anomalies is consistent with human visual perception.\") is too broad and not well supported:** (a) The study by Mueller & Timney, 2016, only exhibits 1 form of human perceptual bias. Disproving it only shows that LLMs do not have this particular bias. (b) It is not immediately clear to me how this bias manifests in anomaly detection. I am not sure how constant speed change differs from acceleration (which causes a change in speed). This hypothesis would greatly benefit from being fully specified, motivated, and examples of datasets shown. \n5. **Hypothesis 2 (LLMs’ repetition bias (Holtzman et al., 2020) corresponds precisely to its ability to identify\nand extrapolate periodic structure in the time series. is unclear:** The hypothesis has 3 key terms, \"identify\", \"extrapolate\" and \"periodic\", but the authors do not vary any of these key terms. The settings that the authors consider have nothing to do with \"extrapolation\" (forecasting), they only consider periodic or periodic time series with noise (which is still periodic), and the paper provides no evidence if or how well LLMs identify periodic structure, and how that corresponds to their findings. I think examples of the generated dataset and the nature of the noise would benefit this section. Also, correlating the ability of models to identify / extrapolate periodic structure with their ability to identity anomalies in periodic and non-periodic time series, would be a much better way to break down this complex claim in my opinion. \n6. **Hypothesis 1 (\"LLMs do not benefit from engaging in step-by-step reasoning about time series data.\") is too broad:** The authors only provide evidence that the models are not capable of chain of thought reasoning. CoT reasoning is only one way to do step-by-step reasoning. The hypothesis should instead be that these models are not helped by engaging is CoT reasoning or some variant of this claim. \n7. **Use of Images:** This is an interesting finding and is consistent with recent work [3]. I wonder if highlighting the location of the anomaly in the image, just like the authors have done it for illustrative purposes would benefit AD performance even more. I suspect it might be true because I am not convince that models understand this statement: `[{\"start\": 171, \"end\": 178}]` especially because the reader has no idea how the time series plots look, if they have an x and y-axis that's marked with sufficient font size and precision. \n8. **Synthetic data:** I understand why the authors use synthetic data, but without knowledge of the difficulty of the synthetic dataset, and how SoTA and simpler methods perform on these datasets. Also from the tables in the Appendix it seems that a simple threshold-based baseline does really well. This raises the question about the limited nature of anomalies, for example [5] can inject many more varieties of anomalies to real-world signals, which provides a much larger, and diverse set of real-world anomaly examples than the considered work. Also the authors cite [6] but do not use their datasets for their experiments. This might be important given that this studies' primary focus is on measuring LLM's ability to detect anomalies. \n\n### Minor\n1. Line 297 -> Hypotheses -> Hypothesize\n2. Line 335, Missing .\n3. Line 532, To -> to\n\n\n### References\n1. Cai, Yifu, et al. \"Jolt: Jointly learned representations of language and time-series.\" Deep Generative Models for Health Workshop NeurIPS 2023. 2023.\n2. Gruver, Nate, et al. \"Large language models are zero-shot time series forecasters.\" Advances in Neural Information Processing Systems 36 (2024).\n3. Cai, Yifu, et al. \"TimeSeriesExam: A time series understanding exam.\" arXiv preprint arXiv:2410.14752 (2024).\n4. Schmidl, Sebastian, Phillip Wenig, and Thorsten Papenbrock. \"Anomaly detection in time series: a comprehensive evaluation.\" Proceedings of the VLDB Endowment 15.9 (2022): 1779-1797.\n5. Goswami, Mononito, et al. \"Unsupervised model selection for time-series anomaly detection.\" arXiv preprint arXiv:2210.01078 (2022).\n6. Wu, Renjie, and Eamonn J. Keogh. \"Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress.\" IEEE transactions on knowledge and data engineering 35.3 (2021): 2421-2429.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UY7dK0pXks", "reviewer": "Reviewer_nLhj", "summary": "With the increasing popularity of LLMs and applications to various tasks, such as forecasting, their performance for anomaly detection remains largely unexplored. This work studies zero-shot and few-shot scenarios of popular LLMs and formulates hypotheses to debunk current norms in this fastly emerging field. The analysis suggests that several beliefs were wrong so this paper offers a useful tool for helping to move the field forward.", "questions": "W1. Unclear focus on LLMs and not pure focus on time-series Foundation Models.\n\nIndeed, the field started by changing language models, assuming their sequential structure could have some benefit for time series as well. I think from the first moment, the time-series community understood how problematic that was.. and the lack of strong experimental results, comparisons with baselines, etc. exacerbated the situation and more and more folks tried to use such LLMs.. however, very fast, the field moved into pure time-series foundation models and abandoned LLM or VLMs for such use. So even though I appreciate the effort, at the same time I think the picture is already much much bigger to just make claims about LLMs which are obviously problematic for this area. I think a much bigger study to include VLMs/LLMs/FM for this area is needed\n\nW2. Missing evaluation measures\n\nThe study focuses on problematic solutions for assessing anomaly detectors. There has been significant recent progress that is omitted [a].\n\n[a] \"Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection.\" Proceedings of the VLDB Endowment 15.11 (2022): 2774-2787.\n\nW3. Missing benchmarks\n\nSimilarly, limited datasets are used when established benchmark exists for this problem [b]\n\n[b] \"TSB-UAD: an end-to-end benchmark suite for univariate time-series anomaly detection.\" Proceedings of the VLDB Endowment 15.8 (2022): 1697-1711.\n\nW4. Missing experimental comparison against baselines to understand the performance of these methods\n\nEven though I understand this might be somewhat out of scope, I think what matters in the end is how they performs vs. SOTA methods. Let's say even if they greatly understand time-series, in the end where they stand in comparison to SOTA techniques. Therefore, an evaluation using such methods ([a,b] should be a good starting point) is needed as well", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. Timely problem to understand the performance of LLMs in this context\nS2. Reasonable hypotheses formulated and tested\nS3. Interesting findings useful for the community", "weaknesses": "W1. Unclear focus on LLMs and not pure focus on time-series Foundation Models.\nW2. Missing evaluation measures\nW3. Missing benchmarks\nW4. Missing experimental comparison against baselines to understand the performance of these methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p5VZNONONa", "reviewer": "Reviewer_CZBV", "summary": "This paper investigates whether MM-LLMs can effectively detect anomalies in time series data,from the perspective of validating 7 hypotheses about LLM performance on the proposed tasks. By conducting controlled experiments, the authors assess LLM capabilities across various anomaly types and input representations (text vs. visual). The findings reveal that LLMs perform better with visualised time series data. Additionally, the study challenges several assumptions about LLMs, including their reliance on arithmetic abilities and repetition biases, which were previously thought to aid in pattern recognition, and the impact of CoT on time series tasks. Finally the authors show that different model architectures can have significantly different performance in anomaly detection, suggesting that previous results using just one or two architectures may not generalise.", "questions": "Q1. For further validation of the arithmetic ability hypothesis, did the authors consider simply comparing different variations of the same model architectures (e.g. by size) and their performance on standard arithmetic benchmarks and correlating it with performance on anomaly detection? It seems that for many of these hypothesis relying on other real-world benchmark proxies could provide additional verification.\n\nQ2. For validating hypothesis 5, the authors claim the test dataset “is too subtle to be visually detected by humans”. How is this validated? \n\nQ3. I did appreciate that the authors stuck by the claim that humans preferred visual anomaly detection and also presented most of their results visually, however this did make it somewhat difficult to interpret the results. Would it be possible to summarise the most important tests for each hypothesis in a single table? I think this could add to the presentation.\n\nNits:\n\nP.6. “We hypotheses…” grammar.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The authors articulate hypotheses proposed by existing literature or propose some based on analyses of current research and then conduct controlled, synthetic, experiments to test these hypotheses.\n\nS2. Visual vs. Textual Representation: this paper extends existing research into the feasibility of visual detection models for time series analysis by comparing multimodal LLMs' performance on visual versus textual time series data, revealing that visual inputs often yield better anomaly detection. Given the applicability of recent multimodal LLMs for a lot of tasks, the analysis on time series anomaly detection is timely.\n\nS3. The study sets up interesting tests for the hypothesis around arithmetic abilities and repetitive biases. The noise injection and artificial arithmetic reasoning drop seem like a well-constructed approach.", "weaknesses": "W1. While the study is insightful, its primary focus on hypothesis testing rather than direct practical application might limit its immediate utility for practitioners looking to deploy LLMs in anomaly detection systems. This can still be helpful for theoretical progress and providing direction for further time-series anomaly detection research, however I think some of the experiments were not defined very strictly or in such a way that they could be built upon for future analysis which hampers the benefit of the work. \n\nW2. The experiments and the hypothesis validation, primarily rely on synthetic datasets for testing. Introducing real-world datasets, such as the 18 datasets found in the TSB-UAD univariate anomaly detection benchmark (Paparrizos et. al. 2022) can help with further validating these hypotheses in real-world settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qaQQ8I0r36", "reviewer": "Reviewer_M4GK", "summary": "The paper explores the potential of using Large Language Models (LLMs) for analyzing anomalies in time-series data, focusing on the insights in terms of reasoning that can be gained from detecting them. To do this, the paper proposes a set of hypotheses and scenarios to assess the effectiveness of LLMs when processing time-series anomalies, providing an overview of their strengths and limitations, as well as the possible parallels with human reasoning.", "questions": "- What are the prevailing beliefs about LLMs when processing time series? Where had they come from?\n- How to impair a LLM for evaluating the third hypothesis?\n- What are the foundations and support to evaluate a LLM from the perspective of cognitive science?\n\n\nPost rebuttal comments:\nThank you for the authors' rebuttals, which address many of the questions. I thus increase the score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- It introduces an attractive problem related to assessing how LLMs can detect time-series anomalies and explores whether there are similarities in how humans approach this task.\n- It emphasizes the importance of analyzing time-series data from different perspectives, such as using visual features in a multi-modal setting, which may yield better results than relying solely on text input.\n- The paper is generally well-written and relatively easy to follow.", "weaknesses": "- The contributions of the paper are not explicitly defined, making the goal of the paper ambiguous. For example, it is unclear whether the goal was to provide a strategy for assessing how LLMs deal with anomaly detection (via the hypotheses), or to simply present the results of evaluating LLMs under specific settings (as highlighted in the abstract and introduction).\n- The paper lacks a clearly defined and replicable method. The reasoning for establishing the hypotheses needs stronger support from previous work, and the evaluation design should clearly outline how each hypothesis is assessed and the logic behind it.\n- The experiments lack essential details, such as the differences between prompt variants, and the results are inconsistent across all evaluated settings (e.g., combining zero-shot and few-shot in the same figures). This raises doubts about whether the highlighted findings are supported by the experimental results.\n- The paper contains claims like \"prevailing beliefs about LLMs\" or \"expected patterns\" that are not properly discussed. For instance, the Related Work section should explain what these prevailing beliefs are, and the Definitions section should clearly define what is considered an expected pattern.\n- The definitions of time-series are not formalized and are ambiguous. Many of them are not used, leaving it unclear as to why they were included. Additionally, there are inconsistencies between them (see details below).\n\n# Detailed Comments\n## Introduction\n- There is not clear justification of focusing the evaluation solely in the F1 score\n- The content is mostly focused on the experimental findings, so it is not clear what is the paper contribution\n## Related Work\n- It is mostly a literature review, with no discussion over the main topic of reasoning over times-series anomalies\n- There is no discussion about the continuously mentioned controversy about time-series analytics in LLMs. A paper close to that topic, Jin et al. (2024a), was not discussed in this section.\n- The claim of lack for visual M-LLM on time series is not accurate. For instance, Jin et al. (2024a) already included references to that approach\n\n## Definitions\n- Definition of anomaly is very ambiguous. What is an expected pattern? \n- There is no definition of the multiple thresholds\n- Equation 1 needs a better explanation. What is G? A forecaster? If G generates x_t, so where the external x_t comes from?\n- The anomaly detection algorithm is not clear. For example, it is a function that generates anomaly scores for each x_t...\n- Interval-based anomalies definition seems incorrect. Why i /in {1,...,k} for a given y_t? It is probably Y, although it seems an unused definition\n- If the interest in the data-point level anomalies, why using interval-based\n- The few-shot definition is inconsistent with previous definitions and g is undefined. The output is probably referring to Y, instead of the actual s_1 or y_1\n- The definition of zero-shot needs more details, contextualized to the prompt setting\n\n## Time-Series Categories\n- Shuffled time series? It is contractionary to the concept of series and time-dependency\n- Trend example is not that clear. Which gradient?\n- Frequency example is not clear, there is no significant shift in Fig. 1(b)\n- What is an expected pattern? It is a very ambiguous term used frequently\n\n## Time-Series Forecasting\n- There are not clear details about the implication of extrapolation: \"Therefore, both time series forecasting, and anomaly detection rely heavily on extrapolation.\"\n- Connection with of time series and tokens needs to be clearly specified\n\n## Understanding LLM \n- \"To demystify LLMs’ anomaly detection capabilities\" What are the myths? \n- There are many assumptions underlying the description of a LLM from a cognitive science perspective. The argument needs to be carefully elaborated because it is implying that a LLM behaves like a human brain, which is a strong statement. Then, it is not clear how the evaluation between reflexible and reflective modes is conducted\n- It is not clear how the second hypothesis may be evaluated using noisy time series\n- There is necessary to include some support for proposing the fourth and fifth hypotheses\n- Fifth hypothesis evaluation is not as consistent with the approach of the referred paper\n- In general, the hypothesis is not aligned with the paper goal regarding anomalies and are mostly focused on looking for some parallels with human reasoning\n\n## Prompting\n- The prompts seem very important, but there is no information about how they are designed. For instance, it is not clear what the variants are doing \n\n## Experiments\n- The process for generating of data sets does not provide design details, even in the Appendix. It should disclose more technical details, and show some examples, since it is over this data that the hypotheses will be evaluated\n- Metrics depend on a threshold, but there is no mention of it\n- It is not clear what Figures 2,4,5,6,7,8 are showing. It is a comparison between prompts that may or may not be comparable and there is not clear point of reference\n- There is not consistency between the selected prompts, so the results are hardly comparable between anomaly types or LLM engines. Also combining zero-shot and few-shot prompts will introduce unnecessary noise to the evaluation \n- In general, the experiments are not providing enough support for evaluating the hypotheses. For example, with the third hypothesis, how the arithmetic capability is removed? The experiment needs to include all this type of details\n- The visual-text experiment, which is a major claim for the paper, need to explore with more details, how it was conducted, how the prompts were designed to achieve a fair comparison \n- Why the frequency anomalies failed? It needs a better explanation, perhaps showing an example of the actual task that the LLM is doing and the chart with the time-series", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fMP6HEbAAU", "reviewer": "Reviewer_hKqp", "summary": "This study investigates large language models' (LLMs) understanding of time-series anomalies by proposing and testing specific hypotheses.", "questions": "See weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The study incorporates a wide range of prompting and representation techniques.", "weaknesses": "Writing\n1. Some redundancy is present; for instance, the discussion in Section 3.3 lacks direct relevance to problem motivation and is only tangentially related.\n\nExperiment/Study\n1. Some hypotheses are tested without adequate scientific rigor. For example, in the discussion of Hypothesis (7) on architecture bias, several variables—such as model architecture (e.g., QWEN vs. GPT-4o-mini), model size, and the content and size of the pretraining corpus—are not controlled. This lack of control makes it difficult to attribute performance differences specifically to architecture, as they may instead stem from variations in model size or pretraining data. \n\n2. No mention of using statistical tests for hypothesis. For instance, in the \"retrained Hypothesis 1 on CoT reasoning,\" it is unclear if the claimed result is from statistical testing with a null hypothesis of no difference or heuristic. It would be made the study scientific and convincing by reporting some results from, say t-test on accuracy difference through bootstrapping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)