Plots unlock time-series understanding in multimodal models

## Abstract

Abstract While multimodal foundation models can now natively work with data beyond text, they remain underutilized in analyzing the considerable amounts of multi-dimensional time-series data in fields like healthcare, finance, and social sciences, representing a missed opportunity for richer, data-driven insights. This paper proposes a simple but effective method that leverages the existing vision encoders of these models to “see” time-series data via plots, avoiding the need for additional, potentially costly, model training. Our empirical evaluations show that this approach outperforms providing the raw time-series data as text, with the additional benefit that visual time-series representations demonstrate up to a 90% reduction in model API costs. We validate our hypothesis through synthetic data tasks of increasing complexity, progressing from simple functional form identification on clean data, to extracting trends from noisy scatter plots. To demonstrate generalizability from synthetic tasks with clear reasoning steps to more complex, real-world scenarios, we apply our approach to consumer health tasks – specifically fall detection, activity recognition, and readiness assessment – which involve heterogeneous, noisy data and multi-step reasoning. The overall success in plot performance over text performance (up to an 120% performance increase on zero-shot synthetic tasks, and up to 150% performance increase on real-world tasks), across both GPT and Gemini model families, highlights our approach’s potential for making the best use of the native capabilities of foundation models.

## 1 Introduction

Multimodal models like GPT4 (Achiam et al., ) and Gemini (Gemini Team et al., ) are trained to understand visual information natively. However, they are not specifically trained to understand time-series data – in particular, the tokenizers for large language models (LLMs) are not well-suited for representing large sequences of floating point numbers (Spathis & Kawsar, ). This mirrors the human approach; we cannot easily make sense of a long array of floating point numbers - instead our first instinct is often to visualize the data through plotting, followed by extracting insights through statistical analysis.

We investigate the hypothesis that multimodal models understand time-series data better through their vision encoders than through the textual representation of the sequences using synthetic and real-world data experiments. Our synthetic data experiments allow us to closely control the difficulty of tasks through the addition of noise and by changing the number of points in each function. We also use a mix of tasks that require a differing number of reasoning steps, as well as different kinds of reasoning, to get a correct answer.

Fall detection and activity recognition are both real-world tasks that make use of inertial measurement units (IMUs) from mobile phones or wearable devices. IMUs are 6-dimensional waveforms consisting of 3 axes of acceleration data and 3 axes of angular velocity data. The fall detection task consists of classifying an IMU waveform segment into one of three classes: Fall, Active Daily Living (ADL) or Near Fall (a hard negative class). The activity recognition task consists of classifying a waveform segment into one of five classes: Sitting, Standing, Walking, Cycling or Stairs.

By contrast, the readiness assessment task is a binary classification of 28 days of training load data from a single user into a state of undertraining or overtraining. Because of the tabular nature of the data, the plot version is presented as a bar plot. This is not the ideal setting for our method - we believe it’s best used when the amount of data exceeds what’s reasonably presentable in a text table.

Our findings show that when using our plot-based approach multimodal models perform much better on tasks where the result is dependent on understanding the overall trend. We find specific examples of this when identifying the functional form, the number of clusters, the correlation between two functions, and on the real-world pattern-recognition tasks of activity recognition and fall detection. For example, GPT4o using plots on the functional form identification task shows a performance improvement of 122% over using the text representation. On other tasks that require more advanced reasoning such as multi-step or connecting trend shapes with sequence magnitudes (e.g. identifying derivatives), and on tasks with tabular data (e.g. readiness assessment), the performance is equivalent. However, there is a substantial cost difference between vision and text prompts, which is particularly pronounced on very long-context tasks, as the same information in a long sequence that requires many (10,000’s to 100,000’s) text tokens can be represented in one plot with many fewer (100’s to 1000’s) vision tokens. While vision tokens are more expensive than text tokens, the difference in unit cost is much lower than the orders of magnitude difference in overall prompt length, so that the total cost is still much lower using the vision approach. This difference is especially relevant on tasks where extensive few-shots are required to achieve good performance, and optimizing token efficiency translates to significant resource savings. Not only does this plot-based approach achieve better performance while being more efficient, it is also completely generalizable across any task that involves reasoning about a long, complex time-series as it requires zero additional model training.

## 2 Related Work

We use the term “time-series understanding” in this paper to distinguish from time-series forecasting. Time-series forecasting predicts future data points based on points seen so far, whereas we are primarily interested in the setting where we connect the time-series data to a multimodal model for further analysis. In particular, we want to show that multimodal models can reason about overall trends, the relationship between multiple time-series, overall clustering of data, and other time-series understanding tasks. For a closer look at the literature, the survey paper by Zhang et al. () tracks a wide range of time-series understanding and forecasting work.

We deliberately avoid using multimodal models to generate any of the questions themselves (e.g. SPIQA (Pramanick et al., ), CharXiv (Wang et al., )) as this can introduce biases during evaluation that are hard to account for (e.g. favoring their own output as in (Panickssery et al., )).

LLMTime (Gruver et al., ) shows that with careful tokenization, text-only LLMs can perform well at forecasting tasks. We perform ablations based on their methods and select the best tokenizations accordingly for our text baselines.

There are several existing approaches that train time-series encoders for specific tasks or domains. For example, Chan et al. () train a domain-specific encoder for multimodal medical time-series analysis. Similarly, Cosentino et al. () train an encoder for the sleep data in their Digital Well-being task. In this paper, we are not claiming that our approach would outperform a task-specific encoder on a specific task – we claim that one can achieve much better performance from a foundation model by exploiting its native multimodal capabilities compared to using only text. We want to show that simply plotting the data is at the very least an easy first step, and might be a helpful approach when training a task-specific encoder from scratch may not be feasible due to the requirements on having additional paired data, compute and expertise.

Closely related to our work are those methods that use vision-embedding models like Contrastive Language–Image Pre-training (CLIP) by Radford et al. (). Wimmer & Rekabsaz () use CLIP to embed plots of financial time-series data, from which features are extracted for use by downstream classifiers. Since we use the vision encoders of the multimodal foundation models directly, there is no need for further feature extraction or downstream classifiers in our approach. IMU2CLIP (Moon et al., ) and ImageBind (Girdhar et al., ) use video and image data paired with waveforms to learn joint embeddings. Both these works rely on existing paired waveform and video data to “bind” the modalities together, whereas we can simply plot the waveforms and use the existing multimodal vision encoder to derive our time-series embeddings.

Perhaps an inversion of our approach, DePlot (Liu et al., ) translates plots to tables and operates on the tabular data. This approach may be sound for discrete data where the number of points remains small – cases where a human would be expected to understand a table of data well.

Finally, Rahmanzadehgervi et al. () suggest that multimodal models are unable to reason about visual inputs, although some follow-up work by Corin () indicates that prompt engineering can fix losses. In any case, our results do not necessarily contradict this - for many of our tasks humans may be able to get perfect scores, and indeed the multimodal models do not. Regardless, our main claim that plots are better suited than text as input to a multimodal model for time-series understanding holds true.

## 3 Methodology

We evaluate our visual prompting method on both synthetic data and real-world use-cases. Synthetic data allows us to control the difficulty of the task by adding noise and altering the number of data points, and to investigate specific kinds of reasoning in isolation. We chose the synthetic tasks to align with the different steps of reasoning we hypothesize are required for the representative real-world use cases we test on. These include understanding the local and global longitudinal signatures (trend and magnitude) of a time-series, and potentially comparing it with several other time-series (as in the case of multidimensional sensing). We summarize in Section 4.1 the different tasks in our experiments, along with the type of reasoning we are probing.

### 3.1 Structured Prompting

We used the open-source structured prompting library Langfun (Peng, ) for all tasks in this paper, with the exception of the Readiness task (Section 4.2) which is processed in a privacy-preserving sandbox environment. The prompts and Langfun code snippets for all tasks (except Readiness) are provided in Supplementary Information S.1.4 for reproducibility. The structured prompting approach in Langfun allows us to use target schemas for outputs, though we do not use the controlled generation feature (Gemini models) or structured output (GPT4o models), simply relying on the native formatting of the model to the correct schema.

### 3.2 Models

We tested all synthetic data tasks on two frontier models: Gemini Pro 1.5 (gemini-pro-001) and GPT4o (gpt4o-2024-08-06) and two smaller models Gemini Flash 1.5 (gemini-flash-001) and GPT4o-mini (gpt4o-mini-2024-07-18).
We use a temperature of 0.1 for all our experiments, Supplementary Tables S33-S35 includes our ablations on temperature. All other sampling parameters remain at API defaults.

### 3.3 Floating Point Representation

In order to find the best textual representations, we ran ablations (Supplementary Information S.1.3) inspired by LLMTime (Gruver et al., ) on which floating point precision (2, 4, 8, 16) and separator (space or comma and space) to use. We also tested the scaling approach suggested by LLMTime. We found that the lower precision led to better performance. On synthetic tasks, the best performing separator differs per model, on Gemini we make use of the space separator whereas on the GPT4o family we use comma and space. On real-world tasks we make use of the space separator for all models as we did not observe a difference in performance on these tasks and the space separator uses fewer tokens.

### 3.4 Statistical Methods

For our aggregate results, we aggregate individual model responses to an overall performance quality metric (accuracy or mean absolute error (MAE)) over the task dimensions as described in Supplemental Section S.1.1.1. This produces multiple points from which we extract a distribution presented as a box-plot where the central line is the median, the edges of the boxes are the inter-quartile range (IQR), the whisker lengths extend to 1.5 times the IQR and outliers are presented as individual points. For our more detailed plots in Supplementary Information S.1.2 we show 95% confidence intervals constructed from 1.96 times the standard error of the mean of the metric.

For real-world tasks, since we don’t have the ability to regenerate the same problem, we instead make use of bootstrapping (with 1,000 replicates) to produce distributions of the macro-averaged $F_{1}$ scores from which we construct similar box-plots as for the synthetic tasks. Note that the distributions plotted in the real-world box-plots are thus expected to be tighter than the synthetic task plots, as they don’t reflect independent replicates.

In Table 2, we present the median and IQRs of the differences between plot and text performances, with the difference taken such that a positive value always means the plot method performs better than the text method. For synthetic tasks, the median and IQR are calculated by directly creating the distribution of differences between the plot and text performances of different instances of the experiment, while for real-world tasks we create a distribution of differences by randomly sampling 1,000 random pairs of the bootstrapped metric distributions described immediately earlier.

For synthetic tasks only, we test for significant differences between the plot and text performances with a two-sided Wilcoxon signed-rank test (Wilcoxon, ). We apply a Bonferroni (Bland & Altman, ) correction for multiple comparisons within a task block. We could not apply the same hypothesis testing framework to the real-world tasks as the performance distributions were bootstrapped and thus not independent, violating the assumptions of the Wilcoxon test.

## 4 Experiments and Results

In this section we provide a summary of each task and the differences in plot- versus text-based model performances.
Table 1 summarizes the tasks and the reasoning required for each, and Table 2 summarizes the model performances.
More details about each task can be found in Supplementary Section S.1.1.

**Table 1: Summary of the eight tasks we study in this paper, including the type of reasoning each requires and the scale of the number of points in the time-series.**
| Type of data | Task | Reasoning | Num points |
| --- | --- | --- | --- |
| Synthetic | Functional form id. | Understanding of one overall trend | 10’s - 1,000’s |
| Correlation of two lines | Understanding of two overall trends | 10’s - 1,000’s |  |
| 2D cluster counting | Understanding and counting $N$ overall trends | 10’s - 1,000’s |  |
| Derivative id. | Multi-step reasoning connecting two overall trends | 10’s - 1,000’s |  |
| Quadratic derivative id. | Multi-step reasoning connecting two trends and magnitudes | 10’s - 1,000’s |  |
| Real-world | Fall detection from IMU data | Classify a pattern based on local spikes in multiple signals | 10,000’s |
| Activity recognition from IMU data | Classify a pattern based on global trends in multiple signals | 10,000’s |  |
| Readiness from wearable measures of training intensity | Compare a local trend with a global trend | 10’s |  |

**Table 2: Summary of results across all tasks. Rows are different tasks, potentially with different numbers of few-shot examples, and columns are the gains in using plots over texts for the specified models for the specified metric (a positive number always means plots are better). Cell entries contain the median and inter-quartile range (IQR) of the metrics as described in Section 3.4. Bold values are a visual clue that the plot method performs better than the text as judged by a positive median difference between the two performances. Stars in the synthetic tasks indicate statistically significant differences between the plot and text metrics at 95% confidence corrected for multiple comparisons; we could not perform the same hypothesis testing on the real-world tasks due to non-independence of bootstrapped samples (see Section 3.4 for more information). See Supplementary Table S1 for relative differences between plot and task approaches.**
| Task<br>(Metric) | Shots | GPT4o-mini | Gemini Flash 1.5 | GPT4o | Gemini Pro 1.5 |
| --- | --- | --- | --- | --- | --- |
| Functional form id.<br>(Accuracy) | 0 | 0.32<br>(0.18, 0.41)* | 0.22<br>(0.11, 0.40)* | 0.46<br>(0.31, 0.52)* | 0.04<br>(-0.08, 0.25) |
| Correlation of two lines (Accuracy) | 0 | 0.33<br>(0.17, 0.33)* | 0.25<br>(0.17, 0.33)* | 0.17<br>(0.00, 0.17)* | 0.33<br>(0.17, 0.50)* |
| 2D cluster counting<br>(MAE) | 0 | 1.02<br>(0.53, 1.09)* | 1.67<br>(1.49, 1.80)* | 2.29<br>(1.84, 2.44)* | 1.82<br>(1.36, 2.06)* |
| Derivative id.<br>(Accuracy) | 0 | 0.16<br>(0.12, 0.24)* | 0.08<br>(-0.04, 0.20) | 0.00<br>(-0.18, 0.12) | -0.02<br>(-0.16, 0.08) |
| Quadratic derivative id.<br>(Accuracy) | 0 | 0.27<br>(0.23, 0.38)* | 0.15<br>(0.02, 0.30)* | -0.17<br>(-0.30, -0.10)* | 0.17<br>(-0.01, 0.24) |
| 1 | 0.12<br>(0.07, 0.24)* | 0.23<br>(0.12, 0.33)* | 0.05<br>(-0.03, 0.14) | 0.25<br>(0.12, 0.40)* |  |
| 2 | 0.20<br>(0.17, 0.31)* | 0.23<br>(0.10, 0.37)* | 0.07<br>(-0.11, 0.23) | 0.30<br>(0.26, 0.35)* |  |
| 3 | 0.17<br>(0.12, 0.33)* | 0.32<br>(0.13, 0.34)* | 0.10<br>(-0.07, 0.17) | 0.28<br>(0.19, 0.43)* |  |
| Fall detection<br>($F_{1}$ score) | 1 | 0.03<br>(0.02, 0.05) | 0.11<br>(0.09, 0.12) | 0.32<br>(0.31, 0.34) | 0.13<br>(0.10, 0.15) |
| 10 | 0.21<br>(0.19, 0.22) | 0.17<br>(0.15, 0.19) | 0.50<br>(0.49, 0.52) | 0.40<br>(0.38, 0.42) |  |
| Activity detection<br>($F_{1}$ score) | 1 | 0.09<br>(0.07, 0.11) | 0.12<br>(0.10, 0.14) | 0.03<br>(0.01, 0.06) | 0.20<br>(0.18, 0.22) |
| 5 | 0.11<br>(0.09, 0.13) | 0.23<br>(0.21, 0.25) | 0.18<br>(0.15, 0.20) | 0.23<br>(0.21, 0.25) |  |
| Readiness<br>($F_{1}$ score) | 0 | – | -0.08<br>(-0.11, -0.06) | – | 0.07<br>(0.05, 0.09) |

### 4.1 Synthetic Data Tasks

Figure 1 summarizes all the zero-shot versions of the synthetic data tasks showing that plot-based methods outperform the text-based methods across GPT and Gemini model families, with few exceptions.

Figure: (a) Functional form id.
Refer to caption: x1.png

Functional form identification (id.)
This is the simplest task that requires only identifying one overall trend and correctly classifying it into one of five functional tasks (linear, quadratic, cubic, exponential or periodic).

Correlation of two lines
This task now requires understanding the trends in two lines and comparing them against each other to correct identify whether the lines are positively or negatively correlated.

2D cluster counting
In this task, the model needs to correctly identify the $N$ number of clusters present in a set of points.

Derivative identification
This is a harder task: the model must now identify the correct first derivative (out of four choices) of the function provided in the question. The function and choices are presented as either plots or text. We pass various known functions to the model alongside four synthetic first derivatives, each corresponding to different functional classes, and ask it to identify which of the multiple choices corresponds to the true derivative.

Quadratic derivative identification
As a hard variant of derivative identification, the model must now identify the correct linear function (out of four choices) that corresponds to the quadratic function in the question. Here the model must pay attention to both the sign and magnitude of the slope.

In order to investigate the quadratic derivative task further, we also ran experiments providing few-shot examples with reasoning traces with results shown in Figure 2. Here we find that GPT4o text zero-shot remains an outlier in its strong performance, but for the other models plot outperforms text, with few shots improving performance in the Gemini family for both plot and text, but reducing performance in the GPT family of models for both plot and text.

Figure: Figure 2: Few-shot quadratic derivative identification
Refer to caption: extracted/5893188/figures/20240924141055_Quadratic_Derivative_Identification_FewShot_Summary_Boxplot.png

### 4.2 Real-world Tasks

Fall detection from inertial measurement units (IMUs)

Figure: Figure 3: Results of fall detection task for all models, for 1, 3, 5 and 10 few-shots. We also report that the best performing plot models on 10-shot have (sensitivity, specificity) as follows: Gemini Pro 1.5 - (0.84, 0.95) and GPT4o - (0.92, 0.81). The state-of-the-art result reported by Aziz et al. () on this dataset is a purpose-built support vector machine (SVM) trained on 50% of the data which achieves (0.96, 0.96).
Refer to caption: extracted/5893188/figures/20240924141848_Fall_Detection_Summary_Boxplot.png

An IMU segment is a 6D-vector composed of 3-axes accelerometer signals and 3-axes gyroscope signals. The first real-world task we evaluate is to classify whether a 15-second IMU segment recorded at 128hz contains a fall, a “near” fall or showed “active daily living” (ADL). The dataset used in the open-source IMU Fall Detection Dataset (IMUFD, Aziz et al. ()).

Few-shot fall detection is a pattern-recognition task - typically a fall shows up on the IMU as a big spike in magnitude on multiple axes.
What makes the task hard is the inclusion of the hard negative class of “Near” falls, where the participants of the study pretend to trip but recover before actually falling, creating similarly large changes in magnitude on the IMU.

Activity recognition from IMUs

Figure: Figure 4: Results of activity detection task for all models, for 1, 3, 5 and 10 (where context length allowed) few-shots. The state-of-the-art result reported by Kumar & Selvam () is an average $F_{1}$ score evaluating their deep learning activity recognition model on HHAR.
Refer to caption: extracted/5893188/figures/20240926085225_Activity_Recognition_Summary_Boxplot.png

A further real-world IMU task we evaluate is to classify whether a 15-second IMU segment is one of five activity classes: “sit”, “stand”, “stairs”, “walk” or “bike”. The dataset used in the open-source Heterogeneity Human Activity Recognition dataset (HHAR, Stisen et al. ()). As with Fall Detection we test performance with 1, 3, 5 and 10 few-shot examples, with 383 samples per model and number of few-shots. For this task the text representation of the 10 few-shot examples exceeded the GPT4o and GPT4o-mini 128k context windows, so these results are only shown for the Gemini models.

Activity recognition requires evaluating the entire IMU segment and correlating signals between different axes and sensors to determine the likely activity, as the noisy signals may only subtly change between “sit” and “stand” or “walk” and “stairs”. The HHAR dataset was deliberately collected to be heterogeneous containing data collected from four different types of smartphone and two different types of smartwatch. The classes in the dataset aren’t balanced so performance is reported using an $F_{1}$ score.

Figure: Figure 5: Results of readiness task for Gemini models only (as the dataset cannot be sent to other models).
Refer to caption: x6.png

Readiness

Estimating fitness readiness for a workout is a multicomponent task that involves assessment of health metrics, sleep, training load, and subjective feedback (Cosentino et al., ). Among those, training load analysis can be evaluated quantitatively and involves plot interpretation. Therefore, we framed the task as a binary classification problem (training load trending upwards or downwards) and use the calculated acute-chronic workload ratio (ACWR) to obtain ground truth labels.

ACWR is a ratio of acute training load (total training impulse, or TRIMP, over the past 7 days) divided by chronic training load (28-day average of acute load). An ACWR equal to 1.0 means that the user has exercised at the same intensity continuously over the past week compared to the month, below 1.0 means that they are trending downward, and above 1.0 means they are trending upwards. Precise ACWR calculation involves multiple mathematical operations, so we assess the model’s ability to understand the trend from monthly TRIMP values without explicit calculation.

We use training load data from 350 fitness case studies (Cosentino et al., ) and present it as tables or TRIMP bar plots. Each case study contains data from 30 consecutive days. We use a simplified version of the textual prompt and visualization from Cosentino et al. () and do not split TRIMP in different heart rate zones. We tested Gemini 1.5 Pro and Gemini 1.5 Flash for both plot and text approaches as zero-shot tasks. Since this task involved analyzing just 30 data points we did not expect the plot prompt to excel here. Interestingly, models of different sizes showed opposite trends: Gemini 1.5 Pro had a slight increase in performance when using plots, while Gemini 1.5 Flash had a slight increase in performance when using text, though the magnitudes of the gains were small.

### 4.3 Ablations

We considered a variety of text and plot ablations to confirm if there were any large gains. All ablations were performed on Gemini Pro 1.5. We used the function identification task to test for any performance differences; details and results are reported in Section S.1.3.

### 4.4 Cost

Using plots for time-series data can often be more cost-efficient and token-efficient. Token efficiency matters when your context is large and the context-window is limited; for example we needed to downsample our raw signals to fit them into the 128k context window for GPT4o(-mini), particularly with the large few-shot experiments (Section 4.2).
For example, when using the Gemini API (Google, ), images account for 258 tokens if both dimensions are less than 384 x 384 pixels, after which 4 additional crops are added for a total of 1290 tokens. Text tasks can easily be 10x larger (e.g. 10-shot activity recognition Section 3) requiring more than the entire 128k context available.

Plot experiments also end up being cheaper. As an example, for our most expensive experiment on few-shot activity recognition, we can estimate the input token cost of our 5-shot experiment for both plot and text on GPT4o (OpenAI, ).
The text version of this task required nearly 128k text tokens (costing $0.32 per 5-shot question); by contrast, the plot version required 50 images (costing $0.032), a 10x difference in overall costs for input tokens.

## 5 Conclusions and Future Work

Our key finding is that engaging the vision encoder of a multimodal foundation model through the use of plot representations leads to significant performance and efficiency gains on time-series understanding tasks, compared with relying on the text encoder. By processing data visually instead of textually, these models can better capture temporal patterns and relationships. We established our results on synthetic data with well-controlled characteristics and reasoning types, and also showed that this approach holds on real, noisy and complex tasks related to making sense of consumer health signals.

The method presented here is powerful in its simplicity and generalizability – we believe that it is particularly useful when the following conditions are met:

- •
You want to use an off-the-shelf multimodal model to interpret your time-series data.
- •
Your use-case is not restricted to a specific task or modality, and generalizability across tasks is more important than accuracy on a single task. We showed that plots act as a generalizable time-series encoder across many tasks, even though they may not be better than a task-specific encoder trained for one task. Training task-specific encoders for multimodal models can be limited by availability of paired training data, compute and expertise.
- •
You don’t want to downsample your data. In many cases the textual representation of real time-series outstrips the maximum context length, and so the plot-based approach is the only way to present the data without downsampling.

Our focus in this work is specific to time-series understanding (i.e., reasoning about known data). Forecasting is also important to time-series analysis and in the future we suspect that leveraging the vision components of multimodal models might yield positive results in this area too.

All plots in this work were generated by human-written code in order to avoid any bias. Looking forward, in real applications we anticipate that plotting could be part of a tool-use framework, where the model is prompted to choose how and when to plot the data, after which it uses the plot representation it created.

## 6 Reproducibility Statement

Our evaluations are run on publicly available models that have publicly available APIs. The exact model versions used are detailed in Section 3.2.

Our structured prompting methods are detailed in 3.1 and we include the actual prompts and target dataclasses used in Supplementary Section S.1.4.

All the data for the synthetic tasks is by definition synthesized and the detailed task descriptions in Supplementary Section S.1.1.1 provide the necessary details to recreate these synthetic datasets.

The IMU Fall Detection Dataset (IMUFD, Aziz et al. ()) and Heterogeneity Human Activity Recognition dataset (HHAR, Stisen et al. ()) used respectively for the fall detection and activity recognition tasks are both publicly available. Pre- and post-processing steps are detailed in Supplementary Section S.1.1.2.

The dataset for the Readiness task is not currently publicly available. However the task details in Section 4.2 would enable reproduction of the results with access to a comparable dataset.

## Acknowledgments

We would like to thank Sebastien Baur for early discussions on the idea, and Diego Ardila and Yun Liu for helping review the paper.

## Appendix S.1 Supplementary Information

### S.1.1 Detailed task descriptions

In this section we provide further details of each task implementation to assist with reproducibility of results.

#### S.1.1.1 Synthetic tasks

Functional form identification

- •
We generate $(\bm{x},\bm{y})$ series of linear, quadratic, cubic, exponential and periodic functions with variable number of points and noise (injected into the function domains) over the range $\bm{x}\in[-10,10]$.
- •
We perform five repeats across different numbers of points (50, 500, 1000 and 2500) and noise levels (0.0, 0.5, 1.0, 2.0 and 5.0), giving 500 samples per model across number of points, noise level, function type and random replica dimensions.
- •
These results are then passed either as a stringified series of $\bm{x}$ and $\bm{y}$ vectors (“text” task), or as a matplotlib figure (the “plot” task) to the model.
- •
This task only requires that the model is able to understand and label the global longitudinal trend of the function, without overly needing to reason about magnitudes.

Figure: Supplementary Figure S1: Plots used for functional form identification task examples chosen at random representing at various noise levels (from left to right): cubic, exponential, linear, periodic, quadratic.
Refer to caption: extracted/5893188/figures/20240927142556_Supplementary_Examples_Functional_Form_Identification.png

Correlation of two lines

- •
We generate $(\bm{x},\bm{y_{1}})$ and $(\bm{x},\bm{y_{2}})$ series that represent linear functions $y=m\cdot x$ with variable pairs of slopes $(m_{1},m_{2})\in\{(1,2),(-1,1),(5,-1),(-2,-5),(-3,2),(2,3)\}$.
- •
We perform trials across different numbers of points (50, 500, 1000 and 2500) and noise levels (0, 0.25, 0.5, 1, 1.5, 2.0, 3.0 and 5.0) over the range $\bm{x}\in[-10,10]$, with 192 samples per model across number of points, noise level and random replica dimensions.
- •
These results are then passed either as a stringified series of $\bm{x}$, $\bm{y_{1}}$ and $\bm{y_{2}}$ vectors (“text” task), or as a matplotlib figure (the “plot” task) to the model. The model is then asked to classify whether the two lines $\bm{y_{1}}(\bm{x})$ and $\bm{y_{2}}(\bm{x})$ are positively or negatively correlated.
- •
This task requires first understanding two global trends, and then comparing them with each other.

Figure: Supplementary Figure S2: Plots used for correlation of two lines task examples chosen at random representing positive and negative correlations at various noise levels.
Refer to caption: extracted/5893188/figures/20240927142643_Supplementary_Examples_Correlation_of_two_lines.png

2D cluster counting

- •
We generate a series of points corresponding to $N$ distinct clusters, parameterised by the standard deviation from the cluster center which also controls the difficulty of the task.
- •
The cluster centers are chosen randomly, and we enforce a minimum distance between clusters.
- •
We perform five repeats across different levels of standard deviation (0.025, 0.05 and 0.075), different levels of number of points per clusters (5, 50 and 100) and with the number of clusters from 1 to 9, giving 405 samples per model across standard deviation, number of clusters, number of points per clusters and random replica dimensions.
- •
Extending the correlation task, this task now requires that the model is able to simultaneously identify and keep separate track of $N$ different patterns.

Figure: Supplementary Figure S3: Plots used for 2D cluster counting task examples chosen at random representing varying cluster parameterizations.
Refer to caption: extracted/5893188/figures/20240927142648_Supplementary_Examples_2D_Cluster_Counting.png

Derivative identification

- •
We generate $(\bm{x},\bm{y})$ series of linear, quadratic, cubic, exponential and periodic functions and their derivatives $\bm{y}^{\prime}(\bm{x})$ over the range $\bm{x}\in[-10,10]$.
- •
We perform five repeats across different numbers of points (50, 500, 1000 and 2000) and noise levels (0.0, 0.5, 1.0, 2.0 and 5.0) giving 500 samples per model across number of points, noise level, function type and random replica dimensions.
- •
There are four multiple choices per sample, and each choice is the derivative series $\bm{y}^{\prime}(\bm{x})$ of a random selection of function types, with the same noise level and number of points as the function in question.
- •
These results are then passed either as a stringified series of $\bm{x}$ and $\bm{y}$ vectors (“text” task), or as a matplotlib figure (the “plot” task) to the model.
- •
This task represents a multi-step extension of the function identification task: here we require the model first to understand a function, next reason about what the functional trend implies about the characteristics of its derivative, and then finally identify those characteristics within the set of multiple choices. Beyond simply introducing a multi-reasoning requirement, we also focus on derivative understanding as rates of change are key components of time-series analysis and understanding. Note that because the choices are different functional classes, the model can achieve good accuracy without reasoning about the functional magnitudes.

Figure: Supplementary Figure S4: Each row shows plots used for a randomly selected derivative identification task example. The leftmost plot in the row is the function to identify the derivative of, and the remaining plots are the four multiple choices.
Refer to caption: extracted/5893188/figures/20240927142655_Supplementary_Examples_Derivative_Identification.png

Quadratic derivative identification

- •
We generate $(\bm{x},\bm{y})$ series of quadratics (of form $y(x)=A\cdot x^{2}$) and their derivatives $\bm{y}^{\prime}(\bm{x})$ over a range of scales $A\in\{-10,-5,-1,1,5,10\}$ over the range $\bm{x}\in[-10,10]$.
- •
We perform five repeats across different numbers of points (50, 500, 1000 and 2000) and noise levels (0.0, 0.5, 1.0, 2.0 and 5.0), with 600 samples per model and number of few-shots across number of points, noise level, function type and random replica dimensions.
- •
There are four multiple choices per sample, and each choice is a random selection of derivatives of quadratic functions with a range of scales $A\in\{-20,-15,-10,-5,-1,1,5,10,15,20\}$, with the same noise level and number of points as the quadratic function in question.
- •
We create few-shot examples in the same manner as the main task, with the quadratic functions being randomly sampled from a range of scales $A\in\{-20,-15,-3,3,15,20\}$ with 0 noise and 50 data points.
- •
These results are then passed either as a stringified series of $\bm{x}$ and $\bm{y}$ vectors (“text” task), or as a matplotlib figure (the “plot” task) to the model.
- •
This hard variant of the derivatives task introduces a new requirement for the model to reason about function magnitudes as all the possible choices are the correct functional form (i.e., linear).

Figure: Supplementary Figure S5: Each row shows plots used for a randomly selected quadratic derivative identification task example. The leftmost plot in the row is the function to identify the derivative of, and the remaining plots are the four multiple choices.
Refer to caption: extracted/5893188/figures/20240927142702_Supplementary_Examples_Quadratic_Derivative_Identification.png

#### S.1.1.2 Real-world tasks

Fall detection
The task is hard to define as zero-shot, since there isn’t a natural way of explaining the plots and text, so we frame this as a few-shot task. We test few-shot examples with 1, 3, 5 and 10 examples, with 480 samples for each body location per model and number of few-shots. Model API errors were ignored as long as at least one body location succeeded for that example.

*Pre-processing and post-processing*

Due to context-window limitations for the text-only task, we apply a 1D average pool with a kernel size of 10 and stride of 10 to the data. This allows us to use up to 10 few-shot examples in the text-only tasks and still fit into the GPT4o and GPT4o-mini 128k context window.

The dataset provides multiple IMUs from 7 body locations. We consider a subset of head, waist, left and right thigh. We sample IMUs from each as separate examples - when evaluating either text or plot approach, the final prediction is a majority vote from the predictions from each of these body parts.

The dataset is stratified into train (20%) and test (80%) based on participant ID. Few-shot examples are chosen from the “train” set while eval data is chosen from the “test” set. This ensures the model never sees any data from the same participant.

Activity detection

*Pre-processing*

As with the fall detection data, we apply a 1D average pool over the raw data to limit the number of text tokens. As the raw IMU sample rates varied between 70-200Hz the kernel size, and matching stride, was chosen to target a downsampled frequency of 10Hz for each example. We select few-shot examples using leave-one-out cross-validation at the dataset user level to maximise the number of examples used for validation while ensuring we exclude any few-shot examples from the same user, even those from different device types.

The raw HHAR dataset consists of examples with varying durations and longer examples typically contain multi-second gaps with no samples from the IMU. We chunk each raw example by splitting on any gaps longer than 2 seconds and take a central 15 second crop of the longest chunk to create the examples used in this study. Any examples with mismatching sample rates for the accelerometer and gyroscope are filtered out. The raw labels “stairsup” and “stairsdown” are coalesced into a single “stairs” class.

### S.1.2 Further Results

We first present the relative differences in task performance between plot and text approaches in Supplementary Table S1.

**Supplementary Table S1: Percent differences in median plot performances relative to median text performances.**
| Task<br>(Metric) | Shots | GPT4o-mini | Gemini Flash 1.5 | GPT4o | Gemini Pro 1.5 |
| --- | --- | --- | --- | --- | --- |
| Functional form id. (Accuracy) | 0 | 70.8 | 82.4 | 122.7 | 11.1 |
| Correlation of two lines (Accuracy) | 0 | 66.7 | 42.9 | 20.0 | 50.0 |
| 2D cluster counting (MAE) | 0 | 55.8 | 65.5 | 86.4 | 85.0 |
| Derivative id. (Accuracy) | 0 | 40.0 | 31.6 | -5.3 | -0.0 |
| Quadratic derivative id. (Accuracy) | 0 | 100.0 | 85.7 | -37.5 | 47.4 |
| 1 | 37.5 | 107.1 | 23.8 | 113.3 |  |
| 2 | 92.9 | 100.0 | 8.0 | 128.6 |  |
| 3 | 108.3 | 112.5 | 9.1 | 172.7 |  |
| Fall detection ($F_{1}$) | 1 | 11.6 | 36.5 | 114.4 | 27.3 |
| 10 | 120.6 | 49.1 | 153.0 | 92.4 |  |
| Activity detection ($F_{1}$) | 1 | 36.0 | 36.5 | 6.3 | 69.3 |
| 5 | 48.0 | 77.2 | 39.7 | 64.6 |  |
| Readiness ($F_{1}$) | 0 | – | -10.8 | – | 10.0 |

In Supplementary Table S2 we compare the fall detection task results from most performant foundation models (GPT4o and Gemini Pro 1.5) with a task-specific model reported in (Aziz et al., ) using the sensitivity and specificity metrics that they report.
While we don’t achieve the same sensitivity and specificity as expected for a general model, our plot results are of the same order of magnitude.

**Supplementary Table S2: Comparison of most performant foundation models with task-specific fall detection support vector machine (SVM) as reported in (Aziz et al., ).**
| Model | Sensitivity | Specificity |
| --- | --- | --- |
| GPT4o, plot, 10-shot | 0.92 | 0.81 |
| GPT4o, text, 10-shot | 0.70 | 0.49 |
| Gemini Pro 1.5, plot, 10-shot | 0.84 | 0.95 |
| Gemini Pro 1.5, text, 10-shot | 0.61 | 0.70 |
| Task-specific SVM | 0.96 | 0.96 |

Next, we present results of the synthetic task performances as functions of the various dataset parameters that control the difficulty of the task example.
The variable model responses over different combinations of the dataset parameters described in this section create the distributions shown in Figure 1.
Depending on the task, these dataset parameters include:

- •
Number of points: the number of points per series. For all tasks except for 2D cluster counting, this means the number of function samples between $x=-10$ and $x=10$. For the cluster counting task, this is simply the number of points per cluster.
- •
Noise level: the amount of noise injected into the functions, i.e. $y=y(x+noise_level)$. This parameter is relevant for all synthetic tasks except the 2D cluster counting.
- •
Standard deviation: for the 2D cluster counting task, this controls the tightness of each cluster.

We also show the performance of each task over the set of possible correct answers specific to that task.

#### S.1.2.1 Functional form identification

Figure: (a) Results as a function of number of points
Refer to caption: x7.png

#### S.1.2.2 Correlation of two lines

Figure: (a) Results as a function of number of points
Refer to caption: x10.png

#### S.1.2.3 2D cluster counting

Figure: (a) Results as a function of number of points
Refer to caption: x13.png

#### S.1.2.4 Derivative identification

Figure: (a) Results as a function of number of points
Refer to caption: x16.png

#### S.1.2.5 Quadratic derivative identification

Figure: (a) Results as a function of number of points
Refer to caption: x19.png

### S.1.3 Ablations

#### S.1.3.1 Methodology

We perform the ablations described here on the functional form identification task using Gemini Pro 1.5.

For the text-only task we tested the effect of comma versus space separation of numbers. We were also guided by the approach in LLMTime (Gruver et al., ) and tested the effect of fixed precision of floating point numbers (2, 4, 8, 16) and rescaling the input numbers.

For the plot tasks, we considered the following ablations:

- •
Resolution (in dpi): 25, 50, 100, 200, 400
- •
Figure size: $(3.5,3.5)$, $(4,3)$, $(7,7)$, $(8,6)$, $(12,12)$
- •
Plot style: Default, classic, ggplot, seaborn-whitegrid, seaborn-darkgrid
- •
Different color palettes for text, background and scatter
- •
Marker types: circle, square, triangle, x-mark, plus
- •
Marker sizes: small (10), medium (50), large (100)
- •
Plot components: all (title, axis labels, spines, ticks, grid, axes), minimal (grid and axes only) or none
- •
Temperature: 0.0, 0.1, 0.3, 0.55, 1.0

We also test the combined plot and text version of the task, across both possible prompt orderings (i.e., text followed by plot, and plot followed by text).

Table cells contain mean accuracies with 95% confidence interval derived from 1,000 bootstrap repeats in brackets; rows and columns refer to different combinations of ablation and dataset parameters. We note that generally speaking, nothing had a strong positive effect on the results.

#### S.1.3.2 Text-based results

**Supplementary Table S3: Text-based functional form identification ablation results: modifying value separator for different numbers of points.**
| Separator | 50 | 500 | 1000 | 2500 |
| --- | --- | --- | --- | --- |
| Comma and space | 0.63 (0.59, 0.67) | 0.50 (0.46, 0.54) | 0.50 (0.46, 0.54) | 0.36 (0.32, 0.40) |
| Space | 0.62 (0.58, 0.66) | 0.54 (0.50, 0.59) | 0.48 (0.44, 0.53) | 0.36 (0.32, 0.40) |

**Supplementary Table S4: Text-based functional form identification ablation results: modifying value separator for different noise levels.**
| Separator | 0.0 | 0.5 | 1.0 | 2.0 | 5.0 |
| --- | --- | --- | --- | --- | --- |
| Comma and space | 0.71 (0.66, 0.75) | 0.49 (0.44, 0.54) | 0.44 (0.39, 0.49) | 0.46 (0.41, 0.51) | 0.39 (0.34, 0.44) |
| Space | 0.73 (0.69, 0.78) | 0.49 (0.44, 0.54) | 0.46 (0.41, 0.51) | 0.42 (0.37, 0.48) | 0.40 (0.34, 0.44) |

**Supplementary Table S5: Text-based functional form identification ablation results: modifying value separator for different function classes.**
| Separator | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| Comma and space | 0.12 (0.08, 0.14) | 0.91 (0.88, 0.94) | 0.48 (0.43, 0.53) | 0.73 (0.69, 0.78) | 0.26 (0.21, 0.30) |
| Space | 0.10 (0.07, 0.12) | 0.83 (0.80, 0.87) | 0.45 (0.40, 0.50) | 0.87 (0.84, 0.91) | 0.26 (0.22, 0.30) |

**Supplementary Table S6: Text-based functional form identification ablation results: modifying floating point fixed precision for different numbers of points.**
| Fixed precision | 50 | 500 | 1000 | 2500 |
| --- | --- | --- | --- | --- |
| 2 | 0.70 (0.64, 0.75) | 0.53 (0.47, 0.59) | 0.51 (0.45, 0.57) | 0.31 (0.25, 0.37) |
| 4 | 0.62 (0.55, 0.68) | 0.53 (0.47, 0.59) | 0.50 (0.44, 0.56) | 0.40 (0.34, 0.46) |
| 8 | 0.60 (0.54, 0.66) | 0.51 (0.45, 0.57) | 0.52 (0.46, 0.58) | 0.36 (0.30, 0.42) |
| 16 | 0.59 (0.53, 0.64) | 0.52 (0.46, 0.59) | 0.44 (0.38, 0.51) | 0.36 (0.31, 0.43) |

**Supplementary Table S7: Text-based functional form identification ablation results: modifying floating point fixed precision for different noise levels.**
| Fixed precision | 0.0 | 0.5 | 1.0 | 2.0 | 5.0 |
| --- | --- | --- | --- | --- | --- |
| 2 | 0.77 (0.71, 0.82) | 0.51 (0.43, 0.57) | 0.45 (0.38, 0.52) | 0.43 (0.36, 0.51) | 0.40 (0.33, 0.47) |
| 4 | 0.75 (0.68, 0.81) | 0.46 (0.40, 0.53) | 0.46 (0.40, 0.53) | 0.46 (0.39, 0.53) | 0.43 (0.36, 0.50) |
| 8 | 0.69 (0.62, 0.75) | 0.45 (0.38, 0.52) | 0.48 (0.41, 0.55) | 0.47 (0.40, 0.53) | 0.41 (0.34, 0.48) |
| 16 | 0.68 (0.61, 0.74) | 0.56 (0.49, 0.63) | 0.42 (0.35, 0.49) | 0.41 (0.34, 0.47) | 0.33 (0.27, 0.40) |

**Supplementary Table S8: Text-based functional form identification ablation results: modifying floating point fixed precision for different function classes.**
| Fixed precision | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| 2 | 0.12 (0.07, 0.16) | 0.95 (0.92, 0.98) | 0.58 (0.52, 0.65) | 0.55 (0.48, 0.62) | 0.35 (0.29, 0.42) |
| 4 | 0.10 (0.07, 0.15) | 0.95 (0.93, 0.98) | 0.41 (0.35, 0.48) | 0.81 (0.76, 0.86) | 0.27 (0.20, 0.32) |
| 8 | 0.07 (0.04, 0.12) | 0.84 (0.79, 0.89) | 0.45 (0.38, 0.51) | 0.89 (0.84, 0.93) | 0.24 (0.18, 0.30) |
| 16 | 0.12 (0.08, 0.17) | 0.74 (0.68, 0.80) | 0.40 (0.33, 0.47) | 0.96 (0.94, 0.98) | 0.17 (0.12, 0.23) |

**Supplementary Table S9: Text-based functional form identification ablation results: rescaling values as suggested in (Gruver et al., ) for different numbers of points.**
| rescaling | 50 | 500 | 1000 | 2500 |
| --- | --- | --- | --- | --- |
| llmtime ($\alpha=0.5,\beta=0.0$) | 0.61 (0.52, 0.69) | 0.31 (0.23, 0.40) | 0.32 (0.24, 0.40) | 0.40 (0.32, 0.48) |
| llmtime ($\alpha=0.5,\beta=0.15$) | 0.55 (0.46, 0.65) | 0.38 (0.30, 0.46) | 0.35 (0.27, 0.43) | 0.30 (0.22, 0.38) |
| llmtime ($\alpha=0.5,\beta=0.3)$ | 0.50 (0.42, 0.59) | 0.37 (0.28, 0.46) | 0.33 (0.26, 0.41) | 0.27 (0.19, 0.35) |
| llmtime ($\alpha=0.5,\beta=0.5)$ | 0.57 (0.48, 0.66) | 0.38 (0.29, 0.46) | 0.34 (0.26, 0.43) | 0.30 (0.22, 0.38) |
| llmtime ($\alpha=0.7,\beta=0.0)$ | 0.57 (0.48, 0.66) | 0.32 (0.24, 0.41) | 0.29 (0.21, 0.38) | 0.38 (0.30, 0.47) |
| llmtime ($\alpha=0.7,\beta=0.15$) | 0.54 (0.46, 0.62) | 0.35 (0.27, 0.44) | 0.30 (0.22, 0.38) | 0.30 (0.22, 0.38) |
| llmtime ($\alpha=0.7,\beta=0.3)$ | 0.57 (0.48, 0.66) | 0.34 (0.26, 0.43) | 0.30 (0.22, 0.38) | 0.30 (0.22, 0.38) |
| llmtime ($\alpha=0.7,\beta=0.5)$ | 0.54 (0.45, 0.62) | 0.40 (0.32, 0.49) | 0.32 (0.24, 0.40) | 0.28 (0.21, 0.36) |
| llmtime ($\alpha=0.9,\beta=0.0)$ | 0.52 (0.43, 0.61) | 0.33 (0.24, 0.41) | 0.29 (0.20, 0.37) | 0.38 (0.30, 0.47) |
| llmtime ($\alpha=0.9,\beta=0.15$) | 0.52 (0.43, 0.61) | 0.38 (0.30, 0.46) | 0.32 (0.25, 0.40) | 0.27 (0.19, 0.35) |
| llmtime ($\alpha=0.9,\beta=0.3)$ | 0.50 (0.41, 0.58) | 0.31 (0.23, 0.39) | 0.28 (0.21, 0.36) | 0.30 (0.23, 0.38) |
| llmtime ($\alpha=0.9,\beta=0.5)$ | 0.53 (0.44, 0.62) | 0.32 (0.24, 0.39) | 0.32 (0.24, 0.40) | 0.31 (0.23, 0.39) |
| llmtime ($\alpha=0.99,\beta=0.0$) | 0.56 (0.47, 0.65) | 0.31 (0.23, 0.40) | 0.31 (0.24, 0.39) | 0.41 (0.33, 0.50) |
| llmtime ($\alpha=0.99,\beta=0.15$) | 0.54 (0.46, 0.62) | 0.34 (0.25, 0.42) | 0.38 (0.29, 0.46) | 0.32 (0.24, 0.40) |
| llmtime ($\alpha=0.99,\beta=0.3$) | 0.54 (0.46, 0.63) | 0.33 (0.24, 0.41) | 0.32 (0.24, 0.40) | 0.29 (0.22, 0.38) |
| llmtime ($\alpha=0.99,\beta=0.5$) | 0.51 (0.43, 0.60) | 0.34 (0.26, 0.42) | 0.33 (0.25, 0.41) | 0.30 (0.22, 0.37) |
| minmax | 0.61 (0.53, 0.70) | 0.42 (0.34, 0.51) | 0.35 (0.27, 0.43) | 0.24 (0.17, 0.32) |
| none | 0.62 (0.54, 0.70) | 0.58 (0.49, 0.66) | 0.57 (0.47, 0.66) | 0.35 (0.27, 0.43) |

**Supplementary Table S10: Text-based functional form identification ablation results: rescaling values as suggested in (Gruver et al., ) for different noise levels.**
| rescaling | 0.0 | 0.5 | 1.0 | 2.0 | 5.0 |
| --- | --- | --- | --- | --- | --- |
| llmtime ($\alpha=0.5,\beta=0.0$) | 0.41 (0.31, 0.50) | 0.43 (0.33, 0.53) | 0.43 (0.33, 0.52) | 0.39 (0.29, 0.48) | 0.39 (0.30, 0.49) |
| llmtime ($\alpha=0.5,\beta=0.1$5) | 0.48 (0.38, 0.58) | 0.42 (0.32, 0.51) | 0.40 (0.31, 0.50) | 0.36 (0.26, 0.45) | 0.31 (0.22, 0.40) |
| llmtime ($\alpha=0.5,\beta=0.3$) | 0.48 (0.38, 0.57) | 0.42 (0.33, 0.52) | 0.42 (0.32, 0.52) | 0.28 (0.19, 0.37) | 0.24 (0.16, 0.33) |
| llmtime ($\alpha=0.5,\beta=0.5$) | 0.49 (0.39, 0.59) | 0.44 (0.34, 0.54) | 0.42 (0.33, 0.51) | 0.36 (0.27, 0.46) | 0.27 (0.18, 0.36) |
| llmtime ($\alpha=0.7,\beta=0.0$) | 0.33 (0.24, 0.42) | 0.44 (0.34, 0.54) | 0.43 (0.33, 0.53) | 0.35 (0.26, 0.44) | 0.40 (0.31, 0.49) |
| llmtime ($\alpha=0.7,\beta=0.1$5) | 0.45 (0.35, 0.54) | 0.47 (0.37, 0.56) | 0.34 (0.25, 0.44) | 0.34 (0.25, 0.43) | 0.26 (0.18, 0.34) |
| llmtime ($\alpha=0.7,\beta=0.3$) | 0.43 (0.34, 0.53) | 0.39 (0.30, 0.48) | 0.37 (0.27, 0.47) | 0.38 (0.29, 0.47) | 0.31 (0.22, 0.40) |
| llmtime ($\alpha=0.7,\beta=0.5$) | 0.49 (0.38, 0.58) | 0.47 (0.36, 0.57) | 0.41 (0.31, 0.50) | 0.30 (0.20, 0.39) | 0.25 (0.17, 0.33) |
| llmtime ($\alpha=0.9,\beta=0.0$) | 0.39 (0.30, 0.49) | 0.41 (0.31, 0.50) | 0.42 (0.32, 0.52) | 0.38 (0.29, 0.48) | 0.30 (0.21, 0.39) |
| llmtime ($\alpha=0.9,\beta=0.15$) | 0.45 (0.35, 0.55) | 0.40 (0.32, 0.49) | 0.39 (0.30, 0.49) | 0.35 (0.26, 0.44) | 0.27 (0.19, 0.36) |
| llmtime ($\alpha=0.9,\beta=0.3$) | 0.39 (0.29, 0.49) | 0.41 (0.31, 0.51) | 0.35 (0.26, 0.44) | 0.34 (0.25, 0.43) | 0.25 (0.17, 0.33) |
| llmtime ($\alpha=0.9,\beta=0.5$) | 0.44 (0.34, 0.54) | 0.45 (0.35, 0.54) | 0.44 (0.34, 0.53) | 0.33 (0.24, 0.43) | 0.19 (0.12, 0.27) |
| llmtime ($\alpha=0.99,\beta=0.0$) | 0.41 (0.32, 0.50) | 0.43 (0.33, 0.53) | 0.38 (0.29, 0.48) | 0.38 (0.29, 0.47) | 0.39 (0.29, 0.48) |
| llmtime ($\alpha=0.99,\beta=0.15$) | 0.50 (0.40, 0.60) | 0.43 (0.34, 0.53) | 0.44 (0.35, 0.53) | 0.30 (0.21, 0.40) | 0.30 (0.21, 0.40) |
| llmtime ($\alpha=0.99,\beta=0.3$) | 0.43 (0.34, 0.54) | 0.44 (0.35, 0.53) | 0.40 (0.31, 0.49) | 0.27 (0.19, 0.36) | 0.31 (0.22, 0.40) |
| llmtime ($\alpha=0.99,\beta=0.5$) | 0.47 (0.38, 0.57) | 0.45 (0.35, 0.55) | 0.40 (0.31, 0.50) | 0.27 (0.18, 0.36) | 0.25 (0.17, 0.34) |
| minmax | 0.39 (0.30, 0.49) | 0.53 (0.43, 0.62) | 0.48 (0.38, 0.58) | 0.35 (0.26, 0.44) | 0.28 (0.20, 0.36) |
| none | 0.77 (0.68, 0.85) | 0.54 (0.44, 0.64) | 0.55 (0.45, 0.64) | 0.45 (0.35, 0.55) | 0.34 (0.25, 0.44) |

**Supplementary Table S11: Text-based functional form identification ablation results: rescaling values as suggested in (Gruver et al., ) for different function classes.**
| rescaling | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| llmtime ($\alpha$=0.5, $\beta$=0.0) | 0.01 (0.00, 0.03) | 0.68 (0.58, 0.76) | 0.50 (0.40, 0.60) | 0.62 (0.53, 0.72) | 0.24 (0.15, 0.33) |
| llmtime ($\alpha$=0.5, $\beta$=0.15) | 0.01 (0.00, 0.03) | 0.58 (0.49, 0.68) | 0.55 (0.45, 0.64) | 0.58 (0.49, 0.67) | 0.25 (0.17, 0.34) |
| llmtime ($\alpha$=0.5, $\beta$=0.3) | 0.01 (0.00, 0.03) | 0.59 (0.50, 0.69) | 0.42 (0.33, 0.51) | 0.57 (0.47, 0.67) | 0.25 (0.17, 0.33) |
| llmtime ($\alpha$=0.5, $\beta$=0.5) | 0.01 (0.00, 0.03) | 0.56 (0.46, 0.65) | 0.52 (0.42, 0.62) | 0.60 (0.50, 0.69) | 0.29 (0.21, 0.38) |
| llmtime ($\alpha$=0.7, $\beta$=0.0) | 0.02 (0.00, 0.05) | 0.71 (0.62, 0.79) | 0.44 (0.34, 0.54) | 0.62 (0.52, 0.72) | 0.16 (0.09, 0.24) |
| llmtime ($\alpha$=0.7, $\beta$=0.15) | 0.00 (0.00, 0.00) | 0.54 (0.44, 0.64) | 0.47 (0.38, 0.56) | 0.61 (0.51, 0.70) | 0.24 (0.16, 0.32) |
| llmtime ($\alpha$=0.7, $\beta$=0.3) | 0.00 (0.00, 0.00) | 0.62 (0.52, 0.72) | 0.46 (0.36, 0.56) | 0.57 (0.47, 0.67) | 0.23 (0.15, 0.31) |
| llmtime ($\alpha$=0.7, $\beta$=0.5) | 0.03 (0.00, 0.07) | 0.57 (0.47, 0.67) | 0.49 (0.40, 0.59) | 0.59 (0.49, 0.69) | 0.24 (0.16, 0.33) |
| llmtime ($\alpha$=0.9, $\beta$=0.0) | 0.01 (0.00, 0.03) | 0.68 (0.59, 0.77) | 0.36 (0.27, 0.45) | 0.61 (0.52, 0.70) | 0.24 (0.16, 0.32) |
| llmtime ($\alpha$=0.9, $\beta$=0.15) | 0.00 (0.00, 0.00) | 0.61 (0.52, 0.70) | 0.48 (0.39, 0.58) | 0.57 (0.47, 0.67) | 0.20 (0.12, 0.28) |
| llmtime ($\alpha$=0.9, $\beta$=0.3) | 0.02 (0.00, 0.05) | 0.48 (0.38, 0.57) | 0.40 (0.31, 0.50) | 0.65 (0.55, 0.74) | 0.19 (0.12, 0.27) |
| llmtime ($\alpha$=0.9, $\beta$=0.5) | 0.03 (0.00, 0.07) | 0.53 (0.43, 0.63) | 0.53 (0.44, 0.63) | 0.53 (0.43, 0.63) | 0.23 (0.15, 0.32) |
| llmtime ($\alpha$=0.99, $\beta$=0.0) | 0.01 (0.00, 0.03) | 0.71 (0.62, 0.80) | 0.40 (0.30, 0.50) | 0.62 (0.53, 0.71) | 0.25 (0.17, 0.34) |
| llmtime ($\alpha$=0.99, $\beta$=0.15) | 0.00 (0.00, 0.00) | 0.59 (0.48, 0.69) | 0.48 (0.38, 0.58) | 0.63 (0.54, 0.72) | 0.27 (0.18, 0.36) |
| llmtime ($\alpha$=0.99, $\beta$=0.3) | 0.01 (0.00, 0.03) | 0.61 (0.51, 0.71) | 0.45 (0.35, 0.55) | 0.58 (0.48, 0.68) | 0.20 (0.12, 0.28) |
| llmtime ($\alpha$=0.99, $\beta$=0.5) | 0.01 (0.00, 0.03) | 0.51 (0.42, 0.60) | 0.47 (0.37, 0.57) | 0.61 (0.51, 0.70) | 0.24 (0.16, 0.33) |
| minmax | 0.00 (0.00, 0.00) | 0.57 (0.47, 0.66) | 0.59 (0.49, 0.67) | 0.67 (0.58, 0.76) | 0.20 (0.13, 0.28) |
| none | 0.17 (0.10, 0.25) | 0.92 (0.87, 0.97) | 0.48 (0.38, 0.58) | 0.79 (0.71, 0.87) | 0.29 (0.20, 0.37) |

#### S.1.3.3 Plot-based results

**Supplementary Table S12: Plot-based functional form identification ablation results: modifying figure dpi for different numbers of points.**
| dpi | 50 | 500 | 1000 | 2500 |
| --- | --- | --- | --- | --- |
| 25 | 0.66 (0.57, 0.74) | 0.67 (0.60, 0.75) | 0.70 (0.62, 0.78) | 0.66 (0.58, 0.74) |
| 50 | 0.59 (0.51, 0.67) | 0.70 (0.62, 0.78) | 0.70 (0.62, 0.78) | 0.74 (0.66, 0.82) |
| 100 | 0.67 (0.58, 0.75) | 0.70 (0.62, 0.78) | 0.68 (0.60, 0.76) | 0.74 (0.66, 0.81) |
| 200 | 0.72 (0.64, 0.80) | 0.70 (0.62, 0.78) | 0.72 (0.64, 0.80) | 0.73 (0.65, 0.81) |
| 400 | 0.63 (0.55, 0.71) | 0.71 (0.63, 0.79) | 0.71 (0.63, 0.79) | 0.73 (0.65, 0.81) |

**Supplementary Table S13: Plot-based functional form identification ablation results: modifying figure dpi for different noise levels.**
| dpi | 0.0 | 0.5 | 1.0 | 2.0 | 5.0 |
| --- | --- | --- | --- | --- | --- |
| 25 | 0.91 (0.85, 0.96) | 0.83 (0.76, 0.90) | 0.61 (0.52, 0.71) | 0.61 (0.51, 0.70) | 0.41 (0.30, 0.51) |
| 50 | 0.92 (0.86, 0.97) | 0.87 (0.80, 0.93) | 0.68 (0.59, 0.77) | 0.59 (0.49, 0.69) | 0.35 (0.26, 0.44) |
| 100 | 0.95 (0.90, 0.99) | 0.89 (0.82, 0.95) | 0.65 (0.56, 0.74) | 0.60 (0.50, 0.70) | 0.40 (0.31, 0.50) |
| 200 | 0.98 (0.95, 1.00) | 0.89 (0.83, 0.95) | 0.69 (0.60, 0.78) | 0.61 (0.52, 0.70) | 0.42 (0.33, 0.52) |
| 400 | 0.94 (0.89, 0.98) | 0.89 (0.82, 0.95) | 0.68 (0.59, 0.77) | 0.58 (0.49, 0.68) | 0.39 (0.30, 0.49) |

**Supplementary Table S14: Plot-based functional form identification ablation results: modifying figure dpi for different function classes.**
| dpi | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| 25 | 0.22 (0.14, 0.30) | 0.95 (0.91, 0.99) | 0.99 (0.97, 1.00) | 0.37 (0.28, 0.47) | 0.84 (0.77, 0.91) |
| 50 | 0.38 (0.29, 0.47) | 0.95 (0.91, 0.99) | 0.98 (0.95, 1.00) | 0.32 (0.23, 0.41) | 0.78 (0.70, 0.85) |
| 100 | 0.37 (0.28, 0.47) | 0.96 (0.92, 0.99) | 0.97 (0.93, 1.00) | 0.45 (0.35, 0.55) | 0.74 (0.65, 0.82) |
| 200 | 0.42 (0.32, 0.51) | 0.97 (0.93, 1.00) | 0.98 (0.95, 1.00) | 0.45 (0.36, 0.55) | 0.77 (0.68, 0.84) |
| 400 | 0.43 (0.33, 0.53) | 0.94 (0.89, 0.98) | 0.98 (0.95, 1.00) | 0.39 (0.29, 0.49) | 0.74 (0.65, 0.83) |

**Supplementary Table S15: Plot-based functional form identification ablation results: modifying figure size for different numbers of points.**
| figsize | 50 | 500 | 1000 | 2500 |
| --- | --- | --- | --- | --- |
| (4, 3) | 0.67 (0.59, 0.75) | 0.70 (0.62, 0.78) | 0.71 (0.63, 0.79) | 0.71 (0.63, 0.79) |
| (3.5, 3.5) | 0.71 (0.63, 0.79) | 0.73 (0.65, 0.81) | 0.76 (0.68, 0.83) | 0.77 (0.70, 0.83) |
| (6.4, 4.8) | 0.69 (0.60, 0.76) | 0.72 (0.65, 0.80) | 0.70 (0.63, 0.78) | 0.72 (0.65, 0.80) |
| (8, 6) | 0.65 (0.57, 0.74) | 0.70 (0.62, 0.78) | 0.69 (0.60, 0.77) | 0.72 (0.65, 0.79) |
| (7, 7) | 0.63 (0.55, 0.72) | 0.74 (0.66, 0.81) | 0.71 (0.63, 0.79) | 0.75 (0.67, 0.82) |
| (12, 12) | 0.70 (0.62, 0.78) | 0.66 (0.58, 0.74) | 0.68 (0.60, 0.75) | 0.71 (0.63, 0.79) |

**Supplementary Table S16: Plot-based functional form identification ablation results: modifying figure size for different noise levels.**
| figsize | 0.0 | 0.5 | 1.0 | 2.0 | 5.0 |
| --- | --- | --- | --- | --- | --- |
| (4, 3) | 0.98 (0.95, 1.00) | 0.86 (0.79, 0.93) | 0.65 (0.55, 0.75) | 0.63 (0.54, 0.72) | 0.37 (0.28, 0.47) |
| (3.5, 3.5) | 0.98 (0.95, 1.00) | 0.90 (0.83, 0.95) | 0.76 (0.68, 0.84) | 0.69 (0.60, 0.77) | 0.38 (0.29, 0.48) |
| (6.4, 4.8) | 0.97 (0.93, 1.00) | 0.89 (0.83, 0.95) | 0.66 (0.57, 0.75) | 0.59 (0.49, 0.69) | 0.43 (0.34, 0.53) |
| (8, 6) | 0.95 (0.90, 0.99) | 0.88 (0.82, 0.94) | 0.67 (0.59, 0.75) | 0.59 (0.49, 0.68) | 0.35 (0.26, 0.44) |
| (7, 7) | 0.95 (0.90, 0.99) | 0.88 (0.81, 0.94) | 0.76 (0.68, 0.84) | 0.60 (0.51, 0.70) | 0.35 (0.26, 0.44) |
| (12, 12) | 0.94 (0.89, 0.98) | 0.79 (0.70, 0.87) | 0.69 (0.60, 0.78) | 0.63 (0.54, 0.72) | 0.38 (0.29, 0.48) |

**Supplementary Table S17: Plot-based functional form identification ablation results: modifying figure size for different function classes.**
| figsize | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| (4, 3) | 0.35 (0.26, 0.44) | 0.95 (0.90, 0.99) | 0.99 (0.97, 1.00) | 0.39 (0.29, 0.48) | 0.81 (0.73, 0.89) |
| (3.5, 3.5) | 0.57 (0.47, 0.66) | 0.96 (0.92, 0.99) | 0.99 (0.97, 1.00) | 0.36 (0.27, 0.46) | 0.83 (0.76, 0.90) |
| (6.4, 4.8) | 0.37 (0.28, 0.46) | 0.96 (0.92, 0.99) | 0.99 (0.96, 1.00) | 0.48 (0.38, 0.57) | 0.74 (0.65, 0.83) |
| (8, 6) | 0.40 (0.30, 0.50) | 0.94 (0.89, 0.98) | 0.97 (0.93, 1.00) | 0.39 (0.30, 0.49) | 0.74 (0.66, 0.83) |
| (7, 7) | 0.51 (0.41, 0.61) | 0.93 (0.88, 0.98) | 0.95 (0.90, 0.99) | 0.35 (0.26, 0.44) | 0.80 (0.72, 0.88) |
| (12, 12) | 0.31 (0.22, 0.40) | 0.91 (0.85, 0.96) | 1.00 (1.00, 1.00) | 0.43 (0.33, 0.53) | 0.78 (0.70, 0.85) |

**Supplementary Table S18: Plot-based functional form identification ablation results: modifying plotting style for different numbers of points.**
| Plot style | 50 | 500 | 1000 | 2500 |
| --- | --- | --- | --- | --- |
| None | 0.66 (0.57, 0.74) | 0.72 (0.64, 0.79) | 0.70 (0.62, 0.78) | 0.72 (0.64, 0.79) |
| classic | 0.70 (0.62, 0.78) | 0.72 (0.64, 0.79) | 0.70 (0.62, 0.78) | 0.73 (0.65, 0.80) |
| ggplot | 0.65 (0.56, 0.73) | 0.70 (0.62, 0.79) | 0.70 (0.62, 0.78) | 0.74 (0.67, 0.82) |
| seaborn-v0_8-darkgrid | 0.68 (0.60, 0.76) | 0.71 (0.63, 0.78) | 0.69 (0.60, 0.77) | 0.74 (0.66, 0.82) |
| seaborn-v0_8-whitegrid | 0.66 (0.58, 0.74) | 0.70 (0.62, 0.78) | 0.70 (0.62, 0.78) | 0.72 (0.63, 0.80) |

**Supplementary Table S19: Plot-based functional form identification ablation results: modifying plotting style for different noise levels.**
| Plot style | 0.0 | 0.5 | 1.0 | 2.0 | 5.0 |
| --- | --- | --- | --- | --- | --- |
| None | 0.95 (0.91, 0.99) | 0.92 (0.86, 0.97) | 0.65 (0.55, 0.74) | 0.56 (0.47, 0.65) | 0.41 (0.31, 0.50) |
| classic | 0.96 (0.92, 0.99) | 0.91 (0.85, 0.96) | 0.67 (0.57, 0.76) | 0.59 (0.50, 0.68) | 0.43 (0.33, 0.52) |
| ggplot | 0.94 (0.89, 0.98) | 0.91 (0.85, 0.96) | 0.68 (0.59, 0.77) | 0.60 (0.50, 0.70) | 0.37 (0.28, 0.46) |
| seaborn-v0_8-darkgrid | 0.96 (0.92, 0.99) | 0.87 (0.80, 0.93) | 0.67 (0.57, 0.76) | 0.61 (0.51, 0.70) | 0.41 (0.32, 0.51) |
| seaborn-v0_8-whitegrid | 0.96 (0.92, 0.99) | 0.87 (0.81, 0.94) | 0.67 (0.58, 0.75) | 0.58 (0.48, 0.68) | 0.41 (0.32, 0.51) |

**Supplementary Table S20: Plot-based functional form identification ablation results: modifying plotting styles for different function classes.**
| Plot style | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| None | 0.39 (0.29, 0.48) | 0.97 (0.93, 1.00) | 0.97 (0.93, 1.00) | 0.43 (0.34, 0.53) | 0.73 (0.64, 0.82) |
| classic | 0.41 (0.32, 0.51) | 0.96 (0.92, 0.99) | 0.98 (0.95, 1.00) | 0.46 (0.37, 0.56) | 0.75 (0.67, 0.83) |
| ggplot | 0.42 (0.32, 0.51) | 0.94 (0.89, 0.98) | 0.96 (0.92, 0.99) | 0.43 (0.34, 0.53) | 0.75 (0.67, 0.84) |
| seaborn-v0_8-darkgrid | 0.36 (0.26, 0.45) | 0.95 (0.91, 0.99) | 1.00 (1.00, 1.00) | 0.46 (0.36, 0.55) | 0.75 (0.65, 0.84) |
| seaborn-v0_8-whitegrid | 0.35 (0.26, 0.45) | 0.97 (0.93, 1.00) | 0.97 (0.93, 1.00) | 0.47 (0.38, 0.57) | 0.73 (0.64, 0.82) |

**Supplementary Table S21: Plot-based functional form identification ablation results: modifying color palette for different numbers of points.**
| Color palette | 50 | 500 | 2500 |
| --- | --- | --- | --- |
| black and white | 0.50 (0.48, 0.51) | 0.64 (0.63, 0.65) | 0.67 (0.66, 0.69) |
| default | 0.50 (0.49, 0.52) | 0.63 (0.61, 0.64) | 0.65 (0.64, 0.67) |
| high contrast | 0.49 (0.48, 0.51) | 0.60 (0.58, 0.61) | 0.63 (0.61, 0.64) |
| invert | 0.51 (0.50, 0.53) | 0.64 (0.62, 0.65) | 0.66 (0.65, 0.68) |
| low contrast | 0.48 (0.47, 0.50) | 0.61 (0.60, 0.63) | 0.65 (0.63, 0.66) |

**Supplementary Table S22: Plot-based functional form identification ablation results: modifying color palette for different noise levels.**
| Color palette | 0.0 | 1.0 | 5.0 |
| --- | --- | --- | --- |
| black and white | 0.88 (0.87, 0.89) | 0.63 (0.62, 0.65) | 0.30 (0.29, 0.32) |
| default | 0.88 (0.87, 0.89) | 0.63 (0.62, 0.65) | 0.27 (0.26, 0.29) |
| high contrast | 0.85 (0.84, 0.86) | 0.62 (0.60, 0.63) | 0.25 (0.24, 0.26) |
| invert | 0.88 (0.87, 0.89) | 0.64 (0.62, 0.65) | 0.30 (0.28, 0.31) |
| low contrast | 0.86 (0.85, 0.87) | 0.61 (0.60, 0.63) | 0.27 (0.25, 0.28) |

**Supplementary Table S23: Plot-based functional form identification ablation results: modifying color palette for different function classes.**
| Color palette | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| black and white | 0.35 (0.33, 0.37) | 0.78 (0.76, 0.79) | 0.90 (0.88, 0.91) | 0.30 (0.28, 0.32) | 0.71 (0.69, 0.72) |
| default | 0.32 (0.30, 0.34) | 0.76 (0.74, 0.77) | 0.90 (0.89, 0.91) | 0.31 (0.29, 0.33) | 0.69 (0.67, 0.71) |
| high contrast | 0.27 (0.25, 0.28) | 0.73 (0.71, 0.74) | 0.90 (0.89, 0.91) | 0.28 (0.26, 0.30) | 0.69 (0.68, 0.71) |
| invert | 0.32 (0.30, 0.34) | 0.79 (0.77, 0.80) | 0.90 (0.89, 0.91) | 0.31 (0.29, 0.33) | 0.70 (0.69, 0.72) |
| low contrast | 0.29 (0.28, 0.31) | 0.75 (0.74, 0.77) | 0.88 (0.87, 0.89) | 0.29 (0.27, 0.30) | 0.69 (0.67, 0.71) |

**Supplementary Table S24: Plot-based functional form identification ablation results: modifying plot makers for different numbers of points.**
| Plot marker | 50 | 500 | 2500 |
| --- | --- | --- | --- |
| + | 0.48 (0.47, 0.49) | 0.61 (0.60, 0.63) | 0.64 (0.63, 0.66) |
| ^ | 0.50 (0.49, 0.52) | 0.64 (0.63, 0.66) | 0.67 (0.66, 0.69) |
| o | 0.52 (0.51, 0.54) | 0.63 (0.62, 0.65) | 0.65 (0.64, 0.67) |
| s | 0.50 (0.48, 0.52) | 0.62 (0.60, 0.64) | 0.64 (0.63, 0.66) |
| x | 0.49 (0.47, 0.50) | 0.61 (0.59, 0.62) | 0.65 (0.64, 0.67) |

**Supplementary Table S25: Plot-based functional form identification ablation results: modifying plot makers for different noise levels.**
| Plot marker | 0.0 | 1.0 | 5.0 |
| --- | --- | --- | --- |
| + | 0.86 (0.85, 0.87) | 0.62 (0.60, 0.63) | 0.26 (0.25, 0.27) |
| ^ | 0.88 (0.87, 0.89) | 0.64 (0.63, 0.66) | 0.30 (0.29, 0.32) |
| o | 0.88 (0.87, 0.89) | 0.63 (0.62, 0.65) | 0.29 (0.28, 0.31) |
| s | 0.88 (0.87, 0.89) | 0.62 (0.60, 0.63) | 0.27 (0.26, 0.28) |
| x | 0.86 (0.85, 0.87) | 0.62 (0.61, 0.64) | 0.27 (0.25, 0.28) |

**Supplementary Table S26: Plot-based functional form identification ablation results: modifying plot makers for different function classes.**
| Plot marker | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| + | 0.29 (0.27, 0.31) | 0.75 (0.74, 0.77) | 0.89 (0.88, 0.90) | 0.28 (0.26, 0.30) | 0.68 (0.66, 0.69) |
| ^ | 0.31 (0.29, 0.33) | 0.77 (0.75, 0.78) | 0.89 (0.87, 0.90) | 0.34 (0.33, 0.36) | 0.73 (0.71, 0.74) |
| o | 0.32 (0.30, 0.34) | 0.78 (0.77, 0.80) | 0.90 (0.89, 0.91) | 0.30 (0.28, 0.32) | 0.71 (0.69, 0.73) |
| s | 0.33 (0.31, 0.35) | 0.75 (0.74, 0.77) | 0.91 (0.90, 0.92) | 0.28 (0.26, 0.29) | 0.68 (0.66, 0.70) |
| x | 0.30 (0.29, 0.32) | 0.75 (0.73, 0.76) | 0.89 (0.87, 0.90) | 0.28 (0.26, 0.30) | 0.70 (0.68, 0.72) |

**Supplementary Table S27: Plot-based functional form identification ablation results: modifying plot maker sizes for different numbers of points.**
| Markers size | 50 | 500 | 2500 |
| --- | --- | --- | --- |
| large | 0.51 (0.49, 0.52) | 0.62 (0.61, 0.63) | 0.65 (0.64, 0.66) |
| medium | 0.50 (0.49, 0.51) | 0.62 (0.61, 0.63) | 0.65 (0.64, 0.66) |
| small | 0.49 (0.48, 0.50) | 0.63 (0.62, 0.64) | 0.65 (0.64, 0.66) |

**Supplementary Table S28: Plot-based functional form identification ablation results: modifying plot maker sizes for different noise levels.**
| Marker size | 0.0 | 1.0 | 5.0 |
| --- | --- | --- | --- |
| large | 0.87 (0.86, 0.88) | 0.63 (0.62, 0.64) | 0.27 (0.26, 0.28) |
| medium | 0.87 (0.86, 0.88) | 0.63 (0.61, 0.64) | 0.28 (0.27, 0.29) |
| small | 0.87 (0.86, 0.87) | 0.62 (0.61, 0.63) | 0.28 (0.27, 0.29) |

**Supplementary Table S29: Plot-based functional form identification ablation results: modifying plot maker sizes for different function classes.**
| Marker size | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| large | 0.32 (0.30, 0.33) | 0.75 (0.73, 0.76) | 0.91 (0.90, 0.92) | 0.30 (0.28, 0.31) | 0.70 (0.68, 0.71) |
| medium | 0.31 (0.30, 0.33) | 0.76 (0.75, 0.77) | 0.90 (0.89, 0.91) | 0.29 (0.27, 0.30) | 0.70 (0.69, 0.72) |
| small | 0.30 (0.29, 0.32) | 0.78 (0.76, 0.79) | 0.88 (0.87, 0.89) | 0.30 (0.29, 0.32) | 0.69 (0.68, 0.71) |

**Supplementary Table S30: Plot-based functional form identification ablation results: modifying plot components for different numbers of points.**
| Plot components | 50 | 500 | 2500 |
| --- | --- | --- | --- |
| all | 0.53 (0.51, 0.54) | 0.65 (0.64, 0.66) | 0.68 (0.67, 0.69) |
| minimal | 0.47 (0.46, 0.48) | 0.61 (0.60, 0.62) | 0.63 (0.62, 0.64) |
| none | 0.50 (0.49, 0.51) | 0.61 (0.60, 0.62) | 0.66 (0.64, 0.67) |

**Supplementary Table S31: Plot-based functional form identification ablation results: modifying plot components for different noise levels.**
| Plot components | 0.0 | 1.0 | 5.0 |
| --- | --- | --- | --- |
| all | 0.89 (0.88, 0.90) | 0.65 (0.63, 0.66) | 0.32 (0.31, 0.33) |
| minimal | 0.83 (0.82, 0.84) | 0.60 (0.59, 0.61) | 0.28 (0.27, 0.29) |
| none | 0.89 (0.88, 0.90) | 0.63 (0.62, 0.65) | 0.24 (0.23, 0.25) |

**Supplementary Table S32: Plot-based functional form identification ablation results: modifying plot components for different function classes.**
| Plot components | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| all | 0.37 (0.36, 0.39) | 0.82 (0.81, 0.83) | 0.94 (0.94, 0.95) | 0.29 (0.28, 0.31) | 0.66 (0.65, 0.68) |
| minimal | 0.24 (0.23, 0.25) | 0.75 (0.74, 0.76) | 0.89 (0.88, 0.90) | 0.25 (0.24, 0.27) | 0.71 (0.69, 0.72) |
| none | 0.32 (0.30, 0.33) | 0.71 (0.70, 0.72) | 0.85 (0.84, 0.86) | 0.34 (0.33, 0.35) | 0.72 (0.71, 0.74) |

**Supplementary Table S33: Plot-based functional form identification ablation results: modifying temperature for different numbers of points.**
| Temperature | 50 | 500 | 1000 | 2500 |
| --- | --- | --- | --- | --- |
| 0.00 | 0.70 (0.62, 0.77) | 0.71 (0.63, 0.79) | 0.68 (0.60, 0.75) | 0.72 (0.64, 0.80) |
| 0.10 | 0.66 (0.58, 0.74) | 0.71 (0.63, 0.78) | 0.70 (0.62, 0.78) | 0.73 (0.65, 0.80) |
| 0.30 | 0.72 (0.64, 0.80) | 0.72 (0.64, 0.80) | 0.68 (0.60, 0.77) | 0.73 (0.65, 0.80) |
| 0.55 | 0.69 (0.60, 0.78) | 0.70 (0.62, 0.78) | 0.70 (0.62, 0.77) | 0.74 (0.66, 0.82) |
| 1.00 | 0.71 (0.64, 0.78) | 0.70 (0.62, 0.78) | 0.68 (0.59, 0.76) | 0.74 (0.66, 0.81) |

**Supplementary Table S34: Plot-based functional form identification ablation results: modifying temperature for different noise levels.**
| Temperature | 0.0 | 0.5 | 1.0 | 2.0 | 5.0 |
| --- | --- | --- | --- | --- | --- |
| 0.00 | 0.96 (0.92, 0.99) | 0.90 (0.84, 0.96) | 0.67 (0.58, 0.75) | 0.58 (0.47, 0.67) | 0.40 (0.31, 0.50) |
| 0.10 | 0.96 (0.92, 0.99) | 0.90 (0.84, 0.96) | 0.67 (0.58, 0.76) | 0.57 (0.48, 0.66) | 0.40 (0.31, 0.51) |
| 0.30 | 0.96 (0.91, 0.99) | 0.93 (0.87, 0.98) | 0.67 (0.58, 0.76) | 0.62 (0.53, 0.71) | 0.38 (0.29, 0.48) |
| 0.55 | 0.95 (0.90, 0.99) | 0.90 (0.84, 0.95) | 0.69 (0.60, 0.78) | 0.59 (0.49, 0.69) | 0.41 (0.32, 0.51) |
| 1.00 | 0.96 (0.92, 0.99) | 0.91 (0.85, 0.96) | 0.68 (0.58, 0.77) | 0.61 (0.52, 0.70) | 0.38 (0.28, 0.47) |

**Supplementary Table S35: Plot-based functional form identification ablation results: modifying temperature for different function classes.**
| Temperature | cubic | exponential | linear | periodic | quadratic |
| --- | --- | --- | --- | --- | --- |
| 0.0 | 0.37 (0.27, 0.46) | 0.95 (0.91, 0.99) | 0.97 (0.93, 1.00) | 0.48 (0.38, 0.58) | 0.74 (0.65, 0.82) |
| 0.1 | 0.40 (0.30, 0.50) | 0.94 (0.89, 0.98) | 0.97 (0.93, 1.00) | 0.45 (0.35, 0.54) | 0.74 (0.66, 0.83) |
| 0.3 | 0.42 (0.33, 0.52) | 0.92 (0.86, 0.97) | 0.99 (0.97, 1.00) | 0.47 (0.38, 0.57) | 0.76 (0.67, 0.84) |
| 0.55 | 0.39 (0.30, 0.49) | 0.96 (0.92, 0.99) | 0.99 (0.97, 1.00) | 0.46 (0.37, 0.56) | 0.74 (0.65, 0.82) |
| 1.0 | 0.37 (0.28, 0.46) | 0.96 (0.91, 0.99) | 0.96 (0.92, 0.99) | 0.48 (0.38, 0.58) | 0.77 (0.69, 0.85) |

#### S.1.3.4 Combined modality results

Figure: Supplementary Figure S11: Results of functional form identification task across all modalities, including combined plot and text.
Refer to caption: extracted/5893188/figures/20240924135719_Functional_Form_Identification_PlotAndText_Boxplot.png

### S.1.4 Prompts and target dataclasses

For reproducibility, we provide here the prompt templates and target dataclasses we provided to Langfun for each task, except for readiness as that was performed on a proprietary dataset.

basicstyle=,
captionpos=b,
frame=single,
breaklines=true,
breakatwhitespace=false,
breakindent=0pt,

#### S.1.4.1 Functional form identification

Figure: Target Dataclass

Figure: Plot Prompt

Figure: Text Prompt

Figure: Text and Plot Prompt

Figure: Plot and Text Prompt

#### S.1.4.2 Correlation of two lines

Figure: Target Dataclass

Figure: Plot Prompt

Figure: Text Prompt

#### S.1.4.3 2D cluster counting

Figure: Target Dataclass

Figure: Plot Prompt

Figure: Text Prompt

#### S.1.4.4 Derivative identification

Figure: Target Dataclass

Figure: Plot Prompt

Figure: Text Prompt

#### S.1.4.5 Quadratic derivative identification

Figure: Target Dataclass

Figure: Plot Prompt - zero-shot

Figure: Text Prompt - zero-shot

Figure: Plot Prompt - few-shot

Figure: Text Prompt - few-shot

#### S.1.4.6 Fall detection from IMU data

Figure: Target Dataclass

Figure: Plot Prompt

Figure: Text Prompt

#### S.1.4.7 Activity recognition from IMU data

Figure: Target Dataclass

Figure: Plot Prompt

Figure: Text Prompt

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 3
presentation: 2
questions: 1. How are synthetic time series generated? 
2. How are the options of the time series understanding questions generated? 
3. What kinds of correlation between two lines are your measuring? Cross-correlation?

See other questions above.
soundness: 2
strengths: 1. The paper shows an interesting finding on a few synthetic and real-world tasks. 
2. The authors conduct rigorous statistical experiments to evaluate differences between textual and visual presentation of time series inputs.
summary: This paper demonstrates that for some synthetic and real-world time series understanding problems (for example, detecting the functional form of a time series, and recognizing human activity), using the visual component of multimodal foundation models is better than using their text component. This is an interesting finding which can likely lead to better time series understanding performance, with fewer tokens.
weaknesses: 1. **Writing:** I would encourage the authors to spend some more time improving the writing of the manuscript. For example, there's a lot of forward references to the appendix and supplementary material with key information. This not only makes reading the manuscript harder, but the leaves the reader to wonder about some basic questions, for example how is the synthetic time series generated? The captions should be written so they communicate a story, rather than just the mechanics of the plot. The rationale behind some design decisions are not explained, for example the use of structured prompting. Also I would encourage the authors to use the space more effectively, some of the tables and figures can be made smaller, some figures and long descriptions of statistical tests can also be moved to the appendix. The extra space can then be used to have a more thorough related work section, for example covering how this work is positioned in the context of time series foundation models [1--4], multimodal time series and text models [5], how the are synthetic time series generated, why are some tasks chosen versus others, etc. 
2. **Baselines:** Looking at the paper it is hard to judge whether the models are even doing well in these understanding tasks because only improvement of vision-based pipeline over the text-based pipeline is shown. Moreover, how do humans, simple time series models, other foundation models do? Only in Figure 4, do the authors mention "state-of-the-art", but fail to mention which model they are referring to. Moroever, the authors only use closed-source models, while open-source multimodal models are available. Including, results from these models are likely to aid reproducibility, and improve the rigour of the experiments. I would also encourage the authors to improve the readability of Table 1, for example I am not sure what "understanding of one overall trend" means, or what does "Num points" correspond to.
3. **Tasks:** The authors claim that they evaluate the "reasoning" capability of these models, but it is unclear how that is done? It's also unclear why some of these tasks were chosen, versus other tasks, or if reasoning elicitation techniques such as Chain-of-Thought or its variants were used. I would encourage the authors to review some recent work on evaluating reasoning in time series models [6] and LLMs [7]. These studies seem to have been made public very close to the ICLR deadline, but still present some useful and interesting results and experimental setups which might be of value in improving this manuscript. 
4. **Plots:** Some questions about the nature of plots remain unanswered. What kinds of problems are benefited by visual processing. I am assuming problem which can be reasoned about by the shape of the time series. Also how were the plots generated? How does the plot generation mechanism affect the results? 
5. **Cost:**: I like the cost argument, but one might also argue that the raw time series can be sub-sampled to answer the same questions. For a synthetic time series where the goal is to answer if something is periodic or not, the model doesn't need 1000 time points, and 100 uniformly sub-sampled points might suffice. 

### References
1. Das, Abhimanyu, et al. "A decoder-only foundation model for time-series forecasting." arXiv preprint arXiv:2310.10688 (2023).
2. Woo, Gerald, et al. "Unified training of universal time series forecasting transformers." arXiv preprint arXiv:2402.02592 (2024).
3. Goswami, Mononito, et al. "Moment: A family of open time-series foundation models." arXiv preprint arXiv:2402.03885 (2024).
4. Ansari, Abdul Fatir, et al. "Chronos: Learning the language of time series." arXiv preprint arXiv:2403.07815 (2024).
5. Cai, Yifu, et al. "Jolt: Jointly learned representations of language and time-series." Deep Generative Models for Health Workshop NeurIPS 2023. 2023.
6. Potosnak, Willa, et al. "Implicit Reasoning in Deep Time Series Forecasting." arXiv preprint arXiv:2409.10840 (2024).
7. Cai, Yifu, et al. "TimeSeriesExam: A time series understanding exam." arXiv preprint arXiv:2410.14752 (2024).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. What is the prevalence of plot vs. numerical text data in the typical available data? The benefit of this method seems dependent on currently having substantial, unused plotted time-series data. 
2. Using the vision encoder to interpret visual input improves performance over text-only input on only some of the pairs of real-world example task and foundation model. Why does the method perform well or poorly on various tasks, and why is there so much variation in the performance of the method between different foundation models (e.g., Gemini Flash 1.5 vs GPT4o in Figure 3)?
soundness: 2
strengths: 1. The comparison to human interpretation of numerical versus visual input helped to motivate why this enables a new mode of reasoning in foundation models.
2. On some synthetic and real data tasks, visual interpretation of the data does lead to a notable improvement over text interpretation on many pattern-recognition tasks.
summary: Multimodal foundation models have a native capability to “see,” but this capability is currently underutilized. This paper introduces a method for leveraging the vision encoders of foundation models to interpret visual time-series data displayed in plots, allowing the foundation models to “see” the data; since LLMs are currently not well-suited for interpreting large sequences of numbers in time-series data, introducing the capability of visual interpretation can improve a model’s performance on a variety of time-series understanding tasks. These tasks include the ability to reason about trends, the relationship between multiple time-series, cluster data, and perform other time-series understanding tasks. The authors show that in both synthetic and real-world tasks, the plot performance improves substantially over text performance and is cheaper, with no additional model training.
weaknesses: 1. It seems like the actual task of recognizing the types of visual patterns achieved by this method are not inherently novel – Section 2 shows that models are already able to parse plots and tables. So the contribution of the paper would be more clear if the authors provided (1) comparisons to actual timeseries baselines to show an improvement over prior work or (2) further justification on why one would ever feel constrained to using a foundation model to interpret time-series data (i.e., how often or why text interpretation would be the only alternative), which is currently unclear to me.
2. The related work makes it unclear what has been done on (1) visual parsing of plots in general and (2) making use of the native visual capabilities of foundation models. The discussion of related work could use more description of trends in past work and how the current work improved upon them directly. 
3. If the justification for why it is ok to have comparable performance on real-world tasks is that the vision encoder is more cost-efficient than the text encoder, more rigorous analysis of this improvement in cost efficiency seems necessary to show that the model provides any practical improvement in the real-world analysis. 
4. The description of the method and tasks are very vague – there does not seem to be enough detail in the main text to understand what the method actually does, or what the tasks are measuring (what is being measured, how are they being evaluated, what is the significance of each task). 

Other, more minor comments: 
1. Table 2 is hard to parse – the visualization in Figure 1 seems to convey more information and more effectively (such as true value rather than just relative performance). Could you remove the table and use the space to describe the tasks in more detail? 
2. Results across different noise/number of points rather than just aggregate performance for that task type would be interesting.
3. The real-world IMU data patterns seem considerably more complex than the pattern recognition tasks in the synthetic data. It would be interesting to see more description of how the results from synthetic and real-world tasks complement or relate to each other.

---

confidence: 2
contribution: 1
initial_rating: 1
presentation: 1
questions: - Could you provide references or empirical evidence to support the claim that the model's approach mirrors the human approach?
- Could you consider grouping the related work into subsections based on different approaches to make it clearer?
- Could you add a clear mathematical formulation to define the problem and solution, which would make it more rigorous?
- Could you consider how to enhance the depth of analysis to meet the standards of high-quality research for ICLR?
soundness: 1
strengths: - The paper introduces a creative idea of using vision encoders to understand time-series data by visualizing it as plots.
- Multimodal models for time-series understanding through visualization potentially reduce the token usage, leading to lower API costs.
- The experiments cover both synthetic and real-world datasets, providing an assessment of the proposed method's strengths and weaknesses across different contexts.
summary: This paper proposes a method for a multimodal foundation model to interpret time-series data by converting it into visual plots. 
This approach employs the vision encoders of models like GPT-4 and Gemini to see visual time-series data instead of processing it as text, which, according to the authors, results in improved performance and reduced computational costs.
The paper evaluates the proposed method through synthetic and real-world tasks, including fall detection and activity recognition.
weaknesses: - The core idea of using multimodal models to interpret time-series data via visual plots may not be sufficiently novel.
- The experimental methodology lacks rigorous justification. The paper does not provide enough comparative analysis with well-established baselines or detailed theoretical grounding to support its hypothesis, which weakens its scientific contribution.
- The related work section is not well-organized, making it difficult for readers to understand the context and significance of the contribution. This lack of structure makes it challenging to discern the novelty and value of the proposed method.
- The paper makes strong claims, such as the multimodal models' capability to mirror human understanding of visual plots, without adequate empirical backing or reference to previous research. Overstated claims with insufficient evidence can raise concerns about the validity of the conclusions.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How are multiple time series handled? Please add more details. For example, for the fall detection task on IMU data, is the input {img} a scatter plot containing six signals? Does a scatter plot containing multiple points from various variables lead to errors in the results? Including descriptions or figures about the input plots would enhance understanding.

2. Regarding the ablation experiments on visual elements in the matplotlib figure (including size, plot style, color palettes, markers, and plot components), which setting combination is reasonable or optimal? Is there a relationship between plot settings and the number of shown points? Since this paper focuses on plot representation, such a discussion is noteworthy and would be helpful for the follow-up work.

3. From the supplementary tables, it appears that results on cubic and periodic functions are worse on the plot-based approach. Some discussions on this finding are appreciated.

4. It might be beneficial to consider elaborating on the scalability of the approach for large-scale datasets.
soundness: 3
strengths: The paper tackles how to enhance interpretability and efficiency when dealing with time-series data using multimodal models. It demonstrates examples of identifying clusters, recogniting real-world patterns, detecting, identifying the correlation between functions, and so on. This paper contains a wealth of comparative experiments, ablation studies on visual elements, and provides prompt templates in the supplementary material to ensure reproducibility. Overall, I appreciate the contribution of visual time-series representations and believe it add significant value to the ICLR community.
summary: The primary goal of the paper is to propose a novel yet practical method for enhancing time-series data analysis by using vision tokens and multimodal foundation models. This approach avoids the limitations inherent in processing long sequences of floating-point numbers by transforming them into visual representations (plots).  It showcases that the plot-based approach performs much better on tasks where understanding the overall trend is crucial, while also being more resource-efficient, highlighting a practical solution rather than a purely theoretical one.
weaknesses: Lack of insights from a large number of experiments and ablation studies.

Meta-review: This paper has been evaluated by 4 knowledgeable reviewers and their initial opinions varied: 1 strong rejection, 1 marginal rejection and 1 marginal acceptance. The authors provided extensive rebuttals but that did not help the scores. Even though the key concept presented in the paper is relevant to ICLR and sufficiently well motivated, reviewers pointed out unjustified claims, missing important baselines and relevant literature, and some questioned the novelty of the core idea.

Decision: Reject