LETS-C: Leveraging Text Embedding for Time Series Classification
Authors: Rachneet Kaur, Zhen Zeng, Tucker Balch, Manuela Veloso
Accepted at: acl25

LETS-C: Leveraging Text Embedding for Time Series Classification

Rachneet Kaur    Zhen Zeng    Tucker Balch    Manuela Veloso
J.P. Morgan AI Research
{rachneet.kaur, zhen.zeng, tucker.balch, manuela.veloso}@jpmorgan.com

Abstract
Recent advancements in language modeling have shown promising results when applied to time series data. In particular, fine-tuning pre-trained large language models (LLMs) for time series classification tasks has achieved state-of-the-art (SOTA) performance on standard benchmarks. However, these LLM-based models have a significant drawback due to the large model size, with the number of trainable parameters in the millions. In this paper, we propose an alternative approach to leveraging the success of language modeling in the time series domain. Instead of fine-tuning LLMs, we utilize a text embedding model to embed time series and then pair the embeddings with a simple classification head composed of convolutional neural networks (CNN) and multilayer perceptron (MLP). We conducted extensive experiments on a well-established time series classification benchmark. We demonstrated LETS-C not only outperforms the current SOTA in classification accuracy but also offers a lightweight solution, using only 14.5% of the trainable parameters on average compared to the SOTA model. Our findings suggest that leveraging text embedding models to encode time series data, combined with a simple yet effective classification head, offers a promising direction for achieving high-performance time series classification while maintaining a lightweight model architecture.

1 Introduction
Time series classification (Bagnall et al., 2017; Abanda et al., 2019; Ismail Fawaz et al., 2019) has gained attention due to its wide-ranging applications in various domains, such as finance (Passalis et al., 2017), healthcare (Lipton et al., 2016), and activity recognition (Yang et al., 2015). The increasing availability of time series data has driven the need for efficient and accurate classification methods. Advances in natural language processing (NLP) and large language models (LLMs) (Achiam et al., 2023) have demonstrated strong promises in modeling sequential data (Achiam et al., 2023). Inspired by this, researchers have explored applying LLMs to time series via prompting (Gruver et al., 2024; Liu et al., 2024; Merrill et al., 2024; Chu et al., 2024) or fine-tuning pre-trained LLMs (Jin et al., 2023; Zhou et al., 2024), achieving state-of-the-art (SOTA) performance on well-established benchmarks for tasks including classification and forecasting.

However, LLMs for time series classification face a major drawback—their large size, often with billions of parameters, making them computationally expensive and impractical in resource-limited settings (Bommasani et al., 2021). Fine-tuning partially frozen pre-trained LLMs still requires millions of trainable parameters (Zhou et al., 2024). To mitigate this, we propose an alternative approach to leverage the success of language modeling in the time series domain. In particular, we propose a novel approach, LETS-C (Leveraging Text Embeddings for Time Series Classification), which utilizes off-the-shelf text embedding models instead of fine-tuning LLMs for time series classification. To the best of our knowledge, this work is the first to explore the potential of text embeddings in time series analysis, specifically classification, and demonstrate SOTA performance.

LETS-C combines text embeddings with a simple yet effective classification head composed of convolutional neural networks (CNNs) and a multilayer perceptron (MLP). By projecting time series data using text embedding models, we capture the intricate patterns and dependencies present in the temporal data. The embeddings and time series are then fed into the classification head, which learns to discriminate between different classes. Through extensive experiments on a well-established benchmark containing time series datasets from various domains, we demonstrated that LETS-C outperforms 27 baselines including the previous SOTA
method. Moreover, LETS-C is significantly more efficient, using much fewer trainable parameters than the previous SOTA. Our key contributions are:

• Text Embeddings for Time Series: We introduce LETS-C, the first work to leverage text embeddings for time series analysis, specifically for classification tasks.

• State-of-the-Art Performance: LETS-C achieves SOTA performance in classification accuracy on a well-established benchmark containing time series datasets across different domains, surpassing 27 baseline models.

• Computationally Lightweight: LETS-C is significantly more efficient, achieving higher accuracy while using much fewer trainable parameters (14.5%) than the existing SOTA method.

In addition, we conducted comprehensive analyses to showcase the effectiveness of LETS-C:

• Built-in Discriminative Power of Time Series Embeddings: We demonstrate the advantage of text embeddings for time series, showing that embeddings of time series from the same class are more similar than those from different classes, explaining the boost in classification accuracy.

• Generalization Across Various Text Embedding Models: LETS-C with different text embedding models consistently outperforms previous SOTA with much fewer trainable parameters, further validating the effectiveness of our approach.

• Optimizing Model Size with Minimal Accuracy Loss: LETS-C retains a high percentage of accuracy even as the classification model size shrinks considerably, enhancing computational efficiency with minimal impact on performance.

2 Related Work

We review the related works in three key areas: time series classification, the application of language models to time series data, and text embeddings. See Appendix A for an extended review.

Time Series Classification Time series classification has been an active research area for decades. Early methods focused on distance-based approaches (Abanda et al., 2019), such as Dynamic Time Warping (Berndt and Clifford, 1994) and distance kernels with Support Vector Machines (Kampouraki et al., 2008). Others used feature extraction with classifiers like eXtreme Gradient Boosting (XGBoost) (Chen and Guestrin, 2016) and LightGBM (Ke et al., 2017). Deep learning-based approaches, including CNNs (Wu et al., 2022a; Zhao et al., 2017), MLPs (Zhang et al., 2022a), and Recurrent Neural Networks (RNNs) like Long Short-Term Memory (LSTM) (Lai et al., 2018), later gained popularity for learning complex patterns and handling long sequences. More recently, Transformer-based models (Vaswani et al., 2017) have been adapted from NLP to time series (Nie et al., 2023; Zhou et al., 2022; Zhang et al., 2022b; Eldele et al., 2021; Wu et al., 2021; Zhou et al., 2021; Koval et al., 2024), leveraging self-attention for long-range dependencies. Additionally, unsupervised representation learning methods pre-train models with masked time series modeling to minimize reconstruction error, followed by fine-tuning for downstream tasks like classification (Franceschi et al., 2019; Tonekaboni et al., 2021; Yue et al., 2022; Eldele et al., 2021; Zerveas et al., 2021; Goswami et al., 2024). However, these complex models often require larger sizes and higher computational costs, particularly for training.

Language Models for Time Series The success of LLMs in NLP has inspired their application in time series analysis. Surveys (Zhang et al., 2024; Jiang et al., 2024) provide insights into key methodologies, challenges, and future directions. Recent studies explore integrating time series and language, including structured time series-text modeling (Khadanga et al., 2019; Deznabi et al., 2021), natural language descriptions of time series (Murakami et al., 2017; Jhamtani and Berg-Kirkpatrick, 2021; Fons et al., 2024), and various applications (Li et al., 2024a; Drinkall et al., 2024; Kawarada et al., 2024). Pre-trained LLMs have been used for time series forecasting via prompting (Gruver et al., 2024; Liu et al., 2024; Cao et al., 2023), while (Yu et al., 2023) explored their potential for generating explainable financial forecasts. Time-LLM (Jin et al., 2023) maps time series to the language embedding space, enabling LLM-based forecasting (Yang et al., 2021). More importantly, recent work, OneFitsAll (Zhou et al., 2024) achieved SOTA performance on various time series tasks by fine-tuning LLMs like GPT (Radford et al., 2019). In contrast, we propose leveraging text embeddings instead of directly using LLMs for time series analysis. Our approach establishes new SOTA performance on time series classification tasks with significantly fewer trainable parameters, making it a more efficient alternative.
Text Embeddings  Text embeddings are crucial in NLP, mapping words or sentences into dense vector spaces to capture semantic and syntactic information. Text embedding techniques range from word-level embeddings like Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) to contextualized embeddings from pre-trained models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b). In time series, unsupervised methods have been proposed for learning embeddings (Franceschi et al., 2019; Eldele et al., 2021; Zerveas et al., 2021; Yue et al., 2022; Sun et al., 2023; Goswami et al., 2024). However, large-scale datasets are scarcer in time series than in NLP, making it more challenging to learn embeddings from scratch. To our knowledge, we are the first to leverage well-trained text embeddings from NLP for time series classification.

3 Methodology

Given a time series classification dataset \( \mathcal{D} = \{ (\mathbf{x}_i, y_i)_{i=1}^N \} \), where \( \mathbf{x}_i \) is a multivariate time series sample, and \( y_i \in \{1, 2, \ldots, C\} \) is the corresponding class label, the goal is to learn a classifier that accurately predicts the class label \( \hat{y}_i \) for each time series. As illustrated in Figure 1, we propose LETS-C framework that harnesses text embeddings for time series classification tasks. Specifically, we 1) initially preprocess the time series data to normalize it, then 2) subsequently generate text embeddings from the normalized time series, 3) fuse embeddings with the time series data, and finally 4) feed the fused representation to a classification head that consists of CNNs and MLP. The choice of a simple classification head is intentional, we aim to test the hypothesis that the text embeddings of the time series provide sufficiently powerful representations for effective classification.

Preprocessing  To ensure consistent scales across all model inputs, each feature dimension of time series \( \mathbf{x}_i \) is min-max normalized to the range [0, 1] based on the minimum and maximum feature values of each dimension across the training data.

Text Embedding of Time Series  It is crucial to carefully format the preprocessed time series into strings before using text embeddings, as the tokenization of numerical strings can significantly affect the embeddings. (Liu and Low, 2023) has shown that tokenization impacts a model’s arithmetic abilities, with commonly used subword tokenization methods like Byte Pair Encoding (BPE) arbitrarily subdividing numbers, causing similar numbers to appear very differently. To mitigate this, we adopted a digit-space tokenization strategy, as suggested by (Gruver et al., 2024), where each digit is spaced, commas are added to separate time steps, and decimal points are omitted for fixed precision. For instance, a series to be formatted with a precision of two decimal places, such as 0.645, 6.45, 64.5, 645.0, would be converted to "6 4 , 6 4 5 , 6 4 5 0 , 6 4 5 0 0" prior to tokenization. This method ensures separate tokenization of each digit, preserving numerical integrity and enhancing pattern recognition in language models.

Next, we utilized the text-embedding-3-large model (OpenAI, 2024) to embed the formatted time series into the embedding space. It is important to note that we are using only the text embedding model, unlike the LLMs used in previous works (Jin et al., 2023; Zhou et al., 2024; Gruver et al., 2024). This model was selected for several reasons: it is highly ranked on the Massive Text Embedding Benchmark (MTEB) leaderboard (Muenninghoff et al., 2022; MTEB, 2024), known for its effectiveness in a variety of downstream tasks such as text search and sentence similarity; it supports a high maximum token length of 8191, accommodating our time series datasets; and it offers a high-dimensional vector space of 3072 dimensions. This large dimensionality captures a broad spectrum of temporal features, additionally the model allows for truncation to reduce dimensions as needed for specific applications without substantial loss of semantic information (Kusupati et al., 2022). This capability to truncate dimensions is particularly advantageous for optimizing efficiency while maintaining robust performance, aligning well with our goal of a lightweight framework.

In particular, we generate a text embedding for each dimension of \( \mathbf{x}_i \), transforming \( \mathbf{x}_i \in \mathbb{R}^{d \times l_x} \) into embeddings \( \mathbf{e}_i \in \mathbb{R}^{d \times l_e} \). Here, \( d \) is the number of dimensions in the multivariate time series, \( l_x \) is the series length, and \( l_e \) is the embedding length. Specifically, each dimension of \( \mathbf{x}_i \) (of length \( l_x \)) is first converted into a string, resulting in \( d \) separate strings. Each string is then independently embedded using an embedding model, as studies (Nie et al., 2023; Zhou et al., 2024; Goswami et al., 2024) suggest that channel-wise modeling is an effective strategy for multivariate time series. Our approach differs slightly depending on the type of
Figure 1: Left: Conventional text embedding. Right: Our proposed LETS-C framework normalizes time series and formats them to tokenize each digit separately. It then embeds the time series, fuses the embeddings with the original series via element-wise addition, and uses a simple CNN-MLP classification head to perform classification. Only the lightweight CNN and MLP head are trained. For illustration, we show a single-dimensional time series \( x_1 \in \mathbb{R}^{l_x} \), transformed into an embedding vector \( e_1 \in \mathbb{R}^{l_e} \).

embedding model used: 1) Proprietary model: For OpenAI’s text-embedding-3-large, the API directly returns an embedding vector of size \( l_e \) for each string; 2) Open-weight models: We extract the last token’s embedding from the model’s final hidden states to represent the entire sequence, effectively transforming the dimension from \( l_x \) to \( l_e \). This approach captures the final contextual representation by condensing the sequence into a single embedding derived from the last token. Embeddings for each dimension are then combined into a \( d \times l_e \) matrix, effectively transforming the input from \( d \times l_x \) to \( d \times l_e \). This transformation preserves independent contextual representations for each dimension while ensuring a consistent embedding structure. Note that the embedding computation in LETS-C is a one-time pass, then the computed embeddings of the time series can be stored and reused for further training, in contrast to the persistent computational cost caused by fine-tuning parts of LLMs. To validate the suitability of off-the-shelf text embeddings for time series classification, we compared cosine similarities between text embeddings of time series from the same and different classes. Intra-class similarity was consistently higher, demonstrating their effectiveness (see Section 5.2).

Fusing Embedding and Time Series Next, we fuse the embedding with the preprocessed time series using element-wise addition, applying zero padding for dimensional consistency. The time series embedding vector serves as a feature representation of temporal patterns present in the entire time series, and adding the raw time series integrates both information sources. This approach is well-established, particularly in ResNet (He et al., 2016), where raw data is combined with feature representations via shortcut connections. Additionally, fusing embeddings from different modalities is widely supported (Manzoor et al., 2023; Guo et al., 2019; Poria et al., 2018; Baltrušaitis et al., 2018; Jabri et al., 2016), with element-wise addition being a common and effective method for combining multimodal embeddings. This fusion allows both time series and text embeddings to contribute meaningfully to the final representation, improving the model’s overall performance. As shown in Section 5.3, using either the text embedding or the raw time series alone, as well as alternative fusion methods to addition, is less effective. While element-wise addition is an empirical design choice in our architecture, our experiments demonstrate that it achieves SOTA performance while maintaining minimal model complexity compared to alternatives such as concatenation, or fusion.

Lightweight Classification Head Lastly, we pair the fused time series representation with a simple classification head composed of 1D CNNs and an MLP for time series classification. The output from CNNs are flattened and fed through the final MLP
head with a softmax activation, which outputs a vector of the probabilities of each time series class. As detailed in the Appendix I, hyperparameter search determines the number of convolutional blocks in CNN, the number of linear layers in MLP, and the use of batch normalization, dropout, activation, and pooling. With a simple classification head, our model is lightweight and requires much fewer trainable parameters compared to the existing SOTA built on transformers, as detailed in Section 5.1.

4 Experimental Protocol and Details

Datasets and Evaluation Metrics  We evaluated LETS-C against a well-established benchmark for multivariate time series classification, commonly adopted for comparison in various studies (Zhou et al., 2024; Li et al., 2024b; Wu et al., 2022a; Zerveas et al., 2021). This benchmark, selected from the UEA Archive (Bagnall et al., 2018), is purposefully curated to cover challenging and diverse domains, including the following datasets: Ethanol-Concentration (Large et al., 2018), FaceDetection (Rik Henson, 2023), Handwriting (Shokoohi-Yekta et al., 2017), Heartbeat (Liu et al., 2016), JapaneseVowels (Kudo et al., 1999), PEMSSF (Cuturi, 2011), SelfRegulationSCP1 (Birbaumer et al., 1999), SelfRegulationSCP2 (Birbaumer et al., 1999), SpokenArabicDigits (Bedda and Hammami, 2010), and UWaveGestureLibrary (Liu et al., 2009). This commonly adopted benchmark offers a comprehensive testing environment, with multivariate dimensions ranging from 3 to 963, time series lengths up to 1751, and up to 26 classes. See Appendix B for details on each dataset.

To assess the classifiers, we used metrics including classification accuracy and AvgWins. AvgWins is defined as the average number of times that a method outperforms other methods across benchmarked datasets, with ties also being counted towards this average. Additionally, we analyzed models’ computational efficiency in terms of trainable model parameters.

Baselines  We included 27 baseline models to ensure a comprehensive comparison. We utilize the same supervised learning baselines outlined by (Zhou et al., 2024; Wu et al., 2022a), namely: Classical methods: 1) Dynamic Time Warping (DTW) (Berndt and Clifford, 1994), 2) eXtreme Gradient Boosting (XGBoost) (Chen and Guestrin, 2016), and 3) RandOm Convolutional KErnel Transform (ROCKET) (Dempster et al., 2020); MLP-based methods: 4) LightTS (Zhang et al., 2022a), and 5) DLinear (Zeng et al., 2023); RNN-based models: 6) Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997), 7) Long- and Short-term Time-series Network (LSTNet) (Lai et al., 2018), and 8) Linear State Space Layer (LSSL) (Gu et al., 2021); CNN-based models: 9) Temporal Convolutional Network (TCN) (Franceschi et al., 2019), and 10) TimesNet (Wu et al., 2022a); Transformer-based models: 11) Transformer (Vaswani et al., 2017), 12) Reformer (Kitaev et al., 2020), 13) Informer (Zhou et al., 2021), 14) Pyraformer (Liu et al., 2021), 15) Autoformer (Wu et al., 2021), 16) Non-stationary Transformer (Liu et al., 2022), 17) FEDformer (Zhou et al., 2022), 18) ETSformer (Woo et al., 2022), 19) Flowformer (Wu et al., 2022b), 20) Patch Time Series Transformer (PatchTST) (Nie et al., 2023); and LLM-based model: 21) OneFitsAll (Zhou et al., 2024). For details, see Appendix C. Note that some of these methods were adapted from forecasting to classification tasks, without altering the core design of the models (Appendix D). Additionally, we included the unsupervised representation learning baselines outlined by (Goswami et al., 2024), namely: CNN-based methods: 22) T-Loss (Franceschi et al., 2019), 23) Temporal Neighborhood Coding (TNC) (Tonkaboni et al., 2021), 24) TS2Vec (Yue et al., 2022); Transformer-based models: 25) Time Series representation learning framework via Temporal and Contextual Contrasting (TS-TCC) (Eldele et al., 2021), 26) Time Series Transformer (TST) (Zerveas et al., 2021), and 27) MOMENT (Goswami et al., 2024). For details, refer to Appendices E (unsupervised baselines) and F (adapting unsupervised models for classification). Reproduction details for baselines and LETS-C implementation details are in Appendices G and H, respectively.

5 Results and Analysis

5.1 Performance and Efficiency

Comparison to State-of-the-art  Table 1 and Figure 3 present a comparative analysis of LETS-C against 27 baseline models on the benchmark introduced above1. We observe that LETS-C con-

1To maintain consistency with prior works, we use the same settings as in TimesNet and OneFitsAll, where the UEA benchmark includes only training and testing sets (with no validation set). Consequently, the reported performance reflects the model’s upper bound measurement. We collect supervised baseline model results from (Zhou et al., 2024) and (Wu et al.,
Table 1: Comparison of classification accuracy (%) and AvgWins (%). Red: Best, Blue: Second best. Abbreviations: EC: Ethanol Concentration, FD: Face Detection, HW: Handwriting, HB: Heartbeat, JV: Japanese Vowels, SCP1: Self-Regulation SCP1, SCP2: Self-Regulation SCP2, SAD: Spoken Arabic Digits, UW: UWave Gesture Library.

<table>
  <tr>
    <th>Model/Dataset</th>
    <th>EC</th>
    <th>FD</th>
    <th>HW</th>
    <th>HB</th>
    <th>JV</th>
    <th>PEMS-SF</th>
    <th>SCP1</th>
    <th>SCP2</th>
    <th>SAD</th>
    <th>UW</th>
    <th>Average ↑</th>
    <th>AvgWins % ↑</th>
  </tr>
  <tr>
    <th colspan="13">Supervised Learning Methods</th>
  </tr>
  <tr>
    <td>Classical methods</td>
    <td>DTW (Bertold and Clifford, 1994)</td>
    <td>32.3</td>
    <td>52.9</td>
    <td>28.6</td>
    <td>71.7</td>
    <td>94.9</td>
    <td>71.1</td>
    <td>77.7</td>
    <td>53.9</td>
    <td>96.3</td>
    <td>90.3</td>
    <td>66.97</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>XGBoost (Chen and Guestrin, 2016)</td>
    <td>43.7</td>
    <td>63.3</td>
    <td>15.8</td>
    <td>73.2</td>
    <td>86.5</td>
    <td>98.3</td>
    <td>84.6</td>
    <td>48.9</td>
    <td>69.6</td>
    <td>75.9</td>
    <td>65.98</td>
    <td>10%</td>
  </tr>
  <tr>
    <td></td>
    <td>ROCKET (Dempster et al., 2020)</td>
    <td>45.2</td>
    <td>64.7</td>
    <td>58.8</td>
    <td>75.6</td>
    <td>96.2</td>
    <td>75.1</td>
    <td>90.8</td>
    <td>53.3</td>
    <td>71.2</td>
    <td>94.4</td>
    <td>72.53</td>
    <td>20%</td>
  </tr>
  <tr>
    <td>MLP</td>
    <td>LightTS (Zhang et al., 2022a)</td>
    <td>29.7</td>
    <td>67.5</td>
    <td>26.1</td>
    <td>75.1</td>
    <td>96.2</td>
    <td>88.4</td>
    <td>89.8</td>
    <td>51.1</td>
    <td>100</td>
    <td>80.3</td>
    <td>70.42</td>
    <td>10%</td>
  </tr>
  <tr>
    <td></td>
    <td>DLinear (Zeng et al., 2023)</td>
    <td>32.6</td>
    <td>68</td>
    <td>27</td>
    <td>75.1</td>
    <td>96.2</td>
    <td>75.1</td>
    <td>87.3</td>
    <td>50.5</td>
    <td>81.4</td>
    <td>82.1</td>
    <td>67.53</td>
    <td>0%</td>
  </tr>
  <tr>
    <td>RNN</td>
    <td>LSTM (Hochreiter and Schmidhuber, 1997)</td>
    <td>32.3</td>
    <td>57.7</td>
    <td>15.2</td>
    <td>72.2</td>
    <td>79.7</td>
    <td>39.9</td>
    <td>68.9</td>
    <td>46.6</td>
    <td>31.9</td>
    <td>41.2</td>
    <td>48.56</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>LSTNet (Lai et al., 2018)</td>
    <td>39.9</td>
    <td>65.7</td>
    <td>25.8</td>
    <td>77.1</td>
    <td>98.1</td>
    <td>86.7</td>
    <td>84</td>
    <td>52.8</td>
    <td>100</td>
    <td>87.8</td>
    <td>71.79</td>
    <td>10%</td>
  </tr>
  <tr>
    <td></td>
    <td>LSSL (Gu et al., 2021)</td>
    <td>31.1</td>
    <td>66.7</td>
    <td>24.6</td>
    <td>72.7</td>
    <td>98.4</td>
    <td>86.1</td>
    <td>90.8</td>
    <td>52.2</td>
    <td>100</td>
    <td>85.9</td>
    <td>70.85</td>
    <td>10%</td>
  </tr>
  <tr>
    <td>CNN</td>
    <td>TCN (Franceschi et al., 2019)</td>
    <td>28.9</td>
    <td>52.8</td>
    <td>53.5</td>
    <td>75.6</td>
    <td>98.9</td>
    <td>68.8</td>
    <td>84.6</td>
    <td>55.6</td>
    <td>95.6</td>
    <td>88.4</td>
    <td>70.25</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>TimesNet (Wu et al., 2022a)</td>
    <td>35.7</td>
    <td>68.6</td>
    <td>32.1</td>
    <td>78</td>
    <td>98.4</td>
    <td>89.6</td>
    <td>91.8</td>
    <td>57.2</td>
    <td>99</td>
    <td>85.3</td>
    <td>73.57</td>
    <td>0%</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>Transformer (Vaswani et al., 2017)</td>
    <td>32.7</td>
    <td>67.3</td>
    <td>32</td>
    <td>76.1</td>
    <td>98.7</td>
    <td>82.1</td>
    <td>92.2</td>
    <td>53.9</td>
    <td>98.4</td>
    <td>85.6</td>
    <td>71.9</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>Reformer (Kitaev et al., 2020)</td>
    <td>31.9</td>
    <td>68.6</td>
    <td>27.4</td>
    <td>77.1</td>
    <td>97.8</td>
    <td>82.7</td>
    <td>90.4</td>
    <td>56.7</td>
    <td>97</td>
    <td>85.6</td>
    <td>71.52</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>Informer (Zhou et al., 2021)</td>
    <td>31.6</td>
    <td>67</td>
    <td>32.8</td>
    <td>80.5</td>
    <td>98.9</td>
    <td>81.5</td>
    <td>90.1</td>
    <td>53.3</td>
    <td>100</td>
    <td>85.6</td>
    <td>72.13</td>
    <td>20%</td>
  </tr>
  <tr>
    <td></td>
    <td>Pyraformer (Liu et al., 2021)</td>
    <td>30.8</td>
    <td>65.7</td>
    <td>29.7</td>
    <td>75.4</td>
    <td>98.4</td>
    <td>83.2</td>
    <td>88.1</td>
    <td>53.3</td>
    <td>99.6</td>
    <td>83.4</td>
    <td>70.75</td>
    <td>10%</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>Non-stationary Transformer (Liu et al., 2021)</td>
    <td>31.6</td>
    <td>67.4</td>
    <td>31.7</td>
    <td>76.6</td>
    <td>98.2</td>
    <td>82.7</td>
    <td>94</td>
    <td>54.6</td>
    <td>100</td>
    <td>87.5</td>
    <td>71.07</td>
    <td>10%</td>
  </tr>
  <tr>
    <td></td>
    <td>FEDformer (Zhou et al., 2022)</td>
    <td>31.2</td>
    <td>66</td>
    <td>28</td>
    <td>73.7</td>
    <td>98.4</td>
    <td>80.9</td>
    <td>88.7</td>
    <td>54.4</td>
    <td>100</td>
    <td>85.3</td>
    <td>70.66</td>
    <td>10%</td>
  </tr>
  <tr>
    <td></td>
    <td>ETSFomer (Woo et al., 2022)</td>
    <td>28.1</td>
    <td>66.3</td>
    <td>32.5</td>
    <td>71.2</td>
    <td>95.9</td>
    <td>86</td>
    <td>89.6</td>
    <td>55</td>
    <td>100</td>
    <td>85</td>
    <td>70.96</td>
    <td>10%</td>
  </tr>
  <tr>
    <td></td>
    <td>Flowformer (Wu et al., 2022b)</td>
    <td>33.8</td>
    <td>67.6</td>
    <td>33.8</td>
    <td>77.6</td>
    <td>98.9</td>
    <td>83.8</td>
    <td>92.5</td>
    <td>56.1</td>
    <td>98.8</td>
    <td>86.6</td>
    <td>72.95</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>PatchTST (Nie et al., 2023)</td>
    <td>26.2</td>
    <td>68.5</td>
    <td>25.9</td>
    <td>66.8</td>
    <td>90.7</td>
    <td>87.9</td>
    <td>85.7</td>
    <td>53.3</td>
    <td>97.2</td>
    <td>85.0</td>
    <td>69.25</td>
    <td>0%</td>
  </tr>
  <tr>
    <td>LLM</td>
    <td>OneFitsAll (Zhou et al., 2024)</td>
    <td>34.2</td>
    <td>69.2</td>
    <td>32.7</td>
    <td>77.2</td>
    <td>98.6</td>
    <td>87.9</td>
    <td>93.2</td>
    <td>59.4</td>
    <td>99.2</td>
    <td>88.1</td>
    <td>73.97</td>
    <td>20%</td>
  </tr>
  <tr>
    <th colspan="13">Unsupervised Representation Learning Methods</th>
  </tr>
  <tr>
    <td>CNN</td>
    <td>T-Loss (Franceschi et al., 2019)</td>
    <td>20.5</td>
    <td>51.3</td>
    <td>45.1</td>
    <td>74.1</td>
    <td>98.9</td>
    <td>67.6</td>
    <td>84.3</td>
    <td>53.9</td>
    <td>90.5</td>
    <td>87.5</td>
    <td>67.37</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>TNC (Tonekaboni et al., 2021)</td>
    <td>29.7</td>
    <td>53.6</td>
    <td>24.9</td>
    <td>74.6</td>
    <td>97.8</td>
    <td>69.9</td>
    <td>79.9</td>
    <td>55.0</td>
    <td>93.4</td>
    <td>75.9</td>
    <td>65.47</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>TS2Vec (Yue et al., 2022)</td>
    <td>30.8</td>
    <td>50.1</td>
    <td>51.5</td>
    <td>68.3</td>
    <td>98.4</td>
    <td>68.2</td>
    <td>81.2</td>
    <td>57.8</td>
    <td>98.8</td>
    <td>90.6</td>
    <td>69.57</td>
    <td>0%</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>TS-TCC (Eldele et al., 2021)</td>
    <td>28.5</td>
    <td>54.4</td>
    <td>49.8</td>
    <td>75.1</td>
    <td>93.0</td>
    <td>73.4</td>
    <td>82.3</td>
    <td>53.3</td>
    <td>97.0</td>
    <td>75.3</td>
    <td>68.21</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>TST (Zerveas et al., 2021)</td>
    <td>26.2</td>
    <td>53.4</td>
    <td>22.5</td>
    <td>74.6</td>
    <td>97.8</td>
    <td>74.0</td>
    <td>75.4</td>
    <td>55.0</td>
    <td>92.3</td>
    <td>57.5</td>
    <td>62.87</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td>MOMENT (Goswami et al., 2024)</td>
    <td>35.7</td>
    <td>63.3</td>
    <td>30.8</td>
    <td>72.2</td>
    <td>71.6</td>
    <td>89.6</td>
    <td>84.0</td>
    <td>47.8</td>
    <td>98.1</td>
    <td>90.9</td>
    <td>68.4</td>
    <td>0%</td>
  </tr>
  <tr>
    <td></td>
    <td><b>LETS-C (Ours)</b></td>
    <td>52.9</td>
    <td>68.9</td>
    <td>23.8</td>
    <td>78</td>
    <td>99.2</td>
    <td>93.1</td>
    <td>93.2</td>
    <td>62.8</td>
    <td>99.2</td>
    <td>90.6</td>
    <td><b>76.17</b></td>
    <td><b>40%</b></td>
  </tr>
</table>

Table 2: Comparison of trainable parameters (millions) for LETS-C vs. OneFitsAll. The Ratio (%) = 100 × # of trainable parameters in LETS-C / # of trainable parameters in OneFitsAll, quantifying the efficiency of LETS-C relative to OneFitsAll.

<table>
  <tr>
    <th>Model/Dataset</th>
    <th>EC</th>
    <th>FD</th>
    <th>HW</th>
    <th>HB</th>
    <th>JV</th>
    <th>PEMS-SF</th>
    <th>SCP1</th>
    <th>SCP2</th>
    <th>SAD</th>
    <th>UW</th>
    <th>Average ↓</th>
  </tr>
  <tr>
    <td>Trainable parameters</td>
    <td>LETS-C (Ours)</td>
    <td>0.28</td>
    <td>0.003</td>
    <td>0.15</td>
    <td>0.04</td>
    <td>0.14</td>
    <td>0.56</td>
    <td>0.30</td>
    <td>0.33</td>
    <td>0.14</td>
    <td>0.26</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td></td>
    <td>OneFitsAll</td>
    <td>1.42</td>
    <td>2.37</td>
    <td>1.73</td>
    <td>2.03</td>
    <td>1.32</td>
    <td>10.23</td>
    <td>0.98</td>
    <td>1.04</td>
    <td>1.82</td>
    <td>1.0</td>
    <td>2.39</td>
  </tr>
  <tr>
    <td><b>Ratio (%)</b></td>
    <td>19.89</td>
    <td>0.16</td>
    <td>8.89</td>
    <td>2.28</td>
    <td>11.19</td>
    <td>5.51</td>
    <td>30.83</td>
    <td>32.06</td>
    <td>7.77</td>
    <td>26.21</td>
    <td><b>14.48</b></td>
  </tr>
</table>

sistently demonstrates robust performance across all datasets, achieving the highest average accuracy of 76.16% and AvgWins of 40%, compared to 27 benchmark models. This includes the most recent SOTA model OneFitsAll (accuracy: 73.97%, AvgWins: 20%) and an older SOTA TimesNet (accuracy: 73.57%, AvgWins: 0%). Notably, LETS-C surpasses OneFitsAll on six out of ten datasets by a significant margin. LETS-C is particularly effective on challenging datasets like PEMS-SF and EthanolConcentration (EC). PEMS-SF is characterized by exceptionally high dimensionality with 963 features, and EC contains an extremely long time series at length of 1751. These results showcase the competitive edge of LETS-C against the previous SOTA methods, thus establishing a new benchmark for time series classification.

Computational Efficiency Next, we aim to assess how well LETS-C balances performance with computational efficiency, which is crucial for usage in resource-constrained environments. Table 2 provides a detailed analysis of the trainable parameters associated with LETS-C compared to the previous SOTA model, OneFitsAll. Our method achieved higher performance with only 14.48% of the trainable model parameters on average, compared to OneFitsAll. Despite the advantage of OneFitsAll over other leading models like TimesNet and FEDformer on its reduced parameter count, OneFitsAll still requires much more trainable parameters

2022a), except for PatchTST, which is based on our own reproduction. Additionally, we gather unsupervised representation learning baseline results from (Goswami et al., 2024) and (Yue et al., 2022).
than our approach. We show that LETS-C offers a lightweight approach to time series classification while achieving the SOTA accuracy. It’s important to note that in LETS-C, the text embedding computation is a one-time operation, unlike models like OneFitsAll, which continue to incur computational costs while fine-tuning partially frozen LLMs.

5.2 Effectiveness of LETS-C

Built-in Discriminative Power of Text Embeddings on Time Series To analyze text embeddings extracted from time series and assess their effectiveness, we compared embeddings of time series from the same and different classes. This study revealed the built-in power of text embeddings to readily distinguish time series from different classes. Specifically, we computed the average cosine similarity for time series pairs within the same class and across different classes. We average similarities from multiple channels to handle multivariate time series effectively. This approach helps us quantify how closely time series in the embedding space are related, both within each class (intra-class) and across distinct classes (inter-class). Figure 2 (left) visualizes these embedding similarities through heatmaps. The heatmaps are scaled using min-max normalization, where warmer colors in the heatmap represent higher similarities and cooler shades indicate lower similarities. Diagonal entries show intra-class similarities, while off-diagonal entries reveal inter-class similarities. We observe that intra-class similarity consistently exceeds inter-class similarity, demonstrating that text embeddings effectively retain and convey significant information from the underlying time series.

Generalization Across Various Text Embedding Models To assess the generalization of our approach across various text embedding models beyond text-embedding-3-large, we evaluate LETS-C using three additional embeddings: e5-mistral-7b-instruct (Wang et al., 2024), gte-large-en-v1.5 (Li et al., 2023), and nomic-embed-text-v1 (Nussbaum et al., 2024). These models were selected for their high rankings in the MTEB overall leaderboard and their variations in embedding dimensions, maximum token lengths, and model sizes. Details on embedding models are in Appendix N. Table 3 presents detailed accuracy metrics and trainable parameters for these various embedding models in the LETS-C framework. We observe that LETS-C consistently outperforms current baselines in terms of average classification accuracy and AvgWins, while utilizing only a fraction of the trainable model parameters across all explored text embedding models. Specifically, text-embedding-3-large achieves an average accuracy of 76.17%, while e5-mistral-7b-instruct, gte-large-en-v1.5, & nomic-embed-text-v1 achieve accuracies of 74.89%, 76.02%, and 75.12% respectively. These results not only surpass the previous SOTA accuracy of 73.97% but also require significantly fewer trainable model parameters—just 14.48%, 15.62%, 9.24%, and 5.31% compared to OneFitsAll. Among these text embedding variations, higher text embedding dimensionality leads to more trainable parameters on average. e5-mistral-7b-instruct requires the most trainable parameters (0.25M) due to its 4096 dimensions, followed by text-embedding-3-large (3072D, 0.22M), gte-large-en-v1.5 (1024D, 0.19M), and nomic-embed-text-v1 (768D, 0.09M). Consequently, our approach generalizes across diverse text embedding models, demonstrates superior performance, with the benefit of being lightweight.

Optimizing Model Size with Minimal Accuracy Loss Next, we examine the trade-offs between model accuracy and model size in our approach. To vary model size, we adjust the number of linear and convolution layers in the classification head, ranging from 1 to 5 layers each, creating model variants of different sizes. Across all datasets, we find that significant reductions in model parameters result in only a minimal loss of accuracy. Figure 2 (right) illustrates this trade-off, showing accuracy and parameter retention relative to the optimal performance of LETS-C (see Tables 1 and 2 for reference optimal values). The results highlight the efficiency of LETS-C, demonstrating its ability to maintain high accuracy with significantly fewer parameters across all datasets. While the trade-off between model size and accuracy is data-dependent, in general, substantial parameter reductions lead to only slight accuracy decreases. For detailed results, see Appendix O and Table 9. This analysis confirms that while reducing parameters can lower accuracy, the impact remains manageable, making LETS-C suitable for applications requiring efficient inference.
Figure 2: Left: Heatmaps illustrating within- and between-class cosine similarities of text embeddings derived from the training time series in JapaneseVowels (far left) with 9 classes and SpokenArabicDigits (middle) with 10 classes. Right: Trade-off between the percentage of accuracy retention and model parameter retention relative to LETS-C’s optimal values. The optimal LETS-C accuracy (%) and parameters (M) for each dataset are in the legend.

Table 3: Comparison of LETS-C with various embedding models against OneFitsAll. Performances surpassing OneFitsAll are in bold, with the best in Red. AvgWins scores above 50% indicate consistent superiority, calculated as 1 for outperforming OneFitsAll and 0 otherwise. Ratio (\%) = \( 100 \times \frac{\# \text{ of trainable parameters in LETS-C}}{\# \text{ of trainable parameters in OneFitsAll}} \) measures computational efficiency relative to OneFitsAll.

<table>
  <tr>
    <th rowspan="2">Method/Dataset</th>
    <th colspan="8">Accuracy ↑</th>
    <th rowspan="2">Average ↑</th>
    <th rowspan="2">AvgWins % ↑</th>
  </tr>
  <tr>
    <th>EC</th><th>FD</th><th>HW</th><th>HB</th><th>JV</th><th>PEMS-SF</th><th>SCP1</th><th>SCP2</th><th>SAD</th><th>UW</th>
  </tr>
  <tr>
    <td>OneFitsAll</td>
    <td>34.2</td><td>69.2</td><td>32.7</td><td>77.2</td><td>98.6</td><td>87.9</td><td>93.2</td><td>59.4</td><td>99.2</td><td>88.1</td>
    <td>73.97</td><td>—</td>
  </tr>
  <tr>
    <td>text-embedding-3-large</td>
    <td>52.9</td><td>68.9</td><td>23.8</td><td>78</td><td>99.2</td><td>93.1</td><td>93.2</td><td>62.8</td><td>99.2</td><td>90.6</td>
    <td>76.17</td><td>80%</td>
  </tr>
  <tr>
    <td>LETS-C<br>e5-mistral-7b-instruct</td>
    <td>55.5</td><td>68.7</td><td>23.3</td><td>77.6</td><td>99.2</td><td>84.4</td><td>93.9</td><td>59.4</td><td>98.5</td><td>88.4</td>
    <td>74.89</td><td>60%</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5</td>
    <td>57.8</td><td>68.8</td><td>24.7</td><td>77.6</td><td>98.4</td><td>91.3</td><td>94.2</td><td>60</td><td>99</td><td>88.4</td>
    <td>76.02</td><td>60%</td>
  </tr>
  <tr>
    <td>nomic-embed-text-v1</td>
    <td>52.9</td><td>68</td><td>24.8</td><td>76.6</td><td>99.2</td><td>88.4</td><td>93.9</td><td>59.4</td><td>98.6</td><td>89.4</td>
    <td>75.12</td><td>60%</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Method/Dataset</th>
    <th colspan="8">Trainable Parameters (M) ↓</th>
    <th rowspan="2">Average ↓</th>
  </tr>
  <tr>
    <th>EC</th><th>FD</th><th>HW</th><th>HB</th><th>JV</th><th>PEMS-SF</th><th>SCP1</th><th>SCP2</th><th>SAD</th><th>UW</th>
  </tr>
  <tr>
    <td>OneFitsAll</td>
    <td>1.42</td><td>2.37</td><td>1.73</td><td>2.03</td><td>1.32</td><td>10.23</td><td>0.98</td><td>1.04</td><td>1.82</td><td>1.0</td>
    <td>2.39</td>
  </tr>
  <tr>
    <td>text-embedding-3-large</td>
    <td>0.28</td><td>0.003</td><td>0.15</td><td>0.04</td><td>0.14</td><td>0.56</td><td>0.30</td><td>0.33</td><td>0.14</td><td>0.26</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>Ratio %</td>
    <td>19.89</td><td>0.16</td><td>8.89</td><td>2.28</td><td>11.19</td><td>5.51</td><td>30.83</td><td>32.06</td><td>7.77</td><td>26.21</td>
    <td>14.48</td>
  </tr>
  <tr>
    <td>LETS-C<br>e5-mistral-7b-instruct</td>
    <td>0.13</td><td>0.40</td><td>0.39</td><td>0.17</td><td>0.16</td><td>0.30</td><td>0.24</td><td>0.24</td><td>0.33</td><td>0.16</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>Ratio %</td>
    <td>9.48</td><td>16.93</td><td>22.78</td><td>8.54</td><td>12.55</td><td>2.97</td><td>25.06</td><td>23.59</td><td>18.39</td><td>15.93</td>
    <td>15.62</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5</td>
    <td>0.18</td><td>0.31</td><td>0.31</td><td>0.12</td><td>0.04</td><td>0.56</td><td>0.03</td><td>0.07</td><td>0.22</td><td>0.09</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>Ratio %</td>
    <td>12.91</td><td>13.44</td><td>18.27</td><td>6.11</td><td>3.36</td><td>5.51</td><td>3.77</td><td>7.65</td><td>12.34</td><td>9.00</td>
    <td>9.24</td>
  </tr>
  <tr>
    <td>nomic-embed-text-v1</td>
    <td>0.03</td><td>0.03</td><td>0.36</td><td>0.07</td><td>0.05</td><td>0.19</td><td>0.02</td><td>0.10</td><td>0.06</td><td>0.02</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>Ratio %</td>
    <td>2.21</td><td>1.31</td><td>20.99</td><td>3.58</td><td>3.99</td><td>1.85</td><td>3.02</td><td>10.03</td><td>3.34</td><td>2.78</td>
    <td>5.31</td>
  </tr>
</table>

5.3 Additional Analysis

Ablation Study To empirically evaluate the benefits of fusing text embeddings with time series over variants using only one modality, we conduct an ablation study. As shown in Appendix P and Table 10, we observe that the combination of both embeddings and time series achieves the highest average accuracy, at 76.17%, and the best number of AvgWins, compared to the ablated versions. These demonstrate the significant performance gains from fusing embeddings with time series data, which are essential for optimal model accuracy.

Alternative Methods for Fusing Time Series with Embeddings We explore two additional methods for fusing time series and embeddings beyond simple addition. The first method involves a Fusion network that first processes embeddings and time series data through convolutional and dense layers in two separate branches, then merges the features from both branches into a final dense network. The second method employs Concatenation, where the time series and embeddings are concatenated and processed through a lightweight classification head. Despite cross-attention being another alternative for fusing different modalities, we didn’t include it in this study due to the computational complexity it adds to the model. Appendix Q and Table 11 present details on classification accuracy and trainable model parameters for these variations. We observe that the addition approach in the LETS-C architecture achieves the highest average classification accuracy (76.11%) compared to the fusion network (73.40%) and concatenation (74.22%). It
also records the best AvgWins at 70%. Further, the number of parameters increases with both the concatenation and fusion network approaches, resulting in additional complexity compared to the addition method. As our goal was to develop a lightweight model, we opted for the addition approach due to its simpler parameter structure.

6 Conclusion

We introduced LETS-C, a novel approach that leverages text embeddings for time series classification. To our knowledge, this is the first exploration of using off-the-shelf text embeddings in time series analysis, particularly for classification. By projecting time series through text embedding models and employing a simple yet effective classification head, LETS-C achieves SOTA performance on a well-established benchmark covering various domains, surpassing 27 baselines. Additionally, LETS-C is significantly more lightweight than previous SOTA methods, achieving higher accuracy with fewer trainable parameters. Our comprehensive analysis highlights LETS-C’s robustness across different text embedding models, the advantage of using text embeddings for time series classification, and the trade-off between accuracy and model size.

7 Limitations and Broader Perspective

Limitations and future work: Our approach has a few key limitations. One is the maximum sequence length constraint of text embedding models, which restricts the handling of longer time series and may necessitate truncation or downsampling, potentially affecting the capture of long-term dependencies. Additionally, while we adhered to established benchmarks for consistency, we did not evaluate our method on a newly constructed or strictly held-out dataset to check for potential data leakage into the OpenAI embeddings. Testing on a novel dataset could further validate our results and ensure that the approach’s effectiveness is not influenced by pre-existing knowledge. Another limitation is the monetary cost associated with using OpenAI’s text embeddings, which we discuss in more detail in Appendix K.

Future research could explore alternative time series tokenization strategies, as well as extensions to other time series tasks. We believe our findings will inspire further exploration of text embeddings in the time series domain, paving the way for more powerful and efficient methods for various time series tasks.

Broader perspective: The embeddings generated may lack interpretability, making it difficult to understand model decisions in high-stakes applications. Additionally, the resources accompanying this study will be responsibly released for research purposes only.

Datasets: The benchmarks used in this study are publicly available from the UEA Archive (Bagnall et al., 2018) and were curated by previous research. Specifically, they include the following datasets: EthanolConcentration (Large et al., 2018), FaceDetection (Rik Henson, 2023), Handwriting (Shokoohi-Yekta et al., 2017), Heartbeat (Liu et al., 2016), JapaneseVowels (Kudo et al., 1999), PEMS-SF (Cuturi, 2011), SelfRegulation-SCP1 (Birbaumer et al., 1999), SelfRegulation-SCP2 (Birbaumer et al., 1999), SpokenArabicDigits (Bedda and Hammami, 2010), and UWaveGestureLibrary (Liu et al., 2009). We abide by their terms of use.

Acknowledgments

This paper was prepared for informational purposes by the Artificial Intelligence Research group of JP-Morgan Chase & Co and its affiliates (“J.P. Morgan”) and is not a product of the Research Department of J.P. Morgan. J.P. Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.

References

Amaia Abanda, Usue Mori, and Jose A Lozano. 2019. A review on distance based time series classification. Data Mining and Knowledge Discovery, 33(2):378–412.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. 2018. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075.

Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. 2017. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data mining and knowledge discovery, 31:606–660.

Claus Bahlmann, Bernard Haasdonk, and Hans Burkhardt. 2002. Online handwriting recognition with support vector machines-a kernel approach. In Proceedings eighth international workshop on frontiers in handwriting recognition, pages 49–54. IEEE.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271.

Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443.

Mouldi Bedda and Nacereddine Hammami. 2010. Spoken Arabic Digit. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C52C9Q.

Donald J Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd international conference on knowledge discovery and data mining, pages 359–370.

Niels Birbaumer, Nirn Ghanayim, Thilo Hinterberger, Iver Iversen, Boris Kotchoubev, Andrea Kübler, Juri Perelmouter, Edward Taub, and Herta Flor. 1999. A spelling device for the paralysed. Nature, 398(6725):297–298.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. 2023. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948.

Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794.

Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. TimeBench: A comprehensive evaluation of temporal reasoning abilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pages 1204–1228, Bangkok, Thailand. Association for Computational Linguistics.

Marco Cuturi. 2011. Fast global alignment kernels. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 929–936.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359.

Angus Dempster, François Petitjean, and Geoffrey I Webb. 2020. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5):1454–1495.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Iman Deznaei, Mohit Iyyer, and Madalina Fiterau. 2021. Predicting in-hospital mortality by combining clinical notes with time-series data. In Findings of the association for computational linguistics: ACL-IJCNLP 2021, pages 4026–4031.

Felix Drinkall, Eghbal Rahimikia, Janet Pierrehumbert, and Stefan Zohren. 2024. Time machine GPT. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3281–3292, Mexico City, Mexico. Association for Computational Linguistics.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. 2021. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization.

Elizabeth Fons, Rachneet Kaur, Soham Palande, Zhen Zeng, Tucker Balch, Manuela Veloso, and Svitlana Vyetrenko. 2024. Evaluating large language models on time series feature understanding: A comprehensive taxonomy and benchmark. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 21598–21634, Miami, Florida, USA. Association for Computational Linguistics.

Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. 2019. Unsupervised scalable representation
learning for multivariate time series. Advances in neural information processing systems, 32.

Jonas Geiping and Tom Goldstein. 2023. Cramming: Training a language model on a single gpu in one day. In International Conference on Machine Learning, pages 11117–11143. PMLR.

Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. 2017. The reversible residual network: Backpropagation without storing activations. Advances in neural information processing systems, 30.

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. 2024. Moment: A family of open time-series foundation models. In Forty-first International Conference on Machine Learning.

Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. 2024. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36.

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. 2020. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474–1487.

Albert Gu, Karan Goel, and Christopher Ré. 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396.

Wenzhong Guo, Jianwen Wang, and Shiping Wang. 2019. Deep multimodal representation learning: A survey. Ieee Access, 7:63373–63394.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. 2019. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917–963.

Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. 2016. Revisiting visual question answering baselines. In European conference on computer vision, pages 727–739. Springer.

Young-Seon Jeong, Myong K Jeong, and Olufemi A Omitaomu. 2011. Weighted dynamic time warping for time series classification. Pattern recognition, 44(9):2231–2240.

Harsh Jhamtani and Taylor Berg-Kirkpatrick. 2021. Truth-conditional captions for time series data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 719–733, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Yushan Jiang, Zijie Pan, Xikun Zhang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. 2024. Empowering time series analysis with large language models: A survey. ArXiv, abs/2402.03182.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. 2023. TimeLm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728.

Argyro Kampouraki, George Manis, and Christophoros Nikou. 2008. Heartbeat time series classification with support vector machines. IEEE transactions on information technology in biomedicine, 13(4):512–518.

Masayuki Kawarada, Tatsuya Ishigaki, Goran Topić, and Hiroya Takamura. 2024. Demonstration selection strategies for numerical time series data-to-text. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7378–7392, Miami, Florida, USA. Association for Computational Linguistics.

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30.

Swaraj Khadanga, Karan Aggarwal, Shafiq Joty, and Jaideep Srivastava. 2019. Using clinical notes with time series data for ICU management. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6432–6437, Hong Kong, China. Association for Computational Linguistics.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. 2021. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations.
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451.

Ross Koval, Nicholas Andrews, and Xifeng Yan. 2024. Financial forecasting from textual and tabular time series. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8289–8300, Miami, Florida, USA. Association for Computational Linguistics.

Mineichi Kudo, Jun Toyama, and Masaru Shimbo. 1999. Japanese Vowels. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5NS47.

Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. 2022. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233–30249.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95–104.

James Large, E Kate Kemsley, Nikolaus Wellner, Ian Goodall, and Anthony Bagnall. 2018. Detecting forged alcohol non-invasively through vibrational spectroscopy and machine learning. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 298–309. Springer.

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848.

Nian Li, Chen Gao, Mingyu Li, Yong Li, and Qingmin Liao. 2024a. EconAgent: Large language model-empowered agents for simulating macroeconomic activities. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pages 15523–15536, Bangkok, Thailand. Association for Computational Linguistics.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.

Zekun Li, Shiyang Li, and Xifeng Yan. 2024b. Time series as images: Vision transformer for irregularly sampled time series. Advances in Neural Information Processing Systems, 36.

Zachary C Lipton, David Kale, and Randall Wetzel. 2016. Directly modeling missing data in sequences with rns: Improved classification of clinical time series. In Machine learning for healthcare conference, pages 253–270. PMLR.

Chengyu Liu, David Springer, Qiao Li, Benjamin Moody, Ricardo Abad Juan, Francisco J Chorro, Francisco Castells, José Millet Roig, Ikaro Silva, Alistair EW Johnson, et al. 2016. An open access database for the evaluation of heart sound algorithms. Physiological measurement, 37(12):2181.

Haoxin Liu, Zhiyuan Zhao, Jindong Wang, Harshvardhan Kamarthi, and B. Aditya Prakash. 2024. LST-Prompt: Large language models as zero-shot time series forecasters by long-short-term prompting. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7832–7840, Bangkok, Thailand. Association for Computational Linguistics.

Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. 2009. uwave: Accelerometer-based personalized gesture recognition and its applications. Pervasive and Mobile Computing, 5(6):657–675.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019a. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. 2021. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations.

Tiedong Liu and Bryan Kian Hsiang Low. 2023. Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks. arXiv preprint arXiv:2305.14201.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2022. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881–9893.

Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421–2425.

Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng, Preslav Nakov, and Shangsong Liang. 2023. Multimodality representation learning: A survey on evolution, pretraining and its applications. ACM Transactions on Multimedia Computing, Communications and Applications, 20(3):1–34.

Mike A Merrill, Mingtian Tan, Vinayak Gupta, Thomas Hartvigsen, and Tim Althoff. 2024. Language models still struggle to zero-shot reason about time series. In Findings of the Association for Computational
Linguistics: EMNLP 2024, pages 3512–3533, Miami, Florida, USA. Association for Computational Linguistics.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26.

MTEB. 2024. Massive Text Embedding Benchmark (MTEB) Leaderboard. https://huggingface.co/spaces/mteb/leaderboard. Accessed: 2024-05-13.

Niklas Muenninghoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.

Soichiro Murakami, Akihiko Watanabe, Akira Miyazawa, Keiichi Goshima, Shihiko Yanase, Hiroya Takamura, and Yusuke Miya. 2017. Learning to generate market comments from stock prices. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pages 1374–1384.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations.

Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. Nomic embed: Training a reproducible long context text embedder. Preprint. arXiv:2402.01613.

OpenAI. 2024. OpenAI Embedding Models. https://platform.openai.com/docs/guides/embeddings, https://openai.com/index/new-embedding-models-and-api-updates/. Accessed: 2024-05-13.

Victor Pan. 2017. Fast approximate computations with cauchy matrices and polynomials. Mathematics of Computation, 86(308):2799–2826.

Victor Y Pan. 2012. Structured matrices and polynomials: unified superfast algorithms. Springer Science & Business Media.

Nikolaos Passalis, Avraam Tsantekidis, Anastasios Tefas, Juho Kanninainen, Moncef Gabbouj, and Alexandros Iosifidis. 2017. Time-series classification using neural bag-of-features. In 2017 25th European Signal Processing Conference (EUSIPCO), pages 301–305. IEEE.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.

Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Erik Cambria, Alexander Gelbukh, and Amir Hussain. 2018. Multimodal sentiment analysis: Addressing key issues and setting up the baselines. IEEE Intelligent Systems, 33(6):17–25.

Jacob Portes, Alexander Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, and Jonathan Frankle. 2024. Mosaicbert: a bidirectional encoder optimized for fast pretraining. Advances in Neural Information Processing Systems, 36.

Ofir Press, Noah A Smith, and Mike Lewis. 2020. Shortformer: Better language modeling using shorter inputs. arXiv preprint arXiv:2012.15832.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

UEA Rik Henson. 2023. DecMg2014 - Decoding the Human Brain. https://www.kaggle.com/c/decoding-the-human-brain/data. Accessed: 2024-04-15.

Tim Salimans and Durk P Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29.

Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202.

Hiroshi Shimodaira, Ken-ichi Noma, Mitsuru Nakai, and Shigeki Sagayama. 2001. Dynamic time-alignment kernel in support vector machine. Advances in neural information processing systems, 14.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-Lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053.

Mohammad Shokoohi-Yekta, Bing Hu, Hongxia Jin, Jun Wang, and Eamonn Keogh. 2017. Generalizing dtw to the multi-dimensional case requires an adaptive approach. Data mining and knowledge discovery, 31:1–31.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.

Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. 2023. Test: Text prototype aligned embedding to activate lnn’s ability for time series. arXiv preprint arXiv:2308.08241.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. 2021. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations (ICLR).

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Aaron Voelker, Ivana Kajić, and Chris Eliasmith. 2019. Legendre memory units: Continuous-time representation in recurrent neural networks. Advances in neural information processing systems, 32.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pages 11897–11916, Bangkok, Thailand. Association for Computational Linguistics.

Xiaoyue Wang, Abdullah Mueen, Hui Ding, Goce Trajcevski, Peter Scheuermann, and Eamonn Keogh. 2013. Experimental comparison of representation methods and distance measures for time series data. Data Mining and Knowledge Discovery, 26:275–309.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. 2022. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2022a. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations.

Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2022b. Flowformer: Linearizing transformers with conservation flows. arXiv preprint arXiv:2202.06258.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419–22430.

Chao-Han Huck Yang, Yun-Yun Tsai, and Pin-Yu Chen. 2021. Voice2series: Reprogramming acoustic models for time series classification. In International conference on machine learning, pages 11808–11819. PMLR.

Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xi-aoli Li, and Shonali Krishnaswamy. 2015. Deep convolutional neural networks on multichannel time series for human activity recognition. In Ijcai, volume 15, pages 3995–4001. Buenos Aires, Argentina.

Xinli Yu, Zheng Chen, and Yanbin Lu. 2023. Harnessing LLMs for temporal data - a study on explainable financial time series forecasting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): Industry Track, pages 739–753, Singapore. Association for Computational Linguistics.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. 2022. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980–8987.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121–11128.

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2114–2124.

Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. 2022a. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. 2022b. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems, 35:3988–4003.

Xiyuan Zhang, Ranak Roy Chowdhury, Rajesh K. Gupta, and Jingbo Shang. 2024. Large language models for time series: A survey. ArXiv, abs/2402.01801.

Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. 2017. Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics, 28(1):162–169.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning, pages 27268–27286. PMLR.

Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. 2024. One fits all: Power general time series analysis by pretrained lm. Advances in neural information processing systems, 36.
Appendix

Table of Contents

A Related Works 15
B Datasets 15
C Comparison Baselines: Supervised Learning Methods 18
D Adapting Supervised Forecasting Baseline Models for Classification 25
E Comparison Baselines: Unsupervised Representation Learning Methods 25
F Adapting Unsupervised Representation Learning Models for Classification 27
G Reproduction Details for Baselines 27
H Implementation Details for LETS-C 28
I Hyperparameter Settings for LETS-C 28
J Model Performance 29
K Monetary Costs Analysis 29
L Assessing Text Embeddings with Cosine Similarity 29
M Numerical Precision for Tokenization 29
N Generalization Across Various Text Embedding Models 30
O Trade-offs: Model Accuracy vs. Parameter Complexity 31
P Ablation Study 34
Q Alternative Methods for Fusing Time Series with Embeddings 34

A Related Works

Time series classification has been an active research area for decades. Early approaches investigated distance-based approaches (Abanda et al., 2019) for time series classification. Some built nearest neighbor classifiers based on explicit time series distance measures such as Dynamic Time Warping (DTW) (Wang et al., 2013; Jeong et al., 2011; Berndt and Clifford, 1994). Others have used distance kernels instead and learned Support Vector Machines (SVMs) (Kampouraki et al., 2008; Bahlmann et al., 2002; Shimodaira et al., 2001), or extracted features and learned linear (Dempster et al., 2020) or tree-based classifiers such as eXtreme Gradient Boosting (XGBoost) (Chen and Guestrin, 2016).

Later, deep learning-based approaches are widely adopted because of their ability to learn complex patterns. Convolutional Neural Networks (CNNs) have proven to be successful in learning local patterns in time series data (Wu et al., 2022a; Franceschi et al., 2019; Zhao et al., 2017). Similarly, Multilayer Perceptron (MLP) can provide simple but effective time series classifiers (Zhang et al., 2022a). Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM) effectively handle long sequence modeling (Gu et al., 2021; Lai et al., 2018; Hochreiter and Schmidhuber, 1997).

More recently, transformer-based models (Vaswani et al., 2017) have revolutionized the NLP domain, and these models have been adapted to the time series domain (Zhou et al., 2022; Wu et al., 2021; Zhou et al., 2021). The self-attention mechanism in transformers is known for modeling long-range dependencies in sequence data. However, the increasing complexity of these models often comes with larger model sizes and higher computational costs, especially for training.

B Datasets

We benchmark our model using the following 10 multivariate datasets from the UEA Time Series Classification Archive Bagnall et al. (2018). See Table 4 for their data characteristics.

B.1 EthanolConcentration

EthanolConcentration (Large et al., 2018) comprises raw spectra from water-and-ethanol solutions contained within 44 unique, real whisky bottles, featuring ethanol concentrations of 35%, 38%, 40%, and 45%. Scotch Whisky regulations require a minimum alcohol content of 40%, a standard that producers adhere to in order to comply with labeling specifications. The dataset presents a classification task to identify the ethanol concentration from spectral readings of any given bottle. Each record includes three spectral readings from the same bottle and batch, obtained by positioning the
Table 4: Dataset Characteristics. Abbreviations: EC: EthanolConcentration, FD: FaceDetection, HW: Handwriting, HB: Heartbeat, JV: JapaneseVowels, PEMS: PEMS-SF, SCP1: SelfRegulationSCP1, SCP2: SelfRegulationSCP2, SAD: SpokenArabicDigits, UW: UWaveGestureLibrary

<table>
  <tr>
    <th>Characteristic</th>
    <th>EC</th>
    <th>FD</th>
    <th>HW</th>
    <th>HB</th>
    <th>JV</th>
    <th>PEMS-SF</th>
    <th>SCP1</th>
    <th>SCP2</th>
    <th>SAD</th>
    <th>UW</th>
  </tr>
  <tr>
    <td>Train Size</td>
    <td>261</td>
    <td>5890</td>
    <td>150</td>
    <td>204</td>
    <td>270</td>
    <td>267</td>
    <td>268</td>
    <td>200</td>
    <td>6599</td>
    <td>120</td>
  </tr>
  <tr>
    <td>Test Size</td>
    <td>263</td>
    <td>3524</td>
    <td>850</td>
    <td>205</td>
    <td>370</td>
    <td>173</td>
    <td>293</td>
    <td>180</td>
    <td>2199</td>
    <td>320</td>
  </tr>
  <tr>
    <td>Number of Dimensions</td>
    <td>3</td>
    <td>144</td>
    <td>3</td>
    <td>61</td>
    <td>12</td>
    <td>963</td>
    <td>6</td>
    <td>7</td>
    <td>13</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Series Length</td>
    <td>1751</td>
    <td>62</td>
    <td>152</td>
    <td>405</td>
    <td>29</td>
    <td>144</td>
    <td>896</td>
    <td>1152</td>
    <td>93</td>
    <td>315</td>
  </tr>
  <tr>
    <td>Number of Classes</td>
    <td>4</td>
    <td>2</td>
    <td>26</td>
    <td>2</td>
    <td>9</td>
    <td>7</td>
    <td>2</td>
    <td>2</td>
    <td>10</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Type</td>
    <td>Spectro</td>
    <td>EEG</td>
    <td>Motion/Human Activity Recognition</td>
    <td>Audio</td>
    <td>Audio</td>
    <td>Occupancy rate</td>
    <td>EEG</td>
    <td>EEG</td>
    <td>Speech</td>
    <td>EEG</td>
  </tr>
</table>

bottle between a light source and a spectroscope. These spectral readings, which cover wavelengths from 226nm to 1101.5nm at a 0.5nm resolution, were recorded over a one-second integration time using a StellarNet BLACKComet-SR spectrometer. The methodology deliberately avoids optimizing for clarity or consistency in the spectral path, aiming to simulate the varied conditions typical of rapid screening tests that may be performed on batches of spirits for quality assurance.

B.2 FaceDetection

The FaceDetection dataset originates from a 2014 Kaggle competition (Rik Henson, 2023). The challenge involves identifying whether a subject is viewing a picture of a face or a scrambled image using magnetoencephalography (MEG) data, independent of the individual subject. This dataset specifically includes only the training portion from the competition, organized by patient. It comprises data from 10 training subjects (subject01 to subject10) and 6 testing subjects (subject11 to subject16). Each subject has approximately 580 to 590 trials, resulting in a total of 5,890 training trials and 3,524 test trials. Each trial features 1.5 seconds of MEG data, initiated 0.5 seconds before the stimulus is presented, and is associated with a class label—Face (class 1) or Scramble (class 0). The data were down-sampled to 250Hz and subjected to a high-pass filter at 1Hz, producing 62 observations per channel.

B.3 Handwriting

The Handwriting dataset (Shokoohi-Yekta et al., 2017) consists of motion data captured from a smartwatch while subjects wrote the 26 letters of the alphabet. Developed at the University of California, Riverside (UCR), this dataset includes 150 training cases and 850 test cases. It features six dimensions, comprising three accelerometer readings and three gyroscope readings.

B.4 Heartbeat

The Heartbeat dataset originates from the PhysioNet/CinC Challenge 2016 Liu et al. (2016) and consists of cardiac sound recordings from a diverse pool of participants, both healthy individuals and patients with cardiac conditions. Recordings were made in various settings, clinical and non-clinical, and captured from multiple body locations including the aortic, pulmonic, tricuspid, and mitral areas, among up to nine potential sites. The dataset categorizes these sounds into two primary classes: normal and abnormal. Normal heart sounds were obtained from healthy subjects, while abnormal sounds were recorded from patients diagnosed with cardiac ailments, predominantly heart valve defects such as mitral valve prolapse, mitral regurgitation, aortic stenosis, and post-valvular surgery conditions, as well as coronary artery disease.

The audio recordings, inclusive of contributions from both children and adults, were uniformly truncated to five seconds. Spectrograms of each truncated audio were generated using a window size of 0.061 seconds with a 70% overlap. This multivariate dataset is structured with each dimension representing a frequency band derived from the spectrogram. There are 113 instances in the normal class and 296 in the abnormal class.

B.5 JapaneseVowels

The Japanese Vowels dataset Kudo et al. (1999), sourced from the UCI Machine Learning Repository, comprises recordings from nine male speakers who pronounced the Japanese vowels ‘a’ and ‘e’. Each utterance was analyzed using a 12-degree linear prediction to extract a 12-dimensional time-series representation, with lengths varying origi-
nally from 7 to 29. For consistency, all instances in the dataset have been padded to the maximum length of 29. The objective of the classification task is to identify the speaker; hence each 12-by-29 instance matrix is associated with a single class label, ranging from 1 to 9. This dataset serves as a benchmark for assessing the efficacy of time-series classification models in distinguishing speakers based on LPC cepstrum coefficients obtained from their speech patterns.

The dataset includes a total of 640 time-series instances. A training set consists of 30 utterances per speaker, totaling 270 instances. The test set, however, comprises 370 instances and varies in distribution—ranging from 24 to 88 instances per speaker—owing to external factors such as timing and availability during the experimental setup.

B.6 PEMS-SF

The PEMS-SF dataset Cuturi (2011) contains 15 months of daily data sourced from the California Department of Transportation. This dataset details the occupancy rates, ranging from 0 to 1, across various car lanes on the freeways of the San Francisco Bay Area. The data spans from January 1, 2008, to March 30, 2009, with measurements taken every 10 minutes. Each day is treated as an individual time series with a dimension of 963, corresponding to the number of sensors that consistently functioned throughout the observation period. The length of each time series is 144 data points (6 per hour x 24 hours). The dataset excludes public holidays and two anomalous days (March 8, 2009, and March 9, 2008) when sensors recorded no data between 2:00 and 3:00 AM, resulting in a total of 440 valid time series. The classification task involves identifying the day of the week for each series, labeling them with integers from 1 (Monday) to 7 (Sunday). Each attribute within a record reflects the occupancy rate recorded by a sensor at a specific timestamp throughout the day.

B.7 SelfRegulationSCP1

The SelfRegulationSCP1 dataset, sourced from Birbaumer et al. (1999), involves recordings from a healthy subject who was instructed to control a cursor on a screen using cortical potentials. This process was facilitated by tracking the subject’s slow cortical potentials (Cz-Mastoids), where cortical positivity resulted in downward cursor movements and cortical negativity caused it to move upward. Each trial, lasting six seconds, was designed to capture these dynamics, with visual feedback provided between the second 2 and 5.5 of the trial. During each trial, a goal was visually indicated at either the top or bottom of the screen starting from 0.5 seconds to the end of the trial, guiding the subject to generate negative or positive potentials correspondingly. The usable data for each trial, however, spans only 3.5 seconds—from the second 2 to 5.5—corresponding to 896 samples per channel given the sampling rate of 256 Hz.

Data capture involved a PsyLab EEG8 amplifier and a PCIM-DAS1602/16 A/D converter, recording over channels positioned according to the 10/20 system. The dataset includes a training set of 268 trials—168 from the first day and 100 from the second, mixed randomly—and 293 test instances, with class labels indicating positivity or negativity.

B.8 SelfRegulationSCP2

The SelfRegulationSCP2 dataset Birbaumer et al. (1999) includes data from an artificially respirated ALS patient who was tasked with controlling a cursor on a computer screen using cortical potentials. Auditory and visual cues were used to guide the patient, with slow cortical potentials measured at the Cz-Mastoids. A positive potential moved the cursor downward, whereas a negative potential moved it upward. Each trial lasted 8 seconds, with the cursor movement direction (up for negativity, down for positivity) indicated both visually and auditorily from the 0.5 to 7.5 second marks. Auditory instructions were given precisely at the 0.5-second mark, and visual feedback was available from seconds 2 to 6.5. Only the data from this 4.5-second feedback period, translating to 1152 samples per channel at a 256 Hz sampling rate, are used for training and testing.

EEG data were collected from several sites according to the 10/20 system and included channels for detecting vertical eye movements (vEOG). The EEG signals were not corrected for EOG artifacts, providing a raw view of the cortical activity. The dataset comprises 200 trials for training, evenly split between two classes, and an additional 180 trials for testing, recorded on the same day but after the training session data. Each trial spans 7 dimensions and a series length of 1152.

B.9 Spoken Arabic Digits

The Spoken Arabic Digits dataset Bedda and Hammami (2010) consists of 8,800 time series data entries derived from the vocal utterances of 88 native
Arabic speakers (44 males and 44 females, aged between 18 and 40). Each dataset entry represents one of ten Arabic digits, spoken ten times by each speaker. The dataset captures 13 Mel Frequency Cepstral Coefficients (MFCCs) for each sound snippet, which are extracted under the following audio processing conditions:

• Sampling rate: 11025 Hz

• Bit depth: 16 bits

• Window function: Hamming

• Pre-emphasis filter: \( 1 - 0.97Z^{-1} \)

Each line in the database corresponds to one frame of analysis, listing the 13 MFCCs separated by spaces. These coefficients effectively capture the spectral properties essential for recognizing spoken digits. This structured approach facilitates robust time-series analysis for speech recognition tasks involving Arabic numerals.

B.10 UWaveGestureLibrary
The UWaveGestureLibrary Liu et al. (2009) comprises a set of eight simple gestures, each generated from accelerometer data. The dataset records the X, Y, and Z coordinates corresponding to each gesture’s motion. Every time series within this dataset consists of 315 data points.

C Comparison Baselines: Supervised Learning Methods
To provide a thorough comparison, we evaluate our approach against 21 supervised baseline models for time series classification. These baselines can be categorized into Classical methods, models based on MLPs, RNNs, CNNs, transformers, and LLMs. The details of which are provided below.

C.1 Classical methods
1) Dynamic Time Warping (DTW) Berndt and Clifford (1994) is a method for measuring similarity between two time series, \( X = (x_1, x_2, \ldots, x_M) \) and \( Y = (y_1, y_2, \ldots, y_N) \), which may differ in length and are sampled at equidistant points in time. DTW identifies the best alignment between these series by minimizing the effects of distortion and shifting in time, allowing for the comparison of similar shapes across different phases.

The core of DTW is the construction of a local cost matrix, \( C \in \mathbb{R}^{M \times N} \), with entries \( c_{i,j} = \| x_i - y_j \| \) for \( i \in [1, M] \) and \( j \in [1, N] \), which represents the pairwise distances between points in the two series. The objective is to find a warping path \( p = (p_1, p_2, \ldots, p_L) \) where each \( p_l = (p_{i_l}, p_{j_l}) \) lies within \( [1, M] \times [1, N] \). This path aligns the series by following the route that minimizes cumulative distance, adhering to several constraints: it must start and end at \( p_1 = (1, 1) \) and \( p_L = (M, N) \) (boundary condition), maintain the temporal ordering of points \( m_1 \leq m_2 \leq \ldots \leq m_L \) and \( n_1 \leq n_2 \leq \ldots \leq n_L \) (monotonicity condition), and prevent large temporal jumps (step size condition). The optimal warping path is identified through a recursive process aimed at minimizing the total cost associated with \( p \), calculated as \( c_p(X, Y) = \sum_{l=1}^L c(x_{m_l}, y_{n_l}) \). This path, \( P^* \), where \( c_{P^*} = \min_{p \in P} c_p(X, Y) \), defines the DTW distance, quantifying the similarity between the series. However, the computational cost of this process is \( O(MN) \), where \( M \) and \( N \) are the lengths of the two series, rendering it computationally demanding for large datasets.

For classifying time series, the DTW distance is integrated with the k-nearest neighbors (k-NN) algorithm. This approach computes the DTW distance of a target series to all other series in a training dataset and assigns a classification based on the most common class among the k-nearest neighbors. Thus, DTW effectively accommodates series with time shifts, providing a robust, distance-based method for classifying time series.

2) eXtreme Gradient Boosting (XGBoost) (Chen and Guestrin, 2016) is a state-of-the-art machine learning algorithm that primarily uses decision trees as base learners to construct a robust ensemble model. XGBoost sequentially builds a series of weak learners—typically decision trees—and enhances each successive tree by correcting errors made by its predecessors, a technique known as boosting. This process involves an additive model where each new decision tree is improved by leveraging the cumulative knowledge of the trees that came before it, optimizing for maximum information gain at each
split using a greedy algorithm. To prevent overfitting, XGBoost incorporates regularization directly into its loss function and employs shrinkage to moderate the learning rate. Additionally, the optimization method, which is gradient-based, minimizes a cost function by iteratively adjusting the model’s parameters in response to the gradients of the errors. XGBoost also refines the decision tree construction process by using a Similarity Score and Gain to determine the most effective node splits, further improving the model’s accuracy and efficiency.

3) RandOm Convolutional KErnel Transform (ROCKET) (Dempster et al., 2020) is a method for time series classification that employs random convolutional kernels to transform series data, which is then used to train a linear classifier. Unlike traditional convolutional neural networks (CNNs) that rely on learned kernels, ROCKET utilizes a broad array of random kernels. Each kernel is uniquely characterized by random properties such as length, weights, biases, dilation, and padding. This configuration forms a single-layer convolutional neural network, where the randomized kernel weights contribute to generating input for a softmax layer, thus optimizing the feature extraction process. ROCKET also efficiently scales for large datasets due to its linear complexity relative to the length of the time series and the number of training samples. The key advantages of ROCKET include:

– Number of Kernels: ROCKET utilizes a substantial number of kernels in a single layer. The non-learned nature of these kernels reduces computational costs significantly, enabling the use of numerous kernels without substantial overhead.
– Variety of Kernels: Unlike typical CNNs, where kernels might share characteristics, each ROCKET kernel is distinct in its attributes, enhancing the diversity and the ability to detect various patterns.
– Kernel Dilation: Dilation in ROCKET is randomly assigned to each kernel, differing from the exponential increase with depth seen in traditional CNNs. This randomness is vital for identifying patterns across different scales and frequencies.
– Feature Extraction: Beyond employing global max pooling techniques through the maximum value of feature maps, ROCKET utilizes a novel metric—the proportion of positive values. This metric allows classifiers to assess the prevalence of patterns more accurately within the time series.

C.2 MLP-based methods

4) LightTS (Zhang et al., 2022a) is an MLP-based time series forecasting model that employs simple MLP structures to manage both short-term and long-term temporal dependencies. This model includes two downsampling strategies: interval sampling, which targets long-term dependencies, and continuous sampling, which focuses on short-term local patterns. These strategies are based on the principle that downsampling typically preserves the majority of a time series’ crucial information, thus maintaining model efficiency. LightTS utilizes an MLP-based framework on top of these downsampling techniques, enabling effective information exchange among different down-sampled subsequences and time steps. This configuration allows LightTS to adaptively select relevant information for forecasting and to efficiently handle very long input sequences by processing only a fraction of the data after downsampling.

5) DLinear (Zeng et al., 2023) is a recent non-transformer model developed in response to the difficulty transformer-based models face in capturing ordering information within time series. DLinear integrates a decomposition scheme similar to those used in Autoformer and FEDformer but relies on linear layers for processing. Initially, it decomposes a raw data input into a trend component using a moving average kernel and a remainder (seasonal) component. Subsequently, two single-layer linear layers are applied independently to each component. The outputs from these layers are then summed to produce the final prediction. This approach allows DLinear to enhance performance over a standard linear model by explicitly handling trends within the data.

C.3 RNN-based models

6) Long Short-Term Memory (LSTM) Hochreiter and Schmidhuber (1997) addresses the
vanishing gradient problem that plagues vanilla RNNs in processing longer sequences. This issue arises as the network propagates forward, and the small weight values in the hidden layers are multiplied repeatedly, causing the gradients to diminish rapidly. As a result, the weights in the initial layers become increasingly difficult to train, which impacts the training of subsequent weights, making RNNs challenging to train overall. LSTM mitigates this problem by incorporating a memory cell equipped with various gates that regulate the flow of information into and out of the cell, enabling it to handle long-short term dependencies effectively. An LSTM unit utilizes a cell state and three gates—input, forget, and output—to manage information. Each gate includes a sigmoid layer \( \sigma \) that outputs values between 0 and 1, representing the proportion of information allowed through the gate, and a point-wise multiplication operation. Specifically, the forget gate \( f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) \) determines which information to discard from the previous cell state \( c_{t-1} \) by analyzing the current input \( x_t \) and the previous hidden state \( h_{t-1} \). The input gate \( i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) \) decides which new information to update, and the update to the cell state \( \tilde{c}_t = \tanh(W_c \cdot [h_{t-1}, x_t] + b_c) \) is computed. The cell state is then updated to \( c_t = f_t \cdot c_{t-1} + i_t \cdot \tilde{c}_t \). Finally, the output gate \( o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) \) determines what portion of the cell state to output, with the output hidden state given by \( h_t = o_t \cdot \tanh(c_t) \).

7) Long- and Short-term Time-series Network (LSTNet) (Lai et al., 2018) incorporates both CNN and RNN components to perform comprehensive time series analysis. The CNN extracts short-term local dependency patterns from multi-dimensional input variables, while the RNN is tasked with capturing complex long-term dependencies in time series trends. To address the issue of scale insensitivity commonly found in neural network models, LSTNet integrates a traditional autoregressive model. Additionally, LSTNet features a Recurrent-skip structure designed to effectively capture very long-term dependence patterns and to facilitate easier optimization, leveraging the periodic properties of the input time series signals. Further enhancing its robustness, LSTNet also employs a traditional autoregressive linear model in parallel with its nonlinear neural network components. This dual approach makes the network particularly adept at managing time series that exhibit significant scale variations.

8) Linear State Space Layer (LSSL) (Gu et al., 2021), which is part of the structured state space sequence model, introduces a parameterization for state space models (SSM) designed to enhance computational efficiency. LSSL modifies the structured state matrices by decomposing them into a low-rank and a normal term (Gu et al., 2020; Voelker et al., 2019). This modification facilitates the computation of the truncated generating function in frequency space, rather than expanding the standard SSM in coefficient space. Furthermore, LSSL employs the Woodbury identity to adjust the low-rank term, and the normal term is stably diagonalized. This approach simplifies the computations by involving processes associated with a Cauchy kernel (Pan, 2012, 2017), known for its stability in theoretical contexts. These modifications allow LSSL to efficiently manage both computational and memory resources.

C.4 CNN-based models

9) Temporal Convolutional Network (TCN) (Bai et al., 2018; Franceschi et al., 2019) is a CNN-based architecture designed to capture extended historical data with long memory capabilities and requires minimal tuning in practice. TCNs utilize dilated causal convolutions to ensure that predictions do not prematurely incorporate future data. The dilations significantly expand the network’s receptive field, enabling it to cover a broader range of historical context. Furthermore, TCNs integrate residual connections to facilitate the effective training of deeper models.

A TCN model comprises a series of \( n \) TCN residual blocks, where \( n \) is a hyperparameter. Each block contains two dilated causal convolutional layers, which are fully convolutional to ensure that the output size is consistent with the input size. These causal convolutions guarantee that the output at any given time \( t \) depends only on inputs from time \( t \) and earlier. The convolutional layers are ap-
plied with a stride of 1, and padding adjustments maintain the convolutional nature of the network. Each convolutional layer applies a dilation factor \( d \), typically set as \( d = 2^i \) for the \( i \)-th block, to exponentially increase the receptive field as the network deepens. Mathematically, a convolution with dilation factor \( d \) on an element \( x \) of a 1D input \( g \) with a filter \( f \) of length \( k \) is computed as
\[
(g *_{d} f)(x) = \sum_{j=0}^{k-1} f(j) \cdot g(x - d \cdot j).
\]
Following the convolutional layers, the sequence of operations includes weight normalization (Salimans and Kingma, 2016), ReLU activation, and a dropout layer. Weight normalization improves gradient conditioning and accelerates convergence by reparameterizing each weight vector \( \mathbf{w} \) as \( \mathbf{w} = \frac{g}{|\mathbf{v}|} \mathbf{v} \), where \( \mathbf{v} \) has a fixed norm and \( g \) is a scalar. This process decouples the magnitude of the weight vector from its direction, enhancing network optimizability. Each block concludes with an element-wise addition of the block’s input and the estimated residual mapping, followed by a ReLU activation. Dimension alignment for this addition is achieved with a \( 1 \times 1 \) convolution.

10) TimesNet (Wu et al., 2022a) is a method that overcomes the limitations of traditional 1D time series representation by transforming these series into 2D tensors organized across multiple periods. It captures short-term intraperiod variations and long-term interperiod trends by embedding them into the columns and rows of the 2D tensors, respectively. This transformation allows for more efficient modeling of temporal variations using 2D convolutional kernels, thereby extending the analysis into a more comprehensive 2D space. TimesNet ensures simultaneous representation of both intraperiod and interperiod variations, with modules specifically tailored to emphasize the unique temporal patterns of each period.

The central component of TimesNet, the TimesBlock, is a versatile and adaptive structure designed to detect multiperiodicity and extract complex temporal patterns from these 2D tensors. Its parameter-efficient Inception-block (Szegedy et al., 2015) based architecture boosts the model’s analytical capabilities, facilitating a detailed examination of distinct temporal variations associated with different periods. This approach allows TimesNet to transcend the constraints of 1D representations, enabling a unified and thorough analysis of temporal variations.

C.5 Transformer-based models

11) Transformer (Vaswani et al., 2017) comprises a dual-component architecture with both an encoder and a decoder, each containing stacked self-attention and point-wise, fully connected layers. Each component consists of six identical layers. In the encoder, each layer includes a multi-head self-attention mechanism and a position-wise fully connected feed-forward network, complemented by a residual connection and layer normalization. The decoder replicates this configuration but adds a third sub-layer for multi-head attention on the encoder’s output. It also modifies its self-attention mechanism to prevent forward-looking attention, thus preserving the model’s auto-regressive properties for sequential generation.

The Transformer utilizes multi-head attention to enable nuanced interactions between its encoder and decoder and to facilitate detailed processing across different positions in the sequence. This attention mechanism projects queries, keys, and values through multiple linear transformations, enabling diverse representation and integration of information across subspaces. It is applied in three distinct forms: encoder-decoder attention allows decoder queries to attend to all positions in the encoder output; self-attention within the encoder lets each position process information from all preceding positions; and self-attention within the decoder restricts attention to prevent future positions from influencing the sequence, maintaining sequential integrity. This structured approach helps the Transformer effectively manage and process long sequence dependencies, making it adaptable for a broad range of sequence-based applications.

12) Reformer (Kitaev et al., 2020) enhances the efficiency of the transformer model with two significant modifications, making it more suitable for processing long sequences. Firstly, it
replaces the traditional dot-product attention with a locality-sensitive hashing mechanism. This change reduces the computational complexity from \( O(L^2) \) to \( O(L \log L) \), where \( L \) is the sequence length. Secondly, Reformer employs reversible residual layers, which regenerate the activations of any layer from the subsequent layer’s activations using only the model parameters. This approach eliminates the need for storing multiple copies of activations for each layer, substantially reducing memory usage. The reversible layers, introduced in (Gomez et al., 2017), require only a single set of activations to be stored for the entire model, significantly reducing the memory cost usually multiplied by the number of layers (\( N \)). Moreover, the Reformer processes activations in chunks within feed-forward layers, further decreasing memory demands. These adjustments, along with the use of locality-sensitive hashing for attention computation, not only minimize memory and computational overhead but also maintain the model’s performance on par with the traditional transformer model for long sequences.

13) Informer (Zhou et al., 2021) is a transformer-based model designed specifically to tackle challenges in long sequence time-series forecasting, such as quadratic time complexity, high memory usage, and constraints of the traditional encoder-decoder architecture. The Informer introduces several modifications to enhance efficiency and effectiveness in processing long sequences:

– ProbSparse Self-Attention Mechanism: This mechanism replaces the conventional self-attention in Transformers. It is engineered to achieve \( O(L \log L) \) in both time complexity and memory usage, efficiently managing dependency alignments without compromising performance.

– Self-Attention Distilling: This process improves attention management by concentrating on dominant attention scores and halving the input size for each cascading layer. This method effectively manages extremely long input sequences and significantly lowers the space complexity to \( O((2 - \epsilon)L \log L) \).

– Generative Style Decoder: Diverging from the typical step-by-step decoding process, the generative style decoder in Informer predicts long time-series sequences in a single forward operation. This approach accelerates inference for long-sequence predictions and reduces the propagation of cumulative errors during the inference phase.

The Informer leverages these enhancements—ProbSparse self-attention for efficient processing, self-attention distilling to focus on important attention scores, and a generative style decoder for rapid sequence generation—to improve its performance in forecasting long sequences and capturing long-range dependencies between extensive time-series inputs and outputs.

14) Pyraformer (Liu et al., 2021) is a transformer-based model that employs a pyramidal attention module (PAM) for efficient management of time series data. This module uses inter-scale and intra-scale connections to summarize features at different resolutions and capture temporal dependencies across various ranges. The design integrates a tree structure for inter-scale connections and neighboring intra-scale connections to achieve a multi-resolution representation of time series. This architecture ensures that Pyraformer scales linearly with the input series length, optimizing computational efficiency while maintaining a constant signal path length relative to the sequence length \( L \), and keeping both time and space complexity linear with \( L \).

Pyraformer operates by first embedding observed data, covariates, and positions in a manner similar to the Informer (Zhou et al., 2021). It then constructs a multi-resolution \( C \)-ary tree through a coarser-scale construction module (CSCM), where each coarser scale node aggregates information from \( C \) finer scale nodes. This structure allows Pyraformer to model temporal dependencies efficiently across different scales through sparse intra-scale connections, thus reducing computational overhead. Depending on the specific needs of different downstream tasks, Pyraformer adapts its output structure to effectively meet the requirements of diverse time series analyses.
15) Autoformer (Wu et al., 2021) is a transformer-based model that incorporates time series decomposition, drawing inspiration from classical time series analysis methods. Unlike traditional transformers that rely on self-attention mechanisms to capture long-range dependencies, Autoformer introduces an auto-correlation mechanism as an alternative. This change addresses the issues that traditional models face with intricate temporal patterns and long-term forecasting, where identifying reliable dependencies can be challenging. To enhance efficiency in handling long series, traditional transformers sometimes use sparse versions of self-attention, which can limit the effective use of information.

Autoformer integrates decomposition blocks directly into its structure, moving away from the typical preprocessing approach of series decomposition. These blocks are specifically designed to progressively isolate long-term trends from the data during the forecasting process, allowing the model to refine and decompose the data iteratively. This setup improves the handling of complex time series. The auto-correlation mechanism, inspired by stochastic process theory and based on the periodicity of the series, focuses on identifying dependencies and aggregating representation at the sub-series level, effectively capturing and utilizing similarities derived from underlying periodic patterns.

The architecture of Autoformer adheres to a residual and encoder-decoder framework. The encoder eliminates long-term trend-cyclical components through the series decomposition blocks and focuses on modeling seasonal patterns. In contrast, the decoder accumulates the trend component extracted from the hidden variables, using past seasonal information to enhance forecasting accuracy. Additionally, Autoformer incorporates a moving average within its decomposition blocks to smooth out periodic fluctuations and highlight long-term trends, thereby facilitating a more targeted analysis of stable trend components within the time series.

16) Non-stationary Transformer (Liu et al., 2022) addresses the challenges posed by non-stationary real-world data, where the joint distribution changes over time, often leading to the degradation of transformer performance. Non-stationary Transformer consists of two interdependent modules: series stationarization and de-stationary attention. Series stationarization normalizes the input data to unify its statistical properties, enhancing predictability, and adjusts the output to restore the original statistics. This module utilizes a straightforward normalization approach without additional parameters. De-stationary attention, on the other hand, aims to counteract the potential over-normalization by reintroducing the intrinsic non-stationary characteristics of the data into the model’s temporal dependencies. The de-stationary attention module approximates how attention mechanisms would function on unnormalized data and integrates these insights back into the model to maintain crucial temporal dynamics. This setup allows stationerform to balance the predictability benefits gained from normalized data with the rich, detailed patterns inherent in the raw non-stationary series.

Structurally, Non-stationary Transformer adapts the traditional encoder-decoder setup. The encoder extracts information from past observations, while the decoder aggregates this information to refine predictions. The framework modifies the standard transformer by applying series stationarization to both the input and output of the model, and replaces the conventional self-attention mechanism with de-stationary attention. This adaptation aims to enhance the model’s ability to predict non-stationary series by effectively managing the challenges associated with data variability over time.

17) Frequency Enhanced Decomposed Transformer (FEDformer) (Zhou et al., 2022) is a transformer-based method that incorporates a time series decomposition scheme to address the limitations of traditional transformers, particularly their high computational demands and challenges in capturing global time series trends. By combining transformers with the seasonal-trend decomposition method, FEDformer aims to separate the broad trends from more detailed fluctuations in time series data. This allows the transformer component to focus on more granular details while the decom-
position handles the overall profile of the series.

The architecture of FEDformer includes specialized blocks such as Fourier-enhanced and Wavelet-enhanced blocks within the transformer framework. These blocks serve as substitutes for the conventional self-attention and cross-attention mechanisms, and they enable the model to analyze important structures through frequency domain mapping. FEDformer employs a selective approach to incorporating Fourier components, which helps keep the computational complexity and memory usage linear in relation to the length of the time series. Specifically, FEDformer is structured as a deep decomposition architecture that integrates Frequency Enhanced Block (FEB), Frequency Enhanced Attention (FEA) connecting the encoder and decoder, and the Mixture Of Experts Decomposition block (MOEDecomp). This setup leverages both seasonal-trend decomposition and distribution analysis to facilitate the processing of time series data.

18) ETSformer (Woo et al., 2022) is a transformer-based architecture tailored for time series forecasting, integrating exponential smoothing techniques. ETSformer uses two novel mechanisms, Exponential Smoothing Attention (ESA) and Frequency Attention (FA), which are designed to replace the traditional self-attention mechanism in standard transformers. ESA utilizes attention scores based on relative time lags, enabling efficient handling of growth components with a computational complexity of \( O(L \log L) \) for a length-\( L \) lookback window. Similarly, FA employs Fourier transformations to identify dominant seasonal patterns, selecting bases with the highest amplitudes in the frequency domain to achieve the same level of complexity.

ETSformer is structured with modular decomposition blocks that allow it to dissect time-series data into distinct, interpretable components such as level, growth, and seasonality. This design facilitates layer-wise decomposition of the time series into these components, enhancing the model’s ability to capture and represent complex temporal dynamics. The architecture systematically extracts latent growth and seasonal patterns through a deep, multi-layered approach, where each layer progressively refines the extraction of these temporal features. The final forecast generated by ETSformer integrates these decomposed elements—level, trend, and seasonality—into a cohesive output that is both practical and interpretable for human analysts. By emphasizing recent observations, the model aligns with the principles of exponential smoothing, ensuring that more recent trends carry greater weight in the forecast.

19) Flowformer (Wu et al., 2022b) modifies the traditional transformer architecture by integrating flow network theory to tackle the scalability issues typical of standard transformers. The conventional attention mechanism in transformers is known for its quadratic complexity, which limits their ability to process a large number of tokens and scale to larger models effectively. To address this, Flowformer introduces the Flow-Attention mechanism, grounded in the principles of flow conservation. This mechanism reimagines attention as information flowing from sources (values) to sinks (results) via learned flow capacities (attentions), aiming to achieve linear complexity.

The Flow-Attention mechanism manages the flow of information by regulating incoming flow at sinks to initiate source competition and outgoing flow at sources for sink allocation. This management of flow helps in aggregating relevant information without relying on specific inductive biases and aims to prevent the common issue of degenerated attentions found in typical attention mechanisms. Flowformer embeds flow conservation within its attention mechanism to streamline the process of information aggregation and refinement. By integrating these elements, Flowformer seeks to provide an alternative approach that could potentially handle large datasets and complex time series data more efficiently than traditional transformers, without the computational complexity typically associated with these models.

20) Patch Time Series Transformer (PatchTST) (Nie et al., 2023) is a transformer-based model for multivariate time series forecasting. It
is built on two key principles: (i) segmentation of time series into subseries-level patches, which serve as input tokens to the transformer, and (ii) channel-independence, where each channel contains a single univariate time series that shares the same embedding and transformer weights across all series. The patching mechanism provides three main advantages: it retains local semantic information in embeddings, significantly reduces the computational and memory complexity of attention maps for a given look-back window, and enables the model to attend to longer historical dependencies. The model operates as follows: multivariate time series data is divided into independent channels, each processed separately while sharing the same transformer backbone. Each univariate time series undergoes instance normalization before being segmented into patches, which then serve as input tokens for the transformer. PatchTST employs masked self-supervised learning, where patches are randomly selected and set to zero, requiring the model to reconstruct the missing values. The training objective minimizes Mean Squared Error (MSE) loss between the predicted and ground truth values, demonstrating the effectiveness of transformers when time series data is segmented into patches at the subseries level. For downstream time series tasks, the same transformer encoder is extended and trained with a linear layer for task-specific predictions.

C.6 LLM-based models

21) OneFitsAll (Zhou et al., 2024) employs pre-trained language and computer vision models, developed from billions of tokens, for time series analysis. This approach, termed the Frozen Pretrained Transformer (FPT), retains the original self-attention and feedforward (FFN) layers of the pre-trained models, which hold the majority of the learned knowledge. The model is fine-tuned for various time series classification tasks, with adjustments made only to the positional embeddings and layer normalization layers to adapt to specific downstream tasks. Additionally, to more effectively manage local semantic information, OneFitsAll incorporates a patching technique as described by (Nie et al., 2023). This method aggregates adjacent time steps into a single patch-based token, thereby increasing the historical time horizon that can be processed by the model without increasing the token length, reducing information redundancy within transformer models.

D Adapting Supervised Forecasting Baseline Models for Classification

It is important to note that some of our supervised learning-based baselines were adapted from forecasting to classification tasks without altering the core design of the models. We employ a lightweight multi-layer perceptron (MLP)-based projection layer on top of the model’s encoded features (originally designed for forecasting) to map them to classification labels. The parameters of this layer are learned during training, enabling the model to adapt effectively for classification tasks. This approach aligns with methods used in prior works such as OneFitsAll (Zhou et al., 2024) and TimesNet (Wu et al., 2022a) when adapting forecasting models for multivariate classification.

E Comparison Baselines: Unsupervised Representation Learning Methods

E.1 CNN-based models

1) T-Loss (Franceschi et al., 2019) is an unsupervised method for learning general-purpose representations of multivariate time series while handling variations in length. It trains a scalable encoder, structured as a deep convolutional neural network with dilated convolutions, to produce fixed-length vector representations regardless of input length. The loss function is designed as a triplet loss with time-based negative sampling, leveraging the encoder’s ability to process time series of unequal lengths. The objective is to ensure that similar time series obtain similar representations without requiring supervision to learn such similarity.

2) Temporal Neighborhood Coding (TNC) (Tonkaboni et al., 2021) is a self-supervised approach that leverages the local smoothness of signals to define temporal neighborhoods and learn generalizable representations for time series. It builds on the smoothness of the generative process of signals to capture meaningful representations for time series windows by ensuring that, in the representation space,
signals close in time are distinguishable from those farther apart. In other words, temporal proximity remains identifiable in the encoding space. For each sample window, a neighborhood distribution is first defined. The encoder learns the distribution of windows in the representation space, and samples from these distributions are fed into a discriminator, which predicts the probability of the windows belonging to the same neighborhood. TNC is a general framework, making it agnostic to both the nature of the time series and the architecture of the encoder. The encoder can be any parametric model suited to the signal properties. For the discriminator, a simple multi-headed binary classifier is used.

3) TS2Vec (Yue et al., 2022) is a universal contrastive learning framework for time series representation learning across all semantic levels. It performs hierarchical contrastive learning over augmented context views, enabling robust contextual representations for each timestamp while capturing multi-resolution temporal dependencies. To learn representations at different granularity, TS2Vec applies a simple aggregation mechanism. The representation of an arbitrary sub-sequence is obtained via max pooling over the corresponding timestamps, allowing the model to generate fine-grained representations at any resolution. The framework hierarchically discriminates positive and negative samples across both instance-wise and temporal dimensions, reinforcing the ability to capture contextual relationships in time series data.

The contrastive objective is based on augmented context views, ensuring that representations of the same sub-series in different augmented contexts remain consistent. To achieve this, two overlapping subseries are randomly sampled from an input time series, and consistency of contextual representations is encouraged on the common segment. The encoder is optimized jointly with temporal and instance-wise contrastive loss, with the total loss summed across multiple scales in a hierarchical manner. The encoder consists of three key components: an input projection layer, a timestamp masking module, and a dilated CNN module. The input projection layer maps observations at each timestamp into a high-dimensional latent space via a fully connected layer. The timestamp masking module then masks latent vectors at randomly selected timestamps to generate an augmented context view. Importantly, masking is applied at the latent vector level rather than raw input values, as time series values may be unbounded, making it impractical to use a special token for raw data.

E.2 Transformer-based models

4) Time-Series Representation Learning via Temporal and Contextual Contrasting (TS-TCC) (Eldele et al., 2021) is an unsupervised framework for learning time-series representations from unlabeled data using contrastive learning. It transforms raw time-series data into two different but correlated views through weak and strong augmentations. TS-TCC consists of two key modules: a temporal contrasting module and a contextual contrasting module. The temporal contrasting module enhances robustness by enforcing a cross-view prediction task, where past latent features from one augmentation predict the future of another. This design encourages the model to learn transformation-invariant representations while handling perturbations from different timesteps and augmentations. The contextual contrasting module further refines representation learning by maximizing similarity among different contexts of the same sample while minimizing similarity across samples. By building on the representations learned from temporal contrasting, this module enhances the model’s ability to capture discriminative features. TS-TCC employs simple yet effective augmentations applicable to any time-series data, ensuring adaptability across diverse datasets.

5) Time Series Transformer (TST) (Zerveas et al., 2021) is a transformer-based framework for unsupervised representation learning of multivariate time series. It employs a masked Mean Squared Error (MSE) loss to train a transformer model, extracting dense vector representations through an autoregressive input denoising objective. Specifically, parts of the input are masked (set to zero), and the model is trained to predict the missing values. Since transformers are inherently insensitive
to input order. TST incorporates positional encoding to capture the sequential nature of time series data. Only the predictions on the masked values contribute to the MSE loss. The pre-trained model can be applied to various downstream tasks, including classification. For classification, the final representation vectors from all time steps are concatenated into a single vector, which serves as input to a linear output layer with parameters corresponding to the number of classes.

6) MOMENT (Goswami et al., 2024) is an open-source foundation model for general-purpose time series analysis. It follows a masked time series modeling approach, where the goal is to reconstruct masked input time series using a lightweight reconstruction head. First, a univariate time series is segmented into disjoint fixed-length sub-sequences (patches). Before patching, reversible instance normalization (Kim et al., 2021) is applied to the time series. Each patch is then mapped to a embedding, using a trainable linear projection if all time steps are observed, and a designated learnable mask embedding if some patches are masked. During pretraining, patches are randomly masked by replacing their patch embeddings with a special learnable mask embedding. These patch embeddings are then fed into a transformer encoder to learn patch representations. The transformed patch embeddings are used to reconstruct both masked and unmasked time series patches through a lightweight reconstruction head. Pretraining ensures that embeddings and high-level representations of the time series are effectively learned, with the objective of minimizing the masked reconstruction error (Mean Squared Error) between the ground truth and predicted patches.

To process multivariate time series, MOMENT operates independently on each channel, aligning with recent studies (Nie et al., 2023; Zhou et al., 2024) that support channel-wise modeling as an effective strategy. Unlike traditional architectures that use a decoder of similar size to the encoder, MOMENT employs a lightweight reconstruction head, enabling task-specific fine-tuning with minimal trainable parameters while preserving the encoder’s high-level learned features. For classification, MOMENT operates in a zero-shot setting by replacing its reconstruction head with a linear classification head, mapping patch representations to logits corresponding to class labels.

F Adapting Unsupervised Representation Learning Models for Classification

There are two approaches to adapting unsupervised representation learning baseline models for classification:

• Two-stage approach: First, sequence-level representations for each time series are obtained without access to labels, using the unsupervised representation learning model. In the second stage, a machine learning classifier (e.g., a Support Vector Machine with an RBF kernel) is trained on these representations with labeled data. The classifier applies a softmax function to generate a probability distribution over classes, with cross-entropy loss computed against the categorical ground truth labels. This approach is used for all unsupervised baselines except MOMENT, specifically for T-Loss, TNC, TS2Vec, TS-TCC, and TST.

• Direct adaptation approach: Instead of a two-stage process, the reconstruction head in the unsupervised representation learning model is replaced with a linear classification head that maps patch representations to logits corresponding to class labels. MOMENT adopts this approach by replacing its reconstruction head with a linear head that outputs logits equal to the number of classes.

G Reproduction Details for Baselines

Table 5 lists the original code sources for each baseline model. All models, except MOMENT, were trained on each dataset individually—either with labels for supervised learning methods or without labels for unsupervised representation learning methods. We collect supervised baseline model results from (Zhou et al., 2024) and (Wu et al., 2022a), except for PatchTST, which is based on our own reproduction. Additionally, we gather unsupervised representation learning baseline results from (Goswami et al., 2024) and (Yue et al., 2022).
Table 5: Code source for each baseline.

<table>
  <tr>
    <th>Model</th>
    <th>Source</th>
  </tr>
  <tr>
    <th colspan="2">Supervised Learning Methods</th>
  </tr>
  <tr>
    <td>DTW</td>
    <td>https://github.com/markdregan/K-Nearest-Neighbors-with-Dynamic-Time-Warping/tree/master</td>
  </tr>
  <tr>
    <td>XGBoost</td>
    <td>https://github.com/dmlc/xgboost</td>
  </tr>
  <tr>
    <td>ROCKET</td>
    <td>https://github.com/angus924/rocket/tree/master</td>
  </tr>
  <tr>
    <td>LightTS</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/LightTS.py</td>
  </tr>
  <tr>
    <td>DLinear</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/DLinear.py</td>
  </tr>
  <tr>
    <td>LSTNet</td>
    <td>https://github.com/laiqgoukun/LSTNet</td>
  </tr>
  <tr>
    <td>LSSL</td>
    <td>https://github.com/state-spaces/s4</td>
  </tr>
  <tr>
    <td>TCN</td>
    <td>https://github.com/White-Link/UnsupervisedScalableRepresentationLearningTimeSeries</td>
  </tr>
  <tr>
    <td>TimesNet</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/TimesNet.py</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/Transformer.py</td>
  </tr>
  <tr>
    <td>Reformer</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/Reformer.py</td>
  </tr>
  <tr>
    <td>Informer</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/Informer.py</td>
  </tr>
  <tr>
    <td>Pyraformer</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/Pyraformer.py</td>
  </tr>
  <tr>
    <td>Autoformer</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/Autoformer.py</td>
  </tr>
  <tr>
    <td>Non-stationary Transformer</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/Nonstationary_Transformer.py</td>
  </tr>
  <tr>
    <td>FEDformer</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/FEDformer.py</td>
  </tr>
  <tr>
    <td>ETSformer</td>
    <td>https://github.com/thuml/Time-Series-Library/blob/main/models/ETSformer.py</td>
  </tr>
  <tr>
    <td>Flowformer</td>
    <td>https://github.com/thuml/Flowformer/tree/main/flowformer_TimeSeries</td>
  </tr>
  <tr>
    <td>PatchTST</td>
    <td>https://github.com/yuqinie98/PatchTST</td>
  </tr>
  <tr>
    <td>One Fits All</td>
    <td>https://github.com/DAMO-DI-ML/NeurIPS2023-One-Fits-All</td>
  </tr>
  <tr>
    <th colspan="2">Unsupervised Representation Learning Methods</th>
  </tr>
  <tr>
    <td>T-Loss</td>
    <td>https://github.com/White-Link/UnsupervisedScalableRepresentationLearningTimeSeries</td>
  </tr>
  <tr>
    <td>TNC</td>
    <td>https://github.com/sanatonek/TNC_representation_learning</td>
  </tr>
  <tr>
    <td>TS2Vec</td>
    <td>https://github.com/zhihanyue/ts2vec</td>
  </tr>
  <tr>
    <td>TS-TCC</td>
    <td>https://github.com/emadedleene24/TS-TCC</td>
  </tr>
  <tr>
    <td>TST</td>
    <td>https://github.com/gzerveas/mvts_transformer</td>
  </tr>
  <tr>
    <td>MOMENT</td>
    <td>https://github.com/moment-timeseries-foundation-model/moment</td>
  </tr>
</table>

H Implementation Details for LETS-C

The experiments were conducted on a Linux machine equipped with an NVIDIA T4 Tensor Core GPU with 16GB of memory. We utilized the PyTorch v2.4.0 deep learning platform running on Python 3.11 for all models. We set a fixed random seed for all experiments to ensure reproducibility. All configurations employed the RAdam optimizer Liu et al. (2019a), with its default hyperparameters settings \((\beta_1, \beta_2) = (0.9, 0.999)\). Further, we explored LETS-C’s performance across three different text embedding models: e5-mistral-7b-instruct (Wang et al., 2024), gte-large-en-v1.5 (Li et al., 2023), and nomic-embed-text-v1 (Nussbaum et al., 2024), in addition to text-embedding-3-large in Section 5.3. Exploratory hyperparameter optimization was conducted, revealing that 1-3 1D convolutional layers and 1-2 linear layers are optimal for the performance of LETS-C across all datasets. For a detailed description of the hyperparameters, see Appendix Section I. For tokenization, we found that maintaining a precision of one decimal place optimizes performance (see Appendix Section M for details), thus we adopted that precision throughout the experiments.

I Hyperparameter Settings for LETS-C

I.1 Hyperparameter Search Space

Our hyperparameter search for the lightweight classification head included determining the optimal number of convolutional layers, whether to use dropout or pooling within the convolutional blocks, the number of dense layers, the type of activation function for each layer, and whether to employ batch normalization. We explored 1 to 5 convolutional layers and 1 to 4 linear layers. We also tested various learning rates, ranging from 0.001 to 0.05, different activation functions such as ReLU, GELU, and tanh, and truncation lengths for the text-embedding-large from 64 to 1024.

I.2 Optimal Hyperparameters

Table 6 presents the optimal configurations for our experiments across various datasets. All configurations employed tokenization with a precision of one decimal place and utilized the text-embedding-3-large embeddings. It is crucial to note that the count of linear layers includes the output layer; therefore, a configuration with one linear layer means that this single layer functions as
the output layer within the model architecture. Furthermore, all convolutional layers utilized a kernel size of 3. Additionally, each configuration used the RAdam optimizer Liu et al. (2019a) with its default hyperparameter settings \( (\beta_1, \beta_2) = (0.9, 0.999) \).

J Model Performance

Figure 3 displays a comparison of models based on the average classification accuracy across all datasets, as summarized in Table 4.

K Monetary Costs Analysis

Our approach incurs monetary costs when utilizing OpenAI’s text-embedding-3-large model (OpenAI, 2024). To compute the cost of generating text embeddings for our datasets, we followed the pricing structure of $0.130 per million tokens. Each time series sample, including all time steps, was embedded separately for each dimension. To estimate the number of tokens per sample, we considered the numerical format of the time series values, where each time step consists of a three-digit number. These digits are separated by spaces, and consecutive time steps are delimited by commas, resulting in an estimated six tokens per time step. The total token count for each dataset was then computed as:

\[
\text{Total Tokens} = \text{Total Samples} \times \text{Num. Dimensions} \times \text{Tokens per Series},
\]

where the estimated tokens per series were derived from the series length multiplied by six. The dataset characteristics, including train and test sizes, number of dimensions, and series lengths, are detailed in Table 4. Using this approach, the total number of tokens across all datasets was approximately 1.05 billion, leading to an estimated embedding cost of $137.07. It is important to note that API costs may fluctuate over time, leading to potentially different or outdated values. Furthermore, cost remains a common limitation for all LLM-based approaches that rely on closed-source paid models.

L Assessing Text Embeddings with Cosine Similarity

Figure 4 visualizes the within-class and between-class cosine similarities of text embeddings derived from the testing time series. Each matrix entry is scaled using min-max normalization to range from 0 to 1, where warmer colors in the heatmap represent higher similarities and darker shades indicate lower similarities. Diagonal entries show within-class similarities, highlighting intra-class cohesion, while off-diagonal entries reveal between-class relationships

Similar to the training set, we observe that within-class similarity consistently exceeds across-class similarity, thereby validating the hypothesis that text embeddings effectively retain and convey significant information from the underlying time series data.

M Numerical Precision for Tokenization

To explore the impact of numerical precision on the computation of embeddings, we analyzed classification accuracy across precisions 1 to 6 using four datasets: Handwriting, Heartbeat, JapaneseVowels, and UWaveGestureLibrary. We selected these datasets because they were the smaller ones among the 10 available, making the computation of embeddings more computationally affordable. Figure 5 illustrates the average classification accuracy (%) across these numerical precisions. Detailed results can be found in Table 7.

Studies in the NLP domain have shown that longer inputs do not perform well with language models (Press et al., 2020; Levy et al., 2024). Our empirical analysis supports this claim, revealing a decrease in classification accuracy with increased numerical precision when computing text embeddings. The average accuracy starts at 72.8% with precision 1 and declines to 69% at precision 6. Additionally, the percentage of AvgWins is highest at precision 1. As numerical precision increases, so does the length of the time series and the input to the text embedding model. Note that the maximum token length for text-embedding-3-large embeddings is 8191, and thus keeping precision of 1 ensures that context length doesn’t exceed the maximum permissible token length. This issue is especially problematic for datasets with longer time series, such as the EthanolConcentration dataset, which includes 1751 time steps per sample. This finding led us to opt for precision 1 in our study.

Note that these precision results also depend on the type of tokenization selected, and defining an appropriate tokenization for time series is one of the potential future directions for this work.
Table 6: Optimal hyperparameter configuration for LETS-C.

<table>
  <tr>
    <th rowspan="2">Dataset/Configuration</th>
    <th colspan="4">Model Hyperparameter</th>
    <th colspan="2">Training Process</th>
  </tr>
  <tr>
    <th>Embedding dimension</th>
    <th>Conv layers</th>
    <th>Linear layers</th>
    <th>Activation</th>
    <th>Learning rate</th>
    <th>Batch size</th>
  </tr>
  <tr>
    <td>EthanolConcentration</td>
    <td>64</td>
    <td>2</td>
    <td>1</td>
    <td>tanh</td>
    <td>0.007</td>
    <td>64</td>
  </tr>
  <tr>
    <td>FaceDetection</td>
    <td>64</td>
    <td>1</td>
    <td>1</td>
    <td>tanh</td>
    <td>0.007</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Handwriting</td>
    <td>16</td>
    <td>1</td>
    <td>2</td>
    <td>tanh</td>
    <td>0.007</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Heartbeat</td>
    <td>512</td>
    <td>1</td>
    <td>1</td>
    <td>tanh</td>
    <td>0.007</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Japanese Vowels</td>
    <td>512</td>
    <td>1</td>
    <td>2</td>
    <td>GELU</td>
    <td>0.007</td>
    <td>64</td>
  </tr>
  <tr>
    <td>PEMS-SF</td>
    <td>1024</td>
    <td>3</td>
    <td>1</td>
    <td>tanh</td>
    <td>0.007</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Self-Regulation SCP1</td>
    <td>64</td>
    <td>1</td>
    <td>1</td>
    <td>tanh</td>
    <td>0.007</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Self-Regulation SCP2</td>
    <td>1024</td>
    <td>2</td>
    <td>1</td>
    <td>GELU</td>
    <td>0.007</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Spoken Arabic Digits</td>
    <td>512</td>
    <td>1</td>
    <td>1</td>
    <td>tanh</td>
    <td>0.001</td>
    <td>64</td>
  </tr>
  <tr>
    <td>UWave Gesture Library</td>
    <td>1024</td>
    <td>1</td>
    <td>1</td>
    <td>tanh</td>
    <td>0.001</td>
    <td>64</td>
  </tr>
</table>

Table 7: Numerical Precision for Tokenization: This table reports classification accuracy (%). Red: Best performance.

<table>
  <tr>
    <th>Dataset / Precision</th>
    <th>1</th>
    <th>2</th>
    <th>3</th>
    <th>4</th>
    <th>5</th>
    <th>6</th>
  </tr>
  <tr>
    <td>Handwriting</td>
    <td>23.2</td>
    <td>15.9</td>
    <td>11.6</td>
    <td>11.1</td>
    <td>12.0</td>
    <td>11.2</td>
  </tr>
  <tr>
    <td>Heartbeat</td>
    <td>78.0</td>
    <td>78.5</td>
    <td>79.0</td>
    <td>78.5</td>
    <td>79.5</td>
    <td>78.0</td>
  </tr>
  <tr>
    <td>JapaneseVowels</td>
    <td>99.2</td>
    <td>98.4</td>
    <td>97.6</td>
    <td>96.8</td>
    <td>95.9</td>
    <td>95.1</td>
  </tr>
  <tr>
    <td>UWaveGestureLibrary</td>
    <td>90.6</td>
    <td>90.6</td>
    <td>92.5</td>
    <td>89.1</td>
    <td>90.3</td>
    <td>91.9</td>
  </tr>
  <tr>
    <th>Average ↑</th>
    <th><span style="color:red">72.75</span></th>
    <th>70.85</th>
    <th>70.175</th>
    <th>68.875</th>
    <th>69.425</th>
    <th>69.05</th>
  </tr>
  <tr>
    <th>AvgWins % ↑</th>
    <th><span style="color:red">50%</span></th>
    <th>0%</th>
    <th>25%</th>
    <th>0%</th>
    <th>25%</th>
    <th>0%</th>
  </tr>
</table>

N Generalization Across Various Text Embedding Models

To evaluate the generalization capabilities of our approach across different text embedding models beyond text-embedding-3-large (OpenAI, 2024), the performance of LETS-C was tested using three alternative embedding models: e5-mistral-7b-instruct (Wang et al., 2024), gte-large-en-v1.5 (Li et al., 2023), and nonic-embed-text-v1 (Nussbaum et al., 2024). A summary of the embedding models utilized in this study is provided in Table 8.

N.1 e5-mistral-7b-instruct

The e5-mistral-7b-instruct model (Wang et al., 2024) is based on the pretrained Mistral-7b checkpoint (Jiang et al., 2023) and was fine-tuned using the RankLLaMA training methodology (Ma et al., 2024). It employed LoRA (Hu et al., 2021) with a rank of 16 on a mixture of multilingual datasets, which included both synthetic data and a collection of 13 public datasets, yielding approximately 1.8 million examples after sampling. Proprietary LLMs, such as GPT-4, were prompted to generate diverse synthetic data with instructions in multiple languages. Leveraging the strong language understanding capabilities of the Mistral model, e5-mistral-7b-instruct achieved state-of-the-art results across nearly all task categories on the competitive MTEB benchmark. Techniques such as gradient checkpointing, mixed precision training, and DeepSpeed ZeRO-3 were applied to further reduce GPU memory requirements.

N.2 gte-large-en-v1.5

The gte-large-en-v1.5 model (Li et al., 2023) is a general text embedding (GTE) model that utilizes contrastive learning on an open-source large-scale dataset comprising unsupervised text pairs extracted from various sources. To enhance the quality of the learned text representations, high-quality text pairs with human labels from multiple sources were employed for contrastive fine-tuning. The model utilizes a multi-stage contrastive learn-
Figure 3: Model comparison based on classification accuracy, averaged across all datasets listed in Table 4. For detailed results, refer to Table 1 in the main paper.

Table 8: Summary of selected embedding models used in the study.

<table>
  <tr>
    <th>MTEB Rank</th>
    <th>Model</th>
    <th>Embedding Dimensions</th>
    <th>Max Token Length</th>
  </tr>
  <tr>
    <td>15</td>
    <td>text-embedding-3-large (<a href="https://openai.com/">OpenAI, 2024</a>)</td>
    <td>3072</td>
    <td>8191</td>
  </tr>
  <tr>
    <td>6</td>
    <td>e5-mistral-7b-instruct (<a href="https://huggingface.co">Wang et al., 2024</a>)</td>
    <td>4096</td>
    <td>32768</td>
  </tr>
  <tr>
    <td>9</td>
    <td>gte-large-en-v1.5 (<a href="https://arxiv.org">Li et al., 2023</a>)</td>
    <td>1024</td>
    <td>8192</td>
  </tr>
  <tr>
    <td>35</td>
    <td>nomic-embed-text-v1 (<a href="https://nomic.ai">Nussbaum et al., 2024</a>)</td>
    <td>768</td>
    <td>8192</td>
  </tr>
</table>

ing approach and benefits from a diverse training data mixture, enabling it to achieve strong generalization performance for single-vector embeddings.

N.3 nomic-embed-text-v1
The nomic-embed-text-v1 model (<a href="https://nomic.ai">Nussbaum et al., 2024</a>) is a fully reproducible, open-source English text embedding model with an 8192 context length that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on short and long-context tasks. To accommodate long sequence lengths, the model adapts the BERT architecture (<a href="https://arxiv.org">Devlin et al., 2019</a>) with several optimizations: replacing absolute positional embeddings with rotary positional embeddings (<a href="https://arxiv.org">Su et al., 2024</a>), using SwiGLU activation instead of GeLU (<a href="https://arxiv.org">Shazeer, 2020</a>), implementing Flash Attention (<a href="https://arxiv.org">Dao et al., 2022</a>), setting Dropout to 0 (<a href="https://arxiv.org">Geiping and Goldstein, 2023</a>), and ensuring the vocabulary size is a multiple of 64 (<a href="https://arxiv.org">Portes et al., 2024; Shoeybi et al., 2019</a>). These modifications result in a 137M parameter encoder.

O Trade-offs: Model Accuracy vs. Parameter Complexity

Table 9 illustrates the trade-off between model accuracy and the complexity of training parameters in our model, which utilizes both embeddings and time series data as inputs to a lightweight framework. To vary model size, we adjust the number of linear and convolution layers in the classification head, ranging from 1 to 5 layers each, creating model variants of different sizes.
Figure 4: Heatmaps illustrating within-class and between-class cosine similarities of text embeddings derived from the testing time series data in Japanese Vowels (left) with 9 classes and Spoken Arabic Digits (right) with 10 classes. On both axes, x and y represent different classes. Diagonal entries indicate within-class similarities, and off-diagonal entries represent between-class similarities. Warmer colors signify higher cosine similarities, while cooler colors suggest lower similarities.

Figure 5: Average classification accuracy (%) across numerical precisions 1 to 6. Results are averaged from four datasets: Handwriting, Heartbeat, JapaneseVowels, and UWaveGestureLibrary. See Table 7 in the Appendix for detailed results.

Across all datasets, we find that significant reductions in model parameters result in only a minimal loss of accuracy. This trade-off between model accuracy and parameter complexity is data-dependent. However, we generally observe a trend where a reduction in parameters leads to only a slight decrease in accuracy. Next, let’s take a closer look at three datasets: Heartbeat, PEMS-SF, and Spoken Arabic Digits, to understand how these trade-offs manifest in different contexts.

• Heartbeat: In the Heartbeat dataset, we retain 99.48% of the optimal model’s accuracy using only 75% of its trainable parameters. Specifically, the optimal model achieves an accuracy of 78% with 46,426 trainable parameters, while the second-best model achieves 77.6% accuracy with 34,820 parameters. Moreover, we retain 96.28% accuracy with a further reduction to 57.95% of the parameters.

• PEMS-SF: In the PEMS-SF dataset, the optimal model starts with an accuracy of 93.1% and 564,231 trainable parameters. Reducing the parameters to 173,866 (30.81% of the original), the model maintains 98.06% of its optimal accuracy at 91.3%. Further parameter reductions to 85,210 (15.10%), 69,077 (12.24%), and 62,901 (11.14%) result in accuracies of 90.8%, 87.9%, and 87.3%, respectively. These reductions illustrate that even significant reductions in parameters only lead
Table 9: Trade-off between model accuracy and the complexity of training parameters. The accuracy and parameters of the best model are highlighted in bold. The accuracy difference is calculated as the raw difference between the accuracies of the reduced model and the best model. The % Delta in accuracy and parameters is defined separately for each as \( \frac{\text{Accuracy of the reduced model}}{\text{Accuracy of the optimal model}} \) and \( \frac{\text{Parameters of the reduced model}}{\text{Parameters of the optimal model}} \), quantifying the accuracy and computational efficiency of the reduced model relative to our best model.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Accuracy (%)↑</th>
    <th rowspan="2">Trainable Parameters ↓</th>
    <th colspan="2">Difference | % Delta in Accuracy ↑</th>
    <th rowspan="2">% Delta in Parameters ↓</th>
  </tr>
  <tr>
    <th>Delta in Accuracy ↑</th>
    <th>% Delta in Accuracy ↑</th>
  </tr>
  <tr>
    <td>EthanolConcentration</td>
    <td>52.9<br>46</td>
    <td>283950<br>105344</td>
    <td>-6.9<br>86.95</td>
    <td>-</td>
    <td>37.09</td>
  </tr>
  <tr>
    <td rowspan="3">FaceDetection</td>
    <td>68.9</td>
    <td>3842</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>68.6<br>67.9<br>66.4</td>
    <td>2402<br>962<br>482</td>
    <td>-0.3<br>-1.0<br>-2.5</td>
    <td>99.56<br>98.54<br>96.37</td>
    <td>62.51<br>25.03<br>12.54</td>
  </tr>
  <tr>
    <td>23.8</td>
    <td>154526</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">Handwriting</td>
    <td>23.2<br>22.7<br>20.2<br>19.4</td>
    <td>107226<br>53626<br>20394<br>13426</td>
    <td>-0.6<br>-1.1<br>-3.6<br>-4.4</td>
    <td>97.47<br>95.37<br>84.87<br>81.51</td>
    <td>69.39<br>34.70<br>13.19<br>8.68</td>
  </tr>
  <tr>
    <td>78</td>
    <td>46426</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Heartbeat</td>
    <td>77.6<br>75.1</td>
    <td>34820<br>26908</td>
    <td>-0.4<br>-2.9</td>
    <td>99.48<br>96.28</td>
    <td>75.00<br>57.95</td>
  </tr>
  <tr>
    <td>Japanese Vowels</td>
    <td>99.2<br>98.9<br>98.6<br>98.4</td>
    <td>148233<br>123401<br>105353<br>100857</td>
    <td>-0.3<br>-0.6<br>-0.8</td>
    <td>99.69<br>99.39<br>99.19</td>
    <td>83.24<br>71.07<br>68.03</td>
  </tr>
  <tr>
    <td rowspan="4">PEMS-SF</td>
    <td>93.1<br>91.3<br>90.8<br>87.9<br>87.3</td>
    <td>564231<br>173866<br>85210<br>69077<br>62901</td>
    <td>-1.8<br>-2.3<br>-5.2<br>-5.8</td>
    <td>98.06<br>97.52<br>94.41<br>93.77</td>
    <td>30.81<br>15.10<br>12.24<br>11.14</td>
  </tr>
  <tr>
    <td>Self-Regulation SCP1</td>
    <td>93.2<br>92.5</td>
    <td>302626<br>99657</td>
    <td>-0.7</td>
    <td>99.24</td>
    <td>32.93</td>
  </tr>
  <tr>
    <td rowspan="4">Self-Regulation SCP2</td>
    <td>62.8<br>58.9<br>57.8<br>57.2<br>56.1</td>
    <td>334402<br>166306<br>111106<br>83330<br>76386</td>
    <td>-3.9<br>-5.0<br>-5.6<br>-6.7</td>
    <td>93.78<br>92.03<br>91.08<br>89.33</td>
    <td>49.73<br>33.22<br>24.91<br>22.84</td>
  </tr>
  <tr>
    <td>Spoken Arabic Digits</td>
    <td>99<br>98.4<br>98.1<br>98<br>97.8<br>97.6<br>96.6</td>
    <td>141790<br>70066<br>30964<br>20646<br>15487<br>10328<br>5308</td>
    <td>-0.2<br>-0.8<br>-1.1<br>-1.2<br>-1.4<br>-1.6<br>-2.6</td>
    <td>99.79<br>99.19<br>98.89<br>98.79<br>98.58<br>98.38<br>97.37</td>
    <td>49.41<br>32.90<br>21.83<br>14.56<br>10.92<br>7.28<br>3.74</td>
  </tr>
  <tr>
    <td rowspan="2">UWave Gesture Library</td>
    <td>90.6<br>88.4<br>87.5</td>
    <td>263338<br>211556<br>176298</td>
    <td>-2.2<br>-3.1</td>
    <td>97.57<br>96.57</td>
    <td>80.33<br>66.94</td>
  </tr>
</table>
to a slight decrease in performance.

• SpokenArabicDigits: For the Spoken Arabic Digits dataset, reducing the number of trainable parameters generally correlates with a minor decline in accuracy, though the trade-off is modest. The optimal model, achieving an accuracy of 99.2% with 141,790 trainable parameters, shows that even with substantial reductions to 70,066 parameters (49.41% of the original), the accuracy remains high at 99%, retaining 99.79% of the original model’s accuracy. Further reductions to 46,658 (32.90%), 30,964 (21.83%), 20,646 (14.56%), 15,487 (10.92%), 10,328 (7.28%), and 5,308 (3.74%) yield accuracies of 98.4%, 98.1%, 98%, 97.8%, 97.6%, and 96.6% respectively.

These examples highlight that efficiency in terms of trainable parameters does not linearly correspond to a loss in model accuracy across various datasets. While fewer parameters generally lead to a lower accuracy, the decrement is often proportional and manageable, making these models highly suitable for deployment in resource-constrained environments or for applications requiring rapid processing with minimal computational overhead.

P Ablation Study

Table 10 presents the results of this study, comparing the performance of our default approach, which adds embeddings to time series, to variants that exclude either embeddings or the time series.

Q Alternative Methods for Fusing Time Series with Embeddings

We explored two additional methods for fusing time series and embeddings beyond simple addition. The first method involves a Fusion network that first processes embeddings and time series data through convolutional and dense layers in two separate branches, then merges the features from both branches into a final dense network. The second method employs Concatenation, where the time series and embeddings are concatenated and processed through a lightweight classification head. Despite cross-attention being another alternative for fusing different modalities, we didn’t include it in this study due to the computational complexity it adds to the model. Table 11 presents the classification accuracy and trainable model parameters for these variations.

We observe that the addition approach in the LETS-C architecture achieves the highest average classification accuracy (76.11%) compared to the fusion network (73.40%) and concatenation (74.22%). Notably, when compared to all baselines, concatenation emerges as the second-best method after addition, which achieves the highest accuracy. The fusion approach ranks fifth overall, with only OneFitsAll and TimesNet outperforming it, following addition and concatenation, in terms of average accuracy.
Table 10: Comparison of classification accuracy (%) for different configurations: ours (both embeddings and time series), embeddings only, and time series only. Red: Best.

<table>
  <tr>
    <th>Method/Dataset</th>
    <th>EC</th>
    <th>FD</th>
    <th>HW</th>
    <th>HB</th>
    <th>JV</th>
    <th>PEMS-SF</th>
    <th>SCP1</th>
    <th>SCP2</th>
    <th>SAD</th>
    <th>UW</th>
    <th>Average ↑</th>
    <th>AvgWins % ↑</th>
  </tr>
  <tr>
    <td>LETS-C (Ours)</td>
    <td>52.9</td>
    <td>68.9</td>
    <td>23.8</td>
    <td>78</td>
    <td>99.2</td>
    <td>93.1</td>
    <td>93.2</td>
    <td>62.8</td>
    <td>99.2</td>
    <td>90.6</td>
    <td><b>76.17</b></td>
    <td><b>60%</b></td>
  </tr>
  <tr>
    <td>embedding only</td>
    <td>38</td>
    <td>59.4</td>
    <td>20.1</td>
    <td>80</td>
    <td>99.2</td>
    <td>92.5</td>
    <td>88.7</td>
    <td>63.9</td>
    <td>99.2</td>
    <td>88.4</td>
    <td>72.94</td>
    <td>40%</td>
  </tr>
  <tr>
    <td>time series only</td>
    <td>42.6</td>
    <td>69.6</td>
    <td>25.1</td>
    <td>76.1</td>
    <td>98.1</td>
    <td>89</td>
    <td>93.2</td>
    <td>58.3</td>
    <td>98.9</td>
    <td>83.8</td>
    <td>73.47</td>
    <td>30%</td>
  </tr>
</table>

Table 11: Comparison of classification accuracy (%) and trainable model parameters (millions) for alternative methods of fusing time series with embeddings. Higher AvgWins and accuracy averages indicate superior performance, while lower model parameter averages suggest greater computational efficiency. Red: Best performance.

<table>
  <tr>
    <th colspan="11">Accuracy ↑</th>
  </tr>
  <tr>
    <th>Method/Dataset</th>
    <th>EC</th>
    <th>FD</th>
    <th>HW</th>
    <th>HB</th>
    <th>JV</th>
    <th>PEMS-SF</th>
    <th>SCP1</th>
    <th>SCP2</th>
    <th>SAD</th>
    <th>UW</th>
    <th>Average ↑</th>
    <th>AvgWins % ↑</th>
  </tr>
  <tr>
    <td>LETS-C (Addition)</td>
    <td>52.9</td>
    <td>68.9</td>
    <td>23.8</td>
    <td>78</td>
    <td>99.2</td>
    <td>93.1</td>
    <td>93.2</td>
    <td>62.8</td>
    <td>99.2</td>
    <td>90.6</td>
    <td><b>76.17</b></td>
    <td><b>70%</b></td>
  </tr>
  <tr>
    <td>Fusion network</td>
    <td>44.1</td>
    <td>66.5</td>
    <td>23.6</td>
    <td>76.6</td>
    <td>98.1</td>
    <td>86.1</td>
    <td>92.8</td>
    <td>56.1</td>
    <td>99.2</td>
    <td>90.9</td>
    <td>73.40</td>
    <td>20%</td>
  </tr>
  <tr>
    <td>Concatenation</td>
    <td>43</td>
    <td>65.1</td>
    <td>22.5</td>
    <td>79</td>
    <td>98.9</td>
    <td>93.1</td>
    <td>93.9</td>
    <td>58.9</td>
    <td>99</td>
    <td>88.8</td>
    <td>74.22</td>
    <td>30%</td>
  </tr>
  <tr>
    <th colspan="11">Trainable Parameters (M) ↓</th>
  </tr>
  <tr>
    <th>Method/Dataset</th>
    <th>EC</th>
    <th>FD</th>
    <th>HW</th>
    <th>HB</th>
    <th>JV</th>
    <th>PEMS-SF</th>
    <th>SCP1</th>
    <th>SCP2</th>
    <th>SAD</th>
    <th>UW</th>
    <th>Average ↓</th>
  </tr>
  <tr>
    <td>LETS-C (Addition)</td>
    <td>0.28</td>
    <td>0.003</td>
    <td>0.15</td>
    <td>0.04</td>
    <td>0.14</td>
    <td>0.56</td>
    <td>0.30</td>
    <td>0.33</td>
    <td>0.14</td>
    <td>0.26</td>
    <td><b>0.22</b></td>
  </tr>
  <tr>
    <td>Fusion Network</td>
    <td>0.19</td>
    <td>0.35</td>
    <td>0.33</td>
    <td>0.28</td>
    <td>0.16</td>
    <td>5.54</td>
    <td>0.37</td>
    <td>0.27</td>
    <td>0.23</td>
    <td>0.04</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>Concatenation</td>
    <td>0.42</td>
    <td>0.009</td>
    <td>0.26</td>
    <td>0.17</td>
    <td>0.11</td>
    <td>0.28</td>
    <td>0.23</td>
    <td>0.39</td>
    <td>0.09</td>
    <td>0.46</td>
    <td>0.24</td>
  </tr>
</table>

Reviewer paper summaries: LETS-C leverages text embedding models to encode time series data and combines it with a lightweight CNN-MLP classification head for efficient time series classification. This method directly uses text embedding model to embed time series, which avoids expensive fine-tuning for efficiency. Very solid work. The writing is clear, the logical flow is fluent, the experiments are solid, and the discussion is comprehensive.

The paper suggests a "lightweight" solution for time-series classification that is leveraging LLM embeddings for a given time series.
Summary of strengths: LETS-C leverages text embedding models to encode time series data and combines it with a lightweight CNN-MLP classification head for efficient time series classification. This method directly uses text embedding model to embed time series, which avoids expensive fine-tuning for efficiency. Very solid work. The writing is clear, the logical flow is fluent, the experiments are solid, and the discussion is comprehensive.

The paper is really well-written. The idea is clear and the evaluation is convincing.
Summary of weaknesses: Here are two suggestions to improve this work:
1.	In the baseline models, only one baseline is an LLM-based model. However, as far as I know, many LLM-based methods have been developed for temporal data. Adding two or three LLM-based models to the baselines would make the comparison more convincing.
2.	Since the fusion of the original time series and time series embeddings performs better than using either of them alone, they must complement each other's knowledge. Deeper discussion of why this happens could be provided to enhance understanding.

The only issue that I have with the paper is the novelty of the results and the impact of the contribution. In principle, it is well known in NLP community that LLM embeddings provide rich dense vector representations for sequential data and naturally using such reach representations as input could enable a light-weight classifier to achieve good results.
Comments, suggestions and typos: Here are two suggestions to improve this work:
1.	In the baseline models, only one baseline is an LLM-based model. However, as far as I know, many LLM-based methods have been developed for temporal data. Adding two or three LLM-based models to the baselines would make the comparison more convincing.
2.	Since the fusion of the original time series and time series embeddings performs better than using either of them alone, they must complement each other's knowledge. Deeper discussion of why this happens could be provided to enhance understanding.

I think that paper could be a great fit for a workshop specifically oriented towards time series and/or representation learning.
Overall assessment scores: 3

3.5
Metareview score: 3.5
Metareview: The paper introduces LETS-C, a lightweight method for time series classification that leverages pre-trained text embeddings from LLMs without fine-tuning. By encoding time series data into text-like sequences and combining raw signals with embeddings via element-wise addition, LETS-C achieves state-of-the-art performance on the UEA benchmark while using fewer trainable parameters than existing approaches.
Summary of reasons to publish: 1. First to demonstrate that off-the-shelf text embeddings (e.g., GPT-4, BERT) effectively encode time series data, enabling competitive performance with minimal computational overhead. This bridges NLP and time-series analysis in a novel, practical way.
2. Extensive experiments (40+ datasets) validate SOTA results against 15 baselines, including recent LLM-based methods (e.g., MOMENT, UniTS). The parameter efficiency addresses overparameterization concerns in LLM-for-time-series research.
3. Well-structured writing, code/software release, and thorough ablation studies (e.g., cross-attention vs. element-wise fusion).