TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series
Authors: Chenxi Sun
Hongyan Li
Yaliang Li
Shenda Hong
Venue: ICLR 2024 poster (2024)
Keywords: Time Series
Embedding
LLM
Primary area: general machine learning (i.e., none of the above)

Abstract: This work summarizes two ways to accomplish Time-Series (TS) tasks in today's Large Language Model (LLM) context: LLM-for-TS (model-centric) designs and trains a fundamental large model, or fine-tunes a pre-trained LLM for TS data; TS-for-LLM (data-centric) converts TS into a model-friendly representation to enable the pre-trained LLM to handle TS data. Given the lack of data, limited resources, semantic context requirements, and so on, this work focuses on TS-for-LLM, where we aim to activate LLM's ability for TS data by designing a TS embedding method suitable for LLM. The proposed method is named TEST. It first tokenizes TS, builds an encoder to embed TS via instance-wise, feature-wise, and text-prototype-aligned contrast, where the TS embedding space is aligned to LLM’s embedding layer space, then creates soft prompts to make LLM more open to that embeddings, and finally implements TS tasks using the frozen LLM. We also demonstrate the feasibility of TS-for-LLM through theory and experiments. Experiments are carried out on TS classification, forecasting, and representation tasks using eight frozen LLMs with various structures and sizes. The results show that the pre-trained LLM with TEST strategy can achieve better or comparable performance than today's SOTA TS models, and offers benefits for few-shot and generalization. By treating LLM as the pattern machine, TEST can endow LLM's ability to process TS data without compromising language ability. We hope that this study will serve as a foundation for future work to support TS+LLM progress.

Published as a conference paper at ICLR 2024
TEST: TEXT PROTOTYPE ALIGNED EMBEDDING TO ACTIVATE LLM's ABILITY FOR TIME SERIES
Chenxi Sun 1,2,3 Hongyan Li Yaliang Li Shenda Hong
NNational Key Laboratory of General Artificial Intelligence. Peking University
2 Key Laboratory of Machine Perception (Ministry of Education), Peking University
School of Intelligence Science and Technology, Peking University
PPKU-WUHAN Institute for Artificial Intelligence
5 Alibaba Group
National Institute of Health Data Science, Peking University
Institute of Medical Technology, Health Science Center of Peking University
(chenxi_sun) @pku. edu, en ya yaliang 11 i @a baba- inc. com, hongshendafpku.etu..om
ABSTRACT
This work summarizes two ways to accomplish Time-Series (TS) tasks today's s Large Language Model (LLM) context: LLM-for-TSS (model-centric) designs and trains a fundamental large model, or fine-tunes a pre-trained LLM for TS data; TS-for-LLM (dala-centric) converts TS into a model-friendly representation enable the pre-trained LLM to handle TS data. Given the lack of data, limited resources, semantic context requirements, and on, this work focuses TS- for-LLM,, where we aim to activate LLM's ability for TS data by designing TS embedding method suitable for LLM. The proposed method is named TEST. It first tokenizes TS, builds an encoder to embed TS via instance-wise, feature-wise. and rex--roootyppe-aignee contrast, where the TS embedding space is aligned LLM's embedding layer space, then creates soft prompts to make LLM more open that embeddings, and finally implements TS tasks using the frozen LLM. We also demonstrate the feasibility of fS-for-LLM through theory and experiments. Experiments are carried out on TS classification, forecasting, and representation tasks using eight frozen L.Mss with various structures and sizes. The results show that the pre-trained LLM with TEST strategy can achieve better or comparable performance than today's s SOTA TS models and offer benefits for few-shot and generalization. By treating LLM as the pattern machine, TEST can endow LLM's ability process TS data without compromising language ability. We hope that this study will serve as a foundation for future work to support TS+LLM progress.
INTRODUCTION
Implementing Time-Series (TS) tasks, such as medical, industrial, and meteorological, research- intensive field [Snn et al.) (2020). The relevant models evolved from statistical models to RNNs, CNNs, and Transformers. Nowadays, we see fast growth and remarkable performances of Large- scale pre-trained Language Models (LLM) in NLP and CV fields Zhao et al. (023). Consequently, seems natural to inquire whether LLMs can be used for TS tasks. However, according to experi- ments, most pre-trained L.MSs have not made significant progress in relation to abstract TS.
In answer to this requirement, we envision two ways to achieve the paradigm of TS$LLM M
LLM-for-TS (model-cene) cenericc modify LLM). For TS data, design and train fundamental Large Model from scratch (LM-of-TS)) then fine-tune the model accordingly for various downstream tasks. Or, fine-tune the existing pre-trained LLM and convert it from text tasks to TS tasks;
"Corresponding authors
'This categorization focuses on the requirement for changing the model. But from technology. LLM+TS he achieved by pre-training, fine-tuning, tool-augmented methods, external encoders, and their ensemble.

---

Published as a conference paper at ICLR 2024
TS- (data-centric, modify TS). Based on the existing LLMs, furthest freezing them, design some mechanisms to customize TS for them by creating fiiendyyy TS representation.
We acknowledge that the first way, particularly developing and training model from scratch, is the most essential solution since pre-training is the crucial step of instilling knowledge to the model. And the second way is actually challenging to break beyond the model's original capabilities. How- ever, in this work, we still focus on the second way due to the following three considerations:
Data perspective. LLM-for-TS methods, especially when building foundation model, necessitate large dataset, but TS is professional, the largest dataset is less than 10GB, which is much smaller than that for NLP Zhou et al.] (2023); -S-foo-LLMM methods can
relatively small dataset as its objective is solely to assist the existing LLM in inferring TS; Model perspective. LLM-for-TS methods focus on vertical industries. Because of the major disparities in TS across domains, various large models targeting medical TS, industrial TS, etc. must be built and trained from the start; TS-for-LLM methods need little or even no training. By utilizing plug-in modules, it makes the utilization more general and convenient: Usage perspective. LLM-for- methods are appropriate for instances involving specialists; TS-for-LLM methods maintain LLM's textual capabilities while providing rich complementing semantics, being easily accessible and user-friendly.
Without changing the existing model, the most natural approach is treating TS as text data. For example, a possible dialogue is: [Q] Diagnose if patient has sepsis through the following mean arterial pressure sequence in mm Hg: 88, 95, 78, 65, 52, 30. [A] Yes. However, TS is often multivariate while text is univariate. For example, excepting mean arterial pressure, dozens of vital signs, and laboratory values, such as heart rate, lactic acid, etc.. need to be included when diagnosing sepsis. One intuitive method is divide multivariate TS into multiple univariate sequences and input them into L.M one by one. However, this will lead to three drawbacks. First, different prompt sentences, data order, and connection statements will produce different results; Second, long input sequence likely to make LLM inefficient and hard to remember the previous univariate TS; Third, the crucial aspects of multivariate dependency in TS will be ignored.
To address the above issues and achieve TS-forLLLM, we do not directly input TS into LLM, but instead, we first tokenize TS, then design an encoder to embed them, finally skip the embedding layer to input them into LLM. In this way, the core is to create embeddings that the LLM can understand.
High-quality TS embedding can be employed as the computational phenotype that the deep learning model can understand Hong et al. (2023). To make the embedding understandable by language models. Most multimodal approaches use alignment, for example, aligning text embedding and image embedding through text descriptions of the image |Wan et al. (2023). However, TS lacks visual cues and has an annotation bottleneck caused by its complex characteristics. Only a few specific TS, such as ECG, have text descriptions in each segment, where the image-text matching route could be implemented. But in most cases, it's not feasible.
Contrastive Learning (CL) can avoid the annotation bottleneck through designing pretext tasks by utilizing intrinsic information instead of relying on pre-defined prior knowledge. Currently, CL methods for TS data has also advanced Meng al. (023b)). These methods evaluate the effective- ness of TS embedding through follow-up classification, prediction, or clustering models, such as SVM Franceschi al. (2019). However, these simple and newly-trained models are considerably different from the complex and pre-trained L.M.. The representation vector generated by uncon- strained CL likely to deviate greatly from the LLM's cognitive embedding space.
To address the above issues, we propose an embedding method for TimE Series tokens to align the Text embedding space of LLM (TEST), Based on CL, TEST uses text embedding vectors as prototypes to constrain TS' embedding space and highlights feature- wise patterns. We show that TEST can activate LLM's 's ability as pattern machine. The contributions of this work are:
Summarize two TS+LLM paradigms, LLM-for-TS. TS, fS-MorLLLM with their potential methods;
Propose TEST for TS-for- LLM. TEST can produce the similarity- based, instance-wise, feature- wise, and text-prototyper pe-tyype-allggeee plaaigeed embedding for TS tokens. We prove that prompt tuning is al- most equivalent to supervised fine-tuning when TS embedding and word embedding are aligned;
Experiments on TS classification, forecasting, few-shot, and representation tasks demonstrate that TEST can activate LLM's capability to archive TS tasks, where the random and unsatisfac- tory results produced by original LLMs can be elevated to the baseline.
2

---

Published as a conference paper at ICLR 2024
Category
Mcans
Prus
Cons
Work
Specialized, accurate
Not universal,
Pre-training Ma al. 2023
LM-of-TS
Training
large datasets
Earth transformer Bi et al
2023
End-to-end, accurate
More experiments,
GPT4T et al. 2023
Tuning
lose language ability
I.L44T Chang ct al. 2023
LLM-for-TS
PrompiCasi Xue & Salmm
2023
Tool
Parameter -ffeccceett less experiments
Need experts, need annotation
Health Learner Liu et al.
2023
augmented
METS Li al. 2024
Text2HC Sxh2HCCCChung.o al. (0033
External
Parameter-efficient multiple abilities
TS-
Weak robust
TEST
encoder
Table I: Existing Work about TS+LLM
As the name of TEST implies, it's forward-lookee looking test that hope to lay the groundwork for future study. And it does give LLM new capabilities and highlight its qualities as pattern machine.
RELATED WORK
2.1 TIME SERIES AND LARGE LANGUAGE MODEL
There hasn't been much research done TS+LLM because this field is still in its infancy. We summarize the existing work in Table D I.LM-for-TS with changing the model can be achieved through tuning tool augmented means; TS-for-LLM with changing the data can be achieved through building the external encoder.
LM-of-TS Ma et al. (023) trains fundamental and accurate model based on accumulated domain TS data, but it can be difficult to construct large well-labeled dataset due to data acquisition and annotation costs. By comparison, Supervised Fine-nuning (SFT) in LLM-for-
Chang et al. (2023) has a relatively smaller workload than pre-training, but it can make the LLM lose its language capabilities and its advantages over sophisticated model designed specifically for TS tasks unclear. Regarding TS as the text sequence and using prompts as the augmented tool Liu et al. (2023) could input numerical TS into LLM directly, but it is inaccurate. requires more experience, and will fail for multivariate TS. The multimodal methods Li et al. (2024) could align the text and TS, but apart from ECG, most TS datasets have no segment annotation.
2.2 TIME SEIRES EMBEDDING
TS embedding can provide identities by including typical, associated, and dependant attributes. CL-based methods can get the data representation Chen et al. (2020), employing the instance dis- crimination pretext task to bring similar pairs closer while pushing dissimilar pairs apart in the embedding space. Some efforts have been made to implement instance-level contrast Woo et al. (202bb) Zheng al. (2023), lemp contrast (Meng et al. (2023c); Franceschi et al. (2019), and clustering-level contrast Meng et al. 2023a on TS data, with promising results. However, the direct contrast cannot bridge TS embedding and the LLM's comprehensible space. In our setting. we prefer to freeze the pre-trained L.MM and let the embedding compromise. That is, we use the text token embedding in LLM limit and guide the TS token embedding.
Inspired by the prototype-level contrast Caron et al. 2020), which goes beyond the independence assumption and exploits latent cluster information present within samples. We can select some text embeddings as basic prototypes to lead the learning. However, in addition to the alignment, we still need consider issues of prototype selection, differentiation MMnn et al.) (2023c), uniformity Wang & Isola (2020), stability Huang et al. (2023) and etc.
3 METHODS
TEST has two key steps: In Figure [ build an encoder to embed TS; In Figure 2 create prompts to make the LLM can accept TS embeddings as input.
3

---

Published as a conference paper at ICLR 2024
Autoencading
Feature matrix
contrest
Pojectos
Positive
Augmentation
Anchor
Shape
Encodir
Similarity
Text prototype
Negative
Frequency
alignment
Value
Frequency
Value
Shape
Valoa
Shape
high
Frequency
Figure 1: Text-protooype-aligned TS Embedding by Instance-wise and Feature-wiss Contrast
3.1 TS TOKEN AUGMENTATION AND ENCODING
Definition (Token Embedding of Time Series) multivariate time series x {2 {TdiT.D
1t=1,d=1 has to M-dimensional representation space by an embedding function fe 8& E RRDX + Ex RRM, Finally, the token embedding list of is {cx)
D variables and T time points. It can be segmented to list of K non-overlapping subsequences {sx] by segmentation function fa - 8, where the length of sæ Zı,a, is arbitrary, 1 S t <t₃ S T. We call as the token list of time series x. Further, each token can be embeded
fe(ff(())
through the classical sliding window method in representation learning Yuc al. (2022) = fz). We define TS token s as the anchor instance. Its positives 8* are the augmented instances, a gweak Tweekk (jitte--ndd- scale strategy, adding random variations to the signal and scale up its magnitude), s satong ~ Tstrong ppermutation-anddjiteee
We first tokenize TS into some cegnrentiiosssseeeepeessstttttnn
strategy, splitting the sequence into a random number of segments and ran- domly shuffling them) Eldele al. (2021). Its negatives s from non-overlapping instances which do not have the same subsequence as 8.
built neural network the encoder embed in- stance into vector = fe(s). We also trained a decoder fa by using the auto-encoding loss Lae き Σi sim(s, sa(e)) to ensure the representativeness of the embedding and subsequent verification. Because our primary goal is to retrieve the encoder, this decoder can likewise un- built without harming the future process.
After getting
3.2 INSTANCE-WISE AND FEATURE- WISE CONTRAST
The basic instance- wise CL treats each instance independently and design the instance discrimina- tion pretext task to keep similar instances close and dissimilar instances far away. To prevent embed- ding space collapse, we treat augmented views of the same instance as the unique positive pair, and all remaining ones within the B size minibatch negative pairs He et al. (2020). The instance-wise contrastive loss is shown in Equation| Where given the instance embedding t//- we construct projection head fp. which is one-layer MLP to obtain fp(e). a oec,e=/~) is used to calculate the similarity between two projected vectors through a similarity function sim like cosine similarity with the instance-level temperature parameter
explo(e,
Line - llog
explo et)) expexp(((eee)
(I)
o(c, sim(sp(e), (o(et//()t)))
4

---

Published as a conference paper at ICLR 2024
We also propose feature-wist contrast method to break the independence between instances. As shown in Figure after embedding, a feature matrix R Bx M is formed by the representation vectors of instances in minibatch. Where each row is embedding of a instance, thus rows could be regarded as soft labels of instances which are used in Equation [ In addition to rows, columns of feature matrix also have semantic information. [Li et al. (2021) proposed that the columns could be further regarded as cluster representations. However such cluster-wise methods require prior knowledge to pre-specify the number of clusters, which is non-trivial for the unlabeled TS data in this work. Thus, we propose to regard the columns as the soft labels of features and perform discrimination between groups of similar features.
For an anchor feature matrix m, where m is the B-th row copy of the vector we obtain a positive feature matrix m+ and a negative feature matrix m where m = [e:] E BxM, We mark the columns in the matrix m m². As expressed by the item before the right arrow in the Equation 2 the feature-wise contrast mainly align and differentiate the same feature column among the positive and negative. However, this may cause the representation space shrink within a small area. We find that ensuring differences between features can better address this issue. That is, we suggest the contrast between different feature columns as shown in the item after the right arrow.
M Σ (a(ms, m,)
M
exp(o(ms,, (m)
Liee
amm. m,
j=1 (ex m²)) + exp(o my
Aligument
Difference
category uniformity
More importantly, the injection of feature column differences can also greatly assist in the sub- sequent implementation of text-prototype-alinnee contrast. Because that contrast will apply the selected text token embedding to the feature columns, like coordinate axes.
3.3 TEXI-POOTOYYE-ALIGNES CONTRAST
The pre-trained LLM has its own token embedding. e.g., small, medium, and big GPT-2 embed text tokens from word dictionaries into representation spaces with 768, 1024, and 1280 dimensions. Naively, we can align the token embedding of TS and text using the similarity estimation. Although TS tokens lack text annotation, we can place their embedding near typical text descriptions of TS, such as value, shape, and frequency. In this fashion, it is intuitively expected that various TS tokens can represent various descriptive terms such as small, big, up, down, stable, fluctuating, and so on. Naturally, the example above is based on the closest neighbor principle because the embedding space of text token is discrete, akin to vector table, but that of our TS token continuous.
However, of course, the actual outcomes will not match what we expect because we are not providing the supervised label or ground truth. For example, the embedding of a subsequence with an upward trend may be very close to that of decline word, or even that does not describe the trend. But it irrelevant whether semantics can be understood by us. As usual, the fact is that humans cannot comprehend the model's perceptual mode.
Recently, researchers proved that LLMs are pattern machines Mirchandani et al. (023). Thus, in this work, we achieve "TS - pattern text" activate LLM's ability for TS tasks. The choice of text prototype can be relaxed, not necessarily the description related to TS.
In this work, choose P representative text embedding tp pivots/prototypes, and map TS em- bedding to them. In high dimensional space, almost all vectors are pairwise orthogonal Hopcroft & Kannan (203)), thus the number of prototypes rather than the type does matter, and their differences can be reflected in single dimension/featuue. Thus, the modeling function of the text prototype tp realized by feature-wise contrast. As expressed by Equation 3 the alignment term guarantees that the two space ranges are roughly the same through the similarity constraint, the contrast term uses fp as the coordinate axis to map the TS embedding, making the representation values in text coordinate axes of similar instance similar. The feature matrix is no longer obtained through the projector but through the prototype mapping e lp + m.
P
Liext
[sim(tpr,e)
- Cfeale -t,, tp,et - tp,e - tp)
(3)
Text alignment
contrast
5

---

Published as a conference paper at ICLR 2024
3.4 LEARNABLE PROMPT EMBEDDING
Even TS has been described using an embedded representation that the LLM can understand, LLM still has be instructed how do subsequent TS tasks.
Prompt engineering like template and chain-of-thought intuitive. Their con- texts are coherent in human semantics, but a TS embedding list has no human se- mantics, it is more about a pattern se- quence. Thus, to create a more consistent prompt pattern, we train soft prompt by p-tuning Lester et al. (202T) make LLM be casier to understand the imput. These soft prompts are task- specifcciiiic embedding. learning through the loss from LLM's ' out- put and task ground truth in Equation
Trainable layer
Language model
Soft prompr
Classifier
Classificrtion
[ik)
Encoder
Decoder
promp = Lyog/ett /els ltrgg/etscooocatt (concat(pe, e)) (4)
Token
Question
Regression
GPT4TS Zhou al. 2023 has proved the feasibility that SFT can make LLM apply TS. Based on this, we demonstrate the feasibility of TEST by proving the equíva- lence between soft prompt and SFT.
Train
Fine-tune
Fine-tune
Figure 2: Framework of LLM for TS Tasks
Consider a conditional generation task where the input x is a context and the output is sequence of tokens. Assume an autoregres- sion LLM (o(yaa) (x) with parameter é, [x; y]. The inference of pre-trained LLM is computing h2 as a function of 2; and the past activations in its left context, Y L.Mo(z,,h) The past h, in
the soft prompt turning with prompt peo h = peoli.:),
if E peidx The SFT from LLM (M; h2), otherwise
TS-LLM is Equation Its transformation shows that the soft prompt tuning approximately equivalent to SFT.
max pox max
Σ
log Po (2/(/be<))= = Σ
logPo+a + 6zilhez)
tEYia
YEYia
log P (z: hee))
log (6zz(hes)
ie Yidx
REpedax
(5)
log
log Jogpa(6zin) (ō21(22))
REYid
SEpeids
Text TS alignment
Prompt pea
Frosen L.I.M
Equation 5 also suggests that the projection space TS tokens should preferably cover the complete set of text embedding space. Thus, we utilize clustering to find P represeentative text prototypes. The process of using LLM to infer TS shown in Figure 2 this framework, the text data is input into the embedding layer of LLM, while the prompts and TS embeddings skip this layer.
EXPERIMENTS
The core of TEST is train encoder Se and soft prompt pe described in Algorithm The encoder must can extract relevant information from TS, needs to be time- and memory-efficien, and has allow variable-length inputs. Thus, we build a causal TCN with 10 layers of convolution blocks. Each convolution block is sequence of GELU, DilatedCony, BatchNorm, GELU, Dilated- Cony, with skip connections across each block. The DilatedConvs have dilation of in each layer i of convolution block. A final convolution block is used to map the hidden channels to the output channel whose size is the same as the LLM's embedding size.

---

Published as a conference paper at ICLR 2024
Algorithm Training TEST
for in epochs do
for e in epochs do
2: // UPDATE ENCODER
10:
UPDATE PROMPT
3: f,. = 8,. - o, (Lins + Ltest)
11:
pe PE n Ppromp
4: UPDATE DECODER (OPTIMAL)
12:
// FINE TUNE DECODER (OPTIMAL)
5: Of, U3a -, Fae
13:
Ofa- Osa V Lreg
6: // UPDATE PROJECTOR
14:
I/ UPDATE CLASSIFIE (OPTIMAL)
7: f% 03p 8,
8,, Lins
15:
B,. 8,. nyVe Lets
8: end for
16: end for
Model
Size
Embed. dimension
The used LLMSs are as listed in Table Each encoder and soft prompt of LLM are trained using the Adam optimizer on 20 NVIDIA Tesla V100-SXM2 GPU with CUDA 11.3.
Ben Devlin et al. 2018
110M, 335M
748, 1024
GPT2 Radford el al. 2019
117M, 34SM, 774M
768, 1024, 1280
ChatGLM Du ct a. 2022
6B
4096
LLaMa2 Touvron al. 2023
7B,
13B
4096
Table 2: The Used Language Model
We compare method to 5 kinds of methods including 12 basclines: 1) LLM--A methods Xue & 2023) y Liu et al. (2023) with the classification template Classify the given [domain) sequence either lclass label] [class label]: [numerical sequence]. [A] and the forecasting template [Q] Forecast the next value of the given [domain] sequence: [numerical sequence). [A]: 2) SFT LLM-
Salim @alimm2023))
for-TS method GPT4TS Zhou al. (2023); ; 3) classical TS models DWT, DWTD Bagnall al. (0018). INNED, and TCN Tan et al. (202T); 4) SOTA TS models Informer Zhou et al. (2021).
DLinear Zeng et al. (023). and TimesNet Wu al. (2023): 5) SOTA CL-based TS models Tloss Franceschi et al. (2019), TS2 Yue et al. (2022), and CoST Woo et al. (2022a).
(The appendix has more compared classical SOTA models and detailed results about long-term, short-term, few-shot, and zero-shot forecasting, multivariate time series classification, and representation tasks.). Overall, after using TEST, when the size of LLM reaches about 300M, their accuracy comparable to SOTA model.
The overall results are shown in Figure (The
4.1 CLASSIFICATION
We present accuracy scores for all 128 kinds of univariate TS datasets in UCR archive Dau et al. 2019) and all 30 kinds multivariate TS datasets UEA archive Bagnall et al. (2018).
Accuracy. In Figure 3 (a-b), TEST makes the classification accuracy of LLM increase significantly. LLM's original classification performances are demonstrated through two QA results. almost guesses the classification labels random, especially for multivariate TS. After using TEST, GPT2- 774M, which has the median accuracy among all models, can improve accuracy by at least 18% for univariate TS and 25% for multivariate TS. TEST makes most LLMs comparable to, if not better than, the existing models. When the size reaches about 300M, the accuracy can exceed TS baselines; When the size reaches about 700M, the accuracy can exceed SOTA TS transformers.
Ablation. In Figure (c-d), different text prototypes will lead to different results. We set 3 groups of text prototypes: embeddings value, shape, frequency, and embeddings of 3 or 10 cluster centers. Choosing prototype group that more accurately represents LLM's entire text embedding space can improve the performance. This is also suggested by Equation 5 Different prompt types, initializa- tion, and length will lead to different results. We compare the soft prompt with the hard prompt of Classify the given [domain] sequence as either [class label] or [class label]: [TS embedding). The accuracy differs by at least 10%. We set random initialization from uniform distribution and task description initialization from Classify the given sequence. The latter makes the training converge faster. When the model reaches IB. prompt length of 10 can achieve excellent results.
4.2 FORECASTING
We present short-fore foott-forecatiing MSE scores for all 19 kinds of varied time series datasets in TSER archive [Tan et al. (202I), and foreccstc MSE scores for popular real-world benchmark datasets including weather, traffic, electricity, IIII, and ETT from Wu et al. (023)
7

---

Published as a conference paper at ICLR 2024
Univariate Casiification
TSER forecastinn MSE.
Multivariate Clessification
Representation
(GPT2-77MM)
6
General Forecasting
Foracasting
TS2Vec
Long-term Forecasting
Fow-ssot Farecasting
cisseiiiation accursty
00-
forecazting
0.8
0.
70
04
60
0.0
ZUM
MtM
opt2 sis
opty 6173-714
Litto 150
Interned
Touetter
Tass
8007
Me EST Dury
Corara
dlossircatton acouracy
100
90
Specilic foracasting
Generol forecasting
60-
20
16
GPD 7/
Dloud
stow
BSAT 85RT-300 0972.171
200M
(972.5400
0922-716
71277744
Chartrad Licate-ng
Latto Lrogs
oWID
Blineer
Pash
6P72
LoW loun)
Clos cal
S01A moses
strtt Sereecatingg M55
UCB UCRCiinniiiceio accurecy
0.0
0.6
parpoo adtuted petopw
4972 3A51
0970
Scthelssittation ercuratty
100
rspresenaaiion
0.7
dlsse ficatinn
85
GPT-117M
GPT-345M
GPT-774M
60
Alter cose
Pakk
Abcor
1.00
Prompt longth
Figure 3: Experiment Results. (a-d) shows the classification results; (e-h) shows the forecasting
results; (i) shows the representation results. The red dashed line represents the best result.
Accuracy. In Figure (e-f), TEST makes the forecasting accuracy of LLM increase significantly and comparable to SOTA models. When the size reaches about 300M, the accuracy can exceed SOTA TS transformers.
Generalization. We fuse 19 datasets into I dataset and test the method on this fused dataset. As shown in Figure 3 (g), compared with baselines, LLMe-bsse models have better generality.
Few-shot. LLM has demonstrated remarkable performance few-shot learning. Based on the settings in Zhou et al. (2023). we present fhw-shot forecasting for 10% time steps in training datasets. As shown in Figure (b), TEST achieves the best performance and demonstrates a relative average MSE reduction of 23.5%.
8

---

Published as a conference paper at ICLR 2024
4.3 REPRESENTATION
Representation learning. Learning universal rep- resentations for TS is a fundamental but challeng- ing problem. Both TEST's first step (creating TS embedding) and second step (LLM's output) can achieve this task. Based on the classical representa- tion learning task, we evaluated the effectiveness of
Active
important final night
TEST representation using SVM classifier on UCR dataset. Note that using simple classifier can bet- ter reflect the presentation effect. In Figure (i),
the embedding in TEST's first step is comparable to
White important change loap happy actively limit finally
SOTA representation methods, and the embedding in TEST's second step outperform them. This
indicates that after using LLM, the representation of TS becomes discriminative.
Figure 4: Matching TS Embedding to Words
Case. We use nearest neighbor method to find the text that TS token matches to in the word embedding space of frozen LLM. In Figure 4 the majority of the identified words are sentiment- related adjectives and nouns. We speculate that by prompting, the model will treat TS classification task as sentiment classification task. Thus, introducing prompt is like introducing shortcut for LLM. Besides, the matched words are like a kind of textual Shapelet for TS segmentation, representing TS through a series of patterns. Instead of regarding TS as a sequence of numbers, we suggest using words to identify patterns in TS as LLMs without SFT are not good for math when performing digital tasks, but they are good at extracting knowledge as pattern machine. The semantics of the patterns be perplexing to us, but it makes sense to LLM.
DISCUSSION AND CONCLUSION
This paper proposes an instance-wise, feature-wise, and text-proootype-alinedd TS embedding method achieve TS-for-LLM. It activate LLM's ability for TS tasks while maintaining its original language ability. Experiments on classification. forecasting, and representation tasks show that using TEST, LLM can archive comparable performance SOTA methods.
TS-for-LLM can enrich LLM's capabilities. SFT LLM may be more effective than TS-for-LLM, yet its superiority over customized TS models remains unclear, Training customized models may be more accurate in TS tasks, yet TS for-LLM offers all notable benefits of LLM additionally.
TL- can explore LLM's mechanism as a pattern machine. The essence of TS-for-LLM is: TS <> TS embeddings <+ patterns <>> text/word embedding <+ text. Although TEST gives the impression of a forcibly aligning operations between TS and text, it dose convert TS into an under- standable pattern sequence for LLMs, that clearly demonstrates that the essence of LLM is pattern recognition. fact, TS is objective data, whereas images, text, and speech are subjective data that can be perceived by human senses. TEST aligns objective TS data and subjective text data at the machine level, but how to align them at the human perception level requires future research.
Mcanwhile, in addition to text prototypes and prompts, LLM size and type also affect the results. related to downstream tasks, where the bidirectional
The impact of model type intuitive, structure is beneficial for classification, and the generated structure is beneficial for forecasting. The impact of model size, where a larger model produces more accurate results, can be attributed to various reasons. Aside from the impact of additional parameters, we believe that the datasets used in the pre-training process are also important, with the size, diversity, and corpus type all having an impact. We conjecture that more training data will provide the model with more opportunities to learn temporal patterns. As a result, we intend to conduct more experiments to investigate deeper correlations between corpora and TS data Chen al. (2023).
ACKNOWLEDGMENTS
This work is supported by National Natural Science Foundation of China INo.62172018 No.62102008) and Wuhan East Lake High-Tech Development Zone National Comprehensive Exper- imental Base for Governance of Intelligent Society. We extend our heartfelt gratitude to Eamonn's lab and THUMI. for generously providing the UCR datasets and TSlib datasets utilized in this study.

---

Published as a conference paper at ICLR 2024
REFERENCES
Anthony J. Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn J. Keogh. The UEA multivariate time series classification archive, 2018. CoRR, 10011011.0005. 2018.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, pp. 1476-4687, 2023, doi: 011588/441886-023-654444
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Floo Florina Balcan, and Hsuan-Tien Lin in Neural Information Processing Systems, 2020.
Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. LLM4TS: two-stage fine-tuning for time-series forecasting with pre-trained llms. CoRR, has/2308.08460 2023.
Daoyuan Chen, Yilun Huang, and et al. Data-juicer: A one-stop data processing system for large language models. CoRR, abs/2309.0203, 2023.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning. volume 119, pp. 1597-1607, 2020.
Hyunseung Chung, Jiho Kim, Joon-Myoung Kwon, Ki-Hyun Jeon, Min Sung Lee, and Edward Choi. Text-to-ecg: 12-lead electrocardiogram synthesis conditioned on clinical text reports. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5, 2023.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The time series archive. JEEE/CAA Journal of Automatica Sinica, 6: 1:12--1330,, 2019. doi: 10. 10.11000AS.2219.9111777
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of Anmual Meeting of the Association for Computational Linguistics, volume 1, pp. 320-335, 2022.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In International Joint Conference on Artificial Intelligence, pp. 2352-2359, 2021.
Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems, pp- 4652 4663, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In Computer Vision and Pattern Recognition, pp- 9726-9735, 2020.
Shenda Hong, Hongyan Li, Chenxi Sun, and Junyuan Shang. Research and applications of extracting computational phenotype from vital sign time series. China Seience and Technology Achivements, 10, 2023. doi: 10.3772/f.issn.1009. 100.::55.....0055559999 00359.223.11 10.002.
John Hopcroft and Ravindran Kannan. Computer science theory for the information age. Cambridge University press, 2013.
Zhizhong Huang. Jie Chen, Jumping Zhang, and Hongming Shan. Learning representation for clus- tering via prototype scattering and positive sampling. IEEE Trans. Pattern Anal. Mach. Intell., 45 7522,4 2023. doi: 10.1 009/TPAMM..022.3111555
10

---

Published as a conference paper at ICLR 2024
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficiet prompt tuning. In Proceedings of Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021. doi: 10.18653/w120221.mmml .etp:/02..m.mmmmmmmmmma 24in.243.
Jun Li, Che Liu, Sibo Cheng, Rossella Arcucci, and Shenda Hong. Frozen language model helps ecg shot learning. In Medical Imaging with Deep Learning, pp. 402-415, 2024
Yunfan Li, Peng Hu, Jerry Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In AAAI Conference on Artificial Intelligence, pp. 8547-8555, 2021.
Xin Liu, Daniel McDuff, Geza Kovacs, Isaac R. Galatzer-Levy, Jacob E. Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak N. Patel. Large language models few-shot health learners. CoRR, abs/2305. 15525, 2023. doi: 10.485550/rxxv.230055
Qianli Ma, Zhen Liu, Zhenjing Zheng, Ziyang Huang, Siying Zhu, Zhongzhong Yu, and James T. Kwok, A survey on time-series pre-trained models. CoRR, abs/2305.10716, 2023. doi: 10.48550/ D.Xiv.3055100166
Qianwen Meng, Hangwei Qian, Yong Liu, Lizhen Cui, Yonghui Xu, and Zhiqi Shen. MHCCL: masked hierarchical cluster-wise contrastive learning for multivariate time series. In AAAI Con- ference on Artificial Intelligence, pp. 9153-9161, 2023a. doi: (1160//aai.v3778826666..6
Qianwen Meng, Hangwei Qian, Yong Liu, Yonghui Xu, Zhiqi Shen, and Lizhen Cui. Unsupervised representation learning for time series: A review. CoRR, abs/2308.01 2023b. doi: 10.48550/ arXiv.2308.0155
Qianwen Meng, Hangwei Qian, Yong Liu, Yonghui Xu, Zhiqi Shen, and Lizhen Cui. Unsupervised representation learning for time series: review. CoRR, abs/2308.015 2023c.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Are- nas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng- Large language models as general pattern machines. In Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pp. 2498-2518, 2023.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAI, 2019.
Chenxi Sun, Shenda Hong, and et al. A review of deep learning methods for irregularly sampled medical time series data. CoRR, abs/2010. 12493, 2020. doi: 20.445550/rxx..2100..2222
Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey Webb. Time series extrinsic regression. Data Mining and Knowledge Discovery, pp. 1-29, 2021. doi: https://doi.org/10 1007/ s10618-021-00745.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In Proceedings of International Conference on Machine Learning, volume 119, pp. 9929-9939, 2020.
Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Mach. Intell. Res., 20(4):447 482, 2023. doi: 11.1007/s11133-022 1410-8.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In Interna- tional Conference on Learning Representations, 2022a.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trende representations for time series forecasting. In The Inter- national Conference Learning Representations, 2022b.

---

Published as a conference paper at ICLR 2024
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang. and Mingsheng Long. Timesnet: Temporal 2d variation modeling for general time series analysis. International Conference on Learning Representations, 2023.
Hao Xue and Flora D. Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. CoRR, abs/2210.08964, 2023.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAI Conference on Artificial Intelligence, pp. 8980-8987, 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI Conference on Artificial Intelligence, pp. 11121-11128, 2023. doi: 10. 1609/aaii. v3719.26317
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji- Rong Wen. A survey of large language models. CoRR, abs/2303.1822 2023. doi: 10.48550/ MrXiv:2303.1822.
Xiaochen Zheng. Xingyu Chen, Manuel Schürch, Amina Mollaysa, Ahmed Allam, and Michael Krauthammer. Simts: Rethinking contrastive representation learning for time series forecasting. CoRR, abs/2303.18200 2023.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Con- ference on Artificial Intelligence, pp. 11 1110011111 11115, 2021. doi: 10.1609/aaii 135.122....351
Tian Zhou, PeiSong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all:power general time series analysis by pretrained Im. Conference and Workshop on Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
RELATED WORK
Our work mainly involves two research fields: Universal Representation Learning (URL) for time series based on Contrastive Learning (CL) and Large Language Model (LLM) Time Series (TS).
1.1 CL-BASED URL FOR TS
Unsupervised URL approaches aim to learn discriminative feature representations from unlabeled data, without the requirement of annotating every sample. Enabling URL is extremely crucial for time series data, due to its unique annotation bottleneck caused by its complex characteristics and lack of visual cues compared with other data modalities.
Contrastive methods learn meaningful representations from time series by optimizing self- discrimination tasks. Instead of directly modeling the complex raw data, they employ pretext tasks that leverage the underlying similarity between samples, which eliminates the need for re- constructing the complete imput and allows for the discovery of contextualized underlying factors of variations. Contrastive methods typically generate augmented views of the raw data through vari- ous transformations and then learn representations by contrasting positive samples against negative samples. Thee existing CL-based URL for TS are listed in Table $2
Instance-level contrastive models treat individual samples independently for the purpose of instance discrimination. They utilize data augmentations to transform original inputs into new embedding space. Within this space, augmentations derived from the same sample are considered as positive pairs, while those from different samples are treated as negative pairs. During training, these models are optimized by maximizing the similarity between representations of positive pairs, while simul- tancously minimizing the similarity between representations of negative pairs.
Prototype-level contrastive models break the independence between samples and explore to exploit the implicit semantics shared by samples in the same cluster. They address the limitation that instance-level contrastive learning models tend to treat semantically similar samples as negatives.
Temporal-leve contrastive models instead focus capturing scale- invariant representations at each individual timestamp. By cosidering both instance-level and temporal-level representation learning strategies, researchers aim enhance the capability of contrastive learning methods in capturing the complexities inherent in time series data.
Category
Reconstructio--unt
Disregard insignificant data may contain noise
Collapse of embedding space;
TimeNe Wi et al
2023
Unable measure feature relations
SimMTM Dong
2023
Adversarial
Eliminate need for expensive manual labeling
Difficulty in model convergence:
TimeGAN Youn al GAN Broply
2019
Unable measure fenture relations
Predicative
Self-supervised
7erveas
TS-Tc
Contrastive
Self-supervised
Different datasets require different data augmentation methods and similarity evaliuations
Table
Table S1: Representation Learning Methods of Time Series Methods
13

---

Published as a conference paper at ICLR 2024
Type
Methods
Instance-level
SimCI.R Chen al.
2020)
TimeCI Yang al.
2023
MrCol He 2020
BYOI Grill et
2020
CPC van den Oord ot
2018
SimSiam Zhonng ot
2025
MCL Wickstren et
2022
Prototype-level
SwAV Caron et CC 21 20216
2020
PCL Liet al.
2021b
CCL Sharma et
2020
SCCL Zhang
2021
SLIC Khoraggam el al
2022
MIC Meng et
Temporal level
TS2Vec Yue et al.
2022
TS TCC Fildel et al 2
TNC Tonekahoni et al
2021
TCL
T-Loss Franceschi et
BTSF Yang Hong
2122
Woo 20224
Table S2: Contrastive Learning based Universal Representation Methods for Time Series
Mcans
Pros
Cons
Work
Training
Specialized, accurate
Not universal,
Pre-training et
2023
large datasets
Earth transtormer Bs
2025
TS Transformers Wa
Tuning
End-to-end, accurale
experiments,
GPT4TSZhou
2023
lose language ability
LLM C nang 20723
LLMTime Grave a
Time MLne-LLMJinn
PromptCast Xue Salim
2023
Parametes-cfficient, experiments
experts.
Health Learner Lau
Tool Augmented
annotation
METS
Tex12ECC Chhing ct
2023
Extemal Fincoder
Parameter efficient, multiple abilities
Weak robust
TEST
Table S3: Existing Work about TS+LLM
LLM+TS
Model tuning
Data tuning
Model Centric
Data Centric
oM
LIM Tor
TS for L.I.
Pre-training
Fine-tuning
Multimodal
Data adoptation
W
MWN
M
M
M
M
Figure S1: Technical Route of LLM+TS
1.2 LLM+TS
Large models, specifically referred as large language models (LLMs) and pre-trained foundation models (PFMs), have witnessed remarkable success across multitude of tasks and domains, such as natural language processing (NLP),, computer vision (CV). Given the remarkable achievements of large models in these diverse fields, intriguing question emerges: can large models be effectively employed to analyze TS data?
TS data has long been studied and proven to be indispensable in myriad of real-world applica- tions, encompassing fields such as geoscience, transportation, energy, healthcare, environment, and finance. While large models have made significant progress in various fields, the arena of time ries analysis has followed a more gradual path. Traditional analytical methods have predominantly relied on statistical models. The advent of deep learning has galvanized the research community
14

---

Published as a conference paper at ICLR 2024
explore potent data-driven models, typically built the basis of Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers. Nonetheless, the majority of these models remain relatively small in scale and tailored for specific tasks, thereby lacking the capacity to acquire comprehensive semantic and knowledge representations from large-scale data for multi-task reasoning.
There hasn't been much research done on TS+LLM because this field is still in its infancy. We summarize the existing work in Table S3 Different from the main text, we category work here through technical means.
2
EXPERIMENTS
2.1 MODEL
2.1.1 ENCODER
The core of TEST is to train an encoder and a soft prompt. The encoder must can extract relevant information from TS, needs to be time- and memory-efficient, and has to allow variable-length inputs. Thus, as shown in Figure S2 we build causal TCN with 10 layers of convolution blocks. Each convolution block a sequence of GELU, DilatedConv, BatchNorm, GELU, DilatedConv, with skip connections across each block. The DilatedConvs have dilation of 2i in each layer i of convolution block. A final convolution block used to map the hidden channels to the output channel whose size is the same the ILM'' embedding size.
The detailed architecture is: Number of channels in the intermediary layers of the causal network is 40; Number of layers (depth of the causal network) is 10; Kernel size of all convolutions is 3; Negative slope of the leaky ReLU activation is 0.01; Number of output channels of the causal network (before max pooling) is 640; Dimension of the representations is the embedding size (e.g. 1024 for gpt2).
the LLM's
Output
Diltion 2
Tanky
Weight
Dilstion 2
Ceocel Conolition Dilxinn
Convolatio Kernel up- dowe- samping)
Leoky ReL.t
Dilation 2
Weight Norm
Convotation Dilation
Timo
Figure $2: Illustration of Three Stacked Dilated Causal Convolutions and Composition of the i-th Layer The Chosen Architecture
We train our models with the following parameters for time series classification. Note that no hy- perparameter optimization was performed on the encoder hyperparameters: Optimizer is Adam with learning rate a = 0.001 and decay rates = (0.9, 0.999); Number negative samples is K (1, 5, 10} for for univariate time series, K {5. 10, 20) for multivariate ones: Batch size is
10; Number optimizations steps 2000for K < 10 (i.e., 20 epochs for a dataset of size 1000), 1500 otherwise.
15

---

Published as a conference paper at ICLR 2024
2.1.2 LLM
The used LLMs are as listed in Table S4 the Adam optimizer 20 NVIDIA Tesla
Each encoder and soft prompt of LLM are trained using
V100-SXM2 GPU with CUDA 11.3.
Model
Embed. dimension
Berl Devlin et al.
2018)
1O0M, 335M
748, 1024
GPT2
20
117M, 345M, 774M
768, 1024, 1280
ChaiGi M Du et al
6B
4096
La Touvron
7B, 13B
4096
Table S4: The Used Language Model
2.2 FORECASTING TASKS
All the deep learning networks are implemented in PyTTrrch and trained NVIDIA V100 32GB GPUs. We use mean square error (MSE) and mean absolute error (MAE) as metrics. For zero- shot learning, mean absolute percentage error (MAPE) is used for TOURISM; symmetric MAPE (sMAPE) used for M3 and M4; normalized deviation (ND) is used for ELECTR, All experiments repeated 3 times and the mean of the metrics is used in the final results.
2.2. DATASET DETAILS
The details of long-term forecasting and few-shot forecasting datasets are: ETT datasets Zhou et al. 2021) contain electricity load various resolutions (ETTh & ETTm) from two electricity stations; Weather Matase/Wetttesseeee
(2017) contains 21 meteorological indicators of Germany within year; Illness CCDC (2021) contains the influenza-like illness patients in the United States. IIII is not used for few-shot learning for the limited quantity that is hard to follow the definition of few-shot; Electricity dataset [SJ & B] (2017) contains the electricity consumption; Traffic dataset PeMS (202T) contains the occupation rate of freeway system across the State of California. Table S5 summarizes details of feature statistics.
Dutaset
Length
Dimension
Frequency
ETTb
17420
hour
Weather
52696
22
10 min
IIII
966
days
Electricity
26304
321
hour
Traftic
17544
862
hour
Table S5: Long-term Forecasting and Few-shot Forecasting Dataset Details
Dataset Horizon
Mapping
Longth
M3 Yearly
645
Yearly
Quarterly
756
Quarterly
M3 Monthly
1428
18
Monhll
Others
Monthly
M4 Yearly
23000
18
Yearly
Quarterly
Quarterty
M4 Monthly
8
48000
Monthly
M4 Weekly
359
Monthly
M4 Daily
4227
14
Monthly
M4 Hourly
414
Montlyy
TOURISM Yearly
518
Yearly
Yearly
TOURISM Quarteely
427
Quasterly
Quarterty
ROURISM Montbly
366
Momthly
Monthly
ELECTR
1311
Hourly
Monbly
Table S6: Zero-term Forecasting Datasets and Mapping Details of Zero-shot Learning
16

---

Published as a conference paper at ICLR 2024
The details of zero-shot forecasting datasets are: M4 is a large and diverse dataset that contains time series of various frequencies and fields, including business, financial and economic forecasting; M3 is smaller than M4, but also contains time series from diverse domains and frequencies; TOURISM is the dataset of tourism activities with different frequencies and contains a much higher fraction of erratic series compared with M4; ELECTR represents the electricity usage monitoring of 370 customers over three years. Table S6 summarizes details of the datasets and zero-shot mapping between source and target.
2.2.2 BASELINE DETAILS
For long-shot forecasting, we refer to the SOTA methods reported in Wu al. (0023; TimesNet
Wu et al. (2023), ETSformer Woo et al. (022b), DLinear Zeng et al. (202)). FEDtormer Zhou et al.
2022). Informer Zhou et al. (2021), and LLM for TS method GPTT4'S Zhou al. (2023).
For few forecasting, we refor to the SOTA methods reported in Zhou et al. (023): DLinear
Zeng et al. (023). PatchTST Nie et al. (202)) TimesNet Wu et al. (2023), FEDrormer Zhou et al.
2022)) Autoformer Wu et al. (2021). Stationary [iu et al. (2022) FTSformer Woo et al. (022b)
Informer Zhou al. (2021), Reformer Kitaev et al. (2020)
For zero-shot forecasting, we refor to the SOTA methods reported in Zhou et al. (2023): N-BEATS
Oreshkin et al. (2020), DLinear Zeng et al. (2023), PatchTST Nie et al. (2023). TimesNet Wu et al.
2023). FEDformer /Zhou et al. (2022). Autoformer Wu et al. (2021). Stationary Liu et al. 2022
ET Sfoomee Woo et al. (2022b). Informer Zhou et al. [202] lb. Reformer Kitaev et al. (2020)
Methods
TEST
GPT4TS
TimesNot
ETSformer
DLincar
FEDformer
Informer
TCN
LSTM
96
0.293 0.3-1
0.292 0,346
0.325 0.398
0,338 0.375
0,345 0,372
0.375 0.398
0.672 0.571
0.863 0.664
0.863 0.664
192
0.369
0.332 0.372
0.324
0,408 0.410
0,380
0.426 0.441
0.795 0.669
0,837 0.700
ETTml
336
0.368 0.392
0.366 0.394
0,360 0.411 I
0.435 0.428
0.413 0.413
0.445 0.459
1,212 0.871
1.124 0.832
1,267 0,832
720
0.418 0,420
0.417 0.421
0.428 0.450
0.499 0.462
0.474 0.453
0.543 0.490
1.166 0.823
1.153 0.820
1.324 0.858
Avg
0.353 0,382
0.352 0.383
0.350 0.406
0.429 0.425
0.403 0.407
0.448 0.452
0.961 0.734
0.929 0.725
1.142 0.782
96
0.372 0,400
0.376 0.397
0.384 0.402
0.494 0.479
0.386 0.400
0.376 0.419
0.865 0.713
0.878 0.740
1.044 0.773
192
0.414 0.422
0.416 0.418
0.436 0.429
0,538 0.504
0.437 0.432
0.420 0.448
1.008 0.792
1.037 0.824
1.217 0.832
ETThl
336
0.422 0.437
0.442 0.433
0.491 0.469
0.521
0,481
0.465
1.107 0.809
1,238 0.932
0.841
720
0.447 0.467
0.477 0.456
0 0.500
0.562 0.535
0.519 0.516
0,506 0.507
1.181 0.865
1.135 0.852
1.271 0,838
Avg
0.414 0.431
0.427 0.426
0.458 0.450
0.542 0.510
0.456 0.452
0.440 0.460
1.040 0.795
1.072 0.837
1.198 0.821
96
0.275 0.338
0.285 0.342
0.340 0.374
0.340 0.391
0,333 0.387
0.358 0,397
3.755 1.525
2.116 1.197
2.522 1,278
192
0.340 0,379
0.354 0.389
0.402 0.414
0.430 0.439
0.477 0.476
0.429 0.439
5.602 1,931
4.315 1.635
3.312 1.384
ETTh2
336
0.329 0.381
0.373 0.407
0.452 0.452
0.485 0.559
0.594 0.541
0.496 0.487
4.721 1.835
1.124 1.604
3.291 1.388
720
0.381 0.423
0.406 0.441
0.462 0,468
0,500 0.497
0.831 0.657
0.463 0.474
3.647 1.625
3.188 1.540
3.257 1.357
Avg
0,331 0,380
0,354 0,394
0.414 0.427
0,439 0.452
0.559 0,515
0.4370.449
4.431 1.729
2.686 1.494
1.352 3.095
96
0.132 0.223
0.139 0,238
0.168 0.222
0,187 0.304
0.197 0.282
0.193 0,308
0.274 0.368
0,25% 0.357
0.375 0.437
192
0.158 0.241
0.153 0.251
0,184 0.239
0.199 0.196
0,285 0.201
0.315 0.296
0.386 0.266
0.368 0.348
0.442 0.473
Electricity
336
0.163 0.260
0.169 0,266
0.198 0.260
0,212 0.329
0.209 0.301
0.214 0.329
0,300 0,394
0.280 0,380
0.439 0.473
720
0.199 0.291
0.206 0.297
0,220 0.300
0.235 0.345
0.245 0.335
0,246 0.355
0.373 0.439
0,283 0.376
0.980 0.814
Avg
0.162 0,253
0.167 0,263
0.192 0.245
0,208 0.323
0.212 0.300
0.214 0.327
0.311 0.397
0.313 0.401
0.559 0.549
96
0.407 0.282 0
0.388 0.282
0.593
0.607 0.392
0.650 0.396
0.587 0.366
0.719 0.391
0.684 0.384
0.843 0.453
192
0.423 0.287
0.407 0,290
0.617 0.336
0.621 0.399
0.598 0,370
0.604 0,373
0.696 0.379
0.685 0.390
0.847 0.453
Traflic
336
0.430 0.296
0.412 0.294
0.629 0.336
0.622 0.396
0.605 0.373
0.621 0,383
0.420
0.734 0.408
720
0.463 0,315
0.450 0,312
0.640 0.350
0.632 0.396
0.645 0.394
0.626 0.382
0.864 0.472
0.717 0.396
1,500 01805
Avg
0.430 0,295
0.414 0.294
0.620 0.336
0.621 0.396
0. 0.625 0.383
0.376
0.764 0.416
0.705 0.395
1.011 0.541
96
0.150 0.202
0.162 0.212
0.152 0.220
0.197 0.281
0.196 0.255
0.217 0.296
0.300 0.384
0.458 0,490
0.369 0,406
192
0.198 0.246
0.204 0.248
0.209 0.261
0.237 0.312
0.237 0.296
0.276 0.336
0.598 0,544
0.658 0.589
0.416 0.435
Weather
336
0.245 0.286
0.254 0.286
0.280 0.306
0,298 0.353
0,283 0,335
0.339 0,380
0.578 0.521
0.797 0.652
0.455 0.454
0.324 0.342
0.326 0.337
0.365 0.359
0.288
0.345 0.381
0.403 0,428
1.059 0.741
0.869 0.675
0.535 0,520
Avg
0.229 0.271
0.237 0,270
0.236 0.287
0,271 0.334
0.265 0.317
0.309 0.360
0.634 0,548
0.696 0.602
0.444 0..544
24
1.974 0,886
2.063 0,881
2.317 0.934
2.527
1,000
2.398 1.040
3,228 1,260
5.764 1.677
4,480 1.444
5.914 1.734
36
2.028
0.976
1.868 0.892
1.972 0.900
2.615 1.007
2. 2.646 1.088
2.679 1.080
4.755 1.467
4,799
1.467
6.631 1.845
ILI
48
2.353 1.115
1.790 0.884
2.238 0.900
2.359 0.972
2.614 1.086
2,622
1.078
4.763 1.469
4,800
1.468
6.736 1.857
60
2.425
1,203
1.979 0.957
2.027 0.928
2.487 1.016
2. 2.804 1.64
2.857 1.15
5.264 1.564
5.278
1.560
6.870 1.879
Avg
2.195 1.045
1.925 0.903
2.139 0.901
2.497 1.004
2.616 1.090
2.847
1.144
1.544 5.137
4,839
1.485
6.538 1.829
1d count
4
0
0
0
0
as the backbone. The past sequence length is set as 36 for ILI and 96 for the others. All the results are averaged from
Table S7: Long-term Forecasting Results (MSE, MAE). TEST uses
4 different prediction lengths, that is {24, 36, 48, 60) for ILI and (96. 192, 336, 720} for the others.
17

---

Published as a conference paper at ICLR 2024
2.2.3 LONG-TERM FORECASTING
We follow the classical experiment settings and the results of SOTA models in Wu et al. (2023) (ICLR 2023). The results are shown in Table S7 Overall, TEST achieves comparable performance to SOTA models TimesNet and Dlinear, and outperforms other baselines.
2.2.4 FEW-SHOT FORECASTING
For the few-shot forecasting task, only 10% percentage timesteps of training data are used, and the other two parts remain unchanged. We follow the classical experiment settings and the results of SOTA models in Zhou et al. (2023) (NeurIPS 2023). The results are shown in Table S8 Overall, TEST has comparable performance with the SOTA baselines PatchTST and Dlinear, and SOTA LLM for TS method GPT4TS.
Methous
TEST
GPT4TS
DLinear
PuichTST
TimesNet
FEDiommer
Auitofomerr
Stationary
ETSformer
LighTTS
Informer
Reformer
96
0.166 0.213
0.163 0.215
0.171
0.224
0.165 0.215
0.184 0.230
0.188 0.253
0.221 0.297
0.192 0.234
0.199 0.272
0.217 0.269
0.374
0.335 0,380
192
0.230 0.263
0.210 0.254
0.215
0,263
0.2100.257
0.245 0.283
0,250 0. 0.3
0.270 0.322
0.269 0.20 2.295
0.279 0.332
0.259 0,304
0.552 0.478
0.522 0.462
Weather
336
0.278 0.282
0.256 0.292
0.258
0.299
0.259..29)
0.305 0.321
0.312 0.346
0.320 0.351
0.370
0.356 0.386
0.303 0.334
0.724 0.541
0.715 0.535
720
0.301 0.328
0.321 0.339
0.320
0.346
0.332
0,381 0.371
0.387
0.393
0.390 0.396
0.441 0.405
0.437 0.448
0.377
0.382
0.739 0,558
0.611 0,500
Avg
0.243 0.272
0.238
0.275
0.241
0,283
0,242 0.279
0.279 0.301
0,284 0.324
0.300 0.342
0.318 0.323
0.318 0.360
0.289
0.322
0.597 0.495
0.546 0.469
96
0.455 0.457
0.458
0.456
0.492
0,495
0,516
0.485
0.861 0.628
0,512 0.499
0.613 0.552
0.918 0.639
1.112 0.806
1.298 0.838
1.179 0.792
1.184 0.790
192
0.572 0.519
0.570 0.516
0.565
0.538
0.598 0.524
0.797
0.593
0.624 0.555
0.722 0.598
0.915 0.629
1.155 0.823
1.322 0.854
1.199 0.806
1,295 0.850
E
336
0.611 0.531
0.608 0.535
0.721
0.622
0.657 0.550
0.941
0. 0.648
0.691
0.574
0,750 0.619
0.939 0.644
1.1 0.832
1.347 0.870
I 1,202 0.811
1.294 0.854
720
0.723
0.594
0.725 0.591
0,986
0.743
0,762 0.610
0.877
0.641
0.728
0.614
0.721
0.616
0.887 0.645
1.273 0.874
1.534
1.217 0.825
1.223 0.838
Avg
0.479
0,525
0.590 0,525
0.691
0.600
0.633 0,542
0.869
0.628
0.639
0.561
0,702
0.596
0.915 0.639
1,180 0.834
1.375
0.877
1.199 0.809
1.249 0.833
96
0.332 0.374
0.331
0.374
0.357
0,411
0.353
0.389
0.378
0.409
0.382
0.416
0.413 0.451
0.389
0.411
0.678
0.619
2.022
1.006
3.837
1,508
3,788 1,533
192
0.401 0.433
0.402 0,411
0.569
0.519
0.403 0.414
0,490
0.467
0.478
0.474
0.474
0.477
0.473
0.785 0.666
2.329
1.104
3.856
1.513
3.552. 1.483
ETT12
336
0.408 0,440
0.406
0,433
0.671
0.572
0,426
0.441
0,537
0.494
0.504
0.501
0,547
0.543
0.507 0,480
0.839 0.694
2.453
1.122
3.952
1.526
3,395
1.526
720
0.459 0.480
0.449
0.464
0.824
0.648
0.477
0.480
0.510
0.491
0,499
0. 0.509
0.516 0 0.523
0.477 0.472
1.273 0.874
3.816
1.407
3.842
1.503
3,205
1.401
Avg
0.401
0:432
0.397
0.421
0.605
0.538
0,415
0.431
0.479
0,465
0.466
0:475
0.488
0.499
0.462 0.455
0.894 0.713
2.655
1.160
3.872
1.513
3.485 1.486
0.392 0,401
0.390
0.404
0,392
0,410
0.419
0.583
0.501
0,578
0.774
0.634
0.761
0.688
0.921
1.162 017785
1.442 0.847
192
0.423
0,426
0.429 0.423
0.382
0.412
0.437
0.434
0.630
0.528
0,617 0.546
0.754 0.592
0.781 0.574
0.955 0.703
0.957 0,701
1.172 0.793
1.444 0.862
ETTml
336
0.471 0.444
0.469 0.439
0.419
0.434
0.476
0.454
0.568
0.998 0.775
0.869
0.677
0,803 n.587
0.991 0.719
0.998 0.716
1.227 0.908
1.450
720
0.552
0.501
0.569
0,498
0.490
0.477
0.681
0,556
0.769
0.549
0.693
0.579
0,810
0.630
0.844 0,581
1.062 0.747
1.007 0,719
1.207 0.797
1.366 0.850
Avg
0.574
0.443
0.464 0.441
0.411
0.429
0.501 0.466
0.677
0.537
0,722 0.605
0.802
0.628
0.797
0.578
0.980 0.714
0.971 0.705
1.192 0.821
1. 1.426 0.856
96
0.233 0.262
0.188
0.269
0.213
0,303
0.191
0.274
0.212 0.285
0,291 0.399
0.352 0.454
0.229
0.308
0.331 0.430
0.813
0.688
3.203
1.407
4.195 1.628
192
0.303 0.302
0.251
0.309
0.278
0.345
0.252 0.317
0.270
0.323
0.307 0.379
0.694 0.691
0.291 0.343
0.400 0.464
1.008 0. 7,768
3.112 1,387
4.042. 1.601
ETTm2
336
0.359 0.341
0,307 0.346
0.338
0,385
0.306
0.353
0.323
0. 0.353
0,543 0.559
2.408 1.407
0.348 0.376
0.469
0.498
1.031 0,775
3.255
1.421
3,963
1,585
720
0.452 0.419
0.426
0.417
0.436
0.440
0.433
0.427
0.474 0.449
0.712
1,913 1.666
0.461 0.438
0.557
0.791 1.096
3.909
1.543
3,711 1.532
Avg
0.317
0.309
0.293
0.335
0.316
0.368
0.296
0.343
0.320
0.353
0,463 0.488
1.342 0.930
0.332 0,366
0.447 0.487
0.987
0.756
1.440 3.370
3.978
1,587
96
0.143
0.235
0.139
0.237
0,155
0.253
0.140
0,238
0.299
0.373
0.231
0.323
0,261
0.348
0.440
0,466
0.599
0.587
0.350
0.425
1,259
0.919
0.993
0.784
192
0.158 0.255
0.156 0.252
0.164 0.264
0.160 0.255
0.305 0.379
0,261 0.356
0.338
0.406
0.441 0.459
0.620 0.591
0.376 0.448
1.160 0.873
0.938 0.753
Electricity
336
0.6 0.275
0.1 0.270
081
0.282
0.180 0.276
0.319 0.391
0.360 0.445
0.410 0.474
0.434 0.473
0.662
0.619
0.428 0.485
1.157 0.872
0.925 0.745
720
0.230
0.233 0.317
0.223
0
0,241
0.323
0.369
0.426
0,530 0.585
0.715
0 0.685
0,510 0.521
0.757
0.664
0.61
0.597
1.203 0.898
1.004 0.790
Avg
0.176 0.269
0.176 0.269
0.180
0.280
0.180
0.273
0.323 0.392
0.346 0:427
0.431 0.478
0.444 0.480
0.660 0.617
0.441
0.489
1.195 0.891
0.965 0.768
0.415
0.414 0.297
0.298
0.403
0.289
0.719
0.416
0.639
0.405
1.412
8.802
1.643 0.855
1.157 0.636
1.557 0.821
1.527 0.815
192
0.425
0.300
0.426 0.301
0.434
0.305
0,296 0.415
0.748
0.428
0.637 0.416
0.727
0.424
1.419 0.806
1.641 0.854
1,207 0.661
1.454 0.765
1.538 0.817
Traffic
336
0.436
0.310
0.434 0.303
0.449
0,313
0.426 0.
0.853
0.471
0.655
0.427
0.749 0.454
1.443 0.815
1.711 0.878
1.334 0.713
1.521 0.812
1,550 0.819
720
0.489
0.338
0.487
0,337
0.484
0.336
0.474
0,331
1.485
0.825
0.722
0.456
0.847
0.499
1.539 0.837
2.660 1.157
1.292
0.726
1.605 0.846
1.588 0.833
Avg
0.441 0.316
0.440 0.310
0.447
0.313
0.430 0.305
0.951 0.535
0.663 0.425
0,749 0.446
1.453 0.815
1.914 0.936
1.248 0.684
1 1.534 0.81
1,551 0.821
1st
5
0
0
0
Table S8: Few-shot Forecasting Results (MSE, MAE) TEST uses GPT2-Medium as the backbone. All the results are averaged from 4 different prediction lengths, that is (96, 192, 336, 720].
2.2.5 ZERO-SHOT FORECASTING
Zero-shot Forecasting task can evaluate the cross datasets adaption ability. Which means that the method is evaluated to perform on dataset (without any training data from this dataset) when it is trained from another dataset. The results are summarized in Table S9 TEST outperforms all recent SOTA methods. TEST is comparable to N-BEATS without any meta-learning design and GPT4TS.
18

---

Published as a conference paper at ICLR 2024
Methods
M4
M3 sMAPE
TOURISM MAPE
ELECTR NDx100
Metric
SMAPE
Average
1 count
N-BEATS
11.70
12.44
18.82
17.8
15.19
2
DLinear
15.33
14.03
28.51
17.6
18.86
0
TimesNet
13:55
14.17
28.84
19.3
18.96
PatchTST
13.22
13.06
27.10
17.3
17.67
0
LighTTS
13.62
17.90
66.99
19.6
29.52
0
Stationary
13.32
15.29
43.75
22.0
23.59
FEDformer
15.04
13.53
31.55
18.4
19.63
Autnformer
20.02
15.87
40.39
33.9
27.54
Informer
35.82
21.2
22.97
Reformer
14.09
13.37
25.48
21.6
18.63
GPT2(6)
TEST
13.10
12.56
18.17
17.9
15,93
Table S9: Zero-shot learning results. Dataset-specific metrics aggregated over each dataset. A lower value indicates better performance. The source dataset M3, Tourism, Electricity are M4. For M4, the source data for N-BEATS is FRED, and M3 for other models.
2.3 CLASSIFICATION TASKS
All the deep learning networks are implemented in PyTorch and trained on NVIDIA V100 32GB GPUs. We use Area Under Curve of Receiver Operating Characteristic (AUC-ROC) metrics. Meanwhile, we compute the average rank, the number of Top-1, Top-3, and Top-5 accuracy to show the robustness of different methods. All experiments are repeated 3 times and the mean of the metrics is used in the final results.
2.3.1 DATASET DETAILS
We present accuracy scores for all 30 kinds of multivariate TS datasets in UEA archive Bagnal et al. 2018). UEA consists 30 different datasets, Details of these datasets are shown Table S10
Dataset
Train Cases
Test Cases
Dimensions
Length
Classes
Articulary Wordooottioooonniioonnnn
Atrilfiibillation
15
640
BasicMotions
40
100
Character Trajetoriiss
1422
1436
182
20
Cricket
108
72
17984
5
DuckDuckGoese
Eigen Worms
128
131
17984
Epilepsy
137
oonentation
261
263
1751
ERing
30
FaceDetection
5890
3524
FingerMovementss
316
100
28
2
HandMooemmeuuDirectin
Handwriting
150
850
152
26
Heartbeat
204
495
NawueeseVowels
270
370
Libras
180
280
15
LSST
2459
2466
14
InscetWingbett
30000
20000
200
78
10
Motoslmagery
NATOPS
180
180
24
S1
PenDigits
7494
3498
10
PEMS SF
267
173
963
144
7
Phoneme
3315
3353
217
39
RacketSports
152
SelfRegulationCCPI
268
293
896
SellRegulationSCPP
SpokenArahicDigiee
6599
2199
93
10
StandWalkJump
UWaeeGsstureLibrrr
120
320
315
Table S10: UEA Classification Dataset Details
19

---

Published as a conference paper at ICLR 2024
2.3.2 BASELINE DETAILS
For classification, we refer to the SOTA methods: Three benchmarks Bostrom et al. (0018) (EDI, DTWI, and DTWD) are based on Euclidean Distance, diinension-indeeendee dynamic time warp- ing, and dimension-dependeni dynamic time warping; MLSTM-FCNS Karim et al. (2019) applies an LSTM layer and stacked CNN layers to generate features; WEASEL-MUSEESchhie
& Leser (2017) is a bag-of-pattern based approach which extracts and represents features to words. Scalable Rep- resentation Learning (SRL) Franceschi et al. (2019)) employs negative sampling techniques with an encoder-basee architecture to learn the representation; TapNet ZZhng et al. (2020) is recent model with an attentional prototype learning in its deep learning- carning-based network; Shape [ii et al.) 2021a) projects the subsequences into unified space and applies clustering to find the shapelets; Rocket and MiniRocket Dempster et al. (202T) use random convolutional kernels to extract features from univariate time series; RL-PAM Gao et al. (2022) introduces reinforcement learning to the
pattern mining; TStamp Transformer Zerveas et al. (2021 takes the values at each timestamp as the
input for transformer encoder; SVP-T al. 2023 uses differnt variables and positions (time
interval) as the inputs (shape-level).
2.3.3 MULTIVARIATE TIME SERIES CLASSIFICATION
We follow the classical experiment settings in multivariate time series classification tasks Bostrom ctt al. (2018). The results are shown in Table Overall, TEST achieves comparable performance
to SOTA models and outperforms most baselines.
EDI
DTWI
DTWD
MLSTM-FCNS
MESSEL+MUSE
SRL
TapNet
ShapeNet
Rocket
MiniRocket
RLPAM
TStamp
SVP-T
TEST
AWR
0.970
0.980
0.987
0.973
0,990
0.987
0.987
0.987
0.996
0.992
0.923
0.983
0.993
0,994
AF
0,267
0.267
0,220
0.267
0.333
0.133
0.333
0.400
0.249
0.133
0.733
0.200
0.400
0.420
BM
0.676
1.000
0.975
0,950
1.000
1.000
1,000
1,000
0.990
1.000
1.000
0.975
1,000
1.000
CT
0.964
0.969
0.989
0.985
0.990
0.994
0.997
0,980
N/A
0.993
0.978
N/A
0.990
0.989
CK
0.944
0.966
1,000
0.917
1,000
0.986
0.958
0.986
1,000
0.986
1,000
0.958
1,000
L.000
DDG
0.275
0,550
0.600
0.675
0,575
0.675
0.575
0.725
U.461
0.650
0,700
0.480
0,700
0.675
0.549
N/A
0.618
0.504
0.890
0.878
0.489
0.863
0,962
0,908
NIA
0,923
0.878
EP
0.666
0.978
0.964
0.761
1.000
0.957
0,971
0,987
0.991
1.000
0.978
0.920
0.986
0,985
ER
0.133
0.914
0.929
0.133
0.133
0.133
0.133
0.133
0.981
0.981
0.819
0,933
0.937
0.937
EC
0.293
0.304
0.323
0.373
0.430
0.236
0.323
0.312
0.447
0.468
0.369
0.337
0.331
0.373
FD
0,519
0.000
0.529
0,545
0.545
0.528
0,556
0.602
0.694
0.620
0.621
0.681
0,512
0.512
FM
0.550
0,520
0.530
0.580
0.540
0.530
0.580
0.550
0.640
0.776
HMD
0.278
0.306
0.231
0.365
0.365
0.270
0.378
0.338
0.446
0.392
0.635
0.608
0.392
0.444
0,200
0.316
0.286
0.286
0.533
0.357
0.452
0.567
0.507
0.522
0.305
0.433
0.431
HB
0.619
0.658
0.717
0.663
0.727
0.737
0.751
0.756
0.718
0.771
0.779
0.712
0,790
0.791
TW
0.128
N/A
N/A
0.167
N/A
0.160
0.208
0.250
N/A
0.595
0.352
0.684
0.184
0.572
TV
0.924
0.959
0.949
0.976
0.973
0.969
0.965
0.984
0.965
0.989
0.935
0.994
0.978
0.991
LB
0.833
0.894
0.870
0.856
0.878
0.867
0,850
0.856
0.906
0.922
0.794
0.844
0.883
0.884
LSST
0.456
0.575
0.551
0.373
0.558
0.568
0.590
0.643
0.643
0,381
0.666
0.595
MI
0,510
N/A
0.500
0,510
0,500
0.540
0.590
0.610
0.531
0.550
0.610
N/A
0.650
0.650
NT
0,850
0.850
0,883
0.889
0.944
0.939
0.883
0.885
0.928
0.950
0,900
0.906
0.902
PD
0.705
0.939
0.977
0.978
0,948
0.983
0.980
0.977
0.996
N/A
0.982
0.974
0.983
0.979
PM
0.973
0.734
0.711
0.699
0.000
0.688
0.751
0.751
0.856
0.522
0.632
0.919
0.867
0.860
PH
0.104
0.151
0.151
0.110
0.190
0.246
0.175
0.298
0.284
0.292
0.175
0.088
0.176
0.196
RS
0,868
0.842
0.803
0.803
0.934
0.862
0.868
0.882
0.928
0.868
0.868
0.829
0.842
0.851
SCP1
0.771
0.765
0.775
0.846
0.782
0.925
0.925
SCP2
0.483
0.533
0.539
0.472
0.460
0.556
0.550
0.578
0.514
0.522
0.632
0,589
0.600
0.579
SAD
0.967
0.959
0.963
0.990
0.982
0.956
0.983
0.975
0.630
0.620
0.621
0.993
0.986
0.982
SWJ
0,200
0.333
0.200
0.067
0,333
0.400
0.400
0.533
0.456
0.333
0.667
0.267
0.467
0.468
UGL
0.881
0.868
0.903
0.891
0.916
0. 888
0.894
0.906
0.944
0.938
0.944
0.903
0.941
0.933
AvgkRank
Rank
10.933
9.450
8,821
8.756
6.890
7.120
6.956
5,523
5.423
5.013
5.059
7.484
4.032
4.012
Top-1
1
1
1
1
2
3
5
6
4
3
1
2
1
6
3
7
12
14
16
17
Nop-Top-5
2
2
3
12
13
17
16
20
19
10
23
24
P-value
0.000
0.000
0.000
0.000
0.006
0.003
0.000
0.118
0.217
0.765
0.967
0.047
0.044
0.040
Table S11: Accuracies on All Datasets of the UEA Archive
2.4 REPRESENTATION TASKS
We assess the quality of our learned representations on supervised tasks in a standard manner by using them for time series classification Franceschi al. (2019)). All the deep learning networks
20

---

Published as a conference paper at ICLR 2024
implemented in PyTorch and trained on NVIDIA V100 32GB GPUs. We use Area Under Curve of Receiver Operating Characteristic (AUC-ROC) as metrics.
2.4.1 DATASET DETAILS
We represent the results for all 128 kinds of univariate TS datasets in UCR archive Dau et (019). which is a standard set of varied univariate datasets.
2.4.2 BASELINE DETAILS
The compared method includes SOTAs of unsupervised time series representation: T-Loss Franceschi et (2019b). Tldee et al. 2021b), TST et et. (2021) and TNC[Tone]
aboni et al. 2021). Vee Yue et al. (2022).
2.4.3 CLASSIFICATION BASED ON REPRESENTATION
We assess the quality of our learned representations on supervised tasks in a standard manner by using them for time series classification Franceschi al. (2019b). In this setting, we show that our method outperforms SOTA unsupervised methods, and notably achieves performance close to the supervised SOTA method as shown in Table
For each considered dataset with train / test split, we unsupervisedly train an encoder using its train set. We then train an SVM with radial basis function kernel on top of the learned features using the train labels of the dataset, and output the corresponding classification score on the test set.
FEST
TCN
TS2Vec
T.l.oss
TNC
Adiac
0.776
0.768
0.765
0.675
0.726
ArrowHead
0.825
0.857
0.817
0.766
0.703
Beef
0.766
0.768
0.633
0.667
0.733
BeetleFly
0.853
0.900
0.900
0.800
0.850
BirdChicken
0.808
0.803
0.750
Cur
0.883
0.834
0.700
0.833
0.683
1,000
1.000
ChlorineConeennratiio
0.810
0.832
0.812
0.749
0.760
CinCECGToSs
0.815
0.829
0.825
0.713
0.669
Collee
1,000
1,000
1,000
1.000
1,000
Computers
0.632
0.660
0.660
0.664
0.684
CricketX
0.623
Cricket
0.754
0.749
0.769
0.728
0.597
0.787
0.794
0.790
DiatomSizz:eduutioo
0,980
0.985
0.987
0.984
0.993
0.776
0.761
0.757
0.775
0.754
DoaalPaiianOOuiineegeeaa Ogtunne
0.714
0.727
0.719
0.727
0.741
DistalPhalanxTT
0.662
0.698
0.683
0.676
0.669
Earliquakes
ECG200
0.893
0.920
0.880
0.940
0.830
ECG5000
0.935
0.935
0.934
0.933
0.937
ECGFiveDays
1,000
1,000
1,000
1.000
0.999
ElectricDevices
0.714
0.721
0.719
0.707
0.700
FaceAll
0.789
0.771
0.805
0.786
0.766
FaceFour
0.834
0.932
0.932
0.920
0.659
FacesUCR
0.939
0.926
FiftyWords
0.781
0.771
0.774
0.732
0.653
0.937
0.926
0.937
0.891
0.817
FordA
0,940
0.936
0.948
0.928
0.902
FurdB
0.789
0.794
0.807
0.793
0.733
GunPoint
0.983
0.980
0.987
0.980
0.967
Ham
0.714
0.714
0.724
0.724
0.752
HandOutines
0
0.925
0.930
0.930
Haptics
0,510
0.526
0.536
0.490
0.474
Herring
0.625
0.644
0.609
0.594
0.594
InlineSkate
0.389
0.418
0.407
0.371
0.378
Invect FinehWiingeeSSoomd
0.620
0.630
0.624
0.597
0.549
Italy PewerenndDem
0.969
0.925
0.960
0.954
0.928
AangeKicchnnAApllasss00
0.855
0.845
0.875
0.789
0.776
Lightning2
0.846
0.869
Lightning?
0.866
0. 0.863
0.822
0.795
0.767
Mallat
0.915
0.944
0.873
0.951
0.871
Meal
0.950
0.952
0.967
0.950
0.917
Medicallmages
0.792
0.789
0.793
0.750
0.754
0.811
0.838
0.825
0.825
0.818
21

---

Published as a conference paper at ICLR 2024
MiddllePhalmxxTK TW
0.591
0.584
0.636
0.578
0.656
1550
1550
0,825
0,643
MoteStrain
0.923
0.857
0.930
0.861
0.863
6166
0.00
0.898
0.940
0.938
0.935
0.919
09122
OliveOil
0.903
0.901
0.940
0.867
0.833
OSULTeaf caf
0.872
0.851
0.843
0.760
0.723
PPalangesQulinnssCooeee
0.787
Phoneme
0.296
0.312
0.309
0.276
0.180
Plane
1,000
1,000
0.900
0.990
1,000
ProximalPhalans Outineeooeeetieeeeereo
0.876
0.887
0.900
0.859
0.866
ProcimalPhalamx ProxinalPPaaaannnullnneeeee Cgnnnee
0.844
0.837
0.829
0.844
0.854
ProsimalPhallmxTTT TW
0.785
0.824
0.805
0.771
0.810
RefrigerationDeviies
0.587
0.586
0.589
0.515
0.565
ScreenType
0.397
ShapeletSim
0.989
1.000
0.994
0.672
0.589
ShapesAll
0.897
0,902
0.905
0.848
0.788
SmallKKichhesApliiaees
0.723
0.731
0.733
0.677
0.725
Sony AonyAABBRRoaaSSrrface
0.874
0.903
0.900
0.902
0.804
GonyATBORohaSSurface
0.893
0.871
0.889
0.889
0.834
StatLightCurves
0.970
0.968
0.971
0.964
0.968
Straberry
0,962
0.966
0.965
SwedishLeaaf
0.939
0.945
0.942
0.914
0.880
Symbols
0.973
0.977
0.972
0.963
0.885
SyntheticContrel
0.997
0.997
0.993
0.988
1,000
ToeSegmentation
0,933
0.917
0.947
0.939
0.864
TreSegmentation2
0,915
0.899
0.900
0.900
0.831
Trace
1,000
1,000
1,000
0.990
1.000
TwoLesDECG
0,982
0.986
TwoPatterns
1,000
1,000
1,000
0.999
1,000
LaveeGetuueLirrrry
0.810
0.795
0.801
0.785
0.781
U Bee-GeeturLLibbrrggr
0.729
0.719
0.720
0.710
0.697
L Weveereturel abbary/
0.761
0.774
0.768
0.757
0.721
U (WaveGesturLLibbury
0.935
0.930
0.934
0.89
0.908
Water
0.995
0.998
0.998
0.992
0.994
Wine
0.788
0.880
WordSynonyms
0.699
0.679
0.704
0.691
0.630
Worms
0.704
0.701
0.701
0.727
0.623
WolnsTwoClas
0.805
0.806
0.753
0.792
0.727
Yoga
0.883
0.883
0.877
0.837
0.812
ACSFI
0.849
0.910
0.730
AllGeesturWWimmeexx
0.744
0.777
0.751
0.763
0.703
AiGGestrrWimmsee
0.754
0.796
AllGestureWimote
0.744
0.749
0.770
0.723
0.646
BME
0.979
0.992
0. 9.900
0.993
0.973
Chinatown
0.969
0.964
0.959
0.951
0.977
Crop
0.753
0.754
0.758
0.722
0.738
20GHorizontaliignal
0.544
0.569
0.522
EOG SSgtiallall
0.467
0.503
0.472
0.434
0.392
Ethanoll
0.480
0.468
0.424
FreezeReguulaTTuii
0.983
0.996
0.983
0.956
0.991
FreezerSmalllTain
0.893
0.875
0.872
0.933
0.982
Pungi
0.967
0.958
0.946
1.000
0.527
GesturcMidAirDt
0.637
0.608
0.615
0.608
0.431
GesureMiddArr22
0.479
GestunMiddAieD!
0.346
0.492
0.300
0.285
0.292
GesturePehble771
0.878
0.930
0.919
0.378
GesturePebble222
0.842
0.873
0.848
0.899
0.316
GunPoint AgeSSnn
0.994
0.987
0. 0.968
0.994
0.984
1,000
1,000
1.000
0.997
0.994
GuaPointOlddarre Yorsus
1.000
1.000
1.000
1.000
1.000
HouseTwenty
0.782
InvecdEPGRegolarTTin
1,000
1.000
1.000
1.000
1.000
InsecEPGSmallTrrin
1,000
1.000
1,000
1.000
1.000
0.954
0.959
0.956
0.944
0.942
MixedShapes
0.915
0.917
0.922
0.905
0.911
SaaddhaapsSmallTTaii
0.884
0.861
0.856
0.813
PsckupGestture Wüimote'2
0.800
0.823
0.760
0.740
0.620
AigAir Pressure
PigArtPressure
0.962
0.966
0.966
0.928
0.808
PigCVP
0.803
0.815
0.870
0.788
0.649
PLAID
0.551
0.561
0.549
0.555
0.495
PowerCons
0.967
0.961
0.972
0.900
0.933
Rock
0.660
0.700
0.580
SemngHaandGemmdeCh222
0.952
0.963
0.962
0.890
0.882
0.897
0.891
0.944
0.952
0.942
0.920
0.820
SmoothSubspace
0.967
0.980
0.993
0.960
0.913
UMD
1,000
1.000
0.993
0.993
0.993
22

---

Published as a conference paper at ICLR 2024
Avg
0.832
Table S12: Accuracies on All Datasets of the UCR Archive
2.5 ABLATION
TEST contains two contrastive learning strategies: instance-wise contrast and feature-wise contrast, and can different text embedding vectors as prototypes, show the impact of these strategies.
2.5.1 1
CONTRASTIVE LEARNING STRATEGIES
As shown in Table ST3 and S14 both two contrastive learning strategies can increase the accuracy.
ETTml
ETTm2
ETTh1
ETTh2
Electricity
Traffic
Weather
IL]
Instance
0.621 0,550
0.755 0.630
0.493 0.453
0.612
0.293 0.396
0,788 0.620
0.463 0.349
3,301 4.535
Feature-wise wise
0.741 0.559
0.793 0.634
0.699 0.493
0.585 0.628
0.286 0.390
0.821 0.629
0.453 0.388
3.139 5.931
0.382
0.293 0.334
0.414 0.431
0.331 0.380
0.162 0.253
0.295
0.229 0.271
2.195 1.045
Table S13: Long-term Forecasting Results (MSE, MAE). TEST uses different contrastive learning stragegy. All the results are averaged from 4 different prediction lengths, that is (24, 36, 48, 60) for II.I and (96, 192, 336, 720} for the others. The results are average.
Instance-wise beature-wise TimesNat N-BEATS
DLinear FEDiommer Stationary Autoformer
Reformer
SMAPE
11.927
13.525
16.987
11.829
11.851
14.718
13,639
12,840
12.780
12.909
14.086
18:200
MASE
1.613
2,111
3.265
1,585
1,599
2.408
2.095
1.701
1,756
1.771
3.010
4.223
OWA
0.861
1,051
1,480
0.851
0.835
1.122
1.051
0.918
0.930
0.939
1.230
1.775
Table S14: Short-term Forecasting Task on M4. The prediction lengths are in [6, 48) and results are averaged from several datasets.
2.5.2 TEXT PROTOTYPES
The number and the type of text prototypes will lead to different results.
As shown in Table S15 We randomly select 1, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22 prototypes. The accuracy and number are basically positively correlated. The results of 10 prototypes are almost optimal.
As shown in Table $16 consistent. Therefore, the type of prototypes has almost no impact on the results.
We randomly select 10 prototypes 10 times. The accuracy is basically
SMAPE
30.901
20.20]
17.415
16.97
13,820
11.927
11.710
11.638
11.094
11.098
10.953
10.885
MASE
6.590
4.515
3.910
3.595
2.580
1.613
1.408
1.155
1.301
1.306
1.471
1.310
OWA
3,779
2.050
1.451
1.484
0.990
0.861
0.872
0.801
0.910
0,902
0.838
0.830
Table S15: Short-term Forecasting Task on M4. The results are reported with different number of text prototypes.
6
7
8
9
Avg.
SMAPE
11.907
11.920
11.927
11.926
11.925
11.925
11.950
11,890
11.728
11.910
11.90]
0.059
MASE
1.612
1.610
1.653
1.603
1.619
1.620
1.625
1.623
1.613
0.016 0.870
1.591
1.617
OWA
0.70
0.872
0.872
0.872
0.872
0.872
0.849
0.862
0.876
0.868.
0.009
Table $16: Short-term Forecasting Task on M4. The results reported with different types of text prototypes.
Considering why the type of text prototype does not significantly affect results, figure that in high dimensional space, almost all vectors are pairwise orthogonal Hopcroft & Kannan (2013). Which means that, in high-dimensional space, it is easy generate large number of almost orthogonal
23

---

Published as a conference paper at ICLR 2024
vectors to represent different attributes. Thus, randomly selecting the same number of vectors, the represented space size and expressed number of features are almost the same. Therefore, the key is the number rather than the type.
In terms of probability, "two vectors orthogonal" is equivalent to "two vectors perpendicular" is equivalent to "two vectors uncorrelated" is equivalent "cos 8 0". For a n-dimensional space, randomly two vectors have: Vc, lim, P( cos & > c) = O. As shown in Figure [S3] as the dimension increases, the probability of two random vectors being similar decreases. For LLM,
n > 1024, P(O = 0) < 0.00001.
041
0.7 68 0.8
Figure S3: Probability Density of the Angle between Two Random Vectors in n-dimensional Space
REFERENCES
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn J. Keogh. The UEA multivariate time series classification archive, 2018. CoRR, abs/1811 111.00100.... 2018.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, pp. 1476-4687, 2023. doi: 10. 10.10338//415886-23---66616
Aaron Bostrom, Anthony Bagnall, Eamonn Keogh, Hoang Anh Dau, James Large, Jason Lines, Michael Flynn, and Paul Southam. The uea multivariate time series classification archive, 2018, 2018.
Eoin Brophy, Zhengwei Wang, Qi She, and Tomás Ward. Generative adversarial networks in time series: A systematic literature review. ACM Comput. Surv., 55(10):199:11199331 2023.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. 2020.
CDC. Illness. 2021. doi:
Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. LLM4TS: two-stage fine-tuning for time-series forecasting with pre-trained llms. CoRR, 0a5.2338.04666 2023.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. Proceedings of International Conference on Machine Learning, volume 119, pp. 1597-1607, 2020.
Hyunseung Chung, Jiho Kim, Joon-Myoung Kwon, Ki-Hyun Jeon, Min Sung Lee, and Edward Choi. Text-to-ecg: 12-lead electrocardiogram synthesis conditioned on clinical text reports. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5, 2023.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The time series archive. JEEF/CAA Journal of Automatica Sinica, 6: 6:129331305, 2019. doi: 10. (09/AAS.209..110
Angus Dempster, Daniel F. Schmidt, and Geoffrey I. Webb. Minirocket: A very fast (almost) de- terministic transform for time series classification. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 248-257, 2021. doi: 10. 054114///44554844722311
24

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-trat pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang. Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. CoRR, abs/2302.00861,, 2023.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang. and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Pruceedings of Annual Meeting of the Association for Computational Linguistics, volume 1, pp. 320-335, 2022.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 2352- 2359, 2021a.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh. Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, In International Joint Conference on Artificial Intelligene, pp. 2352-2359, 2021b.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems, pp. 4652 4663, 2019a.
Jean- Yves Franceschi, Aymeric Diculeveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems, pp. 4662-4663, 2019b.
Ge Gao, Qitong Gao, Xi Yang, Miroslav Pajic, and Min Chi. A reinforcement Saaminng-infrmmer pat- tern mining framework for multivariate time series classification. In Proceedings of International Joint Conference Artificial Intelligence, pp.
2022. doi: 2224446//CAAA.22222222
Bann-Batieen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to sule-supe learning. In Advances in Neural Information Processing Systems, 2020.
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero- shot time series forecasters. CoRR,
2023. doi: 10.48550/ARXIV..310.0777
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In Computer Vision and Pattern Recognition, pp- 9726-9735, 2020.
John Hoperoft and Ravindran Kannan. Computer science theory for the information age. Cambridge University press, 2013.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-Ilm: Time series forecasting by reprogramming large language models. CoRR, abs/2310.01722S 2023. doi: 10.48550/ARXIV. 2310.01728.
Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Samuel Harford. Multivariate Istm-fens for time series classification. Neural Networks, 16::237-245, 2019. doi: NEUNET. 2019.04.014.
Salar Hosseini Khorasgani, Yuxuan Chen, and Florian Shkurti. SLIC: self-supervised learning with iterative clustering for human action videos. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16070-16080, 2022. doi: 10.1 199CCPP5228.2222001155
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In
International Conference Learning Representations, 2020.
25

---

Published as a conference paper at ICLR 2024
Guozhong Li, Byron Choi, Jianliang Xu, Sourav S. Bhowmick, Kwok-Pan Chun, and Grace Lai- Hung Wong. Shapenet: A shapelet-neural network approach for multivariate time series classification. In AAA! Conference Artificial Intelligence, pp. 3375-8383, 2021a. doi:
00.1609/AAALV3511.1100
Jun Li, Che Liu, Sibo Cheng, Rossella Arcucci, and Shenda Hong. Frozen language model helps ecg seoo learning. Medical Imaging with Deep Learning, pp. 402-415, 2024.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. Prototypical contrastive learning of
unsupervised representations. In International Conference on Learning Representations, 2021b.
Yunfan Li, Peng Hu, Jerry Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In AAAI Conference on Artificial Intelligence, pp. 8547-8555, 2021c.
Xin Liu, Daniel McDuff, Geza Kovacs, Isaac R. Galatzer-Levy, Jacob E. Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak N. Patel. Large language models are few-shot health learners. CoRR, abs/2305. 15525, 2023. doi: 10. 10.48500arX...2055.5555 15552
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long, Non-stationary transformers: Exploring the stationarity in time series forecasting. In Advances in Neural Information Processing Systems, 2022.
Qianli Ma. Zhen Liu, Zhenjing Zheng, Ziyang Huang, Siying Zhu. Zhongzhong Yu, and James T. Kwok. A survey on time-series pre-trained models. CoRR, abs/2305.10716, 2023, doi: 10.48550/ arXiv.2305.1 10716.
Qianwen Meng, Hangwei Qian, Yong Liu, Yonghui Xu, Zhiqi Shen, and Lizhen Cui. MHCCL: masked hierarchical cluster-wise contrastive learning for multivariate time series. CoRR, abs/2212.011411, 2022.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: neural basis expansion analysis for interpretable time series forecasting. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
PeMS. Traffic. 2021. doi: http://pems.dot.ca.gg
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI, 2019.
Patrick Schäfer and Ulf Leser. Multivariate time series classification with WEASEL+MUSE. CoRR, abs/1711.1133 1134, 2017.
Vivek Sharma, Makarand Tapaswi, M. Saquib Sarfraz, and Rainer Stiefelhagen. Clustering based contrastive learning for improving face representations. In IEEE International Conference Au- tomatic Face and Gesture Recognition, pp. 109-116, 2020. doi: 1011009F6478800220000000
Taylor SJ and Letham B. Forecasting at scale. In PeerJ Preprints, pp. 5:e3190v2, 2017. doi: http://ppeerrrpprrntt.....
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference Learning Repre- sentations, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. 2023.
Aäron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding, CoRR, abs/1807 007.03748, 2018.
Wetterstation. Weather. 2017. doi: https//www.bbgg-eem....d//eeeeee
26

---

Published as a conference paper at ICLR 2024
Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-superrised representation learning for time series. Pattern Recognit. Lett., 155:54-61, 61, 2022.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting, In Interna- lional Conference on Learning Representations, 2022a.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Etsformer: Expo- nential smoothing transformers for time-series forecasting. CoRR, abs/2202.01381, 2022b.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans-
formers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, pp. 22419-22430, 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
Hao Xue and Flora D. Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. CoRR, abs/2210.0820 2023.
Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilin- ear temporal-spectral fusion. In International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25038-25054, 2022.
Xinyu Yang, Zhenguo Zhang, and Rongyi Cui. Timeclr: A self-supervised contrastive learning framework for univariate time series representation. Knowl. Based Syst., 245:108606, 2022.
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial net- works. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5509-5519, 2019.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAI Conference on Artificial Intelligence, pp. 8980-8988 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI Conference on Artificial Intelligence, pp. 11 11121-11128, 2023. doi: 10. 1609/aaai 137792263177
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2114-2124, 2021. doi: 10. 10.114573447548.44444000
Dejiao Zhang, Feng Nan, Xiaokai Wei, Shang-Wer Wen Li, Henghui Zhu, Kathleen R. McKeown, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. Supporting clustering with contrastive learning. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5419-5430, 2021. doi:
Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang- - Lu. Tapnet: Multivariate time series classification with attentional prototypical network. AAAI Conference on Artificial Intelligence, pp. 6845-6852, 2020. doi: 10. 1..00//AAAL..34444..1.1..
Xiaochen Zheng, Xingyu Chen, Manuel Schürch, Amina Mollaysa, Ahmed Allam, and Michael Krauthammer. Simts: Rethinking contrastive representation learning for time series forecasting. CoRR, abs/2303.1 18205, 2023.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li. Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Con- ference Artificial Intelligence, pp. 11 1006-11115. 2021. doi: 10. 1331
27

---

Published as a conference paper at ICLR 2024
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 27268- 27286, 2022.
Tian Zhou, PeiSong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all:power general time series analysis by pretrained Im. Conference and Workshop on Neural Information Processing Systems, 2023.
Rundong Zuo. Guozhong Li, Byron Choi, Sourav S. Bhowmick, Daphne Ngar-yin Mah, and Grace Lai-Hung Wong. SVP-T: shape-level variable-position transformer for multivariate time series classification. AAA/ Conference Artificial Intelligence, pp. 11497-11505, 2023. doi: 0AA009/AAALLLV7799.22323333
28


Reviews:
[{"review_id": "ozPSbYrsfW", "reviewer": "Reviewer_3p6a", "summary": "The paper introduces TEST, a TS-for-LLM method that leverages LLMs in time series (TS) data analysis without the need for fine-tuning. TEST focuses on transforming TS data into LLM-understandable embeddings. It shows potential in enhancing few-shot learning performance by harnessing the inherent knowledge within LLMs. The method involves training an encoder using contrastive learning, comprising three objectives: instance-wise, feature-wise, and text-prototype objectives. The instance-wise objective pulls closer to similar instance embeddings. The feature-wise objective is applied to each feature column to further discriminate instances. The text-prototype objective aims to align the TS embeddings and prototype embedding using feature-wise loss. Learnable soft prompts are also employed to assist LLMs in understanding time series data. Experimental results demonstrate that TEST can rival or outperform current SOTA methods.", "questions": "Have you considered the potential for parameter-efficient fine-tuning methods, such as LoRA, to outperform the use of soft prompts? Similar to using soft-prompts, LoRA does not require updating the model's entire parameter set.\n\nReference:\nLoRA: Low-Rank Adaptation of Large Language Models, Hu et. al.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The exploration of using language models for time series data without extensive pre-training or fine-tuning is an intriguing avenue of research.\n2. I am impressed by the non-fine-tuning approach to achieve performance comparable to a supervised baseline. It demonstrates the effectiveness of the proposed method on the time-series problem. Additionally, exploring the TS-for-LLM method is a promising direction for few-shot settings.", "weaknesses": "1. The paper lacks clarity regarding how the representative prototypes are chosen. There are three groups of prototypes (value, shape, frequency), and it's unclear how these are determined. Does their selection depend on the dataset, and what criteria are used to assess their effectiveness?\n2. I would like to see an examination of the performance impact of removing the decoder component. It looks like the decoder is not a necessary component in the proposed method.\n3. The paper could be stronger from an ablation study to thoroughly investigate the effectiveness of the proposed components and objectives. For example, what's the performance without using soft-prompt or feature-wise objectives? This would help readers assess the contributions of each component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M0lEoIBwjQ", "reviewer": "Reviewer_YYDc", "summary": "The paper explores two ways to accomplish Time-Series (TS) tasks using Large Language Models (LLMs): LLM-for-TS (model-centric) and TS-for-LLM (data-centric). The authors focus on the TS-for-LLM approach and propose a method called TEST (TimE Series tokens to align the Text embedding space of LLM). TEST tokenizes TS data, embeds it using instance-wise, feature-wise, and text-prototype-aligned contrast, and aligns the TS embedding space with LLM's embedding layer space. Soft prompts are then created to make LLM more receptive to the embeddings. The paper demonstrates the feasibility of TS-for-LLM through theoretical analysis and experiments on TS classification, forecasting, few-shot, and representation tasks. The results show that pre-trained LLMs with the TEST strategy can achieve better or comparable performance to state-of-the-art TS models and offer benefits for few-shot learning and generalization. The authors hope this study will serve as a foundation for future research in the TS+LLM domain.", "questions": "see above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The organization of this paper is well-structured, making it easy to read and comprehend.\n2. The paper proposes a novel embedding method (TEST) that aligns Time Series tokens with the text embedding space of Large Language Models, effectively enabling LLMs to handle Time Series data without compromising their language abilities. This study will serve as a foundation for future research in the TS+LLM domain.\n3. Through experiments on various Time-Series tasks, the paper demonstrates that the TEST strategy can achieve better or comparable performance to state-of-the-art models, offering benefits in few-shot learning and generalization, thus validating the feasibility of the TS-for-LLM approach.", "weaknesses": "1. Based on the experience with LLM and multimodal fusion, it is possible to integrate LLM with image/video/speech representations using techniques such as adapters once good representations are obtained. Similarly, LLM4TS and TS4LLM can be fused with each other, for instance, when we obtain a good TS token embedding using TS4LLM, we can directly fine-tune the large model using LLM4TS, thereby achieving good performance. Therefore, is it appropriate to isolate LLM+TS into LLM4TS and TS4LLM as two distinct categories?\n2. Time series prediction is a crucial task, and as observed from the experiments in this paper, there is a significant gap between TS4LLM, LLM4TS, and traditional methods when performing this task. Have the authors considered analyzing the reasons for this discrepancy?\n3. In the statement \"...we choose P representative text embedding tp as pivots/prototypes...\", what specific type of text does P represent?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fL1nIKoFjV", "reviewer": "Reviewer_M1Ks", "summary": "This research presents two approaches to handle Time-Series (TS) tasks with Large Language Models (LLMs): one that trains or fine-tunes a model for TS data, and another that transforms TS data for compatibility with existing LLMs. The focus is on the latter approach, introducing a method named TEST, which casts TimeSeries data into a format that LLMs can understand and then utilizes the LLM for TS tasks with the help of soft prompts. Experiments reveal that frozen LLMs equipped with the TEST strategy match or surpass current state-of-the-art TS models, enabling them to process TS data without losing language capabilities.", "questions": "Do you have concrete examples of how a Time-Series data is transformed into text/(any format consumable by LLM), and the output of an LLM to such time-series data(of course, with the soft-prompt included)?\n\nIt would be helpful to include such information, if not in main text, in the Appendix. \n\nHow scalable is the proposed method. \nWill you be releasing the encoder model, along with the soft-prompt for the task that the paper used?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper made clear the difficulty of using LLM for TS data: \"In this way, the core is to create embeddings that the LLM can understand.\"\nTEST method proposed a \"similarity-based, instance-wise, feature-wise, and text-prototype-aligned embedding for TS tokens\" for this problem. \n\nThe explanation and writing about the method, as formally shown in Algorithm-1, is straight-forward clear. \n\nThe theretical analysis about casting Time-Series data into a pattern, and then translating the pattern into text, is an interesting hypothetis. \n\nThe experiment design, which includes 7 dimensions and 5 baseline models, is comprehensive and convincing. The improvements of TEST can be clearly visualized by the shaded pink area in top-left Figure-3, which is quite significant.", "weaknesses": "A picture to illustrate the connection between the \"Time-series\" data, and then the corresponding \"text data to LLM\", will be helpful for readers to understand the input and output of the TEST method. And it would be interesting to investigate the connection between resulting target text's semantics and the actual content of \"Time-series\" data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QTg5DfEf9K", "reviewer": "Reviewer_t5Ny", "summary": "The authors propose an interesting way to do time series forecasting and classification with pretrained Large Language Models (LLMs). Whereas LLMs are designed to be used via prompts, the authors propose to find time series embeddings that are introduced in the model instead of finding suitable prompts. In this way, by considering an end-to-end approach, where the embeddings are adjusted according to the given dataset keeping the LLM weights frozen, the authors show that the proposed methodology does present a competitive performance. In particular, the larger the model, i.e. LLaMa-13B, the better the performance, potentially outperforming SOTA models like transformers. Finally, the authors present a brief analysis on the embeddings learn by the model, which suggests that these are mainly composed by adjectives/sentiments.", "questions": "1. What is jitter-andscale strategy?\n\n2. can the authors please expand a bit on what the following line means? _In this work, we choose P representative text embedding tp as pivots/prototypes, and map TS embedding to them_\n\n3. typo: Figure 3.g: spesific → specific\n\n4. typo: Figure 2: Fine-turn → Fine-tune.\n\n5. it seems that a general trend is that the LLaMa-13B performs best. It I understand correctly, this is as well the largest pretrained model considered. Further, one can see that for GPT2, the larger the model, the better the performance. Do we know, or can we do a conjecture, on what is the limiting performance in terms of model size (number of parameters)?\n\n6. Why is LLM-QA not showing any variance? Or why is it depicted in this way? \n\n7. Why is one-fits-all not used in *all* forecasting experiments? If I understand correctly, one can use it for comparison.\n\n8. Why are not other models considered here? for instance DeepAR, MQCNN, Wavenet, NHiTS, NBeats, etc for these experiments? Even models like ARIMA, ETS, seasonal naive would be interesting to consider.\n\n9. From 3.d it seems that the prompt length should be restricted to not be particularly large, but suprisingly the smaller the number of parameters (GPT-117M) the better the performance. How does this fit with the previous observation that larger models perform better?\n\n10. On Embeddings (section 4.3): how does the embeddings change per model? For instance, what are the embeddings when using BERT-335M, GPT2-774M, etc?\n\n11. When it comes to forecasting, is the proposed approach a point-wise or a probabilistic forecaster?\n\n12. What is the performance if the LLM taken is the one of the first checkpoint, i.e. when the weights of the model are random? What is the gap between these case those taken for the current experiments?\n\n13. What is the time execution/complexity of the proposed approach. Does this depend on the pretrained model in consideration? For instance, would it take longer of LLaMa-13B than for BERT-110M?\n\n14. What is the performance per dataset when it comes to time series forecasting? The current experimental setting only reports results aggregated across different datasets.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors motivate the proposed approach by clarifying that there are basically two ways to extend LLMs to time series: either by 1/creating a LM from a large enough time series corpus, which would imply an non-negligible amount of computational resources, or 2/ adjusting time series so that one can leverage available LLMs. Given that the later approach is more accessible to practitioners, the authors then start providing a way to embed time series to leverage existing LLMs for time series.\n\nThe strongest point of the paper is the notion of introducing time series embeddings to LLMs rather than finding the right prompt for it. Interestingly, the authors consider that by freezeing the weights of the LLM one can obtain embeddings that are leading to a competitive performance in terms of forecasting and classification. \n\nPerhaps the most interesting point would be to understand the kind of embeddings that are obtained from the proposed methodology.", "weaknesses": "1. Given that the proposed approach relies on contrastive learning, I would suggest the authors to provide a more self-contained and better written text when it comes to contrastive learning. Currently the paper assumes a non-trivial amount of familiarity to the topic and makes it difficult to follow. For instance, what does the following term mean: _jitter-and scale strategy_?\n\n2. There is no code included. This is relevant as I wonder if one can reproduce the experiments given the current writing quality of the paper.\n\n3. There are no evaluations for zero-shot evaluations. According to the motivation of the paper, it might be relevant as well to consider the case where the time series embedding is developed in a particular dataset, and then tested in another one. Although this might not be necessarily too far from the few-shot evaluation already presented.\n\n4. There is no analysis on time execution. The authors do not report how long it takes to obtain the time series embeddings. \n\n5. The results are presented in an aggregated way across datasets. It would be of interest for researchers and practitioners how the proposed approach performs per dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)