TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis
Authors: Sabera J Talukder
Yisong Yue
Georgia Gkioxari
Venue: Accepted by TMLR ()

Abstract: This work studies the problem of time series analysis with generalist (or foundation) models, which are models trained across many data domains. Drawing inspiration from the widespread success of large language models, we consider the simple strategy of discretely tokenizing time series data drawn from a myriad of datasets via self-supervision, then using the fixed tokenization to solve a variety of tasks across many data domains. Canonically, time series models are either trained on a single dataset or built in a task-specific manner (e.g., a forecasting-only model), where many use patches of time as inputs to the model. As such, performant generalist, discrete representation time series models explored across many tasks are of value. Our method, TOkenized Time Series EMbeddings (TOTEM), produces such generalist time series models with minimal or no fine-tuning while exhibiting strong zero-shot performance. We evaluate TOTEM extensively over nearly 500 experiments on three commonly-studied time series tasks with real-world data: imputation (17 baselines, 12 datasets), anomaly detection (19 baselines, 25 datasets), and forecasting (14 baselines, 12 datasets). We conclude that TOTEM matches or outperforms existing state-of-the-art models in both the canonical specialist setting (i.e., training one model on one domain) as well as the generalist setting (i.e., training a single model on many domains), which demonstrates the efficacy of tokenization for general time series analysis. The open-source implementation is available here: https://github.com/SaberaTalukder/TOTEM; a video summary is available here: https://www.youtube.com/watch?v=OqrCpdb6MJk.

Published in Transactions on Machine Learning Research (12/2024)
TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis
Sabera Talukder Yisong Yue
sabero@caltenn edu yyue@caltech., edu georgia@calteche edu
Georgia Gkioxari
California Institute of Technology
Reviewed on OpenReview: https: //
// openreviewww.... net/ formm? id- QUGG
Abstract
This work studies the problem of time series analysis with generalist (or foundation) models, which are models trained across many data domains. Drawing inspiration from the widespread success of large language models, we consider the simple stralegy of discretely tokenizing time series data drawn from a myriad of datasets via self-supervision, then using the fixed tokenization to solve a variety of tasks across many data domains. Canonically, time series models are either trained on a single dataset or built a task-specific manner (e.g., a forecasting-only model), where many use patches of time as inputs to the model. As such, performant generalist, discrete representation time series models explored across many tasks are of value. Our method, TOkenized Time Series EMbeddings (TOTEM), produces such generalist. time series models with minimal or no fine-tuning while exhibiting strong zero-shot performance. We evaluate TOTEM extensively over nearly 500 experiments on three commonly-studier time series tasks with real-world data: imputation (17 baselines, 12 datasets), anomaly detection (19 baselines, 25 datasets), and forecasting (14 baselines, 12 datasets). We conclude that TOTEM matches or perforrsm existing state-of-the-a models in both the canonical specialist setting (i.e., training one model on one domain) as well as the generalist setting (i.c., training single model many domains), which demonstrates the efficacy of tokenization for general time series analysis. The open-source implementation is available here:
Talukderr/ra TOTTM, video
summary is available here:
1
Introduction
Time series are fundamental data modality, generalizing large classes of time-varying data from many domains, like weather phenomena, electrical grid activity, or traffic flow. Most commonly, time series analysis
is first restricted to one such domain, then to a specific task, like imputation (Luo et al.) 2018 2019 Talukder
et all 2022 anomaly detection (Xu et al. 2021 He & Zhao 2019). or forecasting (Wu et al. 2021 Woo et al, 2022 among others. Though these domains and tasks are quite distinct, a natural question whether it is possible design domain-agnostic models adaptable to any task. This question is the subject of our work.
Generalist models are those trained on many data domains simultaneously (e.g., weather, electricity, traffic, single time series domain (e.g., weather only), shown in Figure A (Zhou et al.) 2023 Wu et al.) 2022: Nic et al. 2022). Both generalist and specialist models can be tested in two ways: in-domain lesting, where a model is lested on the same domain(s) on which it was trained, and zero-shot testing, where it is tested on different domain(s) (see Figure [B). Performing zero-shot testing on spocialist models is not uncommon. For example, some works have studied zero-shot forecasting, where a forecaster trains on one dataset then predicts on a separate dataset (Zhou et al., 2023), or trains on subset of channels (which call sensors) from one dataset then forecasts zero-shot on the remaining sensors in the
etc.), while specialist models are those trained

---

Published in Transactions on Machine Learning Research (12/2024)
a
Training Paradigms
X
ID CNN
Z
quantize
2
ID TCNN
Generalist
2
&
D
D
MMM w
T/F
T/F+
WMM W MMM
Specialist
'no
MMM
freeze codebook
2
K
Figure 1: TOTEM Overview, Training Paradigms, Inference Paradigms & Tasks. (a)
Inference Paradigms
impute
tokenize
detect
5
anomaly
or
TOTEM's VQVAE enables generalist training, i.e. on many Zero Shot
forecast
Domain
E
data domains jointly, and specialist training, i.e. on one data domain at a time. The TOTEM VQVAE architecture consists of 1D strided CNN encoder E, quantizer, latent codebook, and 1D strided transpose
CNN decoder D. (b) TOTEM's discrete, sclf-supervised codebook is frozen then leveraged for both in domain and zero shot testing across many tasks.
same dataset (Liu al.; 2023). However, we emphasize that both of the preceding examples are specialists, as they were trained only (or subset of one) dataset. In contrast, goal in this paper is instead the design of generalist models, which we evaluate in both the in-domain and zero-shot testing regimes.
Not only are most modern time series models specialists, they typically operate over patches Zhou et al.
2023: Wu et al.
2022 Liu et al. 2023 Zhang & Yan 2022 Nie et al. 2022 Li et al., 2019: Zhou et al. 2021
Wu et al. 2021 and are trained for only a single task (Das et al.) 2023b Ansari et al. 2024 Liu et al. 2023
Zhang & Yan 2022 Zhou et al. 2021: Wu et al.) 2021). Our core hypothesis is that many of the design decisions in prior works hinder the development of generalist. models, and that by adopting practices more
commonly used in language (Gage 1994 Radford et al. 2018) and vision modcling (Van Den Oord al.
2017
Esser et al.) 2021 Rombach et al. 2022) we can boost the ceneralization performance of resulting time
series models. While there exist works that train in an unsupervised manner (Yue et al.) 2022 Yang & Hong
2022
Tonekaboni ct al. 2021 Barnum et al.) 2020 Franceschi et al. 2019) or use discrete represontations
RRbaanse et al. 2020b Van Den Oord et al. 2017 Lin el al. 2007), few works have explored the combination
of generalist models and discrete representations over many tasks in a systematic manner (i.e., both the in-domain and zero-shot testing regimes). Thus, the contributions of our work are twofold.
TOTEM. We develop Tokenized Time Series Embeddings (TOTEM), simple tokenization method for time series data that employs a self-supervised pre-training stage to learn fixed number of discrete tokens over a multi-domain corpus (Section 3.2). Surprisingly, we demonstrate that TOTEM is effective for solving a variety of downstream tasks a agnnsiic manner even though the tokens only encode the shape of univariate waveforms. This allows TOTEM to generically tokenize multivariate data, of differing size by simply stacking collections of univariate tokens.
Comprehensive Experiments. We test our hypothesis extensively on three distinct tasks, each with their datasets and baselines: imputation (17 baselines and 12 datasets), anomaly detection (19 baselines and 25 datasots), and forecasting (14 basclincs and 12 datascts). We find that in the specialist scttings, TOTEM matches or outperforms the performance of most state-of-the-art (SOTA) task-specific models, despite minimal or no task-specific design. Similarly, TOTEM also matches or outperforms SOTA generalist models. We conduct thorough ablations showing that discrete tokens outperform patches and that generalist training improves model performance independent of TOTEM's modeling choices. Our experiments are some of the most extensive in the literature, comprising hundreds of seeded runs (see Sections 5 and 6
2

---

Published in Transactions on Machine Learning Research (12/2024)
2
Related Work
We categorize related works in three ways: whether they (i) study specialists or generalists, (ii) use patched or discrete data representations, and (iii) train and evaluate models for multiple distinct time series tasks. Unlike TOTEM, no prior works study the use of discrete data representations for training generalists across multiple lasks (see Table for a comparison).
Specialist vs. Generalist Training. Historically, the specialist (i.e., single-domain) training paradigm is most common amongst time series models (Zhou et al. 2023 Wu et al.) 2022 Nie et al. 2022 Zhang & Yan 2022 These specialist models are primarily evaluated via in-domain lesting, where the test set is held-out set from the same training domain. Some concurrent and subsequent works have begun exploring generalist time series foundation models, including forecasters from Google and Amazon (Das et al. 2023b Ansari
gt al.. 2024 We compare to the concurrent MOMENT model (Goswami et al. 2024 in limited evaluations (see Tables 13 and 20 as also studies multiple tasks, and find that TOTEM generally outperforms it.
Patched vs. Discrete Data Representations. In order to pass time series data to downstream model, it is necessary to choose some latent data representation. As in ViTs (Dosovitskiy et al. 2020), the prevailing strategy is to patch time scries data, either temporally (Liu et al. 2023 Zhang & Yan 2022 Nie et al.) 2022) or spatially Li et al. (2019) Zhou et al. (021)); Wu et al. 2, then to linearly project the patches to some latent embedding on which a model like a transformer or MLP can operate. We emphasize that patched representations are dynamic in the sense that the embedding associated with the patch is determined entirely by the layers in the downstream model which project the patches to the embodding space. Therefore, patchod representations are trained end to end.
Patching is fundamentally at odds with tokenization, wherein fixed "vocabulary" of embeddings is deter- mined before training the downstream model, which then operates on the fixed, tokenized representations. Tokenization (learned or otherwise) has been leveraged for training models in fields like language and vision
modeling (Gage 1994 Radford et al. 2018 Van Den Oord et al. 2017 Esser et al. 2021 Rombach et al.) 2022). Some prior work in time series modeling has explored discrete representations using binning (Rabanser
et al. 2020b a Lin et al. (007) or quantization (Baevski et al., 2020 Van Den Oord al. 2017 Oord et al,
2016 in domain- or task-specific ways. Inspired by the success of vector quantized variational autoencoders (VQVAEs) in both audio and vision (Van Den Oord et al. 2017 Esser al. 2021 Rombach et al. 2022),
build on these works by showing that the VQVAB is also ellective for learning discrete representations for
general time series modeling.
Generalist Discrie Tokenization Multiple Tasks
GPT2
Zhou et al. 2023
Prior
TiNet
Wu el al. 2022)
W2V2.0
Baevski et al. 2020
SAX
Lin et al.
2007
TimesFM
Das et al. 2023b)
X
5
Chronos
Ansari et al.]
2024
MOMENT
Goswami et al.
2024
TOTEM (Ours)
Table 1: Related Work Overview. TOTEM is designed for generalist training using discrete tokenization for any task. No prior and concurrent/subsqqueen (C/S) works study all three at once.
Time Series Tasks. Prior works on time series modeling study a variety of tasks, like forecasting, anomaly delection, imputation, and classification.. Many prior and concurrent works focus on a single task (Zhang & Yan, 2022 Nie et al. 2022 Xu et al.) 2021 Ansari et al. multiple specialist trained models on many tasks (Zhou et al.
Das et al. 2023b), with a few exploring 2023 Wu et al. 2022). TOTEM is most closely 2024), which are focused on generalist models
2024
related to concurrent works like MOMENT (Goswami
which are effective any one of the above tasks. For detail each task, see Sections and
3

---

Published in Transactions on Machine Learning Research (12/2024)
3 Method
3.1
Task Definitions
This work considers three tasks: imputation, anomaly detection, and forecasting- In impulation, models intake masked time series Xm Sx and impute the missing values to recover the reconstruction & E I xgSx,,, In anomaly detection, models intake time series corrupted at known level Xcorr R and predict which times anomalous, e {0, 1 Lastly, in forecasting, models intake time series x E SxI and predict future values y E R where Tin and Tout signify the durations of the preceding and succoeding time series, respectively. A core design goal for TOTEM is to learn a representation suitable for any of these three tasks using the same architecture and without leveraging any task- or domain-specific knowledge.
3.2 Design Decisions
This section discusses TOTEM's key design features: a self-supervised training stage, exchusively-temporal tokenization, and no domsin-specifie data engineering.
Self-supervised Tokenizer Training. As described in Section 2 TOTEM learns fixed codebook of tokens over multi-domain corpus of time series data independently from the training of any downstream model. This disentangles the choice of data representation from the choice of task-specific architecture and permits the learning of representations from a large, diverse set of data, which aids in zero-shot generalization.
First, we elect to use a discrete, deterministic encoder to produce time series tokens. This decision largely motivated by large language models (and in particular, tokenization methods in NLP like byte pair encoding (BPF) (Gage 1994: Radford et al.] 2018)), in which a downstream model learns tokens. Moreover, in methods like BPE, the tokenization operation is lossless and reversible because it is deterministic (though non-unique). This suggests that vector quantization quantization-based models could be effective for tokenizing time series data. Two popular vector quantization methods are VQVAEs (Van Den Oord et al., 2017 and VQGANs (Esser et al. 2021 ). In this work, we choose to use a VQVAE, as VQGANs are more commonly used for encoding images. Moreover, the use of VQVAEs has been studied in neural audio models (Oord al. 2016 Van Den Oord et al. (2117), including followup works with audio-specific models (Baevski et al.
finite number of dist inct
2020 which suggests that they may be effective for modeling general time series.
Tokenization. A time series dataset consists of E' examples, S sensor channels, and T time steps, LE and can be formally expressed as {x,}/=1 C RSTT Prior work
Exclusively-
commonly patches along either the scnsor dimension [Li ct al.
2019 Zhou el al.] 2021 Wu et al. 2021 Liu al.
2021), or
W
time dimension (Liu et al. 2023 Zhang Yan 2022
Nie et al.
2022 When training specialists, it is reasonable to tokenize
S
M
S
M
across any combination of these or the example dimension (e.g..
W
in neuroscience data, it is common to group recordings by day,
where the subject exhibits different behavior on a daily basis
E
E
Talukder et al. 2022
Figure 2: Left. Specialist models can tok- enize along any of the E, S, or T dimensions. Right. Generalist models can only tok- enize along T, since the learned tokenization must apply diverse set of domains with any possible data dimensionality.
However, in the generalist case, bocause the sensors associated with each domain have distinct semantic meanings, perform- ing sensor- example-wise tokenization will capture domain- specific relations, hindering the tokenizer's generalization. Thus, we choose to exclusively tokenize over the temporal dimension, such that the tokens represent univariate wavelorms. Further, this is crucial for testing the tokenizer in the zero-shot regime, where the dimensionality of the testing domain may differ significantly from that of the training domain(s). Specifically, TOTEM tokenizes time series data with non-overlapping temporal chunks of length T/F, where F some compression factor for downsampling
the data.
4

---

Published in Transactions on Machine Learning Research (12/2024)
No Domain-specific Data Engineering. Many prior works (especially in time series forecasting) leverage domain-specific knowledge to handcraft features that encode critical information. For instance, works that study calendar-based time series often add auxiliary features that denote landmarks like the first day of a month or holidays (Chen et al. 2023 Salinas et al. 2020). Other works propose highly-ongineered architectures that convert Lime series into Trequency space representations. For example, TTimesNet operales on the assumption that most time scries exhibit multi-resolutional periodicity, and convert a time scries into frequency-space image by computing the Fourier transform on several subsets of the time series (Wu et al, 2022 Similarly, FedFormer represents time series with a random subset of its Fourier components and complex mixture-offeexperss model (Zhou et al. 2022). In contrast, TOTEM uses only reverse instance normalization (RevIN) (Kim et al. 2021 to represent temporal waveforms in a normalized space (see Figure 3 which requires no assumptions on the form of the data. This allows TOTEM to generalize across domains and outperform the prior handcrafted methods on many distinct tasks using simple, generic architectures.
3.3 Tokenizer Implementation
W
Though TOTEM is a VQVAE, the design of the encoder and decoder differ substantially from the original model and similar works like WaveNet, which use dilated convolutions (Oord et al.] 2016 Van Den Oord et al.) (017).. The dilations in these architectures skip many time steps, allowing the convolutional filters to operate on a larger input area at a coarser scale, improving model efficiency. However, this design decision is motivated by the high sampling rates of digital audio waveforms, which is not a universal trait across time series domains (see Table 6 In contrast, TOTEM uses a stack of strided 1D convolutions with a dilation of 1 such that it can account for every time step. Using a long input (e.g, 96 time steps for standard forecasting lasks) allows TOTEM to maintain a large receplive field. Lastly, the use of RevIN allows TOTEM remain effective by only learning small set of normalized waveforms, and if the unnormalized reconstruction is required for downstream task, the normalization parameters can also be passed to the decoder (see Figure
m
SxE
Figure 3: TOTEM flattens the sensor and example dimensions and learns a discrete representation along the time dimension in normalized space.
Formally, TOTEM accepts a batch of univariate time series {x, E R obtained by flattening the sensor channel of the multivariate data, An encoder consisting of a stack of strided 1D convolutions then temporally compresses the data by a total factor of F to recover a latent variable where D is the latent feature dimension. The latent variable is then quantized into an element 2 of the codebook C {c, ],K, consisting of K D-dimensional codewords c, RR following the relation & C2, where L arg min, || cill.. The decoder D mirrors the encoder's architecture, mapping the quantized embedding à to a reconstrueted time series & = D(ž) E RT.
E(x) E R //xx
As in Van Den Oord et al.
2017 we train E, D, and C by optimizing the objective
L =
Σ
|x & +
sg|z] 2 12 +B sg(2]11
(L)
E.S
where sg[-] is the stop-gradient operator and B is the commitment loss weight. For additional details, see Appendices A.11 and A.12 In all experiments, we use compression factor of F 4, (sce Table 32
3.4 Forecasting Model Implementation
In contrast with prior works, TOTEM is capable of solving the imputation and anomaly detection tasks with the tokenizer alone (see Figures and 12 Therefore, the only downstream model we must design is the forecasting model. First, each sensor's observations X; E RTi. are converted into a sequence of Tw//P discrete tokens 2, The forecaster processes adds temporal positional embeddings to those tokens, passing them through a transformer encoder consisting of a series of multi-head attention layers that attend along the time dimension to predict normalized measurements y; E Ro for = 1,.., S.
5

---

Published in Transactions on Machine Learning Research (12/2024)
A separate prediction head predicts the mean [4 and standard deviation associated with each univariate time series X, such that the final forecasted prediction is y, = 0, y, + l4 for = 1, 1. The forccaster is trained in supervised fashion by minimizing three smooth I1 losses between predictions (5x, (i; 0; ) and their ground truth values respectively. Crucially, this architecture is used for all domains in our fore- casting experiments, demonstrating that TOTEM can competitively perform forecasting in a domain- agnostic manner.
W
linear layer
forecast
transformer encoder
linear layers prediet
future mean & std
add positional embedding
0
tokenize with pretrained codebook
T/F
4 Experimental Setup
Figure 4: The Forecaster Model. The forecaster takes in a tokenized version of normalized time series observations (obtained using TOTEM's encoder) and predicts normalized time series over some specified horizon along with parameters that allow the model to unnormalize the prediction.
This section explains the experimental setup for each task, including the baselines and datasets used for evaluation. The results and analyses are presented in Section 5 We compare two families of approaches: methods designed for multiple tasks (multi-task), like TOTEM, and methods designed for specific task (single-task). Many single-task methods have frequently been adapted by others to tasks besides the ones for which they were originally designed, and in those cases, we compare against the best reported results for the adapted model. For all tasks, we trained GPT2 generalist baseline from scratch, and for forecasting, we additionally trained a GPT2 specialist.
4.1 Imputation
Baselines.
In the main text, we compare TOTEM against. 12 baselines with varying model architectures We further compare against 5 additional baselines with different architectures for completeness in the Appendix
A.2 In total, evaluate against 17 baselines. See Table 2A for summary.
Datasets. For the in-domain testing regime, we test on 6 datasets, and for the zero-shot testing regime, we evaluate on an additional 5 datasets. We also perform additional evaluations in the appendix on the PhysioNet Challenge 2012 dataset. In total, we evaluate on 12 distinct dalasels. See Table B for sunmary.
Metrics. We report the mean squared error (MSE) and mean absolute error (MAE) of the imputed versus the ground truth signals.
Imputation Baselines
Type
Model
Abbr. Arch.
Citation
B. Imputation Datasets
Multi-task
GPT2 Gunzzliit
GPT2- Specialist
GPT2 TF
TF
Traine by
Regime
Dataset
Abbr. Citation
TimesNes
TiNet Conv
Wa ct al
Weather
W Zhou es al
X723
PS
Patch
TF
20/2
Electrictty
Zbom
ETSFormer
Podformer
FED
ETS
TF
TF
Zhou
Woo et a1
2024
2022
In Domatn
DTTu2
m1
m22
Zbou cs ml
Zbou
2025
Non-stationayy Trans.
TF
20221
ETTh
h1
Zaom
2025
Single-task
Antofurmer
Auto
TF
Wi st ml.
2021
WTTTh2
Informer
DLinear
Refermer
LighTS
LITS
Inf
DLin
TF
Limar
TF
Linesr
Zhon
Zeng
Xinagg at al
Kitacv
(0/2
21223
2020
Zero-
Saugees River Flow
Neuro2
Neuroō
Births
N5
Goddloowa ct 41
Peterson a1
2022
7027
2022
V-RIN
VAR
Mulyael al
21021
Sumspot
Godalhew
Appendix
RDIS
Diffuioa
Choi et al.
2023
Appendlis PlysioNel
Silva
2012
Unaveditional CSDI
Diffasion
Tsehiru
2722
Table Imputation baselines and datasets.

---

Published in Transactions on Machine Learning Research (12/2024)
4.2 Anomaly Detection
Baselines. In the main text, we compare TOTEM against 16 baselines, and in the Appendix A.3 an additional 3 for total of 19 baselines (see Table 13 See Table 3A for a summary.
Datasets. For the in-domain testing regime, we test 5 datasets, and for the zero-shot regime, test another 5. For additional signal, we also test 15 distinct anomaly detection datasets from Wu & Keogh 2021 in Appendix A.3 (see Table 13 In total, we evaluate on 25 datasets. See Table 33 for a summary.
Metrics. We report the precision (P), recall (MSE), and adjusted F1 score.
A. Anomaly Detection Baselines
Type
Model
Multistaok
GPT2- Gemeralist
GPT2
Triinwd
GPT2 - Specialist
GPT2
TF
Zhou et al 2022
В. Anomaly Detection Datasets
TimmsNet
TiNes
Conv
Wa et
2022
Regione Datasct
Abbr. Citation
Anomaly Trans.
ATtaa
TF
Su ml 2021
SMD
Zhon al
2073
PatdTTT
Putch
MSL
Zhoa ot al
2028
ETSFormer
In-Dumain SMAP
Zhou
Fulloemer
FED
TF
Zioe et
(2022
SWAT
Zhou
Non-stationary Trans.
Stat
TF
Lit
20223
PSM
Khoa at ml
2023
Autoformer
Single-bac
Peraformer
Informer
Inf
Pyza
TF
TF
Liu et m]
Zhou
2021
1122
Neunvd
Neuno2
N2
K5
Patenson et m.
Putemson
2722
A122
Reformer
2020
Shot
Saugeen River
Godalews
LogTrams
Lo
TF
Li ml.
2019
U.S. Births
Godahews et
2021
Tranfformer
Tham
TF
Vaagmant
2017
Suraped
Gire ceeensheuss
2021
LightTS
Zhang
22022
DLinear
DLis
Linea
Zetag et ai.
2020
Appondix 15 Wa ad. Dacasete
Wa & Keagh
2021
DCILL
Chol.a ct al
2222
Appondia
MOMENT-0
Goswami
2224
MOMENT-LP
Table 3: Anomaly detection baselines and datasets.
4.3 Forecasting
Baselines. Tn the main text, we compare against. baselines, with an additional 2 in Appendix A.4 (see Table 20). For the GPT2 specialist that we trained from scratch, we choose lookback length of 96 for fair comparison with the other models in this paper. In total, have 14 baselines. See Table 4 for summary.
Datasets. For the in-domain testing regime, we test on 7 datasets, and for the zero-shot regime, we test on additional 5. In total, we evaluate on 12 datasets. See Table 4 for a summary.
Metrics. We report the mean squared error (MSE) and mean absolute error (MAE) of the predicted versus the true forecast values.
A. Forecasting Baselines
Type
Model
Abber Arch. Citation
B. Forecasting Datasets
Multi-task
GPT2 Generalist
GPT2 TF
Traiued
Regime
Citation
GPT2- Specialiat
Trained
w/ lookbook length
TimesNet
TYNes Conv
Wa et al
2027
Electrictty
Lau <5
2023
Transfrmer
iTrnn TF
Tau st
2025
Teaflic
Liu
2025
PatchTST
Patch TF
In-Domaen
Crossformar
Yua
DTTTm
m2
L70 05
2023
Fedlormer
FED
TF
Zhou ct al.
20222
ETTh
hi
1i
2023
Single-tack
Nonostationary Trans
Stai
TF
Lin a..
20021
ETTh2
Neurol
N2
Peterson
2022
RLiacar
BLin
Linear
< 0l
2023
Neuroß
DLinear
DLin
Lioez
Zang Al 22123
Zero Shot
Saugee River Fl
Gindslewss at
2021
SeaNet
U.S. Darths
Godabees
2021
Appendix
N-Bends
MOMENT
Drsshkin
Ciowwam
220/2
Suuspot
Godalows
Table 4: Forecasting baselines and datascts.

---

Published in Transactions on Machine Learning Research (12/2024)
4.4 Task Selection
In the time series literature, there are five canonically studied tasks: impulation, anomaly detection, short- and long-term forecasting, and classification. In this work, we study imputation, anomaly detection, and long-term forecasting. We exclude short-term forocasting and classification for the following reasons.
Non-Standardized Baselines. The long-term forecasting task uses standardized input and output lengths across all datasets (in particular an input length of 96 timesteps and output lengths of 96, 192, 336, and 720
timesteps), as enforced by large body of existing work Liu al. (2023); Wu et al. (2022): Liu et al. (2022b): Zhou et al. (2022) among other This allows us to fairly baseline TOTEM without rerunning thousands of experiments dozens of models trained from scratch.
In rast, the shorl-lerm forecasting task lypically uses non-standard and datasel-specifie input and output dimensionalities (see Table 19 for details), which makes systematic, fair comparisons of TOTEM against prior works extremely challenging in the generalist setting Thus, we exclude short-term forocasting from our main results.
Leaky Baselines. In both classification and anomaly detection, the modern SOTA baselines are leaky (Zhou et all, 2023 Wu ct al. 2022 Xu et al., 2021), where Icakage is defined as using the test set as the validation set during training. In particular, the cited works that report SOTA results all use models that were trained with either early stopping or with the best model checkpoint on the validation (i.e., the test) set. We felt strongly that we should not propagate faulty baselines, so we did not compare to these models in our work. Subsequent to the initial release of this paper, followup works have demonstrated on neural classification tasks that TOTEM, when compared to baselines trained a non-leaky manner, achieves SOTA performance (Chau et al. 2024a b).
For anomaly detection, the benchmark datasets used by Zhou al. (0023) Wu et al 2022 Xu et al 2021 contain numerous flaws besides training leakage flawed (see Wu & Keogh 2021 for detailed account) However, since Wu & Koogh (2021) released a large set of new, unflawed benchmarks, we elected to compare TOTEM to both the flawed and a subset of the unflawed baselines (see the comparisons to Wu & Keogh [2021) in the Appendix). Because we find that TOTEM convincingly achieves SOTA performance in both cases, we report our results to establish an unflawed baseline for future comparison.
In summary, due to non-standardized and leaky baselines, we only report systematic results on the imputation, anomaly detection, and long-teni term forecasting tasks.
5
Main Results
The primary goal of our experiments is to systematically evaluate TOTEM multiple tasks simultaneously against new generalist benchmarks and strong specialist baselines (i.e., models trained on data from many domains versus one domain). In particular, for each task, we report evaluations against (i) specialists on the in-domain testing regime, (ii) generalists on the in-domain regime, and (iii) generalists on the zero-shot regime. We emphasize that no domain, sampling rate, or sensor dimension is shared between the training sets and zero-shot testing sets (sce Table G for additional datasct details).
Throughout the main text, report summary results. The full numerical results can be found throughout the Appendix. Moreover, all results are reported as the mean of 3 seeded runs, with standard deviations available in the Appendix. Since evaluation metrics differ across tasks, (1) will denote metric where lower is better and (t) will denote a metric where higher is better. Given the varied metrics, we calculate the average number of best results, or AvgWins, for each method and highlight the best method. For a summary of training and testing domains, see Table 7 for a comparison of generalist paramcter counts and training times, see Section A.S for additional architecture and training details, see Sections A.11 and A.12
2ome methods utilize 512-dimeusional input, which makes consistent comparisons challenging: despite this field-wide inconsistency, we include some of these results in Appendix 20 TOTEM outperforms other methods across lookback lengths 96, 512 at 58.3% AvgWina while the next best model is GP12 at 8.3% AvgWins 3 Despite this, we show that TOTEM and GPT2 outperform all other methods small of short term forecasting lengths and datasets in Appendix T8
8

---

Published in Transactions on Machine Learning Research (12/2024)
B. Generalist In- In-oomain
A. Imputation Performance Summary
Model TOTEM
GPT2
Specialist Domain (Table
Metric MSB
MSE
MAT
Avg. Wirs (%)
60-
40
35.4
Singletask Method
Multitask Melhod
12.5% 0.029 0.060
$1.3%
25% 0.050
50%
0.032
0.038
0 0.000
0 0:062
0 087
0.029
0.033
0.017
0.043
0.048
0.054
0.061
0.045
C. Generalist Zero-Shot
Model
Metric
MSB
TOTEM
MAE
MSB
GPT2
MAE
Ours
18.8
12 3%
25%
0.065
0.080 0.189 0.108
0.171
0.0000
0.213
0. 0.186
57.5%
12.5%
0.041
00.029
0.139
0.120
0.000
0.017 0.145
0.191
50%
0,095
0 0.205
132
0.236
50%
0.066
0.160
131
0.288
0
12.5%
0.041
0.132
0.052
0.141
12.5%
000077 0.085
0.021
0.095
25%
0.011
0-135
0.065
0.154
0.019
0.090
0. 0.024
0.107
TOTEM
GPT2
TiNet
Patch
FED
Stat
Auto
Inf
Re
DLin
S1 27-376
0.048 0-139 0.085
0,058
O.
0.177
19196
0,,029
0.022 0. 0.09%
0.1
0.03
0.123
12 5%
0 0.040
12 2.5%%
10.071 0.109) 093 0.119
Generaalsst Dumain (Teble 26
Generalist Zero Shot (Table 2C)
25%
0.041
0 126
0.033
0.101
25
0.08T 0.117
125 0.134
100
37.32
0.043
0.121 0.038
0.110
3F
0.112 0.129
0. 16
0.154
56.3
80
502
0.048
0 136
0.046
0.121
50%
D. 4N
D. 147
220
0.182
Avg. Wins (%)
60
80
12.5%
0.100 0. 20
0.313
0.217
12.5%
00.612
0.642
30.392 0.496 0.6GE
43.8
25%
0.108
0.209
0.131
0.231
0-444
0.523
40
Ours
40-
60
Ours
31.3%
0.144
0.237
182
0,266
50%
0.827
0.718
0.591
0.599
20-
20
31.3%
12.3%
2026
0.075
0.093
0.076
0. 195
0 071
0.177
0.077 0.167
0.067
0.071
0.160
0.155
5
37 47.5%
12 5%
25%
00.057
0.069 0.178
0.061
0.160
0.168
0.070
1.0 0.209
081
0. 189
0.173
0
0 192
0.0N6
0.179
50%
0.0082 0.
1.2 0.234
TOTEM
GPT2
TOTEM
GPT2
Avglins
58.3%
43.8%
AvgWins
80.0%
20.0%
D. Specialist In-Domain
Model
TOTEM
GPT2
TiNet
Patch MAR
ETS
FED
Stat
Auto
Inf
Re
LIT'S
Dlin
Metric
MBB
MAE
YSE
MAB
MSB
YAE
XSE
MSE
MAE
MSSE
MAB
MAR
MSE
MAR
MAE
ISEE
MAR
ISS
MAE
MSE
MAR
12.55
0.028
0.046
0.026
0.049
0.025
0.045
0.029 O (LD49 0. 0.057 0.141 (0.041 0.1077
0.051 0.026 0.D47 0.037 0.031 0.076600.04
0.101 0.030 0.D84
25%
0.029
0.047
0.025
0.052
0.029
0.052
0.031 D. D.053 0 065 0.155 0.064 O. 1616 0. 029 D. 0.056 0.030 0.D51 0.042 D. 1O0KL0S5 0.082 0.052 0.1.1 0.0
0.108
37.37
0.031
0.048
0.631
0.080
0.031
0.057
0.035 0. 1.088 0. 081 0.180 0. 107 0.228 0.033 0.062 0.032 0.060 0.049 0.1 11 0.040 0.031 0.058
0.121 0. 0.057 0.1
5028
00,033
0 052
0.037
0.065
0.054
0.062
0388
D63
0. 102 D 202 R) 183 0. 0 037 O
0.087
D.T 1 0.053
D. 111 KL OLG D. D.02
055
0
0. 0.066
131
12.4%
0.154
0.080
0.
0.085
0.20/
0.055
0. 196 0.321 107
093 0.210 0.080 012210 0.
190 0.308
0.220
2
0.205
00.667
0.169
0.094
0.004
0.213
0.076
0.189 0.
196 266
102 O 222 104 0.229 0 22222 0.328
203 0.315 141 0.273 144 0.276
50%6
00.079
0.183
0.
101
0.220
0.100
100
0.221
091
0.20S 0 235 3.3 D. 158 O 28 1 108 O 222 O. 113 O. 2323 0 228 0.331 R 210 O.3 0.319 0 160 O 2913 O. 0.175 0.305
12.5%
0.049
0.125
0.017
0.085
0.019
0.092
0.041 0.130 O. 067 0.188 00.035 O. 1.555 O. 026 O. 0.1 0.034
24 0.047 0.155 55 NJ 0L0 12 0.075 0.180 0.058 62162
25%
0.062
0.128
0.022
0.096
0.023
1.10
0.0
D. 135 O OBG D 229 K 0.2 O. 16B 032 O.L 19 0G D.1. 14 O OES 180 KI 12 D. 16
093 0.201 O. 080 193
37.5%
0.055
0.112 0,020
0.111
0.029
0.111
0. 0.040
D. 0.143 O. 135 0.271 R 06069 O. 191 ORO O 1111 O. O5T D. 16 0.0 0.200 D 063 0.
113 O 0.231 O. 1.103 0.215
50%
061
0. 139
0.040
0.128
O.
036
0.124
05S
151 O. 18G 3:3 I ONN O
047 D 14 OIT 174 D OBS 212 EI ON2 0.20
134 O 0.255 O. 1322 248
12 12.5%
00:016 0 0.078
0.017
0.076
0.018
0.080
1
108 239 1O 0E6 D 39 121 O DSS 023 D92 133 270 I0S
030 D 1.1 062 LG
25%
0,081
0,020
0.085
0.
O 030
135 O 136
042 0.143 0.085
37
0.018
30300
222
30%
0.020
0 08.088800
0
025
0,085
0.026
0.098
0
034
10
0.
323
0.121
156 O. 227 030
LOS 035
19 0. 200 0.353 E1
O. 0.329
059 0.174
131 0.24T
12.5%
0.119
0 012
0. 0.043
0.
0 0.057
0. 159
083
0.20 201
0. 12G 263 1O 070 O. 190 O. OGO O. GG 0.074 D. 182 O. 114 D. 0.234 074 O. 191191000 200 O 0.345 O. 151 0.20 A.26T
= 25%
0.127
0.220
0.054
0.156
0.069
0.178
0. 107
0.217
0. 169 0.304 R. 106 O. 236 O8O O. L8D 0.090 0.201 O. 1:00 0.262 RL 102 0.227 0. 262665 O. 0.364 0. 180 0.292
37.35
0.138
0.230
0.072
0.180
0 0.184
0. 0.6
0. 120 0 0.2230
0 220 3.34T O. 124 D. 0.255 102 01212 0. 104 122222 O. 174 D 0.2433L K. 135 0.261
1.296 D 0.3882 0.216 0.318
30%
0.157
0,247
0.107
0,216
002
0.215
0. 141111110 0.248
0. 253 D. 402 10 165 O. 0,29H 135 O 0.240 O. 187 2.2 012255 0.325
11,111111 179 0.298
(1 334 0. 0.101 0.255 0.347
12.5%
0.640
0.129
0.039
0.5
0.040
0.130
0.057 0.152 0. 187 0.319 00.055 0.213 044 O. 133 0. 0.044 0.138 0.305 0.451 10. 16. 0,289
101 0.231 0.100 0.218
25%
0..41
01.181
0.044
0.135
0.046
0.141
0.066 0.1558 0 27777 0.300 0. 137 D 0.255 049
0.147 0.050 19149 0.
0.444
K1:20G 0
0.246 0. 0.247
0.151
0. 065
4
0.060
0.355
0.257
0.158
M/%
0.047
0.142
0.038
0.158
0.000
0.0
0.174
0.602
292
065 0.170
0.173
0.472 V1 31.316
0,419
136 0
(1,200)
Avgline
52.1%
35.4%
18.8%
0%
0%
0%
0%
0%
0%
0%
0%
0%
Table 5: Imputation Summary, In all categorics TOTEM has SOTA AvgWins In the specialist TOTEM has 52.1% AvgWins ; in generalist in domain TOTEM has 38.3%; in generalist zero shot TOTEM has 80.0%.
Since we only use 3 seeds, run a non-parametric permulation lest on the generalist models in Appendix A.6 to analyze the performance of TOTEM vs. GPT2 (Table 24). and TOTEM vs. PatchTOTEM (Table 25 We find that TOTEM statistically significantly (p S 0.05) outperforms GPT2 in terms of AvgWins on all tasks for both the in-domain and zero-shot lesting paradigns. Additionally, TOTEM outperforms PotchTOTEM in a statistically significant (p S 0.05) manner for in-domain and zero-shot testing.
5.1 Imputation
In imputation, models intake masked time series Xm E RSTTi,, and then impute the signal & E RS (see Figure 11 We experiment with four canonical masking percentages at 12.5%, 25% 37.5%, 50%, and report the resulting MSE and MAE.
Specialists. Figure BA and Table 5D compare TOTEM to specialist baselines. All models are trained and evaluated on the same datasct (in-domain). TOTEM has the highest AvgWins with 52. 12,1, followed by GPT2 at 35.4%, and TiNet at 18.8% TOTEM's performance on ml and h1 is lower, but since these datasets are the minute and hour resampling of the same raw data respectively, we expect. their results to be correlated.
9

---

Published in Transactions on Machine Learning Research (12/2024)
Generalists. Figure 5A and Tables BB&C compare TOTEM to GPT2 (the best two models in the specialist in-domain regime) in the gemeralist setting, when both models are trained on the aggregate of the W, E, ml, m2, h1, and h2 datasets. We evaluate them on both the in-domain and zero-shot test sets. TOTEM outperforms GPT2 in-domain, 58.3% vs. 43.8%, and by a much larger margin zcro-shot, 80% vs. 20%- TOTEM's performance across all experiments demonstrate that tokens are performant representation for imputation. We visualize codebook examples Figure 13 and imputation examples Figure
5.2 Anomaly Detection
In anomaly detection, models intake corrupted time series Xcorr SXT= and predict which times corre- spond to anomalies via a binary mask ý E 10,1) where the amount of corruption is considered known, at A% (see Figure 12h. We report Precision P (t), Recall R (t); and F1 Score (t). In the main text, we compare against the flawed baselines from prior work (sce Section 4.4) for ease of comparison. We compare against a subset of 15 "correct" basclines from Wu & Keogh| (2021) in Table 13 In both cases, find that TOTEM achieves SOTA results.
Anomaly Detection Performance Summary
Spec In Domain
40
33.3
Multitask Method
Singletask Method
30-
Aug. Wins
Ourg
13.3
10
0
0
0
0
TOTEM
GPT2
TiNet
Afran
Pate
Prs
FED
Stat
Auto
pyra
inf
RB
Log
DLin
Generalist In Domain
Generalist Zero Shot
100
60
73.3
Avg Wins (%)
80
Specialists. Figure 5 and Table 10 test TOTEM against specialist, baselines. TOTEM has the highest AvgWins at 33.3% followed by a tie between GPT2, TiNet, ATrans, ETS, and LogTr at 13.3%.
60
40
Ours
40
Ours
26.7
20
20
TOTEM
GPT2
TOTEM
Generalists. Figure 5 and Table 11 compare generalist-trained TOTEM and GPT2. On the in- domain and zero-shot regimes, TOTEM outperforms GPT2 80% to 20% and 73.3% to 26.7% respectively.
Figure 5: Anomaly Detection Results. In all cases, TOTEM has SOTA AvgWins. Vs. specialists, TOTEM has 33.3%; vs. generalists in-domain, TOTEM has 80.0%; vs. generalists zero-shot, TOTEM has 73.3%.
TOTEM's AvgWins across the specialist and gener-
alist settings demonstrate that tokens are a performant representation for anomaly detection.
5.3 Forecasting
In forecasting, models intake a time series x E ST and predict future readings E Sx where S is the sensor dimension and Tin, Tout signify the durations of the preceding and succeeding time series, respectively.
All models have a lookback window of Tin = 96, with prediction lengths Tout = {96, 192, 336, 720). Results
for basclines are from Liu al. (2023). We run GPT2 with Tin = 96 as Zhou et al. 2023 originally use inconsistent dataset-specilic lookback lengths. See Figure for a summary.
Specialists. Figure G and Table 14 show that TOTEM achieves the highest AvgWins at 28.6% followed by iTann at 26.8%. In particular, TOTEM has first-place finishes in five datasets while iTrans' first-place finishes are concentrated in only the electricity and traffic datasets.
Gencralists. Figure t and Table 15 compare the generalist-trainel TOTEM and GPT2 models. TOTEM outperforms GPT2 in both the in-domain (67.9% vs. 33.9%) and zero-shot (90.0% vs. 12.5%) regimes. TOTEM's AvgWins across both regimes show that tokens are a performant representation for forecasting.
6 Ablations
We present ablation studics: (1) testing tokens vs. patches for a fixed TOTEM architecture, (ii) testing tokens vs. patches using both transformer and MLP forecasters, (iii) a codebook size study, and (iv) study of TOTEM's zero-shot performance when trained datasets of different. sizes.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Forecasting Performance Summary
Discrete Ablation
Specialist In Domain
Discrete Tokens (TOTEM) vs. Patches (PatchTOTEM)
Specialist in Domain
Generalist in Domain
Generalist
30
28.6
26.8
Multitask Method
Zero Shot
Avg Vann (%)
Singletask
67.9
80
78.6
67.5
20
Ours
Method
10
1.8
1.8
3.6
50
50
TOTEM
GPT2
TiNet
ITrans
Patch
Cross
FED
Stat
TIDE
RLin
DLin sci
Aug. Wir
40
40
35
Generalist In Domain
Generalist Zero Shot
80
100
90
30
30
30-
Avg. Wins (%)
75
20
40
20
25
12.5
TOTEM
PatchTOTEM
TOTEM
PatchTOTEM
TOTEM
PatchTOTEM
TOTEM
GPT2
Figure 6: Forecasting Summary. In all cate- gories TOTEM has SOTA AvgWins In the specialist TOTEM has 28.6%; generalist in domain TOTEM has 67.9%; in generalist zero shot TOTEM has 90.0%.
Figure 7: Discrete Token Ablation. In all categories, the discrete token representation (TOTEM) has SOTA AvgWins over the patch representation (PatchTOTEM),
Tokens vs. Patches. The experiments in Section 5 show that the combination of discrete tokenization and TOTEM's generalist architecture achieve SOTA performance. We now fix the architecture while varying only the representation (TOTEM vs. PATEMT on forecasting task test what proportion of the performance is attributable to tokenization. We find that in all testing regimes used in the main results, TOTEM greatly outperforms PatchTOTEM, with 67.9% vs. 39.3% AvgWins in the specialist in-domain regime, 78.6% vs. 23.2% AvgWins in the generalist in-domain regime, and 67.5% vs. 35.0% AvgWins in the generalist zero-shot regime (see Figure and Table 21
Downstream Architecture Study. In Figure Table 21 we explore the effect of discrete tokens vs. patches for each of common downstream forecasting models: the transformer encoder introduced in Section 3.3 and Figure and an MLP
Discrete Ablation Varying Downstream Model (TOTEM) (PatchTOTEM)
fransformer Specialist
MLP Spec alist
[Ekambaram et al.) 2023 Das et al. 2023a Zeng et al.] 2023).
70-
66.1
The MLP has 3- layeesss ReLU activations, uses dropout with = 0.1 after the second layer, and concludes with a layernorm; this architecture modcled after similar architectures in the literature like Das al. (2023a). The patch-based MLP lakes in an uncompressed time series. We find that for both the MLP and transformer architectures, the discrete token representation outperforms the patch representation (in the transformer 67.9 67.9% to 39.3% AvgWins and MLP 66.1% to 37.5% AvgWins). This shows that TOTEM's strength in forecasting is not due to the strength the transformer forecaster, but because the choice to use discrete tokens.
60
Ang Westti 196
30.3
37.5
10:
0
Figure 8: Discrete Token vs. Patches with MLP. For both the transformer (left) and MLP (right) the discrete token repre- sentation (TOTEM) outperforms the patch respresentation (PatcHTOTEM).
Codebook Size. In Figure g we explore the effect of the codebook size K on the VQVAE's reconstruction performance. As expected, find that K increases from 32 to 256 to 512, the reconstruction performance improves. However, for downstream lasks like forecasting, is more parsimonious to model interactions between fewer codewords. Thus, we elect to use K = 256 codewords, as the reconstruction performance is similar to that of K = 512. We note that the the average generalist codebook error (see Table 21D), is substantially lower than the corresponding downstream forecasting error, demonstrating that larger proportion of error is attributable to the difficulty of the forecasting task rather than poor reconstruction. This provides evidence that time
11

---

Published in Transactions on Machine Learning Research (12/2024)
Dataset Size Study
Dataset Codebook
0.471
0.0463
0.36
0.0393
0.312
MSE
9.34
0.0209
Barz
0.0211
0.0000
0.055
d0i01
0.8129
0.32
Avg. Wins (Sc
K-256
K-512
0:446
0.252
3.15
1.2204
202222
0.0977
0.1027
0.0740
0.0673
0.0133 23
0,087a
dosss
0.10
20
12.5
0.35
2.5
K-256
K-512
Goneralist
Eloc.
Traffic
Weather
Speciallat Specialist
Figure Codebook Size Ablation. As the codebook size K increases, the reconstruction loss of the VQVAE decreasos varioty of datasets.
Figure 10: Dataset Size Study, As expected, the generalist has the highest zero-shot perfor- mance at 85.0% Avgwins, but the electricity spe- cialist outperforms the traflic specialist, even with a smaller training dataset. This confirms that dataset diversity may be more important than dataset scale for generalization.
series can have a single unified representation across multiple domains, akin to BPE in language modeling. We note that this same trend holds for the specialist models as well.
Dataset Size Study. One natural question whether TOTEM's strong generalization performance is driven by the size of the dataset or the diversity of the training samples. We study this in minimal setting by comparing the TOTEM generalist. model against two TOTEM specialists trained the two largest domain-specifee datasets: traffic (10.2M examples) and electricity (5.8M examples). As expected, the results in Figure 10 show that the TOTEM generalist significantly outperforms the two specialists in the zero-shot setting. However, the electricity specialist outperformed the traffic specialist even though the training dataset was about half the size. This provides some preliminary evidence that simply training on more data is insufficient for achieving generalization the types of data are also crucial. For related exploratory studies on generalist models, see Appendix A.7
7 Conclusion
We present TOTEM: a simple, performant tokenizer that. designed to learn domain-agnostic discrete representations for time series data, paving the way for time series foundation models. TOTEM demonstrates strong in-domain and zero-shot capabilities versus large array of both generalist and specialist basclines across dozens of domains and datasets over hundreds of seeded experiments. Overall, TOTEM unlocks domain generalization while performing at above existing SOTA levels, demonstrating the potential of adopting training and modeling techniques from language and vision modeling for time series modeling.
There are many exciting directions for future work. First, our proposed architectural design decisions were very simple, which suggests that there are many possible performant extensions. Further, while we have collated millions of existing time series, TOTEM's promising initial results suggest that scaling up the generalist training dataset size by order of magnitude more could unlock true domain- and task-agnostic generalizability. Such followup works could allow a more systematic study of the relationships between generalist data representations, token length, data size, and domain diversity.
12

---

Published in Transactions on Machine Learning Research (12/2024)
8 Broader Impact Statement
There are immediate ethical concerns that arise from our work. However, as with all data driven methods, cerlain societal consequences are important to be discussed, in this case surrounding time series modeling, A few are reported below:
Privacy Concerns. Time scries data, especially when sourced from personal devices or applications, can contain sensitive information about individuals, e.g. for health domains. In this work, no time series were sourced from personal devices.
Misuse. Time series forecast models can be misused. For instance, if model forecasts stock prices market movements, it could be exploited for insider trading or other illegal financial activities. In this work, we are focused on domains pertinent to scientific disciplines.
Economic Impacts. Automated forecasts and decisions based on time series models can significantly impact industries and labor markets both positively and negatively. For instance, if model can accurately predict weather patterns, it might affect farmers and their crop decisions, or if can forecast energy consumption, it could impact the energy sector.
9 Acknowledgments
We thank Albert Hao Li for helpful discussions and edits, Addison Hu for insights into statistical modeling, and Angela Gao and Jack Wilding for discussions surrounding applications to earthquake data.
References
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shehur, Syama. Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403. 07815, 2024.
Alexci Bacvski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli, wav2vec 2.0:
framework for self-supervised learning of speech representations. Advances neural information processing syslems, 33: 12449 12449-12460, 2020.
George Barnum, Sabera Talukder, and Yisong Yue. On the benefits of early fusion in multimodal representation learning. arXiv preprint ar arXiv:2011.07191 2020.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, 31, 2018.
Cristian Challu, Peihong Jiang, Ying Nian Wu, and Laurent Callot. Deep generative model with hierarchical latent factors for time series anomaly detection. In International Conference on Artificial Intelligence and Slatistics, pp. 1643 1654. PMLR, 2022.
Geeling Chau, Yujin An, Ahamed Raffey Iqbal, Soon-Jo Chung, Yisong Yue, and Sabera Talukder. General- inability under sensor failure: Tokenization+ transformers enable more robust latent spaces. arXiv preprint arXiv:202.11 18546, 2024a.
Geeling Chau, Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati Soedarmadji, Yisong Yue, Boris Katz, and Andrei Barbu. Population transformer: Learning population-level representations of intracranial activity. ArXig, 2024b.
Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting. arXiu preprind wwX:::3333006555 2023.
Tac-Min Choi, Ji-Su Kang, and Jong- Hwan Kim. Rdis: Random drop imputation with self saff-training for incomplete time series data. IEFE Access, 2023.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprinl (xXim22344084222 2023a.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. orXiv preprinl arXiv:2310. 10688, 2023b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arx preprint arXiv:2010. 11999, 2020.
Vijay Fkambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 459-469, 2023.
Patrick Esscr, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognilion, pp. 12873 12883, 2021.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Adeances in neural information processing systems, 32, 2019.
Philip Gage. A new algorithm for data compression. C Users Journal, 12(2):23-35 38, 1994.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob Hyndman, and Pablo Montero-Manso Monash time series forecasting archive. In Neural Information Processing Systems Truck on Dalasels and Benchmarks, 2021.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint arx:222402 (020854,, 2024.
Yangdong He and Jiabao Zhao. Temporal convolutional networks for anomaly detection in time series. In Journal of Physicss Conference Series, volume 1213, pp. 042050. TOP Publishing, 2019.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Tacsung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Ho Choi, and Jacgul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. orXiv preprint arXi::2001.04451, 2020.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-ter term time series forecasting: An investigation on linear mapping. arXiv preprint ar 1055:23005.10721, 2023.
Jessica Lin, Famonn Keogh, Wei, and Stefano Lonardi. Experiencing sax: a novel symbolic representation of time series. Data Mining and knowledge discovery, 15:107 144, 2007.
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35:5816 5828, 2022a.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference learning represendations, 2021.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 55:9881-9893, 2022b.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint 0rXiv:2310.06622 2023.
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. Multivariate time scries imputation with generative adversarial networks. Advances in neural information processing systems, 31, 2018.
Yonghong Luo, Ying Zhang, Xiangrui Cai, and Xiaojic Yuan. E2gan: End-to-end generative adversarial
network for multivariate time series imputation. In Proceedings of the 28th international joint conference on artificial intelligence, pp. 3094 3094-3100, AAAI Press Palo Alto, CA, USA, 2019.
Ahmad Wisnu Mulyadi, Eunji Jun, and Heung-Il Suk. Uncertainly-aware variatioonllrecurrenl imputation network for clinical time series. IEEE Transactions on Cybernetics, 52(())9684 -994, 2021.
Yugi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. ar preprint arXix:2211.14790 2022.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint XrXi::1609.0 08499, 2016.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXic:1905. 100:119905,1,,7, 2019.
Steven M Peterson, Satpreet Singh, Benjamin Dichter, Michael Scheid, Rajesh PN Rao, and Bingni W Brunton. Ajile12: Long-ter tergtermman naturalistic human intracranial neural recordings and pose. Scientific data, 9 (1):184, 2022.
S Rabanser, T Jamuschowski, Flunkert, D Salinas, and J Gasthaus. The effectiveness of discretization in forocasting: An empirical study on neural time series models. arxiv 2020. ar preprint ftw:2005.0011. 2020a.
Stephan Rabansor, Tim Jamuschowski, Valentin Flunkert, David Salinas, and Jan Gasthaus. The effective- ness of discretization in forecasting: An empirical study on neural time series models. ar preprinl arXiv:2005.10111, 2020b.
Alec Radford, Karthik Narasimhan, Tim Salimans, Nya Sulskever, et al. Improving language understanding by generative pre-training 2018.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 10684 10695, 2022.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Jamuschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3)1118111 2020.
Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/computtin in cardiology challenge 2012. In 2012 computing in cardiology. pp. 245 248. IEEE, 2012.
Sabera Talukder, Jennifer Sun, Matthew Leonard, Bingni W Brunton, and Yisong Yue. Deep neural imputation: A framework for recovering incomplete brain recordings. arXiv preprint arXiv:2206. 08094, 2022.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-basod diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34: 2021.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg, Unsupervised representation learning for time series with temporal neighborhood coding. arXie preprinl arXiv:2106.00754 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint orXix:2202.01381 2022.
Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34: 22419-201 22430, 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modcling for general time serics analysis. arXiv preprint arXiv:2210.02188, 2022.
Renjie Wu and Eamonn J Keogh. Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress. IEEE transactions on knowledge and data engineering, 35(3):2421 2429, 2021.
Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long, Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint (iXv:2110000642. 2021.
Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion. In International Conference on Machine Learning, pp. 2022.
PMLR,
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8980 8000--98,-- 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAT conference artificial intelligence, volume 37, pp. 11121-11128, 2023.
Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light campling-oriented mlp structures. arXiv preprint 2XXi:2207.01188 2022.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multi- variate time series forecasting. In The Eleventh International Conference on Learning Representations, 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106 11115, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pp. 17268-272888 PMLR, 2022.
Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained Im. XrXiv preprint ar Niv:2230 1939, 2023.
16

---

Published in Transactions on Machine Learning Research (12/2024)
A Appendix
A.1 Dataset.
Dataset
Sampling Rate
Number Sensors
Imputation & Forecasting Training Sets
Weather
Every min
21
Traffic
Every hour
862
Flectricity
Every hour
321
Etthl, ETTh2
Every hour
7
Ettml, ETTT2
Every 15 min
7
Anomaly Detection Training Sets
SMD (Sever Machine)
Every min
38
MSL
(Mars Rover)
Every min
55
SMAP
(Soil Moisture)
Every min
25
SWAT
(Water Treatment)
Every sec
51
PSM (Pooled Server)
Every min
25
Zero Shot Tessinn Sets for
Imputation, Forecasting & Anomaly Detection Every 0.002
Neuro2
72
Neuro5
Every 0.002 sec
106
Saugeen River Flow
Every day
1
US Birth Rate
Every day
1
Sunspot
Every day
Table 6: Dataset Information Table. Notably no sampling rate or sensor number shared between the training sets and testing sets for any task.
Task
Training Domains
Training Domains Explored (Appendix) Healthcare
Zero Shot Testing Domains Explored
Explored (Main Paper)
Imputation
Weather, Electricity,
Neuroscience (ECoG), River Flow,
Transformer Temperature
U.S. Birth Rate, Sunspot Neuroscience (ECoG), River Flow, U.S. Birth Rate, Sunspot
Anomaly Detection
Server Machiues, Mars Science Lab,
Insect Feeding, Walking Acceleration, Air Temperature, 3D Gait Phase, Heart Beat (including ECG datasets), Parkinsous Asymmelry, Till Table Beal, Internal Bleeding, Accelerometer on Whale,
Soil Moisture, Water Treatment
NASA SpaceCraft Increase Rate Tourism, Imports & Exports,
Forecasting
Traffic, Weather, Electricity, Transformer Temperature
Neuroscience (BCoG), River Flow, U.S. Birth Rate, Sunspot
Real Estate, etc. (aggregated in the W4, W3, etc. datasels), Demographies
Table 7: Domain Diversity Table. Here we list various domains that are used for each task across training and testing in both the main paper and Appendix.
A.2 Imputation. Imputer
ID CNN
Z
quantize
1D TCNN
MV.
W
D
D
K
T/F
Figure 11: Imputation Visualization. The VQVAE architecture does not change for the imputation task. The data passed in has mask applied to it so that the VQVAE solves the task of reconstruction and imputation simultancously.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Table 8: Means & Stds. for the Imputation Task. A. is the TOTEM specialist, B. is the TOTEM generalist, C. the GPT2 generalist which we setup to run in a generalist manner.
B. TOTEM Generalist Imputation (L) C. GPT2 Generalist Imputation (4)
Metricl
NSF
MAR
Metriel
MBF
MAR
12. 12.0%/00.029
0.0012
0.060 +: 0.0847
12.52
12.52%10.02
0.0010
0.045 1: 0,03066
25%
0,030
0.0006
0.060
0.00447
0.033
0.0006
0.045
0.00/06
47.0% 0.032
0.062
0.108891
37.0% 0LAST
(UXXT1
0.0136
0.0012
12.5%/0.065
0.0020
0.171 0.0032
12.5% 0.008
0.0020
0.186
0.0085
257 (0.071
0.0015
0.179 +
0.00831
25%
0.001
0.0020
0.197
0,0025
$7.426 0.000
0.0025
0.180 0.08622
37. 5%
0.0021
0.213
0.0026
50%
0.095
0.0026
0.215 0L001S2
0,132
D.0026
0.0026 0,236
TOTEM Specialist Imputation (4)
12.0%60..441
1.132-2
12.5% 15%
0.0012
0.0016
Metriel
25%
17.5% 0.04S
0.00036
0. 0,139- 0.00440
97.5%
O. 085
0.0038
0.171 0.0022
12-1% 00.028
0.0000
0.046
0.0008
MIS 0.008
0.0010
152 + 0LOG0II
N0%
0.
0.002
0.196
0,0026
= 37.3%
25%
50%
[0.051
100351
0229
0.0006
0.0000
0.0000
0.017
0.011
0.052
0.0015
0.00 0.0010
0.0008
37.0% 0.00010
12.5%Vo..640 0:00 0.0020
25% 0.641
0.0015
0015
0.126 0L005S
0.129
0.125 + 0.0067
37.0% OLISS
12.5% 029
0.033
DJ0006
0.0000
0.101 0.00106
0.095
0.3 0UXIT'
0.0306
12. 054
1,0006
0.154
9.0015
0.0008
31.1% 0.067
50%
0.074
0.0006
0.0012
181
0.0012
0.0012
3T ST.95/..112 0.122 0.0064
12.5%
25% D. 0.108
0.0049
0.009
0.201 0.0049
0.200
0.220 + 0.0044
0.00888
ST 57.5%
12.5%
25%
1.11
0.131
151
0.0012
0.0012
0.0010
0.231
0.247 0.0017
0.217
0.0015
0.0021
12. 00.49
0.0000
0.125
0.0008
0.141 50%
0.0078
0.237 0.001
O. 0.182
DJ0016
0.0012 0,286
2 2526 0.052
ST 055
0.0008
00O0
0.132
0.124
00.0000000 0006
0.0008
24
12 12.5%10.075
0012
0.175- 0000
12 12.5%
0010
0.155
0.0015
37.3% 15% 0.093
0122
0.195 0.0200
97.5%
0.077
0.0010
0.167 0.0015
12. 12.1% 0.016
0.0006
0.0010
0.080 MIS
DOTO
0. 1922
+ 0100333
50%
(LOIE
0.0022
0.170
24 25
D.0 ST.3 3% D0.18
0,00066
0 0.081
0.0006
Zero-Shan
Zero-Sha Shoo
0,0000
0
0.000
50%
0.020
10000
0.084
0.0000
12.5%/D..29 1: 0.0015
0.120 +. 0L0045
12.5%/0..04
0.0006
0.145 0.0015
12. 22.3% 10111-9-00 1.0010
0.212
0.0006
$7.3%% 0.611
25% 0.033 = 0010
0.0006
0.127 + 0.00335
0.139 0012
37 0.000
25% 0.DG1
0.001
DJ0036
0.164 0.0015
0.191 0.0082
0.138
0.0012
0.200
0.0000
50%
157-
0.0006
01217 0010
12.5% 01T
0.00000
5%
0006
0.0012
29 2525
37 3%
12. 12.5% 00040
D 0.043
0.041
400
0 0,0006
0.0010
0.0006
0.129
0.138
0.131
0.0017
0.0012
0.0006
7. 3T.5% 0.022 D.022
25% 0.019
0,029
0.0006
0.0010
0.098 1 0L0025
0.050
0.11 + 0.0022
0.0030
37 5% 0.000
25 O 024
MOX
DLINS
0.0006
0.0015
0.0015
0.107 0.0010
0.123 0.0015
0.0023 0. 45
0.0012
12 10.571 0.0070
0.109 + 0.0040
12 12.5%/0.000
0.0010
0.119 0.0015
25%
0.00
0050 50% 145
0. 0.129 0.00335
O.
0.0042
0,0052
0.147
0.0023
50%
220
0.0045
0,00157 0,182
12 5%/00663
0087
0.642 + +. 0.006S
12 5%
302
0064
0.196
0.0023
2522 D 8933
0OTO
O + 0.0847
25%
D. 41
DOTT
0.523
0.0029
57.3% 0.766
0.0055
0.692 0.0023
S7 5%%
O.
DJ0080
0.0023 0.553
0.0051
0,0700
5.5%%
1,100 0.0023
0.0017
25% 0.061
0.0000
0.168 0.0021
25%
0 084
0.0010
0.189 0.0015
3T.3% 0.069 D..66
0.0000
0.178
1. 0.0021
37.5%
0.00
0.0010
0.200 0.0021
50% 0.082 = 0.0010
0.193 + 0.0015
50%
00 128
0.0015
0.234 + 0.0021
Table 9: Imputation on PhysioNet 2012 Dataset. We report MAE where lower is better. TOTEM has the best performance in all three scenarios of percent missing.
Method 10% Missing 50% Missing 90% Missing
V-Rin BRITS RDIS
0.271
0.365
0.606
()2844
0.368
0.517
0.319
0.419
0.613
Unconditional
0.326
0.417
0.625
CSDI
0.217
0.301
0.481
TOTEM (Ours)
0.126
0.134
0.143
18

---

Published in Transactions on Machine Learning Research (12/2024)
A.3
Anomaly Detection.
Anomaly Detector
quantize
ID CNN
1D TCNN
Clean Data
MMM
E
D
D
K
4-T/F-
4 TTFFF
freeze codebook
K
Anomalous Data
16
43
91
7
tokenize
Here 25% Anomaly
Figure 12: Anomaly Detection Visualization. The VQVAE architecture does not change for the anomaly detection task. The training data passed in must be clean such that the VQVAE can learn clean representations. At test time, when anomaly data is passed in with anomaly A% (in this case 25%),, the worst A% reconstructed is set to the anomaly.
Table 10: Specialist Anomaly Detection (t). TOTEM has the highest AvgWins at 33.3% followed by a five-way tie between GPT2, TiNet, ATrans, ETS, and LogTr at 13.3%. Some prior methods use the test set as a validation set for early stopping of the lcarning algorithm, which can inflate performance. We do not adopt this practice and train TOTEM for number of iterations.
Model TOTEM GPT2
TiNet ATran Patch
ETS
FED Stat Auto Pyra Inf
Re LogTr Trans LITS DLin
62
86.89
84.61
85.49
84
84.62
83 13
885
08
84 62
85
03
65
32
76
79 00.56
82 2:445
84
85.03
29.05
06
E
94.02
$3.10
83
06
25
69
62
10
50.52
69.97
69
80.31
95.57
97.13
97.34
75.40
82.08
76.74
76.07
SMD
76.06
$4.98
81.54
82
$2.14
79 23
92.39
81.21
82
35
80
77
23
69 24
70.13
76.13
78
82.85
73.36
57
70.06
54 03
80.07
80
55
86
57-37
87
94.04
10.00
30
32
80
00
09.02
37
97
52
SMD
89
87.91
88.91
26
87
86
$3.46
83
89.54
79.61
88.34
80 23
40
00
53
86
40
02.08
92.20
90.75
91.10
90.02
90.17
N9
29
72
08
07.38
98.51
08 35
03 03.06
62
37
98
AvgWins
33.3%
13.3%
13.3%
0%
13.3%
0%
6.77%
0%
0%
0%
0%
13.3%
0%
0%
0%
19

---

Published in Transactions on Machine Learning Research (12/2024)
Table 11: Generalist Anomaly Detection (t). We train TOTEM & GPT2 on all datasets and then perform in-domain and zero-shot evaluations. A. In-Domain Performance. TOTEM outperforms GPT2: 80.0% vs. 20.0%. B. Zero-Shot Performance. TOTEM again outperforms GPT2: 73.3% vs. 26.7%.
A. In-Domain Performance
B. Zero-Shot Performance
Model TOTEM
GPT2
ITOTM
GPT2
SMD
78.61
51.29
39.02
$3.29
F1
51
93337
39.15
76.88
93.00
SMID
86.06
38.40
46.45 43
38-30
ND
47.72
Avgwine
80.0%|
20.0%
AvgWins
73.3%
26.7%
Table 12: Means & St.s. for the Anomaly Detection Task. A. is the TOTEM specialist, B. is the TOTEM generalist, C. is the GPT2 generalist which we setup to run in generalist manner.
B. TOTEM
Generalist Anomaly Detection (1) C. GPT2 Generalist Anomaly Detection (t)
Moan
Moan Stdd
SMD
0.7664 1.1066
SMD
0,7978 0.0055
MSI.
0.8329 0.0000
MSL.
0.8017 - 0.0085
SMAP
0.9951 + 0.0014
SMAP
0.6706 + 0.0011
SWAT
0.487 à 0.1005
SWAT
0.8022 = 0.0016
0.9678 0.0002
PSM
0.907 0.0779
A. TOTEM- Spocialist Anomaly Detection (t)
0.5129 0.0227
0,3902 0.0095
Moan + Std
0.5128 1,1000
0.4219 0.0047
0.8889 0.0625
0.3614 0.0011
SMD
0.7902 0.0157
0.4915 1. 0,0229
B
0,2081 :: 0.0482
MST.
0.0052
0.5217 a 0.1111
0,3812 = 000022
E, SMAP
0,9102 0.0006
SWAT
0.9427 + 0.0006
0.7207 0.1065
0,7342 0.0559
PSM
0.9587 + 0.0008
SISL
0,8296 111006
MSL
0,7848 0.0277
SMAP
0.9188 0.0020
SMAP
0,5542 0.0051
SMDD
0.7606 + 0.0237
SWAT
0.9613 1. 0.0OLD
SWAT
0,8755 1: 0.0003
MST.
PS5
O 500000 de 0.1004
PSM
0.8776 = 00.0224
SMAP
0 70888 1 0.0594
0.3900 1: 0.0602
SWAT
0.9301 + 0.0012
0.0082
0,3677 0.0498
PSM
0.9121 + 0.0001
0.0595
0.0018
SMI
0.8354 + 0.0054
O 03731 0.0540
17177 0.0426
MST
0.8732
0.7736 L 0,0381
0,3185 L: 0.0648
SMAP
110000 0A0G4
SMD
5095 0.0114
SMD
0,8744 1: 0.0029
SWAT
0.0088 + 0.0033
0.5364 0.0014
0.0130
PSM
0.9778 1 0.0012
SMAP
SMAP
SWAT
0 0,9268 0.0001
SWAT
0,9183 0.0006
PSM
0.9774 + 0.0002
PSM
0.9538 0.0025
0 0.3849- + 0.0359
N2
0.4613 - 0.0601
0.3868 1 0.00244
N5
0.8988: 0.0006
0.3802 0.0502
0.4630 = 0.0139
0.0172
0.2533 0.0498
0.2833 1.1125
0.4772
20

---

Published in Transactions on Machine Learning Research (12/2024)
Table 13: Extra Anomaly Detection (t). We present the the Adj. F1 metric the table (higher is better). then calculate the AvgWins The selection criteria for the 15 datasets from (Wu Keogh 2021 Goswami et al, 2024) was the following. First, based only on the names in (Goswami et al. 2024 it was often ambiguous which data file used. In these cases, excluded the dataset. Second, we had difficulty verifying whether the default hval//tt/tttttt ratios specified in the (Goswami et al. 2024) code matched what was reported. We found for the majority of datasets that the defaults resulted in test sets with no anomalies, when anomalies should be present. These were also excluded. From the results we could obtain, TOTEM matches or beats all other methods.
Model
TOTEM ATran MNT-0 MNT-LP
DGHL GPT2 TiNet
CIMIS44Air OMISSMAirTemeeraatum Temperature3
73.8
6.0
100.0
98.0
50.0
18.0
47.0
MP711Marker[PMM
96.7
76.0
69.0
97.0
31.0
48.0
90.0
InternalBleedings
100.0
94.0
100.0
100.0
100.0
92.0
100.0
MesoplodonDensirotti
99.4
100.0
91.0
84.0
79.0
100.0
100.0
100.0
100.0
83.0
95.0
100.0
16.0
12.0
95.0
Walking Aceleration5
100.0
99.0
100.0
100.0
91.0
87.0
93.0
insect FP2
100.0
12.0
11.0
23.0
14.0
81.0
96.0
Itstabs30791A.
100.0
100.0
100.0
100.0
100.0
100.0
100.0
park3m
67.2
15.0
56.0
64.0
20.0
63.0
93.0
221001mMB
100.0
69.0
65.0
71.0
15.0
5.0
8.0
sddb49
99.8
89.0
100.0
100.0
88.0
94.0
100.0
sel sel840m ECGI
99.5
16.0
61.0
66.0
28.0
21.0
36.0
sel840m 1.440E0CG CGG2
86.8
15.0
36.0
39.0
32.0
28.0
21.0
tiltA BB2
68.5
92.0
96.0
98.0
36.0
83.0
38.0
tiltAP B3
23.4
17.0
48.0
85.0
3.0
5.0
9.0
AvgWins
53.5%
13.3%
33.3%
53.5%
13.3%
13.3%
33.3%
Avg. Best Adj. F1
87.7
58.9
75.2
81.7
46.9
55.8
68.4
21

---

Published in Transactions on Machine Learning Research (12/2024)
A.4 Forecasting.
Table 14: Specialist Forecasting (L). TOTEM has the best AvgWins (28.6%), followed by iTrans (26.8%). Notably, TOTEM has first place finishes in 5 datasets, while i'Trans' first places are roteent in only electricity and traffic. All models have lookback n,. 96.
Model
TOTEM
GPT2
TiNet
iTrans
Patch
Cross
FED
Stat
TIDE
RLin
DLin
SCi
Metric
MAR
USS
PAE
MSE
PAE
MAE
XAE
Lasg
KAE
MAi
MAE
KAE
MSE
MAE
MAB
ISEE
417
290
m1
m0
1O
MOOIR 45
5.09
0.397
0.778 O. 15S10 100
33500 533
AvgNins 28.6%
1.8%
1.8%
26.8%
14.3%
3.6%
5.4%
0%
0%
25%
0%
0%
Table 15: Generalist Forecasting (4). Here we evaluate the generalist TOTEM and GPT2 models. A. In-Domain Performance. TOTEM outperforms GPT2: 67.9% to 33.9%. B. Zero-Shot Performance.
TOTEM outperforms GPT2: 90.0% to 12.5%.
In- Domain Performance.
Model Metric
TOTEM
GPT2
MSB
MAB
USE
MAB
2
72)
0.344 0334
B. Zero- Shot Performance
96
O0 179
MNO
Model Metric
MSB
TOTEM
MAE
MaB
GPT2
MAE
96
SUT 8
2
1.138
8
1332
530
01:02
0.2
277
0.50
O.
262
0--077
0
0
298 0-343
D.446 0.422
443
0.525
0.625
AvgWins
67.9%
33.9%
AvgVine
90.0%
12.5%
22

---

Published in Transactions on Machine Learning Research (12/2024)
Table 16: Means and Stds. for the Forecasting Specialits. A. is the TOTEM specialist, B. is the GPT2 specialist which setup to run with a consistent lookback.
A. TOTEM Specialist Forecasting (0) B. GPT2 Specialist Forecasting ing. Lookback of 96 J
Moan 4: Std
Mean # Stdl
Metrie
MBR
MAR
Metric
XSE
MAE
0.200
D 20102
0.300
D 0022
0.000S
D 0010
0
190
0.00US 0.0014
2-2
105533 O
0-003
0.000S
O0116
0.301
120 R GOOD
0004
BOOS
0.253
D 9010
3
O000
DEST
96
3.7
9-0007
0.0004
0005
Table 17: Means and Stds. for the Forecasting Generalist. A. is the TOTEM generalist, B. is the GPT2 generalist which we sctup to run in a generalist manner.
A. TOTEM
Generalist and Zhro-Shot Foreratting (1) B. GPT2- Generalist and Zero-Shot Forecusting ())
Metric
Moan 4 Sid
Metric
K2R
Moan d Std
11201 D.80
I THE
8331 0.351
WORKS
17%
8:133 + W.000
SU
033-
0. 0/.75
ART
0.0300
0.403 + 0.0035
122
10.245
002
8428 R000O
11
12
Zero-Shot
Zero-Shot
NN
NS
N5
+
086 02020
Metric [Tin- >> -Tout] Train
Test
TOTEM (Ours)
GGT Patch
DLin
Re
Inf Fed LITT
sMAPEI 24 +18 IM1-M M3-M
144
14.1
14.0
14.77
15.. 16.9115 24
sMAPEI 48 -224
IM3-M M4-M
14.6
14.6
16.2
11.7
14.8 6123 5 25. J L11 15.2
MAPE 122 -44 IM1-Y Tour.Yi
31.8
27.2
35.6
33.2
133.9141.2 51. 41138.2
NDx1001 ++168 IM4-H Elec,-H!
17.6
17.2
19.3
17.3
17.6 12.22.222222 33.9 118.118.411 19.6
Table 18: Short term forecasting results (lower is better). We randomly choose settings across varying input-to-output dimensionalites, train and test datasets, and find that TOTEM (Ours) and GPT2 outperform all other methods.
23

---

Published in Transactions on Machine Learning Research (12/2024)
Table 19: Long term vs. short term forecasting lookback and lookahead lengths. We see that long torm forecasting is far more stereotyped, and therefore casier to build gencralist models for, than short term lorecasling.
Distased
Inpott -Oulput
Long
Tarn
Foroeasting;
Ia Domeain Tosting (2022h
All Datasets (enforeed
Liu
Wi
Zhou al.
2022
p6 96, 192, 336, 720
Long
Term Forecasting: Zero Shot Testing
All Datasets
96
96, 192, 336, 720
Short Term Forecasting; In Domain Testing
18
Short T Term Forecasting: Zero Shot Testing
M4-Y M
12
M4-Q M3-Q
24
M1-Ml. MS-M
24
18
MS-O
M3-M.
M.-Y M4-
M3-M, MI-W
63 13
M3-M. M4-D
9
MS-0; M4-H
48
ML-Y. Tour.YY
12
M1-Q. Touz.
24
Tour-V
168
Yo=Yynrl, Q-Quarterly M-Monthly, W-Wookly, D=Duily, H-Hoorly, O=Other
Table 20: 96 and 512 Lookback Lengths. We compare various forecasters with lookback length of 96 and 512, across all lookback lengths and datasets TOTEM has the most Avgwins at 58.3% followed by GPT2 at 8.3%.
Tin 512 Model
[TOTEM (Ours)
GPT2
MNT
Patch
N-Beats
Run By
TOTEM (Ours)
GPT2
MNT
Patch
MNT
Dataset
Metric
MSE, MAE
MSE, MAE
ISE, MAEIMSE, MAEJMSE, MAE
W
96
0.147, 0.196
0.162, 0.212
0.154, O. 0,200 149, O. 0.198J0.152 0,210
W
192
0.195, 0.242
0.204, 0.248
N/A, N/A 10.194, 0.241 N/A, N/A
W
336
0.248, 0.283
0.254, 0.286
N/A, N/A 10.245, 0.282| N/A, N/A
720
0.314, 0.330
0.326, 0.337
0.315, 0.336/0.314, 0.334)0.331, 0.359
96
0,135, 0,231
0.139, 0.238
0.138, 0.242 129, 0.222/0.131 0.228 N/A, N/A 14. 0.240 N/A, N/A
192
0.151, 0.245
0.13,, 0.251
E
336
0,168, 0.265
0.169,
0.266
N/A, N/A 16.163, 0.259 N/A, N/A
720
0.200.
0.292
0.206, 0.297
0.2111 0.305/0.191 0.29010.208, 0.298
T
96
0.369.
0.241
0.388, 0.282
0.391, O. 0.28200.360 0.249 375.. 0.259
192
0.383, 0.242
0.407, 0.290
N/A, N/A (0.379, 0.256 N/A, N/A
T
336
0.397, 0.248
0.412, 0.294
N/A, N/A J0.392, 0.264 N/A, N/A
720
0.446. 0.275
0.450, 0.312
0.450, 0.310/0.433 0.286|0.500 508, 0.335
Tin=96
96
Model
[TOTEM (Ours)
GPT2
MNT
Patch
N- Beats
Run By
TOTEM (Ours)
TOTEM (Ours)
N/A Trans
N/A
96
0.165. 0.208
0.184, 0.224
N/A, N/A UU.177, 0.218
N/A. N/A
W
192
0.207, 0.250
0.231, 0.263
N/A, N/A 10.225, 0.259
N/A, N/A
W
336
0.257, 0.291
0.285, 0.302
N/A, N/A (0.278, 0.297
N/A, N/A
720
0.326, 0.340
0.362, 0.351
N/A, N/A (0.354, 0.348
N/A, N/A
96
0.178, 0.263
0.186, 0.272
N/A, N/A JU. 195, 0.285
N/A, N/A
192
0.187, 0.272
0.190, 0.278
N/A, N/A J0. 199, 0.289 N/A, N/A
336
0.199. 0.285
0.204, 0.291
N/A, N/A 10.215, 0.305 N/A, N/A
720
0.236. 0.318
0.245, 0.324
N/A, N/A 10.256, 0.337 N/A, N/A
T
96
0.523, 0.303
0.471, 0.311
N/A, N/A 00.344, 0.359 N/A, N/A
192
0.530. 0.303
0.479, 0.312
N/A, N/A |0.540, 0.354 N/A, N/A
T
336
0.549, 0.311
0.490, 0.317
N/A, N/A (0.511, 0.358 N/A, N/A
T
720
0.598, 0.331
0.524, 0.336
N/A, N/A 00.886, 0.375 N/A, N/A
MAvgWine|
58.3%
8.3%
0%
35.4%
0%
24

---

Published in Transactions on Machine Learning Research (12/2024)
A.5 Ablation Details.
Table 21: Ablations (4). Across the Tokens vs. Time (TvT) experiments tokens out perform time, (A) spocialist: 67.9% to 39.3%, (B) in-domain generalist: 78.6% to 23.2% and (C) zero-shot generalist: 67.5% to 35%. (D) As the codebook size K increases the VQVAE reconstruction performance improves.
A. TvT Specialist
B. TVT In- Domain Generalist
TOTEM TOTEM Metric
Model
Model
TOTEM
TOTEM
MSE
MAE
MSE
MAR
Metric
MSE
MAR
MSE
NAE
166 0 20
209
0 0.257
0 291
206
0
285
0.325
96
O. 175
0.262
D6
264
0.207
0.18
0.236 0.31
021
0.325
0,223
96
0.523
0-303 02524
0310
O0 507
055
H
0.39
5.500
F
0.51
202
720
0.3
220
0.309
0.612
96
326
m
J22
LAr
0-102
0.382
S
D.
95
O.
0.176
25310-177
M
m2
22
220
0.5
D.
0551
96
0 293
0.338
0.204 0.338
34 0.3000
0.346
2
0.00
0.aby
Avgusna
67.9%
39.3%
AvgMins
78.6%
23.2%
C.T Shot Generalist
Model
TOTEM TOTEM
Met Metric
MSE
MSE
2
0.470
D. Codebook Size Ablations
N6
98
0.483
0.484
0 0.483
Codebook Size K
0.488
0.201
120
MSE
R
96
0.578
0.628
All
10.0451
0.0192
0.0184
0.0312 0 0.0120
0.0101
0.506
0.0463 0.0209
0.0152
96
0.805
W
0.03393 0.011
0.0128
m
330
80809
MAE
All
0. 0..1-60 0. 0937
0.0913
0.362
0.1204
0.0685
D 8-538
0. 0.1520 0.1027
0.0878
0.717
0.2122 0.0673
0.0607
AvgVins
67.5% %
35.0%
AvgWins
0%
0%
100%
E. TVT MLP Specialist
Model
TOTEM TimcTOTEM
96 Metric
NSE
MAE
NSE
M6E
180
224
0.208
0273
0,330 0.342
0.355
0330
268
220
0.2
0.323
8:0
0:313
96
0.539
0.330
556
R2
0-552
50L
96
12
8-138
0.533
Avgäine
66.1% 1%
37.5%
25

---

Published in Transactions on Machine Learning Research (12/2024)
Table 22: Mean & Stds. for the PatchTOTEM Ablation. Left is the specialist, right is the generalist.
Generalist In Domain & Zero Shot Forecasting
Mean + Std
Metric
MSE
MAP
0.0012 0000
0.0006
0.201
0.0000
(126Z 010000
(1299
(1,0006
720
0.3
0.0010
0.341
0.0006
96
08183
0.0012
0.267 = 0.0012
0.189
0.0000
0,0000
Specialist. Forecasting
T20
0.242
0.0006
8.325
0.0006
Mean
+
Stdl MAB
96
0.517
0.00001
0,0029
MSE
0.229
0.0030
0.206
0,000
96
164
000000
0 209
0.0023
0.0006
330
F20
0.002
0.0036
1010
0.326
D. 15
00
179
0.4
0.0006
E
192
96
0.428
0.438
1.409
0 0000
0002
0.437
0.420
0.422
0.0042
O 0,0000
0.00 10
0 260
0.0000
T20
(100333
0.289
0.0061
0
55
+ 0020
(100040
0.325
0.0036
EF
200
0.001
96
528
0.0008
(LOG0G
10 10.310
349
00.000000000 0000
0.0099
136
720
0.521
1.358
0.013
0.1199
1,0026
0.1122
0.365
0.0852
96
0006
0.0006
0.398
=
102
15
0
0121 01010
0.000610-20 355
0.000G
720
336
= 1000
110200
0.208
0.510
0.0006
0.0017
m
0.0006
0.0015
-
0.0006
020000
2N
96
0
303
010009
100
02222 1000
0.366
0.0006
mZ
0.001010
0,0020
0.0006
0.531
0.0026
0.0000 0,330
0.002
0.411
0.0044
2N
96
1.127
0100
0.00
0.0006
0.0010
0.00100
100 195
0.0006
010006
1.00 0.428
0.0013
220
0.700
0.0010
0.522
0.001
NS
1S4
1001
0.0000
0.0006
0.00211
0.0010
0000
0003 0.133
189
0.0032
0000
01000
0.015
0. 1,550
R
100000
0.628
0.0017
0.613
0.00
720
0
0.596
0.00 0,0020
96
0.
0.820
0 0.009
0.751 0.0076
33G
102
1000
8:513
0L261
0 00
(1001
720
01131
(1 (1444
0.00151
0.481 0.0010
0.499 0000
00102
0.0025
0.63T
0,0006
Table 23: Mean and Stds. for the Codebook Ablation (4)
Mean + Std MAE
K
MSE
0.0040 0.0000
256
0.0007 0.00555.....0000...........
512
32
1..093 0.0005
+ 0.0064
0.00011 0.00004
0.0011
0.0032
32
0.000000..00
0.0001
0.0016
- 0.0001 10000000000000000000000000000100 1.02577-000000
0.0029
0.000 0.0887 0.000 0.0004
0.0000
32
10.0310 + 0.000
0.0008
[
256 0.000000000000000000000000000000
0.0003 0.0749
0.0007
0.000 00..00055000000000
512
0.0044
26

---

Published in Transactions on Machine Learning Research (12/2024)
A.6
Statistical Significance.
Despite the fact that much prior and concurrent work only reports results on 1 seed Wu et al. (2022 Goswami et al. (2024) and Zhou et al. (2023) (except for Table 15), perform statistical analysis generalist results.
Our results are statistically significant with 3 seeds using an exact one-sided permutation test. The exact one-sided permutation test repeatedly randomly permutes the reported metrics (e.g., MSE) between two competing methods (e.g., TOTEM vs. GPT2) and generates a distribution of all the possible metric assignments. This is an appropriate test because it returns a p-value indicating how rare it is observe our reported results relative to all the permuted outcomes. Importantly, this test is non-parametrie, in low-sample regime. The reason we expect this statistic to return p<-0.05, even with 3 seeds, is because the training algorithms in this setting are actually quite stable (unlike other areas of ML such as deep RL Henderson et al. 2018)).
it is valid
We perform this analysis on the generalist models for each experimental trial/metris (e.g., for the MSE metric of the Weather dataset in the forecasting task), for all tasks (where we compare TOTEM vs. GPT2), and for the token vs. patch analysis (where we compare TOTEM vs. PatchTOTEM).
The following tables report the proportion of trials where the p-value is <= 0.05, i.c., where TOTEM statistically significantly outperforms the competing method (GPT2 24 or PatchTOTEM 25 While the results in the main paper double-count the ties between method (e.g.. if TOTEM and GPT2 tied, a win was counted for both) to prove the strength of TOTEM, the tables below compare the percentage of experiments in which TOTEM strictly out perforrs the baseline (we also provide the reported win percentage from the main paper for case of comparison). It is clear that even the statistical sctting, TOTEM still wins more than baselines.
Table 24: TOTEM vs. GPT2 Generalist Statistical Significance. Here we compare the TOTEM and GPT2 generalists and calculate the proportion of trials where the p-value is <= 0.05, i.e., where TOTEM statistically significantly outperforms GPT2 (right column). These results are in line with those reported in the main paper.
AvgWins
Original with ties Originil//Saaaaaaa Stafistical
(as in 5 6)
no ties
no ties
Imputation In Domain Imputation Zero Shot Anomaly Detection In Domain
58.3%
56.3%
56.3%
80.0%
80.0%
80.0%
80.0%
80.0%
66.6%
Anomaly Detection Zero Shot
73.3%
73.3%
73.3%
Forecasting In Domain Forecasting Zero Shot
67.9%
66.1%
62.5%
90.0%
87.5%
82.5%
Table 25: Tokens vs. Patches Generalist Statistical Significance. Here we compare the TOTEM and PatchTOTEM generalists and calculate the proportion of trials where the p-value is <= 0.05, ie., where TOTEM statistically significantly outperforms PatchTOTEM (right column). These results are in line with those reported in the main paper.
AvgWins
Original with Original Statistica
ties (as in 7
no ties
no ties
In Domain Zero Shot
78.6% 67.5%
76.9% 65.0%
66.1%
60.0%
A.7 Further Exploration Details.
Generalist Codebooks. To further explore the capabilities of a generalist codebook data representation we train models that utilize general codebook but datasct-specific transformer forccasters, i.c., a TOTEM VQVAE trained on multiple domains with forecaster trained only on electricity, Table 26 We compare these mixed models to generalist. and specialist models trained the same domains. All models the same
27

---

Published in Transactions on Machine Learning Research (12/2024)
codebook hyperparameters (number of codewords K 256, compression factor F = 4, code dimensionality D = D as well as the forecaster transformer architecture to ensure a fair comparison.
Since we are evaluating specialists, mixed-models, and a generalist on in-domain test data, one might expect the TOTEM specialists to significantly outperform all models in all domains. Surprisingly, this intuition not correct. We find that the fully-generalist model (right Table 26) significantly outperforms the mixed-models (middle Table 26 in traffic (T) and electricity (E). This performance puzzling until considering the training sizes.
The largest training set across domains belongs to traffic (T) at 10.2M training examples. In dataset T, the fully generalist models achieves 100% AvgWins. The second-largest training set belongs to electricity (E) at 5.8M Iraining examples, with 75% AvgWins for the fully-generalistt model. Unfortunately, there is a sharp drop off in training set sizes, with the rest the data domains collectively comprising 1.6M training examples. These results cvoke questions. For instance: does training on the smaller datascts act like form of regularization? How does in-domain generalist performance scale with dataset size? We leave these exciting directions for future work. The generalist codebook's 's performance across datasets highlights the potential of unified, discrete, token representations for in-domain evaluations.
Table 26: Specialist models, mixed models, and generalist models.
Codebook
Specialist Specialist
Generalist Specialist
Generalist Generalist XSE
Forecaster
Metric
MSE
MAE
MSE
MAE
MAE
0.165
0.208
0.164...4- 0.208
0. 0.172 0.216
W
0.2
207 0.250
0.208
0.231
0.257
0.501
0.258
0.290
720
0.326
0.340
0.329
0.338
0.34
0.342
96
0.178
0.26310.178
0.26310.17
0.179 0.264
0.187
0.212
(1
0.181
0.267
336
0.199
0.585
0.109
0.85
0.196 0.283
720
0.236
0318
0.320 0.314
96
0.523
0.303
0.521
30301005507
284
H
192
0.300
02
0.530
0.303
336
0.549
0.3
0.559
0. 292
720
0.598
033
0.605
0351
0.309
0.320
0.344.44000
0.32
0352
0.884
m1
0.399
0.38
380
0.406 0.402
0400 440
01
0.471
0.400
0.176
0.27
m2
0.302
0.302
1310
0.426 0.410
0.3 0.348
0.410
0.3
0.410
0.345
031
96
0.380 0.394
0.395
h1
0.434
0.
0.490 0.459
0.4
0.40
0.507
0.463
0.539
D.5
01566
0.512
96
0.293 0.333.338111
h2
0.390
0.37
0.39
0.431 0.42
0.610
0.567
0.567
0,557
AvgWins
57.1%
35.7%
30.4%
28

---

Published in Transactions on Machine Learning Research (12/2024)
Table 27: Zero Shot Vignette: Training Size & Diversity
Model
TOTEM Generalist ALL
TOTEM Specialist Traffic
TOTEM Specialist
Train Domain
Electricity
Sensor Num (S)
862
321
Raw Length (T)
17544
26304
Train Size
17.6M
10.2M
Metric
MSE
MAE
MSE
MAE
MSE
MAE
96
1. 1.138 0.777
1.194
0.798
193
0.802
N2
1449
0.785
121
808
1.022 0-429
[-00
804
1,055
0.784
96
0.483 0.484
0.515
0.505
0.459
49490 1.9
N5
0-339
0-01
0-468
0-453
0.92
0.2
0.582
1.71
0 0-07
0.579
R
1.242
1.237 6266
0.028
720
1.182
0.604
198
0.6
236
0.633
96
0.805
0.
739
0
749
0.820
0.756
0.836
0. 152
(1809
0-748
0.741
0.896
0.51
0.446 0.482
0
0.508
460
0.487
0.462 0.491
0
0-025
0.270
0-208
0.717 0.625
AvgWins
85.0%
2.5%
12.5%
Table 28: Means and Stds. Mixed Models Forecasting (L)
Mean
+ Std
Metric
MSF
MAE
0.001010
208 0.001
258
0.00120 700
0.0015
0.002 0.33
0.0015
96
203
0.000
301
192
0 530
0.00230
00012
336
0050
0.317
003610
0X022
96
0,00000 100.23555
0,0010
m2
0,000 10
1OOTO
01003
0.012
00032
96
294
0.001000..000
0.0012
391
331
0.610
0.0089
0.0075
29

---

Published in Transactions on Machine Learning Research (12/2024)
Table 29: Mean and Stds.
Traffic Only Specialist Zero-Shot Performance (4)
Mean + Stdd
Metric
MSE
0,006
96
10.515
0.002600
505
0,0012
01503
0.005
0-514
0.002X
0.500
0.0064
0.0052
96
0.002
635
0.0019
192
273
0000
0
612
749
267
0.820
0.004
720
0.919
0.006
0.00ST
96
170
0.001210
508 0.0012
0.795
Table 30: Means and stds.
Electricity Only Specialist Zero-Shot Performance (1)
Mean +
Std MAE
Metric
MSE
96
0059 802
N2
832
N5
0.001
0.508
0.00440 0,32
0.503
0.0010
0.0006
0.006600.77 579
0.002
R
0.022
0.005 0.66
0.0070
10 756
0.00
0.00
0,0022
0.741
0.001S
0.00
0. 790
0.0020
0.0010
330
0.000
2000
243
010011
720
0.764
0.0046
0.641
11 0.0014
30

---

Published in Transactions on Machine Learning Research (12/2024)
A.8 Generalist Training Time Comparisons.
TOTEM (Ours) Parains
TOTEM (Ours) Training
GPT2 Params
GPT2 Training
Time on A100
Time on 1 A100
Imputation
345,000
1.5 hours
-60,700,000
Several days
Anomaly Detection
345,000
1.5 hours
-4,,500,000
Several days
Forecasting
-005,000 for VQVAE,
-.5 hours for VQVAE, day for transformer
-8,0000,000
Several days
-1,600,000 for downstream
transformer
Table 31: Comparison of Parameters and Training Time between TOTEM (Ours) and GPT2 generalist models.
A.9
Codebook Visualizations.
Codowarda Latert Scace
Trafic: Cocowards da Latent Space
Bictrictty Codewarda Latart Space
Weather- Cocowends sazart Space
10
20
26
66
20
26
80
20
26
20
25
Space
Codoworss Space
Bectriity:
Weather
Space
Figure 13: TOTEM Codebooks. We visualize all 256 codes for the generalist (All), and three specialists (Traffic, Electricity, and Weather). The top row visualizes codes in the latent space, the bottom row visualizes codes in the decoded time space. We additionally highlight codeword pairs matched via low MSE between All-Traffie, All-Electricity, and All- Ala-heether in the bottom row.
31

---

Published in Transactions on Machine Learning Research (12/2024)
A.10 TOTEM Examples.
Example Forecasts
La
Trath
Prediction
Example Imputations
Inpot
Ground
Truth
Figure 14: TOTEM Examples. In the top row we visualize four weather forecasts for Tin-96 and Tout=96. In the bottom row we visualize four ETTm2 imputations. In all cases the model input grey, the predictions are in blue, and the ground truth in green.
A.11 Architecture Details.
VQVAE. For imputation, anomaly detection, and forecasting the VQVAE's number of residual layers 2, residual hidden size 64, and block hidden size 128 for all datasets. Each residual block has 2 non-causal, non-dilated ID convolutional layers. The residual blocks are paired with additional non-causal, non-dilated 1D convolutional layers, where the number of additional layers is determined by the desired compression factor. See Table 32 for more hyperparameter details.
Table 32: VQVAE Hyperparameters (A) Imputation generalist (All) and specialists. (B) Anomaly detection generalist (All) and specialists. The anomaly %s for all of the zero shot datasets are 2%. (C) Forecasting generalist (All) and specialists.
A. Imputation.
B. Anomaly Detection.
Dataset
LR
Iter.
BS
# CW Dim. CFF
Dataset
LR]
Iter.
BS #ff CWCCW Dim,CH
Anomaly
All
le-
12000001
512
Elec.
Ie- 15000
8 8192
5
All
1e-3112000-1096
1021
64
Varies by test
Weather
1-3 15000
X192
212
SMD
le-
60000 4096
1024
64
0.5
m
Le- 15000
8 8192
5
l-
15000
4096
1024
64
m2
le-3 15000
8 8T92
512
le-
60000 1096
102
64
15000
15000
8 8192
97
512
le-3
15000
15000 4096
4096
1024
1024
64
64
C. Forecasting.
Dataset
LR
Iter.
BS |
CWICW Dim, CFF
All
le-31
15000 40961
256
Elec
1 5000 1096
5-5
Weather
l-3 5000
4096
550
Trafic
16-3 5500
5:0
ml
Te-3 5000
5:0
le-3
5000
4096
250
le-
15000
4096
250
h2
1e-3
15000 40961
256
64
Downstream Forecaster. The rownstream forecaster has two components the transformer encoder that intakes codes and outputs a normalized time forecast, and the feedforward neural network that takes time and outputs predictions for the forccast's mcan and standard deviation. The downstream forecaster is a transformer encoder with model dimension 64, hidden dimension 256, number of heads 4, number of layers = 4. The transformer encoder applies sin cos positional embedding along the time dimension and
32

---

Published in Transactions on Machine Learning Research (12/2024)
applies its attention mechanism to each sensor independently. There is single linear layer applied after the transformer encoder output. The feedforward neural network takes in the input time steps, and predicts the future's mean and standard deviation.
A.12 Training Details.
In imputation, anomaly detection, and forecasting the VQVAE is trained with a learning rate of 0.001 using the Adam optimizer, embedding dimension of 64, commitment cost of 0.25, and compression factor of 4; see Table 32 for more hyperparameters. The codewords are uniformly randomly initialized over [ k where K the number of codewords and D is the latent dimension. In all tasks there is a global normalization, and local normalization Kim et al. (2021); both are standard throughout prior work. In imputation we only leverage global normalization, anomaly detection and forecasting we utilize both global and local normalization. In anomaly detection evaluate the models we run, TOTEM and GPT2, with both local normalized data and non-local normalized data for each method and report whichever schema leads to the best performance. In forecasting the downstream model is transformer encoder with 4 layers and 4 attention heads and a feed-forward hidden dimension of 256. We train using Adam with a base learning rate of 0.0001 and a one cycle learning rate scheduler in accordance with Nic et al. (0022 on A100s.
33


Reviews:
[{"review_id": "Je5w8seLKY", "reviewer": "Reviewer_FqZp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is explained and written very clearly and that is the main strength. It builds upon simple concepts and proposes a tokenised model as opposed to a patch-based model\n2. Experiments and ablation studies are another big strength for the paper. They are extensive.\n\nWeaknesses:\n1. While the experiments are extensive, the domains can be more diverse. E.g: how does the TOTEM that is trained on an EEG dataset translate to ECG and vice versa? These are important questions to be answered.\n2. What other architectures were considered?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is well-researched and clearly written supported by elaborate experiments. The paper is novel with a simple idea of using VQVAE based architecture to build a generalized time-series with baselines and datasets. \n\nThere are some concerns regarding the choices of architectures and decisions about why VQ-VAE was chosen but overall, it is a good paper and definitely worthy of TMLR.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is well-written for the most part and the following changes are required. \n\n1. Explanation and motivation behind the choice of VQ-VAE and the reason for having to use Straight-through estimator.\n2. More experiments in diverse domains and how these results compare and contrast. Example time series: ECG time series, communication channels, motion activity data", "broader_impact_concerns": "Not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4dxtn3KrzH", "reviewer": "Reviewer_JMhz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: Exploring Time Series Analysis Through Tokenization is Interesting: The application of tokenization, a technique commonly used in natural language processing, to time series analysis is a creative and innovative approach. \nS2: Rigorous Evaluation of the Method: The proposed method is not only conceptually interesting but also rigorously tested across various real-world scenarios\n\n\nW1: Risk of Losing Fine-Grained Temporal Information: While tokenization is a novel and powerful technique, it introduces the risk of losing important fine-grained temporal details. Time series data often contain subtle temporal dependencies and trends that may not be fully captured when discretized into tokens. \nw2: Lack of Comparison of Training Time with Competitors: Although the model's performance is evaluated in terms of accuracy and robustness, the paper does not provide a detailed comparison of training time against competing methods. Since deep learning models, especially transformer-based ones, can be computationally expensive and time-consuming to train, it would be important to understand how TOTEM performs in terms of computational efficiency.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper \"TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis\" presents TOTEM, a framework specifically designed to tackle general time series analysis by transforming time series data into tokenized embeddings. The proposed method is rigorously evaluated across three different real-world scenarios.\n\nI have done my best to provide a thorough and objective review of the paper. However, as I am not an expert in this specific field, my feedback may appear somewhat superficial or lack the depth that a more experienced reviewer could provide.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.The three tasks are only introduced in Section 3.2. To make the paper more accessible to non-expert readers, I suggest briefly introducing these tasks in the introduction.\n2. Training time comparison\n3. The tables are somewhat difficult to read; I recommend improving their visual presentation for better clarity.", "broader_impact_concerns": "na", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "74kVV8Qgdx", "reviewer": "Reviewer_nhYG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n- The paper's experimental breadth is staggering, covering a wide variety of baselines, problems, settings, and even a few ablations.\n- The basic premise of the work is compelling: the design of a generic time series model that requires minimal fine-tuning is a great premise.\n- The visuals in Figure 1 and 4 helps to elucidate aspects of the design of the method.\n\n__Weaknesses__\n- While the breadth of experimentation is clearly a strength, the care of reporting results, depth, and experimental rigor are lacking. The paper reports \"For all experiments & models in the main paper, we run three seeds and report the mean; standard deviations in the Appendix.\" Consequently, the experimental data present weak evidence, despite the breadth of baselines and domains. As a result, I find it hard to draw strong conclusions. I encourage the authors to perform a more careful statistical comparison, or at the very least increase the number of seeds used on a smaller number of domains in order to arrive at some statistically meaningful conclusions. As an example, the question of whether tokens or patches are more useful is an interesting one, but the evidence presented is not strong enough to support the claim \"TOTEM’s performance demonstrates that tokens, when compared to patches, lead to better performance\" simply because there is no indication of how meaningful these data are. Regarding care in reporting the results, the tables are extremely difficult to read. While this is subjective, I would encourage using more white space, and presenting these data in a different way. The imputation performance summaries are relatively easy to process, but B. C. and D. of Table 2 are quite difficult to parse.\n\n\nUpdate: Following the response from the authors, I believe the proposed statistical tests and corresponding strength of the evidence in the paper to resolve my concern above, and I am updating my response to the question \"Claims and Evidence\" in light of that.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a generalist method for time series analysis called Tokenized Time Series Embeddings (TOTEM). The approach makes use of the VQ-VAE at its core, which, in addition to other design aspects, makes TOTEM both a _generalist_ model in the sense that it can be trained on many different domains, and a specialist model in the sense that it can also be trained on a single domain. The bulk of the paper is then spent evaluating TOTEM on a large variety of tasks and settings, including (i) imputation, (ii) anomaly detection, and (iii) forecasting. Beyond these three problems, however, the paper also studies other aspects of the design of TOTEM, such as whether tokens or patches are more useful (Figure 7), or how the side of the codebook impacts performance (Figure 9). Collectively, these ablations and the three analyses on the core problems mentioned above comprise the bulk of the experimental work. The stated findings from the experiments is that TOTEM regularly matches (or outperforms) state-of-the-art.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "__Writing Suggestions__\n\nOverall:\n- My primary suggestion is to focus on improving the strength of the evidence in your experimental data. This does not require any further baselines or settings, as the paper is already extremely broad in that sense. Rather, I believe that some statistical analysis of the data would add a great deal of support to your main claims, so long as the claims made also match the strength of the findings. At the moment, it is very difficult to read the main tables, or to interpret the data, or to identify the strength of the evidence. In its current form the paper's claims do not match the strength of the evidence provided.\n- In general, if you are attributing a fact or claim from another piece of work without stating the authors' name(s) directly, use the parenthetical citation: for example, \"on a single time series domain Zhou et al. (2023); Wu et al. (2022a); Nie et al. (2022), Figure 1A\" should be \"on a single time series domain (Zhou et al. 2023; Wu et al. 2022a; Nie et al. 2022), as visualized in Figure 1A.\n\nOther low-level writing suggestions that had less impact on my evaluation:\n- I don't quite understand the use of color in the annotation of Figure 1 (why is \"in\" purple? It appears to be referencing some aspect of the purple color in the visual itself, but I can't decipher it.)\n- I believe aspects of the related work discussion could be made more specific, and more useful. The initial blocks of citations (as in \" ranging from statistical methods Winters (1960); Holt (1957); Anderson (1976); Hyndman & Athanasopoulos (2018); Taylor & Letham (2018) to multilayer perceptrons (MLPs) Zeng et al. (2023); Li et al. (2023); Das et al. (2023a); Challu et al. (2023); Chen et al. (2023); Zhang et al. (2022); Oreshkin et al. (2019)...\") is not all that helpful to the reader. Providing slightly more insight or discussion on these approaches would be valuable, rather than summarizing a vast body of literature with statements like \"Many models are hybrid solutions that blend aforementioned approaches.\" That said, the discussion that follows this paragraph is quite valuable.\n- I would move Section 3.2 before 3.1. It is easier to digest the design details when they are contextualized by the problem framing. I believe this would also benefit from moving the \"Operating Across the Time Dimension. \" paragraph to Section 3.2.\n- \"any dimension E, S, or T, Figure 2 left.\" --> \"any dimension E, S, or T, visualized in Figure 2.\"\n- I had a hard time parsing this: \"this two-step represent then solve-task process \"", "broader_impact_concerns": "None of note.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by three expert reviewers. The reviewers initially raised concerns about the  diversity of the explored domains, the information loss due to discretization and the significance of the reported results. These concerns were addressed by the authors in their response, and two reviewers recommended acceptance of the paper, while one reviewer recommended weak acceptance. I thus think that the paper is now ready for publication.

To further strengthen the paper, I would encourage the authors to discuss the following:
- Language models rely on the structure of language which is shared across all corpora on which these models are trained. However, time series data collected from different domains can be very diverse with different patterns occurring in the different series. Why are those models expected to learn patterns useful for other time series domains?
- Would models that do not produce discrete data representations, but are trained on many data domains simultaneously also achieve strong zero-shot performance and why? 

I also request the authors to fix the issue with the parenthetical citations raised by Reviewer nhYG.
Decision: Accept (poster)