Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Authors: Luca Masserano
Abdul Fatir Ansari
Boran Han
Xiyuan Zhang
Christos Faloutsos
Michael W. Mahoney
Andrew Gordon Wilson
Youngsuk Park
Syama Sundar Rangapuram
Danielle C. Maddix
Bernie Wang
Venue: ICML 2025 poster (2025)
Keywords: time series
forecasting
foundation models
wavelets
tokenization
frequency
pretrained model
nonstationary
Primary area: deep_learning->sequential_models_time_series
TL;DR: We develop a wavelet-based tokenizer and pretrain a foundation model for time series forecasting on time-localized frequencies. Results show excellent generalization performance and superior ability to capture complex patterns of practical relevance.

Abstract: How to best develop foundational models for time series forecasting remains an important open question. Tokenization is a crucial consideration in this effort: what is an effective discrete vocabulary for a real-valued sequential input? To address this question, we develop WaveToken, a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon. By decomposing coarse and fine structures in the inputs, wavelets provide an eloquent and compact language for time series forecasting that simplifies learning. Empirical results on a comprehensive benchmark, including 42 datasets for both in-domain and zero-shot settings, show that WaveToken: i) performs on par or better than recently proposed foundation models for forecasting while using a much smaller vocabulary (1024 tokens), and is competitive with modern deep learning models trained specifically on each dataset; ii) exhibits superior generalization capabilities, achieving the best average rank across all datasets for three complementary metrics; and iii) easily captures complex temporal patterns of practical relevance that are challenging for other recent pre-trained models, including trends, sparse spikes, and non-stationary time series with varying frequencies evolving over time.

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Luca Masserano 1* Abdul Fatir A Boran Han Xiyuan Zhang? Christos Faloutsos 311
Michael W. Mahoney Andrew Gordon Wilson 34f Youngsuk Park
Syama Rangapuram"
Danielle C. Maddix
Yuyang Wang?
Abstract
1. Introduction
How to best develop foundational models for time series forecasting remains important open ques- tion. Tokenization is crucial consideration in this effort: what is an effective discrete vocabu- lary for a real-valued sequential input? To address this question, we develop WaveToken, wavelet- based tokenizer that allows models to learn com- plex representations directly in the space of time- localized frequencies. Our method first scales and decomposes the input time series, then thresh- olds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to fore- cast coefficients for the forecast horizon. By decomposing coarse and fine structures in the inputs, wavelets provide an eloquent and com- pact language for time series forecasting that simplifies learning. Empirical results com- prehensive benchmark, including 42 datasets for both in-domain and zero-shot settings, show that TaveToken: i) performs on par or better than recently proposed foundation models for fore- casting while using much smaller vocabulary (1024 tokens), and is competitive with modern deep learning models trained specifically each dataset; ii) exhibits superior generalization capa- bilities, achieving the best average rank across all datasets for three complementary metrics; and iii) easily captures complex temporal patterns of prac- tical relevance that are challenging for other re- cent pre-trained models, including trends, sparse spikes, and non-stationary time series with vary- ing frequencies evolving over time.
Time series forecasting is integral to decision-making pro- cesses in many domains, including finance, healthcare, sup- ply chain optimization, and climate science. Over the last decade, the field has seen a gradual but steady adoption of "global" deep learning models in lieu of traditional "lo- cal" statistical models (Lara-Beníter et al., 2021; Benidis et al., 2022). Recently, the success of large language models (LLMMs) on natural language and vision applications has spurred an increasing interest for developing similar "foun- dation models" in other fields (for example, Subramanian et al. (2024); Golling et al. (2024); Ansari et al. (2024); Das et al. (2023)). These efforts aim at building general-purpoot machines able to learn complex representations from vast amounts of data and generalize based on the premise that LLMs are general pattern rec- ognizers (Mirchandani et al., 2023). In other words, if problem can be reduced to that of modeling an arbitrary sequence of tokens defined on a discrete vocabulary, then an autoregressive transformer might be capable of learning non- trivial relationships via next-token prediction, regardless of the inputs representing text or not. The sequential nature of time series forecasting aligns seamlessly with this per- spective, which is why several recent works have proposed adapting transformer-
wide variety of tasks,
architectures into foundation models for time series (see Appendix B for a review).
Tokenization is a crucial, albeit still understudied (Dagan et al., 2024), component of LLMs, as it provides the vocab- ulary on which token streams are defined and the autore- gressive structure is learned. While transformers can, in principle, learn arbitrary dependencies, it matters in prac- tice whether the architecture can efficiently leverage specific structures in the inputs. In the context of time series forecast- ing, it is then important to answer the following question: what is the most appropriate discrete vocabulary for a con- tinuous rreue-valued) sequential input? In other words, what is the correct "language" for time series forecasting? As the goal is to develop unified model with excellent forecast- ing performance on unseen datasets, an ideal dictionary of tokens should be as expressive as possible while also being
'Wor done during an internship at Amazon Web Services. Work done at Amazon. 'Cannegie Mellon University AAmazon Web Services AAmazon "New York University. Correspondence to: Luca Masserano <Imassera@e andrew.emu.cduu. Abdul Fatir Ansari <ansardd amazon.com>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Ground Truth
Median Forecast
80% Interval
Chronos-Base
TimesFM
Moirai-Large
Takee-okn-BBal Baae (Ours)
WMN---0-MI W
Willll 10-
Figure I: WaveToken-Base (199M parameters) provides excellent forecasts with very low uncertainty. Performance of different foundation models for time series forecasting on complex patterns of practical relevance: Chronos-Bar (201M), TimesFM (200M) and Moirai-Large (31 struggle capture exponential trends (top row), sparse spikes (second row). and non-stationary signals with 2 and 5 frequencies evolving over time (bottom two rows).
compact, in order efficiently represent the extremely high variety of non-stationary real-world time series. In addi- tion, different "languages" can exhibit various dependency structures: although time series have natural "order" dic- tated by time, one might ask whether there exists a mapping providing a more eloquent re-ordering in a suitable space that exposes important features and simplifies learning. Pop- ular existing frameworks leverage temporal patches (e.g., Das al. (2023)) or quantization into prescribed bins (e.g.. (Ansari et al., 2024)) and lead to competitive performance in standard settings. However, by simply preserving the natural temporal dependency in the input, these approaches tend to struggle to simultaneously capture both local and global patterns that often represent relevant cases in practi- cal applications (see, e.g., Figure
models, recent LL-M-based forecasters, and state-of-the-art deep learning models. Empirical results show that
i) WaveToken achieves excellent forecasting performance and performs on-par or better than all other baselines with re- spect to three complementary metrics, while using a smaller vocabulary (1024 tokens) than recent foundation models for time series.
ii) WaveToken exhibits superior generalization capabilities, achieving the best average rank across all datasets across all metrics for the in-domain benchmark, and across WQL and MASE for the zero-shot benchmark.
ii) WaveToken can easily capture complex temporal patterns in several edge cases relevant for practical applications, such as exponential trends, sparse spikes and signals with differ- ent frequencies varying over time, as shown in Figure 1.
Main contributions. In this work, propose and ana- lyze tokenizer based on wavelets, which are families of basis functions used to decompose signal into small waves with high resolution in both time and frequency domain (Mallat, 2009). In particular, we develop a tailored pre- and post-processing tokenization pipeline - WaveToken that allows the model to learn directly in the space of sparse wavelet coefficients. This yields compressed but highly expressive vocabulary that facilitates the encoding of com- plex non-stationary time series. In addition, it provides an explicit multi-scale structure in the inputs, which the model learns to exploit by effectively forecasting from coarser to finer resolutions. We pair our hevelet-based tokenizer with the popular T5 encoder-decoder architecture of Raffel al. (2020) and pre-train it on large corpus of time series from different domains. We evaluate WaveToken on com- prehensive in nn-oomiin and zero-shot benchmark made of 42 real-world datasets and compare it with traditional statistical
Overall, our findings not only suggest that wavelet- based tokenization can provide a compelling language for time se- ries forecasting with LLMs, but also position our approach as a promising avenue for developing general-purpont information-efficien forecasting models.
2. Language modeling of time-localized frequencies
In this section, we introduce a wavelet-based tokenizer that allows the model to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresh- olds and quantizes the wavelet coefficients, and finally it pre-trains an autoregressive model to predict the tokenized wavelet coefficients for the horizon window.
2

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Wavelet-based Tokenizer
Time Series Forecasting
Real-Valued Time Series
200
100
Real-Valued Time Series (Context)
200 100
Z-Score Scaling
Discrete Wavelet Transform (DWT)+ Thresholding
Concatenated Coefficients (Context)
Approximation, Level
Datail, Level
M
-2.
Approximation, Level 2
2.5
Detail, Level
T:R V
Language Model
Quantization
Sampled Coefficients (Horizon)
Time Series Forecast
Discrete Token Stream: Concatenated Coefficients
www.
Figure 2: High-level depiction of our method. (Left) WaveToken first re-scales the input time series by computing i. = (z, - (M:cc//o:cc then it applies the DWT and possibly thresholds the resulting detail coefficients to zero (red crosses). Finally, wavelet coefficients are quantized to bins of optimal size given their empirical distribution, and then concatenated (excluding the first approximations). (Right) After pretraining on a large corpus of time series, at inference time the model samples autoregressively from the categorical output distribution and yields coefficients at all decomposition levels, which are pushed through the inverse tokenizer to obtain forecast.
2.1. A brief tour of wavelets
via the maximally decimated Discrete Wavelet Transform (DWT), where K Rs N/2 for a signal of length N. The DWT decomposes a signal and preserves its length (N) by applying a cascade of high -pats and low-pass conjugate mir- ror filters via successive convolutions and down- wam--sampinng operations in O(N) time.
We begin with a brief introduction to the main concepts and terminology concerning wavelets, see Appendix A and references therein for more details. The Wavelet transform (WT) was introduced to address some limitations in existing mathematical tools - namely the Fourier transform, which implicitly assumes signal is stationary by employing quickly decaying zero-mest eroo oscillatory function, known as the mother wavelet, which inherently adapts its time- frequency resolution to the signal's 's characteristics. This dual localization property, achieved through the modula- tion of the wavelet's scale parameter, enables the WT to efficiently analyze non-stationary signals achieving localiza- tion in both time and frequency domain (Daubechies, 1992: Mallat, 2009).
2.2. Tokenization via wavelet decomposition
We ask the following question: given a real-valued univari- ale time series x1:C = . [cc] where Cs the context length,
we find an optimal map T : R V that encodes the input with compact but expressive discrete vocabu- lary V? To this end, we develop WaveToken, a tokenization pipeline divided in scaling, wavelet decomposition, thresh- olding and quantization. See Section 3.4 for an extensive analysis of the chosen
We can divide wavelet families into two sets of basis func- tions: the father wavelet, which captures low frequencies (the approximation), and the mother wavelet, which focuses on high frequencies (the detail). Both sets can be modulated via scaling and translation to achieve multi-resolution de- composition of an arbitrary signal (z) into the J-th lowest approximation component combined with the J 1 suc- cessive details. In what follows, we obtain approximation {ax Jy and detail coefficients {d Vk E
Scaling. Re-normalizing time series is standard practice in modern forecasting, it brings all inputs to common scale and avoids numerical and optimization issues, espe- cially for globally-learne deep learning models (Benidis et al., 2022). Popular normalization techniques are based on affine transformations ī, = (x, - m)/s, with m and appropriately chosen. We set m = P1.C = 3/ 1t=1 It
3

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
and s d1.C V c-1 Ei - mI,)), also known as 7-score scaling. By applying this transformation, we make sure the model receives consistent fimeunnnyeency representations.
to finer coefficients to reflect granularity and downsampling of the DWT.
VisuShrink (Donoho, 1995): de, = sign(da,)) - >), with À = ov2log. NN and ø estimated from the finest detail coefficients dx.j=1. The DWT of noisy data can be seen as a maaximum-likelihon estimate of the wavelet coefficients, and it can be shown that this threshold reduces the expected reconstruction error (i.e., estimator's 's risk) close to the possi- ble minimum, under certain assumptions. We explore both the soft- and hard-thresholdingg variants of this method.
Decompasition. We then apply the maximally decimated DWT (Daubechies, 1992) to decompose the scaled time series C into its constituent time-localized frequencies up to the J-th level, which yields approximation coefficients {ag}s J and a series of detail coefficients {d=) Vk. In addition to being faster than the FFT as mentioned in Sec- tion A.2, this wavelet transform leads to concise representa- tions, i.c., it tends to concentrate the majority of the signal energy onto only a few significant wavelet coefficients while preserving the input size- - a signal of size N results in N coefficients." ! This is especially true for time series with sharp spikes or localized features, and is a useful property for compression and denoising. As each coefficient group is the outcome of convoluting a filter with the signal, the autoregressive structure is also preserved within each group.
FDRC (Abramovich & Benjamini, 1996): see Appendix C for details on the algorithm. By leveraging the connection between thresholding and multiple hypotheses testing, this procedure improves on VisuShrink by adaptively choosing À to control the expected proportion of incorrectly included coefficients (similar to false discovery rate) among those chosen for the wavelet reconstruction.
See Section 3.4 and Appendix C for results and justifications on the chosen thresholding technique.
Thresholding. The inherently sparse representations that the DWT induces imply that we can encode complex fea- tures with only a few significant wavelet coefficients, while the rest are close to zero and thus can potentially be dis- carded without significantly addressing reconstruction qual- ity. Several thresholding techniques, many tailored to spe- eific applications, have been developed over the years and are effectively used in practice (e.g., in image compres- sion: Chang et al. (2000); Vetterli & Kovacevic (1995); Christopoulos et al. (2000)). As the main downstream task in this work is to pre-train a language model on a large corpus of diverse time series datasets, any thresholding technique must be adaptive the underlying noise and com- plexity level of the input, as to retain most of the signal energy without discarding essential information. In what follows, we always keep the approximation coefficients un- altered since they model the low-resolution coarse structure of the time series, and explore the following techniques for thresholding detail coefficients:
Quantization. At this point, the resulting coefficients {as}, and {de ) are still real-valued and need be con- verted into discrete tokens to be directly processed by lan- guage models. Expanding upon the techniques explored by (Ansari et al., 2024). we construct the vocabulary V by bin- ning the raw coefficients according to their joint empirical distribution on the training set, and choose the optimal bin size according to the Freedman-Diaconis rule (FD) to mini- mize the reconstruction error (Freedman & Diaconis, 1981). In symbols, given the optimal B bin centers G1 < < CB and B edges C; < E < C++1 for i = 1,..., - 1, we map each wavelet coefficient w E (fax (d to V as follows: q(w) = i 1 (e;-1 I w < <i). We further enrich V to be immediately compatible with language mod- els by adding two special tokens: a PAD token that signals missing values in the wavelet coefficients - resulting from missing values in the time series from padding of short instances and an EOS token that signals the end of the sequence. In our context, single shared vocabulary jointly encodes time-localized low and high frequencies, thereby leading to a compressed but expressive codebook.
No-thresholding: dx.j de.j If the chosen wavelet family preserves signal energy (in the sense of Parseval's 's theorem), not thresholding the coefficients avoids any information loss at this stage.
2.3. Model training, objective function and forecasting
CDF-thresholding:
dz,; = dx., d > |T2) ()
Given a time series context X1:C, we apply the steps detailed in Section 2.2 and then concatenate the resulting discrete to- kens for approximation and detail coefficients into a vector Z = [a., d...... dr), so that the model can learn and forecast ttarse-to-fine structures in multi-scale fashion. To minimize ad-hoe modifications to the language model, we opt for an encoder-decoder architecture based on the T5 family (Raffel et al., 2020), which has recently been shown to achieve excellent zero-shot performance on a compre-
where F-1 is the (empirical) inverse-CDF of the absolute values of the detail coefficients d,. Vj (1..... J). A coefficient is set to 0 if it is in the lower tail of the empirical distribution of the details' magnitude at the corresponding j- th level, where the cutoff grows exponentially from coarser
'To be precise, longer filters and specific boundary conditions to handle the signal edges might yield slightly higher number of coefficients (Torrence& & Compo, 1998).
4

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
hensive benchmark (Ansari et al., 2024). Other proposed LLM-based forecasters either reformulate the problem crestion-answering task or use patches of the input as tokens, and would therefore not be immediately compatible with our wavelet-based tokenizer (see Appendix B for an exhaustive review). For brevity, in what follows we refer to WaveToken as being both the tokenizer and the model paired together.
lar task-specific models trained for each dataset separately, namely DeepAR (Salinas et al., 2020), PatchTST (Nie et al., 2022) and TFT (Lim et al., 2021); and ii) recently proposed foundation models for time series forecasting - namely, TimesFM (Das et al., 2023), Chronos Mini-Laarge (Ansari et al., 2024), Moirai Base & Large (Woo et al., 2024), and Lag-Ilama (Rasul et al., 2023) - which do not perform task-specific training, but are trained only once on large corpus and then deployed across all evaluation datasets. Appendix H details all the hyper-parameters for the men- tioned baselines. A reference implementation of WaveToken is available at https: thub. com/ commonazo-- science/ chronos- Corecastinggtreee/vveeee
We train the model via next-token prediction by minimizing the cross-entropy between the predicted distribution and the categorical output distribution over the corresponding label tokens from the horizon window 2C:C+ H. which are obtained by tokenizing the ground-trute -outhttruthhaa future values of the time series, XC:C+ H- Formally, the loss function for a time series (including EOS) is
Datasets and training strategy. We train and evaluate WaveToken on the publicly available datasets comprehen- sively collected by Ansari et al. (2024). These span variety of domains and exhibit diverse properties in terms of size, frequencies and prediction lengths, and can be divided in i) pre-training only: datasets exclusively used for training (13 datasets); ii) in-domain: datasets employed for training, whose validation set is also used for evaluation (Benchmark I, 15 datasets); and iii) zero-shot: datasets used solely for evaluation (Benchmark 11, 27 datasets). See Appendix H for detailed information about all datasets. The context length of the sequences in each training batch is set to 512 and the prediction length is set to 64. In addition, we adopt the data augmentations techniques introduced by (Ansari et al., 2024): each of the sequences is generated with probability 0.9 from TSMixup set, which takes convex combinations of different time series, and with probability 0.1 from a syn- thetic dataset generated from Gaussian Processes based on randomly combined kernels. Both of these have been shown to be effective in domains such as time series forecasting, where data is inherently scarce relative to standard language modeling applications.
- -EX+' E.', ((C log pop = (Z(
While the cross-entropy loss does not induce a metric onto the underlying vocabulary, it offers greater flexibility to learn output distributions of arbitrary shapes. This property lends itself well to forecasting applications, where it is often important to capture the correct shape or pattern of the time series without imposing structural limitations such as those inherent in traditional loss functions like MSE and MAE (Le Guen & Thome, 2019).
Learning autoregressive model on concatenated groups of wavelet coefficients might seem counter-intuitive: the temporal structure is only preserved within each coefficient group and is broken as the input transitions from, e.g., ap- proximations to details. In practice, this turns out to be surprisingly helpful as offers the model a natural way to break down complex sub- stuusttuutsres in inputs and outputs by dividing them into a hierarchy of time-localized frequency bands. As we will see in Section 3.3, the model easily learns to exploit these partitions and is able to attend to the right coefficients to improve the overall forecasting accuracy.
Evaluation tasks and metries. For both in-domain (I) and zero-shot (II) benchmark datasets, use the last H obser- vations of each time series a held-out test set. For the long- horizon benchmark of Appendix F, instead, we increase the forecast length H of each time series in Benchmark II by a factor of 2 and 3. We compute the weighted quantile loss (WQL) to assess the quality of probabilistic forecasts on 9 uuiformly-spaces quantile levels {0.1, 0.2, .... 0.9} and the mean absolute scaled error (MASE; Hyndman & Koehler (2006)) to evaluate the quality of point forecasts. In addi- tion, we compute the Visual Relative Squared Error (VRSE; Posam et al. (2024)), which measures the relative squared difference of the amplitudes at all frequencies between the ground truth and the (median) forecast. This metric serves a complementary role to avoid common pitfalls of standard scores, which can fail to properly assess important edge cases that would otherwise be visually obvious, as shown in Figure 8. In order to aggregate these metrics and provide
At inference time, the model produces sample paths over the vocabulary via autoregressive sampling from the pre- dicted distribution p= Z1.C+h), h I,..., H. To obtain time series forecast, first de-quantize the tokens by mapping them to the corresponding bin center: -- = c.. We then apply the inverse discrete wavelet transform (IDWT) and un-scale the reconstructed series by multiplying it by diec and adding f1.0, as in Figure 2.
3. Experiments
3.1. Setup of empirical evaluation
Models and baselines. We pre-train WaveToken with T5 models of four sizes Mini (19.2M), Small (44.5M), Base (199M) and Large (705.8M) for 200K steps on 8 A100 GPUs, and we compare their performance against i) popu-
5

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Chronos (Large)
0.964
Chronos (Large)
0.695
TimesFM
0.521
WaveToken (Large)
WavaToken (Large)
0.698
WaveToken (Base)
0.540
WaveToken (Base)
0.573
Chronos (Base)
0.706
Wave (Large)
0.540
Chronoa (Base)
TimeeFM
0.580
WaveToken (Small)
WaveToker oken (Base)
Chromoo (Smail)
0.708
6.718
0.727
Chronos (Large)
Chronos (Base)
Token (Mini)
0.355
0.550
WaveToke ok (Smal))
WaveToken
0.729
TImeMOE (Base)
0.565
Model
Wave Token (Minl)
Chronoe (Mini)
0.592
Chronos (Min))
PatchTST
WaveToken (Small)
Chronos (Small)
PatchTST
0.601
TimeeFM
9.745
Chronos (Mini)
0.591
(Small)
Moirai (Large)
0.800
PatchTST
0.608
Moiral (Large)
0.669
6.821
Moirai (Large)
0.664
Moirai (Base)
DeepAR
0.676
0.690
TImeMDE (Base)
Moiral (Base)
TFT
0.376
0.939
Molral (Base)
TTM-R2
DeepAR
0.894
0.734
Seasonal Nalive
1.000
TFT
0.921
Lag-Llama-
0.987
TTM-R2
1.030
Seasonal Nalve
1,000
Seasonal
1.000
Lag
Lag-Llama
0.0
0.5
0.0
0.5
1.0
0.0
0.5
1.0
1.5
Agg. Relative WQL
Agg. Relative MASE
Agg. Relative VRSEE
Benchmark (In- Ln-Domain for Wave Token)
TFT
WaveToken (Large)
Moiral (Large)
WavaToken (Large)
0.644
PatchTST
6.810
Taken (Large)
0.584
Chronos (Large)
WaveToken (Base)
0.817
WaveToken (Small)
0.585
WavaToken (Small)
WaveToken (Base)
0.655
0.655
WaveToken (Small)
Chronos (Large)
Chronos (Base)
0.823
WaveToken oken (Base)
Chronos (Small)
Moiral (Base)
0.604
Chronos (Base)
0.662
Chronoe (Small)
0.841
Chronos (Large)
0.623
Model
Chronos (Small)
Chronos (Mini)
0.678
TImesFM
DeepAR
0.843
0.346
chronos (Base)
Chronos (Mini)
0.629
0.624
WaveT Token (Mini)
(Minl)
PatchTST
0.684
WaveToken (Mini)
TFT
0.667
TimeeFM
Chronos (Mini)
0.850
PatchTST
0.732
Moirai (Base)
0.698
Moirai (Large)
Moiral (Base)
0.875
0.908
TImeMOE (Base)
TimeoFM
0.773
6,810
Moirai (Large)
TImeMOE (Base)
DeepAR
DeepAR
0.739
Seasonal Naive
1.000
Seasonal Nalve
1.000
Seasonal
T TTM-R2
1.120
TTM-R2
1.087
Liga-Lama-
1.007
Lia-Laam
1.291
Lag-Llama
1.842
0.0
0.5
0.0
0.5
1.0
1.5
Agg. Relative WQL
Agg. Relative MASE
Agg. Relative VRSE
(b) Benchmark II (Zero-Shot for Wave
Figure 3: WaveToken performs on par or better than recently proposed foundation models for forecasting while using a much smaller vocabulary, and competitive even with modern deep learning models trained specifically on each dataset. The plots show forecasting performance on Benchmark (Panel A, Top) and Benchmark II (Panel B, Bottom) in terms of WQL, MASE and VRSE. Note: Since the values of the VRSE metric for individual datasets may be close zero. the aggregated relative VRSE score, which is computed by taking the geometric mean, may be sensitive to these datasets. We refer the reader to Appendix E for VRSE results on individual datasets.
fair comparisons, we compute each model's score divided by the score baseline model (here, Seasonal Naive). These relative scores are then aggregated across all datasets using the geometric mean. See Appendix D for more details. All results for Chronos and WaveToken averaged across three different seeds.
separately on each dataset. Note again that WaveToken uses a vocabulary size 1024, while Chronos, for example, uses four times as much tokens with | 4096. This empirical finding confirms the well-known theoretical properties of wavelets, which can provide compact but very expressive representations able to condense complex structures into compressed codebook. Finally, on the datasets of Bench- mark I, WaveToken achieves the best average rank across all metrics, as shown in Figure 9.
3.2. In-Domain & Zero-Shot Benchmarks
Figure 3 summarizes the forecasting performance of mod- els the in-domain datasets of Benchmark I. Wave Token outperforms all other baselines with the exception of Chronos-Large on WQL and MASE, and is competitive with respect to VRSE. Considering each model size for each metric, WaveToken achieves lower (i.c., better) scores than Chronos 75% of the times and largely improves on other recent LLM--bsee forecasters. Similarly, method is considerably better even than task-specific models trained
Benchmark II focuses on the forecasting performance on datasets that were never included the training corpus (for brevity, zero-shot). As shown in Figure 3, WaveToken exhibits superior generalization capabilities which lead it to i) outperform all other foundation models for time se- ries across all metrics, with 83% success rate against Chronos models of the same size; and ii) be competitive on WQL and MASE, and much better on VRSE, relative to task-specific models specifically trained on each zero-
6

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Chronos
WaveToken (Ours)
48
96
144
Context
Context
192
240
Attention
288
0.6
0.4
384
432
432
0.2
480
480
40
56
15
64
0.0
Approximations
Details
Horizon
Horizon
Figure 4: Wavelet-based tokenization induees structured patterns in the cross-attention maps. Cross-attention weights for the eighth decoder layer when forecasting the spiky data of Figure (second row). Chronos Prronos-Basee (left) repeats the same patterns for all steps, while WaveToken-Base (right) shows high values the detail coefficients corresponding to the spikes.
shot dataset. WaveToken also achieves the best average rank across WQL and MASE on the datasets of Benchmark II, as shown in Figure 10.
patterns in the cross-attention layers of Chronos and our model, as they use the same T5 encoder-decoder archi- tectures. Figure 4 shows heat-maps of the cross-attention weights in the eighth decoder layer of Chronos-Baa (left) and LaveToken-Base (right), when forecasting the sparse spikes in the second row of Figure 1. Two main things are worth noticing: first of all the attention map for our model is clearly divided in four quadrants, of which the upper- left and lower-right ones exhibit generally larger attention values. These clearly show that, to forecast approximation coefficients in the horizon (time-steps 0-34 on the r-axis), the model learning to attend more to approximation coef- ficients in the context (time-steps 0-258 on the y-axis), and to forecast detail coefficients in the future (time-steps 35-68 the æ-axis), the model attends more to detail coefficients in the context (time-steps 259-516 on the y-axis). Second, we notice an interesting pattern: the detail coefficients for the horizon present two clear columns of larger attention weights (time-steps 45 and 56 on the z-axii) that map to the detail coefficients representing the spikes in the context. In addition, other detail coefficients attend to everywhere else in the bottom-right quadrant (corresponding the flat regions in time domain for the spiky data of Figure 1) ex- cept to those same positions corresponding to spikes (see the white horizontal lines interrupted by the vertical column in the magnified portion). Similar patterns are present for the approximation coefficients, albeit less defined. None of this structures is present in the attention weights of Chronos- Base shown in the left panel, which repeats the same patterns for all steps and indeed fails to forecast the correct position and intensity of the spikes in time domain.
Figure in Appendix shows results on long-horizon benchmark constructed by increasing the forecast length of each dataset in Benchmark II (Zero-Shot) by factor of 2 and WaveToken outperforms other foundation models across almost all combinations of metrics and long horizons, the only exception being H x 3 with respect to WQL, where TimesFM performs slightly better. Appendix E shows the raw WOL, MASE and VRSE values for each dataset in Benchmark and II.
3.3. Qualitative Analysis
So far we have seen how wavelets allow a transformer-bast architecture trained on large corpus to achieve excellent forecasting performance especially on previously unseen datasets, which hints at their superior generalization ca- pabilities. It is then worth analyzing more deeply some edge cases of practical relevance to showcase how wavelets can efficiently capture a wide variety of complex patterns, frequencies and local structures within a compressed rep- resentation. Figure shows examples of synthetically- generated time series which exhibit strong trends, sharp spikes and several frequeenies evolving over time. We eval- uate WaveToken against popular recent foundation models for time series forecasting. Both Chronos, TimesFM and Moirai clearly underestimate trends, struggle at isolating sudden spikes and are not able to capture non-stationary behaviours. On the other hand, our model is able to leverage the different concatenated coefficient groups which repre- sent the time-localized frequency bands, thereby providing accurate forecasts with very low uncertainty.
Appendix G shows the remaining attention maps for layers missing in Figure 4, which all show similar patterns with varying intensities.
This phenomenon can be further explained by looking at

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
In Domain (MASE)
Daubechles-2
Blothhogona--222
In Domain (WQL)
1.0
Avg. Scaled Score
Zero Shot (MASE)
96
Zero Shot (WQL)
1.0
0.85
0.9
0.80
0.8
0.75
0.8
0.70
0.7
0.7
0.65
0.6
32
128
512
1634
2048
4098
Daubh22 2
Daubh
Bior-2.2
Bior-4.4
R-Bier-2.2
Dec1
Dec2
De63
Dec1
Deci2
Dec3
Vocabulary Size
Decomposition Level
Wavelet Family
Figure 5: Effect different vocabulary sizes, wavelet families and decomposition levels on downstream forecast- ing accuracy of WaveToken-Smaa Small. The optimal hyper-parametend were single-level wavelet decomposition with a Biorthogonal -2. family and vocabulary size of 1024. See Section 3 for more details.
3.4. Ablation study
analyze the performance of all four techniques described in Section 2. Empirically, we observe discrepancy in the results obtained when training for the same number of steps on I or 8 GPUs, where the latter is the final configuration used to train the optimal models. As Figure 7 shows, in the single-GPU setting, chosen to streamline computations during hyper- syper-parameter optimization, VisuShrink (Donoho, 1995) appears be the best, a result which is then reverted in favor of No-thresholding in the 8-GPU setting, which increases the total number of samples processed at each step, thereby allowing the model to learn the richer frequen- cies preserved in the input. VisuShrink, on the other hand, leads to smoother signals by crudely cutting off coefficients, which eventually harms generalization. These empirical ob- servations led us to not threshold coefficients, although we note that thresholding is still implicitly happening during the quantization step. which collapses small coefficients to the bin centered at 0. This value is then used at inference time to forecast tokens mapped to this bin. Optimal bins for the discretization procedure - detailed in Section 2 were selected between the lower and upper bounds [-30, 30). chosen empirically by scanning the training corpus.
Figure 5 shows the effect of different vocabulary sizes, wavelet families and decomposition levels on the forecast- ing accuracy of WaveToken-Small trained for 200K steps on one A100 GPU. Regarding vocabulary size, which de- termines the precision with which tokens encode wavelet coefficients, we observe a gradual but consistent improve- ment until | = 1024, which we select as the optimal value. For higher vocabulary sizes, WQL and MASE remain flat worsen both in-domain and zero-shot benchmarks. This phenomenon can be ascribed to the intrinsic compres- sion properties of wavelets (Mallat, 2009): by concentrating most of the signal energy onto few coefficients, we can effectively capture more information with a smaller code- book, while a larger one would only reserve more tokens to spurious coefficients. Note that our vocabulary is much smaller than comparable models: Chronos for example, that leverages the same architectures, uses | = 4096. As to the many wavelet families available, the central panel of Figure shows that the Bior thorthogonal-22 2... 2 basis achieves optimal performance. This family uses two separate filters with two vanishing moments for analysis and synthesis, a dual structure that allows for symmetric or near-symmetric wavelet functions that help preventing distortions near the signal edges. This is known to be advantageous in many applications, including image compression Families with higher vanishing moments capable of capturing higher order polynomial did not prove effective. As to the decomposition level, we observe that first-level coefficients are sufficient to achieve good forecasting accuracy. Albeit deeper levels pro- vide more granular licelloc frequencies, the higher number of groups in the concatenated coefficients make it harder for the attention mechanism to identify the relevant ones at each prediction step. Regarding rhresholding, we
4. Conclusion
In this work, we develop WaveToken, a tokenization pipeline tailored to a specific goal: constructing a general-purpose forecasting model capable of capturing a wide variety of complex patterns while consuming as little information as possible, thereby leading to excellent generalization perfor- mance unseen datasets. We leverage dxcently-ropposee framework to pre-train an encoder-decoder architecture in the context of time series (Ansari et al., 2024) and re- purpose it to learn an autoregressive model in the space of time-localized frequencies. The resulting wavelet-based vocabulary is both compact - using 1024 tokens, i.e. one quarter of Chronos and very expressive, and leads to
Biorthogonal wavelets with the maximally decimated DWT are in fact used in popular compression standards, such as JPEG- 2000 (Christopoulos et al., 2000; Usevitch, 2001).
8

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
excellent forecasting performance with respect all other baselines in terms of three complementary metries:
Benidis, K., Rangapuram, S. S., Flunkert, V., Wang. Y., Mad- dix, D., Turkmen, C., Gasthaus, J., Bohlke-Schneider, M., Salinas, D., Stella, L.,, et al. Deep learning for time se- ries forecasting: Tutorial and literature survey. ACM Computing Surveys, 1-66), 2022.
weighted quantile loss for probabilistic forecasts, mean ab- solute scaled error for point forecasts, and visual relative squared error to measure discrepancies in the frequency content relative to the ground truth.
Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv: /4/65, 2020.
ti) superior generalization capabilities, with WaveToken being the best model across all datasets in terms of average rank- across all metrics for Benchmark I, and across WQL and MASE for Benchmark II. In addition, our method can easily capture complex temporal patterns in several edge cases relevant for practical applications, as shown in Figure 1.
Chang, S. G., Yu, B., and Vetterli, M. Adaptive wavelet thresholding for image denoising and compression. IEEE transactions on image processing,
2000.
Choi, S., Mousavi, S., Si, P., Yhdego, H. G., Khadem, F., and Afghah, F. Eegbert: Understanding hidden lan- guage of eegs with self-supervised representation learn- ing, arXiv preprint arXiv:2306.06::4 2023.
As potential future directions, we foresee that exploring techniques to automatically handle context lengths larger than 512 in the tokenizer would allow models to capture longer-range dependencies if needed. The DWT is in fact a natural tool to leverage in these settings, as it can encode long contexts without increasing the input length through convolutions and wwwn-sampling. Furthermore, WaveToken exploits an autoregressive model on wavelet coefficients. As such, it suffers from slower decoding and inference times rel- ative to recently proposed alternatives, such patch- mutch-based models. Applying WaveToken to patch-based architectures represents an interesting area for future research.
Christopoulos, C., Skodras, A., and Ebrahimi, T. The jpeg2000 still image coding system: overview. IEEE transactions consumer electronics, 46(4): 100-)1107711 2000.
Dagan, G., Synnaeve, G., and Roziere, B. Getting the most out of your tokenizer for pre-training and domain adap- tation. In Proceedings of the 41st International Confer- ence on Machine Learning, volume 235, pp. 9784-9805 PMLR, 2024.
Das, A., Kong, W., Sen, R., and Zhou, Y. A decoder- only foundation model for time-series forecasting. arXiv preprint arXiv:2310. 10688, 2023.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Daubechies, I. Ten lectures wavelets. SIAM, 1992.
Donoho, D. De-noising by soft-thresholding. IEEE transactions on information theory, 41(3):613-627, 1995.
References
Farebrother, J., Orbay, J., Vuong, Q., Taïga, A. A., Cheb- otar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., al. Stop regressing: Training value func- tions via classification for scalable deep rl. arXiv preprint W:Xh::2403003950 2024.
Abramovich, F. and Benjamini, Y. Adaptive thresholding of wavelet coefficients. Computational Statistics & Data Analysis, 22(4)3351-361, 1996.
Alexandrov, A., Benidis, K., Bohlke-Schneider, M., Flunkert, V., Gasthaus, J., Januschowski, T., Maddix, D. C., Rangapuram, S., Salinas, D., Schulz, J., al. Glu- onts: Probabilistic and neural time series modeling in python. Journal of Machine Learning Research, 21(116):
Freedman, and Diaconis, P. On the histogram as a density estimator: 2 theory. Zeitschrift für Wahrscheinlichkeets- theorie und verwandte Gebiete, 57(4):453-476, 1981.
Gage, P. A new algorithm for data compression. The Users Journal, 12(2):23-38, 1994.
1-6, 2020.
Alyafeai, Z. Al-shaibani, M. S., Ghaleb, M., and Ahmad, I. Evaluating various tokenizers for arabic text classification. Neural Processing Letters, 55(3):2911-293 2023.
Garza, F., Canseco, M. M., Challú, C., and Olivares, K. G. Statsforecast: Lightning fast forecasting with statistical and econometric models. PyCon: Salt Lake City, UT, USA, 2022.
Ansari, A. F. Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.078/ 2024.
Golkar, S., Pettee, M., Eickenberg, M., Bietti, A., Cranmer, M., Krawezik, G., Lanusse, F., McCabe, M., Ohana, R., Parker, L.., et xval: continuous number encoding for

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
large language models. arXiv preprint arXiv:2310.02988, 2023.
Mallat, S. A Wavelet Tour of Signal Processing: The Sparse Way. Academic Press, 3rd edition, 2009.
Golling, T., Heinrich, L., Kagan, M., Klein, S., Leigh, M., Osadchy, M., and Raine, J. A. Masked particle modeling on sets: Towards self-supervised high energy physies foundation models. Machine Learning: Science and Technology, 5(3):035074, 2024.
Mallat, S. G. Multiresolution approximations and wavelet
orthonormal bases of (IR). Transactions of the Ameri- mathematical society, 3 315(1):69-87) 1989.
Mattar, W., Levy, I., Sharon, N., and Dekel, S. Wavelets are all you need for autoregressive image generation. arXiv preprint arXiv:2406. 19997, 2024.
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., and Dubrawski, A. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885. 2024.
Meyer, Y. Wavelets and operators: volume 1. Cambridge university press, 1992.
Gruver, N., Finzi, M.. Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.
Mirchandani, S., Xia, F,, Florence, Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models general pattern machines. arXiv preprint arXiv:2307.0472 2023.
Guth, F., Coste, S., De Bortoli, V., and Mallat, S. Wavelet score-based generative modeling. Advances in neural information processing systems, 35:478-491, 2022.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211 144:32, 2022.
Haar, A. Zur theorie der orthogonalen funktionensysteme. Mathematische Annalen, 71(1)338-53, 1911.
Oppenheim, A. V. and Schafer, R. W. Discrete-Time Signal Processing. Pearson, 2010.
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319-326, 2025.
Papadimitriou, S., Brockwell, A., and Faloutsos, C. Adap- tive, hands-off stream mining. In Proceedings 2003 VLDB Conference, pp. 560-571. Elsevier, 2003.
Posam, L., Shekhar, S., Lee, M.-C., and Faloutsos, C. Difffind: Discovering differential equations from time series. Pacific-Asia Conference on Knowledge Discov- ery and Data Mining, pp. 175-187. Springer, 2024.
Hyndman, R. J. and Kochler, A. B. Another look at mea- sures of forecast accuracy. International journal of fore- casting, 22(4):679-688 2006.
Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F,, Pan, S., et al. Time-Ilm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023.
Purwins, H., Li, B., Virtanen, T., Schlüter, J., Chang, S.-Y., and Sainath, Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing, 13 (2):206 219, 2019.
Koenker, R. and Hallock, K. F. Quantile regression. Journal of economic perspectives, 15(4):143-156, 2001.
Rabanser, S., Januschowski, T., Flunkert, V., Salinas, D., and Gasthaus, J. The effectiveness of discretization in forecasting: An empirical study on neural time series models. arXiv preprint arXhv:2005.10111 2020.
Lara Hara-Benitez, P., Carranza-Gareía, M., and Riquelme, J. C. An experimental review on deep learning architectures for time series forecasting. International journal of neural systems, (03)): 330001, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (1 (140):1-67, 2020.
Le Guen, V. and Thome, N. Shape and time distortion loss for training deep time series forecasting models. Advances in neural information processing systems. 32, 2019.
Rasul, K., Ashok, A., Williams, A. R., Khorasani, A., Adamopoulos, G., Bhagwatkar, R., Biloš, M., Ghonia. H., Hassen, N. V., Schneider, A., et al. Lag-Ilama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.0827 2023.
Lim, B., Ank, S. Ó., Loeff, N., and Pfister, T. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4): 14641764, 2021.
10

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3):1181-113) 2020.
Usevitch, B. E. A tutorial on modern lossy wavelet image compression: foundations of jpeg 2000. IEEE signal processing magazine, 18(5):22-35, 2001.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Sasal, L.., Chakraborty, T., and Hadid, A. W-transformers: a wavelet-basee transformer framework for univariate time series forecasting. 2022 s IEEE international con- ference on machine learning and applications (ICMLA),
Vetterli, M. and Kovacevic, J. Wavelets and subband coding. Prentice-hall, 1995.
pp. 671 671-676. IEEE, 2022.
Woo, G., Liu, C.,, Kumar, A., Xiong, C., Savarese, S., and Sahoo, D. Unified training of universal time series fore- casting transformers. arXiv preprint arXiv:2402. 02599, 2024.
Singh, A. K. and Strouse, D. Tokenization counts: the im- pact of tokenization on arithmetic in frontier llms. arXiv preprint ar arXiv:2402. 140:2402..4903, 2024.
Stewart, L., Bach, F., Berthet, Q.. and Vert, J.-P. Regression as classification: Influence of task formulation on neural network features. In International Conference Artifi- cial Intelligence and Statistics, pp. 11563-11582. PMLR, 2023.
Xue, H. and Salim, F. D. Promptcast: A new prompt- based learning paradigm for time series forecasting, IEEE Transactions on Knowledge and Data Engineering, 2023.
Yeo, K. and Melnyk, I. Deep learning algorithm for data- driven simulation of noisy dynamical system. Journal of Computational Physics, 376:1212-125 1231, 2019.
Strang, G. and Nguyen, T. Wavelets and filter banks. SIAM, 1996.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-5077 50,, 2021.
Subramanian, S., Harrington, P., Keutzer, K., Bhimji, W., Morozov, D., Mahoney, M. W., and Gholami, A. To- wards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, X., Jin, X., Gopalswamy, K., Gupta, G., Park, Y., Shi, X., Wang, H.,, Maddix, D. C., and Wang, Y. First de-trend then attend: Rethinking attention for time-series forecasting, arXiv preprint arXiv:22/2.0815// 2022.
Talukder, S., Yue, Y., and Gkioxari, G. Totem: Tokenized time series embeddings for general time series analysis. arXiv preprint ar arXiv:2402.1 144. 2024.
Zhang, X., Chowdhury, R. R., Gupta, R. K., and Shang, J. Large language models for time series: A survey. arXiv preprint arXiv:2402.0188)) 2024.
Tian, K., Jiang, Y., Yuan, Z., Peng. B., and Wang. L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint 2000::2200402900, 2024.
Zheng, W., Sharan, S., Jaiswal, A. K., Wang, K., Xi, Y., Xu. D., and Wang. Z. Outline, then details: Syntacti- cally guided coarse-to-fine code generation. In Inter- national Conference Machine Learning, pp. 42403- 42419. PMLR, 2023.
Tolmachev, A., Kawahara, D., and Kurohashi, S. Juman++: A morphological analysis toolkit for scriptio continua. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstra- lions, pp. 54-59, 2018.
Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R. Fedformer: Frequency enhanced decomposed trans- former for long-term series forecasting. In Interna- tional conference machine learning, pp. 27268-27286. PMLR, 2022.
Torgo, L. and Gama, J. Regression using classification algo- rithms. Intelligent Data Analysis, 1(4):275-292, 1997.
Torrence, C. and Compo, G. P. practical guide to wavelet analysis. Bulletin of the American Meteorological society, 79(1):61-78, 1998.
Zhu, Z. and Soricut, R. Wavelet-based image tokenizer for vision transformers. arXiv preprint arXiv:4405.18626 2024.
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Haar
Biorthogonal-2.2
Traffic Dataset
Time
Approximation,
Detail, Level
0.4
08
4.10
0.30
8:10
0.28
1.3
(Lx)
Pancla)
Time
Mosther
Approximation, Level
Detail, Level 2
0.55
8.59
0.5
1.00
Vosc(e)
Mracl))
Time
Figure 6: (Left) Mother and father wavelets for the Haar and siorthogonal-2.1 2.2 families. Note the dual structure of the latter, which uses two filters: one for decomposition and one for reconstruction. (Right) Example of discrete wavelet transform (DWT) applied time series from the traffic dataset. Red boxes highlight the coefficients that are returned by the decomposition.
A. A brief tour of wavelets
A.1. Motivation: From the fourier transform to the wavelet transform
map a signal from the time domain the frequency domain by computing the signal's projection onto basis of complex exponentials, which represent sine and cosine functions of varying frequencies. This process effectively decomposes the signal into its constituent frequency components. The main drawback of the Fourier Transform is that, while having high resolution in the frequency domain, it has zero resolution in the time domain. In other words, it cannot tell at which location time these frequencies occur in the original signal. The Short-Time Fourier Transform (STFT) tries to overcome this issue by splitting the original signal in different windows and applying the Fourier Transform in each of them. Nonetheless, the fixed window immediately implies a trade-off: the smaller the size of the window, the higher the resolution time domain, the lower the resolution in frequency domain, and vice versa. This widely known the Uncertainty Principle (Oppenheim & Schafer, 2010).
The Fourier Transform (FT) allows
The Wavelet Transform (WT) addresses this limitation by employing quickly decaying zero-mean oscillatory function, known as the mother wavelet, which inherently adapts its time-fiequency resolution to the signal's ''s characteristics. By dilating or compressing the wavelet, its time support varies inversely with its frequency, thus providing high time resolution for high-frequency components and high frequency resolution for low-frequency components. This dual localization property, achieved through the modulation of the wavelet's scale parameter, enables the WT efficiently analyze non-stationary signals whose spectral content evolves over time, phenomenon often referred to as time-frequency localization (Daubechies. 1992; Mallat, 2009).
A.2. The haar wavelet and the discrete wavelet transform
Wavelet families consist of basis functions that can be divided into two primary types: the scaling function (often referred to as the "father" wavelet), which captures the low-frequency, coarse structure of the signal (the approximation), and the wavelet function (often referred to as the "mother" wavelet), which captures the high-frequency components (the detail) of the signal. The "father" and "mother" wavelets give rise to "son" and "daughter" wavelets through scaling and translation. Here, we briefly formalize these concepts in the context of the Haar wavelet (Haar, 1911) 1), a simple yet pedagogically significant wavelet family. For an extensive introduction to wavelets and their applications, see, for example, Mallat (2009), Daubechies (1992) and Strang & Nguyen (1996).
12

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
The Haar father wavelet is given by x(x)) I1(x E [0, I)). The corresponding son wavelet is
Φx,(T) = 20/2 x -k) -k)= [21/2, if s k+1 2
0, otherwise.
where j E N. Note that (x))(>)) generates an orthonormal basis for functions with jumps at the integers, whose space we denote by Vo- The dilations and translations (x))(x) span orthonormal spaces Vj for functions with jumps at k/2', and satisfy V3+1 S V C ...... Vo. With these bases, we can represent an arbitrary function f(x) in Vj+1 by taking a component in Vg (the approximation) plus the orthogonal complement of V to V which we denote by W (the detail).
In other words, we represent in V5+1 by taking its orthogonal decomposition 2+1 Vg + Wj. The space Wo is spanned by the mother wavelet
= 2/2 1,
(x) = (2xx) 6222 1)
if s
0,
otherwise
whose dilations and translations v/.. = 24/2 (2'x k) form an orthonormal basis for each W, Putting this all together, we can represent a function at different resolution levels by breaking down the level-j approximation component Vj to level-(j - 1) detail and approximation components, and so on: V, = VJ @ W = Vo @ Wo @ W1 WJ-1. wavelet decomposition is then linear combination of elements in these subspaces:
J
f(x) = Σ a (z) +
Σ Σ
(k.(),))))
000
where only a finite number of coefficients ag and dx.j are non-zero.
How do we compute ag and dR,j in practice? The multi-resolution theory of Mallat (1989) and Meyer (1992) offers an elegant and efficient solution: any wavelet that generates an orthonormal basis can be characterized by conjugate mirror filter. The mapping from a discretized signal to a sequence of wavelet coefficients is then implemented as a filter bank made of a cascade of high-pass and low-pass filters associated with the mother and father wavelets, and is called Discrete Wavelet Transform (DWT).. For the Haar family, the approximation components are associated with averaging filters (low-pass), while the detail components are associated with distance filters (high-pass). The DWT convolves these filters with the signal and then downsamples by factor of two eliminate the repeated information. The result is an array of detail coefficients {d+}; } and approximation coefficients {ax Jj- The latter can be further decomposed into new approximation and detail coefficients at the next level and the process can continue recursively, resulting in a hierarchical and multi resoutionooltion decomposition of the signal. Figure 6 (right) shows this for a particular time series.
The DWT has a computational complexity of O(V), with N being the length of the input signal. This is due to i) the filtering operation requiring a constant amount of work proportional to the signal length, and ii) the down- samplin operation halving the signal at each of the log2 N levels: this leads total amount of work equal to N + 1 + 1 +...., ), which converges to 2N. Note that this is even faster than the FFT algorithm, which has a computational complexity of O(N log N).
B. Related Work
Tokenization in LLMs. Most tokenizers originate from the natural language processing (NLP) literature and have been developed and studied for various domains such as math (Singh & Strouse, 2024), code (Zheng et al., 2023), and several languages (e.g., Tolmachev et al. (2018); Alyafcai al. (2023)). In modern LLMs. the most popular tokenizers learn a dictionary directly on data, such as variants of Byte-Pair Encoding (BPE; Gage (1994)). Tokenization of real-valued numbers has received particular attention due to the diffieulty of finding what is the most appropriate discrete vocabulary for continuous input. One of the most popular methods is training Vector Quaatized-Variationaal AutoEncoder (VQ-VAE) (Van Den Oord et al., 2017), which learns a dictionary of k-dimensional codewords to capture latent representations. Instead of learning a token for each numerical value, Golkar et al. (2023) proposes a fixed scheme that allocates a dedicated embedding vector for numerics and scales it by the number value, thereby improving efficiency and generalization. In this work, we similarly adopt a fixed scheme, but employ a tailored wavelet decomposition that enhances the crucial spectral properties of time series signals.
Wee always refer to the decimated DWT. For a comparison with the undecimated DWT, see Mallat (2009).
13

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Algorithm 1 False Discovery Rate of Coefficients (FDRC)
1: For each j compute two-sided p-value pa.j = 2(1 (((dk.//o)) for 11 dk.. = 0
2: Order Pk.j such that P(1) S .... Pm
3: Let in = arg max; P(o) $ (i/m)g, with q being error-rate under Ho
4: Let Aig (1 - Pio /2)
5: Threshold all detail coefficients at level Xio
Wavelet-based forecasters. Several approaches have tried to embed wavelets in forecasting pipelines. An early example that of Papadimitriou al. (2003), who integrate the discrete wavelet transform with ARIMA modeling capture complex patterns over long time periods. More recently, Zhou et al. (2022) proposed to substitute attention blocks with Fourier- and wavelet-enhanced blocks in the transformer architecture. Within this line of work, Zhang et al. (2022) proposed to model trend and seasonality separately with an MLP and Fourier attention, respectively. In addition, they showed that under linear transformations, attention models in time domain, Fourier domain and wavelet domain have the same representation power. Sasal et al. (2022) leverage a redundant wavelet transform that yields J series of coefficients (one per decomposition level), each having the same length as the original time series. They then learn a separate transformer model for each scale. To the best of our knowledge, WaveToken is the first application of a maximally decimated wavelet transform to build a tokenizer tailored for time series forecasting with LLMs.
Modeling and generation of signals. Different methods in the signal processing literature propose to integrate spectral or wavelet decomposition with deep learning architectures to enhance the capabilities of these models on a variety of tasks. Apart from the aleetdd- VQ-VAE (Van Den Oord et al., 2017), several works exploit spectrograms and log-mel spectra, for example, as pre- pre-preessiing steps on medical or audio data (Choi et al., 2023; Purwins et al., 2019). Similarly, audio codecs (Zeghidour et al., 2021) are emerging as critical technique to bridge the gap between continuous waveforms and token-based language models. As for image generation, Guth al. (2022) proposed speed up denoising by learning diffusion models in the wavelets domain. Recent concurrent works by Tian et al. (2024), Mattar et al. (2024) and Zhu & Soricut (2024) learn a transformer model on multi-scale sequence or in the space of 2D wavelet coefficients.
Regression via classification. WaveToken converts the regression problem of forecasting into a classification problem on the space of quantized wavelet coefficients. Apart from its use in recent works such as Ansari et al. (2024), Farebrother et al. (2024) and Hollmann et al. (2025), this technique of performing regression via classification has been studied in prior literature (Torgo & Gama, 1997; Stewart et al., 2023). It has also been used for time series forecasting ( (Yeo & Melnyk, 2019; Rabanser et al., 2020; Alexandrov et al., 2020). However, in contrast to WaveToken which operates on quantized wavelet coefficients, all these prior works directly quantize the time series. This enables WaveToken to capture the local and global patterns in the time series.
Foundation models for time series forecasting. Several approaches are being developed to adapt LLMs to other domains, such as time series forecasting. Recent efforts in this direction include, e.g., Xue & Salim (2023), which converts time series into text and re-frames the task as a question-answe answering problem; Gruver et al. (2024), which tokenizes real-valued data as strings of digits and leverages models such as GPT-3 (Brown, 2020) and Llama (Touvron et al., 2023); and Jin et al. (2023), which prompts frozen L.M with prefix describing the task and patch embeddings of time series aligned with text prototypes. Recently, Rasul et al. (2023); Goswami et al. (2024); Das et al. (2023); Ansari et al. (2024); Woo et al. (2024); Talukder et al. (2024) proposed different paradigms to pre-train transformer-based architectures on a large corpus of time series. See also Zhang et al. (2024) for recent survey. While these works adopt domain-speciffe designs such as patching, lags and time features, none of them tackles the problem by learning an autoregressive model in the expressive space of time-localized frequencies.
C. Additional results on different thresholding techniques
Algorithm outlines the FDRC thresholding method step-by-step. See Abramovich & Benjamini (1996) for more details. During our experiments, we chose the standard g 0.05, which corresponds to the type-I error control level of the resulting hypotheses tests.
Figure 7 shows the effect of the different thresholding techniques detailed in Section 2.2 on the downstream forecasting accuracy in terms of weighted quantile loss (WQL.,, mean absolute scaled error (MASE), and visual relative squared error
14

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
0.95
In Domain (MASE)
In Domain (WQL)
Zero Shot (MASE)
Avg. Scaled Score
0.85
Zero Shot (WQL)
In Domain-8GPUL (MASE)
0.80
In Domain- (WQL)
0.75
Zero Shot-8GPUs (MASE)
Zero Shot-8GPUs (WQL)
0.70
0.65
0.60
None
CDF
VieuShrink
FDRC
Thresholding Method
Figure Effect of the different thresholding techniques Section 2.2 on the forecasting accuracy.
(VRSE). This ablation study complements those on vocabulary size, wavelet family and decomposition level outlines in Section 3.4. Apart from the four thresholding techniques analyzed in this paper, several more exist the literature for signal processing and image compression. We leave comprehensive analysis of these methods and deeper study of their effect on downstream performance to future work.
D. E Evaluation metrics
Consider a collection N time series {x; = 1......... +I,++Hl)= that include both context and horizon. Section we evaluated WaveToken and all the other baselines with respect three metries, which we now describe more detail: weighted quantile loss (WQL), mean absolute scaled error (MASE), and visual relative squared error (VRSE)
Weighted quantile loss (WQL). We use this metric to evaluate probabilistic forecasts L,C++ obtained by generating
N 20 samples from the model for an input i at nine quantile levels E {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). WQL aggregates the standard quantile loss at level QL, (q, (Koenker & Hallock, 2001) over multiple horizon steps
1,..., H and series by taking a weighted average:
2 Ei, Qu Qlo((ect422ss)
WQL 1/9 -
Σ
Eie [aae]
where 9 is the number of quantiles used.
Mean absolute scaled error (MASE), We use this metric to evaluate point forecasts &; (ll....,,,,C+4 which we take to be the median quantile g across = 20 samples for probabilistic models. The MASE (Hyndman & Koehler, 2006) scales the mean absolute error by the empirical error the seasonal naive model:
C- E Et=++ ( Zi,el
C+H
MASE( x;) = H CC-8 Fitt - Fass
where S is seasonality parameter.
Visual relative squared error (VRSE). We this metric to measure the frequency content of (point) forecasts relative to the ground truth. As for MASE, we take the median forecast g.55 as input to the metric for probabilistic models. VRSE (Posam et al., 2024) is defined as follows:
(Ax, -x (f)
VRSE x;)
E,(xx,
15

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Truth Constant Forecast
0.04
2.0
SARIS
0.2
Shifted Forecast
RMSE
0.03
MASE
1.5
VRSE
1.0
0.1
0.05
0.01
0.5
20
40
60
100
0.00
0.0
0.00
Time
Foreact Type
Figure 8: Pitfall of standard evaluation metries. RMSE and MASE fail distinguish poor constant forecast from visually much more accurate shifted forecast. By comparing the amplitudes at all frequencies, VRSE captures the difference and assigns lower score to the shifted forecast.
WaveToken (Large)
4.73
WaveToken (Large)
K.47
WaveToken (Base)
7.00
Chronos (Large)
4.87
Chronos (Large)
WaveToken (Large)
WaveToken (Base)
WaveToken (Base)
5.47
WaveToken (Small)
TimesFM
5.47
Chronos (Base)
6.00
Chronos (Large)
7.47
Tavenoken (Small)
Wave (Small)
PatchTST
6.73
6.73
TimesFM
(Base)
Chronos
TimesFM
7.47
TimeMOE (Bas
8:27
WaveT okkn (Mini)
6.80
(Min))
(Mini)
Model
PatchTST
Chronos (Small)
Chronos (Small)
Chronos (Min)
8.13
Chronos (Mini)
8.47
Chronos (Min))
9.13
Chronos (Small)
8.47
TimeMOE (Base)
10.53
Moirai (Large)
(Large)
Moiral (Large)
10.67
PatchTS1
9.40
DeepAR
10,80
DeepAR
12.20
DeepAR
9.80
Moirai (Base)
12.53
(Base)
10.53
TFT
TFT
13.80
TTM-R2
Moirai (Base)
12:13
Lag-Llama
14.33
TFT
12.93
Lag-Liama
13.33
15.47
Seasonal
13.00
Seasonal Nalve
Seasonal Naive
15.93
Lag-Llama
15
15
Avg. Rank (VRSE) Figure 9: WaveToken achieves best average ranks on in-domain datasets. Average rank of models on Benchmark I (in-domain) in terms of WQL, MASE and VRSE.
Avg. Rank (WQL)
Avg. Rank (MASE)
where A2(f) is the amplitude of the Fourier transform coefficient frequency for input x. It serves complementary role to WQL and MASE: by computing the discrepancy between forecasts and ground truth in the amplitudes of the Fourier transform coefficients at all frequencies, this metric captures whether the forecast has the correct overall "shape", instead of looking at it point-wise. Thus, it provides a valuable alternative in cases where other metrics clearly fail, as shown in Figure 8.
E. Additional results: Benchmarks & II
Figures 9 and 10 report the average rank across all datasets in the corresponding benchmark (in-domain and zero-shot, respectively) achieved by all the evaluated models. Tables 1-3 report the raw per-dataset values of all metrics for all models on the in-domain benchmark, along with the corresponding Aggregate Relative Score and Average Rank. Similarly, Chronos, PatchTST, DeepAR and
Tables 4-6 report the same values on the zero-shot benchmark. Results for TFT are averaged over three seeds.
16

---

time series Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Wake (Large)
5.63
WakeToken (Large)
5.63
Moiral (Large)
5.44
TFT
5.78
PatchTS
Moirai (Base)
PatchTST
6.04
Chronos (Large)
WaveToken (Large)
TimesFM
6.26
TakeToken (Base)
7.00
Taken (Base)
7.59
Chronos (Base)
WaveToken (Small)
7.81
Chronos (Large)
TimesFM
PatchTST
DeepAR
7.00
DeepAR
7.59
DeepAR
6.44
Model
WaveToken (Base)
7.07
7.78
(Large)
Chronos (Base)
WaveToken (Small)
WaveToken (Mini)
WaveTor Token (Small)
8.00
Chronos (Small)
8.37
TimesFM
9.44
Chronos (Small)
0.56
WaveToken (Mini)
9.33
9.74
(Large)
9.30
Moiral (Large)
10.15
Chronos (Base)
WaveT oken (Mini)
9.33
Chronos (Mini)
10.37
Chronos (Small)
9.85
Moirai (Base)
11.04
Chronos (Mini)
10.07
Moiral (Base)
Seasonal Naive
TimeMOE (Base)
11.11
Chronos (Mini)
0.74
TimeMOE (Base)
13.41
Seasonal Naive
12.78
Lag- Lag-Liama
14.48
TTM-R2
15.22
13.59
Seasonal Nalve
Lag-Llama
Lag-Llama
10
20
Avg. Rank (WQL)
Avg. Rank (MASE)
Avg. Rank (VRSE)
Figure 10: WaveToken achieves best average ranks on zero-shot datasets. Average rank models on Benchmark
(zero-shot) terms of WQL, MASE and VRSE.
Table 1: Raw per-dataset values of WQL for all models on the in in-domiin benchmark.
Whuttkkn ravel
Moseel
Strpet
Lap
Eicntriy Min
HMTT
8/76
0.900
01877
B:965
0167
0/75
B.85
6.19
01101
B131
0.877
0.993
0:189
6117
Steencnty (Welly)
Bectisty (Houd))
11.00
6.114
0.20
0.105
1001
8J89
8001
0.104
0.100
0.36
0.114
0.998
0.10
0257
[1113
11123
0200
0.112
1891
621
0.53
8189
0.869
0.118
0.330
0.005
0.125
01108
0.167
0.55
ML (forly)
0.04
800
0034
8025
0.07
0/38
0.53
0.68
544 (3auiiky)
0J091
0.00
0.88
600
0.000
0,099
6991
0256
0.00
086
0.85
0.01
0.09
0.196
Mi (Warth)
8017
0.17
030
000
8.00
0.79
8/16
0016
B000
000
0.06
0.04
8.51
0.6
Palcstiar Cours
5.244
0.33
0236
06255
0210
0210
0.355
0252
0.50
0223
0.57
022
0.361
0.19
0.10
6131
B116
0.177
0.117
BM
015
5.33
0106
0312
0.13
TaspostereRRa
0.718
(Buil))
0.151
0.766
0.176
Agp Relatire Score
1.564
0.50
0.00
0.944
0.991
0.51
0,560
0.97
0.00
0.600
B319
0.01
0.67
0.754
1:00
Arg. Rock
4.677
5,077
8.467
8.133
6.800
3.53
4.67
4.33
13333
12133
10.33
5463
7.377
00.500
11.400
15
Table 2: Raw per-dataset values of MASE for all models on the in-domain benchmark.
Pottrated Midde da Decsan))
Pestaited Klakh (Olke))
Talk Spedil Modih
Local Maddi
GritheeonBBn Btomi
PAASTOT
Eaceicyy (llaciy)
1/50
1.50
1.71
1338
1.53
1.50
1.64
1.009
15T1
1113
1.09
1.30
1.0
1.39
1.3%
17380
1.50
Elanicy (Rarchy)
1.58
133
1954
157
3354
1.71
1.31
2.17
3.0
Londen Sumar Moes
0U53
3.355
01771
0.99
0.55
0.59
0353
0.52
6S88
0.76
055
0/56
1196
36 (istlel
51144
1100
312
1134
3130
3355
1115
3377
5.36
6400
6397
140
590
329
522
(U90)
3550
835
0.50
2060
1033
1.58
0.8%
1351
130
199
1.0
344 (4kkool
1390
2521
210
2135
2137
2072
2005
1588
280
240
2300
3233
$397
1396
2386
258
270
3.355
0.38
Tedl 3D
1.030
3.89
0961
034
033
0.51
0.48
8333
100
1376
1281
1056
1303
1.311
1.57
115
1113
1100
3.555
8329
1001
696
3988
1811
1199
099
Ube TLC (Htar)))
8871
067
0191
0.76
0.77
0.71
0.79
8711
0/2
3756
087
651
0.99
0.556
0.00
0756
(981
Arg.
4.997
3230
5457
6.30
3.482
446
1435
12335
1666
7,481
13.47
10333
12280
11100
13.93
F. Additional results: Long-Horizon Forecasting
Figure 11 shows results long-horizon benchmark constructed by increasing the forecast length H of each dataset in Benchmark II (Zero-Shot) by factor of 2 and 3, implying maximum horizons of 1 112 and 168 (for NN5 Daily). Benchmark I (In- (n-Domain) was not used as increasing the forecast length over the prescribed ones (see Table 9) would imply mixing the test and training portions of the datasets. The datasets Benchmark II without sufficient history to allow for longer horizons have been skipped. WaveToken- Baakenn-Bseeee outperforms other foundation models across all three metrics the H setting. When 3, TimesFM performs better with respect WQL only. All results for Chronos and
17

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Table 3: Raw per-dataset values of VRSE for all models on the in-domain benchmark.
Protrateed Modeccla Deraii))
Protraized Madile (Dile)
Tod:: Spdili Medek
Local Moddl
Soall
Suataln Baad
Elnttecyy Mesl
3534
0.812
Elatiicy (Woaih))
0.015
3534
0831
0831
0.19
0.206
8.84
1379
6011
0.58
0317
0.33
0057
0.150
KDDD
0.99
3388
8.14
6115
0113
0:18
0:23
0111
2222
(19)
0.77
0:17
6131
62%
M4 (ibll))
0000
3.87
0.97
003
689
0.83
3353
3358
0335
0.955
003
03m
000
itkeerto
3:00
1081
693
003
6.01
0.01
0101
2001
000
0:91
0.01
000
(Wodkty
3.59
0.34
4.63
0.83
3555
003
0130
Podcaraa Crante
3334
0325
0111
0.09
0.355
0.99
6126
3:33
0111
BJ66
0821
0.25
3.54
022
11.90
2.33
0312
0312
0.155
0.535
0.83
6423
6:10
8489
033
B-43
0.38
0.854
65T
1712
Ube (Baty
3.53
1902
0834
0128
0.33
0.84
1067
1007
3534
0/822
0.33
0.033
0824
6831
811
Agg Relativ Seare
0.590
3.50
0.58
0.57
0.531
0.33
1.50
0161
3581
0.53
0381
0.535
0.28
0161
0931
1/00
Table 4: Raw per-dataset values of WQL for all models on the zero-shot benchmark.
Pritraine Meddi (Zem Shat)
Pretraised Meekds (Cther)
Tiok Specitr Modek
Local Madkls
Sarr Mal
Werbaee Largo
Warhaa
VAASTON
Aurisiiaa Eleanarty
0.07
0183
9635
0.11
101
101
0::89
0.687
0187
110
Cosld
Domirics
ERCOT Load
0.85
0.09
0,332
BD16
B1355
2015
(011
1000
0050
1550
2015
0000
100
0:001
0.56
0.00
KTT (Bioely
ELS Mine)
01733
0.683
1181
0360
6064
0080
0003
5001
0913
100
1089
0.004
0,009
1005
Eudhung Rate
0:13
0014
0001
0014
0003
1017
0003
0011
8015
FRED
0.021
6417
2002
0100
9022
0.000
0:39
2001
0122
Hospial
0087
0000
9009
0000
0000
2000
0000
0.006
0063
XI (Qualarls)
0.003
0130
0110
5106
0.000
0.50
M3 (Manifly
M4 (teartys
0.151
0.995
11381
0.897
0.000
0.112
0.1%
0.09
0.099
0.177
D606
0161
0091
0.955
01155
03341
5.19
9105
1.K.
8.993
0164
0113
0
6.130
(10%
0340
0226
545 (Taaby)
M3 (Quanady))
0.151
0.074
0.155
8.076
0.799
0.135
0.199
0001
(.140
0.079
D1149
D617
6.51
0175
0.145
0075
0.102
0.134
D000
2.155
0065
81111
0123
011
0.00
1000
0101
ME (Qummdy)
01817
0/881
0109
0161
0100
11099
0117
01111
0000
6:222
0.00
1000
B119
M& (Yzaly)
0.336
0.140
0.136
8111
0128
111
9110
0.61
MS
0.915
0.597
0.34
0.59
0207
0,80
1634
XNS (Wickly)
0.091
0.90
0.91
0692
0.02
0.266
0106
0125
Tpenen (Monttayy
0.000
11:16
0.111
0.10
1L117
01100
0196
0.213
5.121
0.15
0.02
0.184
Teerson (Quanelyy
0.600
0.69
0.744
0.071
D670
0.955
0.63
0.307
51000
0.85
0360
0.02
0.04
0110
Therion (Yarly)
0:187
0.307
0.00
0.718
61181
01997
0.191
0.197
0.714
5167
0163
0:15
815
6137
070
Tullic
0.255
0.264
0.263
0.244
0.240
01250
0.260
0.200
0.256
8.225
0251
8.184
0.56
0.33
0226
0362
Warthrr
0.179
0.41
0.191
0.140
0.179
0.132
0.164
8.138
0153
0.100
014
0.147
0.51
0317
Rank
8.556
9.61
8.000
5650
15
9.66
6.259
7.000
6.77
WaveToken are averaged across three different seeds.
18

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Table 5: Raw per-dataset values of MASE for all models on the zero-shot benchmark.
Preteated Medch Shat)
Preinsianl Modch (Otbrr)
Took Specile Medes
Lecd Moddi
swetilee Seell
Caapol
Anstalia Eletricay
1335
1339
1.9
E334
Les7
1337
1404
1,10
L8t5
1250
1401
1355
0,09
0.571
1-10
0,50
1233
Ca Purt
(190)
3349
3361
330
3368
636
055
03.6
1111
1500
E581
1535
1100
0.815
6796
6.79
1.301
Coold Deaks
13.533
17.09
12.00
15.0%
11161
(L.56
66:41
16.06
14.30
33,35
18
Deceacc
B338
05P
2333
0333
3323
0322
1:33
2190
1050
1120
1.394
1108
0.577
0251
0.96
[USTI
EACOT Load
0AI7
2250
5288
0253
1921
122
2259
P-86
1143
0353
119
620
0.211
IIT (Ikaely)
0775
5784
035
6387
0385
1963
3895
E560
5465
0.5
0211
Phrkurge
3:411
3336
1847
1226
198
110
-20
1390
1011
1700
Itspial
2330
0.15
0.33
6961
1965
1399
5321
1024
E255
1151
1025
0289
341 (bboedts)
1029)
2.117
1,00
1.34
1.23
1115
101
1009
135
1.71
12291
1001
1315
1132
125
112
1200
1314
anell)
1131
1/59
4337
1751
1260
1350
1.50
-350
1161
4200
400
100
756
4.91
M7 (hloeddy)
08877
3.48
0.85
2453
0.53
5358
6388
1395
3917
1411
1987
1253
116
(Quanatt)
1181
1.26
1.32
1221
1216
2599
1443
149
1158
2113
1207
122
120
1206
142
(Qunaat)))
1,316
1221
1,36
1.355
1213
1200
2555
1333
1229
1100
1355
1180
120
125
1AC
(htulyy)
3.00
1.78
1/51
1181
3663
1350
1956
1.56
4176
2014
5100
1971
332 (1Na0)
0,573
2233
0A15
0.32
0613
0992
2300
0/26
DA25
1201
0030
1500
0356
125
NR2 (Wodly)
2525
2.47
0.58
3551
0948
0081
1.44
2551
0096
2065
1103
1009
0000
1.5%
100
3337
1.70
1936
1865
3030
3113
1911
1.41
1234
135
158
120
Tre ar (haadty
3.765
3.000
1901
1881
3235
1553
3860
1280
1355
1868
135
1234
1366
13M
113
300
380
12G
Trafk
3338
0.87
0150
0385
3317
0341
6317
0.33
3325
0750
RX28
344
B,96
076)
622
R55
1ITT
0913
Agg- Reiative Seare
0.475
3.32
0.81
3.50
3348
B338
0311
52108
1741
3468
0.825
B346
1120
0981
6310
030
0667
100
Table 6: Raw par-dataset values of VRSE for all models on the zero-shot benchmark.
Spacile
Boee
Rostein Suge?
Anduduu maaniiay
0.955
0.801
8367
Bd10
0153
0883
5165
0331
0.8
0310
1986
5353
1333
[131)
097
3.63
0331
0383
(19%
01811
DIES
Doabs
0.085
1004
si65
1961
03254
2060
1856
0315
6801
0.545
0.91
Dheensk
0.75
$250
0362
0251
5211
0:35
01355
023)
0.21
0401
INT Mis
0.801
0351
120
2m1
1361
0356
035
ITTT (llaalyy
0.007
0357
196
603
2011
0813
0367
0.00
6000
Duchungs Bas
0.000
8160
1220
0320
1000
0330
1000
0301
0.80
1.9
600
Hoopeal
0163
6354
0.54
2004
0000
0985
8007
631
6954
0.944
0965
0.00
0010
XI Kiibbesss
0.89
1000
1101
6961
0305
1082
0001
000
315
023
0.002
0.1
0.5
0230
0004
Oliuity
0.877
0099
6101
081
6963
033
000
2844
0141
0.00
13777
343 Olleedd)
0.03
6433
8107
6811
0.50
2005
1009
0925
1094
080
0003
0554
0.84
8101
0029
0.26
0305
0334
1999
0321
5103
2004
080
0131
McHQkerery)
0.891
000
0002
033
0.50
0000
0009
0.559
8094
8311
2034
0034
0805
035
0030
M4 Yowryy
0106)
0090
$163
090
0.90
016
0090
01856
2014
010
003
5055 (Dul))
1147
[U36
0145
0.51
094:
0250
03340
(136)
0181
216
01550
0.41
0.35
054
0.71
330 (Wuddo)
0.805
B006
2102
0016
0317
0005
0006
0/14
5011
0001
0/23
0.63
0155
6017
DICs
01336
0067
0318
6965
Trenww (Towlyy
0.003
0007
800
08%
0.30
6005
0.556
8155
100
100
0834
0.61
038
030
0mL
Trals
0.333
6.93
078
0322
0335
6/41
0736
077
6188
2114
0L985
0333
0721
0/76
Asa. Reistive Scare
0.825
8968
0824
0332
5331
6610
0.554
1363
0.54
6553
6:713
1187
0.32
0134
0187
1000
Ang.
9915
18014
3.99
7313
1,95
13411
4352
3.44
5.44
13.98
11.101
93333
9/41
12715
19

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-bascl Tokenization
WaveToken (Base)
0.c01
WaveToken (Base)
WaveToken (Base)
Datenets:
Horizom: IJ x
Model
TimesFM
Chronos (Base)
Chronos (Base)
Chronos (Bass)
(Large)
-Wi (Duateet)
54 Oierr))
(Large)
0.722
Moirai (Large)
0.597
TimesFM
MT JAKETY (Quatier)
"Brram Hearty
Seasonal Naive
10000
Seasonal Naive
1.000
Seasonal Naive
1.000
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
Agg. Relative WQL
Relative MASE
Relative VRSE
Horizors
Ti
WaveToken (Basa)
WaveToken (Base)
0.621
FFIZD up
Model
WaveToken (Base)
Chronos (Base)
Moiral (Large)
(19 min)))
Holppoa
Chronos (Bass)
Chronos (Base)
Socher Hatt
MS Dhanting
(Large)
0.743
(Large)
0.686
TimasIM
0.719
Sessonal Naive
1.000
Seasonal Naiie
1.003
Seasonal Naive
1.000
Auataa
RNN
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
-Oir Parts
Barkoo
Relative WQL
Relative MASE
Agg. Relative VRSE
Figure 11: Long-horizon benchmark constructed by increasing the forecast length of each dataset in Benchmark II (Zero-Shot) by factor of 2 and 3. (Top row) Results with horizon multiplied by 2: WaveToken-Bazt outperforms other foundation models across all three metrics. (Bottom row) Results with horizon multiplied by 3: WaveToken-Baat outperforms other foundation models with respect MASE and VRSE. TimesFM performs better with respect to WQL. (Right) List of all datasets included in the two long-horizon, benchmarks. All results for Chronos and WaveToken are averaged across three different seeds.
20

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
G. Additional results: Qualitative analysis
Figures 12, 13 and 14 show the cross-attention maps for the 12 decoder layers Chronos-Base (left) and WaveToken-Bai (right) when forecasting the spiky data of the second row in Figure 1. Figures 15, 16 and 17 show example forecasts generated by Bave on the evaluation datasets considered in Section
H. Datasets and Baselines
In this section we report details on all the datasets (Tables 7 and 9) and baselines (Table 8) used for the experiments of Section 3. For the ScasonalNaive baseline, we relied on the implementation available in the StatsForecast library (Garza al., 2022). For the task-specific deep learning models, we used their implementations available in the GluonTS library (Alexandrov al., 2020). Finally, we used the corresponding reference implementations for Lag-Ilama' (Rasul et al., 2023). Moirai (Woo al., 2024), TimesFM (Das et al., 2023) and Chronos" (Ansari et al., 2024).
Table 7: Breakdown of the datasets and baselines used for training and evaluation.
Data Subset
Datasets
# Series
Usage
Evaluation Baselines
Pretraining-only
13
795,936
pretraining
SeasonalNaive, DeepAR, TFT, PatchTST Lag-Llama, Moirai-1.0-E (Base & Large), Chronos (Mini-Large), TimesFM
pretraining and in-domain evaluation
Benchmark I
15
97,272
Benchmark II
27
190,674
zero-shot evaluation
All of the above
Table Baseline models and hyper-parameter choices. Syyper-parameter not specified are set to defaults in their respective implementations. C' stands for context length, d,, for hidden layer dimension, nz for number of layers, and nyr for number of heads.
Model
Model Type
Implementation
Probabilistic
Hyperparamcters
SeavonalNaive
Local
StatsForecast
Yes
N/A
DeepAR
Task-specific
GluonTS
Yes
10, T
TFT
Task-specific
GluonTS
Yes
d, = 32, nz =
PatchTST
Task-specific
GluonTS
Yes
Patch length: 16, Stride: 8, d, 32, n. nh
Lag-Llama
Pretrained
Reference
Yes
C 32
Moirai-1.0-R
Pretrained
Reference
Yes
C 1024, Patch length: selected by dataset- specacc- validation
TimesFM
Pretrained
Reference
Yes
All hyperparameters set defaults from the released models
Chronos
Pretrained
Reference
Yes
All hyperparameters set to defaults from the released models
WaveToken
Pretrained
Reference
Yes
All hyperparameters set defaults as Chronos
21

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Chronos
WaveToken (Ours)
2,00
Aesredimations
2.00
1.75
1.75
144
1.50
Layer [Log Scale)
Coers Attertion
Context
182
Context
192
1.25
240
288
1.00
336
384
384
480
480
16
24
32
40
56
16
24
40
56
Approximations
Horizon
Horizon
Chronos
WaveToken (Ours)
1.6
Context Jegroximations
1.4
1.75
144
1.2
4000 1110 Goog
1.50
Layer [Log Scalo)
Cross Aitontion
Context
192
1.0
192
1.25
240
0..8
1.00
288
-0.5
336
384
00.8
384
432
0.2
480
480
16
24
60
48
56
64
0.0
8
16
24
32
40
45
56
64
Approximations
Details
Horizon
Horizon
Chronos
WaveToken (Ours)
48
Angrocimations
144
0.0
Coos Aroontion
244
Layer (Log Scalo)
Cross Attentonn
Context
240
0.0
144 Log (cadoo
Context
192
1.25
1.00
336
336
0.75
384
002
432
480
480
16
24
40
48
56
64
0.0
16 Approximations
8
24
40
48
56
64
0.00
Details
Horizon
Horizon
Chronos
WaveToken (Ours)
48
prosimattions
144
Context
0.8
CC plteetttooo
Context
192
240
Attentioon
0.66
288
336
Detall
432
0.
480
00.0
32
45
64
Approximations
Details
Horizon
Horizon
Figure 12: Cross-attention maps for the first to fourth (from top) decoder layers of Chronos-Ban (left) and WaveToken-Base (right) when forecasting the spiky data of the second row in Figure
22

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Chronos
WaveToken (Ours)
1.2
40
144
dddee [Log Pe0a
144
Layer (Log Scale)
Context
+0.0
Couse Attention
Context
240
1.5
0..6
260
336
Detall:
884
432
0.5
452
56
64
0.0
Approximations
Details
Horizon
Horizon
Chronos
WaveToken (Ours)
48
1.4
0.7
1.2
Context
144
1.0
10000 [Log 99
(Zaas Alhention
Context
144
Layer 6 (Log Scale)
240
0.0
240
0.4
336
384
0.1
480
480
32
48
Approximations
Horizon
Horizon
Chronos
Wake (Ours)
1.4
2.00
96
1775
144
Context
192
100 (Log Scale)
Cross Attention
Context
1ut Log (cale)
Cross Attention
0.08
240
288
0..6
268
1.00
336
0.75
384
0.4
284
-0.50
-0255
480
480
24
32
40
48
56
64
Approximations
Horizon
Horizon
Chronos
WaveToken (Ours)
aegreximatoons
96
96
1.0
Context
192
Aiterttion Soooo Sooe 100
10/00
Context
192
Laper Alttertioon
0.88
240
288
0.6
336
0.4
384
480
480
24
32
40
48
56
64
-0.0
16 Approximations
32
48
64
Horizon
Horizon
Figure 13: Cross-attention maps for the fifth to eighth (from top) decoder layers of Chronos-Baad (left) and WaveToken-Base (right) when forecasting the spiky data of the second row in Figure 1.
23

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Chronos
WaveToken (Ours)
1.8
Aggrosimations
96
-14
1.2
Layer's (Log Scale)
Cross Attention
Context
192
Context
240
0.8
(Log Scalo)
288
288
0.6
0.6
384
0.4
384
480
24
32
40
56
64
45
Approximations
Horizon
Horizon
Chronos
WaveToken (Ours)
Nagroxinationss
144
1.2
100 Bon 01 1.44M
0.8
Layer 10 [log Scale)
Context
1.0
Ri (013
Context
392
240
200
336
0066
Details
336
00.
432
432
0.2
-0.0
16
24
32
48
0.0
Approximations
Details
Horizon
Horizon
Chronos
Wave (Ours)
0.8
2.00
96
0.7
1.75
0.6
0000 600 1..fr
144
Context
192
Miti0 (000)
Context
240
288
0.4
268
1.00
0..3
0.75
384
-0.2
204
0.50
480
480
16
24
32
40
48
56
64
56
Approximations
Horizon
Horizon
Chronos
WaveToken (Ours)
48
1.0
Nnprodmmation
1.4
144
10V0 66 21 (aid
Context
Sonuoa (000
Context
240
0.6
240
Aitectionn
336
-0.4
Detalls
0.6
432
480
-0.0
16
48
Approximations
Horizon
Horizon
Figure 14: Cross-attention maps for the ninth twelfth (from top) decoder layers of Chronos- Base (left) and WaveToken-Base (right) when forecasting the spiky data of the second row in Figure 1.
24

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Gruund Truhh
Mediae
20%
Australian Electricty
CIF
2000
5000
833
02:28 00
03-28 12
03-23 06
00-20
90-11
00-11 12
06-01.00
24-0112
$6-22.00
1475-41
1976-07
1977-81
1977-47
197--21
197607
1676-1
1976-67
1985-01
Parts
Covid Deaths
1.0
1933
0.4
0.0
22353
20000
18993
19983
12003
1992 1992-10 1992 1023-112880 1992 2992-11-188 1992 1992-11 10-0-11-000 1992-11 1992-11-17
08:27
08-2113
092812
09:29
00-2112
03.30
20-3212 083180
ETT Min)
ETT (Hourly)
052282
06-2000 06-25 85-2515
06051
35-26.00
86-22.0
06-22
66-3400
06-25.0
86-2512
Electricity (18 Min.)
Electricity (Hourly)
33-
Arm
15-
13-
Electriiity (Weekhy)
Exchange
10000
0.93
2014-06
2014--7
2016-33
5814-09
2014-10
2314-11
2016-12
2010-29
2016-10
2018-11
2016-12
2019-21
2019-02
FRED-MD
Hospital
15
19900
13-
13000
201.
2013
Figure 15: Example of forecasts from WaveToken-Base -ase the test datasets used in experiments.
25

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
230
190
2019 242319 23 232819 es 252319 sa 272319 ts zezene 27a81s 300818 es 312018 04
19:00
1912
02 2092
20
02 21
21
02
02
MI (Monthy)
MI (Quarterly)
1000
50832
800
600
30002
400
23802
1963
2001
146
(Yearly)
(Monthay)
3.50
1976
1984
1988
1998
2000
MB (Quanerly)
MB (Yearly)
6000
5932
2092
1907
1968
1464
1942
1942
1982
1944
1946
1988
1490
M4 (Daily)
(Iteurly)
232
703
592
(Montht)
(Quarterly)
14000
1902
12000
1993
10000
1203
1992
2011
2312
2513
2014
2055
2514
2009
2370
2011
2012
2013
2514
2514
2074
MM (Weekty)
(Yearty)
20000
6333
34000
Figure 16: Example of forecasts from WaveToken-Bass the test datasets used in experiments.
26

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
2018 912079
2010 012010 0415
2010 012018
2016 DO 212318
1997-11
1997
7999
1999
1998
1998
1998
NNG (Woskdy)
Pedestrian Dounts
190
140
1.00
200
1998-32
9-13 20-12-30221111
18-12-002318-
18-12-11 18-11-112201--12222
Texi (30 Min.
12-15.00
13-15 12-1880
12-1830
12-1812
01:28
01-2812
cn-d012
01-a0
01-40
01-41
Temperatura-Raa
Tourtem (Menthly)
MM
w
30003
10003
2017 on 02017 01:15 2017 91777 1977 09 02077 23 15 2017 0e 02017 04
7999
2000
2007
2002
2005
2004
2005
2009
2317
Touriem (Quartetty)
Touriam (Yearly)
40000
19000
12500
10000
30000
Traffic
Uber (Daily)
155
3.00
13:21.0
12-2312
13-23 12:29
12:30
12:0012
12-3103
12-3112
01:1000
2015:00-03 2015-06-00 201506-13 2015-06-77 2115-6621 2015-26-25 2315-18250741
Uber TLC (Hourhy)
Weather
1.0
0.8
0.4
0.0
1981-57-9000-57-15
Figure 17: Example of forecasts from WaveToken-Ban on the test datasets used in experiments.
27

---

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Table 9: Details of all datasets used for experiments, as collected by Ansari et al. (2024), partitioned according to how they are used for training and evaluation of Wokenoken models.
Series Length
Prediction
Dataset
Domain
Freq. Num. Series
min
avg
max
Length (H)
Pretraining-only
Solar (5 Min.)
energy
5min
5166
105120
105120
105120
Solar (Hourly)
energy
1h
5166
8760
8760
8760
Spanish Energy and Weather
energy
lh
66
35064
35064
35064
Taxi (Hourly)
transport
2428
734
739
USHCN
nature
ID
6090
5906
38653
59283
Weatherbench (Daily)
nature
ID
225280
14609
14609
14610
Weatherbench (Hourly)
nature
Ih
225280
350633
350639
350640
Weatherbench (Weekly)
IW
225280
2087
2087
2087
Wiki Daily (100k)
100000
2741
Wind Farms (Daily)
energy
ID
337
71
354
366
Wind Farms (Hourly)
energy
1h
337
1715
8514
8784
In-domain evaluation
Electricity (15 Min.)
energy 15min
370
16032
113341
140256
24
Electricity (Hourly)
energy
Ih
26304
26304
26304
Electricity (Weekly)
energy
156
KDD Cup 2018
nature
1h
270
9504
10897
10920
48
London Smart Meters
energy
30min
5560
288
29951
39648
48
M4 (Daily)
various
ID
4227
107
2371
9933
14
M4 (Hourly)
Ih
414
748
901
1008
(Monthly)
various
48000
234
2812
M4 (Weckly)
various
IW
359
93
1035
2610
13
Pedestrian Counts
transport
lh
66
576
47459
96424
48
Rideshare
transport
Ih
2340
541
541
S41
24
Taxi Min.)
transport
30min
2428
1488
Temperature-B
nature
32072
725
725
Uber TLC (Daily)
transport
ID
262
181
181
181
7
Uber TLC. (Hourly)
transport
lh
262
4344
4344
4344
24
Zero-shot evaluation
Australian Electricity
energy
30min
5
230736
231052
232272
48
CIF 2016
banking
IM
Car Parts
retail
2674
51
Covid Deaths
healthcare
ID
266
212
212
212
30
Dominick
retail
ID
100014
201
296
399
8
ERCOT Load
energy
1h
8
154854
154854
154854
24
ETT (15 Min.)
energy
15min
69680
69680
69680
ETT (Hourly)
energy
Ih
17420
17420
17420
Exchange Rate
finance
IB
8
7588
7588
7588
30
FRED-MD
economic
IM
107
728
728
728
12
Hospital
healtheare
IM
767
84
84
MI (Monthly)
various
90
MI (Quarterly)
various
3M
203
18
48
114
8
MI (Yearly)
various
IY
181
15
24
58
6
M3 (Monthly)
various
IM
1428
66
117
144
M3 (Quarterly)
various
3M
756
48
72
8
M3 (Yearly)
various
645
47
M4 (Quarterly)
various
3M
24000
24
100
874
8
M4 (Yearly)
various
IY
23000
19
37
841
6
M5
retail
30490
124
1562
1969
28
NNS (Daily)
finance
791
791
NNS (Weekly)
finance
IW
113
113
8
Tourism (Monthly)
various
IM
366
91
298
333
24
Tourism (Quarterly)
various
IQ
427
30
99
130
8
Tourism (Yearly)
various
IY
518
24
47
4
Traffic
transport
Ih
862
17544
17544
17544
Weather
nature ID
3010
1332
14296
65981
30
28


Reviews:
[{"review_id": "Ai7Zhis8KH", "reviewer": "Reviewer_BBt2", "summary": "The manuscript describes a method to tokenize univariate time series data using wavelet transform. The proposed method consists of discrete wavelet transformation followed by a quantization, which generates tokens equal size to the length of input data. Then, the model is trained to forecast the wavelet coefficients in the forecasting horizon. It is shown that the proposed method achieves comparable accuracy with other state-of-the-art models.\n\n## update after rebuttal ##\nI believe that the authors tried to address my concerns. I updated the score as I believe the manuscript is worth reporting in ICML.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments that have helped us improve our manuscript. Below we provide point-by-point answers to each section in your review.\n\n---\n> why not train Chronos with WaveToken?\n\nChronos itself is based on the T5 architecture (Ansari et al. (2024)) and we use the same model hyper-parameters (Table 8, Appendix H in our paper). Thus, Chronos effectively serves as an ablation of WaveToken with respect to a different tokenization method (wavelet tokenizer vs simple discretization). Hence, the performance gain of WaveToken can be attributed entirely to the wavelet tokenizer.\n\n---\n> The experiments do not include some of the best performing models, such as TabPFN-TS, TTM, and MOMENT.\n\nThank you for these suggestions. We ran preliminary experiments for the pre-trained TTM-R2 and TimeMOE models (also suggested by Reviewer Yi8n). TTM-R2 and TimeMOE are point forecasters, so the WQL metric is not meaningful for them.\n| Model | Bench. I (MASE) | Bench. II (MASE) |\n|--|--|--|\n| TTM-R2 | 1.029 | 1.114 |\n| TimeMOE | 0.887 | 0.973 |\n| WaveToken (Large) | 0.698 | 0.810 |\n\nBoth TTM-R2 and TimeMOE exhibit significantly inferior performance compared to WaveToken. These findings are consistent with the limited zero-shot capabilities of these models noted on benchmarks like GIFT-Eval.\n\n- MOMENT: We faced some challenges setting up experiments correctly with its linear probing setup. We will finalize the experiments for the final manuscript. We note that while MOMENT has performed well in contexts beyond forecasting (e.g., classification and anomaly detection), there isn’t enough evidence in existing benchmarks showing that MOMENT is a state-of-the-art zero-shot forecasting model.\n\n- TabPFN-TS: The paper and codebase was released publicly on Jan 5, 2025. This falls well within the four months time frame specified by the concurrent work guidelines for ICML 2025. Nevertheless, we thank the reviewer for this suggestion and will include the results for TabPFN-TS in the final version of our paper. \n\n---\n> I find that a few notations are used without clearly defined.\n\nWe agree that some of the notation is not clear. Thank you for pointing this out. We will make sure to update the final version of the manuscript with the clarifications below:\n- In $[a_k]_J$, the symbol $k=1,\\dots,K$ indexes approximation coefficients from the DWT. For an input of size (say) 512, we get about $K=256$ approximation coefficients (depending on the wavelet basis).\n- In $F^{−1}_{|d_j|} (b^{J−j+1})$, $F^{-1}$ is the (empirical) inverse-CDF of the absolute values of the detail coefficients $|d_j|, \\forall j=1,\\dots,J$. The thresholding percentile $b^{J−j+1}$ grows exponentially in the decomposition level so that finer detail coefficients, which tend to capture more noise, are thresholded more aggressively. We set $b=2$\n- $z_{1:C}$ denotes the concatenated wavelet coefficients after decomposing the time series context. \n- $z_{C+1:C+H}$ denotes the coefficients obtained from the time series horizon at training time, which are used to compute the loss.\n\n---\n> Can you be more clear about compactness in what sense?\n\nThe “compactness” of wavelets manifests itself in two ways:\n1. Wavelets concentrate most of the signal energy (input variance) on a few coefficients of high magnitude. Thus, after thresholding the effective number of non-zero coefficients can be much lower than the size of the input\n2. Empirically, WaveToken has excellent performance while using a much smaller vocabulary (1024 tokens) relative to Chronos (4096 tokens), which instead directly quantizes the time series.\n\n---\n> I guess w∈[[ak]J,[dk]j] should be w∈[[ak]J,[dk]j=1J]. Right? Anyway, w is a vector, e.g., [ak]J Then how do you quantize a vector?\n\nThanks for catching the typo! Yes, $j=1,\\dots,J$ is correct. Also, note $w$ is a single coefficient, without distinguishing between approximation and detail. In other words, we quantize all the coefficients element-wise by assigning each to bins based on their empirical distribution.\n\n---\n> Also, in the quantization, how did you decide the bin size that can be universally applied to diverse data set?\n\nWe construct the vocabulary by scanning the training set and choosing the optimal bin size according to Freedman and Diaconis (1981), which selects the bin width minimizing the reconstruction error. Clearly, the universality of this binning scheme relies on having a representative training corpus, the reason why we trained WaveToken on 28 different real-world datasets from different domains. \n\n---\nThanks for pointing out the reference to Yeo and Melnyk (JCP 2019) regarding prior quantization approaches. We’ll add this reference in the final version.\n\n---\nThank you again for your time and engagement in the review process. We hope our responses above have satisfactorily addressed your concerns. If so, we request you to consider raising your score to reflect that. If you have further questions, we would be happy to respond to them.", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "The proposed method is relatively straightforward. While it looks like there are some limitations, such as an extension to multi-variate time series and variable length forecast, I think the proposal is interesting and worth reporting to the ML community. The evaluation criteria also make sense.", "theoretical_claims": "This paper is empirical.", "experimental_designs_or_analyses": "It should be noted that the experimental results are obtained by training T5 with the features obtained by WaveToken. Even though, the model achieved good accuracy, it is unclear if it is due to the use of WaveToken or due to T5. A more straightforward approach would be training one of the models used in the benchmark with WaveToken and compare accuracy. For example, Chronos is anyway trained with the cross-entropy loss. Then, why not train Chronos with WaveToken?\n\nThe experiments do not include some of the best performing models, such as TabPFN-TS, TTM, and MOMENT.", "relation_to_broader_scientific_literature": "Time series tokenization is of current interest, not only because it allows use of LLM for time series problems, but also because it has shown that tokenizing time series seems to help the models better learn the representation. I believe that the idea of wavelet-based tokenization is timely.", "essential_references_not_discussed": "The idea of quantizing time series data was proposed long before Ansari et al (2024). It was proposed in Yeo and Melnyk (JCP 2019) with a method to impose a structural constraint in the distribution.", "other_strengths_and_weaknesses": "Strength:\nThe paper is well written and the experiments are well designed to demonstrate the strength of the proposed method.", "other_comments_or_suggestions": "I find that a few notations are used without clearly defined. For example, there is $[a_k]_J$ on page 3. What is the dimension of $k$? On page 4, in CDF-thresholding, What is $F$ and what is $b^{J-j+1}$? $z_{1:C}$ is defined, but what is $z_{C+1:C+H}$? While it may not be difficult to guess what they are, I would suggest to proofread the manuscript and clearly define every mathematical notations to improve clarity.", "questions_for_authors": "1. In the first paragraph in page 4, it is mentioned that \"wavelet transform leads to compact representations\". But as you mentioned in the following sentence, \"a signal of size N results in N coefficients\", which does not look like a compact representation. Can you be more clear about compactness in what sense?\n\n2. I think it will be helpful to elaborate the quantization, probably adding more details in SM. In quantization section, I guess $w \\in [[a_k]_J,[d_k]_j]$ should be $w \\in [[a_k]_J, [d_k]\\_{j=1}^J]$. Right? Anyway, $w$ is a vector, e.g., $[a_k]_J$. Then how do you quantize a vector?\n\n3. Also, in the quantization, how did you decide the bin size that can be universally applied to diverse data set? The bin size essentially determines the approximation accuracy. When the data generating distribution changes, one set of bins that is optimal to one data set may perform terribly for another data set.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Zk7IiAqmE", "reviewer": "Reviewer_6tR2", "summary": "This paper introduces WaveToken, a wavelet-based tokenization method for time series forecasting. It decomposes time series into wavelet coefficients, which are then used to autoregressively predict future values. The method involves standardizing, decomposing, thresholding, and quantizing the coefficients, and training with cross-entropy loss.WaveToken performs well across various datasets, especially in zero-shot settings, using a smaller vocabulary while achieving competitive forecasting performance. It effectively handles complex patterns like non-stationary signals and trends, offering an efficient approach to time series forecasting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for these useful comments and questions that helped improve our paper. Below we provide point-by-point answers to each section in your review.\n\n---\n> I would like to know if the proposed method can handle variable-length inputs and outputs with a single model.\n\nThe current implementation can readily handle variable-length inputs and outputs. More specifically:\n- At training time, any input of size up to 512 and output of size up to 64 can be processed directly. If the inputs or outputs are shorter, we simply pad them with NaNs to ensure consistency of the shapes. In practice, we observed that these maximum lengths cover the vast majority (if not all) of practical applications. Note that other popular foundation models for time series forecasting adopt the same strategy, e.g. Chronos by Ansari et al. (2024) and TimesFM by Das et al. (2024).\n- At test time, the model can provide forecasts of unlimited length due to its autoregressive nature.\n\n---\n> Regarding thresholding, are the filtered coefficients set to zero or completely removed? In other words, does the number of input coefficients remain C? Additionally, how are excessively large coefficients handled?\n\nThe filtered coefficients are set to zero. Removing them would lead to variable-length coefficient vectors and a potential mis-representation of different groups of coefficients. In other words, since the model processes concatenated groups of wavelet coefficients, it is important to keep the length of each coefficient group fixed (i.e. no removals) so that the model can learn to forecast (and attend to) approximation coefficients differently from details coefficients.\n\n---\n> Is there a possibility to allow the model to dynamically select the thresholding method based on the data characteristics?\n\nAll the thresholding methods we tested do indeed try to adapt to the data characteristics. CDF-thresholding, for example, directly looks at the empirical distribution of the detail coefficients (in absolute value) to determine the cutoffs. VisuShrink, on the other hand, applies a threshold based on an estimate of the variance of the input time series. In general, one could in principle choose entirely different thresholding methods for different datasets or domains. Although we have not implemented this feature yet, it represents an interesting area for future work. Thank you for raising this point!\n\n---\n> From both theoretical and experimental perspectives, why is quantization necessary?\n\nand\n\n> Ablation on quantization ia required, i.e. directly usinig the raw coefficients without as quantization input and perform continuous autoregressive with MSE loss.\n\nThe quantization step is not strictly necessary to apply the ideas presented in our paper. However, the primary focus of this work was to improve tokenization in the context of time series foundation models with _discrete vocabularies_. Therefore, we need the quantization step to construct a finite vocabulary of tokens that can be directly processed by the T5 architecture without substantial modifications. We focus on models with discrete vocabularies because they present an exciting alternative method to address forecasting problems compared to traditional regression-based approaches and have demonstrated superior performance in prior works. \n\nIn principle, one could also sidestep the quantization process by feeding the raw coefficients as embeddings to the transformer, or, as you suggested, by learning to forecast the raw coefficients directly via a continuous-input loss such as MSE/MAE. Both these approaches are perfectly valid ideas. However, they entail completely different models with different analyses and are better suited for independent future works rather than ablations of WaveToken. It would also be interesting to explore ideas based on Wavelets in the context of models that operate on patches (e.g., TimesFM and Moirai). We are hopeful that the community will build upon our work to investigate these ideas. \n\n---\n> Do outputs share the same quantization bins with inputs?\n\nYes, the vocabulary is shared between inputs and outputs. Both are mapped to and from the same bins using the procedure described in Section 2.2 - Quantization.\n\n---\nThank you again for your time and engagement in the review process. We hope our responses above have satisfactorily addressed your concerns. If so, we request you to consider raising your score to reflect that. If you have further questions, we would be happy to respond to them.", "claims_and_evidence": "Claims are well supported.", "methods_and_evaluation_criteria": "Overall, the methods and evaluation are convincing. I have a few questions as follows:\n1. Since wavelet transformation is closely related to the input length, and the experiments were conducted under a fixed setting of input 512-output 64, I would like to know if the proposed method can handle variable-length inputs and outputs with a single model.\n\n2. Regarding thresholding, are the filtered coefficients set to zero or completely removed? In other words, does the number of input coefficients remain $C$? Additionally, how are excessively large coefficients handled?\n\n3. The paper tested various thresholding methods and provided comprehensive experimental results, with different model settings requiring different thresholding techniques. Is there a possibility to allow the model to dynamically select the thresholding method based on the data characteristics?\n\n4. From both theoretical and experimental perspectives, why is quantization necessary? This is because we could also directly use the raw coefficients as input and perform autoregressive prediction using continuous loss functions such as MSE or MAE.\n\n5. Do outputs share the same quantization bins with inputs?", "theoretical_claims": "Not applicable, no new theoretical claims is proposed.", "experimental_designs_or_analyses": "Ablation on quantization ia required, i.e. directly usinig the raw coefficients without as quantization input and perform continuous autoregressive with MSE loss.", "relation_to_broader_scientific_literature": "The objective of this paper is to enhance efficiency and generalization of time-series foundation model by using wavelet-based discrete tokenization.", "essential_references_not_discussed": "Not applicable, references are generally comprehensive.", "other_strengths_and_weaknesses": "The structure of the paper is very clear and easy to follow.", "other_comments_or_suggestions": "See my comments and suggestions above.", "questions_for_authors": "1. See above questions in **Methods And Evaluation Criteria** and **Experimental Designs Or Analyses**.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OWtJu4Ef8J", "reviewer": "Reviewer_QuAc", "summary": "This paper introduces WaveToken, a wavelet-based tokenization framework designed to enhance foundation models for time series forecasting. The approach leverages the multi-resolution properties of wavelets to transform real-valued time series into compact, expressive token sequences, enabling efficient learning of temporal patterns across diverse domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for these useful comments and questions that helped improve our paper. Below we provide point-by-point answers to each section in your review.\n\n---\n> The code is not publicly available, hindering replication.\n\nWe plan to release a user-friendly research package complete with details on how to train and evaluate our tokenizer and models. Unfortunately, we could not share our code for review at this stage due to pending legal approvals. \n\n---\n> The autoregressive nature of the T5 architecture paired with wavelet tokenization may lead to slower inference times compared to non-autoregressive models (e.g., patch-based approaches like TimesFM). This could hinder real-time applications requiring rapid predictions.\n\nWe agree that competitive inference times are paramount to achieve an effective utilization of these models in real-time application. We note that in practice the wavelet tokenizer does not influence inference speed and adds only a minimal overhead, since it is implemented as a convolution-and-downsampling operation which runs in linear time. Below we report running times (averaged over 10 repetitions) for both Chronos (Base) and WaveToken (Base) — which use the same underlying architecture and only differ in the tokenization pipeline — when forecasting a batch of 32 time series, with context_length=512 and prediction_length=64.\n- Chronos: 6.56s +/- 1.02ms\n- WaveToken: 6.86s +/- 1.14ms\n\nThe difference in inference times is negligible, and could be further reduced by applying the wavelet decomposition in parallel on the GPU.\n\nSimilar to Chronos, what limits inference speed for WaveToken is the autoregressive nature of forecast generation. Crucially, this is the reason why patch-based methods such as TimesFM are faster than WaveToken. It would be interesting to apply ideas from our work to patch-based architectures as part of future work. In this way, one would retain the expressivity of the time-frequency domain while achieving significant speed improvements.\n\n---\n> The choice of wavelet family (e.g., Biorthogonal-2.2) and thresholding method (e.g., no-thresholding) is task-dependent and may not universally optimize performance across all datasets. Hyperparameter sensitivity could limit reproducibility in diverse real-world scenarios.\n\nAs detailed in Section 3.4, we choose hyperparameters by analyzing their probabilistic and point forecasting performance across all the tasks in the in-domain and zero-shot benchmarks, not just on a specific task/dataset. These benchmarks are already quite large (42 datasets in total), hence they ensure that the chosen hyperparameters indeed provide optimal performance for a variety of real-world domains, seasonalities and frequencies. These choices lead to superior generalization performance, as it can be seen from the results on the Zero-Shot benchmark of Figure 3 (panel B), where WaveToken even outperforms (or is competitive with) task-specific models that were trained separately on each single dataset.\n\n---\n> While WaveToken performs well for H×2 horizons, its WQL performance slightly lags behind TimesFM for H×3 forecasts. This suggests potential limitations in extrapolating patterns over very long horizons, which may require architectural adjustments (e.g., hierarchical attention).\n\nWe note that TimesFM seems to outperform WaveToken for $H \\times 3$ horizons only with respect to weighted quantile loss, which measures the accuracy of probabilistic forecasts. On the other two metrics, namely mean absolute scaled error and visual relative squared error (VRSE), WaveToken outperforms all other models.\n\n---\nThank you again for your time and engagement in the review process. We hope our responses above have satisfactorily addressed your concerns. If so, we request you to consider raising your score to reflect that. If you have further questions, we would be happy to respond to them.", "claims_and_evidence": "Claims about wavelet decomposition provides a compact and expressive representation are supported by the superior performance of WaveToken on 42 datasets, particularly in capturing complex patterns like exponential trends and sparse spikes (Figure 1). Meanwhile, the authors claim that WaveToken generalizes well to unseen datasets in both in-domain and zero-shot settings. This is supported by empirical results showing WaveToken outperforming state-of-the-art models like Chronos and TimesFM.", "methods_and_evaluation_criteria": "The proposed wavelet decomposition, thresholding, and quantization pipeline is justified for encoding temporal and frequency information. The metrics (WQL, MASE, VRSE) cover probabilistic accuracy, point forecasting, and frequency-domain fidelity.", "theoretical_claims": "The paper focuses on empirical validation rather than theoretical proofs. Key claims (e.g., wavelet sparsity, multi-scale learning) are supported by prior wavelet theory (e.g., Mallat, 2009) but lack formal derivations.", "experimental_designs_or_analyses": "Ablation studies (vocabulary size, wavelet family, decomposition level) systematically validate hyperparameters. Results are averaged over three seeds, and comparisons include both foundation models and task-specific models. However, the use of T5’s encoder-decoder architecture may introduce bias in autoregressive performance.", "relation_to_broader_scientific_literature": "For Wavelet forecasting, this paper extends prior work (e.g., Sasal et al., 2022) by integrating wavelets into a foundation model framework.", "essential_references_not_discussed": "For non-autoregressive models, PatchTST (Nie et al., 2022) is cited but not discussed in the context of inference speed.", "other_strengths_and_weaknesses": "1. This paper proposes an innovative Wavelet-based tokenization, which utilizes Wavelets decompose time series into hierarchical frequency components (approximations and details), enabling the model to capture both global trends and local patterns efficiently.\n2. The proposed methos can achieve high expressiveness via sparse wavelet coefficients while reducing computational overhead, which uses only 1024 tokens (1/4 of Chronos), .\n3. The attention mechanism analysis reveals the model’s ability to exploit wavelet coefficient hierarchies, enhancing transparency.", "other_comments_or_suggestions": "The code is not publicly available, hindering replication.\n\n## update after rebuttal\nI appreciate the authors' efforts in addressing my concerns, particularly regarding the inference efficiency, the rationale behind wavelet tokenizer design choices, and the scope of zero-shot generalization. I believe the authors have addressed my questions, and I maintain my original score.", "questions_for_authors": "1. The autoregressive nature of the T5 architecture paired with wavelet tokenization may lead to slower inference times compared to non-autoregressive models (e.g., patch-based approaches like TimesFM). This could hinder real-time applications requiring rapid predictions.\n2. The choice of wavelet family (e.g., Biorthogonal-2.2) and thresholding method (e.g., no-thresholding) is task-dependent and may not universally optimize performance across all datasets. Hyperparameter sensitivity could limit reproducibility in diverse real-world scenarios.\n3. While WaveToken performs well for H×2 horizons, its WQL performance slightly lags behind TimesFM for H×3 forecasts. This suggests potential limitations in extrapolating patterns over very long horizons, which may require architectural adjustments (e.g., hierarchical attention).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2UG7NRkwGD", "reviewer": "Reviewer_Yi8n", "summary": "To build an effective discrete vocabulary for a real-valued sequential input, this paper develops WaveToken, a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. The proposed method performs well while using a much smaller vocabulary and exhibits superior generalization capabilities.\n\n## update after rebuttal\nThank you very much for your responses. I would like to keep my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for these useful comments and questions that helped improve our paper. Below we provide point-by-point answers to each section in your review.\n\n---\n**Additional baselines**\n\nBased on your and Reviewer BBt2’s suggestion, we evaluated TimeMOE and TTM-R2 models. WaveToken clearly outperforms these models on both benchmarks. See our reply to Reviewer BBt2 for specific details.\n\n*Task-specific baselines*: We already reported results for TFT and PatchTST, which are known state-of-the-art models. Recent [independent work](https://arxiv.org/abs/2502.14045) found PatchTST performs comparably to iTransformer on many datasets. Additional task-specific results are also available in Ansari et al. (2024). We thank you for suggesting iTransformer, but due to the limited rebuttal time, we couldn’t finalize experiments on all 42 datasets. We’ll include these results in the final manuscript.\n\n--- \n> For in-domain settings, why WaveToken only performs better than Chronos on VRSE?\n\nComparisons with Chronos have to be made by looking at model pairs of the same size. Doing so, WaveToken achieves lower (i.e., better) scores than Chronos 75% of the times across all metrics, and achieves the best average rank across all metrics for both Benchmarks I and II (Figures 9 and 10).\n\nWaveToken’s superiority in VRSE specifically arises from its wavelet decomposition, explicitly capturing complex time-frequency structures. VRSE measures forecast-truth similarity in terms of frequency magnitude, aligning naturally with wavelet-based tokenization.\n\nThe core motivation of our work is to develop a general purpose tokenizer that seamlessly captures global and local patterns. Qualitative results (Figure 1 and Section 3.3) demonstrate the impressive performance of WaveToken on complex edge cases that are pervasive in practical applications, where existing foundation models fail almost completely. \n\n---\n> For zero-shot settings, PatchTST performs on par with WaveToken on MASE and TFT performs on par with WaveToken on WQL, but they lag far behind on other metrics. What causes this difference?\n\nThis is due to the original design principles of these models. PatchTST was originally a point forecaster. We use the implementation from GluonTS, which adapts PatchTST for probabilistic forecasting. However, it is possible that some design elements of PatchTST are better suited for point predictions, hence its superior performance on the MASE. Similarly, TFT was originally designed to be a probabilistic forecaster trained with quantile regression, which is similar to the WQL up to scaling. \n\nNeither PatchTST nor TFT directly aim at producing forecasts whose frequency content closely matches that of the true time series, which is the target property of the VRSE metric.\n\n---\n> [...] in zero-shot settings with VRSE as metric, a small size model WaveToken(Small) is much better than a large size model WaveToken(Base), can you provide some discussion on it?\n\nWe indeed noticed this pattern as well while conducting experiments. Chronos (Ansari et al. (2024)) exhibits a similar phenomenon. Both Chronos and WaveToken share the same architecture (T5), so the answer might lie in this model. However, we do not currently have a definitive explanation.\n\n---\n> Can you provide some detailed comparison results about inference time?\n\nThe wavelet tokenizer does not influence inference speed and adds only a minimal overhead, since it is a linear-time convolution-and-downsampling operation. Inference times (average of 10 runs, batch of 32 series, context=512, horizon=64) for both Chronos Base and WaveToken Base — which only differ in the tokenizer — are reported below. The difference is negligible, and could be further reduced by applying the DWT on a GPU.\n- Chronos: 6.56s +/- 1.02ms\n- WaveToken: 6.86s +/- 1.14ms\n\nThe main bottleneck remains the autoregressive T5 structure.  Future work could explore integrating wavelets with PatchTST or TimesFM architectures so that they can process wavelet coefficients and retain the expressivity of the time-frequency domain, while achieving significant speed improvements.\n\n---\n> Please provide the visualization results on some benchmark datasets to make the evaluation results in Section 3 more persuasive.\n\nFigure 1 demonstrates WaveToken’s superiority over existing models on controlled practical scenarios (e.g., non-stationarity, sparse spikes). Due to the tight rebuttal deadline, we prioritized additional baseline evaluations requested by Reviewers BBt2 and Yi8n. We'll include further visualizations on a diverse set of real-world datasets in the final manuscript, so as to strengthen the qualitative evaluation of Section 3 as you suggested. \n\n---\nThank you again for your time and engagement in the review process. We hope our responses above have satisfactorily addressed your concerns. If so, we request you to consider raising your score to reflect that. If you have further questions, we would be happy to respond to them.", "claims_and_evidence": "n/a", "methods_and_evaluation_criteria": "This paper conducts experiments on many datasets for evaluation. These datasets make sense for the problem or application at hand.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "Baselines:\n1. Please provide the comparison with some latest time series forecasting foundational models, such as Timer [1] and TimeMoE [2].\n2. This paper claims that the proposed method is better than some task-specific models. To better support this claim, can you provide the comparison with more latest task-specific models, such as ModernTCN [3], iTransformer [4]?\n\n[1] Timer: Generative Pre-trained Transformers Are Large Time Series Models\n\n[2] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts\n\n[3] ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis\n\n[4] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting\n\nMetrics:\n\nThe evaluation results vary under different metrics. To make the results persuasive, please provide more discussion on the evaluation metrics.\n1. For in-domain settings, why WaveToken only performs better than Chronos on VRSE?\n2. For zero-shot settings, PatchTST performs on par with WaveToken on MASE and TFT performs on par with WaveToken on WQL, but they lag far behind on other metrics. What causes this difference?", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n1. This paper develops WaveToken, a novel wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies.\n2. The proposed method performs well while using a much smaller vocabulary and exhibits superior generalization capabilities.\n\nWeaknesses:\n1. More discussion on the evaluation metrics is needed. Please see Experimental Designs Or Analyses.\n2. Efficiency. In line 435, the authors point out the slower decoding and inference time as a limitation of the proposed method.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "Scaling:\n\nFor the relationship between model size and model performance, most of the results are as expected. But in zero-shot settings with VRSE as metric, a small size model WaveToken(Small) is much better than a large size model WaveToken(Base), can you provide some discussion on it?\n\nEfficiency:\n\nIn line 435, the authors point out the slower decoding and inference time as a limitation of the proposed method. Can you provide some detailed comparison results about inference time? This can give the reader a fuller understanding of the advantages and disadvantages of the proposed method.\n\nVisualization:\n\nFigure 1 only shows the visualization results on synthetically generated time series. Please provide the visualization results on some benchmark datasets to make the evaluation results in Section 3 more persuasive.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to tokenize univariate time series data using wavelet transform. It decomposes time series into wavelet coefficients, which are used autoregressively to predict future values. The idea was deemed interesting and straightforward but lacks a formal derivation. While the results are systematic and extensive, there are concerns regarding the key element that contributes to the success of the method, given the reliance on T5 architecture.

Despite these concerns, the authors replied to most of the issues raised by the reviewers. The remaining concerns raised by the reviewers are minimal, and while the paper could benefit from more descriptions, the existing contribution will be beneficial for the community.

Thus, I recommend its acceptance.
Decision: Accept (poster)