SimpleTM: A Simple Baseline for Multivariate Time Series Forecasting
Authors: Hui Chen
Viet Luong
Lopamudra Mukherjee
Vikas Singh
Venue: ICLR 2025 Poster (2025)
Keywords: Time Series Forecasting
Wavelets
Primary area: learning on time series and dynamical systems

Abstract: The versatility of large Transformer-based models has led to many efforts focused on adaptations to other modalities, including time-series data. For instance, one could start from a pre-trained checkpoint of a large language model and attach adapters to recast 
the new modality (e.g., time-series) as ``language''. Alternatively, one can use a suitably large Transformer-based model, and make some modifications for time-series data. These ideas offer good performance across available benchmarks. But temporal data are quite heterogeneous (e.g., wearable sensors, physiological measurements in healthcare), and unlike text/image corpus, much of it is not publicly available. So, these models need a fair bit of domain-specific fine-tuning to achieve good performance -- this is often expensive or difficult with limited resources. In this paper, we study and characterize the performance profile of a non-generalist approach: our SimpleTM model is specialized for multivariate time-series forecasting. By simple, we mean that the model is lightweight. It is restricted to tokenization based on textbook signal processing ideas (shown to be effective in vision) which are then allowed to attend/interact: via self-attention but also via ways that are a bit more general than dot-product attention, accomplished via basic geometric algebra operations. We show that even a single- or two-layer model gives results that are competitive with much bigger models, including large transformer-based architectures,
on most benchmarks commonly reported in the literature.

Published as a conference paper at ICLR 2025
SIMPLETM: A SIMPLE BASELINE FOR MULTIVARIATE TIME SERIES FORECASTING
Hui Chen' Viet Luong'
Lopamudra Mukherjee Vikas Singh'
University of Wisconsin-Madison Uniierriity of Wisconsin- Whitewater
(chen7955wise edu mukher @www.edu
vhluong@wisc. edu vs vsingh@biostat. wwnghheiosttt..wss...om wws..wsc.... edu
ABSTRACT
The versatility of large Transformer-based models has led to many efforts focused on adaptations to other modalities, including time-series data. For instance, one could start from pre-trained checkpoint of a large language model and attach adapters to recast the new modality (e.g., time-series) as "language". Alterna- tively,
use suitably large Transformer-basee model, and make some modifications for time-series data. These ideas offer good performance across available benchmarks. But temporal data are quite heterogeneous (e.g., wearable sensors, physiological measurements in healthcare), and unlike text/image corpus, much of not publicly available. So. these models need a fair bit of domain- specific fine-tuning to achieve good performance this is often expensive or dif- ficult with limited resources. In this paper, we study and characterize the perfor- mance profile of a non- non-generaliit approach: our Simple'IM model is specialized for multivariate time-series forecasting. By simple, we mean that the model is lightweight. It is restricted to tokenization based on textbook signal processing ideas (shown to be effective vision) which are then allowed to attend/interact: via
but also via ways that bit more general than dot-product at- tention, accomplished via basic geometric algebra operations. We show that even single- or two-layer model gives results that are competitive with much big- ger models, including large transformer-based architectures, on most benchmarks commonly reported in the literature.
INTRODUCTION
Multivariate time-series (MTS) data are ubiquitous in various disciplines such as finance and eco- nomics Andersen (2006). climate science Mudelsee (219)) healthcare Zeger al. (2006), geophysics Gubbins (2004), and industrial monitoring Truong et al. (2022). Consequently. MTS data processing and analysis techniques have been extensively studied, going back to works in vec- tor autoregressive models Lütkepohl (2013), dynamic factor models Molenaar et al. (1922), state- space models |Rangapuram et al. (2018) and others. The literature provides rich theory and various solutions depending the assumptions that make the most sense for the data at hand, e.g., ho- moscedasticity versus heteroscedasticity Rodríguez Ruiz (2005), degree of autocorrelation Bence 1995), and stationarity versus non-stationarity Das & Nason (2016). Such models refined decades are well-studied in economics, computational finance and statistics. While progress in deep learning architectures over the last 10+ years has led to the most significant gains in performance capabilities for tasks involving image and natural language data, there is growing body of literature (discussed below) describing strategies for harnessing these models for multivariate time-series data
Liu et al. (2024a) Huang et al. 2023); Zhang & Yan 2023).
Deep Architectures for MTS data. Most types of widely used deep architectures from con- volutional neural networks LeCun et al. (1998); [Simonyan & Zisserman (2015); He et al. (2016) to graph neural networks (GNN) Kipf Welling (2017); Hamilton et al. (2017) transformers Vaswani et al. (2017); DDellin et al. (2019) Dong et al. (2021) have all been adapted and attempted
for various types of MTS data Bagnall et al. (0088). For instance, Zhang & Yan (2033: Zhou et al.
"Code available GitHub:
http:ttthh..cc...tooeooeoe ngh- group/ooppoo... Srrppp/Sippeell

---

Published as a conference paper at ICLR 2025
(022b): Wu et al. (2021): ; Liu et al. (2024a) use attention mechanism to model the long term interaction between different time points whereas approaches using GNNs Cheng et al. (2022) p Li
et al. 2202); Jin et al. (2023) seek to extract interaction adaptively between different series. However, as noted in Huang et al. al. (2023), all methods face challenges in handling temporal fluctua- tions and heterogeneity between variables (i.e., different time-series in the same data). But perhaps more importantly, there is an immense degree of variability between different MTS datasets. For instance, MTS data from wearable sensors will bear little to no similarity to electroenceepaalggrmm (EEG) recordings of brain activity. We know that internet-scale text and image data corpus have been used, train large language and vision models, where the sheer size of the dataset provides the model some ability to handle heterogeneity. But while the raw sizes of MTS data produced or acquired each day (e.g., in physiological recordings or wearable sensors) is enormous, only a minor fraction of it is publicly available due to strict privacy regulations (HIPAA) or laws surrounding sharing of consumer behavior data or entirely non-legal reasons (proprictary, competitiveness). To summarize, such data remain scarce and thereby, deploying pre- pre-trained models in specific setting involving our own MTS dataset requires care and often significant fine-tuning. In fact, Zeng et al. (2023) found that for number of publicly available datasets, simple one-layer linear model can frequently outperform generic approaches based on Transformers, suggesting that translating the same backbone to complex and heterogeneous MTS data is challenging. Promisingly, in the last year, number of interesting approaches Liu et al. (2024a); Wang et al. (2024); Nie et al. (2023); Chen al.] (2024), have been proposed which make specific adjustmentsmmoddicaiieee
the ar- chitecture to better handle the nuances and complexity of MTS data, and show robust/reproduciblc
performance. Several of these models will serve as our baselines later.
Repurposing LLMs for Time-series data. A related but distinct line of work seeks to re- re-interpret Lnee-srriss data as natural language, and allows operating on top of powerful large language models Jin et al. (2024). Such an approach can benefit from the vast amount of text data the language model has already been trained on, which kept frozen, and one assumes that a mechanism to map chunks of time-series to word embeddings can be estimated based sufficiently large MTS dataset. This mapping is often accomplished by training specialized adapters placed before and after the LLM in the pipeline. This direction is evolving rapidly and providing promising results, but of now, deploying the model on a domain specific dataset with its own specific characteristics of stationarity and seasonality, while possible, remains quite compute intensive.
This work. Our paper aligns more closely with the aforementioned non- gen-generallist approaches in that the intended use of the model will only be for multi-variate time-series data. Instead of modify- ing a large Transformer-basee Similar to LLMs, we also use tokenization but given the well-defined application scope dime-series data), we will use ideas based directly on classical signal processing Haykin & Veen (1998). Then, we borrow the self-attention module and make a small modification to it, so it can capture a richer dependency structure between tokens, allowing it to capture dependencies across-time and across- dimensions. The key contributions of our work are summarized as follows:
backbone, we will add in modules, one by one, quite conservatively.
(i) We propose SimpleTM, a simple yet effective architecture that uniquely integrates classical signal processing ideas with a slightly modified attention mechanism.
(ii) We show that even single-layer configuration can effectively capture intricate dependencies in multivariate time-series data, while maintaining minimal model complexity and parameter require- ments. This streamlined construction achieves performance profile surpassing (or par with) most existing baselines across nearly all publicly available benchmarks.
2 PRELIMINARIES: PROBLEM SETUP AND NOTATIONS
Univariate time-series. Let (z1, L be single historical (or lookback) time-series of length L where &, E I denotes the measurement/vall valuu at the t-th timestep. Let (y1,... (H) be a single time- series of length H in the future. We call H the forecast/horizon window length and L the lookback window length. Time-series forecasting asks if we can predict (.i,...,, ,yH) from (I1, (0,,".,,, ,I)).
Multi-variate time-series. Let X E RCxt, f. and Y RCx f be two matrices, jointly drawn from some distribution P. We also write x, and y, as the t-th column of X and Y respectively. That is, observe L measurements for each of C channels or variables from X. Our goal is to "forecast"
2

---

Published as a conference paper at ICLR 2025
the time-series of H. timesteps, each timestep t in the forecast window is vector of length C, collectively called Y.
Multiple Multi-variate time-series. Denote by N the sample size: the number of vulti-variate time-series we observe. We can use as a generic index for a specific sample for E [M]]
Remark. This multivariate setting captures scenarios where C' different channels or variables in a synchronized manner, which becomes particularly valuable when there correlations dependencies among these variables.
measuring time-series data for
Definition (Forecasting error) Assume a multi-variate time-series (X, Y) ~ P, where X E RCXL and Y RCxH For any mapping f :RCXL + RCXH we call a forecasting function. We define the forecasting error with regards to J as
[()) == PXX I(X)]LFF
where || ||I denotes the Frobeneus norm of the matrix and E is the expectation over the joint distribution P. We are given set of i.i.d. samples ((Xi, Y:)) We define the empirical risk with regard to forecasting function f as
N L(f) = N - f(x
-
Our goal is to optimize over f to minimize the empirical forecasting error. We will now introduce the specific modules in our overall model, after which we will describe the experimental evaluations.
3 MODULE 1: TOKENIZATION VIA STATIONARY WAVELET TRANSFORM
Motivation/Rational For this first module, we seek a tokenization scheme for MTS data that re- lieves the amount of work that the downstream mod- ules need to do discovering all local/global depen- dencies - which raises both the compute footprint and also the sample sizes needed. Ideally, if our tokens could capture temporal information across multiple scales (rapid, short-term variations slow, long-term trends), and capture both local/global pat- terns within each of the C' variables/sites, then the task of exactly how to synthesize this information for forecasting Y would be simplified. If can allow scale-specific processing, then arguably the synthesis task can benefit from the specific modules processing each scale, acting collaboratively.
Lookbuck
Inpat
Lincar Projectinn
Langh
Langth
cluundd
Length z.'
Token
Filter
Learnable
Figure 1:
For each channel, time-series mea- surements are passed through a stationary wavelet transform followed by linear projection to obtain L' tokens.
One possible solution. The reader will immediately see that the Wavelet transform is first principles based solution to the requirements outlined above, and this idea has recently found use processing image data in Transformer models Yao al. 2022); Zhu & Sorieut (2024). It offers multi-scale decomposition of each signal while maintaining temporal localization. We will treat each wavelet scale separately which will allow learning scale-specific interactions within each variable. If our forecast window is dominated by dependencies that are prominent at one scale but not the other, such a construction provides the downstream modules relevant information to operate with.
Details of the construction. We now present our tokenization scheme, Let X = ( , E RCXL denote our multivariate time-series, where C is the number of channels as before.
A) Linear Projection. We first apply a linear projection g(:; 0) RR_ + R to embed each channel into a hidden/latent space which gives X = [w1, ãg, _ g(X).
B) Stationary Wavelet Transform (SWT). To achieve multi-scale representation, we learnable stationary wavelet transformation (SWT). Sws & Silverman (1955) is defined as
SW7(: ho: gn) CxL' Cx x(S+1)

---

Published as a conference paper at ICLR 2025
where ho, Po R are learnable filters with kernel size k, and S is the decomposition level.
This transformation produces a set of time-frequency tokens {uf u!") ,...,, u ) information at different temporal scales for each channel independently, see Figs. D2 SWT is
capturing
suitable since provides a time-invariant decomposition while preserving the original temporal structure. This is achieved by avoiding downsampling at each decomposition level, thus maintaining the up-scaled length/size L'. SWT is also iniat-invariant making it effective in capturing localized events across multiple scales. At the core of the SWT are the mother wavelet (i(t) and scaling function (t)). The family of discrete wavelets can be expressed as:
(t) 2 2-2 k) and (s, 2- kc).
where s controls the scale (dilation) and k determines the position (translation).
Leeph
Inbeted
C) Obtaining Wavelet Coefficients. The embedded time series { undergoes decomposition via the station- ary wavelet transform (SWT), yielding approximation co- efficients a(") and detail coefficients u²²) at each level For clarity, we present the process for univariate series. SWT uses two main filters: a low-pass filter and high- pass filter y. derived from the scaling function d(l) and the wavelet function (i) respectively:
+
Seale Leppennteentt
Lourek
Longl
Figure 2: Tokenization via SWT: The input series is padded and processed through learnable filters. At each scale, SWT inserts zeros between filter co- efficients, producing a non-decimated output. This approach, demonstrated for a scale-2 approximation, allows shift-invariant feature extraction while preserving temporal resolution.
t((k) (o(t), ((2 k)) and g(k) (t) k)).
(0) Starting with at Tt, the decomposition level is computed as:
(s+1) Σ (l
(5) and u, (s+1) (
a,
Here, (2(x) and (()) are upsampled versions of h and g. obtained by inserting 2 - zeros between each original filter coefficient. This upsampling preserves the signal length, ensuring time invariance. Instead of keeping filter coefficients fixed, we allow the coefficients h and to adapt to the data, i.e., mak- ing them learnable (Michau al., 2022) allowing them capture relevant patterns and features at each variate level more effectively. Our experiments demonstrate that the learned filters exhibit correlation patterns that resemble those in the respective variables/channels but switching this adap- tivity/lcarning off does not adversely impact the results much.
Summary of tokenization scheme. The iterative decomposition yields final approximation u. at and set of wavelet coefficients {uf S { at each time point across various scales. This decomposition allows for complete reconstruction of the up-projected time series if desired:
(
3.
Σ
leug
(e) S.S
ΣΣ
Σ
(5) u (t).
In our tokenization scheme, each fime-freyuency point u, serves token, encapsulating infor- representation provides rich, struc-
mation at specific scale s and time t. This tured view of the data, where each token inherently retains both its temporal context and frequency information. This approach is simple but aligns well with our initial objectives.
4 MODULE 2: A SMALL MODIFICATION OF SELF-ATTENTION
Motivation. Recall that each token represents multiple channels at specific "pseudo" time point (pseudo because the length is L' and not L) for a specific wavelet scale. SWT already captures some temporal/frequency information. But we also want to characterize the full range of inter-channel dynamics, cheaply. In finance, some asset prices move together and others move inversely, and this can change over time. Tokens from fine resolution might show high linear independence for
4

---

Published as a conference paper at ICLR 2025
rapidly changing variable, capturing short-term dynamics, while those from coarser scales can reveal long-tene long-term correlations between different channel subsets, reflecting slower, persistent patterns. The degree of inter-channel complementarity or linear independence is not fully encoded by a scalar. In a five channel (or variable) system, tokens (1, 1,0, O, 0) and (0, 0, 1, 1,0) give zero dot product, but span a 4D subspace, indicating high complementarity. This could reveal, for instance, that the first two channels and the next two channels are behaving as coupled pairs. Leveraging such information explicitly may be unnecessary in large Transformer model with many layers where conjecture that these complex dependencies may get picked up anyway. But in smaller model, endowing the model with such a capability explicitly appears like a good idea.
One possible solution. It turns out that geometric algebra product Artin (2011) offers these abilities. extends classical linear algebra to provide a unified setup for expressing geometric constructions. Put simply, we obtain small generalization of self-attention which preserves the capabilities of standard dot-product attention. Note that Transformer models based on Clifford Algebra have been proposed recently BBrehmer et al. (2023); de Haan et al. (2024) these are broadly applicable but computationally heavy, see Chytas et al. (2024). This is because the size of the geometric product scales exponentially with the number of dimensions involved in the product. Our design is quite light, involves minimal changes to self-attention and well suited for our problem.
Details of the construction. We summarize few concepts before describing the low-level details.
A) Brief Geometric Algebra Review. Geometric Algebra (GA) provides a framework for rep- resenting and manipulating geometric objects. We focus on G2, the GA over 2-dimensional vector space because we consider pairs of to- kens in our attention mechanism, regardless of the tokens' dimensionality, The fundamental object in G2 is the multivector, expressed as M (Mo + (M) + (M)2, where (M) the k-vector part of M for k {0, 1, The key operation in GA is the geometric product, denoted by: a = β a A B, where
axp
Scalar
Vector
Bövector
Trivector
afp
(b)
Figure 3: Geometric product objects. (a) shows the
oriented parallelogram of the wedge product while (b) shows the progression from scalars to vectors, bivectors, and trivectors.
denotes the dot product and A denotes the wedge (or outer) product. The wedge product A, also known the exterior product, represents the oriented vectors. For vectors a and |3, the wedge product a A B results in a bivector (a 2-dimensional ele- ment in the algebra). As an example in G2. consider a aej beg and B 00 + dez, where ej and are orthonormal basis vectors. Their wedge product is a A B = (ad bc) A e2). Here (ad bc) represents the area magnitude, while e e2 indicates the orientation in the plane.
of the parallelogram spanned by two
B) Instantiating Geometric Product in our case. We can reformulate the attention mechanism using the geometric product. For tokens t and t', instead of just computing their dot product, we can use the geometric product which com- bines the dot product (scalar part) with the wedge product (bivector part), encoding both magnitude-ban and geometric relationships between the tokens. So, we capture not only the scalar similarity between tokens (via dot product) but also their linear independence and the ori- entation of the space they span (via wedge product). This allows detecting complementary information across chan- nels and changing inter-channel dynamics. For two tokens and /3 for different time points across C channels, the a f3 part is the scalar similarity, while a A B tells us how these tokens complement each other across the channels.
Geametrie
similarity
Scaler
Blrector
Tukes
Leursabee
Figure 4: A simplified illustration of ge- ometric product attention. The entries of the attention matrix are multi-vectors.
C) Linear Projection. Given time-freyuency tokens U++) = {u un scale and shared weights Wq. Wx. Wy E E.L'' LL''L', the query, key, and value matrices are:
E RCCL' for
Q (()) Ul. k( Wk, v(t) - Wv, for scale (0, 1. ,S). (4)
5

---

Published as a conference paper at ICLR 2025
To keep the number of channels/variables unchanged, we apply the linear projection along L'. D) Geometric attention calculation. Consider the expression a 3 + + and let us evaluate how minimally modify the self-attention block to mimic this behavior. We can consider two different 's) say V*(a) and Vg(") The dot-product attention between the tokens can act upon Vi(e) for the term ((x)* K(s), Separately, the matrix of wedge-product objects acts upon Vi(e) for the second term B = (Bre) } for t,t' E (1,-.-,'') with Bi = qe A k
The first part can be viewed simply as the vanilla attention mechanism, needed. The wedge product results in a matrix of bivector objects, where each element Bi is bivector for the pair-wise tokens; we indeed compute it element-wiss for each pair t and t'. Viewing V/(*) column by column, the operation (V well defined and is closed within the algebra. For example, B is L' L' matrix where each element B is bivector resulting from the wedge product of the 1-th query vector and the 1'tth key vector, while V2(") is an I, x C matrix. The the geometric product between bivector B and entries from column of V,(1) The only remaining task is to combine this result with vanilla self-attention, and do so, need map these bivectors down. For this, function (+)) to match dimensions. There is much flexibility in choosing ((-): ): it can be the bivector's input.
special treatment is
operation be written explicitly the
reduction
magnitude trainable MLP that takes both magnitude and orientation
Summary of geometric product attention mechanism. The geometric attention mechanism is:
GeoProdAttm Q, K, V) = softmax (dot-prod Q, K)
+ +5((
Q, K)
VC
VC
(5)
where C' is a scaling factor and we have used V instead of two separate variables. Also, the matrix of bivectors acts upon V individually for each column in V.
MODULE 3: RECONSTRUCTION OF MULTIVARIATE TIME SERIES
Motivation/Rationale After processing the fime-freyuency tokens through the geometric product attention module described above, we need to reconstruct the signal in the time domain. This is achieved using learnable ISWT(-; (1.91 91). where h and 91 are the learnable synthesis filters for the low-pass and high-pas -assspasssss components, respectively. These filters are the direct counterparts to the analysis filters hg and gu used the forward SWT.
Details of the construction. The reconstruction is performed iteratively, starting from the coarsest scale and progressing to the finest scale. Given the initial approximation coefficients à á (0) and the processed tokens each scale s, denoted {ei(") ai!") ..... 1s 1 the reconstruction
can be written as:
(a-1)
Σ (+) (
à
where ( and gl are the upsampled versions of h and gi level The reconstruction process iteratively computes (e-1) using (+) and (=) until we reach (0))
Summary of reconstruction. The final reconstructed time series X (#1, ......... .t given by a(0), This reconstructed representation preserves the temporal structure of the original input while incorporating the multi-scale information processed via geometric product attention. This reconstructed time-domain representation X is then passed through feed-forward network and layer normalization for final refinement, which produces the forecast output to calculate the loss. We perform end-to-end training.
EXPERIMENT
In this section, cover experimental findings in detail. We divide our experimental protocol into two phases: evaluating the quality of forecasting both for long-term and short-term and an ablation study to evaluate the efficacy of proposed model, Sim

---

Published as a conference paper at ICLR 2025
Lookback Input MTS
Lincar Projection
FFN & Layer Norm
* SWT -pp-
Length
MSE
channels
Learnable
Backward Propagation
Figure 5: An overview of the proposed SimpleTMM model framework.
6.1 SETUP AND BASELINES
Baselines. We compare our model with 15 well-known forecasting models for MTS data, including
(a) MI.P-hased methods: TimeMixer Wang et al. (2024). TiDE Das et al. (2023) RLinear Li et al.
2024 DLinear Zeng al. (2023);
(b) Transformer-bant baaed methods: Transformmr Liu et al. 2024a). PatchTST Nie et al.
2023
Crossformer Zhang & Yan (2023), FEDformer Zhou al. 2002bb
Autoformer
Wu al. 202
FiLMZZou et al. (2022a), Stationaryl Liu et al. (2022b);
(c) CNN-based methods: TimesNet Wu et (003
SCINet LIU (2022). MICN Wang al.
2023);
(d) GNN-based method: CrossGNN Huang et 2023
Datasets. The datasets that are covered in our experiments include:
(a) Long-term forecasting: We evaluate our model 8 widely recognized benchmarks: the ETT datasets (ETTh1, ETTh2, ETTm1, and ETTm2), which provides seven factors of electricity trans- former data recorded hourly and 15-minute intervals, as well the Weather, Solar-Energy. Electricity, and Traffic datasets, which include diverse metcorological, power production, con- sumption, and road occupancy data Wu et al. (202T).
(b) Short-term forecasting: We adopt the PEMS dataset Chen et al. (2001) with four public traffic subsets (PEMS03, PEMS04, PEMS07, and PEMS08) recorded every minutes. We also evaluate the forecastability of all datasets, following Wang et al.] (2024), and observe that ETT and Solar- Energy pose modeling challenges due to their low forecastability. Further details on the datasets can be found in Appendix
6.2 EVALUATION RESULTS
Long-term forecasting results: Forecast results from our experiments are presented in Table with optimal performance denoted in red and second-best in blue. A lower MSE/MAE values indicates superior prediction accuracy. SimpleTM demonstrates robust performance across diverse bench- marks, achieving optimal MSE/MAE in out of 8 datasets. Complete results are in Appendix B We briefly discuss comparisons with two the closest methods in terms of performance.
(a) TimeMixer Wang et al. (2024): : Our method exhibits MSE reductions of 8.1% for ETTh2 and 8.8% for ECL compared to TimeMixer. the Solar-Energy dataset, acknowledged for its complex- ity, our model attains the best MSE, surpassing TimeMixer by 14.8%. Although TimeMixer uses multi-scale approach, it underperforms high-dimensional datasets because its mixing mechanism is limited to linear or lower-order interactions. Its reliance average pooling and ensemble pre- dictions leads to information loss during scale transitions. SimpleTM overcomes these challenges with geometric attention mechanism in G2 space, combined with classical tokenization method, enabling to capture complex, higher-order relationships by considering both the magnitude and orientation of token pairs.

---

Published as a conference paper at ICLR 2025
Table I: Long-tere term forecasting results for various prediction horizons (96, 192, 336, 720} with fixed lookback window of L = 96. Reported values are averaged across prediction lengths.
Modd
SimgheTM
(Ours)
2004
20044
Ca
2023)
RLinsar
2004
PoshIST
3023
Creasfomeer
ping
TIDE
2021
2013
202
SCISs
1022
Metni
MSG
MAG
MSL
MAE
MSL
MAE
MSE
MAG
MSL
MAL
MSG
MAG] MSG
MAE
MSB
MAG
MSE
MAE
MSG
MAB
MSD
MAG
RF7ml
[0:81
00396
1
0.385 0.900 [0407
0,410
(0.991
0/404
10,414 0.407
10.67 0,000 (0513 84466
[0419
0,419 (0400)
0.40K
0.404
0,407 00485
0.481
E17m2
0.278
0.322
00.27
0.325
[02288
0.332
(0283
0:330
0.226
0.327
[0.28] 0.326
(0755 0.60
|6.000.358
00.58
0,408 (0.99
0:333
10,350
0,401
0.571
0.537
11422
0.822
0.507
0,453
0.647
BTTh2
[0383
0,391
[0.384
0.407
[0.363
0,407
[0.30
0..34
0.39
[0.367
0,407
(0,92 0.644
[0,61] 0,500
0.414
0.427
(0.559
0,515
0.954
0.725
FCL.
[L166
0.300
10.182
0.22
[0.128 0.270
[0.00]
0.300
(0,219
0.20
[0.005 0.390
(0244
0.334
[0.25] 0.544
0.192 0.205
[0,212 0.900
0.266
0.365
Traffic
0.444
0.389
10,684
0.297
[0,226 0.252
[0:58]
0.223
(0,006
0.508
[0:46] 0.304
(0.50)
0.50
[0.060
0,473
(0630)
0.396
(0.525
0,583
0.804
0.509
Weother
[0.243
0.271
[0.245
0.276
[0.258
0.278
[0.247
0.289
[0.272
0.291
[0.259
0.281
[0.259
0.315
[0.27] 0,520
0.259
0:287
[0,265
0,517
0.292
0,363
Sole-Firrrgy
0.733
0.720
0/417]
0.401
0.375
Table 2: Short-term forecasting results on PEMS datasets. Results are shown for prediction horizon H = 12 with a fixed lookback window of = 96. Lower metric values indicate more accurate predictions.
Model Matric
Simple IM Firandermee
TimeMaser
Crossformer
PuchiST
TimesNett
MICN
DLineer 2023
FILM 2012aa
FEDiermer 20225
Auteloemer
20124a
2034
2023
20226
MAE
14:85
13.15
14.30)
13.14
18.05
16.41
13.71
19.70
21.56
19:00
17.64
15:08
PEM503
MAPE
14.79
19.19
14.79
13.74
17.29
15.17
13.67
18.35
18:35
18.57
17.56
18.75
RMSE
23.58
28.86
23.58
25.56
30.15
26.72
24.55
32.35
35.07
30.05
28.57
27.82
MAR
18.74
33.42
18.07
20.
24.86
23.63
21.62
24.62
26 74
26.51
22. 22.34
25.00
PEM504
MAPE
12.11
17.83
12:21
12.84
16.65
13:15
13.53
16.12
16.46
16.76
14.85
16.70
RMSE
30.46
35.75
30.70
32.41
40.46
34.90
34,39
39.51
42.86
41.81
35.47
38.02
MAF
20.25
22.54
20.76
22.54
27.87
25.
22. 28
28.65
28.76
27.92
26.02
26.92
PEM507
MAPE
8.55
12.77
8.77
9.38
12.69
10.60
9.57
12.15
11:21
12.29
11.75
11.83
RMSE
33.06
33.92
31.71
35.49
42.56
40.71
35.40
45.02
45.85
42.29
42.34
40.60
MAF
14.92
18.79
15.23 26
17.56
20.35
19.01
17.76
20 -6
22. 11
20.56
19.20
20.47
PEMS08
MAPE
9.36
12.19
9.71
10.92
13.15
11.83
10.76
12.09
12.81
12.41
12.31
12.27
RMSE
23.80
28.86
24:15
27.21
31.04
30.65
27.26
32.38
35.13
33.97
38.62
31.52
(b) iTransformer Liu et al. (2024a): Against iTransformer, our method achieves MSE reductions of 6.4%, 7.0%, and 4.7% for ETTml, ETTh1, and ECL respectively. While iTransformer excels on high-dimensional time-series datasets, such as Traffic (862 variables/cchnnels), struggles with the rapidly fluctuating ETT datasets due to its variate-wise tokenization, which fails to capture fine- grained local patterns and lacks sufficient inter-channel context in lower-dimensional scenarios. In contrast, SimpleTM uses wavelet- tokens that prioritize intra-variable local interactions and effectively capture oscillatory patterns across multiple resolutions.
Short-term forecasting results: Table 2 presents the short-term forecasting results for the high dimensional PEMS datasets. This is evaluated using three metrics: Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Root Mean Square Error (RMSE), where lower values indicate better predictions. Our SimpleTTM demonstrates superior or comparable performance across all four PEMS datasets (PEMS03, PEMS04, PEMS07, and PEMS08), consistently achieving the best results. These results validate our model's good performance for high-dimensional, short- term forecasting tasks, complementing its strong performance in long-term forecasting scenarios. Additional results for larger prediction horizons are provided in Table 10 in Appendix B.
6.3 MISCELLANEOUS ADDITIONAL ANALYSIS: ABLATIONS, WAVELETS
Ablation Study. We conducted an ablation study to evaluate two key architectural elements: geo- metric attention mechanism and the stationary wavelet transform. Table 3 summarizes the results across four datasets, with metric values averaged over four prediction horizons. The findings consis- tently demonstrate that both geometric attention and SWT contribute to our model's performance. Detailed results can be found in Appendix C
Filters in the wavelet decomposition. We now briefly check the properties of the learned fil- ters. We randomly initialized the wavelet basis with fg normalization and compared the result- ing filters to the wavelet bank for the ETTh2 dataset, identifying the most similar ground truth wavelet for each learned wavelet. As shown in Fig 6a (left). the learned filters from random initialization occasionally approximated wavelet-like structures, displaying higher ampli-
Table 3: Ablation study results comparing model variations across datasets.
Model
ETThl
ETTml
Solar
Weather
MSE
MAF
MSE
MAE
MSR
MAF
MSR MAR
SimpleTM
0.271
w/o Aitn
0.437
0.440
0.385
0.398
0.194
0.253
0.245
0.273
w/o SWT
0.432
0.435
0.386
0.398
0.246
0.289
0.247
0.274
8

---

Published as a conference paper at ICLR 2025
Comelation Marr Them Acrey Chernet
Compactinn Merne Oripne Oursel
(h)
Figure 6: Analysis of learned filters and their correlations in sevelet-hased time series forecasting. (a) Com- parison of learned filters with theoretical wavelet bases (Bior3.3), 3). Left: synthesis low-pass filter; Right: forward high-pass filter. (b) Correlation heatmaps of learned filters (left) and original channels (right).
tude peaks and maintaining overall patterns, although with some amplitude variations. This suggests that the model can inherently discover wavelet-like features without explicit wavelet priors. In con- trast, filters initialized with standard wavelets (the right in Fig 6a) retained their core structure while exhibiting subtle adaptations, indicating that a wavelet initialization provides strong inductive bias for refining theoretically grounded filters based on empirical data. A comparison of the correlation heatmaps in Fig. 6b between the filters (all initialized with identical standard wavelets) and the orig- inal channels reveals meaningful patterns. The filter correlation matrix shows distinct block-like structure with high correlations (0.7 0.9), primarily due to shared initialization. However, dark horizontal and vertical lines suggest that some filters have developed lower correlations, indicating degree of specialization. In contrast, the original channel correlation matrix shows weaker overall correlations (0.4 - 0.6) with a less pronounced block structure. The persistence of some block-litt like patterns in both matrices, albeit at different scales, implies that the model retains aspects of the original data structure while enhancing certain relationships through learned filters.
Multi-scale visualization. Fig. 7 presents representative example of multiseale forecasting results. The MTS forecasting panel illustrates the model's effectiveness accurately predicting overall patterns, including significant peaks and troughs, while capturing the cyclical nature of the data. Our model excels at decomposing and reconstructing the time series across multiple scales. The Scale I and Scale panels highlight the model's s ability capture high-frequency fluctuations, while the Scale 0 panel reveals the underlying low-frequency trend. This nusti-resolution analy- sis enables the model to extract relevant features from various timescales and integrate them into coherent forecast in the original domain. In summary, our multiscale forecasting technique demon- strates robust performance in capturing both macro trends and micro fluctuations.
Lookback Window
Prodicted Window
Forceast Window
1.5
1.0
0.1
0.1
0.13
0.1
0.00
2.0
125
35 50 135 190
25 50 75
100 125
150 175 250
50 75 135
MIS forocasting
Scale 1 (dctail)
Scale 2 (dctail)
Scale 0 (spproximation)
Figure 7: Multiscale forecasting visualization: MTS prediction shows global pattern and cyclical nature. Scale decomposition demonstrates the ability to capture low-frequency trends (Scale 0) and progressively higher- frequency fluctuations (Scales 1-2).
Compute and Memory Footprint. A key advantage of our single- or two-layer design is that it often captures complex dependencies in MTS data without resorting to larger, more cumber- some backbones. In our reported experiments, we prioritize memory and computation efficiency by choosing the bivector's magnitude for the reduction function ((.). This approach not only matches or surpasses the accuracy of competitive baselines, but also reduces memory requirements and pa- rameter counts. For instance, on the Weather dataset, our model uses only 0.3% of iTransformer's parameters and 13% of TimeMixer's parameters, requires 38% and 66% less memory, and runs 1.7x and 3.4x faster, respectively. Further details provided in Appendix D

---

Published as a conference paper at ICLR 2025
7 RELATED WORK
Time Series Forecasting. Time-series forecasting is mature topic and has advanced from tradi- tional statistical models like AIRMA (AutoRegressive Integrated Moving Average) Box & Jenkins (1994) and ARMA (AutoRegressive Moving Average) Markidakis & Hibon (1997) / models to so- phisticated deep learning approaches that better handle the complexity of time-series data. These approaches can be broadly categorized as follows.
1. CNN models effectively capture local temporal patterns in time series data. TCN Bai et al. (2018) introduced causal and dilated convolutions, while SCINet [IC al. (2022) employed sample con- volutions. TimesNet Wu et al.] (2023) used 2D variation modeling with inception blocks to capture both inter-period and intra-period patterns. CNNs sometimes struggle with long-range forecasting due to their limited receptive field.
2. Graph Neural Network (GNN) methods are capable of capturing inter-variable relationships in MTS data. MTGNN Wu et al. (2020) used graph learning to infer variable interactions, and Cross- GNNnHuang et al. (023) further refined this with cross-scale and cross- cross-variable modeling to manage noise in MTS data. However, GNNs can often be quite computationally intensive.
3. MLP models Zeng et al. (2023) offer a balance between simplicity and efficiency. TimeMixer Wang et al. (2024) introduced multi-scale mixing however average pooling in this context leads to some information loss when transferring between scales. RLiinar Li et al. (2024) showed that linear models, with careful design, could effectively capture periodic features, achieving competitive performance with more complex architectures.
4. Transformer models Wu et al. 2202 y Zhou al. (2021) Wu al. (2022); Zhou et al. (002bb) are prominent results that have demonstrated efficacy in capturing long-range dependencies. Cross- former Zhang & Yan (2023) introduced cross-dimension self-attention, iTransformer [Luu al. (2024)) applied attention to channel-token but lacked the resolution to capture fine-grained lo- cal patterns and can struggle to gain sufficient inter-channel context, and PatchTST et al. (2023)
used a patch-based representation with channel-independent processing and fixed resolution.
Multi-scale Modeling, Capturing patterns at different resolutions is very common in vision Fan et al.] (2021): Lin et al.] (2017); Tao et al. (2020) and has also been used to obtain efficient self-attention modules Nguyen et al. (2023); Zeng et al. (2022). Inspired by these successes, multi-scale modeling has been adapted to time-series forecasting as well. N-HiTS Challu et al. (202)) constructed a hierarchical forecast with multi-rate sampling, while Scaleformer (Shabani et al. (202)) progressively refined forecasts through repeated upsampling and downsampling op- crations. Pathformer [Chen et al. (2024) applied dual attention over patches of varying temporal size. Pyraformer Liu et al. (2022a) used pyramidal attention structure to handle inter-scale depen- dencies. Time.Mixer Wang et al. (2024) used a decomposable mixing approach, combining seasonal and trend components separately across scales for both past and future temporal variations. Different subsets of these methods face different challenges. Manually designed scales can make the model inflexible to adapt dynamic time series, while average pooling often results in the loss of fine- grained details. In some models, the aggregation and reconstruction mechanisms are fragmented, requiring ensemble strategies or more complicated architectures.
CONCLUSIONS
Our work introduces a novel approach to multivariate time series (MTS) analysis that integrates a simple wavelet-based tokenization and generalized form self-attention that captures both multi- scale temporal dynamics and complex inter-channel relationships. Our empirical results demonstrate competitive performance against most existing baselines across various MTS tasks. Our experi- ments suggest that exploiting inter-channel dependency does not always yield improvements, and the performance varies from one dataset to other other. The construction is simple and can provide sensible lightweight baseline for more sophisticated methods although in its current form, cannot easily be extended to token-by-token generation. We must acknowledge that as MTS datasets that publicly available grow in size, larger models may be better suited to maximize performance. Nonetheless, we believe that the individual components utilized in our formulation can still mean- ingfully inform the design of specialized adapters and/or embedding/tooennzatiin schemes.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. The authors are grateful to Sourav Pal, WonHwa Kim, Melanie Boly and Guilio Tononi for discussions and imput. Partial support for this work came from a contract to UW-Madison under the DARPA Strengthen program.
REFERENCES
Torben G. Andersen, Tim Bollerslev, Peter F. Christoffersen, and Francis X. Diebold. Chapter 15 Volatility and Correlation Forecasting, pp. 777-878. Handbook Economic Forecasting. 2006. ISBN 9780444513953. doi: 00.001//$1774-000600511001111110 of Andersen, Bollerslev and Diebold was supported by grants from the U.S. National Science Foundation, and the work of Christoffersen was supported by FQRSC, SSHRC and IFM2. The authors gratefully acknowledge extremely helpful comments from two anonymous referees. The usual disclaimer applies.
Funding Information: The work
Emil Artin. Geometric Algebra. Wiley Classics Library. Wiley, 2011. ISBN 9781118164544 URL /tttss://boks.... google..oo.ooooooooooooooo...... com/ booksooooksseooooo (toookss7dd-333990pppmmB8..
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron George Bostrom, Paul Southam, and Eamonn J. Keogh. The uea multivariate time series classification archive, 2018. ArXiv, 0bs/1811.00075, 2018. URL org/Corpus ::15566665353
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.01271, 2018.
James R. Bence. Analysis of short time series: Correcting for autocorrelation. Ecology, 76 (2):628-639) 1995. doi: http://dbiiorg/1100337////////// Inlinelln innerfrrry.... wi wwwwwllee ley. com doi abs//b////// 2307//1...3001110 011/00711911118
URL
George Edward Pelham Box and Gwilym M. Jenkins. Time Series Analysis: Forecasting and Con- trol. Prentice Hall PTR, USA, 3rd edition, 1994. ISBN 01 0130607746.
Johann Brehmer, Pim de Haan, Sönke Behrends, and Taco Cohen. Geometric algebra transformer. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 35472-35496. Curran Associates. Inc., paperapppppp-- http://///pppppeeee/e//////e///p otp.aaa-eee..--aa---------eeeeee
2023. URL
Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza Ramirez, Max Mergen- thaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 376))6989-6997, URL battps://ojs aaai. org/ /i//inndxxx.///////////////ii//. php/
Jun. 2023. doi: w37.:/9//...........22... AAAI DLAA/arttcclevvew//25555
Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway perfor- mance measurement system: Mining loop detector data. Transportation Research Record, 1748 (1):96-102, 2001.
doi: 10.3141/1748-12. URL //db:::://dd. 0rg/10.3141//74--1
Peng Chen, Yingying ZHANG, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. Pathformer: Multi-scale transformers with adaptive pathways for time series forecasting. In The Twelfth International Conference Learning Representations, 2024. URL
eew net w
Dawei Cheng, Fangzhou Yang, Sheng Xiang, and Jin Liu. Financial time series forecasting with multi-mod-ie modality graph neural network. Pattern Recogn., 121(C). January 2022. ISSN 0031-3203. doi: 10. (0.10 2021.1..001 URL https: //oo: www.llorr....... 1016/j. patcog. 2021 108218
Sotirios Panagiotis Chytas, Rudrasis Chakraborty, and Vikas Singh. Geometric algebra based en- coding for graph prompting. In ICML 2024 Workshop Efficient and Accessible Foundation Models for Biological Discovery. 2024. URL https: http::://ppeeeeviieeee net het.:orumtt.i...
L.2Ve.JBDFw

---

Published as a conference paper at ICLR 2025
Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan K Mathur, Rajat Sen, and Rose Yu. Long-term forecasting with tiDE: Time-series dense encoder. Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL
pCbC3aQB5W
Sourav Das and Guy P. Nason. Measuring the degree of non-stationarity of a time series. Stat, (1):295-305, 2016. doi: http:///oo..og//1..0000/////aa 125. URL http://s///mmieelllliiite/ee/ w1 htty.cco//oo//aaeece wttpa////s..11000/.o.o.ooo..
Pim de Haan, Taco Cohen, and Johann Brehmer. Euclidean, projective, conformal: Choosing a geo- metric algebra for equivariant transformers. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, volume 27, 2024. URL https: / httpaaaxiivvi... abs/
2511.04744
Jacob Devlin, Ming-Wei, Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Pupers), pp. 4171-4 -116. Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 1653/v1/N19-1453 URL https: http://caaaathhoooooo egy.oolly.oorrrrr
N19-1423/
Yihe Dong, -ann-Bap Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang (eds.), Proceedings the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. dong html
PMLR, 18-24 Jul 2021. URL https:
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pp. 6824-6835, October 2021.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski.
Moment: a family of open time-series foundation models. In Proceedings of the 41st International Conference on Machine Learning, (4.4 JMLR.org. 2024.
David Gubbins. Time Series Analysis and Inverse Theory for Geophysicists. Cambridge University Press, 2004.
1017 /tt.://.111101000077000111110000
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. htt/:tpppe//prreeeiise..eeoooo/p 2ppp:/0077A1///e00000aaaaaaaaaa2
files/
Simon S. Haykin and Barry Van Veen. Signals and Systems. John Wiley & Sons, Inc., USA, 1st edition, 1998. ISBN 0471138207.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10.1109/CVPR.2016990
Qihe Huang, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. Crossgnn: Confronting noisy multivariate time series via cross interaction refinement. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 46885 Curran Associates. Inc., files/pape:/2023/ oaa.ae/reee---aaee---ooeeeeeeeee
2023. URL http::://proeeddiggss.eezoio
File/
Ming Jin, Yu Zheng, Yuan-Fang Li, Siheng Chen, Bin Yang. and Shirui Pan. Multivariate time series forecasting with dynamic graph neural odes. IEEE Trans. on Knowl. and Data Eng., 35 (9):9168 9180, September 2023. ISSN 1041-4347. doi: 10.11099KKDE22022.222222 httpps://dos http:///ooo.or/11/..0 1109/ TKDE. 2022. 3221989
URL
12

---

Published as a conference paper at ICLR 2025
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Ye Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2024. URL htt.ttts:: /wppr//peeree//ee//e ew net. foatttforrnng for http://orruunnnnnnnnneeeee
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https:
//openreview.o net. www.ee//wwooeeieeeeeeeeeeooeeeee dtt.:ummmmmmsuuua
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18, pp. 95-104, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781 478155535677. doi: 10.1145/3209978. 3210006. URL https: /ttts::/ddoi.oo /ttp::///oooor////////
011.:/99978.21000000 3210006
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1) (1)(1)):20 1):2222-23,4, 1998.
Jianxin Li, Qingyun Sun, Hao Peng. Beining Yang, Jia Wu, and Philip S. Yu. Adaptive subgraph
neural network with reinforced critical structure mining. IEEE Trans. Pattern Anal. Mach. tell., 45(7):5063-808888 July 2023. ISSN 0162-8828 doi: (0.L100/PAMM2222..55599
URL
https::/ddi.orgge
1109 TPAMI. 2023. 3235931
Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on affine mapping, 2024. URL pettps://operreee
net. forum?id
197kxcting
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 936-944, 2017. doi: 10.1 N0.110//CVPR.201..006
Minhao LIU, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia LAI, Lingna Ma, and Qiang Xu. SCINet: Time series modeling and forecasting with sample convolution and interaction. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL https: htttp:://opeeeew/etee /wwp//pwwiwww...o//// net/ forum?id Aya is TAzmql
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dust- dar. Pyraformer: low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2022a. URL https: //openzeview. net/ffrum/ d=00xxF
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transform- ers: Exploring the stationarity in time series forecasting. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9881-9893. Curran Associates. Inc., 2022b. URL /tttps://proceeddii ngs www.resuui...e ct.. http:///errr.llleepppppp//////// otw.://000....22202222222------- Conference
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang. Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfih In- ternational Conference on Learning Represenlations, 2024a. (tRl:t//s:/ ptttpps///operrviiee
net /tt////oumiiiddeee -ee.iii-aJePAABBaal
Yong Liu, Haoran Zhang, Chenyu Li. Xiangdong Huang. Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning. vol- ume 235 of Proceedings of Machine Learning Research, pp. 32369-32399, PMLR, 21-27 Jul 2024b. URL hhttps://pposste. igs. ml www.press//ll httesss/v23/// u2
13

---

Published as a conference paper at ICLR 2025
Helmut Lütkepohl. Vector autoregressive models. In Handbook of research methods and applica- tions in empirical macroecononics, pp. 139-164. Edward Elgar Publishing. 2013.
Spyros Markidakis and Michele Hibon. Arma models and the box- methodology. Journal of Forecasting, 1633):17--63,, 1997. doi: http:///booooooll.oo0o/////c1111
3(44::AID-FFOR62)33.0000222
URL
com/
0t1p66/8810022880000001111111112 BAAID-FOR65243B5 0..
Gabriel Michau, Gaetan Frusque, and Olga Fink. Fully learnable deep wavelet transform for un- supervised monitoring of high-frequency time series. Proceedings of the National Academy of Sciences, (196)))c20069991 19, 2022. doi: 10. 10.1073/pnas.21ee 2006598.01 URL https:/ //t::///wwwwww
pnas www.s/org/ooio http///i/abs///i abs/1ass00b. 1073/ 2003/p/s..... 21 065 06198119
Peter C. M. Molenaar, Jan G. De Gooijer, and Bernhard Schmitz. Dynamic factor analysis of nonstationary multivariate time series. Psychometrika, 57(3):333-344 1992. doi: 10.1007/
BF02295422. TRI.https://// http:::://oooooro//////oo www//o............ 0ttp://1.100/BFOOO 1077000229//
Manfred Mudelsee, Trend analysis of climate time series: A review of methods. Earth- Science Reviews, 190:310-322, 2019. ISSN 0012-8252. doi: https://bb..rr//10011////. 12.005. URL
2001282522 8303726
G. Nason and B. W. Silverman. The Stationary Wavelet Transform and some Statisti- cal Applications, pp. 281-299. Springer New York, New York, NY, 1995. ISBN 978-1- 4612-2544-7. doi: 1111077978-114412-255444
URL httph://ppe. oog.ioorg/1... 10077
978-1-4613 200-225447722
Tan Minh Nguyen, Tho Tran Huu, Tam Minh Nguyen, Minh Pham, Nhat Ho, and Stanley Osher. Transformers with multiresolution attention heads, 2023.
forum?1d=L.qkkr _ht]
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https:/ http::///ppereeee..weeee net. forum??n JodcOvTOcol
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: imperative style, high serfoorrfrmencce deep learning library. Curran Associates Inc., Red Hook, NY, USA, 2019.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neurul Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL ht https://proceddngg www.:ss..riiiic..o.e. ccppppr files/paper/2018) File/ /ttp///666655110.668222285225580
Julio Rodríguez and Esther Ruiz. A powerful test for conditional heteroscedasticity for financial time series with highly persistent volatilities. Statistica Sinica, pp. 505-525, 2005.
Mohammad Amin Shabani, Amir H. Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. In The Eleventh International Con- ference on Learning Representations, 2023. URL https: http:p:://opnnreviee net.iwewwweeeeeeeeeememmmm net foot/ffoum?
Ad=sCrallCtjjo
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA. USA. May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http://aaxiv.orrg/////
1556
14

---

Published as a conference paper at ICLR 2025
attention for semantic seg- Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical wttp://.rrggab////000000 /ara//aas/200... 10821
httpivxxivv.......//aaa org/ mentation. CoRR, 2tts/22055.082..... 2020. URL CtLtatts:://// /tt/att///aax.aa.o/oo
Huong Thu Truong, Bac Phuong Ta, Quang Anh Le, Dan Minh Nguyen, Cong Thanh Le, Hoang Xuan Nguyen, Ha Thu Do, Hung Tai Nguyen, and Kim Phuc Tran. hight-weight federated learning-basee based anomaly detection for time-series data in industrial control systems. Computers in Industry, 1 140:2, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
ngs neur.. www.ips.rrcccccco ottp wirrffflssppaaae wwt.ss//ppppp//1117 File/ 2tlle/3550e222554777611110110012
paf
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Mien: Multi- scale local and global context modeling for long-term series forecasting. In The eleventh inter- national conference on learning representations, 2023. URL ptttps://openeeview.o nett/
Eorum?id 21
Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou Timemixer: Decomposable multiscale mixing for time series forecasting. In ICLR, 2024.
net/ htt.:/oommmm......sfff..........
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In Proceedings of the 41st International Conference on Machine Learning, ICML'224 JMI.. 2024.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long- Long-term series forecasting. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, 2021. URL http:tttt///oeeee.ee
net/ forum? (ammmnidd=44jj3o30 6d5Qm
Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Lineariz- ing transformers with conservation flows In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 24226-24242. PMLR, 17-23 Jul 2022. URL https: htttp:://prcceedingg ptr.ppssssm.......///em....... V162 2262//uw2mm html
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d- modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023. htl/tttps://opereviiee net forum?
1d= Uqw3840g
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang, Con- necting the dots: Multivariate time series forecasting with graph neural networks. In Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Min- ing. KDD '20, pp. 753-763, New York, NY, USA, 2020. Association for Computing Machin- ery. ISBN 4503999999. doi: 10.1145/3394886.4000 18. URL httttp::///ii.oooooooottttt org/ 1145/339
3403113
Ting Yao, Yingwei Pan, Yehao Li, Chong-Wal Ngo, and Tao Mei. Wave-vit: Unifying wavelet andnbsp:transfommers fornbsp:visual representation learning. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27 2022, Proceedings, Part XXV, pp. 328 328-345, Berlin, Heidelberg. 2022. Springer- ISBN 0777888-031-198000
doi: 1001/07/
978-3-031- 10800-9-----9999 19. URL https: /tts:///ooi. /oii. www./00.0o...... 1011001100000000 11-1110007700---111
19
Scott L. Zeger, Rafael Irizarry, and Roger D. Peng. On time series analysis of public health and biomedical data." Anmual Review of Public Health, 27(Volume 27, 2006):57- 79, 2006.
ISSN 1545-2093. doi: https://doi.org//0.. http://ddhooggg0011111aaaammnnnn 14t6://mmmeyyyplllllll/.22211111
15

---

Published as a conference paper at ICLR 2025
annurev 2ub.health.....22222 22t.:/........0.................
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI Conference Artificial Intelligence, 370):11121-11122,, Jun. 2023. doi: 10.1609/aaii v37.a............. v37i9.26317. URL httph::/toosssss. /aap:///ossaaaa..........
php/ aaal. wwt./ooggg/./.oe oag.././/ee.....eedx......
AAAI http///rtccc//ee//1111111
Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh. Multi resolution analy- sis (MRA) for approximate self-attention. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25955 25972. PMLR, 17-23 Jul 2022. URL https: htttp:://pxoceeeiigg... mlr. htt.:eeiiiii.....sesssssees v1 62 /etg2/nng222...m html
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023. URL
.ew. www.eeeeeoonnniiviie.e.eee. j9eie
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan- cai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecast- ing. In Thirty- AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thine Con- ference Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 11106-11115. AAAI Press, 2021. doi: 10.16099/AAALL.35112...11 https: //oi./do../// www.rrrgg...... 609/ aaai. www.v2.511211.... 17325
URL
Tian Zhou, Ziqing MA, xue wang, Qingsong Wen, Liang Sun, Tao Yao. Wotao Yin, and Rong Jin. Film: Frequency improved legendre memory model for long-term time series forecasting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 12677-12690. Curran Associates, Inc.,
2022a. URL htttps://proceeedings..ce nwwrips...n.ppp..p... wwwppp//pppppeeeeeeeeeeeeeeeeeee httpc/ppprr_ffllsseeeeeeeeeeeeee files//aper///22/
Ltl1://2..2222000077772222662220 Paper oor.raeeeeeeececoeooooooeeeeeeee wwn.erenneeeccccccoooccoooooc pdff
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer:
Frequency enhanced decomposed transformer for long-term series forecasting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Pro- ceedings of Machine Learning Research, pp. 27268-2277777 27286. PMLR, 17-23 Jul 2022b. URL https://pzoceedingg hrrprceeediiggnn...rrress//////e press 2hoze//22g.2///. .htll]
Zhenhai Zhu and Radu Soricut, bavelet-based image tokenizer for vision transformers, 2024. URL
https://arxiv. wttp:/wwwwwgg/bbb/////////
16

---

Published as a conference paper at ICLR 2025
A EXPERIMENT DETAILS
Datasets. We evaluate our model on several benchmark datasets covering both long-term and short- term forecasting tasks. Among long-term forecasting tasks, we evaluate method on 1) ETT Zhou et al. (2021): a dataset of electricity transformer data, which includes four subsets (ETTh1, ETTh2, ETTml, ETTm2). E and ETTh2 record data hourly, while ETTml I and ETTm2 record data every 15 minutes; 2) Weather Wu et al. (202T): a dataset comprising 21 meteorological parameters and is collected at a 10-minute interval: 3) Solar-Energy Lai et al. (2018): a dataset recording power generation data from multiple plants, with data collected every 10 minutes in 2006; 4) Electricity Wu al. (202T): a dataset of electricity consumption for 321 clients; 5) Traffic Wu et al. (2021): ): a datasets monitoring hourly road occupancy rates through 862 sensors in San Francisco from 2015 to 2016. Additionally, we evaluate our method on PEMS dataset for short-term forecasting. The PEMS dataset collects traffic network data from various locations and covers four subsets (PEMS03, PEMS04, PEMS07, PEMS08), which has been widely adopted as benchmarks since LIU et al. 2022).
We mainly follow the experimental configurations in Wu et al. (2023). including the same data processing and splitting protocol. For both the long-term and short-term forecasting settings, fix the lookback window length to 96 for all datasets and baselines. The prediction lengths vary according to the forecasting tasks: for ETT family, Weather, Solar-Energy, ECL,, and Traffic datasets in the long-term forecasting task, we use prediction lengths of {96, 192, 336, 720), while for the PEMS dataset in the short-term forecasting task, we employ prediction lengths of {12, 24, 48, 96). Details of the dataset are provided in Table 4
Table 4: Dataset statistics. The dimension indicates the number of channels/variates, and the dataset size is organized in (training, validation, testing).
Tasks
Dataset
Dim.
Prediction Length
Dataset Size
Frequency
Domain
ETTml
7
(96, 192, 336, 720}
(34465, 11521, 11521)
15 min
Temperature
ETTm2
7
(96, 192, 336, 720}
(34465, 11521, 11521)
min
Temperature
ETThl
7
(96, 192, 336, 720}
|
(8545, 2881, 2881)
1 hour
Temperature
Long-term
ETTh2
7
(96. 192, 336, 720}
(8545, 2881, 2881)
hour
Temperature
Forecasting
Flectricity
321
(96, 192, 336, 720}
(18317, 2633, 5261)
Hourly
Flectricity
Traffic
862
(96, 192, 336, 720)
(12185, 1757, 3509)
Hourly
Transportation
Weather
21
{96. 192, 336, 720}
(36792, 5271, 10540)
10 min
Weather
Solar- Energy
137
(96. 192, 336, 720)
(36601, 5161, 10417)
10 min
Electricity
PEMS03
358
(12, 24, 48, 96}
(15617.5135 5135)
5 min
Transportation
Shor-term
PEMS04
307
{12, 24, 48, 96}
(10172, 3375, 3375)
5 min
Transportation
Forccusting
PEMS07
883
{12, 24, 48, 96}
(16911.5622. 5622)
min
Transportation
PEMS08
170
{12, 24, 48, 96}
(10690, 3548, 265)
min
Transportation
Hyperparameter search. Table 5 summarizes the hyperparameters and training settings used in our experiments. Our hyperparameter selection followed a systematic approach, combining grid search with domain-speeific considerations. The number of layers was fixed at 1, and the input length L was set to 96 to ensure fair comparisons across benchmark datasets. The pseudo length L' was configured based on the input dimensionality of each dataset. Larger values (L' 256) were assigned to datasets with more input channels, and smaller values (L' = 32) were used for datasets with fewer channels to balance computational efficiency with model capacity.
The selection of wavelet initialization types was guided by both systematic evaluation and signal processing principles. We explored common wavelet families {dbl, db4, db8, db12, bior3.1), with specific choices informed by the temporal characteristics (i.e., sampling frequency) of each dataset. Specifically, the dbl (Haar) wavelet was primarily employed for datasets exhibiting high total varia- tion (e.g., hourly-sampled datasets such as ETThi, ECL, Traffic) due to its effectiveness in capturing sharp transitions. Conversely, longer filters (bior3.1, db4, db8) were utilized for frequency data (c.g., minute-level datasets such as ETTm2, Weather, their smoother temporal patterns. We acknowledge that any of them are suitable initializations. The scale parameter S was fixed at 3.
PEMS04) to better capture
17

---

Published as a conference paper at ICLR 2025
For training parameters, performed grid search over learning rates within logarithmic scale from 10-3 3 to x 10-? Batch sizes and training epochs were systematically evaluated within the ranges (16, 24, 256} and {10, 20), respectively. Larger batch sizes (256) and fewer training epochs (10) were typically assigned to long-term forecasting tasks, while smaller batch sizes (16) and more training epochs (20) were used for short-term forecasting scenarios. This approach hyperparame- ter optimization enabled us to achieve good performance while accounting for the distinct temporal and structural characteristics of each dataset.
Fair comparison settings. To ensure fair comparison, we maintained a consistent lookback win- dow length of 96 across all experiments. Our baseline comparisons mimic the experimental pro- tocols established in TimesNet Wu et al. (2023), including identical data processing and splitting procedures. We applied early stopping to all baselines when the validation loss failed to decrease for three consecutive epochs. Recent baselines, such as I'Transformer Liu et al. (2024a), Time.Mixer Wang et al.] (2024), and CrossGNN [Huang et al.] (2023), adopted the same fair comparison set- tings. Therefore, their experimental configurations required no modifications, and we utilized their official repositories directly for reproduction. For baselines published prior to 2024, we used the long-term forecasting results provided in the TimesNet Wu et al. (2023) repository. These results were built on the experimental configurations provided by each model's original paper or official code. We verified that all hyperparameters for these baselines were selected from their respective official repositories while ensuring consistency with the fair comparison settings, where the only change were the input and output sequence lengths of all baseline models. Additionally, we adopted all baselines' short-term forecasting results directly from TimeMixer Wang et al. (2024). consistent with the fair comparison settings established by TimesNet.
Implementation Details. All experiments were conducted using PyTorch Paszke et al. (2019) on a single NVIDIA A100 40GB GPU. The model was trained using the Adam optimizer Kingma & Ba (013) with Mean Absolute Error (MAE) as the loss function specifically for the PEMS datasets, following TimeMixer Wang al. (2024). and Mean Squared Error (MSE) as the loss function oth- erwise.
Table 5: Summary of the experimental configurations for all datasets with prediction length of 96.
Dataset/ Configuration
Model Hyperparamcter
Training Process
Layers Input Length
Pseudo Length E'
Waveler Initialization Scale
Learning Rate
Attention
Batch Size
Epochs
ETTb]
96
dbl
10
Geometric
256
ETTh2
96
32
bioe3.1
6 10-3
Geometric
256
10
BTTml
db1
10
Geometric
256
BTTm2
1
96
32
bior3.1
x 10
Geometric
256
10
Weather
dhs
10
Geometric
256
10
Solar Energy
96
128
dbs
10
Geometric
256
10
Electricity
256
dbl
10
Geometric
256
Traffic
96
256
dbl
x 10
Geometric
21
10
PEMS03
256
bior3.1
10
Geometric
20
PEMSO4
96
256
bior3.1
2. 10
Geometric
16
20
PEMS07
256
db12
10
Geometric
20
PEMSO8
96
256
db1
10
Geometric
16
20
B COMPLETE RESULTS OF FORECASTING TASKS
B. FULL RESULTS
Long-term forecasting task. In the long-term forecasting results presented in Table of the main paper, we reported only the averaged performance across four prediction lengths due space con- straints. Tablef6provides a comprehensive breakdown of empirical results for each prediction length. Within each row, the lowest MSE and MAE scores highlighted in red, and the eecondllowest scores are underscored in blue. Our proposed method consistently achieves near top-2 performance across all evaluations.
Additional baselines in long-term forecasting, We evaluated our approach against conventional statistical time series forecasting methods, specifically ARIMA and ETS models, using the ETTh1
18

---

Published as a conference paper at ICLR 2025
Table 6: Complete results of the long-term forecasting task, with an input length of 96 for all tasks. The reported metrics include the averaged Mean Squared Error (MSE) and Mean Absolute Error (MAE) across four prediction horizons, where lower values indicate better model performance.
StapleTSE
POLATST
Conoderace
TIRE KENE
SCINGT
Stakeear
Veria
VINT
MEXTT
MENT
SANT
XISA
MESP
MINT
SENT
MEAT
RENIT
XISI MEAT
8.361
8-351
395
B-3T
9:315
8.77M
18h
0:475
14900
1000
8.77
8:90
B-00
136
8
199
8/481
0.501
8.080
0:41
8-53
0.910
16464
0.48
8499
04463
BAS9
8441
8442
0.40
0449
BA6S
14500
8.477
0.47%
9490
M4244
0.455
0.500
B243
0.555
0236
(4571
0.591
14301
0.59
1.20
140
BAT0
0.35
1.64
14414
0.46
0.49
B313
407
0.490
0.40
1.40
0.497
1440
0.40
6.465
1642
0.461
D429
14,00
0.37
0.357
017%
185
036
111
0356
0365
0.06
0.88
335
8100
0.32
0.56
0.79
0.367
0.93
0.55
0.355
0339
01216
0.500
0568
8:34
0.443
0.251
0.991
0.354
01:99
0.372
1.00
0.29
3460
14344
0.222
1770
04413
0.417
0443
0145
0412
1.351
DAD
0.50
811
0.57
15310
0.36
0.32
5371
0368
0.386
3365
B464
0.364
3440
0.306
0.76
1411
0.11
0446
0.64
5499
8.000
1300
11300
0.20
0.09
11,00
0395
0033
03211
6:00
8.07
8.06
01000
0.208
0.13
0449
0.5% 8455
1.413
0.40
8:44
1064
0.50
B679
0.53
1248
0.5%
0.474
0.62
0.515
0511
10353
0.500
3,427
14550
10054
0.722
01440
0.530
0.49
0489
0:36
8:588
0781
0-50
0.34
0.85
0.315
0.36
072
1130
0.30
0.331
8:33
0.297
0.331
14566
0.354
0.244
0.773
0.231
53511
0.35
0,79
0:466
00205
0.222
0321
0.256
0.561
10.01
6-36
1.:00
0.25
1:30
0.29
B:44
10:04
02331
0:44
0.99
3:59
0.390
0,96
0:214
0.35
0,55
14.27
0.335
0500
0:39
0.35
0:300
5555
0.96
3:11
8:00
0.78
6753
0.00 999
0-55
0.36
0.53
0.33
0.11
1377
13313
0.166
8:34
0:397
BA77
0.775
0.56
0.315
0.577
8337
9:51
1L314
B440
0.2%
0,20
1829
0.335
1239
3475
0333
0311
1000
0.530
0-36
0025
0381
0.50P
19376
0.53
0.55
5379
Morkes
11.90
8:18
0.990
0333
0.561
01775
0.196
0355
0.30
8:21
16266
0.36
5336
1036
10/07
0:30
0.20
0:88
1000
0.29
1000
2:27
(1:09)
5:00
1120
033
0..7%
03399
11,00
0.21
0.:0
0.76
8.387
0775
03385
8:30
0.99
0.30
0300
0.225
339
B416
0.246
0:118
0.20
0750
0 500
0385
0:90
0155
0337
0.54
0192
19206
0.97
0389
1.20
0315
B795
0058
3388
B-30
0.39
0330
0.333
6064
0.30
0
0.28
01311
0.352
0318
5.17
(324
0313
0:50
B417
3339
0330
0.31
0.111
dataset. The evaluation covered four prediction horizons {96, 192, 336, 7203. with standardized
preprocessing procedures applied across all models to ensure a fair comparison. Our proposed model consistently outperformed these methods across all prediction horizons, as shown in Table This performance gap widened with increasing prediction lengths, aligning with our observations of these models. Statistical methods like ARIMA and ETS require sufficient lookback period for robust parameter estimation and struggle with longer forecasting lengths due to error accumulation.
Table Performance comparison of additional statistical models on the ETTh1 dataset across vary-
ing prediction lengths.
Model
E9
199h
ETTh1-336
ETTh1-721
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
SimpleTM
0.366
0.392
0.422
0.421
0.440
0.438
0.463
0.462
ETS
|
1.145
0.658
1.185
0.855
1.234
0.963
2.298
1.818
ARIMA
I
1.010
0.719
1.033
0.635
1.204
0.700
2.269
1.072
We also compare our model's performance with zero-shot LLM Results Liu et al. 2024bb; Woo al. (2024); Goswami et al. (022)). We refer the zero-shot performance reported by TIMER Liu et al. (2024b). However, because LLM-based models often uses substantially larger context window (say 672 steps) than our 96-step lookback. direct comparison is inherently limited. To obtain a more rigorous assessment, one would need to match context windows and also evaluate fine-tuned versions of the I.M,, an effort that would require significant computational resources. Consequently, we focus demonstrating that specialized model trained for a fixed-horizon forecasting task can still achieve strong performance, as shown in Table g
Short-term forecasting task. For short-term forecasting, we conducted additional comparisons with TimeMixer Wang et al. (2024). the previous ofate-of-the-ar model, using its official repository and experimental configuration. The evaluation covered four prediction lengths {12, 24, 48, 96). Ta- ble preseenss the averaged RMSE/MAE values, along with their pooled standard deviations. Our model demonstrates consistent and statistically significant improvements across all PEMS datasets,
19

---

Published as a conference paper at ICLR 2025
Table Performance comparison of additional LLM-based models with prediction horizon of 96.
Dataset
SimpleTM
TIMER
TIMER 16B
TIMER 28B
MOIRAL-S
MOIRAI-M
MOIRAH-
MOMENT
ETThl
0,366
0.438
0.364
0.393
0,441
0,383
0.394
0,674
ETTh2
0.281
0.314
0.294
0.308
0.295
0.295
0.293
0.330
ETTml
0.321
0.690
0.766
0.420
0.562
0.448
0.452
0.670
ETTm2
0,173
0.213
0.234
0.247
0,218
0.225
0.214
0.257
ECL
0.141
0.192
0.139
0.147
0.212
0.162
0.155
0.744
Traffic
0.428
0.458
0.399
0.414
0.616
0,425
0.399
1.293
Weather
0.162
0,181
0.203
0.243
0.195
0.197
0.221
0.255
with error reductions ranging from 5.4% to 17.4%. The most substantial improvements were ob- served on PEMS08, where our model reduced RMSE by 15.8% and MAE by 17.4%. Notably, our model also shows more stable performance, as evidenced by the considerably smaller pooled standard deviations across all metrics and datasets.
B.2 STABILITY ANALYSIS
Pooled standard deviation. The pooled standard deviation is calculated as
Σ1 (ma o:)²
sd
4 (n 1)
where n is the repeat times, n - is the degree of freedom within each prediction length, indexes the prediction lengths, j indexes the repeats, Fij represents individual measurements, and T; is the mean of repeats for each prediction length.
Significance test. To establish statistical significance, used Type II ANOVA analysis assess the model effects (our proposed model versus other baseline model) while accounting for predic- tion length variations. The blocking design for prediction length effectively removed this source of variation from our етrоr term, and increased statistical power to detect genuine differences be- tween model architectures. The p-values reported Table test the null hypothesis that there is no difference in performance between the proposed model and the baseline model.
Results. We compare our proposed model with the second-best linear-based model, TimeMixer Wang et al. (2024) and third-best transformer-based model, i'Transformer Liu et al. (2024a) across three repeats and four prediction lengths for both long-term and short-term forecasting tasks. shown in Table T0, the pooled standard deviations are consistently smaller across all datasets, in- dicating the stability of our model's performance regardless initialization. The consistently low p-values (p 0.05) across all datasets confirm that the superior performance of model is sta- tistically significant and not attributable to random chance or prediction length variability. This is further supported by our additional short-term forecasting results with extended prediction lengths {12, 24, 48, 96).
C ABLATION STUDY
C.I ABLATIONS ON ARCHITECTURAL COMPONENTS
To rigorously validate our approach, we conducted additional experiments across four datasets (ETThl, ETTml, Weather, Solar-Encrgy with four prediction lengths, each repeated three times. Through systematic component ablation, we evaluated two key architectural elements: geometric attention mechanism and the stationary wavelet transform.
Geometric attention mechanism. For geometric attention, we performed a direct comparison with vanilla attention. While not every dataset benefits equally-depending the degree of cross-talk between channels-out findings show consistent performance improvements. For example, we ob- served 3.55% MSE reduction ETThh and 5.43% reduction on Solar-Energy, all achieved
20

---

Published as a conference paper at ICLR 2025
Table 9: Performance comparison of models on long-term forecasting tasks. The table reports the av- eraged Mean Squared Error (MSE) and Mean Absolute Error (MAE) across four prediction lengths, along with their pooled standard deviations (SD). Lower values indicate better model performance.
Dataset
Model
MSE (Pooled SD)
MAE (Pooled SD)
MSF p-value
MAF p-value
SimpleTM
0.166 (0.00 (0.0008)
0.260 (0. (0.0006)
ECL
TimeMixer
0.182 (0.0012)
0.272 (0.0006)
0.000 0.000
0.000 0.000
iTransformer
0.175 (0.0009)
0.267 (0.0008)
SimpleTM
0.422 (0.0015)
0.428 (0.0007)
ETTh1
TimeMixer
0,456 (0.0111)
0.444 (0.0071)
0.000 0,000
0.000
i'rrasffrmme
0.456 (0.0035)
0,448 (0.0024)
0.000
SimpleTM
0.353 (0.0021)
0.391 (0.0015)
ETTh2
TimeMixer
0.386 (0.0074)
0,407 (0.0043)
0.000
0.000
i'rrasformer
0.384 (0.0017)
0.407 (0.0010)
0.000
0.000
SimpleTM
0.381 (0.0009)
0.396 (0. (0.0008)
ETTml
TimeMixer
0.385 (0.0048)
0.399 (0.0032)
0.022
0.003
Transformer
0.408 (0.0012)
0.412 (0.0010)
0.000
0.000
SimpleTM
0.275 (0.0012)
0.322 (0.0011)
ETTm2
TimeMixer
0.278 (0.0026)
0.325 (0.0018)
0.001
0.000
iTransformer
0.292 (0.0011)
0.335 (0.0010)
0.000
0.000
SimpleTM
0.184 (0.0016)
0.247 (0.0031)
Solar
TimeMixer
0.237 (0.0088)
0.290 (0.0242)
0.000
0.000
iTranfformer
0,235 (0.0032)
0,262 (0.0010)
0.000
0.000
SimpleTM
0.440 (0.0013)
0.292 (0.0003)
Traffic
TimeMixer
0.497 (0.0087)
0,300 (0.0029)
0.000
0.000
iTransformer
0.422 (0.0008)
0.282 (0.0005)
0.000
0.000
SimpleTT
0.243 (0.0005)
0.271 (0.0007)
Weather
TimeMixer
0.245 (0.0012)
0.275 (0.0019)
0.000
0.000
iTransformer
0.261 (0.0023)
0.281 (0.0021)
0.000
0.000
Table 10: Performance comparison of models on short-term forecasting tasks. The table reports the averaged Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) across four predic- tion lengths И E {12, 24, 48, 96), along with their pooled standard deviations (SD). Lower values indicate better model performance.
Dataset
Model
RMSE (Pooled SD)
MAE (Pooled SD)
RMSE p-value
MAE p-value
PEMS03
SimpleTM TimeMixer
29.08 (0. 154)
17.96 (0.065)
31.73 (0.529)
19.22 (0.278)
0.000
0.000
SimpleTM
32.91 (0.121)
20.34 (0.077)
PEMS04
TimeMixer
34.78 (0.472)
21.99 (0.304)
0.000
0,000
SimpleTM
38.00 (0.139)
23.36 (0.085)
PEMS07
TimeMixer
40.65 (0.498)
25.44 (0.363)
0.000
0.000
SimpleTM
27.42 (1.)
17.09 (0.069)
PEMS08
TimeMixer
32.58 (2.453)
20.68 (1.776)
0.000
0.000
without any increase model parameters. The pooled standard deviations, as shown in Table I1 are small across all datasets, indicating that the performance advantages are stable/reproduciblc To test statistical significance, we performed a Type II ANOVA analysis with the null hypothesis that there is no difference in performance between baseline model with and without geometric attention.
The consistently low p-values (p < 0.05) across all datasets confirm that the observed improvements are statistically significant and not attributable to random chance or prediction length variability.
Stationary wavelet transform. For the SWT, we conducted three types of ablation experi- ments: (i complete removal of the SWT decomposition and reconstruction; (ii) replacement with the Fast Fourier Transform (FFT) as the tokenizer and inverse FFT as the de-tokenizer; (iii) replacement with seameter-matched 1-D convolution layers to ensure fair comparison. The Performance/Parameterr A% column in Table [1 the percentage change performance and total trainable parameters relative the baseline model.
The results in Table II suggest that removing SWT leads to substantial performance degradation, particularly evident in the Solar dataset where observed 34.8% MSE increase despite only
21

---

Published as a conference paper at ICLR 2025
reducing parameters by 5.27%. Even parameter matchhe alternatives underperformed compared to our model: replacing SWT with equivalent convolutions increased MSE by 16.8% on Solar and 9.8% on Weather. Additionally. FFT-based variants showed performance drops, with 20.7% and 9.8% MSE increases on Solar and Weather, respectively. All of these differences are statistically significant (p 0.01), showing that both architectural components contribute to the performance of our simple baseline presented here. Particularly interesting is the substantial performance gap be- tween SWT and its convolution-bast based replacement, which suggests that SWT's effectiveness stems from its multi-resolution analysis capabilities rather than merely adding model capacity.
Table 11: Ablation study results for different models across various datasets. Metries include Mean Squared Error (MSE) and Mean Absolute Error (MAE) with their pooled standard deviations (SD), along with percentage changes in performance and parameter counts relative to the baseline model. Lower MSE/MAE values indicate better performance, while a negative performance delta signifies performance degradation.
Modell SimpleTM
Number parameters changes!
MSS (Pooled SD) Perfomance WorS??
MAE (Ponted SD: Perticmancc warse? p-valuc p-vulue
0:422 (0.0015)
0.428 (0.007))
w/o GlanmAto w/o SWT
None
0.457 (0.0011)
Yes, by - -1.55
(0.000))
Yes, by -2. 2..005
0.000
01.000
ETThl
Yes. 0.436%
0.452 (0.0050)
Yes. by 33%%
0.435 (0/001)))
Ves. by 1.00%
0:000
0.000
Yes, by
Yes, by
PFT-ST
+0,436%
0.435 (010640)
SimpleTM
0.381 (0.0009)
0.395 (0.0028)
w/o ClamAno
None
0.385 (0.0011)
Yes, by -1 1.05%
(0.000))
Yes, by -0: 0.515
0.000
8.000
cTTml
40.436%
Yos. by
Yos. by 48.51%
None 40,436%
00255)
Yes. by
by -0.76%
PFT:SW
0.399 (000009)
SimpleTM
0. (0.0066)
0.247 (0.0631)
w/o GeomAAtt
None
0 194 (0.0122)
Yes, by 5.43%
0.253 0.253 (0.012))
Yes, by 2.40%
0.018
0.120
Solar
You, -..7%
(0.00110)
Yos. by
Yes. by 17.0PP
None
Yes. by
Yes, by
PFT:SW
Yes, -5.27%
2.222 (0.0187)
-20.7%
(0.0158))
SimpleTM
0.243 (0.0005
0.271 (00007)
w/o Geom/Att
None
0.265 (0.0021)
Yes, by 0.82%
0.275 (00018)
Yes. by 0.74%
0.007
0,002
Weather
40.426%
(0.081:4)
Yos. by 1.65%
(8.00055
Yes. by -1.11%
None
Yes. by
by
FFT:SW
+0,426%
(0.0006)
-9.85%
0.286 (0.0025)
-3.54%
C.2 ADDITIONAL. ABLATIONS
Inter-channel dependencies. In our tested datasets, recent results (e.g., PatchTST Nie et al. (223)
have shown that individual channels were often sufficient for making reasonable forecasts, indicating limited direct correlations between channels. However, experiments show that incorporating all channels in the token embedding improves forecasting performance compared to single-channel embeddings (where the bivector reduces a scalar), as shown in Table 12
This improvement likely comes from how our attention mechanism uses the channel information. While not explicitly mixing channels through projection layers, it computes attention weights using all channels simultancously as well as using cross cross-channel relationship through the wedge product. This allows features across all channels to collectively determine how much each token's 's full channel vector contributes to the final representation, creating an implicit form of channel interaction. We hypothesize that the model adaptively captures useful channel relationships when they exist, while avoiding imposing artificial correlations when they do not.
Table 12: Performance comparison of SimpleTM with and without the Independence feature across three datasets. Metrics include Mean Squared Error (MSE) and Mean Absolute Error (MAE) with pooled standard deviations (SD), along with p valuus for statistical significance.
Dataset
Model
MSE (Pooled SD)
MAE (Pooled SD)
MSE p-value
MAE p-vulue
SimpleTM + Independence
0.422 (0.0015)
0.428 (0.0007)
ETTh1
0.451 (0.0073)
0.444 (0.0035)
0.000
0.000
SimpleTM + Independence
0.381 (0.0009)
0.396 (0.0008)
ETTml
0.394 (0.0079)
0.400 (0.0055)
0.000
0.007
SimpleTT + Independence
0.243 (0.0005)
0.271 (0.0007)
Weather
0.268 (0.0024)
0.286 (0.0019)
0.000
0.000
22

---

Published as a conference paper at ICLR 2025
D EFFICIENCY ANALYSIS
To provide a thorough efficiency comparison, we evaluated our model against two of the most competitive baselines: the transformer-based i'Tranffrmmer Liu et al. (2024a) and linear-based TimeMixer WWang et al. (2024). Our experimental setup used consistent batch size of 256 across all models and measured four key metrics: total trainable parameters, inference time, GPU memory footprint, and peak memory usage during the backward pass. Results for all baseline models were compiled using PyTorch.
Our findings demonstrate remarkable efficiency improvements: On the Weather dataset, our model achieves better accuracy while using only 0.3% of iTransformer's parameters (13K vs 4.8M) and 13% of TimeMixer's parameters (13K vs 104K). Our memory footprint is 38% smaller than iTrans-- former's and 66% smaller than TimeMixer's. iTransformer and 3.4x faster than TimeMixer. These efficiency gains are even more pronounced on the larger Solar-Energy dataset, where our model uses just 1.3% of TimeMixer's parameters (166K vs 13M) while achieving 24% better accuracy. Our memory consumption is 73% lower than TimeMixer's s, and inference speed is 5.8x faster. Notably, these improvements come without com- promising performance, datasets.
terms of speed, our model is 1.7x faster than
model maintains superior or comparable MSE scores across both
In the reported experiments, prioritize memory and computation efficiency by choosing the bivector's s magnitude for the reduction function <(-). However, we have a fair bit of flexibility to upgrade the reduction function later for additional performance gains.
Table 13:
Comparison of model performance and resource utilization across different datasets. Met- rics include Mean Squared Error (MSE), total parameter count, inference time (seconds), GPU mem- ory footprint (MB), and peak memory usage (MB).
Dataset
Model
MSE
Total Params
Inference Time (s)
GPU Mem Footprint (MB)
Peak Mem (MB)
SimpleTM
0.162
13,472
0.0132
994
181.75
TimeMixer
0.164
104,433
0.0453
2,954
2,281.38
Weather
iTransformer
0.176
4,833,888
0.0222
1,596
847.62
SimpleTM
0.163
166,304
0.0455
2,048
1,151.56
TimeMixer
0.2155
13,009,079
0.2644
7,576
6,632.40
Solar
iTransformer
0.203
3,255,904
0.0663
4,022
2,776.50
Potential benefits from Parallelization. While our current implementation follows standard SWT approach, additional opportunities for further optimization are possible. In the standard SWT, each level's approximation is convolved with an upsampled wavelet filter to yield the next level's ap- proximation and detail coefficients. Because no downsampling occurs, there is no inherent data dependency between levels, yet an iterative formulation does not exploit this property. By rec- ognizing that each scale be expressed directly convolution of the original signal with an appropriately composite filter, one can bypass the step-by-step procedure and compute all levels in parallel. This approach leverages convolution's 's associativity to collapse the iterative chain into set of direct convolutions, each mapping from the original input to a given scale's approximation or detail.
Let h-2--1 and 9+2--1 denote the upsampled low-past and high- pass filters at level s by inserting 2--1 zeros between each tap. We can define composite filters H, and G, via the recurrences
H, H.-1 h42 and G, H J12" 1, for 1,
starting from Hi = h and G₁ = This recurrence builds up the composite filter by succes- sively convolving the upsampled filters from previous levels, capturing the cumulative filtering. By associativity, the outputs at level a become
(a) u Hg and d(s) = u G.,
thus eliminating the need to explicitly compute (=-1) first. This parallelized approach would allow all convolutions to be applied directly to the original signal u and launched simultancously on mod- hardware accelerators, potentially reducing computational overhead in real-time large-scale
23

---

Published as a conference paper at ICLR 2025
wavelet-based applications. The direct implementation of these convolutions using the composite filters would involve operations with dense matrices, giving up at least some of the parallelization gains. To address this, further optimization using structured matrices, such as circulant matrices. This circulant approximation replaces the upsampled filters with wrapped versions, allowing the of FFT. Here, are leveraging the shift- invariance of circulant matrices, with the understanding that trade-off's exist between speed and accuracy. We leave the implementation and evaluation of these modifications for future work, but note that it is promising direction for improvements.
be achieved by approximating these convolutions
E SHOWCASING AND CHECKING FORECASTING CAPABILITIES
Our forecasting model demonstrates its ability to predict trends across various time series datasets, including ECL, Traffic, Solar Energy, and Weather. Each example uses 96-step input to generate 96-seep predictions. In the visualizations, the blue lines represent the lookback window, the orange lines indicate the ground truth forecasting window, and the red lines show the model's predictions. The model's ss strengths lie in pattern recognition and trend prediction. It is good at identifying and
extrapolating recurring patterns, particularly evident in the Traffic dataset (Fig 8 where accurately captures cyclical nature and oscillations. In the Solar-Energe dataset (Fig 10 the model successfully
predicts overall directional trends.
However, there are areas for improvement. The model sometimes struggles with precise amplitude prediction, as seen in the ECL dataset, where predicted peaks and troughs do not always align perfectly with the ground truth. Phase shifts between predicted and actual values are also observed in some Traffic dataset forecasts (Fig suggesting a need for improved timing mechanisms. Handling anomalies shows another challenge. The model occasionally struggles with sudden spikes or dips, particularly evident in the Solar Energy dataset. Additionally, in longer predictions, the model shows signs of instability drift, as observed in certain forecasts for the ECL and Weather dataset.
In summary, while the model demonstrates adaptability to different scales and patterns, there's room for improvement in amplitude accuracy. phase alignment, anomaly handling, and long-term stability. Future work should focus on addressing these limitations to enhance the model's 's robustness and accuracy across diverse time series forecasting tasks.
24

---

Published as a conference paper at ICLR 2025
-10
Figure 8: Forecasting examples from the ECL dataset with a 96-step input and 96-step predictions.
Figure 9; Forecasting examples from the Traffic dataset with a 96-step input and 96-step predictions.
25

---

Published as a conference paper at ICLR 2025
-130
Figure 10: Forecasting examples from the Solar dataset with 96-step input and 96-step predictions.
Produced serio
MVLL
0.4
202
20
300
1/2
1,10
300
3.4
M/1
Figure 11: Forecasting examples from Weather dataset with 96-step input and 96-step predictions.
26


Reviews:
[{"review_id": "MY6jFHcbDq", "reviewer": "Reviewer_L7y3", "summary": "The authors propose a model for multivariate time-series (MTS) forecasting that utilizes a stationary wavelet transform (SWT) for tokenization, a geometric product-based self-attention mechanism to capture inter-channel relationships, and a learnable inverse SWT to reconstruct the signal in the time domain. The experiments show that the model demonstrates competitive performance on both long-term and short-term forecasting tasks, outperforming state-of-the-art methods like TimeMixer and iTransformer on a variety of datasets. The ablation experiments show that the geometric attention mechanism consistently improves performance, and that the wavelet-based tokenization provides an inductive bias for learning data-driven filters. Despite the above, the authors note that while the model is effective and lightweight, it may not scale well to larger datasets or more complex tasks, such as token-by-token generation. They also emphasize the potential for their model components to inform future work in specialized tokenization and embedding schemes.", "questions": "* Appendix is referenced in Table 1 but does not exist in the submission\n\n* The authors mention that the difference in results between fixed and learned wavelet filters is minimal - is there insight into why this is the case? In addition methods such as discrete fourier transforms are widely used in time series analysis. While the use of wavelet transform is well motivated, it would be beneficial to characterize other methods in context of the proposed tokenization scheme\n\n* Experimental results for TimeMixer, the closest baseline to the proposed model, differ from those reported in the original paper, for both long and short term forecasting. Where does the difference come from? Error bars will help to assess if differences might be significant or not", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written with easily understandable structure\n\n* The experiments are extensive, covering 8 multi-variate time series datasets and 15 baseline models that were compared against the proposed model\n\n* The proposed tokenization and attention schemes show notable performance improvements over previous works across datasets. The idea is simple and effective as demonstrated by the experiments. It is worth noting that the model is computationally efficient with a low memory footprint, making it feasible for resource-constrained settings", "weaknesses": "* While the model performs well on smaller datasets, the authors acknowledge that it may not scale efficiently to larger datasets or more complex tasks. The model's architecture is not well-suited for token-by-token generation, and larger models may be needed as datasets grow in size.\n\n* The paper could be strengthened by including details on the number of parameters in the proposed model compared to other large-scale models, as well as results exploring the impact of increasing the number of layers in the proposed model. Providing additional information about the characteristics of the datasets used and the hyperparameter search settings would also improve clarity and improve reproducibility", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CcybdIj6dH", "reviewer": "Reviewer_Lbi6", "summary": "The paper presents a new model to do multivariate point forecasting.\n\nThe model is based on a transformer architecture, with two main modifications. The first one is that the usual dot product in the attention mechanism is replaced by a combination of a dot product and a wedge product. The second one is that the original input, after being transformed by a trained linear layer, is then transformed by a trained wavelet layer into various channels which contains information about various time frequencies.\n\nThe model is then compared with various state-of-the-art point forecasting models, and comes up ahead. An annealing is also made where the model is tested against itself, but without its wedge product modification of the attention mechanism, and it shows that the version with the wedge product is slightly better than the version without it.", "questions": "1. In figure 6b, the filtering procedure makes the correlations between time steps very close to 1, and never negative. This is counterintuitive, since there doesn't seem to be any bias which should favor positive correlations instead of negative correlations in the wavelets parametrization. Also, having only correlations close to 1 should make it harder for a model to distinguish the information contained in the initial signal. Do you have an explanation for how the model is still able to function with this correlation pattern?\n2. Some datasets have a tendency to have almost no meaningful correlation between channels (the single channel forecast being mostly enough to do good forecasts). Do you know whether the datasets you have tested your model have this property or not?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The model is compared with a vast array of recent timeseries models, on standard testing datasets.\n2. The idea of using a wedge product in the attention mechanism to allow the model to give more weight to dissimilar tokens seems novel.", "weaknesses": "1. A lot of important hyperparameters of the models are missing. Such as $\\zeta$, $L'$, and others.\n2. The training procedure is missing. It would be useful to know how reliable the results are when training the models multiple times from various randomized initial parameters.\n3. Ablation experiments are lacking. Firstly, the minimal score improvement when adding the wedge product to the attention mechanism could easily be caused by the associated addition of more trainable parameter. A proper comparison should be done where the extra parameters are added to the traditional scalar product version of the attention mechanism. Secondly, the usefulness of the wavelet transform should also be tested through ablation. For example, by having replacing by two linear layer (or a MLP), one in the encoding, and one in the decoding part of the model.\n4. More detailed information should be provided concerning the number of parameters and memory footprint of this model, in particular in how they compare to those of other models. One concerning point is that only a constant memory footprint is provided, while the number of tokens sent to the attention layer should grows with the dimensionality of the dataset, which should therefore lead to a variable memory footprint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZdpThIJuyC", "reviewer": "Reviewer_avrv", "summary": "This paper introduces a simplified transformer-based model for multivariate time series (MTS) forecasting that performs competitively with larger models while using only 1-2 layers. The proposed architecture includes three key innovations:\n\nStationary Wavelet Transform (SWT) Tokenization: The authors apply SWT to efficiently capture local and global dependencies across different time frequencies. Unlike discrete wavelet transforms, SWT maintains time invariance by avoiding downsampling, allowing the model to capture both short- and long-term patterns with high temporal resolution.\n\nGeometric Product Attention Mechanism: To address the multidimensionality in MTS data (including variate, time, and frequency dependencies), the authors adapt a Clifford Algebra-inspired attention mechanism to capture both magnitude-based and spatial orientation relationships among tokens. This lightweight modification enhances interpretability and accuracy compared to the traditional dot-product self-attention.\n\nLearnable Inverse SWT (ISWT) for Reconstruction: Finally, the model employs a learnable ISWT module to reconstruct the MTS from the processed time-frequency tokens, facilitating a more accurate and coherent forecast by synthesizing the multi-scale information captured in the initial stages.\n\nExperiments and Analysis:\nThe authors evaluate their model on diverse MTS datasets for both short- and long-term forecasting, demonstrating competitive or superior performance relative to established baselines. They also perform ablations, analyze learned filter effects in decomposition, and examine multi-scale forecasts, showcasing the model’s ability to capture both macro trends and finer details across scales.", "questions": "Overall, this paper has the potential to be a good contribution because the motivation is sound. After getting clarifications for the concerns below, mostly on the experiments and their results, I would consider increasing my score.\n\n- For the lack of statistical significance tests: Could the authors provide statistical tests for Tables 1 and 2 to demonstrate whether the observed performance differences are statistically significant?\n\n- Regarding the ablation study: Could the authors clarify if the ablation study evaluates the impact of the Stationary Wavelet Transform (SWT) module, and if so, how this contribution is isolated and measured?\n\n- For LLM-based models: Is any of the baselines LLM-based? If not, how does thus the claim that the proposed model is competitive with much larger, even LLM-based models, verified?\n\n- For model size comparison: Could the authors include model size comparisons and discuss how their model compares in size and parameter count to the baselines? Moreover the discussion on compute and memory footprint can be extended to multiple competitive baselines to get a better understanding of where the proposed model stands.\n\n- About TimeMixer results: Could the authors confirm whether the TimeMixer results were obtained from scratch or directly from prior work? Have they checked the results from the official TimeMixer repository (https://github.com/kwuking/TimeMixer)? Both hyperparameter-tuned and non-tuned results seem much better than the ones reported in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- While wavelet transforms have been applied in forecasting before, this is the first known application for tokenization within a transformer architecture for time series. The approach is well-motivated, addressing the limitations of viewing time series as a single token stream and effectively handling various seasonalities and frequencies. The proposed end-to-end architecture builds on this foundation and includes complementary modules to support the overall design.\n\n- The paper is largely well-written, with comprehensive mathematical explanations for each module. Visuals effectively clarify the concepts, making the technical details accessible.", "weaknesses": "- Results in Tables 1 and 2 lack statistical tests to validate significance. For instance, in Table 1, differences between the proposed model and the next-best model are minimal for half of the datasets, with changes ranging from 0.001 to 0.004, which may not be statistically significant.\n\n- The ablation study is challenging to interpret. While the authors mention component replacement and removal experiments, the table only compares geometric versus vanilla attention across datasets. It’s unclear whether the study assesses the contribution of the Stationary Wavelet Transform (SWT), which is fundamental to the architecture.\n\n- The authors claim their model is competitive with much larger, even LLM-based, models. However, there’s no discussion of model sizes of each baselines. Moreover there is no LLM-based baselines in the comparisons (correct me if I am wrong), which weakens this claim.\n\n- The TimeMixer results in Table 1 is not directly comparable to results shared in their official repository. With the official TimeMixer results, the proposed model’s competitive standing may be less compelling, though further exploration could be valuable if the proposed model size is indeed significantly smaller as authors claim.\n\nThings to improve the paper that did not impact the score:\n\n- The introduction reads more like a related work section, making it difficult to grasp the proposed method and its motivation. It heavily references prior work, which distracts from understanding the main contribution. Much of this content might be more suitable for the related work section.\n\n- A background section on Wavelet transformations would be helpful for readers unfamiliar with the concept. Currently, such readers may need to consult external sources to follow the paper. If space is limited, this background could be included in the appendix.\n\n- The illustrations in Figures 6 and 7 are quite small, which makes them difficult to interpret. Enlarging these figures would improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8KDpSso2Bf", "reviewer": "Reviewer_8zFK", "summary": "The authors present a simplified approach to multivariate time series forecasting by using stationary wavelet transforms (SWT) for tokenization and geometric product attention. The use of geometric product attention allows the model to capture both the magnitude-based similarity and geometric relationships between tokens. The authors state that even a single or two layer model, using this architecture, yields results competitive with much bigger models on most benchmarks reported in the literature.", "questions": "Please see weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality and Significance: The paper leverages SWTs for time series tokenization, a concept inspired by recent successes in image processing with transformers. The authors go beyond simply applying SWTs by introducing learnable filters within the SWT framework. Further, the paper's most original contribution lies in its introduction of the geometric product attention mechanism. All of these contribute to the originality of the paper.\n\nQuality: The paper demonstrates the effectiveness of its approach through rigorous experimental evaluation on a diverse range of benchmark datasets. The model achieves state-of-the-art performance on numerous long-term and short-term forecasting tasks, outperforming established baselines like TimeMixer and iTransformer.", "weaknesses": "Clarity: Sections explaining the method (3,4,5) are too long, and can be summarized first before delving into the details. Also, the contributions of the paper needs to be summarized in the introduction, which is not currently done. (Note that I currently don't consider these points in my score, and they are suggestions)\n\nMore empirical evaluation necessary. These are the major points that made me choose a score of 6 for this paper. Addressing them can potentially increase my score: \n\n* It would be worth adding simple baselines like ARIMA and ETS, just so we know how much better the transformer-based models are. \n* At the same time, it would be worth adding a note on the number of parameters in each model, and the inference time of each model, which are both important aspects for practical forecasting. \n* Especially since a new geometric product attention mechanism is proposed, detailing the memory footprint and computational complexity of the mechanism compared to that of other models would be useful. \n* The results don't report the standard error. It's not unclear how many seeds were used in the experiments for each model.  \n* The authors claim that the mechanism captures inter-channel dependencies, but there is no empirical evidence that this is being useful for forecasting. An ablation to demonstrate that would be useful. You can plot the interchannel dependencies learned for each dataset and also show that meaningful dependencies were learnt.\n* Details on training: Since the baselines and the proposed model were trained for each dataset, the authors should provide details on how the baselines were trained (how hyperparameter selection was done, what protocol was used etc.) in the appendix. It is important to check if the evaluation compares all models in a fair manner. This is important to check how sensitive the model is to different hyperparameter values.\n\n**Update**: My concerns are addressed. Thanks to the authors for the efforts. I increase my score to 8, but I want to note that I consider this paper a 7, considering the strengths of the paper and the additions that the authors have done (I am unable to choose 7 in OpenReview).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)