Title: OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time series forecasting; Linear-based model; Transformer
Abstract: This paper presents $\mathbf{OLinear}$, a $\mathbf{linear}$-based multivariate time series forecasting model that operates in an $\mathbf{o}$rthogonally transformed domain. Recent forecasting models typically adopt the temporal forecast (TF) paradigm, which directly encode and decode time series in the time domain. However, the entangled step-wise dependencies in series data can hinder the performance of TF. To address this, some forecasters conduct encoding and decoding in the transformed domain using fixed, dataset-independent bases (e.g., sine and cosine signals in the Fourier transform). In contrast, we propose $\mathbf{OrthoTrans}$, a data-adaptive transformation based on an orthogonal matrix that diagonalizes the series' temporal Pearson correlation matrix. This approach enables more effective encoding and decoding in the decorrelated feature domain and can serve as a plug-in module to enhance existing forecasters. To enhance the representation learning for multivariate time series, we introduce a customized linear layer, $\mathbf{NormLin}$, which employs a normalized weight matrix to capture multivariate dependencies. Empirically, the NormLin module shows a surprising performance advantage over multi-head self-attention, while requiring nearly half the FLOPs. Extensive experiments on 24 benchmarks and 140 forecasting tasks demonstrate that OLinear consistently achieves state-of-the-art performance with high efficiency. Notably, as a plug-in replacement for self-attention, the NormLin module consistently enhances Transformer-based forecasters. The code and datasets are available at https://github.com/jackyue1994/OLinear.

OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain

Abstract

This paper presents OLinear , a linear -based multivariate time series forecasting model that operates in an o rthogonally transformed domain. Recent forecasting models typically adopt the temporal forecast (TF) paradigm, which directly encode and decode time series in the time domain. However, the entangled step-wise dependencies in series data can hinder the performance of TF. To address this, some forecasters conduct encoding and decoding in the transformed domain using fixed, dataset-independent bases (e.g., sine and cosine signals in the Fourier transform). In contrast, we utilize OrthoTrans , a data-adaptive transformation based on an orthogonal matrix that diagonalizes the series' temporal Pearson correlation matrix. This approach enables more effective encoding and decoding in the decorrelated feature domain and can serve as a plug-in module to enhance existing forecasters. To enhance the representation learning for multivariate time series, we introduce a customized linear layer, NormLin , which employs a normalized weight matrix to capture multivariate dependencies. Empirically, the NormLin module shows a surprising performance advantage over multi-head self-attention, while requiring nearly half the FLOPs. Extensive experiments on 24 benchmarks and 140 forecasting tasks demonstrate that OLinear consistently achieves state-of-the-art performance with high efficiency. Notably, as a plug-in replacement for self-attention, the NormLin module consistently enhances Transformer-based forecasters. The code and datasets are available at https://github.com/jackyue1994/OLinear .

1 Introduction

Multivariate time series forecasting is critical in fields such as weather [Wu et al., 2023a], transportation [Ma et al., 2021], energy [Zhou et al., 2021], and finance [Chen et al., 2023]. Time series forecasters typically adopt the temporal forecast (TF) paradigm [Wang et al., 2025a, Liu et al., 2024a, Nie et al., 2023, Wang et al., 2024a], which encodes time series into latent representations and decodes them back, all within the time domain. However, this paradigm struggles to fully exploit the forecasting potential in the presence of entangled intra-series dependencies [Yi et al., 2023, Yue et al., 2025]. To mitigate this issue, recent studies apply Fourier [Yi et al., 2024a, 2023, Yue et al., 2025] or wavelet [Masserano et al., 2024] transforms to obtain the decorrelated feature sequence and perform encoding and decoding in the transformed domain. Nevertheless, these methods rely on dataset-independent bases, which fail to exploit the dataset-specific temporal correlation information.

In this paper, we introduce OrthoTrans , a dataset-adaptive transformation scheme that constructs an orthogonal basis via eigenvalue decomposition of the temporal Pearson correlation matrix [Gray and Davisson, 2004]. Projecting the series onto this basis obtains the decorrelated feature domain, providing a disentangled input for linear encoding and empirically improving forecasting performance. Notably, OrthoTrans is modular and can be integrated into existing forecasters to enhance their performance. In-depth ablation studies reveal that OrthoTrans promotes representation diversity and increases the rank of attention matrices in Transformer-based models.

As OrthoTrans transforms complex temporal variations into decorrelated features, the representation learning process can be effectively handled by linear layers [Yi et al., 2023]. Specifically, we employ a linear-based Cross-Series Learner (CSL) and Intra-Series Learner (ISL) to model multivariate correlations and sequential dynamics, respectively. To motivate our design, we note that in the classic self-attention mechanism, the attention entries are all positive with the row-wise L1 norm fixed as 1. Inspired by this, we design a new lin ear layer in CSL, called NormLin , where the weight matrix entries are made positive via the Softplus function and then row-wise norm alized. Surprisingly, the NormLin module consistently outperforms the self-attention mechanism while improving computational efficiency in the field of time series forecasting. The overall model is referred to as OLinear , and our contributions can be summarized as follows:

In contrast to the commonly used TF paradigm, we introduce OrthoTrans , a datasetadaptive transformation scheme which leverages the orthogonal matrix derived from the eigenvalue decomposition of the temporal Pearson correlation matrix. As a plug-in, it consistently improves the performance of existing forecasters.

For better representation learning, we present the NormLin layer, which employs a rownormalized weight matrix to capture multivariate correlations. Notably, as a plug-in, the NormLin module improves both the accuracy and efficiency of Transformer-based forecasters. It also adapts well to decoder architectures and large-scale time series models.

Extensive experiments on 24 benchmarks and 140 forecasting tasks (covering various datasets and prediction settings) demonstrate that OLinear consistently achieves state-ofthe-art performance with competitive computational efficiency.

2 Related work

2.1 Transformed domain in time series forecasting

Deep learning-based forecasters typically adopt the TF paradigm [Wang et al., 2025a, 2024a, Yu et al., 2024, Wang et al., 2024b, Liu et al., 2024a, Nie et al., 2023, Wu et al., 2023b, Zeng et al., 2023], with the entire process performed in the time domain. However, this paradigm may underperform in the presence of strong intra-series correlations [Yi et al., 2023, Wang et al., 2025b]. To mitigate this, recent models propose to forecast in the frequency domain [Yi et al., 2024a, Xu et al., 2024, Yi et al., 2023] or the wavelet domain [Masserano et al., 2024] where step-wise correlations are reduced and performance improvements are observed. FreDF [Wang et al., 2025b] incorporates a frequency-domain regularization term into the loss function to address the issue of biased forecast. However, classical Fourier and wavelet transforms employ fixed, dataset-agnostic bases that do not explicitly use the specific statistical characteristics of the dataset. In this work, we leverage orthogonal matrices derived from the temporal Pearson correlation matrix to decorrelate the series data. This approach provides a more suitable input for linear encoding and improves forecasting performance.

2.2 Time series forecasters

Deep learning based time series forecasters can generally be categorized into Transformer-based [Wang et al., 2025a, Nie et al., 2023, Liu et al., 2024a, Wang et al., 2024b, Yu et al., 2024], linearbased [Wang et al., 2024a, Zeng et al., 2023, Yi et al., 2023, Xu et al., 2024, Yi et al., 2024a], TCN-based [Wu et al., 2023b, Luo and Wang, 2024], RNN-based [Lai et al., 2018, Rangapuram et al., 2018], and GNN-based [Huang et al., 2023a, Yi et al., 2024b] models. Recently, research interests increasingly focus on Transformer-based and linear-based methods, each offering distinct advantages. Transformer-based forecasters typically exhibit strong expressiveness, whereas linear-based models offer better computational efficiency. In this paper, we aim to achieve state-of-the-art performance using an efficient linear-based model. Specifically, we employ the linear-based NormLin module to

Instance

Norm.

Flatten & Linear

Inverse Instance

Norm.

Dimension

Extention

Input

Output

Mat

Mul

Mat

Mul

Linear

Encoding

Orthogonally Transformed Domain

Cross-Series

Learner (CSL)

Intra-Series

N

d

τ





Cross-Series Linear Learner

Linear

Mat

Mul

Learn.

Softplus

Linear

Add &

LayerNorm

NormLin

Layer

L1 Norm

Row-wise

Intra-Series Linear Learner

Linear

Add &

LayerNorm

Linear

GELU

Representation Learning

OrthoTrans

NormLin

Module

N

T



N

T



N

d

T





N

d

T





T

T



T

i

Q

N

d

D





N

d

D





τ

τ



o

Q

N

d

D





N

τ



N

τ



Linear

Decoding

N

N



L



N

τ



robustly model inter-variate correlations. Remarkably, the NormLin module consistently outperforms the self-attention mechanism in Transformer-based forecasters, highlighting the strong capability of linear layers for time series forecasting.

3 Preliminaries

Real-world time series often exhibit temporal dependencies, where each time step is influenced by its predecessors. In this section, we derive the expected value of a future step given its past, assuming a multivariate Gaussian distribution. Without loss of generality, we consider a series x ∈ R t , and the following time step y ∈ R . We now present the following theorem.

Theorem 1 (Expected value of y ) . We assume that x ∼ N ( µ x , Σ x ) , y ∼ N ( µ y , σ 2 y ) , and the joint distribution [ x , y ] ∼ N ( [ µ x , µ y ] , [ Σ x Σ x y Σ T x y σ 2 y ]) . Here, N ( · , · ) denotes a Gaussian distribution t × t

with the specified mean vector and covariance matrix. Σ x ∈ R is symmetric and positive definite. Σ x y ∈ R t × 1 denotes the covariance between x and y . Then, the expected value of y given x is

The proof of Theorem 1 is provided in Appendix A. Equation 1 shows that for a temporally correlated series [ x , y ] , the expected value of y depends not only on its own mean µ y , but also on its past observations x . The second term in Equation 1 introduces additional difficulty to forecasting under the TF paradigm. In contrast, decorrelating the series simplifies the forecasting task. In this work, we introduce OrthoTrans to transform the original series into a decorrelated transformed domain, converting the temporal forecasting problem into the feature prediction task. Experiments in Section 5.2 validate the effectiveness and generality of this transformation strategy.

4 Method

Problem formulation For multivariate time series forecasting, given a historical sequence X = { x 1 , · · · , x T } ∈ R N × T with T time steps and N variates, the task is to predict the future τ time steps Y = { x T +1 , · · · , x T + τ } ∈ R N × τ . Our goal is to approximate the ground truth Y as closely as possible with predictions ˆ Y .

4.1 Overall architecture

As shown in Figure 1, OLinear adopts a simple yet effective architecture. Given the input series X , a RevIN [Kim et al., 2021] layer first performs instance normalization to mitigate non-stationarity. A dimension extension module then enhances expressiveness [Yi et al., 2023] by computing the outer product with a learnable vector ϕ d ∈ R d , where d is the embedding size. Next, the time domain is decorrelated by multiplying with a transposed orthogonal matrix Q T i ∈ R T × T , which will be detailed in Section 4.2. This process can be formulated as Z = (RevIN Norm ( X ) ⊗ ϕ d ) Q T i ∈ R N × d × T , where ⊗ denotes the outer product.

We then perform encoding and forecasting in the transformed domain. Specifically, a linear layer first encodes the decorrelated features Z to the model dimension D . The cross-series learner (CSL) and intra-series learner (ISL) subsequently capture multivariate correlations and model intra-series dynamics, respectively. After passing through L stacked blocks, the representation is decoded to the desired prediction length τ , and then mapped back to the time domain via multiplication with the orthogonal matrix Q o . The overall process is summarized as:

Finally, the flattened output is mapped to shape N × τ via a linear layer, then de-normalized to yield the final prediction: ˆ Y = RevIN DeNorm ( FlattenLinear ( ˜ Y )) .

4.2 Orthogonal transformation (OrthoTrans)

One effective approach for decorrelating the series is based on the Pearson correlation matrix. Let X train ∈ R N × M denote the training set, where M is the length of the training series. For each variate j , we generate T lagged series with temporal offsets from 0 to T -1 : s j i = X [ j, i : M -T + i ] , i = 0 , · · · T -1 , where s j i denotes the i -th temporally lagged series of variate j . We then compute the Pearson correlation matrix CorrMat j t of { s j i } , whose ( p, q ) -th entry is Cov( s p , s q ) √ Var( s p ) · Var( s q ) , p, q ∈ { 0 , · · · , T -1 } .

Here, Cov( · ) denotes the covariance of two series, and Var( · ) denotes the variance. The final Pearson correlation matrix is then obtained by averaging over all variates: CorrMat t = 1 N ∑ N -1 j =0 CorrMat j t .





































t





































Following the above procedure, the resulting CorrMat t is symmetric with all diagonal entries equal to 1. Based on the properties of symmetric matrices, we perform eigenvalue decomposition as CorrMat t = Q i Λ Q T i , where Q i is an orthogonal matrix [Horn and Johnson, 2012] whose columns are the eigenvectors of CorrMat t [Bishop and Nasrabadi, 2006], and Λ is a diagonal matrix containing the corresponding eigenvalues.

For the input series x with temporal correlation matrix CorrMat t and unit variance, the covariance matrix of the transformed vector Q T i x is CovMat ( Q T i x ) = Q T i CorrMat t Q i = Q T i Q i Λ Q T i Q i = Λ , which is a diagonal matrix. Therefore, the entries of Q T i x are linearly uncorrelated, removing sequential dependencies. Similarly, we can compute Q o ∈ R τ × τ , and recover the temporal correlations

by multiplying with Q o . Note that Q i and Q o are pre-computed and used throughout training and inference. Ablation studies on these two matrices are discussed in Appendix J.3. Moreover, as shown in Appendix I.10, OLinear remains robust even when the orthogonal matrices Q i and Q o are computed with limited training data.

Discussion (1) Mathematically, Q T i x projects x onto the eigenvectors of the temporal correlation matrix CorrMat t . The transformed domain exhibits energy compaction, and noise is suppressed in the primary components [Bishop and Nasrabadi, 2006]. (2) Compared to DFT and wavelet transforms, or to the TF paradigm without any transformation, OrthoTrans produces higher-rank attention matrices for Transformer-based forecasters (see Figures 10- 14), indicating greater representation diversity and enhanced model expressiveness. This partly explains why OrthoTrans enhances other forecasters when used as a plug-in module.

4.3 Representation learning

CSL and NormLin Inspired by the properties of attention matrices in self-attention [Vaswani et al., 2017], we impose two constraints on the weight matrix of the linear layer for robust multivariate representation learning: (1) all entries must be positive, and (2) each row must sum to 1. To enforce these constraints, we apply the Softplus function followed by row-wise L1 normalization to the learnable weight matrix W . The resulting layer is referred to as NormLin, and is defined as follows:

The variants of NormLin are discussed in Appendix J.1. Incorporating the other components in Figure 1(b), the CSL process is formulated as:

where Linear( · ) and NormLin( · ) operate on the sequence and variate dimensions, respectively. We define the NormLin module as NormLinModule( · ) ≜ Linear (NormLin (Linear ( · ))) . Ablation studies on the two linear layers in NormLinModule( · ) are presented in Appendix J.2.

ISL We adopt two linear layers separated by the GELU activation function as a powerful predictive representation learner [Li et al., 2023]. It has been well established that MLPs are highly effective for encoding sequential dynamics and decoding future series [Zeng et al., 2023]. Similar to CSL, residual connections and LayerNorm are also applied. The complete ISL process is defined as:

Discussion Compared to the self-attention mechanism, the NormLin module offers the following advantages: (1) As shown in Table 1, it reduces computational complexity by half and decreases memory footprint by a factor of 1 /h , where h is the number of attention heads; (2) The learned weight matrix in NormLin naturally exhibits high rank , in contrast to the low-rank nature of self-attention (see Appendix E.2). The low-rank issue in self-attention could arise from the sharp value concentration of the Softmax function. A higher-rank weight

Table 1: Comparison of FLOPs and memory usage between the NormLin module and multi-head self-attention (MHSA). Here, h denotes the number of attention heads, typically set to 8.

(or attention) matrix often better preserves the rank of the representation space and thus improves the model's expressiveness [Han et al., 2023]. (3) From the perspective of gradient flow , the NormLin layer provides a more direct backpropagation path for optimizing weight entries. Appendix B shows that the Jacobian matrices of Softmax( · ) in self-attention and Norm L1 (Softplus( · )) in the NormLin layer share a similar structure, but the latter offers greater flexibility. (4) The query-key mechanism of self-attention induces correlations in attention matrix entries, restricting the optimization space . In contrast, NormLin decouples these dependencies, allowing for more flexible parameter updates.

Model

OLinear

(Ours)

2024a

FilterNet

2024a

DLinear

2023

TimeMix.++

2025a

2024

CARD

2024b

Fredformer

2024

iTrans.

2024a

PatchTST

2023

TimesNet

2023b

Metric

MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE

ETT(Avg) 0.359

0.376

0.367 0.388 0.375 0.394 0.442 0.444

0.349

0.377 0.367 0.387 0.366 0.380 0.366 0.385 0.383 0.399 0.380 0.396 0.391 0.404

ECL

0.159 0.248

0.182 0.273 0.173 0.268 0.212 0.300 0.165

0.253 0.169 0.263 0.168 0.258 0.176 0.269 0.178 0.270 0.208 0.295 0.192 0.295

Exchange 0.355 0.399 0.387 0.416 0.388 0.419 0.354 0.414 0.357

0.409 0.354 0.402 0.362 0.402

0.333 0.391

0.360 0.403 0.367 0.404 0.416 0.443

Traffic

0.451

0.247

0.485 0.298 0.463 0.310 0.625 0.383

0.416

0.264 0.467 0.294 0.453 0.282 0.433 0.291 0.428 0.282 0.531 0.343 0.620 0.336

Weather

0.237

0.260

0.240 0.272 0.245 0.272 0.265 0.317

0.226

0.262 0.242 0.272 0.239 0.265 0.246 0.272 0.258 0.279 0.259 0.281 0.259 0.287

Solar

0.215

0.217

0.216 0.280 0.235 0.266 0.330 0.401

0.203

0.258 0.230 0.264 0.237 0.237 0.226 0.262 0.233 0.262 0.270 0.307 0.301 0.319

PEMS03

0.095 0.199

0.167 0.267 0.145 0.251 0.278 0.375 0.165

0.263 0.107 0.210 0.174 0.275 0.135 0.243 0.113 0.221 0.180 0.291 0.147 0.248

PEMS04

0.091 0.190

0.185 0.287 0.146 0.258 0.295 0.388 0.136

0.251 0.103 0.210 0.206 0.299 0.162 0.261 0.111 0.221 0.195 0.307 0.129 0.241

PEMS07

0.077 0.164

0.181 0.271 0.123 0.229 0.329 0.395 0.152

0.258 0.084 0.180 0.149 0.247 0.121 0.222 0.101 0.204 0.211 0.303 0.124 0.225

PEMS08

0.113 0.194

0.226 0.299 0.172 0.260 0.379 0.416 0.200

0.279 0.122 0.211 0.201 0.280 0.161 0.250 0.150 0.226 0.280 0.321 0.193 0.271

1

st

Count

5

9

0

0

0

0

0

0

4

0

0

0

0

0

1

1

0

0

0

0

0

0

Model

OLinear

(Ours)

2024a

FilterNet

2024a

DLinear

2023

TimeMix.++

2025a

2024

CARD

2024b

Fredformer

2024

iTrans.

2024a

PatchTST

2023

TimesNet

2023b

Metric

MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE

ILI

1.429 0.690

1.864 0.806 1.793 0.791 2.742 1.126 1.805

0.793 1.725 0.777 1.959 0.822 1.732 0.797 1.715 0.773 1.905 0.804

1.809

0.807

COVID-19

5.187 1.211

5.919 1.350 5.607 1.322 8.279 1.601 5.974

1.369 5.251 1.285 5.536 1.314 5.279 1.287 5.301 1.293 5.836 1.362

6.106

1.369

METR-LA 0.587

0.311

0.608 0.372 0.603 0.366 0.580 0.422

0.567

0.363 0.603 0.367 0.639 0.350 0.617 0.369 0.627 0.373 0.614 0.372

0.617

0.370

NASDAQ 0.121

0.201 0.120

0.204 0.127 0.211 0.150 0.251 0.125

0.210 0.128 0.211 0.125 0.207 0.127 0.210 0.133 0.217 0.128 0.209

0.161

0.247

Wiki

6.395

0.415

6.443 0.439 6.457 0.439 6.420 0.510 6.430

0.443 6.417 0.433 6.419 0.427

6.318

0.429 6.422 0.432 6.368 0.424

7.633

0.572

SP500

0.146 0.250

0.153 0.265 0.164 0.279 0.178 0.298 0.157

0.270 0.163 0.282 0.147 0.252 0.167 0.286 0.161 0.279 0.159 0.277

0.150

0.262

DowJones

7.686 0.619

8.499 0.633 8.283 0.633 7.893 0.626 8.895

0.643 8.257 0.633 7.699

0.619

8.041 0.625 8.177 0.630 7.991 0.626 10.960 0.737

CarSales

0.330

0.305

0.333 0.322 0.328 0.319 0.387 0.376 0.337

0.321 0.335 0.322 0.347 0.324 0.335 0.325

0.311

0.307 0.327 0.318

0.334

0.328

Power

1.248

0.835 1.234

0.840 1.309 0.870 1.278 0.870

1.234

0.841 1.295 0.868 1.288 0.847 1.302 0.870 1.324 0.874 1.311 0.873

1.317

0.871

Website

0.225 0.311 0.279 0.358 0.297 0.367 0.302 0.389 0.260

0.344 0.264 0.348 0.303 0.366 0.266 0.351

0.179 0.297

0.284 0.362

0.251

0.341

Unemp

0.729

0.461

1.581 0.708 1.286 0.627

0.565

0.509 1.506

0.678 1.502 0.689 1.163 0.596 2.048 0.789 1.408 0.666 1.237 0.624

2.328

0.852

1

st

Count

4

10

2

0

0

0

1

0

2

0

0

0

0

1

1

0

2

1

0

0

0

0

5 Experiments

Datasets and implementation details We extensively evaluate OLinear using 24 diverse realworld datasets: ETT (four subsets), Weather , ECL , Traffic , Exchange , Solar-Energy , PEMS (four subsets), ILI , COVID-19 , METR-LA , NASDAQ , Wiki , SP500 , DowJones , CarSales , Power , Website , Unemp . The weighted L1 loss function from CARD [Wang et al., 2024b] is adopted. The embedding size d is set as 16. Dataset description and more implementation details are presented in Appendices C and D, respectively.

5.1 Forecasting performance

Baselines We carefully choose 11 well-acknowledged state-of-the-art forecasting models as our baselines, including (1) Linear-based models: TimeMixer [Wang et al., 2024a], FilterNet [Yi et al., 2024a], FITS [Xu et al., 2024], DLinear [Zeng et al., 2023]; (2) Transformer-based models: TimeMixer++ [Wang et al., 2025a], Leddam [Yu et al., 2024], CARD [Wang et al., 2024b], Fredformer [Piao et al., 2024], iTransformer [Liu et al., 2024a], PatchTST [Nie et al., 2023]; (3) TCN-based model: TimesNet [Wu et al., 2023b].

Main results Comprehensive long-term and short-term forecasting results are presented in Tables 2 and 3, respectively, with the best results highlighted in bold and the second-best underlined. Lower MSE/MAE values indicate more accurate predictions. Across a wide range of benchmarks, OLinear















































































Table 4: Ablation study on transformation bases. S1 and S2 represent 'Input-12, Predict-{ 3 , 6 , 9 , 12 } ' and 'Input-36, Predict-{ 24 , 36 , 48 , 60 } ', respectively. Wavelet1 and Wavelet2 use the Haar and discrete Meyer wavelets. Identity denotes no transformation. Full results are in Table 20.

consistently outperforms state-of-the-art Transformer-based and linear-based forecasters. Notably, these gains are achieved with high computational efficiency. (Figure 3 and Table 41).

We attribute this superior performance to the adopted OrthoTrans and NormLin modules. The effectiveness of the simpler NormLin module challenges the necessity of the widely adopted multi-head self-attention mechanism, which has been a dominant design in prior works. To further validate robustness, we evaluate OLinear under varying lookback lengths (Table 29), where it consistently outperforms existing stateof-the-art methods.

Figure 4 shows the prediction visualizations of OLinear. Moreover, Table 14 demonstrates that









































OLinear(Ours)

1GB, 17ms

DLinear

0.03GB, 1.4ms

iTrans.

0.4GB, 10ms

PatchTST

1.3GB, 29ms

Fredfm.

TimeMixer

CARD

2.2GB, 36ms

DLinear

0.07GB, 1.6ms

OLinear(Ours)

0.3GB, 9ms

iTrans.

3GB, 40ms

Fredfm.

6GB, 100ms

TimeMixer

11GB, 170ms

PatchTST

8GB, 170ms

CARD

10GB, 180ms

OLinear exhibits greater robustness to random seeds compared to state-of-the-art Transformer-based forecasters such as TimeMixer++ and iTransformer.

5.2 Model analysis

Transformation bases We replace OrthoTrans with several commonly used bases, including the Fourier basis, two wavelet bases (Haar and discrete Meyer wavelets), and three polynomial bases. As shown in Table 4, our method consistently outperforms all alternatives. Specifically, it achieves a 5.5% reduction in MSE over the Fourier basis, and a 10.0% improvement over the no-transformation baseline (i.e., the TF paradigm) on PEMS03.

OrthoTrans as a plug-in We further integrate OrthoTrans into three classic forecasters: iTrans-

former, PatchTST, and RLinear [Li et al., 2023]. As shown in Table 5, OrthoTrans yields average MSE improvements of 5.1% and 10.1% for iTransformer and PatchTST, respectively, highlighting the benefit of incorporating dataset-specific statistical information into the model design. This improvement can be attributed to the increased attention matrix rank introduced by OrthoTrans (see Appendix E.3), which indicates enlarged representation space and enhanced model capacity.

Representation learning To validate the rationality of our CSL and ISL designs, we conduct ablation studies by replacing or removing their core components-NormLin and (standard) linear layers. As shown in Table 6, our design-applying NormLin along the variate dimension and standard linear layers along the temporal dimension-consistently achieves the best performance. Notably, applying NormLin along the variate dimension consistently outperforms its self-attention counterpart (last row), with reduced computational cost. Furthermore, removing the ISL module (third-last row) results in a 6.2% performance decline, highlighting the importance of updating temporal representations. Interestingly, on small-scale datasets with fewer variates (e.g., NASDAQ and ILI), the model with only temporal linear layers (third row) exhibits competitive performance, implying that NormLin is more beneficial when handling a larger number of variates.

OLinear-C As shown in Figure 5, the learned weight matrix resemble Softmax(CorrMat v ) , where CorrMat v is the Pearson correlation matrix across variates. Motivated by this, we replace the learnable weights in NormLin with the pre-computed Softmax(CorrMat v ) , resulting in a simplified variant: NormLin c ( x ) = Softmax(CorrMat v ) x . This indicates that the weight matrix in NormLin is optimized to align with the correlation matrix among channels CorrMat v . We refer to the model with this NormLin variant as OLinear-C . Table 7 shows that OLinear-C performs comparably to OLinear with reduced learnable parameters. Comprehensive evaluation of OLinear-C and the variants of NormLin c are discussed in Appendix H.

5.3 Generality and scalability of the NormLin module

In our CSL module, we employ the NormLin module-comprising the NormLin layer and its associated pre- and post-linear layers (Equation 4)-to capture multivariate correlations. Despite its simple architecture, the NormLin module demonstrates strong capability, generality, and scalability. Specifically, it consistently outperforms the classic multi-head self-attention mechanism and its variants, offering a compelling alternative for token dependency modeling in time series forecasting.

Comparison with attention mechanisms To assess its effectiveness, we compare NormLin with classic attention variants, such as Reformer [Kitaev et al., 2020], Flowformer [Wu et al., 2022], FLatten [Dao et al., 2022], and Mamba [Gu and Dao, 2023]. As shown in Table 8, NormLin

















consistently outperforms these methods, indicating that simply normalizing the weight matrices can be more effective than self-attention for time series forecasting (see Paragraph 4.3).

NormLin as a plug-in To demonstrate the generality of the NormLin module, we replace selfattention with NormLin in Transformer-based forecasters. As shown in Table 9, this substitution leads to notable MSE improvements-6.7% for iTransformer and 10.3% for PatchTST-validating its plug-and-play effectiveness. These results also validate NormLin's capability to model dependencies across multiple token types (e.g., variate, temporal patch, and frequency-domain patch), highlighting its potential as a universal token dependency learner for time series forecasting. Furthermore, the NormLin module consistently improves both training and inference efficiency across these models (see Table 42); for example, it boosts iTransformer's inference efficiency by an average of 53%.

Scalability of the NormLin module Decoder-only Transformers are currently the de facto architecture of large time series models [Liu et al., 2024b, Ansari et al., 2024, Das et al., 2024]. We take Timer [Liu et al., 2024b] as a representative model and replace its self-attention with our NormLin module. To align with Timer's decoder-only structure, we apply a causal mask by zeroing out the upper triangular part of NormLin's weight matrix. The modified model is pre-trained on the UTSD dataset [Liu et al., 2024b], which spans seven domains and contains up to 1 billion time points. As shown in Table 10, NormLin improves performance in both zero-shot and fine-

tuning scenarios. For example, the zero-shot MSE on ETTh2 and ETTm1 is reduced by 12.7% and 8.4%, respectively. These results demonstrate that the NormLin module adapts well to decoder-only architectures and scales effectively to large-scale pre-training scenarios.

6 Conclusion

In this work, we present OLinear, a simple yet effective linear-based forecaster that achieves state-ofthe-art performance, built on two core components: (1) OrthoTrans, an orthogonal transformation that decorrelates temporal dependencies to facilitate better encoding and forecasting, and (2) the NormLin module, a powerful and general-purpose token dependency learner. Notably, both modules consistently improve existing forecasters when used as plug-ins. We hope OLinear will serve as a strong baseline and inspire future studies in time series forecasting.

Acknowledgments

This work was supported by Beijing Natural Science Foundation under Grant No. L247029, and the National Natural Science Foundation of China (NSFC) under Grant No. 62371009 and Grant No. 623B2002.

References

Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with a unified deep model. Nature Machine Intelligence , 5(6):602-611, 2023a.

Changxi Ma, Guowen Dai, and Jibiao Zhou. Short-term traffic flow prediction for urban road sections based on time series analysis and lstm_bilstm method. IEEE Transactions on Intelligent Transportation Systems , 23(6):5615-5624, 2021.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI , volume 35, pages 11106-11115, 2021.

Zonglei Chen, Minbo Ma, Tianrui Li, Hongjun Wang, and Chongshou Li. Long sequence time-series forecasting with deep learning: A survey. Information Fusion , 97:101819, 2023.

Shiyu Wang, Jiawei Li, Xiaoming Shi, Zhou Ye, Baichuan Mo, Wenze Lin, Shengtong Ju, Zhixuan Chu, and Ming Jin. Timemixer++: A general time series pattern machine for universal predictive analysis. In ICLR , 2025a.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In ICLR , 2024a.

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. ICLR , 2023.

Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting. In ICLR , 2024a.

Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mlps are more effective learners in time series forecasting. In NeurIPS , 2023.

Wenzhen Yue, Yong Liu, Xianghua Ying, Bowei Xing, Ruohao Guo, and Ji Shi. Freeformer: Frequency enhanced transformer for multivariate time series forecasting. arXiv preprint arXiv:2501.13989 , 2025.

Kun Yi, Jingru Fei, Qi Zhang, Hui He, Shufeng Hao, Defu Lian, and Wei Fan. Filternet: Harnessing frequency filters for time series forecasting. Advances in Neural Information Processing Systems , 37:55115-55140, 2024a.

Luca Masserano, Abdul Fatir Ansari, Boran Han, Xiyuan Zhang, Christos Faloutsos, Michael W Mahoney, Andrew Gordon Wilson, Youngsuk Park, Syama Rangapuram, Danielle C Maddix, et al. Enhancing foundation models for time series forecasting via wavelet-based tokenization. arXiv preprint arXiv:2412.05244 , 2024.

Robert M Gray and Lee D Davisson. An introduction to statistical signal processing . Cambridge University Press, 2004.

Guoqi Yu, Jing Zou, Xiaowei Hu, Angelica I Aviles-Rivero, Jing Qin, and Shujun Wang. Revitalizing multivariate time series forecasting: Learnable decomposition with inter-series dependencies and intra-series variations modeling. In ICML , 2024.

Xue Wang, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Card: Channel aligned robust blend transformer for time series forecasting. In ICLR , 2024b.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In ICLR , 2023b.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI , volume 37, pages 11121-11128, 2023.

Hao Wang, Lichen Pan, Yuan Shen, Zhichao Chen, Degui Yang, Yifei Yang, Sen Zhang, Xinggao Liu, Haoxuan Li, and Dacheng Tao. Fredf: Learning to forecast in the frequency domain. In ICLR , 2025b.

Zhijian Xu, Ailing Zeng, and Qiang Xu. Fits: Modeling time series with 10 k parameters. In ICLR , 2024.

Donghao Luo and Xue Wang. Moderntcn: A modern pure convolution structure for general time series analysis. In ICLR , 2024.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR , pages 95-104, 2018.

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. NeurIPS , 31, 2018.

Qihe Huang, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. Crossgnn: Confronting noisy multivariate time series via cross interaction refinement. NeurIPS , 36:46885-46902, 2023a.

Kun Yi, Qi Zhang, Wei Fan, Hui He, Liang Hu, Pengyang Wang, Ning An, Longbing Cao, and Zhendong Niu. Fouriergnn: Rethinking multivariate time series forecasting from a pure graph perspective. Advances in Neural Information Processing Systems , 36, 2024b.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In ICLR , 2021.

Roger A Horn and Charles R Johnson. Matrix analysis . Cambridge university press, 2012.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS , 30, 2017.

Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305.10721 , 2023.

Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In ICCV , pages 5961-5971, 2023.

Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, and Yasushi Sakurai. Fredformer: Frequency debiased transformer for time series forecasting. In SIGKDD , 2024.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ICLR , 2020.

Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Linearizing transformers with conservation flows. In ICML , 2022.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memoryefficient exact attention with io-awareness. NeurIPS , 35:16344-16359, 2022.

Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Transformers for time series analysis at scale. In ICML , 2024b.

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815 , 2024.

Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning , 2024.

Hao Wang, Zhiyu Wang, Yunlong Niu, Zhaoran Liu, Haozhe Li, Yilin Liao, Yuxin Huang, and Xinggao Liu. An accurate and interpretable framework for trustworthy process monitoring. IEEE Transactions on Artificial Intelligence , 5(5):2241-2252, 2023.

Haotian Wang, Haoxuan Li, Hao Zou, Haoang Chi, Long Lan, Wanrong Huang, and Wenjing Yang. Effective and efficient time-varying counterfactual prediction with state-space models. In The Thirteenth International Conference on Learning Representations , 2025c.

Lei Wang, Shanshan Huang, Chunyuan Zheng, Jun Liao, Xiaofei Zhu, Haoxuan Li, and Li Liu. Mitigating data imbalance in time series classification based on counterfactual minority samples augmentation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages 2962-2973, 2025d.

Hao Wang, Haoxuan Li, Xu Chen, Mingming Gong, Zhichao Chen, et al. Optimal transport for time series imputation. In The Thirteenth International Conference on Learning Representations , 2025e.

Hao Wang, Licheng Pan, Zhichao Chen, Xu Chen, Qingyang Dai, Lei Wang, Haoxuan Li, and Zhouchen Lin. Time-o1: Time-series forecasting needs transformed label alignment. Advances in Neural Information Processing Systems , 2025f.

Xinyu Li, Yuchen Luo, Hao Wang, Haoxuan Li, Liuhua Peng, Feng Liu, Yandong Guo, Kun Zhang, and Mingming Gong. Towards accurate time series forecasting via implicit decoding. Advances in Neural Information Processing Systems , 2025.

Xingjian Wu, Xiangfei Qiu, Zhengyu Li, Yihang Wang, Jilin Hu, Chenjuan Guo, Hui Xiong, and Bin Yang. CATCH: Channel-aware multivariate time series anomaly detection via frequency patching. In ICLR , 2025a.

Xingjian Wu, Xiangfei Qiu, Hongfan Gao, Jilin Hu, Bin Yang, and Chenjuan Guo. K 2 VAE: A koopman-kalman enhanced variational autoencoder for probabilistic time series forecasting. In ICML , 2025b.

Xingjian Wu, Xiangfei Qiu, Hanyin Cheng, Zhengyu Li, Jilin Hu, Chenjuan Guo, and Bin Yang. Enhancing time series forecasting through selective representation spaces: A patch perspective. In NeurIPS , 2025c.

Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Zhenli Sheng, and Bin Yang. TFB: Towards comprehensive and fair benchmarking of time series forecasting methods. In Proc. VLDB Endow. , pages 2363-2377, 2024.

Xiangfei Qiu, Xingjian Wu, Hanyin Cheng, Xvyuan Liu, Chenjuan Guo, Jilin Hu, and Bin Yang. DBLoss: Decomposition-based loss function for time series forecasting. In NeurIPS , 2025a.

Xiangfei Qiu, Xingjian Wu, Yan Lin, Chenjuan Guo, Jilin Hu, and Bin Yang. DUET: Dual clustering enhanced multivariate time series forecasting. In SIGKDD , pages 1185-1196, 2025b.

Xiangfei Qiu, Zhe Li, Wanghui Qiu, Shiyan Hu, Lekui Zhou, Xingjian Wu, Zhengyu Li, Chenjuan Guo, Aoying Zhou, Zhenli Sheng, Jilin Hu, Christian S. Jensen, and Bin Yang. Tab: Unified benchmarking of time series anomaly detection methods. In Proc. VLDB Endow. , pages 2775-2789, 2025c.

Jiaming Ma, Binwu Wang, Pengkun Wang, Zhengyang Zhou, Xu Wang, and Yang Wang. Bist: A lightweight and efficient bi-directional model for spatiotemporal prediction. Proceedings of the VLDB Endowment , 18(6):1663-1676, 2025a.

Jiaming Ma, Zhiqing Cui, Binwu Wang, Pengkun Wang, Zhengyang Zhou, Zhe Zhao, and Yang Wang. Causal learning meet covariates: Empowering lightweight and effective nationwide air quality forecasting. In International Joint Conference on Artificial Intelligence , 2025b.

Jiaming Ma, Binwu Wang, Guanjun Wang, Kuo Yang, Zhengyang Zhou, Pengkun Wang, Xu Wang, and Yang Wang. Less but more: Linear adaptive graph learning empowering spatiotemporal forecasting. In Advances in Neural Information Processing Systems , 2025c.

Jiaming Ma, Binwu Wang, Pengkun Wang, Zhengyang Zhou, Yudong Zhang, Xu Wang, and Yang Wang. Mobimixer: A multi-scale spatiotemporal mixing model for mobile traffic prediction. IEEE Transactions on Mobile Computing , 2025d.

Jiaming Ma, Binwu Wang, Qihe Huang, Guanjun Wang, Pengkun Wang, Zhengyang Zhou, and Yang Wang. Mofo: Empowering long-term time series forecasting with periodic pattern modeling. In Advances in Neural Information Processing Systems , 2025e.

Jiaming Ma, Binwu Wang, Pengkun Wang, Zhengyang Zhou, Xu Wang, and Yang Wang. Robust spatio-temporal centralized interaction for ood learning. In Forty-second International Conference on Machine Learning , 2025f.

Qihe Huang, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. Crossgnn: Confronting noisy multivariate time series via cross interaction refinement. Advances in Neural Information Processing Systems , 36:46885-46902, 2023b.

Qihe Huang, Lei Shen, Ruixin Zhang, Jiahuan Cheng, Shouhong Ding, Zhengyang Zhou, and Yang Wang. Hdmixer: Hierarchical dependency with extendable patch for multivariate time series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 12608-12616, 2024.

Qihe Huang, Zhengyang Zhou, , Yangze Li, Kuo Yang, Binwu Wang, and Yang Wang. Many minds, one goal: Time series forecasting via sub-task specialization and inter-agent cooperation. In Advances in Neural Information Processing Systems , 2025a.

Qihe Huang, Zhengyang Zhou, Kuo Yang, Zhongchao Yi, Xu Wang, and Yang Wang. Timebase: The power of minimalism in efficient long-term time series forecasting. In Forty-second International Conference on Machine Learning , 2025b.

Wenxuan Wang, Kai Wu, Yujian Betterest Li, Dan Wang, and Xiaoyu Zhang. Synthetic series-symbol data generation for time series foundation models. Advances in Neural Information Processing Systems , 2025g. URL https://arxiv.org/abs/2510.08445 .

Sai Surya Duvvuri and Inderjit S Dhillon. Laser: Attention with exponential transformation. arXiv e-prints , pages arXiv-2411, 2024.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. NeurIPS , 34:22419-22430, 2021.

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. NeurIPS , 35: 5816-5828, 2022.

Yuzhou Chen, Ignacio Segovia-Dominguez, Baris Coskunuzer, and Yulia Gel. Tamp-s2gcnets: coupling time-aware multipersistence knowledge representation with spatio-supra graph convolutional networks for time-series forecasting. In ICLR , 2022.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Ian T Jolliffe. Principal component analysis for special types of data . Springer, 2002.

Qihe Huang, Zhengyang Zhou, Kuo Yang, Zhongchao Yi, Xu Wang, and Yang Wang. Timebase: The power of minimalism in efficient long-term time series forecasting. In ICML , 2025c.

Wenzhen Yue, Xianghua Ying, Ruohao Guo, DongDong Chen, Ji Shi, Bowei Xing, Yuqing Zhu, and Taiyan Chen. Sub-adjacent transformer: Improving time series anomaly detection with reconstruction error from sub-adjacent neighborhoods. arXiv preprint arXiv:2404.18948 , 2024.

Hui Chen, Viet Luong, Lopamudra Mukherjee, and Vikas Singh. SimpleTM: A simple baseline for multivariate time series forecasting. In ICLR , 2025.

Shengsheng Lin, Haojun Chen, Haijie Wu, Chunyun Qiu, and Weiwei Lin. Temporal query network for efficient multivariate time series forecasting. In ICML , 2025.

Xiaowen Ma, Zhenliang Ni, Shuai Xiao, and Xinghao Chen. Timepro: Efficient multivariate longterm time series forecasting with variable- and time-aware hyper-state. In ICML , 2025g.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NIPS , 33:1877-1901, 2020.

Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprint arXiv:2304.08424 , 2023.

Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In ICLR , 2023.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML , pages 27268-27286. PMLR, 2022.

Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems , 36:19622-19635, 2023.

Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts. In ICLR , 2025.

Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In ICML , 2024.

Yong Liu, Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Sundial: A family of highly capable time series foundation models. arXiv preprint arXiv:2502.00816 , 2025.

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In ICML , 2024.

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 14138-14148, 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims in the abstract and introduction clearly state our contributions and accurately match the theoretical and experimental results in Sections 3, 4 and 5, respectively.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Appendix L.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theorem 1 is formally stated in Section 3, with its complete proof provided in Appendix A. Additionally, the derivations of the Jacobian matrices of the NormLin layer are detailed in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The implementation details are presented in Appendix D. The source code and datasets are available at https://github.com/jackyue1994/OLinear .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Data and code are available at this anonymous repository: https://github. com/jackyue1994/OLinear .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental settings and implementation details are provided in Appendices C and D. The source code is publicly available.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are reported in Appendices F and H.2. Specifically, Table 13 presents standard deviations over seven random seeds, and Table 14 shows 99% confidence intervals of averaged performance across four prediction lengths.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources are presented in Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work strictly conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a technical work on time series forecasting without direct societal impact, hence we select [NA] . Broader technical implications are discussed in Appendix M.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The datasets and models used in this work are standard in time series forecasting and pose no known safety or misuse risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use publicly available datasets and open-source code, all of which are properly cited in the main text. All terms of use have been respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The released code repository includes all necessary source code to reproduce the experimental results and is well documented with usage instructions. All assets are anonymized for review.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs are not used as an important, original, or non-standard component of our methods.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Proof of Theorem 1

Proof. For clarity, we denote z ≜ [ x y ] ∈ R t +1 , µ z ≜ [ µ x µ y ] ∈ R t +1 , and Σ ≜ [ Σ x Σ x y Σ T x y σ 2 y ] . According to the definition, the probability density function of z , i.e, the joint density of x and y , is

where | Σ | denotes the determinant of Σ . In the following, we ignore the constant coefficient and focus on the exponential term.

Using the standard block matrix inverse formula [Horn and Johnson, 2012], we have

where the scalar d = σ 2 y -Σ T x y Σ -1 x Σ x y is the Schur complement [Horn and Johnson, 2012]. Letting b ∈ R t × 1 denote d -1 Σ -1 x Σ x y , the matrix A becomes Σ -1 x + d · bb T .

For notational simplicity, we define α ≜ x -µ x ∈ R t × 1 and β ≜ y -µ y ∈ R . Then, the exponent in Equation 6 (ignoring the constant factor -1 / 2 ) becomes

Since p ( x ) ∝ exp ( -1 2 α T Σ -1 x α ) , it follows that

Therefore, the conditional density of y give x is also a Gaussian distribution, whose mean is

and the variance is d = σ 2 y -Σ T x y Σ -1 x Σ x y . This completes the proof.

B Jacobian matrix comparison of self-attention and NormLin

In this section, we analyze the gradients of the non-linear transformations in the self-attention mechanism and the NormLin module, focusing on the attention/weight rows, denoted as a ∈ R N .

B.1 Jacobian matrix of Softmax in self-attention

Let c ≜ Softmax( a ) ∈ R N . We first compute the partial derivatives element-wise, and then rewrite the results in matrix form. Based on the definition of Softmax , the i -th element of c is c i = e a i ∑ N k =1 e a k . The partial derivative of c i with respect to a i is:

For j = i , the partial derivative of c i with respect to a j is:

Combining Equations (11) and (12), we obtain:

where δ ji = 1 if j = i , otherwise 0. Therefore, the Jacobian matrix ∂ c ∂ a can be written as

where Diag( c ) is the diagonal matrix with c as its diagonal. We can observe that the Jacobian matrix is a function of c . Since the Softmax function could cause sharp value concentration, the Jacobian matrix may exhibit sparsity, with most entries being close to zero [Surya Duvvuri and Dhillon, 2024].

B.2 Jacobian matrix of the NormLin layer

Let c ≜ Norm L1 (Softplus ( a )) ∈ R N , where Norm L1 ( · ) denotes the L1 normalization. For notational simplicity, let b ≜ Softplus( a ) . We first analyze ∂ c ∂ b . Since all entries of b are positive, we have:

c

=

b

N

k

=1

i

.

(15)

̸

Therefore, it follows that

̸

For j = i , the partial derivative ∂ c i ∂ b j is:

Combining Equations (16) and (17), we obtain the Jacobian matrix:

i

∑

b

k



















where 1 ∈ R N is the all-ones vector.

For the derivative of b i with respect to a j with 1 ≤ i, j ≤ N , we have

Therefore, ∂ b ∂ a can be written as

where Sigmoid( · ) operates element-wise, and ˜ b ≜ Sigmoid( a ) is defined for clarity. Using the chain rule, the Jacobian matrix of c with respect to a can be derived as follows:

where we use the fact that 1 T Diag ( ˜ b ) = ˜ b T , and ¯ b = b ∥ b ∥ 1 is the normalized b .

Equations (21) and (14) share similar structures, with the former introducing an additional learnable scaling factor 1 ∥ b ∥ 1 , which offers greater flexibility. The entries of ˜ b = Sigmoid( a ) in Equation (21) are generally larger than those of c = Softmax( a ) in Equation (14), particularly when a contains small values near zero. This property may contribute to stronger gradients in NormLin. Figure 6 illustrates the Jacobian matrices of the self-attention mechanism and the NormLin layer under the same input a , highlighting that NormLin tends to produce stronger gradient values.

C Dataset description

In this work, the following real-world datasets are used for performance evaluation. The dataset details are presented in Table 11.

ETT datasets [Zhou et al., 2021] record seven channels related to electricity transformers from July 2016 to July 2018. It contains four datasets: ETTh1 and ETTh2, with hourly recordings, and ETTm1 and ETTm2, with 15-minute recordings.

Weather [Wu et al., 2021] contains 21 meteorological variables (e.g., air temperature, humidity) recorded every 10 minutes in 2020.

ECL [Wu et al., 2021] records hourly electricity consumption of 321 consumers from July 2016 to July 2019.

Traffic [Wu et al., 2021] includes hourly road occupancy rates from 862 sensors in the Bay Area from January 2015 to December 2016.

Exchange [Wu et al., 2021] collects daily exchange rates for eight countries from January 1990 to October 2010.

Solar-Energy [Lai et al., 2018] records the solar power output every 10 minutes from 137 photovoltaic plants in 2006.

PEMS [Liu et al., 2022] provides public traffic sensor data from California, collected every 5 minutes. We use its four subsets (PEMS03, PEMS04, PEMS07, PEMS08) in this study.

ILI 2 contains weekly records of influenza-like illness patient counts provided by the U.S. CDC from 2002 to 2021.

COVID-19 [Chen et al., 2022] includes daily records of COVID-19 hospitalizations in California in 2020, provided by Johns Hopkins University.

METR-LA 3 collects traffic network data in Los Angeles every 5 minutes from March to June 2012. A total of 207 channels are included.

NASDAQ 4 includes daily NASDAQ index and key economic indicators (e.g., interest rate and gold price) from 2010 to 2024.

Wiki 5 records daily page view counts for Wikipedia articles over two years (2018-2019). The first 99 channels are used in this study.

SP500 records daily SP500 index data (e.g., opening price, closing price, and trading volume) from January 1993 to February 2025.

DowJones collects daily stock prices of 27 Dow Jones Industrial Average (DJIA) component companies from January 1999 to March 2025.

CarSales collects daily sales of 10 vehicle brands (e.g., Toyota, Honda) in the U.S. from January 2005 to June 2023. The data are compiled from the Vehicles Sales dataset 6 on Kaggle.

Power contains daily wind and solar energy production (in MW) records for the French grid from April 2020 to June 2023. The data are compiled from the Wind & Solar Daily Power Production dataset 7 on Kaggle.

Website 8 contains six years of daily visit data (e.g., first-time and returning visits) to an academic website, spanning from September 2014 to August 2020.

Unemp contains monthly unemployment figures for 50 U.S. states and three other territories from January 1976 to March 2020, sourced from the official website of the U.S. Bureau of Labor Statistics 9 .

D Implementation details

OLinear is optimized using the ADAM optimizer [Kingma and Ba, 2015], with the initial learning rate selected from { 10 -4 , 2 × 10 -4 , 5 × 10 -4 } . The model dimension D is chosen from { 128 , 256 , 512 } , while the embedding size d is set to 16. The batch size is selected from { 4 , 8 , 16 , 32 } depending on the dataset scale. The block number L is chosen from { 1 , 2 , 3 } . Training is performed for up to 50 epochs with early stopping, which halts training if the validation performance does not improve for 10 consecutive epochs. We adopt the weighted L1 loss function following CARD [Wang et al., 2024b]. The experiments are implemented in PyTorch [Paszke et al., 2019] and conducted on an NVIDIA GPU with 24 GB of memory. Hyperparameter sensitivity is discussed in Appendix I.8. For baseline models, we use the reported values from the original papers when available; otherwise, we produce the results using the official code. For the model FilterNet [Yi et al., 2024a], TexFilter is adopted in this work. The code and datasets are available at the following anonymous repository: https://github.com/jackyue1994/OLinear .

E Showcases

E.1 Time domain and transformed domain

Figure 7 illustrates the temporal domain and its corresponding transformed domain. In the transformed domain, the correlations along the sequence are effectively suppressed. Adjacent time series exhibit strong consistency in this new domain, which is desirable for forecasting tasks. The orthogonal transformation corresponds to projecting the time series onto the eigenvectors of the temporal Pearson correlation matrix CorrMat t . The eigenvalues of this symmetric positive semi-definite matrix CorrMat t typically decay rapidly [Jolliffe, 2002], with only a few being dominant. Consequently, the transformed series exhibits sparsity, with most of the energy concentrated in just a few dimensions. Moreover, since noise tends to be evenly distributed across the transformed dimensions, the signal-tonoise ratio (SNR) in the leading components is improved [Bishop and Nasrabadi, 2006].

E.2 Low-rank attention matrix and high-rank NormLin weight matrix

We replace the NormLin module in OLinear with a standard self-attention mechanism and observe the typical low-rank property in the resulting attention matrices, as shown in Figure 9. This phenomenon can be attributed to the sparsity induced by the transformed domain and the sharp focus introduced by the Softmax operation. The low-rank attention matrix could limit the expressive capacity of the model. In contrast, the weight matrices in NormLin exhibit higher rank, better preserving representation diversity.

Furthermore, the learned NormLin weights closely resemble the across-variate Pearson correlation matrix CorrMat v , suggesting that the NormLin layer effectively captures multivariate correlations. By directly replacing the learnable weight matrix with Softmax(CorrMat v ) , we obtain OLinear-C , which also achieves competitive performance, as demonstrated in Appendix H.

Reason analysis A potential reason behind this phenomenon is that OrthoTrans can mitigate the intrinsic low-rank property of time series data. The empirical results are presented below. Following TimeBase [Huang et al., 2025c], we compute the median singular values of the correlation matrix among non-overlapping time series patches (patch length = 24, number of patches = 30). As shown in Table 12, OrthoTrans yields larger singular values than other transformation bases (with 'Identity' denoting no transformation). In other words, OrthoTrans alleviates low-rank tendencies of time series data and enhances data diversity.

E.3 OrthoTrans enhances attention matrix rank

Figures 10-12 visualize the attention matrices of OLinear (with the NormLin module replaced by self-attention) under different transformation bases. Similar results for iTransformer and PatchTST are presented in Figures 13 and 14, respectively. As shown, OrthoTrans typically yields higher-rank attention matrices compared to DFT, wavelet transforms, or no transformation. As stated earlier,









































t





































































































































































higher-rank attention matrices better preserve the representation space and can potentially enhance the model's expressive capacity [Han et al., 2023]. This may explain why integrating OrthoTrans as a plug-in consistently improves the performance of Transformer-based forecasters (as shown in Table 5).

ECL

Traffic

Weather

Solar-Energy

PEMS03

ETTh1

ILI

Power

Website

0.2

0.0

0.2

0.4

0.6

0.8

1.0

F Robustness under various random seeds

Table 13 presents the standard deviations for different datasets and prediction lengths using seven random seeds. OLinear demonstrates strong robustness across independent runs. Furthermore, as shown in Table 14, our model exhibits better robustness than state-of-the-art Transformer-based models, TimeMixer++ and iTransformer, as measured by the 99% confidence intervals.

Note that averaging over M prediction lengths reduces the standard deviation by a factor of 1 √ M , making the results more robust to the choice of random seeds. Therefore, we prefer to report the average results in this work to mitigate the influence of randomness.

We perform a significance test with 7 random seeds using Student's t-test to compare OLinear and iTransformer across multiple datasets and prediction horizons. As shown in Table 15, OLinear consistently outperforms iTransformer, with the improvements being statistically significant at the 0.05 level ( p < 0 . 05 ).

G Full results

Table 16 lists the simplified tables from the main text and their full versions in the appendix.

























































v







v

H OLinear-C

H.1 Forecasting performance

Table 25 compares OLinear and OLinear-C on both short- and long-term forecasting tasks. OLinear-C achieves performance comparable to OLinear while using fewer learnable parameters and offering improved efficiency (see Table 41). However, due to its fixed weight matrix, NormLin c lacks the flexibility required to serve as a plug-and-play module for other forecasters. Therefore, we focus on OLinear in this work.

H.2 Robustness

Table 26 reports the standard deviation of OLinear-C across seven random seeds, demonstrating its robustness to independent runs. For a broader robustness comparison, Table 14 presents 99% confidence intervals for OLinear, OLinear-C, TimeMixer++, and iTransformer.



























































H.3 Ablation studies

We conduct ablation studies on different transformations of the Pearson correlation matrix CorrMat v , including Softplus , Sigmoid , ReLU , and the identity function (i.e., no transformation). All variants are followed by row-wise L1 normalization. As shown in Table 27, Softmax(CorrMat v ) consistently outperforms the other options and is adopted in the main experiments.

I More experiments

I.1 Quantitation of temporal decorrelation of OrthoTrans

We evaluate the decorrelation effectiveness of OrthoTrans, DFT, and Haar wavelet transforms using a window size of 96. The metric is the off-diagonal Frobenius norm of the temporal correlation matrix, normalized by the number of off-diagonal elements. As shown in Table 28, OrthoTrans reduces the correlation in the original series by 96%, and achieves relative improvements of 84% and 93% over DFT and Haar wavelet transforms, respectively.





























































I.2 Performance under varying lookback horizons

We evaluate the forecasting performance of OLinear and its variant, OLinear-C, under varying lookback horizons. Instead of using a fixed lookback length, we search for the optimal horizon within the range of 96 to 720. As shown in Table 29, both models consistently achieve state-of-the-art results across different forecasting lengths and datasets, demonstrating their robustness and strong performance.

I.3 More baselines

We further compare OLinear with several lightweight baselines, including SimpleTM [Chen et al., 2025], TQNet [Lin et al., 2025], TimePro [Ma et al., 2025g], and TimeBase [Huang et al., 2025c]. The lookback length is uniformly set to 96 for all methods. As shown in Table 30, OLinear achieves the lowest MAE across all datasets, outperforming SimpleTM, TQNet, TimePro, and TimeBase by 5%, 5%, 7%, and 16% on average, respectively. These results demonstrate that OLinear serves as a strong and efficient new baseline for lightweight time series forecasting.



























































I.4 Few-shot and zero-shot forecasting

To evaluate OLinear's adaptability to sparse data and its transferability to unseen datasets, we conduct few-shot and zero-shot learning experiments, with the results shown in Tables 31 and 32, respectively. As a linear-based model, OLinear exhibits surprisingly strong generalization capabilities, a characteristic more commonly associated with Transformer-based models [Brown et al., 2020]. Notably, OLinear achieves this with high computational efficiency (see Table 41). Furthermore, OLinear delivers performance on par with or superior to TimeMixer++ [Wang et al., 2025a], despite the latter's architectural complexity, which includes multi-scale design, series imaging, and dual-axis attention mechanisms.

This performance advantage may be attributed to the high-rank property of the NormLin weight matrix (see Appendix E.2) and its enhanced gradient flow (see Appendix B). In contrast, attention matrices derived from the classic query-key mechanism typically exhibit low-rank characteristics (see Figures 10- 14), leading to a compressed representation space [Han et al., 2023].





























































I.5 Increasing lookback lengths

Performance under increasing lookback lengths reflects a model's ability to effectively utilize historical information [Liu et al., 2024a]. Linear-based forecasters have strong potential to benefit from longer lookback windows [Zeng et al., 2023]. As shown in Figure 15, OLinear demonstrates consistent improvements as the lookback horizon increases from 48 to 720, and consistently outperforms state-of-the-art forecasters.

I.6 Training with less data

Training with less data reflects a model's adaptability and learning efficacy. As shown in Figure 16, OLinear adapts well to decreasing training set sizes, from 100% down to 5%. Our model consistently outperforms others across all training ratios, highlighting its robustness to data-sparse conditions.

I.7 OLinear versus large time series models

We further compare OLinear with the large pre-trained time series models in the full-shot, fewshot and zero-shot scenarios. As shown in Table 33, OLinear surprisingly achieves comparable



























































performance to Timer [Liu et al., 2024b], despite not being pre-trained on large-scale datasets. When the full training set is used, OLinear outperforms the fine-tuned Timer on the ECL, Traffic, Weather, and PEMS datasets. Even when only 5% of training data are available, OLinear still performs better on the ETTm1 and ETTm2 datasets. In zero-shot forecasting, our model-trained on ETTh1 and ETTm1 respectively-outperforms state-of-the-art large time series models when evaluated on ETTh2 and ETTm2. These results not only demonstrate OLinear's strong generalization capability, but also imply that large time series models are still in the early stages of development.

I.8 Hyperparameter sensitivity

We evaluate the hyperparameter sensitivity of OLinear with respect to four key factors: the learning rate ( lr ), the number of blocks ( L ), the model dimension ( D ), and the embedding size ( d ). As shown in Figure 17, OLinear generally maintains stable performance across different choices of lr , L , and D , although each dataset exhibits its own preferences. Regarding the embedding size d , Table 34 indicates that d = 16 represents a sweet spot , which we adopt as the default setting in our experiments.

I.9 More metrics

We further conduct a comprehensive evaluation of forecasting performance using three scale-free metrics: the Coefficient of Determination ( R 2 ), the Pearson Correlation Coefficient ( r ), and the Mean Absolute Scaled Error (MASE), which are defined as follows:





































































Here Y n : and ˆ Y n : denote the n -th variate of the ground truth Y and the prediction ˆ Y , respectively. Y n,t refers to the t -th timestep of the n -th variate, and ( · ) represents the mean.

As shown in Table 35, OLinear consistently achieves state-of-the-art performance on these metrics. For example, it an average Pearson correlation coefficient of r = 0 . 903 on the ECL dataset, indicating strong alignment between predictions and ground truth. On the Traffic dataset, OLinear achieves a MASE of 0.764, outperforming the second-best model, CARD [Wang et al., 2024b], by a notable margin of 11.4%.

I.10 Less training data to compute Q matrices

To evaluate the amount of training data required for computing the orthogonal transformation matrices Q i and Q o , we conduct experiments using only partial data of the training set. As shown in Table 36, OLinear exhibits strong robustness, with minimal performance variation across different data proportions. This property not only enables more efficient preprocessing before training, but also ensures robust performance in few-shot learning scenarios (see Appendix I.4).

J More ablation studies

J.1 Ablations on NormLin design

In the NormLin layer, the Softplus function followed by L1 row-wise normalization is applied to the learnable weight matrix W to produce attention-like matrices, i.e., RowNorm L1 (Softplus( W )) . To assess the effects of different transformations and normalization strategies, we consider transformation

functions from the set { Softplus , Identity , Softmax , Sigmoid , ReLU } and normalization methods from { L1 , L2 } (all applied row-wise). Here, Identity denotes no transformation.

As shown in Table 37, the combination ' Softplus + L1 ' yields the best performance among the 10 candidates. On average, L1 normalization leads to a 10% performance improvement over L2, while Softplus slightly outperforms Softmax and other transformation functions.

 























 























































































































J.2 Pre- and post-linear layers in NormLin module

In CSL, the pre- and post-linear layers, named based on their positions in the execution sequence, are applied around the NormLin layer (see Equation 4): Post _ Lin (NormLin (Pre _ Lin( · ))) .

We conduct ablation studies to evaluate the contributions of these two layers. As shown in Table 38, incorporating both pre- and post-linear layers yields an average performance gain of 6% compared to the variant without them, highlighting their effectiveness in refining inputs for multivariate correlation modeling and downstream series representation learning.

J.3 Ablations on Q matrices

The effectiveness of OrthoTrans has been demonstrated in Tables 4 and 5. To further analyze the individual contributions of Q i and Q o , we conduct ablation studies on these transformation matrices.

As shown in Table 39, using both Q i and Q o achieves the best performance-reducing the average MSE by 5.3% compared to the baseline without transformations, by 4.4% compared to using Q o only, and by 0.8% compared to using Q i only. Notably, the variant using only Q i outperforms the one using only Q o , highlighting the critical role of temporal decorrelation in effective encoding and representation learning.

J.4 Swapping Q matrices

Table 40 reports the forecasting performance when both training and testing are conducted using Q matrices ( Q i and Q o ) derived from different datasets. As shown, using the Q matrices from the same dataset yields the best performance, highlighting the importance of incorporating dataset-specific temporal correlation information in time series forecasting.

K Model efficiency

As a linear model, OLinear achieves remarkable training and inference efficiency while delivering state-of-the-art forecasting performance. Figure 3 shows that OLinear achieves better efficiency than Transformer-based forecasters, benefiting from its relatively simple architecture. Detailed resource consumption for both training and inference is summarized in Table 41. Moreover, Table 42 presents the efficiency differences before and after applying the NormLin module to Transformer-based forecasters.

L Limitations

While this work includes extensive experiments, it primarily focuses on time series forecasting, and our claims are confined to this setting. Future work includes larger-scale pre-training and a wider range of time series analysis tasks (e.g., imputation and anomaly detection), in order to further validate the robustness of OLinear and its core components, OrthoTrans and NormLin. In addition, although NormLin is linear-based, it still incurs O ( N 2 ) computational and memory complexity with respect to the number of variates N . Developing simpler yet effective alternatives for modeling multivariate correlations-and more generally, for learning general-purpose token dependencies-remains a promising direction.

M Broader impacts

Multivariate time series forecasting is a fundamental field with a wide range of real-world applications across domains such as energy, finance, healthcare, and transportation. While this work does not target a specific application domain, OLinear could have general real-world applicability (e.g., weather forecasting and traffic planning). We contribute two key insights: (1) the OrthoTrans scheme, which reformulates temporally correlated forecasting as inter-independent feature prediction; and (2) the linear-based NormLin module, which effectively models multivariate correlations, enhances representation learning among temporal and frequency tokens, and remains compatible with decoder architectures and large-scale pre-training. As a step toward simpler and more efficient token dependency learners, the NormLin module shows potential for broader use in deep learning and may inspire future research in this direction.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces *OLinear*, a multivariate time-series forecaster that operates in an *orthogonally transformed* domain rather than the raw time domain. For each input window, the authors compute the temporal Pearson-correlation matrix, perform an eigen-decomposition, and project the signals onto the resulting orthogonal basis. They then replace self-attention with *NormLin*. Extensive experiments indicate improvements over strong baselines, and the accompanying code is available.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The authors evaluate OLinear on 24 publicly available datasets spanning electricity, traffic, weather and financial domains, and consider multiple forecast horizons for each dataset. Reporting 99 % confidence intervals indicates an effort to convey result variability rather than single-run numbers.\n\n2. The paper is well structured: problem statement, method, ablations and main tables flow logically. Figures are labelled and referenced properly, and notation is introduced before use, which makes the technical content easy to follow.\n\n3. Code and data-processing scripts are released, allowing reviewers and future researchers to reproduce the results or to integrate the proposed method into their own pipelines with minimal effort.\n\n4. The plug-and-play lowers the entry barrier for practitioners who already have Transformer-based models and wish to test the proposed ideas without redesigning their entire architecture.\n\n5. While Fourier and wavelet bases dominate current transform-domain forecasters, constructing an orthogonal basis from the data’s own temporal correlation matrix has been less explored. This adaptive approach could inspire further work. \n\n## Weaknesses\n\n1. Results seem to degrade when the look-back window exceeds 96, casting doubt on long-horizon robustness. For example, with 15-minute data, 96 steps cover only 24 h, yet the model may need to predict up to 720 steps (≈ 1 week) ahead in long-term forecasting, raising concerns about interpretability.  \n\n2. Many of the hyper-parameters, such as model dimension, embedding size, number of blocks and learning rate, are dataset-dependent.\n\n3. The assumption of decorrelation after OrthoTrans is not demonstrated. Two series can be decorrelated in time yet correlated in frequency; an orthogonal transform does not necessarily remove such cross-frequency dependencies. Moreover, decorrelating in the time domain removes second-order dependencies but says nothing about higher-order or non-linear dependencies.\n\n4. *The novelty remains unclear.* The manuscript does not cite the previous existing work [1]. That work already establishes that forecasting Transformers suffer from excessive sharpness, and that this can be mitigated by (i) using a different optimiser and (ii) reducing the architecture to a *single* channel-wise attention head—whose complexity is exactly $O(N^2)$, the same as NormLin with $h=1$ (see the Table 1). These two changes were shown to raise the attention-matrix rank and therefore boost the model’s expressivity. The study re-uses those insights in a purely linear setting, which by construction loses the non-linear capacity of a Transformer, yet none of this prior art is acknowledged. Consequently, the claims regarding higher rank, multivariate correlations, and expressivity are not novel in the context of multivariate time-series forecasting.  \n\n5. NormLin trades away non-linearity while retaining the same $O(N^2)$ cost.   \n\n**References**\n\n[1] SAMformer : Unlocking the potential of Transformers in Time Series Forecasting\n\nQUESTIONS:\n1. *Could you run a statistical significance test across ≥ 5 random seeds for every dataset–horizon pair and report the resulting p-values?*\n\n2. The existing work already demonstrates that a specific optimiser combined with a single channel-wise attention head (complexity $O(N^2)$) raises attention-matrix rank and boosts accuracy while using a channel attention mechanism to capture multivariate correlations. *Please clarify the novelty and compare the baseline with this existing work. Especially, how the model compares to a shallow transformer trained with a strong regularization like SAM and a channel-wise attention mechanism ?*  \n\n3. The claim that OrthoTrans removes step-wise correlations needs quantitative confirmation in both domains.  *Could you provide a metric that shows (i) a drop in the off-diagonal magnitude of the time-domain covariance matrix and (ii) that no new dependencies appear in the frequency domain—for instance, by comparing the Frobenius norm of the off-diagonal entries in the covariance matrix and in the cross-spectral density before and after applying OrthoTrans?*\n\n4. NormLin removes non-linearity yet keeps quadratic cost.  *Is there a way to reduce this cost?*\n\n5. In the experiments, you use different learning rate, model dimension, embedding size and number of blocks for each dataset. *Could you please freeze one single architecture, run it on all the datasets and compare this performance with the best architecture for each dataset?*\n\nLIMITATIONS:\nWhile the authors include a brief limitations paragraph in Appendix, a fuller discussion into the main paper, covering details like compute cost and dataset-specific hyper-parameters, is needed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces OLinear, a linear-based model for multivariate time series forecasting with two key components: \n- OrthoTrans: A data-adaptive transformation that diagonalizes the temporal Pearson correlation matrix, decorrelating features and simplifying forecasting.\n\n- NormLin: A lightweight linear layer with row-wise normalization that outperforms self-attention in accuracy and efficiency.\n\nExperiments on 24 datasets demonstrate state-of-the-art performance, particularly for long-term forecasting, while maintaining low computational overhead.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ Theorem 1 formally connects the temporal-correlation structure of the data with the orthogonal transformation used by OrthoTrans, giving the model a clear theoretical underpinning and enhancing interpretability.\n+ NormLin replaces multi-head self-attention with a row-normalized linear layer, cutting both FLOPs and memory footprint by roughly 50 % while preserving—or improving—forecasting accuracy.\n+ OLinear achieves state-of-the-art results on 140 forecasting tasks drawn from 24 public datasets, demonstrating robustness across domains and long-horizon scenarios.\n+ Both OrthoTrans and NormLin are modular, training-free components that can be plugged into existing forecasters to boost performance with minimal engineering effort.\n+ The experimental section is notably comprehensive, featuring extensive ablations, scalability tests, and comparisons against a wide range of baselines.\n\nWeaknesses\n- While training-time speed and memory are reported, the paper does not quantify inference-time latency or serving cost—metrics that are crucial in streaming or high-frequency settings; deployment-cost analysis is therefore incomplete.\n- Important modern linear or correlation-based baselines (e.g., RLinear, TSMixer, StemGNN, DEPTS) are omitted, leaving open whether OLinear’s gains hold against the strongest current alternatives.\n- The claim that NormLin is a “universal token dependency learner” appears overstated: its superiority is demonstrated only on time-series data, with no evidence provided for other modalities or tasks.\n\nQUESTIONS:\n- Why does NormLin show better gradient dynamics than attention? Are there any theoretic supports.\n- What is the relationship OrthoTrans with PCA?\n- Theorem 1 assumes Gaussian. How sensitive is OLinear to deviations from this assumption in real-world data?\n- How does performance degrade with noisy/missing data or non-stationary correlations? \n- What is the inference latency for ultra-long sequence (1M)? And some related references for long-term forecasting such as Pyraformer (ICLR 2022) should be discussed.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces OLinear, a computationally efficient alternative to transformer-based models for multivariate time series forecasting. The architecture combines two key components that can also be used as plugins: OrthoTrans, which applies orthogonal transformations to decorrelate input features, and NormLin, a linear layer that aims to reproduce channel-wise attention. OLinear consists of a Cross-Series Learner (CSL) using NormLin layers for multivariate correlations and an Intra-Series Learner (ISL) with MLPs for individual series processing. Results show OLinear achieves competitive or superior performance while reducing computations.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths:*\n- The two-component design (OrthoTrans, NormLin for cross-variable interactions) creates a clean separation of concerns where temporal decorrelation is handled by preprocessing and multivariate correlations by learned linear layers.\n- These two modules can work as plugins for existing transformer architectures, with validated improvements on iTransformer (+5.1%) and PatchTST (+10.1%), which is really interesting \n- The empirical benchmark is exhaustive, covering 24 datasets and 140 configurations, including zero-shot and few-shot scenarios and seems to show strong results. The ablation study is extensive.\n- Rank analysis demonstrates good expressiveness of the model.\n\n*Weaknesses:*\n- The proposed way to evaluate the correlation matrix is confusing and lacks proper formalization. I guess you want to estimate the autocorrelation of a block of T timesteps (context length) but what does this mean formally? The mathematical framework should be more rigorously defined as the concept is non-trivial.\n- In addition, when reading the description of the approach, two intuitive baselines came to my mind: (a) instead of eigenvector decomposition of the correlation matrix, we can directly perform data rotation by PCA (i.e., working with $XX^T$), (b) the correlation matrix can alternatively be estimated through a sliding window approach by computing cross-correlation between the timestamps across different windows. It will be valuable to perform additional experiments to compare with these approaches in order to have more insights on the chosen methodology.\n- Averaging correlations across N variables dilutes variable-specific temporal structures without theoretical justification. Why did not you apply decorrelations for each channel separately?\n- Decorrelation effectiveness is not shown, no metrics show Q^T CorrMat Q ≈ Λ or residual correlation analysis on a validation set, for example.\n- When looking at the source code, it appears that there are some implementation errors. The authors consider the Pearson correlation, but in `dataset/Generate_corrmat.ipynb`, we have `cov_matrix = cov_matrix / diag_vec` which seems to divide only by σ_i, not σ_i × σ_j, producing an invalid asymmetric correlation matrix.\n- Some other architecture choices are not well-explained nor well-motivated (e.g., the CSL block, the dimension extension). Please see my questions below.\n- Reducing the number of FLOPs is beneficial, but the complexity (memory for example) remains quadratic with respect to D, which can still result in substantial computational overhead for high-dimensional multivariate series.\n\nQUESTIONS:\n- On the asymmetric normalization: The cov_matrix / diag_vec normalization produces an asymmetric matrix that isn't a true correlation matrix. Is this an implementation error or deliberate choice? If intentional, what's the theoretical justification? Why is this implementation in a notebook? Maybe I didn't find the right file that saves the matrices.\n\n- On the decorrelation architecture: Have you explored architectures where each variable has its own $Q^{(j)}$ matrix to preserve variable-specific temporal structures? What information is lost by averaging correlations across all variables? Did you try the ablation studies with PCA on $XX^\\top$? What is the intuition? Did you try an approach with learnable $Q^{(j)}$? \n\n- What is the intuition behind the dimension extension? To what extent does it differ from the linear encoder? What does it bring? Why decorrelating after the dimension extension?\n\n- In the CSL: What is the purpose of the first linear layer, what is the intuition behind it? From my understanding, you want to reproduce the architecture of self-attention blocks, but then I don't see why we need the first layer. CSL:\n$$\\tilde{H} = \\text{LayerNorm}\\left(\\tilde{H} + \\text{Linear}\\left(\\text{NormLin}\\left(\\text{Linear}\\left(\\tilde{H}\\right)\\right)\\right)\\right)$$\n\n- In Table 6, could you elaborate on what \"Temporal NormLin\" baseline exactly is?  Do you keep the intra-series learner? Could you be more explicit about this architecture?\n\n- Could you also have a comparison with other lightweight multivariate methods such as SAMFormer [1] and TSMixer [2]?\n\n\n*References:*\n\n - [1] (Ilbert et al., 2024) SAMFormer: Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention.\n - [2] (Chen et al., 2023) TSMixer: An all-mlp architecture for time series forecasting.\n\nLIMITATIONS:\n- As said earlier, the memory complexity remains quadratic with respect to D.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles entangled temporal dependencies in multivariate time series forecasting through two main contributions: a data-adaptive orthogonal transformation based on eigenvalue decomposition of Pearson correlation matrices, and a computationally efficient normalized linear layer (NormLin) that serves as an alternative to self-attention mechanisms. The proposed method leverages dataset-specific statistical properties to decorrelate temporal features before applying linear forecasting models. The comprehensive evaluation across multiple datasets and and tasks in this paper demonstrates consistent performance gains over strong baselines, with both components showing modularity as plug-in enhancements for existing architectures.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThis paper presents a well-executed combination of theoretical innovation and empirical rigor. The core contribution using eigenvalue decomposition of temporal Pearson correlation matrices for orthogonal transformation, provides an elegant theoretical foundation with clear justification. The work demonstrates state-of-the-art performance across multiple datasets and tasks, with both components serving as effective plug-in modules that consistently improve various baseline models. I am particularly impressed by the comprehensive experimental evaluation, which includes extensive ablation studies, and evaluation across carious tasks with different prediction horizons.\n\n\n**Weaknesses:**\n\nSee questions below\n\nQUESTIONS:\n1. How does the method perform on highly non-stationary time series where the correlation structure changes significantly over time? Have you considered adaptive or online updates to the orthogonal matrices? \n\n2. You mention that OrthoTrans improves the rank of attention matrices in Transformer models. Can you provide more detailed analysis or proof of this phenomenon? What is the relationship between decorrelation and representation diversity?\n\n3. Have you compared against other adaptive bases beyond the eigenvalue decomposition, such as those learned through neural networks or other unsupervised methods?\n\n4. The orthogonal matrices Qi and Qo are computed from the training data. The paper only briefly mentions robustness to limited training data in the appendix but doesn't thoroughly investigate how performance degrades with smaller training sets or distribution shifts.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}