UNDERSTANDING TRANSFORMERS FOR TIME SERIES FORECASTING: A CASE STUDY ON MOIRAI
Authors: Dennis Wu
Yihan He
Yuan Cao
Jianqing Fan
Han Liu
Venue: ICLR 2026 Poster (2026)
Keywords: time series forecasting
Primary area: learning on time series and dynamical systems
TL;DR: We study transformers for time series forecasting, with a focus on MOIRAI.

Abstract: We give a comprehensive theoretical analysis of transformers as time series pre-
diction models, with a focus on MOIRAI (Woo et al., 2024). We study its ap-
proximation and generalization capabilities. First, we demonstrate that there exist
transformers that fit an autoregressive model on input univariate time series via
gradient descent. We then analyze MOIRAI, one of the state-of-the-art multivariate
time series prediction models capable of modeling arbitrary number of covariates.
We prove that MOIRAI is capable of automatically fitting autoregressive models
with an arbitrary number of covariates, offering insights into its design and em-
pirical success. For generalization, we establish learning bounds for pretraining
when the data satisfies Dobrushin’s condition. Experiments support our theoretical
findings, highlighting the efficacy of using transformers for time series forecasting.

Published as a conference paper at ICLR 2026
UNDERSTANDING TRANSFORMERS IN TIME SERIES FORECASTING: A CASE STUDY ON MOIRAI
Dennis Wu 23- Yihan He Yuan C Jianqing Fan", Han Liu Northwestern University, CCntter for Foundation Models and Generative AI, NNSF- AI Institute for the Sky, *Princeton University, "The University of Hong Kong (hibb, hanlin j @u northwestern. .du (yihan. he, jqfan) ) @princeton. edu yuancao@hku. hk
ABSTRACT
We give a comprehensive theoretical analysis of transformers as time series pre- diction models, with focus on MOIRAI (Woo et al., 2024). We study its ap- proximation and generalization capabilities. First, we demonstrate that there exist transformers that fit an autoregressive model on input univariate time series via gradient descent. We then analyze MOIRAI, one of the state-of-ther time series prediction models capable of modeling arbitrary number of covariates. We prove that MOIRAT is capable of automatically fitting autoregressive models with an arbitrary number of covariates, offering insights into design and em- pirical success. For generalization, we establish learning bounds for pretraining when the data satisfies Dobrushin's 's condition. Experiments support our theoretical findings, highlighting the efficacy of using transformers for time series forecasting.
multivariate
INTRODUCTION
Recent advancement of transformers is reshaping the field of time series forecasting. Numerous studies have demonstrated transformers are an effective architectures for various time series analysis tasks such forecasting (Woo et al., 2024; Ansari al., 2024; Liang et al., 2024; Das et al., 2023), anomaly detection (Zhang and Luo, 2025; Wen et al., 2025) and more. While recent works have made efforts to enhance transformers for times series forecasting by modifying model architecture (Zhang and Yan, 2023; Liu et al., 2023; Wu et al., 2021) or data processing (Reneau et al., 2023; Nie al., 2022), the research community has yet to understand how transformers perform so well on forecasting tasks even on the simplest settings. Moreover, existing methods (Zhang and Yan, 2023; Wu et al., 2021; Woo al., 2024; Liu al., 2023) predominantly depend heuristic reasoning and are notably deficient in rigorous theoretical analysis. A deeper understanding of transformers in time series forecasting will guide the design of more effective architectures by future practitioners.
A critical challenge for transformers to handle time series data is to handle arbitrary number of covariates as the architecture proposed in Vaswani et al. (2017) is designed to handle fixed vocabulary size. Therefore, recent studies have developed several ways to address this issue. MOIRAL (Woo et al., 2024) propose an unified way to concatenate all covariates into a single long univariate time series (any-variate encoding). MOIRAI also propose novel any-variate mechanism for disambiguating different covariates. I'raasformers (Liuu et al., 2023) treat each covariate as tokens. Compared to traditional approaches that perform attention along the temporal dimension, they perform attention along the variate attention Other models choose to discard covariate features by only considering
univariate time series (Ansari et al., 2024: Rasul et al., 2023). In this paper, we start by theoretically analyzing MOIRAL AI and its novel mechanisms. The main goal of this paper is to address the central question: What characteristics of architectures in MOIRAI contribute to their strong performance on time series data?
To understand MOIRAI in time series learning, we begin by examining one of the most widely used algorithms for time series regression: the auto-regressive (AR) regression (Hamilton, 2020). 1. We show that transformers are indeed capable of performing AR regression on univariate time series, indicating they can handle time series principled, algorithmic fashion. 2. We apply our theoretical
cqual contribution

---

Published as a conference paper at ICLR 2026
results ofate-of-the-arr time series model, MOIRAI (Woo al., 2024), wransformer-based model that can handle arbitrary number of covariates in various time series tasks. While several novel designs in MOIRAI are for engineering purpose and heuristic, we show that these unique designs enable it to perform AR regression on arbitrary number of covariates. 3. Our theoretical results contribute understand the effectiveness of recent time series pretraining approaches (Ansari et al., 2024; Woo et al., 2024; Jin et al., 2023; Das et al., 2023; Rasul et al., 2023; Liang et al., 2024). These methods involve pretraining transformers on large collections of time series, often spanning diverse domains, and have demonstrated strong empirical performance on a wide range of domains. To analyze this phenomenon, we derive generalization error bounds for pretrained transformer models under formal statistical framework. The contributions of this paper is threefold:
From an algorithmic approximation view, we prove the existence of a transformer capable of fitting AR model any given univariate time series via gradient descent. When extending this to the multi- variate setting, we theoretically verify that MOIRAI, with its novel any-variate attention can automatically adjust the dimensionality of the AR model to fit time series with arbitrary number of covariates. Our approximation results not only explain the strong performance of MOIRAL, but
also justify the encoding and architecture design of MOIRAL
We present the first pretraining generalization bound for MOIRAL on time series learning. We show that when the pretraining data satisfies Dobrushin's condition, the test error can be effectively bounded even when the data does not satisfy i..d. Specifically, when pretraining MOIRAI on multi-variate time series with length T, the test error decays by a rate of I/VAI.
Our experimental results on both synthetic and real-world data match our theories by showing that the prediction error of transformers reduces as the input time series length increases, corresponding to approximation result. This empirical finding not only verifies our theoretical results, but also demonstrates transformers are capable of acquiring algorithmic capability through training.
Organization. Section 2 introduces our problem setup and preliminaries. Section 3 describes how transformers simulate different AR regression algorithms and case study on MOIRAL. Section 4 analyzes the pretraining guarantee for learning transformers over different time series. Section conducts experiments to verify our theoretical results synthetic and real-world datasets. Section 6 discusses how our findings generalize to other models and limitations. Additional Related works are discussed in the Appendix C.
Notations. We use the following notation conventions. The vector-valued variable is given by boldfaced characters. We denote [n] := (1....,%) n) and [i j == (ú, i ..) for < j. The universal constants are given by C' and are ad hoc. Considering sequence of vectors (æ), WT). we use æ without index to represent the whole sequence, and Ziyy represents (x, ... x.)) for i < j. We impose periodic boundary conditions for the negative index, i.e., .- æT. For vector e we denote [|0]|| as its L2 norm. For a matrix A E Rm>x we denote its operator norm as ||||2 supeesa- ||Au||2. Random variables are given by calligraphic characters X,, and elements from a domain set are given by normal font x. For more details, see Table 1.
2 PROBLEM SETUP
This section presents backgrounds and formula definitions of the transformer model, and then introduce the auto-regressive model.
Transformers. We consider sequence of N input vectors ( C C We introduce H = [h,,... Aw] DDN, Given any H DxN, we define the attention layer follows.
Definition 2.1 (Attention layer). A self-attention layer with M heads is denoted as Att (.) with parameters 6, {(Vn), (Q.). (Km)) (mm)) DDD, The self-attention layer processes any given input sequence H DDN
M H I x (QT (KmH)),
Aun (H) H
m=1
where (1)= = ReLU())/N is the ReLU function normalized by N.
Next, we introduce the varivar attention, where Woo et al. (2024) uses it to replace the standard attention in transformers. The vay-variate attention introduces two learnable variables: Attention
2

---

Published as a conference paper at ICLR 2026
Bias U1, U2 R, for disambiguation between variates, In (Woo et al., 2024), they use this mechanism to allow transformers to distinguish different covariates.
Definition 2.2 (Any-variate Attention.). An any-variate attention layer with M heads is denoted as Atmm, (.) with parameters 01 {(Vm), (Qm), (Km), (m,,), (um)) With any input H E DDN,, we have
M
1 Atmo (H) +
E N
V H) x ((Q (K H) + uby * tmm t):
where (()) - ReLU())N.
RNN is a block diagonal matrix with block size T E N+, such that each block consists of Is, U U U, and denotes element-wise multiplication.
Definition 2.3 (MLP Layer). We denote an MLP layer with hidden state dimension D' as MLP
with parameters 02 (W1, W2) E R XD DDD The MLP layer processes any given input sequence H E DxN as MLP (H) = + (,,(W,H)
Finally, we define a L-layer transformer.
Definition 2.4 (MOIRAI Transformer). We define the L-layer MOIRAI transformer (Woo et al., 2024), (Fe(:), as
TFo H) -MIPPg (Armet (-.MIPOg (Armoy (H))
Note that this transformer is equipped with any-variate attention instead of the standard attention. For transformers with standard attention, we denote it as T.)
We use denote the vectorization of parameters a transformer and super-index & denote the parameter of the L-th layer. Thus, the parameter of transformer is defined by
2.f θ {{(10 K V um Um IMEEM] W. W (eue))
We denote the "attention-only" transformers with W define the following norm of a MOIRAI transformer as
= as TFg for shorthand. We
M
| op = max max de L] mE(M")
{110 l1.. K 12, /u//u/
+
IV 12 + || lW
m==
where M the number of heads of the E-th Attention layer.
Auto-regressive Model. Here, first consider the case where we aim to find a multi-layered transformer that performs regression via In-context learning (ICL). Specifically, we assume our data generated from an autoregressive process (q) follows, where q. d denotes the steps of lag and number of covariates, respectively. Consider sequence of data æ E BdxT = (æ)...... Dr where æ; ( ... = E Rdd Assuming our target (variate of interest) univariate, we define ARa(4) process generates xt as follows:
a. ol +
(w , 2/-
(2.1)
where 4 ~ N(O, 1), of E R!. We denote the concatenation of all weights w = (w), wa) E Rnd We assume bounded features ||xt-g:-1l1 < B, for all t E [T]. The first equation writes the AR process in scalar form, and the second writes it in vector form. In the following chapters, will start by considering the uni-variate case (AR;) and then move on to the multi-variate case (AR,).
3 TRANSFORMERS SIMULATE AUTO-REGRESSITE REGRESSION
We first investigate how MOIRAI performs univariate AR regression. Next, will move to variate AR regression and analyze how MOIRAT's unique design and pre-processing methods enable its advantages.

---

Published as a conference paper at ICLR 2026
3.1 WARM UP: UNIVARIATE AUTOREGRESSIVE REGRESSION
We start our analysis with a warm-up example on the AR, (q) model. We show that standard trans- formers are capable of performing gradient descent via in-context learning on autoregressive data. The results in this subsection apply to univariate time series models such as (Ansari et al., 2024). Here, we consider an input sequence with the following form
T1 P1
E R Dx(++1)
p: =
E 1T44+
(3.1)
P2
PT
PT+1
[ii < T
where is an one-hot vector with 1 at the i-th entry, and d' + +T + 3 = D. Here, our goal is to predict .T41-
Remark 3.1. Most in-context learning studies (Akyürek al., 2023; Bai et al., 2024; Li al., 2023) make an assumption on the input data, where they assume it is formatted with features and labels in the same column, i.e., y/
IN In contrast, adopt natural approach that leverages the raw structure of the data, particularly for the ARa(g) process. In this setting, each time step' label also serves feature for future steps. Further, the unknown value of q complicates the task of achieving such a format in Remark 3.1.
22
Yı
Our next lemma shows that transformers are indeed capable of reformatting into the form in Remark 3.1. Notably, we relax the assumption in Remark 3.1 of previous studies well.
Lemma 3.2. Given a sequence of token in the form of Equation 3.1, there exists a one-layer, quax head attention layer, such that for any q S qmaxo the columns of Attn' (H) has the following form:
0g -
Atnne (H); = [ra F4-1
Ii-g P'l
D¹ =
6
(3.2)
1
lii
< TH
The proof in Appendix E.1. Lemma 3.2 crucial in our analysis as connects the theoretical results in ICL (Bai et al., 2024) to univariate time series forecasting. When data formats in the form of Remark 3.1, Bai et al. (2024) show that there exists a multi-layer transformer that performs linear regression via gradient descent on the first N - data points and evaluates the N-th one. Thus, Lemma 3.2 implies transformers are also capable of performing linear regression on time series data.
This lemma applies to both any-variate attention and standard attention, the latter can be viewed as a special case of any-variate attention by setting w², u² = 0. Additionally, the construction of single layer with q heads is not strict requirement; the lemma also holds for e layers of q/c head attention, for any c satisfies q/c 2
With Lemma 3.2, AR1 (4) case. Consider the data generated by the AR process in Equation 2.1. Given an input time series R we define the least squares estimator the empirical risk minimizer over the time series, i.e.,
able to apply the in-context learning results in (Bai al., 2024) the
frey( (w, Ft -i-11 [ww, [z] lat 2 1st (1) x!]
2
(oee x)) T
Leg (w, 01-111-)) WFRM == argmin Trog (w, a). wCRd
t=1
where (v; denotes the concatenation between vectors, [#]-1a-gi x2 1:t-g (mr_1; 2 . x;-g)) E R224, ã denotes masking out the last time step of the target variate, and Lreg is loss, which is a-strongly convex, and 3-smooth over Rdd We make the following assumption and then present our first result on univariate time series (d = 1).
Proposition 3.3 (Univariate Autoregressive Regression via MOIRAD, Assume the regression problem dell-defined and has bounded solution and fix Quax > For any 0 S S 3 with 2
4

---

Published as a conference paper at ICLR 2026
Bu > O, and B2Bw/2, there exists L-layer MOIRAI transformer TF (.), with
L = L L2, L1 12x log
BBBu
)1, L2
9mmax MOKEE M
3, | op S |4R 83 83
2e
(R = max Bg Bw: BR: 13), the following holds. On any input data generated by any AR1 (q) process such that 0 < q S (max, | Bw/2, we have
(ūERM [01-1a-qi-- a - Se
(3.3)
where öт read(TFg (H)) read H) operation reads out the first entry of Tth column H.
This proposition follows immediately from Lemma 3.2 and (Bai et al., 2024, Theorem 4). The above result applies for standard transformers as well as in our construction u 2 in all heads and layers. Further, one can replace the least squares ERM with lasso or ridge ERM and obtain a similar result by applying Theorem 4, 7, and 13 of (Bai et al., 2024).
So far, we show that transformers are capable of solving univariate AR regression with, at best, one additional layer compared to the results in (Bai al., 2024). The result above provides insights on transformer-base
univariate time series foundation models (Ansari et al., 2024; Rasul al., 2023; Das et al., 2023). To study MOIRAI, we then include two ingredients into our analysis: the any-variate encoding and the covariates in the following chapters.
3.2 CASE STUDY: MOIRAI TRANSFORMER
In this subsection, we extend our results to the multivariate autoregressive process (d > 1) and the encoding method of MOIRAL Note that in the multi-variat case, we focus on MOIRAI as it is compatible with arbitrary numbers of covariates. For other transformer-based models with the same feature, we leave it to future work. We start by introducing the any-variate encoding.
Encoding. Woo al. al. (2024) propose to flatten a d d-dimensional time series, x E RdxT into 1- 1-dimensional sequence, i.e., x' IaxTd This operation transforms time series with arbitrary number of covariates (d), into a long sequence with fixed dimension, enabling consistent input dimension for transformers. Following the flattening operation, Woo et al. (2024) also proposes to add two types of indices into the input sequence: the time and variate ID. We term the above operations as the any-variate encoding, which transforms a multivariate sequence x E RdxT as follows:
of
x²
2 P1
27
2 P1
Pr
Pr
(3.4)
el
e1
6d
ed
where e; is the variate index, one-hot vector with i-th entry being 1, and p; is the time index, which defined the same as Equation (3.1). This is without loss of generality because the discrete- time and variate ID used in (Woo et al., 2024) can be easily transformed into our constructed positional encoding with the patch embedding layer. Note that only the target variate has length T, we highlight oy as it is our prediction target and will be masked as O.
Now we define the history matrix (g) E Ra+1xT for the i-th covariates (af,"". , with order g-
such that
Ar(g),. x x2- for E [d], v [q].
where in the j-th column of As As(g). it contains historical values of 21 with lag q 0.
Lemma 3.4. Fix qmax D E N+ Given any I > 0, > > 0, > such that T > q, 9max 2 For any input matrix H in the form of any-variate encoding Equation 3.4, such that H R Dxf" There exists a one layer, qmax head any-variate attention that performs the following operation.
A1(g)
Aa(g)
Aa(g) 0 XT
T
x
x7
0
02'2
pi
PT
pi
pr
P1
pr
4
61
e1
62
63
ed
ed
where d' d' qmax-
5

---

Published as a conference paper at ICLR 2026
The proof is in Appendix E.2. Intuitively, the above operation performs the same operation in Lemma 3.2 but in a variate-wise fashion. Lemma 3.4 is crucial to our understanding to MOIRAI as it shows that any-variate attention is capable of organizing the history of each variate efficiently and in a parallel way. To again achieve the format in Remark 3.1, one has to stack all Az(g) in the same columns, which can be easily done by single layer of attention via Lemma 3.2 and (Bai et al., 2024, Proposition A.5) (details in Appendix E.2). This lemma serves as a foundation for MOIRAI to handle multi-variate time series with in-context learning which we present as the theorem below.
Remark 3.5. Comparing to Lemma 3.2, Lemma 3.4 is specifically for any-variate attention in our construction, where we demonstrate that several special mechanisms any-variate attention enables variate-wisg wise operations in parallel.
Theorem 3.6 (Any-variate Autoregressive Regression via MOIRAI). Let L1 [Smmx + 1, L2 [2glog B_B=], with some dmaxx Qmax > O. For any S a S 3 with k. = 2. B > O, and E < B, B,,2,, there exists an (L1 + Lly-l of MOIRAl transformer equipped with any-variate
Attention, satisfies the following
max (11,1)
( S 3, 111@1 S |4R+88
Σ M dmax +
Gmax:
D [(iax 1)dmax + T + 2,
such that for any input time series x with length T generated from an ARa(1) process, where RdxT"" S 9max: d S dmaa it satisfies
||
(3.5)
where 11 = g and H E RDN the any-variate encoding of
Remark 3.7. Theorem 3.6 indicates there exists a MOIRAI transformer that fits an autoregressive model on time series as long as the number of covariates no greater than d and lags no greater than qmax- This shows its ability infer the underlying AR model in a principled way and provides a possible explanation for its zero-shot performance on a wide range of datasets. To modify the setting from multi-variate prediction, one can easily apply Bai et al. (2024, proposition A.5), where the hidden dimension scales linearly w.r.t. the number of variates to predict.
Remark 3.8. Here the max number of lags and covariates depends on the hidden size D. If MOIRAI takes more than dmax covariates, in our construction, it will only fit AR with the first dmaax covariates. Therefore, while exceeding dmax will not completely destroy model performance, certain performance degradation is expected. The same concept applies to Jmax as well. However, we would like to highlight a counterexample that one can find an AR model that has non-zero weights on 4max lags, in this case, the model might not be able to generate meaningful prediction.
The proof is in Appendix D. Observe that there exist two trade-offs in Theorem 3.6. First, fmax is upper bounded by the D (up to constant), which is natural trade-off in our construction. Second, the approximation error is roughly O(e-"), suppressed exponentially by the number of layers, as in our analysis, each layer of MOIRAI performs single step of gradient descent on Lugg
4 ANALYSIS ON PRETRAINING
In this section, we investigate the generalization bound of pretraining MOIRAI multiple non IID time series. Specifically, we assume the time series satisfy the Dobrushin's 's condition, which is a regularity condition commonly used for dependent data. Informally, we show that the generalization error decays roughly with the order of O ViT) ), where n is the number of time series and T the length of each time series. This provides insights and justifications on the large scale pretraining for time series transformer-baat based models.
Let be meta distribution, and each distribution drawn from it P(T) ~ n, satisfies Dobrushin's 's condition (Dobrushin, 1968) (which we will introduce shortly). For pretraining data, we first sample distributions PTT) i.i.d. from and for each distribution, we sample time series (æ1j. ... æT,). for j E [n)) and each of them contains no more than d covariates and with lag step more than q. For each time series, we encode it with any- varivari encoding into an input matrix denoted as H E DxN We define each pretraining sample as z; == (H,,V;) Uy)) where yj = æh.- We consider
DDe any-variate encoding, N dT.

---

Published as a conference paper at ICLR 2026
the squared loss between model prediction and the label, i..
E(z,. 0) = 1 clipA, (read, (TF (H) ))}²
where Clipp, (1) = max(min((, Br); - B. and TFş is the MOIRAI transformer defined in Definition 2.4 with Clip()) applied after each layer. The pretraining loss and test loss defined as the following:
T ΣΣ 8 23t); 1(0) B [((0
T n7 (=1==
(4.1)
The goal of our pretraining algorithm is to find an empirical risk minimizer (ERM) over MOIRAI transformers with L layers, M heads, and norm bounded by B:
argmin I(O),
L. D
M,
{0 (0(1.), oQ(12i) (0..1200$$111
max M s M, max D < S'D 110|lpp BB. FE[M]
(4.2)
4.1 NON IID TIME SERIES
Here, we consider the training data æ to be drawn from distribution P satisfying Dobrushin's condition. Under this condition, we are able to present several generalization bounds on pretraining. Definition 4.1 (Dobrushin's Uniqueness Condition). Let X = (X1..... (T) be a sequence of random variables over D- The Dobrushin coefficient a of X and the influence I of variable X, on variable X; are defined as
axX))= = max ΣI,
I)-((0)=
max x---3
| (1x--15) Pri (1=i--):<1111
ISiST
171
where x-i-j E DD x/ ED., |1-1lry denotes the total variation distance, andxx_ represents the vector x after omitting the i-th element. We say the variable satisfies Dobrushin's 's uniqueness condition if (x)))) < L. For distribution P. we denote a(P) = supxx= ((X)).
Definition 4.2 (Log Dobrushin's Coefficients). Let X. = (Xi,,.. "',T'" be a random variable over Df and let Ps denote its density, Assume that Ps > on all D For any + E [T], the log influence between j and is defined as:
1 sup log P(a,,j,,---sl P 1,,,,
(2)
P(R,2),3-111] P 1,,
where the sup is taken over overx-i-j;-a x max Ejji (2C). Note that a(-) has a natural bound in [0, T 1), with a = 0 implies i.i.d. Remark 4.3 (On the choice Dobrushin's condition). general, time series analysis (non IID requires certain level of regularity condition on data, to derive meaningful learning bounds and exclude extream cases. For example, mixing properties (Mohri and Rostamizadeh, 2010; Steinwart and Christmann, 2009; Kuznetsov and Mohri, 2014), hypercontracttvity (Ziemann and Tut, 2022), or Dobrushin's 's condition (Dagan et al., 2019). Dobrushin's condition is not only a commonly used regularity condition for time series data, it also fits several common statistical models, which makes our derivation suitable for further analysis on different models. An example is shown in our section 4.3. One more benefit of having this condition instead of mixing properties is that do not need to assume stationarity of the data, which can easily be violated in practice. We highlight that verifying all those conditions difficult in practice, as the true distribution of data intractable.
and the log-coefficient of DC is defined as mog (XC)
4.2 GENERALIZATION BOUNDS OF MOIRAI
Theorem 4.4 (Pretraining Generalization Bound), Let LL,M.D'.E B be the parameter space defined in Equation 4.2. Assume 0noo (P(T)) < 1/2. Then with probability at least 1 E, ERM Ô satisfies the following:
B 1- (P(P))
LIMDz DD log(1/s)
L( S inf
L( 0
L.BLMM. M.
n.
where C. is an universal constant, and Ç 0(log(2 + max B,R, Br, T,d)))
7

---

Published as a conference paper at ICLR 2026
The proof is in Appendix E.4. Here we observe that the bound mainly depends two terms, the Dobrushin's 's coefficient, and the demoninator nT. Under Dobrushin's condition, increasing n alleviates the performance degradation of 1/1-a exponentially in Further, increasing n also tightens the bound with the order 1/vm. Further, in Theorem 4.4, we do not assume our data is generated from the AR process, only its Dobrushin coefficient. When the data is generated by the AR process, we able to give a more explicit bound on the same test loss as described below.
Proposition 4.5 (Test Error Bound). Following the setup in Theorem 4,4, if pretraining samples are generated by some AR process with noise sampled from N NO.,02 o?)², then with probability A(1 - e),, ERM satisfies the following:
-L
B
LIMD DD'X log(1/s)
L(Ø) 00
Bu exp
+
1 (P(T)
nT
where A (11 (re/DaBue 422))7). C is universal constant, and < O(log(2
max ( R. B2, T.d)))
Considering the model parameters (M, D', d) are of constant level, one is able to further optimize the bound to L L()) S (nT) -/a by selecting appropriately. Next we provide an example of the application of Proposition 4.5 on AR(1) process with the following form
2++1 (w, + Et, e <~ N=(,00, o²), with æt E Rd, w E B C V 2/+1-
To satisfy the condition of (P) < 1 we assume B < In (o?), || < 1. The first condition
2.
comes from the fact that require the pair-wise potential of this time series to be less than 1/2 (For more details, see Appendix E.5). The second condition comes from the requirement of it being stationary.
Corollary 4.6 (Generalization Bound for MOIRAI AR(1)). Considering AR(1) processes with each of its Dobrushin's 's coefficient bounded by 1/2. With probability at least 8(1 - s). ERM θ satisfies the following:
L
o²
L(MD2 + DD'X + log(1/E)
L(Ō) =
v1
B,
exp
+
a (AR
nf
with
1 2/ 2/m By |w||
a<AB(1))
S min
and - Olog(2 + max B, R, Bz, d))). If we Te further optimize the bound by viewing the hyperpa- rameters as constants, the test error obeys O
WT w.h.p. whenever is small.
Remark 4.7. Corollary 4.6 provides an application of Dobrushin's 's condition on common statistical models, indicating that Dobrushin's condition can be further used to certify stability and derive generalization bounds for broad class of time series processes beyond the i...d. setting.
Setup. We use MOIRAI-base, 12 layer MOIRAI transformer. The hyperparameters of this experiment are in Table 2. We use MSE loss for pretraining instead of NLL loss for simplicity. For pretraining, we follow the setup Woo et (2024) but the patch size as minimize the impact of patch embedding. During pretraining, each time series is randomly sampled. We evaluate the pretrained model on our test data with d = (3. 4, 5). = and の² = 1 with different input length. We compare MOIRAL AI with LSR performing different gradient descent steps. For LSR, we assume known. When MOIRAI takes T length input, LSR is trained on T 1 samples with each having dg features. A detailed example standard transformers and MOIRAI with ReLU replacing Softmax, which we term it as MOIRAI-relu. For standard transformers, keep the any-variate encoding but standard attention. DDtaiis of data generation can be found in Appendix F.3.
implementation is in Appendix F.4. We also include the
5 EXPERIMENTS
To verify our analysis, we first train transformers on synthetic datasets generated from AR process with different parameters. The goal of this experiment is to verify the existence of transformer
2 Here we assume fixed d, q across all samples as one can describe a lower dimension/order AR process with zero coefficients.
In (Woo et al., 2024), without any-variate attention, the error of MOIRAT-small increases roughly 40%.
8

---

Published as a conference paper at ICLR 2026
4
d 5
1.20
Model
1,20
Model MOIRA
1:25
Model
1.18
KOIBAI
MOHHAI
1.16
1.15
MDIRA-ra
MCHRAL-eolu
framer
transformer
transfommr
1.14
(tor=100)
1.10
[iter= 100)
1:35
LS (ttr-
1.12
(Ror=50)
1.10
(Rorw50)
1,05
1.00
1.00
1.05
30
100
130
50
100
150
30
100
330
Lookback Size
Lookback Size
Lookback Size
< 10,
d 2,
3,
1.10
Model
1.08
Model
1.10
Model MOHIAJ
MOIRAI
1.00
US (ler-100)
1.05
L5 liler 100)
1.08
LS (iter- 100)
1.06
(ter=50)
liter-
1.06
US (ttrm30))
1.04
1.04
1.04
1.02
1.00
1.00
1.00
50
100
150
50
100
150
50
300
150
Lookback Size
Lookback Size
Lookback Size
Figure 1: Top: Model performance on data with different number of covariates. For both Mossd--based models, we observe their performance behave like least squares. As in our construction, the longer the lookback size is, the more examples available for transformers to an AR. Note that our test data has variance o² 1. thus the MSE for both models are expected to converge to 1 as the increases. Buttom: Generalization to unseen values of d. q. From left to right, we have MOIRAI's generalization power (pretrained on E {4, 5): {4,5}) on high dimensional (d 10), low dimensional data (d 2) and high lag step + low dimensional data (d = 3, q = 7). Note that high and low is compared with pretraining data. We observe that even when MOIRAI did not learn from any time series with d = 10, it is still able generalize well and shows even better sample complexity than LSR. Finally, even when both q, d are unseen, it does not impact MOIRAT's ability to make accuracy predictions. The lines are the average over 5 runs.
that performs least squares regression (LSSR) on input time series with bounded lags and number of covariates. Next we study whether pretrained transformer is capable of generalize such an ability to unseen values of d, q. More empirical results such as on real-world data are in Appendix F.5.
Since our test data generation process obeys noise variance = 1, when fitting a linear model, the expected MSE will converge towards as lookback size increases. Based on Theorem 3.6, the length of input time series also corresponds to the number of examples model perform least squares on via gradient descent. We observe that as the input length increases, the predictive error of MOIRAI decreases similarly to least squares, which verifies Theorem 3.6. Next, when pretrained on diverse dataset, pretrained MOIRAL is able to adapt to different number of covariates and perform least squares accordingly. Further, when replace softmax with ReLU, the performance gap is negligible. For standard transformer, while it also behaves similar to other models, it does present higher error comparing other baselines, indicating the advantages of using any-variate attention.
Results.
Next we are interested in whether a pretrained transformer can generalize to unseen values of d and 9. Thus, we train transformers (MOIRAI) on synthetic data generated with AR with E {4, 5). In our construction, pretrained transformer is compatible with lower order and dimension AR data. We evaluate the trained model on data with unseen values of d. We select d = 2, d = 10, to represent the scenario when the number of covariates is lower and higher than pretraining data.
5), and
Results. We observe that even when facing data with unseen number of covariates, MOIRAI is still capable of performing AR regression effectively. Note that for d = 10, LS require higher sample complexity to obtain similar performance to = 5 case. However, the pretrained MOIRAI is able to outperform it from such aspect. For d = 2 all models perform well, again verifies our theoretical results. Finally, when facing data with unseen both and q, it is still capable of performing well.
DISCUSSION
Here we discuss the implications of our results. First, we demonstrate how transformers perform univariate AR regression, classic time series prediction algorithm. Next, demonstrate that the unique varivari attention proposed by Woo et al. (2024), indeed provides benefits for transformers to process each covariate in parallel fashion (Lemma 3.4). This observation not only leads to

---

Published as a conference paper at ICLR 2026
next result, showing MOIRAI can perform AR regression arbitrary number of covariates, also provides a justification on the design of any-variate encoding (Woo et al., 2024). The above approximation results provide profound insight on designing transformer- baasd time series models. Next, for pretraining, we derive generalization bounds when data is under Dobrushin's 's condition, where the bound scales roughly by VnT It is worth investigating how to further relax this assumption, which we leave to future work. For predicting multiple time steps or nultiple variates, one can easily apply Bai et al. (2024, Proposition A.5) to obtain the same result with hidden dimension D scales linearly. More discussions on the limitations of this work is in Appendix B.
Which transformers do we cover? Our results in Section 3.1 also covers notable SOTA models including classic transformers (Vaswani el al., 2017), Chronos (Ansari et al., 2024) (Theorem E.5), MOMENT (Goswami et al., 2024). Architectures using rrassorrmerddecoderss are also compatible, such as TimeGPT (Garza and Mergenthaler-Cansee Canseeco, 2023), TimesFM (Das et al., 2023). see Appendix B (Bai et al., 2024). Furthermore, the use of patch embedding Nie et al. (2022) does not affect our theoretical result an AR sequence remains an AR sequence after patch embedding transformation (linear), thus solvable by our AR regression framework. One simply needs to use weight-tying trick (for the last layer embedding). to obtain the prediction of the original input. Univariate models with patch embedding also fits Section 3.1.
Notable wransformer-based SOTA models do not cover include iTransformer (Liu al., 2023), CrossFormer (Zhang and Yan, 2023), as they also applied attention mechanism on the covariate dimension. This novel technique requires further analysis thus we leave future work. Autoformer (Wu et al., 2021) also utilizes a technique named auto-correlation, which requires different theoretical tool study. Similarly, Time-LLM (Jin et al., 2023) and Lag-Llama (Rasul et al., 2023) utilize either natural language inputs, or the lag features, therefore cannot be covered by our theoretical analysis.
Can we verify Dobrushin's condition on time series data? Dobrushin's 's condition is a property of the joint distribution, not of the observed time series values alone. Therefore, one general cannot verify this condition without knowing the true distribution behind the data. However, practical solution we suggest to first fit commonly used models on the observed time series such as AR, the fitted model. The sufficient conditions for several
ARMA etc, and then verify the condition models to satisfy Dobrushin's 's condition is derived in Appendix E.5. We also provide a python code that automatically test input time series with the approach described above. Note that it is possible to switch from Dobrushin's 's condition to 3-mixing Mohri and Rostamizadeh (2010), a more strict condition on non IID data, in our generalization analysis. By doing so, there existing an algorithm Mcdonald et al. (2011) to estimate the 8-mixing coefficient via histograms. Finally, the authors wish highlight that those dependency assumptions are mainly used for theoretical studies as they excluded many unlearnable examples. These assumptions do not imply they are necessary to the success of model training.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
DW supported by the Northwestern Advanced Cognitive Science Fellowship. YC supported in part by NSFC 12301657 Hong Kong RGC ECS 27308624, and Hong Kong RGC GRF 17301825. JF research was supported part by the NSF Grants DMS-22 10833 and DMS-24 2202 and the ONR Grant N00014-25-1-255 Simons Foundation MPS-AI-0001000
HL is partially supported by NIH ROILMI372201 NSF AST-2421845, AbbVie, Dolby and Chan Zuckerberg Biohub Chicago Spoke Award. This research supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility at Northwestern University which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Teccnology.
REFERENCES
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023.
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic and neural time series modeling in python. Journal of Machine Learning Research, 21(116):1-6 2020.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815. 2024.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Yuval Dagan, Constantinos Daskalakis, Nishanth Dikkala, and Siddhartha Jayanti. Learning from weakly dependent data under dobrushin's condition. In Conference on Learning Theory, pages 128. PMLR, 2019.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. decoder-only foundation model for time-series forecasting, arXiv preprint arXiv:2310.
2023.
L. Dobrushin. The deseription of a random field by means of conditional probabilities and conditions of its regularity. Theory of Probability and Its Applications, 13:197-224, 1968.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Azul Garza and Max Mergenthaler-Canee Canseco. Timegpt-1, 2023.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I Webb, Rob J Hyndman, and Pablo Montero- Manso. Monash time series forecasting archive. arXiv preprint arXiv:2 100.06643, 2021.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In International Conference on Machine Learning, 2024.
James D Hamilton. Time series analysis. Princeton university press, 2020.

---

Published as a conference paper at ICLR 2026
Yihan He, Yuan Cao, Hong-Yo Yu Chen, Dennis Wu, Jianqing Fan, and Han Liu. Learning spectral methods by transformers. arXiv preprint arXiv:2501.01312, 2025.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yux- uan Liang, Yuan- Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint wxXiv:2310001720 2023.
Christof Külske. Concentration inequalities for functions of gibbs fields with application to diffraction and random gibbs measures. Communications in mathematical physics, 239:29-51, 2003.
Vitaly Kuznetsov and Mehryar Mohri. Generalization bounds for time series prediction with non- stationary processes. In International conference on algorithmic learning theory, pages 260-274. Springer, 2014.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
Yingcong Li, Muhammed Emrullah Iidiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference Machine Learning, pages 19565-15 19594. PMLR, 2023.
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6555-6565, 2024.
Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforce- ment learning via supervised pretraining. arXiv preprint arXiv:2310.08566, 2023.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.
Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.03576, 2023.
Daniel Mcdonald, Cosma Shalizi, and Mark Schervish. Estimating beta-mixing coefficients. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 516-524. JMLR Workshop and Conference Proceedings, 2011.
Mehryar Mohri and Afshin Rostamizadeh. Stability bounds for stationary e--mxxing and B-mixing processes. Journal of Machine Learning Research, 11(2), 2010.
Eshaan Nichani, Alex Damian, and Jason Lee. How transformers learn causal structure with gradient descent. arXiv preprint arXiv:2402. 14735, 2024.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint 1rXiv:2211.1423 2022.
Arushi Rai, Kyle Buettner, and Adriana Kovashka. Strategies to leverage foundational model knowledge object affordance grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1714-1723, 2024.
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Hassen, Anderson Schneider, et al. Lag-Ilama: Towards foundation models for time series forecasting. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
Alex Daniel Reneau, Jerry Yao-Chieh Hu, Ammar Gilani, and Han Liu. Feature programming for multivariate time series prediction. In International Conference on Machine Learning. pages 29009-29029. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Michael E Sander, Raja Giryes, Taiji Suzuki, Mathieu Blondel, and Gabriel Peyré. How do trans- formers perform in-context autoregressive learning? arXiv preprint arXiv:2402.05777, 2024.
Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on relu and softmax in transformer. arXiv preprint arXiv:2302.0646/ 2023.
Ingo Steinwart and Andreas Christmann. Fast learning from non-iid observations. Advances in neural information processing systems, 22, 2009.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174. PMIR, 2023.
Martin Wainwright. High-dimensional statistics: non-asymptotic viewpoint, volume 48. Cam- bridge university press, 2019.
Mi Wen, ZheHui Chen, Yun Xiong, and YiChuan Zhang. Lgat: A novel model for multivariate time series anomaly detection with improved anomaly transformer and learning graph structures. Neurocomputing, 617 617:129024, 2025.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo.
Unified training of universal time series forecasting transformers. arXiv preprint arXiv:440..02592, 2024.
Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Replacing softmax with relu in vision transformers. arXiv preprint 2rXiv:2309008886 2023.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition transformers
with auto-correlation for long-term term series forecasting. Advances in neural information processing systems, 34:22419-2244 22440, 2021.
Biao Zhang, Ivan Titov, and Rico Sennrich. Sparse attention with linear units. arXiv preprint arXiv:2104.07012, 2021.
Wenxin Zhang and Cuicui Luo. Decomposition based multi- scale transformer framework for time series anomaly detection. Neural Networks, 187:107399, 2025.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
Yufa Zhou, Yixiao Wang, Surbhi Goel, and Anru R Zhang, Why do transformers fail to forecast time series in-context? arXiv preprint arXiv:2510.09776. 2025.
Ingvar Ziemann and Stephen Tu. Learning with little mixing. Advances in Neural Information Processing Systems, 35:4626-4637,, 2022.
13

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIAL
A Table of Notations
14
Limitations and Impact
15
Related Works
15
D Additional Theoretical Background
16
E
Proofs
19
E,1
Proof of Lemma 3.2
19
E.2
Proof of Theorem 3.6
19
E.3
Proof of the Lipschitzness Any- Variati Transformers
24
E,4
Proof of Theorem 4.4
27
E.5
Analysis of Corollary 4.6
32
E.5.1 Further Analysis on Dobrushin's Condition Common Models
32
E.6
Additional Details
34
Experimental Details
35
F.1
Environment..
35
F22
Model Architecture
35
F33
Synthetic Data Generation
35
F.4
Baselines
35
F.5
Additional Experiments
36
F66
Evaluation on Real- World Datasets.
36
F.7
Ablation Study Attention Bias
37
A TABLE OF NOTATIONS
Table Mathematical Notations and Symbols
Symbol
Description
The i-th component vector æ: Inner product for vectors a, b Rd Index set {1.... I}, where E N+ Spectral norm, equivalent to the 2o-norm when applied to a vector The largest L2 norm of column vectors of a matrix
(a, b)
2,00
Aij
The element on the i-th row and j-th column of matrix A
The sub-sequence of sequence æ from coordinate to Concatenation between column vectors v + u (w u Concatenation between two row vectors
[u; v]
N
Length of transformer input sequence Number time steps time series Number of attention heads.
T
M
g
Lag of an AR process.
d
The number of covariates in an AR process
0
Vector (bold lower)
A
Matrix (bold upper)
X
random variable (calligraphic)
element from domain set Domain of random variable X
Dr
one-hot vector with its i-th entry
Probability distribution of X The probability P w w|
Pa/w w)
14

---

Published as a conference paper at ICLR 2026
B LIMITATIONS AND IMPACT
Limitations. One limitation in our analysis is that consider ReLU instead of softmax in attention mechanisms. While the same approach also is in theoretical (5; 22; 15) and empirical works (41; 43; 33), one might obtain different approximation bound comparing to Theorem 3.6. However, in our generalization analysis, the difference is small as softmax does not affect the model complexity too much. Another aspect is that we mainly focus on AR processes. While in the appendix, we do show the approximation result for non-linear AR processes generated by ReLU network, to achieve universal forecasting, more general assumption on data is required. Another limitation is we consider MSE loss instead of NLL loss used in (40) for pretraining.
Impact Statement. This paper studies the theoretical aspect of transformers as time series foun- dation models. No negative societal impacts that the authors feel should be specifically highlighted here.
C RELATED WORKS
Transformers in Time Series Forecasting, The recent progress in foundation models (35; 6; 29) has begun reshape the field of time series forecasting, critical task of predicting the future based on history (14). However, there are two major challenges in building time series foundation model: (a) the model must be able handle an arbitrary number of covariates, and (b) the model must generalize to unseen time series domains. To circumvent (a), several studies simplify the task by considering only univariate time series (4; 30; 8). (8) propose decoder-only transformer pretrained on both real and synthetic datasets. (30) incorporate lag features and the Llama architecture to pretrain a large univariate time series foundation model. (4) leverage the power of large language models (LLMs) by using pretrained LLMs backbones.
Recently, (40) proposed MOIRAI, the first time series foundation model capable of handling an arbitrary number of covariates. It addresses (a) by concatenating all covariates into mimeiimensi sequence, ensuring a consistent input dimension across datasets. It addresses (b) by pretraining on large collection of time series datasets (12; 42; 19) spanning domains such as weather, traffic, electricity, and industry. MOIRAI not only generalizes across a wide range of domains, but its zero- shot performance also surpasses several strong supervised learning baselines (23; 28; 45). However, the machine learning community has yet to provide a suitable explanation for MOIRAI'S impressive performance. Therefore, this paper the first offer theoretical guarantees for MOIRAI as a time series foundation model.
In-Context Learning. In-context learning (ICL. an emerging capability of large foundation models, enabling them to learn diverse and unseen tasks from given examples. (6) first provide empirical evidence of ICL in large language models (LLMs); by presenting several examples of (ær y) pairs, GPT-3 effectively infers the relationship between æ and y. (10) then conduct quantitative experiments on simple function classes, such as linear regression. Their results demonstrate that large foundation models can learn the parameters of these function classes. Subsequently, several theoretical studies (5; 37; 1; 2) have proven that different types of transformers can implement algorithms such as gradient descent. This discovery provides theoretical foundation for the empirical findings in (10).
The closest studies to this paper are (27; 32). However, (32) examines ICL in the context of next-token prediction using a linear transformer. While their theoretical results relate to in-context learning on sequential data, they are insufficient explain transformers' success time series forecasting. (27) explores another case where the data is modeled as Markov chain generated by transition matrix. They demonstrate the existence of induction heads that enable transformers to perform next-token prediction. However, their scenario does not align with multivariate time series, which is where our main contribution lies.
The biggest different between our work to existing ICL studies (5; 27; 37; 24) is we do not consider structured inputs as in practice, time series data is considered a long sequence instead of many feature-label pairs. Next, in our case study, we show that MOIRAL Al can perform AR regression on different dimensions of feature, while ICL studies mostly focused on a fixed dimension of features. Last, our pretraining analysis considers the case where data is non IID, which is critical in time series data due to the dependent nature. Therefore, this paper indeed provides new insights on using transformers for time series learning. Another related work both time series analysis, in-context
15

---

Published as a conference paper at ICLR 2026
learning and transformers is (46). In our paper, study the function class of MOIRAL, where (46) studies transformers with LSA with no-MLPs. Next, while in our paper, our construction also reshapes input sequence into Hankel-lik like matrices (sliding window features) and then performs LSR by gradient descent. On the other hand, (46) directly feeds the Hankel matrix H as input and shows that the sigma-algebra generated by I.S((H) is strictly coarser than generated by H itself, so LSS inevitably loses information relative to OLS. Also, our construction is algorithmic, and (46) focuses on representation space, and our results rely on two key powers that (46) does not have (Lemma 3.2, 3.4, Prop 3.3, Thm 3.6). Further, our construction uses model depth to unroll GD steps and (46) uses model depth to compose cubic maps, which cannot simulate least square regression. Thus, the difference between our work and (46) lies in our settings and goals. Finally, we have added short discussion on this work in the related work section.
Learning Theory on Time Series Data. Efforts time series learning theory have developed
frameworks for non-i..d. data using mixing ergodic assumptions (7; 47; 26; 34). Yet these results are largely agnostic to model architecture and thus offer limited insight into the specific advantages of
transformers. In short, clear theoretical understanding of how transformer architectures contributes to time series forecasting remains elusive.
Relationships to (5). Here we first summarize the contribution of (5). Their major contribution is to demonstrate there exists transformer to perform gradient descent in-context, which provides great starting point to provide possibile explanation on the empirical success of transformers in ICL. Their generalization results focus pretraining on ICL tasks. However, they mainly focus on the standard transformer architecture (with ReLU activation) and IID samples. They did not cover:
A Arbitrary number of covariates (feature dimension in their paper)
B Any-variate encoding (different input format)
Any-variate attention (MOIRAI's new architecture)
D Non-IID learning time series
On the other hand, we focus on MOIRAL, specific architecture and encoding method designed to address the problem where standard transformers cannot handle an arbitrary number of covariates. Their design/approach is simple but powerful. We extend the results from Bai et al. (5) to tackle the above setting differences (which is realistic to MOIRAI) with our novel results. To address (B), use Lemma 3.2, deomnstrate how can apply Bai et al. (5)'s results to MOIRAI with minimal effort. For (A, C), we provide Lemma 3.4, which demonstrates a key insight on how any-variate attention can handle (A, C). Both Lemma 3.2 and Lemma 3.4 are novel results and not based on Bai et al. (5). Finally, for non-lID time series pretraining, we introduce Dobrushin's condition and new proving techniques (Corollary E.16, Lemma E.18) to obtain generalization time series learning.
D ADDITIONAL THEORETICAL BACKGROUND
Here, include several technical lemme that intensively used throughout our paper. The Lipschitzness of an MI.P layer is obtained in (5, Lemma 1.1), which we restate below
Lemma D.1 ((5)). For a single MLP layer, 0₂ = (W1, W2), we introduce its norm
||102|| = W Willop llop + Wellopp
For any fixed hidden dimension D', consider
2 (02 1| < B}.
Then for H Ha 02 O2,B, the function (0,, H) MLPe,, is (BR)-Lipschitz w.r.t. 02 and (1 + Liipscic w.r.t H.
The following lemma shows any-variate attention is capable of performing variate-wise operation on arbitrary number of covariates under any-vari encoding.
Lemma D.2 (Group-Wise Operation via Any-Variate Attention). Let || 12, := (EN 11/2/1)
denote the column-wise (2, pnoom of H. For any input matrix H (h1,'.. .hT) such that
16

---

Published as a conference paper at ICLR 2026
H 2,,00 suppose :()): B RXT is sequence-to-sequeeeee function implemented by
single layer standard transformer (TF)) such that
(H)
Then there exists single layer MOTRAI transformer TFo()) such that for any input
=
H2 Hkl.
where H3 E RXT, TFo()) performs
(F)(H)) = (1))
(H2)
(HK)].
Proof of Lemma D.2. start by showing the case of single-head, single-layer standard transformer. Let
MLPe, Atm ( MLPa, (Ho KH)) HHHH.
where Aи ( (QH, )H))
Let 1/1 (H) H Н. to apply group-wise operation (-)()) on some input such that 1 ( =
4/1 (H2)
1/1 (H)))
Let O E RTxx be zero matrix, and E RTxT
x be Is matrix, for for any input ||** S R. one
can find some u² 0 to decompose (-) into the following form.
2/1 (H*)
V ( H₂AH,
H Ahx
AH
0
A H2
0
V
Ahк-
VH*x x
QH KH QH2 KH
QH,, KH2) QH2, KH2
QH, KHR) QH2 KHR)
u²
n2
u 1
0
122
(QHK, KH (QHK, KH
QQK KHK)
Lu
1
u2
Further, observe that operations an MLP layer are either left multiplication or element-wise operations, which implies group-wir wise as well. We then finish the proof by setting u 0.
Theorem D.3 ((38, Section 5.6)). Suppose 1p [0, +-00 is convex, non-decreasing func- tion satisfying (x + y) 2 (bxx)(()))) (V)). For any random variable X, we consider the Orlicz norm induced by vp ||||| inf 0 (XX]KK) S 1. Suppose that {Xo} is a zero-mean random process indexed by such that [| - X S p(0.0) 0') for some metric p on θ. Then the following holds
Xxp - Xorl (8(++))) <
1
P sup 0.
((/DD)' for allt 2 0,
where D is the diameter the metric space (Q,p), p), and the generalized Dudley entropy integral J is given by
J J= J + (N(6,
where N(8; O.p) p) is the d-covering number of (O.
The next technical lemma is in (5). Let ( = [-R, R| denotes the standard Eo ball in IR* with radius R. 0.
17

---

Published as a conference paper at ICLR 2026
Definition D.4 (Sufficiently smooth v-variable function). We say function g: RR + R (R, Ct)-
smooth for = [[k 1)/2 + 2, 9 is a C frccionn on B (R). and
sup
(=) = sup
sup
O 2 S Li
ZEB (R)
ZEB (R) -1 J.E
for all i = 0, with maxogiss R S Ce.
Lemma D.5 (Approximating smooth k-variable functions). For any Eapprox > 0, 1 1,C, > 0, we have the following: Any (R, -mmoohhtth function g 12t + R is (Eapproxs R. M, -ppprooximall by sum of relus with M S C( log(1 + Ce/ and C & C(k)C& where C(k) > 0 is constant that depends only on k, ie..
f(z)
Σ Cm ( (amlz:1))
with Σ
1mml S C,
max ||am 11 S. mE[M]
such that sup (uPze[-R.Fi) 1f(z) g(2)] Eapprex-
18

---

Published as a conference paper at ICLR 2026
E PROOFS
E. PROOF OF LEMMA 3.2
Here we prove slightly simpler result with the positional encoding containing only zero vectors and one-hot vector. One can easily extend the proof by padding the weight matrices.
H
F1 Pi
T2
O PT+1
E EDDITHD
O0 e,
E RCT
(E.I)
P2
PT
Lemma E.1 (Lemma 3.2 Restate). Given a sequence of token H in the form of Equation E.1, there exists one-layer, 1 head ReLU attention layer, such that the columns of Attno H has the following form:
T2
1i-1
0.'-9 1 < T 1}
Attnle (H);
=
where p', =
RR-
(E.2)
24 2 pt
Proof. Consider an input of the following form
T1
J22
TT
0
0
0
G1
C2
er
where x, Rd P. RR, for all as following,
1,
T. We construct weights of the m-th head W Wa
T
0
0 T
O
0
e2
0
K
W
eq
where we define the negative index as rotational index, i.e. e-1 eT, e cr-1-
We have
e1
e
(WX')*
(W
eT
Teim
e2
-IT
er
Note that the result of (WWX)) (W is rotation matrix, where right multiplication X will rotate the columns of X. Therefore, we have W that performs row-wise shifting and the attention matrix ((Wx)) (Wxx)) performs column-wise shifting.
E.2 PROOF OF THEOREM 3.6
Autoregressive Linear Regression under Any- Anr-Variate Encoding, The ultimate goal of this setup to perform the following mechanism. Let æ be the target variate we wish to predict, z be the j-th covariate of for j E [M]. We denote the lookback window size q. and each covariate has length (T -ime steps.). We denote the time encoding as p; for i E [T], and the variate encoding as q, for j E [M]. Finally, goal is to predict 27--
19

---

Published as a conference paper at ICLR 2026
A (4) A2(1)
x Pt
BT
x p1
1
PT
pr
pl
PT
E1
E1
2/2
62
ed
Ed
Aa(9)
Here, different colors represent different covariates. The motivation for performing such operation to apply the in-context learning property of transformers proved in (5).
Lemma E.2 (Lemma 3.4 Restate). Define the matrix (4) for the i-th covariates (a), . xT)), with order q, such that
of
22
xt
x441
4
TT
T-
x
It
F1+1
A:(g)=
F--1
TT--
x2-2
x1-1
It.
FT
Do T--+1
o 8-9++
It--+2
where in the j-th column of A.(g)), it contains historical values of z) with lag q.
Given fixed D,T e N+. where T >> q. For any input matrix H in the form of Any-Variate Encoding in Equation 3.4, such that H R xTT" and D' D, T < T. There exists I-layer, q head Any-Variate Variate Attention that performs the following operation.
x7
x
T
A1
A2(g)
Ad(g)
P1
Pr
p1
PT
p
PT
+
E1
e1
62
62
ed
ea
Proof. The proof comes as a direct corollary of Lemma D.2 and (5, Proposition A.5). By Lemma 3.2, there exists a single layer standard transformer layer (with W1, W2 being 0s) that generates (. for each univariate (covariate). It then left applying Lemma D.2 for variate-wise operation and applying (5, Proposition A.5) to keep the time indices p: unchanged.
Corollary E.3. There exists dmaa head standard attention layer that performs the following
(g)
A2(g)
(4
A2(g)
Aa(g)
for any d S duaxx
ia(a)
where Ā.(g) is Ag(g) without the first row.
Proof. Note that this operation in Corollary E.3 is straightforward with Lemma 3.2 and (5, Proposi- tion A.5). As for each [d]. a 1, the attention layer performs two operations to each element of Ag)::
iT columns to the left Umax rows below zero out
right multiplication
left multiplication
if in first row (left multiplication)
Note that one can simply construct weight matrices to perform the above permutations and masking. In total, need dmax heads perform such operations for each As(g), for any d < dmax- For
20

---

Published as a conference paper at ICLR 2026
q Jmaxx the remaining entries will be zero padded. Finally, with at best 2 layers of dmax head any-variate attention, we then obtain
3 TT-2
Ag)
(g)
(()
RT
Ad(g)
p
d
1T
IT
PT-1 PT
el
el
where p is the matrix of (P PPr, e is the matrix of (ei.".,, ,1).
Note that ot in red is the target we wish to predict (masked as 0) initially), and the entries in blue is considered the input feature of our AR model (a linear regression model in this case), and we are able to directly apply several theoretical results in (5) with input H Specifically, for Theorem 3.6, it follows directly from (5, Theorem by setting À
Next, present several approximation results from (5), which our approximation results follows immediately from. Considering the general form of autoregressive data: æ E Rdxx' == (æi...., Dr).
where H, (zt,-. ... 21)) E. Assuming our target (variate of interest) is in dimension assume the autoregressive process generates z as follows:
z f( + Et.
(E.3)
where & ~ N(O, of E R', and f is function of interest. We then present several results when varies.
Non-Lineea AR. Here we analyze that when the autoregressive process is generated by a 2 layer ReLU network with look back window size q. Suppose the prediction function pred (x, u))
Ek= uur œ) is given by a two-layer neural network, parameterized by w = [wxx uk]aE[K] E R(d+1), Consider the ERM problem:
min (u) wEW
1
Σ ((pred(2; w) vi)
1
upr( (u 2..), x+
2N
2N
where (8) is a bounded domain and T; E R is flatten version of Ft-qitt E Rd Proposition E.4. Fix any Bw, B, 0. L 3, O, and O. Suppose that
1. Both the activation function r and the loss function is -mmmooth.
2. W is closed domain such that W C {w [owi ua]kETK E kdd(d+1); 1|uxll2 < Bo, luxl S Bu). and Proje MLPo, for some MLP layer with hidden dimension D and ||02] op I Cw.
Then there exists (L1 + (-222))laaee -layer MOIRAI transformer with
(1) <0(5=
D(E) S 5Ō(= + Dan
max 11(12+1,2112]
max
L
|1011,p op < 0(1 + + C
Σ
M = dmax + Imax
E=1
where we hide the constants K, Bz, Bu, Bu, C2, satisfies the following
② w 2 < - nL,))e,
where Ly = supwew v
21

---

Published as a conference paper at ICLR 2026
Maximum Likelihood Estimation (Gaussian) via Transformers. The next result shows that MOIRAI transformers are also capable of performing maximum likelihood estimation on any in- put multi-variate time series. Given data generated by some AR,(g) process with parameter (wi,". "* wgg) C IR:: (x)1 ... ,r) C Rd the conditional likelihood f(-) of observing æ; is
(-(x:
(w,, Σi (w;; Tr-))
faz T1-1," Lt-g)
2 exp
20²
The goal is to estimate the mean vector (u01,'.. wg) and the variance o² by minimizing the negative log-likelihood loss. Note that with 2 d, the loss is strongly convex. The optimization over the NLL Loss has two steps: estimating the mean vector: w, and then derive the variance ō² with the following dosed-form solution:
T
a²
Lt
Σ 1=1
e xi
Theorem E.5. Given a set of input data generated by some ABa(q) process: X, Raxd Y E R", considering the following negative log-likelihodd loss, the goal is to find set of parameters w Rdd 20,022 E IR+ to minimize the following loss
9 Et Σ
LNLL(W, o) == (og((mo²))
12
Σ
W x
20
t=
We denote w", o² as the ERM satisfying the NII. Loss. There exists a (I1 + L2 + 2)-layer MOIRAI Transformer such that its first L1 + L2 layers follow the same requirement in Theorem 3.6, and the last two layers each has two and one heads, it estimates w, with bounded error:
1
w0 w0llss Se.
and the estimated variance is bounded by
6² - x2
S 2EBBS + Big = Ō(e +²),
where E E BB1 + Bu), and hides the values dependent on Bgg Bw
Proof of Theorem E.5.
T
INLL σ) = n log (2TTT +
Σ
Σ ww x
Dt
20
[=1
Following Theorem 3.6, the first L1 + Le layers of MOIRAI obtains w such that the L1 + L2 + 1-th layer takes the following as input
h
(22
[a) xt 1 1x-gii qi:t x -i-iiiii w + = 0;;0;;;;;
where w + E IRad is the flatten mean vectors. For the simplicity of notations, for the i-th column, we denote z1 with yi, and denote [x+ is-gi 22 1a ; 1:4-1 as H, E Rad as they correspond to the label and feature of our AR model, respectively. e E dR satisfies
||||| & E (n..).
Now we start to construct the (I1 + L2 1)-th layer. One can then construct
Q +++ h [0; ~. 0], K -- h = @:; 0], V L+1 h [0; 10:0
Q h 00;
T. 0),
K 2- 10-hhh h = - -0:-w;;0 L+tng h 10:-
The above construction gives
h 1++1
h
σ 100t+11 h KI+ h/)) VL+1h! hb
j=1
1 के eh 0: 1; + (a((w, (w) x.)) o(-(w, x;))) [0: 1:0
I के कb; (10, 3 0: 1;
22

---

Published as a conference paper at ICLR 2026
Next, we construct the last layer as Q [...i Vi (w, :.)
11 h
K n2j [...;;) - (0, 2y);
Vj h 10:1:0]
Finally, the result becomes
h4
[...;
1
(Z,s w) = [..::o;:.....
71
1=1
Thus, we complete the proof.
23

---

Published as a conference paper at ICLR 2026
E.3 PROOF OF THE LIPSCHITZNESS OF ANY- VARIATE TRANSFORMERS
We first show the Lipschitzness of each component in an Any-Variate Transformer. For any p E [1, o0], || ||<|ll2) 1/r denote the column-wise (2. p)-mm of H. For any radius R > 0, we denote Hyg H |2,,0 SR) be the ball of radius under norm 11-112,00'
let 12,p <Σ D=
Lemma E.6. For a single Any-Variale attention layer, 0 {(Vm: m: Qm, Km u u )) ME(N)) we introduce its norm
0ill
ll
max {11@ mE m
11 llap |u /u Σ I lop
For any fixed hidden dimension D', consider
1 = :0::11 - B).
Then for H Hz, 01 11,B the function (01, H) Arme, is (1 + 1) -i-isscchitz w.r.t 01, where = max RB²R² + T + (T 1)d, B(T 1)d), and (1 + B²R²) J-ipsshit w.r.f. H.
Proof. Given some e > 0, some set X and a function class F. If T is L-Lipschitzness, i.e., [[f(z)) f(x2)) - L |||| -22l, for all f E F.
Then, the following holds
((, S X(H1I,
Define
+ aatn.B {0am ||| |mmm |l| B).
The output of the Any-Variata Attention
[h.] is given by
M
N
Tr. = hi +
Σ
Σ a (1Qml Kmlgi) Vinity + un, * U + u * U).
m==
1=1
We also define 6'mm
(V mat
Q' K' m u um MEEMM 7h. as
11
M
N
Ts. = h4 +
σ ((0'm his Kn)ht)
Vinti + u * U + u2 * U).
N
m=
j=1
Now we bound
||Attng,
H
Attmo (H)
(H) 2,00 = -max; hu - hi as follows
M Σ
h4
a (1Qmhr Km) + unn U + u * U) Vnn,-
N
m=1
j=1
((''m hu, K hg) + w U + u U) V hi
M
N
(QQmii Kmhj) + un U + u U) Vin,
N
((O'm his K h2) + uy * U + 21 * U) V m
h
M
h 1bgll or h1 Kmhg) + u * U + u + U) Vm
N
((Q'm his ky) + u/ * U + 21 + U) V
24

---

Published as a conference paper at ICLR 2026
Let
A (Qmly Kmhg) + um U + um U
(Q h2s Km h2) + un U + u U.
By triangle inequality, we have
[A)(V o(B)Vall [A)))) IIV Villlap + +Ilo(A) o(B)llap l
Note that (()) is 1-Lipschitz, we get
(A) - BB) S ||A Bllop
Qmli Kmhg) (Q'n hi hn) + (u/ " + (u -2% op
<
Qmlli Kmby) - (Q'm h4, K ks) + [1( - um) U||
+ |1(u - - * 0
For the first term in the last inequality, have
(Qmli Kmitt) - ('m hi, km)gg) z 11Qm - Qm [| [1 Kmll= + 1| K [|/]]]]]]]]]]]]]] | |m/ll
= B (11Qm Q + Km - Kl.l))
Further, we have
[1(!) - U
luin - ||||| - Tlu w/ll
where T' is the length of each variate (lookback window size).
(u u
0
S J
uM
|0|| S (T u) 2n11
where d the number of variates.
Thus, we have
(A) - o(B) lml
< B BR² (1 Qm - 2. + |m - K'mll) + T (lu)) - whhl + (T - 1)d ((w2 - 2221))
B max B, (T 1)d} (IlQ Q'all + [mm Kmmll |u,, =ull + | 4
Next, we bound
[|)())) - ||A| up < B²R² + (T + (T 1)d),
due to the fact that
||||| < 11Qmhill ||
| U n U||
Overall, the Any-Variate Variate Attention is max m RB
(T 1)d, B(T -ipshhips 0,-
Proof. We start by considering H' [h]] and
h = h4
((Qm Kmhti) + w/. U uz Ú) Vnh...
25

---

Published as a conference paper at ICLR 2026
We then bound
h
h() - (h, n.)11
M
N
lo (Qmt Kmltj) + um U + um Ú) Vnh, - -(Q M, Kmhg) + um U + +mm U) V m h2j
N
==1
[Vml mllla lop |oo (1Qm Knhy) + umn U + um U) h, (1Qmh' Kmhjj + um U n UU h1il22
M
[Vmllop oo for (1, km) + um U + u U) ||,; - n11
N
m=1
+ ((Qmhe k)) + um U + u U) a ((Qm Kn km + un U + n Ú) 1-) -1115ll
+ l ((Qmh; hi, Km h2) + umm
U + in U) - a ((Qm h/, Km 11y) + u U + 2 U) [//11
IV lop 3 QiQllllppp |m lop
R² 1, h2gl12
S R²
H
H
2,00
Where the third inequality comes from the fact that ReLU is I-Lipschitzness, and the fourth and fifth inequality comes from the AM-GM inequality. For more details, refer (5, Section J.2)
Corollary E.7 (Lipschitz Constant of Single Layer Moirai Transformer). For fixed number of heads M and hidden dimension D', we consider
TTF,1,8 = {0 = (0,,02) 02) M heads, hidden dimension D', 1100l S B.
Then for the function TF given by
TF (0, H) + clipa (Atmng (H))), E @TF,I,B H E Hqg
TF is Lipschhh w.E.t @ and By Lipschitz w.E.. H, where Be (1-BB)(++))+BR(L++BB)) and Bи = (1 (1+ Proposition E.8 (Lipschitz Constant of Moirai Transformer). For a fixed number of heads M and hidden dimension D', we consider
1IF.L. {0= -00 (1(1. 00 of M M D', D',11011
S B.
Then for the function TF is (LB 1is--Bay Be) Liipchiit in 0 E OTF,L, for any fixed H.
Proof. For any 0 = (01, H E Hy. and 0' = =0%; 02), we have |TF (Ht) TFe (H)) < ||MLPe, (Autne, (H)) MLPe, (Attng (H))
+
2,00
||LP (Attmo (H)) M (Attng: (H))
2,0c
< (1 + 2B2 | (H) Attno; (H) 2,00 BBR | 02llop
< (1 + (1 1) ||0 fllop + BR 00 02 lop S Ba ||0 0' 0'lppp
where R = R B L max R² + T + (T- - 1)d, B(T 1)d}. The second inequality comes
from the fact ||Attn (H)l < R BdR
Further, for H' E Hz. we have
|[Too(H) TFp H
112,0
< (1 + B²) [|)]|||n ||ttno, (H) Attno, (H')ll < + B²) B²R²) H H' 1,,000
For the multi-layer case, one can simply follow (5, Proposition J.1) to conclude the proof.
26

---

Published as a conference paper at ICLR 2026
E.4 PROOF OF THEOREM 4.4
We first describe steps of our proving technique..
We write the pretraining data generation and objective.
(ii) We show that the objective of pretraining on n time series satisfying Dobrushins condition with length T, is equivalent to pretraining on single time series with length nı' under Dobrushin's 's condition with the same coefficient.
(iii) We bound the complexity of transformers, and apply learning bounds from (7).
Let be meta distribution, and each distribution drawn from p(T) satisfies the Dobrushin's condition with max coefficient as a. We then define the single-pane average loss as
1 T
Σ g.erry = T 1=1
2(0, z1)
Ez-per) [E(0, x)]
Now, we assume our pretraining data is generated by the following
1. Sample n distributions from π i.i.d. to get p for j = ,,
2. For each distribution (T), we sample (Zj,1, ,
Assumption E.9, We assume that for each E [m), (zj,) has marginals equal to some distribution D for 1 ,. T.
We first present several lemma and theorems that will be used later.
Lemma E.10 ((88, Example 5.8)). Given any well-defined norm I1-11'- Let be the Rd unit-ball in 11-11,, i.e. B {0 e Rd ||0||' s 1}. we have
log (6, IB, 11-11") S dlog 1+ (
2
Theorem E.11 ((7, Theorem 5.3)). Given a function class F, such that S. B, for all f J. Let p(T) be distribution over some domain Z(T) assuming Assumption E.9 holds and Oog. (p(T)) 1/2. Then for all I. >> O,
T Σ
f(2) - - Ex[f( > >c >c(em)(3)+
Bt VT )
S 21-2
PROPeT)
sup fEJ T
for some universal constant whenever 1/2 croo (llog((22) (P(2)) is bounded away from zero.
The following theorem is from (7; 17).
Theorem E.12. Let (T) be a distribution satisfying the Dobrushin's 's condition with coefficient
a(P!")). Let (Zi...., 2T) p(T), and let f (T)) + IR be real-valued function with the following bounded difference property, with parameters ' .....,TT
AT 2
1f(z) - ((z))f(z))1+ < Σ 12zzziA
Then for all t O,
(1 (1- a)t
P(f((z) - -[((()))((())]
exx
23,, X2
The following corollary directly follows from the above result
Corollary F.1 Following Theorem 12, les
1 T
((z) = T
((z)
1=1
27

---

Published as a conference paper at ICLR 2026
where 0 S ((z)) S B for all 1, T and all 2s. Then the variance of E(:) is bounded by
[((=) {=2)\ < B.
Then, the following holds
-(1 (1-aa)
P((((z) E[((z))1 2 1) $ 2xx
2 2 B
Direct Application of Theorem E.11. By Theorem E.11, if Assumption E.9 holds, with probability over /22 ///,/,. for any E 6,, Olgg (P) < 1/2 we have
s S (e(O)) + 2
sup og
Yo,eer)
VT
where ((O) denotes the function class of E(O,) for all 0 E 9, and C 0 is an universal constant. Note that the above bound presents the naive learning bound for learning a single time series, which direct result from (7).
Here we show properties of a time series generated by concatenating n series under Dobrushin's condition with bounded coefficient.
Lemma E.14 (Dobrushin submuliiplicativity). Let P, P,Q be Markov kernels on measurable space (X.F) with Dobrushin coefficients
P(P) sup P(x,,) P(x', IllVV, xxz²
a(Q) == sup 1Q(x, Q(x', IllTv.
2,
Then (PQQ) S ((P)
Proof. For any probability measures μ4, V on X,
||PQ uPQ = PaMP-PPQQlmn S uPlly S aa(Q) l| u/|rv. Taking supp,y Dirac measures yields a (PQ) S (P)a(QQ)
Lemma E.15 (Concatenation preserves Dobrushin). Let (P:]T- be kernels with a(P)) s te For the T-step kernel Pı. = P.P....PP Pr,
T
a(PI.T) S
II 8t.
In particular, if S < for all then (PPIT) SRT.
Proof. Apply the previous lemma repeatedly:
((Pi.)) s (P.) <
II Kt
Corollary E.16 (Concatenating Dobrushin blocks). Let (P!
jorj 1,...,,,,. be kernels for < (1) < 1. For the concatenated nsT-step chain
n blocks, and suppose a p!()
( 2)
p
block block
block
its per-step coefficients are bounded by N. maxja Kit
())
< 1, hence the entire n -se- kernel has
IIII j=1t=
p(d
! S K
j=
If all blocks share a common bound K < then the concatenated process is Dobrushin with the same per-step bound and nT -sep contraction & TT
28

---

Published as a conference paper at ICLR 2026
Proof. A direct result after applying Lemma E.15.
Remark E.17. A "reset" step (next state independent of current) has a and only improves the bounds. The above results tell us that by concatenating independent length T Dobrushin time series, we end up having a length n'I' Dobrushin time series.
Concatenation with resets. For block j E [m], let (P!)))T, be the Markov kernels and let p()) be its prescribed initial law. Define the reset kernel R (3r, n1ا) (.). The concatenated -tTp chain has kernels
R(),
P
P
R(2)
p(n)
block
block 2
block
Lemma E.18. For any probability measure the reset kernel R(x, = uu.)) satisfies a(R) = Consequentl, if (a(p()) S kit has
(5)
for all blocks and steps, then the concatenated chain with resets
T
II
p ... ...PPPP) (()))
S
IIII 1=1==
Lemma E.19 (Reduction a single chain of length nT), Let P(nI) be the concatenation with resels.
Then: (a) Mlg (Pa) S ā < for all 8 S nT; (b) the marginal at every time under P(xT) equals D; (c) for every 0,
nf
E ((nT)
E(O, z1) 1 n j=1
Σ
E
(T) T
E
e(0, 23,
nT
t=1
Proof. (a) Direct result from the submultiplicative property of Dobrushin series; resets satisfy
aog (()) 0, hence sup, olog Olog(Ps) S ã. (b) By DP D and RU) having output D, induction over time yields marginal D at all positions. (c) Linearity and the blocking ( (j 11T 8. O
Lemma F.20.
T
T e(O, 23,ر)l
1
X8 =
21,)
E Bptr
Σ
T
T
j=1
1=1
and
nT
1 xf
Yo
E(0, 21)
E Epiiry nT
Σ
E(0, 21)]
nT
Then {Xo)o and (Yoto have the same expectation, i.e.
nT
Σ
g(2)))
1
Σ
E (T) Σ 9(
nt
72 j=1
Proof of Lemma E.20. For all 2, x', R(x,-) = R(x',) = μr hence |[(x, :) (( = 0, so a (R) = 0. Submultiplicatiiity gives (<L)) S a(K) (LK))(())))))) and a a(R()) = 0. Use lincarity for the expectation identity.
Remark F.21 (Equality in law of objectives). Under Lemma E.20, for each fixe 0, Xo and Ya have the same mean (0) and the same Lipschitz modulus in the sample coordimates. In particular, it suffices to bound sup, [oo
Lemma E.22 (Increment control in 0). For any (21,..., 2xT)
nT fo(z))
nf
Σ for (z)) & Le p(0,0')
nf
nf
Hence Yo Yor is subgaussian with variance proxy
1.2 000 [1-x.)ni nT.
29

---

Published as a conference paper at ICLR 2026
Proof. Direct application of He et al. (15, Theorem 2.3).
Lemma E.23 (Suprema coincide under concatenation). Let Xa and Yo be
T
Xo =
[(0, 20,1) - B +
Σ
E(0,
23)))
f=1
1=1
nT
Yo
((0, zx) Pppnnr
WT Σ ((0, =t)
nT
where (z1..... ZxT) the concatenation (with resets) of the blocks ((3g);-.., Then for every realization of the data and for every 0,
Xo Yo:
hence
sup 1-
Xol
sup |o)-
PEN
Proof. Write
X nT
1
E(0, 2j,0)
1 n
B p T
Σ
e(o 23.2
By construction (21,.... 2nr) is just the stacked sequence, so
1 ΣnT E(0, 21)
#T
Σt 8(0, Zz,e). Expectation alignment (concatenation with resets) gives
E Bptarr
Σ
((0, 21)
E
E Σ e(0, 21.0
Substitute these two equalities into the definition of Y, to obtain Xe Yo pointwise, hence the equality of suprema.
Proof of Theurem 4.4. Recall from Lemma E.23, sufficient to bound sup, Yo
Now, to take supremum over Yg. we get
sup AE0
X9
sup 0E 7
0
sup (7)
n1 j= BEB
To upper bound sup Yol. we take a similar approach to (5, Proposition A.4).
Assuming the index set is equipped with a distance metric p and diameter D. We assume that for any ball @ of radius in O, there exists some constant C₁ such that the covering number admits upper bound
log N(,, O', .'.p) S dlog(2Ar/6)),
for all 0 < 8 2r.
Now we select Ou such that it is (Do/2) of O. The above assumption guarantees us that can have Op such that log 14,l x dllog(2AD/ D.)) By Corollary E.13, Yo is 212/1-a)- subgaussian (a a(p(nT))). Then, with probability at least 8/2,
2B
sup e
Yol < C
(1 a) /llog(2D//D)) log(2/8).
Note that the uniform bound for independent subgaussian random variables still applies here as for each 0, re-sampling chain from new distribution sampled from
30

---

Published as a conference paper at ICLR 2026
Assume that Ou {01.-.. }x Now for each [m]], consider 3 is the ball centered at f, of radius Do in p). With Theorem D.3, for each process Neo,, then
B'
+ = 222
Yo Yor
VnI (0.
where xx)x)ss is B² p(0, 0*) 00:0000'ssssuuuu subgausssin random variable.
We then get
Yorl (V
dloo((2)) nT +1 :11) - 2exp(-nT(t), for all / 2 0.
P
sup 0,0
If further take & S /log(2m/0)) ö)/Tn, then with probability least -8/2, it holds that for all
j E [m].
2d ld log(4/8)
Yo 1 Talls'i SC'' Do
sup c0,
nT
By chaining, we have
Yol S Yo,1 + 1 For1-
Hence with probability at least holds that
2B Y Yo,1 s Cn (1 (1-a) +B'
dlog(2AD/ Do) + (2/) nT
sup Yol S sup [ sup aee,
AEG
AE
Next by taking Do D/K, x= = B'D
(1-a) 2B
get
2B Vol S C"I 150
d dlog(2AN) + (2/8)
sup
(1-a)
B' DR)
nT
Last, check whether the assumptions make above hold for our function class Le Below, slightly abuse our notation by using D as the dimension for weight matrices in TFp. By Lemma E.10, holds that
log (5, BIHHI, (r), (1-1pp s L(3MD2 + DD' log(11 + 2r/0),
where BiH- (r) is a ball of radius r under norm 11-1lap
check that
Ba) ||e(0, 2) ((0', 2)]l I BL
0' 1lpp
where it direct result from Proposition E.8. By plugging all the parameters, we get
B2
L(3MD2 DD' + 2) + 1og(2/8)
sup Y S C(
(1 - \
IEA
n.T
where log(2 + 2(LB -
Be) B 1-0
By plugging the ERM 0, we get
(( inf L(0) 2 sup Vol.
8
Finally, apply Lemma E.15, we have < a.
31

---

Published as a conference paper at ICLR 2026
E.5 ANALYSIS OF COROLLARY 4.6
Definition E.24 (Markov Random Field (MRF) with pairwise potentials). The random vector 2 = La) over Zd is MRF with pariwise patentials if there exist functions 1 Z IR and Pij = + for # jjE {l....,d ,d} such that for all E Z"
d
P I =
(Z)))
II
Igisisd
The functions 1/5 are called as element-wise potentials and diş are pairwise potentials. Definition E.25. Given an MRF Z with potentials {ppt) and {Vigh. we define
(L)(Z) == sup ZzZZjeZ
Ipii (Z,, 2i]l: 8(2) = max Σ Boy(Pd).
13isd
JØL
Lemma E.26. Given an MRF z with pairwise potentials, for any i + j. (z-i(2) s Byz(z).
1 [-) (Z) (Z)
Lemma E.26 implies that to satisfy the condition a.)5(.) < 1/22, it is sufficient to show that B(-) < 1/2, leading to the following condition.
1 + (o.))
(war,, 2(+1) In 2
(E.4)
Observe that
(wr,, 2e+1) S ||ma|| max ||ar|||
B Brr
2 + (a.) 0.3.
E.5.1 FURTHER ANALYSIS ON DOBRUSHIN'S CONDITION ON COMMON MODELS Here we derive the conditions for (1) ARMA and (2) VAR (with certain stability) to satisfy the Dobrushin's condition.
ARMA. Consider the scalar ARMA(p, q) model defined as
F1
Σ OETI +
Σ
Et a~~1 n(0,00
1=1
Similar to the above derivation, we assume
[oal < Bu,
Vt.
Its joint log-density Gaussian MRF is
T ((LTT)
p(Z1:T & exp
Σ
= exp
-
20 -
20
where J is symmetric, banded precision matrix. Decomposing the quadratic form under MRF, have
(I,( 20
Jyy,E i
The pairwise potentials is bounded by
B |din (1t, S |, S Jzyl-
20
20
Therefore,
B² By(X)= supl 24y1 S Jayl.
20
Ii.7,
32

---

Published as a conference paper at ICLR 2026
and
B
B
B(X) max
Σ
(x)
max
J:j
||
c0,0f
20
20²
j#1
j#1
From Lemma E.26, require
Olog (X) S B(X).
Thus a sufficient condition is
B
1 2
J
20,off
2
Further, we use the AR(oo) representation:
U monn++e Exe
with absolutely summable (vm), satisfies the crude bound
|| oo
Hence the sufficient condition is also
1 Oog () < 2
B Σ
whenever
Mm is sufficiently small.
vAR(p). Consider the vector autoregressive model
X1 Σ
AxX t
11(0, Ee),
with X, E Rdd. Assume the usual stability condition: the spectral radius of the companion matrix strictly less than 1, ensuring unique stationary Gaussian process. Throughout this section restrict attention to a bounded domain
1 S B,,
At.
The joint density of the block X1.7 can be written as
T
Σ AnXet11k)
E.- "x²((x011
E A x km
p(X1:r) & exp
X441
t=1
Expanding the quadratic term gives
(X1+1 - Σ Ae ) 2
1 (Xet1 Σ A
k=1
k=1
-X Xx41 2 Ex Xx A + (terms not involving X(41).
km1
The cross-term between Xe41 and X
d Σ
XI A
(3 Ax),
x() x
H+1
141-k)
i=1
==
Hence, in the Gaussian Markov random field representation, the pairwise potential between node (t + 1, and node (t + k,j) is
014110,04115511 kil (()) (1)
( 211-)) cu Ab) (1) 22
33

---

Published as a conference paper at ICLR 2026
where dj is absolute constant (absorbing the factor 2). On the bounded domain ||eel S BR.
(1141,)000++1)))000 ) S SCB2 (E Ax)is
for some absolute constant C. Thus,
((0 CB (E. As) (b)))
For fixed node (t + 1,1), summing over all its neighbors yields
(() CB
E A&t At)
k=1j=1
j=1
Taking the maximum over all (1, i), we obtain
B(X) S C Σ
Axl
k=1
where
M 1c0 = max
May
(row-sum norm).
By Lemma E.26, the log-Dobrushin coefficient satisfies
alog (X) I 3(X).
Therefore, sufficient condition for Olog (X) is
C B E A Axlloc k=1
E. = Ia this simplifies to
B 2
A < Co,
for numerical constant cp-
E.6 ADDITIONAL DETAILS
The History Matrix. The matrix form of Ax(g) is presented below
x 4
T -1
x
zt
Ag)))= = x7-1
---2
x
T
(E.5)
BT -+1+1
r
x
ft-g+2
34

---

Published as a conference paper at ICLR 2026
F EXPERIMENTAL DETAILS
F.I ENVIRONMENT
We mostly train our model NVIDIA-HI00 GPUs with 2 cores each with 128GB RAM. 2 GPUs are sufficient for all of our experiments. We use PyTorch 2.1 and our code is based on the open source published by (40). Training and evaluate takes roughly 12 hours for one run.
F.2 MODEL ARCHITECTURE
For most of our experiments, we use MOIRAI-base model. The hyperparameters are listed in Table 2.
Table 2: Hyperparameters
parameter
values
batch size
64
loss
MSE
initial learning rate learning rate decay
1c-3
cosine annealing
hidden dimension
768
MLP dimension
3072
number of heads training steps sequence length
12
20k
512
optimizer beta (81, (22) weight decay
AdamW (0.9, 0.98)
le-1
warm up steps (linear)
10k
F.3 SYNTHETTCC DATA GENERATION
We generate the AR synthetic data similar to Equation (2.1) but use normalization to stabilize the values. The parameters of synthetic data are in Table 3. Consider sequence of data RdXTT = (I....., Rr), where x, = (xt,... ,I²)) E Rd. Assuming our target (variate of interest) in dimension 1, assume the ARd(g) process generates z! as follows:
2
t- t
(F.I)
where e, ~ 1),, of ~ () E R. After recursively generating the time series, we remove its first 50 time steps burnout. Each AR time series has number of covariates between 1 to 5. For training data, we sampled 100 different time series, each with 20k time steps. For test data, we randomly generate
time series with time step 5k; and evaluate model all time steps. We set q.d S our experiments.
Seasonality. We also conduct experiments on datasets with seasonality information. Specifically, we consider monthly information. After generating multi variate time series with T time steps ØdxT we then add the seasonality information. For each time step f, its seasonal information is
2TT
a sin
ER
where a E R is the amplitude, f N+ is the frequency which is 30 for monthly information. The whole seasonal information will be added to the time series.
F.4 BASELINES
Least Squares Regression. Consider MOIRAI taking an input AR sequence æ E RRxXT" to match our theoretical results (Theorem 3.6), we transform æ into the following input-label pairs
Σ = ((x). ... xg): 11
T ((0g, Bz 1). 1gg2)
35

---

Published as a conference paper at ICLR 2026
Table Parameter of Synthetic Data
parameter
values
lag size
{1, 2, 3, 4, 5) unif(0.1, 1,
variance
length (T)
20k
number of covariates (d)
{1, 2, 3, 4, 5}
amplitude
unif 1.5)
frequency
30
After fitting least squares on this transformed dataset with samples, it predicts the T + 1-th time step with the following input
Bies = (#T- (0t---+1;"" FT)
For least squares, we use learning rate as 0.1, and perform full gradient descent with 50, 100 iterations.
F.5 ADDITIONAL EXPERIMENTS
Seasonality Data. Here we present the experimental results on training transformers on seasonality
data. The data generation is the same described above. We use the same setup for seasonality data, where our training data comes from time series with E (1, 3, 4, 5), and g = {1, 2, 3, 3,4, 5}. The evaluation results seasonality data is in Figure 2. We observe that transformers of inferring data with seasonality. Note that transformers are capable achieving nearly optimal performance, while least squares regression fails, indicating that transformers capable of fitting more complicated model than AR on given time series.
capable
Seasonality Data (d=5, q=5)
Model
1.8
MOIRAI
(iter=100)
1.6
LS (iter=50)
MSE
1.4
1.2
1.0
50
100
150
Lookback Size
Figure 2: We observe that when least squares regression fails to obtain the optimal error rate for prediction, transformers capable of having their MSE converge towards the lookback size increases. This indicates that these models are capable of fitting a more complex model other than linear regression given time series.
F.6 EVALUATION ON WORLD DATASETS.
Here we conduct similar experiment to Section 5 to real-world datasets. We first pretrain MOIRAI transforner to simulate AR regression, and then we test it on real-world datasets to see if it can handle real-world dataset in principled way. Specifically, we use the same checkpoint of transformers in Section 5, and we pick two common datasets: ETTh1, ETTml and II.I test transformers. Finally, also include another statistical model: VARIMA another bascline. Note that the context length here for AR models are not their lookback size, but related to their training data size as we use these AR models under the same ICL setting of MOIRAL While for real-world datasets, we can see the performance indeed got worse. The results are in Figure 3.
AR(92), common lookback size
We compare the pretrained transformer with AR models with different lag size and VARIAM model. Each point in the figure is the average over runs, the variance small thus we omit for better visualization.
36

---

Published as a conference paper at ICLR 2026
Figure 3: Comparison between pretrained transformer and AR models. While different lag size shows different performance, we are able observe that transformers can indeed outperform all other AR models. Indicating transformers predictive power might be more than just simulating the AR algorithm.
Datasets. ETTh and ETTml are datasets recording the electricity usage through out time. ETThl records the usage every hour, and ETTml records the usage every 15 minutes. The two datasets are in the same domain, but with different frequency. We test all models the whole dataset. IIII is Influenza Activity time series dataset of different states, is it sampled weekly.
F.7 ABLATION STUDY ON ATTENTION BIAS
our theory, the any-variate attention bias plays an vital role in handling arbitrary number of covariates. Here conduct ablation study without the matrix. We train MOIRAI on a mixed different synthetic AR data. Next, we evaluate the trained MOIRAI some other synthetic AR data with unseen q, d. We pretrain MOIRA Al E [5, 10) E [6, 10). The results are visualized in Figure 4.
whether the model performance would change
q=10 10 d=5
q=2 d=15
3.5
3.0
MOIRAI
MOIRAI MOIRAI (w/o bias)
MOIRAI (w/o bias)
3.0
MSE loss
MSE loss
2.5
2.5
2.0
2.0
1.5
200
400
600
200
400 CTX Length
600
CTX Length
Figure 4: We observe that when using MOIRAI with the U matrix, the performance has obvious gain when evaluated on unseen values of g, d. Specifically, when 15, which is explicitly shown to MOIRAI (comparing g is latent variable), the performance gain is more clear.
37


Reviews:
[{"review_id": "nbcBS2e7L5", "reviewer": "Reviewer_TXYd", "summary": "This paper provides a theoretical analysis of how transformers can perform well for time series forecasting, focusing particularly on the set of time series modeled by AR processes. First, the authors show that transformers with linear attention can approximate AR regression on univariate, as well as multivariate case (based on the any-variate attention in MOIRAI). Second, the authors use Dobrushin's condition to show a generalization bound for why pretraining on diverse datasets works. Finally, the authors ran experiments on synethetic AR data by first pretraining on generated AR data, and then compare the performances on in-distribution AR processes and out-of-distribution AR processes against least squares regression.", "questions": "1. I don’t think the setting in Appendix F6 EVALUATION ON REAL-WORLD DATASETS is appropriate. The window size seems to be too small for ETTm1 and ETTh1 to cover a whole period of time series. This might be the reason that MSE increases along with window size in Figure 3 left, which is counter-intuitive.\n2. Line 80: \"We impose periodic boundary conditions for the negative index, i.e., x−1 = xT.\" Why do we need this? It seems unconventional.\n3. A recent paper \"Why Do Transformers Fail to Forecast Time Series In-Context?\" (https://arxiv.org/pdf/2510.09776) argues that a simple linear model has direct access to the full history of the time series, while an LSA-based model must compress this entire history into the fixed-dimensions in the Q/K/V matrices. Based on this observation, they show that for any AR(p) process, an optimally parameterized LSA model cannot achieve a lower expected MSE than the classical optimal linear predictor. Can you compare your analysis with theirs, and share your thoughts on why your findings are different?\n4. Section 4 is a bit difficult to follow. Consider adding/moving more insights and intuitions to somewhere earlier in the paper.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is mostly easy to follow. It extends the ideas in Bai et al. to show that a transformer can approximate any AR model, and also shows the generalization ability of time series pretraining. It also conducts a few controlled experiments to validate the findings.", "weaknesses": "1. Writing. I can follow the paper, but there are quite a few typos here and there, and even in the title “Time Seires” should be “Time Series”,\n- I think the following description of iTransformer might be a bit misleading: “iTransformer (Liu et al., 2023) propose to use a pooling technique to reshape arbitrary number of covariates into a unified size.”\n- Line 298-299: “.” instead of “,”. For each time series, we encode it with any-variate encoding into an input matrix denoted asH ∈ RD×N , 1 We define each pretraining sample as zj := (Hj , yj ), where yj = x1T j . \n- This seems to be an overstatement. I believe there are other transformer models, or TSFMs that can handle an arbitrary number of covariates. “Note that in the multi-variate case, we only focus on MOIRAI as it is the only transformer-based model that is compatible with arbitrary number of covariates.”\n- Move Figure 1 closer to the experiment section.\n- Line 143, w* = (w1, ..., wj ) \\in R^{qd} should be  (w1, ..., wd )?\n2. The proof on approximating any AR model with a transformer seems to be a straightforward application of the proof in Bai et al., as the main difference is the construction of the feature and label pairs. However, I am not very familiar with related literatures. The authors can correct me if I am wrong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N4OP8DTsy3", "reviewer": "Reviewer_Fmb5", "summary": "This work provides a clean theory for why transformer-style time series foundation models, by focusing on the MOIRAI work. They're showing a transformer can learn to perform least-squares autoregression on univariate series via in-context gradient descent. The authors then prove MOIRAI's any-variate encoding and any-variate attention let the model infer an AR predictor with an arbitrary number of covariates, explaining its strong zero-shot and few-shot behavior across domains.", "questions": "I'd encourage the authors to check the Weaknesses part first, and here I have two more questions.\n\n(1) Your theory assumes d≤dmax and q≤qmax with qmax·dmax tied to the hidden size D. How should we expect the model to behave when test-time d or q exceed these limits, and can we anticipate meaningful degradation vs sharp failure from the construction? Would be good to know the answer to it\n\n(2) Any-variate attention uses a block matrix U with fixed block size T, so I was wondering how the framework handles irregular sampling or missing timestamps across covariates where a clean block structure does not exist and what time encoding is implied in that case? \n\nThanks again for your hard work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I found the originality, quality, clarity and significance of the work very well. The paper gives a fresh, concrete theory for TS transformers by showing a transformer can do least-squares AR via in-context gradient descent and by explaining how MOIRAI’s any-variate attention builds per-variate lag histories to support multivariate AR, extending prior ICL views to time series with unknown lags. MOIRAI is a perfect paper in my view, and I am happy that a team is working on it. The technical core is strong, and the theory is then stress-tested on real data. The paper also defines the attention variants and the MOIRAI transformer cleanly and keeps the flow from setup easy to follow. Finally, the authors unify why MOIRAI-style FMs work in forecasting, give a generalization story for non-IID series, and show practical gains over tuned AR baselines on ETTh1/ETTm1, so the results matter both for theory and for building stronger pretrained TS models.", "weaknesses": "While I really enjoyed reading the methodology and approaches that the authors followed, I'd like to raise some weaknesses that came to my mind, and I'm looking forward to a nice discussion on these with the authors. \n\n(1) From my understanding, the theory and constructions use a ReLU-based attention, and not softmax, so I'm curious to know if the authors are willing to either extend the proofs or add tests showing the key claims still hold with softmax MOIRAI. \n\n(2) Another point is that the pretraining bounds hinge on Dobrushin's condition, so I was wondering if the authors could report a practical check for this on ETTh1/ETTm1 or at least share their thoughts on it.\n\n(3) Additionally, I found the real-data study a bit narrow because only ETTh1/ETTm1 and comparisons only to AT models were discussed, so maybe adding another domain and strong time series foundation model baselines would make the empirical case much stronger. Do the authors have any thoughts on it? Would be good to know.\n\n(4) Finally, the loss uses output clipping (ClipBx), and theory/results depend on qmax and dmax, so I was thinking maybe a short sensitivity study and guidance on choosing these would improve usability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B2uOnG0GAd", "reviewer": "Reviewer_AzNh", "summary": "The paper develops a theoretical lens on transformer models for time-series forecasting, with a focus on MOIRAI. It claims three main contributions: (i) existence proofs that transformers can implement (via in context learning -ICL-) least-squares AR regression on univariate series and—via MOIRAI’s “any-variate” mechanism—extend to multivariate AR with an arbitrary number of covariates; (ii) generalization bounds for pretraining under Dobrushin’s condition, yielding error decaying on the order of (1/$\\sqrt{nT}$); and (iii) experiments (synthetic + some real-world) showing prediction error decreases with longer lookbacks, consistent with the theory. The motivation is to replace architectural heuristics with a principled understanding of why transformer time-series foundation models (particularly MOIRAI) work well and handle variable covariate counts.", "questions": "- Can you characterize classes of TS (e.g., ARMA, VAR with certain stability) that satisfy Dobrushin and provide diagnostics to check it in practice?\n\n\n- How sensitive are your AR-via-ICL constructions to positional encoding choices and patching (MOIRAI uses patching by default)? Can you provide ablations?\n\n\n- Is it possible to add stronger empirical baselines and calibration (e.g., CRPS or PIT histograms) on real datasets to complement theory?\n\n\n- Beyond AR, can your constructions cover state-space or seasonal/trend components? Even partial results would broaden scope.\n\n\n- Where does any-variate bias (u₁, u₂) matter most? An ablation isolating those terms would be informative.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "In my understanding, this paper draws a new connection between transformer ICL and AR regression for time series, arguing MOIRAI’s any-variate attention and concatenation scheme enables automatic AR dimensionality selection across variable covariate sets. This is a useful theoretical complement to recent empirical TSFMs. The pretraining generalization analysis under non-IID dependence (via Dobrushin) is also timely, given TSFM pretraining on heterogeneous corpora. The positioning against prior ICL theory (largely fixed-dimensional, next-token settings) and time-series FM work is appropriate; the contribution is incremental on the ICL side but significant for time-series where clear theory is scarce.\n\n- Bridges ICL theory to time-series AR regression; explains MOIRAI’s ability to handle arbitrary covariate counts.\n- Non-IID pretraining generalization bound under Dobrushin is relevant to TSFM.\n- Clear limitations section; transparent about ReLU vs softmax and AR scope.", "weaknesses": "The empirical section aims to validate theoretical trends rather than chase SOTA: synthetic AR data confirms error decreases with longer context and that pretrained MOIRAI adapts across (d,q); there’s a limited real-world section in the appendix. I would like to see stronger baselines beyond LSR (e.g., well-tuned ARIMA/ETS, simple RNN/Temporal-Conv) and ablation isolating any-variate bias terms’ role in practice. Also, reporting statistical variability (multiple seeds) and calibration (since pretraining uses MSE) would strengthen claims.\n\n- Empirical findings can be extended: limited real data, modest baseline coverage, lacking variance/calibration analysis.\n- The Dobrushin assumption is nontrivial; guidance on when it holds in common TS domains would help external validity.\n- Several results rest on formatting assumptions and constructed sequences; practical robustness across typical TS preprocessing is less clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)