Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility
Authors: Annan Yu
Danielle C. Maddix
Boran Han
Xiyuan Zhang
Abdul Fatir Ansari
Oleksandr Shchur
Christos Faloutsos
Andrew Gordon Wilson
Michael W. Mahoney
Bernie Wang
Venue: ICLR 2026 Poster (2026)
Keywords: time series
foundation models
rank structure
attention
embedding
Primary area: learning on time series and dynamical systems

Abstract: Transformers are widely used across data modalities, and yet the principles distilled from text models often transfer imperfectly. In this paper, we analyze Transformers through the lens of rank structure. Our focus is on the time series setting, where the structural properties of the data remarkably differ from those of text or vision. Time-series embeddings, unlike text or vision, exhibit sharply decaying singular spectra: small patch sizes and smooth continuous mappings concentrate the data into low-rank subspaces. From this, we prove that the associated $Q/K/V$ projections admit accurate low-rank approximations, and that attention layers become compressible in proportion to the decay of the embedding spectrum. We introduce the concept of *flow-of-ranks*, a mechanism by which nonlinear mixing across depth inflates the rank, explaining why early layers are most amenable to compression and why rank schedules should grow with depth. Guided by these results, we compress Chronos, a large time series foundation model, achieving a reduction of $65\\%$ in inference time and $81\\%$ in memory without loss of accuracy. These findings provide principled guidance for allocating width, depth, and heads in time series foundation models, and for exploiting their inherent compressibility. Our code is available at https://github.com/amazon-science/tsfm-compression.

Published as a conference paper at ICLR 2026
UNDERSTANDING TRANSFORMERS FOR TIME SERIES: RANK STRUCTURE, FLOW-OF-RANKS, AND COMPRESSIBILITY
Annan Yu, Danielle C. Maddix, Boran Han," Xiyuan Zhang, 2
Abdul Fatir Ansari, Oleksandr Shchur,? Christos Faloutsos,
Andrew Gordon Wilson, Michael W. Mahoney, Yuyang Wang?
1 Center for Applied Mathematics, Cornell University
2 Amazon Web Services Amazon Selling Partner Services
4 Amazon Supply Chain Optimization Technologies
ABSTRACT
Transformers are widely used across data modalities, and yet the principles dis- tilled from text models often transfer imperfectly to models trained to other modalities. In this paper, analyze Transformers through the lens of rank structure. Our focus is on the time series setting, where the structural proper- ties of the data differ remarkably from those of text or vision. We show that time-series embeddings, unlike text or vision, exhibit sharply decaying singu- lar value spectra: small patch sizes and smooth continuous mappings concen- trate the data into low-rank subspaces. From this, we prove that the associated Q/K/V projections admit accurate low-rank approximations, and that attention layers become compressible in proportion to the decay of the embedding spec- trum. We introduce the concept of flow-of-ranks, a phenomenon by which non- linear mixing across depth inflates the rank, explaining why early layers are most amenable to compression and why ranks grow with depth. Guided by these the- oretical and empirical results, we use these insights compress Chronos, large time series foundation model, achieving a reduction of 65% in inference time and 81 % in memory, without loss of accuracy. Our findings provide principled guidance for allocating width, depth, and heads in time series foundation mod- els, and for exploiting their inherent compressibility. Our code available https: / /tle://gitttte /ttptb.tttuu.ggg//g////.gg...... com/ amazon science/ts/ie compression.
INTRODUCTION
Transformers, originally designed for language (Lewis et al., 2020; Achiam et al., 2023), are now widely deployed, e.g., time series (Ansari et al., 2024a; Das et al., 2024; Shi et al., 2025; Wolff et al., 2025), images (Liu et al., 2021; Dosovitskiy et al., 2020), molecules (Maziarka et al., 2020; Leon et al., 2024), and DNA sequences (Ji et al., 2021: Le et al., 2021: Nguyen et al., 2024). A common approach to apply Transformers to these other data modalities is to directly transfer architectural parameters (e.g., width, heads, depth) from text-based models, for text should generalize. However, this assumption is fragile. As an example, we show that time series differ fundamentally from language in how signals are tokenized and embedded. This leads to the more general question: how well do community insights on pretraining and hyperparameter tuning, based largely on Transformers applied to text data, port Transformers applied to other data modalities? Understanding the answer to this question is particularly important when Transformers are applied in domains where data are less abundant than in the text domain.
the assumption that what works
Here, we address this question in the context of time-series data and time-series forecasting, A priori, the answer to this question is not obvious: time series data have similarities with text data (e.g., they have obvious sequential properties), but they also have many differences (e.g., it is not clear that they
Work done during an internship at AWS.
Correspondence Danielle C. Maddix <dmmaddi x@amazon. com?.

---

Published as a conference paper at ICLR 2026
We
W
Wi
Wix
W
W
Fribeddedd Inpul
W
Ougas
W
Gutpu
W
Tiwe
Serics Eribadiing Low
Language
244 13mh
Layer Insul
W
W
W
Laser Outbut
Siane, was ihe vocre od Nupess
W
syens
Apprrcs loction
Figure 1: Overview of our results. We show that the embedded inputs of Transformers trained with time-series data have much lower ranks than those of other modalities, including Vision Transform- ers and Transformers trained with language data (see section 2); we prove that attention matrices on low-rank inputs well-approximated by low-rank matrices (see section 3); and we introduce and demonstrate a concept called ofw-of-ranks, describing how attention matrices in earlier layers are more compressible than those in later layers (see section 4).
are well-modelable by discrete set of tokens). The question, however, is timely: time series data are ubiquitous in many domains, including scientific (Abhishek et al., 2012; Zhang & Gilpin, 2025; Lai et al., 2025), industrial (Hong et al., 2016), and financial applications (Zhang et al., 2001), where they facilitate critical tasks, e.g., forecasting (Hyndman & Athanasopoulos, 2018), imputation (Yoon et al., 2018), and anomaly detection (Blázquez-Gare García et al., 2021). In addition, there has been recent growing interest in developing so-called time series foundation models (TSFMs) (Brown et al., 2020; Radford et al., 2021; Ansari et al., 2024a). These models are large pretrained models, designed with the hope of providing a foundation (Bommasani et al., 2021), to be adaptable to a wide range of domains and time series tasks. As with other models that aim provide such "foundation," " TSFMs are appealing because they reduce the need for task-specific architectures and parameter computation, thereby enabling the transfer to new settings with relatively little effort.
In this paper, we develop a framework for analyzing design decisions in Transformers; see Figure for an overview and Appendix A for related work. Our view is that these decisions are guided by an understanding of the structural properties of the data modality, which are then inherited by properties of the model. Our framework enables detailed linear algebraic analysis of the attention layers within a Transformer, which consist of three linear transformations to form the queries, keys, and values. Our approach is general, and we apply it the time series domain. Lastly, we show that practical tool for compressing TSFMs. We summarize the flow of our
we can use approach paper and our main contributions as follows:
1. Data modality and rank structure. We compare Transformers trained to different data modali- ties through the lens of numerical rank, and we demonstrate that time-series data lead to partic- ularly low-rank structure, at the level of inputs. To the best of our knowledge, our work is the first to directly study which data modality features make Transformer models wpll-approximated via a truncated singular value decomposition (SVD), and why. We also show how standard time-series embeddings preserve low-rank structure in the hidden space, which differs from lorge-vocabulary text and other modalities. (See section 2.)
2. From low-rank inputs to low-rank attention. Next, we provide the first general theoretical results that connect low-rank embeddings low-rank attention matrices, and we make clear how width and the number of heads control the quality of low-rank approximations. While these results apply generally any low-rank embeddings, we illustrate them in the case of time series. In addition, our results are sharp: we show that high-rank embedded inputs, which appear in data modalities such as text and vision, lead to incompressible attention matrices. (See section 3.)
3. Flow-of-ranks. We introduce the "flow-of-ranks" concept to describe how the numerical rank changes across layers in deep Transformer. This extends our earlier analysis from a single attention layer to the setting of a deep Transformer, where nonlinear activations, residual mixing, and normalization gradually increase the rank of a representation. We note that "flow-of-ranke ex explains why early layers are often better approximated by SVD than later ones. (See section 4.)
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of embedding strategies and patch sizes across TSFMs.
Model
Chronos
WaveTooknn
TOTEM
Time-MOE
Chronos-Bolt
TimesFM
Strategy
Quantization
Quantization
Quantization
Continuous
Continuous
Continuous
Patch Size
1
1
1
16
32
4. Compressibility of real-world TSFMs. Finally, we illustrate one application of our insights: compressing real-world TSFM. We demonstrate that the same set of hyperparameters (i.e., width, depth, and number of heads) more severely over- parameterizes TSFMs than it does LLMs. This leads us to develop two complementary compression strategies: compressing pretrained model and pretraining model that is compressed by design. In particular, we show that by compressing Chronos model, we can reduce the inference time by 65.4% and memory usage by 81.4%. at cost of predictive performance. Overall, our results demonstrate and explain why sfate-of-the-art TSFMs are highly compressible, in practice, compared to (language-trained) LLMs of the same size. (See section 5.)
Additional supporting material may be found in the appendices.
2 DATA MODALITY AND RANK STRUCTURE OF EMBEDDING
In this section, we investigate the structure of time-series embeddings and compare them embed- dings from other data modalities. Our goal is to understand, from both theoretical and empirical perspective, why time-series inputs often look low-rank after embedding.
We consider univariate time series. In particular, let x = (Z1...., .T) E R RIXT be a univariate input of length T. Note that, unlike other data modalities, the input is a rank-1 matrix." The first step in a TSFM is to map into high-dimensional sequence via embedding function 4: RRIT - RdxL where d denotes the hidden dimension of the model and L denotes the new sequence length, possibly different from T due to patching. This embedding is typically constructed by applying trainable function Φ _ + gd to disjoint patches of size k. Assuming = T/k is an integer, this gives:
xx) (#(=1....... Fk), ((I F22)
((I(L-())-+)))) 1)- ILk = zT)).
These patch embedding functions Φ fall into two main categories (see Figure 8 in Appendix B):
A quantization- embedding partitions the input space & into disjoint regions, R*
R2. Each region R. is mapped to a unique trainable vector u, E Rd, so that d(x) =i=1 (XER,) This approach is used in several TSFMs, e.g., Chronos (Ansari et al., 2024a), WaveToken (Masserano et al., 2025), and TOTEM (Talukder et al., 2024).
a parameterized function, typically neural network Φ(: 0), to map patches directly. This strategy is also used in many TSFMs, e.g., Chronos-Boll (Ansari et al., 2024b), Moirai (Woo al., 2024), TimesFM (Das et al., 2024), and Time-MoB (Shi al., 2025).
A continuous embedding
Table summarizes the design choices for these prominent TSFMs.
In most TSFMs, the patch size is significantly smaller than the hidden dimension (i.e., k < d), meaning & maps from a low-dimensional space to a higher-dimensional one. Intuitively, if Φb is well-behaved, it should embed the low-dimensional space R* into corresponding low-dimensional submanifold (fR+) C .dd_ To formalize this notion of dimensionality, we use singular values. Let U be a linear operator between some Hilbert spaces and 01 2 2 on 2 0 be its singular values, where 1 < n < co. While the algebraic rank of an object U, i..., the number of its non-zero singular values, is a strict measure, weal-world data is noisy; therefore, we use the more practical concept of numerical rank. For tolerance ε > O, the c-rank of U denotes the number of its singular values that are significant relative to the largest one:
rank. (U) = Ii (U((o) > e]].
In the case of few-variate time series, x becomes a few-rank matrix, making the analysis in the paper directly generalizable by viewing the number of variates another patch dimension. We note that in some time-series applications with a large number of input variahles, the embedded inputs may not be low-rank, and our analysis may not directly apply in such cases.
3

---

Published as a conference paper at ICLR 2026
A low numerical rank implies U is well-approximated by an operator Ü with rank(U) rank (U).
Our central hypothesis is that for large corpus of input patches (x(*) Ji=1- the resulting embedded matrix, via quantization or continuous embedding, U = [d(x(1)
((x(A))] has a low nu- merical rank, which is smaller than the ambient dimension d. To test this hypothesis, we sample thousands of patches from diverse signals (e.g., sinusoids, exponential functions, and white noise), and we compute the singular value decay of their embeddings. For contrast, we perform the same analysis tabular foundation model (TFM) (Mitra) (Zhang & Robinson, 2025) using 1000 syn- thetic tables, a T5 LLM processing text from Dickens' Tale of Two Cities, and a VIT processing 1000 randomly sampled images from CIFAR-10.
SVD of Input Embedding
Embedding of Chronos
Embedding of T5
10%
4000
peguar vector
810
3000
25000
10-2
20h lingiie
2000
15000
10000
10 3
5000
Vima Ma3
10 0
100
200
300
400 500
singular vector (b)
(a)
(c)
Figure 2: (a): Singular values of the embedded input matrices from many different TSFMs, TFM, a ViT, and an L.M.. (b, c): Embedding space of Chronos and a T5 L.M,, respectively, visualized by projecting them onto the leading two singular vectors of the embedding matrix.
As shown in Figure (a), the singular values of TSFM embeddings decay dramatically faster than those from the tabular and language models, which confirms their significantly lower numerical rank. Figure 2 (b, c) visualize the embedding spaces of Chronos (quantized) and T5 (language tokens) by projecting them onto their top two singular vectors. The Chronos embedding, mapping a quantized real line, reveals clear low-dimensional structure, whereas the T5 embedding, likely due to its vocabulary properties, appears far less well-structured.
One may wonder: why it that Chronos-Bolt (k 16) produces lower-rank embedding than Chronos (k = 1)? This seeming surprise arises from their different embedding mechanisms. A quantization- based model like Chronos initially maps adjacent values to random, unstructured vec- tors; it must learn the geometry of the real line during training. In contrast, a continuous embedding like Chronos-Bon uses smooth neural network ( 0). This architectural choice imposes smooth- ness from the start, ensuring that even a randomly initialized model maps the low-dimensional patch space R a low-dimensional submanifold in Ted (see Appendix H).
Although understanding quantization- based embedding requires looking into the training dynam- ics, which beyond the scope of this paper, we theoretically analyze how a continuous embedding preserves low-rank structures. The following theorem formalizes this intuition.
Theorem 1. Given any hidden dimension d >> 1, let Φ= : [-1, 1] + Idd be function that embeds [-1, 1] into Rd. Given L arbitrary points .i...., L sampled from [-1, 1, 1]. define
(x))
(ai)
(1(x)) :
E RR
1-111
V
E 12dx
(x(
Φa(x1)
sa(x1)
Let 31 2 2 ss 0 and σi 2 ... oa 2 be the singular values of the quasimatrix and matrix V, respectively. Then, the following statement holds:
1. If, for some V 0, v 2 and every 1 1 d, we have o, and its derivative through o(") are absolutely continuous on [-1, and 0. is of bounded variation V, then we have
4V o()-VV). J4+15 TV(j -
2V ddL 00 VdL), v+i<j5d-1.
8441 < Tu(j-1-vy)
4

---

Published as a conference paper at ICLR 2026
2. If, for some MM > 0 and every i d. d, has analytic continuation to the Bernstein ellipse of radius > (see Trefethen (2019)), whose infinity norm is no greater than M, then we have
4M dM -(p Vd), 0- p-1 $ P
2M AM -06 0sisd-1.
We refer interested readers to Townsend & Trefethen (2015) for the precise definition of quasima- trix (informally, it is "matrix" which one of the dimensions is discrete as usual but the other is continuous). Theorem 1 guarantees that for univariate patches (k = 1), smooth embedding func- tion yields singular values with guaranteed decay rates: polynomial decay of order v for functions with / continuous derivatives, and exponential decay for analytic functions.
See Appendix B for a proof of Theorem I using classic univariate polynomial approximation tech- niques. Using this result, we can directly explain the low-rank structure observed in models like Time-MoE (see Figure 2 and Corollary 2). While multivariate polynomial approximation results enable us to extend Theorem the size of the polynomial basis up to a fixed degree increases ex- ponentially with k, which makes it less practically relevant for a larger patch size k. Instead, for Chronos-Bote Bolt, where k = 16, we seek an ad hoc result that works when the embedding cb:; () an MIP (see Appendix C for proof of Theorem 2 below).
Theorem 2. Consider the input embedding defined by two-layer residual MLP:
( = W,X W2 (W,X), XER WIER W2ERddd, W; W.
where h denotes the patch size, L denotes the number of patches, dy denotes the hidden-layer dimension in an MLP, d > & denotes the hidden dimension of the Transformer, and w denotes any activation function satisfying that [x)] S [x] for every R. Then, for any >> O, we have
Ki o, (x)) > all Welle|| X||2|| S minfd, (1 + e )k]].
Theorem states that the numerical rank of the continuous embedding is bounded by a quan- tity dependent linearly the patch size k not the much larger ambient dimension d. The term ||2ll2/| W W|X/12 1|2 reflects the natural scaling of 01 ( (X)) in the definition rank.. In practice, we have k < d, meaning that MIP embeds input patch into low-dimensional subspace in R.d.
To illustrate this theorem, we pretrain 6 Chronos-Bai -Bot models of different patch sizes, and we com- pute the singular values of their embeddings. Figure 3 (a) shows that the numerical rank of the embedding increases with the patch size k. We also perform an in-depth analysis, where for each pair of embedded inputs of ()) and (xx()) E IRd, we compute the angle between them as follows: ((((((()) ((x(1))) (x) (Tp(T(xx))) (((llxx(()
"Mllllll((("MMalll(x(())))))) (llllo(xx)))
[0, n/2).
(2)
The larger the 0, the more linearly independent the two vectors are. We illustrate this in Figure 3 (b), where brighter the heatmaps correspond to higher rank matrices. We see that the embedded input, (XX) from Chronos- Bolt, which is subset of Rd IR 768 spans subspace of significantly smaller dimension, i.e., the image of R = R under Φ
3 FROM LOW-RANK INPUTS TO LOW-RANK ATTENTION MATRICES
Let U E Rdxx l. be an input embedded in the hidden space. Recall that in section 2 we showed that for TSFMs, U often has low numerical rank. This immediately implies U can be expressed in a low- rank format: U ~ U2 where U₁ Radd and U2 E RdxL for some d < This yields faster matrix-matrix products with U. but limitation is that this representation requires an expensive rank-revealing matrix factorization (Damle et al., 2024), which adds overhead, particularly during backpropagation. If so, how do we leverage the low-rank structure of TSFM embeddings?
From basic linear algebra, it is known that for a linear operator T Rd + Rd to act on an r- dimensional subspace, one only needs to specify the operator in directions, in which case the operator can then be aell-approximmted by a low-rank matrix T whose rank scales with instead of the full width (Damle et al., 2024; Ipsen & Saibaba, 2024). An attention layer, defined by
U W UWW
Attention( U; We, WK, Wv) = Ww softmax
We, WK. Wv E Rdxd
Va
5

---

Published as a conference paper at ICLR 2026
SVD of Chronos-Bolt Boott's Embedding
Embedding of Sine Waves
Embedding White Noises
10
k=
40
40
K-4
0.8
10-1
80
80
0.4
1
120
50
100
120
50
Random Vectors in 16
Random Vectors in Jt 7d8
n2
1/2
10-3
12
1.2
40
0.0
10-4
80
80
04
0
25
50
75
100
125
J
00
120
50
100
0.0
(a)
(b)
Figure 3: (a): Singular values of the embedded input matrices from Chronos-Boll models pre- trained with different patch sizes k. (b) : Angles between Chronos- Chronos-Bool's embedded vectors in Fd 708 defined in eq. (2), where the patches x(0) are from sinusoidal wave and Gaussian white noises, respectively. We also plot the angles between i.i.d. random Gaussian vectors in R 16 and Rd = IR 768 for comparison.
while nonlinear, contains three linear transformations: namely, the queries, the keys, and the values. In this section, we establish Theorem 3, which supports the low-rank representations of Wq. WK, and Wv v given a low-rank (see Appendix for the proof).
Theorem 3. Let C 0 be a constant. Let
XN] E Rx be given for some N 2 1, x, E Rd, and ||xy||2 S C' for all S j S N. Let Wq. WK, Wy dddd be matrices such that I Wxll2 I Cvd, and ||Wrll2 I C.v. The following two statements hold:
[X]
1. (Attention matrices are compressible on low-rank inputs.) For any d < d such that d+1 = ( I L, there exist Wo, WK. K, Wv E Rdxd with rank (Wq) = rank (x) vank(Wv) d. | Q W xl|2 < [[W Q W k|12, and ||Wrl S ||Wrlla, such that given any matrix U E RdxL for any L 2 1, where each column of U is a column of E, we have that
Attention Wo. WK. Wv) Attention(U,, U; Wo. WK. Wv) so
lo co(Voaat). (3)
where the constant in the O-notation only depends on C.
(Attention matrices are incompressible on high-rank inputs.) The upper bound in eq. (3) is tight up factor of Vd. That is, fix some 1, 2 d and 1 2 σ1 2 02 2.. 2 aa 0. There exist U E Rdx with oy (U) σj for all 1 S j s d, and Wo, W Id such that for any d < d, any orthogonal matrix Wv Rddd and any rank-d matrix Wy Rdxdd have
Attention( U; Wo, WK. K, Wv) Attention (U; Wo. WK, Wy)
1 > 7
(4)
Note that Theorem a purely numerical statement that holds for any low-rank embedding E. applies to time series because our analysis section 2 reveals the low-rank nature of E, but nonetheless works for other modalities provided that E is low-rank. The first statement of Theorem says, at a high level, that if the inputs U come from low-rank vocabulary E, i... du+1 small, then one only needs low-rank attention matrices WQ: WK. Wy on U. Here, it is important that we fix the low-rank embedded space E\t and prove uniform bound eq. (3) that holds for all input U, so that our low-rank approximation Wo: WK. K, Wr is not input-dependent. Since TSFMs have low-rank vocabulary 3 (see section 2), high-rank attention matrices in TSFMs can be approximated by low-rank ones. See Appendix for a numerical experiment. We emphasize that this low-rank property does not depend on the temporal simplicity of a particular time series (e.g., being constant), but rather on the intrinsic low- www-dimensionality of the input embedding space 3 itself, which makes U for any time-series input. For other data modalities than time-series, e.g., TFMs, ViTs, and LLMs (see Figure 2). where has a higher rank, the second statement of Theorem suggests that the attention matrices are less compressible,
will show in section 5.

---

Published as a conference paper at ICLR 2026
To illustrate the concepts in Theorem 3 on TSFMs, take pretrained Chronos models and T5 LLMs of different hidden dimensions d. These two models have the same Transformer size and architecture, and differ only in the pretraining data modality. For each model and fixed > 0, compute the averaged e-rank of all query projection matrices W from the model. Figure 4 (a, b) show that the size of Wa increases, the matrix We stays low-rank in Chronos model, and its rank scales almost linearly with d in a T5 LLM. For Chronos models, where the vocabulary is embedded in a low-rank subspace, low-rank attention matrices suffice capture most information in the inputs. This empirical observation goes beyond Theorem 3, which proves the sufficiency of low-rank W W//K/V, but it does not guarantee their appearance via training. Figure (a) shows that training, while independent from expressiveness, gives rise to low-rank weights (see Appendix G for more insights). This provides motivation for pretraining a compressed model and for compressing a pretrained one (see section 5). In Appendix G, we show an analysis of the distribution of the singular values of attention matrices, in pretrained foundation models and also during training.
Rank of We (Chronos)
Rank of W Q (T5)
Rank of We (Chronos)
2000
c-10
1000
1-10
<=10
1500
2-10-21
800
c-10-22
(Wa)
C-10-25 25
c-10-25
300
600
1000
400
200
200
100
0
256
512
1024
2048
256
512
1024
0
8 rank of input embodding
2 3
32
d
(a)
(b)
(c)
Figure 4: The averaged e-rank of query projection matrices We in pretrained Chronos models and TS LLMs. In (a, b), , we vary the hidden dimension d. The light blue curves are contours of the ratio between the horizontal and the vertical axes in the semilog-x scale. In (c), we fix the hidden
dimension 512 and change the rank of the fixed input embedding E (see Appendix I).
To further corroborate the role of the rank d in Theorem 3, we pretrain Chronos models with fixed vocabulary. We increase the (algebraic) rank of =\t from 1 to 32 (see Appendix I), where we observe that the numerical ranks of attention matrices increase with rank (=) (see Figure 4 (c)).
While our discussion in this section assumes a single head in the attention layer, in Appendix E we extend the result to the multi-head case. From the standpoint of representation complexity, Theo- rem 3 applies equally to multi-head attention, and is independent of the number of heads (see The- orem 5 in Appendix D). In practice, however, we observe that the W W//K/V matrices in pretrained multi-head layers tend to have higher numerical ranks (see Figure 9). This effect can be understood through the lens of numerical linear algebra. via concept called "sketching" (see Appendix E). It helps explain why additional heads can improve robustness and training stability, even though the underlying complexity result is head-agnostic (see Theorem 6 in Appendix E).
FLOW-OF-RANKS, MOVING THROUGH A TRANSFORMER
Our prior discussions in section 3 focused on compressing a single attention layer, but a Transformer in a TSFM has many layers. While we showed that the input into the first attention layer is low- rank (see section 2), one may wonder: what about the inputs into later layer? If you apply linear transformation to low-rank input, then it at most preserves, if not decreases, the rank of the input; however, each layer in a Transformer generally nonlinear. Here, we demonstrate that these nonlinear layers can increase the rank of the input.
We refer to the phenomenon of the increasing rank of the input as it goes deeper into the layers of a model as the flow-of-ranks. Our contribution here is twofold: we quantify this rank growth across layers in a deep Transformer by proving Theorem and then show what that means to Trans- formers applied to time-series data. To see the "flow-of-ranks" in practice, we show the numerical ranks of attention matrices per layer in a Chronos model and T5 LLM in Figure (a, b). The numerical ranks of attention matrices become higher for deeper layers because of the flow-of-ranks (see Figure 5 (c)). For a T5 LLM, only with a large E do we observe an increase in the e-rank, because the g-rank with small e is already saturated (i.e., close to the matrix's 's dimension) to cap-
7

---

Published as a conference paper at ICLR 2026
ture the high-rank vocabulary E. increases the small singular values of a low-rank input matrix. Conceptually, this "flow-of-ranks' can be viewed as a mechanism by which the model lifts a low-dimensional but complex signal into a higher-dimensional space where the underlying autocorrelation becomes simpler. This is analo- gous to the Koopman operator framework, which similarly transforms nonlinear dynamics into higher-dimensional representation with approximately linear evolution.
To formalize this discussion, prove how attention layer
Rank W (Chronos)
Rank We (T5)
SVD of Inputs to Each Layer
600
700
100
500
600
10
400
500
10
300
400
10-3
210-
200
300
10
100
200
2
8 10
8
0
50
100
150
200
layer index
layer index
(a)
(b)
(c)
Figure 5: (a,b): The e-rank of every query projection We in the encoder of Chronos model
and a T5 model of the same size, respectively. (c): Singular values of the input matrix to each Chronos' encoder layer, starting with constant signal (31, (#1,--.,,77 .T)
(0, (0......)) 0)-
Theorem 4. Given positive integers S L, let U RdxL be an input matrix with singular values
1 == 01 2 ... 2 σa > 0. Let D > 1 be the number of layers of the model. Let h be the number of heads and d, be the per-head dimension so that d = h x d2. The following statements hold:
1. Suppose vD 2 h. For every 1 S i < let l
w (
12 I Vdh, and || wwlle l2 be
any attention matrices (see eq. (18) for the notation). For any < d, we have
(o--SH1+
h
o (Z)// (Z) S
min Sisk
VD (6-1))1)))
(5)
where Z U-UY/DD and Y MH-Attention( U; WQ. WK, Wv,
2. The upper bound above is tight. More precisely, given any d, L, N, h, and singular values 1 σ1 2 ... 22220 aa > O, there exist an imput U E Rdx with singular values oI..... Jd, and matrices
given
WQ. WK. and Wv with (w( ll2 1, such that for any 1 < k I d. we have
o (Z) Z N (ON-I)d
+
VD Tw/1
(6)
for every < h such that [h//1] < dr.
See Appendix F for the proof of Theorem 4 and Appendix J for a numerical experiment. The matrix can be interpreted as the output of residual attention layer given input U, where the scaling factor Transformer to stabilize and accelerate training (De Smith, 2020). The upper bound in eq. (5) provides guarantee that the rank of the output cannot be arbitrarily large the input is low-rank. In this bound, there are two terms balancing each other: 0 which increases. Without knowing the precise distribution of the singular values, the j which minimizes eq. (5) cannot be determined. To understand why the upper bound tight, we prove a corollary in a simplified one-layer setting.
1/ 1/vD is commonly used
increases j increases, and ((g-1)/A++1 which decreases
Corollary 1. Using the notations in Theorem 4 and assuming D
have (Z(Z) S ((h)a [#++))((++))++1 In addition, the lower bound eq. (6) satisfies that (Z) 2 O(1)aTu/M for every k S d de. The constants in both O-notations are universal.
The corollary states that when U goes through an attention layer, the 4tth singular value of the output Z can be increased to at most the order of magnitude of the [k/h] 1tth singular value U. The lower bound suggests that this can be achieved by some inputs, which proves the sharpness of the upper bound. Interestingly, this result says that the number of heads h plays an important role in increasing the ranks of input matrix. The term note here is not the O(h) factor multiplied to the upper
8

---

Published as a conference paper at ICLR 2026
Rank of Wq (Moirai)
Rank of We (Waee
Rank of We (VisionTSS)
750
$00
550
(W
350
rank (Wa)
400
(Wo)
350
422
303
300
300
203
250
150
100
layer index
layer index
layer index
(a)
(b)
(b)
Figure 6: Flow-of-ranks is also observed for many TSFMs other than Chronos, including Moirai- 1.0-R-base (Woo et al., 2024), WaveTToken (base) (Masserano al., 2025), and VisionTS (Chen et al., 2025). The hidden dimension d in all three models are 768.
bound but the division by h+1 in the subindex (1++))/(+++++ When h is large and the singular values decay fast, d [[k+1)/(b++ [)]+++]]+++ +) (U) can be significantly larger than oh (U) (see Figure 17).
In Figure 6, we also verify the flow-of-ranks in many TSFMs beyond Chronos. general, we observe that the rank of an attention weight matrix in many TSFMs grows as function of the layer index. The three models compared in Figure 6 also have distinct input space dimensions, with that of VisionTTS significantly larger than those of Moirai and WaveToken, which is also reflected by the overall larger numerical ranks of the weight matrices of VisionTS.
USING THESE INSIGHTS: How TO COMPRESS A TSFM?
In this section, we apply our theoretical framework to compress large TSFMs. We pursue two approaches: first, motivated by the observation that attention matrices in many pretrained TSFMs are already low rank, we apply truncated SVD to each attention matrix of pretrained model. Second, to achieve stronger compression, we design architectures that parameterize attention matrices in low rank from the start and pretrain them from scratch, using a layer-dependent hyperparameter schedule that accounts for the ffow-of-ranks!! We present results on compressing Chronos, and provide more results on another TSFM in Appendix to support the broad applicability of our methods. Unlike LoRA's low-rank fine-tuning updates, factorize our attention weights themselves, yielding inference-time efficiency and enabling pretraining with a compressed backbone.
Compressing a Pretrained Model. In Figure 4, model usually do not have a full numerical rank. That means we can well-approximate a large matrix with low-rank one. More precisely, let W [gdxd be attention matrix and let W Σj=19 be the singular value decomposition (SVD) of W. Then, for a fixed > 0, the
that attention matrices in large pretrained
truncated SVD satisfies relative error bound: W E a,u.v S < < 2|2 (see eq. (1)).
For fixed e > O, we apply the truncated SVD to every attention matrix of the pretrained Chronos model without fine-tuning, which reduces the number of parameters. We compute the WQL and MASE losses (see Ansari et al. (2024a)) and their geometric means relative to the original model. A score below means the reduced model performs better while it performs worse otherwise.
Table 2 shows that we can compress the attention matrices up to around 23.7% 73.7% of the original size without any loss of performance. Reducing the size further takes away key information in attention matrices and results in a rapid performance deterioration, leading us to the next compression method.
Pretraining Compressed Model. Table shows a hard limit of compression: compressing the size to below about 20% makes the performance significantly worse. If one is given enough budget, more robust method to obtain smaller TSFM by pretraining compressed model. We parameter- ize d d attention matrix by a rank-d representation. Driven by "floksof-ranks" from section 4, we choose to let d be layer-dependent. In the ith layer of the model, we use d, d(i)) = [d + 1) where d, > and a 2 0 are two hyperparameters. This design is motivated by the fact that the numerical rank of input to layer increases as the layer index, and need attention matrices of higher ranks to capture them (see Theorem 3). In Figure 7, we see how pretraining a compressed model allows us expand the time-accuracy Pareto frontier of TSFMs, which show that pretrain-

---

Published as a conference paper at ICLR 2026
Table 2: Results of compressing pretrained Chronos or T5 model. We apply truncated SVD with specific which results in model whose attention matrices are compressed to the "Ratio" of the original size. We compare the performance scores relative to the original pretrained model, where "LPPL" stands for the logarithm of the perplexity. Overlap is obtained by selecting the top-10 tokens from the original model and the compressed one, and computing their Jaccard overlap.
WOL inLinnd
Lug-Parp ooly
Chroncs Zero-Shot WQL
Ratio
In-Domain WQL
MASE
MASE
Overlip
1.P11
Overlap
Jeccardyy 12 (Chronos)
Jaccandg30 (73)
0.073
4,400
4.095
3.562
3.435
0.308
3.313
0:227
Saching 5000
64 Eo
0.151
1.901
2.412
1.586
1.576
0.717
2:530
0.301
0.237
1,053
1,6855
1,830
1,011
0.883
1.652
0.345
0.353
1.000
1,024
0.990
0.594
0.379
1.544
0.568
to wwerlan
0.569
1.000
0.952
0.945
0.935
0.997
1.200
0.730
0.755
1:007
1,088
200
1.000
compress-to ratio
ing compressed model is more robust than compressing a pretrained one. In fact, in Table of Appendix K. we show that pretraining a compressed Chronos-Bolt allows us to outperform even traditional local methods on both time and accuracy.
Finhedding From Soratch
Reuse Embeckiing
dg
Size Ratio
In-Domain
Zero-Shot
In-Domain
Zero-Shot
Time
Space
WQL 1 MASE
WQL
MASE
WQL.
MASE
WQL J MASE
0.075
0.186
1.084
0.988
0.966
1.031
1.067
1.084
0.35
0.150
0.358
0.312
1.048
0.982
1.080
1.055
1.025
0.530
0.936
0.969
0:40
0.250
0.494
0.440
1.021
0.949
0.996
1.019
0.999
1,007
0.949
0.943
1.000
1.000
1.000
1,000
1.000
1.000
1.000
1.5
Nsive
1.4
Nalve
1.3
1.2
AutoTheta
1.1
Lag-Llama AutoETS
AutoTheta
1.0
Scasonal Naive
AutoETS
SCUM
MASE
Seasonal Naive
SCUM
Liaia
AutoARIMA
0.8
0.9
Moirai-1.0-R (Base)
AUTOARIMA
Moira (Large)
Moirai-1.08 (Base)
A Moirai-1.0R (Large)
LCc Rocee
licks
Chronos [Small) Chranos-GPT?
Chronos (Base)
0.6
Laca Ricee Restor Midic
Chronos (Small) Chronos
Chronos (Base)
0.7
P Chrones (Large)
Chronos (Large)
10c
10a
10
10
10
10
inference time
inference time
Figure 7: Results of pretraining a compressed Chronos model. We compare the performance scores relative to the original pretrained model. We show prediction losses for both models whose em- bedding matrix is randomly initialized and models whose embedding matrix is inherited from the original pretrained model. The last row is the baseline.
To show the generality of our analysis in this paper, we also pretrain compressed Moirai-1.00 Satraai1..---aee 1-0---bae models. We show the results in Table of Appendix K. It clearly shows the benefit of the flow- of-ranks design: given fixed compression ratio, model with a different reduced rank per layer performs significantly better than one whose rank remains layer-independent.
CONCLUSION
We have developed a data/moddaltyyydeeeddenn framework via the lens of rank structure for analyz- ing the structure of and design decisions for Transformers, and we have applied it to Chronos and Chronos-Bols two popular TSFMs. Our results highlight how properties of the model depend on and interact with properties of the input data, and they lead to concrete principles for the design of models, parameters, and hyperparameters. We illustrate our proof-of -f--iiccippeee results by showing the compressibility of TSFMs in comparison to Transformers trained on text data.
ACKNOWLEDGMENTS
The authors would like to thank Yongjun He, Junming Yin, Dmitry Efimov, and Boris Oreshkin for many fruitful discussions this work.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT.
Code used to produce the experiments in this paper, along with scripts for data process- ing and figure generation, is available at ts tsfm-commpree com-oomressccn.
com/ amazon science/
REFERENCES
Kumar Abhishek, Maheshwari Prasad Singh, Saswata Ghosh, and Abhishek Anand. Weather fore- casting model using artificial neural network. Procedia Technology, 4:311-318, 2012.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (30.xx::3330077444 2023.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language time series. arXiv preprint arXiv:2403.078/1 20244,
Abdul Fatir
Ansari,
Caner Turkmen, Oleksandr Shchur, and Lorenzo Stella. zero-shot forecasting with Chronos-Bolt and Autogluon,
Fast and accurate
2024b.
URL https: //ap::://www../ http://wwaaaaaaaaa.......a...aas amazon www.blogs cowmbloogs//acccinneeeeeeeeeeeee wwwps//aaaaaneeeeennnnnnee learning/ fast hast--ad--ccuratte-erro--------- hrg.a//eee..e............e.eeee.
Ane Blazquez-Garcfa, Angel Conde, Usue Mori, and Jose Lozano. A review outlier/anomaly detection in time series data. ACM computing surveys (CSUR), 54(3):1-33, 2021.
Rishi Bommasani et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 /08. 07258, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecast- ing. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 6989-6997, 2023.
Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Uni- fying sparse and low-rank attention. Advances in Neural Information Processing Systems, 34: 17413-17426, 2021.
Mouxiang Chen, Lefei Shen, Zhuo Li, Xiaoyun Joy Wang, Jianling Sun, and Chenghao Liu. Vi- SionTS: Visual masked autoencoders free-lunch zero-shot time series forecasters. PMLR, 2025.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. International Conferences Leurning Representations, 2021.
Anil Damle, Silke Glas, Alex Townsend, and Annan Yu. How reveal the rank of matrix? arXiv preprint arXiv:2405.044: 04330, 2024.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning, 2024.
Soham De and Samuel I.. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In Neural Information Processing Systems Foundation, 2020.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27, 2014.
11

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
P. Drineas and M. W. Mahoney, RandNLA: Randomized numerical linear algebra. Communications of the ACM, 59:80-90, 2016.
Shibo Feng, Peilin Zhao, Liu Liu, Pengcheng Wu, and Zhiqi Shen. Hdt: Hierarchical discrete transformer for multivariate time series forecasting. arXiv preprint arXiv:2502.0855 08302, 2025.
Everette S Gardner Jr. Exponential smoothing: The state of the art. Journal of forecasting, 4(1): 1-28, 1985.
Leon Götz, Marcel Kollovieh, Stephan Günnemann, and Leo Schwinn. Byte pair encoding for efficient time series forecasting. arXiv preprini arXiv:2505. 14411, 2025.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint ar arX1v:23/2.00752, 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.00390 2021.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971- 35983, 2022.
Haokun Gui, Xiucheng Li, and Xinyang Chen. Vector quantization pretraining for eeg time series with random projection and phase alignment. In International Conference on Machine Learning, pp. 16731-167500 PMLR, 2024.
Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, and Rob J Hyndman. Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond, 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. ОTexts, 2018.
Isse CF Ipsen and Arvind K Saibaba. Stable rank and intrinsic dimension of real and complex matrices. arXiv preprint arXiv:2407.21594, 2024.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 2014.
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language genome. Bioinformatics, 37(15):2112-2233 2021.
Jeffrey Lai, Anthony Bao, and William Gilpin. Panda: pretrained forecast model for universal representation of chaotic dynamics. arXiv preprint arXiv:2505. 13755, 2025.
Nguyen Quoc Khanh Le, Quang-Thai Ho, Trinh-Trung-Duong Nguyen, and Yu-Yen Ou. A trans- former architecture based on bert and 2d convolutional neural network to identify dna enhancers from sequence information. Briefings in bioinformatics, 22(5):bbab005, 2021.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned ep-decomposition. arXiv preprint arXiv:1412.6553, 2014.
Miguelangel Leon, Yuriy Perezhohin, Fernando Peres, Aleš Popovič, and Mauro Castelli. Compar- ing smiles and selfies tokenization for enhanced chemical language modeling. Scientific Reports, 14(1):25016, 2024.
12

---

Published as a conference paper at ICLR 2026
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequennce-o--sequnee to-sequence pre-training for natural language generation. translation, and comprehension. Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, 2020.
Bryan Lim, Sercan Ô Arık, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International journal of forecasting, 37(4): 1748-1764, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Spyros Makridakis and Michele Hibon. Arma models and the box-jenkins methodology. Journal of forecasting, 16(3):147-163 1997.
Charles Martin and Michael W Mahoney. Implicit self-regularizatioon in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021.
Luca Masserano, Abdul Fatir Ansari, Boran Han, Xiyuan Zhang, Christos Faloutsos, Michael W Mahoney, Andrew Gordon Wilson, Youngsuk Park, Syama Rangapuram, Danielle C Maddix, et al. Enhancing foundation models for time series forecasting via wavelet-hased tokenization. In International Conference Machine Learning. PMLR, 2025.
Lukasz Maziarka, Tomasz Danel, Slawomir Mucha, Krzysztof Rataj, Jacek Tabor, and Stanislaw Jastrzebski. Molecule attention transformer. arXiv preprint wrXiv:2002.08266. 2020.
Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam Bartie, Armin W Thomas, Samuel H King, Garyk Brixi, al. Sequence modeling and design from molecular to genome scale with evo. Science, 386(6723)cca09336 2024.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:221..14330 2022.
Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting, arXiv preprint arXiv:1905.1043, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8766 PmLR, 2021.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International journal of forecasting, 36(3):1188 1191, 2020.
Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.1355. 2023.
Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts. International Conference on Learning Represenlations, 2025.
Sabera Talukder, Yisong Yue, and Georgia Gkioxari. Totem: Tokenized time series embeddings for general time series analysis. arXiv preprint arXiv:2402. 16412, 2024.
Alex Townsend and Lloyd N Trefethen. Continuous analogues of matrix factorizations, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 47 (0021))):2014088,, 2015.
Lloyd N Trefethen. Approximation theory and approximation practice, extended edition. SIAM, 2019.
13

---

Published as a conference paper at ICLR 2026
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.047668 2020.
Andrew Gordon Wilson. Deep learning is not so mysterious or different. PMLR, 2025.
Malcolm L Wolff, Shenghao Yang, Kari Torkkola, and Michael W Mahoney. Using pre-trained LLMs for multivariate time series forecasting. arXiv preprint arXiv:2501.063886 2025.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo.
Unified training of universal time series forecasting transformers. 2024.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings the AAAI conference on artificial intelligence, volume 35, pp. 14138-14148, 2021.
Jinsung Yoon, William R Zame, and Mihaela Van Der Schaar. Estimating missing data in temporal data streams using nulti-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, (665))1477-1445 2018
Annan Yu, Arnur Nigmetov, Dmitriy Morozov, Michael W Mahoney, and N Benjamin Erichson. Ro- bustifying state-space models for long sequences via approximate diagonalization, arXiv preprint arXiv:2310.01699, 2023.
Annan Yu, Michael W Mahoney, and N Benjamin Erichson. Hope for a robust parameterization of long-memory state space models. arXiv preprint (.ti::2005.13975.. 2024.
Michael Yuanjie Zhang, Jeffrey R Russell, and Ruey Tsay. A nonlinear autoregressive conditional duration model with applications to financial transaction data. Journal of Econometrics, 104(1): 179-207, 2001.
Xiyuan Zhang and Danielle Maddix Robinson. Mitra: Mixed synthetic priors for enhanc- ing tabular foundation models, 2025. URL https: htttps://ww.... htwp://wwwaaaazo..mmmm/mmm zoa www.:aaae....cieeeceeeeeeee scw.nce/blocccccc//ooooocc sa--miidd-sytteeeeeeeeeee
Yuanzhao Zhang and William Gilpin. Zero-shot forecasting of chaotic systems. International Con- ference on Learning Representations, 2025.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
A.1 TIME-SERIES MODELING
Time series data are ubiquitous in many domains, including scientific (Abhishek et al., 2012; Zhang & Gilpin, 2025; Lai et al., 2025), industrial (Hong al., 2016), and financial applications (Zhang et al., 2001), where they facilitate critical tasks, such as forecasting (Hyndman & Athanasopou- los, 2018), imputation (Yoon al., 2018), and anomaly detection (Blázquez-García et al., 2021), I). Classical time-series forecasting (Hyndman & Athanasopoulos, 2018) has deep roots in traditional methods such as ARIMA (Makridakis & Hibon, 1997) and exponential smoothing (Gardner Jr, 1985). Many modern approaches involve deep learning: for example, sequence models such as DeepAR (Salinas et al., 2020) popularized probabilistic forecasting at scale, while Lim et al. (2021) combined the attention mechanism with interpretability for multi-horizon tasks. There are other neu- ral forecasters without attention, such as N-BEATS (Oreshkin et al., 2019) and N-HiTS (Challu et al., 2023). An evolving modern alternative that targets long contexts efficiently is the recently developed state-space models (Gu et al., 2021) and Mamba (Gu & Dao, 2023). These nettor-- methods are usually considered task- that is, given task, one trains model to obtain specific set of weights tailored to that task.
A.2 TIME-SERIES FOUNDATION MODELS
Recent work pretrains general-pune purpose time series models across domains and tasks. Examples include TimesFM (Das et al., 2024), Chronos (Ansari al., 2024a), Moirai (Woo et al., 2024), and Time MOE (Shi et al., 2025). These models differ in tokenization choices, training corpora, and zero-shot protocols, but they share the goal of one model that transfers across datasets and horizons.
A.3 TIME-SERIES TOKENIZATION AND EMBEDDING
Designing the input representation is the key to leveraging the high flexibility and expressive power of large Transformers. Patching turns small motifs into tokens, in Nie et al. (2022) and Das et al. (2024). Discrete tokenization via quantization has been explored by Ansari et al. (2024a) and Talukder al. (2024), and other discrete designs include HDT (Feng al., 2025) and vector- quantized methods (Gui et al., 2024). Another line of research is by using frequency-based tokeniz- ers. such as WaveTooenn (Masserano et al., 2025). Lately, traditional language embedding strategies such as Byte-Pair Encoding are also considered (Götz et al., 2025).
A.4 LOW-RANK STRUCTURES IN DEEP LEARNING
Low-rank structure shows up in the deep learning community as both an inductive bias (Martin & Mahoney, 2021; Wilson, 2025; Yu et al., 2024) and a compression tool (Sharma et al., 2023; Gu et al., 2022; Yu et al., 2023). There is also a line of research that looks into parameter-efficient finetuning called LoRA (Hu et al., 2022), which learns low-rank updates to weight matrices. We note that LoRA is different from low-rank model compression strategy because LoRA essentially learns a fatt--rann-lu-oooeee
expression of the weight matrices, and does not facilitate storage inference of a model. Earlier work reduced cost via matrix or tensor factorizations, especially in the CNN community (Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2014). These ideas motivate studying when low-rank operators suffice for sequence models, while our paper provides clean justifications for why low-rank operators suffice for a time-series foundation model.
A.5 LOW-RANK STRUCTURES IN TRANSFORMERS
Low-rank structure in Transformers and attention has been extensively exploited for both effi- ciency and interpretability. A first line of work directly approximates the quadratic self-attention matrix with low-rank kernels: Linformer (Wang et al., 2020) projects keys and values to low- dimensional subspace, yielding self-attention with linear complexity in sequence length, while Nyströmformer (Xiong et al., 2021) uses a Nyström approximation of the softmax kernel based on a small set of landmark tokens. Random- feature methods such as Performer (Choromanski et al., 2021) further view softmax attention kernel and approximate it with low-rank random feature
15

---

Published as a conference paper at ICLR 2026
maps, obtaining tinear-time and memory. Scatterbrain (Chen et al., 2021) analyzes when sparse ver- sus low-rank attention yields better upproximations and proposes unified sparse+low-rad estima- tor that attains lower eπror than either component alone. Complementary approaches parameterize the weight matrices themselves low rank: LoRA (Hu et al., 2022) learns low-rank updates on top of frozen full-rank weights for parameter-efficient fine-tuning of large language models, and Sharma et al. (2023) perform layer-selective rank reduction to probe and improve the reasoning behavior of Transformers. These works treat low rank primarily as an architectural algorithmic assumption used to accelerate or regularize attention. By contrast, our analysis starts from the observed low numerical rank of time-series embeddings, proves when attention matrices over such embeddings are provably compressible, and then uses this data-driven perspective to design TSFMs whose atten- tion layers are low rank by construction and compressible, which accelerates both pretraining and inference.
16

---

Published as a conference paper at ICLR 2026
B PROOF OF THEOREM AND COROLLARY 2
Before go into the proofs of Theorem and Corollary 2, present figure to illustrate the differ- ences between the quantization- embedding and continuous embedding discussed in Section 2.
Quantization-based Embedding
Continuous Embedding
R
R
④ R
R"
R
R
Figure 8: A comparison of the quantization -aseed embeddings and continuous embeddings. In quantization- hased one, we discretize the input domain into a number of regions and map each of them to trainable vector. In continuous embedding, to map each element into the hidden space Rdd The red parts in the figure highlight the trainable components.
trainable function (usually MLP)
The proof of Theorem relies classic polynomial approximation results. Intuitively, if a function Φ is well-approximated by low-degree polynomial, then the low-degree polynomial can be repre- sented with small basis, spanning low-dimensional subspace. We formalize this notion by providing the proof.
Proof of Theorem I. Fix some S d. Let Pag be the best degree-(j 1) polynomial approxi- mation of Φ; in the infinity norm, and let
6, = max || - P.ollo=((-1.))) Isidd
Since P.. a polynomial of degree < j 1, we can write it into
(z)(2) Σ aia Te(x),
k=
where T3 is the degree-k: Chebyshev polynomial. Then, we can write =\t into
(z))(z)
(Ø1 (z,)))))
0100
01j-1
To(
E=
+
+E,
Pa.s (x)
(da- (z)
0d.0
ad
T Tz)
E,
Since B, is a quasimatrix of rank at most j. by (Townsend & Trefethen, 2015, Thm. 4.2), we have that
8j+1
|EIlr
Σ |0:
Pa P
2
Σ b P.. P - $ 22dd (7)
17

---

Published as a conference paper at ICLR 2026
Similarly, note that
P1o(F1)
PIo(Ec)
[di P1))(z1)
(01 (oL)(EE)
+
Pa)(21)
PaS(IL)
(oa- - Pa1)(
(ou - (OL)
Ta1,0
a1
To(
To(#1)
+Ff.
ad,0
0d,
(x-)(1))
T3-1(01)
Since the rank of y is at most j. by the Eckart- Young inequality, we have that
F |Fylle VdLōg.
J++1
From (Trefethen, 2019, Thm. 7.2 & 8.2), we have that
2V
8, 2Mp
2M 1ー
6, < Tv(j 11
and
(9)
p
when d; satisfies the condition in the first and the second statement of the theorem, respectively. Hence, the two statement are proved by combining eq. (7), (8), and (9).
The activation function swish used in Time MOE is analytic, and its domain of analyticity increases as 3 + 0+. Hence, we can use Theorem to prove that Time MOE's s embedding is low-rank.
Corollary 2. The embedding used in Time-MoE, defined by (x(z) swish (w,x) (v,z) (wara? e Ax), where we assume |waes| S and 3 > 0, satisfies that e=ax),
JJ+1 J=+1 - - 0 (VaL(B + 3- (1 (299))
where o++1 defined in Theorem for any V and the constant O-notation universal.
Proof. The function (b(2) is meromorphic function with poles at (2 exp(-32) -1} {(2k+ + 1)ri/B k E ZJ. Set that
+ (233). Then, within the Bernstein ellipse E, of radius we have
|x|² I-c 0(1+8~9).
Re (e 30)
lo)(z)
<
E Ep-
where 0 <<< 1 universal constant and the constant in the O-notation is also universal. The corollary follows from Theorem 1.
18

---

Published as a conference paper at ICLR 2026
C PROOF OF THEOREM 2
The intuition Theorem is that the MLP does not have a nonlinear activation function, then the linear MLP clearly maps low-rank matrix to a low-rank matrix. While a nonlinear activation function complicates things, a contractive function does not increase the Frobenius norm of a matrix, which is equivalent the e2 -oomm of all singular values.
Proof of Theorem 2. Since X is rank-k matrix, the rank of W X is at most k. Hence, the Frobe- nius norm the matrix satisfies that
W W
Σ oy <o (W,
Given that [xxx) S |o|| for every x E IR we have
min(dy.L)
ko koi x [wxx]]
Σ
o,(w(W,X))
Using the singular value inequalities, we have that min(d,,4.
min(dy.L)
oj w (WWW(WWx))'
S (W
Σ
(W)()))))
S (ko,(W2) 01 (W,X)² = k W
This is, control the number of large singular values of W2 w W X by
Ki (W2
Wella W, [1 2k.
Moreover, the rank of the matrix W3X is at most k. Using the singular value inequality that
o+tj- B) < o, oo, (B),
A.BRRa itj-1 ----1 min(d, D),
have
(+()( S WoW -W j i min(d, L) k.
This gives us
K Holo,(((X)) > Welle / ([, W W +
S (1+
which proves the result.
19

---

Published as a conference paper at ICLR 2026
D PROOF OF THEOREM 3
To prove the upper bound in Theorem 3, we will prove a stronger result concerning not only single-head attention but also a multi-head one.
Theorem 5. Let
xN] E IRd be an embedding matrix, where the hidden dimension, N is the vocabulary size, x, E Rd, and ||x S C for all multi-head attention matrices with h heads, such that l(we) (0) w ( k ll2 I Cydd. and [W V 2l S Cy for every 1 S h. For any d < d such Q
be a constant. Let =
is j - N. Let Wo, WK, Wv E Rdxdd
that d+1 = 0+1 (=) < 1, there exists a stable low-rank approximation Wo, W K, Wv V E Rdxd with rank Wq) Q) rank W k) rank Wv) d, [ WxWle 1l2 S I k ll,, and [W & || such that given any input matrix U E IR for any L 2 1, where each column of U is column of E. we have that the low-rank attention matrices uniformly approximate the original one:
MH Atteentnnionn U.WQ. Q. WK. W h) - MM--AAttetiion W..Ww)/l 3o (vdo
(10)
where the constant in the O-notation only depends on C.
Proof. Fix a d d. For the sake of simplicity, we assume, without loss of generality, that C' 1. Let
= V
Ua E Rdxd
S E dxdd E
be the truncated SVD E. Let Q2 = V E R Therefore, we have
= U U2lll S (1))
Define Wo. WK. K. and Wy by We U Raxd
Wx UUU Raxd
Wv = WrU,U Rdddd
Since U has orthonormal columns, we have that (Ual2a = | 1. By the sub- cub-multiplicity of the spectral norm, we have that
| IW W x < I Wxlk,
Wv < W
which proves the stability of the low-rank representation. Let Qu E 2x be the matrix defined by the condition that the ith column of Qu is the jth column of Q, if and only if the ith column of U is the jth column of E, and let
A U,QU U.
Since every column of A is a column of = - U2Q. its norm is no greater than 0 (E), i.e.,
Au(s)))l s o (3),
(11)
Aull - | S VL (E)
Next, we have
A
(K)
112 A 0 VdLL (E),
(6)
w , W Uuuff
V
-WO U w U
U w U - U,
where the first incquality comes from the sub-multiplicity of the spectral norm. Morcover, every column of A (6) is a column of - -w(i) A and must satisfy that
A (8) (31)l2a da (E)(()),
ISi5 L.
(12)
20

---

Published as a conference paper at ICLR 2026
Similarly, have that
||
(1) Q 2 I -00
(6)
= U (W
U K U'UJU
Q
-
T U
= -U (W
U K
U Q
U
= -
U- (W Q K A A Q
) T K Au.
where the inequality is obtained by recalling that | & VL and A y la S VL a (=):
Moreover, since we have [ll & and AA(;))]) & 02 (3) for all S j S N, every
entry of A (6) satisfies that
( (t,j)ll2 < (E)
(E) + EE)
L ISiSL.
(13)
Q
For each fixed
I i i h, define the notations
= ( U
( U (w TWUU
vdn
ddn
9 ( G(,)),
9 G(:))
For simplicity, we drop the superscripts (i) on G, G. g. and g. Since we assumed ||uر||2 S 1 for all 1 isj S and ((w())
Q T w K
l2 I dh, we have that
1 Imax I 1, 1g, lmax S 1
1 isish.
Denote by of and 5j the tth entry gj and 9j. respectively. For every fixed S j S and 1st L, we have
exp(ot)
exp(3f)
softmax(g,) - softmax(gg))
Ele exp(g))
Ek exp(o))
E (exp(o))exp(()) -xp(xx
L mMx (exp(() exp
exp(ot) E explot))
I exp(p-
L maxi (xp(())) (exp( exp(x (cm)))))(()
L7 exexp(2)
maxx(exp(Jj(cxp(c))-oo exp
exp((f)
L exp(-2)
(3) a++1 (3) + (20 2+1 (3) + 0 2+1 (3)
Lexp(-2)
Do
(14)
where the last inequality follows from eq. (13). Hence, for each fixed $ S L and S S h, we
have that
w! U softmax(g,, (") w U softmax
g |
v U(; softmax (5²) w WUU U(, softmax(0))
VL max
( (w U(: f) softmax(o)) (w ( U(it)) - (0 (softmax(9)) d3))
ISESL
VT. max 1SiST. (11 v (:, 1) softmax (3) w U(:11)l1 DO)
< VT (Vth th 0. 1 + Vdn o 10 + 22m) /L)) =0 -OVThoa-IVVE)
21

---

Published as a conference paper at ICLR 2026
Since w U softmax(g))) (0)
where the last inequality follows from eq. (12) and (14). w U (oftmax(g,)) is the jth column of
Attention U: W w K. w - Attention U; W W K W
we have that
MHH-Ateention(;; We, WK. Wv, h) Attention(U; We, WK. Wv, h)llF
Attention (U; w 0 W K
w
Attention (U; W w( w/
ΣΣ
w ( U softmax(g))) W U $oftmax(o,"))
= VhLo V -
-0 -(Vooan)) (V(ooa)))
The proof is complete.
Theorem 5 immediately proves the upper bound in Theorem 3. The lower bound needs separate argument.
Proof of Theorem 3. The upper bound follows immediately from Theorem 5 by setting h = 1. To prove the lower bound, let U' Raxd and E gdxL be such that
- U
U' diag(o)...... oa) E RRd
O gdx((-d)
Clearly, the singular values of U are 01, ad- Define We and W such that
2Vd WoWr = log(4d)o Vd Lt.
Then, we have
U W = log((d) o2 2
diag(of diag(of......o3 o
(d
T
0 xI.
((1-d) (f.-))
Let G softmax(T) and let gj be the jth column of G. Then, for every 1 S d, we have
1
i i j.
(L - 1) + exp (log( (oog(4))07700)
4d
9; = 1111
T, 55
(15)
exp (log(Ad)o 20,)
2 2'
i i=
(L - + exp (log( (d) o 2 og)
Write G into
G
G
G
G21
G22
where
G = G' E dxd
G12 E R (I-()-
G21 E ((I-d)xdd
G22 E R (1-
and write G1 into the sum of its diagonal part and off-diagonal part:
G G Gofl.
where Gaing is a diagonal matrix and G Goft.digg is a matrix with zero diagonal entries. by eq. (15), we have
Then,
Gall 12 S Goll || S 1 1
O2 (Ginn)) = min Jf 2
1545d
2'
4d 4'
22

---

Published as a conference paper at ICLR 2026
Hence, by Weyl's inequality, we have
oa (G1) 2 (Giig))))
Goft-diag 11 2 4"
(16)
Using the results above, we have
U W
U'
Wv U softmax
U softmax
Vd
Vd
G
G12
(WrU WrU)G WvU - WrU Oax(n=o)) ]
G21
G22
= (Wr''-wuU)
Gn Gn (WrU'-WuUl G12
but Wy U' is rank-d matrix must have
|Wr - Wr 2 -a-1 (
1+1
(17)
Hence, by the singular value inqualities, we have
U WoWW W
U U
W w U softmax
U softmax
Vd
vd
2 2l(wrU'-wrU)
Gn
20 (WrU'- Wr oe (G()
2
1
4 d
where the last inequality is obtained by combining eq. (16) and (17). The proof is complete.
23

---

Published as a conference paper at ICLR 2026
E COMPRESSING A MULTI-HEAD ATTENTION LAYER
Theorem 3 concerns the compressibility of a single-head attention layer. In practice, most TSFMs use multi-head attention instead:?
Attention( U; W w w())
MH-AAttention U: Wo, WK, Wv,
Attention( U; W (N)
W (h)
K. w(!))
(18)
Q
Wo/kyv Q/K/v)
(W /
W
Q/K/V Rdxxd,
d₁ = d/h.
This leads to the following question: does the number of heads have an effect on the numerical ranks of the attention matrices trained Chronos models? The answer to this is positive, but there are some important subtleties. If we look at the left panel of Figure 9, then we see that if we pretrain Chronos models with fixed hidden dimension = 1024 and only increase the number of heads, the numerical rank of attention matrices increases. At first, it may seem that this happens because a multi-head attention less compressible than a single-head one, but this is not the case. It is straightforward to show that Theorem 3 holds well for multi- head attentions (see Theorem in Appendix D). In particular, if E has algebraic rank of d, then all multi-head attention matrices can be compressed to rank-d without affecting the output.
Rank of W
Rank of per-head W
Angles between Rows of W
200
175
(Wa)
rank, (Wa)
1.0
200
125
1000
400
200
600
200
400
600
Figure 9: The left panel shows the averaged e-rank of query projection matrices We in pretrained Chronos models with varying number of heads. The middle panel shows the averaged e-rank of query projection submatrices w-()) for every head. The right panel shows the angle between every pair of rows of WQ Q in the first layer of 3-head pretrained Chronos model.
Q
Theorem 5 holds for multi-head attention, then why does a multi-heaa attention exhibit a higher numerical rank? To understand this, need understand the mechanism of low-rank weight matrix. We use Wq for illustration, and the same analogous applies to W K and Wv. Let We E Raxd be approximated by rank-d matrix, i.e., Wq R W1 W2, where W1 Raxd and Wa Rdxd We view the action Wº U, which maps every column of U from Rd into IRd, as two- step process. First, we multiply W₂ to U drop the columns of U from Ed to Rdd In numerical linear algebra, this is known "sketching." (Drineas & Mahoney, 2016) i.e., W₂ "sketches" a d- dimensional subspace R(W,U) in R the row space of W.U. from d-dimensional subspace R(U) in RA, 3 If (U)((U) small, then there exists sketching matrix We so that R(W,U) s R(U) R U); hence, to Rdd and have W, (W,U) WoU.
apply matrix W, lift the columns of W₂U from Rd back
we apply a low-rank approximation to a multi-hean attention matrix WQ. then we obtain:
w(1)
[
W
Wo W,W
W₂
w(O E Rdd. xd
(19)
In eq. (18), the dd normalization factor in an attention is changed into 1/Vdg- IIn numerical linear algebra (NLA), sketching is usually considered as operation on the column space. To adapt it to our framework, we sketch the now space instead. This is achieved by taking the transpose of the matrix U and sketching its column space as usually done NLA.
24

---

Published as a conference paper at ICLR 2026
When we apply Wa to U, for each head S i S h, W2 sketches d-dimensional subspace from the row space of U while w forms d, rows from this d-dimensional space. This makes the problem clear: in an "optimal""owwrrnt" low-rank approximation of Wq. the sketching matrix W2 is shared across all heads. Since each head is independent from the others in a multi-head attention, there is no guarantee that, in a pretrained model, the sketching matrices will be shared. In the right panel of Figure 9, we visualize the angles (see eq. (2)) between every pair of rows of WQ. If W is shared across all heads, then all rows of Wo should exhibit some linear dependency. We only see low-rank structures within each head w (i.e., the dark d, x da blocks along the diagonal). This shows that instead of eq. (19), We from a pretrained Chronos model looks more like
w
(1)
(
Q
w(O E Rddaad
W E ERixd
W Q
R
(20)
(
W (1)
The rank of the right-hand side matrix in eq. (20), which consists of h rank-d submatrices, can be as large as hd. This finding explains the phenomenon observed at the beginning of this section, i... while the numerical rank d of the input stays the same, the rank of the attention matrices increases with the number of heads h. The middle panel of Figure 9 confirms this numerically. We that the numerical rank of the projec- tion matrix ( each head remains relatively constant for reasonably large & as we increase the number of heads.
which leverages head-dependent sketching W
Our observation in Figure 9 is empirical and explanatory: it says in pretrained TSFM, the atten- tion matrices W W//K/V are essentially doing the -eadddependent sketching (see eq. (20)). From a methodological point of view, if we train compressed model (i.e., the one that is paramaterized by W1 and W2 instead of W WQ) from scratch, should we adopt a parameter-efficien meaddindeppndent sketching in eq. (19) or head-dependent one, which requires more parameters? If we know the numerical rank d of = so that 0 (E) is tiny, then from an approximation theory perspective, there little loss using eq. (19). The only problem is: we do not know d a priori. If we choose d small enough that 0 [ is still relatively large, then the following theorem (see Appendix E for the proof) shows the benefit of using head dependent sketching w in eq. (20).
Theorem 6. Fix some d = h x dn, for two positive integers h and da and L 2 d. Let C 2 be any stability bound. Let 1 2 0i 2 σ2 2 2 oa be any fixed sequence. There exist an input U E IR mdx with (U) o, for all S S d and three matrices Wq. WK, Wy E Rdxdd such that the following statements hold for any d < d4:
1. Given any rank-d matrix Wy E gaxd with ||Wlll S WiW we have
MMH-Attentio((U) We WK, Wv, h) MHHAAtenttoo((U (U; Wo, WK, Wv, 1) 2 2 i+1
1
There exist low-rank matrices We: We WK. Wy Rdxd with |ell S |olle WKllF S || kl, and |/Wvlll= < [[.,, such that
(1) (11)
Q/K/V1 Q/K/V2
W Wo/k/V=
w Q/K/Va (Rdx((+11)
((()) ///K/v2 (R(
w
Whh (h)
Q/K/V.1 Q/K/V2
where every row of w(( entries, and satisfies that
Q/K/V22 except the last row of w (1 K.2 contains most da non-zero
MH-Attention (U; Q, WK, MHH-At)- W. W Ww.Ww, < <4Vho
Theorem 6 highlights "error-correcting" mechanism of the head-dependent low-rank design in eq. (20) in the following sense: if we choose d where d is large, then the first statement
25

---

(1) Published as a conference paper at ICLR 2026
shows that the shared sketching in eq. (19) inevitably limits the expressiveness of the reduced multi- head attention layer. Using a head-dependent low-rank representation relaxes the error to the order of hd which can be significantly smaller than d when the number of heads h is large and the singular values decay fast. While using eq. (20) requires a larger number parameters, Theorem suggests the potential of sparse parameterization of the sketching matrices W Q/K/V22" We leave this promising future direction.
E. PROOF OF THEORRM
We prove that sparse but head-dependent sketching performs better in theory than a head- independent sketching, especially when d is lower than the numerical rank of the input matrix. The intuition is that if d is too small, then the sketching performed by single sketching matrix must be bad, but we use deadddependent sketchings, then we can leverage multiple of them to still obtain a lot from the input matrix.
Proof of Theorem 6. Set Wv = Ls- Interleave the singular values 01...... Og follows:
(1) (1)
01 02 dn (0i, 001,0h+1.--.. -...ll.................... 1dd-))++
01 02
(2) (2)
dp
(2)
(02, ( -dh.-
(N)
(h) (h)
02 o.., (h) = (oh, 02h, ,.. (d.
Define the input matrix by
U= (oig(dag((01)
diag(o (11))
Odx(e-d)
Up
For every 1 i S and Sj S dn, let (() E Rd be unit vector satisfying that
61) (1)
<
(
Then, consider the following matrix:
W Q
W K
(1)
(1) de 10/
(( /o(
vd
1o Eddi
(h)
Then, by our construction, the matrix U W WxU satisfies that
G.0)= (UW) (i.5) <1. i$
Set
{
min
(1+C) min a ( isich /
/ [2|| 2011),1
By choosing sufficiently large
O, guarantee that
UT Ua Vdn
La
I, -
softmax
Se. (21)
O dixd
L (111(L-dx(L- (]x(L-d)
F
Let query and key matrices We and W be defined in eq. (18) using submatrices and w() K = a W k. respectively, for all < < We use these matrices WQ. Wx. Wy, and U prove the two claims.
( = a
Q
26

---

Published as a conference paper at ICLR 2026
Claim 1: No low-rank parameterization gets to accuracy below a2+1" Let Wy E R be any rank-d matrix. We can write the approximation error as
MMH-Attention(UU We WK. Wv, MH-Attention(U) (U; We, WK. Wr,h)//
wPuG
W UG
W UUG
W (
UG
w UG
@ UG
W - : - G)
UG
W
UG
(h) UUG G)
Eirs
Emn
To control Email we note that
Exail < max 1sish
(W w
WDDUGG-)
S max I5ish
(/w//
+ w/ | (22)
1
< 01+ 2+1
To control Erend, we note that
(W W UG
(WW'-W(" UD L- L-Uplax((-
(W (h) W UG
(w w(()) [ UD L UDD'
Hence, we have
(WiP) w
(w w
Evead
: (W V w U
(w th)
JUG
(23)
WvUD - W. UD o+11
where the last inequality follows from the fact that the singular values of WvUp are exactly oi....,0a oa and WD matrix whose rank at most d. Combining eq. (22) and (23), we have
MH-Attention(U; U: We. WK. hv, MH-AAtention(UU WQ. WK, Wy.
Elead Euail 2 d+r
Claim 2: A sparse parameterization gets to an accuracy below hdd+1 For every S i S h
define (i(t) V.L E Idda xa and (i)
V.R Idddd as follows:
( V.L
( V.R
O dxxds (f-1)
I
O Ox(((d, (((((ddd))+dd (h-d((1-)))
0 (dh ddxd
Then, have
diag (0(1 (1) (1)
O
(i-1
O Odx((dn- - (d) (di ddx(((nn d2
U
V.L
V.R
(du (i-1)
(d, -dxdd
while
(1)
W V
= 0ixdg(6-1)
diag
,..., dd.
27

---

Published as a conference paper at ICLR 2026
Hence, we have that
| w W
o11 Q. w W. Rd(++1xdd be given by
V.UUl
(24)
d+1
Similarly, let W
W
K.L Rdxx(d+1) and w
Q.L.
w
W K,L a
(6)
1
(0 ].
i' mod(i, h) + 1,
Q.L
d
a
1,
1xx
[1/02+1. 0, O,...,. 0]
w
Q.
(i-1)
I
0 (x1 (k-
i> 1,
[1/01, O,.....) 0]
0 1xx
O
1x(((hn -d) +d +o
(1-
w( K
O (i-1n
1a
0 Odx((dn -d) -d) 1/0a]
[1/01,.... (i-nn
O 1
Note that since we have 1 (6) ll2 for every 1 S i S h and 1 S j S dn. we clearly have that llaW x|F for every 1 S i S h. Hence, [Wxle ll [Wa K lr.. From now on, we only argue for the first head, i.e., i = 1. The rest follows easily from symmetry. Set i = 1, we have
||w(0)
Q. w g.lllF & loWelF and || ( w
K.I. K.
we have [Wolle I [Wolr and [
(1)
(1) k isisd, 1sksd,
v1 (2)
1sjsd, dcksd,
U UTw()) w(1) w w Q. Q.L K.L
KRR U (1,))=a² a²
v(2) (1)
k j=d+1, 1sksd,
v (2)
TO)
vi j=d+1, å<ksd.
0,
otherwise.
That is, for j d, the jth column of (1) has exactly one entry that equals 1, which is the jth element for 1 j I d and the (j + 1 1)st element for > d. Moreover, for j > d, the jth column of T()) is zero. Hence, from the definition of a, we have softmax (T (1:d,:) (::dd))-G((:3)) G(1:d,))
||
softmax (ra))(1:))) -1:d.))-G(:dd
| | || G
< (25)
Combine eq. (24) and (25), We have ( w
w
UG W
U softmax
(
V.L V.R
w - (( (!)
UG
(1) V.L
w
UG-
w U softmax (T
V.L V.R
V.
V.R
w.' - w V.L w (I)
UG
w
VL
v.l
(1)
U (G- softmax (Tw)) (1:d,)
w WU (()) W.L
||G
112
U
+o
d+1 |[[[ GG--tmma softmax
(1)
(T()) (1:
2
(1) S [0 d+1 (2 + =) + 2:
w V.L
V.
(1 d+1
(1)
< 40
where the first inequality follows from the triangle inequality, the first equation follows from the
sparsity of ((1) V.t. the second inequality follows from the sub-multiplicity of the spectral norm and eq. (24), the third inequality follows from eq. (24) and eq. (25), the fifth inequality follows from the definition of e, and the last inequality follows from the definition of (1) Notably, this d41'
inequality holds for every head 1 S i S h. Hence, we have
MH-Attention U: W Q. W K. Wv.h) - MH-AAtention UU; Wo. WK. K. Wy, Wy.i)l/
vh max ISiSA
n UG
(( w ( V.J. V.R U softmax (IC0)),, S 44To and+1
The proof is complete.
28

---

Published as a conference paper at ICLR 2026
F PROOF OF THEOREM 4 AND COROLLARY 1
In this section, we prove the idea of flow-of-ranks. The upper bound is proved via a straightforward application of singular value inequalities while the lower bound requires careful construction.
Proof of Theorem 4. We prove the two statements separately.
The upper bound. Fix a head index 1 S i and let
W w dd,
Y W U softmax (
T
Then, every entry of T has a magnitude no greater than because every column of U has a 2- norm no greater than 1 and |[ @ w ll2 I vdh. Hence, every entry of softmax (T))) has magnitude between e and L L- This means
01 [softmax
(T()) ) = softmax (T)))_ = softmax ( se.
Moreover, since (w()
< 1, we have that
oj (x S SE)o;(U);
1 isis de.
Fix some J S d and let jn 10i 1)/h| 11. When we concatenate Y() form Y, using the singular value incquality that o+ts-1 (A + B) S (A) + oB( and defining Y to be the concatenation of ( for all E I, have
YY) S Jjh (Y tos-suta
(Yll-11)
S Ss (r(li) to
(r(l--1) to (yll-2)
Σ ju (v(i))
(Y())
Σ ji (Y S Sehhos,( (U).
+ +00-(b-1)6a-1)
1=1
Hence, applying the same inequality again, we have that
ox ok U+ J v) S Sok-tt1 (U)+a)
1 v x
e²h So--jti++
VN 10- 10-1)/M (26)
Moreover, from the triangle inequality, we have
o 2 (U) oi
VN x) 20)
1
- VN
e 2 ho, (U) ( 2 2 (V)
(27)
Combining eq. (26) and (27), we prove the theorem.
The lower bound. Let o 2 ... 2 aa > 0 be given. Define the input matrix to be
U (di.g(o)......uu σa) Odx(L-4)]
For every isis h, let w be
w
Idh
0 ((dh-1)x(d-aa-))
0 01x(dn-1)
0 (1x
Set = V aa/(5h)) For every S i & h. let p()) E RL be the matrix so that P(0 du, ( ((-1))d++] id,)) = It,- p(() (d,,, k) = 1 for every < k < (i - 1)d, and every > id,, and is zero elsewhere. For sufficiently large 0 determined later on, let
w Q = a [diag(o]') o.. 0 Odnx(d-an))
Odn (dd-1)x(((-1))
0 O (1) [old.
(
diag(o 1)da +1
K
o
(1-1)dn
o
29

---

Published as a conference paper at ICLR 2026
Then, have
T T U W'l
K U
(d, - 1)x ()-((-xxx)))
(d, - (b--ii) (b-
=a [T Od,x(L-dn)]
Idaa
Oa ((--( = -ap(i))
1ix(i-1)da
Therefore, by picking a sufficiently large, we have
1.
h = j+ (i - 1)dh, dh and l (i 1)dh,
GG0 = G ()A))
j dA and idh,
L-1,
h > d,
0,
otherwise.
(28)
Note that we have
w U
diag( 01 Oha-
(1x
1xx -1
0 (1x(L_dn)
Od,, ()x((()))
O (k-
Lda
L Ld
x
1x(i-11dda
1x(--i)da
(Zt-dd,)x( (f-
(I.-da)xd Xx
0 d (b-e)dh
(1 (
diag(as (0...... The 2ha-)) -1)((t-d))
Od diag(o1...... 0ha Od1x(b-i)dd
0Ix(L-d)
(
Note that the mulithead attention output is given by
Y MH-Attention( (U; WQ. WK. Wv,
h)
YY Yead
Y Yural + E,
Y lead
( lead
Y (h)
Y Imall
(1) T trail
(h) trail
lead
where
h
h
E
<
W
1
UG
w
UG S
w
U
- G S he,
1=1
and the layer output is given by
1 E,
Z = U
Y
[Zedd
Zurii]
VN
VN
Ziend diag (1,, oa) +
Y Ylead
Zirail
Y
Since Ziena is diagonal matrix with nonnegative entries, its singular values equal its diagonal en- tries, which are
V Od +1++1+ VN
t1 01
02 v 02
od, -1 VN on
Odn Od +d
Od, +2 VN 02
od, 11) VN o
(h-1))d++ + V (h-11)d++2 + 02
02
((--1)d+(dd--- VN dd de -1
0(h 1)da +d
(29)
Moreover, since Ziead is a submatrix of Z, the singular values of Z are no smaller than the singular values of Ziead Given < i < h and 1 < j < dn. let Ei,j = (ti-1jdat) + 1jzdd1 be the element in row i and column j in the array in eq. (29). Then, have that
E Ev
for any i 1' and j 2 "."
30

---

Published as a conference paper at ICLR 2026
Hence, {i, no greater than the (i jitth singular value of Zeaa That means for every isis h, the kthh singular value Zedd satisfies
Tk (Zeed) 2 St,sk//] oi- +)4-1)1
VN
Using Weyl's inequality, have
he
Th ( 2 σx ox (Zead)
VN (1E) 2 ox (Zead)
VN
1
((-1))d++R//1 + Ta//i
(15))
0T4/68)
VN
Moreover, we have
1
(z S 1 (Zread) + oi (Zuril)
(E) 1
01
Y
amil
he
VN
VN
VN
VN
16
2 VN L
5
Combining the two inequalities above, we obtain the theorem.
The proof Corollary is straightforward manipulation of the ceiling and floor operators.
Proof of Corollary 1. Set j = [(hk + 1)/ 1)J. have
Øk-j+1 0 - (6++1)/(+++)+++ - T (bk (ah- (8+A))(++)]
and
((-1)//]41 = OLLLN-1//00D--)//HH S ((0M+))/((4)-))))11111-- = -000)//44)2
The corollary follows from Theorem
31

---

Published as a conference paper at ICLR 2026
G WATCHING THE WEIGHTS OF LARGE MODELS
The central claims made in this paper heavily based the low-rank structures of weight matrices in large-scale time series foundation models. this section, we provide more empirical analysis of the singular values of these matrices.
G.1 COMPARING THE WEIGHTS OF TSFMs AND LLMS
For the first set of comparisons, we consider three models: T5, Chronos, and Chronos-Bolt While T5 is an LLM, Chronos and Chronos-Bolt are TSFMs. The three models we compare have the same base size, and each model contains 12 encoder layers and 12 decoder layers. From each layer, we take a matrix W, which is either the query projection matrix Wq E R 768x668 the first matrix of the MLP layer WMLP MLP R We apply an SVD to the weight matrix W UEV and construct a histogram out of all relative singular values in diag(E) 1
T$
Chronos
Chronos-Bolt
W MLP
Figure 10: The distribution of all relative singular values in a weight matrix W in T5, Chronos, or Chronos-Bolt model. We use five progressively fainter face colors to indicate the range where the relative singular values are in the (1e-5, 1e-4], (1e-4, 1e-3], (1e-3, 1e-2], (1e-2, 1e-1], and (1e-1, ranges, respectively. Each model contains 12 encoder layers, and we arrange corresponding weight matrices in row-major order. Note that all histograms are made on semilog-x scale.
From Figure 10, make some observations that align with the major claims made in the main manuscript:
1. The relative singular values decay faster for Chronos and Chronos-Boff and they decay much slower for T5. The reason is that TSFMs leverage low-rank embeddings and do not require high-rank attention matrices, which is not the case for T5, which inevitably needs high-rank embedding.
In Chronos and Chronos-Boll when the layers get deeper, we generally have a higher-rank struc- ture. This aligns with the ofow-of-ranks idea: as an input is pushed through more nonlinear attention and MLP layers, its rank gets higher, and we need higher-rank attention matrices.
3. The attention matrices generally more compressible than MLP matrices. This is not sur- prising, because the attention matrices in these models are square matrices, making low-rank structures much easier emerge than in rectangular MI.P matrix.
32

---

Published as a conference paper at ICLR 2026
G.2 WATCHING THE WEIGHTS OF TSFMs DURING TRAINING
Another mysterious finding that used in the main manuscript is that the attention matrices in pretrained Chronos or Chronos-Bov model all demonstrate low-rank structure; however, while The- orem 3 only suggests that the attention matrices preclude these matrices from having a high-rank form. To understand why the attention matrices happen to exhibit low-rank structure, we watch the training dynamics of these attention matrices, where we pretrain a small Chronos model and record the six weight matrices in its six encoder layers few different training steps. Unlike Figure 10, in Figure 11, we do not show the relative singular values 01/01. The reason will be clear later.
be written in low-rank form, it does
Step 33
Step 436
Step = =3138
Step 7314
Step 10000
Step 50000
Figure 11:
The distribution of all absolute singular values in a weight matrix We a Chronos model over training. The model contains encoder layers, and we arrange the corresponding weight matrices in row-major order. Note that all histograms are made on a semilog-x scale.
We note that the initialization of Chronos typically relies on very small scaling factor. That is, the weights are all initialized to be small. From Figure 11, we see that the weight matrices are eventually learned to be larger: that is, by looking at the absolute singular values instead of the relative ones, we see that the leading singular value, i.e., the norm the weight matrix WQ, eventually gets larger. When they get larger, note that the low-rank structure evolves. This "learning the leading singular direction" interpretation is more plausible than if the weight matrix W Q is initialized large, because has no incentive to "forget the residual ranks" when does not need to.
G.3 WATCHING THE WEIGHTS IN A MULTIHEAD ATTENTION
In Figure 9, we showed the "correlation matrix" of a three-head attention matrix WQ- In Figure 12, show more of these matrices when different number of heads h. For each number of heads h, we can see clearly h low-rank blocks on the diagonal, corresponding to the in-head low-rank attention.
One interesting observation we can make that the off-diagonal parts of the heatmaps, while much more yellowish than the diagonal parts, also have block structures. This is expected: if the rows in each head are very colinear, then when you compare rows between two different heads, they should share similar angle. That is, we have duality that not only holds for attention matrices but also
33

---

Published as a conference paper at ICLR 2026
h 3
h = 6
h = 12
h = 24
48
h = 96
10-
16
10 140
Figure 12: We pretrain Chronos model with different number of heads h. As h changes, show the angle between every pair of rows Wq in the first encoder layer.
holds general: the darker a diagonal block is, the clearer the edges of off-diagonal blocks in the same row of blocks will be.
The off-diagonal parts of attention matrices, while much more orthogonal than the rows within each head, are still far from random. This means the heads themselves are also somewhat correlated, which is not surprising given that the row spaces query, key, and value matrices are subspaces of the potentially low-dimensional row space of the input and that the models are trained with single objective. There are two forces pulling against each other: a head-dependent random initialization, which "orthogonalizes" different heads, and head-ine seaddinneeeedeett training objective that tries to align these heads.
There is third trend (and the second duality) that is also very interesting: darker diagonal blocks seem to correspond to lighter off-diagonal blocks. There is one potential explanation for that: if the diagonal blocks input. In order to obtain good results, other heads must incorporate the remaining bits of the input, resulting in very different sketchings and larger angles. On the other hand, if the diagonal blocks are brighter, that means it already contains a fair amount of information about the input, and lots of the input information is shared across heads. Hence, they should be much more correlated during training.
very dark, that means each head only attends to tiny bit of information of the
34

---

Published as a conference paper at ICLR 2026
H VISUALIZATION OF TOKENIZERS
In section 2, we consider a large corpus of inputs. Here, we perform two case studies to look into how each of the embeddings works. In particular, we select two time-series input data:
A sinusoidal wave sin(t).
A random Gaussian input, where each entry is i..d. (N 1).
Let X E _d be the embedded input. For each pair of vectors x; and x, of X, we compute the their correlation:
x
8. arccos
x
||,
These correlation matrices are shown in Figure 13. There are interesting observations make: first. for WokenToken, the correlation matrix given a sinusoidal input has clearly four blocks corresponding to the low-frequency wavelets and high-frequency ones. For Time MOE, the angles are so much darker because we apply continuous embedding on one-dimensional input space, leading to an ultimately low-rank structure.
Chronos
Chronos-Bolt
WaveToken
Time MOE
random
sinusoidal
SVD of Random Input Embedding
SVD of Sinusoidal Input Embedding
chronos chronds zutt
10
fine Mice
o,(X)/oi(X)
o,(X//nxXX) (X)
22'44
12'
1044
10m4
]
Figure The heat maps show the correlation matrix of the embedded input with a random context or sinusoidal one, using four different embedding strategies. The two line plots show the relative singular values of the embedded matrix X.
We also observe that the numerical rank of the embedded input X is generally higher when the context is random than sinusoidal. This is not surprising either, because the temporal relationship within a random context much more complex than that in sinusoidal one.
35

---

Published as a conference paper at ICLR 2026
1 MORE ON THE CHEBYSHEV EMBEDDING
In Figure 4, we show an experiment where we increase the rank of a fixed input embedding and watch the numerical ranks of pretrained Chronos models with that embedding. Here, we further ex- plain how we can control the rank of this fixed input embedding function using Chebyshev polyno- mials. To motivate our design, we first consider how we can compute rank-1 embedding function D1 I + F. In this case, what Φ needs to do is to map the real line linearly onto a one- dimensional subspace of F That is, we can set $ (z)) = zu for some fixed unit vector R
Now, how can we use this idea to design rank-k embedding (x(x)) One way to do that is by considering the following extension of &;::
2 : R
I f + fa(
I
As long as the functions f1, fi..... fK are linearly independent, the image of the real line, & (& is subset of k-dimensional subspace of Rd. The only question that remains is: how to choose (x). Perhaps the easiest way to choose such basis by making them monomials: [x()) x/. However, the monomial basis often suffers from many numerical stability issues and is not ideal for the embedding, e.g.. for large j. the set (z..... x²) } is very ill-conditioned. To choose a well-conditioned basis, we use Chebyshev polynomials, which are orthogonal on [-1, That is, set (z) ( where maxx is the maximum number considered in quantization, which equals 15 in the case of Chronos. Given a set points Ti,..., JL to embed, we assemble the embedded matrix X E gdx as follows:
1. Sample orthonormal random column vectors .,,..., ug E IR RdX
2. Compute row vectors fl.... ,.....,f,, fg E RRI2L, where the ith entry of fj is TSTT (Ti//ummxx)
3. Compute the outer product X = Σ n,f
We show the visualization of our Chebyshev embeddings in Figure 14, where as that as increase the rank k, the embedding becomes more "complicated."
degree = 1
degree = 2
degree 3
350
150
degree =
degree = 32
100
-000
180
280
Figure 14: Visualization of the Chebyshev embeddings. For each embedding, we visualize the first two dimensions out of the 768 in the hidden space. There are 4096 points embedded in the hidden space, whose corresponding values the input domain range from 15 to 15.
36

---

Published as a conference paper at ICLR 2026
J NUMERICAL EXPERIMENTS TO CORROBORATE OUR THEOREM
In this appendix, provide numerical experiments, simulated in MATLAB R2024b, that verify the theoretical statements we made in the main manuscript.
J.1 TWO NUMERICAL EXPERIMENTS ON THEEREM
To verify Theorem 3, we fix the hidden dimension to be d - 512 and the vocabulary size N = 4096. We create random embedded matrix E Rx with singular values s E IR ranging from e e-6 to el and uniformly distributed on a logarithmic scale. This matrix is computed by randomly sampling an orthogonal matrix U Rdxd and matrix V E Rdxd with orthonormal columns, via QR- decomposing random matrices, and setting X = U diag(s) VT.
Next, we randomly sample three attention matrices Wq. WK. Wv, and for each reduced-order = I.....d d - 1, we use the constructive formulas given in the proof of Theorem 3 to compute the reduced matrices We. Wk, Wv. We set the input to be the entire vocabulary matrix X and compute the output of the original attention layer, defined by Wq. WK. Wv, as well that of the reduced one, defined by We. W K. Wv. We evaluate the Frobenius norm of the difference between the two outputs.
10
5x10
2
3x10
8x10
10
10-8
10.6
10
3
1
10
50
100
300
1000
d
0+1
Figure 15: The left panel shows the relationship between the reduced-order and the approxima- tion error of randomly sampled attention layer applied to randomly sampled input matrix X with controlled singular values o1....; Od. The reference line has a slope in the log-log plot. The right panel shows the relationship between the dimension the hidden space d and the approxima- tion error of a fixed-degree reduced-orde model. For each hidden space d, we randomly resample the attention matrices and embedded matrix X, holding leading singular values unchanged. The reference line has slope of 1/2 in the log-log plot.
Figure 15 shows two controlled experiments by changing different variable. On the left, change the truncation degree d, which in turn controls the singular value 0 The reference line has slope of 1, revealing a linear relationship between Y - and 1+ as indicated in Theorem 3. On the right, we change the size of the hidden space d, and the reference line has slope of 1/2, which verifies the vd factor in the statement of Theorem 3.
J.2 A NUMERICAL EXPERIMENT ON THEOREM
The essence of Theorem 6 is that sparse multi-head sketching is more effective than dense single- head sketching. We use an experiment to verify that. In our setting, we set d = 2048 and L 4096, and randomly sample input matrix X E dd with exponentially decaying singular values, as shown in the left panel Figure 16. Our sampling method is the same as the one outlined the previous experiment.
37

---

Published as a conference paper at ICLR 2026
We set our reduced rank = 4. If we just use single head sketch rank-d space from the row space of X, then its accuracy power-bounded by d which is still a large number. To explore the potential of sparse head-dependent sketching, increase the number of heads from h 1 to 128 and keep it an integral divisor of d. For each h, we use the sparse sketching; that is, we assemble random matrix
W2,6 2.4
w (1).
W (1).T 2
W 2.6
E dxd
The matrix W2,4 is sparse in the sense that it has exactly d, = d/h non-zero entries, whose posi- tions are randomly chosen, and for each non-zero position, we sample value i.i.d. from ((0, 1). Note that given this sparse design, 2,,4 has the same number of non-zero entries for any h.
Now, our question is: what the different between the row space of 2 X and that of words, how good is the sketching using W2,2? To this end, we clearly have that R (22,, 2, C RX). where we use the notation for the row space of matrix. Hence, the remaining question how much in R(X) is not filled by R(W2, This can be measured by projecting R(X) onto R 2, X) and measure the loss by the projection:
other
d(R(X), R(W 2, x.) ||orth(X WT.)) orth Taoh(xTWiiaa W X - -
(30)
We show this distance in the right panel of Figure 16, and we relate this to the singular values of X. In that sense, since the vertical rules in the left panel almost evenly spaced, shows that the "effective rank" of R (22, X) grows proportionally with respect to h, indicating that the quality of a sparse multi-head sketching comparable the quality of dense multi multi-head sketching.
10
R 22
fa (X)
1011
10
X
2
10
10 10
102
103
10
10
102
j
number of heads h
Figure 16: On the left, we show the singular values of the input matrices. On the right, we compute the "sketching error" defined in eq. (30), as we change the number of heads sparse sketching. We map the sketching errors back to the singular value plot to indicate the index such that our sketching error achieves an error below aj+1-
J.3 A NUMERICAL EXPERIMENT ON THEOREM
Our final numerical experiment considers the flow-of-ranks. Theorem 4 suggests that larger num- ber of heads also facilitates the flow-of-rank, and this is hard validate empirically with pretrained Chronos models. In our targeted numerical experiment, we fix an input matrix X with predefined exponentially decaying singular values, which is, again, randomly sampled from the product of random orthogonal matrix, predefined diagonal matrix, and the transpose of random matrix with orthonormal columns (see the previous two subsections). Then, we randomly select Wo. W.. Wy and compute the output of the attention mechanism defined by these three matrices together with number heads parameter h. Figure 17 gives us that the output matrix
Y MH-Attention X; Wo, W K, W v. h)))
38

---

Published as a conference paper at ICLR 2026
higher-rank h increases.
10
10
(Y) (Y)
10
10
Y (A-1)
(8=2)
10
(R=1)
Y (A-8)
(h=16)
10
j
Figure 17: Relative singular values of the input matrix X and output matrices Y MHH-Attention(X, WQ. Wk, Wv, h)) + X we change the number of heads h. We see that the numerical rank of increases with h.
39

---

Published as a conference paper at ICLR 2026
K ADDITIONAL EXPERIMENTS AND DISCUSSIONS
Pretraining a Compressed Chronos-Bol Bolt and a Compressed Moirai. In addition to Chronos, we also pretrain a compressed Chronos-Bol model. To show the promise of compression, we start with an already-small Chronos-Boit (small) model, based on the T5 (small) architecture. The table and figures in Table can be read in the same way as those in Figure 7. In particular, we see that for both MASE and WQL, the compressed Chronos-Bol models completely form the Pareto frontier on our evaluation benchmark. That is, given any local method or pretrained foundation model, there exists a compressed Chronos-Bov model that is simultaneously faster and more accurate.
Table 3: Results of pretraining a compressed Chronos-Bol (small) model. We compare the perfor- mance scores relative to the original pretrained model. The last row is the baseline.
Inference
In- Domaonn
Zero-Shot
dg
Time
Space
WQL
MASE
WQL +
MASE +
0.25
0.517
0.802
1.005
1.004
1.013
0.999
3
0.50
0.601
0.821
1.005
0.996
1.009
1.001
0.70
0.740
0.840
0.980
1.003
0.991
1.010
0.00
1.000
1.000
1.000
1.000
1.000
1.000
14
Ohure
1.0-
Scaronal Navv
4g Uame
MANRTS
MASE
WQL
30 Bood
Maiie o Jaace
Moini L0k cargel
Chrona Strati
Serah
Ceceesizet Sosee
Chescz Laage
Ovente Largo
inference time
inference time
While it is hard to find rule of thumb that works for any task, we propose guideline that should work fine in most cases. Let D be the number of layers and the hidden size. Set the per-layer target rank
r = r (rD - (:)
(=,,...,
where we can set Tmin the median numerical rank of a small sample of input embeddings (or 16 if unknown), Tmax = d/2, and 7 = 0.5. This design guarantees that 7ę grows smoothly and monotonically from Tmin to max (concave in depth), aligning capacity with the observed flow-of- ranks while keeping early layers compact.
Table 4: Results of pretraining a compressed Moirai- 1.aii-1..-----reee model. We compare the perfor- mance scores relative to the original pretrained model. We show prediction losses when the flow- of-ranks is or is not used. The last row is the baseline.
With Flow-of-ranks
Without Flow-of-ranks
Size Ratio
dg
WQL
MASE 4
dg
WQL | MASE
0.250
8
0.34
1.001
1.014
0.00
1.069
1.050
0.500
10
0.58
0.996
1.007
32
0.00
1.038
1.036
1.000
64
0.00
1.000
1.000
40

---

Published as a conference paper at ICLR 2026
L THE USE OF LARGE LANGUAGE MODELS (LLMs)
LLMss are used for polishing the writing and word choices of a few sections in the main text. They not used in the conceptualization and implementation of research.
41


Reviews:
[{"review_id": "DFZLKVs7FC", "reviewer": "Reviewer_Qkww", "summary": "This paper presents a rigorous linear-algebraic analysis of time series Transformer models. It first examines the ranking structure of time series embeddings, revealing unique low-rank characteristics that distinguish them from other modalities. The authors then theoretically infer the potential low-rank nature of the attention matrices in time series Transformers. Meantime, the authors introduce the concept of “flow-of-ranks” to describe how representation ranks evolve and increase across Transformer layers due to nonlinear transformations. Finally, leveraging these insights, the paper proposes two effective compression strategies for time series foundation models, achieving up to 65% reduction in inference time and 81% reduction in memory usage on the Chronos model, without compromising accuracy.", "questions": "1. Could the observed low-rank and compressibility properties be interpreted as universal characteristics of time series Transformer models? If so, does this imply that current architectures are inherently low-rank and may therefore lack sufficient expressiveness to capture more complex temporal dynamics?\n\n2. The success of pre-trained compressed model suggests that standard TSFMs are severely over-parameterized. Does this imply that the common practice of scaling up model size for time series is misguided ?  Is the low rank of TSFMs a inherant feature, or is it a sign that we are not yet challenging them with tasks of sufficient complexity?\n\n3. Theorem 3 suggests that the attention matrix may be less compressible when dealing with more complex or higher-rank input time series (e.g., noisy financial data). Are there datasets of this nature that could be used to empirically test the boundaries of the proposed low-rank assumption?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Strong Theoretical Grounding: Theorems 1 and 2 formally connect patch size and embedding smoothness to low-rank structure. Theorem 3 crucially links low-rank inputs to compressible attention layers, while Theorem 4 quantifies the \"flow-of-ranks.\" These theories provide novel insights to guide time series foundation model design.\n\n2.  The translation from theory to practice is seamless and compelling. Each theoretical claim is supported by corresponding empirical evidence, making the overall theory framework more convincing and practically relevant. And the results are striking, showing that TSFMs are significantly more over-parameterized than LLMs and can be compressed dramatically. The layer-dependent rank schedule is a simple yet powerful idea derived directly from the \"flow-of-ranks.\"\n\n3. Clarity and Organization: Despite the complex mathematical content, the paper is well-structured and readable. The flow from data structure to single-layer analysis to depth-dependent phenomena and finally to applications is logical and easy to follow.", "weaknesses": "1. The core experiments only focus on a single architecture family. All experiments are conducted on Chronos and Chronos-Bolt, which are based on the T5 architecture. While the principles are argued to be general, validation on other TSFM architectures (e.g., TimesFM, Moirai) would have further strengthened the claim of universality.\n\n2. The paper provides elegant existence proofs—such as the low-rank properties of time series embeddings and the W_Q/K/V matrices—based on the core assumption that time series embeddings are intrinsically low-rank. However, this assumption may be an artifact of current TSFM design choices (e.g., small patch sizes and simple MLP-based embedding layers). As more tokenization methods emerge (e.g., VisionTS [1], Wavelet-based Tokenization [2]), it remains unclear whether these conclusions will still hold.\n\n[1] Chen M, Shen L, Li Z, et al. Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters[J]. arXiv preprint arXiv:2408.17253, 2024.\n\n[2] Masserano L, Ansari A F, Han B, et al. Enhancing foundation models for time series forecasting via Wavelet-based tokenization[J]. arXiv preprint arXiv:2412.05244, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qmvMRtBcKv", "reviewer": "Reviewer_eiE9", "summary": "This paper investigates why Transformers trained on time series data behave differently from those trained on text or vision. The authors analyze the rank structure of embeddings and attention matrices to explain why time-series foundation models (TSFMs)—such as Chronos are highly compressible without losing much accuracy.\nSeveral key points include: time series are naturally low-rank, and flow-of-rank perspective.", "questions": "1. Can the authors explain footnote 1? Does it mean that if we have $n$-variate data, $x$ is then rank-$m$, where $n = m$? Is it possible that $m < n$?\n2. I wonder how naive Thm 1 and Thm 2 are? Since this paper mainly shows existence proof, if the input data is low-rank, it seems like its straightforward that we can find low-rank weights to model it. Are there any counterexamples?\n3. Are there any other features in the data assumption that makes it a time series? Or would the result hold for all low-rank input data?\n\nOverall, I think this paper will be a good contribution to the field and am happy to adjust my score if the authors address my concerns?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To my understanding, transformer theories in time series are generally lacking or provides limited practical guidance. This paper serves as a good entry point to understand how time series data differs from other modalities.\n2. The results are intuitive to me, making this paper easy to follow.", "weaknesses": "1. This paper considers univariate time series. which is limited as several TSFMs can handle any-variate inputs.\n2. The paper assumes input data X is 1-rank (or low-rank). I think this is a pretty strong assumption, which may not hold in many high-dimensional data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FNM5ozfmA2", "reviewer": "Reviewer_uH8K", "summary": "The paper titled “UNDERSTANDING TRANSFORMERS FOR TIME SERIES: RANK STRUCTURE, FLOW-OF-RANKS, AND COMPRESSIBILITY” analyzes Transformer models for specifically time series (TSFMs). They show that in this scenario, the transformer models ( as a consequence of the data being passed to it) possess a uniquely low-rank structure compared to similar architectures for text or vision. They postulate thus due to the continuous nature of time-series embeddings. This low-rank input structure leads to that the Attention layer matrices being highly compressible. The authors introduce the concept of \"flow-of-ranks,\" which describes how the numerical rank of a representation gradually increases with model depth due to nonlinear mixing, explaining why earlier layers are more amenable to compression. By leveraging these insights, the researchers demonstrate that large TSFMs like Chronos are severely over-parameterized and can be significantly compressed, achieving up to a 65% reduction in inference time and 81% in memory without losing predictive accuracy.", "questions": "na", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The low rank property of time-series may prove very useful in their application using transformers, in terms of designing models with fewer parameters. Theoretically, this low-rank property is proven for continuous embeddings, with guaranteed polynomial or exponential decay of singular values for smooth or analytic functions, respectively (Theorems 1 and 2).\nThe work provides the first general theoretical results (Theorem 3) connecting low-rank input embeddings to the compressibility of the internal Attention matrices (W_Q, W_K, W_V)\nThe paper introduces and quantifies the concept of \"flow-of-ranks,\" which explains how  non-linear components (like activations, residual connections, and normalization) across deep layers gradually increase the rank of a representation (Theorem 4)", "weaknesses": "The majority of the empirical validation and compression experiments focus almost exclusively on the Chronos family of Time Series Foundation Models. While there are references to other models the core compression techniques and deep rank analysis (flow-of-ranks, impact of heads) are primarily demonstrated on Chronos. This limits the generality of the practical findings and compression results to other TSFM architectures\nWhile the paper's core claims about rank structure are presented as a modality-dependent framework, the empirical evidence is often constrained to a small number of specialized experiments. How do we know this structure will prevail in other transformer architectures or other datasets.\nThe core theoretical analysis (Theorems 1 and 2) is derived for the univariate time series case. The authors mention in passing that this is extendible to few-variate time series but a detailed discussion is lacking", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xsT0RyL6Xl", "reviewer": "Reviewer_u1Ge", "summary": "This paper analyzes Transformers for time-series data through the lens of rank structure. The authors show that time-series embeddings are inherently low-rank, unlike those in text or vision. They prove that this structure induces low-rank attention matrices, introducing the concept of flow-of-ranks to describe how rank gradually increases with layer depth due to nonlinear mixing. They demonstrate that time-series foundation models, such as Chronos, can be compressed by up to 65% in inference time and 81% in memory without accuracy loss.", "questions": "1. How does the proposed compression compare quantitatively with existing low-rank or sparse-attention baselines (e.g., LoRA, Linformer)?\n2. Does the low-rank structure persist after fine-tuning a compressed model on downstream tasks?\n3. Can the flow-of-ranks pattern be empirically confirmed on other TSFMs beyond Chronos?\n4. How would the theory extend to multivariate or irregularly sampled time series?\n5. Could you provide a simple practical guideline (e.g., rank schedule formula) for designing TSFMs from scratch?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper combines linear-algebraic theory with well-designed experiments confirming the predicted low-rank behavior in TSFMs.\n2. The authors introduce a new analytical lens (flow-of-ranks) that connects data modality to model design. The findings have real design implications for TSFMs.", "weaknesses": "1. The main validation focuses on Chronos. Testing on other TSFMs (like TimesFM and Time-MoE) would strengthen the generality claim.\n2. The comparison to prior compression methods (LoRA) is missing, which makes it unclear how much gain stems from modality vs. technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)