Is isotropy a good proxy for generalization in time series forecasting with transformers?
Authors: Rashed Shelim
Shengzhe Xu
Walid Saad
Naren Ramakrishnan
Venue: Accepted by TMLR ()

Abstract: Vector representations of contextual embeddings learned by transformer-based models have been shown to be effective even for downstream tasks in \emph{numerical domains} such as time series forecasting. Their success in capturing long-range dependencies and contextual semantics has led to broad adoption across architectures.  But at the same time, there is little theoretical understanding of when transformers, both autoregressive and non-autoregressive, generalize well to forecasting tasks. This paper addresses this gap through an analysis of isotropy in contextual embedding space. Specifically, we study a log-linear model as a simplified abstraction for studying hidden representations in transformer-based models. In this formulation, time series embeddings are mapped to predictive outputs through a softmax layer, providing a tractable lens for analyzing generalization. We show that state-of-the-art performance requires embeddings to possess a structure that accounts for the shift-invariance of the softmax function. By examining the gradient structure of self-attention, we demonstrate how isotropy preserves representation structure, resolves the shift-invariance problem, and provides insights into model reliability and generalization. Experiments across $22$ different numerical datasets and $5$ different transformer-based models show that data characteristics and architectural choices significantly affect isotropy, which in turn directly influences forecasting performance.  This establishes isotropy as a theoretically grounded and empirically validated indicator of generalization and reliability in time series forecasting. The code for the isotropy analysis and all data are publicly available.

Published in Transactions on Machine Learning Research (10/2025)
Is isotropy a good proxy for generalization in time series forecasting with transformers?
Rashed Shelim
msheds@rttcc edu
Deportment of Electrical and Computer Engineering 8 Department of Computer Science
Virginia Tech
Shengzhe Xu Department of Computer Science Virginia Tech
shengzn@@ut. edu
Walid Saad
walids@ol. edu
Department of Electricol and Computer Engineering Virginia Tech.
Naren Ramakrishnan
naren@cs. vt. edu
Deportment of Computer Science
Virginia Tech
Reviewed on OpenReview: https: // openreview. net/ form? id= dUtDYVQ=Fg
Abstract
Vector representations of contextual embeddings learned by transformer-base models have been shown to be effective for downstream tasks in numerical domains such time series forecasting. Their success in capturing long-range dependencies and contextual semantics has led to broad adoption across architectures. But at the same time, there is little theoretical understanding of when transformers, both autoregressive and non-autoregressive, generalize well to forecasting tasks. This paper addresses this gap through an analysis of isotropy in contextual embedding space. Specifically, we study a log-linear model as a simplified abstraction for studying hidden representations in transformer- In this formulation, time series embeddings are mapped to predictive outputs through softmax layer, providing tractable lens for analyzing generalization. We show that state- of-the-art performance requires embeddings to possess structure that accounts for the shift-invarii invariance of the softmax function. By examining the gradient structure of self-attention, we demonstrate how isotropy preserves representation structure, resolves the shift-invarianco problem, and provides insights into model reliability and generalization. Experiments across 22 different numerical datasets and different. transformer--asee models show that data characteristics and architectural choices significantly affect isotropy, which in turn directly influences forecasting performance. This establishes isotropy theoretically grounded and empirically validated indicator of generalization and reliability in time series forecasting. The code for the isotropy analysis and all data are publicly available
models.
1 Introduction
Transformer-based models models have been proven effective across various downstream tasks in numerical
domains, such as finance (Garza and Saieentthaer-aanseee 2023 Yu et al. (2023) ), energy (Gao et al.
This work supported in by US National Science Foundation grants CNS-2225511, 25092509 DBI-2412369, CMMI- 2240402, IIS-2312794, and CCF-1918770. Any opinions, findings, and conclusions recommendations expressed in this material those of the author(s) and do necessarily reflect the views the sponsor(s). Corresponding author: Rashed Shelim (e-mail: rashed
httpss//github.. com.Shheggheeuustt..eoee trust, indicator traii

---

Published in Transactions on Machine Learning Research (10/2025)
(2024) climate science (Jin et al. (2024)), healtheare (Wang and Zhang (2024))
transportation signals Xu
et al. 2)244), synthetic tabular generation (Dinh et al. [0022] Borisov et al. 20233 Xu et al. (2024)) among
others. The significant success of emergent transformer based models in capturing long-range dependencies
and contextual semantics has led to their widespread integration across a wide variety of architectures. Several
methods have been developed recently
(Gruver et al. (2024): Dooley et al. (2023); Nie et al. (2023 | Rasu
ct al. (2024); Woo et al. (2024); Jin et al. (2024): Ansari et al. [2024]) that extend or apply transformertbasol models to numerical domains for time series forecasting. For many of these numerical downstream tasks, training a linear classifier on top of the hidden-layer representations generated by the transformer-basel models has been shown to achieve near state-of-the-art performance (Jin et al. (2024); Ansari et al. (2024))
However, the existing models in (Gruver et al. (024)); Dooley et al. (2023): ; Nie et al. (2023); Rasul et al.
(2024); Woo et al. (2024); Jin et al. (2024); Ansari el al. (024)) are treated as a 'black box' where numerical forecasts are controlled by complex nonlinear interactions between many parameters. This makes it difficult
to comprehend how models arrive at their predictions and raises fundamental challenges in assossing the
reliability and generalization of model outputs in numerical domains.
Most scientific domains, e.g., finance, healthcare, rely on machine learning models to exhibit prediction reliability. Thus, in such domains, it is necossary to open up the black box behind transformer-basee models and develop explanatory tools that can serve as good proxies for performance. Although recent empirical studies (Jin et al. (2024): Nie et al. (2023 Liu et al. (2024)) demonstrate the benefits of vector representations of embeddings learned by transformer- basedd models in various numerical downstream tasks, there is little theoretical understanding of their empirical success. This motivates the need for simplified yet expressive abstractions that enable the study of hidden representation geometry independent of specific architoctural details. Thus, fundamental question ariscs: "When (or how) do the hidden representations of transformer-bueed models exhibit structural properlies that enable reliable generalization in time series forecasting?'
The main contribution of this paper is a novel approach for answering this question by exploiting the isotropic property of conssoorrr--bsee model hidden representations in the contextual embedding space. Isotropy refers to the geometric property wherein vector representations in the embedding space are uniformly distributed in all directions, a characteristic critical for maintaining the expressiveness of the embedding space AAooa et al. 2016); : Mu and Viswanath (2018)). To achieve reliable generalization in numerical domains, we show that the hidden representations of transformer-based models must exhibit. structured form in contextual embedding space that accounts for the shift-invariance problem (Singla et al. (2021); Jacobsen itt al. 2020 Rojas-Gomaa et al. (2022) of the softmax function (i.c., the softmax output remains unchanged when all logits are shilted by a constant). Without such structure, the model can shift the logits while keeping the training loss unchanged, thereby leaving the logits ineffective for numerical downstream tasks. By formulating a gradient structure of self-attention in transformer-basee models, we show how the isotropy property of transformer-bast based model embeddings in the contextual embedding space preserves the underlying structure of representations, thereby resolving the shift-invariance problem of the softmax function. In a nutshell, our key contributions include:
We consider a log-lincar model (Arora et al. 2016): Mu and Viswanath (2018), Andreas and Klein (2015): Peters and Klakow (2000): Nelakanti et al. 2013)) as simplified abstraction to analyze the hidden representations of transformer- representations must exhibit structure to address the shift-invariance problem of the softmax function.
models and demonstrate theoretically why such
We take a deeper look into the hidden representations of transformer- based models and show how isotropy preserves the structural integrity of reprosentations. In particular, we derive an upper bound for the Jacobian matrix which collects all first-order partial derivatives of self-attention with respect to the input pattern and show that the largest. eigenvectors of the transformer-basedd model hidden representations minimize the gradient norm of self-attention. Then, by projecting the representations into lower dimensions using these m largest cigenvectors, we find the isotropy within the clusters in the contextual embedding space.
Finally, we provide a comprehensive evaluation across 12 real and 10 synthetic time scries datascts over 5 different transformer- based models. Our experiments demonstrate that the isotropy of transformer- based model hidden representations varies significantly based on the input data characteristics (i.e.,
2

---

Published in Transactions on Machine Learning Research (10/2025)
domain, context length and noise level) and model design choices (i.e., tokenization techniques and architecture), which in turn strongly influences forecasting performance in numerical domains.
2 Problem Setup in Numerical Domains
Overview of Section 2 In this section, a general problem setup is presented for transformer-based modeling of numerical time series, Time series inputs are tokenized and processed through self-attention layers, leading to contextual embeddings that are mapped to pre- dictive outputs. While the precise output head may vary across architectures (e.g., regression or categorical distributions), we adopt log-linear abstraction omelry. This abstraction enables Iractable foundation for isotropy analysis in the contextual embedding space. A numerical downstream task is also defined in terms of logits, casting time serics forecasting as a prediction problem on tokenized sequences. Together, these for- mulations establish the theoretical basis for studying the relationship between isotropy and forecasting performance.
Time series
MMM St
simplified model to study embedding go-
Tokenization techniques
Quantization, Patching etc.
Time series tokens
h2
Time Series Tokens and Similarity Measure. In transformer- based forecasting models, both autoregressive and non-autoregressive architectures can be viewed as learning predictive distributions over future values from historical context. Formally, given a time series X1.+ = [Z],..., ZT...., [,,....,T,...... .T++I. where
Figure 1: Time series tokenization.
the first T time instances give the historical context, the next L time instances constitute the forecast region, and 2, E IR the observation of each time instance, we are interested in predicting the joint distribution of I the next L time instances, ((X++1:7+7 (x:r). Since transformer-basse models operate on tokens from a finite vocabulary, using them for time series data requires mapping the observations to finite set of tokens. Based different numerical applications and caansfomeer-bseed model architectures, various tokenization techniques, c.g., quantization and scaling (Ansari et al. (2024); Rasul et al. (2024)), patching (Woo et al. (2024); Jin et al. (2024); ; Nie et al. (2023)) ), and adaplation of language model tokenizer in numerical domains (Gruver et al. 2004):
; Dooley et al. 2023)), ), can be applied to tokenize the time series and create a time series vocabulary V of N time scries tokens, i.c., |V| N, as shown in Figure Then, the realization of the next time instances can be obtained by computing predictive distributions p(kt k-)) for / E [I,...,I,, where k- denotes the tokenized input sequence and k; is time series token in vocabulary V.
Let V = [1) v be the set of all contextual embedding instances produced by a transformer-based model for the time series token k,. Here, different contexts in the time series sequences yield different selfatttention-based embeddings of kg By constructing Ekev L we define the inter-token cosine similarity
as:
Sens
A Bizz (cos(tss v))).
where v. and dyy are random samples from V The expectation is taken over all pairs of different tokens. This metric quantifies how distinct or overlapping contextual embeddings are across tokens, thereby providing a measure of isotropy in the embedding space.
Model. We consider a general model for numerical data and open the black box of the transformer based model. To analyze representation geometry, we adopt the log-linear model commonly used prior thcoretical
work AAora et al. (2016); Mu and Viswanath (2018); ; Andreas and Klein (2015); Peters and Klakow (2000); Nelakanti et al. (2013)). This adaptation is not intended to replicate the full transformer inference process, which also involves multi- head attention, feedforward layers, normalization, and residual connections. Instead, it serves as tractable abstraction of the softmax paramcterization commonly used in prediction heads. In this formulation, contextual embeddings (generated through self-attention and subsequent transformations) interact with token embeddings via inner products, and probabilities are obtained through normalization. This provides a mathenatical framework for studying isotropy in contextual embodding spaces. Formally, we define the conditional distribution of future time series token ky given tokenized input sequence k-; using woftmax-parameterinee log-linear form, typical of many prediction heads:
3

---

Published in Transactions on Machine Learning Research (10/2025)
pk k-) exp((#",(k-i), v8)),
(2)
where of E RR is token embedding corresponding to the observation 22, and (k-1) encodes the inpul soquence k- into contextual representation in RD. The notation (,,)) denotes the inner product between context and token embeddings. Moreover, we do not consider any prior distribution for input, which makes our model more general than previous latent models in (Arora et al. (2016); Wei et al. (2021)).
To define the numerical downstream task, let 2," (k, ([(,,))=<<>>(() (k_2(k-)), vir)) be the i-th logit of the ground-truth model, and assume that the numerical downstream tasks are defined by a function of the logits, i..., (z"). Also let (* E: exp(2; (k, ()) be the partition function. In this abstraction, the partition function normalizes the output probabilities. Then, the normalized ground-truth model ViE V is then
pr (K- Ie k-1 exp((V> " (k-1)) v1)) exp(2 (k,
Z (k,
(k,
Since we do not know the ground-truth model in reality, we do not have access to the ground-trut truth components 1/r and (k-)) Instead, we only have access to the trained model embeddings dh and 1 (k-((k-1) that aim to minimize training loss. We can then define the trained model's logits z(k, (()-))))>>((()) (( (.) } }= Intuitively, 2 are the contextualized reprosentations learned by the trained transformer-baser model during training. The downstream lask is to learn a function [(k; l). Finally, the output distribution of the trained model Vie V is
p(k= k-1 exp(2((k,1)
(3)
7(k,
Loss Function. While transformer- based forecasting models in practice are trained with a variety of objectives (e.g., mean squared error, likelihood-based losses, or cross-entropy), for theoretical analysis we adopt cross-entropy formulation. This abstraction allows to express the training objective compactly in expectation form and to connect it to KL divergence, thereby providing a tractable framework for studying how embedding representations relate to generalization and reliability. For tokenized input sequence k-1, the loss is given by:
4 = E& - 1² (k== = k))) log p(k) k-))]
Ex DKL((' p² (-)) ((k k [ + Ex HH(p (kal ki)]]
(4)
where p(-lk=:) is the categorical distribution predicted by the trained model, p (-Ik-1) is the ground-truth distribution, DKL is the KT. divergence, and H()- denotes entropy. This formulation highlights that minimizing cross-entropy is equivalent to minimizing KL divergence up to an additive constant given by the ground-truth entropy. Under large-scale training, the cross-entropy loss becomes small, implying that the KL divergence term is also small. This supports the common assumption that the trained model distribution approximates the ground-truth distribution in practice.
Downstream Numerical Task. We formulate time series forecasting as a downsstream task, in which a trained causformer-basee model is used to predict future values of a time series. While practical models often generate continuous outputs (e.g., through regression or lkkelihodd-based heads), for theoretical analysis we adopt categorical abstraction. Specifically, the continuous observations are mapped into a finite vocabulary of discrete tokens, so that forecasting can be cast as a conditional prediction problem over V. The downstream task is then defined as a function of the logits z 2k, I) generated by the ground truth model, where each logit corresponds to a token in V given the input sequence k,,.. For interpretability, one can consider a linear predictor that operates directly on the contextual representation (k=): f"kk = (w=,(kk)): u²)
at a (ks I). where u² E of E R and at are coefficients. Although this linear form provides intuition about how logits contribute to predictions, it does not guarantee reliable generalization, as KT. divergence is less sensitive to sign changes in f((k, when logits have small magnitude Wu et al. 22023). To improve robustness, we instead model the downstream task using a nonlinear activation on the logits:
[
f* Σ aj b3)
aj (WV2, (k_)) vt) -b))
4

---

Published in Transactions on Machine Learning Research (10/2025)
where is the ReLU function and by is threshold paramcter. This abstraction suppresses low-confidence logits that contribute minimally to KT, divergence, thereby focusing the task on high-relevance predictions. While not intended as a literal description of forecasting architectures, this formulation provides a theoretical device for analyzing robustness and highlights how isotropy in the embedding space supports reliable gencralization in time series forecasting.
Despite the empirical success of transformer- approaches in numerical domains, there remains a fundamental gap in understanding when these models generalize reliably. A key challenge lies in the mismatch between training and inference performance, i.e, good training performance does not necessarily translate to robustness at test time. To address this, we introduce a theoretical framework based on isotropy in the contextual embedding space. Within this framework, strong isotropy in hidden representations stabilizes the partition function and mitigates the shift-invariance issue observed in noftmax-based abstractions, thereby leading to more reliable inference. The next section formalizes this insight and provides a theoretical justification for using isotropy as a key indicator of model reliability in time series forecasting.
3 The Role of Isotropy in Transformer-Based Models for Time Series Forecasting
Overview of Section 3 This section develops a theoretical perspective on why isotropy in contextual embeddings is critical for reliable generalization of transformer-based models in numerical forecasting. Although strong performance is often observed during training, it does not always transfer to unseen scenarios, particularly across heterogeneous numerical domains. To explain this mismatch, we introduce framework that links isotropy of hidden representations to the stability of the partition function. We show that isotropy mitigates the shift-invariance issuc in softmax-based abstractions, thereby enabling more robust generalization. This establishes formal connection between isotropy and inference reliability, motivating its use as an indicator of model performance in numerical domains.
As discussed in Scction 2 we consider transformer-based forccasting models where outputs are often parameterized through lask-specific heads (e-g., regression, likelihood, or classification). For theoretical analysis, we adopt a softmax- baseed formulation as an abstraction to connect logits and probabilities, enabling isotropy analysis. The relationship between logits and this softmax-based abstraction directly impacts downstream forecasting performance. However, the softmax function is inherently shift-invariant, i.c., its output remains unchanged if all logits are shifted by the same constant. Shift invariance has been extensively studied in the deep learning literature (e.g.. Singla et al. (2021); : Jacobsen et al. (2020); Rojas- Gomez et al. (2022) and is recognized as a practical concern across various domains such as NLP and vision. For instance, (iiglla et al. (2021]) shows that transformers can be sensitive to input shifts, and (Jacobsen et al. (2020) shows that such invariance can distort representation geometry and affect generalization. In time series forecasting, this implies that unless hidden representations enforce structural stability, outputs under the softmax- based abstraction may remain unaffected by meaningful representational changes. Since we cannot control the logit shifts of a trained model on unseen data, strong predictive performance during training does not necessarily imply reliable generalization in out-o--distribution scenarios. This lack of reliability due to uncontrolled logit shifts is formalized in the following theorem:
Theorem 1. Let the logils of the fround-trulh model be bounded. Then for any (k(k,),, there exists set of functions (k, D) such that for all k and T1+,, the predictive distribution of the trained model & (kt k-1) matches that of the ground-truth model p" (kr 1-) while [k:, 1) = O. In other words, there exists a trained model with the same training loss as the ground-trus truth model, but with logits ineffective for rumerical
downstrram lasks.
Proof. The proof is provided in Appendix A
Theorem I highlights that without additional structure in the hidden representations of transformer-based models, the logits can be arbitrarily shifted while preserving training loss, rendering them ineffective for forecasting tasks. Thus, theoretical guarantees for downstream performance require embedding spaces that enforce geometric stability.
To prevent the invatinnv problem from undermining forecasting reliability, it is necessary to stabilize denote the contextual embeddings of the tokenized
the partition function. Let v = (w-,,...., ((p))) E
5

---

Published in Transactions on Machine Learning Research (10/2025)
time series. Within our softmax-baset abstraction, the stability of the partition function can be assessed
through the isotropy of the embedding space (Arora et. al. (2016) Mu and Viswanath (2018))
(((+))) ming.ee 2(k,
55)
max Z(k,2)'
where C= T L is the input correlation matrix of input patterns and 1,.... L. From equation we can see that when the partition function is constant (i.e., stable) for different samples, I((x))) becomes close to 1, which indicates that the contextual embedding space is closer to isotropic and thus geometrically stable
5
(Arora et al. (2016); ; Mu and Viswanath (2018)). In this abstraction, probabilities depend consistently on
logits without being distorted by global shifts, thereby mitigating the softmax shift-invariance problem.
Building on this theoretical foundation, we now turn to the following empirical question: "How can isotropy be quantified and interpreted in practice, and how does relate to forecasting generalization across numerical domains?" Motivated by Theorem 1 we analyze structural properties of hidden representations, such as effective dimensionality and cluster organization, in transformer-baxee abstraction. These analyses reveal how isotropy manifests in contextual embeddings and how its presence correlates with generalization ability across diverse numerical domains, thereby providing practical indicator of model reliability. Section g introduces quantitative methods, including spectral alignment, principal component analysis (PCA), and cluster-based isotropy metrics, which link theoretical reliability to empirical generalization.
models under our max-baxe
4 Study of isotropy in transformer-based models hidden representations
Overview of Section 4 This section builds on the theoretical foundation of isotropy to address the empirical question of how isotropy can be mcasured and interpreted in practice, and how it relates to generalization across numerical domains. It introduces techniques such as spectral alignment, PCA, and cluster-based isotropy metrics to examine the structural geometry of hidden representations in transformer- based models. These methods uncover how isotropy manifests in model embeddings and how its presence correlates with forecasting performance. Through this analysis, the section establishes a crucial link between theoretical reliability and empirical generalizability.
Analysis Settings. For this study, consider five different transformer- based models including Chronos- T5 (Ansari et al. (2024)), Chronos- Bolt PStchTST (Nic et al. (2023) ), Moirai-1.0-R (Woo et al. (2024)). and Llama Rasul et al. (2024)). ). For illustration, we randomly select real dataset (i.e., finance- Dataset 1) from a broader collection of 22 numerical datasets that we use in this paper since we see similar results with all of these datasets. The details of these models and datasets could be found in Section
4.1 Effective Dimensions
In each layer of each model, we start with a data matrix A E R xD where [V] rep- resents the number of tokens in the input time series sequence, and D corresponds to the embedding dimension. We apply PCA to reduce the dimensionality from D to m i.c., A E xm Then, the fraction of vari- ance captured by the reduced representation 5m-1
Table 1: The effective dimension d(0.8)
Layer
1
2
3
4
5
6
8
9
10
11
12
Chronos- T5
Chronos-Boo Boll
1
1
1
1
1
1
1
1
1
PatchTST
Moirai
1
1
1
Llama 2
2
is given by: Tm E where a; de-
notes the i-th largest cigenvalue of the covariance matrix of A. We define the c-effective dimension d(c) - arg minm I' 2 c. For instance, if d(0.8) = 3, then three principal dimensions retain 80% of the variance. A higher d suggests a more isotropic space (Cai et al. (2021)), where information is spread across multiple dimensions rather than being concentrated in a narrow subspace. Table D presents the values of d(0.8) for different layers and modcls. Surprisingly, all of these modcls have very small effective dimensions
htttp::/hhugginggface.e http:///ggaaaeeeee/c///o/////cca

---

Published in Transactions on Machine Learning Research (10/2025)
compared to original embedding dimensions. For instance, Chronos-Boo has very small effective dimensions, with d(0.8) = 1 for layers I through 12, as compared to its original embedding dimensions D = 512. The small effective dimensionality is another way of telling that that Chronos- Bolts's embedding vectors lie in a subspace defined by a very narrow cone (Ethayarajh (2019)), and consequently, their inter-token cosine similarity is large. If all the embedding vectors lie on a 1-dimensional line, the inter-token cosine similarity would be close to 1, and there would be hardly any model capacity. Surprisingly, despite having such low effective dimensionality, these transformer-basee models still perform well in numerical domains. This counterintuitive result motivate to look deeper into the contextual embedding space.
4.2 Spectral Alignment for Generalization in Numerical Settings
Let G(V)) (1) softmax( T where A= We W E R and W WR are the parameter matrices for the query and key matrices of self-attention. The lemma below provides insights into how the isotropic property of mansformer-bassee models enables generalization in numerical domains. The proof of this lemma follows the analysis in (Kim ct al. (021)) and is provided in Appendix B for completeness.
BIV xD RI be the function for self-attention, i.e., 9i(
Lemma 1. Consider the Jacobian matriz _ which represents the gradient of the self attentttnnion mapping
G()) with respect to the input time series token embeddings. Then the spectral norm of J satisfies ||J |
| MM (pL, + 2)) db E) Pigul +A.
The residual term A and the station weight Pt,j is defined in Appendix B For notatioanl simplicity, we express the term Σn pbs Σ} P in Lemma as r. From Lemma we can see that, in order to minimize the norm of the gradient [|]] we essentially need make I small. When A is small and all the input time XP series token embeddings are centered at the origin, 1 eb =0, we have I ~ MM 1 - v u (see Appendix
Next, we prove that A minimizes the objective T and contains the m largest eigenvectors of correlation matrix V vp of time series token embeddings, where m is the rank of A.
Theorem 2. Let the eigenvalues of the correlation matriz v Th let E R for i = 1..... D denote their associated eigenvectors. Then, the matrix A that minimizes the quantity I has the optimal form A Em
vp be ordered as x1 2 22 2.... AD. and
Proof. The proof of Theorem 2 is provided in Appendix
Theorem 2 shows that the self-attention mechanism effectively projects input time series tokens onto a low-dimensional contextual embedding space defined by the top eigenvectors of the correlation matrix & T v. This result reveals that the self-attention mochanism in transformer- based models implicitly aligns with the dominant directions (i.e., top eigenvectors) of the contextual embedding space, and hence, suggesting that isotropy is not just geometric artifact but learned structural property that supports effective generalization to numerical downstream tasks.
While the self-attention aligns imput representations with the dominant eigenvectors of the embedding space, the alignment may vary across different subregions of the contextual embedding space due to variations in the input sequences, token types, or contextual patterns. As a result, the degree of isotropy may differ across subregions of the contextual embedding space, which motivates the need to assess isotropy at local (i.e., cluster) level rather than relying solely global metric. The next section explores these local structural patterns and examines the geometry of the hidden representations through principal component analysis (PCA), which helps reveal how variance is distributed across embedding dinensions.
M
Thh Jacobian matrix J 0g. 90,
represents the gradient of the self-attention mapping G(P) with respect to the
s-==
input time series token embeddings.
7

---

Published in Transactions on Machine Learning Research (10/2025)
var ratio 0.52
var ratio 0.94
var ratio 0.97
var ratio 0.83
(a) Chronos-T5 layer 8 (b) Lag-Liama layer 8
(c) PatchTST layer 2
(d) Moirai layer 2
Isolated or slightly overlapping cluster islands exist in the contextual embedding space for all models. For brevity, we only show few representative middle layers from each model.
Figure 2:
4.3 Clusters in the Contextual Embedding Space
Motivated by the results of Lemma [ and Theorem 2 this section investigates local structural patterns by projecting the transformer- based models' hidden representations into lower-dimensional space using the top m=3 eigenvectors via PCA, as shown in Figure ? The three axes of the figure represent. the first three principal components of the covariance matrix of transformer-base based model representations of each layer. For instance, in Figure 2 and] 1 the first three principal components account for 94% of the total variance in layer 8 of Lag-Laan Llama and 83% in layer 2 of Moirai. From Figure 2 2
b. 2 c and 2 d, we can see that there are disconnected or slightly overlapping islands that are far away from each other. In equation 1 the space isotropy is measured on pairs of arbitrary time series token representations, which could reside in two disconnected clusters. However, given that the variance is dominated by distances between clusters, such estimation would be biased by the inter-cluster distances. Hence, it is more reasonable to consider a per-cluster (i.c., local) investigation rather than global estimate.
Isotropy within Clusters. We start by performing clustering the transformer-based model represen- lations in the contextual embedding space. There are various methods for performing clustering, such K-means and DBSCAN algorithm (Ester et al. (1996)). We select K-means clustering method because it is reasonably fast in high embedding dimensions. We use the classical silhouette score analysis (Rousseeuw (1987)) to determine the mumber of clusters |C] in the contextual embedding space (see Appendix D for details). Since each transformer-based model contextual embedding instance tà belongs to a particular cluster through clustering, the cosine similarity should be mcasured after shifting the mean the origin (Mu and Viswanath (2018)). Accordingly, we subtract the mean for each cluster (i.e., centroid) and calculate the adjusted Seos in Section 2 Assuming have total of |C] clusters, let &; {bi v. ....... be the set of token h; 's contextual embeddings in cluster E C, and vis be one random sample in D. We define the adjusted inter-token cosine similarity as
Soos E
Bizj foos (Vr., [1,
(6)
where J. 1/. Es, [ Here E. is the average over different clusters, and v. is the original contextual embedding shifted by the mean, with the mean taken over the samples in cluster c {Kim et al. (2021) ). The inter-token cosine similarity takes values between -1 and 1. A value close to 0 indicates strong isotropy and ensures the existence of structure in the transformer-based model representations. The stability of the partition function 7(k, depends on balanced inter-token similarities (i.e., strong isotropy) in the contextual embedding space. However, as shown in Section 4.2 hidden representations often form disconnocted or weakly overlapping clusters where global isotropy can be misleading due to the variance being dominated by large distances between cluster centroids. By analyzing local isotropy within each cluster in equation 0 meaningful intra-cluster geometry can be captured which ensures that no cluster disproportionately skews the partition function normalization. This leads to more stable and interpretable model outputs.
To put it in a nutshell, this section provides a theoretical foundation showing that self-attention projects input tokens onto low-dimensional subspace aligned with the dominant cigenvectors of the embodding correlation matrix. This alignment induces isotropy in transformer-basee based model hidden representations, stabilizing the partition function and preserving the structure needed for reliable numerical downst task performances.
8

---

Published in Transactions on Machine Learning Research (10/2025)
In Section 5 extend this analysis by empirically evaluating how isotropy in different transformer- models' hidden representations correlates with time series forecasting performances across a wide range of numerical datasets, varying context lengths, and noise levels.
5 Experiments
Overview of Section This section empirically validates the theoretical insights developed in the preceding sections throngh a combination of controlled and broad-based experiments. We begin with a matching controlled experiment that implements a simple transformer- basedd model with a softmax output head trained under cross-entropy loss which reflects the assumptions of our theoretical framework in Section [ This serves a sanity check that bridges theory and practice, confirming that isotropy is tightly linked to forecasting reliability when the model setup matches the analytical formulation. We then extend the evaluation to diverse transformer-based models and datasets, analyzing both synthelic (qualitative) and real- world (quantitative) domains. These broader experiments investigate how factors such as model architecture, tokenization strategies, context length, and noise influence the isotropic structure of learned representations. Across all settings, we find that higher isotropy consistently correlates with better generalization, reinforcing its utility as a reliable, label-free diagnostic for the robustnesss of transformer-based models time series forecasting.
Baselines. We consider popular mannssfomer-bbssee models as the baselines for numerical downstream tasks, including Chronos-T5 (Ansari et al. (2024)) and Chronos-Bolt Bolt (https:/ (htttps/hhuggiggeee http///ggggaaa.co////aaaa// www.a//glgoooo////ioo.oooo /chronos- bolt-base), PatchTST (Nie et al. 2023 Moirai-1.0-R (Woo et al. 20224 ) and Lag- Llama (Rasul et al. (2024)) )
The considered models use different architectures, time scries tokenization techniques and hyperparameters for numerical downstream tasks. For instance, Lag-Llama use decoder only transformer, PatchTST and Moirai-1.0-R use vanilla Transformer encoder, while Chronos-T3 and Chronos-Bol use encoder-decoder transformer. Different baselines achieve contextual embedding in different ways. For example, PStchTST focuses tokenizing time series patches and usee self-attente attentitnnoo for modeling dependencies within each patch and across patches, while Chronos-T5, T5 and CHRONOS Bolt adapt language modeling architectures minimally and generate categorical tokens by applying scaling and quantization. The details of these basclines are sunmarized in Table 2 We lake the released weights of of the baselines and apply them directly to our experimental datasets without fine-tuning. This setup allows us to study the generalization of transformer- basod models in time scries forccasting, since models trained once are evaluated on unscen time scries datasots. In this context, the isotropy of embeddings reflects how well transformer-based models generalize their forecasting capability to numerical domains.
Table 2: Transformer-based models, architectures, time series tokenization techniques and hyperparameter choices. L stands for context length, d, for hidden layer dimension, ny for number of layers, nH for mumber of heads, and for learning rate.
Model
Architecture
Tokenization Technique
Hyperparameters
Incoder-Decoder with autoregressive forecasting
Chronos-T5
Scaling & Quantization
Default
Deooder with multi-step forecasting
Chronos-Bolt
Scaling & Quantization
Default
PatekTST
Vanilla Encoder
Patching
Patch lengthe 16, Stride: 8, d, = 32, , = 2, nM =
Moirai
Encoder
Patching
I 1024, Patch length: selected by dataset-speelfic validation
Llama
Decoder
Lag Feature
32
Datascts. We conduct a comprehensive evaluation using 12 different real time scries datascts from various numerical domains, including energy, nature, finance, healthcare, retail and transportation. The sources of these open-source datasets along with their descriptions, including how each dataset used across different transformer-based model can be found in Table of Appendix
We also illustrate our findings using E in Appendix for details), method that generates 10
KernelSynth (Ansari et al. (2024)) (sce Algorithm additional synthetic datasets via Gaussian processes in Section
5 We select two different datasets from each

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Real and Synthetic Datasets
Data Subset
Domain
Dataset
Dataset 2
Energy
Australian Flectricity - Queensland State
Australian Electricity South Australia Rainfall
Weather
Solar Radiation
Finance
Exchange Rate
NN5 Weekly Cash Withdrawals
Real Datasets
Healtheare
Hospital Putient Counts
COVID-19 Deaths
Trausportation
Transportation Signaling
Trausportation Signaling
Retail
Car Sales
Dominick
Linear
DotProduct kernel (C=0)
DotProduct kernel (C=1)
seasonality
seasonality kernel (period = 0.5W)
seasonality kernel (period = 0.25H)
Synthetic Dalasets
Trend
RudioulQuddratic kernel (a 1)
Kutionuluadrrati keruel (a = 10)
Nou-Linear
RBP kernel (length scale = 0.13
RBF kernel (length scale 1]
Stochastic
WhiteKernel (noise level 0.1)
WhiteKernel (noise level 1)
numerical domain (as shown in Table 3) and then perform qualitative analysis with synthetic datasets and quantitative analysis with real datasots. The results these analyses are provided in the next two soctions.
5.1 Matching Controlled Experiment
To bridge our theoretical framework in Section 2 with the empirical evaluation in Section 5.2 and Section 5.3 in this section, we conduct a matching controlled experiment using a simple transformer-base based time series forccasting model closely aligned with our problem formulation. Specifically, we adapt an open-source vanilla transformer (Vaswani et al. (2023)) implementation for time series forecasting by modifying its output layer and loss function to match our theoretical assumptions. This controlled setup allows us to evaluate isotropy under conditions that directly reflect the log-linear abstraction and KL divergence formulation introduced in Scction 2
The controlled model consists of two standard transformer encoder layers, each comprising multi-head self-attention with 10 heads, residual connections, layer normalization, and a feed- forward network. The architectural details are chosen to align with our problem formulation in Section 2 We introduced the following adaptations:
Model Architecture.
Input tokenization: Raw numerical time series data is discretized via uniform quantization into V = 512 bins. Each quantized value represents time series token, forming categorical vocabulary consistent with the log-linear model in Section 2 These tokens are then embedded into 250-dimensional vectors, and sinusoidal positional encodings are added to preserve temporal order. This transformation converts the continuous sequence of values into a sequence of categorical tokens suitable for the transformer, thereby ensuring that the log-linear abstraction in equation and 4 is realized in practice.
KL-divergence formulation equation
Transformer encoder: A two-layer transformer encoder (n- 2), where cach layer contains multi-head self-attention with 10 heads (ny = 10) and a feed-forward block (d, 250). Softmax normalization is applied within each attention layer, matching the theoretical derivation in Section 2
Output projection: The contextual embeddings are mapped the vocabulary size |V] 512 through linear transformation, followed by softmax layer to yield predictive distributions over quantized values.
De-tokenization: At inference, predicted token distributions are mapped back to real values by taking the expectation over the quantization bins, enabling comparison with the ground-truth numerical series.
Training Setup. The model was trained for 200 epochs with batch size 250 using the AdamW optimizer (learning rate 5 10-4). The loss function was categorical cross-entropy, which directly minimizes the KL divergence between predicted and ground-truth token distributions, as in equation g Input sequences of length L = 128 are used with a forecast horizon of 1 step. Here, we train a lightweight vanilla transformer from scratch on synthetic datasets generated from Gaussian process kernels (linear, trend, seasonality, non-lime linear, and stochastic; see Algorithm E in Appendix E). For each domain, a different dataset from Table g is used to evaluate the controlled model. This setup not only realizes the log-linear and KL divergence assumptions
10

---

Published in Transactions on Machine Learning Research (10/2025)
practice but also tests whether isotropy remains reliable proxy for generalization when transferring across datasets within and across domains. To ensure robustness, each experiment was repeated across
10a
NMSE
10-2
random seeds.
10-4
Results. In Figure we compare the NMSE vs isotropy across different synthetic datasets to evalu- ate the controlled model. For instance, in the linear- Datasct 2, strong isotropy (inter-token cosine sim- ilarity in equation 6 near 0, i.e., 0.0000023) corre- sponds to NMSE of 9.6x1001 10-6, while in the stochastic- Datasct 1, a weaker isotropy (inter-token cosine sim- ilarity far from 0, ie., 0.12) corresponds to NMSE of 0.038. Similar patterns can be observed across all synthetic domains, where models with higher isotropy consistently stabilized the partition func- tion and generalized better to unseen datasets. These results in Figure 3. therefore, validate our theoret- ical justification by showing that the isotropy of the contextual embedding space correlates strongly with time series forecasting performance.
10
Isotropy
10022
10
Linear(Data2)
Non-Linear(Datal Trend(Dataz)
Seasonality(Datal)
Stochastic(Datal)
NMSE vs isotropy comparison across dif- ferent synthetic datasets for the controlled model. All reported results are averaged over 6 independent trials, with error bars indicating the standard deviation across runs to capture variability.
Figure 3:
Implications. The matching cont rolled experiment in this section validates our theoretical framework Section under assumptions precisely aligned with softmax outputs of self-attention and KL divergence in the loss function. It demonstrates that isotropy serves reliable proxy for generalization in practice, extending beyond the abstract log-linear formulation. Importantly, isotropy is computed solely from the embedding geometry of the input sequences, independent of ground-truth outputs. Tn this sense, the diagnostic does not. require labels and can be applied in scenarios where only input data are available. In our study, outputs are used exclusively for validation, i.c., to empirically correlate isotropy with forecasting error through NMSE. Thus, these experiments already provide indirect empirical evidence that isotropy can function as label-free reliability signal in time series forecasting tasks where outputs may not be accessible. Building on this foundation, the following sections extend the analysis to diverse transformer-based architectures and datasets, demonstrating the broader applicability of isotropy beyond the controlled setting.
5.2 Qualitative Analysis
We now analyze the time series forecasting by the baseline transformer- basedd models qualitatively. We focus on synthetically generated time series for controlled analysis of different types of time series patterns which belong to different domains, such as linear, seasonality, trend, non-linear and stochastic. We are partieularly interested in the isotropic measurement (through equation 0 in the transformer-basee model's last layer it is related to the logits and probabilistic inference as explained in Section 2 So all isotropic measure provided in this section is based on the last layer of the basclines.
We begin by analyzing time series forecasting performance (i.e., NMSE) for different baselines and its relation with isotropy in Figure 4 For instance, in Figure 4 b, we have (NMSE = 0.0000066 and cosine similarity 0.00076 for scasonality- -Dttasttt- 1 and (NMSE 0.00012 and cosine similarity 0.0047) for Sonsonality-Datasee 2 for Chronos-T5, This shows that stronger isp exists (i.e., inter-token cosine similarity value is close to 0) in Chronos-T5: embedding space for seasonality-Dataset 1 which preserves the structure in its hidden representations and causes good downstream task performance. On the other hand, a weaker isotropy exists (i.e., inter-token cosine similarity value is far from 0) in Chronos-TT5 embedding space for seasonalit--Dttaset 2, which, in turn, causes a lack of structure in its hidden representations, thereby leading to bad forecasting performance as compared to Sesoonality-Dataaeee 1. The NMSE and inter-token
*The inter-token cosine similarity value close to indicates strong isotropy, with zero representing perfect isotropy. Since both positive negative deviations from the origin reduce isntropy, we report absolute values (e.g., |- 0.x() to emphasize the distance from zero, which is the quantity of interest.
11

---

Published in Transactions on Machine Learning Research (10/2025)
10a
100
10
Datwoet
NMSE
Dataset
NASE
NWSS
Datoset
-0-1
10"
10-
Dataset
satron
10"1
Isotropy
10
Detacet
Dataser
10
10-
10-3
10
10-3
10
Deluset
Chranos 15
Chrones Bolt
PatchTST
Moiral
Lag-Liama
Chronos T3
Chrones Bolt
PatchTST
Motral
Lag-llama
Chrones T5
Chranos Bolt
PatchTST
Moirel
Lag-Laama
(a) Linear
(b) Seasonality
(c) Trend
Dataset
NMSE
NMSE
Dataset
Decaset
Decaset
10
teztrape
-1
Icatroay
10-3
Dataset 1
Debecet 2
Chronas
Chronos PatchTST
Molcal
Lag L'ama
Chronos T5
Chrones Boll
PotchTST
Motrai
Lag-Uarma
(d) Non-Linear
(e) Stochastic
Figure 4: NMSE vs isotropy analysis for different synthetic datasets of different domains. All reported results are averaged over independent trials, with error bars indicating the standard deviation across runs to capture variability.
10a
10a
105
01
10
10-1
Domain Dataset
Linea D2
10-1
Demats Dataset
Lnnew 02
NMSE
10
Linnar
honLinear
NMSF
10-2
NonLinear 02
Tend D2
NMSE
Thend D1
Linee 01
honLitear
Tend D1
20
Thend 02
Seeuuniiity 02
Liner 01
NonL newr 02
Gtcchestic
Sesconaliy 02
NonLineer 01
10-4
Stochastic 02
Sesconality 01
NonLinear D1
10-1
Seasonality 01
Seasonality 02
10
Stachastic
Stochastic DI
10
Stochastic
Stochastic D2
10
10:2
10
10
10:3
10
Isotropy
Isotropy
Isotropy
(a) Chronos
(b) Chronos Bolt
(c) PatchTST
10
10a
Doraain Datasset
Domain Dataser
Linear D2
Tend D2
Trend
Tend
NMSE
Linear D1
NMSE
Linea
Nartmmon
NenLineer
NonLinear D1
Seasonality 02
Linaa
NenLineer 92
Seisonality 31
10-3
Seesonality 01
Seesonality 32
Stochestic D2
Stochestkc
Stochaetic DI
10-
Stothartc
10 Isotropy
10
10
Isotropy
(d) Moiraí
(e) Lag-Llama
Figure 5: NMSE isotropy across numerical domains. The relationship between NMSE and isotropy is consistent across domains, highlighting that stronger isotropy (i.c., inter-token cosine similarity in equation close to 0) leads to lower NMSE and vice versa.
cosine similarity can also vary across different transformer basedd based models and datasets. For example, in Figure p. the NMSE for trend-Dataset 1 is lower for PatchTST and Moirai, but higher for Chronos-T5, Chronos-Bolt, and trend-Datasel 2, the NMSE is lower for Chronos-T5, Caronos-Bolte and Lag-Ilama, but higher for PatchTST
compared to their respective NMSE on trend-Dataset 2. Conversely, for
12

---

Published in Transactions on Machine Learning Research (10/2025)
and Moirai, compared to their respective NMSE for other synthetic datasets and baselines in Figures 4 b, d, and This shows that any dataset from any particular domain may cause different forecasting performances for different baselines, as it generates different hidden reprosentations (sce Appendix F for full visualization) in contextual embodding spaces, and hence, different isotropy measures. We note that in some cases (e-g., Chronos Bolt in Figures I isotropy exhibits higher variability than NMSE. This arises because isotropy captures sensitivity the embedding geometry, which can fluctuate across runs, whereas the prediction head can compensate to keep forecasting error relatively stable. Such discrepancies highlight the complementary role of isotropy as diagnostic tool beyond error metrics. In Figure 5
1. A similar analysis can also be observed
we show NMSE vs isotropy comparison across numerical domains. From
VOST
WGETS
(a)
(b)
Figure 6: Variations in Mironos-T5's hidden representations for different input context lengths for the same synthotic dataset non-lincar-Dataset 1 :: (a) Contextual embedding space for input context length 500. (b) Contextual embedding space for input context length L 100.
the figure, it can be seen that consistent relationship exists between NMSE and isotropy, which justifies our findings, i.e. stronger isotropy leads to lower NMSE and vice versa.
Next, we examine the inffuence of isotropy on forccasting performance in two important scenarios: a) different input context lengths, and b) different levels of noises in the input data. The first scenario is important as it provides an analysis that helps guide in selecting reasonable input context lengths rather than selecting the length through random trials and errors. The second scenario is important as it gives ideas on how the level of noise in noisy data impacts performance, since the data in the real world is mostly noisy.
Isotropy in different input context lengths. We first analyze the effect of isotropy under varying input context lengths. We begin with an illustration in Figure P where we show how the hidden representations of Chronos-TV T5 vary for two different input context lengths, such as L= 500 and L= 100, for non-linear- www.lan---Daasttttt 1, which generates different isotropic measures for different input context lengths.
In Figure 7 we compare the NMSE vs isotropy across different input context lengths, L 500 and L 100, for different synthetic datasets and transformer-basee models. As can be seen from the figure, the isotropy values vary across different input context lengths and datasets. For instance, in Figure 7 b, we have (NMSE= 0.0000066, cosine similarity= 0.00076] and (NMSE= 0.0793, cosine similarity 0.0011) for L = 500 and = 100, respectively, for Chronos-TT T5 with seasonality- significantly increases the NMSE for the input context length L = 100. contrast, in Figure 7 a, we have (NMSE= 0.000025, cosine similarity= 0.2474) and (NMSE= 0.000009, cosine similarity 0.0644) for I. = 500 and L 100, respectively, for Chronos-T5 with linear- the input context length L 100, which causes the decrease in NMSE for chronos-T5. A similar analysis
1. The decrease in isotropy
In this scenario, the isotropy increases for
13

---

Published in Transactions on Machine Learning Research (10/2025)
10
900
10-2
100
NASS
10
L- 100
MASE
10-3
500
10-3
10:4
100
1-100
10a
10-1
500
100
Isorrpy
bootroa
10
100
100
10-2
10-2
10
10-3
Chronos 13
Chrones Bolt
PatchTST
Mairal
Log-liama
Chronos T5
Chrones Bolt
PatchTST
Motral
Lag-llama
Chrones
Chronos Bolt
PetchTST
Moiral
Lag-Ltama
(a) Linear (Data 2)
(b) Seasonality (Data
(c) Trend (Data
NMSE
100
NMSE
300
10-1
109
la25rpp
-0
Enstropy
10
590 100
500
100
Chronos Chronos PatchTST
Molcal
Lag Larna
Chronos T5
Chrones PotchTST
Motrai
Lag-Ularaa
(d) Non-Linear (Data 1)
(e) Stochastic (Data 1)
Figure NMSE isotropy comparison across different input context lengths for synthetic datasets. All reported results are averaged over independent trials, with error bars indicating the standard deviation across runs to capture variability.
also be observed for other synthetic datasets and baselines in Figures 7 d, and 7 In practice, the input context length often selected randomly or through trial and error, which may cause higher forecasting errors for different datasets. Isotropy analysis enables us to understand how varying input context lengths inftuence the hidden representations of the transformer-baat based model. This insight helps guide improvements in forecasting performance by examining the isotropic properties of the contextual embedding space.
10
NWSE
-
NASE
NIKSE
10
10-4
10
betrapy
hotropy
Ixetropy
10-2
10
10-3
Cironos 75
Chronos Boll
PatchTS7
Lag Laaaa
Chronos T3
Chrones PatchTST
Motrai
Lag-Llama
Chronos T5
Chromos Boll
PatchTST
Motrei
Lag-Llama
(a) Linear (Data 2)
(b) Seasonality (Data 1)
(c) Trend (Data 2)
109
NMSSE
NMSE
10-1
10-3
10-2
0.05
10
heatrpp
10
hooropy
10-3
0.05
Chronos 15
Chrenos Bolt
PotchTST
Mokca
Lag-Llama
Chrones 15
Chrones Bolt
PatchTST
Morral
Lag-Llema
(d) Non-Linear (Data 1)
(e) Stochastic (Data
Figure 8: NMSE vs isotropy comparison across different noise levels in synthetic datasets. All reported results are averaged over independent trials, with error bars indicating the standard deviation across runs capture variability.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Isotropy in varying noise levels in datasets. Next, we focus on the second scenario to the impact of noisy datasets on transformer-baser model's performance. Figure g compares the NMSE vs isotropy across two different cases, one without noise, and the other with Gaussian noise with a standard deviation o = 0.05 standard deviation. For instance, in Figure S C, we have (NMSE= 0.000024, cosine similarity= - 0.00022)) and (NMSE= 0.0012, cosine similarity= 0.0040) for a = and a = 0.05, respectively, for Chronos T5 with trend-Dataset 2. The decrease in isotropy significantly increases the NMSE for the noisy dataset. A similar analysis can also be observed for other synthetic datasets and baselines in Figures & b, S d, and 8 In practice, many real-world numerical domains, such those in nature and energy, exhibit noisy and dynamic behavior. In these environments, it is often infeasible to measure noise in real time or to pre-process the imput time series for improved performance. However, the isotropy in the hidden representation ions of transformer-based models can be readily measured, and thus, can be leveraged to enhance forecasting performance by identifying and mitigating the effects of noisy inputs in contextual embedding space.
5.3 Quantitative Analysis
Next, we present our main results on 12 real datascts which belong to different numerical domains including energy, nature, finance, healthcare, retail, and transportation, As our qualitative analysis Section 5.2 we select two different datasets from each numerical domain and the isotropy measure from transformer-basee model's last layer to show the impact of isotropy on NMSE performance for different transformer-based models. As before, the isotropy measure in the figures of this section corresponds to equation 6
10a
Dataset
10
10
NNS5E
10
Dataset 2
NMSE
10
10
Dataset 2
Detavi 1
10-3
Jatwot?
Jilant
10
10
Dataset
10-1
Datenet
setroey
10
Isatrayy
Dateset 2
10
Dateset
10
Delave Dataset
10-3
Chronos 13
Chrones Boll
PatchTST
Molcal
Lag-Llama
Chronos 13
Chrones Boll
PatchTST
Motrai
Lag-Llama
Chronos
Chrones Boll
PatchTST
Motrai
Lag-Llama
(a) Energy
(b) Nature
Finance
10a
100
100
NMSSE
NMSE
10-1
NWSSE
Deloset
Datapet
10-2
10"4
Isotropy
10
fearopy
10-2
10-7
Oataset
Chrones Chronos PatchTST
Molrai
Lag-Liama
Chronas T5
Chronos Belt
PatchTST
Motral
Lag-Ulema
Chronos 15
Chronos Dolt PatchTST
Lag Lag-Ulama
(d) Healthcare
(e) Retail
(f) Transport
Figure 9: NMSE vs isotropy analysis for 12 different real datasets of different domains. All reported results are averaged over independent trials, with error bars indicating the standard deviation across runs capture variability.
In Figure g analyze the time series forecasting performance of different baselines and its relation with isotropy for different real datasets. For instance, in Figure ge, we have (NMSE 0.0061 and cosine similarity = 0.0020) for retail-Datasct and (NMSE = 0.1255 and cosine similarity 0.1931) for retail-Datasct 2 for Moirai. This indicates the existence of stronger isotropy in Moirai's embedding space for relail-Dataset which preserves the structure in its hidden representations and causes good downstream task performance. On the other hand, a weaker isotropy exists Moirai's embodding space for retail-Datasct 2, which yields lack of structure in its hidden representations and, consequently, bad downstream task performance compared to retail-Dataset. 1. The NMSE and inter-token cosine similarity vary across different real
15

---

Published in Transactions on Machine Learning Research (10/2025)
1d4 9c
100
144
144
NMSE
10-1
NMSE
10:1
96
MMSE
10-1
10"3
10-a
10
244
10
144
Isabroyy
toorrpy
10-2
10"4
96
144
30
36
Chronas
Chrenos Bolt
PatchTST
Moiral
Lag-Llama
Chronas
Chronos Bolt
PatchTST
Morral
Lag-llamia
Chrones
Chronos Bolt
PatchTST
Moirai
Lag-Liama
(a) Energy (Data 1)
(b)Nature (Data 2/
(c) Finance (Data 1)
109
NMSE
10
MASE
NMSSE
10-1
164
10.2
10-1
10-1
10"2
Isotrapy
10
licrrpy
10
20
10
Chrones 75
Chronos Boll
PatchTST
Molcal
Lag-Llam
Chronos 15
Chrones Bolt
PatchTST
Motrai
Lag Lagma
Chronos 15
Chrones Boll PatchTST
Lag-Llara
(d) Healthcare (Data 2)
(e) Retail (Data 1)
(f) Transport (Data 1)
Figure 10: NMSE isotropy comparison across different input context lengths for real datasets. All reported results are averaged over 6 independent trials, with error bars indicating the standard deviation across runs to capture variability.
datasets and transformer- based models. For example, in Figure Da the NMSE for meegy-Datasee is lower for Chronos-T5, T5, Chron PatchTST, and Moirai, but higher for Lag-Llama, compared to their respective NMSE on energy-Datasel 2. Conversely, the NMSE for energy-Dataset is lower for Moirai but higher for the other baselines, compared to their respective NMSE on mergy-Datasee 1. A similar analysis can also observed for other synthetic datasets and baselines Figure D and 9 This again shows that datasets from the same numerical domain can cause varying forecasting performance across different baselines, they generate distinct hidden representations in contextual embedding spaces, and hence, different isotropy measures, depending the transformer-baat based model architecture and tokenization strategy.
Finally, in Figure 10 we compare the NMSE vs isotropy for varying input context lengths to observe its impact on the real datasets. We compare the results for two different input context lengths: 1) the recommended input context length L 144 and the reduced input context length = 96. can be seen from the figure, the inter-token cosine similarity values Lag-Llama. become far from 0, i.e., from 0.0097 to 0.0220 for enrgy--ataseed (Figure 10 a) and from 0.0026 to 0.0103 for transport-Dataat decreases the NMSE performances. On the other hand, the inter-token cosine similarity values in Lag-Llame become close to 0, i.e., from 0.109) to 0.0112 for nature-Dataset (Figure 10 b) and from 0.2014 to 0.0133 for financeDattaset 1 (Figure 10 c), which in turn improves the NMSE performances, A similar analysis can also be observed for other real datasets and baselines in Figures I0 a, 10 b, I0 10 d, 10 and I0 f. Thus, the variation in the recommended input context length may not only decrease the NMSE performances, but can also increase for some datasets.
(Figure I0 f), which in turn
In Summary, while error metrics are essential, they are retrospective and require labeled data. In contrast, isotropy offers label-free, structuro-aware diagnostic of embedding quality that reflects transformer-bant model's generalization potential. Our results show that isotropy correlates with performance across diverse settings, capturing representational properties that error metrics overlook. This makes isotropy a necessary tool for assessing robustness and generalizability of monefformr-bbsee models.
6 Conclusion and Limitations
In this work, we introduced novel approach to investigate the role of isotropy transformer-basee model hidden representations for numerical downstream tasks. By deriving an upper bound for the Jacobian matrix which collects all first-order partial derivatives of self-attention with respect to the input pattern, showed
16

---

Published in Transactions on Machine Learning Research (10/2025)
that the self-attention mechanism implicitly aligns with the dominant eigenvectors of the input correlation structure and induces isotropy in the contextual embedding space. The existence of isotropy in the contextual embedding space was found to stabilize the partition function and enable better generalization in numerical downstream tasks across different models and datasets. Our empirical analysis across 10 synthotic and 12 real numerical datasets, and different transformer-based models further validated the consistent relationship between isotropy and forecasting performance, highlighting isotropy as reliable indicator of structured representation learning. These insights open up new interpretability frontier for transformer-based models in numerical domains.
While isotropy offers a principled way to preserve useful structure, there may be alternative approaches to approximating the partition function and guiding numerical reasoning, Moreover, developing mechanisms to recover or enhance structure when isotropy is weak remains an important avenue for future work. particular, promising directions include leveraging isotropy to guide fine-tuning strategies, inform inference- time mariin (c.g., filtering low-quality predictions), and identify optimal representation depths in multi-layer transformers. Exploring these applications, alongside baseline performance comparisons, could translate theoretical findings into practical tools for improving downstream performance. Ultimately, we believe that incorporating structural insights like isotropy into the transformer-basee model design pipcline can significantly improve their reliability and adaptability to mumerical domains.
17

---

Published in Transactions on Machine Learning Research (10/2025)
References
Jacob Andreas and Dan Klein. When and why log-linear models self- nell-noommliziig? In Rada Mihalcea, Joyce Chai, and Anoop Sarkar, editors, Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 244-249, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.3115/v1/N15- 1027. URL http////lannooooooyooorrg///////
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mcrcado, Huibin Shen, Oleksandr Shehur, Syama Syndar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, and Maddix et al. Chronos: Learning the language of time series. arXie preprint 4XXiv2403.078115 2024.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach PMI-based word embeddings. volume 4, pages 385-399, Cambridge, MA, 2016. MIT Press. doi: 10.1 10.1162/tace /tacl
00106.
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, 2023.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference Learning Representations, 2021.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: hanguage-interfacee fine-tuning for non-language machine 2022.
learning tasks. Advances in Newral Information Processing Systems,
Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha Naidu, and Colin White. Forecastpfn: Syutheticalll-traineed zero-shot forecasting, 2023. URL https://aaxxiti wwtp:/o////baaaeeeee 01933
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96, page
AAAT Press, 1996.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Conference on Empirical Methods in Natural Language Processing, 2019.
Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, and Wei Liu. Inducing high energy-latency of large vision-language models with verbose images. In ICLR, 2024.
Azul Garza and Max Mergenthaler-Canneeo. Timegpt-1. arXiv preprint ar Xrxw:23310.03558 2023.
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters, 2024. URL http:://axxvvv....g
07820
Jörn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability, 2020. URL https://arxiv..r org/ abs/ 11111 00401
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time scries forccasting by reprogramming large language models. 2024.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5562 5571. PMLR, 18 24 Jul 2021.
Boxun Liu, Xuanyu Liu, Shijian Gao, Xiang Cheng, and Liuqing Yang. Llm4cp: Adapting large language models for channel prediction. Journal of Communications and Information Networks, 9(2):113-125, 2024. doi: 10.23919/JCIN 202.2.00222221199
18

---

Published in Transactions on Machine Learning Research (10/2025)
Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Represenlations, 2018.
Anil Kumar Nelakanti, Cédric Archambeau, Julien Mairal, Francis Bach, and Guillaume Bouchard. Structured penalties for log-linear language models. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 233-243, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant. Kalagnanam. A time series worth 64 words: Long-tern terg forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.
Jochen Peters and Dietrich Klakow. Capturing long range correlations using log-linear language models. In Verbmobil: Foundations of Speech-to-Speech Translation, pages 79-94. Springer, 2000.
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Hena Ghonia, Rishika Bhagwatkar, Arian Khorasani, Mohammad Javad Darvishi Bayazi, George Adamopoulos, Roland Riachi, Nadhir Hassen, Marin Biloš, Sahil Garg, Anderson Schneider, Nicolas Chapados, Alexandre Drouin, Valentina Zantedeschi, Yuriy Nevmyvaka, and Irina Rish. Lag-llama: Towards foundation models for probabilistic time series forccasting, 2024. URL https://arxis org hwwgg/aba/2300 22110002778000888888
Renan A Mojas-Gomen Tock- Yian Lim, Alex Schwing, Minh Do, and Raymond A Ych. Learnable polyphase sampling for shift invariant and equivariant convolutional networks. Advances in Neural Information Processing Systems, 1135755-35706) 2022.
Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 10:53-65, 1987. ISSN 0377-0427.
Vasu Singla, Songwei Ge, Ronen Basri, and David Jacobs. Shift invariance can reduce adversarial robustness.
In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https: wrtp:///....bbbb....... 03762
Dandan Wang and Shiqing Zhang. Large language models in medical and healthcare fields: applications, advances, and challenges. Artificial Intelligence Review, 57(299):112 2024.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 16158-16170 Curran Associates, Inc., 2021. URL httpps:/pproceedinggs www.eeedddnnnnn.nip....e. wwtpaper.pppppeceeecceeeeee.. files/ pdf
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers, 2024. URL https://arxxv org/ abs//b/s.2....
Chenwei Wu, Holden Lee, and Rong Ge. Connecting pre-trained language model and downstream task via propertics of representation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Shengzhe Xu, Christo Kurisummoottil Thomas, Omar Hashash, Nikhil Muralidhar, Walid Saad, and Naren Ramakrishnan. Large multi-modal models (Imms) as universal foundation models for ai-native wireless systems. Netwrk: Mag. of Global Internetwkg,, 38(5):10-20) July 2024. ISSN 0890-8044. doi: 10.1109 10.110//MET..0 2024..33332......
Xinli Yu, Zheng Chen, Yuan Ling, Shujing Dong, Zongyi Liu, and Yanbin Lu. Temporal data meets Il expeaiaaie financial time series forecasting. arXiv preprint arXiv:2306. 11025, 2023.
19

---

Published in Transactions on Machine Learning Research (10/2025)
A Proof of Theorem
Theorem 1. Let the logits of the ground-truth model be bounded. Then for any (*, (), there exists set of functions (k; 1) such that for all k and Ti+1, the predictive distribution of the trained model i (kz k-t) matches that of the ground-truth model (k) k-i) while f(ki,) 1) = O. In other words, there exists a trained model with the same training loss as the ground truth model, but with logits ineffective for numerical downstream tasks.
Proof. We select 7 E R such that vk, T44,, T < minsev by maxjev 2k 1), and Vk, T4+1, Vj E V. By setting (k, = 23 (k; + T, we get Vj V.
1k 1) - bj < (k,1) + min jEV
bj
max 2 (k:, 1141)= - bi3 S
jEV
this implies that (22(k, bU) 0. Hence, Vk, T++1 and we have f(k O.
B Proof of Lemma
Lemma 1. Consider the Jucobian matriz (v) IV
0v, iij='' which represents the gradient of the self-attention mapping G(DD) with respect to the input time series token embeddings. Then the spectral norm of J satisfies |J]|2 I 2 MM (pr, + 1) |o. E A.
2 + LAlz Σ| 1جاي - by According to the analysis, the gradient
Proof. In Lemma the residual term A is given by A |A/2 Einj Pi. (/)
2
and the attention weights Pr,j are defined as Pi. = exp(v, Avs)
ΣM exp(t) Adta)
of gi( ()) with respect the variable 4 is expressed as Jı.j Beg Pi,, v Q' (TA + F.,
(g.
where the matrix Q' is defined by Q' diag(pt.) - P..P... Here, Pt.: B corresponds to the i-th row of the probability matrix P, Eg.a E | denotes matrix with single entry the ()t))th position and zeros elsewhere, and j.g E {0, 1} is the Kronecker delta. We thus have
M
[J]
<
Ji
kJ=1
Pi.j
T
V T
Q'E,,, 1/////2
i,=1
M
$ < + A//
Σ
Pr.
12,
Σ
Pi Pii
+ |A
Σ i,
vv T Q'e
|
1/1 + A//2
Pioli
Σ
Pig
+
|A
Σ
Pi I (2/b
I pa:)l
1,==1
M2
M
1
AA 2
A
Piç 2/2
labe v Px.
1
|A
Pi.s 1V:
NTT pp..11
|xb2 2
j=1
MV Σ
M
Pi.s + 1 1
|A 2
||
10b
TPp.l"
Pc..
|
|A
Pi.i
Pigg Vig
|
12
q=1
20

---

Published in Transactions on Machine Learning Research (10/2025)
Theorem 2 shows that A minimizing the objective ΣM It: L V A contains the largest m eigenvectors of the correlation matrix y L of input time series token embeddings where is the rank of A.
Lemma implies that one of the key components in the Jacobian's upper bound takes the form |e Consequently, during optimization, it is natural to aim for reduction in the gradient magnitude, which motivates minimizing the expression Σ by ΣS Pir This leads to understand the choice of we and w that minimize ΣM Mt: - E)= Pi which equivalent solving the optimization problem AlrSo EM lt: - EE Pt, where the scalar constraint p regulates the size of A.
To proceed, we consider the objective in the scenario where p is small. In this case, can approximate the 1
attention weights by Pij [
+
v. Athy. Now, we define the average of embedding as = v 1///// It
then follows that ΣM |db V Pa.: 1 EM
1t 144-v0 & TTD A Assuming all input time scries patterns are y DArl? = tr (11 v " Theorem establishes that the optimal A that minimizes Σ i= |24y v T Ap,, spanned by the top m eigenvectors of L T v where m equals the rank of A.
zero-centered, ie., e = 0, we have Σ i= R4
DM
C Proof of Theorem 2
Thcorem 2. Let the eigenvalues of the correlation matrix T be ordered as X 2 XD, and let ni E RDD for 1,..., D denole their associaled eigenveclors. Then, the malrix A thal minimizes the quantity MM (b: Vp T "App has the optimal form = -m v
Proof. Given that We E P and WK Dxmm it follows that the matrix A has rank Hence, we know mInA EM
T Ab 2
1 m Ag- Now, we set A to = Em Gi Xu A then we obtain D
EM 11: V 1 = tr (1- - 76% )
L L
Eq=m++ Ag.
Therefore, the optimal solution A for minimizing > ==
[V] || v 2 is essentially characterized as a linear combination of the top m cigenvectors of v L. Since a small gradient will prefer a small quantity of EM ||1/: - v A A the self-attention mechanism implicitly drives the weight matrices We and WK to align with the dominant eigen-directions of y Tv. v.
D Clustering in the Contextual Embedding Space
Clustering. We begin with the isotropy assesmment by performing clustering on the transformer-based model representations in the contextual embedding space. There are varions methods for performing clustering, such reasonably fast.
as k-means, DBSCAN (Ester al. (1966). ). We select. K-means clustering method because in high embedding dimensions (o.g., Z 768 for GPT2, ELMo, BERT etc.). We use the celebrated silhouette score analysis (Rousseeuw (1987) to determine the number of clusters |C| in the contextual embedding space. After performing K K-means clustering, each observation p (i.e,, one of the J vector representations in V) is assigned to one of C' clusters. For an observation p assigned to the cluster E C, we compute the silhouctte score as follows
b(p) - -appppp))
Σ dist(p, g): a(p) = |C| VEC,p
b(p) = min
Σ gEa
dist q):
s(p) = max b(p))
dgee
where a(p) is the mean distance between an observation p and the rest in the same cluster class p. while b(p) measures the smallest mean distance from p-th observation to all observations in the other cluster class. After computing the silhouette scores s(p) of all observations, a global score is computed by averaging the individual silhouctte values, and the partition (with a specific number of clusters) of the largest average score is pronounced superior to other partitions with a different number of clusters. We select the best |C| that belongs to the partition that scores highest among the other partitions.
21

---

Published in Transactions on Machine Learning Research (10/2025)
E
Dataset Description
Real Datsets. One of our goals in this paper is to study how variations in time series characteristics affect isotropy and forecasting performance. For this, we selected real datasets from the Monash Time Series Forecasting Archive (https:/ httpss:/fforecassinggaaaaaoos org/), a widely used benchmark covering diverse domains and structural properties. Additionally, we included a transportation dataset from separate public source to introduce greater variability.
Table 4:
The complete list of datasets used for quantitative and qualitative analysis. The table is divided
into three sections, representing how the datasets were used for baseline models.
Series Length
Prediction
Dataset
Domain
Freq.
Num. Series
min
max
Length (H)
Australian Electricity
Energy
30min
5
230736
231052
232272
48
Car Parts
Retail
1M
2674
51
51
51
12
Covid Deaths
Healtheare
ID
266
212
212
212
30
Dominick
Retail
ID
100014
201
296
399
Exchange Rate
Finance
IB
8
7588
7588
7588
30
FREDD-M
Economics
107
728
728
728
12
Hospital
Healthcare
1M
767
84
84
84
12
NN5 (Weekly)
Finance
1 LW
113
113
113
Weather
Nature
ID
3010
1332
14296
65981
30
Iransportaion Signal
Transport
ID
3010
1332
14296
65981
30
Synthetic (10 kernels)
Numerical
1000000
1024
1024
1024
64
Synthetic Datasets. We use KernelSynth (Ansari et al. (0244) ), a method generate synthetic dataset using Gaussian processes (GPs). KernelSynth allows generation of large, diverse datasets tailored to specific patterns statistical properties, which is particularly useful when real-workd data is scarce incomplete. In this synthetic data generation process, the GPs are defined by mean function, μ(t), and a positive definite kernel, k(Z,, which specifies a covariance function for variability across input pairs (z), A kornel bank IC (which consists linear, RBF, and periodic kernels) is used to define diverse time series patterns. The final kernel R(x;, is constructed by sampling and combining kernels from IC using binary operations ((#;,;;;)
like + and Synthetic time series are generated by sampling from the GP prior, GP(n(t) The following algorithm presents the pseudocode for KernelSynth which essentially follows the approach in (Ansari et al. (2024)).
Algorithm KERNELSYNTH: Generating Synthetic Sequences via Gaussian Process Kernels Input: Kernel bank K, maximum kernels per time series 5, and length of the time series layn 1024. Output: A synthetie time series Xidayn"
1: j- J{
b sample the number of kernels b sample j kernels from the Kernel bank K.
2: (3((,)))) L1d
+(1,1t))<((1( (l,1() i do
*~ * {+
pick random operator (add or multiply) compose kernels
(t,tt) + n² (t, xm((,tt)))
7: end for
8: Xisayn ~ (P (t,t'))
b draw a sample from the GP prior
9. return X1:4yy
F Full Visualization of PCA plots for different models
The full visualization of PCA plots of different models is provided below. We use the synthetic Datasct 1, and Dataset 2 from non-linear domain for illustration.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Non-Linear (Dataset 1):
Chronos-T5
Chronos-Bolt
PatchTST
Morai
Lag-Lima
23

---

Published in Transactions on Machine Learning Research (10/2025)
Non-Linear (Dataset 2):
Chronos-T5
Chronos-Bolt
PatchTST
Morai
"IGHT
Lag-Lima
24


Reviews:
[{"review_id": "HhW4XDLjZm", "reviewer": "Reviewer_sFkx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The problem tackled is interesting\n- The idea of looking at LLM embeddings makes sense in this context\n- The experiments seem to validate the theoretical findings, although standard deviations are missing\n\n**Weaknesses**\n\nI list below what I believe are weaknesses, but I would be happy to be corrected if I misunderstood some key components of the work.\n1) The validity of the model (log-linear) chosen is unclear to me: to the best of my knowledge, current pre-trained LLMs do not follow the log-linear model. Could the author elaborate on their motivation and why it makes sense in the paper setting? \n2) Follow-up question: Why does assuming that the observation probability of $k_{T_{l+1}}$ satisfies the log-linear model make sense for pre-trained autoregressive LLMs? Some prior works in the literature to motivate such a choice **in this context** would be appreciated.\n3) Why mention a \"student model\" in section 2: is there a teacher model, and if yes, what is it?\n4) The motivation behind the modeling of the downstream task is unclear to me: the authors study later on time series forecasting, but the described downstream task of Section 2 is very different from that.\n5) What is the justification for assuming that the student model achieves a small loss such that KL term of Eq. 4 is also small? Usually, teacher models are very strong, and student models are to be trained; hence, it is not clear why the loss would be directly small on such models. Could the authors elaborate on this? \n6) In the abstract, the authors describe their study in terms of pre-trained LLMs trained for the next-prediction task. However, some of the models considered in their experiments **are not language models**: PatchTST is not pre-trained on language and is simply a transformer model with a specific patching of time series, while Moirai is pre-trained on a large corpus of time series. Neither of them is autoregressive or trained on language, let alone for next-word prediction. Could the authors elaborate on the gap between the claim in the abstract and the practical experiments conducted? This might mean that the authors' analysis could be extended to other models; however, in the current form of the submission, the gap between the claims and the supported evidence is a significant issue on my part.\n8) The benchmarks used in the experiments do not seem to be the de facto choice in long-term forecasting (e.g., the ones used in Chronos, PatchTST, MOIRAI, and Lag-Llama papers). Could the authors elaborate on their choice and/or add experiments on those more common benchmarks (Electricity, Traffic, ETTh1, etc.)?\n9) The numbers in Section 5.1, the qualitative analysis, are very close between methods, and without standard deviation, it is hard to say if they are significant to show that weaker isotropy indeed leads to worse performance. Could the authors elaborate on that and/or add standard deviation?\n\nOverall, I find the main motivation behind the modeling in Section 2 unclear, and in my opinion, there is a gap between the claims and the supported evidence. Put together, I lean towards rejecting the paper, but I remain open to changing my rating depending on the authors' clarifications. Notably, I think that the main findings and ideas are interesting and might be valuable for the TMLR audience, but only if the claims are supported by clean evidence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the capacity of pretrained large language models trained on next-token prediction to be adapted to numerical tasks. They focus on time series forecasting and model LLMs with a log-linear model, where forecasting can be done using its context through a network with a softmax layer in the output layer of LLMs (akin to language model head in self-attention). The authors show that for such a model to be good on numerical data, it needs to verify a specific structure (isotropic property) that preserves the shift-invariance property of softmax.- They formulate self-attention gradient, showing that LLM contextual embedding verifies this property, and as such, they provide performance guarantees for time series forecasting. They conduct experiments on 22 datasets and 5 models, showing how isotropic properties impact the performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1) Could the authors provide more justification for the modeling choices in Section 2?\n2) Could the authors clear the gap between the claims of a study of pretrained LLMs and the presence of PatchTST and MOIRAI in the experiments?\n3) Could the authors correct the typos or unclear sentences listed below?\n\n- page 1, Section 1, first line: \"have been proven to effective\" --> \"have been proven to *be* effective\"\n- All the paper: Citations in the text should have parentheses (e.g., \"such as finance Garza and Mergenthaler-Canseco (2023)\" --> \"such as finance (Garza and Mergenthaler-Canseco (2023))\"\n- page 3, Section 2, first line: \"domians\" --> \"domains\"\n- page 6, Section 4.2: \"the analysis of Kim et al. (2021) is provided in Appendix B\" -->  \"the analysis of Kim et al. (2021) *and* is provided in Appendix B\"\n- In the experiments sections and result tables: \"Lag-Llma\" --> \"Lag-Llama\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YtsrUSgJdL", "reviewer": "Reviewer_NQkF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper aims to improve our understanding of LLMs in and to mitigate hallucinations.\n\n2. The experiments cover a variety of real-world and synthetic numerical domain tasks. The base LLMs also cover multiple architectures and tokenization techniques.\n\n**Weaknesses**\n\n1. It is unclear if the findings apply only to numerical domains or to any (or most) next-token prediction tasks. If they apply mostly to numerical domains, why? And if not, then it would be preferable to present the results more broadly.\n\n2. The paper is motivated by providing performance guarantees, but the guarantees given by the paper's analysis remain vague.\n\n3. While there might be a relationship between normalized mean squared error (NMSE) and isotropy, it is unclear whether it is a causal relationship. Some experiments only show a small amount of data points, for example context length 100 vs 500 (or 96 and 144 for another experiment). Other factors impacting final performance remain unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyses the impact of the isotropy (uniformity in all directions) of LLM contextual representations for numerical domain tasks. The authors introduce a downstream numerical task where shifting the logits can lead to poor performance. For LLM representations, PCA (principal component analysis) shows that many models have small effective representations (<=4 dimensions capturing 80% of the variance or more), and the paper introduces a theorem to explain this observation. The authors then cluster LLM representations and measure the expected isotropy over clusters. Experiments are run with 5 base models over 12 real and 10 synthetic time series datasets, where the authors observe a relationship between normalized mean squared error (NMSE) and isotropy. The experiments consider different scenarios with varying input context lengths and noise levels.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**\n1. Can you clarify how structure and isotropy are related? Are they interchangeable? If you have 2 large sets of vectors (-1,0) and (1,0), the overall average cosine similarity will be very close to 0, but the vectors are not isotropic.\n\n2. Similarly, you mention \"If all the embedding vectors lie on a 1-dimensional line, the inter-token cosine similarity would be close to 1\". How do we know that no vectors point in opposite directions?\n\n3. How does local isotropy (within a cluster) impact the global normalization of the partition function?\n\n4. In the downstream numerical task, can you clarify why you replace the linear equation with a non-linear one (with ReLU non-linearity and bias)? I did not follow the explanation in the paper.\n\n5. If the time series is tokenized, what is the significance of the downstream numerical task?\n\n6. For theorem 1, how do we know that the functions $\\hat{z}_i$ that satisfy the conditions can be represented according to the model parameterization?\n\n7. Why are some of the cosine similarity values shown as $|-0.x|$?\n\n**Would strengthen**\n\nPlease consider revising the notation to make it more easily understandable. For example, in the equation between (2) and (3) (on page 3), `i` is used in 3 different manners: as a token ($k_{T_{l+1}}=i$), as a specific index in $k_i$, and as a summation index $\\sum_{i=1}^{|V|}$. In equation (4), the tokens should be the same for $p^*$ and $\\log{p}$.\n\n**Typos**\n\nP1. have been proven to effective\n\nP3. domians\n\nP4. perfromance\n\nP5. Theorem 3.1\n\nP7 enoder\n\nModel name typos, e.g.  Lag-Llma, Choronos-Bolt, CHORNOS-T5.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lP3dEZt1lU", "reviewer": "Reviewer_uaYx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper links isotropy in the embedding space to performance on a novel task, which provides insight on the inner mechanisms of these black box models.\n- The empirical results are supported by a theoretical justification.\n- The experiments comprise diverse pre-trained models and datasets.\n\n**Weaknesses**\n- The paper, while well written, is difficult to follow, as the causal flow across different sections is not smooth and the notation seems overly complex.\n- The paper appears to talk about LLMs (thus pre-trained on language tasks), but then, in practice, is about models that are trained on time series forecasting. Moreover the numerical domains are indeed time series only.\n- The presented problem formulation seems not aligned with the empirical section. Section 2 seems to present a problem in which the pre-trained model uses the softmax as output function and is trained to minimize the cross-entropy. However, the only model that does such thing in practice is Chronos, while all other models are trained by optimizing different objectives and their last layer does not appear to be a softmax.\n- It is unclear if the shift invariance of the softmax is a problem in practice.\n- The paper does not explain how the relationship between isotropy and performance can be exploited in practice. Evaluating the model's error directly seems more convenient than checking isotropy. Moreover, error can be compared across models and has inherent meaning. Conversely, isotropy can be only used to infer whether a specific model will perform better on task A or task B.\n- It is unclear why grouping datasets by domain is meaningful. The fact that two forecasting tasks belong to the same conceptual domain does not necessarily imply that they are similar.\n- Variance bars are not reported in any plot, so it is difficult to asses how significant some differences are.\n\n**Questions**\n- When selecting the datasets did you ensure in any way that there is no leakage with the data used to train the pre-trained models? \n- Is there any reason why figures 5 and 6 use Chronos while figure 8 uses Lag-Llama? Can you provide relative results for the other models?\n- Regarding figure 5, is there a consistent relationship between isotropy and context length? For instance by increasing context length we have a smooth increase in isotropy up to a plateau, or an increase followed by a decline.\n- Is the isotrpoy/performance relationship consistent across domains? It would be useful to see a plot like that in figure 7 but with every dataset, so that the bars' scale is comparable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper seeks to understand which properties should the embedding space of a pre-trained LLM satisfy in order to provide performance guarantees when applied to numerical tasks. In particular they identify the shift-invariance of the softmax function as a problem for such guarantees, and show that higher isotropy in the embedding space mitigates this problem and aligns with better performance in practice.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Main changes**\n- Readability could be improved by having sections begin with a brief overview of what the reader has to expect and the purpose of the section.\n- The theoretical justification's notation could be simplified in my opinion. In the current state it is difficult to follow through as one needs to trace back the meaning of a lot of symbols.\n- The authors should clarify what they mean with \"performance guarantee\". In the current state it seems that high isotropy should guarantee the optimal performance of a pre-trained model on any downstream numerical task, which I think is not what the authors mean.\n- I think the paper title's could better reflect its contents. I don't see why the considered models should be referred to as LLMs, nor the necessity of talking about numerical domains when the only considered task is time series forecasting. In my opinion a more appropriate title should be something along the lines of \"Is isotropy a good proxy for performance on unseen tasks for transformer based time series forecasting models?\"\n- Plots should report a measure of variance in the isotropy and performance.\n- It should be clarified which metric does the term Isotropy refer to in the plots. I imagine it is related to Equation 6, however it is not clear how.\n\n**Minor changes**\n- There is a typo before Eq.3, \"defines\".\n- In my experience the dot notation is a more common way to refer to the dot product, compared to angular parenthesis, at least in deep learning literature. I suggest that the authors clarify the meaning of the notation to avoid confusion.\n- There is an artifact at the right of the \"Proof of Theorem 1\" line.\n- There is a typo in Lemma 1, \"In order to minimize\".\n- In Figures 5, 6 and 8 the isotropy colorbars do not match the respective tone (light or dark) for the performance bars. Aligning the correspondence with the other figures avoids confusion.\n\n**Suggestions**  \nIn my opinion the paper would greatly benefit from presenting some practical way of exploiting this found relationship between isotropy and performance. I hereby provide some suggestions for the authors to consider.\n- Showing that a pre-trained model that yields higher isotropy on task A than task B can be fine-tuned more effectively on task A.\n- Showing that isotropy can be used at inference time to \"ignore\" bad predictions and reduce the average error on \"kept\" forecasts.\n- In multi-layer transformers, showing that one can evaluate the isotropy at different depths and use representations from the depth with lower isotropy to achieve performance improvements.\n\nAdditionally, in the current state it is difficult to asses whether the performance of the pre-trained models are reasonable on the downstream tasks. Knowing that the correlation between isotropy and performance is present at a level where model performance are \"acceptable\" would strengthen the relevance of isotropy as a proxy of performance. Providing the performance of a baseline model trained on each individual dataset can help better put this into context.", "broader_impact_concerns": "I don't have any broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)