Understanding the Implicit Biases of Design Choices for Time Series Foundation Models
Authors: Annan Yu
Danielle C. Maddix
Boran Han
Xiyuan Zhang
Abdul Fatir Ansari
Oleksandr Shchur
Christos Faloutsos
Andrew Gordon Wilson
Michael W. Mahoney
Bernie Wang
Venue: ICLR 2026 Poster (2026)
Keywords: time series
foundation models
inductive bias
frequency
uncertainty
geometry
Primary area: learning on time series and dynamical systems

Abstract: Time series foundation models (TSFMs) are a potential class of powerful, general-purpose tools for forecasting and related temporal tasks, but their behavior is strongly shaped by subtle inductive biases in their design. 
Rather than developing a new model and claiming that it is better than existing TSFMs, e.g., by winning on existing benchmarks, our objective is to understand how the various "knobs" of the training process affect model quality. 
Using a mix of theory and controlled empirical evaluation, we identify and show how various design choices (e.g., patch size, embedding choice, training objective, etc.) lead to implicit biases in fundamental model properties (e.g., temporal behavior, geometric structure, how aggressively or not the model regresses to the mean, etc.), and how these biases can be intuitive or counterintuitive, depending on properties of the model and data. 
We illustrate in a case study on outlier handling how multiple biases interact in complex ways.

with Published as a conference paper at ICLR 2026
UNDERSTANDING THE IMPLICIT BIASES OF DESIGN CHOICES FOR TIME SERIES FOUNDATION MODELS
Annan Yu, Danielle C. Maddix, 2. Boran Han,
Xiyuan Zhang, 2
Abdul Fatir Ansari, Oleksandr Shehur,² Christos Faloutsos,
Andrew Gordon Wilson Michael W. Mahoney, Yuyang Wang?
1 Center for Applied Mathematics, Cornell University
2 Amazon Web Services 3 Amazon Selling Partner Services
Amazon Supply Chain Optimization Technologies
ABSTRACT
Time series foundation models (TSFMs) are a potential class of powerful, general- purpose tools for forecasting and related temporal tasks, but their behavior is strongly shaped by subtle inductive biases in their design. Rather than developing a new model and claiming that is better than existing TSFMs. e.g., by winning on existing benchmarks, our objective is to understand how the various "knobs" of the training process affect model quality. Using mix of theory and controlled empir- ical evaluation, we identify and show how various design choices (e.g.. patch size, embedding choice, training objective, etc.) lead to implicit biases fundamental model properties (e.g., temporal behavior, geometric structure, how aggressively or not the model regresses to the mean, etc.), and how these biases can be intuitive counterintuitive, depending properties of the model and data. We illustrate in a case study on outlier handling how multiple biases interact in complex ways.
INTRODUCTION
Design decisions associated with modern machine learning models typically involve multiple trial- and-error iterations using one or more of many possible "knobs," " including model architectures, loss functions, and parameters and hyperparameters such as learning rate, batch size, patch size, and em- bedding strategy. A common workflow is to adjust these knobs based on intuitions about the model and then use available data, typically in the form of curated, well-established benchmarks, to train and evaluate the model. For example, the case of time series, intuitions include that nearby data points are similar and should be mapped close to each other, that low-frequency structure is typically fluctuations, and that regression algorithms should
more stable and informative than regress the mean. This process, which is expensive, is iterated upon until one obtains model that is quantitatively "better" on well-established benchmarks.
Unfortunately, model improvements often do not stand the test of time: when the data and compute are much larger, or when the data are qualitarively different than the well-established benchmarks on which the model was trained, the same knob fiddling can have a different, and often delete- rious, effect. This leads to worse results and more model development effort in the longer term. Understanding how various design choices affect downstream model properties of increasing im- portance as the community aims to design foundation-lin like models that are forward-compen qualitatively more and different data.
Here, we consider these issues in the context of els (TSFMs). Our goal is not to develop "yet another" model. Rather, we aim to understand the effects. i.e., the implicit biases, encoded by these various design decisions. We are particularly in- terested in how those effects vary with the properties of the data and the model, particular for data that are similar to well-established benchmarks versus data that are qualitatively different. In time series forecasting, there are multiple well-known public benchmarks, e.g., M4 (Makridakis et al.,
time series foundation mod-
Work done during an internship at AWS.
Correspondence Danielle C. Maddix <dnmaddi x@amazon. com?.

---

Published as a conference paper at ICLR 2026
Geometiic (saction
Regressloo-iDeeeeeeeee (section
gurelisiien laced rekoeliing
21 barad
Pascing
Erbnndnng
Taraforerr
Tondaan
Terporral (section 2j
Outlier (section 5)
Terpooal
Gecemti
M m
Figure 1: An illustration of examples of the three broad classes of biases that we identify and investigate. The temporal bias tends to be induced by patching. which transplants the temporal dependencies from the input sequence to a Transformer's 's hidden space; the geometric bias tends to be related to the input embedding. where a discrete versus a continuous embedding preserves the input space's geometry differently; the regression-to-the--eee training loss, where a continuous loss tends to average different outcomes more than a cross-entropy loss does. We also illustrate the interaction of these biases in a case study of outlier handling.
bias tends to be caused by the
2020), ETT (Zhou et al., 2021), and the large collection from Godahewa et al. (2021) including Kaggle's Tourism (Athanasopoulos et al., 201 1). Electricity (Trindade, 2015), Traffic (Lai et al., 2017), and Dominick (James M. Kilts Center, 2020). There has also a recent explosion of TSFMs, e.g., LLMTime (Gruver et al., 2023), Chronos (Ansari et al., 2024a), Moirai (Woo et al., 2024), TimesFM (Das et al., 2024), Chronos-Boo (Ansari et al., 2024b), WaveTToken (Masserano et al., 2025), and TOTEM (Talukder et al., 2024). If foundation model is to be easily adapted to new forecasting tasks, with minimal task-specific data (Liang et al., 2024), this leads to an important open question: how does one know the biases of various knobs in a TSFM on data that are qualitatively different from the benchmarks on which the TSFM was trained and evaluated?
In this paper, we identify three classes of design choices the "knobs" of patch size, whether the embedding is discrete or continuous, and the choice of loss; and, based on extensive analysis, we characterize three corresponding classes of implicit biases which we call the "temporal bias," the "geometric bias," and the "regression-to-the-meee choice tends affect the corresponding bias; and we show how, depending on the model, this effect could be intuitive or very counterintuitive. We also provide an example of how the design choices can interact in complex ways. Our main results, summarized in Figure 1, are as follows.
bias." We describe in detail how each design
Patch Size + Temporal Bias. Given discrete sequence sampled from a time series, many TSFMs group these inputs into patches of length k with A: = 1 as a "trivial" corner case, and map each patch into a Transformer's 's hidden space. This design choice of the patch size parameter leads to a temporal bias: depending on its value, a TSFM may favor low-frequency information more or less aggressively. A larger patch size, e.g., 16 in Chronos-Bolt, favors low-frequency components in the time series, which can be challenging on chaotic (or even non-chaotic) systems that consist of high frequency and/or mixed- frequenny components. Conversely, smaller patch size, e.g., in Chronos or or 4 or 8 in Chronos-Bole can capture parts of these high frequency components. Importantly, many time series
periodic or puriidie,,, and capturing these two related but distinct phenomena can depend on temporal bias in subtle ways. (See section 2.)
Embedding Strategy - Geometric Bias. In TSFMs, there are two common embedding strate- gies: discrete; and continuous. A discrete embedding quantizes the co-domain of time series (e.g.. the real line for a univariate time-series) into finite number of regions, where each is treated as a "word token," as in an LLM. This is used in Chronos. A continuous embedding defines a trainable continuous function, usually an MLP, that maps the time series into a Trans- former's 's hidden space. This is used in, e.g., Chronos- design choice leads to a geometric bias: continuous embeddings preserve most of the input's ge- ometric structure, while quantization maps nearby values to different discrete tokens. As a result, quantization- embeddings introduce several qualitative distinctions: they emphasize local
Moirai, TimesFM and MoEe This
Periiodic phenomena repeat at precise intervals, e.g., New Year's 's Day, while quasi-periodic phenomena follow complex, non-repeating patterns of roughly regular intervals, e.g., Mother's Day.
2

---

Published as a conference paper at ICLR 2026
information over global context; they treat fine- and large-scale patterns with less discrimination; and they remain invariant to signals that are shifted by an offset. (See section 3.)
Bias. Discrete and continuous embeddings ways to handle the input domain; similarly, the output space can also be treated in a discrete or continuous way. A cross-entropy loss, e.g., in Chronos, computes forecasts as discrete classifi- cation task, and is more aligned with LLMs. A continuous regression-based loss is more aligned with classical statistical methods: the LL-based mean squared loss a popular choice for models that perform point forecasting, e.g., TimesMM, whereas models that predict quantiles are usually trained on an L -based quantile loss or negative oogllieeldood loss, e.g., Chronos-Boll This de- bias: TSFMs that use regression-based loss tend to regress to either the mean or median forecast, while TSFMs that use a cross-entropy loss tend to regress much less aggressively and thus, for instance, have the flexibility to hop between basins for chaotic data and to better learn quasi-periodis information. (See section 4.)
Training Loss- + wegression-to-the-mee
two
sign choice leads to pngression-lo-the-nee
In each case, we provide theory and/or quantification through controlled empirical evaluations to corroborate the existence of each bias and to demonstrate how each bias follows from the corre- sponding design choice. To demonstrate this, mostly compare Chronos versus Chronos-Bolt because they are natural to compare: they use the same T5 Transformer backbone, they have com- parable sizes, and they are pretrained on similar datasets (while most TSFMs are trained on different datasets). We also show in ablations that other TSFMs relying on the same choices of knobs hehave (qualitatively) similarly to Chronos-Bolt. Lastly, as an illustration of how to use our insights, we demonstrate how these various biases can interact and mix with each other through an outlier han- dling analysis. (See section 5.) An important summary conclusion of our investigation that design choices, e.g., large patch size, continuous embedding, and continuous training loss, which may induce intuitive implicit biases that work well on well-established time series benchmarks, may actu- ally be deleterious to qualitatively different datasets such as chaotic systems, and for these datasets the opposite design choices may be beneficial. (See Appendix A.)
Related Work. Recent time-series founda- tion models (TSFMs), e.g., TimesFM (Das et al., 2024), Chronos (Ansari et al., 2024a), Moirai (Woo et al., 2024), PatchTST (Nie et al., 2022), Time-MoE (Shi et al., 2025), Chronos- Bolt (Ansari et al., 2024b), iTransformer (Liu et al., 2023), UniTime (Liu et al., 2024b), TOTEM (Talukder et al., 2024), share the com- mon goal of generalization across domains. Empirical studies compare them (Liang et al., 2025; Zhao et al., 2025) across various bench- marks. Several works also investigated specific design choices of TSFMs, e.g., tokenization strategies (Nie et al., 2022; Talukder et al., 2024), scal- ing laws for model size and dataset coverage (Woo et al., 2024; Shi et al., 2025), and architectural choices, e.g., state-space models vs. Transformers (Gu & Dao, 2023; Ma et al., 2024). In this work, we systematically analyze the inductive biases of Transformer-base
Contrnbus Embedding
Trethec
Model techaatee
Quantizttio Smprsciing
Invake Dnl
Figure 2: Design choices across TSFMs and their induced implicit biases.
TSFMs. (See Figure 2.)
While our experiments include many TSFMs, we extensively focus our comparisons on Chronos (Ansari et al., 2024a) and Chronos-Bolt (Ansari et al., 2024b) for two reasons. First, they share the T5 backbone of the same size and are trained similarly mixed corpora of real- world and kernel-synthetic data. Second, together they span all major design choices discussed above: Chronos-Bol. treats the time-series co-domain as continuous, embeds size-16 patches with a two-layer MI.,, and is trained with an I,- quantile-regressionn loss; in contrast, Chronos "lears the language of time series" by quantizing values into 4096 bins (thus using no patching), assigning each bin a trainable embedding vector, and training via cross-entropy classification of real-valued output "tokens."
2 TEMPORAL BIAS: HOw Do TSFMS LEARN TIME?
A temporal bias occurs when a TSFM represents or learns certain temporal structures (c.g., specific frequencies and periodic patterns) more strongly than others. Time series data are fundamentally
3

---

Published as a conference paper at ICLR 2026
different from discrete language data, in that they are often thought of as evolving over continuous time domain. As a result, notions such as frequency, periodicity, and seasonality play a central role, and these notions often determine the dominant structures that model must capture. Here, distinguish between two related temporal notions (frequency and periodicity), and we describe how they are affected by different design choices. We outline key considerations in this section, and leave additional details to Appendix C.
FREQUENCY. The frequency bias, which occurs when a model learns low-frequency information better than high-frequency one, was observed and studied in general neural networks (Rahaman al., 2019; Yang Salman, 2019; Xu, 2020), and it was later extended to sequential models such as Transformers (Piao et al., 2024) and state-space models (SSMs) (Yu et al., 2025a). Most of these models have been shown, both theoretically and empirically, to be better at learning the low frequencies than high frequencies. One might hypothesize that we generally want to learn low- spectral modes better, because they usually contain less noise and/or describe more dominant "large- scale" structure. However, in many tasks, e.g., forecasting chaotic systems (Zhang & Gilpin, 2025a) or systems with quasi-periodic properties, high-spectral modes contain essential information that steers the trajectories, and their role cannot be downplayed. It is important to understand the cause of the frequency bias and to be able to tune it for the nature of the specific problem.
A TSFM operates defined. Given a continuous time series that can be sampled, we can hope to control this bias by changing the sampling interval At. More commonly, however, time series are recorded at fixed frequency intervals, and the data is a sampled sequence with a given frequency, and we have no control over At. When we talk about the "spectral modes," we imagine the Fourier modes in fixed sequence x = (#1...., TL), indexed by natural numbers from 1 to L. Then, we ask: what determines the spectral content that TSFM will learn? Perhaps surprisingly, frequency bias is induced as soon as x is embedded into the hidden space. The reason is that an embedding function Φ : + + RRd maps a patch of size k, i.e., (Ir,... rz+ 1), into the hidden space. The following result, whose full statement and proof can be found in Theorem 1' in Appendix C.1.1, makes precise a sense in which patches of different frequencies are embedded differently.
discrete sample of continuous time domain, in which time series is
Theorem 1 (See Theorem I' for the full statement). Let k be a patch size. Let V [V1 VnJ E &Rxx be a collection of n patches whose frequencies range in an interval of be collection of n patches whose frequencies are sepa- rated by distance of at least w. For some d, m 1, let ((x) - Wa ReLU(W (x) be a neural network embedding, with W, E Rmmxk W2 E Rddm and b E IR that satisfy certain regularity
size w and U = [u, ... un] E &
assumptions (see Theorem I' in Appendix C.1.1). The following two statements hold:
1. (Similar frequencies embedded in shared subspace.) For any fixed 0 < < S 1, we have
ranke (&(V)) = H6 (v()) > coi()(V))) (1))=11w))
stab-rank (arrrrannkkk))))) (v(V)) == ([l[VVlll/lll(()) = O(w).
2. (Different frequencies are embedded in nearly orthogonal subspaces.) There exists a universal constant & > so that with high probability, & with random Gaussian weights satisfies
rank. (*(U)) = 2(n)), stab-rank($$ = 2(n).
Moreover, the constants in all O and ?-notations universal.
The e-rank and stable rank of a matrix (VV) or Φ(U) measure the "dimension" of its column space. In Theorem 1, V be represented as collection of patches of similar frequencies and U set of patches of heterogeneous frequencies. In essence, Theorem 1 says that patches with similar frequencies are embedded into a low-dimensional subspace of 12d with the "effective" dimension bounded proportionally to the bandwidth w (see also Yu et al. (2025b)). At the same time, low- spectral and high-spectral modes are embedded in almost orthogonal subspaces. (See Figure 3c.) See Figure 10 in Appendix C.1 for a conceptual illustration of Theorem 1. One may wonder: what role does the patch size k play in this story? If is small, then all patches share a similar bandlimit, because we must have < k; as result, all patches are embedded into a low-dimensional subspace of Rdd With a random Gaussian assumption of the weights, Theorem shows that several neural networks embed low- and high-spectral modes into orthogonal spaces. In trained TSFMs, even though weight matrices are not random Gaussian, we empirically find that the orthogonality is easily preserved. (See Appendix C.1.4.)
4

---

Published as a conference paper at ICLR 2026
Porecast Chronos-Bae Bolt
b Froquency Losses
Attemtion Scures Each Patch
Low Frequency Loss
High Frequency Loss
Pr1 Stze
Pach Sta
95-
scoind piocipll compan..
0.5
60
Boo Ste 16
Pouk Six
W.W paicics
07
0.2
0.4 Sequenccy
05
08
-2 -1
principll compereett.
Figure (a), we show the forecasts of a signal consisting of low-spectral and a high-spectral mode using two pretrained Chronos- Haroonos-Bolt models with patch size A = and & = 16, respectively. In (b), we vary the frequencies of the two modes, and we evaluate the spectral loss in predicting each mode. We see that Chronos-Bolt with patch size of 16 fails significantly at capturing the high- spectral information. In (c), we project the embeddings of low-spectral modes and high-spectral modes onto the two principal components, and we show how the attention matrices align with these two principal components on average. We observe that the attention scores are heavily dominated by the low-spectral modes.
Now that we know that when k is large, low-spectral and high-spectral modes are embedded very differently, the next question is: which frequencies does the model learn best? In practice, we find that most TSFMs with a large patch size k learn lower frequencies better than higher frequencies (see Figure 3a,b). This aligns with the conventional notion of "frequency bias" in other deep learning models (Rahaman et al., 2019; Yang & Salman, 2019; Yu et al., 2025a). full understanding of this phenomenon can be obtained by analyzing the training dynamics of Transformers, but even without going into that level of detail, we have an intuitive understanding: in many time series, the dominant large-scale structure lies in the low- wwwfireqeeccy components, while high-frrqueed content often appears more irregular or noisy. As a result, Wq. W K. and Wy tend to align more closely with the subspace containing low-spectral information (see Figure 3c and Appendix C.1.4).
PERIODICITY. The periodicity bias refers to model's capabil- ity of recognizing and aligning (quasi-)periodic motifs in the input. Periodicity is a fundamental property of many time-series applications. By "periodicity." we refer to loosely recurring patterns rather than cycles that repeat at fixed frequency. The perioidicity bias controlled by several design choices. First, patching can directly reshape the periodic structure. If the period of the recurring motif does not align with the chosen patch size, the patching operation effectively sam- ples the signal off-grid locations, introducing aliasing effects that alter the perceived periodicity. These distortions accumulate across layers and change how the model "sees" repetition in the input (see Fig- ure 4c for large-scale experiment over 10,000 sequences showing decreased motif-matching quality as k increases). Intuitively, patching replaces the original time axis with coarser "patch axis." Then, periodic patterns that do not divide evenly into this new grid become misaligned, which makes it more challenging for the model detect consistent repetitions. Second, the underlying
Autocoerciatonn Transformeer Output
Principle Compancuts Contest
Laik (sRO) Hitaa
Lwc ((0())))
159
150
230
Best Matthhing Score the Last Mottr
woc
30
Figure 4: In (a), we show the autocorrelation in the out- put of a Transformer given periodic input context, which measures the periodicity preservation of model. In (b), we show leading principal components of the encoded context of a Chronos-Boo model, with or without the [REG] token, given periodic input. In (c). we show the best matching score, aggregated on 10, 000 randomly sampled time series in Chronos' training dataset, with without patching. The best matching score is computed by comparing the last mo- tif of length 64 to all previous motifs.
5

---

Published as a conference paper at ICLR 2026
architecture influences how well periodicity is preserved. Encoder-only models tend to better maintain periodicity because they attend bidirectionally over the entire context, allowing the model to reinforee long-range regularities. In contrast, decoder-only or encoder- -ncoder-decoder architectures may dilute periodic structure when information flows primarily forward or is concentrated at a single aggregation position (see Figure 4a). In decoder-style attention, each position primarily attends to earlier tokens. Then, the periodic structure that requires comparing future and past motif's cannot be cleanly captured. Third, we find that the presence of [REG] token can significantly disrupt periodic patterns. Although the [REG] token is intended as designated location for aggregating global information, can inadvertently overwrite earlier positions when unmasked. Our analysis shows that large values at the [REG] token propagate backward through self-attention and contaminate otherwise periodic channels in the hidden representation. This effect visible in Figure 4b, where leading principal components lose their sinusoidal structure when the [REG] token is present. Conceptually, the [REG] token acts as a strong anchor point in attention space; when its value is large, it "pulls" nearby representations toward itself, breaking the smooth periodic patterns. In Appendix C.2, we provide a mechanistic study demonstrating that the channels whose periodicity collapses are precisely those receiving large contributions from the [REG] token, and that masking this token restores periodicity.
Summary: Frequency and periodicity biases are two connected views how TSFMs represent time. Frequency bias is controlled by the patch size k: which determines whether embeddings emphasize low- or high-spectral modes. Periodicity bias is controlled by the alignment of patch size with the underlying recurrent motifs, architectural choices, such as encoder versus decoder design, and the use of an unmasked [REG] token.
GEOMETRIC BIAS: How Do TSFMs LEARN GEOMETRY?
In this section, we identify geometric bias, which we define as the way an embedding choice dis- torts preserves the input domain's geometric properties (induced by the standard inner product ()-)) on R*). This bias changes which local versus global patterns and scales are easiest to learn. There two broad classes of embeddings: quantization- based (featured, e.g., in Chronos) and con- tinuous (featured in most TSFMs, including Chronos-Boo Bolt, TimesFM, and Moirai). A quantization- based embedding partitions IR into regions D1 Di.... Dv and defines ΦQ R IRd as step func- tion that is constant each D. A continuous embedding instead parameterizes &c RR - R directly with a neural network. We use Chronos to represent the quantization- based embedding and be representative of the continuous embedding. Note
Chronos-Bolt with patch size of k that we pretrain a new Chronos-Bole -oot model with A: = 1 to eliminate the temporal bias from larger patch sizes, as discussed in Section 2, and be more directly comparable with Chronos.
In a continuous embedding, the topology of I is inherently preserved by the continuity of Φc; but in quantization- based embedding, the ordering of the real line must be (re))earned during training. Since each ΦQ(D,) is initially a random vector, training rarely recovers the full geometry of R. in Rd Here, we focus on three geometric notions (angle between a pair of vectors, distance between a pair of vectors, and norm of a vector), and we describe how they are affected by different design choices, and how they in turn affect the behavior of a TSFM. At first glance, it may seem that it generally good to preserve the geometry (i.e., the angle, distance, and norm) of the input space, as a continuous embedding does. We show that while not preserving the geometry does lead to certain problems, it is also surprisingly beneficial in some cases. We outline key considerations in this section (see Appendix D for details).
ANGLES. Our first measurement of the geometric bias is the angular bias, which makes a TSFM to favor local information over global information. This angular bias is caused by difference in the angles between the embedded vectors:
(1lb(()) ,o(w)
Oc(2, v) = arccos (1 oc("))
0 v) arecos (T00 (x) (
(Iel
for the two types of embedding. In Figure 5a, 0z y) is significantly larger than (x. v) function of x - y/. which makes an input appear more different from its close neighbors. This difference increases with the number of bins used in quantization (see Appendix D.4).

---

Published as a conference paper at ICLR 2026
Angle Tehhlding
Cress-Micatie
107
607
10"
10
10-4.
0004
20
WV
MAST - 3637
Scetrat
goond inth
m
101
602-
Chocaeo
6004
MASE - 2:11
content
VVV
gronnd mrth
WV
-25
00
35
10
10
11996
scoees
time
Figure 5: Panel (a) shows the angles between the embedded vectors ((x) and ((y) for Chronos' embedding @Q and Matronss-Bot's embedding Φc, respectively, where and y denote two real numbers to be embedded. Panel (b) shows a histogram of all self-attention scores when inferring the two models on corpus of contexts. Panel (c) shows the forecasts from both models on context formed by repeating a motif of a chaotic system. Chronos achieves a significantly lower MASE than Chronos-Bolt. The cross-attention scores are aggregated across all layers when generating the first prediction token. See Appendix D for details.
The angular bias affects both self-attention and cross-attention in a Transformer. In Figure 5b, the self-attention scores of Chronos are bimodal, i.e., tokens place high attention weights their im- mediate neighbors, and low attention weights on distant tokens. contrast, the attention scores of Chronos-Bolt more evenly distributed, indicating broader mixing of contextual information. A similar pattern appears in the right panels for cross-attention. Chronos focuses almost entirely on the most relevant part of the context, whereas Chronos-Bolt attends more broadly across the context. Among other things, this makes Chronos better at "parroting" one reason why it performs better than other TSFMs on chaotic systems (Zhang & Gilpin, 2025b). This comes the cost of less mix- ing across the context, which can hurt performance on tasks that require synthesizing information from distant parts of the time series or jointly interpreting multiple interacting patterns. Notably, for quantization embeddings, the size of the vocabulary also controls the bias (see Appendix D.4).
Our second measurement TSFM magnify small scales, and makes nearby numbers appear more distinct in the hidden space. This distance bias caused by the difference the relative distance, defined as:
DISTANCES,
the geometric bias the distance bias, which makes a
d2 v) = [1 go(v)ll2
(x)(2))12 (v))ll de(a,, v) = [[)(()) - (c(1))12
[ (1)
Just as the learned quantization embedding ΦQ bends the input space R in the hidden space Rd. as we saw via angle measurement, it also stretches or squeezes While a continuous embedding naturally expected to map nearby numbers to nearby vectors in the hidden space, learned ΦQ is generally less scale-preserving (see Figure 6a).
Distance of Embrelding
Forncasding Maliacale Signal
Rebaivv Erroe of Forecasting the Tho Stales
Touctte Larg
Southetk 1180
Choonss
10
16'
Real World D3
Chrosos
Malia
1186
181
19887
1002
127M
1911
17380
1180
1318
Modial 8000
1474
1131
1812
17
1460
1756
0116
500
12
Figure 6: Panel (a) shows the distance between the embedded vectors for Chronos' embedding &Q and Chronos-Boo multi-scale time series. Panel (c) quantifies the forecasting quality by computing the relative MSE, as we increase the ratio between the two scales. See Appendix D for details.
embedding Φc. Panel (b) shows Chronos' and Chronos- -Bolts forecasts of
The distance bias results in a mismatch in scale preservation. As result, when a multi-scale struc- ture is present, Chronos sensitive and better at learning fine-scale patterns than a Chronos-
7

---

Published as a conference paper at ICLR 2026
Bolt model. Figure 6b well-illustrates this phenomenon: while both Chronos and Chronos- Bolt are good at learning the large-scale pattern, only Chronos nicely learns the small-scale patterns. A more systematic quantification is given in Figure 6c, where we show that, the ratio between the two scales increases, Chronos-Bolt, TimesFM, and Moirai, all of which rely on a continuous embed- dings, are less capable of learning the fine-scale pattern. Whether this is advantage is again task-dependent (see Appendix A): it can be valuable when fine-scale detail is important, e.g., when learning high-frequency information or when learning to perform context parroting; but may be unnecessary, or even distracting, in simpler situations when large-scald scale trends dominate.
drawback
Norza Embedding
Forreasting Multi-Offiet Signal
c. Relutive Error of Forecusinng the Two Offict
ding
Synfhetic Date
Claons
3.0
0.54
05
0.0
10
5:10
4- 10
64
10'
(Pp
044
Cheona-BBllt
-25
Real-Wood World Data
10312
(1811
10150
181.0
4.1
17007
11022
Malia
18.0
1880
14.00
1089
13177
18201
1/000
1.00
2.
18:0
11111
1.10
1991
11809
172
1224
1971
1.2
18.0
2536
002
Figure 7: Panel (a) shows the norm of the embedded vectors (z) and Φc(x). Panel (b) shows the forecasts of a multi-offset time series for Chronos and Chronos-Bole respectively. Panel (c) quantifies the forecasting quality by computing the relative (to each stage's amplitude) MSE of the forecasts as we increase the offset. See Appendix D for more details.
NORMS. Our last measurement of the geometric bias is the norm bias, which makes TSFM better learning motifs that are larger in magnitude. This norm bias concerns the mapping of a single element, rather than pairs of elements, in the other two cases. For continuous embedding &c. many TSFMs use a ReLU-activated MLP. Because the ReLU activation &c maps small input values to small vectors in Rd and large values to large vectors. A quantization- based embedding Φ2 behaves differently. Since the mapping of each bin, Φo (D:). ). is initialized from the same distribution, it has inherent bias toward preserving norms (see Figure 7a).
positive-homogeneouss
The norm bias is caused by the tendency of Φc to map large numbers to vectors with large norms. In Figure 7b, the scale of each period is the same; the only difference is their offsets. When motif is large in magnitude, its corresponding embedded vectors are also large in magnitude. In Transformer's hidden space, they appear more dominating than the smaller vectors associated with a motif with a small magnitude (see Appendix D.2.2). Conversely, a period near zero is mapped to small vectors that are more easily overwhelmed by the rest of the context. Figure 7c illustrates this effect: as the offset increases (equivalently, as the period's 's oscillation amplitude decreases while the offsse stays fixed), Chronos-Boll struggles to forecast the near-zero period. For regular signals with multiple offsets, this can introduce an undesirable imbalance; but for sparse signals, it can be advan- tageous, by allowing the model to focus on nonzero entries and ignoring the zeros. (See Appendix A for beneficial and deleterious examples.)
Summary of the Geometric Bias: Geometric bias concerns how the geometry of the input do- main is preserved by embeddings, and it is controlled by the choice between continuous and quantization- based strategies. From the angle perspective, quantization induces angular bias, making the model favor local information. From the distance perspective, continuous embed- dings advocate a distance bias, making fine-scale patterns less learnable. From the norm per- spective, continuous embeddings induce a norm bias, weighting large-magnitude elements more heavily, while quantization embeddings treat inputs more evenly.
4 REGRESSION-TO-THE-MEE BIAS: Do TSEMs REGRESS TO THE MEAN?
In this section, identify regression-to-themme bias, which is the tendency of TSFM to collapse future variability toward a central forecast (mean or median) versus maintaining sharper, more extreme multimodal outcomes under uncertainty. When the future is stochastic, should
8

---

Published as a conference paper at ICLR 2026
model settle safe, compromised prediction, e.g.. regress to the mean forecast, or should a model aggressively commit to one of the possible outcomes? Here, we analyze the effects of various design choices how aggressively (or not) TSFM regresses to the mean (see Appendix E for details).
How a model reacts to uncertainty heavily depends on how it is trained. Most TSFMs are trained with either an based loss (e.g., mean-squared loss) an bbssd loss (e.g., quantile loss). A notable exception is Chronos, which is trained with a cross-entropy loss, like LLMs. If our target comes from an unknown probability distribution with large variance, then:
T,²- based loss: favors the mean of the distribution, by averaging all possible outcomes.
-based loss: favors the median, which gives a central split of the probability mass in half.
Cross-entropy loss: models the full probability distribution (Stewart et al., 2023) and can settle on a/the "mode." This can avoid "compromise" between possible outcomes, and it allows the model to represent distributions with large variances and stronger heterogeneity more faithfully.
A L or L' -based loss explicitly tailors the forecast to minimize pointwise emrrr metries such as MAE or MSE. From traditional wisdom, a low MAE or MSE is usually good, but time series fore- casting is not all about pointwise fitting. For instance, in chaotic systems, the fractal dimension is measurement of the long-term geometry of the trajectories, and regressing to the median or the mean can severely damage it (Zhang& & Gilpin, 2025a). (See Appendix A for cases where the bias "ben- eficial" or "deleterious.") Unlike mean- or median based forecasts, "regressing to the mode" lets the prediction reflect a sharp, high-probabillin outcomes, instead of a smoothed central tendency.
a. Deterministc Contest
Random Contest
Chronos on a Larenty System
TimalM Chons
NAWA wwwmmall MWW Wy
M MW
M. my
NAWA
www.mmil
TimeeFM Chirenos- Bolt
MANA
Chrrense
www.mm1
80
0.3
04 0.5 100
200
300
400
diffasoo poobubiity
Figure 8: Panels (a) and (b) show the forecasts from three TSFMs of a purely deterministic and a purely random context, respectively. We also quantify the "regression score" by computing min( |1 D. where denotes the forecast from a model. As the context involves more un- certainty. TimesFM and Chronos-Bole regress to the mean and the median, respectively, more than Chronos. Panel (c) shows three examples where Chronos "parrots" three distinct outcome branches from the context of Lorentz chaotic system. See Appendix E for more details.
We illustrate the regression thetheenn bias using the examples in Figure 8. In (a), we show a periodic walk between 0 and 1. Since the process is fully deterministic. Chronos (cross-entropy loss), Chronos-Bolt (T.1 loss), and TimesFM (I.2 loss) produce similar forecasts. Panel (b) uses a context generated by random walk. With probability 1/2 on each branch, the uncertainty in the context drives the forecast of TimesFM toward the mean, Chronos-Bolt toward the median (any value in [0, 1)), and Chronos toward the mode (either 1). We quantify this effect by "bridging" the purely deterministic and purely random cases. For each 0 < < 1/2, we take a periodic walk and diffuse it with probability p. As p increases, the context becomes more uncertain and models trained with continuous loss regress more strongly than the one trained with cross cross-entropy.
Summary of the Regression-to-the Mean Bias: An - L based loss tends to collapse fu- ture variability into a central forecast, reflecting mean or median behavior and usually resulting in good pointwise fitting. In contrast, cross-entropy- based training represents the full distribu- tion and preserves sharper outcomes, instead of aggressively regressing to a central tendency.
5 MIXTURE OF BIASES: OUTLIER HANDLING
In the previous three sections, we have identified three key knobs that affect three fundamental im- plicit biases in the development of TSFMs. Of course, in practical settings, these biases can interact

---

Published as a conference paper at ICLR 2026
Forecast Error and Outlier Magaitude
Forecast Error and Number of Outliers
Forecast Error and Number of Outliers
10"
Chrons-al Bul
10
10
Clivaao Seit 11-11
10
Chrsss
Chramee- Sodi
10
100
101
102
107
104
107
16
20
20
40
50
100
ofteer raggitud
nantbr of outiions
santher of ofartioos
d. Chronos' Input Embedding
Chrones Bolt's Inpur Embedding
100 200
300
500
100
200 300 400
500
Figure 9: Top row shows the MSE of the baseline models of forecasting a sinusoidal wave with randomly injected outliers. Panel (a) shows the effect of increasing the outlier's 's magnitude, and panels (b) and (c) show the MSE as we increase the number of outliers. We repeat the experiments are 30 times. Panels (d) and (e) show the first principal component of the embedded input in Chronos and Chronos-Bol Boott as we increase the magnitude of the outlier.
in complex ways, and the effect of fiddling with a given knob can be nontrivial to characterize. In this section, we present outlier handling as illustration of how different biases combine in nat- ural ways. We use a sinusoidal wave as the context and inject outliers into it, in two scenarios: (1) increasing the magnitude of fixed number of outliers; (2) increasing the number of outliers sampled ii.d. from a fixed Gaussian distribution. Our results, as shown in Figure 9, are the following:
1. Temporal bias. Comparing the two Chronos-Bolt models with patch sizes k 1 and k: 16 shows that, without outliers, the k = 1 model performs better. As outliers are injected, they inject feghffrequency perturbations, making the larger patch size more effective denoising (see Figure 9c). This crossover behavior is consistent with our temporal-bias analysis: larger patches suppress high-frequency variation by averaging over wider window. Injected spikes are smoothed out while the underlying sinusoid is largely preserved in the embedded sequence.
2. Geometric (Distance) bias. When outlier contamination is limited, i.c., their magnitude and number are relatively small, Chronos outperforms Chronos-Boc by reducing the apparent scale of outliers through its guantization-basen embedding (see Figure 9d). In Chronos, extreme values are mapped into nearby bins whose embeddings are not much farther away to those of the clean signals. Isolated spikes no longer dominate local distances in the hidden space, which makes the learning of the underlying sinusoidal wave more tractable.
Geometric (Norm) bias. Chronos-Bole Bolt's continuous embedding causes to dominate the embedded context. As the outlier magnitude increases, this dominance leads to progressively worse forecasts (see Figure 9a,e). The large embedded vectors in Chronos-Bor pull attention toward the timestamps corresponding to outliers, which reweighs the context in favor of the outliers.
outliers
3. Regression-to-the--N
bias. Although Chronos handles outliers well on average, its perfor- mance variance is larger than that of Aaronos--Bolt. This is because the eggressiontt-the-mee mean bias can cause Chronos to produce an "outlier" when it is uncertain whether the observed outliers part of the underlying context (see Figure 9b). This leads occasional large forecast devi- ations even when the average error is small, which highlights a fundamental trade-off between robustness in expectation and higher conditional variance for TSFMs that seek dominant modes.
6 CONCLUSION
We identify and characterize three broad classes of inductive biases in TSFMs, which are induced by various common design "knobs". These biases influence the strengths and weaknesses of TSFMs across tasks in subtle and sometimes counterintuitive ways. Our findings indicate that there is no single choice of patch size, embedding strategy, or loss function that is uniformly optimal across all forecasting regimes. Each induces biases that help on certain types of data, and hinder performance on others. This highlights a central challenge for TSFMs: unlike language, time-series data exhibit widely varying characteristics, which makes universal modeling substantially more challenging. Understanding these biases helps explain why different models excel in different settings, and clari- fies trade-offs among architectural and training choices. Lastly, our work suggests that dynamically adapting these biases, e.g., through in-context learning, retrieval mechanisms, or reinforcement- learning-basee based adaptation, may be more promising path toward broadly applicable TSFMs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors would like thank William Gilpin and Yuanzhao Zhang for many fruitful discussions on this work.
REPRODUCIBILIIY STATRMENT.
Code used to produce the experiments in this paper, along with scripts for data processing and figure generation, is available at https::/ /ttp::///ihthii com/ AnnanYaaTSaa /SFM Biases.
REFERENCES
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shehur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint
2024a.
Abdul Fatir
Ansari, Caner Turkmen, Oleksandr Shehur, and Lorenzo Stella. Fast and accurate zero-shot forecasting with Chronos-Bolt and Autogluon, URL https: //aws amazon htoom/blog///achhhc 1eaeeelleannngg// fast-andh anc-anncaaae------cc zero- shot Forecasting wi with- catt--choooo-bbbee t- and auddog-uuggoooo uon
2024b.
S. Arora, S. S. Du, W. Hu, Z. Li, and R. Wang. Fine-grained analysis of optimization and generaliza- tion for overparameterized two-layer neural networks. In Inter. Conf. Mach. Learn., pp. 322-332. PMLR, 2019.
George Athanasopoulos, Rob J. Hyndman, Haiyan Song, and Doris C. Wu. The tourism forecasting competition. International Journal of Forecasting. 27(3):822 844, 2011.
R. Basri, D. Jacobs, Y. Kasten, and S. Kritchman. The convergence rate of neural networks for learned functions of different frequencies. 32, 2019.
R. Basri, M. Galun, A. Geifman, D. Jacobs, Y. Kasten, and S. Kritchman. Frequency bias in neural networks for input of non-uniform density. In Inter. Conf. Mach. Learn., pp. 685-694 PMLR, 2020.
Nora Belrose, Quintin Pope, Lucia Quirke, Alex Mallen, and Xiaoli Fern. Neural networks learn statistics of increasing complexity. arXiv preprint arXiv:2402.0444 043662 2024.
A. Bietti and Mairal. On the inductive bias of neural tangent kernels. 32, 2019.
Yuan Cao, Zhiying Fang, Yue Wu, Ding- Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint arXiv: 19/2.01 198, 2019.
W. M. Czarnecki, S. Osindero, M. Jaderberg, G. Swirszez, and R. Pascanu. Sobolev training for neural networks. 30, 2017.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning, 2024.
Nguyen Anh Khoa Doan, Wolfgang Polifke, and Luca Magri. Physies-informed echo state networks. Journal of Computational Science, 47:101237, 2020.
Everette S Gardner Jr. Exponential smoothing: The state of the art. Journal of forecasting, 4(1): 1-28, 1985.
William Gilpin. Chaos interpretable benchmark for forecasting and data-driven modelling. arXiv preprint 10.00:21 2021.
Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting. Annual Review of Statistics and Its Application, 125-1125111,11 2014.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(2): 243-268, 2007.
11

---

Published as a conference paper at ICLR 2026
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero- Manso. Monash time series forecasting archive. In Advances in neural information processing systems track datasets and benchmarks, 2021.
Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson The free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. PMLR, 2024.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885, 2024.
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36: 19622-19665,, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, and Rob J Hyndman. Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond, 2016.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv: 10427, 2021.
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
A. Jacot, Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. 31, 2018.
James M. Kilts Center. Dominick's dataset, 2020. URL seseaacchkiitssrreeeee
data/dominicks.
G. Lai, W. Chang, Y. Yang, and H. Liu. Modeling long- and short-term temporal patterns with deep neural networks. arXiv preprint arXiv:1703.07015. 2017.
Jeffrey Lai, Anthony Bao, and William Gilpin. Panda: A pretrained forecast model for universal representation of chaotic dynamics. arXiv preprint vrXiv:2505.13755 2025.
Shuang Liang, Chaochuan Hou, Xu Yao, Shiping Wang, Minqi Jiang, Songqiao Han, and Hailiang Huang. Tsgym. https: /tt:://gttttt higgtuthhb....gggg www. SUFE- SSUUE-AILAE /TSGmm, 2025.
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models time series analysis: A tutorial and survey. Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 6555-6565, 2024.
Bryan Lim, Sercan Ô Ark,, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International journal of forecasting, 37(4): 1768,176, 2021.
Xinliang Liu, Bo Xu, Shuhao Cao, and Lei Zhang. Mitigating spectral bias for the multiscale operator learning. Journal of Computational Physics, 506:1 12944, 2024a. ISSN 0021-9991.
Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. In Proceedings of the ACM Web Conference 2024, pp. 4106, 2024b.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.0666 10.0622.. 2023.
Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. arXiv preprint arXiv:2402.0225 02366. 2024c.
12

---

Published as a conference paper at ICLR 2026
Haoyu Ma, Yushu Chen, Wenlai Zhao, Jinzhe Yang, Yingsheng Ji, Xinghua Xu, Xiaozhu Liu, Hao Jing, Shengzhuo Liu, and Guangwen Yang. A mamba foundation model for time series forecast- ing. arXiv preprint arXiv:2411.029// 2024.
Spyros Makridakis and Michele Hibon. Arma models and the box-jenn box-jenkins methodology. Journal of forecasting 16(3):147-166) 1997.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1): 54-74, 2020. doi: 0100666ffforrccas22199.0.0.01000
Luca Masserano, Abdul Fatir Ansari, Boran Han, Xiyuan Zhang, Christos Faloutsos, Michael W Mahoney, Andrew Gordon Wilson, Youngsuk Park, Syama Rangapuram, Danielle C Maddix, et al. Enhancing foundation models for time series forecasting via wavelet-based tokenization. In
International Conference Machine Learning. PMLR, 2025.
Depen Morwani, Jatin Batra, Prateek Jain, and Praneeth Netrapalli. Simplicity bias in 1-hidden layer neural networks. Advances in Neural Information Processing Systems, 36:8048-8005 2023.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64
words: Long-term forecasting with transformers. arXiv preprint 1.Xix:2211.117555 2022.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis
expansion analysis for interpretable time series forecasting. Internation Conference on Learning Representations, 2020.
Jaideep Pathak, Alexander Wikner, Rebeckah Fussell, Sarthak Chandra, Brian R Hunt, Michelle Girvan, and Edward Ott. Hybrid forecasting of chaotic processes: Using machine learning in conjunction with knowledge-base based model. Chaos: interdisciplinary journal of nonlinear science, 28(4), 2018.
Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, and Yasushi Sakurai. Fredformer: Frequency debiased transformer for time series forecasting. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 2400-2410, 2024.
Christopher Rackauckas. How chaotic chaos? how some ai for science sciml papers are overstating accuracy claims. https:// www.ssscchsstiiiifstseeeoo com/ how www-chaatic-in caa-chas-- how- some ai-for---------------------oa wor--forr-sss-eee science sciml -apllpappppa-reeeee are- terre-vvertaatng--eee accuracy 2025.
N. Rahaman, A. Baratin, D. Arpit, Draxler, M. Lin, F. Hamprecht, Bengio, and A. Courville. On the spectral bias of neural networks. In Inter. Conf. Mach. Learn., pp. 5301-5310. PMLR, 2019.
Riccardo Rende, Federica Gerace, Alessandro Laio, and Sebastian Goldt. A distributional simplic- ity bias in the learning dynamics of transformers. Advances in Neural Information Processing Systems, 37:96207-96228, 2024.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International journal of forecasingg, 36(3):1181- 1191, 2020.
Larry Schumaker and Annan Yu. Approximation by polynomial splines on curved triangulations. Computer Aided Geometric Design, 92: 102050, 2022.
Xiaoming Shi, Shiyu Wang. Yugi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts. International Conference on Learning Representations, 2025.
H. Son, J.W. Jang, WJ.. Han, and H.J. Hwang. Sobolev training for the neural network solutions of PDEs. arXiv preprint arXiv:2101.08932 2021.
Hwijae Son. Sobolev acceleration for neural networks. 2023.
13

---

Published as a conference paper at ICLR 2026
Sophie Steger, Franz M Rohrhofer, and Bernhard C Geiger. How pinns cheat: Predicting chaotic motion of a double pendulum. In The Symbiosis of Deep Learning and Differential Equations II, 2022.
Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression as classifica- tion: Influence of task formulation on neural network features. arXiv preprint 2i1i::221110566// 2023.
L. Su and P. Yang. On learning over over- parameterized neural networks: A functional approximation perspective. volume 32. Curran Associates, Inc., 2019.
Sabera Talukder, Yisong Yue, and Georgia Gkioxari. Totem: Tokenized time series embeddings for general time series analysis. arXiv preprint arXiv:2402.16412, 2024.
Artur Trindade. Fiettrriyhloadiiggaan00111111112 2015. 2015. URL htthtt http:////o.............aaa//a/// /ttp:///aae/eee/eeeeeeeeeaaaeeee ics uci. /21 adt./dataaeet/221leeeaaee http://tteeeeeese/eeeeeececcctad
C. Tsay. Sobolev trained neural network surrogate models for optimization. Comp. Chem. Eng., 153:107419, 2021. 2021.
N.N. Vlassis and W. Sun. Sobolev training of thermodynamii--nfooneed neural networks for inter- pretable glasto-plasticon models with level set hardening. Compu. Meth. Appl. Mech. Eng., 377: 113695, 2021.
Andrew Gordon Wilson. Deep learning is not so mysterious or different. PMLR, 2025.
David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE trans- actions on evolutionary computation, 1(1):67-82, 2002.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. 2024.
7.-Q. J. Xu. Frequency principle: Fourier analysis sheds light on deep neural networks. Commun. Comput. Phys., 2865)11746-1767, 2020.
G. Yang and H. Salman. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv:1907.10990 2019.
Annan Yu, Arnur Nigmetov, Dmitriy Morozov, Michael W Mahoney, and N Benjamin Erichson. Ro- bustifying state-space models for long sequences via approximate diagonalization. arXiv preprint arXiv:2310.01698 2023a.
Annan Yu, Yunan Yang. and Alex Townsend. Tuning frequency bias in neural network training with nonuniform data. International Conference on Learning Representations, 2023b.
Annan Yu, Dongwei Lyu, Soon Hoe Lim, Michael W Mahoney, and N Benjamin Erichson. Tuning frequency bias of state space models. International Conference on Learning Representations, 2025a.
Annan Yu. Danielle C. Maddix, Boran Han, Xiyuan Zhang, Abdul Fatir Ansari, Oleksandr Shchur, Christos Faloutsos, Andrew Gordon Wilson, Michael W. Mahoney, and Yuyang Wang. Under- standing transformers for time series: Rank structure, flow-of-ranks, and compressibility. arXiv preprint arXiv:25100033555 2025b.
Yuanzhao Zhang and William Gilpin. Zero-shot forecasting of chaotic systems. International Con- ference Learning Representations, 2025a.
Yuanzhao Zhang and William Gilpin. Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning. arXiv preprint 1=Xx::2505.113499 2025b.
Zhiyuan Zhao, Juntong Ni, Shangqing Xu, Haoxin Liu. Wei Jin, and B Aditya Prakash. Timerecipe: A time-series forecasting recipe via benchmarking module level effectiveness. arXiv preprint arXiv:2506. 256402200 2025.
14

---

Published as a conference paper at ICLR 2026
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang- Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11106-11115 AAAI Press, 2021.
Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained LM. Advances in neural information processing systems, 36:43322-4333 2023.
B. Zhu, J. Hu, Y. Lou, and Yang. Implicit regularization effects of the Sobolev norms in image processing. arXiv preprint arXiv:2109.00655,,20 2021.
15

---

Published as a conference paper at ICLR 2026
A TWo SIDES OF EACH BIAS
In the main article, we showed many examples in which bias may be "good" or "bad" in certain situations; and we also emphasized that each bias controls trade-off, and that there are important applications on both ends of the spectrum amples illustrating when bias may be "good" or "bad." (These examples illustrate learnings from developing Chronos- Bolt to "improve" Chronos on well-established benchmarks, and the observa- tion that Chronos-Bou Bolt performs worse than Chronos the wealitttiveel--iifeeeee chaotic nonlinear dynamics data.) The examples in this appendix are not cherry-picked, but they are highly context- specifie; for more generic and systematic analysis, see the main article. See Table 1 for overview of our discussion.
bias. this section, we show concrete synthetic ex-
The way to read Table is follows: for each bias, we compare two models on the two different ends of the spectrum. Then, we come up with two examples: in the first example, we show that having this bias is good for this particular example; in the second example, we show the opposite. A green background always indicates that forecast is qualitatively better, while red background shows otherwise.
A.I TEMPORAL BIAS
As described in Section 2, there are two biases related to the temporal bias: the frequency bias; and the periodicity bias.
A.1.1 FREQUENCY BIAS
The frequency bias has been known to the deep learning community for almost decade; and it is often thought as an implicit regularization. Driven by this, we first come up with an example where the context is
(1) where the noise is obtained by sampling every entry from the discrete sequence i.i.d. according to the standard Gaussian distribution A/0, 1). In this case, sin(t) is the useful low-frequee now-frequency information that one wants to learn, while the Gaussian noise is high-frequency content that we do not wish to learn. We see that with larger patch size, and therefore stronger frequency bias, Chronos-Bolt is more capable of "denoising."
sxt)= = sin(t) + noise,
On the other hand, the example in Figure 3 has already shown a case where biasing towards low frequencies is not desirable:
x (( sin( sin(
<
In this case, both the Now-frequency information sin (ot) and the contain useful information, but Chronos-Bolt with patch size of 16 only enables us learn the low-frequency content.
information (in()
A.1.2 PERIODICITY BIAS
To understand the pros and cons of preserving the periodicity bias, we can first consider a signal formed from chaotic system's 's data:
z (t) 1 (t) +
T A),
(3)
R is motif taken from Lorentz oscillator. Since F is repeated twice in the context, and that the task can be solved by simply "matching" the repeating motifs, we see that model that preserves periodicity better, i.e., Chronos-Bolt, performs better than one that destroys periodicity more, i.e., TimesFM.
where F(-) [0, A] +
However, if one focuses too much the periodic patterns, then one may overlook the important non-periodicities. Consider the following context:
x=(t) 1 R2 (in(()
1,, sin(t),
where R1 is the disjoint union of all even ones from an evenly divided set of intervals, and Rg = [0, oo) \ R1 is the complement of it. We see in Table that Chronos-Bolt focuses heavily the periodic patterns on R2, resulting in a bad forecast, whereas TimesFM gives a more accurate partition of R2 and R2 in its forecast.
16

---

Published as a conference paper at ICLR 2026
Table 1: Illustration of when inductive biases "help" and when they "hurt" performance.
Frequency GOOD
/ 'R' WW
Chronos Bolt (A 16:
carenos Bolt
Frequency BAD
Regression GOOD will
www.o noro
Chronos
Chroros-Pa -Rot
Regression BAD
Locality GOOD
Chrenos
Cazenos Bolt
Locality BAD
Scale GOOD
Chrenog-Bolt
Scale BAD
Offset GOOD
Chnooss-Rolt
Chroroa
Offset BAD
dushed gray lime
Periodicity GOOD
Chrenos-Bols
1'im
Periodicity BAD
17

---

Published as a conference paper at ICLR 2026
A.2 GEOMETRIC BIAS
As described in section 3, there three parts to the geometric bias: angular bias, which cerns the angles between the embeddings; a distance bias, which is due to the distance between the embeddings; and norm bias, which originates from the homogeneity of the ReLU activation. We show interesting examples for each of them.
A.2.1 ANGULAR BIAS
A model that has an angular bias, c.g., Chronos, tends to tailor the "parroting" strategy. When there matching motif in the earlier context, then the model will parrot what is after it. Therefore, we set the context to be perfectly periodic signal
xxf) = sin(t);
(5)
then that Chronos' forecast is pinpoint, while Chronos-Boo suffers from certain small errors.
However, parroting can also be dangerous, not only because of the fact that sometimes there is not much to parrot, but also since it can prevent the model from identifying more subtle patterns. Consider the following context:
(t)) = sin(t))(tt.
(6)
where S(t) bidirectional scaling factor that vanishes as t - =oc. In this case, Chronos is able to identify the recent context to previous motif. Unfortunately, this "drags" Chronos into caveat: simply parroting the previous context results in a forecast with an increasing trend, whereas the actual trend is clearly decreasing over time. In this case, Chronos- Booooos-Bolttte gets better forecast than Chronos.
A.2.2 DISTANCE BIAS
Similar to the frequency bias, fine-scale oscillation may sometimes be simply due to random noise. In that sense, the distance bias could be helpful for denoising. Consider the following context:
xtt) sin(t) + noise,
where the noise is obtained by sampling every entry from the discrete sequence i.i.d. according to the standard Gaussian distribution (O, a) for some small a, and then passing through low-pass filter. The reason why we choose a small is that we want to make the noise small-scale, and the reason why we pass through the low-pass filter is to decouple the example from the frequency bias. Here, we see that Chronos learns these small-scale noises, while Chronos-Bol having a distance bias, only learns the large-scale trend.
Of course, when small-scale properties appear to be important, Chronos performs better than Chronos-Bon Bolt. One example shown in Figure 6, where
zt) sin Sin(t)((t),
where S(1) is scaling factor that periodically alternates between large and small values.
A.2.3 NORM BIAS
Regarding the norm bias. it may not be clear why one should treat signals at different offsets dif- ferently. Indeed, if we consider the signal labeled "Dffset-BAD) Table 1, we see that Chronos is much better at continuing the signal near zero than Chronos-Bolt. However, sometimes the small signals the input domain are really "physically" less important than the large signals. One im- portant example is with sparse signal. For example, consider the signal labeled "Offset-GOOD" in Table 1. This signal is sparse, consisting of mostly zeros and a few non-zero motifs. Essentially, the most important information is contained in these non-zero motifs. Chronos-Bad 's MLP embed- ding assigns the zero values small embeddings, and focuses more on the non-zero parts, resulting in better prediction than Chronos.
18

---

Published as a conference paper at ICLR 2026
A.3 REGRESSION-TO- TOE-HE-MEAA BIAS
As described in Section 4, it may be intuitive that regression algorithms should regress to the mean, but an example where this bias is not desired is provided by the chaotic systems data from Zhang & Gilpin (2025a). (Probabilistic forecasting has become an important paradigm real-world ap- plications, from weather and climate prediction (Gneiting & Katzfuss, 2014), to energy demand forecasting (Hong et al., 2016), to financial risk management (Gneiting et al., 2007); and evidence suggests that similar results may hold more generally for TSFMs.) In particular, we use the Lorentz oscillator to generate our context. We see that since the system is chaotic, neither forecast stays close to the ground-truth, but Chronos' prediction resembles the context much more than Chronos- Bolt's. s. That is, while from MSE or MAE perspective, Chronos is not doing much better than Chronos- Bolt, performs better on other aspects, such as the frequency loss or the fractal dimen- sion (Zhang & Gilpin, 2025a). More colloquially, without showing the ground-truth, Chronos gives a more credible continuation the context and Chronos-Boo
Clearly, of course, in many other cases, regression-to-the-e to-the-mean is desirable. A significant example of this arises when outliers are introduced. For example, consider the following context:
(()) si) + Bernoull(p)):,
where is the Dirac-delta and Bernoullipp) is i..dd random variable with small probability 0 < < 1. This signal can model a demand curve that follows some seasonal trend but can be excited by several random events. In this case, we see that Chronos sometimes predicts these outliers, which we do not want, while Chronos-Boo Bolt generally avoids them.
19

---

Published as a conference paper at ICLR 2026
B RELATED WORK
In this section, we describe several lines of related work.
B.1 TIME SERIES MODELING AND FOUNDATION MODELS
Classical time series forecasting (Hyndman & Athanasopoulos, 2018) is grounded in statistical methods such as ARIMA (Makridakis & Hibon, 1997) and exponential smoothing (Gardner Jr, 1985). More recently, deep learning approaches, e.g., N-BEATS (Oreshkin et al., 2020), have be- come prominent. Sequence models such as DeepAR (Salinas et al., 2020) popularized probabilistic forecasting at scale, while the Temporal Fusion Transformer (Lim et al., 2021) combined attention mechanisms with interpretability for multi-horizon prediction. Building on these advances, recent work has focused pretraining general-purpon time series models that transfer across datasets and tasks. Notable examples include TimesFM (Das et al., 2024), Chronos (Ansari et al., 2024a), Moirai (Woo al., 2024), and Time-MoE (Shi et al., 2025). These models vary in tokenization strategies, training corpora, and zero-shot protocols, but they all share the common goal of develop- ing a single "foundation model" that generalizes across domains and horizons. Such TSFMs, which we compare along several axes in Figure 2, include Chronos (Ansari et al., 2024a), TimesFM (Das al., 2024), Moirai (Woo al., 2024), PatchTST (Nie et al., 2022). Time MOE (Shi et al., 2025), -hron (Ansari et al., 2024b), iTTrnsformer (Liu et al., 2023), Moment (Goswami et al., 2024), Timer (Liu et al., 2024c), GPT4TS (Zhou et al., 2023), UniTime (Liu et al., 2024b), and TOTEM (Talukder al., 2024).
B.2 DESIGN CHOICES OF TSFMS
Design choices for TSFMs have caught research attention, c.g., with empirical work in Liang et al. (2025); Zhao et al. (2025). Beyond these benchmark studies, several papers have investigated spe- eific design dimensions of TSFMs. For instance, recent work has examined the impact of tok- enization strategies (Nie et al., 2022; Talukder et al., 2024), scaling laws for model size and dataset coverage (Woo et al., 2024; Shi et al., 2025), and architectural choices such as state-space models versus Transformers (Gu & Dao, 2023; Ma et al., 2024). Other efforts have focused on evaluation protocols and generalization across domains, highlighting the importance of systematic comparisons to guide the development of more reliable TSFMs.
B.3 DEEP LEARNING MODELS FOR CHAOTIC SYSTEMS FORECASTING
We have used time series from chaotic dynamical systems data as part of our investigation. Re- cent work (Rackauckas, 2025) has highlighted the promise and limitations of using deep learning models to Iearn chaotic systems. A major line of work from Gilpin's is group has made fundamen- tal contributions, including providing a benchmark of many state-of-the-art methods rranhin from reservoir computing to TSFMs showing that large, domain- predictions up to two dozen Lyapunov times into the future (Gilpin, 2021). Further empirical stud- ies demonstrate that TSFMs like Chronos can perform zero-shot forecasting of chaotic dynamics, preserving the geometric and statistical properties of chaotic attractors even when point forecasts degrade (Zhang & Gilpin, 2025a;b; Lai et al., 2025). Beyond TSFMs, physics-informed networks leverage known physical constraints to extend predictability limits (Doan et al., 2020; Steger et al., 2022), while knowledge-basee hybrid models combining mechanistic and data-driven components reliably forecast complex chaotic behaviors far longer than either approach alone (Pathak et al., 2018).
models can achieve accurate
B.4 FREQUENCY BIAS
The frequency bias is related to the spectral bias neural networks, which was observed and studied in Rahaman et al. (2019); Yang & Salman (2019); Xu (2020). The term "spectral bias" comes from the spectral decomposition of neural tangent kernels (NTKs) (Jacot et al., 2018), which approximate the training dynamics of overparameterized networks (Arora al., 2019; Su & Yang, 2019; Cao et al., 2019). By analyzing the eigenfunctions of NTKs, Basri et al. (2019); Bietti & Mairal (2019) formally proved the presence of spectral bias in two-layer overparameterized networks with uniform
20

---

Published as a conference paper at ICLR 2026
input distributions; and these results were later extended nonuniform inputs (Basri et al., 2020; Yu et al., 2023b). Several approaches have since been proposed to tune the strength of this bias, including Sobolev-norm-base training strategies (Vlassis & Sun, 2021; Yu et al., 2023b; Tsay, 2021; Son et al., 2021; Czarnecki et al., 2017; Zhu et al., 2021; Son, 2023; Liu et al., 2024a) and frequency-aware initialization methods (Yu et al., 2023a; 2025a).
21

---

Published as a conference paper at ICLR 2026
C DETAILS OF THE TEMPORAL BIAS (FROM SECTION 2)
In this section, we provide more details the temporal bias, which was discussed in Section 2. We discuss both the frequency bias (Appendix C.1) and the periodicity bias (Appendix C.2).
C.I DETAILS THE FREQUENCY BIAS
In this subsection, provide more details the frequency bias, of the two temporal biases (the other being the periodicity bias; see Appendix C.2), from section 2. Our discussion is split into several parts. First, prove Theorem 1 and discuss some of its consequences (Appendix C.I.I). Then, we provide more results on the attention matrices a Transformer with patching to corrobo- rate the frequency bias (Appendix 1.1. Finally, show the existence and non-existence of the frequency bias in two more interesting scenarios (Appendix C.1.3).
C.1.1 PROOF AND DISCUSSION OF THEOREM
THE PROOF
The proof of Theorem is an application of concentration inequalities and matrix norm bounds once we realize that the inverse Fourier transform is an orthogonal transformation, mapping distinct Fourier modes to orthogonal subspaces in the patch space R
Theorem 1'. Let h be patch size, 1 < < k be any positive integer, and w S min(k/n, n) n) 1 be bandwidth. Let B1. Bg...., B, C [R]] == (1,...... k) be n mutually disjoint sets whose bandwidths are below i.e., [Bil S. w for every 1 S i S Let unit vectors Vi,..., Vn E I be patches that belong to the same band, i.e. supp v:) C B, for 1 S I n and a common j. Let unit vectors u1,...; Up R2 be n patches that belong to different bands, i.e., supp(ú,) Vn R and U In ((x) = WeReLU(Wix + b a neural network embedding, with W1 E Rmxk W2 E Rdxm and b E R The following two statements hold:
B, for
all < S n. Let V = [v1
un R and let
1. (Patches of similar frequencies are embedded in shared subspace.) Let 0
1 be given. Given any neural network width m and any W, and W₂ such that ( and any bias term b such that |lb 1, l|r
[Vo(V)]lz
( ReLU(W V)I we have
rank. ( = H6 = stab-rank (obb-rank((()))) =
> col = O(w).
2. (Patches of different frequencies embedded in nearly orthogonal subspaces.) There exists universal constant и > so that the following statement holds. Assume that W, 1 is a random Gaussian matrix whose entries follow i...d. A/0 a)) distribution and W₂ is random Gaussian matrix whose entries follow i.i.d. (0, ,3) distribution, where a and 3 are any positive numbers.
Set the bias term to be b = - WW m /2TT. Then, for any 0 < 8 < 1, m = M(log(165). (((oog(1)) (rflo(nn/)) (n)))) m n, and m/d d = (1). with probability no less than
have
rank = 2(m), stab-rank ((()nn(((UU))))))))))))))()) = 2(n).
Moreover, the constants in all O and Ω-notations are universal.
Proof of Theorem 1'. We prove the two statements separately.
Proof of the First Statement. the supports of vi,..., v, all belong to set B, with a cardinality no more than w, then the vectors vi,..., Vn are contained in subspace of dimension the discrete Fourier transform is an orthogonal operator, the vectors V1..... V, are also contained in subspace of dimension w. Therefore, have that rank (V) I w and that
Since
||V| W S Vai (v & (V) (Wai(V).
22

---

Published as a conference paper at ICLR 2026
Since the ReLU activation function cannot increase the norm of an input matrix, therefore have
l - [W₂ ReLu(W, V)Ile + [b1, lI
[]]] R,, V)lle s WW.////la//eaaL V)
I [O(L)]Walll//W V & (())
& (())) W Wella W 1/22 Vil2.
By our assumption, we have that d (o(V)) N 2(D)///////////lll/ 1|2, and the statement about the stable rank follows immediately. Thus, we have
(ank((((V)) S Ej= ")
2 (Q(() )²
[
| 2
(
V W2 W Wi l2/l 2 e2 W2l2 W V = =(11) ww.
Proof of the Second Statement. We will break the proof into three steps. In the first step, lower-bound the Frobenius norm of ((U). In the second step, we upper-bound the spectral norm of & ((U). This gives us an estimate of the number of large singular values, which we formally prove as the last step.
Step Since the ReLU activation function is homogeneous on [0, oi), we may assume, without loss of generality, that 3 1. Since the vectors ui...., ü, supported on mutually disjoint sets indices, they are orthogonal to each other. Moreover, since each of them a unit vector, they orthonormal. Since the inverse discrete Fourier transform is orthogonal operator, we have that the matrix has orthonormal columns. Therefore, the matrix W, U is still random Gaussian matrix whose entries are i.i.d. distributed like (N, 1). That is, we clearly have that
E [[ReLLU U) mn,
where the 1/2 comes from the fact that ReLUU fires each neuron with probability of 1/2. More- over, by estimating the lower tail of the x² distribution, we know that for some = (2(log(1/6)) with probability no less than 1 8/5, we have that
3 2 mn.
(RLLU(W.U)IE 2
(10)
From now on, we instantiate W, and assume that eq. (10) is achieved. Let Z = ReLU(W,U), N1 ||]| F. and let Y WzZ. Let w. be the ith row of the matrix W2. We have
|
W 2
w 2
de
|||
w
Z
2
d
E
[(w] 2zy 27.
However, since W ~ (N, I,). we have
E [[w]2,)}] = 1,.112 =1
This gives
[ww] 2,) =
||, = d
j=1
j=1
By another Chernoff bound, we have that for d = (2(lo((n/)) with probability no less than 1 1- 8/5, that
YY 2 2(1) dmn.
(11)
Clearly, for d = ())) with probability no less than 1 - -8/5, we have
6 dm
6 dmn.
W <
W W 22 < x 2m
(12)
5
23

---

Published as a conference paper at ICLR 2026
By union bound, with probability no less than 36/5. have that eq. (10), (11), and (12) hold. When these hold, the triangle inequality of the Frobenius norm gives us
2
{ z Y W221 112lmxn/V22//
dmn > dmn.
(13)
5
Step II: We can write the output of the neural network
(UU) = W,ReLLU(,UU)
W21mxn W2 Rel.U(W.U) 22m
1 mxxn
V2T
R
First, since W2 is random Gaussian matrix whose entries are (O, 1), for ((log(n00)) and m = (llog((6))) defined above and with probability no less than 8/5, we have
W2ll2 ((1) Vm).
(14)
Now, consider the matrix R. Since U has orthonormal columns, we know that the matrix W1 still has i.d. entries, and does R. Moreover, each entry in R clearly has sub- Gaussian distribution. Let R be an entry of R. then we have
E(R] ReLU(Z)
(R)LU(Z)]
=
2
V2x
where Z ~ No, 1) and it is well-known that E [ReLU([Z)] = 1/V2T. m (((og(1/6)) with probability no less than - 8/5, we have
Since we assumed that
||||2 O(1) + vn).
(15)
By a union bound, eq. (14) and (15) hold with probability no less than 28/5. When these equations hold, we have by the sub-multiplicativity of the spectral norm that
|| (UUIl22 = W2R S || [R||2 = ((1)(v + m) = 0(1)vdm,
(16)
where the second last step comes from the fact that and the last step follows from the fact that d/m- = (1).
Step III: By a final union bound, we can assume that eq. (13) and (16) hold with probability of no less than 1 5. Let o..... on be the singular values of (UU). Then, we have that
of
o2
(U))i
2(1) dmn 2(n).
of
dm
Since the constant in the Ω-notation is universal, there must exist universal such that least ((n) of the relative singular values 0j/01 = o3 o are > e. The proof is complete.
INTERPRETATION AND DISCUSSION OF TECHNICAL DETAILS
pictorial description of Theorem 1 can be found in Figure 10.
As next step, notice that our theorem relies on a list of crucial assumptions. We anticipate some questions on them, and now we discuss why certain assumptions are made.
Why do we assume that the support of vector u v strictly in a band?
We make this assumption for the simplicity of the proof and the cleanliness of the final statement. In practice, when w is small, one hardly encounters patch whose support is strictly limited within a narrow band; instead, is often the case that a patch has its Fourier coefficients dense in band and supported with small values elsewhere. To prove a result this case, one can view these patches as small perturbations of strictly band-limited patches, and statement will follow using Weyl-type perturbation analysis.
24

---

Published as a conference paper at ICLR 2026
Discrete Fourler Domain
k
T
Low-Rank Embedding
T
2-layer MLP
22
n
:
High-Rank Rank-Rank E Embedding
T
n
n
d
0
Discrete Fourier Domain
if we sample number of patches from the same narrow frequency band, then they are embedded into a small subspace; while we sample the patches with different frequency contents, then they embedded into different subspaces of the hidden space.
Figure 10: An illustration of Theorem 1:
the first statement, why we assume that [[ 2(11) W2ll2 Will2||V/2?
If we assume no ReLU activation is used, then this condition simply reduces to the general tightness of the uub-muliplicaatiiity of the spectral norm. However, introducing the ReLU activation function usually only increases the spectral norm of the output. Hence, the lower bound is natural to assume.
In the first statement, why do we assume that |[b1]] -(0]]=-OOWW)W//l//] W.ReLU(W.V.))
This assumption simply says that when computing the output of two-layer ReLU- activated MI.,, we are not asymptotically dominated by the bias term, which is natural one to make.
In the second statement, why do we assume that -W,1m/ 2
This is perhaps the most arguable assumption that we have made. It is a technical as- sumption that we will further empirically justify in our numerical experiments. At a high level, if we assume that W1 and W are random Gaussian matrices, the ReLU's output ReLU(W, V) does not have a mean O. Since it does not have a mean of zero, in ex- pectation, it will introduce rank-1 perturbation in the output of the MLP. The bias term is there to recenter the intermediate output ReLU(W) V) to avoid the rank-1 component. Without the bias term, the output will be the of large rank-1 matrix and full-rank matrix, making our statement about the stable rank invalid, but the statement about the g-rank remains true.
25

---

Published as a conference paper at ICLR 2026
NUMERICAL EXPERIMENT
10
10
stable rank
8-rank
102
10
E 10-25
10
rank
10
100
2
6
8
10
12
14
16
band width
Figure 11: The s-ranks and stable ranks of the output matrix of a randomly sampled MLP layer, given an input matrix sampled from single frequency band whose width is limited by w. As w increases, the ranks of the output matrix grow. The reference line has a slope of 1 in this log-log plot.
We conclude our discussion of Theorem by presenting three numerical experiments corroborate the theorem statement and elaborate on some technical assumptions made. Our first experiment validates that the bandwidth limits the rank of the input embedding. For this experiment, set d = 1024, m = 4096, n = 1024, and = 64. Our MIP is obtained by randomly sampling Gaussian matrices W₁ and We and defining the bias term be the centering factor W²lm as explained earlier. For fixed bandwidth w, we sample n patches whose supports are contained in fixed band of width w. We then simulate the two-layer MLP to compute the output matrix.
In Figure 11, we see that as increases, the stable rank of the output matrix follows the linear growth, which corroborates the first statement in Theorem The g-ranks are slightly more compli- cated because they are upper bounded by min(d. d, n) and saturate easily. In practice we see that the growth of the g-ranks is initially faster than linear and eventually slows down.
In the second experiment, we compare the two statements made in Theorem 1. To this end, we design an experiment, where we fix the bandwidth to w We still use 1024, m 4096, and & = 64. However, change the number of patches sampled. For each we sample these patches in two ways:
A sampling that resembles the first statement: we sample the patches from fixed single band that has a width of w.
A sampling that resembles the second statement: we sample the n patches from n mu- tually disjoint bands in the Fourier domain. Note that this requires 2 which the reason why we choose small w for this experiment.
We then simulate the two-layer MLP sampled in the way explained above.
26

---

Published as a conference paper at ICLR 2026
Patches from the Same Band
Patches from Different Bands
30
00
25
25
07
0.
0.5
10
03
50
Figure 12:
The relative singular values of the output matrix. On the left, we sample all n patches in the input matrix from the same w-band in the Fourier domain. On the right, sample n patches from the n mutually disjoint w-bands in the Fourier domain.
From Figure 12, we see that if we sample the n patches from the same band, then there is limited number of large singular values, no matter how large n This corroborates the first statement. On the other hand, if we sample the n patches from different Fourier bands, and therefore inject different frequencies into each of them, then all singular values are fairly large - in fact, none the singular values are below 01 / indicating that the output matrix is numerically full-rank. This corroborates the second statement in Theorem 1.
The last experiment show concerns the necessity of the bias term b in the second statement of Theorem 1. As explained carlier, this term is used to centralize the intermediate output. In this experiment, do not the bias term, and we repeat the experiment shown in Figure 12. We that compared to the case with the bias term, the output matrices are essentially the sum large rank-1 matrix and smaller full-rank matrix. That is, for reasonably small n and reasonably large E, the e-rank of the output matrix is still large and growing with n.
Patches from Different Bands (no
The Second Relative Singular Value
0.7
30-70 %
0.6
median
57
20
88
0.5
16
34
0.4
10
92
0.3
2
4
16
32
Figure 13: On the left, we reproduce the experiment in Figure 12, but without the bias term b in the neural network. We see that removing the bias term makes the full-rank output sum of large rank-one matrix and small full-rank matrix. On the right, we show the gap between the rank-one term and the full-rank term as the number of sampled patches n increases, by measuring the second relative singular value 02/ 01 of the output matrix. The reference line has a slope of -1/2.
As n increases, one needs to set e smaller to still have most of the relative singular values of the output larger than E. In fact, theoretically. it is relatively easy to show that we need so that the g-rank of the output matrix, without the bias term b, is proportional to n as it increases. This is still much larger than the output of patches from the same band. In Figure 13, we also repeat this experiment for 100 randomly sampled neural networks without the bias term, and we show the second relative singular value 02/01 of the output matrix, which measures the gap between the large
1/vm,
27

---

Published as a conference paper at ICLR 2026
one matrix and the small full-rank matrix. We see that this gap seems to grow slightly slower than nn, which is suggested by the theory.
C.1.2 VISUALIZATION OF ATTENTION SCORES
Recall that, in Figure 3, we saw that the low- and high- figh-frequency patches are embedded in two nearly orthogonal subspaces of the hidden space. also the case that the attention scores corresponding to the low-frequency patches are much larger than those corresponding to the high-frequency ones. In Figure 14, we show the magnitudes of the averaged (across all heads) attention scores the first layer of the Prronos-Bolt's encoder. The first shows the attention scores when we give sin (8t) as the input, and the second one shows the attention scores when we input sin(1)))). We can see that since the attention kernel W W is much better aligned with the low-frequency patches, the scores the first panel have a much higher order of magnitude than the scores in the second panel. Therefore, when we superpose the Fourier modes form the composite input, the attention scores mainly follow the periodicity of 8 instead of 117.
Law Fureoneyy
Hug: Frequeney
Overnil
30
30
5 10
12
The attention scores before softmax, i.e., Q K E LXL where L is the length of the in- put sequence (i.e., the number of patches). We see that overall, if the patches are low-frequency, then the attention scores have much larger magnitudes than if the patches are high-frequency. Therefore, when a signal contains both low-frequency content and high-frequency one, the attention scores dominated by the (periodicity of) the low-frequency information.
Figure 14:
C.1.3 ADDITIONAL EVIDENCE: SUPERPOSING TWO FREQUENCIES WITHOUU ALIASING
Recall that, in Figure 3, we saw an example where our context sampled from the signal
f(t) = sin()t) + sin(117t).
In that case, we know that if we use a patch size of = 16, then the low- and high-frequent information is stored separately into different subspaces. Moreover, since the attention scores dominated by the low-frequency information, the high-frequency patterns are ruined during atten- tion, resulting in poor learning capability of the high-frequency information.
28

---

Published as a conference paper at ICLR 2026
46-
-11
Figure 15: On the left, show the forecast of Chronos-Bol Bolt model (with patch size k 16) where the periodicity of high-frequency information in the context does not align with that of the low-frequency information. On the right, we show the forecast when the periodicities of the high- and low-frequency align. We use red color to indicate the same motif in the cow-frequenc, si- nusoidal wave, where we see that the high-frequency information in different motifs looks very different on the left, while it looks the same on the right.
Here, we show one additional example where we change the high-frequency mode by a little bit, then completely change the models' performance. In particular, consider the following context:
f(t) sin(St) sin(128t).
In sense, 128 is even "larger" than 117; but when sample this signal and use Chronos-Bolt model with h = 16 to predict, we see that the performance becomes much better (see Figure 15)! What makes that happen? Notice that 128 is an integral multiple of 8, but 117 is not. Therefore, even if the does cause any issue in preserving the high-frequency information. That is, we do not suffer from aliasing errors. (As an analogy, when you use your cellphone's 's camera to take picture of the screen of your television, you often notice some stripes this is due the mismatch between the resolutions of your digital camera and the television screen - but there are certain cases where the stripes do not appear, which happen exactly when the resolutions match each other.) For example, in Figure 15,
information is attended in the way dominated by the low-frequency, it
red colors to indicate the same motif in each period of the low-frequency wave sin(St), (8t). On the left, when the periodicities of sin(St) and sin(117t) do not match, we see that the high-frequener information in each motif is different; but since the periodicities of sin sin(St) and sin 128)) match each other, we have exactly the same high-frequency content in each red motif.
The significance of this example is two-fold:
1. First, it validates our hypothesis that the attention scores are mainly driven by the low-freqqency content that results in the frequency bias when using large patch size.
Second, it shows that the frequency bias a guaranteed notion. When the periodicities of the high- and low-freae wwwfrrequecc match each other, its impact is significantly reduced.
Another Use Case of the Frequency Bias: Concatenation. Previously, our discussion mainly focused on the where we have superposition of low- and high-free fighufrreeeecy, because this is usually the way that people care about the frequency bias. However, there is another way that two different frequencies come into the same context via concatenation.
29

---

Published as a conference paper at ICLR 2026
Corest Goard Tush
Chooens Be'k
Pott
MW
We show the forecast of two models, Chronos-Bolt with patch size k and one with 16, given a context formed by concatenating low-frequency modes and high-frequency ones. We see that the high-freet fregffrryeency information significantly bleeds into Chronos-Bolt forecast when 1, but the forecast better when the patch size 16.
Figure 16:
Angles between Embedded Vectors
Clomass Bab (2-1)
Closaos Boh (6-15)
WA
A
Magnitude of Attention Scores
005
WW V
Figure 17: We show the angles between different pairs of embedded patches the hidden space, given the context in Figure 16, as well as the magnitudes of the attention scores.
30

---

Published as a conference paper at ICLR 2026
In Figure 16, show the forecast of Chronos-Boo models, with a patch size of k 1 and 16, respectively, given an input formed by concatenating low- and high-frreqeene motifs. In this setting, see that having larger patch size helps prevent the high-frequency information from bleeding into the low-frequency forecast.
This finding is not surprising, and it provides yet another example of Theorem 1. The fact that high- and owwfrequency patches are orthogonal to each other in the hidden space means that they are un- likely to interact with each other in Transformer (see the top row of Figure 17). From the attention scores shown in Figure 17, we see that with a patch size of 1, a lot of the mation attended the low-frequency information. However, when & 16, the patches mainly talk to the high- frequency ones, and similarly for the low-frequency patches, but the low-frequency patches rarely attend to the high-frequency ones. This explains why we have less contamination using a larger patch size.
infor-
C.1.4 DEVELOPMENT OF THE FREQUENCY BIAS DURING TRAINING
In the paper, we see that the attention matrices in a pretrained Chronos-Bolt model are naturally better aligned with fow-frequency patches. In this section, we show how the frequency bias evolves during training.
1000
Uie 1855
20000
Figure 18: The evolution of the kernel weights W Wx x in Chronos-Boo during training. For each step number shown in the figure, we embed many low- wwwfrrquency patches along with many high- frequency patches. We then project these patches onto their leading two principal components the hidden space. The light green Ellipse shows the projection of the attention kernel W two-dimensional space.
Q W K in this
In Figure 18, we save a few checkpoints of Chronos-Boo during training and project the embeddings of low- and high-frequency patches onto their leading principal components. We see clear orthog- onal relationship between the low-frequency embeddings and the siggfrrey embeddings. In addition, Figure 18 shows how the weight matrices W q and W к are trained to obtain the frequency bias.
C.2 DETAILS OF THE PERIODICITY BIAS
In this subsection, we provide more details on the periodicity bias, one of the two temporal biases (the other being the frequency bias; see Appendix C.1), from section 2. Our discussion is split into two parts. First, we perform more detailed analysis of the [REG] token's 's role in the preservation of the periodicity (Appendix C.2.1). Then, we briefly discuss how we measure the periodicity bias induced by patching (Appendix C.2.2).
C.2.1 AN INVESTIGATION OF THE [REG] TOKEN
Recall Figure 4, where we noted that when the [REG] token is present, Chronos-Boll Transformer becomes significantly worse preserving periodic patterns. To understand why this happens, present a case study where our input is a sinusoidal wave. We pass this sinusoidal wave into Chronos- Bolt's encoder (patch size k = 1) and compute the context layer. Here, we have d = 512 and L 512. There are 8 heads, so the first head of z is a matrix (() E R84x512 We plot all 64 channels of (1) in Figure 19. We see clearly some channels where the periodicity of the input context is seriously affected. Those channels are highlighted in red.
Rdx after applying the first attention
31

---

Published as a conference paper at ICLR 2026
Channels of the Encoded Context
55-
400
210
430
200
400
-2.3
-0.5
-15
200
400
210
400
202
420
200
400
25
0.5
3.0
3.5
0.0
1.0
-25
-2.
-0.5
-15
160
350
450
660
25-
2.2
0.5
0.5
3.0
160
200
450
240
400
-2.5
-0.5
-0.5
200
400
210
400
202
420
200
400
25
0.0
-25
-2.5
-0.5
-1.
210
25
2.2
0.5
1.5
3.0
160
120
430
240
400
-5.5
200
400
210
400
202
430
200
400
3.5
3.5
0.5
0.0
-25
-0.5
-0.5
-2.5
32

---

Published as a conference paper at ICLR 2026
55-
400
220
430
400
2.-
-15-
-5.5
-0.5
-4.
200
3.5
3.5-
0.5-
0.5
3.0
2.5
0.0
1.0
-15-
-5.5
-0.5
-1.5
260
430
660
0.5
Row
3.0
400
220
430
400
13
-2.5
-0.5
-1.5
100
200
2.5
0.5
3.0
3.0
0.0
1.0
-35-
-5.5
-0.5
-45
250
650
460
3.5
0.5
3.0
400
220
420
200
400
Embedding of the EOS Token
0-
1-
Figure 19: We give a sinusoidal wave to the encoder of Chronos-Boll model (patch size k = 1) and take the pre- W o output from the first attention layer. We plot the 64 channels in the first head of this output, in the row-major order. We see that the periodicity in several channels, highlighted in red, is much less-well preserved than in others. The bottom panel shows the 64 values corresponding the first head of the [REG] token in the first attention layer, where we see the large values correspond exactly the very non-perion non-periodic channels.
pre-Wo
Now, claim that these serious losses of periodicity primarily due the existence of the [REG] token. In the bottom panel of Figure 19, we show all 64 values corresponding to the [REG] token. We see that there are clearly couple of them that are large magnitude. More strikingly, there is strong correlation between the magnitude of the [REG] token and the loss of periodicity. That is, the entries of large values in [REG] are exactly the channels (!)) that have poor periodicity.
33

---

Published as a conference paper at ICLR 2026
A "Bad" Channel
A "Good" Channel
encoded context
0.75
0.50
0.00
0.25
100
att Score
10'4
16-2
10.34
10
20-4
12-4
100
203
adc
402
550
302 ado 502
channel valuee
1.0
0.02
0.00
0.5
100 203 320
400 502
Figure 20: We show how very non-periodic channel and periodic channel from Figure 19 are computed. On the top row, we plot these two channels, and the stars are the positions that we investigate in the bottom two panels of the figure. The starred position is computed by taking the inner product of the attention scores vector, shown in the middle row, and the value vector, shown in the bottom row. We see that the attention scores used to compute the starred position are particularly large at the [REG] token, and the large value of the [REG] token seriously destroys the periodicity of the signal.
This is not hard to understand. If the [REG] token has large value, without masking, this value is written into a previous position that has large attention score on it, perturbing the periodic pattern. To illustrate this with clear example, severely destroyed, and "good" channel. In Figure 20, we see that the main difference happens at the starred position, where the bad channel, the output blows up at this position. In the middle row of Figure 20, we show the attention scores that are used to compute this position. We see that the last attention score is very large, meaning that a the starred position. More precisely, the starred position is computed by taking the inner product of the attention scores and the channel values:
take "bad" channel of z(1), where the periodicity is
the [REG] token's value is written into
Σ 1=1
where ..%.) are the self-attention scores shown in purple and (Ve,,...., Vx,z) are the values in the kith channel shown in green. Since of- is large, whether the periodicity is well-preserved reduces to a question of whether Vk.r. is large. That is why we see the strong coupling of the periodicity and the magnitude of the [REG] token's 's value in Figure 19.
The question that remains is: does removing the [REG] token help? The answer to this positive. In Figure 21, show two models: in the first model, we compute Chronos- Boll'ss encoded values as usual, which corresponds to exactly the figures shown in Figure 19; in the second model, we apply masking to prevent the [REG] token's 's value from being written into the previous tokens, i.e. the [REG] token is write-only. We show both random channels before and after applying the channel mixing layer Wo. Clearly, masking the [REG] token helps the preservation of the periodicity.
34

---

Published as a conference paper at ICLR 2026
Pre-Wo Context
Post-Wo Context
Sample
[REG] without [REG]
without [REG]
Sample
We give sinusoidal wave the encoder of Chronos-Bool Bolt model (patch size le 1) and take the pre-Wo and p output from the first attention layer. We compute the output two ways: using the default setting where every position is attended another position; and applying mask to prevent the value the [REG] token from being written into the previous outputs. We see that with masking, the periodicity much better preserved.
Figure 21:
C.2.2 DESIGN OF EXPERIMENTS IN FIGURE
Recall Figure where we explained that patching also hides the periodic patterns. This is also not surprising. The simplest way to view this is by considering aliasing. If the patch's periodicity mismatches periodicity of the input signal, then we lose periodic information by patching. To show this, we generate two histograms in Figure 4 as follows:
1. Randomly sample a sequence x RR from Chronos' training dataset (Ansari et al., 2024a), which includes both real-world and synthetic time series. Sample until we have L >>>>> 64.
2. Take the last 64 elements of x: x* X(L-6141):L
Break the context into patches and form motifs length 64: x()) = X1:61, x( =x(1+))((44*k) x (3) (64+2)) where none of them overlaps x".
Compute the R² -core between x defined in step 2 and each x formed in step We call the maximum R² score the "best matching score."
We repeat this step for over 10, 000 sampled sequences to form the histograms in Figure 4. Figure 22 shows results for different patch sizes k.
35

---

Published as a conference paper at ICLR 2026
Maximum Correlation Between Motifs
patch
pasch size
17
27
patch size
pasch size 16
pasch size 32
32
maccimmm anlocorreeation
maximam antoocorrelttinn
I0 maciimam sutocorrelation
Figure 22: The maximum correlation between the last motif in context and each motif in the carlier context, which are defined by patching. We see that as the patch size increases, correlations generally become smaller, indicating more challenges in identifying the periodicity the context.
Minismm Distance Between Matifs
paldi
paidh.
paids size
paidh size
paich size 36
paich sice 82
3/
maximmm antosorrelation
maximam antoeeorelation
TB maxiiman autocorrelation
The minimum distance between the last motif in a context and each motif in the earlier context, which are defined by patching. We see that the patch size increases, distances generally become larger, indicating more challenges in identifying the periodicity the context.
Figure 23:
In addition to the R2-score also consider another metric, which measures the relative distance between the last motif and earlier ones. This is done by changing step from above into the follow- ing:
36

---

Published as a conference paper at ICLR 2026
4'. Compute the relative distance between x" defined in step 2 and each x() formed in step 3:
|x
x
(
[|] 10-
where 10 is there prevent zero denominator.
In Figure 23, we see that as the patch size increases, distances generally become larger, indicating more challenges identifying the periodicity in the context.
37

---

Published as a conference paper at ICLR 2026
D DETAILS OF THE GEOMETRIC BIAS (FROM SECTION 3)
In this section, we provide more details on the geometric bias, which was discussed in Section 3. The discussion is made up of three parts: the angles between the embedded vectors, which lead to a locality preference (Appendix D.I); the distance between the embedded vectors, which gives rise to scale-dependent trealment (Appendix D.2); and the norm the embedded vectors, yielding the offset-awart embedding (Appendix D.3). This appendix is broken into three parts, corresponding to each the three biases, respectively.
D.1 DETAILS OF THE ANGULAR BIAS
D.1.1 DESIGN OF EXPERIMENTS IN FIGURE
Recall Figure 5, where we presented an experiment where we collected the self-attention scores of both the Chronos and Chronos-Bolt models, which revealed that the self-attention scores of the Chronos model have more of bimodal distribution, while those of the Chronos-Bol Bolt model followed a more unimodal distribution. To collect this set of self-attention scores, we sample 1000 random samples from Chronos and Chronos-Bole two models to collect all self-attention scores. Due to the space limitation, however, we were only able to show the histogram in the log-log scale, which hid some useful information.
in-domain evaluation corpus, and pass them into the
DAA
We reproduce the histograms in Figure 5 on four different scales: linear, semilog-x x, semilog-y, y, and log-log, respectively. We can clearly see the bimodal distribution of Chronos' self- attention scores in the semilog-y plot, while the semilog-x plot highlights the existence of small self-attention scores in Chronos, compared to Chronos- Booot's mediocre ones.
Figure 24:
In Figure 24, we reproduce this histogram on all four common scales: the linear scale, the semilog-x scale, the semilog-y y scale, and the log- log scale, respectively. From these plots, we can see at least two clean messages:
1. From the semilog-y plot, i.e., the third panel. we see clear bimodal distribution in Chronos' self-attention scores, which does not appear in Chronos-Boll self-attention scores. Note that the modes in Chronos' bimodal distribution do not have the same weight that is essentially why the two modes are hard to visualize when not using logarithmic y-scale. This is not surprising, however, because if single self-attention score large, then it implies that the remaining I, singular values are all small, the weight of the large-score mode must be much lower than the weight the small-score mode.
From the semilog-x plot, i.e., the second panel, we see that the small attention scores in Chronos much smaller than the attention scores in Chronos-Bole For Chronos-Bole most self-attention scores are small, but not tiny. This corroborates our intuition that Chronos-Bolt performs more context mixing by putting relatively small and uniformly distributed attention score in each element across the sequence.
D.1.2 THREE EXAMPLES TO ILLUSTRATE "PARROTING"
In the main article, and in previous work, e.g., Zhang & Gilpin (2025a;b), the idea of "parroting" arises. This basically says that Chronos, contrast to Chronos-Boi Bolt and other models that use continuous embedding, tends compute the output by "matching" the motif immediately prior to the forecasting start point to a motif in the earlier context, and then predicting by copying what comes immediately after. We identified that this "parroting" strategy (described in detail in Zhang &
38

---

Published as a conference paper at ICLR 2026
Gilpin (2025a;b)) is mainly due to the angular bias of the model. Here, we provide three examples to clearly illustrate the idea and the mechanism of "parroting"" and we suggest when it may be "good" or "bad."
Example Parroting helps Chronos to pinpoint a periodic context. The example in Figure 25 features sinusoidal context wave. For this context, is very easy for the Chronos model parrot the context. In the top panels, we show the distribution all cross-attention scores when Chronos and Chronos-Bolt (patch size are put very heavily and locally near the position corresponding to the forecast position in every period, whereas Chronos- Bolt puts the cross-attention scores more evenly across the entire context. As a result, one cannot eyeball difference between Chronos' forecast and the ground truth, but Chronos Bolt does achieve such pinpoint prediction.
make the first forecast. In Chronos, self-attention scores
Chrooce
Chronce Bolt
carlan
Cariann
Figure 25: We show example where Chronos' parroting strategy makes it perform better than Chronos-Bolt on a perfectly periodic signal. Above the forecast plots, we show the density of all cross-attention scores in the decoder of the models. We see that Chronos' cross-attention scores are highly dense where the next forecast corresponds the earlier periods, while Chronos-Bolt's cross-attention scores are more evenly distributed.
Example II: Parroting makes Chronos lazy and prevents advanced "reasoning." In the next example, we still base our context on the same sinusoidal wave, but we scale it into bidirectional envelope. This envelope is maximized in the middle and decays towards the two ends. The finesse of this design is that a model naively performs parroting, then easy to match the last motif to motif at the beginning, and predict the increasing context after it. By looking at the cross-altention scores, we note that this exactly what Chronos does, and consequently, the signal it predicts has an increasing trend, while should be decreasing, Chronos-Bolt, the other hand, makes relatively accurate prediction by attending mostly to the later context than the earlier ones to apply more advanced "reasoning."
39

---

Published as a conference paper at ICLR 2026
Contect
Capast
Figure 26: We show an example where Chronos' parroting strategy harms performance. From that Chronos forecasts the future by parroting the earlier match
the cross-attention scores, motif, but unfortunately, the carlier motif exhibits an increasing trend, while the true trend in the forecast should clearly be decreasing. Chronos-Bole performs much better by not parroting and mainly focusing on the recent history.
Example III: Chronos can still "reason" if there is nothing to parrot. Looking at the second example, may wonder: is it that Chronos cannot perform advanced "reasoning" that it gets the trend wrong? To test this, we redesign our envelope: instead of having bidirectional envelope, unidirectional one that is monotonically decreasing. The major distinction here is that do not have an increasing period at the beginning of the context, which therefore prevents Chronos from parroting. In Figure 27, see that when Chronos has nothing parrot, it attends to the recent history and gets the overall trend right; in this case, by simply "removing" the first decreasing part from our context, we can make Chronos perform as well as Chronos- Bolt again.
Chronon
Pintat
922
122
We show an example where Chronos and Chronos- Bolt perform similarly. In this ex- ample, the scale of context is monotonically decreasing, making there little to parrot. Both models show some capabilities for "reasoning" about the decreasing scale of the signal.
Figure 27:
D.2 DETAILS THE DISTANCE BIAS
D.2.1 DESIGN OF EXPERIMENTS IN FIGURF
Recall Figure where we showed an experiment based on real-world datasets that Chronos has better capability of learning the fine-scale signals than Chronos-Bolt. We achieved this by augmenting Chronos' in-domain evaluation corpus, which contains 15 popular real-world univariate time series benchmark datasets. Given input context e R and target y E RT, we split x = (x) evenly into two parts x, and X2. Then, we fix a scale ratio a 2 1. we consider two regimes:
40

---

Published as a conference paper at ICLR 2026
1. Predicting the large motif: scale X1 to x1/a and form the multi-scale context Xtarge (x1 12,/ Let y E IR" be model's output, compute the error by calculating the weighted quantile loss (WQL) using the ground truth y and forecast y.
Predicting the small motif: we scale X2 to X2/a and form the multi-scale context as X (x1,X2/02 X2/o). Let ý Th be a model's output, we compute the error by calculating the WQL. using the ground truth y and forecast aý. Note that we need to scale the model's output by a to renormalize.
To compute the overall loss, we follow Chronos' evaluation strategy (Ansari et al., 2024a) by taking the geometric mean of the WQLs across all 15 datasets measured by comparing model to a base- line, which is taken to be the model's 'ss performance when = 1. i.e. no augmentation is done. Note also that when a predicting the large motif collapses to predicting the small motif.
D.2.2 How DOES A TRANSFORMER PROCESS A MULTISCALE SIGNAL?
Norn of Embedded Vectors
Numn Errbeddeed Vectes
www.00
Chronos-Bok.
Chrenos
Encoded
www.mm
hw00
wwwww
WWW.
w00
WANNN
wWW
www
WWAA MWM
ww M00
M
MMNNN
MNNNNI
ANHANA
w
MAN
Amma
We show how each of Chronos and Chronos- Bolt (patch size k 1) processes a multi- scale context, respectively. In the top panels, we show our input multi-scale context. The green curves on the second row of panels show the norm of the embedded vectors in each model, where we see that the fine-scale and the large-scale motifs both become large-scale the embedded con- text, whereas the difference between the scales stays distinguishable, if not more dramatic, in Chronos- Soonos-Bool'ss embedding. Then, we show the encoded context both models, projected onto the top-10 principal components. We see that Chronos' fine-scale motifs got preserved nicely, while the fine scales are contaminated by the large scales in Chronos-Bole
Figure 28:
To illustrate the distance bias of Chronos versus Chronos-Bolt better, we perform detailed analysis given a multiscale context shown in Figure 28. First, given the context x E R',, we let the embedding be ((x) (øI..... Φ1) E d L. where d is the hidden dimension of the Transformer. In the panels on the second row of Figure 28, we show the norm ||0 of these embedded vectors, which indicates clear multiscale structure in Chronos-Boo Bolt's embedding, but not as much in Chronos'
41

---

Published as a conference paper at ICLR 2026
embedding. This is the root cause of the distance bias. To show that the embedding really matters, further take the encoded context z E Rd which is the sequence out of the encoder Transformer, and we plot in its leading 10 principal components along the temporal direction. From Figure 28, we see that the fine-scale oscillation preserved very well by Chronos' encoder, but it is severely contaminated by the large-scale oscillation in Chronos-Bolt's encoder.
Embodied
Narm Embedded Vecton
Chromos
AVM Encoded Content
Ercoded Context
www
We show how each of Chronos and Chronos- Bolt (patch size k 1) processes a multi- offset context, respectively. In the top panels, we show our input multi-offset context. The green curves the second row of panels show the norm of the embedded vectors in each model, where we see that the high-offset and the low-offset motifs both have large norms in the embedded context, whereas in Chronos-Bolt's embedding, the low-offset motifs have much smaller high-offset ones. Then, we show the encoded context in both models, projected onto the top-10 principal components. We see that Chronos' low-offset motifs got preserved nicely, while they are contaminated by the high-offset motifs in Chronos-Bol
Figure 29;
than the
D.3 DETAILS THE NORM BIAS
D.3.1 DESIGN OF EXPERIMENTS IN FIGURE
Recall Figure where showed experiment based the real-world datasets that Chronos has better capability of learning the zero-offset signals than Chronos-Bolt. We achieved this by augmenting Chronos" in-domain evaluation corpus, which contains 15 popular real-world univariate time series benchmark datasets. Given an input context x E g& and a target E R we split (x1, X2, X3) evenly into three parts X1, X2, and X3. Then, fix offset 1, we consider two regimes:
1. Predicting the high-offset motif: we drop X2 to X2- - 3, lift X3 to Xy + 8, and form the multi-scale context as Nnigh (x1xx; 3, x, B). Let ý E R² be model's output, compute the error by calculating the WQL using the ground truth y and forecast ý - B. Note that we need to drop the model's output back by 3 to renormalize.
42

---

Published as a conference paper at ICLR 2026
2. Predicting the low-offset motif: drop X1 X1 - lift x2 X2 + 3. and form the multi-scale context as Xlow = (x1 - 3, X2 + 3, Xy). Let y E R' be model's s output, we compute the error by calculating the WQL using the ground truth y and forecast y.
To compute the overall loss, we follow Chronos' evaluation strategy (Ansari et al., 2024a) by taking the geometric mean of the WQLs across all 15 datasets measured by comparing model to a base- line, which is taken to be the model's performance when 3 = 0, i.e., augmentation is done. Note also that when 3 = O, predicting the large motif collapses to predicting the small motif.
D.3.2 How DOES A TRANSFORMER PROCESS MULTI-OFFSET SIGNAL?
To better illustrate the norm bias of Chronos versus Chronos-Bole we perform a detailed analysis given a multiscale context shown in Figure 29, First, given the context x E R', we let the embed- ding be (x) (di,..., Φr.) E RdxL where d the hidden dimension of the Transformer. In the panels on the second row of Figure 29, we show the norm |1 of these embedded vectors, where we see that Chronos' zero-offset motifs are embedded onto large vectors but Chronos-Boll ' embedded onto small vectors. This is the root cause of the norm bias. To show that the embed- ding really matters, we further take the encoded context z E R which is the sequence out of the encoder Transformer, and plot in its leading 10 principal components along the temporal direction. From Figure 29, we see that the zero-offset oscillation is preserved very well by Chronos' encoder, but it is severely contaminated by the large-offset oscillation in Chronos-Bolt's encoder.
D.4 How DOES THE VOCABULARY SIZE RELATE TO GEOMETRIC BIAS?
While our discussion of the geometric bias is mainly comparison between the quantization- based embedding and the continuous embedding, if one quantizes the real line, the number of bins certainly also plays role. Intuitively, if the number of bins is larger, then it is harder for the model to learn the geometry of the input domain in the hidden space, and consequently the geometry bias is stronger.
Distribution of Attention Score
Error vs. Vocabulary Size of Chronos
10a
vocab size 512
vocab size 512
vocab size 4096
10-1
vocab size 1024
re relotive error
vocab size 2048
10a
vocab size 4096
10
-4
10-?
10:3
10
10-1
10a
10
10 1
10a
attention score
scale ratio
Figure 30: On the left, show the distribution of the attention scores, using Chronos model with a vocabulary size of 512 and one with a vocabulary size 4096. On the right, we show the relative error of predicting the fine-scale motif of multiscale structure, outlined in Figure 6.
This intuition corroborated by the two experiments shown in Figure 30. The histogram shows that larger vocabulary size leads to more large attention scores, introducing a heavier bias for attention to local content. The error plot on the right shows that a larger vocabulary size generally magnifies the scale of fine-scale motif in the hidden space, enabling better learning.
43

---

Published as a conference paper at ICLR 2026
E DETAILS OF THE REGRESSION-T-TTH-MEER BIAS (FROM SECTION 4)
In this section, we provide more details on the gogressiontt-the-mmee bias. First, we use two exam- ples illustrate the loss landscape induced by the three standard loss functions: T, -baee, 1.2-based, and cross-entropy (Appendix E. Then, revisit the examples shown in Figure 8, and we show how Chronos learns the underlying probability distribution, instead of regressing to the mean (Ap- pendix E.2). Finally, we give the details about how we build "bridge" from the deterministic to the random case in Figure 8 (Appendix E.3).
E. A LOSS-LANDSCAPE ILLUSTRATION OF THRRE KINDS OF LOSSES
To illustrate the concept of regression-tes wowpsssoo--tttte--mmmaaaeeeee mean/meaan/m,,,,,, consider the where have ground-truth probability distribution on a simple measurable set f = {0, 1/2, 1} equipped with the
discrete o-algebra. Assume first that the ground-truth probability distribution the one with
P Puu ( [0}) = Puum [1/2]] = Puuth ({1) = 3
In each triangle in Figure 31, we show the barycentric coordinate corresponding to a model's output probability distribution on Ω. The three different losses are shown using colors.
P({0)) = (((M12))= 1/3
Expected MSE (néx G/13
Expected MAE (mix 6/1)
8:88
Expacted Cross- Entropy (mix 3/1)
0.25
0.44
3.5
0.30
3.0
0.40
255
0.36
15
0.14
Figure 31: Assume we have ground-truth probability distribution that equals 1/3 on 0, 1/2, and 1, respectively, and suppose our model generates a probability distribution on the set {0, 1/2, 1). We show the loss landscape induced by the ground-truth probability and the model's forecast. The colors indicate the loss, and the barycentric coordinate in each triangle represents the model's predicted probability on 0, 1/2, and 1, respectively. We use red colors indicate the global minima in the loss landscape induced by each norm.
Note that the MSE and MAE are independent of the probability distribution: they only require final prediction. In that sense, one can imagine that the entire triangle collapses to the line segment from 0 to 1 via orthogonal projection. That is essentially why our final forecast regresses to 1/2, which is both the mean and median of TP muth in this example. For cross-entropy, the optimal the barycenter of the triangle, corresponding to exactly the ground-truth distribution.
Next, we consider slightly different ground-truth probability distribution:
F Ftruth ({0}) = Frothh ({1)) 2 Fiothh ({1/2)) = 0.
From Figure 32, we see that MSE still regresses to the mean, which is the same as in the previous example; the cross-entropy loss eliminates 1/2 from its support, injecting more flexibility. The MAE loss, which tailors the median, gives us a completely flat loss landscape, where every distribution is optimum.
44

---

Published as a conference paper at ICLR 2026
P({0)) - P((1)) 1/2
Expected MBE
Expected MAE
Expected Cross Entropy
0.4
0.59
0.54
0.50
2.5
0.30
1.
ALLOPIIAAL
0.46
0.25
Figure 32: The plots show the same things as in Figure 31, except that set the ground-truth probability distribution to be a bi-modal one with probability of 1/2 on 0 and 1, respectively.
The only question that remains is: if all points optimal against MAE loss, then why is it that empirically, the model usually prefers one closer to the mean (see Figure 8y? One answer is: although every number between [0, 1] is median of this probability distribution, the medians have different stability. That is, if we compute the gradient of the MAE with respect to the probability p assuming that
P Fouth ({0}) p. Fuuth ((1)) =1
Puuth ({1/2)) 0,
then this gradient is large near the two endpoints 0 and 1 and small in the middle (see Figure 33). Why does that matter? In practice, even if the probability distribution is 1/2 on both 0 and 1, when sample the data, they will sometimes "lean" towards 0 and sometimes towards 1. Given that, "the median of the medians" gives us a more robust prediction.
Stability
P((0)) P((3))
Expected
(didol
1.0
1.0
6.6
0.8
0.6
0.6
0.4
0.
Figure 33: On the left, we show the expected value of the model's forecast in the barycen- tric coordinate (Schumaker & Yu, 2022). On the right, we show the stability of the MAE loss given the ground-truth in Figure 32, where the colors indicate the magnitude of the gradient (d/d)) (p), IP montee)) at 1/2, where Pinut(p) is at O and 1 - at
E.2 How CONFIDENT CHRONOS AT BIFURCATION?
In Figure 8, we saw that when the context consists only 0's and I's, Chronos, in forecast, only generates these two numbers. Since Chronos performs regression via classification, we can under- stand this better by looking into the logits of Chronos' forecast. Here, we look into these logits when Chronos forecasts the two contexts
deterministic and one random - in Figure 8. Recall that the Chronos generates the output via binning: preassumes a fine grid B = [B1, Bz.... By on the real-line and computes probability distribution over B3 Here, when Chronos makes forecast, we keep track its generated probability associated with each Bj. In particular, there are two bins By and B₁ whose values correspond to (the nearest grid point from) 0 and 1, respectively.
45

---

Published as a conference paper at ICLR 2026
Forecasting the Rist Tiken by Chronos (Uniform Context)
Forecasting Every Token Chronas (Uniform Context)
1.0
10-
0.6
0.6
0.2
0.2
0.0
0.03
2100
2200
2300
Index
forecant
Forecastiing Firtt by INon-uriform Contest)
Forecasting Every by Chronas (Nomsunffom Contest)
0.8
0.5
0.6
secsaa
0.4
0.44
0.2
0.0
0..0..0
2000
2050
2000
2400
2500
60
Figure 34: We compute the logits when we apply Chronos model to predict the two synthetic signals shown in Figure The top two panels show the results corresponding to the deterministic periodic walk between 0 and 1, while the bottom two panels show the results corresponding to random walk. On the left, we show the logits associated with the first prediction. Each red dot corresponds to the logit associated with one bin, and the logits associated with the bin of "0" and the bin of "1" are highlighted with a blue and green cross-mark, respectively. We see that the probabilities are dense on {0, 1} and tiny elsewhere. On the right, show the logits corresponding to the bins of "0" and "1", respectively, as we make more forecast steps. As a reference, we also show the logits corresponding to the bin of "0.01", which are nearly zero.
From Figure 34, we see that Chronos is almost always very certain in that its forecast is either or 1. That is, the only two non-trivial probabilities that it produces are at 0 and - all remaining or Incidentally,
probabilities are almost zero, even for the bins that are immediately next we should note that the fact that Chronos is capable of distinguishing O and 1 from their nearest bins is also due to the locality and distance bias (see section 3). The only major difference between forecasting the deterministic and the random contexts is that Chronos is much more confident in which one, between 0 and 1, to pick when the context is deterministic, whereas given a random context, Chronos generates close probability on 0 and on 1. This is not surprising, it aligns with our discussion that the cross-entropy loss tailors a model to learn the "ground-truthi probability distribution.
E.3 How TO INJECT UNCERTAINTY IN FIGURE 8?
In Figure 8, we show an interesting quantification, where we start from totally deterministic case, where we walk periodically between the two branches and 1, and end up with the totally deter- ministic case, where for each time step. randomly land at 0 or with we show the details of how to gradually inject uncertainty and make a smooth transition from the deterministic case the random case.
probability. Here,
46

---

Published as a conference paper at ICLR 2026
To this end, first define the periodic context. Here, alternately take 5 steps of each branch. That is, our context is given by
2y 13))po
Given probability of diffusion, simply add random "XOR" perturbation this context with probability p. That is, the new context is given by
i = It + Bernoulli(1 - p):
where the Bernoulli distribution at every i.i.d. It is clear that P(²;) is p on one branch of 0 and 1 and 1 - on the other, and that when we increase to 1/2, our context becomes a random walk.
To understand how much the model regresses 1/2, the mean or the median of the entire sequence, we define the "regression score" by
reg. score(y) = min([1 - yl. lvl).
This is shown in Figure 8, where our experiments are repeated 100 times and the error bars show the 30% to 70% quantiles of all regression scores.
E.4 THE EXPERIMENT IN FIGURE 8 ROBUST TO NOISES.
In Figure 8, we show an experiment with 0-1 walk to show that a model trained with a cross- entropy loss does not always regress the mean, unlike the other models trained with an - or b2-ba loss. The only source of stochasticity, however, comes from the probability of landing on 1. In this section, show that the results hold seamlessly for noisy data. That is, hold the context the same but inject random Gaussian noises to each timestamp.
Noise = 5%
Noise 20%
0.4
regression score
0.4
Chrones-Bol
Chrones Bolt
0.3
TimesFM
0.3
TimesFM
Chronos
Chronos
0.2
0.2
Moirai
Moiral
0.0
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Diffusion Probability
Diffusion Probability
Figure 35: We inject random Gaussian noises the experiments in Figure 8. We see that even with noises, a regression- based model still regresses to the mean, whereas the classification-base model does not.
From Figure 35, we note that with a substantial amount of noises, Chronos still does not regress to the mean, unlike other regression models, when the underlying probability distribution becomes bimodal.
47

---

Published as a conference paper at ICLR 2026
F THE OUTLIER HANDLING
In this appendix, we provide a detailed analysis of a particular case of outlier handling. We see that the context sinusoidal wave with many outliers injected. For this context, Chronos' embedding largely dampens the magnitudes of the outliers in the hidden space, as projected by the geometric bias. Consequently, the clean sinusoidal segment is preserved better during encoding. Also note, however, that Chronos tends to reproduce some of the outliers as it learns a probability distribution that hits both the clean sinusoidal structure and the outlier space; on contrary, Chronos- Bolt does not parrot particular outlier.
CO
JALIAA
KUMM
Figure 36: We show more detailed analysis of the outlier handling. All projections are onto the first 5 principal components of the matrix in the hidden space.
48

---

Published as a conference paper at ICLR 2026
G THE SIMPLICITY BIAS
The term "simplicity bias" refers to the tendeney of neural networks to favor simpler hypotheses, even when more complex, and potentially more robust, solutions are equally compatible with the data. This bias has roots in classical learning theory, which uses concepts like VC dimension and Rademacher complexity to explain how restricting effective hypothesis complexity enables gener- alization (Wolpert & Macready, 2002). Empirically and theoretically, the simplicity bias has been observed in both shallow architectures, c.g., two-layer networks tending to depend on only low- dimensional projection of the input (Morwani et al., 2023), and deep transformers, which typically learn low-order interactions first before higher-order ones (Belrose al., 2024; Rende al., 2024).
Some work (Goldblum et al., 2024) argues that deep models inherently favor lower-order polyno- mial fits when possible (Wilson, 2025), while other studies explore how parameters such as effective rank further constrain learned representations toward simpler, low-rank subspaces (Huh et al., 2021). Together, these findings underscore how a simplicity bias, though still not fully explained, provides a key inductive ad advantage in TSFMs, helping them generalize while highlighting the tradc-offs be- tween robustness and expressivity. In this section, we provide some discussion of this simplicity bias.
The first thing to note is that, unlike the other biases discussed in this paper (where there is typically trade-off - tuning design decision "knob" improves performance one end but sacrifices it on the other, in ways that depend on the specific model), the simplicity bias is different. First, is more of "design goal" than "bias" (at least in the sense have used the word so far). Second, and relatedly, recent work has revealed that larger neural networks often exhibit this simplicity bias, which favors simpler solutions despite their expressive power. While the precise mechanism behind this phenomenon remains an active area of research, it is an important consideration for TSFMs. In particular, from expressiveness perspective, even small models but larger models seem to introduce an additional simplicity bias that helps prevent overfitting. To maintain a simplicity bias via larger model, however, we also pay significant costs in compute and memory. A promising direction, perhaps informed by our insights in this paper, is to retain this bias while reducing overhead through post-training compression techniques.
already highly capable,
G. ILLUSTRATION OF THE SIMPLICITY BIAS
Simplicity Biss Chronos
z-Rank Chronos
Chroaos
Brokgbbbtty
10
30y
clock
10-51
-0-
-
1021
1u-77
50 complexity
768-
10-2
08
05
10
complexity
50
baty
mini
size
small
Chronns
base
large
time
500
Figure 37: The simplicity bias concerns how a model prefers simple solutions over complicated ones, enhancing the generalization capability (see the right column). In the left column, we present two targeted experiments: first, we construct time series using a different number of basis elements, which we call the complexity, and we compute the model's probability of forecasting the correct future; next, we fix a context and evaluate chances that a model computes a higher probability for a simpler solution than more complex one, which we call the "win rate." The larger model consistently prefers simpler solutions more than the smaller ones. The middle panel shows the com- pressibility of the attention matrices in pretrained Chronos models, indicating that larger attention matrices mainly contribute to over parameterization rather than expressiveness (see also Yu et al. (2025b)).
In the experiment shown in Figure 37, we design context XI.z with two continuations: a simple (x) one y iA(L-Ty and complex one y (L(TT)) We compare the rate that model prefers the simple
(c)
49

---

Published as a conference paper at ICLR 2026
solution over the complex onc: Pmeddl (LL((L+T) X1.L) > model (V::L++T) X..)). The finesse
(e)
(yle)
of this experiment design is that when the A-complexity between the two continuations is zero, then any model prefers them equally; also, when the A-complexity gets large, eventually any model simply always prefers the simple solution. Walking from one extreme case the other, in Figure 37, we see that Chronos (large) consistently prefers simple solutions more than the smaller models.
The simplicity bias refers to the tendency of TSFMs to favor simpler solutions, which helps explain their robustness despite their large flexibility. This bias is amplified by model size, as scaling up introduces stronger preferences toward simpler hypothe- ses, It is also closely tied compressibility, suggesting that models can retain simplicity bias while still being pruned or compressed for efficient inference.
Summary of the Simplicity Bias:
We now provide details about the two experiments shown in the first column of Figure 37.
G.2 PREDICTION PROBABILITY AS A FUNCTION OF DATA COMPLEXITY
First, we discuss the details of the first panel in the first column of Figure 37.
Data generation. Each synthetic series is split into a context of length I, and future of length T, = (V1..... 3L, 9L++1.... VL+T). The context is produced by simple parametric mechanism that the model can plausibly capture with L samples. We use two base families: a linear trend ye at + b; and a single sinusoid V = Ao 22 + Φu)- The future is obtained by continuing the same base mechanism and then optionally adding M sinusoidal components,
M
UL+S =
base UL+T
+
Σ m==
Am sin(2 + Φm),
t t= 1=,,..,,7 T
with a short ramp-in at the beginning of the horizon to avoid discontinuities. To control the intrinsic complexity of a sample in way that is measurable and reproducible, we use a bit- budget proxy K K(y) that counts the number of bits required to specify the added components. For each new component, we allocate ky bits to choose frequency from coarse-to-fine grid, ba bits to quantize amplitude in [Amin; Amax), and by bits to quantize phase in [0. 2T); hence ((y) Kusse + EM== (ky + bA + b²). Small nonzero complexities are achievable by allowing very coarse choices (e.g., ky = 0, 2) and low-bit quantization for (A, Φ); larger complexities are obtained by increasing these bit budgets and/or M. After generation, we standardize the future with a shared scaling computed from the simple continuation (mean u1s, std Ø.)) so that the magnitude of log-likellng log-likelihoods is not dominated by trivial scale differences: VL++ = (yL++ - ps)/
Result computation. For model M, we score each future under teacher forcing, using the Chronos tokenizer to encode the context as input IDs and the future as labels. The model's total ligelikelihood of the future given the context is
log VELL) = Σ Loggnm(uL- / VuL BL+H-+T-1).
and we normalize by T to obtain an average log-probability per token s(y) z14 log To summarize how likelihood changes with intrinsic complexity, we bin the samples by K(y) into B evenly populated bins and compute, for each bin b, the sample mean &, and 95% confidence in- terval using the normal approximation 5n + 1.96 0t/ /r, where og is the sample standard deviation and ny the number of sequences in the bin. The baseline plot overlays the bin means for all mod- els as a function of the corresponding bin centers in bits. For downstream plotting, we also save a compact 3 x B array per model containing the lower bound, mean, and upper bound in each bin, along with the vector of bin centers. Interpreting the figure is straightforward: curves that lie higher indicate that the model assigns more probability mass on average sequences at that complexity level; a gentle downward slope with increasing bits is expected, as additional structured variation makes the future harder to predict.
In practice, we use helper that assigns indices by quantiles of K; report the mean of K within each hin as its horizontal location.
50

---

Published as a conference paper at ICLR 2026
G.3 WIN RATE OF THE SIMPLE SOLUTION
Next, we discuss the details of the second panel in the first column of Figure 37.
Data generation. This Occam-pairs experiment asks whether model prefers simpler continuation over a more complex one, when both are plausible given the same past. We first draw a context YI.L from the same base families as above (linear or single sinusoid) and form simple future by continuing the base mechanism into the next I' steps. We then create a complex future by adding controlled extra structure to the simple one:
M Σ
( yi
(9) Yt
Sin((22 &m))
1= I=.....;
with the same ramp-in convention and the same shared standardization with respect the simple future. The key scalar we vary across trials is the complexity gap AK, defined as the additional bit budget needed to describe the extra components. As above, each component pays ky bb + by bits; thus AK = >m= (ky + 64 + by) (optionally with a combinatorial term if we explicitly encode subset of large frequeecc grid). To ensure well-behaved behavior at the endpoints, we include two calibrations. At K = O we set M = O and, if observational noise is added, we add the same noise realization to both futures. This makes the two futures exchangeable, so no model can reliably prefer one over the other. At very large AK we gradually turn off preview leakage from the context and increase the amplitude gain of the extra components over window in AK, so that the complex future departs cleanly and strongly from the pattern established in the context.
To prevent the win-rate curves from trivially saturating near (as they would if the complex future were always obviously inconsistent with the context), we apply a lightweight scheduler that couples nuisance knobs to the target gap A K. As AK increases, we gradually reduce observation noise and ramp-in length, and we suppress any preview of the complex components in the context. At small and mid AK these choices keep the two futures similarly plausible from the same past, so wins are earned rather than guaranteed. We also regularize the construction with shared standardization (both futures are scaled by the simple future's mean and variance), matched energy of the added components across AK (so higher bit budgets translate to finer specification rather than unbounded amplitude), and an explicit tie tolerance E so indistinguishable pairs contribute 0.5 on average. Two anchors complete the calibration: at AK-
we set M = and share the same noise realiza- tion across the two futures; while at very large AK a soft right-end "anchor" smoothly turns off preview and ramp and slightly boosts gain so that all models approach without a discontinuous jump. Together, the scheduler and regularization make the mid-range of AK informative, avoid- ing the confusion that would arise if readers compared to the baseline plot (where single-sequence likelihoods often decline quickly with complexity) and wondered why pairwise win rates would otherwise sit near 1 for almost all AK.
Result computation. For each triple (VL.L, VI.T forced log-likeliing lokellosisoo.s
(8) (C)
SIT.Yi7 YI.T and model M we compute two teacher-
Es log logpM(y(5) yI.z),
fc = log (0) yy:..)
and form the difference AL- = -c. A trial counts as a win for simple if A is positive by more than small tie tolerance ;; | < we record a half-win so that truly indistinguishable pairs contribute 0.5 on average. Repeating this for many independently generated pairs at the same 1 E, w, 1]. We summarize uncertainty with a Wilson score interval at 95% confidence, which is appropriate for a proportion and behaves well even when W ((A)) near 0 1. Plotting W(A against the bit gap produces the Oceam curve for each model. Two anchors validate the construction: at AK = 0 all models sit near 0.5 by symmetry; at sufficiently large AK all models approach 1 because the complex future contains clear superfluous structure absent from the past. The region between these anchors is informative: models whose curves rise earlier and remain higher over a broad mid-range of AK are better at recognizing and penalizing unnecessary structure, thus exhibiting stronger empirical simplicity bias in this controlled setting.
AK yields an empirical win rate ((A
51

---

Published as a conference paper at ICLR 2026
H THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMss are used for polishing the writing and word choices of a few sections in the main text. They not used in the conceptualization and implementation of research.
52


Reviews:
[{"review_id": "BlVcgKzenN", "reviewer": "Reviewer_SCK8", "summary": "This paper presents a systematic investigation into the implicit biases of modern Time Series Foundation Models (TSFMs). Rather than proposing a new model, the authors aim to understand how common design choices (\"knobs\") influence model behavior. They identify three primary classes of biases: 1) Temporal Bias, induced by patch size, which affects how models handle different frequencies and periodicities; 2) Geometric Bias, arising from the choice between continuous and discrete (quantization-based) embeddings, which impacts how the model perceives locality, scale, and offsets; and 3) Regression-to-the-Mean Bias, driven by the training loss function (e.g., L1/L2 vs. Cross-Entropy), which determines a model's behavior under uncertainty. The study uses a mix of theory and controlled empirical evaluations, primarily comparing Chronos and Chronos-Bolt, to demonstrate how these biases manifest and interact, ultimately shaping a model's suitability for different types of time series data (e.g., standard benchmarks vs. chaotic systems).", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a critical need in the TSFM literature. By shifting the focus from \"what is SOTA\" to \"why models behave the way they do,\" it provides lasting insights that will remain relevant even as new models emerge. This is the kind of scientific inquiry that fosters deeper understanding.\n\nThe combination of theory, controlled synthetic experiments, and analysis on real data is a major strength. The use of Chronos vs. Chronos-Bolt as a primary case study is an elegant experimental design choice that isolates the effects of specific design decisions.\n\nThe paper is exceptionally clear. The three-part framework of Temporal, Geometric, and Regression-to-the-mean biases is intuitive, well-supported, and provides a powerful lens through which to view TSFM design. The authors do an excellent job of explaining not just that a bias exists, but how the specific design choice leads to it.\n\nThe paper provides practical takeaways. For example, it explains why a model like Chronos (with quantization and cross-entropy loss) might be better for chaotic systems (due to less regression-to-the-mean and a different geometric bias), while a model like Chronos-Bolt might be more robust for noisy, trend-based series. This helps bridge the gap between theory and practice.", "weaknesses": "While the focus on Chronos/Chronos-Bolt is a strength for control, it is also a potential weakness for the generality of the conclusions. The paper does include other models like TimesFM and Moirai in some experiments (which is great!), but the core narrative and many of the detailed analyses are tightly coupled to the Chronos family. It would strengthen the paper to either include more direct evidence from a wider variety of architectures or to more explicitly frame the conclusions in terms of the design choices themselves (e.g., \"models using continuous embeddings and L1 loss...\") rather than implying they hold for all TSFMs.\n\nThe case study on outlier handling in Section 5 is a good first step toward understanding how biases interact. However, this aspect feels somewhat underexplored compared to the detailed analysis of each individual bias. The real world is messy, and models are always operating under a combination of these effects. A slightly deeper discussion or another case study (e.g., handling non-stationarity) could have made the \"mixture of biases\" a more central contribution.\n\nThe inclusion of Theorem 1 is a strength. However, the theorem relies on standard but strong assumptions (e.g., random Gaussian weights) that are not met in a fully trained network. A brief discussion on the expected qualitative persistence of these results in trained models would be helpful to bridge the gap for practitioners. For instance, do we expect the orthogonality of embeddings for different frequencies to be as clean, or just a general tendency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R7O0gyFXbY", "reviewer": "Reviewer_cdYf", "summary": "The paper discusses how different architectural choices in modern time series foundation models lead to learning biases in the resulting models. In particular, it focuses on biases in modeling the dynamics of input time series and in the structure of the predictions. Each observation is supported by convincing empirical results and well-designed experiments. I found the paper interesting and believe it is a relevant contribution to our understanding of the properties of current time series forecasting architectures. Although there are some presentation issues, I believe the paper is a solid contribution to the conference.", "questions": "Please comment on the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper studies interesting and important phenomena in time series forecasting models.  \n* The findings are particularly relevant in the context of foundation models.  \n* I appreciate the focus on clarifying the impact of design choices that are often overlooked.  \n* The empirical analysis is interesting and well-designed.", "weaknesses": "### Main weaknesses\n\nThe writing and presentation could be improved, and several claims would benefit from additional discussion and supporting evidence.\n\n- The introduction could do a better job summarizing the main takeaways of the paper and explaining why the discussed phenomena are particularly important in the context of foundation models (i.e., when learning a transferable model). Some sentences are difficult to contextualize without having read the full paper — for example:  “that time is continuous and this continuity should be maintained, and that regression algorithms should regress to the mean,” or “while a quantization-based embedding introduces a continuous-to-discrete ‘unrounding’ bias, and then relies on training to (imperfectly) recover continuous information in the hidden space.”  Moreover, the example on Chronos vs. Chronos-Bolt performance on chaotic systems is not self-contained and not particularly informative without the cited paper. I would use the introduction to provide more context and clearer motivations for the study. To clarify, I find the motivations sound — I am only referring to the quality of the presentation.  \n- I found Theorem 1 quite difficult to parse, as the terminology and assumptions are packed into a few lines. I understand its purpose, but I recommend streamlining the presentation and expanding the discussion on assumptions and their implications. While the appendix provides detailed explanations (which I appreciate), the main text does not adequately contextualize the theorem or its significance. For instance, the assumptions on the weight and bias matrices seem unreasonable without further discussion.  \n- The discussion on periodicity bias — specifically the impact of architecture (encoder vs. decoder vs. encoder-decoder), REG token, and patch size — is rather limited in the main body. I understand the space constraints, but as it stands, some claims appear less convincing without looking at the appendices.  \n- In Section 3, please clarify the notation. If I understand correctly, *x* and *y* refer to arbitrary patches, and since *k = 1*, these are scalar. Please make this explicit in the text.  \n- “When a period is far from zero, its embedded vectors occupy a larger portion of the embedded context, which makes it easier for the model to learn. [...]”  This is unclear and should be elaborated on further. Please clarify and/or provide an example.  \n- Please include the related work section in the main paper (you can make it more compact). Since much of the analysis focuses on comparing Chronos and Chronos-Bolt, a more in-depth comparison of the two architectures should be provided — particularly regarding the tokenization mechanism used in Chronos. I understand the space limitations, but I would recommend moving some empirical results to the appendix (or making the introduction more concise) to make room for a clearer context.  \n- While some parts of the paper would benefit from more discussion and detail (preliminaries, related work, etc.), other parts could be streamlined (e.g., Sections 3 and 4).  \n\n### Additional comments\n\n- The paper does not mention plans to release the code, but doing so would be very helpful. I also encourage the authors to include a reproducibility statement, as suggested by the ICLR guidelines.  \n- Most empirical results are based on synthetic datasets with no noise (systematic noise, not outliers). How do the authors expect the presence of noise to affect their observations?  \n\nOverall, I liked the paper, and if the authors address the issues above and improve the presentation, I'd be happy to increase my score.\n\n### Minor comments\n\n* What do you mean by “first-order structure” (e.g., line 221)?  \n* Rather than “periodicity,” I would suggest using “seasonality,” which is the more common term in the time series literature.  \n* In Figure 4a, please clarify exactly what *x* and *y* represent.  \n* All models used in the study are pretrained, correct?  \n* It would be interesting to include experiments on more varied datasets, both synthetic and real.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n06g7RdDYH", "reviewer": "Reviewer_8DhK", "summary": "The paper presents a study of implicit biases that follow from the design choices of time-series foundation models. Five biases are categorized into two categories, namely temporal (2) and geometry (3) biases. The biases are subtle but are revealed through experiments and proofs (given in the appendix).", "questions": "On line 346, what seems like a very important insight is mentioned, but only implicitly. I read it as: Because of the findings in this paper, foundation models for time-series might not be possible because the different design decisions will affect the performance on different tasks. Therefore, TSFMs are task specific, which is not the case for language models as they all operate on the same type of representation (language) with similar characteristics, as words and sentences. Is this the conclusion, or am I misunderstanding something? If this is the case, then the argument should probably be made explicitly in the conclusion. \n\nDoes the summary of results in Figure 1 that is described in text from line 93 reflect the summaries given in the boxes after the description of the biases? For example, Patch Size $\\rightarrow$ Temporal bias. However, patch size is a subset of temporal bias, but it also contains architectural choices and unmasked [REG] tokens as well? Did I understand this correctly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* The topic and analysis are novel. \n* I expect this paper to be highly significant for time-series modelling relying on deep learning in general and not only for research in time-series foundation models.\n* The paper is extremely thorough.\n* The structure of the paper is nice.\n* I believe the conclusion to be stronger than what is stated in the paper (see question section).", "weaknesses": "While the writing generally is good, this paper has some issues that I would like to point out. \n* The different biases are not defined explicitly in text. For example, what is meant by temporal and geometric bases? Please specify.  What is the frequency bias and the periodicity bias? What are the angels, distances and norms biases? Explicitly state this, preferably right after the bold heading for each term.\n* For temporal bias, frequency, periodicity and seasonality are mentioned, but while frequency and periodicity have their own bias, seasonality is not mentioned again explicitly. What is the difference between seasonality and periodicity?\n* On line 149: “In general, these models show different inductive biases …” Which? Be explicit and give examples. \n* The term frequence seems to be overloaded and sometimes mean frequency as measured by Hz and at other times mean “Fourier modes”. When it means one thing and when it means another thing is not explicit. Thie reduces the readability. I would suggest to not use frequency to indicate “Fourier modes” and use a different term instead?\n* For geometry bias, paragraph starting on line 267, it is not clear what geometry bias means, as mentioned above, nor which biases are grouped under this term. This is stated for temporal biases.\n* The term “breaking the geometry” is used on line 280, but as it is not clear what is meant by geometry, the term is not easily understood. \n* Line 314: Please explain what is meant by more complex reasoning. \n* Not clear what is meant by “the geometry of the input domain”.\n* Section 5 Mixture of Biases is a bit short. It would improve with more thorough explanations. \n\nIn summary, this paper is not very easy to read. However, this should be fairly easy to correct given that more space is available for final submission (and in the rebuttal phase). \n\nGiven the depth of the analysis and the complexity of the topic, it would probably improve by having less restrictions on page numbers. Publishing the paper in a journal could be an alternative to a conference because of the share amount of work that is reported. I realize that few journals have the reputation of ICLR and that publication might take longer. \n\nThere are issues with the PDF. It is heavy. The problem is especially noteable on page 7. I assume this is because of Figures 5 and 7. Is it because the figures contain text? Is the text the cause of the problem? They seem like vector graphics and not high-resolution images when I zoom in. Still, the page loads terribly slowly. This issue should be rectified if accepted. \n\nImproving the text will not only improve the presentation score, but also the soundness, as it will be easier to evaluate it with better presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gMNBDeJKSV", "reviewer": "Reviewer_UNdi", "summary": "This paper did a investigation on implicit biases in Time Series Foundation Models (TSFMs). These implicit biases are introduced by several design knobs such as patch size, embedding type, and training loss. Instead of proposing a new model, this paper systematically analyzes how these design choices the model’s internal representations and generalization behavior. In this paper, it mainly identifies and discuss three main biases: temporal bias, geometric bias and regression to the mean bias. Combined theoretical analysis with empirical validation, it explains and shows how each biases cause issues and how to improve them.", "questions": "Most of questions are mentioned in weaknesses.\n\n1: Could you please give more evidence to support how large patches can lead to low-frequency alignment? especially any end-to-end training support?\n\n2: Could you include structural evaluation metrics to show your conclusions for model's structural correctness?\n\n3: Have you tested your conclusions or findings on real industrial datasets with missing values, multi-seasonality, and domain shifts? How will your findings help the improvements?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1: This paper shows a wealth of visualization, including different type of figures. These figures are very easy to understand.\n\n2: The motivation of this paper is very clear. It shifts the focus from performance chasing to understanding inductive biases, highly relevant for Time Series Foundation Models (TSFMs)'s  robustness and transferability.\n\n3: This paper did a deep analysis on each implicit bias, validating findings across several leading Time Series Foundation Models (TSFMs). It also shows practical insights on each bias's investigation.", "weaknesses": "1: For theorem 1, it seems that its theoretical scope is a little bit narrow and relies on simplified assumptions. In Theorem 1, it assumes a linearized ReLU MLP and random Gaussian projections to derive the property of patch embeddings. However, in modern Time Series Foundation Models (TSFMs), they have some attention and temporal positional encoding to dominate their feature engineering. We didn't see any assumptions or discussions on these. We are not sure if this paper's conclusions can hold for transformers trained end-to-end.\n\n2: When this paper discussed about low-frequency preference to large patch size, it didn't provide enough quantitative analysis of how this emerges during training. For example, visualization of any frequency-domain activations through epochs may support this claim. More explanations and rigorous proofs are needed to clarify these points. \n\n3: For its geometric bias, its geometry discussion shows that angle between embedded vectors for discrete embeddings is much larger than that for continuous embeddings. Also, this grows with the number of bins. However, this paper didn't measure vocabulary size systematically over this observation shown in its figure. More ablation study should be discussed and provided to show that whether the reported geometry is an inherent property of discretization or an artifact of one particular size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)