Exploring Representations and Interventions in Time Series Foundation Models
Authors: Michał Wiliński
Mononito Goswami
Willa Potosnak
Nina Żukowska
Artur Dubrawski
Venue: ICML 2025 poster (2025)
Keywords: Time Series Foundation Models
Model Steering
Interpretability
Pruning
Primary area: applications->time_series
TL;DR: We investigate why time series foundation models work, the kinds of concepts that these models learn, and how can these concepts be manipulated to influence their outputs?

Abstract: Time series foundation models (TSFMs) promise to be powerful tools for a wide range of applications. However, their internal representations and learned concepts are still not well understood. In this study, we investigate the structure and redundancy of representations across various TSFMs, examining the self-similarity of model layers within and across different model sizes. This analysis reveals block-like redundancy in the representations, which can be utilized for informed pruning to improve inference speed and efficiency. We also explore the concepts learned by these models, such as periodicity and trends. We demonstrate how conceptual priors can be derived from TSFM representations and leveraged to steer its outputs toward concept-informed predictions. Our work bridges representational analysis from language and vision models to TSFMs, offering new methods for building more computationally efficient and transparent TSFMs.

Exploring Representations and Interventions in Time Series Foundation Models
Michal Wilinski!! Mononito Goswami
Willa Potosnak
Nina Zukowska Artur Dubrawski
Abstract
Time series foundation models (TSFMs) have shown promis- ing performance on multiple modeling tasks such as fore- casting. classification, anomaly detection, and imputation, across a wide range of domains, and in settings with varying amounts of data and supervision. However, the underlying mechanisms and learned representations of TSFMs remain largely unexplored. The nature of learned representations in TSFMs remains largely unknown, including (1) their orga- nizational structure, (2) the human-interpretable concepts they encode and their locations, and (3) how they can be ma- nipulated to integrate prior knowledge for more informed predictions. Gaining deeper insight into the inner work- ings of TSFMs is key to improving their performance and trustworthiness.
Time series foundation models (TSFMs) promise to be powerful tools for a wide range of appli- cations. However, their internal representations and learned concepts are still not well under- stood. In this study, we investigate the structure and redundancy of representations across various TSFMs, examining the self-similarity of model layers within and across different model sizes. This analysis reveals block-like redundancy in the representations, which can be utilized for in- formed pruning to improve inference speed and ef- ficiency. We also explore the concepts learned by these models, such periodicity and trends. We demonstrate how conceptual priors can be derived from TSFM representations and leveraged to steer its outputs toward concept-informed predictions. Our work bridges representational analysis from language and vision models TSFMs, offering new methods for building more computationally efficient and transparent TSFMs.
We address each of these knowledge gaps by proposing ap- proaches to systematically analyze, probe, and intervene in TSFMs through their learned representations. To address the first gap, we explore the fundamental question: Do dif- ferent TSFMs learn and organize knowledge in similar manner? This analysis is crucial for understanding how knowledge is organized in different models and whether rep- resentations are redundant across groups of layers, which can be potentially leveraged to improve inference efficiency. Next, we address the second gap by investigating: What human-interpretable concepts can TSFMs learn, and where do these concepts emerge within the latent rep- resentations of the model? Building representation analysis, we propose new methods to iden- tify and localize these concepts across layers. Finally, we tackle the third gap by asking: Is it possible to intervene in TSFMs during inference by applying conceptual pri- ors to steer time series predictions? Through novel approach, we demonstrate how steer learned concepts by adjusting representations across layers, allowing the model to generate concept-informed predictions.
1. Introduction
Foundation models have taken significant strides in model- ing both textual (Brown et al., 2020) and visual (Dosovitskiy et al., 2020) data, and have made complex language and image processing accessible non-experts. These mod- els pretrained can be used to solve multiple tasks across a variety of do- mains, with little to no adaptation. Recently, growing body of work (Garza& & Mergenthaler-Canseoo, 2023; Goswami et al., 2024; Rasul et al., 2024; Das et al., 2024: Woo et al., 2024; Ansari et al., 2024) has extended the benefits of this paradigm to time series data, a modality prevalent in fields such finance (Taylor, 2008), healthcare (Goswami al., 2021), and climate science (Schneider & Dickinson, 1974).
layer-wist wise
massive internet-scale datasets and
The main contributions of this paper are:
(i) TSFM Representation Similarity Analysis and Pruning. We translate prior work vision trans- formers and convolutional neural networks to quantify the representation similarity in TSFMs. Our analysis reveals that model size substantially influences how representations are organized. In larger models, we observe high CKA similarity across groups of layers,
Equal contribution AAton Lab, School of Computer Science, Carnegie Mellon University. Correspondence to: Michal Wilinski <mwilinsk @ andrew.emut emu."
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Exploring Representations and Interventions in Time Series Foundation Models
indicating redundant knowledge storage. Leveraging this insight, we propose a novel block-wise layer prun- ing strategy to remove redundant representations. This approach reduces model size and speeds up inference while maintaining accuracy.
Deep learning models often operate black boxes, mak- ing their internal mechanisms and learned representations difficult to understand. One approach to gaining insights into these models is by comparing their intermediate rep- resentations. Similarity metrics can be used to determine the similarity or dissimilarity of representations at different stages of a model, revealing the hierarchy, homogeneity, and redundancy of learned features. Previous studies focused quantifying the similarity of neural network representa- tions (Raghu et al., 2017; Kornblith et al., 2019). (Raghu et al., 2021) demonstrated the use of these metrics for ana- lyzing and comparing representations in vision transformers and CNNs, providing valuable insights into their function- ing. (Nguyen et al., 2021) investigated the impact of model size and training data ratios on similarity patterns. They also explored model pruning based on similarity. Building on prior studies, we present the first comprehensive analysis of TSFM representations, providing insights into their internal mechanisms.
(ii) Concept Identification in TSFM Representations. We introduce new method to identify and localize specific time series concepts, such as trends and sea- sonal patterns, across model layers. Our findings show that TSFMs gradually learn these intuitive concepts across layers.
(iii) Steering Methods for Concept-Informed TSFM Predictions. We demonstrate a novel approach to steer model predictions by intervening in TSFM latent representations along specific conceptual directions (e.g., introducing an upward trend in forecasts) as a way to guide model output toward concept-informe predictions without additional training or fine-tuning.
Identifying Learned Concepts in Pretrained Models. Understanding the internal representations learned by pre- trained models has been an active area of research, partic- ularly in the context of LLMs and vision models. Previ- ous studies have explored whether individual neurons or directions in a model's 's latent space correspond to specific features or concepts (Dalvi et al., 2019; Goh et al., 2021; Gurnee et al., 2023; Elhage et al., 2022). These investi- gations often focus on identifying linear representations, where features are encoded as linear combinations of neu- ron activations. Recent work has also employed various probing techniques to classify and interpret these internal representations, addressing aspects such as truthfulness and model robustness (Azaria & Mitchell, 2023: Zou et al., 2023; Burns et al., 2023; Marks & Tegmark, 2024). While prior research has primarily focused on language and vision mod- els, we demonstrate that simple techniques enable us to effectively probe TSFMs to identify and localize concepts.
2. Related Work
Time Series Foundation Models (TSFMs). TSFMs are versatile neural networks pretrained on vast amounts of time series data, and have shown impressive accuracy even in zero-shot settings. Several TSFMs have been proposed (Garza& & Mergenthaler-Cansseo, 2023; Liu et al., 2023; Das et al., 2024; Woo et al., 2024; Goswami et al., 2024; Ansari et al., 2024; Ekambaram et al., 2024; Talukder et al., 2024). Most of these are based on variations of the Transformer architecture (Vaswani et al., 2017), with differences in to- kenization strategies, pretraining datasets, and the specific tasks they are designed to address. Consequently, we focus on developing methods to understand Transformer-base based models. We specifically analyze three representative models: MOMENT (Goswami et al., 2024), Chronos (Ansari et al., 2024), and MOTRAI (Woo et al., 2024), which are fully open-source and offer distinct approaches to time series tok- enization (patch vs. discrete), architecture (encoder-only vs. encoder-decoder), and pretraining objectives (imputation vs. forecasting). Recent advancements in TSFMs have focused mostly improving their capabilities through scaling, spe- cialization, or improving inference efficiency (Liu et al., 2024; 2023), as well as by incorporating long and multi- variate contexts (Liu et al., 2025; Žukowska et al., 2024). However, there has been limited research on understand- ing what specific TSFMs Iearn. For example, Goswami et al. (2024) showed that MOMENT learns some human in- terpretable concepts, while Ansari et al. (2024) investigated the failure modes of Chronos in controlled settings, and Potosnak et al. (2024) explored compositional reasoning within time series models, including TSFMs.
3. Methods
We study TSFMs using three complementary methods of analysis and interventions concerning model layer-wise rep- resentations. In Section 3.1, we examine learned repre- sentations through the lens of similarity, uncovering stored knowledge redundancy in TSFM representations. We lever- age this redundancy to prune multiple layers of pretrained models, thereby improving their efficiency. In Section 3.2, we identify specific concepts learned by TSFMs and local- ize them to specific model layers and token positions. In Section 3.3, we demonstrate how conceptual priors can be derived from TSFM representations and leveraged to steer its outputs toward concept-informed predictions.
Analyzing Representations of Deep Learning Models.
2

---

Exploring Representations and Interventions in Time Series Foundation Models
3.1. Representation Similarity Analysis and Pruning
Similarity Matrix
23
To gain a deeper understanding of TSFM representations, we investigate the following research questions: (RQ1) How similar are the representations learned by models of the same size but belonging to different families? (RQ2) How do these representations differ across models of varying sizes within the same family? (RQ3) How similar are the representations learned by corresponding layers of different TSFMs within the same family?
21
KOMMNN L.LNT aarge laver eaeer
19
15
11
9
FFN
5
We consider several metrics commonly employed in the literature to analyze the similarity between learned repre- sentations. While our primary analysis relies on Centered Kernel Alignment (CKA) (Kornblith et al., 2019), we also explored Cosine Similarity and Singular Vector Canonical Correlation Analysis (SVCCA) (Raghu et al., 2017). For brevity, we provide a brief overview of CKA below, while detailed descriptions of the remaining metrics can be found in Appendix B.
Norm
1
3
11 MOMENT-arre layer
7
35
1719211222
Mult-Head Alln.
Norm
0.0
0.2
0.6 CKA Similarity
0.4
0.8
1.0
Layer
Figure 1: For each identified block of layers exhibiting redundant representations (red), we skip computation of their internal layers (blue). For example, if a block consists of five layers, we skip layers through 4, retaining only the first and last layers reduce representation redundancy while maintaining model integrity. More details on pruning can be found in Appendix C.
Representational Similarity using CKA. CKA measures the similarity of representations by comparing the centered kernel matrices. It has been shown to be effective in captur- ing similarities between layers deep networks (Komblith al., 2019). The general form of CKA between two sets of representations X and Y is defined
Algorithm Block-wise Pruning (Skipping Computation)
HSIC(X, Y)
CKA(X, VHSIC(X, X) HSSICY,Y Y)
(1)
Require:
Trained model M with layers {41, 12. 4n
Identified redundant blocks B = (bz, b2. .op....,k))) for each block by in B do
where HSIC denotes the Hilhert-Schmidt Independence Cri- terion (Gretton al., 2005).
Let b; consist of layers 1, to la (Block edges at l and [e}
For computational efficiency, we utilized a linear kernel in our CKA calculations, resulting in the following simplified formula:
for layer index j = + to e do
Remove layer 15 from model M end for end for
X X
CKAjjnnar XX Y)
Y
(2)
return pruned model M'
where || ||F denotes the Frobenius norm. The denominator in Equation 2 ensures that the metric value falls within the range of 0 to 1, facilitating interpretability. A high CKA value indicates strong alignment between the two sets of representations, suggesting that the layers are likely learning similar features or concepts.
integrity of each block, retain the first and last layers of each block and skip computation by removing these blocks altogether.
To demonstrate the effectiveness of our proposed pruning strategy, we explore two pruning configurations, one in which we prune all redundant blocks, and the other where we prune only single block. We compare the performance of these pruned models to the original, unpruned TSFMs us- ing standard task- accuracy metrics (Mean Squared Error and Mean Absolute Error) and efficiency metrics (in- ference time in milliseconds and theoretical model size in megabytes). We evaluate these models on widely used im- putation (Zhou et al., 2021) and forecasting (Ansari et al., 2024) benchmarks in both zero-shot settings and after linear
Pruning TSFMs Based on Representational Similarity. Large TSFMs typically learn redundant representations, which often manifest as block-like structures in heatmaps de- picting pairwise similarity between layer activations (Figure 1). We leverage this redundancy to prune TSFMs, enhanc- ing inference speed while preserving accuracy. Building on prior work (Nguyen et al., 2021), we propose simple and effective layer pruning strategy, which we call Block-wise Pruning, outlined in Algorithm 1, To preserve the structural
3

---

Exploring Representations and Interventions in Time Series Foundation Models
probing (Goswami et al., 2024). While prior work such as (Nguyen et al., 2021) primarily focused on pruning indi- vidual layers, our approach differs by targeting entire blocks of self- layers, preserving boundary layers to main- tain representational continuity. Furthermore, we extend block-level pruning to large pretrained TSFMs, demonstrat- ing practical effectiveness on diverse real-world tasks be- yond classification, achieving substantial inference speedups (up 52%) with minimal performance degradation, even in zero-shot scenarios.
each layer and token classify time series x as a constant or sinusoid pattern. Classifiers fiz (h (2), (B()) are trained on the hidden representation h, at each i-th layer and each j-th token to update the parameters 0.. Additionally, we per- form probing on representations averaged along the token dimension for each i-th layer. The linear probes are trained to optimize the Fisher Criterion, function that aims to max- imize the distance between class means while minimizing within-class variance:
Drisher (c, 8) (1t= pe)2 o3 + o2
3.2. Identifying and Localizing Time Series Concepts
(3)
Through our experiments, we aim answer the following research questions: (RQ4) Do TSFMs represent concepts as- sociated with specific data generating functions distinctly in the latent space? (RQ5) Are these learned concepts localized to specific layers and tokens within TSFMs?
Here, μ4 and jee correspond the mean embedding values, computed using all time series of a given class. Similarly, of and of correspond to the variance computed across the dimension for each class.
To systematically explore the ability of TSFMs to under- stand intuitive time series concepts, we randomly generate large number of synthetic univariate time series. Each randomly generated time series belong to one of two pattern classes: constant or sinusoidal. Constant patterns, repre- sented by y(t) mt + b, capture long-term non-periodic trends. Sinusoidal patterns, modeled as y() = asin (stt represent periodic processes. By controlling the parameters m, b, a, and f. we can systematically generate time series with varying slope, intercept, amplitude, and periodicity, respectively. Despite their simplicity, these data generation mechanisms capture a wide range of real-world time series patterns, Example series are shown in Fig. 3. For a detailed description of the data generation process, please refer to Appendix A.
Localizing Linearly Represented Features. To localize which layers and tokens learn a specific concept, we com- pute the Fisher's 's Linear Discriminant Ratio (LDR) between the classes using the mean and variance statistics of h))) for each predicted class ý. which is determined using the classifier fúy during linear probing. The goal of LDR is to maximize the separation between the classes by comparing the variance o² within each class to the difference between the class means, p. A larger ratio indicates a clearer separa- tion between the two classes, which can aid in concept local- ization by identifying where the classes are well-separated in the feature space. When applied in the context of neural network activations, LDR helps highlight which layers or features are most discriminative
f
LDR ( (5) = (4 (1,0)18s - 14 14. 1=
Identifying Linearly Represented Features. We build on the investigation approach outlined in (Marks & Tegmark, 2024). We say that a feature is linearly represented in a foundation model M if it is represented as a direction in its latent space. If this feature is linearly represented in M, also want identify which layer I in M learns this concept in the most discriminant way. As concrete example, consider that we want to check whether M can distinguish between constant and sinusoidal patterns.
(4)
(U, - -2a)² 03 + o²
(5)
Here, μı, and 14c correspond to the mean computed across the dimension for each class. Similarly, o and o² correspond to the variance computed across the sample dimension n for each class. Let V = [vi,]] E TR. L be the matrix of LDR values, where v.j represents the LDR value for the i-th layer and j-th token, with / layers and N tokens. The LDR output is scaled between 0 and I using min- max scaling to allow for consistent comparison across layers.
To determine whether the feature (sinusoidal vs. constant time series) is linearly represented, we leverage the afore- mentioned synthetic dataset which comprises of multiple sinusoids and constant time series randomly sampled us- ing our data generating function. Using this dataset, we extract the residual stream of each transformer block after the feed-forward layer. Let h
By visualizing the scaled LDR values as shown in Figure 2, one can identify which layers and tokens exhibit the highest degree of separation between concept classes, offering in- sights into the network's 's internal representations for concept intervention techniques.
(i) E = D denote the hidden representation of a time series x at i-th layer and j-th token of M, where D is the dimensionality of the hidden layer. Linear probing involves training separate linear models for
4

---

Exploring Representations and Interventions in Time Series Foundation Models
Concept Localization
Linear Probing
Concept Steering
1.0
Principal Compament
Tussuet Zasass
Priocipal Companent
Yonstan Veewwe
Steered
0.8
50
0.4
30
-100
-50
100
Layer
Principal Compunent 1
Principal Componant
LDRicaled (DR(()) 19). {c
sc
Sy(h/r..0 00000
Foundation Model
h:
Foundation Model
10
10
-10
Deriving Steering Vectors, Si
10
Multi- Attn.
myd(h,
M
M
M,
10
10
Norm
med(h,
M.
Encoder
Figure 2: Overview of linear probing, concept localization, and steering, During linear probing we train linear classifiers (h,)), o(0) classify time series x into constant and sinusoid classes, using hidden representations h at the i-th layer and j-th token. We localize concepts using Fisher's Linear Diseriminant Ratio (LDR) between the classes at each layer and token. The concept steering vector at the i-th layer is defined as the difference between the median activation matrices of the sinusoid and constant time series classes, M, M, Vectors of all layers are stacked into a steering matrix S to steer model predictions towards desired concepts by updating the embeddings as h; + h; + XS, where À is scalar that controls the strength of the intervention.
250
predictions? For example, can we add periodicity or an upward trend constant time series? (RQ7) possible to combine multiple steering interventions manipulate model predictions towards complex compositions of various concepts? For instance, can we steer a model to add both trend and periodicity to a constant signal?
%
100
200
300
400
500
600
Signal
0
-50
100
200
300
400
500
600
250
Deriving Steering Matrices Model Steering. Once we have identified that feature is linearly represented in the latent space of the M, to manipulate the latent space and generate time series that reflect intended concepts. For instance, to introduce peri- odicity to a constant time series, we can utilize steering matrix S, as illustrated in Figure 2. By strategically interven- ing in M using this steering matrix, we can bias its outputs towards predicting periodic time series. To construct steer- ing matrix, we first derive steering vectors S, NxD for each layer i. These vectors represent the change that activations in layer i must undergo such that M produces periodic outputs. S, is simply the difference between the median activation matrix of the constant time series M,,, from that of sinusoids M,, We stack these vectors for all layers to derive the steering matrix. This matrix allows us to
0
100
200
300
400
500
600
steering interventions
Time
Figure Examples of synthetic data generated for con- cept localization and steering experiments include signals with varying trends (top), sinusoidal signals with varying frequencies (middle), and combinations of signals with vary- ing trends and sinusoidal patterns (bottom).
3.3. Concept-Informee Predictions via Model Steering
Through our experiments, we aim to answer the following research questions: (RQ6) Can we leverage these learned concepts to steer model output toward concept-informed
5

---

Exploring Representations and Interventions in Time Series Foundation Models
simultaneously intervene across multiple tokens and layers during inference, which we found to be more effective than single-token interventions. During inference, to steer model predictions, at each layer i, we update its hidden represen- tation as follows: h; < h; + XS, where À E R is a scalar that controls the strength of the intervention.
Chronos Family
11
9
7
5
3
1
3
5 7 9 11 11131517792122
We explore two intervention techniques: (1) deriving steer- ing vectors using the mean of hidden activations rather than the median, and (2) steering single token versus all tokens throughout the model. While our methods are applicable foundation models, we focus on two prominent TSFM families for brevity: MOMENT (Goswami et al., 2024) and Chronos (Ansari et al., 2024). Both these models are fully open-source, come in different sizes, yet have fundamentally differ- ent design choices. For example, Chronos is based encoder-decoder transformer (in our work we will inves- tigate encoder part) model which takes discretized time series as input, whereas MCMENT is multi-task, encoder- only model which takes continuous time series patches as input. We supplement our representation analysis results with Moirai (Woo et al., 2024), another popular TSFM which comes in different sizes. More information on hyper- parameters can be found in Appendix E.
Tiny vs Mini
(ii) Base vs Large
Figure 4: Similarity between representations learned by different layers in TSFMs of the same family but different sizes. Initial layers tend to learn similar representations, while the similarity gradually decreases in the later layers.
to a wide range of transformer-base
Chronos family, where early representations are more con- sistent across models, but later layers become increasingly specialized as model depth increases as shown in Fig. 6.
Block-wise pruning can improve model throughput, without compromising accuracy. We observed consis- tent improvements in memory efficiency and inference speed compared to unpruned model counterparts. For exam- ple, pruning only Block 3 for MOMENT-Large, resulted in a 11% decrease in estimated model size with a 5% de- crease in inference time. Furthermore, this pruned model had lower zero-shot imputation MAE for 5 of 9 datasets (ETTh2, ETTml, ETTm2, Exchange, and Weather) as shown in Tab. 3. Chronos- Large results for zero-shot experiments are reported in Tab. 4. Detailed results on mem- ory usage and speed improvements can be found in Tab. 7. Although pruning consistently improved memory efficiency and inference speed compared unpruned counterparts, per- formance varied between pruning methods and datasets, and some methods showed considerable degradation. In addition to zero-shot experiments, we conducted fine-tuning experi- ments on pruned models-including MOMENT DOMENT-Large with all block redundancies removed- to evaluate the impact of the most aggressive pruning approach on forecasting perfor- mance. Notably, the pruned model performed on par with its unpruned baseline, as shown in Table 1, demonstrating the potential of our block-wise pruning approach to maintain performance while reducing model complexity. Complete finetuning results are provided in Table 6 in Appendix G.
4. Results
Analyzing representations offers interesting insights. Our analysis of model representations demonstrates that both model size and internal architecture con- siderably influence how representations are organized. Fig. 7 shows heatmaps which reveal that larger mod- els, such as MOMENT-Large Chronos-Large, and Moirai-1. 1-R Large, have similar representations across specific groups of layers forming distinct and in- tricate block patterns, which may reflect unique stages of representation learning. More complex block patterns are observed with increasing model size, indicating that scaling may enhance the richness and organization of internal repre- sentations. However, it may also increase redundant knowl- edge storage through similar representations across layers, as suggested by high CKA similarity measured in block pat- terns. Interestingly, within model families (e.g.. Chronos and Moirrai, scaling does not always result in predictable heatmap changes. Larger models, like Chronos La Largge and Moirai- Large, demonstrate more refined and com- plex transformations of representations that are not easily extrapolated from their smaller versions as shown in Fig. 15). Moreover, cross-model similarity analysis results in Fig. 4 reveal that while carly layers tend to have high similarity across models of different sizes, the similarity measures among later layers diverge more notably, particularly in larger models. This divergence is especially evident in the
TSFMs learn intuitive linear concepts. Our concept lo- calization results in Fig. 7 show that certain concepts repre- sented by BOMENT-Larqe are linearly separable and that this separability is not consistent but rather emerges at spe- cific layers in the model. We also found intuitive differences in the locations where these concepts are learned. We ob- served that certain concepts, such as distinguishing between constant and sinusoidal patterns, require examination of the entire time series. In contrast, differentiating between in-
6

---

Exploring Representations and Interventions in Time Series Foundation Models
0
1.0
1.0
17
17
18
15
Patch Position
12
0.8
0.8
Scaled LDR
24
0.6
0.6
138 713B111222
36
0.4
0.4
(i) MOMENT
(ii) Chronos
(iii) Moirai
48
0.2
0.2
Figure 5: Pairwise similarity of layer measured using CKA for large variants of TSFMs (dark blue - low similarity, yellow + high similarity). All TSFMs learn redundant representations with manifesting as block-like structures.
60
0.0
0.0
0
4
12 Model Depth
8
16
20
Figure Linear separability of concepts the patch level (y-axis) across model layers (x-axis). Yellow indicates higher separability than blue in the colorbar. Linear sep- arability is measured using the Linear Discriminant Ratio (LDR), derived from model embedding statistics for each predicted class (constant vs. sinusoidal patterns). Certain concepts captured by MOMENT-Large are linearly sepa- rable, but this separability not uniform; it emerges at specific layers and patches of the model. More examples of linear separability of concepts shown in Fig. 14.
ETTh1
ETTh2
ETTml
ETTm2
Weather
Vanilla
0.385
0.287
0,290
0.171
3.260
0.153
Pruned
0.388
0.296
0,290
0.173
2.981
0.152
Table 1: MSE of fine-tuned vanilla and pruned MOMENT variants on long-horizon forecasting datasets (Zhou et al., 2021). Pruned models perform on par with the original, while reducing memory consumption by: > 50% and im- proving inference time per sample by =1 ms. Full results are provided in Tables and in the Appendix.
Chrones Family
the robustness of concept steering, we generated datasets using multiple random seeds and confirmed that consistent steering effects emerge across samples. This is supported by analyses of linear concept separability and concept emer- gence in the latent space. Furthermore, in Appendix F, we present steering results on real-world ECG data, demonstrat- ing how to steer time series from the normal to the abnormal heartbeat class in the ECG5000 dataset. Our concept steer- ing method can be effectively leveraged both synthetic and real-world data, helping to validate its robustness and practical applicability.
Tiny (ii) Mini (iii) Small 11 (iv) Base (v) Large
Figure 6: Does model size influence the patterns of learned representations? The emergence of block-like patterns in larger models cannot be directly inferred from the patterns observed in smaller models. (expanded view is Fig. 16).
creasing and decreasing trends can be achieved by focusing on the initial and final patches. However, we did not identify specific locations where models learn to distinguish between time series of different amplitudes. This may be attributed the normalization of input time series, a common practice in many TSFMs, including MCMENT.
To assess the impact steering in the latent space, we analyzed changes the hidden representations before and after applying concept steering by projecting them into two-dimensional space using Principal Component Analysis (PCA). We found that steering in the latent space is reflected in these lower-dimensional representations, as illustrated in Fig. 11. The PCA reduction often captured the concept direction as one of the principal components. This could be attributed to the distinct concepts our synthetic data.
We can effectively steer TSFM predictions. Our concept steering interventions effectively transform the latent space of TSFMs, resulting in model predictions that align with the intended concepts, as demonstrated in Fig. 8. We success- fully introduced periodicity and trend concepts to constant time series and demonstrated the ability to combine multi- ple steering vectors to create more complex patterns. By combining steering vectors representing increasing trends and sinusoidal patterns, we were able to steer model pre- dictions towards a combination of these features. To assess
Interestingly, the method of obtaining the steering matrix, either by computing the mean or median across embedding concept classes, has no notable effect the steered output as shown in Fig. 12. However, applying concept steering interventions across all tokens is necessary to achieve the in- tended steered concept output compared to applying concept steering interventions to a single token. Moreover, the À pa- rameter can have considerable effect on steered output. For

---

Exploring Representations and Interventions in Time Series Foundation Models
Concept Steering: Use conceptual priors to shape time series predictions.
5.0
Pertarbed Sua Pestattoo
Pertucted Nien Pertuched
Pertorbed
Nuz Pertarbed
2.5
0.0
2
:05
2003
300
4.00
100
200
300
405
500
200
300
400
500
Trreetop
Tinsstop
Tizosiep
60
80
60
40
tatudded
10
Non Poctaiked
10
Timcatep
Timestep
Taneotop
(i) Periodicity, B
(ii) Trend & periodicity, 0.5
(iii) Trend, B = 1.0
Figure 8: Visualization of MOMENT reconstruction and Chronos forecasting predictions (bottom), comparing concept steering in the latent space (blue) with the unsteered baseline (orange). The original constant signal and steered output referred to as Non-perturbed and Perturbed in the figure legend, respectively. A constant time series is first provided as input to both the proposed steering approach and the unstecred baseline. The non non-perturbed output remains constant signal, as expected, while the perturbed output reflects the new concept introduced via the steering matrix-such as trend, seasonality, or both- wuth-dependingg on the specified 3 parameter value. For both outputs, we show the raw model predictions (lighter color) and their moving averages (darker color) to reduce noise artifacts. Steering results are shown for the following experiments: (i) steering a constant signal have periodicity (3 0), (ii) steering a constant signal have an increasing slope, or trend (B 1.0), and (iii) steering constant signal to have trend and periodicity (3 0.5).
Chronos, steering required tuning the parameter À R 0.1 for effective performance, whereas MOMENT maintained effective steering with À = 1.
their original training. For example, we can steer vital signs in healthcare based on new treatments or adjust financial forecasts based on emerging events such as positive earnings surprises. Adding these contextual factors, even as simple trends, has significant implications for improving model predictions in zero-shot and disto--distribution scenarios. Steering techniques can also support realistic synthetic gen- eration of time series variations. In experiments with the ECG Arrhythmia classification dataset (ECG5000), we demonstrate that time series classified as normal heartbeats can be steered to produce time series classified as abnormal heartbeats. Such data generation capabilities can be used to augment data for model training or generate new sam- ples for us better understand the decision boundaries in model predictions. Prior work has already shown that data augmentation techniques can improve TSFM generalization by enhancing model robustness and exposing it to wider variety of patterns (Ansari et al., 2024).
5. Discussion
We explored two complementary approaches to probing and intervening in TSFMs. We gained valuable insights into their internal mechanisms and identified opportunities for improvement. For instance, our analysis revealed redun- dancy in learned representations. We leveraged this repre- sentational redundancy, inherent to large over- werr-parameterized TSFMs, devise simple block-wise pruning strategy. This strategy effectively reduced the size and computational cost of these models without compromising performance, demonstrating the potential for distilling smaller, more effi- cient models from larger TSFMs.
We also explored ways to use conceptual priors to shape time series predictions. Concept steering enables controlled, post- training updates model embeddings through lightweight steering matrix, allowing models incorporate new con- cepts events into predictions without requiring retraining or fine-tuning. In addition to reducing the computational cost of further training, concept steering has many practical applications. It enables users to imbue models with new or missing contextual factors after training, which can be partic- ularly valuable if models do not include certain scenarios in
Limitations and Future Work. While our methods are broadly applicable to differrent transformer-bast based foundation models, future research should explore other architectures, such as state space models (Gu & Dao, 2024) and stacked multi-layer perceptrons (Ekambaram et al., 2024). More- over, future studies could evaluate steering methods on other real-world time series datasets and domains, as well as on tasks beyond forecasting and imputation, such as anomaly detection and classification.
8

---

Exploring Representations and Interventions in Time Series Foundation Models
Impact Statement
Trebacz for inspiring discussions and support that helped shape this work.
While our pruned TSFMs demonstrate promising perfor- mance, it is crucial to exercise caution when using them, especially in high-stakes applications such as healtheare. Before deploying these models for critical decision-maker making, we strongly recommend fine-tuning and evaluating them on task-specific, in-domain datasets using relevant metrics. The ability to steer model predictions offers numerous bene- fits but also raises concerns regarding potential biases. We urge users to exercise caution when using our proposed steering strategies and carefully consider the potential implications of manipulating model outputs.
References
Ansari, A. F., Stella, L.., Turkmen, A. C.,, Zhang, X., Mer- cado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., et al. Chronos: Learning the laaggugge of Time Series. Transactions on Machine Learning Research, 2024.
Azaria, A. and Mitchell, T. The Internal State of an LLM
Knows When It's Lying. In Findings of the Association for Computational Linguisties: EMNLP 2023, 2023.
Reproducibility Statement
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, 2020. Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discov- ering Latent Knowledge in Language Models Without Supervision. International Conference on Learning Represenaatinss 2023.
To ensure reproducibility, we have made our code anony- mously accessible through GitHub, All time series foun- dation models used for our analysis are publicly available and open-source: Chronos, MOMENT, and MOIRAI. All models were trained and evaluated on computing clus- ter consisting of 128 AMD EPYC 7502 CPUs, 503 GB of RAM, and 8 NVIDIA RTX A6000 GPUs, each with 49 GiB RAM.
Acknowledgment
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., and Glass, J. What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models. In AAAI Conference on Artificial Intelligence 2019.
Funding This work was partially supported by NSF (awards 2406231 and 2427948), NIH (awards ROINS124660 R01DK131586, and R01NR013912), and the US Army (W91 INF-20-D0000))
Das, A., Kong, W., Sen, R., and Zhou, Y. A decoder- only foundation model for time-series forecasting. In International Conference on Machine Learning, 2024.
Datasets and Tooling We are grateful to the creators and maintainers of the UCR Time-Series Archives (classifica- tion& & anomaly) (Dau et al., 2018; Keogh et al., 2021), the Monash Forecasting Archive team (Godahewa al., 2021), the Informer long-horizon data providers (ETT, Electricity, Exchange Rate, Weather, Traffic, ILI) (Zhou et al., 2021), the custodians of TSB-UAD (Paparrizos et al., 2022), and every other public dataset we relied on. We would also like to thank the open-source community- authors of the li- braries, frameworks, and tooling that make modern machine learning research possible.
Dau, H. A., Keogh, E., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., Yanping, Hu, B., Begum, N., Bagnall, A., Mueen, A., Batista, G., and Hexagon-ML. The UCR Time Series Classification Archive, 2018
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2020.
Discussions We thank the anonymous reviewers and revaerev for their constructive feedback and sugges- tions, which helped improve the quality and clarity of this work. We are grateful to Carnegie Mellon University (CMU), the Robotics Institute, and the Auton Lab for provid- ing the research environment and infrastructure that made this work possible. We also thank Rachel Burein and Dr. John Dolan for their support during the CMU Robotics Insti- tute Summer Scholars program. Additionally, we thank our fellow Auton Lab members Jieshi Chen, Piotr Bartosiewicz, Chi-En Teh, Nicholas Gisolfi, Ignacy Stepka, and Kacper
Ekambaram, V., Jati, A., Nguyen, N. H., Dayama, P., Reddy, C., Gifford, W. M., and Kalagnanam, J. TTMs: Fast Multi-level Tiny Time Mixers for Improved Zero-shot and Few-shot Forecasting of Multivariate Time Series. In Advances in Neural Information Processing Systems, 2024.
Elhage, N., Hume, T., Olsson, C.,, Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds Z., Lasenby, R., Drain,
9

---

Exploring Representations and Interventions in Time Series Foundation Models
D., Chen, C., et al. Toy Models of Superposition. arXiv preprint arXiv:2209. 10652, 2022.
Liu, Y., Qin, G., Huang, X., Wang. J., and Long, M. Timer- XL. Long-Context Transformers for Unified Time Series Forecasting. In International Conference on Learning Representations, 2025.
Garza, A. and Srngenthhler-Canseeo M. TimeGPT-I, 2023.
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., and Montero-Manso, P. Monash Time Series Forecasting Archive. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Marks, S. and Tegmark, M. The Geometry of Truth: Emer- gent Linear Structure in Large Language Model Rep- resentations of True/False Datasets, In Conference on Language Modeling. 2024.
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., and Olah, C. Multimodal Neurons in Artificial Neural Networks. Distill, 2021.
Nguyen, T., Raghu, M., and Korbblihh, S. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. International Conference on Learning Representations, 2021.
Goswami, M., Boecking, B., and Dubrawski, A. Weak Supervision for Affordable Modeling of Electrocardio- gram Data. In AMIA Annual Symposium Proceedings, volume 2021, pp. 536. American Medical Informatics Association, 2021.
Paparrizos, J., Kang, Y., Boniol, P., Tsay, R. S., Palpanas, T., and Franklin, M. TSB-UAD: An End-to-end Bench- mark Suite for Univariate Time Series Anomaly Detec- tion. Proceedings of the VLDB Endowment, 2022.
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., and Dubrawski, A. MOMENT: A Family of Open Time- series Foundation Models. In International Conference on Machine Learning, 2024.
Phang, J., Liu, H., and Bowman, S. R. Fine-Tuned Trans- formers Show Clusters of Similar Representations Across Layers. In BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, 2021.
Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. Measuring Statistical Dependence with Hilbert-Schmice Norms. In International Conference on Algorithmic Learning Theory, 2005.
Potosnak, W., Challu, C., Goswami, M., Wilinski. M., Zukowska, N., and Dubrawski, A. Implicit Reason- ing in Deep Time Series Forecasting. arXiv preprint arXiv: 10840, 2024.
Gu, A. and Dao, T. Mamba: Linear-Time Sequence Model- ing with Selective State Spaces. In Conference Lan- guage Modeling, 2024.
Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. SVCCA: Singular Vector Canonical Correlation Anal- ysis for Deep Learning Dynamics and Interpretability. Advances in Neural Information Processing Systems, 2017.
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., and Bertsimas, D. Finding Neurons in Haystack: Case Studies with Sparse Probing. Transactions on Machine Learning Research, 2023.
Raghu, M.. Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. Do Vision Transformers See Like Con- volutional Neural Networks? In Advances in Neural Information Processing Systems, 2021.
Keogh, E., Dutta Roy, T., Naik, U., and Agrawal, A. Multi- dataset Time-Series Anomaly Detection Competition. Presented at SIGKDD 2021, 2021.
Rasul, K.,, Ashok, A., Williams, A. R., Ghonia, H., Bhag- watkar, R., Khorasani, A., Bayazi, M. J. D., Adamopou- los, G., Riachi, R., Hassen, N., Biloš, M., Garg, S., Schneider, A., Chapados, N., Drouin, A., Zantedeschi, V., Nevmyvaka, Y., and Rish, I. Lag-Llama: Towards Foun- dation Models for Probabilistic Time Series Forecasting, 2024.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Simi- larity of Neural Network Representations Revisited. In International Conference on Machine Learning, 2019.
Liu, X., Liu, J., Woo, G., Aksu, T., Liang, Y., Zimmer- mann, R., Liu, C., Savarese, S., Xiong, C., and Sahoo, D. Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts. arXiv preprint arXiv:2410. 10469, 2024.
Schneider, S. H. and Dickinson, R. E. Climate Modeling. Reviews of Geophysics, 12(3):447-493, 1974.
Liu, Y.. Zhang, H., Li, C., Huang, X., Wang, J., and Long, M. Timer: Generative Pre-trained Transformers Are Large Time Series Models. In International Conference on Machine Learning, 2023.
Talukder, S. J., Yue, Y., and Gkioxari, G. TOTEM: TOk- enized Time Series EMbeddings for General Time Series Analysis. Transactions on Machine Learning Research, 2024.
10

---

Exploring Representations and Interventions in Time Series Foundation Models
Taylor, S. J. Modelling Financial Time Seriex. World Scien- tifie, 2008.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. Attention Is All You Need. In Advances in Neural Information Processing Systems, 2017.
Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., and Sahoo, Unified Training of Universal Time Series Forecasting Transformers. In International Conference on Machine Learning, 2024.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, In Proceed- ings of the AAAI Conference on Artificial Intelligence, 2021.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X.. Mazeika, M., Dombrowski, A.-.., al. Representation Enginecring: A Top-Down Approach to AI Transparency. arXiv preprint arXiv:2310.01405, 2023.
Zukowska, N., Goswami, M., Wiliúski, M., Potosnak, W., and Dubrawski, A. Towards SongCConeext Time Series Foundation Models. arXiv preprint arXiv:240.113530 2024.

---

Exploring Representations and Interventions in Time Series Foundation Models
A. Synthetic Data Generation
Examples synthetic data are provided in Fig.
Generated time series { patterns include:
Constant Pattern (v(() ml + b)
-23
200
Value
Value
Value
24
100
-400
0
200
400
200
400
200
400
Time
Time
Time
(i) No trend
iii) Trend (vary t)
Sinusoidal Pattern (v(() sin(
+ ml b)
50
400
Value
Value
Value
-10
200
-12
0
200
400
200
400
200
400
Time
Time
Time
500
50
Value
250
Value
Value
0
-200
-250
-400
0
200
400
0
200
400
0
200
400
Time
Time
Time
(iii) Amplitude (vary a)
(iv) Periodicity (vary f)
(v) Trend (vary m)
Figure 9: Samples of synthetic time series used in experiments. We two base patterns (constant and sinusoidal). To generate synthetic datasets, vary the periodicity f. amplitude a, intercept and linear trend
Constant Pattern. The constant pattern captures long-term progression of the time series and is modeled as:
y(t) mt + b,
where is the slope and 3 is the intercept. These parameters are sampled from uniform distributions:
Uslopeemi slopemax):
b ~ U (intercept interceptmax)
Sinusoidal Pattern. The sinusoidal pattern captures patterns of periodic variations and is modeled as:
2nt
v(t) sin
mt +
where A is the amplitude and P is the period, both sampled from uniform distributions:
a ~ Ulmmplitude,,n. mmin. amplitude
f (perioddmin periodmax)
12

---

Exploring Representations and Interventions in Time Series Foundation Models
Parameter Ranges:
Constant Case:
a ~ O), (0,00, m - U(,,0, b~ U(-30,30).
Increasing Slope Case:
a ~ O), f~ (O, 0), m m~, 1), b U(-30,30)
Decreasing Slope Case: O), f ~~ (), m m~ m~ U(-1,-1- 6~ U--30,30).
Sine Constant Case (with seasonality parameters):
U(50,50 50), (128, 128),
U0, 6~ U(-30,33)
Sine Increasing Slope Case (with seasonality parameters): 50),
f U128, 128),
i m- (0.5, 1), b b~ U~330-,,,0)
Sine Decreasing Slope Case (with seasonality parameters): a ~ U(50,50), 50),
f~ U128, 128), m U(-1 -055), b~ U(-30,30).
B. Additional Representation Similarity Metrics
To comprehensively analyze the similarity of learned representations, we considered several metrics commonly used in the literature. While our primary analysis relies on Centered Kernel Alignment (CKA) (Kornblith et al., 2019), we also explored two additional similarity metrics, namely Cosine Similarity and Singular Vector Canonical Correlation Analysis (SVCCA) (Raghu et al., 2017). Our findings consistently demonstrated similar patterns across all these metrics, underscoring the robustness of our results. We provide brief descriptions of Cosine Similarity and SVCCA below.
Cosine Similarity: Cosine similarity measures the cosine of the angle between two vectors, providing simple yet effective way to assess similarity. In case, work with activation matrices for multiple samples and compute the average cosine similarity. Given two matrices X and Y, representing the activations from two layers, the cosine similarity is computed as:
(X, cosine similarity (X, Y)
x; y: y
TI
where and y; are the i-th columns of matrices X and Y, and n is the number of samples.
Singular Vector Canonical Correlation Analysis (SVCCA) (Raghu et al., 2017): SVCCA is a method that compares the similarity of representations by aligning subspaces spanned by the top singular vectors. effectively reduces the dimensionality and then compares the correlations of the principal components. The SVCCA similarity between two activation matrices X and Y is computed as follows:
SVCCA(X, Y) CCA(UR; V&)
(7)
where UR and Vy are the top l singular vectors obtained from the singular value decomposition (SVD) of X and Y. respectively, and CCA denotes the canonical correlation analysis.
C. Finding and Pruning Redundant Blocks in TSFMs
The term "block" refers to groups of consecutive layers within transformer that exhibit high representational similarity. Consistent with prior work (Phang al., 2021), we use "layer" refer to individual transformer encoder or decoder blocks consistent with prior work, while "block" refers to higher-level structure made up of multiple such layers that share similar representations (Nguyen et al., 2021).
As an example, consider Figure which illustrates the pairwise similarity between the 24 layers of COMENTTLaarge In this figure, lighter colors (yellow) represent higher representational similarity, as measured by Centered Kernel Alignment (CKA) (Kornblith et al., 2019). The identified blocks are outlined with red bounding boxes. For example, Block I comprises layers 1-5, Block comprises layers 9-18, and Block comprises layers 19-23.
13

---

Exploring Representations and Interventions in Time Series Foundation Models
Our pruning algorithm is summarized in Algorithm 1. Our pruning method involves retaining the first and last layers of each block while zeroing the weights of the intermediate layers. Thhis approach preserves the structural integrity of the block while leveraging the skip connections within transformer blocks to ensure that signals and gradients continue to flow through the network. In the case of LOMENT-Large 's Block 3, composed of layers 19-23, this means that layers 19 and 23 are retained, while the weights of layers 20, 21, and 22 are skipped. We have added a dedicated section on pruning in Appendix C to further clarify this process.
Algorithm 2 Block Identification and Filtering
Require: CKA similarity matrix S, similarity threshold T, minimum block size k Initialize blocks + 0
Initialize current_ block f- 0 Phase 1: Initial block identification for each encoder block 1 do if' Vl, E current_block CKA(4, ()) 2 r then current block - current block U {4} else
if Jcurrent. block) 2 k then blocks + blocks U courrent.block.
end if Burrent_block <- {4:
end if end for
Phase 2: Filter small blocks
Eltered_blocks + {b E blocks: : |6] k}
Phase 3: Verify block-wide self-similarity Initialize final blocks < 0 for each block & filtered.
do Let start, end be the indices of first and last layer in b submatrix S[start: end, start : end]
min mainsiimlarity +- dainnsummttrix if min_simiii similarity 2 then
final + finck_ U {6}
end if
end for return final blocks
C.1. A Simple Algorithmic Approach to Identify Redundant Blocks
We identify redundant blocks in a TSFM through visual inspection, which aligns with prior work (Nguyen et al., 2021). Table 2 lists all the identified blocks in MOMENT Large and Chronos-Large. In addition to visual inspection, redundant blocks can also identified using algorithmic approaches.
Below we propose a simple algorithm to identify redundant blocks in TSFMs. First, we can define block as:
Block = {4....,4}
where i and 4 adjacent transformer encoder blocks
Then, we can systematically identify these blocks using an algorithm that:
1. Identifies initial candidate blocks by grouping adjacent layers with pairwise CKA similarity above a threshold
2. Filters out blocks that are too small be considered meaningful
3. Verifies that each block exhibits high similarity across all its constituent layers by examining the complete submatrix of similarities
14

---

Exploring Representations and Interventions in Time Series Foundation Models
C.2. Identified Blocks in MOMENT & Chronos
In this paper, identified redundant blocks visually. Below we specify, the redundant blocks for -OMEeT-Large and Chronos-Larooe
Models
Block
Block
Block 3
Block 4
MOMENT-Large
1-5 5
9- 18
19 -23
N/A
Chronos- Large
1 1-4
5-9
10 113
-
Table 2: Visually identified blocks of redundant layers in MOMENT-Large and Chronos- rge.
D. On Steering and the parameter
Below we provide some guidance on selecting good values of X and insights into its properties:
Selection and Impact
Optimal Range: Based our empirical experiments, found that the steering strength parameter À is most effective for interventions when its value lies within the interval 2.0).
Lower Bound Considerations: Values of < 0.1 often result insufficient perturbation of activation patterns, leading suboptimal intervention effects that may not manifest visibly in the model's 's output.
Upper Bound Effects: Setting À > 2.0 induces excessive perturbations that push activations beyond their typical distribution bounds, potentially resulting in degenerate or semantically meaningless outputs. In the PCA/latent space visualizations, these cases simply appear as more distant points along the steering direction.
Directional Properties
Reversibility:
Multiplying the steering vector by effectively reverses the direction of intervention, enabling bidirectional control (e.g., transforming concept A B into A).
Example application: For steering vector trained to increase signal magnitude, applying its negative counterpart (-XS) produces controlled signal decrease, demonstrating the symmetric nature of the stecring operation.
Practical Guidelines
Initial Calibration: We recommend starting with À = 1.0 and adjusting based on the observed intervention strength. In most cases value = 1.0 works well and does not need tuning.
Task and Model-Specific Tuning: If X 1.0 does yield satisfactory results, the optimal value requires tuning based on both the specific steering objective and target model, necessitating empirical calibration to achieve the desired intervention strength.
Monitoring: When applying steering interventions, practitioners should monitor both the immediate output and latent space representations ensure meaningful transformations while maintaining output coherence.
E. Model Descriptions and Specifications
E.1. MOMENT
The MOMENT model family is designed for general-purpose time-series analysis, utilizing an encoder-only Transformer architecture. It processes input time series by dividing them into fixed-length sub-sequences (patches) and encoding each patch into a D-dimensional space. The pre- pre-training task involves reconstructing masked patches to learn robust representations that generalize across various tasks.
15

---

Exploring Representations and Interventions in Time Series Foundation Models
rcchteecuue:: Encoder-only Transformer, with patch embeddings and masking for reconstruction.
Input Representation:
Time series split into fixed-length patches, embedded into D-dimensional vectors.
Model Variants: MOMENT-Large
E.2. Chronos
Chronos models utilize sequeenceto--sqquenee (encoder-decoder) Transformer architecture based on the T5 model family, Time series are scaled and quantized into tokens, which are processed by the encoder. The decoder autoregressively predicts future time steps, generating tokens that are mapped back into numerical values.
Architecture: Encoder-dccoder Transformer based on T5, with a reduced vocabulary size of 4096 tokens.
Input Representation: Time series quantized into discrete tokens for sequence modeling.
Model Variants: Chronos is available in the following configurations:
chronos-t5-tinny 8M parameters
chronos-t5-miini: 20M parameters
chronos-t5-smalll 46M parameters
200M parameters
chronos-t55-argee 710M parameters
E.3. Moirai
Moirai time series foundation model built with an encoder-only Transformer architecture designed for universal forecasting. The model handles varying temporal resolutions with multiple patch size projection layers and uses any-variad attention for multivariate time series. A mixture distribution employed to model probabilistic forecasts.
Architecture: Encoder-only Transformer with any-variate attention for multivariate time series forecasting.
Input Representation: Time series processed using multi-patch size projections to handle different frequencies.
Model Variants: Moirai is available in three configurations:
Moi rai -maa 1 1: 14M parameters
-Moirai-basee 91M parameters
- MMoirai-iie maai-laage: 31 LM parameters
16

---

Exploring Representations and Interventions in Time Series Foundation Models
F. Additional Steering Results
5.0
Perturbed Pertubbudd
30
2.5
0.0
-2.5
15
Perturbed
Non Perturbed
10
100
200
300
400
500
10
20
30
40
50
60
Timestop
Timestop
(i) Periodicity, 8 = 0
Perturbed
Non- Perturbed
40
-4
Parturbed Noz-Perturbed
30
66
100
200
400
500
10
30
40
60
Timestep
Timestep
(ii) Some trend, more periodicity, / 0.25
Perturbed
60
Pertarbed
2
Non- Perturbed
Non. Porturbed
10
100
200
300
400
500
10
30 Timestop
60
Timestep
(iii) Trend and periodicity, B 0.5
Perturbed
50
Perturbed
Non Per-tertrrbdd
Non Perturbed
2
-4
30
100
200
300
400
500
20
10
20
30
10
50
60
Timestop
Timestop
(iv) More trend, some periodicity, B = 0.75
Perturbed
80
Perturbed
2
Non-Perturbed
Non-Perturbed
40
Timestep
Timesteg
(v) Trend, 1.0
Figure 10: Visualization of the MOMENT model's reconstruction (left) and the Chronos model's forecasting (right) predictions, comparing concept steering in the latent space (blue) with the unsteered baseline (orange). The steering parameter I can be used to interpolate between sinusodial concepts to increasing trend concepts by varying 3 E (0.0,1.0)
17

---

Exploring Representations and Interventions in Time Series Foundation Models
Class 0
Prteopal Competuntt
Procapa Compretet
20
Class Class
Class 0
-40
Class
-00
-10
-20
40
-00
-30
-10
-00
20
40
(i) Constant to sinusoidal
(ii) Constant increasing
(iii) Constant to decreasing
Figure 1I: Visualization of steering effects in latent space reduced using PCA analysis. Here, we considered steering constant to sinusoidal, constant to increasing, and constant to decreasing series. As shown, the steering behaves as expected in the embedding space, moving selected constant samples, into the neighborhood of sinuooial/moceessn//berraaaggg samples.
1.0
0.5
0.5
0.0
0.5
Perturbed
0.5
Perturbed
-.0
Non-Perturbed
Non-Perturbed
100
200 Timestog
300
400
500
100
200
300
400
500
Timestap
(i) Mean + Single-token
(ii) Median+ Single-token
5.0
Parturbed Non-Perturbed
Non: Perturbed
2.5
2.5
0.0
0.0
-2.5
-2.5
-5.0
-5.0
100
200
300
100
500
100
200
300
400
500
Timestop
Timestop
(iii) Mean Multi-token
Median + Multi-token (iv)
Figure 12: Visualization of different intervention and steering matrix derivation techniques. Applying concept steering interventions across all tokens is necessary to achieve the intended steered concept output (iii, iv) compared to applying concept steering interventions to a single token (i. iii). The method of obtaining the steering matrix, either by computing the mean or median across embedding concept classes, has no notable effect on the steered output.
Application of Concept Steering on a Real-Worla Dataset We demonstrate the practical utility of concept steering on the ECG5000 dataset, which contains clectrocardiogram readings classified as cither normal or abnormal heart patterns. Using MOMENT with an SVM classifier, we achieve strong baseline performance distinguishing between the two classes.
For the steering experiment, we compute steering matrices using the median method capture the concept difference between normal and abnormal patterns. Analysis in the activation space reveals that our steering approach successfully moves samples toward the target concept (Figure 13). This bidirectional movement is evident the PCA visualization of activation patterns.
To validate our approach quantitatively, we applied the steering transformations to 30 samples. While these samples were initially classified correctly with 100% accuracy as normal heartbeats, after steering, all of these samples swapped output classes, confirming successful concept transfer. These results suggest that concept steering can effectively capture and manipulate clinically relevant patterns in physiological data.
18

---

Exploring Representations and Interventions in Time Series Foundation Models
20
Class
20
Class
Principal Component 2
Principal Component 2
10
-10
-20
10
-30
Principal Component
Principal Component
(a) Normal to Abnormal
(b) Abnormal to Normal
Figure 13: Visualization of the impact of concept steering on the activation space of the ECG data model. The figure shows PCA projections of the activation space, where arrows indicate the direction of steering. Blue and red points represent normal (class 0) and abnormal (class samples respectively, with green points showing the original samples and orange points showing their steered versions.
1.0
1.0
1.0
Patch Powtiin
0.
0.8
0.8
5.
o.
Smaled LoX
0.6
3.6
0.6
0.6
Staled
0.4
0.4
0.4
Nitcch
0.2
0.2
022
0.0
0.0
0.0
60
60
12
16
20
12
16
20
12
15
20
Model Depth
Model Degth
Modal Depth
Model
(i) Pattern
(ii) Trend
(iii) Periodicity
(iv) Amplitude
Figure 14: Linear separability of concepts at the patch level (y-axis) across model layers (x-axis). Yellow indicates higher separability than blue in the colorbar. Linear separability is measured using the Linear Discriminant Ratio (LDR), derived from model embedding statistics for each predicted class: (i) constant vs. sinusoidal patterns, (ii) constant vs. increasing trend patterns, (iii) sinusoidal vs. increasing trend patterns and (iv) sinusoidal vs. decreasing trend patterns. Color gradient indicates separability, with dark blue representing low LDR and yellow indicating high LDR. Certain concepts captured by MOMENT- Lag-- exhibit linear separability, but this separability is not uniform across layers; instead, it emerges at specific points in the model.
19

---

Exploring Representations and Interventions in Time Series Foundation Models
G. Additional Pruning Results
Moirai Family
23
11
21
-5
19
9
15
7
13
11
5
3
1
5
7
11 1 5 9 1113151112222 (iii) Large
(i) Small
(ii) Base
Figure 15: Self-similarity heatmaps comparing layer-to-layer CKA similarity matrices across different-sized models in the Moirai family. The Small and Base models exhibit similar patterns, with a distinct dissimilar layer at the end. The Base model has one additional dissimilar layer just before the final one. In contrast, the Large model presents notably different structure, showing two-block pattern where the layers are divided into two distinct groups, with the first block comprising roughly one-third of the model, and the second block covering the remaining two-thirds. This indicates hierarchical representation in the larger model.
pronounced
Chronos Fami ly
23
21
5
11
9
(i) Tiny
(ii) Mini
(iii) Sma 11
(iv) Base
(v) Large rge
Figure 16: How does model size influence the patterns of learned representations? Smaller models exhibit more discrete and distinct block structures, while larger models (Base and Large) display increasingly intricate and clustered patterns, reflecting more nuanced and gradual transformations across layers. Notably, the emergence of blocks-like patterns the Large model appears unpredictable from patterns observed in smaller models.
20

---

Exploring Representations and Interventions in Time Series Foundation Models
Model name
Vanilla
Block 1
Block
Block
All
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
ETTh1
0.395
0.371
0.448
0.449
0.521
0.614
0.420
0.424
0,548
0.673
T22
0.243
0.132
0.268
0.153
0.290
0.176
0.243
0.133
0.296
0.185
ETTml
0.287
0.204
0.321
0.233
0.358
0.307
0.287
0.221
0.345
0.284
ETTm2
0.185
0.080
0.198
0.088
0.218
0.107
0.178
0.076
0.220
0.112
Electricity
0.372
0.250
0.446
0.342
0.716
0.757
0.428
0.327
0.727
0.819
Exchange rate
0,125
0.034
0.124
0.032
0.170
0.061
0.111
0.027
0.174
0.067
Illness
0.393
0.421
0.448
0.502
0.547
0.669
0.423
0.446
0.552
0.648
Traffic
0.492
0.790
0.552
0.906
0.861
1.562
0.606
0.940
0.878
1.633
Weather 0.129
0.079
0.134
0.082
0.170
0.107
0.117
0.074
0.176
0.120
Table 3: Zero-shot imputation performance of MOMENT. Results averaged across four different masking rates: {12.5% 25%, 37.5%, 50%} and five runs with different masking seeds. This table presents the Model Performance Metrics (Mean Absolute Error and Mean Squared Error) on a subset of the Time Series Pile (Goswami et al., 2024; Zhou et al., 2021). The model names include: "Vanilla" MOMENT Large without any pruning, "Block 1-3" for cases where only one block is pruned, and "All" for all three blocks being pruned. The best results per dataset are bolded. For full results refer to
Dataset
Vanilla
Block 1
Block
Block 3
Block 4
All
MASE
WQL
MASE
WQL
MASE
WQL
MASE
WQL
MASE
WQL
MASE
WQL
E1'Th
0.776
0.076
0.088
0.765
0.078
0.778
0.079
0.899
0.098
1.093
0.100
ETTm
0.716
0.068
0.820
0.076
0.864
0.084
0.721
0.063
0.724
0.074
0.978
0.084
2
dominick
0.820
0.331
0.829
0.335
0.824
0.334
0.809
0.332
0.835
0.337
0.886
0.359
creot
0.627
0.020
0.694
0.022
0.697
0.027
0.624
0.018
0.909
0.024
1.281
0.042
exchange.
2.310
0.012
0.014
2.061
0.013
2.256
0.014
0.012
1.713
0.013
5
m4 quarterly
1.217
0.082
1.255
0.084
1.193
0.081
1.223
0.082
1.213
0.082
1.270
0.085
6
m4aycarly m5
3.559
0.133
3.537
0.132
3.495
0.129
3.507
0.131
3.502
0.129
3,359
0.123
0,943
0,586
0.942
0,586
0,954
0.591
0.939
0.587
0.941
0.622
0.956
0.644
1.427
0.076
1.17
0.077
1.372
0.073
L.445
0.083
L.638
0.084
2.593
0.143
monash.car:
0.903
1.041
0.883
1,046
0.867
0.998
0.892
1.027
0.833
0.960
0,816
0.956
10
mmmash_cif_2011
0.989
0.012
1.058
0.019
0.944
0.013
0.998
0.018
1.096
0.011
1.275
0.017
11
monash_covis covid_deuthh
43.251
0.058
42.444
0,052
44.001
0.078
42.357
0.060
41.915
0.053
12
monash_fred. free___ffdd_ddd
0.517
0.021
0.520
0.019
0.507
0.024
0.553
0.029
0.504
0.016
0.600
0.017
13
monash hospital
0.704
0.056
0.724
0.057
0.690
0.055
0.703
0.055
0.718
0.058
0.726
0.060
14
monash ml monthly
1.075
0.127
1.117
0.132
1.057
0.125
1.107
0.130
1.222
0.141
1.321
0.155
15
monash quarterly
1.728
0,106
1.743
0.102
1.659
0.105
1.727
0.092
1.748
0.105
1.753
0.098
16
monash ml yearly
4.336
0.183
4.275
0.174
4.113
0.174
4,375
0.163
4,400
0,182
4.100
0.170
17
monash_ms _montll
0.855
0.096
0.887
0.100
0,845
0.094
0.864
0.096
0.909
0.102
0.968
0.108
18
monash quarterly
1.183
0.075
1.204
0.075
1.178
0.073
1.199
0.075
1.204
0.074
1.301
0.077
19
monash_m3.ycant
3,034
0.145
2,972
0.146
2.908
0.143
2,980
0.143
3.017
0.144
2.870
0.137
20
monash _nn5.weekkl weekly
0.936
0.090
0.961
0.092
0.924
0.090
0.939
0.090
0.941
0.090
0.985
0.095
21
monash_tourism monthly
1.746
0.099
1.847
0.101
1.613
0.088
1.996
0.103
2.178
0.191
2.970
0.217
22
mumash_tourism quarterly
1.647
0.072
1.729
0.065
1.615
1.666
0.063
2.011
0.072
2.034
0.071
23
honusshtourism.yeer yearly
3.565
0.180
3.683
0,185
3.574
0.173
3.724
0.188
3.596
0.174
3.568
0.175
24
minash_traftic
0.794
0.253
0.808
0.247
0.797
0,253
0.808
0.255
0.916
0.275
1.046
0.290
25
monash. weather
0.819
0.139
0.900
0.153
0.826
0.140
0.845
0.144
1.013
0.173
1.002
0.170
26
0.574
0.157
0.584
0.160
0.599
0.165
0.571
0.154
0.219
0.905
0.255
Table 4: Zero-shot forecasting performance of Chronos-Large. This table presents zero-shot performance evaluated with Mean Absolute Scaled Error (MASE) and Weighted Quantile Loss (WQL). Results are presented for the model without any pruning (Vanilla), when individual blocks pruned Block i, and when all blocks are pruned (All).
21

---

Exploring Representations and Interventions in Time Series Foundation Models
All
Block
Block
Block 3
Vanilla
Dataset
Mask Ratio
Mous
Sid.
Mean
Std.
Mean
Std.
Mean
Sid.
Men
Std.
MAR
MSF
MAF
MSF
MAF
MSF
MMAR
MSF
MAF
MSR
MAR
MSF
MAR
MSF
.MAR
MSF
MAF
MSF
MMA.
MSR
FTTb
0.125
0.541
0.662
0.016
0.069
0.448
0:466
0.023
0.084
0.516
0.608
0.016
0.072
0.426
0.425
0.058
0.119
0.398
0.370
0.034
0:091
0.25
0,548
0.667
0.015
0.349
0.445
0.441
0.019
0.156
0,520
0.605
0:021
0.072
0.418
0.420
0.026
0.073
0.393
0.366
0.025
0.061
0.375
0.550
0.674
0.015
0.005
0.447
0.485
0.011
0.033
0.522
0,612
0.016
0.045
0.418
0.422
0.019
0.056
0.394
0.368
0.014
0.035
0..
0,551
0.687
0.011
0.006
0.452
0.463
0.012
0.137
0.525
0.630
0.011
0.041
0.420
0.429
0.011
0.035
0.397
0.300
0.010
0.027
0,548
0.673
0.014
0.946
0.448
0.449
0.016
0,052
0.521
0.614
0.015
0.056
0.420
0.424
0.024
0.071
0.305
0,371
0.021
0.054
CTT52
0.125
0.250
0.179
0.005
0.016
0.299
0.155
0.008
0.016
0.292
0.177
0.010
0.020
0.240
0,120
0.000
0.015
0.245
0.131
0.007
0.015
0.25
0.258
0.188
0.088
0.015
0.271
0.159
0.011
0.019
0.294
0.182
0.010
0.019
0.345
0.120
0.00
0.015
0:247
0,136
0.12
0.017
0.375
0.250
0.186
0.005
0.009
0.296
0.151
0.011
0.015
0.286
0.172
0.008
0.013
0.245
0115
0.000
0.010
0.242
0,120
0.011
0.014
0.5
0.298
0.187
0.005
0.010
0.296
0.150
0.008
0.011
0.288
0.174
0.006
0.011
0.242
0,132
0.006
0.008
0.242
0.129
0.007
0.010
misi
0.206
0.185
0.006
0.012
0.266
0.155
0.009
0.015
0.200
0.176
0.008
0.015
0.245
0.183
0.008
0.012
0.248
0.150
0.009
0.013
FTTml
0,125
0.544
0.015
0.931
0.336
0:915
0,035
0.284
0.216
0.788
0011
0.025
0.343
0.100
0,219
0.200
0,012
0.026
0.304
0.012
0.029
0.284
0,216
0.008
0.284
0.199
0.013
0.028
0.375
0.345
0.005
0.014
0.319
0.230
0,005
0.013
0.302
0.006
0.016
0.288
0,222
0.044
0.011
0.286
0,190
0.006
0.016
0.347
0.004
0.010
0.322
0:238
0.004
0.009
0.309
0.002
0.006
0.291
0.229
0.002
0.010
0.288
0.207
0.013
0.008
mices
0.345
0.284
0.009
0.024
0.321
0.233
0.009
0.018
0.355
0.307
0.010
0.023
0.287
0.221
0.006
0.018
0.287
0.204
0.005
0.020
El7m2
0.125
0.222
0.113
0.005
0.008
0.200
0.00
0.004
0.000
0.221
0.110
0.005
0.004
0.179
0.078
0.005
0.044
0.188
0,060
0.005
0.004
0.25
0.220
0.006
0.197
0.06
0.005
0.004
0.305
0,005
0.177
0.01
0.004
0.384
0.07%
0L004
0.003
0,315
0.219
0.003
0.000
0.198
0.06
0.055
0.000
0.16
0.306
0.001
0.003
0.176
0.05
0.002
0.02
0.384
0.000
0.003
0.002
0.219
0.001
0.100
0.06
0.003
0.307
0.004
0.178
0.005
0.002
0.385
0,000
0L005
0.220
0.004
0.198
0.088
0.004
0.000
0.216
0.307
0.004
0.176
0.076
0.000
0.185
0,050
0.004
0:003
electrictyy
0.125
0.736
0.838
0.019
0.037
0.419
0.365
0.009
0.011
0.723
0.569
0.013
0.025
0.425
0.321
0.000
0.015
0.375
0,253
0.012
0.015
0.25
0,725
0.816
0.011
0.021
0.445
0.336
10006
0.007
0:15
0.555
0.009
0.016
0.428
0335
0000
0007
0331
0.349
0.005
0:007
I.375
0,722
0.809
0.007
0.012
0.414
0.341
10002
0.000
0312
0.750
0.005
0
0.430
1331
0.00
0.053
0366
0.341
0.094
01004
0,5
0.725
0.114
0.002
0,002
0.447
0.345
00034
0.004
0.714
0.754
0.002
0.004
0.431
0332
0.003
0.003
0.371
0.349
0.002
0,002
0.727
0.899
0.012
0,013
0.446
0.342
0.006
0.000
0,716
0.757
0.009
0.016
0.428
0.507
1.000
0.009
0.372
0,250
0.007
0.008
exchange mike
0.125
0.164
0.090
0.100
0.014
0.129
5.00
0,007
0.006
0.178
0.065
0,010
0,008
0.109
0,012
0.000
0.002
0.111
0007
0,000
0.006
0.25
0,172
0.064
0.010
0,012
0012
0.01
0000
0.000
01677
0.089
0,001
0.002
0.108
0005
0.00
0.001
0.122
0000
0.00
0.005
0,315
0.178
0.000
0.005
0.009
0012
0.02
0.00
0.000
0.167
0.059
0,008
0.004
0.112
0.08
0.099
0.004
0.125
0000
0010
0.005
0.006
0.5
0.150
0.073
0.008
0.014
0.123
0.002
00007
0.003
0.170
0.061
0.009
0.000
0.114
0.029
0.005
0.002
0.325
0.114
0.011
0.006
micss
0.174
0.067
0.010
0.013
0.124
0.082
0.006
0.004
0.170
0.061
0.008
0.0006
0.111
0.027
0.008
0.003
0.125
0.054
0.009
0.005
iilness
0,125
0.122
0.521
0:723
0.157
0.897
0.915
0.680
0.454
0:129
0.448
0.509
0.283
0.284
0-425
0,440
0.080
0.080
0.255
0.375
0.512
0.345
0.039
0.808
0.403
0,56
0.204
0301
0.547
0.054
0.208
0/407
0,383
0.036
0.177
0.364
0.348
0.057
0.183
0,521
0.544
0.199
0.084
0.406
0.374
0001
0.199
0.166
0.538
0.032
0.147
0.407
0.361
0.028
0.119
0.360
0,330
0.040
0.130
muzin
0,552
0.648
0.078
0.279
0.448
0.502
0.088
0.359
0.547
0.669
0.008
0.383
0.425
0.446
0.071
0.33
0.303
0.421
0.089
0.312
traftic
0,125
0,830
1667
0.018
0,052
0.359
0.915
0.025
0.050
0.80
1.572
0,015
0.028
0.609
0.950
0.027
0.068
0.505
0,800
0.025
0.068
0.25
0,887
1660
0.015
0.002
0.550
0.909
0.088
0.054
0.71
1.589
0.016
0.018
0.602
0.906
0.023
0 0.065
0.492
0.793
0.014
0.056
0,375
0.874
0.011
0.548
0.901
0.152
1.555
0,028
0.605
0388
0.15
0.487
0.006
0,809
0.012
0.549
0.596
0.055
0.046
1.534
0.033
0.008
0.945
0.007
0.487
0.005
0.878
0.015
0.552
0.905
0.049
1.562
0.033
0.606
0.492
0.015
0.049
weather
0.125
0,175
0.121
0.044
0.007
0.154
0.82
0.003
0.006
0.171
0.108
0.004
0.005
0.115
0.071
0.005
0.009
0.130
0.008
0.004
0:007
0.25
0.176
0.120
0.002
0.005
0.134
0.611
0.055
0.004
0.169
0.105
0,006
0.007
0,116
0,073
0.002
0.003
0.128
0,077
0.004
0,004
0,375
0.177
0.121
0.001
0.002
0.134
0.033
0.005
0.007
0.168
0.107
0.008
0.006
0.117
0,076
0.000
0.006
0.129
0.000
0.005
0,007
01.
0.176
0.119
0.002
0.003
0.135
0.084
0.004
0.007
0.170
0.307
0.004
0.006
0.118
0.076
0.003
0.006
0.130
0.081
0L0044
01007
0.176
0.120
0.002
0.004
0.134
0.082
0.004
0.006
0.170
0.107
0.005
0.006
0.117
0.074
0.003
0.006
0.129
0,077
0.004
0.006
Table 5: Zero-shot imputation performance of MOMENT. Results averaged across five runs with different masking seeds. This table presents the Model Performance Metrics (Mean Absolute Error and Mean Squared Error) on a subset of the Time Series Pile (Goswami et al., 2024; Zhou et al., 2021). The model names include: "Vanilla" MOMENT MOMENT-Large without any pruning, "Block 1-3" for cases where only one block is pruned, and " "All" for all three blocks being pruned.
22

---

Exploring Representations and Interventions in Time Series Foundation Models
Horizon
Horizon
Dataset
96
192
336
720
96
192
336
720
Vanilla
MAE
All Pruned
MAE
ETTh1
0.409
0.426
0.437
0.464
0.409
0.426
0.437
0.470
ETTh2
0.346
0.386
0.409
0.440
0.351
0.389
0.413
0.439
ETTml
0.347
0.373
0.386
0.420
0.346
0.369
0.386
0.422
ETTm2
0.260
0.300
0.337
0.392
0.262
0.304
0,343
0.395
Weather
0.209
0.248
0.285
0.337
0.209
0.248
0.287
0.337
Vanilla
MSE
All Pruned MSE
ETThl
0.385
0.411
0.423
0,443
0.388
0.414
0,424
0.460
ETTh2 0.287
0.350
0.370
0.404
0.296
0.356
0.382
0.404
ETTml 0.290
0.330
0.352
0.409
0.326
0.354
0.414
ETTm2
0.171
0.231
0.287
0.372
0.173
0.236
0.294
0.372
Weather
0.153
0.197
0.246
0.316
0.152
0.198
0.247
0.317
Table Fine-tuned forecasting performance of MOMENT. This table presents model performance metrics (Mean Absolute Error and Mean Squared Error) subset of the Time Series Pile (Goswami et al., 2024; Zhou al., 2021). Metries presented for KOMENT La rge without any pruning (Vanilla) and for all blocks pruned (All). Results gathered from the best performance on the test set across epochs of training with batch size 64 and learning rate of 0.0001.
Encoder Sparsity (%)
Model Sparsity
Estimated Model Size (MB)
Average Time (ms)
Standard Deviation (ms)
Method
Pruned Part
(%)
None
0.00
0.00
1301.76
20.88
0.42
Block 1
12.50
11.29
1154.76
19.42
0.27
MOMENT
Block 2
33.33
30.11
909.76
19.71
0.31
Block 3
12.50
11.29
1154.76
19.56
0.29
All Blocks
58.33
52.70
615.76
19.82
0.43
None
0.00
0.00
2704.48
59.95
0.26
Block 1
8.33
3.55
2608.48
58.22
0.58
Block 2
12.50
5.32
2560.48
56.80
0.78
Chronos-T5
Block 3
8.33
3.55
2608.48 .48
56.94
0.27
Block 4
25.00
10.65
2416.48
57.55
0.84
All Blocks
54.17
23.07
2080.48
56.82
0.58
Table Inference performance under various pruning configurations. This table presents the inference performance Large and Chronos-Large models with different pruning configurations. Results presented
metrics of the for MOMENT - Large and Chronos without any pruning (None), when individual blocks pruned Block i, and when all blocks are pruned (All Blocks). Inference time estimation was performed by aggregating times from 100 passes of one-batch, one-channel sample of length 512.
Statistic Category
Metric
Mean Difference
Standard Deviation
Original Better
Pruned Better
Equal
Pearson Correlation Coeflicient
Classification
Accuracy
0.0128
0.0489
9
0.945
Anomaty Detection
Adjusted Best F, VUS-ROC
0.2117
0.3026
6
0.634
001707
0.0995
3
0.512
Table 8: Summary of differences between MOMENT pruned (all blocks pruned) and MOMENT. Detailed results are provided in Tables and 11. For classification, observed slight deterioration in performance, whereas for anomaly detection the deterioration was more severe.
23

---

Exploring Representations and Interventions in Time Series Foundation Models
Classification Results
1.0
0.8
MOMENT;
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
MOMENT, Pruned
(a) Accuracy
Anomaly Detection Results
1.0
0.634
1.0
MOMENT,, VUSROC
MOMENT,
0.6
0.6
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
MOMENT; Pruned
MOMENT.- VUSROC
(a) Adjusted Best F, Score
(b) VUS-ROC Comparison
Figure 17: Combined visualization of classification accuracy and anomaly detection metrics (F1 Score and VUS- ROC) comparing pruned and non-pruned models. Classification results are based on 91 UCR datasets (MOMENT, and MOMENTo o.HHENTooruuna), where the mean accuracy 79.4% and 78. 1 1 respectively (see Table 10). Anomaly detection uses a subset 44 datasets from the UCR Anomaly Archive (MOMENT Le and MOMENT_e prunea). See Tables 11 and for detailed results.
24

---

Exploring Representations and Interventions in Time Series Foundation Models
Red
VIESRO
IGHE
Indbl
0.500
0.300
2200
0230
0.510
(30
0011
0:130
1330
07318
03300
1033
3880
0.900
03500
0.510
0.190
0.85
039
0996
1380
0.50
0:101
889
0665
0190
0.773
07300
0330
07333
IC11
1.990
135
1030
173
0350
358
0300
0581
13300
8.901
33733
1050
1350
13330
08500
3500
13500
1351
0.500
10710
0350
1.330
0.510
3590
1850
1550
0.570
03530
0331
3380
3.500
0.500
0,500
0370
0.500
0.36
3.00
0.130
0.590
0.90
01115
3830
17700
8.000
0966
0530
0.85
Pevetterent
3.931
3.480
0.50
0055
3880
17720
03300
0330
3.500
1990
0.000
3350
23510
0.50
0000
5.510
2.390
0.30
0.20
2330
0.300
0.96
3391
0330
OUTO
2840
5370
0.500
6.110
3.300
035
0100
530
1086
3390
19955
8.30
0000
399
1331
139
1330
3390
0.30
0770
0131
patk3e
3.390
0.9
0.36
3330
0770)
0.90
0380
3381
3.50
0.40
2340
05330
0.30
0.86
copprnee:
0.30
3350
0350
0.30
0366
334)
8000
1.300
03300
0085
0330
0.30
3340
0.50
3.090
0.00
3880
1.800
07501
096
0310
2.831
0.39
0.90
3310
0370
0.90
0770
8331
vcalval
23337
0.30
0010
3520
8330
0.50
0.80
3330
Table 9: Anomaly detection using Adjusted Best F; and VUS-ROC for a subset of 44 datasets sampled from the UCR Anomaly archive, MOMENT;; and pruned (all blocks) MPMENT
MOMENT Opruned
MOMENT
Mean
0.781
0.794
Std
0.146
0.148
Min
0.300
0.369
25%
0.697
0.714
50%
0.776
0.815
75%
0.915
0.916
Max
1.000
1.000
Table 10: Statistic: Classification accuracy of methods across 91 UCR datasets. MCMENT, without fine-tuning and pruned (all blocks) COMENTo.prunees See full table:11
25

---

Exploring Representations and Interventions in Time Series Foundation Models
Dalmat
MONEEB/ gueend
TS2Vee
T-Lons
TNC
TS-TCC
TST
DTW
CNN
Entrddr
FCN
MCCNNN
MLP
ReaNet
I-LoNe
TWIESN
GestueMilAAi122
0.531
0.608
0.400
0.546
0.362
0.254
0.158
0.606
0.518
0.480
0.651
0.500
0.545
0.668
0.058
0.575
0.12
0.621
0.705
0.765
0.781
0.733
0.569
0.728
0.721
0.771
0.754
0.726
0.768
0.781
0.127
0.605
Gotunceb6h.22
0.671
0.16
0.613
0.899
0.316
0440
0.380
0.61
0.778
0.796
0.781
0.720
0.701
0.777
0.154
0.843
BCG5000
0.940
0.942
0.955
0.933
0.937
0.941
0.228
0.924
0.928
0.941
0.940
0.533
0.930
0.535
0.554
0.922
osill
0.000
0.785
0851
0.260
0.721
0.733
0.53
0.891
0.432
0.554
0.929
0410
0.500
0.580
0.182
0.66
Modicallnnges
0.58
0.66
0.769
0.750
0.784
0.747
0.63
0.53
0.611
0.654
0.778
0623
0.719
0.770
0.514
0644
Hum
0.68
0.581
0.74
0.224
0752
0.743
0.534
0.467
0.730
0.622
0.707
0215
0.699
0.75K
0.514
0.766
0.40
0,612
1669
0.568
0.500
0.671
0.644
0.695
0
0.610
0.663
0.285
0.501
0835
0.56
0.87
0850
1866
0.573
0
0.770
0.754
0880
0.764
0.977
0.866
0.730
0.920
0.684
0.817
0.986
0.882
0.956
0550
1291
0.922
0.899
0957
0.700
0:997
0.906
0.995
0.500
0.946
TeoLedEEGG
0.793
0.847
0.960
1000
0659
129
0.266
0.311
0.905
0877
0.754
0592
0200
0.733
1.000
0.500
0.49
GunPolntMaae Verus
0.991
0.991
1000
0997
0294
0.927
1,000
0.97
0577
0.93
0597
0551
0.900
0.992
0.525
0.988
Trace
1,000
1,000
1000
1550
1800
1.000
1,000
1000
0852
1852
0.200
0000
080
0.306
1.000
0.250
0.34
Smopet
08877
0.20
0999
1950
0813
0.55
0327
05/0
0.90
0561
0.90
0.580
0.583
ModdeeMaalaXIWW
0.565
0572
0336
1291
0270
5000
0205
0.000
0231
8:301
0200
0336
0.895
0.286
0.569
Cynthei:Connal
0.550
0.990
0.
1987
10000
0.220
0
0.203
1967
0.223
0969
0153
0.973
0.997
0.167
0.79
ShapesAll
0.807
0815
0248
0228
0.33
0.688
0617
0.22
0224
0.299
0.776
0.926
0.217
0643
AllGestureWlinee
0,599
0.607
0,77
0253
0200
9997
0.76
1110
0713
0,261
0.77
0.741
0.100
Water
0.993
0.977
0290
0.50
0.60
0981
0.907
01992
0.986
0.598
0.892
0916
Faceloor
0.300
0852
1000
0.777
0.111
0350
0.80
0.500
0:21
0.836
0.585
0.205
0357
CricketX
0.221
0.349
0.300
0:331
0.54
0515
0.644
0734
02213
0.591
0.700
0.044
DatilifataneCrniiinCn Comeet
0275
0217
0.250
0.227
0.770
ChlerinCimmnttzzinn
0.771
0.765
0000
0.75
0.608
0.581
0.533
0.554
Chinatoon
01962
0,965
0965
0.056
0.557
0.977
0.966
0.000
0.945
0.572
0.976
0.726
GestacMilAiii
06654
0.546
0,569
0.569
0.534
0.695
0.515
0.575
0.606
0.088
0.549
UMD
0.494
0.951
0,955
0.461
1.000
0.63
10000000
0009
0256
0000
0991
0.630
0.945
0.617
0.910
0.91
0.500
0.900
0.534
0.771
0.577
0.535
0.964
0.558
0342
0.900
0.522
0.590
0.545
0.333
0.571
0.576
0336
Cre
0755
0.714
0.766
0220
0.538
0.742
0.710
0.655
0.670
0.750
0.734
0487
0.18
0.743
0.042
0.480
GooundhabicZ
0,866
0.840
0.900
000000000000
0358
0.995
0.500
0.701
0.844
0.21
0.500
0.760
0.792
0.501
0.161
0.500
0.668
0.668
0.77
0661
0.50 1
0.531
0.422
0.60
0.568
0.557
0.56]
0.400
0.599
0.617
0.219
0.566
ArowHeaa
0.751
0.743
0.657
0766
0200
0201
0.737
0.771
0.703
0.717
0.630
0.343
0.77
0.784
0.836
0.303
0689
Wine
0.667
0,537
0.870
0315
003150000
0.55
0.77%
0.500
0.574
0.519
0.556
0.611
0.500
0.541
0.722
0.500
0.741
Coffee
0.929
0.853
1000000000
1800
11085
1.000
0.821
1.000
1.000
0.355
1,000
0.579
0.993
1.000
0.507
0.979
Barthqulee
0.748
0.748
0.748
0744
0314
0.748
0.745
0.719
0.709
0.740
0.725
0.345
0.727
0.712
0.743
0.745
Herring
0.59
0,594
0.641
11554
03591
0.5%
0.99
0.5M4
0.531
0.331
0.512
0.664
03572
0.391
0.000
0.594
0625
Beef
0.733
0.35
0.751
1667
03733
0.600
0.500
0.633
0.267
0.207
0.000
0.500
0.313
0.753
0.200
0.227
Midsteehallanennniinncc onreet
0.444
0.667
0836
1881
0.53
0.68
0.24
0.752
0:99
0.996
0.755
0.36
0.570
07441
0.740
0.804
1
1.000
1800
0.578
0.763
0.764
0874
0.842
0.985
0.000
0.973
0.966
0.497
0.725
Yoga
0.22
0.34
0
1
0.550
0.837
0.766
0.753
0.457
0.741
0.35
0.567
0.556
0/26
MoteStrain
Adac
0.32
0.629
0.774
0.688
0.89
0.700
0.702
1851
0825
07725
7
0.765
0.550
0335
0.604
0.85
0.393
0.318
0372
0.34]
0336
0.60
0691
0.555
0339
0,924
0.833
0.599
0.023
0
C
Strawberry
0966
0.911
0958
0891
0.965
0.16
0961
0.52
0.99
0.73
0558
0.999
0.580
0.633
0911
InsecWiingeerSeadd
0.60)
0.89
06.00
0.286
0355
0.85
06000
0.192
0287
0000
0.499
0.091
0.435
Dodgerd oepWiethend
0826
0.36
MAN
MaN
0753
0.69
0.74
0953
0 1:04
087
092
0.953
0.39
0994
Mer
0.33
0.917
2882
9883
0.000
0931
0913
0757
0 1.301
02287
0.991
0.590
0.351
0.99
0886
0.766
0.90
0999
0.441
0.791
0813
0384
0912
0.840
036
0:509
0.000
0.70
FusAlll
0.70
0.911
0.71
0226
0226
9813
0.504
0.006
0.774
0.244
0098
0.220
0.794
0.367
0.080
0673
FR
0.835
0811
0.924
0884
07700
9.63
0.543
0.905
0373
0367
0.43
0.775
0:511
0:954
0.343
0641
AllGstteWWimeee
0.584
0,666
0.29
0120
0200
0.241
0.229
0.000
0.64
0,400
0.511
0.794
0.100
StaketlestteWiinnee
0.900
0000
0,00
0.900
0350
0.500
0.75
0.516
0.368
0364
RME
01940
0914
0.97
0.900
0.381
Foalk
07767
0.788
0.764
0.731
0.740
0.777
0.004
0.513
0.763
0,800
0.92
0361
0.857
0.720
0.83
0.855
0.754
0.061
0.520
0.348
0.581
0.126
0.77
0.827
0.829
0350
0.354
0.907
0.745
0.831
0.31
0.44
0.000
0.504
0.975
0.617
0,635
Fulyy Wahy
0.776
0,802
0.771
0.35
0.53
0.655
0.525
0.600
0.624
0.558
0.646
0811
0.706
0.740
0.125
0.516
0.912
0.925
0.917
1017
0.56
0361
0.50
0.307
0.752
0.598
0.706
0.61
0.559
0.589
0.957
0.526
0.892
FreeseSSallTriin
TwoPanems
ShapeletSim
0.933
0.865
0.969
0
0,951
0,902
1000
0.870
10000000000000000010000000000000
0025
1669
0
1
0.689
0.20
0,466
0.753
0.640
1.000
0.397
0.611
0.530
0.50
0.676
1.000
0.483
0.305
0.370
0.46s
09766
0.688
0.903
0.513
0.686
0,832
1,000
0.500
0.500
0.259
0.917
0.546
0,875
Plane
0.990
0.960
1.00
0000
1
11005
1.000
0.55
1.000
0.62
0.964
1.000
0188
0.977
1200
0.343
1.000
GestuMMidAiD3
0.300
0,369
0.200
0.292
0.355
0226 0292
0.137
0.154
0.33
0337
0.368
0.136
11278
0.382
0.340
0.038
0.275
0.859
0.879
0.964
03584
0395
0
0561
0.967
03554
0.350
0.366
11865
0.909
0.301
0.301
0.914
Cricketz
0.713
0,731
0.792
0.792
02206
0358
0.715
0.003
0.754
03011
0.61
0,310
11484
0.639
0.30
0.062
0.643
Lightaing?
0.712
0.726
03355
0307
0655
0:11
0.726
0344
0.696
0,325
13559
0.516
03377
0.260
0:005
U UWaveCeenneL.eo
0.238
0.738
0.79
0210
1159
0.41
1
0.38
062
1926
0.696
0.622
01699
0.600
0.666
0.121
0.497
Guntsimta geSpan
0.999
0,962
0967
10994
0391
6963
0012
0.800
B5995
0387
0.954
0.997
0.494
0.965
0655
0.669
0.727
0277
1
03355
0:41
6.200
0758
1751
0.11
0322
0.507
0.718
0.433
0:705
Swedishl and
0914
0.93
1941
0514
1880
1
0758
0.200
0120
1056
01007
[181
03365
0.565
0.064
0.337
CBF
0972
0.960
1000
0053
1
0.398
0.000
0.007
1555
0.077
0388
0.596
83522
0.596
Boaliifly
0.60
0.000
0000
0800
0
0350
0.801
1000
0.700
1000
0.200
8010
11530
0:500
0.550
0
0.70
0.790
AllGGestne Wiimmel
0.50
0537
0723
9/467
0663
05710
0.39
0228
0.00
0.225
0
0
03166
Dedget Lopplay
0.73
0:638
0200
12200
0.00
0512
0
0. 0.18
1100
0.300
01000
Sanlbinntikeeeee Young
0943
0.991
10:0
10:0
1600
1000
1000
0.38
1922
0222
082
2:889
0.524
0.925
F
0906
0.91
0990
8928
0800
0. 11566
3.5
0880
0.225
0014
0861
0:957
0.510
0.555
0.73
0.41
0.63
0.91
6054
0840
0220
0854
0965
0354
0345
0.500
0.005
0812
1054
0.964
0:25
0.899
0966
0.951
0.349
0.547
0.062
0.99
04883
087)
0.30
Gunl-int
0.927
0.927
0.960
0.000
0220
0227
0.907
0.48
0.284
1,000
0.007
0.928
0:591
0.493
0989
0.732
0712
0.834
0.71
0,100
0.500
0.80
0.66
0.77
0.791
0.76
0.775
0.767
0.773
0.341
0784
FidkypGssueeWlimeee
0.600
0,620
0.800
0.740
0,20
9.600
0.40
0.000
0.608
0.206
0.74
0,112
0.004
0.704
0.100
0616
Sony SonjAtBORokosSurtaa
0.739
0.729
0.902
0,804
9.899
0.724
0.725
0.660
0.729
0.958
0.655
0.892
0.561
0.29
PowerCons
0.933
0.994
0,033
0.911
0378
0.960
0.91
0.929
0.77
0.379
0.500
0.89
BirdClicaan
0.750
0.850
0.850
0.550
0.650
0.750
0.710
0.510
0.540
0.540
0.740
0.580
0.500
0.60
0.915
0.915
0.892
0.900
0.351
0.877
0.615
0.838
0.752
0.702
0.589
0.649
0.745
0.594
0.315
0.794
CiikeY
0.715
0.746
0.749
0.597
0.718
0.467
0.744
0.582
0.639
0,703
0.521
0.595
0,510
0.085
0,552
DecriDeriies
0.659
0.646
0.721
0.207
0.500
0.686
0.676
0.602
0.656
0.702
0.706
0653
0.383
0.728
0.2-2
0,005
DodgetLoopliame
0.710
0.625
0.841
NaN
NNN
NGN
0.646
0.77
0.866
0.10
0.768
0377
0.355
0,710
0.478
0.716
Fungi
01828
0.888
0.957
1.:00
0.27
0.55
0.366
0.39
0.961
0954
0011
0.051
0.63
0,177
0.063
0.439
UWieDootree 2inre/ annry?
Symbols
0.928
01349
0.916
0.765
0.96
0.70
0.257
0.63
0.721
0.585
0916
0.680
0.655
0.785
0.658
0.950
0.608
0.610
0.554
0.684
09955
0.727
0.64
0.645
0.697
0.3
0,303
0.740
0.121
0.134
0.78
0.571
FC0200
0.800
0.760
0.200
0.46
0.830
0.880
0.350
0.770
0.866
0.844
0.583
0.835
0.914
0.574
0.640
0.834
Table 11: Classification accuracy of methods across 91 UCR datasets. Results are shown for MOMENT, (without fine- tuning) and its pruned version, MOMENTcpprune (all blocks). We report that MOMENT, achieves 54 wins, 9 ties, and 28 losses compared to MOMENTo
26


Reviews:
[{"review_id": "xqXDPyFjBF", "reviewer": "Reviewer_i5c5", "summary": "This paper investigates the internal workings of time series foundation models by analyzing their learned representations. It reveals that these models exhibit block-like redundancy across layers, which can be exploited through block-wise pruning to reduce model size and improve inference speed without compromising accuracy. The study further identifies human-interpretable concepts—such as trends, periodicity, and seasonality—within the latent space and demonstrates how interventions in this space (concept steering) can guide model predictions toward desired outcomes. Overall, the work provides valuable insights for optimizing and controlling time series forecasting models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer i5c5,\n\nThank you for reviewing our paper. We appreciate your recognition of our work's contributions regarding model redundancy patterns, block-wise pruning, and interpretable latent space concepts.\n\nGiven your \"Weak accept\" recommendation, we'd be grateful if you could let us know how we might improve our paper to strengthen your assessment. Are there specific aspects of our analysis, experimental design, or presentation that we could improve to better convey the significance and impact of our contributions?\n\nThank you again for your time and consideration.\n\nSincerely,\n\nThe Authors", "claims_and_evidence": "yes, they are.", "methods_and_evaluation_criteria": "Yes, they do.", "theoretical_claims": "There are no theoretical claims", "experimental_designs_or_analyses": "Yes, I checked all. There are no obvious issues.", "relation_to_broader_scientific_literature": "It extends representation similarity analysis—originally developed in the computer vision and NLP communities.", "essential_references_not_discussed": "All good.", "other_strengths_and_weaknesses": "All have been discussed.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "40vwW8QPk7", "reviewer": "Reviewer_Lqam", "summary": "The paper performs analyses into 3 time series foundation models, Chronos, Moirai, and MOMENT. Using concepts from the interpretability literature, the paper studies i) representation similarity across layers, ii) identification of human interpretable concepts, and iii) model intervention. Via experiments, the paper shows that these models have significant similarity between layers, which can ultimately be pruned and still retain performance. They show that these models indeed learn human interpretable concepts such as (linear) trend and seasonality, and the models can be manipulated into making predictions with these concepts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Lqam,\n\nThank you so much for your time! We are glad that you found that our paper \"very nicely sets up the experimental design to support their [our] claims, is comprehensive in exploring 3 different models, and presents clear and convincing evidence regarding representational similarity\". We have addressed all your comments and suggestions in the current version of the manuscript.\n\nBelow, we address your concerns and questions. If you have any further comments, we'd be happy to address them. *If you feel that we have addressed your concerns, we respectfully ask that you consider increasing your score.*\n\n**Cherry-picked example from Steering**: We appreciate the reviewer’s suggestion to include more plots to showcase the effect of steering. To demonstrate that the steering effect is not limited to a single time series example, we generated datasets with different random seeds and assessed that the same phenomena of linear separability happens, with successful steering from one group to another that is visible in the visualizations. Also, visualizations that show influence of a specific steering intervention in the output of the model also are repeatable across different samples, datasets and setups.\n\nWe also provided additional steering results with real-world data to showcase its effectiveness for other datasets in Appendix F.1. Using our proposed steering approach in MOMENT with time series from a real-world ECG Arrhythmia classification dataset (ECG5000), we demonstrate that time series classified as normal heartbeats can be steered to time series classified as abnormal heartbeats. This result confirms that concept steering can effectively guide time series into different clinically relevant category classifications, which could help enhance our understanding of time series pattern variations that guide model predictions for medical condition classification. Additionally, such steering results highlight the potential utility of the method for synthetic data generation. To provide further evidence of the effects of steering across datasets in different domains, we are currently running experiments on steering concepts in popular forecasting (ETT-h1), classification (MIT-BIH ECG dataset), and anomaly detection (UCR Anomaly Archive) experiments.\n\n**Unclear Figures 7 and 8**: We agree that figures 7 & 8 can be made more clear. We have made the following edits to improve clarity.  \n\nFigure 7- The purpose of these figures is to show where in the model linear separability peaks (as measured by normalized Fisher’s LDA objective range (0,1)). X axis is the model depth - in terms of layers, Y axis is the position of the patch of time series from which we took embeddings. The more yellow the specific (layer, patch) combination the more drastic the separation was. The red line also showcases the linear separability - the idea here was that we averaged embeddings across the patch dimension, so that we have a single time series embedding at the specific layer. Here, the X axis is also model depth, while Y axis is the degree of linear separability. In addition to updating the figure caption with this information, we have clarified that the heatmap and red line use distinct y-axes, which was not immediately clear in the original figure. To improve readability, we have updated the plot by adding labeled axes: the x-axis is now labeled “Model Depth,” the left y-axis (for the heatmap) is labeled “Patch Position,” and the right y-axis (for the red line) is labeled “Fisher’s LDA.”\n\nFigure 8 - we’ve incorporated the suggestions about improving the captions. To clarify the setup-we provided constant time series as an input to the model (not visualized, information provided in the caption), and visualize model outputs with and without steering applied, which are referred to as ‘Perturbed’ and ‘Non-perturbed’ outputs in the figure legend, respectively. As expected, without steering applied, the model outputs a constant time series signal. With steering applied, we obtain a different intended concept output, reflecting a trend, seasonality, or a combination of both, depending on the Beta parameter. For each model output, we show the raw output (lighter color) and the moving average (darker color), which helps filter out noise that is an artifact of the model. We have updated the figure caption to include this information.\n\n### Questions\n1. **Line 226 – What is n?** The symbol $n$ refers to the number of time series samples in the dataset. Each hidden representation $h_i^{(j)}$ corresponds to the output of the $j$-th layer at $i$-th token for all $n$ samples considered, with each embedding having dimensionality $D$. \n\n2. **On Blockwise Pruning and Table 7:** Thank you for your thoughtful observations. Please refer to section **Inference Time of Pruning** in our rebuttal to Reviewer tr30.", "claims_and_evidence": "The paper is largely an experimental analysis into existing models, and very nicely sets up the experimental design to support their claims. Evidence regarding representational similarity is clear and convincing. However the following 2 points are problematic:\n\n1. \"Block-wise pruning can improve model throughput, without compromising accuracy.\" seems to be an overstatement given that the subsequent evidence provided showed minimal throughput improvement (see questions section), and zero-shot performance is not retained. I recommend the authors to reduce the boldness of the statement.\n\n2. The evidence provided for the \"concepts\" and \"interventions\" parts, specifically figures 7 and 8 are unclear. I am unable to understand what the figures mean. For figure 7, it is unclear what is the relationship between the red line and the heat map are, and it is also unclear what the heat map is trying to convey. For figure 8, it took me some time to understand the differences between steering vs compositional steering - the sentences \"Introduce periodicity (i) and trend (ii, iii) to constant time series\" and \"Introduce trend and periodicity to constant time series\" should make it clearer the difference between steering vs compositional steering. I would further recommend \"(MOMENT (top), Chronos (bottom))\" to be labels at the side of the diagram instead. I do not understand what the lines represent, are they ground truth time series? or model predictions? What is the difference between the dark lines vs transparent lines? Where is the demarcation between Chronos inputs vs forecasts?", "methods_and_evaluation_criteria": "The methods and evaluations make sense for the analyses. The paper clearly lays out research questions that it intends to explore, and presents the tools and methods it uses to investigate the questions. The paper is comprehensive in exploring 3 different models.", "theoretical_claims": "No theoretical claims made.", "experimental_designs_or_analyses": "The experimental design is largely sound, my main concern is with regards to the steering section, which uses a single example which could be cherry picked. It would be better to have multiple of such plots, or some kind of dataset level experiment to show this capability. Figure 10 looks promising, but it is unclear how the dataset in appendix A has been used for this set of experiments. More explanation should be given.", "relation_to_broader_scientific_literature": "The paper raises some interesting findings for time series foundation models, and suggests several avenues for deeper exploration. Firstly, it raises the issue of redundancy amongst the layers and suggests that existing models can be much more parameter efficient. It also shows that these models can be steered based on a set of examples - this can possibly be extended to domain based steering.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "1. Writing structure can be improved for easier reading. The writing structure which lists all methodology first, followed by experimental results does not suit this paper. Instead, this paper should take the approach of having the experimental results right after each method subsection, e.g. Explain representational similarity -> similarity results -> explain pruning -> pruning results -> explain time series concepts -> time series concepts results, and so on. \n\n2. Regarding the pruning algorithm, the key idea should be on how to detect blocks rather than how to prune them. While an automatic algorithm to detect them is presented in the appendix, ultimately the blocks for experiments were identified visually (as stated in Appendix C.2). \n\n3. The biggest weakness of this paper is its clarity. Starting from the definition $h_i^{(j)} \\in \\mathbb{R}^{n \\times D}$, it is unclear what $n$ is. It seems that indices $I, j$ can be dropped, since all variables are indexed by them. \"Additionally, we perform probing on representations averaged along the token dimension for each i-th layer.\" - not sure what this statement means. Formal notation should be used to define $\\mu_s, \\mu_c, \\sigma_s, \\sigma_c$. It's not clear why a \"mean embedding value\" is a scalar. Clarity of experimental results are also an issue which have been mentioned above.", "other_comments_or_suggestions": "### Nits\n\n1. CKA should be defined at the first usage in line 46, right column, or avoid the use of the term CKA in that part of the introduction, just mention \"similarity\" without mentioning the metric used.\n\n2. Line 216, right column - ...  identify which layer l in ... - \"l\" is not really required as it is not referred to again.\n\n3. Line 225, left column - the term \"residual stream\" should receive a brief explanation and citation as it is not standard Transformer parlance, but more of a term used within the mech interp community.\n\n### Suggestions\n\n4. Line 291, left column - Fig. 7 should be fig. 5?\n\n5. Fig 4 - Label x and y axes, especially for tiny vs mini\n\n6. Table 4 - include percentage change\n\n7. Line 304, right column - \"... inference speed compared to unpruned ...\" - compared -> comparable", "questions_for_authors": "1. Line 226, left column - $h_i^{(j)} \\in \\mathbb{R}^{n \\times D}$ - what is $n$?\n\n2. How was pruning actually performed, especially in the experiments? Algorithm 1 says \"zero out the weights\". Was inferencing skipped on those layers or not? How would zero-ing out the weights but not skipping the layers incur a speed up?\n\n3. Table 7 - I don't understand where the improvement in inference speed comes from (related to previous question). If it is the case that the layers were skipped, then I'm confused why the improvement is so small, i.e. only 1 ms for MOMENT, and the relationship between block 1/2/3 vs all blocks doesn't make sense to me.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jl6m8YVgZv", "reviewer": "Reviewer_tr3o", "summary": "The paper delves into learned representations of time series foundation models. The authors evaluate the similarity of representations using CKA, revealing that larger models can learn redundant patterns. They propose a block-wise layer pruning strategy to reduce the feature dimensionality while keeping the performance. The authors also introduce a method to identify specific temporal patterns (e.g., trend and seasonality) from the latent space. They propose a steering method to guide the model to adjust predictions without fine-tuning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer tr3o,\n\nThank you for your thoughtful and detailed review of our paper. We appreciate that you found our \"analysis of representation similarity, pruning effectiveness, and concept steering to be detailed and that the claims made in the paper are generally well-supported.\" \n\nWe have addressed your concerns and questions below, and we'd be happy to address any further questions or comments. *If you feel that we have addressed your concerns, we respectfully ask that you consider increasing your score.*\n\n**TSFM representational redundancy**: This is a great point! We found that both supervised models trained on specific datasets, and pretrained text & vision models exhibit representational similarity. While this is common knowledge for other models, prior to our work, we were unsure if this holds for TSFMs. Just like prior work, we provide no explanation behind this phenomenon, but we agree that redundancy in the training data may cause this phenomenon. More importantly, we leverage this finding to aggressively prune TSFMs, and make their use more pragmatic in real-world tasks. \n\n**Additional experiments on real-world datasets:** Please refer to section **Cherry-picked example from Steering** in Rebuttal to Lqam.\n\n### Questions\n1. **Novelty of Proposed Pruning Method** We assume that you are referring to this paper in the following paper [1]. Let us know if we got the wrong paper. While our approach is inspired by Nguyen et al. (2021), it differs in key ways: (1) The authors remove individual layers, whereas we prune entire self-similar blocks at once, preserving their boundaries to maintain a notion of continuity in the model’s internal representations. (2) We are also one of the first to apply this type simple block-level pruning to large pretrained Time Series Foundation Models. (3) Finally, we validate our approach on real-world tasks beyond classification, such as imputation and forecasting-demonstrating up to 52% inference speedup with minimal performance degradation, even in zero-shot settings. \n\n[1] Nguyen, T., Raghu, M., and Kornblith, S. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. In International Conference on Learning Representations, 2021.\n\n2. **Inference Time of Pruning:** You're absolutely right that the inference time improvements reported in Table 7 were marginal—this was due to a bug in our original implementation. Initially, pruning was done by zeroing out weights, which left computation intact and yielded minimal speedups, without utilizing CUDA kernels for sparse computations. After correcting this and implementing a computational graph-level pruning mechanism that fully skips computation of pruned blocks, we observed substantial speedups, with inference time reduced by up to 52% (e.g., from 23.17 ms to 11.16 ms for MOMENT, and from 31.68 ms to 15.33 ms for Chronos). Compared to prior pruning techniques that focus on sparsifying weights or pruning individual layers, our method is simple, structurally aligned with learned representations. These updated results are included in the paper.\n\n3. **Motivation of Steering:** The motivation behind concept steering is to enable controlled, post-training updates to model embeddings, allowing models to incorporate new concepts or events into predictions without requiring additional training or fine-tuning. This approach offers several practical benefits: (1) It enables users to imbue models with new or missing contextual factors after training, which is particularly valuable when models weren't originally trained on certain scenarios. For example, we can steer vital signs in healthcare based on new treatments or adjust financial forecasts based on emerging events such as positive earnings surprises. Adding these contextual factors, even as simple trends, has significant implications for improving model predictions in zero-shot and out-of-distribution scenarios. (2) Steering also supports synthetic data generation of realistic time series variations. In our experiments with the ECG Arrhythmia classification dataset (ECG5000), we demonstrate that time series classified as normal heartbeats can be steered to produce time series classified as abnormal heartbeats. Such data generation capabilities can be used to augment data for model training or generate new samples for us to better understand the decision boundaries in model predictions.\n\n4. **Risk of cherry-picking:** Please refer to section **Cherry-picked example from Steering** in Rebuttal to Lqam.", "claims_and_evidence": "The claims made in the paper are generally well-supported.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria (CKA and LDA) are appropriate for the researched problems.", "theoretical_claims": "The paper does not make strong theoretical claims, focusing on empirical analysis.", "experimental_designs_or_analyses": "The authors conduct a detailed analysis of representation similarity, pruning effectiveness, and concept steering. However, the paper could benefit from additional experiments:\n\n* The authors claim that: \"Large TSFMs typically learn redundant representations, which often manifest as block-like structures in heatmaps”. Does this claim apply to non-pretrained time series models, e.g., a supervisedly trained Transformer on specific datasets? I wonder whether the mentioned redundant representations are caused by the variation redundancy of time series data.\n* The use of synthetic data for concept identification and steering is a strong point. However, the paper could benefit from additional experiments on real-world datasets to further validate the generalizability of the steering method.", "relation_to_broader_scientific_literature": "The authors build on prior work for representation analysis of vision models, extending these ideas to the domain of time series foundation models.", "essential_references_not_discussed": "The paper adequately covers the relevant works, but it could benefit from a wider topic on time series foundation models, such as the scaling law of TSFMs.", "other_strengths_and_weaknesses": "Strength: The paper addresses a timely topic in the field of time series analysis, particularly the interpretability and controllability of large foundation models.\n\nWeakness: \n\n* The proposed pruning method may lack novelty. What is the main inovation beyond Nguyen's prior works? \n* The inference time saved by the proposed block-wise pruning seems to be marginal (Table 7, 20.88ms -> 19.82ms). How does this approach compare to other pruning techniques in relevant literature? \n* The motivation of the steering method is not well presented. What is the benefit (or applications) to guide the model for adjusted predictions with post-informed trend and seasonality? \n* The visualization of the experiment may be derived from a subset of samples. It would be beneficial for the author to explain the rationale behind the selection of these samples and provide relevant statistics to mitigate the risk of cherry-picking.", "other_comments_or_suggestions": "Several charts in the experimental part are not very readable, such as the meaning of numbers in Figure 4 and Figure 5.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the representation of time series foundation models. By using representation similarity across multiple models, they identify redundant layer-wise representations, subsequently developing block-wise pruning schemes to improve runtime while maintaining performance. The reviewers had a number of comments, mostly with the clarity of the approach and exposition, which included several major issues including errors in computed numbers that were later fixed, and lack of clarity on novelty over a similar related work. Regardless the contributions to examining foundation models were appreciated and so if there is room I would recommend acceptance.
Decision: Accept (poster)