Exploring Representations and Interventions in Time Series Foundation Models

## Abstract

Abstract Time series foundation models (TSFMs) promise to be powerful tools for a wide range of applications. However, their internal representations and learned concepts are still not well understood. In this study, we investigate the structure and redundancy of representations across various TSFMs, examining the self-similarity of model layers within and across different model sizes. This analysis reveals block-like redundancy in the representations, which can be utilized for informed pruning to improve inference speed and efficiency. Additionally, we explore the concepts learned by these models—such as periodicity and trends—and how these can be manipulated through latent space steering to influence model behavior. Our experiments show that steering interventions can introduce new features, e.g., adding periodicity or trends to signals that initially lacked them. These findings underscore the value of representational analysis for optimizing models and demonstrate how conceptual steering offers new possibilities for more controlled and efficient time series analysis with TSFMs.

## 1 Introduction

Foundation models have taken significant strides in modeling both textual (Brown et al., ) and visual (Dosovitskiy et al., ) data, and have made complex language and image processing accessible to non-experts. These models are pre-trained on massive internet-scale datasets and can be used to solve multiple tasks across a variety of domains, with little to no adaptation. Recently, a growing body of work (Garza and Mergenthaler-Canseco, ; Goswami et al., ; Rasul et al., ; Das et al., ; Woo et al., ; Ansari et al., ) has extended the benefits of this paradigm to time series data, a modality prevalent in fields such as finance (Taylor, ), healthcare (Goswami et al., ), and climate science (Schneider and Dickinson, ).

Time series foundation models (TSFMs) have shown promising performance on multiple modeling tasks such as forecasting, classification, anomaly detection, and imputation, across a wide range of domains, and in settings with varying amounts of data and supervision. However, the underlying mechanisms and learned representations of TSFMs remain largely unexplored. Little is known about the characteristics of learned representations, the kinds of concepts that these models are learning, and how can these concepts could be manipulated to influence model outputs. A deeper understanding of the inner workings of TSFMs is key to enhancing their performance and trustworthiness.

We address these knowledge gaps, by systematically probing TSFMs and intervening in them. First, we examine these models through the lens of representational similarity, identify learned concepts, and localize them to specific hidden states. We find that TSFMs learn redundant representations, and intuitive concepts such as base patterns (e.g., constant vs. sinusoidal waves), amplitudes, periodicity, and trends; often in specific layers and patches. We leverage these insights to intervene in TSFMs in two ways: (1) we exploit the redundant representations to prune models, reducing their size and improving inference speed while preserving accuracy, and (2) we steer model predictions along conceptual directions (e.g., adding an upward trend to forecasts) using synthetic time series. Our work is the first step towards understanding the inner workings of TSFMs and leveraging this knowledge to develop strategies for improving their capabilities.

## 2 Related Work

Time Series Foundation Models (TSFMs). TSFMs are versatile neural networks pre-trained on vast amounts of time series data, and have shown remarkable capabilities in producing accurate predictions even in zero-shot settings. Recently, a number of TSFMs have been proposed (Garza and Mergenthaler-Canseco, ; Liu et al., ; Das et al., ; Woo et al., ; Goswami et al., ; Ansari et al., ; Ekambaram et al., ; Talukder et al., ). While most TSFMs are based on variations of the Transformer architecture (Vaswani, ), they exhibit notable differences in tokenization strategies, pre-training datasets, and the specific tasks they are designed to address. While our methods are broadly applicable to Transformer-based TSFMs, we will primarily focus on analyzing MOMENT Goswami et al. (), Chronos Ansari et al. (), and MOIRAI Woo et al. (), three representative TSFMs that are fully open-source and offer distinct approaches to time series tokenization (patch vs. discrete), architecture (encoder-only vs. encoder-decoder), and pre-training objectives (imputation vs. forecasting).

Analyzing Representations of Deep Learning Models. Deep learning models often operate as black boxes, making their internal mechanisms and learned representations difficult to understand. One approach to gaining insights into these models is by comparing their intermediate representations. Similarity metrics can be used to determine the similarity or dissimilarity of representations at different stages of a model, revealing the hierarchy, homogeneity, and redundancy of learned features. Previous studies focused on quantifying the similarity of neural network representations Raghu et al. (); Kornblith et al. (). Raghu et al. () demonstrated the use of these metrics for analyzing and comparing representations in vision transformers and CNNs, providing valuable insights into their functioning. Nguyen et al. () investigated the impact of model size and training data ratios on similarity patterns by varying the depth and width of different models. They also explored model pruning based on similarity. Building on these these studies, our work presents the first comprehensive analysis of representations learned by TSFMs, offering useful insights into their internal workings.

Identifying and Manipulating Learned Concepts in Pre-trained Models. Understanding the internal representations learned by pre-trained models has been an active area of research, particularly in the context of LLMs and vision models. Previous studies have explored whether individual neurons or directions in a model’s latent space correspond to specific features or concepts
(Dalvi et al., ; Goh et al., ; Gurnee et al., ; Elhage et al., ). These investigations often focus on identifying linear representations, where features are encoded as linear combinations of neuron activations. Recent work has also employed various probing techniques to classify and interpret these internal representations, addressing aspects such as truthfulness and model robustness (Azaria and Mitchell, ; Zou et al., ; Burns et al., ; Marks and Tegmark, ). While many of these studies rely on meticulously curated datasets to probe language and vision models, we demonstrate that for time series models, synthetic data generated using simple mechanisms can effectively identify, localize, and probe the concepts learned by these models.

## 3 Probing and Intervening in Time Series Foundation Models

We study TSFMs using two complementary analysis and intervention methods. In Section 3.1, we examine learned representations through the lens of similarity, uncovering the redundancy inherent in TSFM representations. We leverage this redundancy to prune multiple layers of pre-trained models, thereby improving their efficiency without compromising accuracy. In Section 3.2, we identify the specific concepts learned by TSFMs, localizing them to specific hidden states. Furthermore, we explore the ability to steer model predictions along these conceptual directions, enabling us to influence model outputs in targeted ways.

### 3.1 Analyzing Representational Similarity for Effective Pruning

To gain a comprehensive understanding of the similarity between learned representations, we considered several metrics commonly employed in the literature. While our primary analysis relies on Centered Kernel Alignment (CKA) Kornblith et al. (), we also explored Cosine Similarity and Singular Vector Canonical Correlation Analysis (SVCCA) (Raghu et al., ). For brevity, we provide a brief overview of CKA below, while detailed descriptions of the remaining metrics can be found in Appendix B.

#### Representational Similarity using Centered Kernel Alignment (CKA).

CKA measures the similarity of representations by comparing the centered kernel matrices. It has been shown to be effective in capturing similarities between layers of deep networks (Kornblith et al., ). The general form of CKA between two sets of representations $\mathbf{X}$ and $\mathbf{Y}$ is defined as:

$$ $\text{CKA}(\mathbf{X},\mathbf{Y})=\frac{\text{HSIC}(\mathbf{X},\mathbf{Y})}{ \sqrt{\text{HSIC}(\mathbf{X},\mathbf{X})\cdot\text{HSIC}(\mathbf{Y},\mathbf{Y} )}}$ (1) $$

where HSIC denotes the Hilbert-Schmidt Independence Criterion (Gretton et al. ()).

Figure: Figure 1: For each identified block of layers exhibiting redundant representations (red), we remove the internal layers of the block by zeroing out their weights (blue). For example, if a block consists of five layers, we prune layers 2 through 4, retaining only the first and last layers to reduce representation redundancy while maintaining model integrity.
Refer to caption: x1.png

For computational efficiency, we utilized a linear kernel in our CKA calculations, resulting in the following simplified formula:

$$ $\text{CKA}_{\text{linear}}(\mathbf{X},\mathbf{Y})=\frac{\|\mathbf{X}^{T} \mathbf{Y}\|_{F}^{2}}{\|\mathbf{X}^{T}\mathbf{X}\|_{F}\cdot\|\mathbf{Y}^{T} \mathbf{Y}\|_{F}}$ (2) $$

where $\|\cdot\|_{F}$ denotes the Frobenius norm. The denominator in Equation 2 ensures that the metric value falls within the range of 0 to 1, facilitating interpretability. A high CKA value indicates a strong alignment between the two sets of representations, suggesting that the layers are likely learning similar features or concepts.

Figure: Algorithm 1 Block-wise Pruning

#### Pruning TSFMs Based on Representational Similarity.

Large TSFMs typically learn redundant representations, which often manifest as block-like structures in heatmaps depicting pairwise similarity between layer activations (Figure 1). We can leverage this redundancy to downsize TSFMs, to improve their inference speed, without affecting their accuracy. We build on prior work (Nguyen et al., ) and propose a simple layer pruning strategy, which we call Block-wise Pruning, outlined in Algorithm 1. To preserve the structural integrity of each block, we retain the first and last layers of each block while zeroing out the weights of the intermediate layers. The skip connections within transformer blocks ensure that signals and gradients continue to flow through the rest of the network.

#### Research Questions and Experimental Setup.

To gain a deeper understanding of TSFM representations, we investigate the following research questions: (RQ1) How similar are the representations learned by different layers within a single model? (RQ2) How do these representations vary across models of different sizes? (RQ3) How similar are representations learned by different TSFMs from the same family? To answer these questions, we use Centered Kernel Alignment (CKA) to measure the similarity between representations at different layers of TSFMs, and visualize the results using heatmaps.

To demonstrate the effectiveness of our proposed pruning strategy, we explore two pruning configurations, one in which prune all redundant blocks, and the other where prune only a single block. We compare the performance of these pruned models to the original, unpruned TSFMs using standard task-specific accuracy metrics (Mean Squared Error and Mean Absolute Error) and efficiency metrics (inference time in milliseconds and theoretical model size in megabytes). We evaluate these models on widely used imputation (Zhou et al., ) and forecasting (Ansari et al., ) benchmarks in both zero-shot settings and after linear probing (Goswami et al., ).

Figure: Figure 2: Overview of linear probing, concept localization, and steering. Linear probing involves training separate linear models for each layer or layer and patch to classify time series $\mathbf{x}$ into constant $c$ and sinusoid $s$ classes. Classifiers $f_{ij}(\mathbf{h}_{i}^{(j)},\theta_{i}^{(j)})$ are trained on the hidden representation $\mathbf{h}_{i}^{(j)}$ at each $i$-th layer and $j$-th token to update the parameters $\theta_{i}^{j}$. Concept localization is achieved by computing Fisher’s Linear Discriminant Ratio (LDR) between the classes at each layer and token using mean and variance statistics of $h_{i}^{(j)}$ for each predicted class, $\hat{y}$. The LDR output is scaled between 0 and 1 using min-max scaling to allow for consistent comparison across layers. Concept steering vector can be derived for each $i$-th layer by calculating the difference between the median activation matrices of the sinusoid and constant time series classes, $\mathbf{M}_{i_{s}}-\mathbf{M}_{i_{c}}$ and then stacked into a steering matrix $\mathbf{S}$ for the whole model. During model inference, the steering matrix can be used to steer model predictions towards desired concepts, or classes, by updating the embeddings as $\mathbf{h}_{i}\leftarrow\mathbf{h}_{i}+\alpha\mathbf{S}_{i}$, where $\alpha$ is a scalar that controls the strength of the intervention.
Refer to caption: x2.png

### 3.2 Identifying, Localizing and Steering Time Series Concepts

#### Generating Synthetic Data for Conceptual Analysis of TSFMs.

To systematically explore the ability of TSFMs to understand intuitive time series concepts, we randomly generate a large number of synthetic univariate time series. Each randomly generated time series belong to one of two pattern classes: constant or sinusoidal. Constant patterns, represented by $y(t)=\alpha t+\beta$, capture long-term non-periodic trends. Sinusoidal patterns, modeled as $y(t)=A\sin\left(\frac{2\pi t}{P}\right)$, represent periodic processes. By controlling the parameters $\alpha$, $\beta$, $A$, and $P$, we can systematically generate time series with varying slope, intercept, amplitude, and periodicity. Despite their simplicity, these data generation mechanisms capture a wide range of real-world time series patterns. For a detailed description of the data generation process, please refer to Appendix A.

In this section, we build on the investigation approach outlined in (Marks and Tegmark, ). We say that a feature is linearly represented in a foundation model $\mathcal{M}$ if it is represented as a direction in its latent space. As a concrete example, consider that we want to identify whether $\mathcal{M}$ can distinguish between constant and sinusoidal patterns. If this feature is linearly represented in $\mathcal{M}$, we also want to identify which layer $l$ in $\mathcal{M}$ learns this concept in the most discriminant way.

Figure: (a) Constant with increasing trend
Refer to caption: x3.png

#### Identifying Linearly Represented Features.

To determine whether the feature (sinusoidal vs. constant time series) is linearly represented, we first generate a synthetic time series dataset by varying this feature. This dataset comprises of multiple sinusoids and constant time series randomly sampled using our data generating function. Using this dataset, we extract intermediate representations of each time series from the residual stream of each layer. Let $\mathbf{h}_{i}^{(j)}\in\mathbb{R}^{n\times D}$ denote the hidden representation of a time series $\mathbf{x}$ at $i$-th layer and $j$-th token of $\mathcal{M}$, where $D$ is the dimensionality of the hidden layer. Linear probing involves training separate linear models for each layer and token to classify time series $\mathbf{x}$ as a constant or sinusoid pattern. Classifiers $f_{ij}(\mathbf{h}_{i}^{(j)},\theta_{i}^{(j)})$ are trained on the hidden representation $\mathbf{h}_{i}^{(j)}$ at each $i$-th layer and each $j$-th token to update the parameters $\theta_{i}^{j}$. Additionally, we perform probing on representations averaged along the token dimension for each $i$-th layer. The linear probes are trained to optimize the Fisher Criterion, a function that aims to maximize the distance between class means while minimizing within-class variance:

$$ $\displaystyle\mathcal{L}_{\text{Fisher}}(c,s)$ $\displaystyle=-\frac{(\mu_{s}-\mu_{c})^{2}}{\sigma_{s}^{2}+\sigma_{c}^{2}}.$ (3) $$

Here, $\mu_{s}$ and $\mu_{c}$ correspond to the mean embedding values, computed using all time series of a given class. Similarly, $\sigma^{2}_{s}$ and $\sigma^{2}_{c}$ correspond to the variance computed across the $n$ dimension for each class.

#### Localizing Linearly Represented Features.

To localize which layers and tokens learn a specific concept, we compute the Fisher’s Linear Discriminant Ratio (LDR) between the classes using the mean and variance statistics of $\mathbf{h}_{i}^{(j)}$ for each predicted class $\hat{\mathbf{y}}$, which is determined using the classifier $f_{ij}$ during linear probing. The goal of LDR is to maximize the separation between the classes by comparing the variance $\sigma^{2}$ within each class to the difference between the class means, $\mu$. A larger ratio indicates a clearer separation between the two classes, which can aid in concept localization by identifying where the classes are well-separated in the feature space. When applied in the context of neural network activations, LDR helps highlight which layers or features are most discriminative

$$ $\displaystyle\text{LDR}(\mathbf{h}_{i}^{(j)}|\hat{\mathbf{y}})$ $\displaystyle=\frac{(\mu_{\mathbf{h}_{i}^{(j)}|\hat{\mathbf{y}}=s}-\mu_{ \mathbf{h}_{i}^{(j)}|\hat{\mathbf{y}}=c})^{2}}{\sigma_{\mathbf{h}_{i}^{(j)}| \hat{\mathbf{y}}=s}^{2}+\sigma_{\mathbf{h}_{i}^{(j)}|\hat{\mathbf{y}}=c}^{2}}$ (4) $\displaystyle=\frac{(\mu_{s}-\mu_{c})^{2}}{\sigma_{s}^{2}+\sigma_{c}^{2}}.$ (5) $$

Here, $\mu_{s}$ and $\mu_{c}$ correspond to the mean computed across the $n$ dimension for each class. Similarly, $\sigma^{2}_{s}$ and $\sigma^{2}_{c}$ correspond to the variance computed across the sample dimension $n$ for each class.
Let $\mathbf{V}=[v_{i,j}]\in\mathbb{R}^{L\times N}$ be the matrix of LDR values, where $v_{i,j}$ represents the LDR value for the $i$-th layer and $j$-th token, with $l$ layers and $N$ tokens. The LDR output is scaled between 0 and 1 using min-max scaling to allow for consistent comparison across layers.
By visualizing the scaled LDA values as shown in Figure 2, one can identify which layers and tokens exhibit the highest degree of separation between classes, offering insights into the network’s internal representations for concept intervention techniques.

#### Deriving Steering Matrices for Model Steering.

Once we have identified that a feature is linearly represented in the latent space of the $\mathcal{M}$, we can use steering interventions to manipulate the latent space and generate time series that reflect intended concepts. For instance, to introduce periodicity to a constant time series, we can utilize a steering matrix $\mathbf{S}$, as illustrated in Figure 2. By strategically intervening in $\mathcal{M}$ using this steering matrix, we can bias its outputs towards predicting periodic time series. To construct a steering matrix, we first derive steering vectors $\mathbf{S}_{i}\in\mathbb{R}^{N\times D}$, for each layer $i$. These vectors represent the change that activations in layer $i$ must undergo such that $\mathcal{M}$ produces periodic outputs. $\mathbf{S}_{i}$ is simply the difference between the median activation matrix of the constant time series $\mathbf{M}_{i_{c}}$, from that of sinusoids $\mathbf{M}_{i_{s}}$. We stack these vectors for all layers to derive the steering matrix. This matrix allows us to simultaneously intervene across multiple tokens and layers during inference, which we found to be more effective than single-token interventions. During inference, to steer model predictions, at each layer $i$, we update its hidden representation as follows: $\mathbf{h}_{i}\leftarrow\mathbf{h}_{i}+\alpha\mathbf{S}_{i},$ where $\alpha\in\mathbb{R}$ is a scalar that controls the strength of the intervention.

#### Research Questions and Experimental Setup.

Through our experiments, we aim to answer the following research questions: (RQ4) Do TSFMs represent concepts associated with specific data-generating functions distinctly in the latent space? (RQ5) Are these learned concepts localized to specific layers and tokens within TSFMs? (RQ6) Can we leverage these learned concepts to bias model predictions towards intended outcomes? For example, can we add periodicity or an upward trend to a constant time series? (RQ7) Is it possible to combine multiple steering interventions to manipulate model predictions towards complex compositions of various concepts? For instance, can we steer a model to add both trend and periodicity to a constant signal?

To address these research questions, we will leverage the techniques outlined in previous sections. We also explore two alternative modalities of intervention: (1) deriving steering vectors using the mean of hidden activations rather than the median, and (2) steering a single token versus all tokens throughout the model.

Figure: Figure 4: These heatmaps visualize the linear separability of various concepts at the patch level. Linear separability refers to the Linear Discriminant Ratio (LDR) computed from model embedding statistics for each predicted class: constant versus sinusoidal patterns (i), increasing versus decreasing trends (ii), high versus low periodicity (iii), and high versus low sinusoidal amplitude (iv). Lighter shades indicate higher separability (dark blue $\rightarrow$ low LDR, yellow $\rightarrow$ high LDR). These heatmaps show that certain concepts represented by MOMENT-Large are linearly separable and that this separability is not consistent but rather emerges at specific layers in the model.
Refer to caption: x6.png

#### Analyzed Time Series Foundation Models.

While our methods are broadly applicable to a wide range of transformer-based foundation models, we focus on two prominent TSFM families for brevity: MOMENT(^1^11[https://github.com/moment-timeseries-foundation-model/moment](https://github.com/moment-timeseries-foundation-model/moment)) (Goswami et al., ) and Chronos(^2^22[https://github.com/amazon-science/chronos-forecasting](https://github.com/amazon-science/chronos-forecasting)) (Ansari et al., ). Both these models are fully open-source, come in different sizes, yet have fundamentally different design choices. For example, Chronos is forecasting on encoder-decoder transformer model which takes discretized time series as input, whereas MOMENT is a multi-task, encoder-only model which takes continuous time series patches as input. Since only the Large variant of MOMENT is publicly available at the time of writing this paper, we supplement our representation analysis results with Moirai (Woo et al., )(^3^33[https://github.com/SalesforceAIResearch/uni2ts](https://github.com/SalesforceAIResearch/uni2ts)), another another popular TSFM which comes in different sizes. More information on model parameters can be found in Appendix C.

Figure: Figure 5: Visualization of the MOMENT’s reconstruction and the Chronos’s forecasting predictions (bottom), with concept steering applied in the latent space (blue) and the baseline without concept steering (orange). Concept steering effectively transforms concepts in the latent space, resulting in model predictions that align with the intended concepts introduced. Steering results are illustrated for the following experiments: steering a constant signal input to produce (i) a sinusoidal output, (ii) a constant signal with an increasing(slope $>$ 0), and (iii) decreasing trend (slope $<$ 0). For compositional steering experiments, $\alpha$ controls the strength of sinusoidal and increasing trend concepts. When $\alpha=0.5$, models are steered towards a combination of increasing trend and sinusoidal pattern. $\alpha=0$ results in steering towards a constant signal with an increasing trend, whereas $\alpha=1$ only introduces sinusoidal patterns (iii). Detailed results are available in the Appendix D.
Refer to caption: x10.png

## 4 Results

Figure: Figure 6: These heatmaps illustrate pairwise similarity of layer measured using CKA. Lighter shades indicate higher similarity (dark blue $\rightarrow$ low similarity, yellow $\rightarrow$ high similarity). We see interest structures in the representations learned by these models. While MOMENT shows substantial redundancy in representations, Chronos shows a more pronounced structure of block patterns. Moirai exhibits the most distinct block patterns, suggesting clearly defined stages of representation learning. These plots offer insights into localized regions of representation utility and redundancy across the three models.
Refer to caption: x19.png

Figure: Figure 7: These heatmaps demonstrate how model size influences the patterns of learned representations. Smaller models exhibit more discrete and distinct block structures, while larger models (Base and Large) display increasingly intricate and clustered patterns, reflecting more nuanced and gradual transformations across layers. Notably, the emergence of blocks-like patterns in the Large model appears unpredictable from patterns observed in smaller models.
Refer to caption: x22.png

#### Analyzing representations offers interesting insights.

Our analysis of model representations demonstrates that both model size and internal architecture considerably influence how representations are organized. Fig. 4) shows heatmaps which reveal that larger models, such as MOMENT-Large, Chronos-Large, and Moirai-1.1-R Large, have similar representations across specific groups of layers forming distinct and intricate block patterns, which may reflect unique stages of representation learning. More complex block patterns are observed with increasing model size, indicating that scaling may enhance the richness and organization of internal representations. However, it may also increase redundant knowledge storage through similar representations across layers, as suggested by high CKA similarity measured in block patterns. Interestingly, within model families (e.g., Chronos and Moirai), scaling does not always result in predictable heatmap changes. Larger models, like Chronos-Large and Moirai-Large, demonstrate more refined and complex transformations of representations that are not easily extrapolated from their smaller versions as shown in Fig. 14). Moreover, cross-model similarity analysis results in Fig. 8 reveal that while early layers tend to have high similarity across models of different sizes, the similarity measures among later layers diverge more notably, particularly in larger models. This divergence is especially evident in the Chronos family, where early representations are more consistent across models, but later layers become increasingly specialized as model depth increases as shown in Fig. 7.

Figure: Figure 8: These heatmaps illustrate the similarity between representations learned by different layers in TSFMs of the same family but different sizes. Initial layers of these models tend to learn similar representations, while the similarity gradually decreases in the later layers.
Refer to caption: x27.png

#### Block-wise pruning can improve model throughput, without compromising accuracy.

We observed consistent improvements in memory efficiency and inference speed over their unpruned counterparts. For example, pruning only Block 3 for MOMENT-Large, resulted in a 11% decrease in estimated model size with a 5% speed up in inference. Furthermore, this pruned model had lower zero-shot imputation MAE for 5 of 9 datasets (ETTh2, ETTm1, ETTm2, Exchange, and Weather) as shown in Tab. 1. Chronos-Large results for zero-shot experiments are reported in Tab. 2. Detailed results on memory usage and speed improvements can be found in Tab. 5. While pruning consistently improved memory efficiency and inference speed compared to unpruned counterparts, performance varied across pruning methods and datasets, with some methods exhibiting considerable degradation. In addition to zero-shot experiments, we conducted experiments where models were fine-tuned post-pruning. For this, we applied MOMENT-Large for forecasting to compare a vanilla (unpruned) model to one with all block redundancies pruned, evaluating the impact of the most aggressive pruning approach. Fine-tuning results in Table 9 show that, notably, the pruned model performed nearly as well as the original, underscoring the potential of our block-wise pruning approach to maintain performance while reducing model complexity. Complete finetuning results are provided in Table 4 in Appendix D.

#### TSFMs learn Intuitive Linear Concepts.

Our concept localization results in Fig. 4 show that certain concepts represented by MOMENT-Large are linearly separable and that this separability is not consistent but rather emerges at specific layers in the model. We also found intuitive differences in the locations where these concepts are learned. We observed that certain concepts, such as distinguishing between constant and sinusoidal patterns, require careful examination of the entire time series. In contrast, differentiating between increasing and decreasing trends can be achieved by focusing on the initial and final patches. However, we did not identify specific locations where models learn to distinguish between time series of different amplitudes. This may be attributed to the normalization of input time series, a common practice in many TSFMs, including MOMENT.

#### We can effectively steer TSFM predictions.

Our concept steering interventions effectively transform the latent space of TSFMs, resulting in model predictions that align with the intended concepts, as demonstrated in Fig. 5. We successfully introduced periodicity and trend concepts to constant time series and demonstrated the ability to combine multiple steering vectors to create more complex patterns. By combining steering vectors representing increasing trends and sinusoidal patterns, we were able to steer model predictions towards a combination of these features. To evaluate the effectiveness of steering in the latent space, we analyzed the impact of our interventions on hidden representation, by projecting in a two-dimensional space using Principal Component Analysis (PCA). We found that steering in the latent space is reflected in these lower-dimensional representations, as illustrated in Fig. 11. Notably, the PCA reduction often captured the concept direction as one of the principal components. This can be attributed to the careful design of our synthetic data generation process.

Figure: Figure 9: We fine-tune the vanilla and pruned variants on MOMENT on widely used long-horizon forecasting datasets (Zhou et al., ) and measure MSE. We found that the pruned model performed on par with the original model, underscoring the potential of our block-wise pruning approach. In this particular pruning setup, we reduced the memory consumption by more than half compared to the vanilla model and improved inference time per sample by $\approx$1 ms.

Interestingly, the method of obtaining the steering matrix, either by computing the mean or median across embedding concept classes, has no notable effect on the steered output as shown in Fig. 12. However, applying concept steering interventions across all tokens is necessary to achieve the intended steered concept output compared to applying concept steering interventions to a single token. Moreover, the $\alpha$ parameter can have considerable effect on steered output. For Chronos, steering required tuning the parameter $\alpha\approx 0.1$ for effective performance, whereas MOMENT maintained effective steering with $\alpha=1$.

## 5 Discussion

We explored two complementary approaches to probing and intervening in TSFMs. We gained valuable insights into their internal mechanisms and identified opportunities for improvement. For instance, our analysis revealed redundancy in learned representations. We leveraged this representational redundancy, inherent to large over-parameterized TSFMs, to devise a simple block-wise pruning strategy. This strategy effectively reduced the size and computational cost of these models without compromising performance, demonstrating the potential for distilling smaller, more efficient models from larger TSFMs.

We also explored ways to influence model predictions along conceptual directions using steering matrices. Concept steering has many practical applications, including the ability to correct prediction errors in pre-trained models that may arise from out-of-distribution inference, reflecting the effects of exogenous factors on model predictions or inducing prior knowledge not fully captured during training. Additionally, steering provides a method to reduce computational costs by minimizing the need for fine-tuning. It can also be used to introduce inductive biases from various domains into the model. This is particularly valuable for pre-trained models that may lack exposure to specific concepts due to limited or restricted training data. Such inductive biases can be useful in domains like healthcare, where, e.g., knowledge of the excitatory or inhibitory effects of treatments can guide pre-trained model predictions about whether a patient’s vital signs should increase or decrease in response to specific interventions. Moreover, concept steering can be used for data generation, improving data augmentation techniques and creating more diverse datasets to imxprove TSFM performance across various tasks. Prior work has already shown that data augmentation techniques can improve TSFM model generalization by enhancing model robustness and exposing it to a wider variety of patterns (Ansari et al., ).

Finally, our findings also underscore the importance of synthetic data in studying and steering TSFMs. As opposed to meticulously curated datasets used to probe large language and vision models, we demonstrate that for time series models, synthetic data generated using simple mechanisms can effectively identify, localize, and probe the linear concepts learned by these models.

#### Limitations and Future Work.

This paper provides insights into how a few basic patterns are linearly represented in time series foundation models. Future work must evaluate whether time series foundation models can learn more complex patterns present in real-world time series and whether steering matrices estimated using synthetic data can be used to steer predictions of out-of-distribution, real-world time series. While our methods are broadly applicable to different transformer-based foundation models, future research should explore other architectures such as state space models (Gu and Dao, ) and stacked multi-layer perceptrons (Ekambaram et al., ). Moreover, future studies should evaluate whether our findings hold for other time series foundation models and tasks such as anomaly detection, and classification. Beyond time series, we hope that our work inspires the use of synthetic data to steer large language and vision models as well.

## Ethics Statement

While our pruned TSFMs demonstrate promising performance, it is crucial to exercise caution when using them, especially in high-stakes applications such as healthcare. Before deploying these models for critical decision-making, we strongly recommend fine-tuning and evaluating them on task-specific, in-domain datasets using relevant metrics. The ability to steer model predictions offers numerous benefits but also raises concerns regarding potential biases. We urge users to exercise caution when utilizing our proposed steering strategies and to carefully consider the potential implications of manipulating model outputs.

## Reproducibility Statement

To ensure reproduciblity, we have made our code anonymously accessible through [https://anonymous.4open.science/r/tsfm-interventions-3452/](https://anonymous.4open.science/r/tsfm-interventions-3452/). All time series foundation models used for our analysis are publicly available and open-source: Chronos ([https://github.com/amazon-science/chronos-forecasting](https://github.com/amazon-science/chronos-forecasting)), MOMENT ([https://github.com/moment-timeseries-foundation-model/moment](https://github.com/moment-timeseries-foundation-model/moment)), and MOIRAI ([https://github.com/SalesforceAIResearch/uni2ts](https://github.com/SalesforceAIResearch/uni2ts)). All models were trained and evaluated on a computing cluster consisting of 128 AMD EPYC 7502 CPUs, 503 GB of RAM, and 8 NVIDIA RTX A6000 GPUs each with 49 GiB RAM. Synthetic datasets used in our study and the pruned time series foundation models will be released publicly upon paper acceptance.

## Appendix A Synthetic Data Generation

Examples of synthetic data are provided in Fig. 10. Generated time series $\{x_{t}\}_{t=1}^{T}$ patterns include:

- •
Constant Pattern. The constant pattern captures long-term progression of the time series and is modeled as:
$y(t)=\alpha t+\beta,$
where $\alpha$ is the slope and $\beta$ is the intercept. These parameters are sampled from uniform distributions:
$\alpha\sim U(\mathrm{slope}_{\min},\mathrm{slope}_{\max}),\quad\beta\sim U(
\mathrm{intercept}_{\min},\mathrm{intercept}_{\max}).$
- •
Sinusoidal Pattern.
The sinusoidal pattern captures patterns of periodic variations and is modeled as:
$y(t)=A\sin\left(\frac{2\pi t}{P}\right)+\alpha t+\beta,$
where $A$ is the amplitude and $P$ is the period, both sampled from uniform distributions:
$A\sim U(\mathrm{amplitude}_{\min},\mathrm{amplitude}_{\max}),\quad P\sim U(
\mathrm{period}_{\min},\mathrm{period}_{\max}).$

Parameter Ranges:

- •
Constant Case:
$A\sim U(0,0),\quad P\sim U(0,0),\quad\alpha\sim U(0,0),\quad\beta\sim U(-30,30).$
- •
Increasing Slope Case:
$A\sim U(0,0),\quad P\sim U(0,0),\quad\alpha\sim U(0.5,1),\quad\beta\sim U(-30,
30).$
- •
Decreasing Slope Case:
$A\sim U(0,0),\quad P\sim U(0,0),\quad\alpha\sim U(-1,-0.5),\quad\beta\sim U(-3
0,30).$
- •
Sine Constant Case (with seasonality parameters):
$A\sim U(50,50),\quad P\sim U(128,128),\quad\alpha\sim U(0,0),\quad\beta\sim U(
-30,30).$
- •
Sine Increasing Slope Case (with seasonality parameters):
$A\sim U(50,50),\quad P\sim U(128,128),\quad\alpha\sim U(0.5,1),\quad\beta\sim U
(-30,30).$
- •
Sine Decreasing Slope Case (with seasonality parameters):
$A\sim U(50,50),\quad P\sim U(128,128),\quad\alpha\sim U(-1,-0.5),\quad\beta
\sim U(-30,30).$

Figure: Figure 10: Samples of synthetic time series used in our experiments. We use two base patterns (constant and sinusoidal). To generate synthetic datasets, we vary the periodicity ($f$), amplitude ($a$), intercept ($c$), and linear trend ($t$).
Refer to caption: x29.png

## Appendix B Additional Representation Similarity Metrics

To comprehensively analyze the similarity of learned representations, we considered several metrics commonly used in the literature. While our primary analysis relies on Centered Kernel Alignment (CKA) Kornblith et al. [], we also explored two additional similarity metrics, namely Cosine Similarity and Singular Vector Canonical Correlation Analysis (SVCCA) [Raghu et al., ]. Our findings consistently demonstrated similar patterns across all these metrics, underscoring the robustness of our results. We provide brief descriptions of Cosine Similarity and SVCCA below.

Cosine Similarity: Cosine similarity measures the cosine of the angle between two vectors, providing a simple yet effective way to assess similarity. In our case, we work with activation matrices for multiple samples and compute the average cosine similarity. Given two matrices $\mathbf{X}$ and $\mathbf{Y}$, representing the activations from two layers, the cosine similarity is computed as:

$$ $\text{cosine similarity}(\mathbf{X},\mathbf{Y})=\frac{1}{n}\sum_{i=1}^{n}\frac {\mathbf{x}_{i}\cdot\mathbf{y}_{i}}{\|\mathbf{x}_{i}\|\|\mathbf{y}_{i}\|}$ (6) $$

where $\mathbf{x}_{i}$ and $\mathbf{y}_{i}$ are the $i$-th columns of matrices $\mathbf{X}$ and $\mathbf{Y}$, and $n$ is the number of samples.

Singular Vector Canonical Correlation Analysis (SVCCA) [Raghu et al., ]: SVCCA is a method that compares the similarity of representations by aligning subspaces spanned by the top singular vectors. It effectively reduces the dimensionality and then compares the correlations of the principal components. The SVCCA similarity between two activation matrices $\mathbf{X}$ and $\mathbf{Y}$ is computed as follows:

$$ $\text{SVCCA}(\mathbf{X},\mathbf{Y})=\text{CCA}(\mathbf{U}_{k},\mathbf{V}_{k})$ (7) $$

where $\mathbf{U}_{k}$ and $\mathbf{V}_{k}$ are the top $k$ singular vectors obtained from the singular value decomposition (SVD) of $\mathbf{X}$ and $\mathbf{Y}$, respectively, and CCA denotes the canonical correlation analysis.

## Appendix C Model Descriptions and Specifications

### C.1 MOMENT

The MOMENT model family is designed for general-purpose time-series analysis, utilizing an encoder-only Transformer architecture. It processes input time series by dividing them into fixed-length sub-sequences (patches) and encoding each patch into a D-dimensional space. The pre-training task involves reconstructing masked patches to learn robust representations that generalize across various tasks.

- •
Architecture: Encoder-only Transformer, with patch embeddings and masking for reconstruction.
- •
Input Representation: Time series split into fixed-length patches, embedded into D-dimensional vectors.
- •
Model Variants: MOMENT-Large

### C.2 Chronos

Chronos models utilize a sequence-to-sequence (encoder-decoder) Transformer architecture based on the T5 model family. Time series are scaled and quantized into tokens, which are processed by the encoder. The decoder autoregressively predicts future time steps, generating tokens that are mapped back into numerical values.

- •
Architecture: Encoder-decoder Transformer based on T5, with a reduced vocabulary size of 4096 tokens.
- •
Input Representation: Time series quantized into discrete tokens for sequence modeling.
- •
Model Variants: Chronos is available in the following configurations:
–
chronos-t5-tiny: 8M parameters
–
chronos-t5-mini: 20M parameters
–
chronos-t5-small: 46M parameters
–
chronos-t5-base: 200M parameters
–
chronos-t5-large: 710M parameters

### C.3 Moirai

Moirai is a time-series foundation model built with an encoder-only Transformer architecture designed for universal forecasting. The model handles varying temporal resolutions with multiple patch size projection layers and uses any-variate attention for multivariate time series. A mixture distribution is employed to model probabilistic forecasts.

- •
Architecture: Encoder-only Transformer with any-variate attention for multivariate time-series forecasting.
- •
Input Representation: Time series processed using multi-patch size projections to handle different frequencies.
- •
Model Variants: Moirai is available in three configurations:
–
Moirai-small: 14M parameters
–
Moirai-base: 91M parameters
–
Moirai-large: 311M parameters

## Appendix D Additional Results

Figure: Figure 11: Visualization of steering effects in latent space reduced using PCA analysis. Here, we considered steering constant to sinusoidal, constant to increasing, and constant to decreasing series. As shown, the steering behaves as expected in the embedding space, moving selected constant samples, into the neighborhood of sinusoidal/increasing/decreasing samples.
Refer to caption: x38.png

Figure: Figure 12: Visualization of different intervention and steering matrix derivation techniques. Applying concept steering interventions across all tokens is necessary to achieve the intended steered concept output (iii, iv) compared to applying concept steering interventions to a single token (i, iii). The method of obtaining the steering matrix, either by computing the mean or median across embedding concept classes, has no notable effect on the steered output.
Refer to caption: x41.png

Figure: Figure 13: Compositional Steering for Forecasting using the MOMENT model. The parameter $\alpha$ influences compositional steering, where both sinusoidal and increasing trend concepts are observed in the steered output when $\alpha$=0.5, showcasing how the steering technique can interpolate between different concept combinations in the latent space (ii). Steered output changes from sinusodial concepts to increasing trend concepts by varying $\alpha\in\{0.0,0.25,0.5,0.75,1.0\}$.
Refer to caption: x45.png

Figure: Figure 14: Self-similarity heatmaps comparing layer-to-layer CKA similarity matrices across different-sized models in the Moirai family. The Small and Base models exhibit similar patterns, with a distinct dissimilar layer at the end. The Base model has one additional dissimilar layer just before the final one. In contrast, the Large model presents a notably different structure, showing a two-block pattern where the layers are divided into two distinct groups, with the first block comprising roughly one-third of the model, and the second block covering the remaining two-thirds. This indicates a more pronounced hierarchical representation in the larger model.
Refer to caption: x50.png

Figure: Figure 15: Compositional Steering for Forecasting using the Chronos model. The parameter $\alpha$ influences compositional steering, where both sinusoidal and increasing trend concepts are observed in the steered output when $\alpha$=0.5, showcasing how the steering technique can interpolate between different concept combinations in the latent space (ii). Steered output changes from sinusoidal concepts to increasing trend concepts by varying $\alpha\in\{0.0,0.25,0.5,0.75,1.0\}$.
Refer to caption: x53.png

### D.1 Pruning experiments

**Table 1: Zero-shot imputation performance of MOMENT. Results averaged across four different masking rates: {12.5%, 25%, 37.5%, 50%} and five runs with different masking seeds. This table presents the Model Performance Metrics (Mean Absolute Error and Mean Squared Error) on a subset of the Time Series Pile [Goswami et al., , Zhou et al., ]. The model names include: ”Vanilla” MOMENT-Large without any pruning, ”Block 1-3” for cases where only one block is pruned, and ”All” for all three blocks being pruned. The best results per dataset are bolded. For full results refer to 3**
| Model name | Vanilla | Block 1 | Block 2 | Block 3 | All |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE |
| ETTh1 | 0.395 | 0.371 | 0.448 | 0.449 | 0.521 | 0.614 | 0.420 | 0.424 | 0.548 | 0.673 |
| ETTh2 | 0.243 | 0.132 | 0.268 | 0.153 | 0.290 | 0.176 | 0.243 | 0.133 | 0.296 | 0.185 |
| ETTm1 | 0.287 | 0.204 | 0.321 | 0.233 | 0.358 | 0.307 | 0.287 | 0.221 | 0.345 | 0.284 |
| ETTm2 | 0.185 | 0.080 | 0.198 | 0.088 | 0.218 | 0.107 | 0.178 | 0.076 | 0.220 | 0.112 |
| Electricity | 0.372 | 0.250 | 0.446 | 0.342 | 0.716 | 0.757 | 0.428 | 0.327 | 0.727 | 0.819 |
| Exchange rate | 0.125 | 0.034 | 0.124 | 0.032 | 0.170 | 0.061 | 0.111 | 0.027 | 0.174 | 0.067 |
| Illness | 0.393 | 0.421 | 0.448 | 0.502 | 0.547 | 0.669 | 0.423 | 0.446 | 0.552 | 0.648 |
| Traffic | 0.492 | 0.790 | 0.552 | 0.906 | 0.861 | 1.562 | 0.606 | 0.940 | 0.878 | 1.633 |
| Weather | 0.129 | 0.079 | 0.134 | 0.082 | 0.170 | 0.107 | 0.117 | 0.074 | 0.176 | 0.120 |

**Table 2: Zero-shot forecasting performance of Chronos-Large. This table presents zero-shot performance evaluated with Mean Absolute Scaled Error (MASE) and Weighted Quantile Loss (WQL). Results are presented for the model without any pruning (Vanilla), when individual blocks pruned Block i, and when all blocks are pruned (All).**
|  | Dataset | Vanilla | Block 1 | Block 2 | Block 3 | Block 4 | All |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | MASE | WQL | MASE | WQL | MASE | WQL | MASE | WQL | MASE | WQL | MASE | WQL |
| 0 | ETTh | 0.776 | 0.076 | 0.848 | 0.088 | 0.765 | 0.078 | 0.778 | 0.079 | 0.899 | 0.098 | 1.093 | 0.100 |
| 1 | ETTm | 0.716 | 0.068 | 0.820 | 0.076 | 0.864 | 0.084 | 0.721 | 0.063 | 0.724 | 0.074 | 0.978 | 0.084 |
| 2 | dominick | 0.820 | 0.331 | 0.829 | 0.335 | 0.824 | 0.334 | 0.809 | 0.332 | 0.835 | 0.337 | 0.886 | 0.359 |
| 3 | ercot | 0.627 | 0.020 | 0.694 | 0.022 | 0.697 | 0.027 | 0.624 | 0.018 | 0.909 | 0.024 | 1.281 | 0.042 |
| 4 | exchange_rate | 2.310 | 0.012 | 2.394 | 0.014 | 2.061 | 0.013 | 2.256 | 0.014 | 1.877 | 0.012 | 1.713 | 0.013 |
| 5 | m4_quarterly | 1.217 | 0.082 | 1.255 | 0.084 | 1.193 | 0.081 | 1.223 | 0.082 | 1.213 | 0.082 | 1.270 | 0.085 |
| 6 | m4_yearly | 3.559 | 0.133 | 3.537 | 0.132 | 3.495 | 0.129 | 3.507 | 0.131 | 3.502 | 0.129 | 3.359 | 0.123 |
| 7 | m5 | 0.943 | 0.586 | 0.942 | 0.586 | 0.954 | 0.591 | 0.939 | 0.587 | 0.941 | 0.622 | 0.956 | 0.644 |
| 8 | monash_australian_electricity | 1.427 | 0.076 | 1.417 | 0.077 | 1.372 | 0.073 | 1.445 | 0.083 | 1.638 | 0.084 | 2.593 | 0.143 |
| 9 | monash_car_parts | 0.903 | 1.041 | 0.883 | 1.046 | 0.867 | 0.998 | 0.892 | 1.027 | 0.833 | 0.960 | 0.816 | 0.956 |
| 10 | monash_cif_2016 | 0.989 | 0.012 | 1.058 | 0.019 | 0.944 | 0.013 | 0.998 | 0.018 | 1.096 | 0.011 | 1.275 | 0.017 |
| 11 | monash_covid_deaths | 43.251 | 0.058 | 42.444 | 0.052 | 44.001 | 0.078 | 42.357 | 0.060 | 41.915 | 0.053 | 45.544 | 0.062 |
| 12 | monash_fred_md | 0.517 | 0.021 | 0.520 | 0.019 | 0.507 | 0.024 | 0.553 | 0.029 | 0.504 | 0.016 | 0.600 | 0.017 |
| 13 | monash_hospital | 0.704 | 0.056 | 0.724 | 0.057 | 0.690 | 0.055 | 0.703 | 0.055 | 0.718 | 0.058 | 0.726 | 0.060 |
| 14 | monash_m1_monthly | 1.075 | 0.127 | 1.117 | 0.132 | 1.057 | 0.125 | 1.107 | 0.130 | 1.222 | 0.141 | 1.321 | 0.155 |
| 15 | monash_m1_quarterly | 1.728 | 0.106 | 1.743 | 0.102 | 1.659 | 0.105 | 1.727 | 0.092 | 1.748 | 0.105 | 1.753 | 0.098 |
| 16 | monash_m1_yearly | 4.336 | 0.183 | 4.275 | 0.174 | 4.113 | 0.174 | 4.375 | 0.163 | 4.400 | 0.182 | 4.100 | 0.170 |
| 17 | monash_m3_monthly | 0.855 | 0.096 | 0.887 | 0.100 | 0.845 | 0.094 | 0.864 | 0.096 | 0.909 | 0.102 | 0.968 | 0.108 |
| 18 | monash_m3_quarterly | 1.183 | 0.075 | 1.204 | 0.075 | 1.178 | 0.073 | 1.199 | 0.075 | 1.204 | 0.074 | 1.301 | 0.077 |
| 19 | monash_m3_yearly | 3.034 | 0.145 | 2.972 | 0.146 | 2.908 | 0.143 | 2.980 | 0.143 | 3.017 | 0.144 | 2.870 | 0.137 |
| 20 | monash_nn5_weekly | 0.936 | 0.090 | 0.961 | 0.092 | 0.924 | 0.090 | 0.939 | 0.090 | 0.941 | 0.090 | 0.985 | 0.095 |
| 21 | monash_tourism_monthly | 1.746 | 0.099 | 1.847 | 0.101 | 1.613 | 0.088 | 1.996 | 0.103 | 2.178 | 0.191 | 2.970 | 0.217 |
| 22 | monash_tourism_quarterly | 1.647 | 0.072 | 1.729 | 0.065 | 1.615 | 0.059 | 1.666 | 0.063 | 2.011 | 0.072 | 2.034 | 0.071 |
| 23 | monash_tourism_yearly | 3.565 | 0.180 | 3.683 | 0.185 | 3.574 | 0.173 | 3.724 | 0.188 | 3.596 | 0.174 | 3.568 | 0.175 |
| 24 | monash_traffic | 0.794 | 0.253 | 0.808 | 0.247 | 0.797 | 0.253 | 0.808 | 0.255 | 0.916 | 0.275 | 1.046 | 0.290 |
| 25 | monash_weather | 0.819 | 0.139 | 0.900 | 0.153 | 0.826 | 0.140 | 0.845 | 0.144 | 1.013 | 0.173 | 1.002 | 0.170 |
| 26 | nn5 | 0.574 | 0.157 | 0.584 | 0.160 | 0.599 | 0.165 | 0.571 | 0.154 | 0.792 | 0.219 | 0.905 | 0.255 |

**Table 3: Zero-shot imputation performance of MOMENT. Results averaged across five runs with different masking seeds. This table presents the Model Performance Metrics (Mean Absolute Error and Mean Squared Error) on a subset of the Time Series Pile [Goswami et al., , Zhou et al., ]. The model names include: ”Vanilla” MOMENT-Large without any pruning, ”Block 1-3” for cases where only one block is pruned, and ”All” for all three blocks being pruned.**
|  |  | All | Block 1 | Block 2 | Block 3 | Vanilla |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Mask Ratio | Mean | Std. | Mean | Std. | Mean | Std. | Mean | Std. | Mean | Std. |  |  |  |  |  |  |  |  |  |  |
|  |  | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE |
| ETTh1 | 0.125 | 0.541 | 0.662 | 0.016 | 0.069 | 0.448 | 0.446 | 0.023 | 0.084 | 0.516 | 0.608 | 0.016 | 0.072 | 0.426 | 0.425 | 0.038 | 0.119 | 0.398 | 0.370 | 0.034 | 0.091 |
|  | 0.25 | 0.548 | 0.667 | 0.015 | 0.049 | 0.445 | 0.441 | 0.019 | 0.056 | 0.520 | 0.605 | 0.021 | 0.073 | 0.418 | 0.420 | 0.026 | 0.073 | 0.393 | 0.365 | 0.026 | 0.061 |
|  | 0.375 | 0.550 | 0.674 | 0.015 | 0.035 | 0.447 | 0.446 | 0.011 | 0.033 | 0.522 | 0.612 | 0.016 | 0.045 | 0.418 | 0.422 | 0.019 | 0.056 | 0.394 | 0.368 | 0.014 | 0.033 |
|  | 0.5 | 0.551 | 0.687 | 0.011 | 0.036 | 0.452 | 0.463 | 0.012 | 0.037 | 0.525 | 0.630 | 0.011 | 0.041 | 0.420 | 0.429 | 0.011 | 0.035 | 0.397 | 0.380 | 0.010 | 0.027 |
|  | mean | 0.548 | 0.673 | 0.014 | 0.046 | 0.448 | 0.449 | 0.016 | 0.052 | 0.521 | 0.614 | 0.015 | 0.056 | 0.420 | 0.424 | 0.024 | 0.071 | 0.395 | 0.371 | 0.021 | 0.054 |
| ETTh2 | 0.125 | 0.290 | 0.179 | 0.005 | 0.016 | 0.269 | 0.155 | 0.008 | 0.016 | 0.292 | 0.177 | 0.010 | 0.020 | 0.240 | 0.130 | 0.010 | 0.015 | 0.243 | 0.131 | 0.007 | 0.015 |
|  | 0.25 | 0.298 | 0.188 | 0.008 | 0.015 | 0.271 | 0.159 | 0.011 | 0.019 | 0.294 | 0.182 | 0.010 | 0.019 | 0.245 | 0.136 | 0.010 | 0.015 | 0.247 | 0.136 | 0.012 | 0.017 |
|  | 0.375 | 0.296 | 0.186 | 0.005 | 0.009 | 0.266 | 0.151 | 0.011 | 0.015 | 0.286 | 0.172 | 0.008 | 0.013 | 0.245 | 0.135 | 0.007 | 0.010 | 0.242 | 0.130 | 0.011 | 0.014 |
|  | 0.5 | 0.298 | 0.187 | 0.005 | 0.010 | 0.266 | 0.150 | 0.008 | 0.011 | 0.288 | 0.174 | 0.006 | 0.011 | 0.242 | 0.132 | 0.006 | 0.008 | 0.242 | 0.129 | 0.007 | 0.010 |
|  | mean | 0.296 | 0.185 | 0.006 | 0.012 | 0.268 | 0.153 | 0.009 | 0.015 | 0.290 | 0.176 | 0.008 | 0.015 | 0.243 | 0.133 | 0.008 | 0.012 | 0.243 | 0.132 | 0.009 | 0.013 |
| ETTm1 | 0.125 | 0.344 | 0.281 | 0.013 | 0.039 | 0.322 | 0.236 | 0.012 | 0.023 | 0.361 | 0.313 | 0.016 | 0.035 | 0.284 | 0.216 | 0.008 | 0.026 | 0.288 | 0.208 | 0.011 | 0.025 |
|  | 0.25 | 0.343 | 0.280 | 0.010 | 0.026 | 0.319 | 0.230 | 0.012 | 0.026 | 0.357 | 0.304 | 0.012 | 0.029 | 0.284 | 0.216 | 0.008 | 0.024 | 0.284 | 0.199 | 0.013 | 0.028 |
|  | 0.375 | 0.345 | 0.283 | 0.005 | 0.014 | 0.319 | 0.230 | 0.005 | 0.013 | 0.356 | 0.302 | 0.006 | 0.016 | 0.288 | 0.222 | 0.004 | 0.011 | 0.286 | 0.199 | 0.006 | 0.016 |
|  | 0.5 | 0.347 | 0.292 | 0.004 | 0.010 | 0.322 | 0.238 | 0.004 | 0.009 | 0.357 | 0.309 | 0.002 | 0.006 | 0.291 | 0.229 | 0.002 | 0.010 | 0.288 | 0.207 | 0.003 | 0.008 |
|  | mean | 0.345 | 0.284 | 0.009 | 0.024 | 0.321 | 0.233 | 0.009 | 0.018 | 0.358 | 0.307 | 0.010 | 0.023 | 0.287 | 0.221 | 0.006 | 0.018 | 0.287 | 0.204 | 0.008 | 0.020 |
| ETTm2 | 0.125 | 0.222 | 0.113 | 0.005 | 0.008 | 0.200 | 0.089 | 0.004 | 0.002 | 0.221 | 0.110 | 0.005 | 0.004 | 0.179 | 0.078 | 0.005 | 0.004 | 0.188 | 0.082 | 0.005 | 0.004 |
|  | 0.25 | 0.220 | 0.112 | 0.006 | 0.005 | 0.197 | 0.086 | 0.005 | 0.004 | 0.216 | 0.105 | 0.006 | 0.005 | 0.177 | 0.074 | 0.004 | 0.003 | 0.184 | 0.078 | 0.004 | 0.003 |
|  | 0.375 | 0.219 | 0.112 | 0.003 | 0.003 | 0.198 | 0.088 | 0.003 | 0.002 | 0.216 | 0.106 | 0.004 | 0.003 | 0.178 | 0.075 | 0.002 | 0.002 | 0.184 | 0.080 | 0.003 | 0.002 |
|  | 0.5 | 0.219 | 0.111 | 0.001 | 0.002 | 0.198 | 0.088 | 0.003 | 0.003 | 0.218 | 0.107 | 0.004 | 0.004 | 0.178 | 0.075 | 0.002 | 0.001 | 0.185 | 0.080 | 0.003 | 0.003 |
|  | mean | 0.220 | 0.112 | 0.004 | 0.005 | 0.198 | 0.088 | 0.004 | 0.003 | 0.218 | 0.107 | 0.005 | 0.004 | 0.178 | 0.076 | 0.003 | 0.003 | 0.185 | 0.080 | 0.004 | 0.003 |
| electricity | 0.125 | 0.736 | 0.838 | 0.019 | 0.037 | 0.449 | 0.345 | 0.009 | 0.011 | 0.722 | 0.769 | 0.013 | 0.025 | 0.425 | 0.321 | 0.010 | 0.015 | 0.375 | 0.253 | 0.012 | 0.014 |
|  | 0.25 | 0.725 | 0.816 | 0.011 | 0.021 | 0.443 | 0.338 | 0.006 | 0.007 | 0.715 | 0.755 | 0.009 | 0.016 | 0.428 | 0.325 | 0.004 | 0.007 | 0.371 | 0.249 | 0.006 | 0.007 |
|  | 0.375 | 0.722 | 0.809 | 0.007 | 0.012 | 0.444 | 0.341 | 0.002 | 0.003 | 0.712 | 0.750 | 0.005 | 0.008 | 0.430 | 0.331 | 0.003 | 0.003 | 0.369 | 0.248 | 0.004 | 0.004 |
|  | 0.5 | 0.725 | 0.814 | 0.002 | 0.002 | 0.447 | 0.345 | 0.004 | 0.004 | 0.714 | 0.754 | 0.002 | 0.004 | 0.431 | 0.332 | 0.003 | 0.003 | 0.371 | 0.249 | 0.002 | 0.002 |
|  | mean | 0.727 | 0.819 | 0.012 | 0.023 | 0.446 | 0.342 | 0.006 | 0.007 | 0.716 | 0.757 | 0.009 | 0.016 | 0.428 | 0.327 | 0.006 | 0.009 | 0.372 | 0.250 | 0.007 | 0.008 |
| exchange rate | 0.125 | 0.164 | 0.059 | 0.010 | 0.014 | 0.129 | 0.035 | 0.007 | 0.006 | 0.178 | 0.065 | 0.010 | 0.008 | 0.109 | 0.027 | 0.010 | 0.002 | 0.131 | 0.037 | 0.010 | 0.006 |
|  | 0.25 | 0.172 | 0.064 | 0.010 | 0.012 | 0.122 | 0.031 | 0.006 | 0.003 | 0.167 | 0.059 | 0.001 | 0.002 | 0.108 | 0.025 | 0.007 | 0.001 | 0.122 | 0.033 | 0.008 | 0.004 |
|  | 0.375 | 0.178 | 0.070 | 0.005 | 0.009 | 0.122 | 0.032 | 0.004 | 0.002 | 0.167 | 0.059 | 0.008 | 0.004 | 0.112 | 0.028 | 0.009 | 0.004 | 0.123 | 0.033 | 0.008 | 0.004 |
|  | 0.5 | 0.180 | 0.073 | 0.008 | 0.014 | 0.123 | 0.032 | 0.007 | 0.003 | 0.170 | 0.061 | 0.009 | 0.006 | 0.114 | 0.029 | 0.005 | 0.002 | 0.125 | 0.034 | 0.011 | 0.006 |
|  | mean | 0.174 | 0.067 | 0.010 | 0.013 | 0.124 | 0.032 | 0.006 | 0.004 | 0.170 | 0.061 | 0.008 | 0.006 | 0.111 | 0.027 | 0.008 | 0.003 | 0.125 | 0.034 | 0.009 | 0.005 |
| national illness | 0.125 | 0.612 | 0.840 | 0.122 | 0.477 | 0.521 | 0.722 | 0.157 | 0.597 | 0.619 | 0.916 | 0.164 | 0.660 | 0.454 | 0.590 | 0.129 | 0.522 | 0.443 | 0.588 | 0.145 | 0.541 |
|  | 0.25 | 0.563 | 0.662 | 0.066 | 0.205 | 0.457 | 0.509 | 0.078 | 0.283 | 0.552 | 0.674 | 0.069 | 0.284 | 0.425 | 0.449 | 0.060 | 0.246 | 0.403 | 0.430 | 0.080 | 0.255 |
|  | 0.375 | 0.512 | 0.545 | 0.039 | 0.148 | 0.408 | 0.403 | 0.056 | 0.204 | 0.501 | 0.547 | 0.054 | 0.208 | 0.407 | 0.383 | 0.036 | 0.177 | 0.364 | 0.348 | 0.057 | 0.183 |
|  | 0.5 | 0.521 | 0.544 | 0.019 | 0.084 | 0.406 | 0.374 | 0.031 | 0.139 | 0.516 | 0.538 | 0.032 | 0.147 | 0.407 | 0.361 | 0.028 | 0.119 | 0.360 | 0.320 | 0.040 | 0.130 |
|  | mean | 0.552 | 0.648 | 0.078 | 0.279 | 0.448 | 0.502 | 0.098 | 0.353 | 0.547 | 0.669 | 0.098 | 0.383 | 0.423 | 0.446 | 0.071 | 0.297 | 0.393 | 0.421 | 0.089 | 0.312 |
| traffic | 0.125 | 0.880 | 1.647 | 0.018 | 0.052 | 0.559 | 0.915 | 0.023 | 0.060 | 0.860 | 1.572 | 0.015 | 0.028 | 0.609 | 0.950 | 0.027 | 0.068 | 0.503 | 0.803 | 0.025 | 0.068 |
|  | 0.25 | 0.887 | 1.660 | 0.015 | 0.032 | 0.550 | 0.909 | 0.008 | 0.054 | 0.871 | 1.589 | 0.016 | 0.018 | 0.602 | 0.936 | 0.023 | 0.065 | 0.492 | 0.793 | 0.014 | 0.056 |
|  | 0.375 | 0.874 | 1.620 | 0.011 | 0.034 | 0.548 | 0.901 | 0.012 | 0.052 | 0.860 | 1.553 | 0.011 | 0.028 | 0.603 | 0.931 | 0.015 | 0.043 | 0.487 | 0.780 | 0.006 | 0.044 |
|  | 0.5 | 0.869 | 1.603 | 0.012 | 0.037 | 0.549 | 0.898 | 0.013 | 0.046 | 0.854 | 1.534 | 0.012 | 0.035 | 0.608 | 0.943 | 0.007 | 0.021 | 0.487 | 0.783 | 0.006 | 0.035 |
|  | mean | 0.878 | 1.633 | 0.015 | 0.043 | 0.552 | 0.906 | 0.015 | 0.049 | 0.861 | 1.562 | 0.014 | 0.033 | 0.606 | 0.940 | 0.018 | 0.049 | 0.492 | 0.790 | 0.015 | 0.049 |
| weather | 0.125 | 0.175 | 0.121 | 0.004 | 0.007 | 0.134 | 0.082 | 0.003 | 0.006 | 0.171 | 0.108 | 0.004 | 0.005 | 0.115 | 0.071 | 0.005 | 0.009 | 0.130 | 0.078 | 0.004 | 0.007 |
|  | 0.25 | 0.176 | 0.120 | 0.002 | 0.005 | 0.134 | 0.081 | 0.005 | 0.004 | 0.169 | 0.105 | 0.006 | 0.007 | 0.116 | 0.073 | 0.002 | 0.003 | 0.128 | 0.077 | 0.004 | 0.004 |
|  | 0.375 | 0.177 | 0.121 | 0.001 | 0.002 | 0.134 | 0.083 | 0.005 | 0.007 | 0.168 | 0.107 | 0.005 | 0.006 | 0.117 | 0.076 | 0.003 | 0.006 | 0.129 | 0.080 | 0.005 | 0.007 |
|  | 0.5 | 0.176 | 0.119 | 0.002 | 0.003 | 0.135 | 0.084 | 0.004 | 0.007 | 0.170 | 0.107 | 0.004 | 0.006 | 0.118 | 0.076 | 0.003 | 0.006 | 0.130 | 0.081 | 0.004 | 0.007 |
|  | mean | 0.176 | 0.120 | 0.002 | 0.004 | 0.134 | 0.082 | 0.004 | 0.006 | 0.170 | 0.107 | 0.005 | 0.006 | 0.117 | 0.074 | 0.003 | 0.006 | 0.129 | 0.079 | 0.004 | 0.006 |

**Table 4: Fine-tuned forecasting performance of MOMENT. This table presents model performance metrics (Mean Absolute Error and Mean Squared Error) on a subset of the Time Series Pile [Goswami et al., , Zhou et al., ]. Metrics are presented for MOMENT-Large without any pruning (Vanilla) and for all blocks pruned (All). Results are gathered from the best performance on the test set across 3 epochs of training with a batch size 64 and learning rate of 0.0001.**
|  |  | Vanilla | All Pruned |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | MAE | MSE | MAE | MSE |
| Dataset | Horizon |  |  |  |  |
| Exchange | 96 | 0.236 | 0.109 | 0.240 | 0.113 |
|  | 192 | 0.333 | 0.215 | 0.335 | 0.218 |
|  | 336 | 0.474 | 0.417 | 0.460 | 0.394 |
|  | 720 | 0.757 | 1.003 | 0.780 | 1.066 |
| ETTh1 | 96 | 0.409 | 0.385 | 0.409 | 0.388 |
|  | 192 | 0.426 | 0.411 | 0.426 | 0.414 |
|  | 336 | 0.437 | 0.423 | 0.437 | 0.424 |
|  | 720 | 0.464 | 0.443 | 0.470 | 0.460 |
| ETTh2 | 96 | 0.346 | 0.287 | 0.351 | 0.296 |
|  | 192 | 0.386 | 0.350 | 0.389 | 0.356 |
|  | 336 | 0.409 | 0.370 | 0.413 | 0.382 |
|  | 720 | 0.440 | 0.404 | 0.439 | 0.404 |
| ETTm1 | 96 | 0.347 | 0.290 | 0.346 | 0.29 |
|  | 192 | 0.373 | 0.330 | 0.369 | 0.326 |
|  | 336 | 0.386 | 0.352 | 0.386 | 0.354 |
|  | 720 | 0.420 | 0.409 | 0.422 | 0.414 |
| ETTm2 | 96 | 0.260 | 0.171 | 0.262 | 0.173 |
|  | 192 | 0.300 | 0.231 | 0.304 | 0.236 |
|  | 336 | 0.337 | 0.287 | 0.343 | 0.294 |
|  | 720 | 0.392 | 0.372 | 0.395 | 0.372 |
| ILI | 24 | 1.307 | 3.260 | 1.219 | 2.981 |
|  | 36 | 1.347 | 3.516 | 1.286 | 3.209 |
|  | 48 | 1.405 | 3.828 | 1.374 | 3.479 |
|  | 60 | 1.428 | 3.989 | 1.387 | 3.602 |
| Weather | 96 | 0.209 | 0.153 | 0.209 | 0.152 |
|  | 192 | 0.248 | 0.197 | 0.248 | 0.198 |
|  | 336 | 0.285 | 0.246 | 0.287 | 0.247 |
|  | 720 | 0.337 | 0.316 | 0.337 | 0.317 |

**Table 5: Inference performance under various pruning configurations. This table presents the inference performance metrics of the MOMENT-Large and Chronos-Large models with different pruning configurations. Results are presented for MOMENT-Large and Chronos without any pruning (None), when individual blocks pruned Block i, and when all blocks are pruned (All Blocks). Inference time estimation was performed by aggregating times from 100 passes of a one-batch, one-channel sample of length 512.**
| Method | Pruned Part | Encoder<br>Sparsity<br>(%) | Model<br>Sparsity<br>(%) | Estimated<br>Model Size<br>(MB) | Average<br>Time<br>(ms) | Standard<br>Deviation<br>(ms) |
| --- | --- | --- | --- | --- | --- | --- |
| MOMENT | None | 0.00 | 0.00 | 1301.76 | 20.88 | 0.42 |
| Block 1 | 12.50 | 11.29 | 1154.76 | 19.42 | 0.27 |  |
| Block 2 | 33.33 | 30.11 | 909.76 | 19.71 | 0.31 |  |
| Block 3 | 12.50 | 11.29 | 1154.76 | 19.56 | 0.29 |  |
| All Blocks | 58.33 | 52.70 | 615.76 | 19.82 | 0.43 |  |
| Chronos-T5 | None | 0.00 | 0.00 | 2704.48 | 59.95 | 0.26 |
| Block 1 | 8.33 | 3.55 | 2608.48 | 58.22 | 0.58 |  |
| Block 2 | 12.50 | 5.32 | 2560.48 | 56.80 | 0.78 |  |
| Block 3 | 8.33 | 3.55 | 2608.48 | 56.94 | 0.27 |  |
| Block 4 | 25.00 | 10.65 | 2416.48 | 57.55 | 0.84 |  |
| All Blocks | 54.17 | 23.07 | 2080.48 | 56.82 | 0.58 |  |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Applying concept steering interventions across all tokens is necessary to achieve the intended steered concept output compared to applying concept steering interventions to a single token. What is the main purpose of concept steering? Is it controllable to achieve some target objectives?
soundness: 2
strengths: 1.	The main aim of this paper is to address the gap in understanding the underlying mechanisms and learned representations of TSFMs, which remain largely unexplored. The representational analysis examined the representational similarity based on Centered Kernel Alignment (CKA), identified concepts, and localized them to specific hidden states. The representation redundancy found in this analysis was used to prune the model, which reduces the model size as well as improves the inference speed. 
2.	The findings in this paper help us understand the TSFMs, and the knowledge can potentially be used to improve their capabilities. For example,  the predictions can be steered along the conceptual directions using synthetic time series.
summary: This paper aims to understand the internal representations of time series foundation models (TSFM). By analyzing the self-similarity of the model layers, this paper reveals block-like redundancy in the structure that can be used for model pruning. This representational analysis also shows that the conceptual steering  potentially enables more controllable analysis for TSFM.
weaknesses: 1.	The analysis is conducted using many existing techniques, which are not specifically designed for time series models. There may be room for improvement by introducing novel techniques specifically designed for TSFMs. 
2.	Section 3.2 is a huge subsection that contains many paragraphs. These paragraphs can be further grouped into subsubsections for better viewing.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Refer to weaknesses
soundness: 3
strengths: 1 By comparing representational similarity between layers of the model, this work provides a new method for analyzing TSFMs. Systematic probing of TSFMs through representational similarity identifies redundant representations suitable for model pruning.

2 The work effectively reduces model size and improves inference speed while maintaining accuracy. It introduces the ability to steer model predictions along conceptual directions, influencing outputs in targeted ways.

3 The study provides comprehensive analytical experiments, verifying that steering interventions can introduce new features into signals, such as periodicity and trends. Moreover, it is not limited to a single model but spans multiple TSFMs, enhancing the generality of the findings.The use of open-source models and data facilitates community reproduction and further research.
summary: This work investigates the internal representations of Time Series Foundation Models (TSFMs), revealing block-like redundancy in their representations. It leverages this redundancy for informed pruning to enhance inference speed and efficiency without compromising accuracy. The study also explores learned concepts like periodicity and trends, demonstrating the potential of latent space steering to manipulate these concepts and influence model behavior, introducing new features to time series signals. The findings highlight the importance of representational analysis for optimizing TSFMs and open new avenues for controlled time series analysis.
weaknesses: 1 Although a pruning strategy is proposed, the effectiveness of reducing computational resource consumption for extremely large models like TimeGPT has not been fully validated.

2 The selection of the parameter α in concept steering may require fine-tuning, increasing the complexity of model application. Have you explored automated methods for selecting the parameter α, or can you provide guidance on choosing appropriate α values for different dataset and models.

3 The study primarily focuses on forecasting and does not explore the generalization of the findings to other time series tasks such as anomaly detection or classification.

4 The computational overhead of calculating and applying steering matrices in real-time scenarios is not addressed, which could be significant for large-scale applications.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * **Q1**: I am a bit unclear about the concept of "block" in the article. Does it refer to some adjacent layers? What does it mean to prune Block3 of MOMENT-large at line 426?

* **Q2**: Can you provide a more detailed explanation on how to control the generation of time series? Does this involve modifying some parameters of the model?
soundness: 3
strengths: The starting point of this article is very meaningful, being at a time when TSFMs are emerging, it is crucial to explore more efficient, interpretable, and controllable models. Additionally, some experimental findings in the article provide guidance for future TSFM architecture design. Moreover, the figures and tables in this paper are also aesthetically pleasing.
summary: In this paper, the author investigates the structure and redundancy of representations across various time series foundation models, examining the self-similarity of model layers within and across different model sizes. This analysis reveals block-like redundancy in the representations, which can be utilized for informed pruning to improve inference speed and efficiency. Additionally, the autjor explores the concepts learned by these models—such as periodicity and trends—and how these can be manipulated through latent space steering to influence model behavior. These findings underscore the value of representational analysis for optimizing models and demonstrate how conceptual steering offers new possibilities for more controlled and efficient time series analysis with TSFMs.
weaknesses: I have no objections to the content of this article. My only concern is that it reads more like a technical report, lacking an overall logical structure. Many figures and text sections appear isolated, which could make it difficult for beginners to read and understand. I suggest the authors expand the Introduction section to provide readers with a more comprehensive overview (currently only half a page) and adjust the layout of the figures to accurately correspond with the text content.

Meta-review: This work examines the internal representations of Time Series Foundation Models (TSFMs), identifying block-like redundancy within these representations. By exploiting this redundancy, the authors propose an informed pruning strategy that enhances inference speed and efficiency without sacrificing accuracy. Additionally, the study investigates learned concepts such as periodicity and trends, demonstrating the potential of latent space steering to manipulate these concepts and influence model behavior. This approach enables the introduction of new features to time series signals, offering a novel way to guide model predictions in specific conceptual directions.  

While the method effectively reduces model size, improves inference speed, and maintains accuracy, a significant concern is that the paper lacks a cohesive logical structure and reads more like a technical report. Furthermore, it fails to offer any substantial new insights to the research community.

Decision: Reject