Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
Authors: Leon Götz
Marcel Kollovieh
Stephan Günnemann
Leo Schwinn
Venue: ICML 2025 poster (2025)
Keywords: Time Series
Token Merging
Transformer
State-Space Model
Primary area: deep_learning->sequential_models_time_series
TL;DR: We introduce local merging, a domain-specific token merging algorithm, to boost the efficiency of time series foundation models by up to 5400% while maintaining prediction quality.

Abstract: Despite recent advances in subquadratic attention mechanisms or state-space models, processing long token sequences still imposes significant computational requirements. Token merging has emerged as a solution to increase computational efficiency in computer vision architectures. In this work, we perform the first investigations of token merging in time series analysis on both transformers and state-space models. We further introduce local merging, a domain-specific token merging algorithm that selectively combines tokens within a local neighborhood, achieving two major benefits:  a) Local merging can adjust its computational complexity from quadratic to linear based on the neighborhood size to effectively scale to long sequences; b) Local merging is the first causal merging scheme enabling token merging in transformer decoders. Further, we identify spectral properties of the input data that reliably predict the potential benefits of local merging without requiring evaluation on downstream tasks. Our comprehensive empirical evaluation demonstrates that local merging offers substantial efficiency gains with minimal impact on accuracy, achieving up to 5400% acceleration on the recently proposed Chronos foundation model.

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
Leon Götz 12 Marcel Kollovieh 234 Stephan Günnemann" 234
Leo Schwinn' 2.3
Abstract
8
Despite recent advances in subquadratic atten- tion mechanisms or state-space models, process- ing long token sequences still imposes significant computational requirements. Token merging has emerged solution to increase computational efficiency in computer vision architectures. In this work, we perform the first investigations of token merging in time series analysis on both transform- ers and state-space models. We further introduce local merging, domain- sppeiffc token merging algorithm that selectively combines tokens within local neighborhood, achieving two major bene- fits: a) Local merging can adjust its computational complexity from quadratic to linear based on the neighborhood size to effectively scale to long se- quences; b) Local merging is the first causal merg- ing scheme enabling token merging in transformer decoders. Further, we identify spectral properties of the input data that reliably predict the poten- tial benefits of local merging without requiring evaluation on downstream tasks. Our compre- hensive empirical evaluation demonstrates that local merging offers substantial efficiency gains with minimal impact on accuracy, achieving up to 5400 % acceleration on the recently proposed Chronos foundation model.
3
5
2
3
6
S
S
Figure 1: Local token merging: Computing token similarity on subset Sioe under locality constraint reduces token merging's quadratic complexity to linear.
nism scales quadratically with the number of input tokens, resulting in high memory requirements. This scalability issue becomes especially pronounced in time series pro- cessing, where sequences frequently comprise thousands of tokens (Godahewa et al., 2021). Consequently, recent foundational models in time series, such as Chronos, exhibit impressive zero-shot generalization capabilities but demand substantial computational resources (Ansari et al., 2024).
Recently, state-space models have emerged tion to mitigate the computational burden of transformers. Their complexity scales subquadratically with the sequence length (Poli et al., 2023), which allows them to process millions of tokens (Nguyen et al., 2023). However, even in state-space models, very long sequences will impose considerable memory and computational demands.
solu-
1. Introduction
Bolya al. (2023), have shown that the efficiency of ViTs can be substantially improved by merging tokens throughout the transformer architecture. Specifically, they compute similarity scores between tokens and combine them into single tokens through convex combination. However, they only explore token merging for VIT architectures.
Since their inception in NLP (Vaswani et al., 2017), trans- formers have extended their influence into various do- mains, including computer vision with Vision Transformers (ViTs) (Dosovitskíy et al., 2021), graphs (Yun al., 2019), and time series processing (Li et al., 2019). However, the computational complexity of the standard attention mecha-
In this work, we for the first time explore token merging within the time series domain. We introduce novel local token merging algorithm whose computational complexity varies from quadratic to linear, based on the neighborhood considered for each token merge. This allows token merging to scale to long sequences and be applicable to state-space models. Further, our local merging preserves causality and
'Volkswagen AG "Technical University of Munich MMunic Data Science Institute 'Munich Center for Machine Learning. Cor- respondence to: Leon Götz <lcon, goetz volkswagen.de>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
is the first viable token merging scheme for transformer decoders. The algorithm is illustrated in figure 1. Through comprehensive empirical evaluations, we analyze the impact of token merging on various time series transformer models and state-space models. Our key contributions are:
State-space models Due to the quadratic scaling of the attention mechanism, transformer architectures suffer from significant computational cost when processing long se- quences. Recently, state-space models have shown promis- ing results in overcoming this challenge. Linear state- space layers solve the sequential processing requirement of RNNs (Gu et al., 2021). The S4 model reduces memory requirements by conditioning the state-space matrix with a low-rank correction (Gu et al.,, 2022). By using implicit con- volutions and data-aware gating mechanism, Hyena (Poli et al., 2023) became one of the first state-space model archi- tectures to match transformers on NLP tasks. Later work uses hardware- hardware-aware aware algorithms to improve the performance on modern accelerators (Gu & Dao, 2023).
Token merging in time series We extend token merging from computer vision to time series analysis and propose local merging as domain- token merging algorithm.
Model acceleration Across five time series transformer architectures (5.1), foundation models (5.3), two state-space models (5.4), and six datasets, token merging reveals sub- stantial computational savings with only slight reductions in accuracy. In some settings, it even improves forecasting performance while accelerating models simultaneously. To- ken merging enhances model throughput by up to 5400 % and improves forecasting performance by up to 9 %.
Reducing tokens Many works reduce the number of pro- cessed tokens to increase the efficiency of transformers in computer vision and NLP, often by pruning (Meng et al., 2022; Goyal et al., 2020). Marin et al. (2021) merge to- kens in ViT architectures to reduce the loss of information associated with pruning. Bolya et al. (2023) enhance the token merging algorithm, which they successfully apply to already trained encoder-only models. Besides initial work on classification tasks (Bolya et al., 2023), subsequent work applies token merging to diffusion models (Bolya & Hoff- man, 2023). Kim et al. (2024) combine merging and pruning, while other works investigate optimal merging and pruning rates (Bonnaerens & Dambre, 2023; Chen et al., 2023). Con- current work adapts token merging to preserve the spectral properties of the token space (Tran et al., 2024). However, their merging algorithm still has quadratic complexity, mak- ing it unsuitable for long sequence processing.
Token merging outcomes We identify three distinct out- comes when using token merging: 1) a consistent decline in performance when merging more tokens, 2) initial improve- ments in accuracy with few merged tokens followed by a drop as merging increases, and 3) scenarios where accuracy remains unchanged regardless of the token merging rate.
Understand token merging Our detailed analysis reveals that token merging acts as an adaptive low-pass filter, se- lectively reducing noise. We further identify model- and sppescc- properties explaining the effectiveness of our token merging algorithms.
2. Related work
Sparse attention and token skipping Besides reducing the number of tokens, sparse attention (Child et al., 2019) and token skipping (Raposo et al., 2024) also decrease the computational requirements of transformer models. In con- trast token merging, sparse attention can only accelerate the attention mechanism itself and not the subsequent MLP, which can take over 60 % of the total computation (Marin et al., 2021). Concurrent work, such as token skipping (Ra- poso et al., 2024), involves the selection of a subset of tokens to be processed in a transformer layer. However, it has only been shown in NLP when training from scratch. Token merging, however, can accelerate already trained models and does not require any training data or fine-tuning. This is especially important for recent foundation models. In our experiments in sections 5.1 and 5.2, token merging success- fully accelerates Informer and Autoformer, which already employ sparse attention. We therefore consider token merg- ing as an orthogonal approach.
Time series transformers Recently, many transformer architectures with inductive biases for time series have been proposed. Most of them reduce complexity by modifying the attention mechanism. Informer uses ProbSparse atten- tion (Zhou et al., 2021), while Autoformer leverages autocor- relation as sequence-basee based similarity measure (Wu et al., 2021). FEDformer uses the frequency domain to model time series effectively (Zhou et al., 2022). Non-stationary Transformers mitigate the effect of the time series distribu- tion changing over time (Liu et al., 2022b).
Due to their success in the vision and NLP domain, transformer-bet based foundation models have lately emerged for time series, often used in zero-shot settings. Many works focus on training transformers directly on large and diverse time series datasets, usually with billions of tokens (Garza & Mergenthaler-Canseeo,, Mergenthaler- Canseco, 2023; Das al., 2023; Rasul et al.,
2023; Woo al., 2024). Inspired by the success of founda- tion models in NLP, the recently proposed Chronos model converts continuous time series data into a fixed vocabu- lary (Ansari et al., 2024).
Here, we propose the first token merging algorithm for the time series domain, which extends beyond previous investigations in ViTs (Bolya et al., 2023; Bolya & Hoffmman,
2

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
2023). We systematically evaluate the potential to reduce computational effort in time-serres--peecfie architectures. See appendix A for more details related work.
has a complexity of 0(1/44, which is suboptimal for se- quential data often consisting of long token sequences (Go- dahewa et al., 2021; Grešová et al., 2023), and state-space models featuring subquadratic complexity (Poli et al., 2023; Nguyen et al., 2023).
3. Token merging
Therefore, we propose local merging a superset of token
Despite recent advances efficient transformers, processing long input sequences still induces considerable memory requirements and computational effort. To address this, we first extend token merging, which successfully boosts throughput in computer vision, to time series models. Next, we propose local merging, a domain-specific and efficient token merging algorithm for state-space models and long sequence processing. Finally, we introduce causal merging as special case of local merging to allow for token merging in decoder architectures and propose dynamic merging to further improve token merging in real-world settings.
merging - by introducing & E N, < < 41/2 as a local- ity constraint where we compute the similarity only on a local subset of tokens:
S = {Mis 1 1 < 1//2, 1 - jl < k).
(1)
Figure 1 illlustrates the proposed merging algorithm. The locality constraint reduces the complexity to:
(tt//2 + (k- - 1)(4 ( R)).
(2)
Varying the locality, we achieve linear complexity by considering only neighboring tokens for merging up to quadratic complexity by considering a global merging pool, possibly exploiting more redundancy. For efficient com- putation, we refactor Sloc into a rectangular tensor. An upper bound for the resulting speed-up can be given by speed up < 3 4 (1" 1)-1. The acceleration of
Global token merging for time series Let a neural net- work f(x) = 22 a LL-1 ..... (x)) consist of L layers denoted +;,, where each layer takes the output of the pre- vious layer as input. We assume that the input X; E R consists of & tokens with dimension d. Thereby, the input tokens are generated by tokenizer g: R= + Rixd out of 2- dimensional input data u. We assume the input u E mmx consists of m time stamps with n variates, and m n = 2. To improve the computational efficiency of token- based time series models, we extend global token merging from computer vision to the time series domain. Following Bolya et al. (2023), we combine the most similar tokens in each layer, reducing the tokens to be processed in layer [+1 to 44+1 = 1 r. For this, we split the set of all tokens into two disjoint subsets A. B in alternation to avoid merging con- flicts and allow for a parallelized computation of merging correspondences. Here A and B contain 2 elements each, denoted as a, and b, respectively. We compute the cosine similarity between all tokens in both subsets S (sty) and merge the top r most similar correspondences by averaging the tokens accordingly. This results in global token merg- ing algorithm with quadratic complexity. Lastly, Bolya et al. (2023) use a fixed r to enable batch processing without needing to pad individual batch elements to the same shape after token reduction. Later, we introduce more adaptíve merging schemes through dynamic merging.
deeper models is expected to increase as more subsequent layers can profit from already merged tokens. Local merg- ing additionally preserves order and locality as an inductive bias for sequence processing.
Some time series transformers use processing mechanisms that require a minimum number of tokens in the forward pass. To universally enable token merging in these archi- tectures, we further introduce q as the minimum number of remaining tokens. When encountering odd numbers of tokens 4, we exclude the most recent token from merging, as we expect it to contain the most relevant information fol- lowing the Markov assumption. We derive the complexity of the token merging procedures in appendix B. and further discuss the interplay of time-series-specifee inductive biases and token merging in appendix B.2.
Causal token merging for decoders Existing merging schemes are not suitable for causal operations, as global token merging transfers information over arbitrary ranges. To remedy this limitation and enable token merging in trans- former decoders, such as for recent decoder-only foundation models (Das et al., 2023) and encoder-decoder architec- tures (Ansari et al., 2024), we propose a special case of local merging: By restricting the merging neighborhood to only adjacent tokens with 1. local merging preserves temporal causality.
Local token merging for time series In this work, we design new token merging mechanisms for time series ar- chitectures and demonstrate run-time and even performance improvements over various datasets and models.
Token merging reduces the number of tokens to be pro- cessed throughout the model. However, many architectures require a fixed number of decoder output tokens or fixed dimensions for linear projection output layers. To main- tain a constant output dimensionality while merging tokens to speed-up the decoder, we unmerge all tokens in a final
Previous work on token merging in image processing ex- plored global merging schemes, where every token of each subset A and B could be merged with each other (Bolya et al., 2023; Bolya & Hoffman, 2023). However, computing the similarity S E R0/2 x t/22 between both sets of tokens
3

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
step. Coherent with our causal merging operation, we clone previously merged token into two neighboring identical ones, to unmerge it. Bolya & Hoffman (2023) propose unmerging algorithm for computer vision. However, they only leverage non-causal global token merging. Moreover, they immediately unmerge after every merge, making it unsuitable for long sequence processing, as it is unable to utilize the cumulative effect of reducing tokens.
et al., 2023) based on the Hyena architecture (Poli et al., 2023) and Mamba (Gu & Dao, 2023) models with the same hyperparameters as Hyena. We use a large input length of m = 16 000 nucleotides, utilizing state-space models' subquadratic complexity.
A Applying local merging Allowing self-attention to transfer information between tokens before merging them is benefi- cial in our experiments. Therefore, we apply local merging after the self-attention in transformer architectures as Bolya et al. (2023). In state-space models, we merge tokens after the Hyena or Mamba operator.
Dynamic token merging A fixed merging scheme allows for batch processing without needing to pad individual time series to the same length. However, it enforces a constant r among layers and batches, which might not always be optimal, as dissimilar tokens might be merged. We intro- duce dynamic merging to further improve token merging in real-world settings, where batch sizes are often small. To this end, we determine the number of tokens to be merged dynamically for every batch element using a token cosine similarity threshold. To avoid padding, we average them throughout the batch. As result, dynamic merging enables optimal merging rates that are adaptive the input data and current network layer.
Reproducibility of measurements We report all results on the same Nvidia A6000 GPU and do multiple measurements to achieve inference time standard deviations < 2 %.
5. Results
We first present our main results for local merging on pre- trained models and models trained with local merging. Next, we scale local merging to large foundation models and ex- plore token merging for state-space models and long se- quence processing. Finally, we investigate if we can gain even higher speed-ups in real-world settings, leveraging dynamic merging schemes.
4. Experiments
We systematically explore local merging in diverse settings on 5 time series datasets and 5 model architectures in dif- ferent sizes each. Additionally, we investigate local merging in large foundation models using Chronos in a zero-shot setting (Ansari et al., 2024). Finally, we demonstrate that local merging can be applied to state-space models for long sequence processing. See appendix C for more details on our experimental settings.
5.1. Local merging in pretrained models
We investigate local merging in both the encoder and de- coder on diverse time series transformer models with dif- ferent inductive biases. All models are trained on the target dataset and local merging is applied only during inference time, as accelerating already trained models is of high practi- cal relevance. We choose local merging hyperparameters as described in appendix C, selecting the fastest token merging trial on the validation set that is within a 0.01 increase in MSE compared to the reference without token merging. If we do not find trial with token merging satisfying these tight criteria, we report results without token merging, mim- icking how local merging might be applied in practice. We perform all selections on the validation set and report all results on the test set.
Datasets We use time series forecasting datasets including ETTh1, ETTml, Weather, Electricity and Traffic for our transformer experiments. For state-space models, we use the long-range Dummy Mouse Enhancers Ensembl dataset.
Model architectures For our main experiments, we use architectures, including Autoformer, FEDformer, In- former, Non-stationary Transformer, and the vanilla Trans- former (Vaswani et al., 2017) as reference. For each model, we evaluate local merging for different model sizes with L E (2, 4, 6, 8, 10} encoder layers, which we train doing hyperparameter optimization. We use an input length of m = 192, following the results of Nie et al. (2023), and a prediction horizon p- 96 samples. Longer sequences would generally benefit token merging.
The vanilla and Non-stationary Transformers have quadratic attention mechanisms, while the remaining architectures feature subquadratic attention complexities of (lo(()))())) for Autoformer and Informer and O(4)) for FEDformer. Re- gardless, our local merging in the encoder together with our casual merging in the decoder substantially increase the throughput of most models, up to 3.80x, often with no change in forecasting quality, table 1 shows. In some experiments, local merging even improves the MSE. In line with our formal analysis of potential speed-up from token merging in section 3, we generally observe higher accel- erations for larger models, as more subsequent layers can
For experiments on the foundation model Chronos, we use the default input length of m 512 and prediction horizon p 64 (Ansari et al., 2024). We compute the median from Chronos probabilistic forecasts and report the MSE. For our experiments on state-space models, we use Hye- naDNA medium, a genomic foundation model (Nguyen
4

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
profit from already merged tokens. Independent of model size, local merging finds Pareto optimal points in 17 of 25 settings and has no negative effect in the remaining cases. In some cases, we do not find model with decent fore- casting quality satisfying our criteria. Here, token merging during test only has larger impact on model accuracy, such as for Autoformer on the Traffic dataset. We address this issue when training with token merging section 5.2.
Token merging T
Accelcration x-fold
Test only Training merging
Test only Training
MSE
2-
0.65
0.60
0.4
0.6
10 Number of layers L
Inference time [ms]
(a) Non-stationary 6 layers Traffic (b) Autoformer Traffic
5.2. Local merging during training
Figure 2: (a) Training with different token merging Ttrain fractions compared applying token merging only during inference. (b) Models that showed high MSE degradation with token merging without training show high accelerations while maintaining MSE when enabling token merging during training.
Here, we apply local merging during training to reduce the models' sensitivities to the algorithm at inference time. As shown in figure 2, models trained with token merging often outperform those trained without it, even if token merging is not applied during testing. This approach enables us to accelerate models such as Autoformer on the Traffic dataset without sacrificing accuracy, which was previously not fea- sible when applying token merging only during inference. Additionally, local merging accelerates the training process itself by up to 2.27x for Autoformer on the Traffic dataset.
tational requirements. Therefore, accelerating foundation models without the need for additional fine-tuning is espe- cially important. Thus, we investigate local merging for foundation models on Chronos, univariate probabilistic model, in a zero-shot forecasting setting (Ansari et al., 2024) and apply local merging only during inference.
In all our experiments, we find Pareto optimal points with token merging. For four out of five datasets, local merging improves both accuracy and throughput simultaneously (see appendix D.I). Our results demonstrate that it is often bene- ficial to choose larger Chronos model with token merging over smaller one without, in figure 3. We report our results in table 2, choosing the best model without token
5.3. Scaling to large models
Foundation models are getting more relevant across do- mains, including NLP (Touvron et al., 2023), computer vi- sion (Kirillov et al., 2023), and time series processing (Das et al., 2023). However, these models have high compu-
Table I: Local merging accelerates various pretrained transformers of different sizes on several multivariate time series datasets. Merging induces minimal change quality (MSEa) compared to the reference without token merging (MSE).
Transformer
Autoformer
FEDrormer
Informer
Non-stationary
Dataset
Layers
MSE
Accel.
MSE
MSE
Accel.
MSE
MSE
Accel.
MSE
MSE
Accel.
MSE
MSB
Accel.
MSE3
0.75
1,38x
0%
0.42
1.00x
0%
0.38
1.29x
0%
0.87
1.40x
0%
0.55
1.36x
0%
4
0.71
1.81x
0%
0.40
1.39x
1%
0.39
1.74x
0%
0.92
1.30x
1%
0.47
1.82x
2%
ETTbl
0.66
2.33x
0%
0.11
2.12x
0%
0.38
2,27x
0%
0.93
2.39x
0%
0.16
2.39
0%
0.84
2.90x
0%
0.41
2.68x
-5%
0.39
2.81x
0%
1.23
2.20x
9%
0.48
x.93x
0%
10
0.69
3.51x
0.39
3.14x
0%
0.38
3.36x
0%
1.16
2.45x
4%
0.57
3.56x
0%
0.32
1.35x
0%
0.44
1.00x
0%
0.36
1.00x
0%
0.65
1.40x
0%
0.42
x.30x
0%
0.58
1.85x
2%
0.43
1.00x
0%%
0.37
L.76x
2%
0.60
1.78x
%
0.48
1.72x
ETTml
0.62
2.11x
4%
0.45
1.00
0%
0.38
1.00x
0%
0.59
2,16x
16
0.38
2.52x
0%
0.60
3.09%
1%
0.58
2.60
0%
0.33
1.00x
0%
0.61
1.61
0%
0.46
2.10%
-2% -2%
10
0.62
3.72x
0%
0.54
1.69
0%
0.36
1.00x
0%
0.57
1.00x
0%
0.41
3.80x
0%
0.25
1.11x
1 %
0.28
1.10x
0%
0.27
1.37x
2%
0.35
1.13x
1 1
0.19
1.46x
1%
4
0.28
1.95%
0 %
0.24
1.00
0%
0.26
1.74x
0%
0.24
1.89
2%
0.19
1.95x
0%
Weather
0.28
2.19x
9 %%
0.26
2.03x
2%
0.27
2.42x
0%
0.21
2.19x
2%
0.20
2.54x
0%
0.32
.200
%
0.26
1.56%
4%
0.27
2.88%
0%
0.30
1.56x
1 1%
0.20
3.14x
0%
10
0.35
2.19x
0.26
1.72x
3%
0.24
1.00x
0%
0.31
1.69
%
0.19
3.70x
0%
0.25
1.30x
0%
0.18
1.00x
0%
0.20
1.24%
0%
0.30
1.23x
%%
0.17
1.31x
0%
0.26
1.75x
0%
0.19
1.00x
0%
0.19
1.61x
0%
0.30
1.60x
%
0.17
1.73x
1%
Electricity
0.25
2,29x
0%
0.19
1.00x
0%
0.20
2.22x
0%
0.29
1.00x
0%
0.17
2,26
0%
0.25
2.84x
0%
0.19
1.00x
0%
0.20
2.72x
0%
0.31
1.00x
0%
0.17
2.76x
0%
10
0.25
3.31x
0%
0.18
1.00x
0%
0.20
3.33x
0%
0.30
1.00x
0%
0.18
2.53x
7%
0.66
1.28%
1%
0.63
1.00x
0%
0.59
1.21x
0%
0.68
1.19x
6%
0.00
1.27
2%
4
0.66
1.56x
3%
0.60
1.00x
0%
0.58
1.65x
0%
0.68
1.00x
0%
0.59
1.68x
1%
Traffie
6
0.64
2.13x
1%
0.61
1.00x
0%
0.57
2.10x
0%
0.69
1.00x
0%
0.62
1.58x
2%
0.68
2.67%
0%
0.60
1.00x
0%
0.59
2.61 x
0%
0.71
1.00x
0%
0.59
x.69
1%
10
0.67
3.25x
-1% 1
0.59
1.00x
0%
0.58
3.12x
0%
0.69
1.00x
0%
0.59
3,16x
0%
5

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
0.65
0.50
1.0
tiny Chronos mini
0.45
Ratio of tokens merged
0.60
Chrenos small No merging
0.8
0.40
MSE
0.55
MSE
0.35
-06
0.50
0.30
-044
0.25
0.45
0.2
0.20
0.15
0.00
0.02
0.06 Inference time [s]
0.04
0.08
0.10
0.00
0.02
0.06 Inference time [s]
0.04
0.08
0.10
(a) ETThl
(b) Electricity
Figure 3: MSE for different token merging in Chronos models during zero-shot testing on two datasets. Choosing larger models with token merging is beneficial compared to smaller ones without.
merging as reference. We illustrate two cases: 1) Sclecting the token merging setting that provides the best MSE, and 2) selecting the setting with the fastest throughput. For 2), we constrain the MSE of token merging trials be lower than the second-best model without token merging, In addition, we allow maximum increase in MSE of 3 % compared to the reference. In our experiments, we can improve Chronos MSE by up to 9 % and speed-up inference by 54.76x.
illustrates that local merging achieves substantially larger speed-up and better accuracy than global merging. This experiment indicates that architecture and domain-specific biases are important when applying token merging. Local merging accelerates HyenaDNA up to 3.62x with 4.9 % decrease in accuracy, whereas global merging substantially reduces the accuracy by 9.5 %. Utilizing less aggressive merging schemes, local merging even boosts accuracy by 1.7 % while still accelerating HyenaDNA 1.68x. x In Mamba models, local merging achieves even higher accelerations up to 4.09 x. In experiments, the similarity computation of local merging adds 14 % of additional execution time to every Hyena block. For global merging, however, this is substantially higher with 68 %, highlighting the importance of local merging's s linear complexity. To our knowledge, this is the first study of merging individual states in state-space models to improve their sequence modeling performance.
Table Local merging accelerates all Chronos foundation models from tiny to large during zero-shot forecasting. Applying local merging, we aim for two objectives: best MSE and fastest acceler- ation. Among Chronos models, choose the best without token merging reference (MSE). As local merging improves MSE (negative MSFa) ) while speeding up models, we are able choose small models while surpassing forecasting quality of larger ones.
Best
Fastest
Dataset
MSE
Accel.
MSE
Accel.
MSE
ETThl
0.45
14.17x
-6%
32x76x
2%
Table 3: Comparison global and local token merging for Hyena and Mamba on the long sequence Dummy Mouse Enhancers En sembl dataset. Best, sccond.
ETTml
0.41
1.23x
-4%
6.47x
3%
Weather
0.17
1.16x
-1%
54.76x
3%
Electricity
0.14
1.02x
0%
2.91 x
3%
Traffic
0.61
1.16x
-9%
2.91x
1%
Hyena
Mamba
Token merging
Accel.
Accuracy
Accel.
Accuracy
No merging
1.00x
78.9
1.00x
76.0 %
5.4. Token merging in state-space models
Local merging fastest
3.62
74.0 %
4.09x
74.4 %
State-space models can process very long sequences with millions of tokens due to their subquadratic complexity. Our proposed local merging algorithm is specifically designed to match this subquadratic complexity, enabling effective token merging in state-space models. Additionally, it preserves locality and order as inductive bias for sequence processing. We compare local and global token merging in Hye- naDNA (Grešová et al., 2023) and Mamba (Gu Dao, 2023), for two objectives: the largest speed-up and the best prediction quality. We use a classification task, where the data consists of long genomic sequences with 16 000 nucleotides each. Our local merging with & = featur- ing linear complexity and locality bias outperforms global merging with k = t//2 and quadratic complexity. Table 3
Local merging best
1,68x
80.6 %
1.65x
76.0 %
Global farting
2.93x
69.4 %
2,81x
74.0 %
Global merging best
1.15x
80.2 %
1.27x
76.4 %
5.5. Dynamic token merging
Dynamic token merging mitigates the issue of dissimilar tokens being merged, potentially improving quality. Here, leverage the single-sample explore dynamic merging in optimal conditions. From practical perspective, this case might be relevant for on- device applications like smartphones or automated driving. We further utilize small batch sizes of 10 elements as they might appear in real-world applications.
with batch size 1 to
6

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
In figure 4, we compare token merging utilizing fixed r to dynamic merging, varying the cosine similarity threshold. Dynamic merging improves quality slightly in most settings. With batch size 10, it is marginally worse compared to the optimal single-sample case. Therefore, we suggest using a fixed merging schedule for applications with large batches and dynamic merging for cases with few batch elements. There is no equivalent r' to dynamic merging schedules as they are similarity- bbased and strongly layer-dependent. We report FLOPs as we observe substantial execution overhead in time measurements.
0.80
1.0
Layers Layens Layers No merging
Ratio of Tokens menger
+2
MSE
066
00300-0--
0.4
0.70
0.2
0.65
0.0
0.2
0.4
0.6
Inference time [ms]
Figure 5: Transformer models on ETThl show constant MSE, independent of the amount of token merging r.
Nevertheless, we find that in most cases, these models still provide reasonable forecasts. In our experiments, trans- former models trained on larger or more complex datasets containing more variates do not show this behavior. We ar- gue that this might be a limitation of transformers on small time series datasets (Zeng et al., 2023; Li et al., 2023). Still, token merging successfully improves the throughput while maintaining accuracy for these models.
Fixed merging
0.55
Dynamec merging, le
Dyrsmme merging, 10
MSE
0.50
0.45
Inference FLOPs
7
1e10
Token merging increases forecasting quality, most prominently in Chronos models, as in figure 3a. We explain this behavior in section 6.2.
Decreasing MSE
Figure 4: Comparison of dynamic merging based on similarity
threshold with fixed r merging for Chronos small on ETTh1.
6.2. Improvement of forecasting quality
We explore if the potential benefits of token merging can be predicted through dataset- and model-specific properties without requiring downstream task evaluation.
6. Further investigations
We first explore different local merging outcomes regarding acceleration and forecasting performance. Next, we find dataset- and model-specific properties explaining why, in some cases, local merging can improve forecasting qual- ity. Lastly, we investigate if we can achieve same effects on model acceleration and forecasting performance with simpler methods than local merging. We further explore dif- ferent token similarity measures in appendix E. compare merging with pruning in appendix E.2, and investigate the influence of tokenization on local merging in appendix E.3.
Selective low-pass filter We hypothesize that local merg- ing improves forecasting quality by selectively reducing noise. Averaging similar tokens smoothes the time series, acting as an adaptive low-pass filter. To validate our hypoth- esis, we low-pass filter the input time series using Gaussian kernels without token merging in figure 6. On E both local merging and Gaussian filtering improve the MSE On the Electricity dataset, token merging and Gaussian filtering do not positively impact the MSF. All of these observations are in line with our hypothesis. Applying token merging together with the Gaussian kernel leads to the best results. Other averaging kernels were significantly worse. We show additional results in appendix E.4.
6.1. Merging outcomes
We observe three distinct merging outcomes when combin- ing tokens in transformer architectures.
Dataset properties We find properties of the target dataset that are particularly amenable to token merging. Using met- rics from signal processing, we can predict how well local merging will perform on new dataset prior to evaluation. Improvement in forecasting quality due local merging in table 2 correlates with the spectral entropy of the dataset. Specifically, local merging achieves higher quality gains on high entropy datasets, such as ETTm1, ETTh1, and Traffic (see table 4). We argue that local merging removes unnec- essary information from complex signals with high entropy using its selective smoothing ability. This allows the model to focus on only the relevant patterns of a signal and to
Increasing MSE As the number of merged tokens in- creases, the MSE increases almost monotonically (see fig- ure 3b). This behavior can be explained due to a loss of information when combining multiple tokens and also oc- curs in the vision domain (Bolya et al., 2023).
Constant MSE For the vanilla Transformer on ETTh1 and for FEDformer on ETTh1, Weather, Electricity, and Traffic, we observe a constant MSE when applying token merging as shown in figure 5. For the Transformer model, we find all tokens to be similar after the first attention block. Thus, token merging does not affect the model performance.

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
0.40
Table 5: Quality degradation due to local merging of models with
0.48
No merging Gousssan filter Token menging:
different token representations.
0.35
MSE
0.46
MSF
0.30
Glusssan miigggg
Model and dataset
MSF
Token similarity
0.25
Informer Layers Traffic Informer 4 Layers Electricity
6%
0.10
7%
0.22
Informer 8 Layers ETThl
%
0.28
0.40
1.5
Informer 6 Layers Weather Informer 6 Layers ETTml
2 %
0.35
Gaussian filter o
Gaussian filter a
-1 %
0.40
(a) ETTh1
(b) Electricity
Non-stationary Layers ETTh1
0%
0.77
Non-stationary Layers ETTh1
0%
0.82
Figure 6: Comparison of low-pass filtering the input time series with a Gaussian filter and token merging for Chronos small. The Gaussian filter has a similar effect on MSE as local merging, sup- porting our hypothesis that local merging selectively low-pass filters data. Besides improving MSE, local merging accelerates models unlike Gaussian filtering.
Non-stationary Layers Weather
0%
0.87
Transformer Layers ETTml
0%
0.99
better comparison, we keep the predicted time series snippet fixed and only adjust the input sequence.
achieve better prediction quality. Besides the spectral en- tropy, the same correlation is evident in the total harmonic distortion (THD). Local merging adaptively low- low-pass tww-aas--ilterrs noisy distorted signals condense the most relevant pat- terns and effectively improves the signal-to-noise ratio. The greater noise in ETTm1, ETTh1, and Traffic compared to Electricity and Weather can also be visually inspected in the respective frequency spectrum in appendix E.5. Therefore, we expect a larger improvement prediction quality when applying local merging on high entropy signals with a low signal-to-noise ratio. While less significant, local merging still improves efficiency on low noise datasets, discuss in appendix E.6 in detail.
Our results demonstrate that varying the input length cannot replace local merging (see also appendix E.7). In figure 7, we investigate input length dependence for two objectives in more detail: First, we explore the token merging setup that leads the best MSE and compare the results to the model without merging. Here, local merging yields considerable throughput increases while improving predictive quality at the same time. Second, we compare the fastest model with token merging, which shows no quality decreases, to a standard model. We find models with local merging to scale favorable to long sequences.
We further explore the redundancy input tokens including the influence of the positional embedding in appendix E.
Table 4: Quality improvement due to local merging on datasets with different signal properties.
0.55
No merging Taken merging
Datasct
MSEA
Spectral entropy
THD
0.50
Acceleration
Inference time
Accoleration
MSS
0.2
ETTml
-4 %
1.64
70.23
0.45
1.4
ETTh1
-6% %
4.55
54.93
Traffic
-9 %
2.96
19.78
0.0
512
1024
1536
2048
0
512
1024 1536 2048
Electricity
0 %
2.21
15.77
Input lenght
Input lenght
Weather
-1 %
1.64
13.15
Best MSE
(b) Fastest execution
Model properties Across all datasets, we identify model- specific properties that benefit local merging. For this, we analyze the average cosine similarity tokens the models from table after the first transformer layer. Local merging accelerates models, such as the Non-stationary Transformer, that learn more similar token representations without quality degradation. For models with dissimilar token representa- tions, like the Informer, we observe quality degradations when applying local merging, as table 5 shows.
Figure 7: Effect of input length (a) forecasting quality and (b) inference time for token merging in Chronos small on ETTh1.
7. Conclusion
In this work, we explore token merging in the time series domain for the first time. We conduct an extensive empirical study on transformer architectures and state-space models in diverse settings using various models and datasets. We demonstrate that token merging can successfully accelerate pretrained models and sometimes even improve their predic- tion quality. We further introduce a domain-specific local merging algorithm with variable complexity and illustrate its effectiveness on Hyena and Mamba models. Addition- ally, local merging the first causal token merging scheme,
6.3. Dependencies on input length
Token merging effectively reduces the number of tokens in a transformer layer. Here, we explore we can achieve similar accelerations while maintaining the same prediction quality by varying the number of input samples m. For
8

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
which we successfully apply in transformer decoders. Fi- nally, we conduct several ablation studies investigate when token merging is most effective, including spectral properties of the analyzed dataset and model- and algorithm- specific properties. We hope that token merging will have a positive effect on reducing the resource consumption and environmental impact of time series models.
Cirstca, R.-G., Guo, C., Yang. B., Kieu, T., Dong, X., and Pan, S. Triformer: Triangular, variable-spocific attentions for long sequence multivariate time series forecasting-fin saalsttingfulllllt version. arXiv:2204. 137677774.777 2022.
Cowen-Rivers, A., Lyu, W., Tutunov, R., Wang, Z., Grosnit, A., Griffiths, R.-R., Maravel, A., Hao, J., Wang. J., Peters, J., and Bou Ammar, H. Hebo: Pushing the limits of sample-efficient hy- perparameter optimisation. In Journal of Artificial Intelligence Research, 2022.
Limitations In our work, we divide all tokens into two sets and restrict merging to occur only between tokens from dif- ferent sets. Future work can explore more flexible merging schemes for -peeseii architectures. Moreover, we do not conduct ablations on all possible hyperparame- ters due to the large number of architectures and datasets evaluated in this work.
Das, A., Kong, W., Sen, R., and Zhou, Y. A decoder-only foun- dation model for semessriis forecasting. urXiv:231011066 2023.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenhorn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S.. Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Impact Statement
Garza, A. and Seagentthalr-Canneeo M. arXiv:2310.0358. 2023.
Timegpt-1.
We demonstrate large accelerations and considerable quality gains throughout a broad range of time series architectures. Our local merging can improve training efficiency and accel- erate already trained models without any fine-tuning. This is especially important for emerging foundation models, which require considerable computational resources. We hope that local merging will contribute to more sustainable time series models and reduce their environmental impact.
Godahewa, R. W., Bergmeir, C., Webb, G. I., Hyndman, R., and Montcro-Mann P. Monash time series forecasting archive. In Neural Information Processing Systems Datasets and Bench- marks Track, 2021.
Goyal, S., Choudhury, A. R., Raje, S., Chakaravarthy, V., Sabhar- wal, Y., and Verma, A. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Con- ference on Machine Learning, 2020.
The results, opinions, and conclusions ex- pressed in this publication are not necessarily those of Volk- swagen Aktiengesellschaft.
Disclaimer
Grešová, K., Martinek, V., Čechák, D., Šimeček, P., and Alexiou, P. Genomic benchmarks: collection of datasets for genomic sequence classification. In BMC Genomic Data, 2023.
Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. In Advances Neural Information Processing Systems, 2023.
References
Ansari, A. F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P.,
Gu, A. and Dao, Mamba: Linear-time sequence modeling with selective state spaces. arXiv:23 2023.
Kapoor, S., Zschiegner. J., Maddix, D. C., Wang, H., Mahoney, M.,
M. W., Torkkola, K., Wilson, A. G., Bohlke-Schneidee and Wang. Y. Chronos: Learning the language of time series. arXiv:2403.07815. 2024.
Gu, A.. Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A.. and Ré, C. Combining recurrent, convolutional, and continuous- models with lincar state space layers. In Advances in Neural Information Processing Systems, 2021.
Bolya. D. and Hoffmann, J. Token merging for fast stable diffu- sion. CVPR Workshop on Efficient Deep Learning for Computer Vision, 2023.
Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations, 2022.
Bolya, D., Fu, C.-Y,, Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster. In International Conference on Learning Representations, 2023.
Kim, M., Gao, S., Hsu, Y.-C., Shen, Y., and Jin, H. Token fusion: Bridging the gap between token pruning and token merging. In IEEE Winter Conference on Applications of Computer Vision, 2024.
Bonnaerens, M. and Dambre, J. Learned thresholds token merging and pruning for vision transformers. Transactions on Machine Learning Research, 2023.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Repre- sentations, 2015.
Chen, M., Shao, W., Xu, P., Lin, M., Zhang, K., Chao, F,, Ji, R., Qiao, Y., and Luo, P. Diffrate: Differentiable compression rate for efficient vision transformers. In IEEE International Conference on Computer Vision, 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, I.,, Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollar, P., and Girshick, R. Segment anything. In IEEE International Conference on Computer Vision, 2023.
Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv: 1904. 10509, 2019.

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y--.., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems, 2019.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, L. u. and Polosukhin, I. Attention all you need, In Advances in Neural Information Processing Systems, 2017.
Li Z. Qi. S., Li, Y., and Xu, Z. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv2200.10721, 2023.
Woo, G. Liu, C., Sahoo, D., Kumar, A., and Hoi, S. Etsformer: Exponential smoothing transformers for time-series forecasting. 2xXi:2202.01388 2022.
Liu, S., Yu, H.. Liao, C., J., Lin, W., Liu, A. X., and Dustdar, S. Pyraformer: Low-complexity pyramidal attention for long- range time series modeling and forecasting. In International Conference on Learning Representations, 2022a.
Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., and Sa- hoo, D. Unified training of universal time series forecasting transformers, vrXiv:2402.02592 2024.
Wu, H., Xu, J., Wang. J., and Long, M. Autoformer: Decompo- sition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
Liu, Y., Wu, H., Wang, J., and Long, M. Non-stationary trans- formers: Exploring the stationarity in time series forecasting. In Advances in Neural Information Processing Systems, 2022b.
Liu, Y., Hu, T., Zhang. H., Wu, H.. Wang, S., Ma, L., and Long, M. Itransformer: Inverted transformers are effective for time series forecasting. wxXiv22310006660 2023.
Yun, S., Jeong. M., Kim, R., Kang, J., and Kim, J. Graph trans- former networks. In Advances in Neurul Information Processing Systems, 2019.
Marin, D., Chang, J.-H. R., Ranjan, A., Prabhu, A., Rastegari, M., and Tuzel, O. Token pooling in vision transformers. arXiv:21 10.038600, 2021.
Zeng, A., Chen, M., Zhang. I.,, and Xu, Q. Are transformers effective for time series forecasting? AAAI Conference on Artificial Intelligence, 2023.
Meng. L., Li. H.. Chen, S.n. S., Wu, Z.. Jiang. Y.-G., and Lim, S.-N. Adavit: Adaptive vision transformers for efficient image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Zhang. Y. and Yan, J. Crossformer: Transformer utilizing cross- dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations, 2023,
Nguyen, E., Poli, M., Faizi, M., Thomas, A., Wornow, M., Birch- Sykes, C., Massaroli, S., Patel, A., Rabideau, C., Bengio, Y., Ermon, S., Ré, C., and Baccus, S. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. In Advances in Neural Information Processing Systems, 2023.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J.. Xiong. H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Conference on Artifi- cial Intelligence, 2021.
Zhou, T.. Ma, Z. Wen, Q.. Wang. X., Sun, L., and Jin, R. Fed- former: Frequency enhanced decomposed transformer for long- series forecasting. In International Conference on Machine Learning, 2022.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-tem term forecasting with transform- ers. In International Conference on Learning Representations, 2023.
Zhou, T., Niu, P., wang, x., Sun, L., and Jin, R. One fits all: Power general time series analysis by pretrained Im. In Advances in Neural Information Processing Systems, 2023.
Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y.. Ermon, S., and Re, C. Hyena hierarchy: To- wards larger convolutional language models. In International Conference on Machine Learning, 2023.
Zhu, L. Thop: Pytorch-opcounter https: / /github.comm wyXeel///pttorchh--oCooott 2022.
Raposo, D., Ritter, S. Richards, B., Lillicrap, T., Humphreys, P. C., and Santoro, A. Mixture-of depths: Dynamically allocating compute in transformer-based language models. WxXiv:244.002258 2024.
Rasul, K., Ashok, A., Williams, A. R., Ghonia, H., Bhagwatkar, R., Khorasani, A., Bayazi, M. J. D., Adamopoulos, G., Riachi, R., Hassen, N., Bilos, M., Garg, S., Schneider, A., Chapados, N., Drouin, A., Zantedeschi, V., Nevayvaka, Y., and Rish, I. Lag-Ilama: Towards foundation models for probabilistic time series forecasting. 2rX 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F. Ro- driguez, A., Joulin, A.. Grave, E., and Lample, G. Llama: Open and efficient foundation language models. 13797 2023.
Tran, H.-C., Nguyen, D. M. H., Nguyen, M., Nguyen, T.-T., Le, N., Xie, P.. Sonntag, D., Zou, J. Y. Nguyen, B. T. and Niepert, M. Accelerating transformers with spectrum-preserving token merging, 2024.
10

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
A. Related work
Here, we discuss related work in greater detail.
Time series transformers In recent years, many transformer architectures with inductive biases for time series have been proposed, successfully outperforming classical and other deep-learning lappenggnnng methods in time series forecasting quality like recurrent neural networks (Li et al., 2019). Most of them focus on reducing complexity by modifying the attention mechanism. LogTTaan uses LogSparse attention (Li et al., 2019), while Informer focuses only on the most relevant queries using ProbSparse attention (Zhou et al., 2021). Additionally, many architectures adopt decomposition techniques to model trend and seasonal patterns (Woo et al., 2022: Wu et al., 2021; Zhou et al., 2022; Liu et al., 2022b). Autoformer leverages autocorrelation as a sequence-based similarity measure in the attention mechanism (Wu al., 2021). FHDformer uses the frequency domain to model time series effectively (Zhou et al., 2022). Non-stationary Transformers further mitigate the effect of the time series distribution changing over time (Liu et al., 2022b). Other works apply hierarchical attention (Liu et al., 2022a; Cirstea et al., 2022), embed subsequences as tokens to capture local semantic information (Nie et al., 2023), or leverage attention between time series variates to better model multivariate patterns (Zhang & Yan, 2023; Liu et al., 2023). Due to their success in the vision and NLP domain, transformer-bant foundation models have lately emerged for time series, often used in zero-shot settings. Many works focus on training transformers directly on large and diverse time series datasets, usually with billions of tokens (Garza & Cergenthaler-Canseee 2023; Das al., 2023; Rasul et al., 2023; Woo et al., 2024). Inspired by the success of foundation models in NLP, the recently proposed Chronos model converts continuous time series data into fixed vocabulary and is trained on both real-world and synthetic data (Ansari et al., 2024). Besides, other research branches focus on fine-tuning vision or NLP models for time series (Zhou et al., 2023) and on applying large language models directly on time series data (Gruver et al., 2023).
State-space models Due to the quadratic scaling of the attention mechanism. transformer architectures suffer from significant computational cost when processing very long sequences. Recently, state-space models have shown promising results in overcoming the quadratic complexity of transformers with respect to input length. Linear state-space layers solve the sequential processing requirement of RNNs through linear state-space representations (Gu al., 2021). The S4 model reduces memory requirements by conditioning the state-space matrix with a low-rank correction (Gu et al., 2022). By using implicit convolutions and a data-aware gating mechanism, Hyena (Poli et al., 2023) became one of the first state-space model architectures to match transformers on NLP tasks. Later work uses hardware-aware algorithms to improve the performance of state-space models on modern accelerators (Gu & Dao, 2023).
Reducing tokens Many works reduce the number of processed tokens to increase the efficiency of transformers in computer vision and NLP, often by pruning (Meng et al., 2022; Goyal et al., 2020). Marin et al. (2021) merge tokens in ViT architectures to reduce the loss of information associated with pruning. Bolya et al. (2023) enhance the token merging algorithm, which they successfully apply to already trained encoder-only models. Besides initial work on classification tasks (Bolya et al., 2023), subsequent work applies token merging to diffusion models (Bolya & Hoffman, 2023). Kim et al. (2024) combine merging and pruning, while other works investigate optimal merging and pruning rates (Bonnaerens Dambre, 2023; Chen al., 2023). Concurrent work adapts token merging preserve the spectral properties of the token space (Tran et al., 2024). However, their merging algorithm still has quadratic complexity, making it unsuitable for long sequence processing.
Sparse attention and token skipping Besides reducing the number of tokens, sparse attention (Child et al., 2019; Li et al., 2019; Zhou et al., 2021; Wu et al., 2021) and token skipping (Raposo et al., 2024) also decrease the computational requirements of transformer models. Sparse attention computes a subset of the attention matrix. Therefore, it can only accelerate the attention mechanism itself and not the subsequent MLP, in contrast to reducing the number of tokens during token merging. According to Marin et al. (2021), this MLP can take over 60 % of the total computation in a ViT layer. Further, altering the network architecture from full attention to sparse attention might require a retraining of the model. Concurrent work, such as token skipping (Raposo et al., 2024), involves the selection of subset of tokens to be processed in transformer layer. However, it has only been shown in NLP when training from scratch. In contrast to sparse attention and token skipping, token merging can accelerate already trained models and does not require any training data or fine-tuning. This is especially important for recent foundation models, which are expensive to train. In our experiments in sections 5.1 and 5.2, token merging successfully accelerates Informer and Autoformer, which already employ sparse attention. We therefore consider token merging as an orthogonal approach.
Here, we propose the first token merging algorithm for the time series domain, which extends beyond previous investigations of token merging in ViTs (Bolya et al., 2023; Bolya & Hoffman, 2023). We systematically evaluate the potential to reduce computational effort in memee--errs-ssseiiiie transformer architectures and space models.

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
B. Local token merging for time series
In the following, we provide derivations and more details on our local merging algorithm. We further discuss the interplay of seeeseeies--peccfic inductive biases and our token merging method.
B.1. Derivations
Here, we derive our theoretical results section 3.
Complexity of local merging To compute Stoe for local merging we need to compute the main diagonal of E g4/2 x t//2 and depending on & also secondary diagonals which are symmetrical but shorter than the main diagonal for h: > 1. We derive the complexity of local merging depending on & in the following:
complexity Stor
+
(p 1)
2
2
4
h
+ 2
2 - p
4 + 2
(k - 1)4
Σ P
2
p=1
(k 1)
(b: 1) k.
+ 2
2
2
11 + (k: 1)( k))
2
Merging speed-up bound We roughly estimate the upper bound of the speed-up we can achieve by merging tokens in L-layer transformer model. Therefore, we only consider attention due to its quadratic scaling with 4, We disregard additional effects reducing speed-up, such as merging overhead, to estimate the upper bound. Further, we assume merging half of the tokens each layer. The attention in the first layer is unaffected by merging, we apply token merging between the attention and MLP.
L
speed up <
2 +
2+)
2
2
+
26
2
L
11-1 (2)
T.
8
1 - y S+1
using geometric series
Σ
for
E/rO (2)
L 1)
1 1- =1)
3
(44 -1)-1
12

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
B.2. Inductive biases
Time series feature cemnain--pecific characteristics, including temporal causality, long sequences, periodicity, trends, and sparsity in the frequency domain. Here, we discuss the interplay of these properties with token merging algorithm.
Our local merging algorithm is specifically designed to exploit two core properties of time series: First, it preserves temporal causality, as real-world time series are generated by causal processes. Second, it maintains linear complexity as time series often consist of very long token sequences (Godahewa et al., 2021; Grešová et al., 2023). This way, we design a very universal token merging scheme, applicable to many model architectures and datasets, as we show in our experiments.
We conduct new investigations where we trace the tokens being merged throughout the transformer model and show that token merging can exploit periodicity and trends without explicitly modeling these inductive biases. As illustrated in figure 8b, our global merging for time series combines local and global information. However, we did not implement these properties as hard inductive biases to maintain the universality of our algorithm: This way, token merging also performs well on sequential data that does not exhibit trend periodicity, such as DNA sequences (Grešová et al., 2023), as we show in section 5.4. Stock prices typically also do not have regular periodic patterns. Further, introducing a periodic bias to the neighborhood our local merging algorithm would break causality, making inapplicable to decoders.
Autoformer and FEDformer transform the tokens the frequency space. Autoformer specifically focuses on the autocorre- lation. Here, token merging natively exploits sparsity in the frequency domain and autocorrelation space.
Our token merging algorithm can exploit inductive biases for time series, including periodicity, trends, and sparsity in the frequency or autocorrelation space, but it is not limited to those. This way, it is universally applicable to many architectures and datasets. Further, it features causality and low complexity as inductive biases for Id-sequence processing.
1.0
0.5
0.5
Value
Value
0.0
0.0
0.5
0.5
1.0
1.0
0
100
200
300
400
500
200
300
400
Time
Time
(a) Global merging, linear trend
(b) Global merging, periodic curve
1.0
1.0
0.5
0.5
Value
Value
0.0
0.0
0.5
0.5
-1.0
-1.0
0
100
200
300
400
500
0
100
200
300
400
500
Time
Time
(c) Local merging, linear trend
(d) Local merging, periodic curve
Figure 8: Global and local merging in Chronos base on data with linear trends and periodic patterns. Time series samples merged into the tokens throughout the transformer visualized in the color (top 3 tokens displayed). Local merging preserves locality and causality. Global merging combines local and global information, capitalizing on periodic patterns.
13

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
C. Experiments
Here, we list additional information concerning our experimental settings and resources.
Datasets We base our experiments on 5 commonly used multivariate time series datasets covering different forecasting applications: E7Th/ and ETTml consist of 7 variates measuring the power load and temperature of electric transformers in hourly and quarter-hourly granularity (Zhou et al., 2021). Weather consists of 21 meteorological quantities, such as air temperature, and is recorded every 10 minutes in 2020.' Electricity measures the energy demand of 321 consumers every hour (Godahewa et al., 2021). Traffic consists of 862 sensors in the San Francisco Bay Area measuring the road occupancy hourly (Godahewa et al., 2021). We use the same data splits for training, validation, and test as Wu et al. (2021) for consistency.
Since the Chronos foundation model operates univariately and requires considerable computational resources, we randomly sample the same 7000 time series from the test set for all Chronos evaluations. For the ETThl dataset, we do not observe relevant differences when comparing the results to the full test set in figure 9.
To explore token merging in an additional sequence-basse based domain and on second task, we use the Dummy Mouse Enhancers Ensembl dataset (Grešová et al., 2023) for classifying genomic data. It contains very long sequences of nucleotides from a mouse.
Applying token merging In our experiments, we generally find it beneficial to allow self-attention to transfer information between tokens before merging them. Therefore, we apply token merging between self-attention and the MLP in all transformer encoders as Bolya al. (2023). Many transformers exhibit quadratic attention, imposing considerable computational cost. As result, we do not find the token merging algorithm to introduce a substantial additional overhead. Thus, we choose k = tz/2 to profit from a global merging pool for transformer encoders. For our main experiments, we also apply our causal local merging with k- = 1 in the transformer decoders between self-attention and cross-attention and finally unmerge all decoder tokens. Therefore, we utilize different merging strategies in transformer encoders and decoders. In architectures utilizing additional tensors like attention masks or positional biases, we merge them using the same correspondences.
In state-space models, we merge tokens after the Hyena Mamba operator and choose k 1 to not introduce an operation with quadratic complexity into the architecture.
Hyperparameter optimization For each transformer architecture, model size, and dataset, we train 32 models without token merging doing hyperparameter tuning of learning rate and dropout using HEBO "Cownn-River et al., 2022). Here, we apply token merging during inference-time only. We choose the best model based on its validation MSE. We train 17 models with the found hyperparameters, the minimum possible (train and different uniformly spaced Tivain until all tokens are merged. We again choose the best model based on the MSE for further evaluation. We do 185 hyperparameter optimization trials of both chosen models, trained with and without token merging, using HEBO to find token merging inference hyperparameters Ntest and glest the validation set. Please note that and might be different for local merging in the encoder and causal local merging in the decoder. Finally, we evaluate once on the test set to report our results.
dyperparameters In table 6 we list the most relevant hyperparameters we used for training the transformer models, including the vanilla Transformer, Autoformer, FEDformer, Informer, and Non-stationary Transformer. For training and testing HyenaDNN (Nguyen et al., 2023) and for testing Chronos (Ansari et al., 2024), we used their default hyperparameters.
Reproducibility of measurements We report all results on the same Nvidia A6000 GPU. For training, we utilize Nvidia V100 and A100 GPUs We measure the end-to-end inference time of the models using 2 warm-ups and 2 measurement runs per batch. The standard deviation of the inference time is generally < 2 % in our experiments. Besides the inference time as practically most relevant quantity, we report FLOPs as a more hardware-independee measure using the thop library (Zhu, 2022). We choose the maximum possible batch size and standardize the results.
Computational effort We estimate the computational effort for reproducing our experiments in table 7. Please note that we base some of our experiments on model checkpoints acquired in previous experiments.
https:: //www, bgc- bgc-bena.......e......o.......ee mpg. de/wetter/
14

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
0.65
Chronos tiny
Chronos Chronos small 7000 subsample Full dataset
0.60
MSE
0.55
0.50
0.45
0.40
0.00
0.02
0.06
0.08
0.10
Inference time [s]
Figure Comparison of Chronos models subsampled ETThl dataset the full dataset.
Table 6: Hyperparameters for training the transformer models.
Hyperparameter
Value
Training
Seed
2024
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate
Search space loguniform| 10-6, 10-2
Learning rate decay
Exponential, 7 = 0.97
Dropout
Search space uniform[0.0, 0.25]
Batch size
32
Epochs
100
Early stopping patience
7
Loss
MSE
Model
Input length
m 192
Prediction horizon
p 96
Token dimension
d = 512
Encoder layers
L E (2, 6, 8, 10}
Decoder layers
Attention heads
8
MLP hidden dimension
2048
Activation
GELU
Table 7: Computational effort to reproduce our experiments.
Experiment
Accelerator
GPU hours
A6000
100
Local merging in pretrained models
V100
6720
A6000
50
Local merging during training
V100
3840
Scaling to large models
A6000
500
A6000
40
Token merging in sate- models
A100
12
Dynamic token merging Improvement of forecasting quality
A6000
140
A6000
30
Dependencies on input length
A6000
80
Redundancy of input tokens
A6000
15

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
D. Results
Here, we show additional results for our main experiments.
D.1. Scaling to large models
In this section, we show complete results on applying token merging to Chronos, a time series foundation model.
0.65
1.0
Chronos tiny
Chronos mini
Ratio tokens merged
0.60
Chronos small
0.8
Chronos basc
Chronos large
0.55
No merging
0.6
MSE
0.50
0.4
0.45
0.2
0.40
0.0
0.0
0.1
0.2
0.3
0.5 Inference time [s]
0.4
0.6
0.7
0.8
0.9
Figure 10: Token merging in different Chronos models on ETThl.
0.9
1.0
Chronos tiny
Chronos mini
0.8
Chronos small
0.8
Ralio of tokens merged
Chronos base
Chronos large
0.7
No merging
0.6
MSE
0.6
0.4
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Inference time [s]
Figure 11: Token merging in different Chronos models on ETTml.
0.23
1.0
Chronos tiny
0.22
Chronos mini
Chronos small
0.8
Ratio of tokens merges
0.21
Chronos base
Chronos large
MSB
0.20
No merging
0.6
0.19
0.4
0.18
0.2
0.17
0.16
0.0
0.0
0.1
0.2
0.5 Inference time [s]
0.6
0.7
0.9
Figure 12: Token merging in different Chronos models on Weather.
16

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
0.5
1.0
Chronos tiny
Chronos mini
Chronos small
0.8
Ratio of tokens merged
Chronos base
Chronos large
No merging
0.6
MSB
0.3
0.4
0.2
0.2
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Inference time [s]
Figure 13: Token merging in different Chronos models Electricity.
1.0
Chronos tiny
Chronos mini
Ratio of tokens merged
Chronos
0.8
Chronos hase
Chronos large
No merging
0.6
MSS
0.8
0.4
X
0.2
0.6
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Inference time [s]
Figure 14: Taken merging in different Chronos models Traffic.
17

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
E. Further investigations
Here, we show additional experiments and results.
E.1. Token similarity measures
Different distance measures can be utilized to determine similar tokens for merging. Here, we explore the 11 and 72 norms as magnitude- aware metrics and the cosine similarity measuring the angular distance. Our results show that the cosine similarity outperforms both the L1 and Lg norms marginally. Bolya et al. (2023) further ablate the similarity metric for the vision domain.
0.575
Cosine similarity
0.550
Ly norm
0.525
MSD
0.500
0.475
0.450
0.425
0.02
0.04
0.06
0.08
Inference time [s]
Figure 15: Different token similarity metrics in Chronos small on ETTh1.
E.2. Token pruning
Here, we compare merging tokens to pruning tokens. Generally, pruning is associated with higher loss information. This is also evident in our results in figure 16, where local merging outperforms local pruning.
Merging Pruning
0.55
MSB
0.45
0.04
0.08
Inference time [s]
Figure Local merging retains more information compared to local pruning, resulting better MSE Chronos small on ETThI.
18

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
E.3. Tokenization methods
Tokenization involves splitting the input time series into smaller units and embedding them in high-dimensional space. In experiments, utilize varying token representations: Transformer architectures in section 5.1 leverage multivariate tokens and continuous embedding space, while Chronos utilizes univariate tokens embedded into discrete space. Autoformer and FEDformer transform tokens to the frequency domain. In table 8, we further include PatchTST (Nie et al., 2023), which embeds fixed-length subsequences as tokens. While most architectures generate tokens from time series, Hyena and Mamba tokenize nucleotide sequences. We argue that the tokenization method of minor importance for token merging. Throughout all of our experiments, local merging consistently performs well on top of all token types.
Table Local merging accelerates PatchTST models. Speed-ups are line with table even though there are only tokens available for merging due to patching.
PatchTST
Dataset
Layers L
MSE
Accel.
MSEA
ETTh1
2
0.37
1.17x
2%
ETTml
2
0.30
1.17x
2 2%
Weather
2
0.16
1.98x
5 5%
E.4. Improvement of forecasting quality - selective low-pass filter
We find token merging to have a smoothing effect, improving MSE, and show our results on all datasets here.
0.48
merging merging" merging
0.35
MSF
0.46
MSE
0.30
MSE
0.44
0.25
0.42
0.5
Gaussian filter
Gaussian filter
Gaussian filter
(a) ETTh1
(b) Electricity
(c) Traffic
0.475
0.1800
MSE
0.450
MSF
0.1775
0.425
0,400
0.1725
2
Gaussian
Gaussian filter
(d) ETTml
(c) Weather
Figure 17: Comparing token merging to smoothing the input time series of Chronos small on different datasets.
19

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
E.5. Improvement of forecasting quality dataset properties
In the following, show the frequency spectrum of ETTh1, ETTmI, Weather, Electricity, and Traffic datasets.
400
1250
300
Magnitude
Magnitude
1000
200
750
500
100
250
1000
1500
2000
4000
6000
Frequency
Frequency
(a) ETTh1
(b) ETTml
2500
1500
2000
Magnitude
Magnitude
1000
750
1000
500
500
250
0
2000
4000
2000
Frequency
Frequency
(c) Weather
(d) Electricity
1250
1000
Magnitude
750
500
250
0
500
1000
1500
2000
Frequency
(e) Traffic
Figure 18: Frequency spectra of different datasets.
20

---

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
E.6. Redundancy of input tokens
Token merging exploits similarities in data. Intuitively, the number of tokens that can be merged without affecting predictive performance should depend on the redundancy of the tokens. We explore factors influencing the redundancy of imput tokens, including their number and positional embeddings. In the following, use Autoformer's time stamp positional embedding for our ablation.
First, we investigate whether scaling the number of input tokens increases average redundancy on the ETTh dataset. As demonstrated figure 19, the same relative number of tokens is merged for given merging threshold, independent of input length. Therefore, we suggest scaling the number merged tokens in each layer linearly with the input length. Positional embeddings add information about the location of token within a sequence. As result, two identical tokens without positional embeddings may show considerable differences when positional embeddings included, potentially preventing merging. However, figure 19 shows that this effect token merging is only marginal. It is worth noting that the attention of the transformer acts as high-din -igh-siimnsional low-pass filter, effectively generating more redundancy throughout the transformer layers, as Marin et al. (2021) show. Therefore, token merging not only relies on redundancy in the input data but also exploits redundancy that is introduced by the transformer itself.
Number of similar tokens
3000
Merging limit
0.99
Raw
Similarity threshold
Pos, embed.
2000
0.93
1000
0.90
0.87
2000
4000
6000
Number of input tokens
Figure 19: Relative number of redundant tokens for different similarity thresholds ETThl with and without added positional embedding.
E.7. Dependencies input length
Here show additional evaluation on applying token merging in Chronos models with different input lengths. is beneficial to choose larger input length with token merging smaller one without.
Ratto inegeed
0.55
1.00
1.8
Accele Teration
Inference time
0.4
No merging Token merging Acailratton
0,6
Input 128
Input 256
0.75
1.6
MSE
loput
0.2
marging
0.50
0.45-
0.25
1.2
s12
1024
1836
2048
0.0
512
1024
1536
2048
0.4
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.00
Input lenght
Input lenght
Inference time [s]
(a) Best MSE
(b) Fastest exccution
(c) Pareto plot
Figure 20: Varying the input length of Chronos small on ETThl.
21


Reviews:
[{"review_id": "9XTD05Jh7Z", "reviewer": "Reviewer_HVPc", "summary": "The paper proposes local token merging to improve transformer efficiency. Building up on Bolya 2023, the proposed method appear to compute similar tokens within a local neighborhood (as opposed to all to all) and merge them. There are other techniques mentioned in the text, but the writeup is not organized enough to point it out explicitly -- either in the introduction or in the methods section. The method and experiment description make is really difficult to identify and summarize the proposed concepts. \n\nThere seems to an attempt of evaluating on many datasets and compare with many models. But the unorganized and sloppy writeup makes it difficult to identify a single narrative or theme.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer HVPc,\n\nThank you for taking the time to read our paper. We would like to address your concerns in the following:\n\n**Q:** Write-up of the paper \\\n**A:** We are sorry, that you were confused by the write-up. We will rework the writeup of our Methods and Introduction section to point out our contributions more clearly.\n\n**Q:** Contributions \\\n**A:** Previous work in CV only used quadratic token merging in non-causal transformer encoders. We design token merging for time series:\n- Long sequences: Time series often consist of very long sequences, as opposed to CV (L122). Here, quadratic global merging introduces a significant computational overhead. We propose local merging with linear complexity. (see tab. 5, 16k long sequences).\n- Decoders: In CV, token merging is applied to non-causal encoders. However, for time series, endoder-decoder or decoder-only models are commonly used. Local merging is the first causal merging scheme for decoders. Further, causality is a good inductive bias for time series (tab. 5, L417-426), improving MSE. In contrast to images, real-world time series are generated by causal processes. \n- Forecasting: In CV, most works focus on classification where only 1 cls token needs to be preserved. We, however, use forecasting as a dense task where reducing tokens might be more difficult. To this end, we propose causal unmerging to restore a required number of output tokens.\n- SSM: Previous work only focuses on transformers. We are the first to merge tokens in ssm.\n- We propose dynamic merging to mitigate the issue of dissimilar tokens being merged. Previous work utilized fixed merging rates per layer.\n- Besides these technical aspects, our detailed analysis of local merging, such as why it improves MSE, is way beyond literature.\n\nDue to the page limit and the large amount of experiments, we had to move some aspects to the appendix. However, that seems to cause confusion. We will list our contributions more clearly in the Method and Introduction section.\n\n**Q:** No experiments regarding causality \\\n**A:** Preserving causality has two advantages. First, it enables token merging in transformer decoders. Our main experiment in tab. 1 therefore demonstrates causal merging implicitly as we merge tokens in the encoder and the decoder. Second, causality is a good inductive bias for time series. We show that when comparing local merging with k=1 to global merging in sec. 5.8 and tab. 5. Global merging can exploit a larger merging pool with more similar tokens. However, local merging is still better due to its causal locality bias (74% vs. 69% accuracy, L417-418, L424-432). Reviewers mSjb and 6Amp find our causality claim well supported. Thank you for pointing us to that. We will mention the link to causality more explicitly in these experiments.\n\n**Q:** The paper seems to be having difficulty taking a single stance on their claims. Lines 243~246, in both columns, seem to claim the proposed local merging improves both accuracy and efficiency. But Lines 354 ~356, some results in Table 1 imply reducing tokens would decrease accuracy. \\\n**A:** We investigate token merging from different points of view. Sometimes token merging improves MSE while accelerating the model at the same time. This is mostly true for chronos models in sec. 5.3 and L82-89. In other settings, token merging accelerates the model at some cost of MSE. This is mostly true for the models in tab 1. We find empirical and theoretical explanations for these different behaviors in sec. 5.4 and 5.5. We think that an evaluation from both points of view (finding the best MSE and finding the fastest model at some MSE cost) is very valuable. Both settings relate to different practical applications, i.e., finding the best or the fastest model. We argue that our objective evaluation is therefore especially valuable, rather than focussing on a single setting. We try to disentangle both settings in the write-up of the final paper.\n\n**Q:** Colors in figure 1 \\\n**A:** Thank you for pointing us to this figure. The colors illustrate how different merging distances correspond to entries in the similarity matrix and to different values of k, i.e., which merging correspondences are valid for a given k and which reduced similarity set has to be computed. To avoid further confusion, we will reduce the matrix in fig. 1 to 3x3 (tokens 1 to 6).\n\nWe hope we could address your concerns and we are happy to answer any follow-up questions. \nFurther, we have made several improvements to our work and present new results in our comments to review 6Amp.", "claims_and_evidence": "The text of the paper makes it difficult to understand a consistent and clear narrative of the claim.\n\nToken merging would reduce the computation of both transformer and state space models. This is intuitive and follows directly from the model design.\n\nThe paper seems to be having difficulty taking a single stance on their claims. Lines 243~246, in both columns, seem to claim the proposed local merging improves both accuracy and efficiency. But Lines 354 ~356, some results in Table 1 imply reducing tokens would decrease accuracy. \n\nI would stick to one narrative: either we say token merging reduces model  processing/time and the accuracy is not affected, i.e.. the accuracy drop is tolerable and sometimes even increases; or we say we fix the MSE at a certain level and show how much efficiency we can improve at this level. \n\nOtherwise, these two issues get tangled up very quickly and obscures the message the manuscript is trying to convey, e.g., Lines243~255 right column, Section 5.5., 5.6.", "methods_and_evaluation_criteria": "**Contribution:** I am not even 50% clear what are contributions of the paper. From Section 3, it appears the contribution is local similarity computation as merging. The section does not clarify other important details, e.g., how often tokens are reduced, by how much, and how the tokens were merged. From Line 175 right column, I am guessing the proposed method adopts the exact setting of Bolya 2023 for these. Is restricting the similarity and merging to local neighborhood the main contribution and sets the proposed method apart from Bolya 2023?\n\nIt seems like the manuscript also proposed merging technique for decoders, which I am guessing Bolya 2023 does not do as it is encoder only. Is this a novel idea? If so, why was it not describes in the methods section and deferred to supplemental? Is it not something that distinguishes this method from Bolya 2023? Or is it not important enough or incremental.\nIf it is indeed an important distinction, the experiment section should clearly show empirical evidences of its benefits under a separate section. Now, the reader has to search and find out the the experiments on chronos pertains to encoder-decoder merging technique.\n\nAlso, is Dynamic merging another technique that is novel and being proposed here? Then, why is it mentioned in page 7 under experiments section?\n\n**Exposition** The manuscript  appears to be too poor to convey even the appreciable findings or efforts from the authors. Apart from what already mentioned above, the title does not mention local token merging which is probably the primary claim the submission makes. Figure 1 confuses the readers more than it clarifies: are the numbers ids of tokens? If so, the matrix should be 8x8 if I understood correctly. How the different colors are illustrating anything about the technique? Line 056 seems to give an impression this is an analysis paper not a model design proposal. In Section 3 methods, it is not clear how token merging in computer vision is relevant for model description whereas the token merging for decoders, which sounds like novel, is pushed to supplementary. \n\n\n**Experimental results:** The experiment section looks like dump of \"all experiments we did\" rather than empirical evidences of the claims we made as contribution. See the comments in section above.", "theoretical_claims": "---", "experimental_designs_or_analyses": "---", "relation_to_broader_scientific_literature": "---", "essential_references_not_discussed": "---", "other_strengths_and_weaknesses": "---", "other_comments_or_suggestions": "I would rethink the whole approach, ponder on what is the approach the paper should suggest to improve how we use transformers or state space models for time series and reorganize accordingly. Right now, the only things the the readers take away from the submission is local token merging reduces computation -- which I dont understand why is not obvious already.", "questions_for_authors": "---", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8pv47IXDPA", "reviewer": "Reviewer_eLzT", "summary": "This paper proposes to apply Token Merging (ToMe), which was originally developed for vision transformers, to time series models.The main difference between the author’s work and standard token merging is the use of local neighborhoods, ie local merging. The original ToMe formulation allowed tokens from different regions to be merged together if they were similar enough, but this paper restricts the similarity comparison to small locl windows or even only adjacent tokens, which they find to be more efficient and better suited to time series modeling. The authors then show that this has similar results to base ToMe on standard time-series datasets and conduct ablations to demonstrate the value of the local merging. Due to the large number of input tokens in time series models, local merging leads to large speedups, with particularly large speedups compared to those shown in the original ToME paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer eLzT,\n\nThank you for taking the time to read our paper and for your valuable comments. We are happy to answer them in the following. \n\n**Q:** Local merging is a minimal contribution. \\\n**A:** We see our main contribution in investigating token merging for time series in great detail. Additionally, we propose new mechanisms to extend token merging to causal decoders, state-space models and long sequences.\n(Please see our Rebuttal on Review 6Amp for a full list of novelties)\n\n**Q:** Does linear complexity translate to wall-clock time? \\\n**A:** We think this is a great point. We compare global merging with local merging in sec. 5.8 and tab. 5. Using subquadratic ssm and a very long context of 16k tokens, we showcase the advantage of linear complexity. Computing the cosine similarity of all tokens in global merging has a significant overhead, resulting in only 2.92x speed up while local merging achieves greater acceleration of 3.62x. For this rebuttal, we investigate the wall-clock time in more detail. The similarity computation of local merging adds 14% of additional wall-clock time to each hyena block. For global merging, however, this is an additional 68% of wall-clock time. Thank you for pointing us to this missing detail, we will clarify that in our paper.\n\n**Q:** Is preserving causality important for time series? \\\n**A:** Thanks for bringing this up, you understood it correctly, and we will explain it more explicitly. Preserving causality has two important aspects: First, it enables token merging in decoder models, which are commonly used in time series processing. Previous work only focused on encoders. Second,  causality and order seems to be a good inductive bias for time series, as we show in tab. 5. Here, it is better to rely on a smaller merging pool while preserving causality (74% vs. 69% accuracy, L417-418, L424-432). Intuitively, real-world time series are generated by causal processes. \n\n**Q:** Gaussian filtering does not demonstrate that this is what token merging does. \\\n**A:** Gaussian filtering smoothes the time series acting, as a low pass filter. We argue that averaging two tokens also reduces noise, but in an adaptive manner as only similar tokens are merged. To validate our intuition, we compare local merging to gaussian filtering on 5 datasets. On datasets where gaussian filtering improves forecasting quality, local merging also improves quality. On Electricity, both gaussian filtering and local merging are not effective (fig. 4, 15). We further validate the low-pass-filtering hypothesis by correlating local mergings MSE improvements to dataset properties related to noise (see Dataset properties in sec. 5.4). Please note that we try to show that local merging acts as an adaptive filtering algorithm. Not that it is the same as gaussian filtering. We agree that our comparison is not absolute, but we gather lots of evidence that local merging can be compared to gaussian filtering.\n\n**Q:** Table 1, Autoformer and Informer have rows with no speed up. \\\n**A:** In these experiments, we apply token merging only during inference. This disturbs some models too much. Thus, we report the results without merging (see L228-233, L251-255). However, in sec. 5.2 we show that we can accelerate these models when applying token merging during training. This way the models can adapt to token merging.\n\n**Q:** The gaussian + merging line  missing from Figure 4 b. \\\n**A:** This is intentional, as you correctly guessed, the gaussian + merging line was exactly following the gaussian line. \n\n**Q:** Acceleration measure?\\\n**A:** Thank you for pointing this out. We base our acceleration on the wall-clock time (see L703) as we think it is the practically most relevant measure. However, not reported in the paper, we notice even larger accelerations in FLOPs in all experiments as they do not measure GPU overhead.\n\n**Q:** Important plots in the appendix. \\\n**A:** Due to the page limit we moved many results to the appendix. We will  restructure the final paper.\n\n**Q:** Ablate neighborhood size k. \\\n**A:** In our preliminary experiments we find that either k=1 (linear complexity, causality, locality bias) or k=tl/2 (global merging pool) leads to best results. In many transformer encoders that already exhibit large computational cost, we utilize k=tl/2 to profit from a global merging pool (L679-683). In decoders, ssm, and for long sequence processing we find local merging with k=1 to work best (sec. 5.8). Other values for k were suboptimal. We will add this crucial information to the paper. \n\n**Q:** How we chose hyperparameter r. \\\n**A:** We do 185 hyperparameter optimization trials to find the best r (see L692-694) satisfying our tolerated MSE drop (see L227-230) for every model. In fig. 2,3,4,5,7 r is plotted as the color bar and the different points in the diagram.\n\nThank you for also finding a typo. \nWe are happy to further discuss any open questions.\nWe present new results in our comment to review 6Amp.", "claims_and_evidence": "Claim 1. Local merging is better suited to time series data, since  is not quadratic complexity and does not rely on computing pairwise similarities between a large number of tokens. The original ToMe method is quadratic complexity.\n\nEvidence: It’s certainly true that the matching step of ToMe has quadratic complexity. What is less clear is whether this actually matters for wall-clock time. I am not particularly convinced it does, since in the original paper, measurements showed that the computation time of this step was negligible, especially compared to self attention. There is no clear analysis that demonstrates this point in the paper besides Table 5, which does not provide much detail.\n\nClaim 2. Local merging allows significant acceleration of time series models without reducing their performance. \n\nEvidence: This is well supported by experiments. \n\nClaim 3. Local merging preserves causality, which is important for time series data.\n\nEvidence: If I understand correctly, “causality” here means that any token T_i is only predictable from tokens directly before it. In this sense, i see that local merging does preserve causality. What is not clear is whether this actually matters for time series data; there is no ablation in the paper that clearly demonstrates this. \n\nClaim 4: Local merging improves forecasting quality by selectively reducing noise. \n\nEvidence: I don’t really see how applying a Gaussian filter demonstrates that Token Merging is reducing noise. I think the experiment here needs a bit more clarity; I elaborate on this in the experiments section.", "methods_and_evaluation_criteria": "The basic setup makes sense, which is testing several state of the art time series models that rely on attention mechanisms and measuring the downstream performance metric (MSE) and whether there is a speedup.\n\nHowever, it’s not clear what the “acceleration” measure is - is this wall-clock time, throughput, GFLOPS? Given the results from the token merging paper, it’s certainly surprising to see such larger improvements, so some clarity would be good here.", "theoretical_claims": "The only theoretical claims are related to the complexity of the proposed methods. These proofs are relatively straightforward and are included in the supplementary material, and seem reasonable to me.", "experimental_designs_or_analyses": "An issue I found is that many of the important plots that support claims in the analysis section were punted to the supplementary material. I think the paper could be better organized so that the core results were included in the main text and the supplement only contained additional results that would be helpful for building intuition. \n\nMain Experiment\n\nThe basic setup here makes sense, However, I don’t understand why the “global merging” (or standard ToMe) was not included here, since comparing local merging to it supports the main novelty in the paper. Secondly, I’m confused what “acceleration” means, and what measure is used here - is it wall clock time? Throughput? GFLOPS? All those base numbers should also be reported, and if that would take too much space, the results should be split into multiple tables. Finally, the Token Merging method (and local merging) depend on a parameter r that controls the number of tokens merged per layer. I dont’ see any discussion of this anywhere except in Figure 3; what was the \nAblation of local merging\n\nThere isn’t really an ablation study, since there isn’t many components. However, an important experiment to include would be the effect of increasing the neighborhood size k. There is no such analysis in the main text or the supplement which is an issue - the main point of the paper is that locality is an important property and this should be the main focus of the experiments apart from the main result.\n\nAdditional understanding of token merging improvement through signal processing.\n\nI think this set of experiments was interesting. The premise of this experiment is that local merging improves forecasting quality, which was not observed in the vision setting but seems to occur according to Figure 2 (which should be much larger, as it is one of the more interesting results of the paper). However, the experimental design to validate this doesn’t make that much sense to me. Applying Gaussian filtering does not demonstrate that this is what token merging actually does, and neither does showing that combining it with Gaussian filtering improves the MSE either. I may be missing something here from the explanation, so please feel free to correct me if my understanding is wrong.", "relation_to_broader_scientific_literature": "Token merging showed that for inputs with many redundant tokens, similar tokens could be combined together to get essentially the same results while significantly speeding up the model. This has been shown mostly in computer vision, but also in language models and vision-language case. This paper demonstrates that the effect occurs too in time series models. Furthermore, the paper’s analysis section attempts to explain why local merging actually improves the performance rather than keeping it the same as occurs in other merging works. This paper, if all experiments are properly conducted and the claims completely validate, would provide a clean way to accelerate time series models while also supporting the method with empirical intuitions.", "essential_references_not_discussed": "To my knowledge, the main references were all included in the related works section. Some papers that could also be discussed are those that learnably prune tokens but this is not essential.", "other_strengths_and_weaknesses": "Strengths: \n\nThe results of the paper are quite impressive. Most works on token merging do not show improvements to the base model, but just a speedup. This paper shows impressive speed-ups and a clear improvement in the downstream metrics. I also think that the idea to analyze the reason why local merging works better is great and if executed correctly would provide some key intuitions for others who use this method.\n\nWeaknesses: \n\nUltimately this paper is an incremental improvement to the original token merging paper, in the context of applying a known technique to a different domain with a small tweak to the original algorithm; there’s not a lot of novelty besides adding the locality.\n\nWhile I see the value in local merging for time series data, the change is extremely minimal - the method consists of restricting the window to small local neighborhoods and is a couple-line code change. \n\nThis would be a more valuable contribution if it were better motivated. In the original ToMe work, the matching process has almost negligible runtime; computing cosine similarity is extremely cheap. Is the complexity of this operation actually meaningful? Theoretical complexity does not really translate to wall-clock time which is what matters for running these models at scale; there is no analysis that justifies this part.\n\nFinally, as discussed in the experiments section, the experiments and analysis need to be cleaned up and clarified; I think a lot of important details are currently missing. In the rebuttal phase, I would like to see convincing explanations from the authors why the experiments as currently shown are sufficient for justifying the claims made in the paper.", "other_comments_or_suggestions": "Typo: L298 \"filer\" should be filter.", "questions_for_authors": "Questions:\nIn Table 1, why does Autoformer and Informer have lots of rows with no speedup? Was token merging not appilcable in these cases? Why or why not?\n\nThe gaussian + merging line seems to be missing from Figure 4 b (electricity). Is this intentional?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m71CGQ9rfO", "reviewer": "Reviewer_mSjb", "summary": "This paper introduces a novel local token merging algorithm for time series models aimed at reducing the computational burden of processing long sequences in transformers and state-space models. By merging tokens within local neighborhoods, the method scales the complexity from quadratic to linear while preserving causality—making it suitable even for transformer decoders.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mSjb,\n\nThank you for your valuable feedback and effort. \n\nThe \"Rough Transformers\" paper is very interesting and we will further discuss it in our related work section. Thank you for pointing us to that. We would like to emphasize that we see local merging as a method to accelerate a variety of models without needing to retrain or finetune them. Local merging is not only applicable to transformers but also to state space models. \"Rough Transformers\" on the other hand is a specifically designed model architecture.\n\n**Q:** Do the authors feel that similar techniques could be used in language? \\\n**A:** This is a great point. We do think that local merging might help to boost the efficiency of language models. Our causal merging and causal unmerging is the first token merging technique applicable to decoders, making it very valuable for decoder or encoder-decoder language models. Previous work in vision breaks causality and is not applicable to decoder models as used in NLP. For long contexts, local mergings linear complexity is particularly helpful.\n\n\nFurther, we have made several improvements to our work and present new results in our comments to review 6Amp. \nWe are happy to further discuss any open questions.", "claims_and_evidence": "I think that the claims made by the authors are supported by a variety of experiments. The idea of token merging seems to be effective at improving computational efficiency while retaining good performance.", "methods_and_evaluation_criteria": "The authors benchmarked on standard datasets used in the literature, and the evaluation is thorough.", "theoretical_claims": "This is mostly an empirical paper, so the authors did not make many theoretical claims.", "experimental_designs_or_analyses": "Yes, the experiments are sound.", "relation_to_broader_scientific_literature": "I think one of the key contributions of this paper lies in the fact that this can be integrated into pre-trained models like Chronos. I think this type of technique could become a widely used heuristic to be integrated into time series models.", "essential_references_not_discussed": "I think that while the authors claim that this is the local merging technique, they do not discuss a recent work that integrates both global and local information of the time series using path signatures [1]. I think that this should be included. \n\n[1] Moreno-Pino, Fernando, et al. \"Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching.\" Advances in Neural Information Processing Systems 37 (2024): 106264-106294.", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "None.", "questions_for_authors": "Do the authors feel that similar techniques could be used in language, or is token merging something exclusively usable in the context of time series?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pXLHmdiio7", "reviewer": "Reviewer_6Amp", "summary": "This paper introduces ​local token merging, a novel algorithm for accelerating time series processing in transformers and state-space models (SSMs). A domain-specific token merging method that computes token similarities within a constrained neighborhood (size k), reducing complexity from quadratic to linear (when k=1). This enables scaling to long sequences while preserving locality and causality. The work bridges token merging—previously limited to vision—to time series, addressing efficiency challenges in long-sequence modeling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6Amp,\n\nThank you for taking the time to read our paper and for your valuable questions. We are happy to answer them in the following. To this end, we conducted 5 new experiments. Please find anonymous results here: https://figshare.com/s/679d2c1d825228385b2d\n\n**Q:** Gap between token merging in CV and time series. \\\n**A:** Prior work in CV only used quadratic token merging in non-causal transformer encoders. We design token merging for time series:\n- Long sequences: Time series often consist of very long sequences, as opposed to CV (L122). Here, quadratic global merging introduces a significant computational burden. We propose local merging with linear complexity. (tab. 5, 16k long sequences).\n- Decoders: In CV, token merging is applied to non-causal encoders. However, for time series, endoder-decoder or decoder-only models are common. Local merging is the first causal merging scheme for decoders. Further, causality is a good inductive bias for time series (tab. 5, L417-426), improving MSE. In contrast to images, real-world time series are generated by causal processes. \n- Forecasting: In CV, most works focus on classification where only 1 cls token needs to be preserved. We use forecasting as a dense task where reducing tokens might be more difficult. To this end, we propose causal unmerging to restore a required number of output tokens.\n- SSM: Prior work focuses on transformers. We are the first to merge tokens in ssm.\n- We propose dynamic merging to mitigate the issue of dissimilar tokens being merged. Prior work utilized fixed merging rates per layer.\n- Besides these technical aspects, our detailed analysis of local merging, such as why it improves MSE, is way beyond literature.\n\nWe will highlight these differences in our paper\n\n**Q:** Dynamic merging in batch settings. \\\n**A:** Thank you for this nice suggestion. We conducted new experiments for dynamic merging in batched settings. To retain rectangular tensors, we average the number of tokens to be merged among the elements in a batch. For batch size 10 dynamic merging is still better than fixed r merging. Variable token counts per batch element do not hinder practical usage.\n\n**Q:** Would results hold on the full ETTh1 test set? \\\n**A:** We subsample 7000 time series for chronos experiments. We have made additional efforts to compare results of the subsampled test set to the full ETTh1 test set. Results also hold for the full set. \n\n**Q:** Section B.1 upper bound for speed up ignores overhead. \\\n**A:** You are correct. As the overhead varies between models and implementation, we derive this upper bound theoretically. For our experiments, however, we report the execution time rather than the FLOPs as it includes all overheads. \n\n**Q:** Fig. 9 uses only Autoformer. \\\n**A:** This is a good point. For this ablation, we only borrow the trained position embedding from Autoformer. It is quite similar to the one Transformer, Informer, FEDformer and Non-stationary use. The actual model has no effect. We therefore hope to make broad statements.\n\n**Q:** Merging and pruning. \\\n**A:** This is a great addition. We conducted new experiments to compare merging and pruning, which is used by DynamicViT. Merging results in better MSE, as it retains more information, and comparable speed up. Further, as opposed to DynamicViT, our pruning does not require any training and is zero-shot-applicable.\n\n**Q:** Tokenization and token merging \\\n**A:** In our experiments, the models use 3 different types of tokenization:\n- Models in tab. 1 use multivariate tokens\n- Chronos uses a discrete vocabulary\n- Hyena builds tokens from nucleotides\n\nAutoformer and FEDformer further use the frequency domain. Local merging works on top of all token types. We argue that the tokenization method is of minor importance. We conduct new experiments to include PatchTST. Preliminary results show, that local merging also works on patches. \\\n**Technical differences:** Patching compresses only the input data and requires model training (architectural change). Token merging is a general method that accelerates a variety of models in zero-shot setting. It exploits redundancy in every layer (L807-809, transformers generate redundancy).\nThanks for pointing us to this missing detail.\n\n**Q:** Mamba as SSM \\\n**A:** Thank you for this suggestion. We added Mamba for this rebuttal. Local merging achieves over 4x acceleration with only 1.6% drop in accuracy.\n\n**Q:** Lack of details for SSM merging \\\n**A:** We are happy to include more details. Do you have specific aspects in mind? We already analyzed the wall-clock time in more detail: The similarity computation of local merging adds 14% of additional wall-clock time to each hyena block, while global merging takes additional 68%. This highlights the value of linear complexity.\n \nThank you also for pointing us to sec. 5.4 spectral analysis. We will clarify \"signal entropy\" and \"noise\".\nWe are happy to further discuss any open questions.", "claims_and_evidence": "Most claims are well-supported.\nSpeedup metrics (Table 1, 2, 5) and FLOPs analysis (Appendix B.1) validate complexity reduction. Successful application to Chronos (decoder-only) and encoder-decoder models (Table 2) supports causality claims.\n\nIssues: While Figure 7 shows slight improvements, the paper does not compare dynamic merging to fixed r in batch settings, leaving practical trade-offs unclear. The upper bound in Section B.1 assumes merging half the tokens per layer, but real-world speedups depend on hardware and implementation (e.g., merging overhead).", "methods_and_evaluation_criteria": "Local merging is well-motivated for time series, where locality and causality are critical. Restricting merging to neighborhoods (k) balances redundancy exploitation and computational cost.\n\nChronos experiments use a subsampled test set (7000 series). While practical for compute constraints, full-test-set validation would strengthen claims.", "theoretical_claims": "The derivation of local merging complexity (O(t + (k−1)(t−k))) is correct. The upper-bound speedup estimate for L-layer transformers is reasonable but idealized (ignores layer-wise overhead).", "experimental_designs_or_analyses": "Results on 7000 samples may not generalize to full test sets.\nThe ablation (Figure 9) uses only Autoformer; broader validation across architectures would strengthen conclusions.", "relation_to_broader_scientific_literature": "The key contribution of this paper is a technique switch from CV to time series processing.", "essential_references_not_discussed": "​Adaptive Token Reduction: Liu et al. (2021, DynamicViT) dynamically prunes tokens in vision; comparison could highlight trade-offs between pruning/merging.\n​Time Series Tokenization: Nie et al. (2023, PatchTST) uses patching for efficiency; discussing how merging complements patching would contextualize contributions.\n​Recent SSMs: Mamba (Gu & Dao, 2023) achieves linear-time inference; analyzing merging in Mamba could strengthen SSM evaluations.", "other_strengths_and_weaknesses": "Strengths:\nFirst token merging method for time series, enabling causal merging and SSM acceleration.\nMassive speedups (54×) in Chronos, relevant for real-world deployment.\nConnects spectral properties to merging efficacy, providing actionable insights.\nWeaknesses:\nNo evaluation in batched settings or comparison to threshold-based pruning.\n Lack of code/details for SSM merging.\nLimited to Autoformer; broader validation needed.", "other_comments_or_suggestions": "Page 5, Table 1: “MSE△” formatting inconsistencies.\nSection 5.4’s spectral analysis could better differentiate signal entropy and noise", "questions_for_authors": "My main concern will lie on the kind of incremental motivation that applies token merging technique from CV to time series processing. What is the main gap between the application of token merge in these two domains (definitely existing), and how has this paper resolved this core issue?  Tehse clarifications should be highlighted.\nHow does dynamic merging perform in batched inference? Would variable token counts per batch element hinder practical usage?\nCould local merging accelerate Mamba? Testing this would broaden SSM applicability.\nWould results hold on the full ETTh1 test set?\n\nI am willing to raise my final rating after the rebuttal phase if the authors well-resolved my concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received mixed initial reviews, with one strong reject (#HVPc), one weak reject (#eLzT), and two weak accepts (#mSjb and #6Amp). After a detailed and constructive rebuttal from the authors addressing technical concerns, experimental clarity, and domain-specific adaptation of token merging to time series models, reviewers #mSjb, #eLzT, and #6Amp all raised their ratings to weak accept or accept. Reviewer #HVPc maintained a negative stance, primarily due to concerns about clarity and perceived lack of novelty, but ultimately expressed no objection to acceptance.

After carefully considering the paper, the reviews, and the rebuttal, the AC agrees with the majority opinion and supports acceptance. However, some concerns remain unresolved, particularly regarding the clarity of the presentation and the articulation of the paper’s core contributions. The AC encourages the authors to revise the manuscript thoroughly, addressing all reviewer feedback and improving the exposition in preparation for the camera-ready version.
Decision: Accept (poster)