Enhancing the Maximum Effective Window for Long-Term Time Series Forecasting
Authors: Jiahui Zhang
Zhengyang Zhou
Wenjie Du
Yang Wang
Venue: NeurIPS 2025 poster ()
Keywords: long-term time series
information bottleneck
Primary area: applications
TL;DR: We enhance long-term time series forecasting by proposing MEW, a new metric, and two modules (IBF and HTM) that help Transformers better use historical data.

Abstract: Long-term time series forecasting (LTSF) aims to predict future trends based on historical data. While longer lookback windows theoretically offer more comprehensive insights, Transformer-based models often struggle with them. On one hand, longer windows introduce more noise and redundancy, hindering the model's learning process. On the other hand, Transformers suffer from attention dispersion and are prone to overfitting to noise, especially when processing long sequences. In this paper, we introduce the Maximum Effective Window (MEW) metric to assess a model's ability to effectively utilize the lookback window. We also propose two model-agnostic modules to enhance MEW, enabling models to better leverage historical data for improved performance. Specifically, to reduce redundancy and noise, we introduce the Information Bottleneck Filter (IBF), which employs information bottleneck theory to extract the most essential subsequences from the input. Additionally, we propose the Hybrid-Transformer-Mamba (HTM), which incorporates the Mamba mechanism for selective forgetting of long sequences while harnessing the Transformer's strong modeling capabilities for shorter sequences. We integrate these two modules into various Transformer-based models, and experimental results show that they effectively enhance MEW, leading to improved overall performance. Our code is available at \url{https://github.com/forever-ly/PIH}.

Enhancing the Maximum Effective Window for Long-Term Time Series Forecasting
Jiahui Zhang Zhengyang Zhou! Wenjie Du Yang Wang
Universitty of Science and Technology of China, China
"Suzhou Institute for Advanced Research, USTC, China kongpingdmail ustc. edu.cn {zzy0929, duwenjie, angyan) edu.
Abstract
Long-term time series forecasting (LTSF) aims to predict future trends based on historical data. While longer lookback windows theoretically offer more com- prehensive insights, Transformer-based models often struggle with them. On one hand, longer windows introduce more noise and redundancy, hindering the model's learning process. On the other hand, Transformers suffer from atten- tion dispersion and are prone to overfitting to noise, especially when processing long sequences. In this paper, we introduce the Maximum Effective Window (MEW) metric to assess a model's 's ability to effectively utilize the lookback win- dow. We also propose models to better leverage historical data for improved performance. Specifically, reduce redundancy and noise, introduce the Information Bottleneck Filter (IBF), which employs information bottleneck theory to extract the most essential subsequences from the input. Additionally, we propose the Hlybrid-Transfonnn Mamba (HTM), which incorporates the Mamba mechanism for selective forget- ting of long sequences while harnessing the Transformer's strong modeling ca- pabilities for shorter sequences. We integrate these two modules into various Transformer-based models, and experimental results show that they effectively enhance MEW, leading to improved overall performance. Our code is available at wwwwwoowwwoooooo/o.oooowo 1y/PIH
model-agnostic modules to enhance MEW, enabling
Introduction
Long-term time series forecasting (LTSFF) holds significant importance across various domains such as traffic management, energy optimization, and financial analysis. Transformer-base meth- ods [31, known for their attention mechanisms that facilitate the automatic learning of sequential dependencies, have emerged as promising tools for LTSE.
Intuitively, extending the lookback window is natural choice enhance the forecasting capability of the model. This allows the model to capture long-term trends more accurately, thereby improving its ability to predict seasonal variations, cyclical patterns, and overall trends. However, prior research has found that current Transformer-ber based models are not effective in leveraging long lookback windows [4]. To quantitatively assess a model's 'ss efficiency in utilizing the lookback window, we propose new metric Maximum Effective Window (MEW); For a given model, as the lookback window increased while keeping other settings constant, there exists a point beyond which further increases in the window do not result in better performance. This point is referred to the model's Maximan Effective Window, which reflects the model's potential utilize historical information.
t: corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.52
Transformer
MSE (T- (T-720)
1.2
Ours
MSE (T=720)
0.50
0.48
1.1
0.46
PatchTST
1.0-
0.44
Ours
48
72
96
120
144
168
192
228
336
512
96
120
144
168
192 228 336 512 1024 1440
(a)
(b)
Figure 1: The changes in MSE with respect to the lookback window for Transformer and PStchTST on the E7Th/ dataset with a prediction length of T = 720. "Ours" refers to the integration of our modules into these two models. The triangular markers indicate the Maximum Effective Window (MEW). "Ours" increases the MEW from 72 and 512 to 288 and 1024, respectively.
As shown in Fig. (a), point-based Transformer (where single time point is treated token) only achieve MEW of 72 for the ETTh/ dataset. A natural question then arises: Can we increase a model's s MEW, thereby enabling
perform better with longer lookback windows? Before answering this, we first analyze the reasons behind the limitation of the MEW. From the information perspective. longer windows may contain more redundant signals (26), which can impede the model's learning process by allocating significant modeling capacity to meaningless or irrelevant noise signals [40]. From the architecture perspective, many studies have shown that Transformers suffer from issues such as attention dispersion and overfitting to noise 37. and these drawbacks are exacerbated with longer sequences.
The patch strategy 23 41
is an effective approach to improving the MEW. By treating consecutive time points single patch, this strategy significantly reduces local-level noise whihin patch. Additionally, it greatly reduces the number of tokens that the Transformer process, it helps mitigate issues such attention dispersion and overfitting to noise. As shown in Fig. (b). PatchTST achieves a MEW of 512, which is substantially higher than the 72 achieved by point-based Transformer. However, the patch strategy is heuristic and lacks adaptability. Firstly, it assumes redundancy and noise only within the points of each patch, which is a heuristic assumption that overlooks potential redundancy and noise between patches. Secondly, when the number of patches increases, the issues of attention dispersion and overfitting to noise in Transformers can still be exacerbated. Lastly, this approach is not suited for models that do not adopt a patch-based design.
In this paper, we propose two model-agnostic modules to address the issues of information redundancy and architectural limitations. As shown Fig. [ after integrating our modules, the MEW for PatchTST (patch-based) and Vanilla Transformer (point-based) increases from 72 and 512 to 288 and 1024, respectively, achieving improved performance.
Specifically, to reduce redundancy and noise in long lookback windows, we introduce the Information Bottleneck Filter (IBF) module based on Information Bottleneck (IB) theory. This module aims to identify informative subsequences while minimizing redundancy and noise [.]. enabling the model to prioritize significant subsequences within the lookback windows. However, directly optimizing the IB objective for sequences is challenging due to their discrete nature [39 38) often leading to training instability and degraded predictions. To address this, we propose probabilistic framework for sequence selection, coupled with a noise injection strategy. The core idea that important subsequences should have low probability of noise injection, whereas injecting larger noise into redundant sequences has minimal impact on predictions. By tailoring a noise prior for each input, the IB objective can yield manageable variational upper bound. To overcome the challenges of attention dispersion and overfitting to noise in Transformers, we introduce Mamba (7). state-space model (SSM) capable of selectively remembering historical information and filtering out noise. Recent studies have suggested that Transformers and SSMs complement each other in modeling [21 24]. Therefore, rather than replacing Transformers with Mamba, we propose a Hybrid-Transfoome Transformer-Mamts (HTM) architecture. First, Mamba captures long-term information while selectively filtering out noise from the extended lookback window. Then, the IBF further reduces noise. Finally, leveraging the temporal characteristics of the time series, introduce two sequence

---

splitting aigorithms-intee -interval split and block split- to divide the noise-filtered long lookback window into multiple short subsequences. These subsequences are processed by the Transformer to capture short-term dependencies. Experimental results show that HTM outperforms both pure Transformer and pure Mamba architectures terms of both computational overhead and predictive performance. In summary, our primary contributions are as follows:
We defined a metric MEW, which reflects the model's ability to leverage the window effectively. We analyzed how to improve MEW of Transformer-basee models from both the cooomatiion-thoreetie and architectural perspectives, and proposed two model-agnostic modules: IBF and HTM.
2. We integrated these two modules into multiple Transformer- models and conducted detailed experiments on seven datasets. The results demonstrate that these modules can enhance MEW while achieving better performance.
3. Notably, by incorporating these modules into the PatchTST model, we developed the PIH model (Patch-IBF- Becch-IB-HTTIE HTM), where the window length is extended to surpassinn the window settings of all currently existing non-LLM-bassee time series models, to the best of our knowledge. The PIH model achieved state-of-the-ar results, demonstrating the effectiveness of enhancing the model'ss windows. Our work paves the way for future research to explore even longer window sizes.
2 Related Work
Transformer-basee Models. Early attempts [28 19 II at directly applying vanilla Transformers to time series data failed in long sequence forecasting tasks, as the self-attention operation scales quadratically with the input sequence length. Existing approaches primarily address this challenge through two avenues. Patch-based methods, exemplified by PStchTST [23] and CrossFormer [4]], conceptualize consecutive time steps as patches, reducing the number of input tokens and augmenting local semantics mitigate redundancy. Another approach focuses sparse attention mechanisms. Models such as Informer [42], Autoformer [35], Pyraformer [17], and FEDformer
43 adapt the self-attention mechanism to achieve complexities of (I)) O log(T.)). These models rely specific designs and often sacrifice representational capacity, thereby compromising performance. Our work is independent of these approaches and can be effectively integrated into them.
Mamba for Time Series. Recently, several approaches have emerged to incorporate Mamba into time series modeling. Bi-Mamba+ [12] introduces novel Mamba+ block by incorporating forget gate within Mamba. This modification enables the selective combination of new features with historical ones in a complementary manner, boosting the model's ability to balance past and present information. S-Mamba 33 adopts a different approach by autonomously tokenizing time points of each variate using linear layer. The method employs a bidirectional Mamba layer to extract inter- variate correlations and a Feed- Ferd-ro Network to learn temporal dependencies. Tachinn 2 takes a broader view time series data by leveraging multi-scale contextual cues. Its architecture integrates quadraple-Mambe design, allowing the model to manage both channel-mixing and Shannel-indeeendenne scenarios. MambaTSS 4 challenges the necessity of causal convolution within Mamba for LTSF. It proposes the Temporal Mamba Block (TMB) as alternative. To further prevent model overfitting, MambaTS incorporates a dropout mechanism that selectively applies to TMB's s parameters, ensuring more stable and generalizable model performance.
Information Bottleneck (IB). The essence of the IB principle lies distilling a compact yet predictive code from the input signal (30). Pioneering work by [3 introduced the concept of variational information bottleneck (VIB), thereby enriching deep learning methodologies. Given random variables X and Y, IB aims to compress X into bottleneck random variable B, while retaining information pertinent to predicting Y:
min - BY: BI B
Here, 33 serves Lagrangian multiplier to balance the two mutual information terms. Presently, IB and VIB find extensive applications deep learning, predominantly in representation learning and feature selection domains [8] 33 36). Strategies such as injecting noise into intermediate

---

Doise
1BF
Transformer
peport
Patching Embeedding
Platten Lineur Head
Input Series
C1 C2
=
Output Series
Mamba
Noise Injection
Spiit
~ Bern(e;)
Transformet
E WWe, o
minI
noüse
min I moise Y)
Figure 2: Overview of the PIH. The input long sequence first undergoes patching and embedding to obtain e, which is then processed by Mamba to reduce redundancy and noise based on its selective mechanism, resulting in The IBF evaluates the importance of tokens the sequence z and adaptively injects noise to produce z n distribution with parameter c, and is random Gaussian noise. The tokens within the dashed box represent less important tokens, with corresponding A; values being smaller, thus z is primarily controlled by random noise correspond to larger Xy, with their values being mainly determined by their inherent semantics X,z After splitting z nhise into multiple subsequences, they are processed by the Transformer. Finally, the output passed through flattening and linear head obtain the prediction.
λz + (1 - AJe, where is sampled from a Bernoulli
Ag)c. Conversely, the important tokens within the solid box
representations of pre-trained networks and subsequently selecting regions with optimal information per dimension have been explored 11 27].
3 Method
We integrate HTM and IBF into the PatchT'T, resulting in the PIH model (B-ch- as illustrated in Fig.
Problem Definition. Given a collection of multivariate time series samples with lookback window I, == (x1:...,XXp Xr.,, where each x, at time step is a vector of dimension C, we aim forecast T future values
3.1 Patching and Embedding
Given our utilization of a channel-independent strategy (23]. we opt for simplicity by converting multivariate time series into univariate ones. The input univariate time series x E 1g initially segmented into patches, which may be either overlapping or non-overlapping. Employing patching strategies enhances locality and captures comprehensive semantic information beyond the point level by aggregating time steps into subseries-level patches. Furthermore, to ensure uniform partitioning of the patch sequence into K equally-sized blocks in subsequent modules (see Section (.3). we employ Padding(-) to extend the input sequence. Denoting the patch length P and the stride (the non-overlapping region between two consecutive patches) as S, the Patch(-) process yields sequence of patches h E NxP, where N denotes the number of patches, N = SK K. Subsequently, we employ embedding layer to map the dimension of each patch from h E XNX to e xd
e Embedding (Patch (Padding (x)))
After obtaining the patch embedding sequence e = (e1, .,...., eN), we use Mamba followed by subsequent Dropout layer to capture long-term dependency:
Dropout (Mamba(())
(3)

---

Mamba can selectively pass or forget information, which helps reduce redundancy and noise in the input e. However, relying solely on Mamba not sufficient. scenarios where the patch sequence length N is large, there still exists the possibility of significant redundancy. Therefore, we propose the IBF to further filter out redundant information from
3.2 IBF Module for Redundancy Filtering
The IBF module seeks to retrieve the most relevant subsequence x for a target prediction Y from the input sequence We adopt the sufficient encoder assumption [29]. implying that the information of the input subsequence x x is preserved in the encoding process, resulting in I(x=bb Y) x I(z Y)
and [(x x) ~ [(2 sub z), where znth is subsequence of z. The Eq. are transformed into:
min - I[z'ub Y) + 3I(2"ub z)
The first term encourages znd to be informative with respect to the label Y, and the second term minimizes the mutual information between z and sh so that z uh retains only limited information from The discrete nature of sequences renders direct optimization of IB objective impractical, as there are N potential subsequences z zit for a patch sequence of length N. Here, we relax patch selection from discrete to probabilistic sampling. Considering z; the representation of the i-th patch, we utilize MLP (.) to assess the importance c₁ of patch 2,:
= sigmoid (MLP (x:))
Consequently, the selection of patch can be obtained by sampling from X Bern(c.), where (ern(c,) represents Bernoulli distribution parameterized by C.. To ensure the differentiability of function for the discrete random variable
the sampling process, utilize the gumbel sigmoid [20] 9 A defined as:
u
A; = -iimoid
log
+ log
where u ~ Uniform (0, 1), and 7 is the temperature. Subsequently, subsequence 2 su can be obtained by z sub Az.
Although we can employ shannon mutual information 5 to quantify the compressed and informative nature of the distribution of subsequences zud the optimization process is inefficient and unstable due to mutual information estimation (39]. Here, we employ an optimization strategy known as noise injection (38). which endows the IB objective with a tractable variational upper bound. The core concept is to allow the model to introduce noise into less informative subsequences while minimizing noise injection into more informative ones (i.e., 2dtb). Initially, noise injection disrupts the flow of information from the input sequence z to the perturbed sequence noise Az + (1 A)e, where e follows random Gaussian distribution. To preserve the semantic, we set c - NN oz), where He and of denote the mean and variance of z. Subsequently. we encourage the perturbed sequence 2 noise to maintain its informative properties relative to the label Y. Finally, zuub derived by removing the noise from noise Eq. can be reformulated as:
min I[z Y) + 3liznoise ,) bol
Minimizing - moise Y). We first examine the first term -I (z Y) in Eq. 7 which encour- ages noise is informative of label Y:
-1 (z noise Y) Ey guise log pa
Y noise) =
Cpred moise Y)
(8)
Here, Po (Y) moise) represents the variational approximation to the true posterior distribution p (Y) noise) (A detailed proof can be found in Appendix D). We model po (Y g00ise) as predictor parametrized by 0, which outputs the model prediction based on the input noise Thus, we can min- imize the upper bound of I noise Y) by minimizing the model prediction loss Lpred (z Y).
We choose to utilize the MSE Epred nziis, Y).
5

---

Minimizing [z z). For the second term [z .2) in Eq.
7 we can derive its variational
upper bound:
[ mosse 2) S Ez (- 1- log A log
A +
B²)
Cromp (ztoiie,
(9)
2N
2N
where A Σ==1 (1 x1) and B EN Ay(2; Ma)
A detail proof is given in Appendix D
Tx
Finally, we can efficiently estimate Eq. and Eq. 9 with the batched data in the training set. The overall loss is:
L = pred mz Y) + 8 3.comp (z noise z.
(10)
3.3 Hybrid-Transtormen Transformer--ammaaHHT
Modeling the imput long sequence with Mamba and then using Transformer to model the partitioned short sequences is a promising paradigm [22 25 13), it can leverage the strengths of both architectures simultaneously. We have designed two split methods capable of retaining semantic information: interval split and block split, denoted as:
b: paise
E
naise :i=j (mod K)}
(11)
b; (U--))N/KONN/K1
noise
(12)
where by represents the i-th sequence block, and K is the number of blocks. The premise for splitting sequences into subsequences is that the latter can still retain the semantic meaning of the original long sequences. Fortunately, time series data often adhere this principle. The interval split is inspired by SCINet [15). which highlights unique property of time series: temporal relations (e.g.. trend and seasonal components) are largely preserved after downsampling into two subsequences. SCINet downsamples the original sequence into two subsequences by separating the even and odd elements, our interval split extends this approach to partitioning sequence into K blocks, distributing contiguous K patches into K distinct blocks. This method preserves the global characteristics of the sequence. Additionally, we propose the block split, where continuous segment of patch subsequence forms a block. This partitioning method is based on the periodicity of time series, where one period (or multiples of a period) is considered as a block, thus preserving the local information of the sequence. The patch operation and partitioning reduce the length of the input sequence for the Transformer from L to L/PK, significantly reducing the computational overhead. For short sequences, the Transformer is highly efficient, more than Mamba (see Appendix E.
4 Experiments
Our experiments aim to address the following questions:
Can the PIH model effectively reduce the noise in the input sequence and thus leverage a longer lookback window? (Section 4.11
Can our approach universally improve the MEW of various Transformer-basee models and achieve better performance? (Section 4.2)
What is the impact of each component? (Section
4.1 PIH Can Effectively Leverage Long Windows
Dataset.
We evaluate PIH on seven popular datasets, including Weather, Traffic, Electricity, and four ETT datasets (ETThl, ETTh2, ETTml, ETTm2). We also provide results on the Solar and PEMS datasets in Appendix C.1
Experimental Settings and Baselines. PIH integrates IBF and HTM modules into the PatchTST model, making PatchTST the primary baseline. To assess how effectively our model utilizes longer lookback windows, we set 1024 for both PIH and PatchTST, which is significantly longer than in previous studies. The other experimental settings be found in Appendix 2. We additionally

---

Models
PIFT
PachTST
Bi-Mambn
S-Mamba
FElilomer
Autuformer
Informer
DLinear
NLmmar
Ours
2024
2014
2004
2022
2001
2020
2013
2003
Meenc
M15
MANE
MSB
MAA
MIN
MSL
MAB
MSL
MEN
MSC
MAAL
MSS
MAL
XISC
M.N
MSL
MEAI
0.340
0.394
0.371
0,405
0.378
0.395
0.386
0.906
0.376
0,415
0.455
0.465
0.941
0.769
0.511
0.520
0.379
0,904
E17
0.396
0.479
0.448
0.446
1.007
0.786
0:414
0.428
0.409
0431
0449
0.471
0.465
0.444
0.488
0.444
0.62
0.487
1.08
0.784
0.43
0458
0.422
0.45
730
0.435
0.466
0.482
0.453
0.470
0.457
0.493
0.488
0.469
0.492
0.515
0.517
1.34
0.857
0.511
0.520
0.70
0.477
96
0.263
0.328
0,277
0.540
0.91
0.342
0.288
0.349
01332
0.374
0.332
0.368
1.549
0.552
0.24
0.361
0.296
0.351
0.324
0.370
0.34
0.385
0.392
0,519
0.388
0.407
0,446
0.426
0.434
3.792
0.430
0.488
0.337
0.382
0.314
0.417
0.447
4215
0.644
0.378
0.403
0.442
0.439
0.431
01412
0.69
0.453
3465
1.619
0.83
0.47
0.456
BT 67
192
56
0.291
0.359
0.339
0.374
0.234
0.334
0.574
0.559
0.361
0.320
0.383
0.360
0.221
0.571
0.367
0.355
0.325
0.365
0.415
0.350
0.30
0.514
0.495
00392
0,525
0.725
0.619
0,500
0.31
0.356
0.391
0.55
0,317
0.352
0.395
0.381
336
0.300
0.386
0.363
0.392
0.
0.386
0.402
0.417
0.488
0.392
0.425
0,510
0.491
1,005
0.741
0.365
0.388
0.374
0.393
0.436
0.064
0.29
01776
0.129
0.26
11.80
0260
0:355
0,462
01664
12000
0.163
192
0.213
0.249
0.226
0.255
0242
0.364
0.253
0.310
012522
0.318
0.78
0.356
0.595
0.556
0.238
0.317
0.216
0.294
336
0.205
0.334
0,268
0.331
0.304
0.344
0.312
0,348
0.224
0.364
0.345
0.319
1.270
0.871
0.266
0.326
0.265
0.326
730
0.343
0.373
0:550
0.58
0
0.402
0.412
0.408
0.410
0.430
0.414
0.419
3,001
1.267
0.388
0.375
0,338
0,375
Weather
0.157
0.354
0,190
0:190
0.241
0.249
0.215
0.275
0.329
0.49
0.434
0.267
0.215
336
0.241
0268
0.265
0.285
0.264
0.291
0.216
0.298
0.339
0.377
0.351
0.391
0.585
0.545
0.255
0.304
0.259
0
720
0.500
0.39
0,386
0.318
0.343
0.346
0.353
0.349
0.389
0.409
0.415
0.426
0.916
0.205
0.313
0.351
0.321
0.342
0.350
0.24
0.504
0:20
0.311
0.355
0.331
0.281
0.516
0.359
0.777
0.171
0.735
0:410
01188
0.275
0.333
0.277
Eleentott Tolti
0.371
0.205
0.397
0.380
0.777
0.279
0.307
0.274
0,392
0.423
0.302
0.416
0.274
0.423
0.605
0:375
0.623
0.387
0.776
0.434
0.413
0.288
0.400
0.430
0.46
0.319
0.388
0.458
0.900
0.621
0.325
0.699
0.395
0.827
0.466
0.400
0.009
0.449
0.901
96
0,127
0.228
0.135
0.256
01140
0.236
0.142
0.288
0.186
0.302
0.196
0.113
0.3%%
0.393
0133
0.22
0.133
0.29
192
0.145
0.240
0.151
0.249
0.155
0.253
0.189
0.267
0.197
0
0.211
0.324
0.327
0.47
0.146
0.243
0.148
0.242
0.160
0.263
0.269
0.178
0.328
0.335
0.260
0.164
0.259
0.192
0.206
0.299
0.293
0.207
0.303
0.348
0:351
0.427
0.292
0.292
Table 1: Long-term forecasting results with different prediction lengths T E {96, 192, 320) The best results are highlighted in bold.
(2) 0.40
(b)
PatchTST ParchitM HTM
FatchT3T
Pift
Thre (min) Merney
20
0.86
10
0.32
ww
0.28
300
512
Lookback Window
Weather
Electrotty
Tratfoc
Figure 3: (a): The performance comparison between PIH and PatchTST at L E {96, 336, 512, 1024]. (b): Comparison of GPU memory (GB) and training time (minutes/cpoch) for PatchTST, PatchTST, HTM, and PIH. (c): Visualization of a sample sequence in the Electricity, highlighting the most important 20 patches identified by the IBF module with green shading.
selected Mamba-based, Transformer-basee and Linear-based models as baselines. For Mamba-based model, we choose S-Mamba 34] and Bi-Mamba [12. For Transformer-basse models, in addition to PSt we selected FEDformer 45. Autoformer [35, and Informer 42). Furthermore, we include two Sanear-based models, DLinear and NLinear 40) Given that these two models were models in handling long lookback windows, also set L 1024 for them. All models follow the same experimental setup, with prediction lengths T E (96, 192, 336, 720]. We use MSE and MAF as evaluation metrics. We conducted five repeated experiments and provided the significance analysis in the Appendix C.5
proposed to address the limitations of Transformer-baa
Results and Analysis. As show in Table [ For models like S-Mamba, Transformer, Autoformer, and Informer, PIH significantly outperforms them. Even for models specifically designed to handle long windows, such as PatchTST, DLinear, and NLinear, PIH still surpasses them, demonstrating its effectiveness in processing longer sequences. is worth noting that we did not intentionally choose an unusual setting like L = 1024 to lower the performance of these three models. In Appendix C.I we also provide their performance under shorter windows (e.g., 336 and 512), where PIH continues to outperform them. Overall, PIH with a much longer window setting achieves better results than other models with shorter windows. Our experiments highlight the potential for further increasing the window size.
7

---

Transfommer
Informer
Autnformer
PuichTTT
Origin
Ours
Origin
Ourr
Origin
Ours
Origin
Qure
MEW
MSE
MEW
MSG
MEW
MSE
NEW
MSE
MEW
MSB
NEW
MSE
MEW
MSE
NEW
MSE
Erih1
72
1.0099
144
0.56
12
1.146
144
1.06.3
72
0.508
192
0.491
512
0.442
1024
01,435
Euhh
336
2.297
512
2,087
48
3,624
96
3.289
120
0.458
168
0.446
336
0.379
1024
0.378
Weather
512
0.504
512
0,482
1,028
120
0.957
141
0.460
228
0.451
1024
0.306
1024
0.309
Traflic
168
0.685
192
(.680)
1,083
72
0,973
144
0.649
192
0.612
512
0.432
1024
0.430
Frml
48
0.974
228
0.843
96
0.977
168
0.910
192
0.529
228
0.517
1024
0.407
1440
0_399
120
2.784
3.956
168
0.419
336
1024
0.350
Electricity
48
0.287
96
0.281
72
0.372
120
0.361
336
0.243
336
0.237
512
0.197
1024
0.192
Table 2: The MEW and MSE of the "origin" Transformer-based models: Transformer, Informer, Autoformer, and PatchTST, well as the versions integrated with our module ("ours"). Larger MEWs and smaller MSEs are highlighted in bold.
PIH Can Effectively Utilize Longer Lookback Windows. As shown in Fig. (a), we set the lookback window to L = {96, 336, 512, 1024] and used the average MSE over 7 datasets with forecasting horizons of T E (96, 192, 336, 720) as the evaluation metric. The results indicate that the performance of PatchTST improves steadily as the window increases from 96 to 512, but declines when extended to 1024. In contrast, PIH exhibits consistent performance improvement as the window size increases from 96 to 1024. This suggests that the HTM and IBF modules help PatchTST improve MEW, leading to better performance with longer windows. Another noteworthy observation is that, except for L = 96, PIH consistently outperforms PatchTST for the same L. We hypothesize that with L = 96, sequence redundancy and noise is low, and the patch strategy alone is sufficient to manage it effectively, rendering IBF and HTM unnecessary. Consequently, PIH lags behind PatchTST at this window size. However, the window length increases and sequence redundancy grows, the IBF and HTM modules become more effective, allowing PIH to surpass PatchTST.
Computational Overhead. In addition performance comparisons, we evaluated computation time and memory usage, as shown in Fig. (b). When using only the HTM module without the IBF (referred as HTM), it demonstrates significant improvements in both computational time and memory usage compared to the pure Transformer architecture (referred to as PatchTST), surpassing it by a notable margin (2 3 times). Additionally, HTM outperforms the pure Mamba architecture (referred to as PatchTSM), which can be attributed to the Transformer's lower computational cost when handling shorter sequences compared to Mamba (see Appendix E.2) Moreover, when both HTM and IBF are integrated (i.e.. PIH), the additional overhead introduced is negligible, as the IBF module only consists of a simple MLP.
Interpretability of IBF. Another advantage of incorporating the IBF module is its ability to enhance interpretability by identifying crucial subsequences for the final prediction. As shown in Fig. 3 (c), we provide a visualization of a sample from the Electricity dataset. Based on Eq. we select the top 20 most important patches and highlight them in green. The results indicate that the model focuses more on sequences at peak positions. This aligns with prior knowledge, as peak positions typically represent changes in the sequence and contain more critical information.
4.2 The Generalizability of Our Modules
We conducted experiments on three different point- based Transformer models Transformer, In- former, and Autoformer-as well as a patch-based PatchTST. Specifically, for Transformer, Informer, and Autoformer, we selected MEW from the following windows: 48, 72, 96, 120, 144, 168, 192, 228, 336, and 512. For PatchTST, the MEW were selected from the set: 72, 96, 120, 144, 168, 192, 228, 336, 512, 1024, and 1 1440. The prediction length was set to T = 720, with MSE used as the evaluation metric for MEW. All other experimental settings remained the same. The default model and the versions with the two modules integrated are denoted as "origin" and "Ours", respectively. As shown in Table 2 the MEW of pasnt-based models tends to be quite small, especially for Informer, where the MEW on all 7 datasets does not exceed 96. On the other hand, the PatchTST, which is patch- basedd exhibits larger MEW. This is because the use of patches effectively reduces the noise in the lookback window and significantly reduces the number of tokens, alleviating the issues of attention dispersion and overfitting to noise in the Transformer. After integrating our modules, the MEW of all four models showed significant improvement, and they also achieved better MSE. This is because the IBF module effectively reduces the noise in the input, while the HTM further diminishes noise and, by decomposing long token sequences into shorter ones, ensures that the Transformer
8

---

0.4
ST
+IB
interval
block
Mean MSE
0.35
+HTM
PIH
Mean MSE
0.3
HHMM
0.30
0.2
0.25
0.1
96
192
336
720
Etthl
Erth2
Etmll
Ettm2
Weather
Electricity Traffic
Figure 4: Left: Ablation experiments of different modules prediction lengths T (96, 192, 336, 720}. Right: Comparison interval split and block split. The average MSE across 7 datasets T (96, 192, 336, 720} is used as the evaluation metric.
processes noise-filtered, shorter sequence. These experiments demonstrate the generalizability of our modules. Furthermore, we observe that for the same model, when the MEW is improved, its performance also increases, indicating that enhancing a model's MEW is promising direction.
4.3 Ablation Study
Component Ablation. To assess the effectiveness of IBF and HTM, utilize PatchTST baseline, upon which we separately introduce IBF, HTM and both simultaneously to obtain three variants: +IB, +HTM, and PIH. Additionally, solely employs Mamba to handle both the original long sequences and the divided short sequences. All experiments maintain consistent settings, with L 1024 and T {96, 192, 336, 720]. The average MSE across 7 datasets is used as the evaluation metric. As illustrated in Fig. 4 the following observations are made: (1) Both IBF and HTM modules enhance the model's ''' performance, and combining these two modules yields superior results. (2) Compared to HMM, HTM exhibits slightly better performance, which can be attributed the different mechanisms between Transformer and Mamba, making each more suited to handling different types of sequences. By combining the strengths of both, the hybrid approach achieves superior results. As discussed earlier, the Transformer has lower computational costs for shorter sequences, while Mamba is more efficient for longer sequences. Therefore, from both performance and computational overhead perspectives, using combination of both architectures is better choice than relying solely on one. (3) At longer prediction lengths, such as T 720, our model demonstrates greater improvements compared to = 96, indicating that larger windows L provide more significant benefits for longer-term predictions (longer T))
introduce variant of +HTM, +HMM, which
Interval Split vs. Block Split. We compared the performance of interval split and block split across various datasets, as shown in Fig. 4 Overall, the effectiveness both split methods is roughly comparable, demonstrating their ability to preserve sequential characteristics. However, slight variations in performance are observed across different datasets. We speculate that this discrepancy arises from the different strengths of each partitioning method in retaining specific sequential patterns. Intuitively, interval splil emphasizes global variations, while block split focuses on variations within specific periods. Determining the most suitable partitioning strategy remains a topic for future exploration.
Other Hyperparameters. Our model incorporates several crucial hyperparameters, including K, determining the number of partitions; 3, which governs the balance between prediction and compression in the IB objective; and the temperature factor т, influencing subsequence sampling. We investigate the impact of K (2, 6, 8). B E (0.0001, 0.001, 0.101,0.1111 1), and set t E (0.1, 0.5, 1, 2). We find that the choice of K does not significantly affect performance, whereas and B exhibit considerable influence on performance, likely due to variations the redundancy levels across different datasets. Detailed hyperparameter experiments can be found in the Appendix C.4

---

Conclusion
In this paper, we propose a MEW metric to evaluate the model's ability to leverage the lookback window. We introduce two model-agnostic modules, IBF and HTM, from both iitormation-thhoreeee and model-architectural perspectives. Experiments show that these modules can effectively improve the model's MEW, and with larger MEW, the model's performance also improves, demonstrating the importance of enhancing MEW. Furthermore, we combine these modules with the patch strategy design the PIH model, which can handle longer windows than previous works and achieves state-of-the-aar results, illustrating the potential of leveraging longer windows.
6
Acknowledgement
This paper is partially supported by the National Natural Science Foundation of China (No. 1s227902) ). The AI-driven experiments, simulations and model training were performed on the robotic Al-Scientist platform of Chinese Academy of Sciences., Anhui Science Foundation for Distinguished Young Scholars 19880554))))) Natural Science Foundation of China (No. 1250441....
References
[1] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):887-22055 2018.
[2] Md Atik Ahamed and Qiang Cheng. Timemachine: A time series is worth 4 mambas for long-term forecasting, 2024.
[3] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. CoRR, abs/1612.00410,, 2016.
[4] Xiuding Cai, Yaoyao Zhu, Xueyao Wang, and Yu Yao. Mambats: Improved selective state space models for long-term time series forecasting, 2024.
[5] Tyrone E Duncan. the calculation of mutual information. SIAM Journal on Applied Mathematics, 19(1):215-220, 1970.
[6] Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Youqiang Zhang, and Junshi Huang. Dimba: Transformer-mann diffusion models. 2024.
[7] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752. 2023.
[8] Maximilian Igl. Kamil Ciosek. Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Advances neural information processing systems, 2019.
[9] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[10] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The s1 international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018
[11] Sheng Li, Xiaoyu Jin, Yue Xuan, Xiang Zhou, Weihua Chen, Yi-Xin Wang, and Xia Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting, arXiv, abs/1907 0000000702200 002300235 2019.
[12] Aobo Liang, Xingguo Jiang, Yan Sun, Xiaohou Shi, and Ke Li. Bi-mambat: Bidirectional mamba for time series forecasting, 2024.
10

---

[13] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwariz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: hybrid wransformer-maber language model. CoRR, abs/2403. 19887, 2024.
[14] Bryan Lim and Stefan Zohren. Time series forecasting with deep learning: A survey, CoRR, abs/2004.13408, 2020.
[15] Minhao Liu, Ailing Zeng. Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet:
Time series modeling and forecasting with sample convolution and interaction. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- - December 9, 2022, 2022.
[16] Minhao Liu, Ailing Zeng. Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction, Advances in Neural Information Processing Systems, 35:5816-588 2022.
[17] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2022.
[18) Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. CoRR, abs/2310.06625, 2023.
[19] Jiajun Ma, Zheng Shou, Alireza Zareian, Hamed Mansour, Anthony Vetro, and Shih-Fu Chang. Cdsa: Cross-dimensional self-attention for multivariate, geo-tagged time series imputation. arXiv, abs/ hewp//90500994,, 2019.
[20] Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A contin- uous relaxation of discrete random variables. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[21] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[22] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur, Long range language modeling via gated state spaces. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[23] Yuqi Nic, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[24] Jonathan Pilault, Mahan Fathi, Orhan Firat, Chris Pal, Pierre-Luc Bacon, and Ross Goroshin. Block-state transformers. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[25] Jonathan Pilault, Mahan Fathi, Orhan Firat, Chris Pal, Pierre-Luc Bacon, and Ross Goroshin. Block-state transformers. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.

---

[26]
Dean Prichard and James Theiler. Generalized redundancies for time series analysis. Physica D: Nonlinear Phenomena, 84:476-493, 1994.
[27] Karl Schulz and et al. Restricting the flow: Information bottlenecks for attribution. In ICLR, 2020.
[28] Haichao Song, Deepak Rajan, Jayaraman J Thiagarajan, and Andreas Spanias. Attend and diagnose: Clinical time series analysis using attention models. In AAAI, 2018.
[29] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12. 2020, virtual, 2020.
[30] Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The information bottleneck method. CoRR, physics/0004057, 2000.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, volume 30, 2017.
[32] Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of
language models, 2024.
[33] Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[34] Zihan Wang. Fanheng Kong, Shi Feng, Ming Wang, Xiaocui Yang, Han Zhao, Daling Wang. and Yifei Zhang. Is mamba effective for time series forecasting?, 2024.
[35] Haixu Wu, Jiehui Xu, Jianmin Wang. and Mingsheng Long. Autoformer: Decomposition transformers with Auto Auto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
[36] Tailin Wu, Hongyu Ren, Pan Li, and Jure Leskovec. Graph information bottleneck. In NeurlPS, 2020.
[37] Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. CoRR, abs/2410.05258. 2024.
[38] Junchi Yu, Jie Cao, and Ran He. Improving subgraph recognition with variational graph information bottleneck. CORR, abs/21 2021.
[39] Junchi Yu, Tingyang Xu, Yu Rong, Yatao Bian, Junzhou Huang, and Ran He. Graph infor- mation bottleneck for subgraph recognition. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[40] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transforrees effective for time series forecasting? arXiv preprint arXiv:2205. 13504, 2022.
[41] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representa- lions, 2023.
[42] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, volume 35, pages I 11106-11115, 2021.
[43] Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference Machine Learning, 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have claimed the paper's contributions and scope in abstract and introduc-
tion
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
The proof could be found in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The code and data are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
The code and data are open.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuiu wwwiii/gguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the data splits, hyperparameters, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided the statistical significance of the experiments result.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide provide sufficient information on the computer resources in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics pattps://neuripppt httppp//wwpppppccacccccctet.....
Answer: [Yes]
Justification:
The research is within the Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All of them are properly respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
19

---

A Limitations
First, our experiments demonstrate that extending the window length to I 1024 still yields performance improvements, suggesting that further exploration of longer windows is promising direction. Secondly, although the modules we propose effectively improve the model's MEW, bottleneck still exists. A more promising avenue of research is to explore whether there is a scaling law for lookback windows, meaning that larger windows consistently result in better performance. Thirdly, the interval split and block split methods are heuristic. Designing an adaptive, end-to sod-to-end split method tailored each training dataset may lead to better results. Lastly, while recent large time- series models have adopted much longer windows, we assert that our approach is orthogonal to theirs. is worth exploring whether our method can be integrated into these large models D.....I.M--bar to further extend their window sizes.
B Dataset Description
In the main text, we use seven popular multivariate datasets provided in 33 for forecasting and representation learning. The Weather dataset collects 21 metcorological indicators in Germany, such as humidity and air temperature. The Traffic dataset records road oceupancy rates from various sensors on San Francisco freeways. The Electricity dataset describes the hourly electricity consumption of 321 customers. The ETT (Electricity Transformer Temperature) datasets are collected from two different electric transformers labeled as 1 and 2, each containing two resolutions (15 minutes and 1 hour), denoted as m and h, respectively. Thus, there are four ETT datasets in total: ETTml, ETTm2, ETTh1, and ETTh2. In addition, we incorporated two additional datasets: Solar [0] and PeMS[[6]. Their experimental results are shown in Table 4 The results indicate that PIH also achieved the best performance on these two datasets.
Table 3: Statistics of popular datasets for benchmark.
Datascts
Weather
Traffic
Eloctricity
ETTh1
ETTh2
ETTml
ETTm2
Solar
PEMS
Features
21
862
321
7
7
7
7
137
358
Timesteps
52696
17544
26304
17420
17420
69680
69680
52,179
21,351
C
E Experiments
C.1 Experiments settings
PIH is built upon the PatchTST and thus incorporates all hyperparameters from PatchTST. To ensure a fair comparison, we adhered strictly to the settings of PatchTST for these shared hyperparameters. with the exception of the learning rate. We conducted hyperparameter search only for those introduced by the HTM and IBF modules, as this was necessary. The only exception is the lcarning rate. Given the introduction of the Mamba and IBF modules, the default learning rate of Ir 0.0001 in PStchTST is suboptimal. Consequently, we set the search space for the PIH learning rate to Ir {0.001, 0.0005, 0.0001). }. To ensure fair comparison, we also performed hyperparameter search for the learning rate in PatchTST, and selected the optimal results. The resulting mean Absolute Error (MAE) values were 0.3 10 and 0.335, which are almost unchanged compared to the default learning rate (Ir = 0.0001), yielding 0.310 and 0.336. Thus, this does not affect our result analysis.
Our model incorporates several crucial hyperparameters, including K, which determines the number of partitions; 8, which governs the balance between prediction and compression in the information bottleneck (IB) objective; and the temperature factor T, which influences subsequence sampling. We set E (2, 4). 3 E (0.0001, 0.001, 0.1, 1}. and T E {0.1, 0.5, 1,2 We selected the optimal hyperparameters based the results from the validation set.
The original PatchTST paper discusses the impact of key hyperparameters in detail, such as the number of Transformer layers, patch length, the number of heads in multi-head attention, and the dimension of the latent space. The default parameters provided their official code represent the best-performing combination overall, so there was no need to repeat hyperparameter searches for PSt Additionally, since our PIH based on PStchTST, any changes to PatchTST would necessitate corresponding adjustments to PIH. Conducting performance comparisons between PIH
20

---

and PatchTST for every parameter setting would render the ablation experiments overly redundant. Therefore, we did not adjust the hyperparameters of PatchTST. The only exception is the learning rate. Given the introduction of the Mamba and IBF modules, the default learning rate of Ir = 0.0001 in PatchTST suboptimal. Consequently, we set the search space for the PIH learning rate to Ir {0.001, 0.0005, 0.0001); }. To ensure a fair comparison, we also performed hyperparameter search for the learning rate in PatchTST, using ir = {0.001, 0.0005, 0.0001 J. and selected the optimal results. The resulting mean Absolute Error (MAE) values were 0.3 10 and 0.335, which are almost unchanged compared to the default learning rate (Ir = 0.0001), yielding 0.310 and 0.336. Thus, this does not affect our result analysis.
C.2 Additional Experimental Results.
Due to space limitations in the main text, we only discussed the results on seven datasets. To better evaluate our model, we also conducted experiments on the Solar and PEMS datasets, with the experimental settings consistent with those described in the main text. As shown in Table 4 our model also outperforms other methods on these two datasets.
Table 4: Multivariate long-term series forecasting results for the Solar and PEMS.
Models
PIH
PatcTTST
iTransformer
DLincar
MICN
FourierGiNN
FiDiformer
Autoformer
Metric
MSR
MAE
MSE
MAF
MSE
MAR
MSR:
MAP
MSE
MAR
MSE
MAF
MSR
MAR
MSF
MAE
0.163
0.230
0.185
012446
0.346
012257
0.243
0.183
0.214
0.316
0.369
Solar
336
192
0.188
0.177
0.247
0.239
0.209
0.201
0.266
0.262
0.217
0.195
0,282
0,263
0.228
0.211
0.285
0.273
0.219
0.205
0,250
0.247
0.205
0.198
0.261
0.256
0.294
0.281
0.378
0.361
0.438
0:418
0.467
0.437
720
0.196
0.255
0.226
0.283
0.208
0,276
0.236
0.294
0.227
0.263
0.202
0.265
0.315
0.406
0.618
0.550
0.163
0.063
0.166
0.167
0.078
0.187
0.094
0.204
0.091
0.394
0.584
0.607
PEMS
24
0.075
0.179
0,080
0.185
0.081
0.187
0.113
0.224
0.116
0.229
0.116
0.232
0.300
0.431
0.672
0.664
48
0,100
1,204
0.109
0.213
0.111
0,215
0.167
0.274
0.147
0.255
0.165
0.271
0.396
0.476
0.879
0.781
0.132
0.233
0.145
0.240
0.212
0.313
0.256
0.362
0.196
1.100
0.895
C.3 Performance of PatchTST, DLinear, and NLinear under Different Window Lengths
In the main text, set L to 1024. This is not done with the intention of deliberately undermining the performance of PatchTST, DLinear, and NLinear under an unusual setting. Here, we conducted experiments with DLinear and NLinear, two linear-based models, under two settings: L = 336 and L = 1024, with the results shown in Table For PatchTST, we set L = 336 and L = 512. As shown in Table 5 even with the shorter window settings, their performance still lags behind that of PIH at L = 1024. Specifically, we can draw the following conclusions:
Linear-based models indeed perform well against noise, with NLinear(1024) generally outperforming NLincar(336). This is consistent with the results of PIH, indicating that larger windows are beneficial.
NLinear(1024) generally outperforms NLinear(336), whereas DLinear underperforms compared to DI DI.inar(330). Thus, directly increasing the window size in linear-based methods is not always effective.
PIH(1024) outperforms NI NLinear(1024),, which can be attributed to the superior represen- tational capabilities of the Transformer and Mamba modules compared to linear modules. Therefore, it is essential to continue exploring the potential of Transformer-based models with longer windows rather than relying solely on binear-based models.
Compared to PatchTST (336), PatchTST (512), and PatchTST (1024) discussed in the main text, PIH (1024) achieves better performance. This can be attributed to the IBF and HTM modules, which enable PIH to leverage larger windows effectively.
C.4 Hyperparameters
Our model incorporates several crucial hyperparameters. including K, determining the number of partitions; 3, which governs the balance between prediction and compression the IB objective; and the temperature factor influencing subsequence sampling. We investigate the impact of K E (2, 4, 6. 8), 3 E (0.0001, 0.001, 0.1, 1}, and set T E (0.1,0.5, 1,2). As shown in Fig. we investigate the effects of the three hyperparameters on the ETTh1, ETTml, and Weather datasets.
21

---

Table 5: Comparison between DLinear, NLinear, PatchTST and PIH with different lookback windows.
Wealher
Traflic
Electricity
ETThl
ETTh2
ETTml
ETTm2
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
DLinear(336)
96
0.176
0.237
0.410
0.282
0.140
0.237
0.375
0.399
0.289
0.353
0.299
0.343
0.167
0.260
192
0.220
0.282
0.423
0.287
0.153
0.249
0.405
0.416
0,383
0.418
0.335
0.365
0.224
0.303
336
0.265
0.319
0.436
0.296
0.169
0.267
0.439
0.443
0.448
0.465
0.369
0.386
0.281
0.342
720
0.323
0.362
0.466
0.315
0.203
0.301
0.472
0.490
0.605
0.551
0.425
0.421
0.397
0.421
innearr (024)
96
0.167
0.225
0:385
0.275
0.132
0.229
0.378
0.403
0.294
0.361
0.314
0.358
0.164
0.260
192
0.267
0.397
0.279
0.243
0.428
0.430
0.448
0.391
0.317
336
0.255
0.304
0.288
0.260
0.458
0.484
0.365
0.388
0.265
0.326
720
0.313
0.351
0.450
0.309
0.195
0.292
0.511
0.520
0,905
0.683
0.410
0.417
0,338
0.375
96
0.182
0:232
0.410
0.279
0.141
0.237
0.374
0.394
0.277
0.338
0.306
0.348
0.167
0.255
192
0.225
0.269
0.410
0.279
0.154
0.248
0.408
0.415
0.344
0.581
0.349
0.375
0.221
0.293
336
0.271
0.301
0.435
0.290
0.171
0.265
0.429
0.427
0.357
0.400
0,375
0.388
0.274
0.327
720
0.338
0.348
0.464
0.307
0.210
0.2297
0.440
0.453
0.394
0.436
0.433
0.422
0.368
0.384
NLinear(
96
0.170
0.226
0.383
0.270
0.133
0.229
0.379
0.404
0.296
0.351
0.317
0.359
0. 0.163
0.257
192
0.215
0.265
0.397
0.274
0.242
0.426
0.337
0.382
0.352
0.381
0.216
0.294
336
0.259
0.298
0.410
0.281
0.164
0.259
0.445
0.407
0.393
0.326
720
0.321
0.342
0.449
0.303
0,203
0.292
0.470
0.477
0.417
0.456
0,409
0.413
0,338
0.375
P(33
96
0.152
0.199
0.367
0.251
0.130
0.222
0.375
0.399
0.274
0.336
0.290
0.342
0.165
0.255
192
0.197
0.243
0.385
0.259
0.148
0.240
0.414
0.421
0.339
0.379
0.332
0.369
0.220
0.292
336
0.2449
0.283
0.398
0.265
0.167
0.261
0.441
0.436
0.331
0,380
0,366
0.392
0.278
0.329
720
0.320
0.335
0.434
0.287
0.2002
0.291
0.449
0.466
0.379
0.422
0.420
0.424
0.367
0.385
P11chTST(512))
96
0.149
0.198
0.360
0.249
0.129
0.222
0.370
0,400
0.274
0.337
0.293
0.346
0.166
0.256
192
0.194
0.241
0.379
0.256
0.147
0.240
0.413
0.429
0.382
0.333
0.370
0.223
0.296
336
0.282
0.264
0.163
0.259
0.440
0.329
0.384
0.369
0.392
0.329
720
0.314
0.334
0.432
0.286
0.197
0.290
0.447
0.468
0.379
0.422
0.416
0.420
0.362
0.385
PIH(1024)
96
0.147
0.198
0.357
0.248
0.127
0.220
0.360
0.394
0.263
0.328
0.291
0.349
0,161
0.253
192
0.191
0.239
0.371
0.255
0.145
0.240
0.396
0.418
0.324
0.370
0.337
0.374
0.213
0.289
336
0.241
0.280
0.392
0.261
0.160
0.256
0.409
0.432
0.314
0.376
0.360
0.386
0.265
0.326
720
0.309
0.329
0.430
0.282
0.192
0.287
0.435
0.466
0.378
0.425
0.405
0.411
0.342
0.375
We find that the choice of K does not significantly affect performance, whereas T and / exhibit considerable influence, likely due to variations in the redundancy levels across different datasets.
C.5 Robustness of Our Results
To verify whether the improvements of PIH over PatchTST are statistically significant, we utilized p-values to evaluate the prediction results. Specifically, for a prediction length of T' = 96, we validated the significance of the improvement achieved by PIH over PatchTST, Table T presents the p-values from 5 experiments conducted at T = 96, 192, 336, 720. The results demonstrating that the performance improvements are sigmificant in out of datasets. (p-value < 0.05)
Table 6: P-values for the significance of PIH improvements over PatchTST at different prediction lengths.
T
Weather
Traffic
Electricity
E
Etth2
Ettm 1
Ettm2
96
0.397
1.19 x 10 6
7.00 x 10
1.10 x 10 5
1.99 x 10 0
0.32
0.002
192
0.49
7.47 10 9
7.78 x 10-5
1.60
10 6
8.65
10
0.20
0.013
336
0.015
6.40
10-7
4.60 x 10-4
2.30 x 10-6
2.44
10
0.008
0.007
720
0.08
6.29 x 10 10-6
2.92 x 10
2.53 x 10
2.05 x 10 10-7
0.36
0.0008
C.6
Integration into the iTransformer
iTransformer 18 is special Transformer-based time series model that, instead of treating points or patches as tokens, treats each channel as token. Therefore, the number of input tokens for iTransformer is determined by the number of channels rather than the length of the lookback windows. Although this differs slightly from the main focus of this paper, we found that IBF and HTM can also be applied to i'Transfrmer. In this case, the goal of these two modules becomes reducing redundancy among channels and decreasing the number of tokens fed into the Transformer to mitigate the attention dispersion problem. We conducted experiments on datasets with large number of
22

---

ETTh1
0.370
0.370
0.370
MSE (T-96)
0.365
0.365
0.365
0.360
0.360
0.360
0.001
0.01
0.1
0.1
0.5
B
T
ETTml
0.31
0.31
0.31
MSE (T-96)
0.30
0.30
0.30
0.29
0.29
0.29
2
0.000
0.01
0.1
0.1
0.5
B
Weather
0.155
0.155
0.155
MSE (T-96)
0.150
150
0.150
0.145
0.145
0.145
2
0.001
0.01
0.1
0.1
0..5
K
B
T
Figure The impact of hyperparameters K, 8, and T in the PIH model on performance across ETTh1, ETThm1, and Weather datasets. The vertical axis represents the MSE at T 96.
Table 7: Comparison of MSE values for different datasets and horizons between our method and iTransformer.
Ours
iTransformer
Dataset
MSE
MAE
MSE
MAE
96
0.129
0.225
0.133
0.229
192
0.146
0.242
0.155
0.251
Electricity
336
0.163
0.260
0.167
0.264
720
0.189
0.285
0.194
0.288
96
0.346
0.252
0.349
0.255
192
0.354
0.254
0.359
0.263
Traffic
336
0.370
0.261
0.379
0.272
720
0.413
0.280
0.417
0.291
12
0.059
0.158
0.064
0.167
24
0.075
0.178
0.081
0.187
PEMS
48
0.101
0.203
0.111
0.215
96
0.133
0.234
0.142
0.240
channels, including Electricity, Traffic, and PEMS. shown in Table C.6 after integrating our module, the performance of iTransformer is further enhanced.
23

---

D Proofs of IB
D.1 Proof of Eq(7)
We first examine the first term -/ noise Y) which encourages Vnoise is informative of label
noise Y) I By -log qo (Y) noise)
(13)
Spedd (znmise, Y)
Here, Po (Y) z00is) represents the variational approximation to the true posterior distribution (Y) z moise) .his equation illustrates that minimizing -I (zmoise, Y) is achieved by minimizing the prediction loss between noise and Y. We choose to utilize the Mean Squared Error (MSE) loss to quantify the disparity between the prediction and the ground truth.
Here we provide more details about how to yield Eq. 13 By the definition mutual information and introducing variational approximation Po Y
noise of intractable distribution (Y 220ise) we
have:
Y
noise
Y
z noise
Ey gaass
log
p p(Y)
po (Y) noise
(14)
By sptire
log
p YY)
+ Eggise
[KL (p (Y z molae) I|po (Y 2 noise)
According to the non-negativity of the KL divergence, we have:
log pe Y moiss))
(Y; z nrisse)
2 Ex aani
p(Y)
Ey gois
[lo Po (Y noise
+ H(Y)
We can ignore HYY) since can be treated constant. We model pe Y moise) as a predictor pa- rameterized by 0, which generates the model prediction Y based on the input muise Thus, minimizing the upper bound of - musee Y) entails minimizing the model prediction loss Cpedd (z Y). We opt to employ the Mean Squared Error (MSE) loss to quantify the difference between the prediction and the ground truth.
D.2 Proof of Eq(9)
We derive the upper bound of I mzoos z) by introducing the variation approximation q (zimise) of distribution p moise)
mzooe
z))
I nons 2)
Ey gnnie
log
p(z)
axose
2)
(15)
Fy, gania
log
(2znniee)
Extire [K L (p (z (20000))) llq ( (zookss)
According to the non-negativity KL divergence, we have:
[(z noise
< Ez KL (ps ((z noise [z) noise
(16)
we assume that g (2 moni is obtained by aggregating the patch representations in a fully perturbed sequences. The noise ~ N (nx, o²) is sampled from a Gaussian distribution where μs and o are mean and variance of z. Choosing sum pooling as the aggregatiion function, since the summation of Gaussian distributions is a Gaussian, we have the following equation:
9 noise
N N.:
(17)

---

Then for ps (z"mise z), we have the following equation:
N
N
N
Po noise (2) 12==N
(Nux
+
23z
Azz
Σ (1
My)
02
(18)
jm
jml
Finally, we have following inequality by plugging Eq. 17 and Eq.
18
into Eq.
16
1
I [x m 2) S Ex
log
A
B
+C
(19)
2N
2N
where A - -=Eド (1 23)², = 2i Ag(2, optimization.
and C' is constant which is ignored during
E
Others
E.1
Relationship with Large Time-Series Models
Although some recent large time-series models are capable of handling longer windows, they rely on significantly more parameters and much larger training datasets compared to experiments. Additionally, when tested on the same datasets we used, these models still employ smaller window sizes. Our work does not conflict with these advancements in large time-series models. This is because the HTM and IBF modules we propose are model-agnostic and can be integrated into large time-series models, a direction worth exploring in future.
E.2 Mamba vs Transformer
analyze HTM from both performance and computational overhead perspectives and find that the hybrid architecture has distinct advantages over using only Mamba or Transformer.
From performance perspective. The ablation experiments presented in Fig.3(b) of main paper indicate that removing the Transformer results in slightly worse performance, highlighting the significant advantage of the combined Transformer and Mamba architecture, This finding is further supported by recent works such as Hamba---HHbbie [32, Dimba [6., and Jamba [3]
Table 8: Comparison GPU memory usage and training time per epoch for a single-layer Transformer and Mamba on the Weather dataset as the lookback window L varies.
Model
Metric
96
192
336
512
1024
Time (s)
18.76
21.63
28.25
36.52
58.47
Mamba
Memory (G)
2.02
3.30
4.90
6.78
9.53
Time (8)
7.33
17.94
27.84
44.70
96.57
Transformer
Memory (G)
0.75
1.64
3.21
5,56
15.05
Considering computational overhead. Our framework employs the Transformer solely to process the partitioned short subsequences, which generally mitigates concerns about the costs associated with the Transformer. To validate this, compared the computation time and GPU memory usage between using a single layer of Mamba and single layer of Transformer under various lookback window settings (with nearly identical parameter counts). As shown in Table when L < 336, the computational overhead of the Transformer is even lower than that Mamba; however, at L = 1024, the computational cost of the Transformer is nearly twice that of Mamba. In our experiments, K is typically set to 4, resulting in subsequence length of L/K = 1024/4 4 < 336. Consequently. the addition of the Transformer module incurs less overhead compared to using only Mamba.
In summary, we conclude that retaining the Transformer module is essential for enhancing perfor- mance while managing computational costs effectively.
25


Reviews:
[{"review_id": "M2CAJNHaWH", "reviewer": "Reviewer_Nemp", "summary": "This paper tackles the challenge of using long historical data windows for long-term time series forecasting. The authors find that while longer data windows should theoretically improve predictions, they often introduce noise and computational issues that hinder model performance. To address this, the paper introduces three main contributions:\n\n1. Maximum Effective Window (MEW): A new metric to measure the maximum lookback window a model can effectively use before its performance plateaus or degrades.\n\n2. Information Bottleneck Filter (IBF): A module designed to reduce noise and redundancy by identifying and retaining the most important subsequences from the input data. It uses information bottleneck theory to achieve this filtering.\n\n3. Hybrid-Transformer-Mamba (HTM): An architectural module that combines the strengths of Mamba and Transformer models. HTM first uses Mamba to process long sequences and filter out noise, then splits the filtered sequence into shorter segments for the Transformer to model short-term dependencies.\n\nBy integrating these modules, particularly into the PatchTST model to create a new model called PIH, the authors demonstrate significant improvements. The PIH model can effectively leverage a much longer lookback window of 1024, outperforming existing models on several benchmark datasets. The paper shows that these modules are \"model-agnostic,\" meaning they can be applied to various Transformer-based architectures to enhance their MEW and overall forecasting accuracy.", "questions": "See Weaknesses. A convincing explanation of the issues raised would lead me to increase my score for this paper.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The writing is clear, easy-to-follow.\n\n2. The paper demonstrates high-quality research through extensive and rigorous experimentation. The authors validate their proposed model-agnostic methods on various previous methods including Transformer-based, Mamba-based, and Linear-based models.\n\nWeaknesses:\n1. There are minor inconsistencies that could suggest a lack of careful proofreading. For instance, the caption for Figure 5 states the prediction length is T=720, while the plots are labeled T=96.\n\n2. Why Mamba + Transformer? The Mamba architecture was introduced specifically to address the performance and computational bottlenecks of Transformers on long sequences, such as quadratic complexity and attention dispersion.The HTM architecture first leverages Mamba to process the long input sequence but then re-introduces the Transformer. This design choice seems paradoxical.\n\n3. While the components themselves (e.g., Mamba, IB theory) are drawn from prior work, the originality lies in their integration and application to LTSF.\n\n4. While the proposed method demonstrates strong empirical performance, its contribution should be evaluated in the context of its incremental nature and the associated overhead.", "was_revised": true, "final_justification": "Borderline accept. The authors can not justify the usage of mamba + transformer combination and they admit the low originality of this work. However, the other parts of this work show consistent improvement.", "rebuttal": "Thank you very much for your valuable feedback and suggestions. Below, we address each point in turn.\n\n---\n\n> **Q1.** _There are minor inconsistencies that could suggest a lack of careful proofreading. For instance, the caption for Figure 5 states the prediction length is T=720, while the plots are labeled T=96._\n\n**Thank you for catching this mistake.** You are absolutely right: the caption should read **T = 96** rather than **T = 720**.\n\n---\n\n> **Q2.** _Why Mamba + Transformer?_\n\nIn Appendix E.2, we provide a detailed explanation of why we adopt this hybrid architecture:\n\n- **Computational trade-off:** Although Mamba is theoretically less expensive than the Transformer for very long sequences (e.g., $L=1024$), the Transformer becomes more efficient for shorter windows (e.g., $L\\le336$).  \n    – **Our design** uses Mamba to process the original long sequence and the Transformer to handle the resulting shorter subsequences, yielding a clear efficiency advantage over either model alone.\n    \n- **Complementary strengths:** Empirically, the hybrid (HTM) outperforms the pure-Mamba variant (see Figure 4, left), likely because Mamba and Transformer capture **different but complementary features**, resulting in a richer sequence representation. Similar findings appear in recent works such as Jamba^[1] and Mamba-2-Hybrid^[2].\n    \n\n---\n\n> **Q3.** _While the components themselves (e.g., Mamba, IB theory) are drawn from prior work, the originality lies in their integration and application to LTSF._\n\nWe fully acknowledge this point. Our main contribution lies in the **novel integration** of the HTM module and the IBF within a Transformer-based framework—along with their **demonstrated effectiveness** in long-term time series forecasting and in **improving the model’s MEW.**\n\n---\n\n> **Q4.** _While the proposed method demonstrates strong empirical performance, its contribution should be evaluated in the context of its incremental nature and the associated overhead._\n\nThank you for raising this important point about overhead and incremental contribution.**\n**Figures 3(b) and 4 (left)** directly address this concern:\n1. **Efficiency gains:**\n    - **Figure 3(b)** reports both computation time and memory usage. Replacing the pure Transformer (PatchTST) with our HTM (Transformer + Mamba without IBF) reduces time and memory by **2–3×**. Adding IBF to form PIH introduces **negligible overhead**, as it is implemented as a lightweight MLP.\n2. **Incremental performance:**\n    - **Figure 4 (left)** compares PatchTST (baseline) against three variants—**+IB**, **+HTM**, and **PIH**—plus a pure-Mamba variant (**+HMM**). Using $L=1024$ and $T\\in{96,192,336,720}$ over seven datasets, we observe:\n        1. **Both IBF and HTM individually improve accuracy**, and their combination (**PIH**) achieves the best results.\n        2. **HTM slightly outperforms HMM**, confirming that Transformer and Mamba excel on different sequence lengths and benefit from hybridization.\nIn summary:\n\n- **HTM delivers a 2–3× efficiency boost**, and **IBF adds virtually zero extra cost**.\n- **Both components yield clear accuracy improvements**, with their combination (PIH) providing the strongest results.\n\n---\n\nWe hope these explanations and corrections satisfactorily address your concerns. Thank you again for your thoughtful review.\n\n[1] Lieber, Opher, et al. “Jamba: A hybrid transformer-mamba language model.” _arXiv preprint arXiv:2403.19887_ (2024).  \n[2] Waleffe, Roger, et al. “An empirical study of mamba-based language models.” _arXiv preprint arXiv:2406.07887_ (2024)."}, {"review_id": "CjmqJYNOnQ", "reviewer": "Reviewer_x9sQ", "summary": "The paper introduces the Maximum Effective Window (MEW) metric to evaluate a model's ability to utilize historical data in long-term time series forecasting (LTSF). It proposes two model-agnostic modules: the Information Bottleneck Filter (IBF) to reduce noise and redundancy in input sequences, and the Hybrid-Transformer-Mamba (HTM) architecture to leverage the strengths of both Transformer and Mamba models. These modules are integrated into the PatchTST model to create the PHI model, achieving a lookback window of 1024, surpassing existing non-LLM-based models. The approach is evaluated on seven datasets, demonstrating improved performance and MEW compared to baselines like PatchTST, DLinear, and iTransformer.", "questions": "1.  Referring to weakness 1: Is the proposed MEW metric related to the prediction horizon, data, and model? If so, what is its precise mathematical definition?\n\n2. Could you quantify the computational overhead of the IBF and HTM modules in more detail (e.g., in terms of FLOPs or additional parameters) to better clarify the trade-offs of the hybrid approach?", "limitations": "Yes. They discussed the \"Limitations\"  in the begining of \"Appendix\".", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Two Modules: The IBF module, grounded in information bottleneck theory, effectively filters redundant and noisy subsequences, while the HTM module combines the strengths of Transformer and Mamba architectures. The hybrid approach is well-motivated, leveraging Mamba’s efficiency for long sequences and Transformer’s effectiveness for shorter ones.\n3. Reproducibility: The paper provides open access to code and data, with clear instructions for reproducing results, adhering to NeurIPS guidelines. The experimental settings are detailed, and statistical significance is reported, enhancing the reliability of the findings.\n\nWeakness \n1. Lack of Mathematical Rigor in MEW Definition: Although the concept of MEW is promising, its definition lacks mathematical rigor, relying solely on empirical discussions. Furthermore, it is unclear whether the proposed MEW depends on factors such as the task (e.g., prediction horizon), data, or model. A precise mathematical definition would strengthen the metric's foundation.\n\n2. Computational Overhead Clarity: Although Fig. 3(b) compares GPU memory and training time, the paper could provide a more detailed quantitative analysis of the computational trade-offs introduced by the IBF and HTM modules, particularly in comparison to pure Mamba or Transformer architectures, to better justify the hybrid approach. Additionally, a theoretical analysis of the increased computational complexity is encouraged.", "was_revised": true, "final_justification": "I would like to thank the authors for addressing my questions. I keep my original rating as \"Borderline Accept\".", "rebuttal": "Thank you very much for your valuable feedback and suggestions. Below, we address each point in turn.\n\n---\n\n> **Q1.** _Referring to weakness 1: Is the proposed MEW metric related to the prediction horizon, data, and model? If so, what is its precise mathematical definition?_\n\nYes. Given fixed settings of the dataset, model architecture, and prediction horizon, the **Maximum Effective Window (MEW)** is defined as the **smallest look-back window length** beyond which increasing the window no longer improves performance. In other words, any window longer than MEW provides **no significant gain** under the same conditions.\n\nThank you for this suggestion. While we included an intuitive explanation in the Introduction section, we provide a formal definition below:\nLet:\n\n| Symbol                                   | Meaning                                          |\n| ---------------------------------------- | ------------------------------------------------ |\n| $M$                                      | Forecasting model (architecture + training)      |\n| $D$                                      | Dataset                                          |\n| $H$                                      | Prediction horizon                               |\n| $E(\\hat{y}, y)$                          | Evaluation metric (_lower is better_, e.g., MSE) |\n| $\\Lambda = {L_1 < L_2 < \\dots < L_K}$    | Candidate look-back window lengths               |\n| $S(L_i) = E_{\\text{val}}(M_{L_i}, D, H)$ | Validation error using look-back $L_i$           |\n| $\\varepsilon \\ge 0$                      | Threshold for **non-significant improvement**    |\n\nThen MEW is defined as:\n\n\n$$\\\\text{MEW} _{\\\\varepsilon}(M, D, H) = \\\\min\\\\left\\\\{ L _i \\in \\\\Lambda : \\\\forall j > i, \\; S(L _j) \\ge S(L _i) -\\\\varepsilon \\\\right\\\\}$$\n\n**In words:** MEW is the smallest look-back length beyond which no further improvement larger than $\\varepsilon$ is observed. Since MEW depends on $(M, D, H, E)$, it is setup-specific and must be re-estimated for different tasks.\n\n\n---\n\n> **Q2.** _Could you quantify the computational overhead of the IBF and HTM modules in more detail (e.g., in terms of FLOPs or additional parameters) to better clarify the trade-offs of the hybrid approach?_\n\nWe appreciate the reviewer’s interest in this aspect. \n\n**IBF introduces negligible overhead**, as it is implemented as a lightweight MLP consisting of two Linear layers with one ReLU activation in between. Both input and output dimensions match the Transformer’s patch embedding size (typically 16 or 128 in PatchTST). This structure results in only a small number of additional parameters and almost no increase in FLOPs, making IBF extremely efficient in practice.\n\n**HTM significantly reduces computational cost** by replacing one Transformer layer with a Mamba layer, while keeping the overall model depth identical to PatchTST (e.g., three layers in total). In our implementation, all $K$ subsequences share the same Transformer, so the main overhead difference lies in the substitution of one Transformer layer with one Mamba layer. Below, we provide detailed comparisons under the typical setting where $L=1024$, patch size = 16, stride = 8 (thus 128 patches), and $d_{\\text{model}} = 128$:\n\nFor datasets like **Weather, Traffic, ETTm1/2, and Electricity** (input shape: 1 × 128 × 128)\n\n| Layer       | Params (K) | MACs (M) | FLOPs (M) |\n| ----------- | ---------- | -------- | --------- |\n| Transformer | 66.05      | 12.58    | 25.43     |\n| HTM         | 58.24      | 2.82     | 5.67      |\n\nFor **ETTh1 and ETTh2** (input shape: 1 × 128 × 16):\n\n|Layer|Params (K)|MACs (M)|FLOPs (M)|\n|---|---|---|---|\n|Transformer|1.09|1.31|2.75|\n|Mamba|1.68|0.33|0.66|\n\nFrom these numbers, we observe that **Mamba yields significantly lower computational complexity** than Transformer, especially in terms of FLOPs, while keeping parameter counts in the same range.\n\n\n\n**The theoretical complexity analysis further supports these observations.** In PatchTST, the patching mechanism reduces the Transformer’s effective input length from $L$ to $L / (P \\cdot K)$, where $P$ is the patch length and $K$ the number of subsequences. Combined with the full-sequence Mamba processing, the overall complexity becomes:\n\n$$O(L/P)+O((L/PK)^2)$$\n\nAlthough the latter term still exhibits quadratic complexity, appropriate choices of $P$ and $K$ can maintain $L/P$ within an acceptable constant range.\n\n**In summary, the proposed architecture offers clear advantages in both efficiency and performance.** IBF adds almost no overhead, while HTM achieves a 2–5× reduction in FLOPs relative to a pure Transformer layer. As shown in Figure 4, this comes with consistent improvements in forecasting accuracy, confirming that combining Mamba and Transformer captures complementary temporal features without increasing computational burden.\n\nWe hope this detailed explanation clarifies the trade-offs and justifies the practicality of our hybrid design.\n\n\n---\n\nWe hope that these detailed explanations address your concerns. Thank you again for your thoughtful and constructive review."}, {"review_id": "BmA5oI2XVK", "reviewer": "Reviewer_TLHM", "summary": "This paper introduces an empirical metric designed to estimate the Maximum Effective Window for transformer-based models. The authors propose the IBF and  HTM as strategies for mitigating noise and selectively forgetting information in long sequences. Experimental evaluations validate the effectiveness of these approaches in enhancing MEW.", "questions": "see above", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Unlike prior work that primarily focused on performance metrics, this paper emphasizes a model’s capacity to utilize historical information—a valuable reference for future model design.\n- The ideas behind IBF and HTM are well-articulated, and the experimental setup is relatively comprehensive.\n- The paper grounds the theoretical underpinnings of IBF in solid information theory concepts, including rigorous mathematical derivations. \n- The model-agnostic design of the proposed components broadens their applicability across various transformer architectures, significantly enhancing their practical relevance.\n    \n\n **Weaknesses**\n\n- The difference between interval split and block split is not thoroughly clarified in the introduction. The methodology lacks specific guidance on how to choose the optimal block size.\n- Expanding the historical window in large-scale models could provide more meaningful insights. Although the authors touch upon this in the  limitations section and acknowledge the computational challenges, further exploration in this direction would be valuable.", "was_revised": true, "final_justification": "Keep the positive score.", "rebuttal": "Thank you very much for your valuable feedback and suggestions. Below, we address each point in turn.\n\n___\n\n> **Q1.** _The difference between interval split and block split is not thoroughly clarified in the introduction. The methodology lacks specific guidance on how to choose the optimal block size._\n\nThank you for this insightful question. We agree that further clarification is warranted.\n\nThe key distinction between interval split and block split lies in the type of temporal semantics each method preserves:\n\n- **Interval split** distributes patches across $K$ blocks in a round-robin fashion, as defined in Eq. (11), i.e., patch $j$ is assigned to block $b_i$ if $j \\equiv i \\pmod{K}$. This method emphasizes **global temporal diversity** within each block, enabling the model to access patterns from different time positions. It generalizes the even–odd downsampling strategy in SCINet^[1] to $K$-way interleaving and is especially suited for data with globally distributed seasonal structures.\n- **Block split**, in contrast, divides the sequence into **contiguous segments**, as defined in Eq. (12), where each block $b_i$ spans a local window. This method is particularly effective for time series with **local periodicity**, where patterns repeat consistently within specific time intervals (e.g., daily cycles in electricity consumption).\n\n\n\n**Interval Split vs. Block Split:** In essence, **interval split captures global variation**, while **block split preserves local coherence**. Both methods aim to retain the semantic meaning of the original long sequence after partitioning. We empirically compared the two splitting strategies across multiple datasets, as shown in Figure 4 (right). The results indicate that while both methods achieve comparable performance overall, their relative effectiveness varies across datasets, depending on the dominant temporal characteristics. For instance, datasets with strong periodicity tend to benefit more from block split. This validates our intuition and suggests that the choice of split strategy may depend on dataset-specific dynamics—an aspect we plan to investigate further in future work.\n\n**Block Size**: Regarding the choice of **block size**, we provide further details in Appendix C.1, where we study the impact of the number of partitions $K$ as a tunable hyperparameter. Under the block split scheme, the block size is computed as $L/K$, where $L$ is the total sequence length (we set $L = 1024$ in our experiments). Our empirical results show that $K = 4$ typically yields the best performance, corresponding to **a block size of 256**. This setting offers a good trade-off between retaining meaningful local patterns and reducing Transformer input length.\n\n___\n\n> **Q2.** Expanding the historical window in large-scale models could provide more meaningful insights. Although the authors touch upon this in the limitations section and acknowledge the computational challenges, further exploration in this direction would be valuable.\n\nThank you for this thoughtful suggestion. We fully agree that extending our method to large-scale models represents an important and promising direction for future research.\n\nIn this work, our focus is on **analyzing and improving the MEW**, and we conduct extensive experiments on widely-used compact models such as **PatchTST, Transformer, Autoformer, and Informer**. These choices allow us to systematically study how our proposed HTM and IBF modules contribute to **enhancing MEW** while keeping the computational cost tractable.\n\nWe acknowledge that scaling to state-of-the-art large models, such as **Time-MoE**^[2] or **Timer-XL**^[3], would offer additional insights—especially in extremely long sequence modeling scenarios involving billions of parameters and trillions of time steps. However, due to the **significant training and resource demands**, this was beyond the scope of the current study.\n\n\n\n\n[1] Liu, Minhao, et al. \"Scinet: Time series modeling and forecasting with sample convolution and interaction.\" _Advances in Neural Information Processing Systems_ 35 (2022): 5816-5828.\n\n[2] Shi, Xiaoming, et al. \"Time-moe: Billion-scale time series foundation models with mixture of experts.\" _arXiv preprint arXiv:2409.16040_ (2024)\n\n[3] Liu, Yong, et al. \"Timer-xl: Long-context transformers for unified time series forecasting.\" _arXiv preprint arXiv:2410.04803_ (2024).\n\n\n\n---\n\nWe hope that these detailed explanations address your concerns. Thank you again for your thoughtful and constructive review."}, {"review_id": "oxGVQC6z7W", "reviewer": "Reviewer_up9s", "summary": "This paper introduces the Maximum Effective Window (MEW) and proposes two model-agnostic modules: Information Bottleneck Filter (IBF) for reducing noise/redundancy using information theory, and Hybrid-Transformer-Mamba (HTM) that combines Mamba's selective long-range modeling with Transformer's short-sequence strengths. Their integrated PIH model successfully extends effective window lengths while achieving state-of-the-art performance across seven benchmark datasets.", "questions": "Please refer to the weaknesses section for the main questions regarding this paper.\nAdditionally,\nRecent studies have demonstrated that certain models can achieve improved performance with input lengths of 1000-2000+ timesteps. Given that your approach focuses on extending effective window lengths, how would your IBF and HTM modules perform when applied to models that are already capable of handling very long sequences?", "limitations": "This paper adequately addressed the limitations", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strength**\n1. While recent studies focus on developing methods that work better with short input sequences, this research takes a different approach by extending the investigation to various input lengths and seeking methodologies to effectively leverage longer sequences.\n2. The paper successfully combines the complementary strengths of Transformers and Mamba architectures, where Mamba handles selective long-range dependencies and Transformers excel at modeling shorter subsequences.\n\n**Weaknesses**\n1. The proposed MEW metric appears to be essentially equivalent to hyperparameter searching over input sequence lengths. The paper fails to clearly articulate how MEW differs from conventional parameter tuning approaches. If MEW is simply finding the optimal input length through grid search, then the performance improvements may be attributed to more extensive hyperparameter exploration rather than a fundamental methodological contribution.\n2. While PIH development is presented as the main contribution, the performance comparisons lack evaluation against more recent state-of-the-art models. If the contribution is modular development, the paper should demonstrate that these modules can be applied to other contemporary models to achieve superior performance. If the contribution is model development, direct comparisons with recent state-of-the-art models are essential to establish the advancement.\n3. The paper provides insufficient analysis of how the IBF functions within the model and contributes to performance improvements. While the theoretical foundation is provided, there is a lack of empirical analysis demonstrating what types of information IBF filters out, how it affects the learned representations, and why this leads to better forecasting performance.", "was_revised": true, "final_justification": "The authors addressed several concerns through additional experiments showing consistent improvements. However, all evaluations are limited to medium-length sequences without testing on ultra-long contexts that modern SOTA models can handle. This weakens their claims about long-range dependency modeling. Without validation on truly long sequences, it's unclear if their method generalizes beyond models that already struggle with longer inputs. Therefore, I maintain my original score.", "rebuttal": "Thank you very much for your valuable feedback and suggestions. Below, we address each point in turn.\n\n\n---\n\n> **Q1.** _The proposed MEW metric appears to be essentially equivalent to hyperparameter searching over input sequence lengths. The paper fails to clearly articulate how MEW differs from conventional parameter tuning approaches. If MEW is simply finding the optimal input length through grid search, then the performance improvements may be attributed to more extensive hyperparameter exploration rather than a fundamental methodological contribution._\n\nThank you for raising this important point. While MEW does involve evaluating model performance across different input sequence lengths—similar to hyperparameter tuning—we emphasize that **MEW is not a tuning strategy**, but rather a **diagnostic metric** designed to quantify a model’s effective ability to leverage long-term historical information.\n\nSpecifically:\n\n- **Objective**: Traditional hyperparameter tuning aims to find the best lookback length to optimize performance. In contrast, **MEW focuses on analyzing where performance saturates**, revealing a model's **limitation in capturing long-range dependencies**, rather than optimizing for best-case performance.\n    \n- **Analytical value**: MEW provides a **model-centric lens** to assess historical information utilization capacity. For instance, a model with a larger MEW can leverage longer-term context effectively, which is crucial for long-horizon forecasting.\n    \n\nAs shown in **Figure 3(a)** and **Table 2**, we apply our proposed HTM and IBF modules to multiple backbone models (PatchTST, Transformer, Autoformer, Informer), and consistently observe that both **MEW increases** and **forecasting performance improves**. This demonstrates that enhancing a model's MEW can lead to more accurate long-term predictions—highlighting MEW as an insightful metric beyond simple tuning.\n\n---\n\n> **Q2.** _While PIH development is presented as the main contribution, the performance comparisons lack evaluation against more recent state-of-the-art models. If the contribution is modular development, the paper should demonstrate that these modules can be applied to other contemporary models to achieve superior performance. If the contribution is model development, direct comparisons with recent state-of-the-art models are essential to establish the advancement._\n\nWe appreciate this suggestion. Our **primary contribution lies in modular design**: specifically, we introduce the IBF and HTM modules, which can be integrated into a wide range of forecasting models to improve their ability to utilize long-range dependencies.\n\nTo support this, we have already validated our modules across multiple backbones from different years and architectures—**Transformer (2018), Informer (2020), Autoformer (2021), PatchTST (2023), and iTransformer (2024)**—as shown in **Table 2** and **Table 7**. In all cases, the integration of IBF and HTM improves both MEW and forecasting accuracy.\n\nTo further address the reviewer’s concern, we have conducted **additional experiments on the recently proposed PowerFormer (2025)**. The table below summarizes the MEW and MSE results after integrating our modules ($L \\in [72, 96, 120, 144, 168, 192, 273 228, 336, 512, 1024, 1440]$,$T=720$):\n\n|             |     | PowerFormer (original) | +IBF+HTM (ours) |\n| ----------- | --- | ---------------------- | --------------- |\n| Etth1       | MEW | 512                    | 1024            |\n|             | MSE | 0.439                  | 0.433           |\n| Etth2       | MEW | 336                    | 1024            |\n|             | MSE | 0.376                  | 0.372           |\n| Ettm1       | MEW | 512                    | 1024            |\n|             | MSE | 0.412                  | 0.397           |\n| Ettm2       | MEW | 336                    | 1024            |\n|             | MSE | 0.351                  | 0.340           |\n| weather     | MEW | 512                    | 512             |\n|             | MSE | 0.310                  | 0.305           |\n| Electricity | MEW | 512                    | 1024            |\n|             | MSE | 0.198                  | 0.193           |\n| Traffic     | MEW | 512                    | 1024            |\n|             | MSE | 0.430                  | 0.425           |\n\n\n\nThese new results further confirm that **our modules generalize well** and can enhance even the latest forecasting models. We will incorporate this additional evaluation into the final version.\n\n---\n\n> **Q3.** _The paper provides insufficient analysis of how the IBF functions within the model and contributes to performance improvements. While the theoretical foundation is provided, there is a lack of empirical analysis demonstrating what types of information IBF filters out, how it affects the learned representations, and why this leads to better forecasting performance._\n\nWe thank the reviewer for raising this important point. While the paper includes theoretical motivation for the Information Bottleneck Filter (IBF), we now elaborate on its **empirical behavior and contribution**, focusing on the following two aspects:\n\n**1. What IBF filters out**  \nIBF selects input patches that are **most informative for the forecasting target**, based on a mutual information approximation. As shown in Eq. (5):\n\n$$\\mathbf{c}_i = \\operatorname{sigmoid}(\\operatorname{MLP}(\\mathbf{z}_i))$$\n\neach input token $\\mathbf{z}_i$ is assigned a learned importance score $c_i$. A well-trained IBF learns to assign **higher scores to informative tokens** and suppress irrelevant or noisy ones.\n\nIn **Figure 3(c)**, we visualize the top-20 patches selected by IBF (i.e., those with the highest $c_i$ values) for a sample from the Electricity dataset. These selected patches are primarily located around **peaks and transition points**, which are critical to the dynamics of the signal. We observe similar behavior across other datasets (e.g., Traffic, Weather, Exchange), where IBF consistently highlights high-gradient, or periodic regions, while suppressing **redundant or low-variance segments**.\n\nThese observations indicate that IBF functions as a **saliency-aware filter**, learning to emphasize structure-rich regions that are more predictive of future values.\n\n**2. Why this leads to better forecasting**  \nBy filtering out uninformative historical inputs and retaining only high-salience patches, IBF improves forecasting in the following ways:\n\n- It **reduces attention dilution**, allowing attention mechanisms to focus on relevant content;\n    \n- It **mitigates overfitting**, especially in long-input regimes where noise can dominate;\n    \n- It **improves temporal abstraction** by ensuring that only informative components are passed to downstream layers.\n    \n\nThis filtering process aligns well with the **Information Bottleneck principle**, enhancing the model’s ability to generalize across time and datasets.\n\nDue to NeurIPS 2025 policy constraints, we regret that we could not include additional visualizations or case-specific analyses in the current submission. However, we will incorporate more case studies and dataset-level statistics (e.g., patch entropy, activation frequency) in the revised version to further illustrate the empirical behavior of IBF.\n\n\n\n---\nWe hope that these detailed explanations address your concerns. Thank you again for your thoughtful and constructive review."}]

Author rebuttal:

Meta-review: This paper introduces the Maximum Effective Window (MEW) metric to assess a model's ability to effectively utilize the lookback window, with the IBM to reduce redundancy and noise and the architecture of Transformer + Mamba. 

The experimental results look convincing. Most doubts of reviewers seem to be fixed well after responses and discussions. However, some reviewers still remain skeptical about the Mamba + Transformer design. The inspired recognition seems not to be straight. Besides, it also seems unclear how much the theoretical analysis of IBM can provide practical guidance.
Decision: Accept (poster)