SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion
Authors: Lu Han
Xu-Yang Chen
Han-Jia Ye
De-Chuan Zhan
Venue: NeurIPS 2024 poster (2024)
Keywords: multivariate time series forecasting
channel interaction
centralized structure
efficiency
Primary area: deep_learning_architectures

Abstract: Multivariate time series forecasting plays a crucial role in various fields such as finance, traffic management, energy, and healthcare.  Recent studies have highlighted the advantages of channel independence to resist distribution drift but neglect channel correlations, limiting further enhancements. Several methods utilize mechanisms like attention or mixer to address this by capturing channel correlations, but they either introduce excessive complexity or rely too heavily on the correlation to achieve satisfactory results under distribution drifts, particularly with a large number of channels. Addressing this gap, this paper presents an efficient MLP-based model, the Series-cOre Fused Time Series forecaster (SOFTS), which incorporates a novel STar Aggregate-Redistribute (STAR) module. Unlike traditional approaches that manage channel interactions through distributed structures, \textit{e.g.}, attention, STAR employs a centralized strategy to improve efficiency and reduce reliance on the quality of each channel. It aggregates all series to form a global core representation, which is then dispatched and fused with individual series representations to facilitate channel interactions effectively. SOFTS achieves superior performance over existing state-of-the-art methods with only linear complexity. The broad applicability of the STAR module across different forecasting models is also demonstrated empirically. We have made our code publicly available at https://github.com/Secilia-Cxy/SOFTS.

SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion
Lu Han; Xu- Xu-Yang Chen; Han-Jia Ye, De-Chuan Zhan
School of Artificial Intelligence, Nanjing University, China National Key Laboratory for Novel Software Technology. Nanjing University, China (hanlu, chenxy, yehj, zhandc) @lamda. njj.. edu. cn
Abstract
Multivariate time series forecasting plays a crucial role in various fields such as fi- nance, traffic management, energy, and healthcare. Recent studies have highlighted the advantages of channel independence to resist distribution drift but neglect channel correlations, limiting further enhancements. Several methods utilize mech- anisms like attention or mixer to address this by capturing channel correlations, but they either introduce excessive complexity or rely too heavily on the corre- lation to achieve satisfactory results under distribution drifts, particularly with a large number of channels. Addressing this gap, this paper presents an efficient MLP-based model, the Series-cOre Fused Time Series forecaster (SOFTS), which incorporates a novel STar RegregatteRddistribuuee (STAR) module. Unlike tradi- tional approaches that manage channel interactions through distributed structures, egg., attention, STAR employs a centralized strategy to improve efficiency and reduce reliance the quality of each channel. It aggregates all series to form a global core representation. which is then dispatched and fused with individual series representations to facilitate channel interactions effeettively. SOFTS achieves superior performance over existing state-of-the of-the-art methods with only linear com- plexity. The broad applicability of the STAR module across different forecasting models is also demonstrated empirically. We have made our code publicly available at putttps//githhng com/
www../ow.eeeeeeeee het///eeiilaaaCee Saliaaaaayy//FSSSS /OFTS
Introduction
Time series forecasting plays a critical role in numerous applications across various fields, including environment [9], traffic management [16], energy [17], communication [45), and healthcare [28]. The ability accurately predict future values based on previously observed data is fundamental for decision-making policy development, and strategic planning in these areas. Historically, models such as ARIMA and Exponential Smoothing were standard forecasting, noted for their simplicity and effectiveness in certain contexts (2). However, the emergence of deep learning models, particularly those exploiting structures like Recurrent Neural Networks (RNNs)
30 and Convolutional
Neural Networks (CNN) [ ( has shifted the paradigm towards more complex models capable of understanding intricate patterns time series data. To overcome the inability to capture long-temn -erg-term dependencies, Transformer-basee models have been popular direction and achieved remarkable
performance, especially on long-term multivariate time series forecasting [5] 29 27].
Earlier on, Transformer-bart based methods perform embedding techniques like linear or convolution layers to aggregate information from different channels, then extract information along the temporal
dimension via attention mechanisms 51 37 52 However, such channel mixing structures were
'Equal Contribution
t Corresponning Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

found vulnerable to the distribution drift, to the extent that they were often less effective than simpler methods like linear models 44 Consequently, some studies adopted frramnel-indeeedeenn Yet, these methods overlooked the correlation between channels, thereby hindering further improvements in model performance. Subsequent studies captured this correlation information through mechanisms such as attention, achieving better outcomes, and demonstrating the necessity of information transfer between channels [50] 35| 27].
strategy and achieved favorable results [2 24 36
However, these approaches either employed attention mechanisms with high complexity [27 or
struggled to achieve state-of-the-ar (SOTA) performance [7]. Therefore, effectively integrating the robustness of channel independence and utilizing the correlation between channels in simpler and more efficient manner is crucial for building better time series forecasting models.
In response to these challenges, this study introduces an efficient MLP-based model, the Series-cOre Fused Time Series forecaster (SOFTS), designed to streamline the forecasting process while also enhancing prediction accuracy. SOFTS first embeds the series on multiple channels and then extracts the mutual interaction by the novel STar Aggregate-Redistriuuee Redistribute (STAR) module. The STAR the
heart of SOFTS ensures scalability and reduces computational demands from the common quadratic complexity to only linear. To achieve that, instead of employing distributed interaction structure, STAR employs a centralized structure that first gets the global core representation by aggregating the information from different channels. Then the local series representation is fused with the core representation to realize the indirect interaction between channels. This centralized interaction not only reduces the comparison complexity but also takes advantage of both channel independence and aggregated information from all the channels that can help improve the local ones 42). Our empirical results show that our SOFTS method achieves better results against current state-of-the-art methods with lower computation resources. Besides, SOFTS channels or time steps, which is difficult for many methods based on Transformer without specific modification. Last, the newly proposed STAR is a universal module that can replace the attention in many models. Its efficiency and effectiveness are validated on various current transformer-based time series forecasters. Our contributions are as follows:
scale to time series with large number of
1. We present Series-cOre Fused Time Series (SOFTS) forecaster, a simple MLP-based model that demonstrates state-of-the-arr performance with lower complexity.
2. We introduce the STar Aggregate-Redistrrbute (STAR) module, which serves the foundation of SOFTS. STAR is designed as a centralized structure that uses a core to aggregate and exchange information from the channels. Compared to distributed structures like attention, the STAR not only reduces the complexity but also improves robustness against anomalies in channels.
3. Lastly, through extensive experiments, the effectiveness and scalability of SOFTS are validated. The universality of STAR is also validated on various attention-based time series forecasters.
2 Related Work
Time series forecasting, Time series forecasting a critical area of research that finds applications in both industry and academia. With the powerful representation capability of neural networks, deep
forecasting models have undergone rapid development 23 40 39 4 5 LS 32 Two widely used methods for time series forecasting are recurrent neural networks (RNNs) and convolutional neural networks (CNNs). RNNs model successive time points based on the Markov assumption
14 30
while CNNs extract variation information along the temporal dimension using techniques such as temporal convolutional networks (TCNs) ① g]. However, due to the Markov assumption in RNN and the local reception property in TCN, both of the two models are unable to capture the long-term dependencies in sequential data. Recently, the potential of Transformer models for long-term time series forecasting tasks has garnered attention due to their ability to extract long-term dependencies via the attention mechanism [1 37 52
Efficient long-term multivariate forecasting and channel independence. Long-term multivariate time series forecasting is increasingly significant in decision-making processes (9. While Trans- formers have shown remarkable efficacy in various domains R4 their complexity poses challenges in long-term forecasting scenarios. Efforts to adapt Transformer-basee models for time series with reduced complexity include the Informer, which utilizes a probabílistic subsampling strategy for more efficient attention mechanisms [51) and the Autoformer, which employs autocorrelation and fast

---

Suar Aggragiit- Sagatt--aadtttuutte Module x Core
Input
111
Aggnegati
Output
Channel
Repeot Concal
Redistributo
MLP
Linear
Residual
Time
Figure 1: Overview of our SOFTS method. The multivariate time series is first embedded along the temporal dimension to get the series representation for each channel. Then the channel correlation is captured by multiple layers of STAR modules. The STAR module utilizes centralized structure that first aggregates the series representation to obtain a global core representation, and then dispatches and fuses the core with each series, which encodes the local information.
Fourier transforms to expedite computations 37 Similarly, FEDformer applies attention within the frequency domain using selected components to enhance performance [52). Despite these innovations, models mixing channels in multivariate series often exhibit reduced robustness to adapt to distribution
drifts and achieve subpar performance [48] [T]. Consequently, some researchers have adopted a
counnel-independent approach, simplifying the model architecture and delivering robust results as well 22 24]. However, ignoring the interactions among variates can limit further advancements. Recent trends have therefore shifted towards leveraging attention mechanisms capture channel correlations [50 35 27]. Even though the performance is promising, their scalability is limited on
large datasets. Another stream of research focuses on modeling time and channel dependencies through simpler structures like MLP (49 7 44). Yet, they usually achieve sub-optimal performance compared SOTA transformer-based methods, especially when the number of channels is large.
In this paper, we propose a new based method that breaks the dilemma of performance and efficiency, achieving state-of-the-ar performance with merely linear complexity to both the number of channels and the length of the lookback window.
3 SOFTS
Multivariate time series forecasting (MTSF) deals with time series data that contain multiple variables, or channels, at each time step. Given historical values X E CRCL where I. represents the length of the lookback window, and C' is the number of channels. The goal of MTSF to predict the future values Y E RCCH, where H >> 0 is the forecast horizon.
3.1 Overview
Our Series-cOre Fused Time Series forecaster (SOFTS) comprises the following components and its structure is illustrated in Figure
Reversible instance normalization. Normalization is a common technique to calibrate the distribu- tion of input data. In time series forecasting, the local statisties of the history usually removed to stabilize the prediction of the base forecaster and restore these statistics to the model prediction (18). Following the common practice in many state-of-the-art models (29 27], we apply reversible instance normalization which centers the series to zero means, scales them to unit variance, and reverses the
normalization on the forecasted series. For PEMS dataset, we follow Liu et al. [27] selectively perform normalization according to the performance.
Series embedding. Series embedding an extreme case of the prevailing patch embedding time series (29), which equivalent to setting the patch length to the length of the whole series [27]. Unlike patch embedding, series embedding does not produce extra dimension and thus less complex than patch embedding. Therefore, in this work, perform series embedding the lookback window.
3

---

Series
Core
Interaction
Attention, Interaction
STAR (ours)
Figure 2: The comparison of the STAR module and several common modules, like attention, GNN and mixer. These modules employ distributed structure perform the interaction, which relies the quality each channel. On the contrary, our STAR module utilizes a centralized structure that first aggregates the information from all the series obtain comprehensive core representation. Then the core information is dispatched to each channel. This kind of interaction pattern reduces not only the complexity of interaction but also the reliance the channel quality.
Concretely, we use a linear projection to embed the series of each channel to So Rxdd where the hidden dimension:
So = EmbeddingggX (X).
Channel interaction.
The series embedding is refined by multiple layers of STAR modules:
S. STAR(S:-1), == 1,2,.....
2
The STAR module utilizes star-shaped structure that exchanges information between different channels, as will be fully described in the next section.
Linear predictor. After N layers of STAR, forecasting results. Assume the output series representation of layer to be SN. the prediction Ý E RC*H is computed as:
linear predictor (TR. R to produce the
Linear(S,y
3.2 STar Sngregate-Redistriinn Module
Our main contribution is simple but efficient STar capture the dependencies between channels. Existing methods employ modules like attention to extract such interaction. Although these modules directly compare the characteristics of each pair, they are faced with the quadratic complexity related to the number of channels. Besides, such a distributed structure may lack robustness when there are abnormal channels for the reason that they rely on the extract correlation between channels. Existing research on channel independence has already proved the untrustworthy correlations non-stationary time series [48 [T]. To this end, we propose the STAR module to solve the inefficiency of the distributed interaction modules. This module is inspired by the star-shaped centralized system in software engineering, where instead of letting the clients communicate with each other, there is a server center to aggregate and exchange the information [3 [0O, whose advantage is efficient and reliable. Following this idea, the STAR replaces the mutual series interaction with the indirect interaction through a core, which represents the global representation across all the channels. Compared to the distributed structure, STAR takes advantage of the robustness brought by aggregation of channel statistics [II] and thus achieves even better performance. Figure Zillustrates the main idea of STAR and its difference between existing
(STAR) module to
models like attention 34], GNN (20 and Mixer 33
Given the series representation of each channel input, STAR first gets the core representation of the multivariate series, at the heart of our SOFTS method. We define the core representation as follows:
Definition 3.1 (Core Representation). Given a multivariate series with C channels (81, $2..... sc); the core representation is a vector generated by an arbitrary function f with the following form:
f(31, 82 ................................ 8c)

---

The core representation encodes the global information across all the channels. To obtain such repre- sentation, we employ the following form, which inspired by the Kolmogorov-Ame representation theorem [1 and DeepSets [46]
0,
Pool(MLPPISS...
(3)
where MLP1 R ++ Rd is a projection that projects the series representation from the series hidden dimension d to the core dimension d', composing two layers with hidden dimension d and GELU [3] activation. Stoch_ Stoch_Pool the stochastic pooling 47 that get the core representation o Rdd by aggregating representations of C' series. Stochastic pooling combines the advantages of mean and max pooling. The details of computing the core representation can be found in Appendix B.2 Next, we fuse the representations of the core and all the series:
F; Repeat_ (yppaat_ooncatSs-11100
(4)
S. (L.) + S
The Repeat_Concat Concat operation concatenate the core representation o to each series representation, and we get the F; E (xx(d+))) Then another MLP (MLP2 Rd-d' + IRd) is used to fuse the concatenated presentation and project it back the hidden dimension i.e., S RCxd xd Like many
deep learning modules, we also add a residual connection from the input to the output [1]
3.3 Complexity Analysis
We analyze the complexity of each component of SOFTS step by step concerning window length L, number of channels C, model dimension d, and forecasting horizon H. The complexity of the reversible instance normalization and series embedding is O(CL) and OCCLd) respectively. In STAR, assuming d' = d, the MLP, is a Rd - Rd mapping with complexity O(Cd)). Stoch_ computes the softmax along the channel dimension, with complexity O(Cd). The ML.P on concatenated embedding has the complexity OCCd²). The complexity of the predictor is OCCHH). In all. the complexity of the encoding part is OC + Cdl + CddH), which is linear to C, L, and H. Ignoring the model dimension d, which is a constant in the algorithm and irrelevant to the problem, we compare the complexity of several popular forecasters Table
Table 1: Complexity comparison between popular time series forecasters concerning window length
L, number of channels C' and forecasting horizon H. Our method achieves only linear complexity.
SOFTS (ours)
iTransformer
PatchTST
Transformer
Complexity O(CL + CH) O(C² + CL + CH) OCC1² + CH) OCCL + L2 HL + CH)
4 Experiments
Datasets. To thoroughly evaluate the performance of our proposed SOFTS, we conduct extensive experiments on widely used, real-world datasets including ETT (4 subsets), Traffic, Electricity, Weather (3]) Solar-Energy [22) and PEMS (4 subsets) 25 Detailed descriptions of the datasets
can be found in Appendix
4.1 Forecasting Results
Compared methods. We extensively compare the recent Linear-based or MLP-based methods, including DL.inaa [48]. TSMixer 77. TiDE [6]. We also consider Transformer-basee methods
including FEDformer [52], Stationary (26). PatchTST [29), Crossformer (50), iTransformer (2] and
CNN-based methods including SCINet (25). TimesNet [38].
Forecasting benchmarks. The long-term forecasting benchmarks follow the setting in In- former [51] and SCINet (25]. The lookback window length (L) is set to 96 for all datasets. We set the prediction horizon (Hf) to [12, 24, 48,96) .6} for PEMS and (96, 192, 336, 720) for others. Performance comparison among different methods is conducted based on two primary evaluation metrics: Mean Squared Error (MSF) and Mean Absolute Error (MAF). The results of PatchTST and TSMixer are reproduced for the ablation study and other results are taken from iTransformer 27
5

---

Implementation details. We use the ADAM optimizer [19] with an initial learning rate of 3 x 10 4. This rate is modulated by cosine learning rate scheduler. The mean squared error (MSE) loss function is utilized for model optimization. We explore the number of STAR blocks N within the set 2,3, 4), and the dimension of series within {128, 256, 512). Additionally, the dimension of the core representation d' varies among (64, 128, 256, 512]. Other detailed implementations are described in Appendix B.3
Main results. As shown in Table 2 SOFTS has provided the best or second predictive outcomes in all 6 datasets on average. Moreover, when compared to previous state-of-the-art methods, SOFTS has demonstrated significant advancements. For instance, on the Traffic dataset, SOFTS improved the average MSE error from 0.428 to 0.409, representing a notable reduction of about 4.4%. On the PEMS07 dataset, SOFTS achieves substantial relative decrease of 13.9% in average MSE error, from 0.101 0.087. These significant improvements indicate that the SOFTS model possesses robust performance and broad applicability in multivariate time series forecasting tasks, especially in tasks with a large number of channels, such as the Traffic dataset, which includes 862 channels, and the PEMS dataset, with varying range from 170 to 883 channels.
Table 2: Multivariate forecasting results with horizon H E {12, 24, 48, (6 for PEMS and И E (96, 192, 336, 720} for others and fixed lookback window length L 96. Results are averaged from all prediction horizons. Full results are listed in Table 6
Models
SOFTS (ours) Transfommer PaccTST
TSMixer
Crossformer
Timesher
Dlinneer
SCINat
PEDformer
Suatonary
Meurie
MSE
MAE MSE MAG MSE MAE MSE MAE MSE MAB MSE MAE) MSE MAE MSE MAE MSE MAE MSB MAE MSE MAE
ECL
0.174
0.264
0.178 0.271 (0.189 0.276/0.186 0:287 0.244 0.334 (0:251 0,344 (0.192 0.295 (0.212 0.300/0.268 (0.268 0.365 (0.214 0.327
0.827|0.199 19.19
Tralicc
0,409
0.267
0:428
0.282
(0:454
0.286/0.522 0,357 0.550 0.304 76.66 0:473 (3.00 0.336 (0.625 0.383 (0.804 0.309j0.610 0.376|0.624 0,340
Weather
0.255
0.278
0.258
0.278
(02256
0.229)
256
0.222 0.259) 0.315 (0.271 0.320 0.287 (0.205 0.317 (0.992 a 0.363 (0.399
0.500|o.288 0.314
Solae Energy
0.229
0.256
0233
0.262
(0.236
0.266/0.260 0.297 0.641 0.639 (0.347 0.017 0,319 (0,330 O 0.101 (0.282 0.375j0.291 0.381
j..26 0.381
ETTaal
0.393
0.403
0.407
0.41
10.
0.405j0.990 398 0.407 0.513 0.406 (0.419 (3,00
0.406 (0.4015 O 0.407 (0-485 0.48 (0.448
(o.52).481 0,4%
ETTm2
0.257
0.330
0.288
0.332
(0.287
2.99 0.333 0.757 0.630 0.358 0.404 0.333
(3.300 0.101 (0577 0.537|0.305 0.349j0.306 0,347
FTTsl
0.449
0.442
0.454
0.447
[0.453
0:446/0
0.452 0.520 0,522 0.507
0.450 (0.4%
(0.747 0664700.40 0.40000.570 0.537
ETTh2
0.373
0.400
0.383
0.407 [0.285
0.410
0:417 0,442 0.684 (0.611 0,550 0.434
0:427 (0.599 0.515 0.954 (0:47 0.449|0.526 0,516
PEMS01
0.104
0.210
0.113
0.221 (0.137 0.240j0.119 0:233 0.169 0.281 (0.306 0:419/0. 14447 0.248 (0.278 0.375 10.114 o 0.22410.213 0.327j0.147 0.249
PEMS04
0.102
0.208
0.111
0.221
(0.145
0.249|0.108 0:215 0.209 0.314 (0.353 0.437
0.241 (0.295 0.288 (0.092 0.202 (0.25] 0.837|o.127 0.240
PEMS07
0.087
0.184
0.101
0.204
(0.144 0.233J0.112
0,217 0.235 0.315 0.350 0.440
0.225 (0.329 0.795 10.119 0.234J0.165 0.283jo.127 0,230
PEMS08
0,138
0.219
0.150
0.226
(0.200 0.275j0.165
0,261 0.208 0,307 (0.44) 0.464
0:271 (0.579 0.416
(0.158 0.244j0.286 0.355 0.276
Model efficiency. Our SOFTS model demonstrates efficient performance with minimal memory and time consumption. Figure 3b illustrates the memory and time usage across different models on the Traffic dataset, with lookback window L = 96, horizon H 720, and batch size 4. Despite their low resource usage, Linear-based or MLP-based models such as DLinear and TSMixer perform poorly with large number of channels. Figure (a explores the memory requirements of the three best- pest-performig models from Figure 36] This figure reveals that the memory usage of both PSt and iTransformer escalates significantly with an increase in channels. In contrast, our SOFTS model maintains efficient operation, with its complexity scaling linearly with the number of channels, effectively handling large channel counts.
4.2 Ablation Study
In this section, the prediction horizon (H) is set to {12, 24, 48, 96} for PEMS and (96, 192, 336, 720) for others. All the results are averaged on four horizons. If not especially concerned, the lookback window length (L) is set to 96 as default.
Comparison of different pooling methods. The comparison of different pooling methods in STAR is shown in Table The term "w/o STAR" refers to scenario where an MLP is utilized with the Channel Independent (CI) strategy, without the use of STAR. Mean pooling computes the average value of all the series representations. Max pooling selects the maximum value of each hidden feature among all the channels. Weighted average learns the weight for each channel. Stochastic pooling applies random selection during training and weighted average during testing according to the feature value. The result reveals that incorporating STAR into the model leads to a consistent enhancement

---

24003
PatckTST
TimesNet
iTranfermer
2000
(ours)
Scacionary
9.60
DL.near
FED
Memory
12009
0.11
TSMIxeF
Memory MB)
Crosformer
0.44
SOFTS Tisansformer
0.40
50
100
150
200
250
300
350
# of Channels
Inference Time (ms/iter)
(a)
(b)
Figure 3: Memory and time consumption of different models. In Figure 3a we set the lookback window L 96, horizon H = 720, and batch size to 16 in a synthetic dataset we conduct. In Figure 3b set the lookback window
96, horizon H 720, and batch size 4 in Traffic dataset. Figure 3a reveals that SOFTS model scales to large number of channels more effectively than Transformer-basee
models. Figure 3b shows that previous Linear-based or MIP. based models such as DLinear and TSMixer perform poorly with a large number of channels. While SOFTS model demonstrates efficient performance with minimal memory and time consumption.
in performance across all pooling methods. Additionally, stochastic pooling deserves attention outperforms the other methods across nearly all the datasets.
Table 3: Comparison of the effect of different pooling methods. The term "w/o STAR" refers to scenario where MIP utilized with the Channel Independent (CI) strategy, without the use of STAR. The result reveals that incorporating STAR into the model leads to consistent enhancement performance across all pooling methods. Apart from that, stochastic pooling performs better than
mean and max pooling. Full results can be found in Table 7
ECL
Traffic
Weather
Solar
ETTh2
PEMS04
Pooling Method
MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
w/o STAR
0.187 0.273 0.273/0.442 0.442 0.28 0.261 0.281 0.247 0.272 0.381 8.406|0.145 143 0.245
Mean
00.174 0.266
(..260004400 0..40 0.277 0.261 0.281 0.234 0.262 0.379 0.404 (.0040.100001 106 0.212
Max
00.800 0.270 0.406 0.271 0.259 0.280 0.246 0.269 0.379 0.401 |0. 116 0.223
Weighted
00.184 0.275 0..440 0.292 0.263 0.284 0.264 0.280 0.379 0.403 00. 109 0.218
Stochastic
0.114 0.264 0.409 0.267 0.255 0.278 0.229 0.256 0.373 0.400 0.102 0.208
The STar Redistribtee (STAR) module is an embedding adap- tation function 41 that replaceable to arbitrary transformer- bassd methods that use the attention mechanism. In this paragraph, we test the effectiveness of STAR on different existing mansformer-based forecasters, such as PatchTST 229 and Crossformer 50 Note that our method can be regarded replacing the channel attention in iTransformer [27] Here we involve substituting
Universality of STAR.
the time attention PatchT'ST with STAR and incrementally replacing both the time and channel attention in Crossformer with STAR. The results, as presented in Table demonstrate that replacing attention with STAR, which deserves less computational resources, could maintain and even improve the models' performance in several datasets.
Influence lookback window length. Common sense suggests that longer lookback window should improve forecast accuracy. However, incorporating too many features can lead to curse
7

---

Table 4: The performance of STAR in different models. The attention replaced by STAR here are the time attention in PatchTST, the channel attention in iTransformer, and both the time attention and channel attention in modified Crossformer. The results demonstrate that replacing attention with STAR, which requires less computational resources, could maintain and even improve the models' performance in several datasets. 1: The Crossformer used here a modified version that replaces the
decoder with flattened head like what PatchTST does. Full results can be found in Table 8)
ECL
Traffic
Weather
PEMS03
PEMS04
PEMS07
Model
Component
MSE
MAE|
MSE MAE
MSE MAE
MSE MAE MSE MAE MSE MAE
Attention
0,189 0.276 0.455
0.286 0.256 0,279 0.137 0,240 0.145 0.249 O. 144 0.233
PatchTST
STAR
0.185 0.272
0.448 0.279 0.252 0.277
0.134 0.233 0.136 0.238 0.137 0.225
Attention
0.202
0.301 0.546
0.297 0.255 0,310
0.100
0.208 0.090 0.198 0.084 0.181
Crosstormer!
STAR
0.198 0.292 0.549 0.292 0.252 0.305 0.100 0.204 0.087 0.194 0.080 0.175
Attention STAR
0,178 0.270
0.428
0.282 0.2558 0.278 0.113 0.221 0.111 0.221 0. 101 0.204
iTransformer
0.174 0.264 0.409 0.267 0.255 0.278 0.104 0.210 0.102 0.208 0.087 0.184
of dimensionality, potentially compromising the model's forecasting effectiveness. We explore how varying the lengths of these lookback windows impacts the forecasting performance for time horizons from 48 to 336 in all datasets. As shown in Figure SOFTS could consistently improve its performance by effectively utilizing the enhanced data available from an extended lookback window. Also, SOFTS performs consistently better than other models under different lookback window lengths, especially in shorter cases.
ETTm2
ECL.
Traffic
0.40
0.56
0.48
Bi iinirr
DLitear
inrar
TSMixer
TSMiser
TSMiter
Parch TST
0.34
PatchTST
0.44
PacckTNT
0.38
Wransformer
iTransformer
0.42
Transformer
SOFTS (ours)
0.52
0.40
SOFTS (ours)
MAR
MAK
0.58
0.36
0,50
0.54
0.34
0.30
0.28
0.26
0.32
336
96
192
336
Lookback Window Length
Lookback Window Length
Lookback Window Length
Figure 4: Influence of lookback window length L. SOFTS performs consistently better than other models under different lookback window lengths, especially in shorter cases.
Hyperparameter sensitivity analysis. We investigate the impact of several key hyperparameters on our model's performance: the hidden dimension of the model, denoted as d, the hidden dimension of the core, represented by d', and the number of encoder layers, N. Analysis of Figure 5 indicates that complex traffic datasets (such as Traffic and PEMS) require larger hidden dimensions and more encoding layers to handle their intricacies effectively. Moreover, variations in d' have a minimal influence on the model's overall performance.
Series embedding adaptation of STAR. The STAR module adapts the series embeddings by extracting the interaction between channels. To give an intuition of the functionality of STAR, we visualize the series embeddings before and after being adjusted by STAR. The multivariate series is selected from the test set of Traffic with look back window 96 and number of channels 862. Figure 6 shows the series embeddings visualized by T-SNE before and after the first STAR module. Among the 862 channels, there are channels embedded far away from the other channels. These two channels can be seen as anomalies, marked as (+) in the figure. Without STAR, i.e., using only the channel
independent strategy, the prediction on the series can only achieve 0.414 MSE After being adjusted by STAR, the abnormal channels can be clustered towards normal channels by exchanging channel
8

---

0.64
0.42
0.38
0.49
0.56
0.36
27 0.32
0.36
0.34
0.32
0.34
0.30
0.30
0.34
0.32
0.33
0.29
0:26
0.24
0.22
0.23
0.22
0.22
0.24
0.20
0.18
0.19
0.16
0.16
64
128
256
512
1004
64
138
256
512
1024
Dimension Series
Dimension of Core
off Encoder Layers
Figure 5: Impact of several key hyperparameters: the hidden dimension of the model, denoted as d, the hidden dimension of the core, represented by d', and the number of encoder layers, N. Full results can seen Appendix
Before STAR
After STAR
PEMS03
MSEE-0.44
0SE-0.376
PRichTST
Irrnsformer
WW
SOFTS
MSE
WV
WW
Scries Embedding
Series Embedding
Strength of Noise
(a)
(b)
(c)
Figure Figure 6b T-SNE of the series embeddings on the Traffic dataset. 6a the series embeddings before STAR. Two abnormal channels (*) are located far from the other channels. Forecasting on the embeddings achieves 0.414 MSE. [b series embeddings after being adjusted by STAR. The two channels are clustered towards normal channels (A) by exchanging channel information. Adapted series embeddings improve forecasting performance to 0.376. Figure 66c| Impact of noise on one channel. Our method is more robust against channel noise than other methods.
information. An example of the normal channels is marked as (A). Predictions on the adapted series embeddings can improve the performance 0.376, 9% improvement.
Impact of channel noise. As previously mentioned, SOFTS can cluster abnormal channels towards normal channels by exchanging channel information. To the impact of an abnormal channel on the performance of three models-SOFTS PatchTTT,, and I'Transformer-wwe select one channel from the PEMS03 dataset and add Gaussian noise with mean of () and a standard deviation representing the strength of the noise. The lookback window and horizon are set to 96 for this experiment. In Figure we observe that the MSE of PatchTST increases sharply as the strength of the noise grows. In contrast, SOFTS and iTransformer can better handle the noise. This indicates that suitable channel interaction can improve the robustness against noise one channel using information from the normal channels. Moreover, SOFTS demonstrates superior noise handling compared to iTransformer. This suggests that while the abnormal channel can affect the model's judgment normal channels, STAR module can mitigate the negative impact more effectively by utilizing representation instead of building relationships between every pair of channels.
Conclusion
Although channel independence has been found an effective strategy to improve robustness for multivariate time series forecasting, channel correlation is important information to be utilized

---

for further improvement. The previous methods faced a dilemma between model complexity and performance in extracting the correlation. In this paper, we solve the dilemma by introducing the Series-cOre Fused Time Series forecaster (SOFTS) which achieves state-of-the-ar performance with low complexity, along with a novel STar Aggregate-Redie Redistribute (STAR) module to efficiently capture the channel correlation.
Our paper explores the way of building scalable multivariate time series forecaster while maintaining equal or even better performance than the state-of-the-art methods, which we think may pave the way forecasting on datasets of more larger scale under resource constraints [53]
Acknowledgments
This research was supported by National Science and Technology Major Project (2022ZD01 14805), NSFC (61773198, 18376118,61120000 Collaborative Innovation Center of Novel Software Tech- nology and Industrialization, CCF-Tencent Rhino-Bird Open Research Fund (RAGR20240101),
References
[1] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolu- tional and recurrent networks for sequence modeling. CoRR, (bbs/103.001271, 2018.
[2] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
[3] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In SSST@ EMNLP, pages 103-111. 11. Association for Computational Linguistics, 2014.
[4] Rabvan-Gabriel Cirstea, Darius-Vale Micu, Gabriel-Marcel Muresan, Chenjuan Guo, and Bin Yang. Correlated time series forecasting using multi-task deep neural networks. In ICKM, pages 1527-1530, 2018.
[5] Yue Cui, Kai Zheng, Dingshan Cui, Jiandong Xie, Liwei Deng, Feiteng Huang, and Xiaofang Zhou. METRO: generic graph neural network framework for multivariate time series forecasting, Proc. VLDB Endow, 15(2):224-236 2021.
[6] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. Long- term forecasting with tide: Time-series dense encoder. CoRR, abs/2304.08424, 2023.
[7] Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mip-mixer model for multivariate time series forecasting. In KDD, pages 459 469, 2023.
[8] Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable represen- tation learning for multivariate time series. In NeurlPS, pages 4652-4663, 2019.
[9] Aleksandra Gruca, Federico Serva, Llorenç Lliso, Pilar Rípodas, Xavier Calbet, Pedro Herruzo, Jih Pihrt, Rudolf Raevskyi, Petr Šimánck, Matej Choma, et al. Weather4cast at neurips 2022: Super-resolution rain movie prediction under spatio-temporal shifts. In NeurIPS 2022 Competition Track, pages 292-313. PMLR, 2022.
[10] Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang, Star- transformer. In NAACL-HLT, pages 1315-1325. Association for Computational Linguistics, 2019.
[11] Lu Han, Han-Jia Ye, and De-Chuan Zhan. The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting. CoRR, abs/2304.05206, 2023.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778. IEEE Computer Society, 2016.
10

---

[13] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, Abss1606.008455 2016.
[14] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
[15] QU Hua, ZHANG Yanpeng, LIU Wei, and ZHAO Jihong. A robust fuzzy time series forecasting method based on multi-partition and outlier detection. Chinese Journal of Electronics, 28(5): 899-905, 2019.
[16] Akhil Kadiyala and Ashok Kumar. Multivariate time series models for prediction of air quality inside public transportation bus using available software. Environmental Progress & Sustainable Energy, 33(2):337-345 2014.
[17] Evaggelos G Kardakos, Minas C Alexiadis, Stylianos I Vagropoulos, Christos K Simoglou, Pandelis N Biskas, and Anastasios G Bakirtzis. Application of time series and artificial neural network models in short-term forecasting of pv power generation. In 2013 48th International Universities' Power Engineering Conference (UPEC), pages 1-6. IEEE, 2013.
[18) Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In ICLR, 2021.
[19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2015.
[20] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR. OpenReview.net, 2017.
[21] Andreui Nikolaevich Kolmogorov. On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables. American Mathematical Society, 1961.
[22] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, pages 95-104, 2018
[23] Bryan Lim and Stefan Zohren. Time series forecasting with deep learning: survey. CoRR, 134/2004413408, 2020.
[24] Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Segrnn: Segment recurrent neural network for long-term time series forecasting. CoRR, abs/2308. 1200, 2023.
[25] Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. In NeurlPS, 2022.
[26] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. In NeurIPS, 2022.
[27] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma. and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. CoRR, abs/2310.06625, 2023.
[28) Mohammad Amin Morid, Olivia R Liu Sheng, and Joseph Dunbar. Time series prediction using deep learning methods healthcare. ACM Transactions on Management Information Systems, 14(1): 29, 2023.
[29] Yugi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.
[30] Syama Sundar Rangapuram, Matthias W. Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. NeurIPS, pages 7796-7805, 2018
11

---

[31] Lawrence G Roberts and Barry D Wessler. Computer network development to achieve resource sharing. In Proceedings of the May 5-7, 1970, spring joint computer conference, pages 543-549, 1970.
[32] LIU Shufen, GU Songyuan, and BAO Tie. An automatic forecasting method for time series. Chinese Journal of Electronics, 26(3):445-42, 2017.
[33] Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In NeurlPS, pages 24261-24272, 2021.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, pages 5998-6008, 2017.
[35] Xue Wang, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Make trans- former great again for time series forecasting: Channel aligned robust dual transformer. CoRR, abs/2305. 10955, 2023.
[36] Zepu Wang, Yuqi Nie, Peng Sun, Nam H. Nguyen, John M. Mulvey, and H. Vincent Poor. ST-MLP: A cascaded spatio-temporal linear framework with channel-independene strategy for traffic forecasting. CoRR, 00t.22233800449, 2023.
[37] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long- series forecasting. NeurlPS, pages 101-112, 2021.
[38] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d variatino modeling for general time series analysis. In ICLR. OpenReview.net, 2023.
[39] Xinle Wu, Dalin Zhang, Chenjuan Guo, Chaoyang He, Bin Yang, and Christian S. Jensen. Autocts: Automated correlated time series forecasting. Proc. VLDB Endow. 15(4):971-98,, 2021.
[40] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In SIGKDD, pages 753-763, 2020.
[41] Han-Jia Ye, Hexiang Hu, De- De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, pages 8805-881 8814. Computer Vision Foundation IEEE, 2020.
[42] Han-Jia Ye, De-Chuan Zhan, Nan Li, and Yuan Jiang. Learning multiple local metrics: Global consideration helps. IEEE Trans. Pattern Anal. Mach. Intell., 1977):6698-1712 2020.
[43] Han-Jia Ye, Lu Han, and De-Chuan Zhan. Revisiting unsupervised meta-learning via the characteristics of few-shot tasks. IEEE Trans. Pattern Anal. Mach. Intell., 45(3):3721-3737, 2023.
[44] Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequeney-domain mlps are more effective learners in time series forecasting. In NeurIPS, 2023.
[45] PENG Yu, LEI Miao, GUO Jia, and PENG Xiyuan. Multiresolution analysis and forecasting of mobile communication traffic. Chinese Journal of Electronics, 22(2):373-376 2013.
[46] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep In NIPS, pages 3391-3401, 2017.
[47] Matthew D. Zeiler and Rob Fergus. Stochastic pooling for regularization of deep convolutional neural networks. In ICLR. 2013.
12

---

[48] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, pages 11 11121-11128, 2023.
[49] Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented MLP structures. CoRR, 2at.22207.011880 2022.
[50] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In ICLR. OpenReview.net, 2023.
[51] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In AAAI, pages 11106-1115,, 2021.
[52] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, volume 162, pages 17268-27286, 2022.
[53] Zhi-Hua Zhou. A theoretical perspective machine learning with computational resource concerns. CoRR, abs/2305.022) 2023.
13

---

A Datasets Description
We detail the description plus the link to download them here:
ETT (Electricity Transformer Temperature) [T] comprises two hourly-level datasets (ETTh) and two 15-minute-level datasets (ETTm). Each dataset contains seven oil and load features of electricity transformers from July 2016 to July 2018.
2. Traffic describes describbss the road occupancy rates. It contains the hourly data recorded by the sensors of San Francisco freeways from 2015 to 2016.
3. Electricity collects the hourly electricity consumption of 321 clients from 2012 to 2014.
4. Weather includes 21 indicators of weather, such as air temperature, and humidity. Its data is recorded every 10 min for 2020 in Germany.
5. Solar-Energy [22] records the solar power production of 137 PV plants in 2006, which is sampled every 10 minutes.
6. PEMS contains public traffic network data in California collected by 5-minute windows.
Other details of these datasets have been concluded in Table 5
Table 5: Detailed dataset descriptions. Channels denotes the number of channels in each dataset.
Dataset Split denotes the total number of time points in (Train, Validation, Test) split respectively.
P Prediction Length denotes the future time points to be predicted and four prediction settings are included in each dataset. Gramularity denotes the sampling interval of time points.
Dataset
Channells
Prediction Length
Dataset Split
Granularity Hourly
Domain
ETTh1, ETTh2
7
(96. 192, 336, 720]
(8640, 2880, 2880)
Electricity
ETTml, ETTm2)
7
(96, 192, 336, 720)
(34560, 11520, 11520)]
15min
Electricity
Weather
21
(96, 192, 336, 720)
(36887, 5270, 10539)
10min
Weather
ECL
321
(96, 192, 336, 720
(18412, 2632, 5260)
Hourly
Electricity
Traffic
862
[96, 192, 336, 720
(12280, 1756, 3508)
Hourly
Transportation
Solar-Energy
137
(96, 192, 336, 720}
(36792, 5256, 10512)
10min
Energy
PEMS03
358
(12, 24, 48, 96)
(15724, 5242, 5242)
5min
Transportation
PEMS04
307
[12, 24, 48, 96)
(10195, 3398, 3399)
5min
Transportation
PEMS07
883
[12, 24, 48, 96)
(16934, 5645, 5645)
5min
Transportation
PEMS08
170
[12, 24, 48, 96)
(10713, 3571, 3572)
5min
Transportation
B
Implement Details
B.1 Overall architecture of SOFTS
The overall architecture of SOFTS is detailed in Algorithm T Initially, a linear layer is employed to obtain the embedding for each series (Lines 1-2). Subsequently, several encoder layers are applied. Within each encoder layer, the core representation is first derived by applying an MI. to the series embeddings and pooling them (Line 4). This core representation is then concatenated with each series (Line 5), and another MLP is used to fuse them (Line 6). After passing through multiple encoder layers, final linear layer projects the series embeddings the predicted series (Line 8).
http: http://pems.dot.. caloott.c...o.. gov
httpa://archive.. http://hhw...aeeeeeeeeeeeeeeattp
https://pems... dot.
14

---

Algorithm Series-cOre Fused Time Series forecaster (SOFTS) applied to multivariate time series.
Require: Look back window X E RLXc;
I: X X. transpose
> X E gexL Get series embedding, So Cxd
2: So Linear (X)
3: for = 1.... ...LLe do
4: 0; Stoch Stoch_Pool (o.)(M))((
Get core representation, o, Rd F; RCx(ddd)) D Fuse series and core, S; Cxd
5:
Repeat_Concal (o
6:
S. MLP(F.) S
7: end for
8: Y Lincar(Sz)
Project series embedding to predicted series, Y RC b b e RXC
9: = Y. transpose
10: return Y
B.2 Details of Core Representation Computation
Core representation. Recall that the core representation for the multivariate time series is defined in Definition 3.1 with the following form:
f(81, 82, 8c)
To obtain the representation, we draw inspiration from the two theorems:
Theorem B.1 (Kolmogorov-Amoll representation (2I)). Let f [0, 1 + an arbitrary multivariate continuous function iff it has the representation
f (I1. (01....,22)) .NN) =
Σ Am (Im)
with continuous outer and inner functions p 1R2M+1 R and Φ R. - 22 The inner function is independent of the function f.
Theorem B.2 (DeepSets [46)). Assume the elements are from compact set in Rd, Le. possibly uncountable, and the set size is fixed to M. Then any continuous function operating on set X, i.e. f: RR + R which is permutation invariant the elements in X can be approximated arbitrarily close in the form of
P Σ )lo)) DEX
for suitable transformations & and p.
The two formulations very similar, except for the dependence of inner transformation the dinate through Am The existence of X determines whether the formulation is permutation invariant not. In this paper, we find in Table 4 that the permutation invariant expression (Theorem B.2) performs much better than the permutation variant one Theorrm B. This may be attributed to the characteristics of channel series being enough induce the index of each channel (coordinate). Introducing extra parameters specific to each channel may enhance the dependency channel coordi- nate and reduce the dependence the history, therefore causing low robustness when encountering unknown series. Consequently, we utilize DeepSets form to express the core representation:
Σ Φ(8) S
o
We propose modifications the expression:
We generalize the mean pooling over the inner transformation by arbitrary pooling methods.
2. We remove the outer transformation p because it is redundant with the MLP during the fusion process.
For 1., we tested several common pooling methods and found that the mean pooling and max pooling outperform each other in different cases. Stochastie pooling (described in the following paragraph) can achieve the best results in averaged cases (Table 3). So, the core is computed as Equation
15

---

Stochastic pooling. Stochastic pooling is a pooling method that combines the characteristics of max pooling and mean pooling [47] In stochastic pooling, the pooled map response selected by sampling from a multinomial distribution formed from the activations of each pooling region. Specifically, we first calculate the probabilities p for each dimension j by normalizing the softmax activations within the dimension:
Pij
(6)
k== A
During training, we sample from the multinomial distribution based on p to pick channel & within the dimension The pooled result is then simply Aej:
0j Acj where P(P1j.P2) PC2)
At test time, we use a probabilistic form of averaging:
0j - Σ PijAi
(8)
This approach allows for a more robust and statistically grounded pooling mechanism, which can enhance the generalization capabilities of the model across different data scenarios.
B.3 Experiment Details
All the experiments are conducted on single NVIDIA GeForce RTX 3090 with 24G VRAM. The mean squared error (MSE) loss function is utilized for model optimization. Performance comparison among different methods is conducted based on two primary evaluation metries: Mcan Squared Error (MSE) and Mcan Absolute Error (MAE). We use the ADAM optimizer [19] with an initial learning rate x 10-4 This rate modulated by cosine learning rate scheduler. We explore the number of STAR blocks N within the set {1, 2, 3, 4}, and the dimension of series d within {128, 256, 512). Additionally, the dimension of the core representation d' is searched in (64, 128, 256, 512], with the constraint that d' does not exceed d.
Mean Squared Error (MSE):
MSE Σ
(Y;--Y)
(9)
Mean Absolute Error (MAE):
H Y: -
MAE
Σ
(10)
H
where Y, Y "xC are the ground truth and prediction results of the future with H time points and C channels. Y, denotes the i-th future time point.
C Full Results
C.1 Full Results of Multivariate Forecasting Benchmark
The complete results our forecasting benchmarks are presented Table 6 We conducted experi- ments using six widely utilized real-world datasets and compared our method against ten previous ofate-of-the-arr models. Our approach, SOFTS, demonstrates strong performance across these tests.
C.2 Full Results of Pooling Method Ablation
The complete results of our pooling method ablation are presented in Table 7 The term "w/o STAR" refers to a scenario where an MLP utilized with the Channel Independent (CI) strategy, without the use of STAR. Mean pooling computes the average value of all the series representations. Max pooling selects the maximum value of each hidden feature among all the channels. Weighted average learns the weight for each channel. Stochastic pooling applies random selection during training and weighted average during testing according to the feature value. The result reveals that incorporating STAR into the model leads to consistent enhancement in performance across all pooling methods.
16

---

Table 6: Multivariate forecasting results with prediction lengths H E (12. 24, 24, 96) for PEMS and H E {96. 192, 336, ]20} for others and fixed lookback window length L. = 96. The results of PatchTST and TSMixer are reproduced for the ablation study and other results are taken from iTransformer 227
Models
SOFTS (ouns) ITrasformmr PancTST TSMixer Croesformer
TIDE
TimesNet
DLincur
SCINet
PEDformer
Stationayy
Metric
MSE
MAE
MSE MAE [MSE MAE] MSE MAEJ MSE MAE] MSE
MAE|MSE
MAE] MSE
MAE MSE MAE (MSE MAE MSE MAEJMSE MAE]
96
11325
0.361
00,334
0.368
10.329 (.333 0 013633/ 00404 0:426 (0.
0.38770.355
0.375
0.345
0.372/0.411 0.438 00.17 0.419)0.386 0.398
192
0.375
0.389
0.377
0.391
0.380
0.394
0.336 0.392 0.450 0.451
0.398
0.404
0.374 0.387
0.380
0.389 0.439
0.450 0.426 0.44] 0.459 0.444
0.426
0.420
0.410
0.407 0.413
0.515 0.428
0.410
0.411
0.413
00490 0.485 0.445 0.459 0.495
720
0.466
0.447
0.491
0.459
0.475
0.453
(.485
0.459
(1666
0.589
0.487
0.46
(1,478
0:450
0.474
0.453
(0.595
0:550
0.490
0.516
Avg
0.3393
0.403
0..407
0,410
|0
306
0.40%
1330
0.407
0.496
(0.419
0.,41/0.4000
014006
01:403
0.407
0.407]0.448
(0.485
0.481 [0.448 0.452/0
0.456]
96
0.180
0.261
0.110
0.264
00.184
0.264
0.182
0.266|0.287
0.366
0.207
(.30550.118
0.267 0.193
0.29220.286
0.377
0.207/0.203 0.28770.19
0.274
ETTR ETTH ETTea
192
0.246
0.306
0.250
0.309
0.246
0.306
0.219
0.309
0.414
0.192
0.290
0.364
(12269
0.309
0.284
0.362
0.399
0.485
0.
0.328 0 0,280 0.339
0.511
0.34K
0.542
0.420
0.351
0.427
00.637
0.591
0.366 0.334 0.36]
720
0.405
0.401
0.412
0.417
0.409
0.402
0.416
0.408
1.750
1.042
0.558
0.534
[1.401
0:403
0.554
0.522
(0.960
0:735
0:441
0.415 (0417 0.413
Avg
002257
0.330
0.288
0.332
[0.287
0.239
0.33:
0.610
00.358
0.40440.220
0.333
0.350
0.401
|0.571
0.537 00.305
0.349|0.306 0.347]
96
0.388
0.399
10.366
0:405
10.394
0.406/0.40]
0,433
0,448
0.470
0.46440.554 0:402
0.316
0,400
00.654
0.599 003366
0.419/0.513 0.491
192
1.455
0441
0:441
01.4%
0.440
0.435
(1455
0:447
0.471
0:474
0.525
0.493 01.436
0.429 0.447
0.432
0.719
0.631
0.420
0.448 (n (3.534 0.504
336
0.480
0.452
0.487
0.458
0.462
(.492
0.463
0.570
0.546
00.565
0.491
0.469
0.481
0.459
0.7778
0.659
0.459
0.535
720
0.499
0.488
0.503
0.491
0.487
0.479
0.507
0.490
0.653
0.621
0.594
0.558
0.5521 0.500
0.519
0.516 (0.836 0.699
0.506
0.507 0.643 0.616
Avg
00.449
0.442
0.:454
0:447
0:453
0.446
0.452|0.550
(0529
0.522 (0.54] 0.507|0
00:458 0,450 0.656 0, 0.452 (0.747 0.647 00.400 0 0.160|0.570 0.537]
96
00.297
0.317
0.2297
0.309
00288
0.340
0.319
0.361
0.745
0.584 00,100
0.440
0.374 0.333 0.387 00.707 0.621
10.358 o 0.397 10:176 0.45s
192
0.373
0.394
(1000)
0:400
0.376
0.306
0.402
0.410
0.877
0.656
0.528
0.509
0.402
0.414
0.447
0.476
0.360
0.689
0.429 0.439 0.512 0.493
336
0.410
0.426
0.418
0.432
0.440
0.45L
0.444
0.446
1.043
0.731 0,643
0.57) 0.452
0.452
0.594
0,541
1.000
0,744
0.196 0.487 0.552 0.551
720
00441
0.435
0.121
0.305
0.136
0.453
0.44
0.450
1.101
0.763 0.874 0.679
0.462 0.168
0.831
0.657
1,219
0.838
0.116 0.174 0.174|0.562 0.560
Avg
(0.373
0.400
0.338
0.407
|0:385
0.410/0.401
0.417|0:942
0.684
(0.611
0.550]0.414
0.427 0.559
0.515
0.515|0.954
10.954
0:723
00.437 0.449|0.533
0.516
96
0.143
0.233
0.144
0.241
0.166
0.251
00.157
0.261
0.314
00.25
0.329
(1. 0.166 0.272 0.197 0.282 [0. 14.247 0.345
0.193 i 0.308 to 169 2733
BCL
192
0.158
0.248
01162
0.253
0.173
0.262
0.173
0.274
0.251
0.322
0.236
0.330
0.184 0.289 0.196
0.285
0.257
0.355 0.201
0.315 00 182 282
336
0.17%
0.269
0.178
0.364
0.
0.190
0.279
0.192
0.295
0.246
0.337
0.249
0.344 0.198 0.300 0.
0.301
0.264
0.369
0.214 0 0.320
o 10000 304
720
0.218
1.225
0.317
0.230
0.2223
0.280
0.363 0..284
(1.200 0.520
0.333
(0.299
0.246
0.355
Aug
00.174
0.264
0.220
10.189
0.276|0.186
0.257
0.334
(0.25) 0 0.344|0..99
0.295 0.212 0.300
0.365
(0.214 | 193199 0.296
96
0.376
0.251
0.395
0. 0.266
0.427
0.272
00,99
0.336
0.290
0.800 0.4903
0.599 0.321 0.650 0.396
10,788 0.499 0.587 0.36610.612 0.335
Trallic
192
0.398
0.261
2.226
0.154
0.289
0.197
0.351
0.530
0.293
0.756 0.174 0.617 0.336 0,598
0.370
0.789
0,505
0.001 o 0.373 0.613 0.340
336
0.415
0.269
1143
0.288
0.400
0.282
0.528
0.361
0.558
0.308
0.762
0.477 0.629
0.336
0.605
0.373
0.797
0:508
0.621
0.383 0.618 0.328
720
0.447
0.287
0.467
0.302
0.484
0.30
0.569
0.380
0.589
0.328
0.719
0.449 0.6440 0.350 0.645 0,394
0.841 0.523 0.626 0.382 0.683 0,355
Avg
00.409
0.267
10:478
0,282
10:454
0.286/0.523
0.357|0
0.304
(0.760 0,473/0.620 0.336 0.625 0.383
10. 10.004 0.509 10.610 0.37610.654
0 0.340
Weather
96
1666
0.208
0.114
0:214
0.176
0.317
(0: 166
0.210
0.158
0.230
0.200 0,261] 00.17 0:220 0. 1996 0,255
10.221 0 0.217 o 0.296 (0. 173 233
192
0.217
0.253
0.221
0.221
0.256
0.215
0.256
6.206
0.277
0.243 0. 0.239 (12661 01.237
0.296
(0. 00.661
D.
0:240 0 3.336 (0 245 o
336
0.282
0.300
0.278
0.296
0.275
0.296
0.287
0.300
0.272
0,335
0.287
0.3350.260 0.306 0.263 0.335
0.309
0.378 0.339
0.380
0.338
720
0.335
0.351
0.358
0.347
0.353
0.346
(.335
0.348
0.398
0.418
0.351
0.3860..55 (0.365 0.359 0,345 0,381
00377
0.427 0.403 0,428 0.4414 0.410
Arg
00.255
0.278
10.258
0.27%
1.2256
0.279/0:226
0.279
0.315
(0.271 0.320j0.259 0.287 0..665 0.317|0.292
0.363 (0.309 0.360)0.288
0.314
Solar Energy
96
002000
0.230
00208
0.257
0,200
0.34610.221
0.275
0.310
0.331
0.312 0. 0,309/0.2220
0.292 0,290 0.378 10.237 D. 0.344 (0.242 0.342 2115 0.249
192
0.229
0.253
0.243
0.261
0:237
0.267 0.266
0.300
0.734
0.725
0.339
0.416 0.299 0.318 0.320 0.398 (0.2800
0.380 0.285
0.380
0,252
336
0.243
0.269
0.248
0.275
0:250
0.276 0.272
0.294
0.750
0.735
0.368
0.430 0.319 0.330 0.353
0.41500.304
0.389
0.282 0.376 0.290 0.296
720
0.245
0.272
(1:40
0..24
0
0.252
01281
0.313
0.709
0.765
0.370
0.425 (0.338
0.337 0.356 0.415 00.008
0.388
0.357
0.427 to
285
0.200
Aug
00.299
0.256
|0.333
0.262
10.236
0.266/00.200
0.297
6461
0.639
00.347 0.417
00.501 0.319 0.330 0.401
10.282
0.375
10.291
|0
0.381
PEMSO
12
0.064
0.165
10.071
0.174
10,073
0.178 10:075
0.09000000
0.203
00.178 0 0.305 0.055.085 0.192 0.122
0.243
10.000
0.122
101126
10:081 0
24
0,083
0.188
0.093
0.201
0.105
0.212 0.005
0.120
0.240
0.257 0.37] 0..11K 0.223 0.201
0.317
0
0.085
0.198
0. 0.275
( 105 0214
48
0.114
0.223
01125
0.159
0.264
1.1
0.202
0.317
0.379
0.463 0.155 0.260
0.333
0.425
00.17
0.238
0.227 o 348
1o 154
257
96
0.156
0.264
0.164
1.275
0.210
0.305
0.154
0.295
0.262
0.367
0.490
0.539 0.228 0.317 0.457
0.515
17.
0.287
0.345
0.43400.247
0.247 0.326
Avg
0..044
0.210
10.13
0.221
00.137
0.240j0.119
0.233J0.166
0.169
0.281 (03326 0.419j0.147 0.248 0,278 0.375
0.37510.114
0,224
222213..13
0.327J0.147
0.249
PEMSON PEMSIT PEMSUN
0.007
10.078
0.183
10.065
17700
0.218
(0.219 (.3400
0.
0) 0,148
0,272
00,073
0.177
101338 0.262 10 196
24
0.088
0.194
0.085
0.205
0.115
(.080
0.201
0.151
0.256
0.293 0 0.398 ( 103 0.215
0.274
0.540
00.084
0.193
0.177 0 0.293 ( 104
0 2216
48
0.110
0.219
0.120
0.255
0.067
0.273
0.111
0.223
0.200
0.326
0.409
0.478 0.116 0.250
0.385 0.437
(0.099
0.211
0.270 0 0.368
o 0.7 0225
96
0.135
0.244
0.150
0.262
0.211
0.3
03133 0.247
0.402
0.457 0.492 0.532 0.199 0.303 0.452 0,504
(0.114
0.227
0.341
0.427 08666 0.257
Avg
[0.102
0.208
00.111
0.221
10.145
0 0.249
103
0,215/0
209
0,314
(0.353
0.437 (0.
0.241
0.245
0.388
0.202
00.331
0.337J0 1337/0.127
0.440
12
0.057
0.152
00067
0.165
10.008
00.073 0
0.200
0.1173 0. 0.304 0.082
0.181
0.115
0.242
(0.068
0.171
10.009 0 0.225
(0.063 0 188
0,073
0.173
0.085
0.190
0.2
0.201
0.090
0 09999
0.247
0.271
0,383
0.101 0.204
0.210
0.329
10.19
0.225
0.125
0.244
( 102 0.207
48
0.096
0.195
0,100
0.215
0.170
0.261
0.124
0.231 0.311 0,369
0.846
0.495 0.134 0.238 0.398
0.458
(0.109
0.237
0.165
0.288
036 02400
96
0.120
0.218
0.139
0.245
0.236
0 0.308
0.165
0.255 0.396
0.142
0.628
0.57770.181
0.279
0.594
0.553
(1.14)
0.234
0.263
0
181 0.287
Avg
00087
0.184
10101
0.204
[0.144
0 0.233
0.217
0.233
0.315
(0.380
0.440
02.14
0.225
0.329
0.395
119
0.234
J0. 1666
0 1223)0.127
230
12
0.074
0.171
1000
0.182
000000
0 202
01.033 0.
0.18990.118
0.214
00.277 0. 1.344 0.112 0.212
0.154
0.276
10.087
0.184
1.173 0.273 (0 109 0.207
0.104
0.201
0.115
0.219
0.162
0.117
0.226
0.260
0.318
0.409
0.141 0.238
0.248
0.353
(0.122 0.221 0.210 0 ( 0.236
48
0.164
0.253
01186
0.236
0.238
0.311 100
0.209
0.315 0.355 0.497
0.510 01 1.19 0.283
0.440
0.470
(0. (8.899
0:270 0.370 o 394 (1:211 294
96
0.211
0.253
0.221
0.267
0 3000
0.318 0.2666
0.331 0.377
0.397
0.721
(.520
0.351 0.674
0.565
(0.236 0.300 (0.422
D 4645 (0 00:345 0.367
Avg
00.138
0.219
00150
0.226
10.200
0.275
10.15
0.261
0.266
0,307
0.441
0.464|0.193
0.464
(0193
0.271
0.379
(..1660.188
0.244]0.286
0.358J0.201 0.276]
1 Count
47
2
1
0
2
5
0
17

---

Table 7: Comparison of the effect of different pooling methods. The term "w/o STAR" refers to scenario where an MLP is utilized with the Channel Independent (CI) strategy, without the use of STAR. The result reveals that incorporating STAR into the model leads to consistent enhancement in performance across all pooling methods. Apart from that, stochastic pooling performs better than mean and max pooling.
Pooling Method
w/o STAR
Mean
Max
Weighted
Stochastic
Metric
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE MAE
96
0.161 0.248
0.146 0.239 0.150 0.241 0.156 0.247
0.143 0.233
192
0.171 0.259
0.166 0.258 0.165 0.256 0.173 0.264
0.158 0.248
ECL
336
0.188
0.276
0.175
0.269 0.188
0.280 0.190 0.284
0.178
0.269
720
0.228 0.311
0.211
0.300 0.216 0.304 0.217 0.305
0.218 0.305
Avg
[0.187 0.273
0.174
0.266 0.10/ 0.270 0.184 0.275
..75//..17 0.264
96
0..414
0.266
0.380
0.255
0.386 0.261 0.410 0.275
0.376 0.251
192
0.428 0.272 0.406 0.268 0.397 0.267 0.434 0.288 0.398 0.261
Traffic
336
0.446 0.284 0.442 0.293 0.406 0.273 0.447 0.295 0.415 0.269
720
0.480 0.303 0.453 0.293
0.433 0.284 0.470 0.308 0.447 0.287
Avg
00.422
0.281 0..40 0.277 0.406 0,271 0.440 0.292|0.409
0.267
96
00.199
0.217 0.4 0.213
0.2 0 0.180 0.222
0.166 0.208
192
0.227
0.259 0.227 0.260 0.226 0.260 0.226 0.261
0.217 0.253
Weather
336
0.281
0.299 0.281 0.299 0.280 0.298 0.284 0.302
0.282 0.300
720
0.357 0.348 0.361 0.352 0.360 0.350 0.360 0.351
0.356 0.351
Avg
J0.261
0.281
(0.261 0.281 0.259 0.280 0.263 0.284
|0.255 0.278
96
00.215
0.250
0.202
0.238 0.206 0.243 0.219
0.230
192
0.246 0.271
0.238 0.260 0.245 0.266 0.255 0.272
0.229 0.253
Solar
336
0.263
0.282
0.248
0.277 0.267 0.284 0.292 0.294
0.243 0.269
720
0.263 0.283
0.247
0.271 0.265 0.284 0.290 0.293
0.245 0.272
Avg
10.247 0.272
0.234
0.262/ 0.269 0.264 0.280
0.280/0.229
00.229 0.256
96
0.298
0.349 0.298
0.348 0.296 0.347 0.292 0.344/0.297
0.347
192
0.375
0.398 0.376
0.396 0.378 0.396 0.387 0.401
0.373 0.394
ETTh2
336
0.420
0.431
0.417
0.430 0.423 0.428 0.428 0.435
0,410 0.426
720
0.433
0.448 0.423 0.442 0.421 0.435 0.409 0.433
0.411
0.433
Avg
10.381
0.406 00.799 0.404|0.377
0.401 0.379 0.403
(0.373
0.400
12
10.044
0.189 0.075 0.177 0.078 0.182 0.077 0.180
0.074
0.176
24
0.113
0.220 0.090 0.69 0.095 0.204 0.094 0.203
0.088 0.194
PEMS04
48
0.164 0.266 0.117 0.225 0.126 0,236 0.120 0.231
0.110 0.219
96
0.209
0.304 0.142 0.250 0.164 0.269 0.147 0.258
0.135 0.244
Avg
10.143 0.245
0. 106 0.212 0.223 0.109 0.218
0.218 0.208
18

---

C.3 Full Results of STAR Ablation
The complete results of our ablation on universality of STAR are presented in Table The STar Aggregate- RedisttittRd (STAR) module is set-to-set function [41] that is replaceable to arbitrary transformer-baeee methods that use the attention mechanism. In this paragraph. we test the ef- fectiveness of STAR different existing transformer-based forecasters, such as PatchTST [29] Note that our method can regarded as replacing the channel attention 27 Here we involve substituting the time attention in PatchTST with STAR and incrementally replacing both the time and channel attention in Crossformer with STAR. The results, as presented in Table g demonstrate that replacing attention with STAR, which deserves less computational resources, could maintain and even improve the models' performance in several datasets.
and Crossformer 500-
in ITrransfommer
C.4 More Results of Lookback Ablation
In this section, we extend the lookback ablation in section 4.2 L E [48, 720]. Figure shows the results in MSE. SOFTS performs almost consistently better than other models under different lookback window lengths. However, we also warn about the potential overfitting when the lookback length is very large, i.e. L = 512 or L 720.
ECL
Traffic
ETTn2
TSMbar
TSMise
0.21
PaschTST
0.7
PaichTST
PatchTST
[lrardoruer
0.2
SOFTS (ours)
MSS
008
SDFTS
MSE
0.32
SOFTS 6xuss)
0:83
0.3
0.25
1522
3.56
31.2
730
56
152
100
522
700
96
532
Lookback Windoww Length
Lookbark Windlww Length
Lookback Windoww Length
Figure 7: Influence lookback window length L E (48, (4,,96, 192, 336, 512, 720). SOFTS performs almost consistently better than other models under different lookback window lengths.
C.5 Full Results of Hyperparameter Sensitivity Experiments
We investigate the impact of several key hyperparameters on our model's performance: the hidden dimension of the model, denoted as the hidden dimension of the core, represented by d', and the number of encoder layers, N. Figure and Figure [ that complex traffic datasets (such as Traffic and PEMS) require larger hidden dimensions and more encoding layers handle their intricacies effectively. Moreover, Figure shows that variations in d' don't 't influence the model's s overall performance so much.
0.48
0.30
0.4
0.18
0.40
0.42
0.16
0.60
0.38
2
0.33
0.36
0.30
0.12
0.3
0.10
0.22
0.30
0.20
0.88
0.28
64
122
356
512
1824
64
123
350
512
1024
64
128
350
512
1024
Dimension Series
Series
Dimension
Figure 8: Influence of the hidden dimension of series Traffic datasets (such as Traffic and PEMS) require larger hidden dimensions to handle their intricacies effectively.
19

---

Table 8: The performance of STAR in different models. The attention replaced by STAR here are the time attention in PatchTST, the channel attention in iTransformer, and both the time attention and channel attention in modified Crossformer. The results demonstrate that replacing attention with STAR, which requires less computational resources, could maintain and even improve the models' performance in several datasets. I; The Crossformer used here is a modified version that replaces the decoder with flattened head like what PatchTST does.
Model
iTransformer
PatchTST
Crossformer
Component
Attention
STAR
Attention
STAR
Attention
STAR
Metric
MSE MAE MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0. 148 0.240 0.143 0.233 0.164 0.251
0.160 0.248 0.156 0.259
0,166 0.263
192
061 0.253 0.158 0.248
0.173 0.262
0.169 0.257 0.182 0.284
0.182 0.277
Electricity
336
0.178 0.269 0.178 0.269
0.190 0.279 0.187
0.275 0,203 0.305
0.200 0.296
720
0.225 0.317 0.218 0.305
0.230 0.313 0.225 0.308 0.267 0.358
0.243 0.334
Avg
0. 0.270 0.174 0.264
0.189 0.276 0.185
0.272 0.202 0.301
0.119 0.292
96 00.355 0.268 0.376 0.251
0.427
0.272
0.423 0.265 0.508 0.275
0.520
0.277
192 0.417 0.276 0.398 0.261 0.454 0.289 0.434 0.271 0.519 0.281
0.535 0.285
Traffic
336 0.433 0.283 0.415 0.269
0.450
0.282
0.447
0.278 0,556
0.304
0.551 0.292
720 0.467 0.302 0.447 0.287
0.484 0.301
0.489 0.301
0.600 0.329
0.591 0.315
Avg
0.428 0.282 0.409 0.267 0.454
0.286
0.448
0.279
0.546
0.297
0.249
0.549
0.292
96
0..14 0.214 0.166 0.208
0.176 0.217
0.170
0.214
0.174
0.245|0.174 0.239
192
0.221 0.254 0.217 0.253
0.221 0.256
0.215 0.251
0.219
0.283 0.220 0.282
Weather
336 0.278 0.296 0.282 0.300
0.275 0.296
0.273
0.296
0.271
0.327 0.272 0.324
720
0.358 0.347 0.356 0.351 0.352 0.346
0.349
0.346
0.351
0.383
0.343 0.376
Avg
0.258
0.278 0.255 0.278 0.256 0.279 0.252
0.277 0.254
0.310
0.310/0.252
0.252
0.305
12
10.071 0.174 0.064 0.165
0.073 0.178
0.071
0.173
0.067 0.170
0.065 0.165
24
0.093 0.201 0.083 0.188 0.105 0.212
0.101
0.206 0.081 0.187
0.081 0.184
PEMS03
48
0.
125
0.236 0.114 0.223 0.159 0.264
0.157
0.256 0.109
0.220
0.109
0.216
96
0.164 0.275 0.156 0.264 0.210 0.305
0.205
0.296 0.142 0.255
0.147
0.250
Avg
0..13 0.222 0.104 0.210 0.1 0.240
0.134
0.208
0.100
0.204
12
10.078 0.183 0.074 0.176 0.085 0.189
0.082
0.184
0.069 0.171
00.011
0.174
24
0.095
0.205 0.088 0.194 0.115 0.222
0.108
0.214 0.082 0.0
0.079
0.185
PEMS04
48
0.120
0.233 0.110 0.219 0.167 0.273
0.155
0.258 0.097 0.207
0.091
0.200
96
0.150 0.262 0.135 0.244
0,211 0.310
0.198 0.297
0.111 0.222
0.106
0.218
Avg
0.111
0.221 0.102 0.208 0. 145 0.249
0.136
0.238 (0.900 0.198
0.087
0.194
12
0.067 0.165 0.057 0.152 0.068 0.163
0.065
0.160 0.056 0.151
0.055
0.150
24
0.088
0.190 0.073 0.173 0. 0.201
0.098 0.195 0.070
0.166
0.067
0.165
PEMS07
48
0.110
0.215 0.096 0.195
0.170 0.261
0.162
0.250 0.090
0.192
0.088
0.183
96
0.139
0.245 0.120 0.218
0.236 0.308
0.222
0.294 0.1 0.215
0.110
0.203
Avg
0.
0.204
0.087 0.184
0. 144 0.233
0.137
0.225
0.084
0.181|
0.080 0.175
20

---

0.38
0.42
0.36
0.34
0.40
0.32
MSE
0.32
0.22
0.30
0.20
8.18
0.28
64
126
512
1824
64
126
256
512
1024
64
136
256
582
1804
Dimeasion of Core
Dimcasion of Core
Dimension of Core
Figure 9: Influence of the hidden dimension of the core d'. Variations in d' have minimal influence on the model's overall performance
ETima
Eriml
Elihl
ECL
Traitic
Westher
FEM5503
PEMSU
PEMSOO
PEMSIR
0.46
0.29
0.18
0.40
0.14
0.76
0.12
0.24
0.32
0.2
0.30
0.20
0.18
0.88
0.36
0.16
0.66
of Eneoder Layers
of Encoder Layers
of Eneoder Layers
Figure 10: Influence of the number of the encoder layers N. Traffic datasets (such Traffic and PEMS) require more encoding layers to handle their intricacies effectively.
D
Error Bar
In this section, test the robustness of SOFTS. We conducted 5 experiments using different random seeds, and the averaged results are presented in Table 9 It can be seen that SOFTS have robust performance over different datasets and different horizons.
Table 9: Robustness of SOFTS. Results are averaged over 5 experiments with different random seeds.
Dataset
ETTml
Weather
Traffic
Horizon
MSE
MSE
MAE
MSE
MAE
96
0.325 0.002 0.361 0.002 0. 166 0.002 0.208 0.002 0.376 + 0.002 0.251 + 0.001
192
0.375 + 0.002 0.389 + 0.003 0.217 = 0.003 0.253 + 0.002 0.398 + 0.002 0.261 = 0.002
336
0.405 + 0.004 0.412 0.003
0.282
0.001 0.300 + 0.001
0.415 + 0.002 0.269 + 0.002
720
0..466 + 0.004 0.447
0.002 0.356 0.002 0.351 + 0.002
0.447 + 0.002 0.287 + 0.001
Dataset
PEMS03
PEMS04
PEMS07
Horizon
MSE
MAE
MSE
MAE
MSE
MAE
12
10.064 + 0.002 0. 165 0.002 0.074 0.000
0.176 + 0.000 0.057 + 0.000 0.152 + 0.000
24
0.083 + 0.002 0.188 + 0.002 0.088 = 0.000 0.194 - 0.000 0.073 0.003 0.173 + 0.004
48
0.114
0.004 0.223 0.003 0.110 0.001 0.219 0.002 0.096 + 0.002 0.195 + 0.002
96
0.156 + 0.001 0.264 + 0.001 0.135 = 0.003 0.244 = 0.003 0.120 + 0.003 0.218 = 0.003
21

---

E
Showcase
E.1
Visualization of the Core
In this section, present visualization of the core. The visualization is generated by employing frozen state of our trained model to capture the series embeddings from the final encoder layer. These embeddings are then utilized inputs to a two-layer MLP autoencoder. The primary function of this autoencoder is to map these high-dimensional embeddings back to the original input series. The visualization shown in Figure TT Highlighted by the red line, this core captures the global trend of all cross all the channels in
Figure 11: Visualization of the core, represented by the red line, alongside the original input channels. We freeze our model and extract the series embeddings from the last encoder layer to train a two-layer MLP autoencoder. This autoencoder maps the embeddings back to the original series, allowing to visualize the core effectively.
E.2 Visualization of Predictions
To provide more intuitive demonstration of model's performance, present prediction showcases on the ECL (Figure (2]. ETTh2 (Figure 13}. Traffic (Figure [1). and PEMS03 (Figure (5) datasets. Additionally, include prediction showcases from iTransformer and PatchTST on these datasets. The lookback window length and horizon are set to 96.
Secund Brrhh
Secund
Secund FiR
22
65
(a) SOFTS (ours)
(b) iTransformer
(c) PatchTST
Figure 12: Visualization of Prediction on ECL dataset with lookback window 96, horizon 96.
E.3 More Results on Adaptation of Series Embedding
In this section, we show more results on the series embedding adaptation of our STAR module, similar showcases in figure and figure (bb) The number of channels should be large enough to show the relationship between channels in the embedding space. Therefore, we select the datasets ECL,, PEMS03, and Traffic with channels 321, 358, and 862 respectively. Figure shows the results on these datasets.
22

---

-02
-00
-04
-014
-1.
-1.
-12
-13
-1.
114
-14
-14
25
25
230
(a) SOFTS (ours)
(b) iTransformer
(c) PatchTST
Figure 13: Visualization of Prediction on ETTh2 dataset with lookback window 96, horizon 96.
20
(a) SOFTS (ours)
(b) iTransformer
(c) PatchTST
Figure 14: Visualization of Prediction Traffic dataset with lookback window 96, horizon 96.
E.4 Visualization of Predictions Abnormal Channels
As stated in Section after being adjusted by STAR, the abnormal channels can be clustered towards normal channels by exchanging channel information. In this section, we choose two abnormal channels in the ECL and PEMS03 datasets to demonstrate our SOFTS model's advantage in handling noise from abnormal channels. As shown in Figure [7 the value of channel 160 in PEMS03 experiences sharp decrease followed by smooth period. In this case, SOFTS is able capture the slowly increasing trend effectively. Similarly, in Figure [8] the signal of channel 298 in ECL resembles the sum of an impulse function and step function, which lacks continuous trend. Here, our SOFTS model provides a more stable prediction compared to the other two models.
F Limitations and Future Works
While the Series-cOre Fused Time Series (SOFTS) forecaster demonstrates sigmificant improvements in multivariate time series forecasting, several limitations must be acknowledged, providing directions for future work.
representation quality. The effectiveness of the STAR module heavily depends on the quality of the global core representation. this core representation does not accurately capture the essential features of the individual series, the model's performance might degrade. Ensuring the robustness and accuracy of this core representation across diverse datasets remains a challenge that warrants further research.
Dependence
Limited exploration of alternative aggregate-redistribunee strategies. Although the STAR module effectively aggregates and redistributes information, the exploration of alternative strategies limited. Future work could investigate various methods for aggregation and redistribution to identify potentially more effective approaches, thereby enhancing the performance and robustness of the model.
23

---

02
16
(a) SOFTS (ours)
(b) iTransformer
(c) PatchTST
Figure 15: Visualization of Prediction PEMS03 dataset with lookback window 96, horizon 96.
STAR
Before STAR
STAR
inver
MSE=0414
W
Naad diymall
W
MSF=0.16
MSF-1002
Series Embedking (a) ECL After STAR
Saries Embedding
Series Embedking
(b) Traffic After STAR
(c) PEMS03
After STAR
MSE=03366
MSE=0061
W.
WW
MSE-0143
Series Emberkligg (d) ECL
Series Embedking (c) Traffic
Series Emberking (f) PEMS03
Figure 16: t-SNE visualization of series embeddings before and after STAR adjustment for ECL with a lookback window of 96 and horizon of 96, Traffic with a lookback window of 96 and horizon of 96 and for PEMS03 with lookback window of 96 and horizon of 12. (a)-(d), (b)-(e), (c)-(f): The abnormal channel (+) is initially located far from the other channels. After adjustment by STAR, the abnormal channel clusters towards the normal channels (A) by exchanging channel information. Adapted series embeddings consistently improve forecasting performance based on the MSE metric.
24

---

Tulh
Sound Prak
25
12
12
(a) SOFTS (ours)
(b) iTransformer
(c) PStchTST
Figure 17: Visualization of Prediction on abnormal channel in PEMS03 dataset with lookback
window 96, horizon 96
Droudd
Groudd 3:
Groudd
52
10
40
-15
(a) SOFTS (ours)
(b) Traasformmr
(c) PatchTST
Figure 18: Visualization of Prediction on abnormal channel in ECL. dataset with lookback window
96, horizon 96.
G Societal Impacts
The development of the Series-cOre Fused Time Series (SOFTS) forecaster has the potential to significantly benefit various fields such as finance, traffic management, energy, and healthcare by improving the accuracy and efficiency of time series forecasting, thereby enhancing decision-making processes and optimizing operations. However, there are potential negative societal impacts to consider. Privacy concerns may arise from the use of personal data, especially in healthcare and finance, leading possible violations if data is not securely handled. Additionally, biases in the data could result unfair outcomes, perpetuating or exacerbating existing disparities. Over-reliance on automated forecasting models might lead to neglect of important contextual qualitative factors, causing adverse outcomes when predictions are incorrect. To mitigate these risks, robust data protection protocols should be implemented, and continuous monitoring for bias is necessary to ensure fairness. Developing ethical use policies and maintaining human oversight in decision decision-making making further ensure that the deployment of SOFTS maximizes its positive societal impact while minimizing potential negative consequences.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly outline the novel SOFTS module, which is supported by experimental results demonstrating its superior performance and efficiency.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided all the information needed to reproduce the main experi- mental results of the paper in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

reproducible, they Answer: [Yes]
Justification: We have provided an anonymous code for review along with the scripts processing the data. The code will be made public once the paper is accepted. The datasets for the main experimental is already open, as mentioned in Appendix A
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm cc/
public/guides http://oddaaamsssooooooooooooooo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pubb http:ss.ccppbblc/guuueeee wttpeeeee/oooSSeeesseeiesiie
ssi.iiolioiiiyyioiocc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have included all the training and test details in Appendix A and Ap- pendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We have provided results accompanied by error bars in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
28

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information computer resources in Appendix B.3 and mem- ory/time consumption our model in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal impacts of our work in Appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

and give the The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper for all the baselines in Section 4.1
URL of the datasets in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "KMz9R9V6rO", "reviewer": "Reviewer_gtby", "summary": "The authors present a framework for modeling correlations between channels in a multivariate time series forecasting task. This framework concatenates each channel embedding with a ‘global core embedding’ which contains information from all channels in the lookback window. The authors present experiments that demonstrate the utility of this concept, both from a performance and efficiency perspective.", "questions": "1. Under what conditions should I choose SOFTS over iTransformer?  Figure 3a makes it appear as though when there are a sufficiently large number of channels in the time series.  Are there other considerations?  Figures 3b and 6c show slight improvements but the difference doesn't seem large enough to discriminate between which framework would be better for a reader's specific time series forecasting efforts.\n2. What are the limitations of SOFTS? Or what are the tradeoffs between using SOFTS vs other models?  A few words of discussion in the conclusion would help improve the clarity of the paper.", "limitations": "As is, there is no discussion of the limitations in the main text of the paper.  To address this partially, section G of the appendix could be moved to the main text.\n\nHowever, the discussion of the limitations of SOFTS currently exists in a vacuum, lacking a comparison/contrast with other models mentioned in the paper, especially PatchTST and iTransformer.  The authors should characterize and highlight the characteristics of datasets and inference tasks where SOFTS outperforms existing methods by wide margin.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Time series forecasting has been an important problem and it continues to grow with the advent of time series foundation models. To the best of my knowledge, it remains an open question for how to best enable multivariate time series forecasting, and this paper provides a conceptually reasonable approach. I believe the authors’ work is of broad interest.", "weaknesses": "In my opinion, the paper would be improved with analysis and discussion of their results.  There is little discussion beyond drawing attention to features of figures and tables, which misses an opportunity to explain why the authors believe they are observing such behavior.  I am specifically interested in a discussion between iTransformer and SOFTS, which appear to be quite similar along many dimensions that SOFTS claims to be superior.  Such a discussion will help guide potential readers through the considerations they should take into account when deciding which framework to implement on their own forecasting problems.\n\nI also recommend adding either a few sentences or a small figure that highlights the differences between PatchTST, transformer, and SOFTS — I see some details in the text of table 4, but I think making this information more prominent would be helpful and make the paper more self-contained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your constructive feedback. We appreciate your recognition of the importance of time series forecasting and your acknowledgment of our work's potential impact. We have carefully considered your comments and suggestions and have made revisions to the manuscript to address your concerns. Here are our responses.\n    \n> Comparison and selection among PatchTST, Transformer, and SOFTS.\n\nWe appreciate your suggestion to provide a more in-depth analysis and discussion of our results. The main difference between the three methods is in the following table. \n\n- Embedding: The way of creating token embeddings. \"Patch\" means tokens are embedded by small patches (or windows) containing values of consecutive time. \"Series\" means embedding the whole series.\n\n- Temporal: The way of extracting the correlation of different time steps. \n\n - Channel: The way of extracting the correlation of different channels.\n\n| Method    | PatchTST  | iTransformer | SOFTS  |\n| --------- | --------- | ------------ | ------ |\n| Embedding | Patch     | Series       | Series |\n| Temporal  | Attention | MLP          | MLP    |\n| Channel   | /         | Attention    | STAR   |\n\n**PatchTST vs SOFTS**\n\nThe main difference between PatchTST and SOFTS lies in their approach to handling channels. PatchTST uses a channel-independent model, which sacrifices channel correlation but eliminates interference from other channels in the prediction process. In contrast, our SOFTS model employs the STAR module to extract channel correlations more robustly and effectively while reducing interference from other channels. By adjusting the representation through channel clustering, as shown in Figure 6, the predictions become more robust, making SOFTS particularly advantageous when there are many channels. Although SOFTS minimizes the negative impact of channel correlation as much as possible, there is still a risk of overfitting in situations where channels are highly independent. Current multivariate datasets may not be so independent so we find SOFTS outperform in most cases.\n\nThe second difference is the patch embedding of PatchTST, which utilizes a sliding window to extract information within consecutive times. PatchTST then exchanges the information at different time windows by attention. Our SOFTS uses MLP which is simpler but achieves SOTA performance as well. \n\n**iTransformer vs SOFTS**\n\nThe main difference between iTransformer and SOFTS is that iTransformer uses attention for channel interaction, while SOFTS uses STAR. As shown in Figure 2, the significant difference is that STAR employs a centralized structure to enhance robustness, avoiding the influence of certain abnormal channels on the predictions of other channels. From Table 2, it is evident that SOFTS has a clear advantage over iTransformer on datasets with a large number of channels, such as Traffic and PEMS. This is likely because, with more channels, attention is more susceptible to the influence of certain abnormal channels, whereas STAR mitigates this effect through aggregation. Additionally, in terms of efficiency, STAR reduces complexity from quadratic to linear, significantly enhancing its scalability. But we also note that SOFTS may have a bottleneck due to interaction through core representation. \n\n**So, under what conditions should I choose SOFTS over iTransformer?**\nBased on the current results, we find that SOFTS generally outperforms iTransformer in time series problems, both in terms of performance and efficiency, especially when the number of channels is large. This might be because robustness considerations outweigh capacity in time series issues. Of course, we do not rule out the possibility of scenarios where the reverse might be true in the future.\n\n> Limitation of SOFTS\n\nWe acknowledge your suggestion to move the discussion of limitations from the appendix to the main text. In response, we have integrated Section G from the appendix into the main body of the paper. We have also enhanced this section by comparing and contrasting the limitations of SOFTS with those of other models, such as PatchTST and iTransformer, like the discussions stated above. \n\nOnce again, we sincerely thank you for your valuable feedback. Your insights have significantly contributed to improving the clarity and impact of our paper. We hope the revised manuscript addresses your concerns and enhances the overall quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AYwvyUfJWC", "reviewer": "Reviewer_mAnn", "summary": "This paper proposes to use a global representation to capture the channel correlations for multivariate time series forecasting. Specifically, it uses stochastic pooling to get the global representation by aggregating representations of individual series and then concats the global representation and individual representations to reflect channel correlations for each series. Experiment results confirm that the proposal is much efficiency and achieves better performance than existing methods.", "questions": "Please refer to the weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes an efficient method to capture the channel correlations for multivariate time series forecasting.\n2. Extensive experiments are conducted to confirm the effectiveness of the proposal.\n3. The paper is well-written and easy to understand in general.", "weaknesses": "1. Some experimental results are unconvincing.\n\n-It is unclear why different datasets and metrics are used for different ablation studies, e.g., the datasets used in Table 3 and 4 are different, MAE is used in Figure 4 but MSE is used in other figures.\n\n-There is no statistical significance tests between the results of the proposal and baselines.\n\n-The results of Lookback Window Length 720 should be given as done in other papers.\n\n-It is better to give the training time for each methods as well.\n\n2. Some places are not clearly described.\n\n-It is unclear the MLP and Linear operations are channel independent or dependent. I can guess they are channel independent, but it is not clearly described in Figure 1 and Section 3.\n\n-The sentence \"... rely too heavily on the correlation to achieve satisfactory results under distribution drift\" in the Abstract is not clearly explained. \n\n3. Minor mistakes or typos: Embedding module is missing in Figure1; oi ∈ Rd' should be oi ∈ RC×d' in Algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our paper. We appreciate your feedback and have made corresponding responses to address your concerns.\n\n> It is unclear why different datasets and metrics are used for different ablation studies, e.g., the datasets used in Table 3 and 4 are different, MAE is used in Figure 4 but MSE is used in other figures.\n\nDatasets of Table 3,4 are selected due to space restriction. We provide full results in the **global rebuttal** part. We found that MSE lines of SOTA methods usually entangle with each other, therefore we display MAE. To alleviate concerns, we have also included performance curves using MSE in Figure 1 of the **global rebuttal PDF**. \n\n\n> Statistical significance tests.\n\nThank you for pointing out the necessity of a significance test. Across 48 settings of the benchmark, our method outperforms 10 baseline methods in most cases (Table 6). Considering the diversity of the datasets (covering electricity, traffic, energy, and climate) and the diversity of the comparison methods (including linear, MLP, CNN, and Transformers), this sufficiently demonstrates the superiority of our proposed method. We also demonstrate the stability under different random seeds in Table 9, indicating that the performance is not coincidental. To alleviate any concerns, we have supplemented our analysis with t-test significance experiments. Due to the large number of experimental settings and the lengthy training time of some methods, we selected iTransformer and PatchTST, which have performances most comparable to ours, for comparison. We repeat 10 times for each method and each setting. The results are as follows. The T-Statistics < 0 and p-value < 0.05 are marked in bold.\n\n|||PatchTST||||iTransformer||||\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|||MAE||MSE||MAE||MSE||\n|||P-Value|T-Statistics|P-Value|T-Statistics|P-Value|T-Statistics|P-Value|T-Statistics|\n|ETTm2|96|0.853|0.191|0.514|**-0.683**|**0.001**|**-4.874**|**0.001**|**-5.199**|\n||192|0.882|**-0.154**|0.748|**-0.332**|**0**|**-7.533**|**0**|**-9.142**|\n||336|0.063|**-2.156**|0.319|**-1.063**|**0**|**-7.514**|**0.003**|**-4.169**|\n||720|0.38|**-0.93**|0.925|**0.098**|**0.021**|**-2.877**|**0.108**|**-1.809**|\n|PEMS08|12|**0**|**-27.817**|**0**|**-42.337**|**0**|**-15.018**|**0**|**-21.369**|\n||24|**0**|**-32.518**|**0**|**-43.611**|**0**|**-17.864**|**0**|**-20.731**|\n||48|**0**|**-33.739**|**0**|**-38.155**|**0**|**-19.017**|**0**|**-36.447**|\n||96|**0**|**-72.713**|**0**|**-122.96**|**0**|**-15.827**|**0**|**-16.639**|\n|ECL|96|**0**|**-63.882**|**0**|**-66.26**|**0**|**-10.45**|**0**|**-7.251**|\n||192|**0**|**-18.627**|**0**|**-25.142**|**0**|**-6.495**|**0.002**|**-4.479**|\n||336|**0**|**-12.705**|**0**|**-18.651**|0.288|**-1.139**|0.551|0.623|\n||720|**0**|**-14.689**|**0**|**-25.911**|0.733|**-0.353**|0.194|**1.419**|\n|Solar|96|**0.013**|**-3.2**|0.225|**-1.315**|**0.033**|**-2.57**|**0.611**|**-0.529**|\n||192|**0**|**-6.401**|0.573|0.587|**0**|**-9.082**|**0.063**|**-2.159**|\n||336|**0.001**|**-4.953**|0.23|**-1.3**|**0**|**-6.172**|0.141|**-1.635**|\n||720|0.735|0.35|0.807|0.252|0.261|**-1.21**|0.68|**-0.428**|\n|Traffic|96|**0**|**-175.79**|**0**|**-117.36**|**0**|**-23.495**|**0**|**-15.247**|\n||192|**0**|**-161.43**|**0**|**-146.71**|**0**|**-96.104**|**0**|**-32.583**|\n||336|**0**|**-92.217**|**0**|**-112.43**|**0**|**-53.061**|**0**|**-15.972**|\n||720|**0**|**-109.05**|**0**|**-81.909**|**0**|**-53.852**|**0**|**-19.795**|\n\nOur method significantly outperforms the previous SOTA methods in most cases.\n\n> The results of Lookback Window Length 720.\n\nWe extend the lookback window analysis from range [48, 336] to  [48, 720] and show the results in Figure 1 of the **global rebuttal PDF**. The figure shows that SOFTS can also achieve superior performance when the lookback window length is extended to 512 and 720.\n\n\n> Training time for each method.\n\nThank you very much for your suggestion. However, training time is influenced not only by the complexity of the model itself but also by the training strategy. Earlier methods, such as FEDformer and Informer, often completed training early due to early stopping strategies to prevent overfitting. These methods have shorter training times than current SOTA methods but perform much worse. To ensure a fair comparison, we comprehensively present model performance, inference time, and memory in Figure 3, where the latter two are only related to the complexity of the model. Therefore, we believe this figure more fairly demonstrates the superiority of our method in terms of performance and efficiency compared to merely showing training time.\n\n> It is unclear the MLP and Linear operations are channel independent or dependent.\n\nYes. To make it clear in the paper, we have specified every MLP in the form of mapping. For example, $\\operatorname{MLP}_1: R^{d} \\mapsto R^{d'}$ in line 147 means it project the axis with dimension $d$ to dimension $d'$. This mapping is shared across other axes including the channel dimension. Therefore, it corresponds to channel independence. They function similarly to the FFN layer of the Transformer.\n\n> The sentence \"... rely too heavily on the correlation to achieve satisfactory results under distribution drift\" in the Abstract is not clearly explained.\n\nThis sentence corresponds to lines 33-35 in the introduction \"However, such channel mixing structures were found vulnerable to the distribution drift\". For rigor, we change it to \"fail to achieve satisfactory results under distribution drift when incorporating the correlation\".\n\n> Minor mistakes or typos: Embedding module is missing in Figure1; oi ∈ Rd' should be oi ∈ RC×d' in Algorithm 1.\n\nThanks. We omitted the embedding module in the figure to spare space, and we will add it to the figure in the final version; $o_{i}$ is the global core representation for the whole multivariate series. It is not channel-specific, so it should be $o_{i} ∈ R^{d'}$ and it is not a typo.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eRRYGjCDmL", "reviewer": "Reviewer_NURe", "summary": "This paper presents an efficient MLP-based model, the Series-cOre Fused Time Series forecaster (SOFTS). SOFTS incorporates a novel STar Aggregate-Redistribute (STAR) module to aggregate all series to form a global core representation, which is then dispatched and fused with individual series representations to facilitate channel interactions. The broad applicability of the STAR module across different forecasting models is also demonstrated empirically.", "questions": "1. It seems that all series share a core representation and MLP layers, but different variables will require different aspects of information. How can the model address these differences?\n2. Please increase the look-back window length of the input series to  512 and 720, and compare the performance with other baseline models.\n3. Please provide more results on abnormal channels, i.e. ETT, ECL, and Traffic dataset.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written.\n2. The proposed STAR module is designed as a core to aggregate and exchange information from the channels efficiently, which is a universal module and can replace the attention mechanism.\n3. The experiments in Figure 6 are interesting and the results are impressive.", "weaknesses": "The paper lacks significant innovation. The stochastic pooling itself is not novel in deep neural networks. A somewhat novel facet of the proposed model is to use the pooling to extract global representations. However, the idea of aggregate-and-dispatch the interactions between variables has already been studied in TimeXer [1].\n\n\n[1] Wang, Y., Wu, H., Dong, J., Liu, Y., Qiu, Y., Zhang, H., ... & Long, M. (2024). Timexer: Empowering transformers for time series forecasting with exogenous variables. arXiv preprint arXiv:2402.19072.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and constructive comments on our paper. We appreciate your recognition of our work's strengths and have carefully considered your suggestions for improvement. Below, we provide detailed responses to each of your points.\n\n> The paper lacks significant innovation. The stochastic pooling itself is not novel in deep neural networks. A somewhat novel facet of the proposed model is to use the pooling to extract global representations. However, the idea of aggregate-and-dispatch the interactions between variables has already been studied in TimeXer [1].\n\nThanks very much for reminding us of this paper. We understand your concerns regarding the novelty of our method. However, our approach differs significantly from TimeXer. Most importantly, TimeXer still uses attention (self-attention, cross-attention) as the module for extracting correlations between tokens. In contrast, the core innovation of our paper, the STAR module (as emphasized in line 10 of the abstract, line 63 of the introduction, and line 126 of Section 3.2), is distinct from attention. This difference is detailed in Section 3.2 and illustrated in Figure 2. Additionally, TimeXer aggregates the information of series and patch embeddings using attention, which is an aggregation in the time dimension. This method does not address the problem our paper focuses on, which is how to efficiently and robustly model channel correlation. Finally, TimeXer investigates the scenario of forecasting with exogenous variables, which differs from our study of multivariate forecasting. Their proposed method is also tailored for that scenario. **Therefore, in terms of application scenarios, problems to solve, and methodology, there is little overlap between our paper and TimeXer. We believe that our work offers sufficient innovation compared to this paper.**\n\nIn conclusion, we sincerely thank you for mentioning this paper and offering your suggestions. To ensure the completeness of our paper, we will add a discussion on the scenarios and methods of TimeXer in the related work and future work sections. The preliminary content is as follows:\n\n**Related Work:**\nTimeXer [1] uses self-attention to aggregate the information of series and patches and employs cross-attention to achieve interaction between endogenous and exogenous variables. It extends iTransformer to the forecasting with exogenous variables scenario. Unlike these two methods, our paper proposes an efficient channel interaction module based on MLP, achieving better performance with lower complexity.\n\n**Future Work:**\nFuture work includes exploring how to apply SOFTS and STAR to more forecasting scenarios, such as forecasting with exogenous variables [1] and forecasting with future variates.\n\n[1] Wang, Y., Wu, H., Dong, J., Liu, Y., Qiu, Y., Zhang, H., ... & Long, M. (2024). Timexer: Empowering transformers for time series forecasting with exogenous variables. arXiv preprint arXiv:2402.19072.\n\n> It seems that all series share a core representation and MLP layers, but different variables will require different aspects of information. How can the model address these differences?\n\nThe MLP module can fuse different parts of the core representation according to the series due to its universal approximation ability. That is $MLP(s,o)$ can approximate any forecasting function if $s,o$ contains enough information. By Kolmogorov-Arnold representation theorem and DeepSets (line 146 and Section B.2). $o$ with equation (3) can approximate the function on all the series. Therefore, even if different variables will require different aspects of information, STAR can also approximate the required function.\n\n\n> Please increase the look-back window length of the input series to 512 and 720, and compare the performance with other baseline models.\n> Please provide more results on abnormal channels, i.e. ETT, ECL, and Traffic dataset.\n\nThanks for your advice. We have added the lookback window and the abnormal channel results (on ECL, Traffic, PEMS) in the **global rebuttal PDF**. The ETT dataset has only 7 channels, and the embedding correlation of the channels is not very clear, so we replaced it with PEMS. And we will make corresponding modifications to the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r6Iq6wsYV5", "reviewer": "Reviewer_hzF8", "summary": "The paper introduces SOFTS (Series-cOre Fused Time Series forecaster), an efficient multivariate time series forecasting model that addresses the gap between channel independence and channel correlation in a novel way. By utilizing a centralized STAR (STar Aggregate-Redistribute) module, SOFTS creates a global core representation aggregated from all series, which is then redistributed and fused with individual series representations. This mechanism allows for efficient channel interaction while reducing dependence on the quality of each channel. The paper demonstrates SOFTS's superiority over state-of-the-art methods in terms of performance and computational complexity, and showcases the STAR module's adaptability across various forecasting models.", "questions": "- Has the author currently tried any other methods for aggregation and redistribution?\n- SOFTS is a linear based model, so how to handle non-linear time series prediction?", "limitations": "- The authors have adequately addressed the technical limitations of their work within the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- SOFTS proposes a unique approach to handling channel correlations in multivariate time series forecasting by employing a centralized STAR module, which aggregates and redistributes series representations efficiently.\n- The model's design and implementation are well-thought-out, and the empirical results show that SOFTS outperforms existing state-of-the-art methods.\n- The paper is well-written and structured, providing clear explanations of the STAR mechanism and its integration into SOFTS.\n- The proposed method has the potential to enhance forecasting accuracy and efficiency across various domains, making it a valuable addition to the field.", "weaknesses": "- Additional experiments evaluating the robustness of SOFTS under varying conditions of distribution drift could provide deeper insight into its reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment of our work and your thoughtful feedback. We are pleased to know that you found our approach and presentation to be excellent and that you see the value of our contributions to the field. We have carefully considered your comments and here are our responses.\n\n\n> Additional experiments evaluating the robustness of SOFTS under varying conditions of distribution drift could provide deeper insight into its reliability.\n\nThanks for your advice. We have tested the robustness of SOFTS on real-world datasets and synthetic data with manually injected noise in Figure 6. More rigorous experiments need further design and we are working on it. Thanks again for your valuable advice.\n\n> Has the author currently tried any other methods for aggregation and redistribution\n\nTo keep our method neat, we use the simplest redistribution by concatenation and MLP fusion. We also experiment with 4 simple variants of aggregation, including mean pooling, max pooling, stochastic pooling, and weighted average as shown in Table 3. This simple structure is found to perform well. We agree that there would be other better designs for aggregation and redistribution. And we leave it for future exploration.\n\n> SOFTS is a linear based model, so how to handle non-linear time series prediction?\n\nThe encoding part of SOFTS (using STAR) is composed of multiple MLP layers. So it can handle non-linear time series prediction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper and for providing valuable feedback. We are delighted that the majority of the reviewers (3 out of 4) have given positive evaluations of our work. Our work is said to "of broad interest" (gtby), "well-thought-out" and "valuable addition to the field" (hzF8), "interesting" and "impressive" (NURe), "well-written" (mAnn,NURe,hzF8) . One reviewer has raised some concerns regarding the experiments and clarifications of our paper. We hope that this rebuttal addresses these concerns effectively, allowing both reviewers and readers to gain a clearer and more comprehensive understanding of our research. We aim to improve the perception and evaluation of our work through these clarifications and enhancements.

**PDF content**: Two reviewers (**mAnn, NURe**) are interested in the performance (in MSE) when the look-back window is 512 and 720. Figure 1 in the pdf displays the comprehensive performance comparison against other methods with look-back lengths in [48, 720]. Figure 2 shows more results on abnormal channels, in which reviewer **NURe** is interested.

Reviewer **mAnn** is also concerned about different datasets used in Table 3, 4. The datasets are selected due to the space restriction. Due to the character restriction of personal rebuttal, we show the full results here:

**Table 3. Comparison of the effect of different pooling methods. The term "w/o STAR" refers to a scenario where an MLP is utilized with the Channel Independent (CI) strategy, without the use of STAR. The result reveals that incorporating STAR into the model leads to a consistent enhancement in performance across all pooling methods. Apart from that, stochastic pooling performs better than mean and max pooling.**
|Pooling Method|**ECL**||**Traffic**||**Weather**||**Solar**||**ETTh2**||**PEMS03**||**PEMS04**||**PEMS07**||
|:-------------|:---------|:---------|:----------|:---------|:----------|:---------|:---------|:---------|:---------|:---------|:---------|:---------|:---------|:---------|:---------|:---------|
||**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|
|w/oSTAR|0.187|0.273|0.442|0.281|0.261|0.281|0.247|0.272|0.381|0.406|0.135|0.235|0.143|0.245|0.143|0.232|
|Mean|**0.174**|0.266|0.420|0.277|0.261|0.281|0.234|0.262|0.379|0.404|0.106|0.212|0.106|0.212|0.090|0.188|
|Max|0.180|0.270|**0.406**|0.271|0.259|0.280|0.246|0.269|0.379|0.401|0.113|0.221|0.116|0.223|0.096|0.198|
|Weighted|0.184|0.275|0.440|0.292|0.263|0.284|0.264|0.280|0.379|0.403|0.118|0.226|0.109|0.218|0.097|0.200|
|Stochastic|**0.174**|**0.264**|0.409|**0.267**|**0.255**|**0.278**|**0.229**|**0.256**|**0.373**|**0.400**|**0.104**|**0.210**|**0.102**|**0.208**|**0.087**|**0.184**|

**Table 4. The performance of STAR in different models. The attention replaced by STAR here are the time attention in PatchTST, the channel attention in iTransformer, and both the time attention and channel attention in modified Crossformer.
The results demonstrate that replacing attention with STAR, which requires less computational resources, could maintain and even improve the models' performance in most datasets.**
|Model|Component|**ECL**||**Traffic**||**Weather**||**Solar**||**ETTh2**||**PEMS03**||**PEMS04**||**PEMS07**||
|---------------------|---------|---------|---------|-----------|-----------|-----------|-----------|---------|---------|---------|---------|----------|----------|----------|----------|----------|----------|
|||**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|
|PatchTST|Attention|0.189|0.276|0.454|0.286|0.256|0.279|0.236|0.266|**0.385**|**0.410**|0.137|0.240|0.145|0.249|0.144|0.233|
||STAR|**0.185**|**0.272**|**0.448**|**0.279**|**0.252**|**0.277**|**0.231**|**0.259**|0.391|0.413|**0.134**|**0.233**|**0.136**|**0.238**|**0.137**|**0.225**|
|Crossformer|Attention|0.202|0.301|**0.546**|0.297|0.254|0.310|0.206|0.258|2.772|1.271|0.100|0.208|0.090|0.198|0.084|0.181|
||STAR|**0.198**|**0.292**|0.549|**0.292**|**0.252**|**0.305**|**0.200**|**0.252**|**1.919**|**1.043**|**0.100**|**0.204**|**0.087**|**0.194**|**0.080**|**0.175**|
|iTransformer|Attention|0.178|0.270|0.428|0.282|0.258|0.278|0.233|0.262|0.383|0.407|0.113|0.221|0.111|0.221|0.101|0.204|
||STAR|**0.174**|**0.264**|**0.409**|**0.267**|**0.255**|**0.278**|**0.229**|**0.256**|**0.373**|**0.400**|**0.104**|**0.210**|**0.102**|**0.208**|**0.087**|**0.184**|

Thank you once again for your constructive comments and support.

Meta-review: This paper has been assessed by four knowledgeable reviewers. Two of them recommended accepting it (straight accept ratings) while two opted weakly for rejection. It presents an approach to modeling correlation between channels of multivariate time series to support forecasting tasks. The authors responded to initial feedback by providing a comprehensive rebuttal and engaged the reviewers in a  discussion. That resulted in improved understanding of the key points made by the authors and in increasing the reviewers' scores. This work is mature enough and sufficiently well presented to be included in the program of NeurIPS.
Decision: Accept (poster)