SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion

Authors: lu  han
xu-yang  chen
han-jia  ye
de-chuan  zhan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:45:43

Abstract: Multivariate time series forecasting plays a crucial role in various fields such as finance, traffic management, energy, and healthcare.  Recent studies have highlighted the advantages of channel independence to resist distribution drift but neglect channel correlations, limiting further enhancements. Several methods utilize mechanisms like attention or mixer to address this by capturing channel correlations, but they either introduce excessive complexity or rely too heavily on the correlation to achieve satisfactory results under distribution drifts, particularly with a large number of channels. Addressing this gap, this paper presents an efficient MLP-based model, the Series-cOre Fused Time Series forecaster (SOFTS), which incorporates a novel STar Aggregate-Redistribute (STAR) module. Unlike traditional approaches that manage channel interactions through distributed structures, \textit{e.g.}, attention, STAR employs a centralized strategy to improve efficiency and reduce reliance on the quality of each channel. It aggregates all series to form a global core representation, which is then dispatched and fused with individual series representations to facilitate channel interactions effectively. SOFTS achieves superior performance over existing state-of-the-art methods with only linear complexity. The broad applicability of the STAR module across different forecasting models is also demonstrated empirically. We have made our code publicly available at https://github.com/Secilia-Cxy/SOFTS.

Main paper: 
 introduction: 
 Time series forecasting plays a critical role in numerous applications across various fields, including environment [9] , traffic management [15] , energy [16] , and healthcare [27] . The ability to accurately predict future values based on previously observed data is fundamental for decision-making, policy development, and strategic planning in these areas. Historically, models such as ARIMA and Exponential Smoothing were standard in forecasting, noted for their simplicity and effectiveness in certain contexts [2] . However, the emergence of deep learning models, particularly those exploiting structures like Recurrent Neural Networks (RNNs) [14, 3, 29] and Convolutional Neural Networks (CNN) [1, 8] , has shifted the paradigm towards more complex models capable of understanding intricate patterns in time series data. To overcome the inability to capture long-term dependencies, Transformer-based models have been a popular direction and achieved remarkable performance, especially on long-term multivariate time series forecasting [48, 28, 26] .Earlier on, Transformer-based methods perform embedding techniques like linear or convolution layers to aggregate information from different channels, then extract information along the temporal dimension via attention mechanisms [48, 35, 49] . However, such channel mixing structures were 2 Related Work Time series forecasting. Time series forecasting is a critical area of research that finds applications in both industry and academia. With the powerful representation capability of neural networks, deep forecasting models have undergone a rapid development [22, 38, 37, 4, 5] . Two widely used methods for time series forecasting are recurrent neural networks (RNNs) and convolutional neural networks (CNNs). RNNs model successive time points based on the Markov assumption [14, 3, 29] , while CNNs extract variation information along the temporal dimension using techniques such as temporal convolutional networks (TCNs) [1, 8] . However, due to the Markov assumption in RNN and the local reception property in TCN, both of the two models are unable to capture the long-term dependencies in sequential data. Recently, the potential of Transformer models for long-term time series forecasting tasks has garnered attention due to their ability to extract long-term dependencies via the attention mechanism [48, 35, 49] .Efficient long-term multivariate forecasting and channel independence. Long-term multivariate time series forecasting is increasingly significant in decision-making processes [9] . While Transformers have shown remarkable efficacy in various domains [32] , their complexity poses challenges in long-term forecasting scenarios. Efforts to adapt Transformer-based models for time series with reduced complexity include the Informer, which utilizes a probabilistic subsampling strategy for more efficient attention mechanisms [48] , and the Autoformer, which employs autocorrelation and fast Figure 1 : Overview of our SOFTS method. The multivariate time series is first embedded along the temporal dimension to get the series representation for each channel. Then the channel correlation is captured by multiple layers of STAR modules. The STAR module utilizes a centralized structure that first aggregates the series representation to obtain a global core representation, and then dispatches and fuses the core with each series, which encodes the local information.Fourier transforms to expedite computations [35] . Similarly, FEDformer applies attention within the frequency domain using selected components to enhance performance [49] . Despite these innovations, models mixing channels in multivariate series often exhibit reduced robustness to adapt to distribution drifts and achieve subpar performance [45, 11] . Consequently, some researchers have adopted a channel-independent approach, simplifying the model architecture and delivering robust results as well [28, 23] . However, ignoring the interactions among variates can limit further advancements. Recent trends have therefore shifted towards leveraging attention mechanisms to capture channel correlations [47, 33, 26] . Even though the performance is promising, their scalability is limited on large datasets. Another stream of research focuses on modeling time and channel dependencies through simpler structures like MLP [46, 7, 42 ]. Yet, they usually achieve sub-optimal performance compared to SOTA transformer-based methods, especially when the number of channels is large.In this paper, we propose a new MLP-based method that breaks the dilemma of performance and efficiency, achieving state-of-the-art performance with merely linear complexity to both the number of channels and the length of the lookback window. 
softs: 
 Multivariate time series forecasting (MTSF) deals with time series data that contain multiple variables, or channels, at each time step. Given historical values X ∈ R C×L where L represents the length of the lookback window, and C is the number of channels. The goal of MTSF is to predict the future values Y ∈ R C×H , where H > 0 is the forecast horizon.Our Series-cOre Fused Time Series forecaster (SOFTS) comprises the following components and its structure is illustrated in Figure 1 .Reversible instance normalization. Normalization is a common technique to calibrate the distribution of input data. In time series forecasting, the local statistics of the history are usually removed to stabilize the prediction of the base forecaster and restore these statistics to the model prediction [17] . Following the common practice in many state-of-the-art models [28, 26] , we apply reversible instance normalization which centers the series to zero means, scales them to unit variance, and reverses the normalization on the forecasted series. For PEMS dataset, we follow Liu et al. [26] to selectively perform normalization according to the performance.Series embedding. Series embedding is an extreme case of the prevailing patch embedding in time series [28] , which is equivalent to setting the patch length to the length of the whole series [26] . Unlike patch embedding, series embedding does not produce extra dimension and is thus less complex than patch embedding. Therefore, in this work, we perform series embedding on the lookback window. The comparison of the STAR module and several common modules, like attention, GNN and mixer. These modules employ a distributed structure to perform the interaction, which relies on the quality of each channel. On the contrary, our STAR module utilizes a centralized structure that first aggregates the information from all the series to obtain a comprehensive core representation.Then the core information is dispatched to each channel. This kind of interaction pattern reduces not only the complexity of interaction but also the reliance on the channel quality.Concretely, we use a linear projection to embed the series of each channel to S 0 = R C×d , where d is the hidden dimension: S 0 = Embedding(X).(EQUATIONChannel interaction. The series embedding is refined by multiple layers of STAR modules:EQUATIONThe STAR module utilizes a star-shaped structure that exchanges information between different channels, as will be fully described in the next section.Linear predictor. After N layers of STAR, we use a linear predictor (R d → R H ) to produce the forecasting results. Assume the output series representation of layer N to be S N , the prediction Ŷ ∈ R C×H is computed as: Ŷ = Linear(S N ).Our main contribution is a simple but efficient STar Aggregate-Redistribute (STAR) module to capture the dependencies between channels. Existing methods employ modules like attention to extract such interaction. Although these modules directly compare the characteristics of each pair, they are faced with the quadratic complexity related to the number of channels. Besides, such a distributed structure may lack robustness when there are abnormal channels for the reason that they rely on the extract correlation between channels. Existing research on channel independence has already proved the untrustworthy correlations on non-stationary time series [45, 11] . To this end, we propose the STAR module to solve the inefficiency of the distributed interaction modules. This module is inspired by the star-shaped centralized system in software engineering, where instead of letting the clients communicate with each other, there is a server center to aggregate and exchange the information [30, 10] , whose advantage is efficient and reliable. Following this idea, the STAR replaces the mutual series interaction with the indirect interaction through a core, which represents the global representation across all the channels. Compared to the distributed structure, STAR takes advantage of the robustness brought by aggregation of channel statistics [11] , and thus achieves even better performance. Figure 2 illustrates the main idea of STAR and its difference between existing models like attention [32] , GNN [19] and Mixer [31] .Given the series representation of each channel as input, STAR first gets the core representation of the multivariate series, at the heart of our SOFTS method. We define the core representation as follows: Definition 3.1 (Core Representation). Given a multivariate series with C channels {s 1 , s 2 , . . . , s C }, the core representation o is a vector generated by an arbitrary function f with the following form:o = f (s 1 , s 2 , . . . , s C )The core representation encodes the global information across all the channels. To obtain such representation, we employ the following form, which is inspired by the Kolmogorov-Arnold representation theorem [20] and DeepSets [43] :EQUATIONwhere MLP 1 : R d → R d ′ is a projection that projects the series representation from the series hidden dimension d to the core dimension d ′ , composing two layers with hidden dimension d and GELU [13] activation. Stoch_Pool is the stochastic pooling [44] that get the core representation o ∈ R d ′ by aggregating representations of C series. Stochastic pooling combines the advantages of mean and max pooling. The details of computing the core representation can be found in Appendix B.2. Next, we fuse the representations of the core and all the series:EQUATIONEQUATIONThe Repeat_Concat operation concatenate the core representation o to each series representation, and we get the F i ∈ R C×(d+d ′ ) . Then another MLP (MLP 2 : R d+d ′ → R d ) is used to fuse the concatenated presentation and project it back to the hidden dimension d, i.e., S i ∈ R C×d . Like many deep learning modules, we also add a residual connection from the input to the output [12] .We analyze the complexity of each component of SOFTS step by step concerning window length L, number of channels C, model dimension d, and forecasting horizon H. The complexity of the reversible instance normalization and series embedding is O(CL) and O(CLd) respectively. In STAR, assumingd ′ = d, the MLP 1 is a R d → R d mapping with complexity O(Cd 2 ).Stoch_Pool computes the softmax along the channel dimension, with complexity O(Cd). The MLP 2 on the concatenated embedding has the complexity O(Cd 2 ). The complexity of the predictor is O(CdH). In all, the complexity of the encoding part is O(CLd + Cd 2 + CdH), which is linear to C, L, and H. Ignoring the model dimension d, which is a constant in the algorithm and irrelevant to the problem, we compare the complexity of several popular forecasters in Table 1 . Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We have provided an anonymous code for review along with the scripts processing the data. The code will be made public once the paper is accepted. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 Datasets. To thoroughly evaluate the performance of our proposed SOFTS, we conduct extensive experiments on 6 widely used, real-world datasets including ETT (4 subsets), Traffic, Electricity, Weather [48, 35] , Solar-Energy [21] and PEMS (4 subsets) [24] . Detailed descriptions of the datasets can be found in Appendix A.Compared methods. We extensively compare the recent Linear-based or MLP-based methods, including DLinear [45] , TSMixer [7] , TiDE [6] . We also consider Transformer-based methods including FEDformer [49] , Stationary [25] , PatchTST [28] , Crossformer [47] , iTransformer [26] and CNN-based methods including SCINet [24] , TimesNet [36] .Forecasting benchmarks. The long-term forecasting benchmarks follow the setting in Informer [48] and SCINet [24] . The lookback window length (L) is set to 96 for all datasets. We set the prediction horizon (H) to {12, 24, 48, 96} for PEMS and {96, 192, 336, 720} for others. Performance comparison among different methods is conducted based on two primary evaluation metrics: Mean Squared Error (MSE) and Mean Absolute Error (MAE). The results of PatchTST and TSMixer are reproduced for the ablation study and other results are taken from iTransformer [26] .Implementation details. We use the ADAM optimizer [18] with an initial learning rate of Model efficiency. Our SOFTS model demonstrates efficient performance with minimal memory and time consumption. Figure 3b illustrates the memory and time usage across different models on the Traffic dataset, with lookback window L = 96, horizon H = 720, and batch size 4. Despite their low resource usage, Linear-based or MLP-based models such as DLinear and TSMixer perform poorly with a large number of channels. Figure 3a explores the memory requirements of the three best-performing models from Figure 3b . This figure reveals that the memory usage of both PatchTST and iTransformer escalates significantly with an increase in channels. In contrast, our SOFTS model maintains efficient operation, with its complexity scaling linearly with the number of channels, effectively handling large channel counts.In this section, the prediction horizon (H) is set to {12, In Figure 3b , we set the lookback window L = 96, horizon H = 720, and batch size to 4 in Traffic dataset. Figure 3a reveals that SOFTS model scales to large number of channels more effectively than Transformer-based models. Figure 3b shows that previous Linear-based or MLPbased models such as DLinear and TSMixer perform poorly with a large number of channels. While SOFTS model demonstrates efficient performance with minimal memory and time consumption.in performance across all pooling methods. Additionally, stochastic pooling deserves attention as it outperforms the other methods across nearly all the datasets.Table 3 : Comparison of the effect of different pooling methods. The term "w/o STAR" refers to a scenario where an MLP is utilized with the Channel Independent (CI) strategy, without the use of STAR. The result reveals that incorporating STAR into the model leads to a consistent enhancement in performance across all pooling methods. Apart from that, stochastic pooling performs better than mean and max pooling. Full results can be found in Table 7 .Pooling [39, 41] that is replaceable to arbitrary transformer-based methods that use the attention mechanism. In this paragraph, we test the effectiveness of STAR on different existing transformer-based forecasters, such as PatchTST [28] and Crossformer [47] . Note that our method can be regarded as replacing the channel attention in iTransformer [26] . Here we involve substituting the time attention in PatchTST with STAR and incrementally replacing both the time and channel attention in Crossformer with STAR. The results, as presented in Table 4 , demonstrate that replacing attention with STAR, which deserves less computational resources, could maintain and even improve the models' performance in several datasets.Influence of lookback window length. Common sense suggests that a longer lookback window should improve forecast accuracy. However, incorporating too many features can lead to a curse Hyperparameter sensitivity analysis. We investigate the impact of several key hyperparameters on our model's performance: the hidden dimension of the model, denoted as d, the hidden dimension of the core, represented by d ′ , and the number of encoder layers, N . Analysis of Figure 5 indicates that complex traffic datasets (such as Traffic and PEMS) require larger hidden dimensions and more encoding layers to handle their intricacies effectively. Moreover, variations in d ′ have a minimal influence on the model's overall performance.Series embedding adaptation of STAR. The STAR module adapts the series embeddings by extracting the interaction between channels. To give an intuition of the functionality of STAR, we visualize the series embeddings before and after being adjusted by STAR. The multivariate series is selected from the test set of Traffic with look back window 96 and number of channels 862. Figure 6 shows the series embeddings visualized by T-SNE before and after the first STAR module. Among the 862 channels, there are 2 channels embedded far away from the other channels. These two channels can be seen as anomalies, marked as (⋆) in the figure. Without STAR, i.e., using only the channel independent strategy, the prediction on the series can only achieve 0.414 MSE. After being adjusted by STAR, the abnormal channels can be clustered towards normal channels by exchanging channel Forecasting on the embeddings achieves 0.414 MSE. 6b: series embeddings after being adjusted by STAR. The two channels are clustered towards normal channels (△) by exchanging channel information. Adapted series embeddings improve forecasting performance to 0.376. Figure 6c: Impact of noise on one channel. Our method is more robust against channel noise than other methods.information. An example of the normal channels is marked as (△). Predictions on the adapted series embeddings can improve the performance to 0.376, a 9% improvement.Impact of channel noise. As previously mentioned, SOFTS can cluster abnormal channels towards normal channels by exchanging channel information. To test the impact of an abnormal channel on the performance of three models-SOFTS, PatchTST, and iTransformer-we select one channel from the PEMS03 dataset and add Gaussian noise with a mean of 0 and a standard deviation representing the strength of the noise. The lookback window and horizon are set to 96 for this experiment. In Figure 6c , we observe that the MSE of PatchTST increases sharply as the strength of the noise grows. In contrast, SOFTS and iTransformer can better handle the noise. This indicates that suitable channel interaction can improve the robustness against noise in one channel using information from the normal channels. Moreover, SOFTS demonstrates superior noise handling compared to iTransformer. This suggests that while the abnormal channel can affect the model's judgment of normal channels, our STAR module can mitigate the negative impact more effectively by utilizing core representation instead of building relationships between every pair of channels. 
conclusion: 
 Although channel independence has been found an effective strategy to improve robustness for multivariate time series forecasting, channel correlation is important information to be utilized for further improvement. The previous methods faced a dilemma between model complexity and performance in extracting the correlation. In this paper, we solve the dilemma by introducing the Series-cOre Fused Time Series forecaster (SOFTS) which achieves state-of-the-art performance with low complexity, along with a novel STar Aggregate-Redistribute (STAR) module to efficiently capture the channel correlation.Our paper explores the way of building a scalable multivariate time series forecaster while maintaining equal or even better performance than the state-of-the-art methods, which we think may pave the way to forecasting on datasets of more larger scale under resource constraints [50] . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper for all the baselines in Section 4.1, and give the URL of the datasets in Appendix A.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We detail the description plus the link to download them here:1. ETT (Electricity Transformer Temperature) [48] Other details of these datasets have been concluded in Table 5 . The overall architecture of SOFTS is detailed in Algorithm 1. Initially, a linear layer is employed to obtain the embedding for each series (Lines 1-2). Subsequently, several encoder layers are applied. Within each encoder layer, the core representation is first derived by applying an MLP to the series embeddings and pooling them (Line 4). This core representation is then concatenated with each series (Line 5), and another MLP is used to fuse them (Line 6). After passing through multiple encoder layers, a final linear layer projects the series embeddings to the predicted series (Line 8).Algorithm 1 Series-cOre Fused Time Series forecaster (SOFTS) applied to multivariate time series.Require:Look back window X ∈ R L×C ; 1: X = X. transpose ▷ X ∈ R C×L 2: S 0 = Linear(X)▷ Get series embedding, S 0 ∈ R C×d 3: for l = 1 . . . L do 4:EQUATION:F i = Repeat_Concat(S i-1 , o i ) ▷ F i ∈ R C×(d+d ′ )6:S i = MLP(F i ) + S i-1▷ Fuse series and core, S i ∈ R C×d 7: end for o = f (s 1 , s 2 , . . . , s C )To obtain the representation, we draw inspiration from the two theorems: Theorem B.1 (Kolmogorov-Arnold representation [20] ). Let f : [0, 1] M → R be an arbitrary multivariate continuous function iff it has the representationf (x 1 , . . . , x M ) = ρ M m=1 λ m ϕ (x m )with continuous outer and inner functions ρ : R 2M +1 → R and ϕ : R → R 2M +1 . The inner function ϕ is independent of the function f . Theorem B.2 (DeepSets [43] ). Assume the elements are from a compact set in R d , i.e. possibly uncountable, and the set size is fixed to M . Then any continuous function operating on a set X, i.e. f : R d×M → R which is permutation invariant to the elements in X can be approximated arbitrarily close in the form ofρ x∈X ϕ(x) ,for suitable transformations ϕ and ρ.The two formulations are very similar, except for the dependence of inner transformation on the coordinate through λ m . The existence of λ determines whether the formulation is permutation invariant or not. In this paper, we find in Table 4 that the permutation invariant expression (Theorem B.2) performs much better than the permutation variant one Theorem B.1. This may be attributed to the characteristics of channel series being enough to induce the index of each channel (coordinate).Introducing extra parameters specific to each channel may enhance the dependency channel coordinate and reduce the dependence on the history, therefore causing low robustness when encountering unknown series. Consequently, we utilize DeepSets form to express the core representation:o = ρ s∈S ϕ(s) .We propose two modifications to the expression:1. We generalize the mean pooling over the inner transformation by arbitrary pooling methods.2. We remove the outer transformation ρ because it is redundant with the MLP during the fusion process.For 1., we tested several common pooling methods and found that the mean pooling and max pooling outperform each other in different cases. Stochastic pooling (described in the following paragraph) can achieve the best results in averaged cases (Table 3 ). So, the core is computed as Equation (3).Stochastic pooling. Stochastic pooling is a pooling method that combines the characteristics of max pooling and mean pooling [44] . In stochastic pooling, the pooled map response is selected by sampling from a multinomial distribution formed from the activations of each pooling region. Specifically, we first calculate the probabilities p for each dimension j by normalizing the softmax activations within the dimension:EQUATIONDuring training, we sample from the multinomial distribution based on p to pick a channel c within the dimension j. The pooled result is then simply A cj :EQUATIONAt test time, we use a probabilistic form of averaging:EQUATIONThis approach allows for a more robust and statistically grounded pooling mechanism, which can enhance the generalization capabilities of the model across different data scenarios.All the experiments are conducted on a single NVIDIA GeForce RTX 3090 with 24G VRAM. The mean squared error (MSE) loss function is utilized for model optimization. Performance comparison among different methods is conducted based on two primary evaluation metrics: Mean Squared Error (MSE) and Mean Absolute Error (MAE). We use the ADAM optimizer [18] with an initial learning rate of 3 × 10 -4 . This rate is modulated by a cosine learning rate scheduler. We explore the number of STAR blocks N within the set {1, 2, 3, 4}, and the dimension of series d within {128, 256, 512}.Additionally, the dimension of the core representation d ′ is searched in {64, 128, 256, 512}, with the constraint that d ′ does not exceed d.Mean Squared Error (MSE):EQUATIONMean Absolute Error (MAE):EQUATIONwhere Y, Ŷ ∈ R H×C are the ground truth and prediction results of the future with H time points and C channels. Y i denotes the i-th future time point.The complete results of our forecasting benchmarks are presented in Table 6 . We conducted experiments using six widely utilized real-world datasets and compared our method against ten previous state-of-the-art models. Our approach, SOFTS, demonstrates strong performance across these tests.The complete results of our pooling method ablation are presented in Table 7 . The term "w/o STAR" refers to a scenario where an MLP is utilized with the Channel Independent (CI) strategy, without the use of STAR. Mean pooling computes the average value of all the series representations. Max pooling selects the maximum value of each hidden feature among all the channels. Weighted average learns the weight for each channel. Stochastic pooling applies random selection during training and weighted average during testing according to the feature value. The result reveals that incorporating STAR into the model leads to a consistent enhancement in performance across all pooling methods. The complete results of our ablation on universality of STAR are presented in Table 8 . The STar Aggregate-Redistribute (STAR) module is a set-to-set function [39] that is replaceable to arbitrary transformer-based methods that use the attention mechanism. In this paragraph, we test the effectiveness of STAR on different existing transformer-based forecasters, such as PatchTST [28] and Crossformer [47] . Note that our method can be regarded as replacing the channel attention in iTransformer [26] . Here we involve substituting the time attention in PatchTST with STAR and incrementally replacing both the time and channel attention in Crossformer with STAR. The results, as presented in Table 8 , demonstrate that replacing attention with STAR, which deserves less computational resources, could maintain and even improve the models' performance in several datasets.In this section, we extend the lookback ablation in section 4.2 to L ∈ [48, 720] . Figure 7 shows the results in MSE. SOFTS performs almost consistently better than other models under different lookback window lengths. However, we also warn about the potential overfitting when the lookback length is very large, i.e. L = 512 or L = 720. We investigate the impact of several key hyperparameters on our model's performance: the hidden dimension of the model, denoted as d, the hidden dimension of the core, represented by d ′ , and the number of encoder layers, N . Figure 8 and Figure 10 indicate that complex traffic datasets (such as Traffic and PEMS) require larger hidden dimensions and more encoding layers to handle their intricacies effectively. Moreover, Figure 9 shows that variations in d ′ don't influence the model's overall performance so much. In this section, we test the robustness of SOFTS. We conducted 5 experiments using different random seeds, and the averaged results are presented in Table 9 . It can be seen that SOFTS have robust performance over different datasets and different horizons. 0.325 ± 0.002 0.361 ± 0.002 0.166 ± 0.002 0.208 ± 0.002 0.376 ± 0.002 0.251 ± 0.001 192 0.375 ± 0.002 0.389 ± 0.003 0.217 ± 0.003 0.253 ± 0.002 0.398 ± 0.002 0.261 ± 0.002 336 0.405 ± 0.004 0.412 ± 0.003 0.282 ± 0.001 0.300 ± 0.001 0.415 ± 0.002 0.269 ± 0.002 720 0.466 ± 0.004 0.447 ± 0.002 0.356 ± 0.002 0.351 ± 0.002 0.447 ± 0.002 0.287 ± 0.001EQUATION0.064 ± 0.002 0.165 ± 0.002 0.074 ± 0.000 0.176 ± 0.000 0.057 ± 0.000 0.152 ± 0.000 24 0.083 ± 0.002 0.188 ± 0.002 0.088 ± 0.000 0.194 ± 0.000 0.073 ± 0.003 0.173 ± 0.004 48 0.114 ± 0.004 0.223 ± 0.003 0.110 ± 0.001 0.219 ± 0.002 0.096 ± 0.002 0.195 ± 0.002 96 0.156 ± 0.001 0.264 ± 0.001 0.135 ± 0.003 0.244 ± 0.003 0.120 ± 0.003 0.218 ± 0.003In this section, we present a visualization of the core. The visualization is generated by employing a frozen state of our trained model to capture the series embeddings from the final encoder layer. These embeddings are then utilized as inputs to a two-layer MLP autoencoder. The primary function of this autoencoder is to map these high-dimensional embeddings back to the original input series. The visualization is shown in Figure 11 . Highlighted by the red line, this core captures the global trend of all cross all the channels in Figure 11 : Visualization of the core, represented by the red line, alongside the original input channels. We freeze our model and extract the series embeddings from the last encoder layer to train a two-layer MLP autoencoder. This autoencoder maps the embeddings back to the original series, allowing us to visualize the core effectively.To provide a more intuitive demonstration of our model's performance, we present prediction showcases on the ECL (Figure 12 ), ETTh2 (Figure 13 ), Traffic (Figure 14 ), and PEMS03 (Figure 15 ) datasets. Additionally, we include prediction showcases from iTransformer and PatchTST on these datasets. The lookback window length and horizon are set to 96. In this section, we show more results on the series embedding adaptation of our STAR module, similar to showcases in figure 6a and figure 6b . The number of channels should be large enough to show the relationship between channels in the embedding space. Therefore, we select the datasets ECL, PEMS03, and Traffic with channels 321, 358, and 862 respectively. Figure 16 shows the results on these datasets. As stated in Section 4.2, after being adjusted by STAR, the abnormal channels can be clustered towards normal channels by exchanging channel information. In this section, we choose two abnormal channels in the ECL and PEMS03 datasets to demonstrate our SOFTS model's advantage in handling noise from abnormal channels. As shown in Figure 17 , the value of channel 160 in PEMS03 experiences a sharp decrease followed by a smooth period. In this case, SOFTS is able to capture the slowly increasing trend effectively. Similarly, in Figure 18 , the signal of channel 298 in ECL resembles the sum of an impulse function and a step function, which lacks a continuous trend. Here, our SOFTS model provides a more stable prediction compared to the other two models.While the Series-cOre Fused Time Series (SOFTS) forecaster demonstrates significant improvements in multivariate time series forecasting, several limitations must be acknowledged, providing directions for future work.Dependence on core representation quality. The effectiveness of the STAR module heavily depends on the quality of the global core representation. If this core representation does not accurately capture the essential features of the individual series, the model's performance might degrade.Ensuring the robustness and accuracy of this core representation across diverse datasets remains a challenge that warrants further research.Limited exploration of alternative aggregate-redistribute strategies. Although the STAR module effectively aggregates and redistributes information, the exploration of alternative strategies is limited. Future work could investigate various methods for aggregation and redistribution to identify potentially more effective approaches, thereby enhancing the performance and robustness of the model. The development of the Series-cOre Fused Time Series (SOFTS) forecaster has the potential to significantly benefit various fields such as finance, traffic management, energy, and healthcare by improving the accuracy and efficiency of time series forecasting, thereby enhancing decision-making processes and optimizing operations. However, there are potential negative societal impacts to consider. Privacy concerns may arise from the use of personal data, especially in healthcare and finance, leading to possible violations if data is not securely handled. Additionally, biases in the data could result in unfair outcomes, perpetuating or exacerbating existing disparities. Over-reliance on automated forecasting models might lead to neglect of important contextual or qualitative factors, causing adverse outcomes when predictions are incorrect. To mitigate these risks, robust data protection protocols should be implemented, and continuous monitoring for bias is necessary to ensure fairness. Developing ethical use policies and maintaining human oversight in decision-making can further ensure that the deployment of SOFTS maximizes its positive societal impact while minimizing potential negative consequences.NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://github.com/zhouhaoyi/ETDatasethttp://pems.dot.ca.govhttps://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014https://pems.dot.ca.gov/ 


Decision: This paper has been assessed by four knowledgeable reviewers. Two of them recommended accepting it (straight accept ratings) while two opted weakly for rejection. It presents an approach to modeling correlation between channels of multivariate time series to support forecasting tasks. The authors responded to initial feedback by providing a comprehensive rebuttal and engaged the reviewers in a  discussion. That resulted in improved understanding of the key points made by the authors and in increasing the reviewers' scores. This work is mature enough and sufficiently well presented to be included in the program of NeurIPS.

Reviews: review: paper_summary: paper_summary: The authors present a framework for modeling correlations between channels in a multivariate time series forecasting task. This framework concatenates each channel embedding with a ‘global core embedding’ which contains information from all channels in the lookback window. The authors present experiments that demonstrate the utility of this concept, both from a performance and efficiency perspective.
strength_weakness: strength_weakness: In my opinion, the paper would be improved with analysis and discussion of their results.  There is little discussion beyond drawing attention to features of figures and tables, which misses an opportunity to explain why the authors believe they are observing such behavior.  I am specifically interested in a discussion between iTransformer and SOFTS, which appear to be quite similar along many dimensions that SOFTS claims to be superior.  Such a discussion will help guide potential readers through the considerations they should take into account when deciding which framework to implement on their own forecasting problems.

I also recommend adding either a few sentences or a small figure that highlights the differences between PatchTST, transformer, and SOFTS — I see some details in the text of table 4, but I think making this information more prominent would be helpful and make the paper more self-contained.
questions: questions: 1. Under what conditions should I choose SOFTS over iTransformer?  Figure 3a makes it appear as though when there are a sufficiently large number of channels in the time series.  Are there other considerations?  Figures 3b and 6c show slight improvements but the difference doesn't seem large enough to discriminate between which framework would be better for a reader's specific time series forecasting efforts.
2. What are the limitations of SOFTS? Or what are the tradeoffs between using SOFTS vs other models?  A few words of discussion in the conclusion would help improve the clarity of the paper.
limitations: limitations: As is, there is no discussion of the limitations in the main text of the paper.  To address this partially, section G of the appendix could be moved to the main text.

However, the discussion of the limitations of SOFTS currently exists in a vacuum, lacking a comparison/contrast with other models mentioned in the paper, especially PatchTST and iTransformer.  The authors should characterize and highlight the characteristics of datasets and inference tasks where SOFTS outperforms existing methods by wide margin.
paper_summary: paper_summary: This paper proposes to use a global representation to capture the channel correlations for multivariate time series forecasting. Specifically, it uses stochastic pooling to get the global representation by aggregating representations of individual series and then concats the global representation and individual representations to reflect channel correlations for each series. Experiment results confirm that the proposal is much efficiency and achieves better performance than existing methods.
strength_weakness: strength_weakness: 1. Some experimental results are unconvincing.

-It is unclear why different datasets and metrics are used for different ablation studies, e.g., the datasets used in Table 3 and 4 are different, MAE is used in Figure 4 but MSE is used in other figures.

-There is no statistical significance tests between the results of the proposal and baselines.

-The results of Lookback Window Length 720 should be given as done in other papers.

-It is better to give the training time for each methods as well.

2. Some places are not clearly described.

-It is unclear the MLP and Linear operations are channel independent or dependent. I can guess they are channel independent, but it is not clearly described in Figure 1 and Section 3.

-The sentence "... rely too heavily on the correlation to achieve satisfactory results under distribution drift" in the Abstract is not clearly explained. 

3. Minor mistakes or typos: Embedding module is missing in Figure1; oi ∈ Rd' should be oi ∈ RC×d' in Algorithm 1.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This paper presents an efficient MLP-based model, the Series-cOre Fused Time Series forecaster (SOFTS). SOFTS incorporates a novel STar Aggregate-Redistribute (STAR) module to aggregate all series to form a global core representation, which is then dispatched and fused with individual series representations to facilitate channel interactions. The broad applicability of the STAR module across different forecasting models is also demonstrated empirically.
strength_weakness: strength_weakness: The paper lacks significant innovation. The stochastic pooling itself is not novel in deep neural networks. A somewhat novel facet of the proposed model is to use the pooling to extract global representations. However, the idea of aggregate-and-dispatch the interactions between variables has already been studied in TimeXer [1].


[1] Wang, Y., Wu, H., Dong, J., Liu, Y., Qiu, Y., Zhang, H., ... & Long, M. (2024). Timexer: Empowering transformers for time series forecasting with exogenous variables. arXiv preprint arXiv:2402.19072.
questions: questions: 1. It seems that all series share a core representation and MLP layers, but different variables will require different aspects of information. How can the model address these differences?
2. Please increase the look-back window length of the input series to  512 and 720, and compare the performance with other baseline models.
3. Please provide more results on abnormal channels, i.e. ETT, ECL, and Traffic dataset.
limitations: limitations: yes
paper_summary: paper_summary: The paper introduces SOFTS (Series-cOre Fused Time Series forecaster), an efficient multivariate time series forecasting model that addresses the gap between channel independence and channel correlation in a novel way. By utilizing a centralized STAR (STar Aggregate-Redistribute) module, SOFTS creates a global core representation aggregated from all series, which is then redistributed and fused with individual series representations. This mechanism allows for efficient channel interaction while reducing dependence on the quality of each channel. The paper demonstrates SOFTS's superiority over state-of-the-art methods in terms of performance and computational complexity, and showcases the STAR module's adaptability across various forecasting models.
strength_weakness: strength_weakness: - Additional experiments evaluating the robustness of SOFTS under varying conditions of distribution drift could provide deeper insight into its reliability.
questions: questions: - Has the author currently tried any other methods for aggregation and redistribution?
- SOFTS is a linear based model, so how to handle non-linear time series prediction?
limitations: limitations: - The authors have adequately addressed the technical limitations of their work within the paper.

Comments: comment: Thank you very much for your reply. Stochastic pooling is indeed the best-performing design, but it is not the core contribution of this paper; it is merely an implementation method for the STAR structure we proposed. In most cases, the performance impact is primarily due to the STAR structure rather than the pooling method. Anyway, we appreciate your positive attitude toward our work.
Thank you for your rebuttal and for addressing my concerns. I still maintain my original score as a borderline accept. I appreciate using stochastic pooling to capture global core representation, while it lacks theoretical analysis. As described in $\underline{\text{Line 437-439}}$, the authors tested several common pooling methods, and stochastic pooling is more of a result-oriented design.
Thanks very much for your recognition and valuable advice for improving our manuscript. We hope our work is helpful to you.
Thank your for your response and your elaboration on some of the questions I had.  In my opinion, your revisions make the paper more broadly accessible and self contained. This should help amplify the impact of your work.  I have raised my score.  Nice work!
Thanks very much for your recognition. We are happy that your concerns have been fully resolved.
Thank you for your detailed rebuttal. The additional experiments on robustness and the clarification on aggregation methods are appreciated. The explanation of SOFTS's capability to handle non-linear predictions is satisfactory. We look forward to seeing the further refined experiments in the final version. Well done!
Thank you for raising the score. If you have any additional concerns or questions, we are willing to address them.
Thank you for the responses. Considering other reviewers' comments as well, I have updated my rating.