DDN: Dual-domain Dynamic Normalization for Non-stationary Time Series Forecasting

Authors: tao  dai
beiliang  wu
peiyuan  liu
naiqi  li
xue  yuerong
shu-tao  xia
zexuan  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 14:01:31

Abstract: Deep neural networks (DNNs) have recently achieved remarkable advancements in time series forecasting (TSF) due to their powerful ability of sequence dependence modeling. To date, existing DNN-based TSF methods still suffer from unreliable predictions for real-world data due to its non-stationarity characteristics, i.e., data distribution varies quickly over time. To mitigate this issue, several normalization methods (e.g., SAN) have recently been specifically designed by normalization in a fixed period/window in the time domain. However, these methods still struggle to capture distribution variations, due to the complex time patterns of time series in the time domain. Based on the fact that wavelet transform can decompose time series into a linear combination of different frequencies, which exhibits distribution variations with time-varying periods, we propose a novel Dual-domain Dynamic Normalization (DDN) to dynamically capture distribution variations in both time and frequency domains. Specifically, our DDN tries to eliminate the non-stationarity of time series via both frequency and time domain normalization in a sliding window way. Besides, our DDN can serve as a plug-in-play module, and thus can be easily incorporated into other forecasting models. Extensive experiments on public benchmark datasets under different forecasting models demonstrate the superiority of our DDN over other normalization methods. Code will be made available following the review process.

Main paper: 
 introduction: 
 Deep neural networks (DNNs) with powerful dependency modeling capability have recently been widely used in time series forecasting (TSF) applications, including weather prediction [1] , energy consumption estimation [2] , and traffic flow forecasting [3] . Despite the great advancements of DNN-based TSF methods [4, 5, 6, 7] , they still suffer from unreliable predictions for real-world data due to its non-stationary nature of real-world time series, i.e., data distribution within the series varies quickly over time (a.k.a, distribution drift [8, 9, 10] ). Such non-stationary challenge limits the real applications of DNN-based TSF methods.To mitigate the problem of distribution drift, the classic reversible normalization [11] has recently been proposed with a two-stage pipeline of normalization and de-normalization. The former stage of normalization eliminates non-stationary factors for converting a non-stationary sequence into a stationary sequence, which has to acquire the mean and standard deviation of the sequence before. The latter stage of de-normalization reconstructs non-stationary information from the distribution prediction model or directly reuses the mean and standard deviation acquired in normalization. information. In contrast, recent Sota methods indicate that two modeling ways can be better: CI (Channel Independent) and CD (Channel Dependent). The primary distinction between the two approaches lies in the former focusing only on cross-time features but the latter incorporating crossdimension features. Theoretically, the latter can leverage more information and achieve higher prediction accuracy [21, 22, 23] . In practice, for relatively short input series, CD methods [24, 25] achieve comparable or even better performance than the CI methods. However, for longer input sequences, the situation is often the opposite [26, 4, 27] . In recent research, this difference can be attributed to the CD having higher capacity but often lacking robustness in predicting distributional drift than CI [28, 29] , while longer series typically experience more severe distributional drift. The superior performance of CI highlights the importance of handling distribution drift, and it is a valuable direction in the current research on time series forecasting.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We have clearly outlined the main contributions of this paper in the abstract and introduction sections.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
stationary for time series forecasting: 
 RevIN [11] was the first work to apply reversible normalization for time series forecasting, which assumes that history and future sequences share the same distribution. It counts distribution statistics of historical sequence for both normalization and de-normalization. Due to its simplicity and impressive effectiveness, it has been widely used in recent works [30, 31] . However, RevIN overlooks the distributional differences between historical and future sequences. Building upon RevIN, Dish-TS [12] proposes different distribution characteristics for historical and future sequences, using a distribution forecasting model to predict mean and standard deviation. Concurrently, NST [13] employs a module to provide more consistent distribution with future distribution, which can refer to appendix B. Furthermore, SAN [14] notes that existing distribution assumptions may not adapt to the scenario that time series points rapidly change over time [32, 33] and proposes a more fine-grained method, which supposes the distribution characteristics of time points is different between slices but same within a slice. Nevertheless, SAN still stops at the slice level, rather than the time series point level. Meanwhile, existing works lack consideration of the discrepancies between low and high frequencies, leading to insufficient consideration of non-stationary information.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have mentioned it in both the methods and experimental analysis sections.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In the realm of multivariate time series forecasting, we consider a historical sequence X ∈ R M ×L and aim to predict the corresponding future sequence Y ∈ R M ×T , M is the number of channels. DDN is a model-agnostic plugin designed to align the distribution characteristics of X and accurately estimate the distribution of Y . In this section, we will comprehensively outline the pipeline of the entire framework and elaborate on how to remove and reconstruct non-stationary factors of time series. To enhance clarity and facilitate understanding of subsequent chapters, the key notations used in this paper are summarized in Table 1 , and the framework can be referred to in Figure 2 .The time steps of the historical/future sequences x i , y iThe i-th historical or future series xi * , ȳi *The x i after normalization and it predicted series µ i , σ iThe i-th mean or standard deviation series of x i µ i y , σ i yThe i-th mean or standard deviation series of y i µ i * , σ i * The distribution forecasting of µ i or σ i Table 1 : Summary of key mathematical notationsAs depicted in Figure 2 , we first eliminate non-stationary factors via both the Frequency Domain Normalization (FDN) and the Time Domain Normalization (TDN). These processes output two stationary sequences and two sets of distribution characteristics. Two stationary sequences weighted to a sequence and input to the time series Forecasting Model (FM) for future sequence forecasting, while two sets of non-stationary factors input to Distribution Prediction Model (DPM) and predict future non-stationary factors. Finally, these factors are weighted together and incorporated with forecasting For each series x i , we perform a sliding window along the temporal dimension to acquire distribution characteristics, then replicate padding that will align the length of sliding statistics to the original series. Finally, sliding mean µ i and sliding standard deviation σ i represent to the distribution characteristics of x i . This process can be described as follows:EQUATIONHere 2k + 1 is the size of the sliding window, and stride is 1. After that, the size of sliding statistics is L -2k. Where µ j i and σ j i represent the mean value and standard deviation value of the j th time point respectively, where j ∈ {k + 1, • • • , L -k}. To make sure each time point possesses corresponding sliding statistics. We copy the sliding statistics closest in time by Pad (•) operation, the obtaining µ i and σ i are used to achieve the transformation from non-stationary sequences to stationary sequences. The process is as follows:EQUATIONHere, xi is the stationary series, ϵ is a positive number to prevent the denominator from zero, and ⊙ denotes the element-wise product. By this sliding normalization, annotated as SlidingNorm (•), we can acquire the non-stationary factors of each time point and convert non-stationary sequences to stationary sequences.Frequency Domain Normalization. In this branch, to exhaustively unveil non-stationary factors and eliminate them accurately. Discrete Wavelet Transform (DWT) is conducted on x i to separate the low-frequency component x i l and high-frequency component x i h . Subsequently, we acquire and eliminate their non-stationary factors. The process is as follows:EQUATIONHere, ϕ l,h is a pair of learnable wavelet bases. xi l , µ i l , and σ i l represent the stationary sequence, sliding mean, and sliding standard deviation of the low-frequency component. EQUATIONWhere xi , μi , and σi encompass the stationary sequences, sliding means, and sliding standard deviations of different frequency components. Through these operations, the output stationary sequence and distribution statistics maintain consistency with the dimensions of the input nonstationary sequence.Time Domain Normalization. We conduct the same manners in the time domain without frequency decomposition. The process can be formulated as follows:EQUATIONThe wavelet transform in FDN typically involves padding, which can potentially distort the statistical distribution information of the decomposed sequences. To address this, we implement sliding normalization directly on the original sequence. Consequently, the resulting distribution information is utilized for predicting future distributions, while the output stationary sequence is weighted with the stationary sequence derived from FDN.Stationary Sequences Weighting. Two stationary sequences from FDN and TDN will be weighted to a stationary output, which can be expressed as follows:EQUATIONHere, α is a trainable parameter and β = 1 -α. The weighted xi serves as the final stationary sequence, which is then inputted into FM for stable forecasting.We acquire two sets of sliding statistics reflecting distribution variations after FDN and TDN. Later, we refer to the structure of existing distribution prediction works [34, 12] to predict future distribution. Initially, we calculate the mean value of each sliding statistic to compute the statistical differences. Subsequently, the difference and original series are inputted for future difference prediction. Ultimately, these predicted differences added to the mean value as future sliding statistics.Frequency Domain Prediction. As shown in Figure 3 , for the distribution statistics of FDN, we predict future statistics by distribution prediction model and formulate as:EQUATIONHere, µ i f and σ i f are the mean values of μi and σi . While μi * and σi * denote the prediction of μi and σi . The MP is a mean prediction branch, and the SP is a standard deviation prediction branch. They are affiliated with DPM and adopt the same network structure.As the above frequency domain prediction process, for the distribution statistics of TDN, this prediction process can be formulated as follows:EQUATIONHere, µ i o and σ i o are the mean values of µ i and σ i , respectively. Likewise, µ i * and σ i * denote the prediction of µ i and σ i . The SP and the MP are noted in frequency domain prediction. De-normalization. After the aforementioned distribution predictions, two sets of estimated distribution statistics will be gained. Which are weighted to reconstruct the non-stationary information of the output of the time series forecasting model. This process can be described as follows:µ i * = µ iDistribution prediction and future series forecasting are essentially a bi-level optimization problem [35, 36, 37] , where distribution outputs significantly impact the future series output. To enhance the training effects of our models, we pre-train the DPM to yield a relatively well-trained DPM. This procedure can be formulated as follows:EQUATION)Where θ d represents the parameters of the DPM, it is noteworthy that the wavelet bases ϕ l,h and the weighted factor α belong to θ d . We select the mean square error (MSE) as our loss function between the predicted distribution and the ground truth of the distribution, acquired from the future sequence through TDN. Subsequently, assuming a total training duration of T epochs, the parameters θ d of the DPM will be frozen, then we train FM for T 1 epochs. Finally, DPM and FM will be subject to collaborative training during the remaining T -T 1 epochs. The process is as follows:EQUATION)Where t denotes the t th epoch during the training process, and ϕ f represents the parameters of the FM. We train DPM and FM concurrently to mitigate potential errors in the pretraining stage, as the ground truth of distribution is drived from future sequences based on distritution assumption. This ground truth is somewhat inconsistent with the actual situation and fails to account for high-frequency distribution changes. Consequently, we pretrain the DPM using assumption-based distribution ground truth, and then collaborative train it jointly based on the loss derived from the future sequences.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We have clearly outlined the assumption and proof of this paper in the Introduction, Method, and Appendix sections. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section, we conduct comprehensive experiments on multiple real-world time series datasets to assess the effectiveness of our proposed reversible normalization method DDN.Datasets We conduct extensive experiments on these seven popular real-world datasets [18] , including Electricity Transformer Temperature (ETT) with its four subsets (ETTh1, ETTh2, ETTm1, ETTm2), Weather, Electricity, and Traffic. The setting of these datasets following original works [18, 20] , and more descriptions about these datasets present in appendix A.1.Baselines. DDN is a model-agnostic method that can be applied to any mainstream time series forecasting model. To demonstrate its versatility, we integrate DDN into several representative models, including the earlier proposed models FEDformer [19] and Autoformer [20] , the CI model DLinear [27] , and the CD model iTransformer [24] .Implementation details. Our experiments were conducted three times with a consistent random seed and averaged to mean values. The Mean Square Error (MSE) and Mean Absolute Error (MAE) are chosen as evaluation metrics, with MSE serving as the training loss. All models use the same prediction lengths T = {96, 192, 336, 720}. For the look-back window L, Autoformer [20] and FEDformer [19] use L = 96, while DLinear [27] and iTransformer [24] utilize L = 336 and L = 720 respectively. The wavelet bases initialize to the "coiflet" bases, the default size of our sliding window is set to 7 for information content and temporal locality balance, and α starts at zero. More implementation details of our experiments can be referred to appendix A.2.As illustrated, the DDN method significantly enhances the predictive performance of the four different baselines across nearly all datasets. For the MSE metric, this improvement is particularly evident in the three relatively large datasets: Weather, Electricity, and Traffic. Utilizing DDN, Autoformer achieves a relative error reduction of 19.2%, 24.7%, and 25.6%, respectively, while FEDformer achieves a relative error reduction of 13.1%, 16.2%, and 22.3%. Similarly, incorporating the DDN into the other models also results in substantial performance gains. Additionally, Autoformer, FEDformer, and DLinear do not employ reversible normalization in official implements. While iTransformer utilizes the RevIN [11] normalization technique based on static statistics. However, replacing the RevIN module in iTransformer with the DDN module still yields significant performance improvements. These results strongly demonstrate that DDN makes the baseline model more robust in forecasting. Quantitative evaluation. In this part, we compare recent representative reversible normalization methods using the average MSE metric across four prediction lengths in {96, 192, 336, 720}. Detailed descriptions of these methods and links for access are provided in the Appendix B. As shown in Table 3 , DDN not only significantly enhances the predictive performance of the baseline models, including RevIN [11] , NTS [13] , and Dish-TS [12] , but also demonstrates superior results when compared to existing reversible normalization methods. For instance, taking the recent slice-level SAN [14] method as an example, DDN achieves substantial improvements even upon its solid foundation. Further comparative details are detailed in Appendix D. Considering that early models often lacked robust generalizability as their naive modeling strategies, we additionally included comparisons with two recent representative models: DLinear [27] (modeling features with CI) and iTransformer [24] (modeling features with CD). These methods already have good non-stationary adaptability, so they can better reflect the performance upper limit of fine-grained normalization methods. As illustrated in table 4 , DDN significantly outperforms the recent stateof-the-art model SAN in handling non-stationary information. Overall, DDN almost achieves the best performance compared to SAN in every forecasting case. Specifically, on the Traffic dataset, SAN [14] struggles to achieve satisfactory predictive performance and even performs worse than the original predictive model. In contrast, DDN demonstrates effects by a finer-grain non-stationarity processing.Qualitative Evaluation. As shown in the figure 4 , we compare DDN with reversible normalization methods at different granularities. It can be observed that there are significant differences among the various reversible normalization methods, which are related to their specific implementations.From subplots (a) and (b), it is evident that RevIN [11] reconstructs non-stationary information based on the distribution characteristics of historical sequences, making the distribution of the predicted sequence closer to that of the historical sequence rather than future sequence. However, as significant To validate the effectiveness of FDN and TDN, we conducted ablation studies comparing the predictive performance when only using FDN or TDN for non-stationary processing. It is important to note that when using FDN only for non-stationary processing, the ground truth of DPM pretraining comes from the non-stationary factors of FDN on future sequences. Correspondingly, wavelet bases will be set to nonlearnable parameters when we use FDN only. Meanwhile, for a fair comparison, both FDN and TDN in DDN utilize the same prediction model and share parameters, thereby avoiding the misconception that the superior performance of DDN stems from a larger parameter space in the prediction model.The results of the ablation studies, as shown in Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We have clearly outlined the experiment details in the Experiments and Appendix sections. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this work, we propose Dual-domain Dynamic Normalization (DDN), a novel method that dynamically captures non-stationary factors in time series forecasting, addressing sudden changes and distribution drifts in both time and frequency domains. Specifically, DDN employs sliding normalization in the time domain to eliminate and reconstruct non-stationary factors at a fine-grained level. In the frequency domain, it decomposes time series into high and low frequencies, effectively capturing rapid variations and sudden changes. As a model-agnostic auxiliary module, DDN significantly enhances the predictive performance of various forecasting models. Extensive experiments on seven real-world datasets validate the superiority of DDN, demonstrating its effectiveness in addressing distribution drift and improving the reliability of time series predictions.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the code link in the abstract. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have clearly outlined the links of these assets in the Appendix sections.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:• You should answer [Yes] , [No] , or [NA] .• [NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.• Please provide a short (1-2 sentence) justification right after your answer (even for NA).The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.IMPORTANT, please:• Delete this instruction block, but keep the section heading "NeurIPS paper checklist",• Keep the checklist subsection headings, questions/answers and guidelines below.• Do not modify the questions and only use the provided macros for your answers.Our baseline methods are described as follows:• Autoformer [20] is a transformer-based approach that adopts a decomposition strategy to learn complex temporal patterns in long-term prediction scenarios, decomposing time series into trend, cycle, and seasonal components, reflecting the long-term and seasonal aspects of the sequence data, respectively. The source code is available at https://github.com/ thuml/Autoformer. • FEDformer [19] Related reversible normalization methods are described in table 7, and they are described as follows:• RevIN [11] introduces a data normalization method that addresses the limitations of simply eliminating non-stationary information, which can result in the loss of valuable data that the model needs to learn effectively. Unlike traditional methods that may lead to the model's inability to capture these critical non-stationary factors, this work proposes, for the first time, an explicit restoration of non-stationary information after the model's output. This ensures that while the model can learn without being affected by data drift, it also retains the essential non-stationary information. The source code is available at https: //github.com/ts-kim/RevIN. • NST [13] unlikes traditional time series forecasting methods that reduce non-stationarity by stabilizing the original data, this approach contradicts the importance of predicting sudden events in time series forecasting and overlooks the prevalence of non-stationary data in real-world scenarios, ultimately leading to overly stabilized modeling and prediction.To address this, this paper proposes a novel network architecture composed of sequence stabilization and inverse stabilization attention mechanisms. The source code is available at https://github.com/thuml/Nonstationary_Transformers. • Dish-TS [12] notes that existing work on distribution shift in time series is often limited by distribution quantization and tends to overlook the potential distribution shift between the look back window and the horizon. To address this, this paper proposes using an MLP-based network to predict mean and standard deviation separately for the look back and horizon windows. The source code is available at https://github.com/weifantt/Dish-TS.• SAN [14] indicates previous efforts on addressing non-stationarity have attempted to reduce it through normalization techniques. However, these methods typically overlook the distributional differences between input sequences and horizon sequences, assuming that all time points within the same instance share identical statistical properties. This overly idealistic approach can lead to suboptimal improvements. To address this issue, this paper introduces a novel slice-level adaptive normalization method. The source code is available at https://github.com/icantnamemyself/SAN.Although NST [13] claims that non-stationary information can enhance feature diversity, previous methods have suffered from over-stationarization and inadequate consideration of non-stationary information utilization. However, it is noteworthy that, upon reviewing the current implementation Point Level Prediction (Ours) Table 7 : Comparative overview of non-stationary processing techniques in time series forecasting. "Granularity" and "Estimation" denote the normalization fineness and the prediction derivation method, respectively. of the non-stationary transformer, the designed non-stationary information extraction module and the corresponding integration of this information into intermediate feature learning can essentially be seen as a mode that employs a distribution prediction network to estimate future non-stationary information and learns through the prediction network. Figure 5 presents a visual comparison of the effects with and without the integration of non-stationary information extraction. It is evident that the non-stationary information extraction and integration module fundamentally enables more accurate reconstruction of non-stationary information, such as mean value.The Discrete Wavelet Transform (DWT) offers a nuanced approach to signal analysis by decomposing a time series into distinct frequency bands at multiple resolutions. This method is particularly adept at pinpointing both frequency and temporal aspects of a signal, making it invaluable for analyzing non-stationary time series. Initially, the general DWT of a time series x(t) is expressed through the wavelet coefficients:EQUATIONwhere a and b denote the scaling and translation parameters, respectively, and ϕ is the mother wavelet function. Building upon this foundation, the DWT isolates the approximation coefficients (AC) and detail coefficients (DC), which capture distinct signal characteristics:EQUATIONwhere ϕ low (t) represents the low-pass filter and ϕ high (t) the high-pass filter. Together, these filters facilitate the separation of the input time series into x i ac and x i dc . The AC coefficients x i ac embody the low-frequency components that outline the overarching trends within the time series, whereas the DC coefficients x i dc encompass the high-frequency components, often associated with transient or noise elements in the signal.The Inverse Discrete Wavelet Transform (IDWT) is then utilized to reconstruct the signal from its wavelet coefficients. The IDWT is the process that combines the AC and DC to form the original signal or an approximation of it. The reconstruction using IDWT can be written as:EQUATIONwhere ϕ ac (a, t) and ϕ dc (b, t) are the reconstruction functions from the approximation and detail coefficients, respectively. The wavelet transform, with its ability to localize both frequency and time, provides a powerful framework for signal analysis, particularly for signals like time series that contain non-stationary elements. As illustrated in Figure 4 , we present visual comparisons of predictions between different reversible normalization methods on the Weather dataset using DLinear [27] . The look-back window L is 336, and the prediction length T is 192. As discussed in Section 4.2, Figures 6, 7 , and 8 highlight the necessity of a distribution change prediction module. Meanwhile, Figures 9 and 10 demonstrate the DDN method's superior capability in capturing details compared to the SAN method. This is particularly evident in the richer rapid changes present in its predictive output, allowing for dynamic adaptation and alignment with the rapidly changing data series in specific datasets.To further substantiate this point, we conducted a visual comparison on the relatively smooth Traffic dataset in Figure 11 , where these rapid changes also corresponded well with those in the original sequence, contrasting with the Weather dataset. This comparison underscores the enhanced detail and adaptability of the DDN method across different datasets. This section comprehensively compares DDN with existing reversible normalization methods. As shown in Table 9 , our method consistently achieves state-of-the-art performance across almost all datasets, with a slight under-performance on the ETTm1 dataset compared to SAN. This demonstrates the versatility and effectiveness of our approach.Although recent works have shown that the two-stage training strategy effectively enhances the training of distribution prediction models, this training overly relies on the distribution ground truth obtained through distribution assumptions, which often contain inaccuracies. To address this, we introduce a collaborative training strategy that adjusts the distribution prediction model using the MSE loss between the actual sequence and the prediction during training. As illustrated in table 10, collaborative training generally yields superior training outcomes, and its lower bound in performance is comparable to that of the two-stage training strategy. This indicates that collaborative training can better mitigate the errors introduced by distribution assumptions and improve the accuracy of distribution prediction models.Following the same settings as our main experiment, we present the univariate forecasting results of Autoformer [20] and FEDformer [19] across three datasets, including Weather, Electricity, and Traffic, in Table 11 . Similar to the multivariate forecasting results, DDN consistently enhances the performance of mainstream forecasting models in nearly all cases. It demonstrates that DDN is applicable in both multivariate time series forecasting and univariate forecasting.https://github.com/zhouhaoyi/ETDatasethttps://www.bgc-jena.mpg.de/wetter/https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014https://pems.dot.ca.gov/ 


Decision: The paper introduces a novel Dual-domain Dynamic Normalization (DDN) framework to address the challenges of non-stationary time series forecasting. By dynamically capturing distribution variations in both time and frequency domains, DDN can enhance the robustness and accuracy of forecasting models. The proposed method is validated through extensive experiments on benchmark datasets, outperforming existing state-of-the-art normalization techniques.

During the rebuttal period, reviewers raised concerns regarding hyperparameter analysis, effects of the way of decomposition methods, more comparisons with SOTA methods, look-back window selection, etc. The authors provided thorough responses with extra experiments, which satisfactorily addressed most of these concerns. While some minor weaknesses exist, they are not critical and can be addressed in the CR version. Given the experimental results and the overall positive feedback from the reviewers (with scores of 7, 7, 5, 5), I recommend accepting the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces a approach to improve the accuracy of time series forecasting by addressing the challenge of non-stationary data, where data distributions change rapidly over time. The authors propose a Dual-domain Dynamic Normalization (DDN) framework that captures distribution variations dynamically in both time and frequency domains using Discrete Wavelet Transform (DWT) and sliding normalization. This method enhances the robustness of forecasting models against non-stationary data, significantly outperforming existing normalization methods in extensive experiments on public benchmark datasets.
strength_weakness: strength_weakness: 1. **Lack of Hyperparameter Sensitivity Analysis**: The paper does not include a sensitivity analysis of the hyperparameters, such as the length of the sliding window.
2. **Insufficient Theoretical and Empirical Justification**: The paper primarily focuses on empirical results without providing a strong theoretical foundation for the proposed Dual-domain Dynamic Normalization (DDN) framework. Additionally, there is a lack of ablation studies or detailed experiments that demonstrate the specific contributions and effectiveness of each module within the DDN framework. This makes it challenging to understand the individual impact of each component and the underlying principles that contribute to the overall performance improvements.
3. **Inconsistent Look-back Window Selection**: The paper does not explain why different models use different look-back window sizes or how these variations impact the experimental results. The lack of rationale behind the choice of look-back windows and the absence of an analysis of their effects on model performance makes it difficult to assess the consistency and fairness of the comparisons.
questions: questions: 1. Could you provide a sensitivity analysis of the hyperparameters, such as the length and stride of the sliding window? Understanding how these parameters affect performance is crucial.
2. Could you share experimental results that illustrate the impact of different look-back window sizes on model performance? This would clarify how the choice of look-back windows influences the results.
3. Could you offer theoretical or empirical analyses to demonstrate the contributions and effectiveness of each module within the DDN framework? Detailed ablation studies or theoretical justifications would be very helpful.
limitations: limitations: N/A
paper_summary: paper_summary: The authors consider the data distribution variations for real-world data and then propose a novel dual-domain dynamic normalization. Unlike the previous methods work in time domain, the proposed method decompose time series into a linear combination of different frequencies, and dynamically capture distribution variations in both time and frequency domains. Besides, the proposed method can serve as a plug-in-play module, and thus can be easily incorporated into other forecasting models. Extensive experiments on public benchmark datasets under different forecasting models demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The proposed method tries to decompose the original time series by low and high-frequency components. However, how to decompose time series has less explored.
2. The proposed Dual-domain method mainly considers time series in both frequency and time domain normalization. However, the effect of domains has less explore. For example, we can also transform the time series into high-dimension latent space.
3. Compared with other normalizaiton methods, the advantages and disadvantages should be further discussed.
questions: questions: Please also see the Weaknesses.
1. What is effects of the way of decomposition methods?
2. What is effects of different domains, such as time/frequency/embedding space?
3. What are the advantages and disadvantages of the proposed methods.
limitations: limitations: The limitations have been discussed and there are no potential negative ethical and societal implications in this work.
paper_summary: paper_summary: This paper proposes a novel Dual-Domain Dynamic Normalization (DDN) method to address the non-stationary variations in real-world time series by operating in both the time and frequency domains. In the frequency domain, wavelet transform is employed to decompose the time series into high and low-frequency components to capture distributional changes across different frequency scales. In the time domain, sliding statistics are introduced to adapt to the rapid changes of non-stationary data. As a plug-and-play module, this method can be easily integrated into existing predictive models. Experimental results demonstrate that DDN significantly enhances the predictive performance of existing models.
strength_weakness: strength_weakness: 1. The paper suggests that using proper windows for low and high-frequency components, which can better capture distributional changes. However, this lacks comparative experiments. A comparison with the traditional approach of using a uniform large window for both high and low-frequency components would better demonstrate the superior ability of the proposed method in capturing dynamic changes.
2. The terminology introduced should be consistent with previous works for easier readability. While the concept of sliding statistics is recommended to replace with a more unified term "rolling statistic", as employed in the referenced paper [1].
3. Some symbols need correction. In Section 3.3, the mean symbols (μ_f^i、σ_f^i) of the mean and standard deviation sequences in the Frequency Domain Prediction part are inconsistent with Figure 3. Additionally, the symbols in Figure 3 appear more like the Time Domain Prediction. 
4. The proposed method's comparison is primarily focused on the current four mainstream models. It is recommended to include more baseline models for comparison, such as Transformer and PatchTST, in the experimental section.

[1] Eric Zivot, Jiahui Wang, Eric Zivot, and Jiahui Wang. Rolling analysis of time series. Modeling financial time series with S-Plus®, pages 299–346, 2003. 2
questions: questions: 1. What is effects of window size for low and high-frequency components?
2. Does the work also work for other SOTA methods, like PatchTST?
limitations: limitations: Please refer to the weaknesses above.
paper_summary: paper_summary: This paper proposes a novel Dual-domain Dynamic Normalization (DDN) method to dynamically capture distribution variations in both time and frequency domains, leveraging wavelet transform to handle time-varying periods. DDN eliminates non-stationarity in time series through normalization within a sliding window and can be easily integrated into various forecasting models as a plug-and-play module. Extensive experiments on public benchmark datasets demonstrate DDN's superior performance over existing normalization methods. The code will be made available after the review process.
strength_weakness: strength_weakness: 1. The writing quality and presentation require necessary improvements. I have identified some obvious typos and errors upon a quick browse of the article:

- Line 119: The period should be replaced by a comma, while the comma should be replaced by a period.

- Line 167: The period should be replaced by a comma.

- Equation 10 and Equation 11: The notation below "argmin" should define the domain or the set of possible values for the variables being considered for minimization. Additionally, Equation 10 should be followed by a comma, as should Equation 11.


2. It is challenging to discern the distinction of their method from Figure 1, as the resulting outcomes of the existing manipulation and the proposed manipulation appear very similar. The authors should consider a better presentation to highlight the differences between their method and existing ones.


3. The description of some key concepts needs refinement to reduce unclarity and ambiguity. The authors challenge previous studies for “a fixed window” as stated in line 37. Before reviewing the entire method, this phrase might be misunderstood as referring to “a fixed window size,” suggesting that the primary motivation of this study is to adaptively adjust the window size. The subsequent expression “time-varying period” also leads to potential confusion. The authors should carefully consider how to present their core ideas with greater accuracy.


4. The implementation and core idea of SlideNorm bear significant resemblance to a method proposed by an existing study [1], necessitating a comparison and clarification of its distinction.

[1] J. Deng et al., "Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting," in IEEE Transactions on Knowledge and Data Engineering, doi: 10.1109/TKDE.2024.3371931.


5. The realm of probabilistic time series forecasting also targets the same task of predicting the distribution of future data with DPM. Demonstrating that DPM surpasses some representative probabilistic forecasting models would make this study more solid and convincing. Furthermore, the authors should justify that the improvements achieved by their method arise from the estimation of data distribution rather than the combined power of two deep learning models.


6. The evaluation is somewhat limited. Assessing effectiveness against different settings of hyper-parameters, such as sliding window size and look-back window size, can enhance this section.
questions: questions: Please address the weak points above.
limitations: limitations: Neither limitations nor broader impacts are discussed at the end of this paper. As

Comments: title: Good response
Reply to reviewer
Response to reviewer
Reminder for post-rebuttal feedback
Thanks for the response!
comment: Thank you for your valuable comments and  positive feedback for acknowledging our efforts in addressing your concerns. As suggested, we would add more experiments and analysis we provided in our revised manuscript.
Thanks for your response. My concerns have been addressed. I'd like to keep my score to 7 (accept).
Thank you for your response, my concerns have been resolved, and considering other reviewers' concerns as well, I have raised my score to 5. Also, it's better to include some of the analysis you provided above in the paper for readers to understand the method further.
Thanks a lot for your valuable comments and recognition for our work.
Thanks a lot for your valuable comments for our work.  If you have any additional question, we are happy to answer any additional question before the rebuttal ends.
Dear Reviewer i5Qt

We greatly appreciate your initial valuable comments. We hope that you could have a quick look at our responses to your concerns. It would be highly appreciated if you could kindly update the initial rating if your questions have been addressed. We are also happy to answer any additional questions before the rebuttal ends.

Best regards
Thanks for the detailed response. My concerns are well addressed. I acknowledge the novelty of using the inherent differences in frequency and the sliding statistics in dynamically capturing distributional changes over time.
I appreciate that the authors have addressed some of my concerns, and as a result, I am inclined to raise my score to 5.