CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting

Authors: jianrong  ding
zhanyu  liu
guanjie  zheng
haiming  jin
linghe  kong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-25 08:45:49

Abstract: \textit{Dataset condensation} is a newborn technique that generates a small dataset that can be used in training deep neural networks (DNNs) to lower storage and training costs. The objective of dataset condensation is to ensure that the model trained with the synthetic dataset can perform comparably to the model trained with full datasets. However, existing methods predominantly concentrate on classification tasks, posing challenges in their adaptation to time series forecasting (TS-forecasting). This challenge arises from disparities in the evaluation of synthetic data. In classification, the synthetic data is considered well-distilled if the model trained with the full dataset and the model trained with the synthetic dataset yield identical labels for the same input, regardless of variations in output logits distribution. Conversely, in TS-forecasting, the effectiveness of synthetic data distillation is determined by the distance between predictions of the two models. The synthetic data is deemed well-distilled only when all data points within the predictions are similar. Consequently, TS-forecasting has a more rigorous evaluation methodology compared to classification. To mitigate this gap, we theoretically analyze the optimization objective of dataset condensation for TS-forecasting and propose a new one-line plugin of dataset condensation for TS-forecasting designated as Dataset \textbf{Cond}ensation for \textbf{T}ime \textbf{S}eries \textbf{F}orecasting (CondTSF) based on our analysis. Plugging CondTSF into previous dataset condensation methods facilitates a reduction in the distance between the predictions of the model trained with the full dataset and the model trained with the synthetic dataset, thereby enhancing performance. We conduct extensive experiments on eight commonly used time series datasets. CondTSF consistently improves the performance of all previous dataset condensation methods across all datasets, particularly at low condensing ratios.

Main paper: 
 introduction: 
 Dataset condensation is a strategy for mitigating the computational demands of training large models on extensive datasets. It is pointed out by previous works [15, 32] that building foundation models [14, 10, 6, 35, 2, 44] on time series forecasting (TS-forecasting) have become a hot topic. However, fine-tuning these large models using full time series datasets can entail considerable computational overhead. Hence, the employment of dataset condensation techniques becomes imperative. In recent years, various methods have been proposed in the field of dataset condensation, such as matchingbased methods [51, 49, 3, 20, 38, 7, 5, 41, 50, 52, 36] and kernel methods [33, 55] . To date, dataset condensation methods have achieved success in classification tasks, including image classification [8, 11, 22] , graph classification [17, 16, 23, 43, 25, 9, 27] and time series classification [26] . However, directly applying these dataset condensation methods designed for classification to the domain of time series forecasting (TS-forecasting) results in performance degradation. The objective of dataset condensation is to generate a synthetic dataset so that when the model M s trained with the synthetic dataset and the model M f trained with the full dataset are given identical input, the two models output similar predictions. However, the concept of similar prediction differs between classification and TS-forecasting. In classification, as shown in Fig. 1(a) , predictions are considered similar if M s and M f assign the same class label, irrespective of differences in the distribution of output logits. Conversely, in TS-forecasting, as illustrated in Fig. 1 (b), the similarity of predictions from M s and M f is indicated by the mean squared distance of the predictions. The predictions are deemed similar only when all data points within the predictions are similar. This distinction in evaluation indicates TS-forecasting imposes more stringent criteria in discerning similar predictions compared to classification. It poses a challenge that previous dataset condensation methods based on classification fail to provide adequate assurance for the similarity between predictions of M s and M f within the realm of TS-forecasting.To mitigate the gap, we propose a novel one-line dataset condensation plugin designed specifically for TS-forecasting called Condensation for Time Series Forecasting (CondTSF) based on our theoretical analysis. We first formulate the optimization objective of dataset condensation for TSforecasting. Then we transform the original optimization objective into minimizing the distance between predictions of M s and M f . Furthermore, to minimize the distance between predictions of M s and M f , we decompose the task into minimizing two terms, namely gradient term and value term. We theoretically prove that plugging CondTSF into previous methods can minimize the value term and gradient term synchronously. Therefore, CondTSF serves as an effective plugin to boost the performance of dataset condensation for TS-forecasting. As depicted in Fig. 1(c) , plugging CondTSF into previous methods yields a significant enhancement in performance.In short, our contributions can be summarized as follows.• To the best of our knowledge, we are the first to explore dataset condensation for TS-forecasting.We conduct a theoretical analysis of the optimization objective of dataset condensation for TS-forecasting, breaking it down into two optimizable terms to facilitate improved optimization.• Leveraging insights from our theoretical analysis of TS-forecasting, we propose a simple yet effective dataset condensation plugin CondTSF. Plugging CondTSF into existing methods enables synchronous optimization of the two terms, leading to performance enhancement.• We conduct extensive experiments on eight widely used time series datasets to prove the effectiveness of CondTSF. CondTSF notably improves the performance of all previous dataset condensation methods across all datasets, particularly under low condensing ratios. 
related works: 
 Time Series Forecasting: Time series forecasting (TS-forecating) is the task of using historical, time-stamped data to predict future values. Previous works utilize different methods to achieve better performance. These models can be mainly categorized into 3 types. (1) Transformer-based Models: Transformer [40] have shown great success in natural language processing, and models based on transformers [53, 42, 24, 54] emerged in TS-forecasting fields.(2) MLP-based Models: Efforts to use MLP-based models have been put into TS-forecasting in recent years [47] since DLinear [45] triumph transformer-based models with a simple MLP structure.(3) Patch-based Models: These models [34, 48, 28, 29] focused on learning representation cross patches instead of learning attention at each time point. Therefore they used a patching strategy before feeding the data to transfomers.Dataset Condensation: Dataset condensation is a task that aims at distilling a large dataset into a smaller one so that when a model is trained on the small synthetic dataset and the full dataset separately, the testing performances of the trained models are similar. Previous works related to dataset condensation can be divided into 3 classes below. (1) Coreset Selecting Methods: These methods aim at selecting data with representative features from source dataset to construct a synthetic dataset [1, 4, 12, 37, 39] .(2) Matching-based Methods: These methods aim at minimizing a specific metric surrogate model learned from source dataset and synthetic dataset. The defined metrics are different, including gradient [51, 18, 46] , features from the same class [41] , distribution of synthetic data [50, 52] and training trajectories [3, 5, 7, 11, 8] .(3) Kernel-based Methods: These methods aim at obtaining a closed-form solution for the optimization problem utilizing kernel ridge-regression [20, 33] .In this way, the bi-level optimization problem of dataset condensation is reduced to a single-level optimization problem. Based on these results, the following works have made significant progress in different areas, including decreasing training cost and time [55] , improving performance [30, 31] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: As shown in Sec.6, we discuss the limitations of this work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Dataset Condensation for TS-forecasting Given a time series dataset, we split the dataset into a train set and a test set. In this paper, we denote the train set as f and the test set as x. We denote the synthetic dataset as s. The synthetic dataset s is a small dataset distilled from the full train set f . Train set f , test set x, and synthetic dataset s are all vectors. We employ M θ as a neural network parameterized by θ. Without losing generality, we suppose the model M θ is using historical sequence x t:t+m with length m to predict future sequence x t+m:t+m+n with length n. Given the test set x, we formulate the test error of M θ as the error between the prediction of M θ on test input x t:t+m and the test label x t+m:t+m+n , as shown in Eq.1.L Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
method: 
 Since test set x is not available during the dataset condensation process, the original optimization objective for dataset condensation in Eq.2 is non-optimizable. To mitigate this gap, in the following sections, we transform the non-optimizable objective into two distinct optimizable terms. Then we develop methods to optimize the two terms, thereby indirectly optimizing the original objective.In this section, we decompose the optimization objective of dataset condensation in Eq.2 into two optimizable terms for better optimization. In the testing process, the initial testing model parameter Meanwhile, there is a non-optimizable error ϵ between the prediction of model M θ f,test and the true label from the test dataset, which is formulated in Eq.4.EQUATIONThen we decompose the upper bound of L test (M θs,test , x) into two terms, as shown in Thm.1. We utilize Taylor Expansion in the proof of Thm. To prove Thm.1, we use linear models for further analysis since linear models can be both effective and efficient in TS-forecasting [45] . Given a linear model M θ (x) = θx, its second and higher order gradient is zero. Therefore first-order Taylor Expansion is sufficient to obtain the accurate prediction of the model. Meanwhile, if M θ is a non-linear model, we ignore the higher-order terms of Taylor Expansion. We prove Thm. We develop a method to optimize gradient term in this section. Given a linear model M θ (x) = θx, its gradient on input is ∇M θ (x) = θ ⊤ . It indicates that the gradient of a linear model on input is the parameter of the model. We apply Cauchy-Schwarz Inequality to the gradient term and get its upper bound. We reformulate the gradient term and get its upper bound as shown in Eq.6.||(∇M θs,test (sEQUATIONFigure 3 : Given the same synthetic data as input, all expert models trained on the full train set f provide similar predictions. The initial parameters of the models are sampled from the same distribution P θ . The visualization of this figure utilized MDS [19] algorithm for dimension reduction.Since test data x t:t+m is not available during the dataset condensation process, the distance between synthetic data and test data ||x t:t+m -s t ′ :t ′ +m || 2 is not optimizable. Therefore we only need to optimize the distance between parameters ||θ s,test -θ f,test || 2 . All previous dataset condensation methods based on parameter matching can minimize this distance. Here we utilize MTT [3] as an example to clarify the optimization process. The optimization objective of trajectory matching isEQUATIONHowever, since θ s,test and θ f,test are trained from testing initial parameter θ 0,test ∼ P θ , they are not available during dataset condensation process. Therefore, in practice, we sample θ 0 0 , . . . , θ k 0 ∼ P θ as initial parameters during dataset condensation process. The initial parameters are trained on synthetic dataset s and full train set f respectively to get θ 0 s , . . . , θ k s and θ 0 f , . . . , θ k f . Then we substitute θ s,test , θ f,test and θ 0,test in Eq.7 with parameters sampled in dataset condensation, making the optimization objective optimizable. The practical optimization objective is shown in Eq.8.EQUATIONIn practice, θ 0 0 , . . . , θ k 0 and θ 0 f , . . . , θ k f are sampled, trained, and stored in a parameter buffer before dataset condensation process. It can be concluded that using trajectory matching methods is intuitively minimizing the distance between θ i s and θ i f for all initial parameters θ i 0 ∼ P θ . By minimizing the upper bound of the gradient term, trajectory matching methods indirectly optimize the gradient term.We develop an optimization objective to optimize the value term in this section. Since θ f,test is trained from θ 0,test , it's unavailable in dataset condensation process. To mitigate this gap, we prove that although θ f,test is unavailable in dataset condensation process, its prediction M θ f,test (s t ′ :t ′ +m ) is still available. To prove this statement, we sample initial model parameters θ 0 0 , . . . , θ k 0 from P θ . Then θ 0 0 , . . . , θ k 0 are all trained with the same full train set f . After training, we get parameters θ 0 f , . . . , θ k f . It is observed that models M θ 0 f , . . . , M θ k f predict similarly given arbitrary synthetic data s t ′ :t ′ +m as input.Since initial testing parameter θ 0,test is also sampled from the same distribution P θ and θ f,test is trained from θ 0,test using the same full train set f , the prediction of M θ f ,test is similar to predictions of an arbitrary expert model M θ i f . The conclusion is formulated in Eq.9. if e mod G ̸ = 0 then 4:EQUATIONSample(θ i 0 , θ i f ) from {(θ 0 , θ f )} k 5:Initialize student parameter θ0 ← θ i 0 6:for each trajectory matching epoch j in range N do /*MTT [3] trajectory matching*/Train M θj with synthetic data 8:θj+1 ← θj -α∇L(M θj (s t:t+m ), s t+m:t+m+n ) for all synthetic data 9: Experiments have proved Eq.9 in Fig. 3 . As shown in Fig. 3 , for each synthetic data input s t ′ :t ′ +m (orange points), the predictions of corresponding expert models (yellow and blue points) are similar. Therefore, although θ f,test is unavailable in the dataset condensation process, its prediction M θ f,test (s t ′ :t ′ +m ) can be obtained using the prediction of an arbitrary expert model M θ i f (s t ′ :t ′ +m ). Now we reformulate the value term and transform it into a practical optimization objective. Firstly, We formulate the upper bound of the value term as shown in Thm.2. Theorem 2. The upper bound of the value term can be formulated as suchL param ← ||θ i f -θN || 2 /||θ i f -θ i 0 || 2||M θs,test (s t ′ :t ′ +m )-M θ f,test (s t ′ :t ′ +m )|| 2 ≤ 2• t ′ ||M θ f,test (s t ′ :t ′ +m )-s t ′ +m:t ′ +m+n || 2 (10)We prove Thm.2 by utilizing the triangular inequality and the prediction optimality of θ s,test on synthetic data s. The complete proof is in App.A.2. Accodring to Thm.2, we obtain an optimizable upper bound of the value term. Therefore the optimization objective for the value term can be naturally defined as minimizing the upper bound of the value term, as shown in Eq.11.EQUATIONAccording to Thm.2, label error L label is the upper bound of the value term. Therefore, by minimizing the upper bound of the value term, the value term is indirectly minimized.In this section, we develop a one-line plugin called CondTSF to minimize the label error L label in Eq.11 so that the value term can be optimized. CondTSF is a lightweight one-line plugin, no backpropagation or gradient is required during the update. CondTSF utilizes a simple yet effective additive method to iteratively update the synthetic data s and minimize the label error L label . In TS-forecasting, when generating training data, the data is usually sampled overlap from the dataset. Inspired by the overlap property, we utilize an additive method in CondTSF to gradually update the synthetic data to avoid vibrations. In the i th update iteration, CondTSF uses the prediction of expert model M f,test (s t ′ :t ′ +m ) to update synthetic label s t ′ +m:t ′ +m+n . The update process is shown in Eq.12. s (i+1) where 0 < β < 1 is the update ratio of this additive update method. This additive update process lowers the label error L label of s in each iteration exponentially, which can be formulated asEQUATIONEQUATIONSince the update ratio has a value of 0 < β < 1, CondTSF lowers the label error L label exponentially in each update iteration and solves the optimization problem for the value term. As a plugin module, CondTSF is used to update once for every G iteration of parameter matching methods. In this way, the gradient term and the value term can be optimized synchronously. The algorithm is shown in Alg.1. We also formulate the complete condensation process using CondTSF, as shown in Fig. 2 . 
experiment settings: 
 Dataset Settings: The efficacy of dataset condensation methods is assessed across eight time series datasets. For all datasets, the model is set to be using 24 steps of data to forecast 24 steps of data. We Model Settings: We plug CondTSF into existing dataset condensation models based on parameter matching, including DC [51] , MTT [3] , PP [21] , TESLA [5] , FTD [7] and DATM [11] to prove the effectiveness of CondTSF. We also conduct experiments on non-parameter-matching based methods, including DM [50] , IDM [52] , KIP [33] , FRePo [55] to prove that optimizing value term only also helps boost the performance. The experiment setting and results are shown in App.E. We use DLinear [45] as the expert model to perform dataset condensation since DLinear is a linear model.The source dataset is first divided into a train set and a test set. All synthetic data is initialized by randomly sampling data from the train set. After a synthetic dataset is finished distilling, it is used to train another five models. After the five models are trained, they are tested on the test set. Their average mean absolute error (MAE) and mean square error (MSE) are recorded. We repeat the process above five times and report the average and standard deviation. While testing the generalization ability of the dataset condensation methods, DLinear [45] is used as the expert model to perform dataset condensation. Meanwhile, MLP, LSTM [13] , and CNN are used as test models when testing the generalization ability of the dataset condensation methods. Implementation Details: As a plugin module, we test CondTSF with all previous methods. Each synthetic dataset is optimized using a standard training process according to the chosen backbone model. CondTSF is set to update every 3 epochs and the additive update ratio β is set to be 0.01. All the experiments are carried out on an NVIDIA RTX 3080Ti.Single Architecture Performance: The results are summarized in Table .1. For each backbone method, the first line shows the performance of the backbone model, the second line shows the performance of a backbone model with CondTSF, and the third line shows the percentage of reduction in MAE and MSE after CondTSF is applied. There's a considerable reduction in error for all backbone models. The results suggest that CondTSF is effective in optimizing the value term and enhancing the performance in dataset condensation for TS-forecasting. However, using CondTSF on DC [51] is not as effective as other methods. The reason is that instead of directly matching parameters, DC matches the gradient of parameters on loss in each iteration. Indirectly matching gradient leads to accumulating errors in parameters, making DC unable to lower parameter error as effectively as directly matching parameters. Therefore CondTSF is not effective enough when applied to DC [51] .Cross Architecture Performance: We also conduct experiments to evaluate the cross-architecture performance of dataset condensation methods. The results are summarized in Table .3. We test all models on all datasets with MLP, LSTM [13] , and CNN as test models. All synthetic data is distilled using DLinear [45] model as experts. We use MTT [3] as the backbone for CondTSF. We observe that CondTSF based on MTT outperformed all other previous models.Test Performance and Errors: We conduct experiments on ExchangeRate dataset with MTT [3] and MTT+CondTSF. As shown in Fig. 4 , trajectory of parameter error||θ f -θs|| 2||θ f -θ0|| 2 , label error L label and test error L test through the distillation process are presented. Regarding the parameter error corresponding to the gradient term, both MTT and MTT+CondTSF converge quickly, suggesting that the incorporation of CondTSF doesn't impact parameter alignment. As for the label error corresponding to the value term, since the initial synthetic data s is randomly sampled from the train set f and the expert model is trained by the train set f , the label error of s is small at the beginning. However, the utilization of MTT results in an elevation of label error, whereas employing CondTSF effectively mitigates this increase in label error. During the test, MTT+CondTSF notably outperforms MTT by concurrently optimizing both the value term and the gradient term. 
limitations: 
 The limitation of this work is that we use linear models in our analysis so that the gradient of a model on input is the parameter of the model. Therefore, only linear models like DLinear [45] are solid enough to be an expert model for dataset condensation. The analysis no longer holds when it comes to more complicated models. However, experiments in App.D and App.E show that CondTSF is also effective with non-parameter-matching methods and non-linear models, which merits further exploration. 
conclusion: 
 In this study, we provide abundant proof that previous dataset condensation methods based on classification are not suitable for dataset condensation for TS-forecasting. We elucidate that these earlier methods, predominantly focused on classification tasks, only address a portion of the optimization objective pertinent to TS-forecasting. To address this issue, we propose a plugin module called CondTSF that can collaborate with parameter matching based dataset condensation methods. CondTSF optimizes the optimization objective that previous methods have neglected and boosts the performance of dataset condensation methods on TS-forecasting. We conduct experiments on eight widely used time series datasets and prove the effectiveness of our proof and method. CondTSF consistently enhances the performance of all previous techniques across all datasets, substantiating its effectiveness in improving dataset condensation outcomes for TS-forecasting applications. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: As shown in Sec.5, the datasets used in this work are all widely used and open-source. There's no risk of releasing unsafe data.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: As shown in Sec.5, we cite the papers that produced the code. The datasets used in this work are all widely used and open-source.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We provide documents with the code.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Complete Proof for Theorem 1 Theorem 1. Given arbitrary synthetic data s t ′ :t ′ +m , the upper bound of the optimization objective of dataset condensation L test (M θs,test , x) can be formulated as suchL test (M θs,test , x) ≤ t ||ϵ|| 2 + ||M θs,test (s t ′ :t ′ +m ) -M θ f,test (s t ′ :t ′ +m )|| 2 Value Term + ||(∇M θs,test (s t ′ :t ′ +m ) -∇M θ f,test (s t ′ :t ′ +m )) ⊤ (x t:t+m -s t ′ :t ′ +m )|| 2Gradient Term (14) Proof. Replacing the true label x t+m:t+m+n in L test (M θs,test , x) with Eq.4, the optimization objective of dataset condensation for TS-forecasting is reformulated as the distance between the predictions of M θs,test and M θ f,test given the same test input. Then the triangular inequality of norm functions is used and the original optimization objective can be transformed to its upper bound, as shown in Eq.15.EQUATIONIn Eq.15, we prove that minimizing the distance between M θs,test (x t:t+m ) and M θ f,test (x t:t+m ) is equivalent to minimizing the upper bound of the original optimization objective. Then we decompose the distance between predictions of M θs,test and M θ f,test into two optimizable terms for better optimization. We use linear models for further analysis since linear models can be both effective and efficient in TS-forecasting tasks [45] . Given a linear model M θ (x) = θx, its second and higher order gradient is zero, i.e. ∇ k M θ (x) = 0, ∀k ≥ 2. Therefore, first-order Taylor Expansion can be utilized to get the prediction of the model M θ on test data x t:t+m using the prediction and gradient of the model M θ on arbitrary synthetic data s t ′ :t ′ +m . The process is formulated in Eq.16. Meanwhile, it is worth mentioning that if M θ is a non-linear model, then the second and higher-order terms of the Taylor Expansion are ignored in Eq.16.EQUATIONThen we expand Eq.15 with Taylor expansion. After that, the triangular inequality of norm functions is used to get its upper bound. In the meantime, by applying the triangular inequality, the optimization objective can be decomposed into two optimizable terms.L test (M θs,test , x) ≤ t ||M θs,test (x t:t+m ) -M θ f,test (x t:t+m )|| 2 + ||ϵ|| 2 = t ||ϵ|| 2 + ||M θs,test (s t ′ :t ′ +m ) + ∇M θs,test (s t ′ :t ′ +m ) ⊤ (x t:t+m -s t ′ :t ′ +m ) -M θ f,test (s t ′ :t ′ +m ) -∇M θ f,test (s t ′ :t ′ +m ) ⊤ (x t:t+m -s t ′ :t ′ +m )|| 2 = t ||ϵ|| 2 + ||M θs,test (s t ′ :t ′ +m ) -M θ f,test (s t ′ :t ′ +m ) + (∇M θs,test (s t ′ :t ′ +m ) -∇M θ f,test (s t ′ :t ′ +m )) ⊤ (x t:t+m -s t ′ :t ′ +m )|| 2 ≤ t ||ϵ|| 2 + ||M θs,test (s t ′ :t ′ +m ) -M θ f,test (s t ′ :t ′ +m )|| 2 Value Term + ||(∇M θs,test (s t ′ :t ′ +m ) -∇M θ f,test (s t ′ :t ′ +m )) ⊤ (x t:t+m -s t ′ :t ′ +m )|| 2Gradient Term (17) Therefore Thm.1 is proved.Theorem 2. The upper bound of the value term can be formulated as such||M θs,test (s t ′ :t ′ +m )-M θ f,test (s t ′ :t ′ +m )|| 2 ≤ 2• t ′ ||M θ f,test (s t ′ :t ′ +m )-s t ′ +m:t ′ +m+n || 2 (18)Proof. We first use triangular inequality and the non-negativity of norm functions to get the upper bound of the value term. The process is shown in Eq. 19 .||M θs,test (s t ′ :t ′ +m ) -M θ f,test (s t ′ :t ′ +m )|| 2 (Value Term) = ||M θs,test (s t ′ :t ′ +m ) -s t ′ +m:t ′ +m+n + s t ′ +m:t ′ +m+n -M θ f,test (s t ′ :t ′ +m )|| 2 ≤ ||M θs,test (s t ′ :t ′ +m ) -s t ′ +m:t ′ +m+n || 2 + ||s t ′ +m:t ′ +m+n -M θ f,test (s t ′ :t ′ +m )|| 2 ≤ t ′ ||M θs,test (s t ′ :t ′ +m ) -s t ′ +m:t ′ +m+n || 2 + t ′ ||s t ′ +m:t ′ +m+n -M θ f,test (s t ′ :t ′ +m )|| 2 (19)For further analysis, we need to step back and formulate the training process of θ s to derive an inequality. By doing dataset condensation, a synthetic dataset s is obtained. Then we formulate the training process of θ s,test on synthetic data s as minimizing the prediction error on s. The training process is formulated in Eq.20.EQUATIONNow we can derive an inequality. We denote s as the synthetic dataset obtained by dataset condensation. We denote M θs,test as the model that is trained on s as shown Eq. We further explore the performance of CondTSF with different distill ratios and compare the results with previous matching-based methods.We distill the dataset into a synthetic dataset with a flexible length for each dataset. The information on condensation in Table .4. The performance is shown in Table .5. We distill the dataset into a synthetic dataset with a flexible length for each dataset. Each synthetic dataset is 3 times larger than the synthetic data in Table .4. The information on condensation is shown in Table .6. The performance is shown in Table .7. We observe that CondTSF consistently improves the performance of backbone models with all condensing ratios, suggesting the effectiveness of CondTSF with different condensing ratios.We distill the dataset using the standard condensing ratio. The information on condensation is shown in Table .4. We conduct experiments on CondTSF with non-parameter-matching based methods. We use DM [50] , IDM [52] , KIP [33] , FRePo [55] as backbone methods. The performance is shown in Table .8.Results show that using CondTSF to optimize only one of the two optimizable terms can also boost the performance. We distill the dataset using the standard condensing ratio. The information on condensation is shown in Table .4. We conduct experiments on distilling dataset with non-linear expert models. We use MTT [3] , TESLA [5] , and DATM [11] as backbone methods. The performance of using a CNN as the expert model is shown in Table .9, and the performance of using a 3-layer-MLP as the expert model is shown in Table .10.Results show that CondTSF is also effective with non-linear expert models. We distill the dataset using the one-shot condensing ratio. The information on condensation is shown in Table .2. We conduct experiments on distilling dataset with CondTSF and a low pass filter respectively. We use DLinear [45] as the expert model. The result is shown in Table .11.Results indicate that using CondTSF is significantly more effective than using a low pass filter to smooth the distilled data. We compare the changing trajectories of test error during the dataset condensation process. Since MTT has been proven to be a suitable backbone for CondTSF, we conduct experiments on different methods of plugging CondTSF into MTT. We utilize the standard condensing ratio as shown in Table .4.Test error is calculated as such. After the synthetic data has been distilled, it is used to train 5 randomly initialized testing models. After training with the synthetic dataset, the models are tested on the test set sampled from the source dataset. MAE error is reported in the figures below. We observe that CondTSF consistently reduces the testing error with different utilization gaps.We also conduct experiments on label error and test error. We visualize the trajectory of label error L label and test error through the distillation process. The results are shown in Fig. 8 .• Model 1: Use MTT to distill for 200 epochs.• Model 2: Use MTT to distill for 160 epochs and then use CondTSF to update for 40 epochs.As shown in Fig. 8 , it can be observed that using MTT [3] leads to an increase in label error L label . While applying CondTSF effectively lowers the label error in the last 40 epochs, and therefore enhancing the performance. We test CondTSF with different update gaps G and additive update ratios β. We utilize the standard distill ratio as shown in Table .4. Our observations indicate that CondTSF displays a notable degree of robustness concerning these parameters. Specifically, the effectiveness of CondTSF persists when the update gap G is moderately sized and additive update ratio β is not excessively small. We provide some visualization of synthetic data distilled by MTT [3] and MTT+CondTSF on all datasets. It is observed that the synthetic dataset distilled with CondTSF is smoother than the ones without CondTSF. Smoother data indicates more generalized features and therefore helps boost the performance. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper introduces a simple and effective method for condensing time series datasets, which can be implemented as a one-line plugin for time series forecasting models corresponding to an additional regularization term.
The paper develops a theory showing that this regularization term allows for upper bounding the objective of the dataset condensation problem.
The resulting algorithm is then empirically validated on 8 time series datasets which confirm the effectiveness of the proposed method.
The reviewers generally agree that the paper is clearly organized and structured. Other strengths of the paper include the simplicity of the resulting method, the theoretical analysis, and the strong performance in experiments. However, some reviewers raised concerns about the ambiguous notation, the lack of motivation and significance of addressing dataset condensation for time-series, and the limited evaluation.
The rebuttals satisfactorily addressed most of these concerns, including adding additional experiments to verify that the methods is applicable to nonlinear models, and including comparisons against more baselines like target smoothing on synthetic data.
Overall, the reviewers found the paper to be a solid enough contribution to warrant acceptance. Reviewers however recommend that acceptance be conditional on the additional results in the rebuttals and notation and presentation improvements be included and prominently showcased in the camera-ready version of the paper.

Reviews: review: paper_summary: paper_summary: This paper provides a simple fix on the parameter-matching based dataset condensation methods for time series forecasting tasks. Besides matching the parameters of the model $M_f$ trained on the full training set and $M_s$ trained on the smaller synthetic dataset, it additionally induces the $M_f$ to perform well in the synthetic data too. It theoretically shows that the additional regularization contributes to an surrogate upper bound of the original objective, and empirically proves that the fix help improve various dataset condensation methods.
strength_weakness: strength_weakness: A few details are not clear. See questions.
questions: questions: 1. In line 162, the author claims that it is "observed" that models initialized with different samples of parameter values predict similarly after trained on the full dataset given arbitrary input. Is it supported by any evidences, either theoretically or empirically?
2. Some clarity on optimizing gradient term can be provided. For example, when the parameters are matched and $M_s$ are well fitted, why the label error of $M_f$ on $s$ will increase?
limitations: limitations: The limitation that the analysis only applies to linear models is discussed in the paper.
paper_summary: paper_summary: In this paper the authors study the question of dataset condensation / distillation in the context of time series forecasting. They propose a decomposition that could be used to empirically estimate bound the point-wise forecast MSE loss. Based on this decomposition the authors proposed a condensation plug in on top of existing data condensation method that updates the distilled dataset with the estimated full model forecast. Through empirical study they show that with this plug in the resulting condensed dataset leads to better models across different model classes and benchmark forecasting tasks.
strength_weakness: strength_weakness: The major weakness is the writing. It is hard to follow in its current shape, in particular the math description of the research question should be more rigorous. To name a few:
- What are the sample space of x and s? They now look like a single univariate time series. It is the case?
- What is t' in Eqn (5)? Is it summed over? If so is it double sum within sum_t?
- How is Eqn (7) and (8) optimized (also in Algo 1).
Similarly regarding writing, the paper does not motivate the research question well, and has not provided enough background details on its base distillation methods (Line 200 Model Settings), especially MTT(3). As a result it is hard to evaluate the novelty of the paper when the main idea is not clearly delivered.

Though mentioned as a limitation, the assumption of linear model trivializes the theories, which ties the proposed method to DLinear expert, and makes it hard to understand how the learning from the synthetic data generalizes beyond the expert's model class. There are similar simplifications, e.g. the empirical approximation as in Eqn (9) that's observational.
questions: questions: Besides the questions in the weakness:

- While the distill ratio in Table 2, e.g. condensing the datasets into one training example, can be used to show the extremity of the distillation process,  it is questionable what this one-shot learning means for the MLP, LSTM, CNN involved. Any insights, especially why to choose them over the few-shot study in, e.g. Appendix F (this one has visualization as well) to be in the main paper? 
- As now the expert model is more actively involved through the two decomposition steps to touch the synthetic data, how does the capacity of the expert impact the fidelity of the synthetic dataset versus the full one? What about at least one non linear expert just to get some insights?
- The method proposed here does not utilize any time series concepts. If so this is an application on multivariate regression with linear expert. What's the novelty beyond that for time series exclusively?
limitations: limitations: The authors addressed the limitations.
paper_summary: paper_summary: This paper explores dataset condensation, which generates a small dataset for training of deep neural networks, for time series forecasting. Specifically, it proposes a one-line dataset condensation plugin designed specifically for time series forecasting. It first formulates the optimization objective of dataset condensation for time series forecasting, and based on which a reformulation is proposed and further theoretical analysis is performed. Based on the proposed theoretical analysis, this paper proposes a simple data condensation plugin CondTSF. Experiments on 8 time series datasets are performed to demonstrate the effectiveness of the proposed CondTSF.
strength_weakness: strength_weakness: 1. Dataset condensation is a strategy specifically designed for training large dataset. However, the data size in time series is usually significantly smaller than that in NLP and CV. In my experience, the computational demand to deal with time series data may not be an urgent request in real-world time series applications. The topic discussed in this paper seems interesting given the current program of different foundation models, but I highly doubt it applicability in the field of time series. 
2.  A related problem is that how the time series foundation model performs remain unclear. Exploring the foundation model for time series may be a promising direction, but it is not widely accepted and deployed in real-world applications. In fact, considering the semantics of time series is highly dependent on the underlying applications, so far I have not observed any foundation model can provide good zero-shot learning capability.
questions: questions: Please refer to the weakness part.
limitations: limitations: Please refer to the weakness part.
paper_summary: paper_summary: In this paper, the authors propose a dataset distillation approach by decomposing the distillation loss into two terms, the value term and the gradient term. Then, they derive bounds for these two terms that enable more effective dataset distillation for time-series forecasting. Strong performance is demonstrated in the conducted experimental evaluation
strength_weakness: strength_weakness: - p4. l121. The authors do not explain why this non-optimizable $\epsilon$ error occurs. 

- l. 137 in principle we could directly optimize this gradient term without deriving any bound. Unless I am missing something, this is supported by DL frameworks (e.g., PyTorch). On the other hand, I understand that there might be computational benefits from deriving a bound that does not involve the gradients. However, authors should justify their choice (why we need to derive this bound) and if this is related to computational efficiency. Ideally, experiments should be provided to support their choice. 

- At various points there are implicit assumptions the effect of which is not appropriately discussed, e.g., Eq. (9) is essentially supported by an experimental result in a Figure and a verbal description. I do not (empirically) disagree with the conclusion, but I can see cases where this would not hold (e.g., in financial time-series forecasting, where using the same data we can have vastly different (non-linear) hypotheses). Whether this alters the conclusions/and or the method, is not appropriately discussed.

- The updating process in (12) essentially moves the targets towards the predictions of the model. This is a reasonable result that is well-known to work. However, the way this is applied is now very close to target smoothing/model distillation approaches that essentially do the same thing (also applied in the context of time series forecasting as far as I know). There is some literature on model distillation approaches (and/or self-distillation) in the context of time-series forecasting and/or regression, which essentially demonstrate that this kind of "smoothing" can indeed increase performance. Therefore, this raises some questions on whether the method should be compared experimentally (or at least theoretically) with such approaches.

- Even though some ablation studies were provided in the Appendix, I didn't manage to find perhaps the most important result: What is the impact of the gradient term and what is the impact of the value term? What would happen if we remove the L_param loss in l. 9 of Alg 1? What would happen if we removed the value term?

- Isn't the method also applicable in the case of regression tasks? If I am not missing anything, I think the same methodology without any change (apart from the input to the models) can be applied also to handle generic regression tasks that also expand over time-series forecasting.


Minor: l.147-152: I think $k$ is used before being introduced.
questions: questions: Please respond to the comments provided in the drawbacks. Overall, strong performance is demonstrated, but there are several aspects that should be improved.
limitations: limitations: Yes.

Comments: title: Plan of rewriting part of our paper
Thanks for the comments and raising the score
Thanks for the comments
comment: Dear reviewer TXkn,

Thank you for your comment and advice on improving the writing of our paper. We apologize for the ambiguity that exists in the current version. We will rewrite the following part of our paper.

1. We will clarify the source of the train set $f$, test set $x$, and synthetic dataset $s$ in Sec.3.
2. We will rewrite App.A1 and texts relate to Eq(5) to clarify the test error and the meaning of $t'$. We will also add an explanation of the effects when nonlinear models are used as expert models.
3. We will add a detailed introduction of MTT on how to optimize Eq(7) and Eq(8) in the appendix to clarify the core novelty of our paper.
4. We will add the complete results of using nonlinear models as expert models in the appendix to show that our method also works well with nonlinear experts.
5. We will add an explanation in Sec4.4 of how the additive method relates to the features of time series.

Meanwhile, we are eager for the opportunity to address any remaining concerns you might have before the discussion period ends. We sincerely appreciate your and other reviewers' dedication and time invested in reviewing our work.

Best,

Authors
Thank you for your rebuttal. I was expecting improvements on the writing quality of the paper. Would it be possible to cite in comment a few rewrites if any?
Dear reviewer s2gY,

We want to thank you for the valuable advice and for raising the score of our paper.

We will incorporate your suggestions, especially clarifying the terms, and adding the experiment results reported in the rebuttal.

Best,

Authors
Dear reviewer DVXT,

We want to thank you for your valuable advice and we will clarify the motivation and the pragmatic usage of dataset condensation for time series in the paper.

Meanwhile, we are eager for the opportunity to address any remaining concerns you might have before the discussion period concludes. We sincerely appreciate your and other reviewers' dedication and time invested in reviewing our work.

Best,

Authors
I would like to thank the authors for providing detailed responses to all of my questions, which were - to a significant degree - indeed addressed. Therefore, I am increasing my rating.
Thanks for the reviewer for the detailed response. I acknowledge I have read the rebuttal.