Title: CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dataset condensation

Time series forecasting
Abstract: \textit{Dataset condensation} is a newborn technique that generates a small dataset that can be used in training deep neural networks (DNNs) to lower storage and training costs. The objective of dataset condensation is to ensure that the model trained with the synthetic dataset can perform comparably to the model trained with full datasets. However, existing methods predominantly concentrate on classification tasks, posing challenges in their adaptation to time series forecasting (TS-forecasting). This challenge arises from disparities in the evaluation of synthetic data. In classification, the synthetic data is considered well-distilled if the model trained with the full dataset and the model trained with the synthetic dataset yield identical labels for the same input, regardless of variations in output logits distribution. Conversely, in TS-forecasting, the effectiveness of synthetic data distillation is determined by the distance between predictions of the two models. The synthetic data is deemed well-distilled only when all data points within the predictions are similar. Consequently, TS-forecasting has a more rigorous evaluation methodology compared to classification. To mitigate this gap, we theoretically analyze the optimization objective of dataset condensation for TS-forecasting and propose a new one-line plugin of dataset condensation for TS-forecasting designated as Dataset \textbf{Cond}ensation for \textbf{T}ime \textbf{S}eries \textbf{F}orecasting (CondTSF) based on our analysis. Plugging CondTSF into previous dataset condensation methods facilitates a reduction in the distance between the predictions of the model trained with the full dataset and the model trained with the synthetic dataset, thereby enhancing performance. We conduct extensive experiments on eight commonly used time series datasets. CondTSF consistently improves the performance of all previous dataset condensation methods across all datasets, particularly at low condensing ratios.

CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting

Abstract

Dataset condensation is a newborn technique that generates a small dataset that can be used in training deep neural networks (DNNs) to lower storage and training costs. The objective of dataset condensation is to ensure that the model trained with the synthetic dataset can perform comparably to the model trained with full datasets. However, existing methods predominantly concentrate on classification tasks, posing challenges in their adaptation to time series forecasting (TS-forecasting). This challenge arises from disparities in the evaluation of synthetic data. In classification, the synthetic data is considered well-distilled if the model trained with the full dataset and the model trained with the synthetic dataset yield identical labels for the same input, regardless of variations in output logits distribution. Conversely, in TS-forecasting, the effectiveness of synthetic data distillation is determined by the distance between predictions of the two models. The synthetic data is deemed welldistilled only when all data points within the predictions are similar. Consequently, TS-forecasting has a more rigorous evaluation methodology compared to classification. To mitigate this gap, we theoretically analyze the optimization objective of dataset condensation for TS-forecasting and propose a new one-line plugin of dataset condensation for TS-forecasting designated as Dataset Cond ensation for T ime S eries F orecasting (CondTSF) based on our analysis. Plugging CondTSF into previous dataset condensation methods facilitates a reduction in the distance between the predictions of the model trained with the full dataset and the model trained with the synthetic dataset, thereby enhancing performance. We conduct extensive experiments on eight commonly used time series datasets. CondTSF consistently improves the performance of all previous dataset condensation methods across all datasets, particularly at low condensing ratios.

1 Introduction

Dataset condensation is a strategy for mitigating the computational demands of training large models on extensive datasets. It is pointed out by previous works[15, 32] that building foundation models[14, 10, 6, 35, 2, 44] on time series forecasting (TS-forecasting) have become a hot topic. However, fine-tuning these large models using full time series datasets can entail considerable computational overhead. Hence, the employment of dataset condensation techniques becomes imperative. In recent years, various methods have been proposed in the field of dataset condensation, such as matchingbased methods[51, 49, 3, 20, 38, 7, 5, 41, 50, 52, 36] and kernel methods[33, 55]. To date, dataset condensation methods have achieved success in classification tasks, including image classification[8, 11, 22], graph classification[17, 16, 23, 43, 25, 9, 27] and time series classification[26].

Model trainedwith

synthetic dataset

Ms

1.0

0.5

0.0

Horse

0.5

0.05

0.25

0.2

Predicted label

Noerror

Test Error

0.6

0.5-

0.8

0.7

MTT

MTT+ CondTSF

Test Error

0.5

0.6

0.9

0.8

0.7

FTD + CondTSF

FTD

Model trainedwith

full dataset

Mf

0.5

0.8

Similarprediction

Good synthetic

↑

0.4

0.3

0.2

WA

0.2

0.3

0.4

0.0

Horse

0.15

Cat Rabbit Bird

0.02

0.03

data

0

Condensation Epochs

40

80

120

160

200

0

CondensationEpochs

40

08

120

160

200

(a)Evaluation of syntheticdataforimageclassification

0.8

KIP

1.0

FRePo

Test Data

Model trainedwith

synthetic dataset

Ms

0.05

0.250.2

Pointwise distance

Large error

Test Error

0.7

0.6

KIP+CondTSF

Test Error

80

0.7

0.9

FRePo + CondTSF

0.5

Dissimilar

0.4

0.5

m

Model trainedwith

full dataset

prediction

↑

0.3

0.4

0.3

Mf

Badsyntheticdata

0.2

0

40

80

120

160

200

0.2

0

40

80

120

160

Condensation Epochs

CondensationEpochs

(b)Evaluation of synthetic datafor time seriesforecasting

(c)Comparisonofperformancefordifferentmethods

However, directly applying these dataset condensation methods designed for classification to the domain of time series forecasting (TS-forecasting) results in performance degradation. The objective of dataset condensation is to generate a synthetic dataset so that when the model M s trained with the synthetic dataset and the model M f trained with the full dataset are given identical input, the two models output similar predictions . However, the concept of similar prediction differs between classification and TS-forecasting. In classification, as shown in Fig.1(a), predictions are considered similar if M s and M f assign the same class label, irrespective of differences in the distribution of output logits. Conversely, in TS-forecasting, as illustrated in Fig.1(b), the similarity of predictions from M s and M f is indicated by the mean squared distance of the predictions. The predictions are deemed similar only when all data points within the predictions are similar. This distinction in evaluation indicates TS-forecasting imposes more stringent criteria in discerning similar predictions compared to classification. It poses a challenge that previous dataset condensation methods based on classification fail to provide adequate assurance for the similarity between predictions of M s and M f within the realm of TS-forecasting.

To mitigate the gap, we propose a novel one-line dataset condensation plugin designed specifically for TS-forecasting called Cond ensation for T ime S eries F orecasting (CondTSF) based on our theoretical analysis. We first formulate the optimization objective of dataset condensation for TSforecasting. Then we transform the original optimization objective into minimizing the distance between predictions of M s and M f . Furthermore, to minimize the distance between predictions of M s and M f , we decompose the task into minimizing two terms, namely gradient term and value term . We theoretically prove that plugging CondTSF into previous methods can minimize the value term and gradient term synchronously. Therefore, CondTSF serves as an effective plugin to boost the performance of dataset condensation for TS-forecasting. As depicted in Fig.1(c), plugging CondTSF into previous methods yields a significant enhancement in performance.

In short, our contributions can be summarized as follows.

To the best of our knowledge, we are the first to explore dataset condensation for TS-forecasting. We conduct a theoretical analysis of the optimization objective of dataset condensation for TS-forecasting, breaking it down into two optimizable terms to facilitate improved optimization.

Leveraging insights from our theoretical analysis of TS-forecasting, we propose a simple yet effective dataset condensation plugin CondTSF. Plugging CondTSF into existing methods enables synchronous optimization of the two terms, leading to performance enhancement.

We conduct extensive experiments on eight widely used time series datasets to prove the effectiveness of CondTSF. CondTSF notably improves the performance of all previous dataset condensation methods across all datasets, particularly under low condensing ratios.

2 Related Works

Time Series Forecasting: Time series forecasting (TS-forecating) is the task of using historical, time-stamped data to predict future values. Previous works utilize different methods to achieve better

performance. These models can be mainly categorized into 3 types. (1) Transformer-based Models: Transformer[40] have shown great success in natural language processing, and models based on transformers[53, 42, 24, 54] emerged in TS-forecasting fields. (2) MLP-based Models: Efforts to use MLP-based models have been put into TS-forecasting in recent years[47] since DLinear[45] triumph transformer-based models with a simple MLP structure. (3) Patch-based Models: These models[34, 48, 28, 29] focused on learning representation cross patches instead of learning attention at each time point. Therefore they used a patching strategy before feeding the data to transfomers.

Dataset Condensation: Dataset condensation is a task that aims at distilling a large dataset into a smaller one so that when a model is trained on the small synthetic dataset and the full dataset separately, the testing performances of the trained models are similar. Previous works related to dataset condensation can be divided into 3 classes below. (1) Coreset Selecting Methods: These methods aim at selecting data with representative features from source dataset to construct a synthetic dataset[1, 4, 12, 37, 39]. (2) Matching-based Methods: These methods aim at minimizing a specific metric surrogate model learned from source dataset and synthetic dataset. The defined metrics are different, including gradient[51, 18, 46], features from the same class[41], distribution of synthetic data[50, 52] and training trajectories[3, 5, 7, 11, 8]. (3) Kernel-based Methods: These methods aim at obtaining a closed-form solution for the optimization problem utilizing kernel ridge-regression[20, 33]. In this way, the bi-level optimization problem of dataset condensation is reduced to a single-level optimization problem. Based on these results, the following works have made significant progress in different areas, including decreasing training cost and time[55], improving performance[30, 31].

3 Preliminaries

Dataset Condensation for TS-forecasting Given a time series dataset, we split the dataset into a train set and a test set. In this paper, we denote the train set as f and the test set as x . We denote the synthetic dataset as s . The synthetic dataset s is a small dataset distilled from the full train set f . Train set f , test set x , and synthetic dataset s are all vectors. We employ M θ as a neural network parameterized by θ . Without losing generality, we suppose the model M θ is using historical sequence x t : t + m with length m to predict future sequence x t + m : t + m + n with length n . Given the test set x , we formulate the test error of M θ as the error between the prediction of M θ on test input x t : t + m and the test label x t + m : t + m + n , as shown in Eq.1.

During dataset condensation process , a distribution of initial model parameters P θ is available for training model parameter sampling, and the full train set f is available for condensation. Subsequently, a synthetic dataset s is distilled from the full train set f using dataset condensation methods. During testing process , initial testing model parameter θ 0 ,test is sampled from P θ . Since θ 0 ,test is sampled in the testing process, it's unavailable during the previous dataset condensation process. Then model parameters θ s,test and θ f,test are obtained by training initial testing parameter θ 0 ,test on synthetic dataset s and the full train set f respectively. The objective of dataset condensation is to ensure model M θ s,test and M θ f,test have comparable performance on test set x . Therefore the practical optimization objective is to ensure that model M θ s,test trained with synthetic dataset s minimizes the test error L test on test set x . The optimization objective is formulated as Eq.2.

4 Method

Since test set x is not available during the dataset condensation process, the original optimization objective for dataset condensation in Eq.2 is non-optimizable. To mitigate this gap, in the following sections, we transform the non-optimizable objective into two distinct optimizable terms. Then we develop methods to optimize the two terms, thereby indirectly optimizing the original objective.

4.1 Decomposition

In this section, we decompose the optimization objective of dataset condensation in Eq.2 into two optimizable terms for better optimization. In the testing process, the initial testing model parameter

Break down to

Non-optimizableobjective

Break down to

optimizableterm

minLtest(Mo,)

optimizableterm

Initial synthetic

8

Distilled synthetic

Full train set

data

Parameter Matching

CondTSF

data

Random

min

110f- 0s112

min Llabel

f

Sample

GradientTerm

8

ValueTerm

*

Optimization

Optimization

θ 0 ,test is sampled from a distribution of initial model parameters P θ . Then we train θ 0 ,test on the synthetic dataset s to get model parameter θ s,test , and train θ 0 ,test on the full train set f to get model parameter θ f,test . Given test dataset x , the optimization objective is formulated as Eq.3.

Meanwhile, there is a non-optimizable error ϵ between the prediction of model M θ f,test and the true label from the test dataset, which is formulated in Eq.4.

Then we decompose the upper bound of L test ( M θ s,test , x ) into two terms, as shown in Thm.1. We utilize Taylor Expansion in the proof of Thm.1. For each real test data x t : t + m , we can arbitrarily choose position t ′ and get synthetic data s t ′ : t ′ + m . Then we can perform Taylor Expansion with s t ′ : t ′ + m to obtain the value of M θ s,test ( x t : t + m ) and M θ f,test ( x t : t + m ) .

Theorem 1. Given arbitrary synthetic data s t ′ : t ′ + m , the upper bound of the optimization objective of dataset condensation L test ( M θ s,test , x ) can be formulated as such

To prove Thm.1, we use linear models for further analysis since linear models can be both effective and efficient in TS-forecasting[45]. Given a linear model M θ ( x ) = θ x , its second and higher order gradient is zero. Therefore first-order Taylor Expansion is sufficient to obtain the accurate prediction of the model. Meanwhile, if M θ is a non-linear model, we ignore the higher-order terms of Taylor Expansion. We prove Thm.1 by applying the property of the first-order Taylor Expansion and triangular inequality of norm functions. The complete proof is in App.A.1. Hence we decompose the optimization objective of dataset condensation for TS-forecasting into two optimizable terms, namely value term and gradient term . For value term , it ensures M θ s,test and M θ f,test are similar in prediction values. For gradient term , it ensures the predictions of M θ s,test and M θ f,test are similar in gradient. Optimizing these two terms can optimize the upper bound of the original optimization objective, and therefore indirectly optimize the original optimization objective in Eq.3.

4.2 Gradient Term Optimization

We develop a method to optimize gradient term in this section. Given a linear model M θ ( x ) = θ x , its gradient on input is ∇M θ ( x ) = θ ⊤ . It indicates that the gradient of a linear model on input is the parameter of the model. We apply Cauchy-Schwarz Inequality to the gradient term and get its upper bound. We reformulate the gradient term and get its upper bound as shown in Eq.6.

Input Synthetic Data St':t'+m

St':t'+m

Train Expert Model Prediction M (st:t'+m)

Similar

Mo;(St:t'+m)

.··

Moftear(St:t+m)

Mo;(St:t'+m)

Visualization of Original Data

Input SyntheticData

Expert ModelPrediction

Empiricallyilustrate: Mostest(St:t'+m) ≈ Mog(St:t'+m) ≈ Mo;(St:t+m) ≈ ..: ≈ Mo(St:t'+m)

Since test data x t : t + m is not available during the dataset condensation process, the distance between synthetic data and test data || x t : t + m -s t ′ : t ′ + m || 2 is not optimizable. Therefore we only need to optimize the distance between parameters || θ s,test -θ f,test || 2 . All previous dataset condensation methods based on parameter matching can minimize this distance. Here we utilize MTT[3] as an example to clarify the optimization process. The optimization objective of trajectory matching is

However, since θ s,test and θ f,test are trained from testing initial parameter θ 0 ,test ∼ P θ , they are not available during dataset condensation process. Therefore, in practice, we sample θ 0 0 , . . . , θ k 0 ∼ P θ as initial parameters during dataset condensation process. The initial parameters are trained on synthetic dataset s and full train set f respectively to get θ 0 s , . . . , θ k s and θ 0 f , . . . , θ k f . Then we substitute θ s,test , θ f,test and θ 0 ,test in Eq.7 with parameters sampled in dataset condensation, making the optimization objective optimizable. The practical optimization objective is shown in Eq.8.

In practice, θ 0 0 , . . . , θ k 0 and θ 0 f , . . . , θ k f are sampled, trained, and stored in a parameter buffer before dataset condensation process. It can be concluded that using trajectory matching methods is intuitively minimizing the distance between θ i s and θ i f for all initial parameters θ i 0 ∼ P θ . By minimizing the upper bound of the gradient term, trajectory matching methods indirectly optimize the gradient term.

4.3 Value Term Optimization

We develop an optimization objective to optimize the value term in this section. Since θ f,test is trained from θ 0 ,test , it's unavailable in dataset condensation process. To mitigate this gap, we prove that although θ f,test is unavailable in dataset condensation process, its prediction M θ f,test ( s t ′ : t ′ + m ) is still available. To prove this statement, we sample initial model parameters θ 0 0 , . . . , θ k 0 from P θ . Then θ 0 0 , . . . , θ k 0 are all trained with the same full train set f . After training, we get parameters θ 0 f , . . . , θ k f . It is observed that models M θ 0 f , . . . , M θ k f predict similarly given arbitrary synthetic data s t ′ : t ′ + m as input.

Since initial testing parameter θ 0 ,test is also sampled from the same distribution P θ and θ f,test is trained from θ 0 ,test using the same full train set f , the prediction of M θ f ,test is similar to predictions of an arbitrary expert model M θ i f . The conclusion is formulated in Eq.9.

Algorithm 1 Dataset Condensation with CondTSF (MTT[3] as backbone)

Input: Synthetic data s ; Parameter buffer { ( θ 0 , θ f ) } k ; Synthetic learning rate α ; Trajectory matching epochs N ; Total condensation epochs E ; Additive update ratio β ; Gap of epochs G between using CondTSF

Output: Optimized synthetic data s

̸

Experiments have proved Eq.9 in Fig.3. As shown in Fig.3, for each synthetic data input s t ′ : t ′ + m (orange points), the predictions of corresponding expert models (yellow and blue points) are similar. Therefore, although θ f,test is unavailable in the dataset condensation process, its prediction M θ f,test ( s t ′ : t ′ + m ) can be obtained using the prediction of an arbitrary expert model M θ i f ( s t ′ : t ′ + m ) . Now we reformulate the value term and transform it into a practical optimization objective. Firstly, We formulate the upper bound of the value term as shown in Thm.2.

We prove Thm.2 by utilizing the triangular inequality and the prediction optimality of θ s,test on synthetic data s . The complete proof is in App.A.2. Accodring to Thm.2, we obtain an optimizable upper bound of the value term. Therefore the optimization objective for the value term can be naturally defined as minimizing the upper bound of the value term, as shown in Eq.11.

According to Thm.2, label error L label is the upper bound of the value term. Therefore, by minimizing the upper bound of the value term, the value term is indirectly minimized.

4.4 CondTSF

In this section, we develop a one-line plugin called CondTSF to minimize the label error L label in Eq.11 so that the value term can be optimized. CondTSF is a lightweight one-line plugin, no backpropagation or gradient is required during the update. CondTSF utilizes a simple yet effective additive method to iteratively update the synthetic data s and minimize the label error L label . In TS-forecasting, when generating training data, the data is usually sampled overlap from the dataset. Inspired by the overlap property, we utilize an additive method in CondTSF to gradually update the synthetic data to avoid vibrations. In the i th update iteration, CondTSF uses the prediction of expert model M f,test ( s t ′ : t ′ + m ) to update synthetic label s t ′ + m : t ′ + m + n . The update process is shown in Eq.12.

where 0 < β < 1 is the update ratio of this additive update method. This additive update process lowers the label error L label of s in each iteration exponentially, which can be formulated as

Since the update ratio has a value of 0 < β < 1 , CondTSF lowers the label error L label exponentially in each update iteration and solves the optimization problem for the value term. As a plugin module, CondTSF is used to update once for every G iteration of parameter matching methods. In this way, the gradient term and the value term can be optimized synchronously. The algorithm is shown in Alg.1. We also formulate the complete condensation process using CondTSF, as shown in Fig.2.

5 Experiment

5.1 Experiment Settings

Dataset Settings: The efficacy of dataset condensation methods is assessed across eight time series datasets. For all datasets, the model is set to be using 24 steps of data to forecast 24 steps of data. We

ExchangeRate

Weather

MLP

LSTM

CNN

MLP

LSTM

CNN

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

Random

0.931

±

0.024

1.246

±

0.057

0.840

±

0.047

1.035

±

0.102

0.910

±

0.038

1.217

±

0.106

0.554

±

0.010

0.632

±

0.016

0.531

±

0.020

0.598

±

0.033

0.570

±

0.006

0.655

±

0.017

DC

0.713

±

0.059

0.740

±

0.114

0.511

±

0.034

0.390

±

0.048

0.588

±

0.049

0.519

±

0.072

0.503

±

0.014

0.540

±

0.022

0.446

±

0.011

0.430

±

0.017

0.517

±

0.016

0.533

±

0.028

KIP

0.483

±

0.012

0.397

±

0.013

0.512

±

0.024

0.422

±

0.026

0.494

±

0.022

0.414

±

0.027

0.293

±

0.008

0.276

±

0.011

0.262

±

0.004

0.253

±

0.004

0.331

±

0.005

0.292

±

0.003

FRePo

0.564

±

0.033

0.537

±

0.041

0.583

±

0.048

0.569

±

0.077

0.599

±

0.025

0.578

±

0.044

0.393

±

0.013

0.401

±

0.011

0.419

±

0.044

0.424

±

0.043

0.434

±

0.011

0.428

±

0.016

MTT

0.421

±

0.007

0.301

±

0.009

0.431

±

0.010

0.313

±

0.009

0.419

±

0.007

0.300

±

0.010

0.286

±

0.006

0.256

±

0.004

0.279

±

0.007

0.249

±

0.004

0.328

±

0.018

0.276

±

0.014

PP

0.383

±

0.009

0.249

±

0.008

0.388

±

0.013

0.252

±

0.011

0.465

±

0.021

0.343

±

0.024

0.279

±

0.019

0.253

±

0.010

0.309

±

0.009

0.271

±

0.008

0.344

±

0.023

0.315

±

0.031

TESLA

0.316

±

0.008

0.172

±

0.007

0.323

±

0.010

0.175

±

0.007

0.439

±

0.034

0.302

±

0.044

0.298

±

0.012

0.266

±

0.007

0.292

±

0.012

0.253

±

0.010

0.331

±

0.004

0.283

±

0.006

FTD

0.425

±

0.007

0.306

±

0.005

0.433

±

0.012

0.310

±

0.011

0.445

±

0.025

0.329

±

0.031

0.286

±

0.010

0.251

±

0.005

0.303

±

0.028

0.264

±

0.017

0.347

±

0.015

0.305

±

0.014

DATM

0.452

±

0.013

0.349

±

0.016

0.229

±

0.045

0.095

±

0.032

0.351

±

0.053

0.209

±

0.049

0.270

±

0.004

0.258

±

0.004

0.275

±

0.012

0.253

±

0.010

0.323

±

0.022

0.282

±

0.021

CondTSF

0.135

±

0.005

0.032

±

0.002

0.135

±

0.004

0.032

±

0.002

0.248

±

0.031

0.101

±

0.022

0.242

±

0.009

0.229

±

0.006

0.248

±

0.004

0.231

±

0.004

0.283

±

0.007

0.256

±

0.004

Electricity

Traffic

MLP

LSTM

CNN

MLP

LSTM

CNN

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

Random

0.790

±

0.016

0.931

±

0.039

0.758

±

0.007

0.866

±

0.016

0.782

±

0.012

0.919

±

0.034

0.743

±

0.015

1.102

±

0.042

0.742

±

0.007

1.088

±

0.015

0.753

±

0.016

1.100

±

0.031

DC

0.778

±

0.007

0.912

±

0.016

0.770

±

0.004

0.884

±

0.009

0.769

±

0.010

0.897

±

0.022

0.730

±

0.011

1.035

±

0.031

0.709

±

0.012

0.989

±

0.030

0.747

±

0.009

1.068

±

0.020

KIP

0.769

±

0.014

0.881

±

0.029

0.700

±

0.018

0.741

±

0.036

0.761

±

0.016

0.864

±

0.035

0.738

±

0.018

1.056

±

0.045

0.714

±

0.017

1.008

±

0.023

0.753

±

0.018

1.074

±

0.023

FRePo

0.620

±

0.009

0.626

±

0.016

0.633

±

0.016

0.625

±

0.029

0.642

±

0.011

0.665

±

0.022

0.645

±

0.007

0.802

±

0.014

0.650

±

0.005

0.811

±

0.012

0.656

±

0.003

0.817

±

0.006

MTT

0.465

±

0.009

0.374

±

0.010

0.467

±

0.013

0.378

±

0.015

0.491

±

0.008

0.405

±

0.010

0.635

±

0.004

0.797

±

0.009

0.634

±

0.008

0.788

±

0.011

0.655

±

0.007

0.817

±

0.007

PP

0.483

±

0.008

0.388

±

0.009

0.481

±

0.008

0.388

±

0.009

0.521

±

0.015

0.444

±

0.021

0.617

±

0.006

0.751

±

0.006

0.610

±

0.008

0.740

±

0.010

0.593

±

0.004

0.745

±

0.010

TESLA

0.515

±

0.006

0.441

±

0.007

0.515

±

0.012

0.439

±

0.011

0.530

±

0.006

0.462

±

0.009

0.623

±

0.009

0.800

±

0.014

0.603

±

0.004

0.778

±

0.011

0.631

±

0.003

0.809

±

0.009

FTD

0.505

±

0.009

0.418

±

0.010

0.500

±

0.015

0.414

±

0.016

0.539

±

0.006

0.470

±

0.008

0.635

±

0.013

0.787

±

0.018

0.644

±

0.016

0.796

±

0.024

0.632

±

0.005

0.783

±

0.011

DATM

0.501

±

0.011

0.416

±

0.012

0.509

±

0.018

0.428

±

0.023

0.511

±

0.005

0.431

±

0.007

0.583

±

0.008

0.707

±

0.015

0.592

±

0.004

0.709

±

0.009

0.598

±

0.005

0.726

±

0.012

CondTSF

0.326

±

0.002

0.231

±

0.002

0.324

±

0.012

0.230

±

0.007

0.373

±

0.008

0.272

±

0.008

0.423

±

0.004

0.498

±

0.003

0.419

±

0.006

0.488

±

0.007

0.454

±

0.005

0.522

±

0.006

ETTm1

ETTm2

MLP

LSTM

CNN

MLP

LSTM

CNN

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

Random

0.697

±

0.009

0.859

±

0.020

0.677

±

0.017

0.801

±

0.033

0.713

±

0.015

0.891

±

0.027

0.732

±

0.017

0.880

±

0.041

0.754

±

0.020

0.927

±

0.054

0.760

±

0.021

0.934

±

0.056

DC

0.662

±

0.006

0.786

±

0.003

0.636

±

0.007

0.741

±

0.011

0.676

±

0.013

0.808

±

0.028

0.623

±

0.036

0.629

±

0.069

0.532

±

0.028

0.459

±

0.048

0.682

±

0.023

0.745

±

0.051

KIP

0.566

±

0.005

0.697

±

0.019

0.555

±

0.008

0.690

±

0.018

0.571

±

0.007

0.694

±

0.015

0.285

±

0.012

0.144

±

0.009

0.290

±

0.021

0.149

±

0.015

0.347

±

0.031

0.201

±

0.028

FRePo

0.599

±

0.007

0.718

±

0.013

0.611

±

0.028

0.738

±

0.048

0.630

±

0.031

0.749

±

0.086

0.476

±

0.032

0.412

±

0.043

0.472

±

0.057

0.395

±

0.089

0.579

±

0.075

0.574

±

0.142

MTT

0.484

±

0.003

0.484

±

0.005

0.515

±

0.033

0.530

±

0.048

0.563

±

0.006

0.608

±

0.019

0.258

±

0.007

0.129

±

0.005

0.246

±

0.005

0.124

±

0.004

0.340

±

0.016

0.193

±

0.017

PP

0.486

±

0.007

0.474

±

0.008

0.527

±

0.031

0.539

±

0.040

0.581

±

0.019

0.644

±

0.036

0.272

±

0.004

0.136

±

0.003

0.269

±

0.002

0.135

±

0.002

0.308

±

0.013

0.167

±

0.010

TESLA

0.519

±

0.003

0.523

±

0.003

0.513

±

0.007

0.516

±

0.007

0.579

±

0.013

0.620

±

0.020

0.272

±

0.004

0.135

±

0.003

0.272

±

0.007

0.135

±

0.006

0.365

±

0.041

0.221

±

0.043

FTD

0.528

±

0.015

0.579

±

0.032

0.631

±

0.015

0.790

±

0.037

0.576

±

0.024

0.626

±

0.041

0.279

±

0.005

0.142

±

0.004

0.290

±

0.011

0.147

±

0.010

0.403

±

0.025

0.254

±

0.024

DATM

0.499

±

0.007

0.516

±

0.006

0.513

±

0.015

0.524

±

0.016

0.577

±

0.030

0.616

±

0.046

0.293

±

0.004

0.153

±

0.004

0.290

±

0.009

0.149

±

0.007

0.377

±

0.030

0.232

±

0.029

CondTSF

0.452

±

0.004

0.455

±

0.001

0.459

±

0.013

0.461

±

0.011

0.520

±

0.018

0.543

±

0.025

0.231

±

0.002

0.107

±

0.001

0.240

±

0.009

0.111

±

0.005

0.273

±

0.021

0.133

±

0.014

ETTh1

ETTh2

MLP

LSTM

CNN

MLP

LSTM

CNN

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

Random

0.670

±

0.011

0.796

±

0.023

0.658

±

0.012

0.773

±

0.022

0.703

±

0.014

0.874

±

0.039

0.732

±

0.012

0.874

±

0.031

0.702

±

0.014

0.799

±

0.031

0.755

±

0.027

0.908

±

0.062

DC

0.643

±

0.019

0.745

±

0.038

0.626

±

0.014

0.718

±

0.013

0.672

±

0.023

0.802

±

0.041

0.619

±

0.019

0.650

±

0.043

0.534

±

0.019

0.481

±

0.036

0.680

±

0.028

0.746

±

0.053

KIP

0.636

±

0.017

0.732

±

0.029

0.608

±

0.016

0.696

±

0.027

0.650

±

0.016

0.758

±

0.021

0.494

±

0.009

0.419

±

0.011

0.431

±

0.012

0.329

±

0.016

0.551

±

0.032

0.492

±

0.051

FRePo

0.653

±

0.004

0.770

±

0.007

0.640

±

0.009

0.754

±

0.019

0.659

±

0.010

0.783

±

0.025

0.570

±

0.033

0.552

±

0.060

0.485

±

0.036

0.405

±

0.054

0.672

±

0.023

0.728

±

0.054

MTT

0.606

±

0.003

0.673

±

0.009

0.613

±

0.005

0.680

±

0.010

0.612

±

0.007

0.692

±

0.011

0.307

±

0.005

0.182

±

0.004

0.305

±

0.014

0.180

±

0.010

0.374

±

0.033

0.246

±

0.036

PP

0.633

±

0.006

0.719

±

0.006

0.630

±

0.006

0.710

±

0.009

0.635

±

0.007

0.730

±

0.007

0.354

±

0.019

0.229

±

0.020

0.292

±

0.012

0.173

±

0.007

0.450

±

0.060

0.346

±

0.085

TESLA

0.602

±

0.005

0.671

±

0.010

0.590

±

0.005

0.651

±

0.013

0.612

±

0.005

0.691

±

0.015

0.308

±

0.007

0.181

±

0.006

0.292

±

0.004

0.170

±

0.003

0.390

±

0.016

0.257

±

0.014

FTD

0.616

±

0.008

0.710

±

0.014

0.618

±

0.008

0.716

±

0.011

0.626

±

0.008

0.725

±

0.014

0.329

±

0.003

0.197

±

0.004

0.312

±

0.007

0.187

±

0.012

0.386

±

0.014

0.249

±

0.023

DATM

0.617

±

0.004

0.681

±

0.010

0.612

±

0.007

0.672

±

0.015

0.637

±

0.004

0.723

±

0.010

0.337

±

0.005

0.208

±

0.006

0.329

±

0.006

0.200

±

0.006

0.398

±

0.015

0.268

±

0.013

CondTSF

0.434

±

0.001

0.397

±

0.001

0.429

±

0.002

0.388

±

0.005

0.473

±

0.006

0.456

±

0.008

0.290

±

0.005

0.168

±

0.004

0.287

±

0.006

0.166

±

0.004

0.342

±

0.008

0.211

±

0.006

set the length of the synthetic dataset as 48, as shown in Table.2. Each synthetic dataset can only generate one training pair. We conduct experiments with two larger distill ratios as shown in App.B.

Model Settings: We plug CondTSF into existing dataset condensation models based on parameter matching, including DC[51], MTT[3], PP[21], TESLA[5], FTD[7] and DATM[11] to prove the effectiveness of CondTSF. We also conduct experiments on non-parameter-matching based methods, including DM[50], IDM[52], KIP[33], FRePo[55] to prove that optimizing value term only also helps boost the performance. The experiment setting and results are shown in App.E. We use DLinear[45] as the expert model to perform dataset condensation since DLinear is a linear model.

Metric Settings: The source dataset is first divided into a train set and a test set. All synthetic data is initialized by randomly sampling data from the train set. After a synthetic dataset is finished distilling, it is used to train another five models. After the five models are trained, they are tested on the test set. Their average mean absolute error (MAE) and mean square error (MSE) are recorded. We repeat the process above five times and report the average and standard deviation. While testing the generalization ability of the dataset condensation methods, DLinear[45] is used as the expert model to perform dataset condensation. Meanwhile, MLP, LSTM[13], and CNN are used as test models when testing the generalization ability of the dataset condensation methods.

110g  00l12

110 - 0s]12

GradientTerm

Llabel

ValueTerm

Ltest

1.0

MTT

100

MTT

1

MTT

MTT+ CondTSF

MTT+CondTSF

0.8

MTT+CondTSF

Error

0.8

80

0.7

Parameter

0.6

Label Error

60

Error

0.6

0.4

40

Test

0.5

0.4

0.2

20

0.3

0.0

0

0.2

0

40

80

120

160

200

0

40

80

120

160

200

0

40

80

120

160

200

CondensationEpochs

Condensation Epochs

CondensationEpochs

Implementation Details: As a plugin module, we test CondTSF with all previous methods. Each synthetic dataset is optimized using a standard training process according to the chosen backbone model. CondTSF is set to update every 3 epochs and the additive update ratio β is set to be 0.01. All the experiments are carried out on an NVIDIA RTX 3080Ti.

5.2 Results

Single Architecture Performance: The results are summarized in Table.1. For each backbone method, the first line shows the performance of the backbone model, the second line shows the performance of a backbone model with CondTSF, and the third line shows the percentage of reduction in MAE and MSE after CondTSF is applied. There's a considerable reduction in error for all backbone models. The results suggest that CondTSF is effective in optimizing the value term and enhancing the performance in dataset condensation for TS-forecasting. However, using CondTSF on DC[51] is not as effective as other methods. The reason is that instead of directly matching parameters, DC matches the gradient of parameters on loss in each iteration. Indirectly matching gradient leads to accumulating errors in parameters, making DC unable to lower parameter error as effectively as directly matching parameters. Therefore CondTSF is not effective enough when applied to DC[51].

Cross Architecture Performance: We also conduct experiments to evaluate the cross-architecture performance of dataset condensation methods. The results are summarized in Table.3. We test all models on all datasets with MLP, LSTM[13], and CNN as test models. All synthetic data is distilled using DLinear[45] model as experts. We use MTT[3] as the backbone for CondTSF. We observe that CondTSF based on MTT outperformed all other previous models.

5.3 Discussion

Test Performance and Errors: We conduct experiments on ExchangeRate dataset with MTT[3] and MTT+CondTSF. As shown in Fig.4, trajectory of parameter error || θ f -θ s || 2 || θ f -θ 0 || 2 , label error L label and test error L test through the distillation process are presented. Regarding the parameter error corresponding to the gradient term, both MTT and MTT+CondTSF converge quickly, suggesting that the incorporation of CondTSF doesn't impact parameter alignment. As for the label error corresponding to the value term, since the initial synthetic data s is randomly sampled from the train set f and the expert model is trained by the train set f , the label error of s is small at the beginning. However, the utilization of MTT results in an elevation of label error, whereas employing CondTSF effectively mitigates this increase in label error. During the test, MTT+CondTSF notably outperforms MTT by concurrently optimizing both the value term and the gradient term.

6 Limitations

The limitation of this work is that we use linear models in our analysis so that the gradient of a model on input is the parameter of the model. Therefore, only linear models like DLinear[45] are solid enough to be an expert model for dataset condensation. The analysis no longer holds when it comes to more complicated models. However, experiments in App.D and App.E show that CondTSF is

also effective with non-parameter-matching methods and non-linear models, which merits further exploration.

7 Conclusion

In this study, we provide abundant proof that previous dataset condensation methods based on classification are not suitable for dataset condensation for TS-forecasting. We elucidate that these earlier methods, predominantly focused on classification tasks, only address a portion of the optimization objective pertinent to TS-forecasting. To address this issue, we propose a plugin module called CondTSF that can collaborate with parameter matching based dataset condensation methods. CondTSF optimizes the optimization objective that previous methods have neglected and boosts the performance of dataset condensation methods on TS-forecasting. We conduct experiments on eight widely used time series datasets and prove the effectiveness of our proof and method. CondTSF consistently enhances the performance of all previous techniques across all datasets, substantiating its effectiveness in improving dataset condensation outcomes for TS-forecasting applications.

References

Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv:1703.06476 , 2017.

Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948 , 2023.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4750-4759, 2022.

Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv preprint arXiv:1203.3472 , 2012.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet1k with constant memory. In International Conference on Machine Learning , pages 6565-6590. PMLR, 2023.

Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688 , 2023.

Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3749-3758, 2023.

Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. Advances in Neural Information Processing Systems , 36, 2024.

Qizhang Feng, Zhimeng Jiang, Ruiquan Li, Yicheng Wang, Na Zou, Jiang Bian, and Xia Hu. Fair graph distillation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Azul Garza and Max Mergenthaler-Canseco. Timegpt-1. arXiv preprint arXiv:2310.03589 , 2023.

Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. arXiv preprint arXiv:2310.05773 , 2023.

Sariel Har-Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. In Proceedings of the twenty-first annual symposium on Computational geometry , pages 126-134, 2005.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 , 2023.

Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196 , 2023.

Wei Jin, Xianfeng Tang, Haoming Jiang, Zheng Li, Danqing Zhang, Jiliang Tang, and Bin Ying. Condensing graphs via one-step gradient matching. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) , 2022.

Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2022.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning , pages 11102-11118. PMLR, 2022.

Joseph B Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika , 29(1):1-27, 1964.

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning , pages 12352-12364. PMLR, 2022.

Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation using parameter pruning. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences , 2023.

Haoyang Liu, Tiancheng Xing, Luwei Li, Vibhu Dalal, Jingrui He, and Haohan Wang. Dataset distillation via the wasserstein metric. arXiv preprint arXiv:2311.18531 , 2023.

Mengyang Liu, Shanchuan Li, Xinshi Chen, and Le Song. Graph condensation via receptive field distribution matching. arXiv preprint arXiv:2206.13697 , 2022.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations , 2021.

Yilun Liu, Ruihong Qiu, and Zi Huang. Cat: Balanced continual graph learning with graph condensation. In Proceedings of the IEEE International Conference on Data Mining (ICDM) , 2023.

Zhanyu Liu, Ke Hao, Guanjie Zheng, and Yanwei Yu. Dataset condensation for time series classification via dual domain matching. arXiv preprint arXiv:2403.07245 , 2024.

Zhanyu Liu, Chaolv Zeng, and Guanjie Zheng. Graph data condensation via self-expressive graph structure reconstruction. arXiv preprint arXiv:2403.07294 , 2024.

Zhanyu Liu, Guanjie Zheng, and Yanwei Yu. Cross-city few-shot traffic forecasting via traffic pattern bank. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , pages 1451-1460, 2023.

Zhanyu Liu, Guanjie Zheng, and Yanwei Yu. Multi-scale traffic pattern bank for cross-city few-shot traffic forecasting. arXiv preprint arXiv:2402.00397 , 2024.

Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems , 35:13877-13891, 2022.

Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. arXiv preprint arXiv:2302.06755 , 2023.

John A Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, I Budak Arpinar, and Ninghao Liu. A survey of deep learning and foundation models for time series forecasting. arXiv preprint arXiv:2401.13912 , 2024.

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations , 2020.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 , 2022.

Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, et al. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08278 , 2023.

Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos N Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 17097-17107, 2023.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017.

Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. Losscurvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics , pages 8606-8628. PMLR, 2023.

Ivor W Tsang, James T Kwok, Pak-Ming Cheung, and Nello Cristianini. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research , 6(4), 2005.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12196-12205, 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems , 34:22419-22430, 2021.

Zhe Xu, Yuzhong Chen, Menghai Pan, Huiyuan Chen, Mahashweta Das, Hao Yang, and Tong Hanghang. Kernel ridge regression-based graph dataset distillation. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) , 2023.

Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering , 2023.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11950-11959, 2023.

Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 , 2022.

Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations , 2022.

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning , pages 12674-12685. PMLR, 2021.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6514-6523, 2023.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR , 1(2):3, 2021.

Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7856-7865, 2023.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning , pages 27268-27286. PMLR, 2022.

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems , 35:9813-9827, 2022.

A Complete Proof

A.1 Complete Proof for Theorem 1

Theorem 1. Given arbitrary synthetic data s t ′ : t ′ + m , the upper bound of the optimization objective of dataset condensation L test ( M θ s,test , x ) can be formulated as such

Proof. Replacing the true label x t + m : t + m + n in L test ( M θ s,test , x ) with Eq.4, the optimization objective of dataset condensation for TS-forecasting is reformulated as the distance between the predictions of M θ s,test and M θ f,test given the same test input. Then the triangular inequality of norm functions is used and the original optimization objective can be transformed to its upper bound, as shown in Eq.15.

In Eq.15, we prove that minimizing the distance between M θ s,test ( x t : t + m ) and M θ f,test ( x t : t + m ) is equivalent to minimizing the upper bound of the original optimization objective. Then we decompose the distance between predictions of M θ s,test and M θ f,test into two optimizable terms for better optimization. We use linear models for further analysis since linear models can be both effective and efficient in TS-forecasting tasks[45]. Given a linear model M θ ( x ) = θ x , its second and higher order gradient is zero, i.e. ∇ k M θ ( x ) = 0 , ∀ k ≥ 2 . Therefore, first-order Taylor Expansion can be utilized to get the prediction of the model M θ on test data x t : t + m using the prediction and gradient of the model M θ on arbitrary synthetic data s t ′ : t ′ + m . The process is formulated in Eq.16. Meanwhile, it is worth mentioning that if M θ is a non-linear model, then the second and higher-order terms of the Taylor Expansion are ignored in Eq.16.

Then we expand Eq.15 with Taylor expansion. After that, the triangular inequality of norm functions is used to get its upper bound. In the meantime, by applying the triangular inequality, the optimization objective can be decomposed into two optimizable terms.

Therefore Thm.1 is proved.

A.2 Complete Proof for Theorem 2

Theorem 2. The upper bound of the value term can be formulated as such

Proof. We first use triangular inequality and the non-negativity of norm functions to get the upper bound of the value term. The process is shown in Eq.19.

For further analysis, we need to step back and formulate the training process of θ s to derive an inequality. By doing dataset condensation, a synthetic dataset s is obtained. Then we formulate the training process of θ s,test on synthetic data s as minimizing the prediction error on s . The training process is formulated in Eq.20.

Now we can derive an inequality. We denote s as the synthetic dataset obtained by dataset condensation. We denote M θ s,test as the model that is trained on s as shown Eq.20. According to Eq.20, M θ s,test has the lowest prediction error on synthetic data s under the given model architecture. Since M θ s,test and M θ f,test share the same model architecture, the prediction error of M θ s,test on synthetic data s is no larger than the prediction error of M f,test on synthetic data s . This inequality can be formulated as such

By applying Eq.21 to Eq.19, we obtain the upper bound of the value term, as shown in Eq.22.

Therefore Thm.2 is proved.

B Performance with Different Distill Ratio

We further explore the performance of CondTSF with different distill ratios and compare the results with previous matching-based methods.

B.1 Standard Ratio Condensation

We distill the dataset into a synthetic dataset with a flexible length for each dataset. The information on condensation in Table.4. The performance is shown in Table.5.

B.2 3-times Standard Ratio Condensation

We distill the dataset into a synthetic dataset with a flexible length for each dataset. Each synthetic dataset is 3 times larger than the synthetic data in Table.4. The information on condensation is shown in Table.6. The performance is shown in Table.7.

We observe that CondTSF consistently improves the performance of backbone models with all condensing ratios, suggesting the effectiveness of CondTSF with different condensing ratios.

C Performance of CondTSF with Non-parameter-matching Based Methods

We distill the dataset using the standard condensing ratio. The information on condensation is shown in Table.4. We conduct experiments on CondTSF with non-parameter-matching based methods. We use DM[50], IDM[52], KIP[33], FRePo[55] as backbone methods. The performance is shown in Table.8.

Results show that using CondTSF to optimize only one of the two optimizable terms can also boost the performance.

D Performance of CondTSF with Non-Linear Expert Models

We distill the dataset using the standard condensing ratio. The information on condensation is shown in Table.4. We conduct experiments on distilling dataset with non-linear expert models. We use MTT[3], TESLA[5], and DATM[11] as backbone methods. The performance of using a CNN as the expert model is shown in Table.9, and the performance of using a 3-layer-MLP as the expert model is shown in Table.10.

Results show that CondTSF is also effective with non-linear expert models.

E Performance Comparison of CondTSF and Smoothing

We distill the dataset using the one-shot condensing ratio. The information on condensation is shown in Table.2. We conduct experiments on distilling dataset with CondTSF and a low pass filter respectively. We use DLinear[45] as the expert model. The result is shown in Table.11.

Results indicate that using CondTSF is significantly more effective than using a low pass filter to smooth the distilled data.

F Ablation Study of CondTSF

We compare the changing trajectories of test error during the dataset condensation process. Since MTT has been proven to be a suitable backbone for CondTSF, we conduct experiments on different methods of plugging CondTSF into MTT. We utilize the standard condensing ratio as shown in Table.4.

Test error is calculated as such. After the synthetic data has been distilled, it is used to train 5 randomly initialized testing models. After training with the synthetic dataset, the models are tested on the test set sampled from the source dataset. MAE error is reported in the figures below.

F.1 Performance of CondTSF with Different Gap





































(a) ExchangeRate





























(b) Weather



































(c) Electricity

































(d) Traffic





























(e) ETTm1

































(f) ETTm2































(g) ETTh1



































(h) ETTh2





































(a) ExchangeRate





























(b) Weather



































(c) Electricity

































(d) Traffic





























(e) ETTm1

































(f) ETTm2































(g) ETTh1





































(h) ETTh2





































(a) ExchangeRate





























(b) Weather



































(c) Electricity

































(d) Traffic





























(e) ETTm1

































(f) ETTm2































(g) ETTh1



































(h) ETTh2

We observe that CondTSF consistently reduces the testing error with different utilization gaps.

F.2 Relationship of Performance and Label Error

We also conduct experiments on label error and test error. We visualize the trajectory of label error L label and test error through the distillation process. The results are shown in Fig.8.

Model 1: Use MTT to distill for 200 epochs.

Model 2: Use MTT to distill for 160 epochs and then use CondTSF to update for 40 epochs.

As shown in Fig.8, it can be observed that using MTT[3] leads to an increase in label error L label . While applying CondTSF effectively lowers the label error in the last 40 epochs, and therefore enhancing the performance.

(a) ExchangeRate

(b) Weather

(c) Electricity

(d) Traffic

(e) ETTm1

(f) ETTm2

(g) ETTh1

(h) ETTh2

CondTSFisused

↑

0.9

0.6

TestErrorofModel1

0.8

125

TestError of Model1

Label Error

60

TestErrorofModel2

0.7

Error

Label Error

100

TestErrorofModel2

0.5

Test Error

0.6

75

40

0.5

50

0.4

20

LabelErrorofModel1

0.3

25

0

LabelErrorofModel2

0.2

0

LabelErrorofModel2

0.3

0

40

80

120

160

200

0

40

80

120

160

200

CondensationEpochs

Condensation Epochs

0.9

1.0

TestError of Model1

150

TestErrorofModel1

150

TestErrorofModel2

0.8

TestError ofModel2

0.9

Label Error

Label Error

0.7

100

0.8

Crror

100

0.7

Test

50

50

0.6

LabelErrorofModel1

0.4

LabelError ofModel1

0.5

LabelError ofModel2

LabelError ofModel2

0

0.3

0

0.4

0

40

80

120

160

200

0

40

80

120

160

200

Condensation Epochs

Condensation Epochs

↑

0.8

0.8

150

TestError ofModel1

TestError ofModel2

40

TestErrorofModel2

0.7

Label Error

100

0.7

Error

Label Error

30

0.6

Crror

0.5

0.6

Test

20

Test

50

0.4

10

LabelErrorofModel1

0.3

0.5

LabelErrorofModel2

LabelErrorofModel2

0

0

0.2

0

40

80

120

160

200

0

40

80

120

160

200

Condensation Epochs

CondensationEpochs

CondTSFisused

0.9

200

1.0

TestErrorofModel1

TestErrorofModel1

0.9

Error

150

TestErrorofModel2

0.8

Error

Label Error

150

TestErrorofModel2

0.8

rror

Labell

100

0.7

Test

100

0.6

0.7

百

e

50

0.6

50

0.5

LabelErrorofModel1

Label Error of Model 1

0.4

LabelErrorofModel2

0.5

LabelErrorofModel2

0

0

0.3

0

40

80

120

160

200

0

40

80

120

160

200

CondensationEpochs

Condensation Epochs

G Parameter Sensitivity of CondTSF

We test CondTSF with different update gaps G and additive update ratios β . We utilize the standard distill ratio as shown in Table.4. Our observations indicate that CondTSF displays a notable degree of robustness concerning these parameters. Specifically, the effectiveness of CondTSF persists when the update gap G is moderately sized and additive update ratio β is not excessively small.





















































(a) ExchangeRate











































(b) Weather





















































(c) Electricity















































(d) Traffic



















































(e) ETTm1













































(f) ETTm2





















































(g) ETTh1























































(h) ETTh2

H Visualization of Synthetic Data

We provide some visualization of synthetic data distilled by MTT[3] and MTT+CondTSF on all datasets. It is observed that the synthetic dataset distilled with CondTSF is smoother than the ones without CondTSF. Smoother data indicates more generalized features and therefore helps boost the performance.

































































































































































































































































































































































































































































































































































































































































































































































































































































NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As shown in Sec.1, we explicitly claim the contributions of this work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: As shown in Sec.6, we discuss the limitations of this work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We propose two theorems in Sec.4. We provide complete proof of the two theorems in App.A.1 and App.A.2.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the detailed algorithm in Sec.4. We provide detailed parameter settings to reproduce the results in Sec.5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We attach the code needed to reproduce the results with the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed hyperparameter settings in Sec.5 that are necessary to reproduce the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeat each experiment five times and the average and standard deviation of errors are reported in Sec.5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: As shown in Sec.5, we provide the model of the GPU we used for experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work does not involve human subjects. As shown in Sec.5, the datasets used in this work are all widely used open-source datasets.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work does not involve human subjects. As shown in Sec.5, the datasets used in this work are widely used and open-source. There's no harm to humans or leak of privacy in this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: As shown in Sec.5, the datasets used in this work are all widely used and open-source. There's no risk of releasing unsafe data.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: As shown in Sec.5, we cite the papers that produced the code. The datasets used in this work are all widely used and open-source.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide documents with the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a dataset distillation approach by decomposing the distillation loss into two terms, the value term and the gradient term. Then, they derive bounds for these two terms that enable more effective dataset distillation for time-series forecasting. Strong performance is demonstrated in the conducted experimental evaluation\n\nSTRENGTHS:\n+ The paper is well-written and easy to follow.\n\n+ Authors attempt to support their methodology through theoretical explanations\n\n+ Strong performance is obtained in the experimental evaluation.\n\nWEAKNESSES:\n- p4. l121. The authors do not explain why this non-optimizable $\\epsilon$ error occurs. \n\n- l. 137 in principle we could directly optimize this gradient term without deriving any bound. Unless I am missing something, this is supported by DL frameworks (e.g., PyTorch). On the other hand, I understand that there might be computational benefits from deriving a bound that does not involve the gradients. However, authors should justify their choice (why we need to derive this bound) and if this is related to computational efficiency. Ideally, experiments should be provided to support their choice. \n\n- At various points there are implicit assumptions the effect of which is not appropriately discussed, e.g., Eq. (9) is essentially supported by an experimental result in a Figure and a verbal description. I do not (empirically) disagree with the conclusion, but I can see cases where this would not hold (e.g., in financial time-series forecasting, where using the same data we can have vastly different (non-linear) hypotheses). Whether this alters the conclusions/and or the method, is not appropriately discussed.\n\n- The updating process in (12) essentially moves the targets towards the predictions of the model. This is a reasonable result that is well-known to work. However, the way this is applied is now very close to target smoothing/model distillation approaches that essentially do the same thing (also applied in the context of time series forecasting as far as I know). There is some literature on model distillation approaches (and/or self-distillation) in the context of time-series forecasting and/or regression, which essentially demonstrate that this kind of \"smoothing\" can indeed increase performance. Therefore, this raises some questions on whether the method should be compared experimentally (or at least theoretically) with such approaches.\n\n- Even though some ablation studies were provided in the Appendix, I didn't manage to find perhaps the most important result: What is the impact of the gradient term and what is the impact of the value term? What would happen if we remove the L_param loss in l. 9 of Alg 1? What would happen if we removed the value term?\n\n- Isn't the method also applicable in the case of regression tasks? If I am not missing anything, I think the same methodology without any change (apart from the input to the models) can be applied also to handle generic regression tasks that also expand over time-series forecasting.\n\n\nMinor: l.147-152: I think $k$ is used before being introduced.\n\nQUESTIONS:\nPlease respond to the comments provided in the drawbacks. Overall, strong performance is demonstrated, but there are several aspects that should be improved.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores dataset condensation, which generates a small dataset for training of deep neural networks, for time series forecasting. Specifically, it proposes a one-line dataset condensation plugin designed specifically for time series forecasting. It first formulates the optimization objective of dataset condensation for time series forecasting, and based on which a reformulation is proposed and further theoretical analysis is performed. Based on the proposed theoretical analysis, this paper proposes a simple data condensation plugin CondTSF. Experiments on 8 time series datasets are performed to demonstrate the effectiveness of the proposed CondTSF.\n\nSTRENGTHS:\n1. It proposes a simple dataset condensation plugin. \n2. A theoretical analysis is provided.\n\nWEAKNESSES:\n1. Dataset condensation is a strategy specifically designed for training large dataset. However, the data size in time series is usually significantly smaller than that in NLP and CV. In my experience, the computational demand to deal with time series data may not be an urgent request in real-world time series applications. The topic discussed in this paper seems interesting given the current program of different foundation models, but I highly doubt it applicability in the field of time series. \n2.  A related problem is that how the time series foundation model performs remain unclear. Exploring the foundation model for time series may be a promising direction, but it is not widely accepted and deployed in real-world applications. In fact, considering the semantics of time series is highly dependent on the underlying applications, so far I have not observed any foundation model can provide good zero-shot learning capability.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nPlease refer to the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors study the question of dataset condensation / distillation in the context of time series forecasting. They propose a decomposition that could be used to empirically estimate bound the point-wise forecast MSE loss. Based on this decomposition the authors proposed a condensation plug in on top of existing data condensation method that updates the distilled dataset with the estimated full model forecast. Through empirical study they show that with this plug in the resulting condensed dataset leads to better models across different model classes and benchmark forecasting tasks.\n\nSTRENGTHS:\nThe paper attempts a new and simple solution to an important training data efficiency problem. It also attempts a novel prospect of analyzing the dataset distillation through the loss decomposition.\n\nWEAKNESSES:\nThe major weakness is the writing. It is hard to follow in its current shape, in particular the math description of the research question should be more rigorous. To name a few:\n- What are the sample space of x and s? They now look like a single univariate time series. It is the case?\n- What is t' in Eqn (5)? Is it summed over? If so is it double sum within sum_t?\n- How is Eqn (7) and (8) optimized (also in Algo 1).\nSimilarly regarding writing, the paper does not motivate the research question well, and has not provided enough background details on its base distillation methods (Line 200 Model Settings), especially MTT(3). As a result it is hard to evaluate the novelty of the paper when the main idea is not clearly delivered.\n\nThough mentioned as a limitation, the assumption of linear model trivializes the theories, which ties the proposed method to DLinear expert, and makes it hard to understand how the learning from the synthetic data generalizes beyond the expert's model class. There are similar simplifications, e.g. the empirical approximation as in Eqn (9) that's observational.\n\nQUESTIONS:\nBesides the questions in the weakness:\n\n- While the distill ratio in Table 2, e.g. condensing the datasets into one training example, can be used to show the extremity of the distillation process,  it is questionable what this one-shot learning means for the MLP, LSTM, CNN involved. Any insights, especially why to choose them over the few-shot study in, e.g. Appendix F (this one has visualization as well) to be in the main paper? \n- As now the expert model is more actively involved through the two decomposition steps to touch the synthetic data, how does the capacity of the expert impact the fidelity of the synthetic dataset versus the full one? What about at least one non linear expert just to get some insights?\n- The method proposed here does not utilize any time series concepts. If so this is an application on multivariate regression with linear expert. What's the novelty beyond that for time series exclusively?\n\nLIMITATIONS:\nThe authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper provides a simple fix on the parameter-matching based dataset condensation methods for time series forecasting tasks. Besides matching the parameters of the model $M_f$ trained on the full training set and $M_s$ trained on the smaller synthetic dataset, it additionally induces the $M_f$ to perform well in the synthetic data too. It theoretically shows that the additional regularization contributes to an surrogate upper bound of the original objective, and empirically proves that the fix help improve various dataset condensation methods.

STRENGTHS:
1. The paper is well-written with clear structure and proper organization, and the motivation of the proposed condTSF plugin is solidly grounded by theoretical analyses
2. The proposed method is simple and easy to implement. 
3. Impressive improvement are observed in the experiments on various baselines and benchmarks.

WEAKNESSES:
A few details are not clear. See questions.

QUESTIONS:
1. In line 162, the author claims that it is "observed" that models initialized with different samples of parameter values predict similarly after trained on the full dataset given arbitrary input. Is it supported by any evidences, either theoretically or empirically?
2. Some clarity on optimizing gradient term can be provided. For example, when the parameters are matched and $M_s$ are well fitted, why the label error of $M_f$ on $s$ will increase?

LIMITATIONS:
The limitation that the analysis only applies to linear models is discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14