Learned Data Transformation: A Data-centric Plugin for Enhancing Time Series Forecasting

Authors: Yuxuan Yang, Dalin Zhang, Yuxuan Liang, Hua Lu, Gang Chen, Huan Li
Primary area: learning on time series and dynamical systems
Keywords: time series, data-centric, data transformation, forecasting, generalization, deep learning

Abstract:
Data-centric approaches in Time Series Forecasting (TSF) often involve heuristic-based operations on data. This paper proposes to find a general end-to-end data transformation that serves as a plugin to enhance any arbitrary TSF model's performance. Our idea is to generate transformed data during an approximating process and to co-train a predictor for evaluating data with the transformation. To achieve this, we propose the Proximal Transformation Network (\model{}), which learns effective transformations while maintaining proximity to the raw data to ensure fidelity. When orthogonally integrated with popular TSF models, our method helps achieve state-of-the-art performance on seven real-world datasets. Additionally, we show that the proximal transformation process can be interpreted in terms of predictability and distribution alignment among channels, highlighting the potential of data-centric methods for future research. Our code is available at \href{https://anonymous.4open.science/r/PTN-2FC6/}{https://anonymous.4open.science/r/PTN-2FC6/}.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730622352549

thread_type:
official_review

<thread_object>
creator: reviewer_gmsx

<content>
summary:
This paper proposes a data transformation model to support long-term time series forecasting. Specifically, it first obtains a transformation of the raw data using Proximal Transformation Networks (PTNs) and then uses the transformed data to train a predictor. Each PTN consists of a convolutional encoder and a decoder with intra-patch attention, channel-wise attention, and a point-wise linear head. Experiments on several benchmark datasets are conducted to evaluate the effectiveness of the proposed model.

strengths:
1. The idea is relatively novel and is supported by some theoretical insights.
2. Extensive experiments from various perspectives are provided.
3. The motivations behind the proposal are well introduced.

weaknesses:
1. The comparison with baselines in Table 9 for look-back length 512 appears to be unfair. For instance, iTransformer should also use a look-back length of 512, and the results of PatchTST in this table are much worse than those reported in the PatchTST paper (PatchTST/64 in Table 3 of its paper).

2. The proposed model does not seem to perform well on complex datasets, such as Traffic. It would be beneficial to provide results on more complex datasets, such as the PEMS datasets used in the iTransformer paper.

3. It would be helpful to include Mean Squared Error (MSE) results in Table 4.

4. It seems inappropriate to claim that MoE is used without a gating network. Additionally, the method for selecting an appropriate number of PTNs, as well as the specific values used in the paper, is unclear.

5. There is no complexity analysis when adding the proposed model to the base models.

6. There are also many unclear points and typos in the paper, such as:

+
Figure 1 is not well explained, e.g., the meaning of "7/8".

+
It is unclear how the outputs of intra-patch attention and channel-wise attention are combined.

+
It is unclear how the prediction process is conducted after training. Should the raw data be directly input to the trained predictor?

+
"An attention-based Encoder" should be "An attention-based Decoder" on Page 2.

+
"Piece-wise Linear Head" should be "Point-wise Linear Head" in Figure 3.

questions:
Same as the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652158357

thread_type:
official_review

<thread_object>
creator: reviewer_t7bj

<content>
summary:
The paper proposes the Proximal Transformation Network (PTN), a plugin for improving time series forecasting (TSF) by learning general, data-centric transformations that enhance model performance while preserving proximity to the original data. PTN, which combines a convolutional encoder and attention-based decoder, can integrate with any TSF model, optimizing both data fidelity and forecasting accuracy. Through experiments on seven real-world datasets, PTN achieves state-of-the-art results, showing its effectiveness across linear and non-linear models and its ability to adapt data distributions for better predictability. Additionally, PTN supports interpretability and transferability, offering potential applications in other time series tasks, like anomaly detection and classification.

strengths:
1. PTN offers a general, model-agnostic approach to improve time series forecasting across diverse datasets.

2. It achieves state-of-the-art results, enhancing accuracy and robustness for both linear and non-linear models.

3. This paper conducts too many experiments to show the effectiveness of their framework.

weaknesses:
1. The paper’s abstract and introduction do not clearly convey the overall research idea and process, making it difficult to understand the framework.

2. The authors mention that PTN shows potential to make time series forecasting more interpretable. However, the enhanced embedding is latent, produced by deep learning. It is unclear how this actually enhances interpretability.

3. The authors provide a caption for the framework, but it does not help clarify the framework’s procedure. It's unclear how the transformed embedding is involved in enhancing the prediction task. I suggest that the authors include a framework overview to illustrate the entire process.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730663041934

thread_type:
official_review

<thread_object>
creator: reviewer_pxsv

<content>
summary:
The paper introduces the Proximal Transformation Network (PTN) as a data-centric plugin for enhancing time series forecasting. The proposed PTN aims to find optimal data transformations that improve model performance while preserving data fidelity. Extensive experiments demonstrate state-of-the-art results when the method is integrated with various forecasting models. The key contributions include a reformulation of the time series forecasting problem, the introduction of PTN, and successful performance on seven real-world datasets. The approach highlights the potential of data-centric methods in advancing time series forecasting research

strengths:
1.The paper is well-organized and easy to understand.
2.The proposed model can be widely applied.
3.The proposed model achieves SOTA performance.

weaknesses:
1.In section 3.2, while two losses are considered, what are the motivations/insights of the losses. The reason why they have an influence on the results should be explained.
2.As a plug-and-play model, whether it is lightweight and easy to use is an important criterion, but the experiment does not analyze the time and space complexity of the proposed model.
3.While the performance of the model is not promising enough, the authors don’t analyze the results or explain the pattern.

questions:
1.What is the motivation of losses in section 3.2?
2.Please study the time and space complexity of the proposed model.
3.Why the performance is not stable compared with baselines?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710398044

thread_type:
official_review

<thread_object>
creator: reviewer_cnmd

<content>
summary:
This paper proposes a Proximal Transformation Network to learn effective transformations while maintaining proximity to the raw data to ensure fidelity. The model includes a convolution-based Encoder and an attention-based Encoder that provide transformation on different levels of proximity. The training involves a co-optimization of the proximity of the transformed data and forecasting accuracy. The method achieves state-of-the-art performance on seven real-world datasets. Additionally, the paper shows that the proximal transformation process can be interpreted in terms of predictability and distribution alignment among channels.

strengths:
* The traditional time series prediction task has been redefined as a "two-step problem," the goal is to learn predictions on the transformed data and align them with the raw series, showcasing innovation.

* The proposed method achieves state-of-the-art performance on seven real-world datasets. The ablation experiments are comprehensive.

* The effectiveness of the proposed module is demonstrated through the distribution of data on the loss surface, revealing its ability to categorize time series into predictable and unpredictable groups in a self-supervised manner, with a particular focus on enhancing performance for the former.

weaknesses:
* When the predictor is a linear model, the complexity of PTN seems to far exceed that of the predictor itself. It is suggested to provide the time and memory complexity analysis of the proposed module. Additionally, for predictors that include modules capturing channel-wise and patch-wise correlations, the proposed PTN appears redundant, which may affect its generalizability.

* Based on the results in Table 10, the PTN module appears to enhance performance primarily for simple linear models, while its effectiveness on more complex models, such as iTransformer and PatchTST, varies against the dataset. Considering that the main objective of this paper is to propose a general plugin, it is essential to select a sufficient range of predictors for experimentation.

* The article contains several errors that require careful proofreading. For example, "Encoder" in line 65 should be corrected to "Decoder," and the shape of the matrix in line 129 needs clarification. Additionally, there are concerns regarding Figure 2(b), where $l_{\text{raw}}$ decreases as $l_{\text{pred}}$ increases, which seems counterintuitive and requires further explanation.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732197823659

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Comments to All (Part 1)

comment:
We appreciate the constructive comments from reviewers, which have led to significant updates in our paper. The revised parts in the **newly uploaded version** have been highlighted with blue colors for clarity.

For the convenience of the reviewers, we post here the experimental results that more than one reviewer concerns about. Reviewers can find corresponding results associated with our responses according to the labels (e.g. **T1**, **T2**). 

**T1** Efficiency improvement with attention mask fusion,  hyper-parameters are set as batch size = 128, and number of variates = 64. The time complexity is measured by execution time in ms/iteration and memory complexity is measured by memory consumption in GB.

|           |                  | Execution Time(**ms/iter**) | Memory Consumption(**GB**) |
| --------- | ---------------- | --------------------------- | -------------------------- |
| **train** | w. mask fusion   | 163.6661                    | 9.40918                    |
|           | w/o. mask fusion | 213.2196                    | 5.612305                   |
| **infer** | w. mask fusion   | 337.8378                    | 2.793945                   |
|           | w/o. mask fusion | 467.2897                    | 2.331055                   |

**T2** Training efficiency improvement with variate sampling. The number of variates sampled is 64. The time complexity is measured by execution time in ms/iteration and memory complexity is measured by memory consumption in GB. The additional cost of our PTN module is measured by the relative increase compared to backbone models in "inc(%)". Since the fused attention can be also applied to inter-patch attention, we observe speed-up for PatchTST.
|         |                |    PTN-DLI    |           |             |          |    PTN-iTr    |           |            |          |    PTN-Pat    |           |            |           |
| :-----: | :------------: | :-----------: | :-------: | :---------: | :------: | :-----------: | :-------: | :--------: | :------: | :-----------: | :-------: | :--------: | :-------: |
|         |                | time(ms/iter) |  inc(%)   | memory (GB) |  inc(%)  | time(ms/iter) |  inc(%)   | memory(GB) |  inc(%)  | time(ms/iter) |  inc(%)   | memory(GB) |  inc(%)   |
| traffic | w./o. sampling |    150.105    |  710.45   |    6.830    |  563.58  |    149.201    |   82.51   |   7.928    |  62.57   |    79.701     |   31.56   |   3.330    |   33.38   |
|         |  w. sampling   |    10.775     | **51.92** |    0.042    | **3.83** |     9.383     | **36.56** |   0.033    | **2.48** |     4.779     | **15.74** |   -0.139   | **-8.64** |
| weather | w./o. sampling |     9.452     |   86.11   |    0.070    |   8.66   |     4.301     |   30.15   |   0.039    |   3.48   |     4.429     |   24.87   |   -0.137   |   -9.84   |
|         |  w. sampling   |     2.449     | **28.30** |    0.035    | **8.39** |     1.941     | **18.40** |   0.023    | **3.90** |    -0.690     | **-5.28** |   -0.008   | **-1.02** |

**T3** Performance of cost-free student model inference. We use a look-back window with the length of 96 and horizon lengths in {96, 192, 336, 720}. The results are averaged on different horizons. The full results are shown in Table 10 in Appendix A.4.4.

|                  | Dlinear |       | +PTN(stu) |           | PatchTST |           | +PTN(stu) |           | iTrasnformer |           | +PTN(stu) |           |
| ---------------- | ------- | ----- | --------- | --------- | -------- | --------- | --------- | --------- | ------------ | --------- | --------- | --------- |
| metrics          | MSE     | MAE   | MSE       | MAE       | MSE      | MAE       | MSE       | MAE       | MSE          | MAE       | MSE       | MAE       |
| ETTh1(avg)       | 0.446   | 0.434 | **0.444** | **0.430** | 0.469    | **0.455** | **0.460** | **0.455** | **0.454**    | **0.448** | 0.498     | 0.479     |
| ETTh2(avg)       | 0.374   | 0.399 | **0.371** | **0.396** | 0.387    | 0.407     | **0.386** | 0.410     | **0.383**    | **0.407** | 0.437     | 0.440     |
| electricity(avg) | 0.219   | 0.298 | 0.222     | **0.296** | 0.205    | 0.290     | **0.199** | **0.281** | 0.178        | 0.270     | **0.169** | **0.261** |
| traffic(avg)     | 0.627   | 0.378 | 0.639     | **0.369** | 0.481    | 0.300     | 0.491     | **0.286** | 0.428        | 0.282     | 0.442     | **0.274** |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732197899076

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Comments to All (Part 2)

comment:
**T4** Performance of PeMS datasets. We use a look-back window with the length of 96 and horizon lengths in {12, 24, 36, 48}. The results are averaged on different horizons. The full results are shown in Table 14 in Appendix A.6. We achieve comparable performance improvement except for iTransformer, as it is more difficult to converge and easier to overfit.

|             | Dlinear |       | \+PTN     |           | PatchTST |       | \+PTN     |           | iTransformer |           | \+PTN     |           |
| ----------- | ------- | ----- | --------- | --------- | -------- | ----- | --------- | --------- | ------------ | --------- | --------- | --------- |
| metrics     | MSE     | MAE   | MSE       | MAE       | MSE      | MAE   | MSE       | MAE       | MSE          | MAE       | MSE       | MAE       |
| PEMS03(avg) | 0.317   | 0.364 | **0.172** | **0.277** | 0.298    | 0.368 | **0.123** | **0.230** | 0.113        | 0.222     | **0.102** | **0.207** |
| PEMS04(avg) | 0.329   | 0.377 | **0.193** | **0.292** | 0.341    | 0.411 | **0.153** | **0.257** | **0.111**    | **0.221** | 0.121     | 0.225     |
| PEMS07(avg) | 0.319   | 0.368 | **0.182** | **0.283** | 0.293    | 0.366 | **0.120** | **0.218** | **0.101**    | 0.204     | **0.101** | **0.194** |
| PEMS08(avg) | 0.321   | 0.372 | **0.259** | **0.308** | 0.299    | 0.372 | **0.200** | **0.252** | **0.150**    | 0.226     | 0.173     | **0.220** |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732708531898

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers：

Thank you all for your active participation in the discussion phase and for providing valuable and constructive feedback. We have undertaken extensive revisions in response to the reviews. For a brief summary, we would like to list our key revisions below to facilitate further discussions. 

**Summary of the Revisions**

1. (Suggested by Reviewers CnMd, pXSV, gMsX) We have expanded Section A.4.3 with a comprehensive 3-page analysis on the computational complexity of PTN, as well as general attention mechanisms within time series transformers. By introducing an attention fusion mask and variable sampling, we significantly enhance both training and inference efficiency. Incorporating these methods, the complexity of PTN has been reduced to a level no higher than that of the backbone model. 
2. (Inspired by Reviewer gMsX) To address concerns regarding complexity, in Section A.4.4, we present an alternative implementation option: a train-time distillation version of PTN. This variant allows for inference at zero additional cost, albeit with a minor compromise in performance. 
3. (Suggested by Reviewer gMsX) Extensive experiments on PeMS datasets, newly included in Table 14, demonstrate the effectiveness of our approach on complex datasets.
4. (Suggested by Reviewer t7bj) An additional section, A.4.2, has been added to better illustrate the overall pipeline of our method. Additionally, Figure 3 has been revised for a clearer presentation of the architecture.
5. The paper has undergone thorough proofreading, correcting previous typos and clarifying any ambiguous points.
We believe these revisions can enhance our paper in terms of soundness, presentation as well as contribution. 

**Looking Forward to Further Discussions**

We hope that further discussions will be made, particularly regarding the core emphasis of our work. At the heart of our paper is to question the effectiveness of current time series datasets. Our approach involves transforming the original data to investigate whether models can learn more effectively from this transformed data. The conclusion is that, by not relying solely on raw data, models often achieve more accurate predictions even when applied back to the raw data. As is widely acknowledged, the scaling law encompasses three dimensions: computation, parameter size, and data size. While much research focuses on scaling parameters, our work offers a new perspective on scaling data, specifically in terms of quality [1,2]. Although the end-to-end data transformation process is straightforward and admittedly still evolving, we successfully feed models with improved data without resorting to heuristics, even under simple designs. We sincerely hope that reviewers will engage in deeper discussions with us during the prolonged rebuttal phase and provide valuable feedback. We would greatly appreciate it if reviewers could reconsider our work from a fresh angle.

[1] ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws

[2] Scaling Parameter-Constrained Language Models with Quality Data

Best regards!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces the Proximal Transformation Network (PTN), a data-centric plugin for enhancing time series forecasting. PTN is designed to optimize data transformations while maintaining proximity to raw data, with the goal of improving forecasting accuracy and interpretability. The proposed framework integrates a convolutional encoder and an attention-based decoder to generate transformed data for model training. 

The reviewers agreed that this paper offers a novel perspective by redefining the time series forecasting problem as learning data transformation and predicting on transformed data. Reviewers had concerns about the computational complexity of the proposed PTN methods, while the authors managed to address this issue well during the rebuttal. Although the paper presents a promising concept, reviewers still have concerns about motivation, technical quality, presentation, and experimental results after the rebuttal. As such, I am inclined to recommend rejecting this paper.