Title: Retrieval-Augmented Diffusion Models for Time Series Forecasting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Models

Times Series Forecasting. Retrieval-Augmented Mechanism
Abstract: While time series diffusion models have received considerable focus from many recent works, the performance of existing models remains highly unstable. Factors limiting time series diffusion models include insufficient time series datasets and the absence of guidance. To address these limitations, we propose a Retrieval-Augmented Time series Diffusion model (RATD). The framework of RATD consists of two parts: an embedding-based retrieval process and a reference-guided diffusion model. In the first part, RATD retrieves the time series that are most relevant to historical time series from the database as references. The references are utilized to guide the denoising process in the second part. Our approach allows leveraging meaningful samples within the database to aid in sampling, thus maximizing the utilization of datasets. Meanwhile, this reference-guided mechanism also compensates for the deficiencies of existing time series diffusion models in terms of guidance. Experiments and visualizations on multiple datasets demonstrate the effectiveness of our approach, particularly in complicated prediction tasks. Our code is available at https://github.com/stanliu96/RATD

Retrieval-Augmented Diffusion Models for Time Series Forecasting

Abstract

While time series diffusion models have received considerable focus from many recent works, the performance of existing models remains highly unstable. Factors limiting time series diffusion models include insufficient time series datasets and the absence of guidance. To address these limitations, we propose a RetrievalAugmented Time series Diffusion model (RATD). The framework of RATD consists of two parts: an embedding-based retrieval process and a reference-guided diffusion model. In the first part, RATD retrieves the time series that are most relevant to historical time series from the database as references. The references are utilized to guide the denoising process in the second part. Our approach allows leveraging meaningful samples within the database to aid in sampling, thus maximizing the utilization of datasets. Meanwhile, this reference-guided mechanism also compensates for the deficiencies of existing time series diffusion models in terms of guidance. Experiments and visualizations on multiple datasets demonstrate the effectiveness of our approach, particularly in complicated prediction tasks. Our code is available at https://github.com/stanliu96/RATD

1 Introduction

Time series forecasting plays a critical role in a variety of applications including weather forecasting [15, 11], finance forecasting [7, 5], earthquake prediction [19] and energy planning [6]. One way to approach time series forecasting tasks is to view them as conditional generation tasks [32, 42], where conditional generative models are used to learn the conditional distribution P ( x P | x H ) of predicting the target time series x P given the observed historical sequence x H . As the current state-of-the-art conditional generative model, diffusion models [12] have been utilized in many works for time series forecasting tasks [28, 36, 30].

process(such as text guidance or label guidance in image diffusion models). This also limits the potential of time series diffusion models.

Prediction

Ground

Truth

Prediction

Model

Prediction

Historical

RATD

Prediction

Retrieval

Database

Historical

Time Series

Guide

Historical

Time Series

Historical

Time Series

(a)

(b)

(c)

References

CSDI

RATD(Ours)

1.0

- 8°0

1.2

RATD(Ours)

CSDI

MG-TSD

MSE

0.6

0.4

0.2

0

5

10

Test Index

15

20

25

The second limiting factor arises from two shortcomings of the time series datasets: size insufficient and imbalanced . Compared to image datasets, time series datasets typically have a smaller scale. Popular image datasets (such as LAION-400M) contain 400 million sample pairs, while most time series datasets usually only contain tens of thousands of data points. Training a diffusion model to learn the precise distribution of datasets with insufficient size is challenging. Additionally, real-world time series datasets exhibit significant imbalance. For example, in the existing electrocardiogram dataset MIMIC-IV, records related to diagnosed pre-excitation syndrome (PS) account for less than 0.025% of the total records. This imbalance phenomenon may cause models to overlook some extremely rare complex samples, leading to a tendency to generate more common predictions during training, thus making it difficult to handle complex prediction tasks, as illustrated in Figure 1.

To address these limitations, we propose the Retrieval-Augmented Time series Diffusion Model (RATD) for complex time series forecasting tasks. Our approach consists of two parts: the embeddingbased retrieval and the reference-guided diffusion model. After obtaining a historical time series, it is input into the embedding-based retrieval process to retrieve the k nearest samples as references. The references are utilized as guidance in the denoising process. RATD focuses on making maximum utilization of existing time series datasets by finding the most relevant references in the dataset to the historical time series, thereby providing meaningful guidance for the denoising process. RATD focuses on maximizing the utilization of insufficient time series data and to some extent mitigates the issues caused by data imbalance. Meanwhile, this reference-guided mechanism also compensates for the deficiencies of guidance in existing time series diffusion models. Our approach demonstrates strong performance across multiple datasets, particularly on more complex tasks.

To summarize, our main contributions are summarized as follows:

To handle complex time series forecasting, we for the first time introduce RetrievalAugmented Time series Diffusion (RATD), allowing for greater utilization of the dataset and providing meaningful guidance in the denoising process.

Extra Reference Modulated Attention (RMA) module is designed to provide reasonable guidance from the reference during the denoising process. RMA effectively simply integrates information without introducing excessive additional computational costs.

We conducted experiments on five real-world datasets and provided a comprehensive presentation and analysis of the results using multiple metrics. The experimental results demonstrate that our approach achieves comparable or better results compared to baselines.

2 Related Work

2.1 Diffusion Models for Time Series Forecasting

Recent advancements have been made in the utilization of diffusion models for time series forecasting. In TimeGrad [28], the conditional diffusion model was first employed as an autoregressive approach for prediction, with the denoising process guided by the hidden state. CSDI [36] adopted a nonautoregressive generation strategy to achieve faster predictions. SSSD [1] replaced the noise-matching network with a structured state space model for prediction. TimeDiff [30] incorporated future mix-up and autoregressive initialization into a non-autoregressive framework for forecasting. MG-TSD [9] utilized a multi-scale generation strategy to sequentially predict the main components and details of the time series. Meanwhile, mr-diff [31] utilized diffusion models to separately predict the trend and seasonal components of time series. These methods have shown promising results in some prediction tasks, but they often perform poorly in challenging prediction tasks. We propose a retrieval-augmented framework to address this issue.

2.2 Retrival-Augmented Generation

The retrieval-augmented mechanism is one of the classic mechanisms for generative models. Numerous works have demonstrated the benefits of incorporating explicit retrieval steps into neural networks. Classic works in the field of natural language processing leverage retrieval augmentation mechanisms to enhance the quality of language generation [16, 10, 4]. In the domain of image generation, some retrieval-augmented models focus on utilizing samples from the database to generate more realistic images [2, 44]. Similarly, [3] employed memorized similarity information from training data for retrieval during inference to enhance results. MQ-ReTCNN [40] is specifically designed for complex time series forecasting tasks involving multiple entities and variables. ReTime [13] creates a relation graph based on the temporal closeness between sequences and employs relational retrieval instead of content-based retrieval. Although the aforementioned three methods successfully utilize retrieval mechanisms to enhance time series forecasting results, our approach still holds significant advantages. This advantage stems from the iterative structure of the diffusion model, where references can repeatedly influence the generation process, allowing references to exert a stronger influence on the entire conditional generation process.

3 Preliminary

The forecasting task and the background knowledge about the conditional time series diffusion model will be discussed in this section. To avoid conflicts, we use the symbol 's" to represent the time series, and the 't" denotes the t-th step in the diffusion process.

Generative Time Series Forecasting. Suppose we have an observed historical time series x H = { s 1 , s 2 , · · · , s l | s i ∈ R d } , where l is the historical time length, d is the number of features per P

observation and s i is the observation at time step i . The x is the corresponding prediction target { s l +1 , s l +2 , · · · , s l + h | s l + i ∈ R d ′ } ( d ′ ≤ d ) , where h is the prediction horizon. The task of generative time series forecasting is to learn a density p θ ( x P | x H ) that best approximates p ( x P | x H ) , which can be written as:

where θ denotes parameters and D is some appropriate measure of distance between distributions. Given observation x the target time series can be obtained directly by sampling from p θ ( x P | x H ) . Therefore, we obtain the time series { s 1 , s 2 , · · · , s n + h } = [ x H , x P ] .

Conditional Time Series Diffusion Models. With observed time series x H , the diffusion model progressively destructs target time series x P 0 (equals to the x P mentioned in the previous context) by injecting noise, then learns to reverse this process starting from x P T for sample generation. For the convenience of expression, in this paper, we use x t to refer to the t-th time series in the diffusion process, with the letter 'P" omitted. The forward process can be formulated as a Gaussian process with a Markovian structure:

Historical

Time Series

Prediction

…

…

𝑥𝑇

Noise

Database

Retrieval

…

References

𝑥 𝐻

𝑥𝑡

𝑥𝑡-1

𝑥0

…

…

𝜇𝜃

𝒙𝒕-𝟏

= 𝝁𝜽(𝒙𝒕 , 𝒙

𝑹 , 𝒙

𝑯 , 𝑰

𝒔

, 𝒕)

𝑥 𝑅

Reverse

Forward

Retrieval

Historical

Time Series

Database

…

Vector

Similarity

Top k

Indices

Feature Set

Pre-trained

Encoder

Feature

𝑥 𝑅

References

Retrieval Mechanism

𝑥 𝐻

𝑬𝝋

𝑬𝝋

Guide

𝐷 𝑅

…

where β 1 , . . . , β T denotes fixed variance schedule with α t := 1 -β t and α t := ∏ t s =1 α s . This forward process progressively injects noise into data until all structures are lost, which is wellapproximated by N (0 , I ) . The reverse diffusion process learns a model p θ ( x t -1 | x t , x H ) that approximates the true posterior:

where µ θ and Σ θ are often computed by the Transformer. Ho et al. [12] improve the diffusion training process and optimize following objective:

where ˆ µ ( x t , x 0 | x H ) is the mean of the posterior q ( x t -1 | x 0 , x t ) which is a closed from Gaussian, and µ θ ( x t , t | x H ) is the predicted mean of p θ ( x t -1 | x t | x H ) computed by a neural network.

4 Method

We first describe the overall architecture of the proposed method in 4.1. Then we will introduce the strategy of building datasets in Section 4.2. The embedding-based retrieval mechanisms and reference-guided time series diffusion model are introduced in Section 4.3.

4.1 Framework Overview

Figure 2(a) shows the overall architecture of RATD. We built the entire process based on DiffWave [17], which combines the traditional diffusion model framework and a 2D transformer structure. In the forecasting task, RATD first retrieves motion sequences from the database base D R based on the input sequence of historical events. These retrieved samples are then fed into the ReferenceModulated Attention (RMA) as references. In the RMA layer, we integrate the features of the input [ x H , x t ] at time step t with side information I s and the references x R . Through this integration, the references guide the generation process. We will introduce these processes in the following subsections.

4.2 Constructing Retrieval Database for Time Series

Before retrieval, it is necessary to construct a proper database. We propose a strategy for constructing databases from time series datasets with different characteristics. Some time series datasets are sizeinsufficient and are difficult to annotate with a single category label ( e.g. , electricity time series), while some datasets contain complete category labels but exhibit a significant degree of class imbalance ( e.g. , medical time series). We use two different definitions of databases for these two different types of datasets. For the first definition, the entire training set is directly defined as the database D R :

where x i = { s i , · · · , s i + l + h } is the time series with length l + h , and D train is the training set. In the second way, the subset containing samples from all categories in the dataset is defined as the database

Conv1x1 and ReLUv

RMA

GAU

Conv1x1

Conv1x1

Concatenate

+

Feature Transformer Layer

Conv1x1

Conv1x1 and ReLUv

Conv1x1

Conv1x1

Diffusion

Timestep

Embedding

Layer 0

Layer 1

+

Next

Layer

Time Embedding

Feature Embedding

+

+

Output

[

𝒙 𝑯

, 𝒙

𝒕

]

𝑥 𝑅

𝐼

𝑠

[𝑥 𝐻

, 𝑥

𝑡

]

Q(n

×

d)

K(m

×

d)

V(m

×

d)

Softmax

A(d

×

d)

X

Softmax

Z(n

×

d)

X

Conv1x1

𝒙 𝑹

Linear Projection

X

Matrix Multiplication

Side Feature

Reference Feature

Time Series Feature

(a) Model Architecture                                                                                            (b) Reference Modulated Attention

…

Temporal Transformer Layer

D R ′ :

where x k i is the i -th sample in the k -th class of the training set, with a length of l + h . C is the category set of the original dataset. For brevity, we represent both databases as D R .

4.3 Retrieval-Augmented Time Series Diffusion

Embedding-Based Retrieval Mechanism For time forecasting tasks, the ideal references { s i , · · · , s i + h } would be samples where preceding n points { s i -n , · · · , s i -1 } is most relevant to the historical time series { s j , · · · , s j + n } in the D R . In our approach, the overall similarity between time series is of greater concern. We quantify the reference between time series using the distance between their embeddings. To ensure that embeddings can effectively represent the entire time series, pre-trained encoders E ϕ are utilized. E ϕ is trained on representation learning tasks, and the parameter set ϕ is frozen in our retrieval mechanism. For time series (with length n + h ) in D R , their first n points are encoded, thus the D R can be represented as D R emb :

where [ p : q ] refers to the subsequence formed by the p -th point to the q -th point in the time series. The embedding corresponding to the historical time series can be represented as v H = E ϕ ( x H ) . We calculate the distances between v H and all embeddings in D R emb and retrieve the references corresponding to the k smallest distances. This process can be expressed as:

where index ( · ) represents retrieved index given v D . Thus, we obtain a subset x R of D R based on a query x H , i.e. ζ k : x H , D R → x R , where | x R | = k .

Reference-Guided Time Series Diffusion Model In this section, we will introduce our referenceguided time series diffusion model. In the diffusion process, the forward process is identical to the traditional diffusion process, as shown in Equation (2). Following [34, 12, 35] the objective of the reverse process is to infer the posterior distribution p ( z tar | z c ) through the subsequent expression:

where p ( x T | x H ) ≈ N ( x T | x H , I ) , p θ ( x t -1 | x t , x H , x R ) is the reverse transition kernel from x t to x t -1 with a learnable parameter θ . Following most of the literature in the diffusion model, we adopt the assumption:

where µ θ is a deep neural network with parameter θ . After similar computations as those in [12], Σ θ ( x t , x H , x R , t )) in the backward process is approximated as fixed. In other words, we can achieve reference-guided denoising by designing a rational and robust µ θ .

Denoising Network Architecture Similar to DiffWave [17] and CSDI [36], our pipeline is constructed on the foundation of transformer layers, as shown in Figure 3. However, the existing framework cannot effectively utilize the reference as guidance. Considering attention modules to integrate the x R and x t as a reasonable intuition, we propose a novel module called Reference Modulated Attention (RMA). Unlike normal attention modules, we realize the fusion of three features in RMA: the current time series feature, the side feature, and the reference feature. To be specific, RMA was set at the beginning of each residual module Figure 3. We use 1D-CNN to extract features from the input x t , references x R , and side information. Notably, we concatenate all references together for feature extraction. Side information consists of two parts, representing the correlation between variables and time steps in the current time series dataset Appendix B. We adjust the dimensions of these three features with linear layers and fuse them through matrix dot products. Similar to text-image diffusion models [29], RMA can effectively utilize reference information to guide the denoising process, while appropriate parameter settings prevent the results from overly depending on the reference.

Training Procedure To train RATD ( i.e. , optimize the evidence lower bound induced by RATD), we use the same objective function as previous work. The loss at time step t -1 are defined as follows respectively:

where ˆ x 0 are predicted from x t , and γ t = ¯ α t -1 β 2 t 2 ˜ β 2 t (1 -¯ α t ) 2 are hyperparameters in diffusion process. We summarize the training procedure of RATD in Algorithm 1 and highlight the differences from the conventional models, in cyan. The process of sampling is shown in Appendix A.

Algorithm 1 Training Procedure of RATD

5 Experiments

5.1 Experimental Setup

Datasets Following previous work [45, 38, 8, 30], experiments are performed on four popular real-world time series datasets: (1) Electricity * , which includes the hourly electricity consumption data from 321 clients over two years.; (2) Wind [20], which contains wind power records from 2020-2021. (3) Exchange [18], which describes the daily exchange rates of eight countries (Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore); (4) Weather † , which documents 21 meteorological indicators at 10-minute intervals spanning from 2020 to 2021.; Besides, we also applied our method to a large ECG time series dataset: MIMIC-IV-ECG [14]. The MIMICIV-ECG dataset contains clinical electrocardiogram data from over 190,000 patients and 450,000 hospitalizations at Beth Israel Deaconess Medical Center (BIDMC).

Baseline Methods To comprehensively demonstrate the effectiveness of our method, we compare RATD with four kinds of time series forecasting methods. Our baselines include (1) Time series diffusion models, including CSDI [36], mr-Diff [31], D 3 VAE [20], TimeDiff [30]; (2) Recent time series forecasting methods with frequency information, including FiLM [46], Fedformer [47] and FreTS [41] ; (3) Time series transformers, including PatchTST [25], Autoformer [38], Pyraformer [22], Informer [45] and iTransformer [23]; (4) Other popular methods, including TimesNet [39], SciNet [21], Nlinear [43], DLinear [43] and NBeats [26].

Evaluation Metric To comprehensively assess our proposed methodology, our experiment employs three metrics: (1) Probabilistic forecasting metrics: Continuous Ranked Probability Score (CRPS) on each time series dimension [24]. (2) Distance metrics: Mean Squared Error (MSE), and Mean Average Error(MAE) are employed to measure the distance between predictions and ground truths.

Implementation Details The length of the historical time series was 168, and the prediction lengths were (96, 192, 336), with results averaged. All experiments were conducted on an Nvidia RTX A6000 GPU with 40GB memory. During the experiments, the second strategy of conducting D R was employed for the MIMIC dataset, while the first strategy was utilized for the other four datasets. To reduce the training cost, we preprocessed the retrieval process by storing the reference indices of each sample in the training set in a dictionary. During the training on the diffusion model, we accessed this dictionary directly to avoid redundant retrieval processes. More details are shown in Appendix B.

5.2 Main Results

Table 1 presents the primary results of our experiments on four daily datasets. Our approach surpasses existing time series diffusion models. Compared to other time series forecasting methods, our approach exhibits superior performance on three out of four datasets, with competitive performance on the remaining dataset. Notably, we achieve outstanding results on the wind dataset. Due to the lack of clear short-term periodicity (daily or hourly), some prediction tasks in this dataset are exceedingly challenging for other models. Retrieval-augmented mechanisms can effectively assist in addressing these challenging prediction tasks.

Figure 4 presents a case study randomly selected from our experiments on the wind dataset. We compare our prediction with iTransformer and two popular open-source time series diffusion models, CSDI and D 3 VAE. Although CSDI and D 3 VAE provide accurate predictions in the initial short-term period, their long-term predictions deviate significantly from the ground truth due to the lack of guidance. ITransformer captures rough trends and periodic patterns, yet our method offers higherquality predictions than the others. Furthermore, through the comparison between the predicted

Reference

RATD

CSDI

iTransformer

D 3  VAE

history

history

history

prediction

prediction

prediction

50

100

150

200

250

300

50

100

150

200

250

300

50

100

150

200

250

300

history

prediction

3

2

50

100

150

200

250

300

50

100

150

200

250

300

results and references in the figure, although references provide strong guidance, they do not explicitly substitute for the entire generated results. This further validates the rationality of our approach.

Table 2 presents the testing results of our method on the MIMIC-IV-ECG dataset. We selected some powerful open-source methods as baselines for comparison. Our experiments are divided into two parts: in the first part, we evaluate the entire test set, while in the second part, we select rare cases (those accounting for less than 2% of total cases) from the test set as a subset for evaluation. Prediction tasks in the second part are more challenging for deep models. In the first experiment, our method achieved results close to iTransformer, while in the second task, our model significantly outperformed other methods, demonstrating the effectiveness of our approach in addressing challenging tasks.

5.3 Model Analysis

Influence of Retrieval Mechanism To investigate the impact of the retrieval augmentation mechanism on the generation process, we conducted an ablation study and presented the results in Table 3. The study addresses two questions: whether the retrieval augmentation mechanism is effective and which retrieval method is most effective. Firstly, we removed our retrieval augmentation mechanism from the RATD as a baseline. Besides, the model with random time series guidance is another baseline. The references retrieved by other methods have all positively impacted the prediction results. This suggests that reasonable references are highly effective in guiding the generation process.

We also compared two different retrieval mechanisms: correlation-based retrieval and embeddingbased retrieval. The first method directly retrieves the reference in the time domain ( e.g. , using Dynamic Time Warping (DTW) or Pearson correlation coefficient). Our approach adopts the second mechanism: retrieving references through the embedding of time series. From the results, the correlation-based methods are significantly inferior to the embedding-based methods. The former methods fail to capture the key features of the time series, making it difficult to retrieve the best references for forecasting. We also evaluate the embedding-based methods with various encoders for comparison. The comprehensive results show that methods with different encoders do not significantly differ. This indicates that different methods can all extract meaningful references, thereby producing similar improvements in results. TCN was utilized in our experiment because TCN strikes the best balance between computational cost and performance.

Effect of Retrieval Database We conducted an ablation study on two variables, n and k , to investigate the influence of the retrieval database D R in RATD, where n represents the number of samples in each category of the database, and k represents the number of reference exemplars. The results in Figure 5q can benefit the model in terms of prediction accuracy because a larger D R brings higher diversity, thereby providing more details beneficial for prediction and enhancing the generation

references.

process. Simply increasing k does not show significant improvement, as utilizing more references may introduce more noise into the denoising process. In our experiment, the settings of n and k are 256 and 3, respectively.

Inference Efficiency In this experiment, we evaluate the inference efficiency of the proposed RATD in comparison to other baseline time series diffusion models (TimeGrad, MG-TSD, SSSD). Figure 6 illustrates the inference time on the multivariate weather dataset with varying values of the prediction horizon ( h ). While our method introduces an additional retrieval module, the sampling efficiency of the RATD is not low due to the non-autoregressive transformer framework. It even slightly outperforms other baselines across all h values. Notably, TimeGrad is observed to be the slowest, attributed to its utilization of auto-regressive decoding.

0

100

200

300

400

500

n

0.175

0.180

0.185

0.190

0.195

MSE Value

k=5

100

200

300

400

500

600

700

0

2000

4000

6000

8000

10000

12000

Inference Time (ms)

MG-TSD

TimeGrad

RATD

SSSD

Effectiveness of Reference Modulated Attention To validate the effectiveness of the proposed RMA, we designed additional ablation experiments. In these experiments, we used the CSDI architecture as the baseline method and added extra fusion modules to compare the performance of these modules (linear layer, cross-attention layer, and RMA). The results are shown in the Table 4.

Through our experiments, we found that compared to the basic cross-attention-based approach, RMAcan integrate an edge information matrix (representing correlations between time and feature dimensions) more effectively. The extra fusion is highly beneficial in experiments, guiding the model to capture relationships between different variables. In contrast, linear-based methods concatenate inputs and references initially, which prevents the direct extraction of meaningful information from references, resulting in comparatively modest performance.

Predicting x 0 vs Predicting ϵ . Following the formulation in Section 4.3, our network is designed to forecast the latent variable x 0 . Since some existing models [28, 36] have been trained by predicting an additional noise term ϵ , we conducted a comparative experiment to determine which approach is more suitable for our framework. Specifically, we maintained the network structure unchanged, only modifying the prediction target to be ϵ . The results are presented in Table 5. Predicting x 0 proves to be more effective. This may be because the relationship between the reference and x o is more direct, making the denoising task relatively easier.

RMA position We investigate the best position of RMA in the model. Front, middle, and back means we set the RMA in the front of, in the middle of, and the back of two transformer layers, respectively. We found that placing RMA before the bidirectional transformer resulted in the most significant improvement in model performance. This also aligns with the intuition of network design: cross-attention modules placed at the front of the model tend to have a greater impact.

6 Discussion

Limitation and Future Work As a transformer-based diffusion model structure, our approach still faces some challenges brought by the transformer framework. Our model consumes a significant amount of computational resources dealing with time series consisting of too many variables. Additionally, our approach requires additional preprocessing (retrieval process) during training, which incurs additional costs on training time (around ten hours).

Conclusion In this paper, we propose a new framework for time series diffusion modeling to address the forecasting performance limitations of existing diffusion models. RATD retrieves samples most relevant to the historical time series from the constructed database and utilize them as references to guide the denoising process of the diffusion model, thereby obtaining more accurate predictions. RATD is highly effective in solving challenging time series prediction tasks, as evaluated by experiments on five real-world datasets.

References

Juan Miguel Lopez Alcaraz and Nils Strodthoff. Diffusion-based time series imputation and forecasting with structured state space models. arXiv preprint arXiv:2208.09399 , 2022.

Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Retrievalaugmented diffusion models. Advances in Neural Information Processing Systems , 35:1530915324, 2022.

Giovanni Bonetta, Rossella Cancelliere, Ding Liu, and Paul Vozila. Retrieval-augmented transformer-xl for close-domain dialog generation. arXiv preprint arXiv:2105.09235 , 2021.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning , pp. 2206-2240. PMLR, 2022.

Jian Cao, Zhi Li, and Jian Li. Financial time series forecasting model based on ceemdan and lstm. Physica A: Statistical mechanics and its applications , 519:127-139, 2019.

Jui-Sheng Chou and Duc-Son Tran. Forecasting energy consumption time series using machine learning techniques based on usage patterns of residential householders. Energy , 165:709-726, 2018.

Alexiei Dingli and Karl Sant Fournier. Financial time series forecasting-a deep learning approach. International Journal of Machine Learning and Computing , 7(5):118-122, 2017.

Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie-Yan Liu. Depts: deep expansion learning for periodic time series forecasting. arXiv preprint arXiv:2203.07681 , 2022.

Xinyao Fan, Yueying Wu, Chang Xu, Yuhao Huang, Weiqing Liu, and Jiang Bian. Mg-tsd: Multi-granularity time series diffusion models with guided learning process. arXiv preprint arXiv:2403.05751 , 2024.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning , pp. 3929-3938. PMLR, 2020.

Pradeep Hewage, Ardhendu Behera, Marcello Trovati, Ella Pereira, Morteza Ghahremani, Francesco Palmieri, and Yonghuai Liu. Temporal convolutional neural (tcn) network for an effective weather forecasting using time-series data from the local weather station. Soft Computing , 24:16453-16482, 2020.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Baoyu Jing, Si Zhang, Yada Zhu, Bin Peng, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. Retrieval based time series forecasting. arXiv preprint arXiv:2209.13525 , 2022.

Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data , 10(1):1, 2023.

Zahra Karevan and Johan AK Suykens. Transductive lstm for time-series prediction: An application to weather forecasting. Neural Networks , 125:1-9, 2020.

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172 , 2019.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations , 2020.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pp. 95-104, 2018.

S Sri Lakshmi and RK Tiwari. Model dissection from earthquake time series: A comparative analysis using modern non-linear forecasting and artificial neural network approaches. Computers & Geosciences , 35(2):191-204, 2009.

Yan Li, Xinjiang Lu, Yaqing Wang, and Dejing Dou. Generative time series forecasting with diffusion, denoise, and disentanglement. Advances in Neural Information Processing Systems , 35:23009-23022, 2022.

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems , 35:5816-5828, 2022.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations , 2021.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 , 2023.

James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management science , 22(10):1087-1096, 1976.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 , 2022.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 , 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning , pp. 8857-8868. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 10684-10695, 2022.

Lifeng Shen and James Kwok. Non-autoregressive conditional diffusion models for time series prediction. arXiv preprint arXiv:2306.05043 , 2023.

Lifeng Shen, Weiyu Chen, and James Kwok. Multi-resolution diffusion models for time series forecasting. In The Twelfth International Conference on Learning Representations , 2023.

Zhipeng Shen, Yuanming Zhang, Jiawei Lu, Jun Xu, and Gang Xiao. Seriesnet: a generative time series forecasting model. In 2018 international joint conference on neural networks (IJCNN) , pp. 1-8. IEEE, 2018.

Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pp. 3531-3539, 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pp. 2256-2265. PMLR, 2015.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems , 34:24804-24816, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations , 2022.

Sitan Yang, Carson Eisenach, and Dhruv Madeka. Mqretnn: Multi-horizon time series forecasting with retrieval augmentation. arXiv preprint arXiv:2207.10517 , 2022.

Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mlps are more effective learners in time series forecasting. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems , 32, 2019.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pp. 11121-11128, 2023.

Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmented motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 364-373, 2023.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pp. 11106-11115, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems , 35:12677-12690, 2022.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning , pp. 27268-27286. PMLR, 2022.

A Sampling Procedure

Like Algorithm 1, we summarize the sampling procedure of RATD in Algorithm 2 and highlight the differences from conventional diffusion models in .

Algorithm 2 Sampling Procedure of RATD Require: The historical time series x H , the learned model µ θ , external database D R , pre-trained E ϕ , the number of references k Ensure: Prediction x P corresponding to the history x H 1: Sample initial target time series x T 2: Embed x H into v H 3: Retrieval the reference x R with v H 4: Compute the side feature I s 5: for t in T, T -1 , . . . , 1 do 6: Predict ˆ x 0 from x t , I s and x R (Equation (10)) 7: Sample x t -1 from the posterior q ( x t | x 0 ) (Equation (2)) 8: end for

B Impletion Details

B.1 Training Details

Our dataset is split in the proportion of 7:1:2 (Train: Validation: Test), utilizing a random splitting strategy to ensure diversity in the training set. We sample the ECG signals at 125Hz for the MIMIC-IV dataset and extract fixed-length windows as samples. For training, we utilized the Adam optimizer with an initial learning rate of 10 -3 , betas = (0 . 95 , 0 . 999) . During the training process of shifted diffusion, the batch size was set to 64, and early stopping was applied for a maximum of 200 epochs. The diffusion steps T were set to 100.

B.2 Side Information

We combine temporal embedding and feature embedding as side information v s . We use 128dimensions temporal embedding following previous studies [37]:

where τ = 10000 . Following [36], s l represents the timestamp corresponding to the l-th point in the time series. This setup is designed to capture the irregular sampling in the dataset and convey it to the model. Additionally, we utilize learnable embedding to handle feature dimensions. Specifically, feature embedding is represented as 16-dimensional learnable vectors that capture relationships between dimensions. According to [17], we combine time embedding and feature embedding, collectively referred to as side information I s .

The shape of I s is not fixed and varies with datasets. Taking the Exchange dataset as an example, the shape of forecasting target x R is [Batchsize (64), 7(number of variables), 168 (time-dimension), 12 (time-dimension)] and the corresponding shape of I s is [Batchsize (64), total channel(144( time:128 + feature:16)), 320 (frequency-dimension*latent channel), 12 (time-dimension)].

B.3 Transformers Details

Our approach employs the Transformer architecture from CSDI, with the distinction of expanding the channel dimension to 128. The network comprises temporal and feature layers, ensuring the comprehensiveness of the model in handling the time-frequency domain latent while maintaining a relatively simple structure. Regarding the transformer layer, we utilized a 1-layer Transformer encoder implemented in PyTorch [27], comprising multi-head attention layers, fully connected layers, and layer normalization. We adopted the "linear attention transformer" package ‡ , to enhance

computational efficiency. The inclusion of numerous features and long sequences prompted this decision. The package implements an efficient attention mechanism [33], and we exclusively utilized the global attention feature within the package.

B.4 Metrics

We will introduce the metrics in our experiments. We summarize them as below:

CRPS. CRPS [24] is a univariate strictly proper scoring rule which ´ measures the compatibility of a cumulative distribution function F with an observation x as:

MAEand MSE. MAE and MSE are calculated in the formula below, ˆ x P represents the predicted time series, and x P represents the ground truth time series. MAE calculates the average absolute difference between predictions and true values, while MSE calculates the average squared difference between predictions and true values. A smaller MAE or MSE implies better predictions.

where ✶ ( x ≤ y ) is the indicator function, which is 1 if x ≤ y and 0 otherwise. The CRPS attains the minimum value when the predictive distribution F same as the data distribution.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction include the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our approach at Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the information needed in the Experiment part.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The full code will be uploaded in the future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the details about the dataset in appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not report the error bars in our experiemnt.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provide sufficient information on the computer resources in paragraph implement details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no obvious societal impact of the work performed

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the original work are sited properly.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed Retrieval-Augmented Time series Diffusion (RATD) model for complex time series forecasting tasks, and designed an Extra Reference Modulated Attention (RMA) module to enable the guidance from the retrieved reference time series during the denoising process of the diffusion models. On five real-world datasets, this paper demonstrated that RATD performed on-par or better on multiple metrics compared to various time series forecasting baselines including diffusion models, transformers, and other methods.\n\nSTRENGTHS:\n1. This paper is well-written and easy to follow.\n2. The methodology design is clear and reasonable to me.\n3. The experiments are comprehensive, and demonstrated the effectiveness of RATD across various retrieval mechanisms on multiple time series datasets using multiple metrics.\n4. In the experiment results, RATD is effective with k = 3 retrieved reference time series, and the inference time is also on the lower side compared to other baselines, which showed that RATD should be relatively computationally scalable and efficient.\n\nWEAKNESSES:\n1. Based on the related work section, there have been several works which investigated  time series forecasting using diffusion models and achieved promising results. RAG is also a well-studied topic in text generation. Given this, I think the technical novelty of this work is moderate, though it's a nice direction to combine RAG with time series forecasting and the experiment results demonstrated its effectiveness. I am wondering whether there is any previous work on time series RAG?\n2. Probably out of the scope of this paper, it would be great if the authors can provide some discussion or intuitive insights on the advantage of using diffusion models for time series forecasting compared to other methods, e.g. is it more robust or generalizable, and also the advantage of diffusion models when it comes to retrieval augmented forecasting. As in table 1, several transformer based methods also performed well on some datasets, thus I am wondering whether there is any justification in choosing diffusion models for time series forecasting given its potential extra computational cost.\n\nQUESTIONS:\n1. In section 4.2, how is $D^{R}$ different from $D^{R'}$ if $\\mathcal{C}$ represents all categories? In 5.3, it seems that when it's $D^{R'}$, only $n$ samples from each category are used to construct the database. Is $n$ a hyperparameter for $D^{R'}$ that also needs to be introduced in section 4.2?\n2. In section 4.3, under $\\textbf{Denoising Network Architecture}$, it is mentioned that all references are concatenated together for feature extraction. I am wondering how this scales with the length of time series, i.e. would there be memory concerns if you are working with long-horizon time series forecasting / long context windows?\n\nLIMITATIONS:\nThe authors discussed the limitations of this work with regards to computational costs incurred by the transformer based diffusion model and the retrieval process, which is also my concern.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Retrieval-Augmented Time series Diffusion model (RATD) to address the limitations of existing time series diffusion models, such as insufficient datasets and lack of guidance. RATD consists of two components: an embedding-based retrieval process and a reference-guided diffusion model. The model retrieves the most relevant time series from a database to guide the denoising process, thereby enhancing the utilization of the dataset and providing meaningful guidance. Experiments on multiple datasets demonstrate the effectiveness of RATD, particularly in complex prediction tasks.\n\nSTRENGTHS:\n- **Novel Approach**: The idea of leveraging retrieved similar historical time series to guide forecasting for current timesteps is novel. References can potentially accelerate the convergence of the diffusion process and improve prediction accuracy.\n- **Guidance Mechanism**: The reference-guided mechanism compensates for the lack of guidance in existing time series diffusion models, which is a significant contribution to the field.\n\nWEAKNESSES:\n- **Incomplete Experimental Comparisons**: The experimental setup and comparisons are insufficient and not aligned with existing studies, making it difficult to verify the performance and correct implementation of this work. For example, the datasets and forecasting horizons used differ from those in classical studies like CSDI and TimeGrad. This discrepancy makes it challenging to ensure fair comparisons and reproducibility.\n  - In CSDI and TimeGrad, solar, electricity, traffic, taxi, wiki datasets and forecasting horizon=24 are used for comparisons. While this work adopts datasets of exchange, wind, electricity, and weather and only report the average results of horizons of 96, 192, and 336. \n  - Given that the code is also not provided at the current stage, though I like the idea, I am not sure about the reproducibility. Hence I suggest the author to align with existing classical studies and provide a more comprehensive experimental results.\n- **Insufficient Ablation Tests**: While Table 3 indicates the importance of choosing good retriever embeddings, there are no systematic ablation tests of the architecture developed (Figure 3). Detailed explanations of model architectures and extensive ablation tests are crucial to understanding the success of conditional diffusion and ensuring computational efficiency. The current paper lacks this information. It is easy for me to understand the conditional diffusion process. but it is extremely challenging for me to understand the insights and tricks in the architectures shown in Figure 3. I believe these are crucial to ensure the success of conditional diffusion and provide efficient computation. I cannot find any contents in the main paper or appendix informing me about these.\n  - Detailed explanation of model architectures and extensive ablation tests are important for me to effectively distinguish it from CSDI. For example, you can still follow the architecture of CSDI while only introducing the depdence on retrived references x^R.\n- **Unknown GPU Memory Usage**: There is no information on GPU memory usage and computation efficiency. Understanding how the introduction of reference series impacts computation and how these challenges are addressed by specific architectural designs is important for evaluating the practicality of the model.\n\nQUESTIONS:\n- More comprehensive and detailed comparisons with existing studies\n  - Covering necessary datasets and both short and long horizons\n- Systematic ablation tests to illustrate your specific architecture designs\n  - Especially for Figure 3, more explanations and ablation tests are indispensable for readers to catch up\n\nLIMITATIONS:\nThe idea of retrieval-augmented diffusion time-series model is interesting and reasonable.\nMy major concerns are about the incomplete experiments and the reproducibility.\n\nI may consider raising the score if my major concerns are properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a retrieval-augmented diffusion model for time series prediction, featuring a simple and straightforward approach. It aims to address two key issues: \n\n1. The lack of semantics and labels in time series data, leading to insufficient guidance during the diffusion model's generation process. \n\n2. The problem of size insufficiency and imbalance in time series data. \n\nSpecifically, the method involves embedding-based retrieval to gather similar time series data from a database to serve as references during the denoising process. Their proposed RMA module performs feature fusion between the retrieved time series and the current diffusion step's results, thereby enhancing the denoising and generation process of the diffusion model.  \n\nIn the experimental section, they explore the impact of different retrieval methods on the performance of the diffusion model. They also investigate the effects of varying the size of the retrieval database and the number of retrieved sequences (k) on the prediction performance.\n\nSTRENGTHS:\nAn additional database was utilized to enhance the diffusion generation process. This method is straightforward and easy to follow. The newly proposed RMA module aids in integrating features from different datasets. The paper is clearly written and achieves state-of-the-art (SOTA) results on the majority of the datasets they selected.\n\nWEAKNESSES:\nThe paper lacks innovation as the retrieval-enhanced diffusion method has already been proposed in the field of image generation, and similar retrieval-enhanced approaches exist in the time-series prediction domain, although their backbone is not a diffusion model. Additionally, the performance improvement largely stems from the representational capabilities of the pre-trained embedding model used.\n\nTypo in title: \"Augment\" -> \"augument\".\n\nQUESTIONS:\n1. What would be the result if the retrieved sequences were directly used for a simple weighted average for the final generation output, or if the performance of selecting the sequence closest to the ground truth (gt) from the retrieved ones was evaluated?\n2. I am uncertain whether the non-diffusion time-series prediction methods used in the author's comparative experiments include retrieval-enhanced modules. If not, I recommend adding experiments with this component to demonstrate the efficacy of the proposed method.\n3. In Formula 1, the symbol $\\mathcal{D}$ is used ambiguously, representing both distance and feature dimensions.\n\nLIMITATIONS:\nThe approach may heavily rely on the capabilities of the embedding model and the construction of the database.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting time series diffusion models are unstable due to insufficient datasets and lack of guidance. The RATD model combines an embedding-based retrieval process with a reference-guided diffusion model to improve stability and accuracy. RATD retrieves relevant time series from a database to guide the denoising process, maximizing dataset utilization and compensating for guidance deficiencies. Experiments on multiple datasets demonstrate RATD's effectiveness, particularly in complex prediction tasks.\n\nSTRENGTHS:\n- The paper introduces a creative combination of embedding-based retrieval and reference-guided diffusion. This approach is innovative in addressing the limitations of existing time series diffusion models.\n- The authors conducted extensive experiments on multiple datasets, demonstrating the effectiveness of RATD in complex prediction tasks. The results show that RATD outperforms existing methods in terms of stability and accuracy.\nThe paper provides a thorough explanation of the model architecture, retrieval mechanism, and training procedure, ensuring reproducibility.\n- The paper is well-organized, with clear sections on introduction, related work, methodology, experiments, and conclusions. The use of figures and tables enhances understanding.\n- The authors explain complex concepts in a concise manner, making the paper accessible to readers with varying levels of expertise.\n\nWEAKNESSES:\n- The proposed RATD lacks justification for architecture.\n- Please refer to the below questions.\n\nQUESTIONS:\n- What is the main difference between RAG and proposed work?\n- How to define stability in time-series data? Someone may not agree with stability in Figure 1-(c). Some may disagree that the red plots are unstable.\n- Why necessarily the proposed RATD is needed in time-series forecasting? In other words, how to link the relationship between RATD and TSAD?\n- There is a typo in the last sentence of the second paragraph from Section 5.2. “.t”\n- Why Table 2 is separated from Table 1 independently?\n- What can you define as the best retrieval for forecasting?\n\nLIMITATIONS:\nThey described their limitations themselves.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a retrieval-augmented time series diffusion model that uses an embedding-based retrieval process and a reference-guided diffusion model. The proposed RATD retrieves relevant time series from an external database as references, which is later utilized to guide the denoising process of the diffusion model for future forecasting. Extensive experiments are conducted on five real-world datasets that demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS:\n- The paper is clearly written and easy to understand. The authors provide clear and detailed explanations and illustrations for the proposed architecture.\n- The idea of applying retrieval augmentation on the diffusion model for time series forecasting is novel and insightful.\n\nWEAKNESSES:\n- The proposed architecture requires retrieval database construction and pretraining the time series encoder $E_\\phi$ before training the forecasting model. The additional complexity and computation cost may limit the efficiency of the proposed method.\n- Experimental results do not contain standard deviation, which potentially limits the confidence and significance of the performance superiority.\n- Some baselines are missing. For instance, aurhors mentioned MQ-ReTCNN and ReTime as retrieval-augemented time series models. However, they are not invovled as baselines in experiments.\n\nQUESTIONS:\n- How is the time series encoder $E_\\phi$ pretrained with representation learning tasks (Line148)? Since the embedding quality highly affects the retrieval precision and quality, some analysis and evaluations should be conducted on the time series embedding quality.\n- What is the architecture of the pre-trained encoder $E_\\phi$? How do you encode the multivariate time series into a single embedding? Do you use any pooling operation? More analysis and explanations on the encoder are beneficial for paper clarity.\n\nLIMITATIONS:\nYes. The authors discussed the limitations of the proposed approach in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}