Retrieval-Augmented Diffusion Models for Time Series Forecasting

Authors: jingwei  liu
ling  yang
hongyan  li
‡ shenda hong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:33:38

Abstract: While time series diffusion models have received considerable focus from many recent works, the performance of existing models remains highly unstable. Factors limiting time series diffusion models include insufficient time series datasets and the absence of guidance. To address these limitations, we propose a Retrieval-Augmented Time series Diffusion model (RATD). The framework of RATD consists of two parts: an embedding-based retrieval process and a reference-guided diffusion model. In the first part, RATD retrieves the time series that are most relevant to historical time series from the database as references. The references are utilized to guide the denoising process in the second part. Our approach allows leveraging meaningful samples within the database to aid in sampling, thus maximizing the utilization of datasets. Meanwhile, this reference-guided mechanism also compensates for the deficiencies of existing time series diffusion models in terms of guidance. Experiments and visualizations on multiple datasets demonstrate the effectiveness of our approach, particularly in complicated prediction tasks. Our code is available at https://github.com/stanliu96/RATD

Main paper: 
 introduction: 
 Time series forecasting plays a critical role in a variety of applications including weather forecasting [15, 11] , finance forecasting [7, 5] , earthquake prediction [19] and energy planning [6] . One way to approach time series forecasting tasks is to view them as conditional generation tasks [32, 42] , where conditional generative models are used to learn the conditional distribution P (x P |x H ) of predicting the target time series x P given the observed historical sequence x H . As the current state-of-the-art conditional generative model, diffusion models [12] have been utilized in many works for time series forecasting tasks [28, 36, 30] .Although the performance of the existing time series diffusion models is reasonably well on some time series forecasting tasks, it remains unstable in certain scenarios (an example is provided in 1(c)). The factors limiting the performance of time series diffusion models are complex, two of them are particularly evident. First, most time series lack direct semantic or label correspondences, which often results in time series diffusion models lacking meaningful guidance during the generation Figure 1: (a) The figure shows the differences in forecasting results between the CSDI [36] (left) and RATD (right). Due to the very small proportion of such cases in the training set, CSDI struggles to make accurate predictions, often predicting more common results. Our method, by retrieving meaningful references as guidance, makes much more accurate predictions. (b) A comparison between our method's framework(bottom) and the conventional time series diffusion model framework(top). (c) We randomly selected 25 forecasting tasks from the electricity dataset. Compared to our method, CSDI and MG-TSD [9] exhibited significantly higher instability. This indicates that the RATD is better at handling complex tasks that are challenging for the other two methods.The second limiting factor arises from two shortcomings of the time series datasets: size insufficient and imbalanced. Compared to image datasets, time series datasets typically have a smaller scale. Popular image datasets (such as LAION-400M) contain 400 million sample pairs, while most time series datasets usually only contain tens of thousands of data points. Training a diffusion model to learn the precise distribution of datasets with insufficient size is challenging. Additionally, real-world time series datasets exhibit significant imbalance. For example, in the existing electrocardiogram dataset MIMIC-IV, records related to diagnosed pre-excitation syndrome (PS) account for less than 0.025% of the total records. This imbalance phenomenon may cause models to overlook some extremely rare complex samples, leading to a tendency to generate more common predictions during training, thus making it difficult to handle complex prediction tasks, as illustrated in Figure 1 .Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction include the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work 2.1 diffusion models for time series forecasting: 
 Recent advancements have been made in the utilization of diffusion models for time series forecasting.In TimeGrad [28] , the conditional diffusion model was first employed as an autoregressive approach for prediction, with the denoising process guided by the hidden state. CSDI [36] adopted a nonautoregressive generation strategy to achieve faster predictions. SSSD [1] replaced the noise-matching network with a structured state space model for prediction. TimeDiff [30] incorporated future mix-up and autoregressive initialization into a non-autoregressive framework for forecasting. MG-TSD [9] utilized a multi-scale generation strategy to sequentially predict the main components and details of the time series. Meanwhile, mr-diff [31] utilized diffusion models to separately predict the trend and seasonal components of time series. These methods have shown promising results in some prediction tasks, but they often perform poorly in challenging prediction tasks. We propose a retrieval-augmented framework to address this issue.The retrieval-augmented mechanism is one of the classic mechanisms for generative models. Numerous works have demonstrated the benefits of incorporating explicit retrieval steps into neural networks. Classic works in the field of natural language processing leverage retrieval augmentation mechanisms to enhance the quality of language generation [16, 10, 4] . In the domain of image generation, some retrieval-augmented models focus on utilizing samples from the database to generate more realistic images [2, 44] . Similarly, [3] employed memorized similarity information from training data for retrieval during inference to enhance results. MQ-ReTCNN [40] is specifically designed for complex time series forecasting tasks involving multiple entities and variables. ReTime [13] creates a relation graph based on the temporal closeness between sequences and employs relational retrieval instead of content-based retrieval. Although the aforementioned three methods successfully utilize retrieval mechanisms to enhance time series forecasting results, our approach still holds significant advantages. This advantage stems from the iterative structure of the diffusion model, where references can repeatedly influence the generation process, allowing references to exert a stronger influence on the entire conditional generation process.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our approach at Section 6 Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 The forecasting task and the background knowledge about the conditional time series diffusion model will be discussed in this section. To avoid conflicts, we use the symbol "s" to represent the time series, and the "t" denotes the t-th step in the diffusion process.Generative Time Series Forecasting. Suppose we have an observed historical time seriesx H = {s 1 , s 2 , • • • , s l | s i ∈ R d },where l is the historical time length, d is the number of features per observation and s i is the observation at time step i. The x P is the corresponding prediction target{s l+1 , s l+2 , • • • , s l+h | s l+i ∈ R d ′ } (d ′ ≤ d), where h is the prediction horizon. The task of generative time series forecasting is to learn a density p θ (x P |x H ) that best approximates p(x P |x H ), which can be written as: minEQUATIONwhere θ denotes parameters and D is some appropriate measure of distance between distributions. Given observation x the target time series can be obtained directly by sampling from p θ (x P |x H ). Therefore, we obtain the time series{s 1 , s 2 , • • • , s n+h } = [x H , x P ].Conditional Time Series Diffusion Models. With observed time series x H , the diffusion model progressively destructs target time series x P 0 (equals to the x P mentioned in the previous context) by injecting noise, then learns to reverse this process starting from x P T for sample generation. For the convenience of expression, in this paper, we use x t to refer to the t-th time series in the diffusion process, with the letter "P" omitted. The forward process can be formulated as a Gaussian process with a Markovian structure: The historical time series x H is inputted into the retrieval module to for the corresponding references x R . After that, x H is concatenated with the noise as the main input for the model µ θ . x R will be utilized as the guidance for the denoising process.EQUATIONwhere β 1 , . . . , β T denotes fixed variance schedule with α t := 1 -β t and α t := t s=1 α s . This forward process progressively injects noise into data until all structures are lost, which is wellapproximated by N (0, I). The reverse diffusion process learns a model p θ (x t-1 |x t , x H ) that approximates the true posterior:EQUATIONwhere µ θ and Σ θ are often computed by the Transformer. Ho et al. [12] improve the diffusion training process and optimize following objective:EQUATIONwhere μ(x t , x 0 |x H ) is the mean of the posterior q(x t-1 |x 0 , x t ) which is a closed from Gaussian, and µ θ (x t , t|x H ) is the predicted mean of p θ (x t-1 | x t |x H ) computed by a neural network.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
method: 
 We first describe the overall architecture of the proposed method in 4.1. Then we will introduce the strategy of building datasets in Section 4.2. The embedding-based retrieval mechanisms and reference-guided time series diffusion model are introduced in Section 4.3.Figure 2 (a) shows the overall architecture of RATD. We built the entire process based on DiffWave [17] , which combines the traditional diffusion model framework and a 2D transformer structure. In the forecasting task, RATD first retrieves motion sequences from the database base D R based on the input sequence of historical events. These retrieved samples are then fed into the Reference-Modulated Attention (RMA) as references. In the RMA layer, we integrate the features of the input [x H , x t ]at time step t with side information I s and the references x R . Through this integration, the references guide the generation process. We will introduce these processes in the following subsections.Before retrieval, it is necessary to construct a proper database. We propose a strategy for constructing databases from time series datasets with different characteristics. Some time series datasets are sizeinsufficient and are difficult to annotate with a single category label (e.g., electricity time series), while some datasets contain complete category labels but exhibit a significant degree of class imbalance (e.g., medical time series). We use two different definitions of databases for these two different types of datasets. For the first definition, the entire training set is directly defined as the database D R :EQUATIONwhere x i = {s i , • • • , s i+l+h } is the time series with length l + h, and D train is the training set. In the second way, the subset containing samples from all categories in the dataset is defined as the database D R ′ : D R ′ = {x c i , • • • , x c q |∀c ∈ C} (6)where x k i is the i-th sample in the k-th class of the training set, with a length of l + h. C is the category set of the original dataset. For brevity, we represent both databases as D R .Embedding-Based Retrieval Mechanism For time forecasting tasks, the ideal references {s i , • • • , s i+h } would be samples where preceding n points {s i-n , • • • , s i-1 } is most relevant to the historical time series {s j , • • • , s j+n } in the D R . In our approach, the overall similarity between time series is of greater concern. We quantify the reference between time series using the distance between their embeddings. To ensure that embeddings can effectively represent the entire time series, pre-trained encoders E ϕ are utilized. E ϕ is trained on representation learning tasks, and the parameter set ϕ is frozen in our retrieval mechanism. For time series (with length n + h) in D R , their first n points are encoded, thus the D R can be represented as D R emb :EQUATIONwhere [p : q] refers to the subsequence formed by the p-th point to the q-th point in the time series. The embedding corresponding to the historical time series can be represented as v H = E ϕ (x H ).We calculate the distances between v H and all embeddings in D R emb and retrieve the references corresponding to the k smallest distances. This process can be expressed as:EQUATIONwhere index(•) represents retrieved index given v D . Thus, we obtain a subsetx R of D R based on a query x H , i.e.ζ k : x H , D R → x R , where |x R | = k.Reference-Guided Time Series Diffusion Model In this section, we will introduce our referenceguided time series diffusion model. In the diffusion process, the forward process is identical to the traditional diffusion process, as shown in Equation ( 2). Following [34, 12, 35] the objective of the reverse process is to infer the posterior distribution p(z tar |z c ) through the subsequent expression:EQUATIONwherep(x T |x H ) ≈ N (x T |x H , I), p θ (x t-1 |x t , x H , x R) is the reverse transition kernel from x t to x t-1 with a learnable parameter θ. Following most of the literature in the diffusion model, we adopt the assumption:EQUATIONwhere µ θ is a deep neural network with parameter θ. After similar computations as those in [12] , Σ θ (x t , x H , x R , t)) in the backward process is approximated as fixed. In other words, we can achieve reference-guided denoising by designing a rational and robust µ θ .Denoising Network Architecture Similar to DiffWave [17] and CSDI [36] , our pipeline is constructed on the foundation of transformer layers, as shown in Figure 3 . However, the existing framework cannot effectively utilize the reference as guidance. Considering attention modules to integrate the x R and x t as a reasonable intuition, we propose a novel module called Reference Modulated Attention (RMA). Unlike normal attention modules, we realize the fusion of three features in RMA: the current time series feature, the side feature, and the reference feature. To be specific, RMA was set at the beginning of each residual module Figure 3 . We use 1D-CNN to extract features from the input x t , references x R , and side information. Notably, we concatenate all references together for feature extraction. Side information consists of two parts, representing the correlation between variables and time steps in the current time series dataset Appendix B. We adjust the dimensions of these three features with linear layers and fuse them through matrix dot products. Similar to text-image diffusion models [29] , RMA can effectively utilize reference information to guide the denoising process, while appropriate parameter settings prevent the results from overly depending on the reference.Training Procedure To train RATD (i.e., optimize the evidence lower bound induced by RATD), we use the same objective function as previous work. The loss at time step t -1 are defined as follows respectively:L (x) t-1 = 1 2 β2 t ∥µ θ (x t , x0 ) -μ(x t , x0 )∥ 2 = γ t ∥x 0 -x0 ∥ (11)where x0 are predicted from x t , andγ t = ᾱt-1β 2 t 2 β2t (1-ᾱt) 2 are hyperparameters in diffusion process. We summarize the training procedure of RATD in Algorithm 1 and highlight the differences from the conventional models, in cyan. The process of sampling is shown in Appendix A. Compute loss L with x0 and x 0 (Equation ( 11))8:Update θ by minimizing L 9: end while 5 Experiments 
experimental setup: 
 Datasets Following previous work [45, 38, 8, 30] , experiments are performed on four popular real-world time series datasets: (1) Electricity * , which includes the hourly electricity consumption data from 321 clients over two years.; (2) Wind [20] , which contains wind power records from 2020-2021. (3) Exchange [18] , which describes the daily exchange rates of eight countries (Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore); (4) Weather † , which documents 21 meteorological indicators at 10-minute intervals spanning from 2020 to 2021.; Besides, we also applied our method to a large ECG time series dataset: MIMIC-IV-ECG [14] . The MIMIC-IV-ECG dataset contains clinical electrocardiogram data from over 190,000 patients and 450,000 hospitalizations at Beth Israel Deaconess Medical Center (BIDMC). Baseline Methods To comprehensively demonstrate the effectiveness of our method, we compare RATD with four kinds of time series forecasting methods. Our baselines include (1) Time series diffusion models, including CSDI [36] , mr-Diff [31] , D 3 VAE [20] , TimeDiff [30] ; (2) Recent time series forecasting methods with frequency information, including FiLM [46] , Fedformer [47] and FreTS [41] ; (3) Time series transformers, including PatchTST [25] , Autoformer [38] , Pyraformer [22] , Informer [45] and iTransformer [23] ; (4) Other popular methods, including TimesNet [39] , SciNet [21] , Nlinear [43] , DLinear [43] and NBeats [26] .Evaluation Metric To comprehensively assess our proposed methodology, our experiment employs three metrics: (1) Probabilistic forecasting metrics: Continuous Ranked Probability Score (CRPS) on each time series dimension [24] . (2) Distance metrics: Mean Squared Error (MSE), and Mean Average Error(MAE) are employed to measure the distance between predictions and ground truths.Implementation Details The length of the historical time series was 168, and the prediction lengths were (96, 192, 336) , with results averaged. All experiments were conducted on an Nvidia RTX A6000 GPU with 40GB memory. During the experiments, the second strategy of conducting D R was employed for the MIMIC dataset, while the first strategy was utilized for the other four datasets. To reduce the training cost, we preprocessed the retrieval process by storing the reference indices of each sample in the training set in a dictionary. During the training on the diffusion model, we accessed this dictionary directly to avoid redundant retrieval processes. More details are shown in Appendix B.Table 1 presents the primary results of our experiments on four daily datasets. Our approach surpasses existing time series diffusion models. Compared to other time series forecasting methods, our approach exhibits superior performance on three out of four datasets, with competitive performance on the remaining dataset. Notably, we achieve outstanding results on the wind dataset. Due to the lack of clear short-term periodicity (daily or hourly), some prediction tasks in this dataset are exceedingly challenging for other models. Retrieval-augmented mechanisms can effectively assist in addressing these challenging prediction tasks.Figure 4 presents a case study randomly selected from our experiments on the wind dataset. We compare our prediction with iTransformer and two popular open-source time series diffusion models, CSDI and D 3 VAE. Although CSDI and D 3 VAE provide accurate predictions in the initial short-term period, their long-term predictions deviate significantly from the ground truth due to the lack of guidance. ITransformer captures rough trends and periodic patterns, yet our method offers higherquality predictions than the others. Furthermore, through the comparison between the predicted results and references in the figure, although references provide strong guidance, they do not explicitly substitute for the entire generated results. This further validates the rationality of our approach.Table 2 presents the testing results of our method on the MIMIC-IV-ECG dataset. We selected some powerful open-source methods as baselines for comparison. Our experiments are divided into two parts: in the first part, we evaluate the entire test set, while in the second part, we select rare cases (those accounting for less than 2% of total cases) from the test set as a subset for evaluation. Prediction tasks in the second part are more challenging for deep models. In the first experiment, our method achieved results close to iTransformer, while in the second task, our model significantly outperformed other methods, demonstrating the effectiveness of our approach in addressing challenging tasks.Influence of Retrieval Mechanism To investigate the impact of the retrieval augmentation mechanism on the generation process, we conducted an ablation study and presented the results in Table 3 . The study addresses two questions: whether the retrieval augmentation mechanism is effective and which retrieval method is most effective. Firstly, we removed our retrieval augmentation mechanism from the RATD as a baseline. Besides, the model with random time series guidance is another baseline. The references retrieved by other methods have all positively impacted the prediction results. This suggests that reasonable references are highly effective in guiding the generation process.We also compared two different retrieval mechanisms: correlation-based retrieval and embeddingbased retrieval. The first method directly retrieves the reference in the time domain (e.g., using Dynamic Time Warping (DTW) or Pearson correlation coefficient). Our approach adopts the second mechanism: retrieving references through the embedding of time series. From the results, the correlation-based methods are significantly inferior to the embedding-based methods. The former methods fail to capture the key features of the time series, making it difficult to retrieve the best references for forecasting. We also evaluate the embedding-based methods with various encoders for comparison. The comprehensive results show that methods with different encoders do not significantly differ. This indicates that different methods can all extract meaningful references, thereby producing similar improvements in results. TCN was utilized in our experiment because TCN strikes the best balance between computational cost and performance.We conducted an ablation study on two variables, n and k, to investigate the influence of the retrieval database D R in RATD, where n represents the number of samples in each category of the database, and k represents the number of reference exemplars. The results in Figure 5q can benefit the model in terms of prediction accuracy because a larger D R brings higher diversity, thereby providing more details beneficial for prediction and enhancing the generation process. Simply increasing k does not show significant improvement, as utilizing more references may introduce more noise into the denoising process. In our experiment, the settings of n and k are 256 and 3, respectively.Inference Efficiency In this experiment, we evaluate the inference efficiency of the proposed RATD in comparison to other baseline time series diffusion models (TimeGrad, MG-TSD, SSSD). Figure 6 illustrates the inference time on the multivariate weather dataset with varying values of the prediction horizon (h). While our method introduces an additional retrieval module, the sampling efficiency of the RATD is not low due to the non-autoregressive transformer framework. It even slightly outperforms other baselines across all h values. Notably, TimeGrad is observed to be the slowest, attributed to its utilization of auto-regressive decoding. In these experiments, we used the CSDI architecture as the baseline method and added extra fusion modules to compare the performance of these modules (linear layer, cross-attention layer, and RMA). The results are shown in the Table 4 .Through our experiments, we found that compared to the basic cross-attention-based approach, RMA can integrate an edge information matrix (representing correlations between time and feature dimensions) more effectively. The extra fusion is highly beneficial in experiments, guiding the model to capture relationships between different variables. In contrast, linear-based methods concatenate inputs and references initially, which prevents the direct extraction of meaningful information from references, resulting in comparatively modest performance. Predicting x 0 vs Predicting ϵ. Following the formulation in Section 4.3, our network is designed to forecast the latent variable x 0 . Since some existing models [28, 36] have been trained by predicting an additional noise term ϵ, we conducted a comparative experiment to determine which approach is more suitable for our framework. Specifically, we maintained the network structure unchanged, only modifying the prediction target to be ϵ. The results are presented in Table 5 . Predicting x 0 proves to be more effective. This may be because the relationship between the reference and x o is more direct, making the denoising task relatively easier. We investigate the best position of RMA in the model. Front, middle, and back means we set the RMA in the front of, in the middle of, and the back of two transformer layers, respectively. We found that placing RMA before the bidirectional transformer resulted in the most significant improvement in model performance. This also aligns with the intuition of network design: cross-attention modules placed at the front of the model tend to have a greater impact. Like Algorithm 1, we summarize the sampling procedure of RATD in Algorithm 2 and highlight the differences from conventional diffusion models in . Predict x0 from x t , I s and x R (Equation ( 10))7:Sample x t-1 from the posterior q(x t |x 0 ) (Equation ( 2)) 8: end for B Impletion DetailsOur dataset is split in the proportion of 7:1:2 (Train: Validation: Test), utilizing a random splitting strategy to ensure diversity in the training set. We sample the ECG signals at 125Hz for the MIMIC-IV dataset and extract fixed-length windows as samples. For training, we utilized the Adam optimizer with an initial learning rate of 10 -3 , betas = (0.95, 0.999). During the training process of shifted diffusion, the batch size was set to 64, and early stopping was applied for a maximum of 200 epochs. The diffusion steps T were set to 100.We combine temporal embedding and feature embedding as side information v s . We use 128dimensions temporal embedding following previous studies [37] :EQUATIONwhere τ = 10000. Following [36] , s l represents the timestamp corresponding to the l-th point in the time series. This setup is designed to capture the irregular sampling in the dataset and convey it to the model. Additionally, we utilize learnable embedding to handle feature dimensions. Specifically, feature embedding is represented as 16-dimensional learnable vectors that capture relationships between dimensions. According to [17] , we combine time embedding and feature embedding, collectively referred to as side information I s .The shape of I s is not fixed and varies with datasets. Taking the Exchange dataset as an example, the shape of forecasting target x R is [Batchsize (64), 7(number of variables), 168 (time-dimension), 12 (time-dimension)] and the corresponding shape of I s is [Batchsize (64), total channel(144( time:128 + feature:16)), 320 (frequency-dimension*latent channel), 12 (time-dimension)].Our approach employs the Transformer architecture from CSDI, with the distinction of expanding the channel dimension to 128. The network comprises temporal and feature layers, ensuring the comprehensiveness of the model in handling the time-frequency domain latent while maintaining a relatively simple structure. Regarding the transformer layer, we utilized a 1-layer Transformer encoder implemented in PyTorch [27] , comprising multi-head attention layers, fully connected layers, and layer normalization. We adopted the "linear attention transformer" package ‡ , to enhance computational efficiency. The inclusion of numerous features and long sequences prompted this decision. The package implements an efficient attention mechanism [33] , and we exclusively utilized the global attention feature within the package.We will introduce the metrics in our experiments. We summarize them as below:CRPS. CRPS [24] is a univariate strictly proper scoring rule which ´measures the compatibility of a cumulative distribution function F with an observation x as:EQUATIONwhere 1 (x≤y) is the indicator function, which is 1 if x ≤ y and 0 otherwise. The CRPS attains the minimum value when the predictive distribution F same as the data distribution.MAE and MSE. MAE and MSE are calculated in the formula xP represents the predicted time series, and x P represents the ground truth time series. MAE calculates the average absolute difference between predictions and true values, while MSE calculates the average squared difference between predictions and true values. A smaller MAE or MSE implies better predictions.EQUATIONNeurIPS Paper Checklist 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Justification: There is no obvious societal impact of the work performed Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the original work are sited properly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.* https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014 † https://www.bgc-jena.mpg.de/wetter/‡ https://github.com/lucidrains/linear-attention-transformer 
 
 Appendix:  


Decision: This well written paper has been assessed by five knowledgeable reviewers. Four of them leaned towards accepting it (one full accept, one weak accept and two borderline accept ratings), while one leaned the other way (borderline rejection). The reviewers predominantly highlighted novelly of the presented approach and clarity of its presentation.  One of the reviewers expressed a dissenting opinion on the novelty of the proposed approach that is a clever combination of otherwise well established techniques. The authors provided a comprehensive rebuttal in response to the initial critique and engaged the reviewers in discussions, which all helped resolve most of the concerns. Given the prevailing sentiment, with which I agree, I conclude that this work satisfies the requirements for being presentable at NeurIPS.

Reviews: review: paper_summary: paper_summary: This paper proposes a retrieval-augmented time series diffusion model that uses an embedding-based retrieval process and a reference-guided diffusion model. The proposed RATD retrieves relevant time series from an external database as references, which is later utilized to guide the denoising process of the diffusion model for future forecasting. Extensive experiments are conducted on five real-world datasets that demonstrate the effectiveness of the proposed approach.
strength_weakness: strength_weakness: - The proposed architecture requires retrieval database construction and pretraining the time series encoder $E_\phi$ before training the forecasting model. The additional complexity and computation cost may limit the efficiency of the proposed method.
- Experimental results do not contain standard deviation, which potentially limits the confidence and significance of the performance superiority.
- Some baselines are missing. For instance, aurhors mentioned MQ-ReTCNN and ReTime as retrieval-augemented time series models. However, they are not invovled as baselines in experiments.
questions: questions: - How is the time series encoder $E_\phi$ pretrained with representation learning tasks (Line148)? Since the embedding quality highly affects the retrieval precision and quality, some analysis and evaluations should be conducted on the time series embedding quality.
- What is the architecture of the pre-trained encoder $E_\phi$? How do you encode the multivariate time series into a single embedding? Do you use any pooling operation? More analysis and explanations on the encoder are beneficial for paper clarity.
limitations: limitations: Yes. The authors discussed the limitations of the proposed approach in the paper.
paper_summary: paper_summary: Existing time series diffusion models are unstable due to insufficient datasets and lack of guidance. The RATD model combines an embedding-based retrieval process with a reference-guided diffusion model to improve stability and accuracy. RATD retrieves relevant time series from a database to guide the denoising process, maximizing dataset utilization and compensating for guidance deficiencies. Experiments on multiple datasets demonstrate RATD's effectiveness, particularly in complex prediction tasks.
strength_weakness: strength_weakness: - The proposed RATD lacks justification for architecture.
- Please refer to the below questions.
questions: questions: - What is the main difference between RAG and proposed work?
- How to define stability in time-series data? Someone may not agree with stability in Figure 1-(c). Some may disagree that the red plots are unstable.
- Why necessarily the proposed RATD is needed in time-series forecasting? In other words, how to link the relationship between RATD and TSAD?
- There is a typo in the last sentence of the second paragraph from Section 5.2. “.t”
- Why Table 2 is separated from Table 1 independently?
- What can you define as the best retrieval for forecasting?
limitations: limitations: They described their limitations themselves.
paper_summary: paper_summary: This article proposes a retrieval-augmented diffusion model for time series prediction, featuring a simple and straightforward approach. It aims to address two key issues: 

1. The lack of semantics and labels in time series data, leading to insufficient guidance during the diffusion model's generation process. 

2. The problem of size insufficiency and imbalance in time series data. 

Specifically, the method involves embedding-based retrieval to gather similar time series data from a database to serve as references during the denoising process. Their proposed RMA module performs feature fusion between the retrieved time series and the current diffusion step's results, thereby enhancing the denoising and generation process of the diffusion model.  

In the experimental section, they explore the impact of different retrieval methods on the performance of the diffusion model. They also investigate the effects of varying the size of the retrieval database and the number of retrieved sequences (k) on the prediction performance.
strength_weakness: strength_weakness: The paper lacks innovation as the retrieval-enhanced diffusion method has already been proposed in the field of image generation, and similar retrieval-enhanced approaches exist in the time-series prediction domain, although their backbone is not a diffusion model. Additionally, the performance improvement largely stems from the representational capabilities of the pre-trained embedding model used.

Typo in title: "Augment" -> "augument".
questions: questions: 1. What would be the result if the retrieved sequences were directly used for a simple weighted average for the final generation output, or if the performance of selecting the sequence closest to the ground truth (gt) from the retrieved ones was evaluated?
2. I am uncertain whether the non-diffusion time-series prediction methods used in the author's comparative experiments include retrieval-enhanced modules. If not, I recommend adding experiments with this component to demonstrate the efficacy of the proposed method.
3. In Formula 1, the symbol $\mathcal{D}$ is used ambiguously, representing both distance and feature dimensions.
limitations: limitations: The approach may heavily rely on the capabilities of the embedding model and the construction of the database.
paper_summary: paper_summary: This paper introduces the Retrieval-Augmented Time series Diffusion model (RATD) to address the limitations of existing time series diffusion models, such as insufficient datasets and lack of guidance. RATD consists of two components: an embedding-based retrieval process and a reference-guided diffusion model. The model retrieves the most relevant time series from a database to guide the denoising process, thereby enhancing the utilization of the dataset and providing meaningful guidance. Experiments on multiple datasets demonstrate the effectiveness of RATD, particularly in complex prediction tasks.
strength_weakness: strength_weakness: - **Incomplete Experimental Comparisons**: The experimental setup and comparisons are insufficient and not aligned with existing studies, making it difficult to verify the performance and correct implementation of this work. For example, the datasets and forecasting horizons used differ from those in classical studies like CSDI and TimeGrad. This discrepancy makes it challenging to ensure fair comparisons and reproducibility.
  - In CSDI and TimeGrad, solar, electricity, traffic, taxi, wiki datasets and forecasting horizon=24 are used for comparisons. While this work adopts datasets of exchange, wind, electricity, and weather and only report the average results of horizons of 96, 192, and 336. 
  - Given that the code is also not provided at the current stage, though I like the idea, I am not sure about the reproducibility. Hence I suggest the author to align with existing classical studies and provide a more comprehensive experimental results.
- **Insufficient Ablation Tests**: While Table 3 indicates the importance of choosing good retriever embeddings, there are no systematic ablation tests of the architecture developed (Figure 3). Detailed explanations of model architectures and extensive ablation tests are crucial to understanding the success of conditional diffusion and ensuring computational efficiency. The current paper lacks this information. It is easy for me to understand the conditional diffusion process. but it is extremely challenging for me to understand the insights and tricks in the architectures shown in Figure 3. I believe these are crucial to ensure the success of conditional diffusion and provide efficient computation. I cannot find any contents in the main paper or appendix informing me about these.
  - Detailed explanation of model architectures and extensive ablation tests are important for me to effectively distinguish it from CSDI. For example, you can still follow the architecture of CSDI while only introducing the depdence on retrived references x^R.
- **Unknown GPU Memory Usage**: There is no information on GPU memory usage and computation efficiency. Understanding how the introduction of reference series impacts computation and how these challenges are addressed by specific architectural designs is important for evaluating the practicality of the model.
questions: questions: - More comprehensive and detailed comparisons with existing studies
  - Covering necessary datasets and both short and long horizons
- Systematic ablation tests to illustrate your specific architecture designs
  - Especially for Figure 3, more explanations and ablation tests are indispensable for readers to catch up
limitations: limitations: The idea of retrieval-augmented diffusion time-series model is interesting and reasonable.
My major concerns are about the incomplete experiments and the reproducibility.

I may consider raising the score if my major concerns are properly addressed.
paper_summary: paper_summary: This paper proposed Retrieval-Augmented Time series Diffusion (RATD) model for complex time series forecasting tasks, and designed an Extra Reference Modulated Attention (RMA) module to enable the guidance from the retrieved reference time series during the denoising process of the diffusion models. On five real-world datasets, this paper demonstrated that RATD performed on-par or better on multiple metrics compared to various time series forecasting baselines including diffusion models, transformers, and other methods.
strength_weakness: strength_weakness: 1. Based on the related work section, there have been several works which investigated  time series forecasting using diffusion models and achieved promising results. RAG is also a well-studied topic in text generation. Given this, I think the technical novelty of this work is moderate, though it's a nice direction to combine RAG with time series forecasting and the experiment results demonstrated its effectiveness. I am wondering whether there is any previous work on time series RAG?
2. Probably out of the scope of this paper, it would be great if the authors can provide some discussion or intuitive insights on the advantage of using diffusion models for time series forecasting compared to other methods, e.g. is it more robust or generalizable, and also the advantage of diffusion models when it comes to retrieval augmented forecasting. As in table 1, several transformer based methods also performed well on some datasets, thus I am wondering whether there is any justification in choosing diffusion models for time series forecasting given its potential extra computational cost.
questions: questions: 1. In section 4.2, how is $D^{R}$ different from $D^{R'}$ if $\mathcal{C}$ represents all categories? In 5.3, it seems that when it's $D^{R'}$, only $n$ samples from each category are used to construct the database. Is $n$ a hyperparameter for $D^{R'}$ that also needs to be introduced in section 4.2?
2. In section 4.3, under $\textbf{Denoising Network Architecture}$, it is mentioned that all references are concatenated together for feature extraction. I am wondering how this scales with the length of time series, i.e. would there be memory concerns if you are working with long-horizon time series forecasting / long context windows?
limitations: limitations: The authors discussed the limitations of this work with regards to computational costs incurred by the transformer based diffusion model and the retrieval process, which is also my concern.

Comments: title: Thank you for your reply
Thanks for your response.
Awaiting your reply
Thank you for you reply
Thank you for your response.
Thank you for your reply
Thank you for your reply
Awaiting your reply
Awaiting your reply
Thank you for your reply
Thank you for your response
Thank you for your support
LGTM
Thank you for your reply
Thank you for adding additional experiments
More Experiment Results
Thank you for your reply
Thank you for your response
comment: Thank you for your reply.

If you have further questions, please feel free to ask.
Thanks for the response. The author rebuttal has addressed most of my concerns. I tend to maintain my score.
Dear Reviewer uLZU:

Should there be any further points that require clarification or improvement, please know that we are fully committed to addressing them promptly.

Thank you once again for your invaluable contribution to our research.

Warm Regards,

The Authors
We are glad to receive your reply! We are also pleased that most of your concerns have been addressed.

Regarding your concerns about the paper's novelty, I would like to provide some additional clarifications. In addition to the lack of an effective guiding mechanism of exisiting time series diffusion models, another core issue is the limited size or quality of existing datasets. Specifically, real-world time series datasets are either too small in scale or highly imbalanced, which may not meet the high-quality data requirements of training diffusion models[1]. Our approach offers a general solution by leveraging useful information from the dataset during the generation process progressively, rather than expending significant resources to augment the existing dataset. In other words, our method makes the most of the limited dataset, thereby addressing the aforementioned core issue to some extent.

 Additionally, a critical challenge in time series forecasting is modeling both the periodic and trend components of the time series simultaneously [2]. Our approach provides substantial assistance in direct and explicit modeling of these temporal components by utilizing guidance from references.
 
 Overall, our method focuses on and addresses these core issues, making it both innovative and practical, which have been recognized by other reviewers.

Regarding the supplementary experiments for A4, the performance advantages of our method stem from three aspects:
* **Advantages of Diffusion Model Framework**:  Unlike methods that directly use a transformer, the multi-stage framework of diffusion models reduces the difficulty of learning from prior distributions to data distributions, which helps the model learn more accurate conditional distributions.
* **Iterative Guidance Mechanism**: Our effective guidance mechanism allows the utilization of conditions (i.e., references) during the diffusion process iteratively (T =100 in our experiment), whereas previous methods only performed feature fusion once. In other words, diffusion models can leverage references as guidance more effectively.
* **Better Feature Fusion Module**: The proposed RMA (Retrieval-Modulated Attention) module performs better in feature fusion compared to the previous work. We designed an extra experiment to prove it (results are shown in the previous response A1). Our experiments found that compared to the basic cross-attention-based method (used by ReTime) and linear fusion methods (used by MQ-ReTCNN), RMA can integrate an extra information matrix (representing correlations between time and feature dimensions), thus realizing the fusion of feature more effectively. 

In summary, the first two advantages stem from the diffusion model framework itself, while the last advantage comes from the new module we proposed. By integrating the three components, our proposed method achieved better experimental results.

Once again, thank you for your reply. We are so honored that you find our work interesting and promising. If you have any further questions, please feel free to ask. Thank you!

[1] Schramowski P, Brack M, Deiseroth B, et al. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 22522-22531.

[2]Shen L, Chen W, Kwok J. Multi-Resolution Diffusion Models for Time Series Forecasting[C]//The Twelfth International Conference on Learning Representations. 2024.
The author has addressed the review comments thoroughly, and most of my questions have been answered satisfactorily.

However, I still have concerns regarding the novelty of the proposed method, a point also raised by Reviewer zh2V. While the author highlights their contributions, such as being the first to propose a retrieval-enhanced diffusion model for time series and incorporating an attention mechanism, I remain uncertain whether the solution effectively addresses the core challenges associated with diffusion models in time series forecasting. Nevertheless, the integration of techniques like retrieval and diffusion modeling is an interesting and promising approach for this area.

Regarding Q4, the base structures of the methods being compared are not built on diffusion models. Could you clarify the specific advantages introduced by your base model or the retrieval-augmented solution?
We are so glad to see your reply!

Also, we are very grateful for your recognition of our work. If you have any further questions, please feel free to ask. 

Thank you!
Thanks for your response, which addresses my concerns. I will raise my score.
Thanks for the reply and sharing the insights! It makes sense to me.
Hello, Reviewer uLZU,
Regarding your concerns, we have provided some responses that you might find useful, including:
* We have highlighted and summarized the innovations of our method from two perspectives: innovation in model design and experimental results.
* Additional experimental results, including treating references as predictions and comparing our proposed method with previous reference-based methods.
* We have explained that the performance of our method is not highly dependent on the pre-trained encoding models.

We sincerely hope our responses address some of your concerns. If you have any further questions, please feel free to ask. Thank you.
Hi, Reviewer AGBG,
We have provided some responses that might address your concerns, including:
* We provided proof of the framework's justification, including additional experimental results and further analysis.
* We clarified the differences between our method and previous works while extra experiments demonstrate that our method has greater stability.
* We also offered some insights that you might be interested in about our proposed method and the related methods. 

We sincerely hope our responses address some of your concerns. If you have any further questions, please feel free to ask. Thank you.
Thank you very much for your reply!

I'm glad that our answer could address some of your concerns. Regarding Q2, I believe the main advantage of diffusion models stems from their framework's ability to learn precise distributions (the multi-stage framework reduces the difficulty of learning from prior distributions to data distributions). Learning the accurate conditional distribution for time series forecasting tasks can improve forecasting accuracy. 

Thank you for your recognition of our work! If you have any further questions, feel free to ask.
Thank you to the authors for addressing my questions.

The answers to Q1, Q3, and Q4 have clarified my concerns, and I appreciate that the authors explicitly pointed out that lack of guidance mechanisms is one of the major barriers to the effectiveness of diffusion models in time series modeling, to further clarify the motivation behind this work.

Maybe I did not make myself clear for Q2, what I intended to ask was why diffusion models would be preferred for certain time series modeling tasks over other non-diffusion model based methods, rather than the disadvantages of current diffusion models for time series. For example, is it because denoising would make the predictions more generalizable or more flexible, compared to other non-diffusion model based time series forecasting methods?

Overall, this paper is interesting to me and I believe it will provide insights to future research in this direction. I keep my score.
Thank you very much for raising the score!

We greatly appreciate your suggestions and support for our work. We will carefully consider incorporating your recommendations to enrich the experimental content in the final version of the paper. 

If you have any further questions, please feel free to reply. Thank you!
I trust your promise to share code and ensure the reproducibility. Most of my concerns have been addressed properly. So I have raised the score.
Thank you very much for your reply! We are glad to see that some of your concerns have been addressed. 

I am happy to share our code with you here through an anonymous GitHub link, but the conference policy does not permit posting links or similar actions. If our paper is accepted, we will immediately make the Github link available in the paper. Even if this process requires a formal application, we will upload and disclose the code as soon as possible. We hope for your understanding. 

Thanks again for your reply! Please feel free to ask any further questions.
I appreciate the efforts the authors have taken to address my concerns.The results look good to me. My last question would be are you planning to open source your code? As I feel that such a retrieval-based diffusion may not be easy to reproduce. If possible, could you share some anonymous implementation during the review period?
Hello! Here we provide more comprehensive experimental results to address any concerns you may have about the completeness of our experiments. Our experimental findings are as follows, with additional experiments on all datasets mentioned in CSDI using identical experimental setups to CSDI(CRPS means CRPS-sum here). 

|Dataset | Traffic|  Electricity | Wind | Wiki | Taxi| Solar | Mimic-IV|
| :-----| :---- | :---- |:---- | :---- |:---- |:---- |:---- |
|Metric | MSE CRPS |  MSE CRPS | MSE CRPS | MSE CRPS | MSE CRPS | MSE CRPS | MSE CRPS | 
| CSDI |0.027 0.020|0.112 0.017|0.462 0.531|0.057 0.047|0.262 0.123|0.381 0.298|0.201 0.261|
| TimeGrad | 0.041 0.044 |  0.121 0.021 |0.491 0.569 | 0.059 0.049| 0.253 0.114|0.359 0.287| 0.217 0.279|
| RATD(Ours)|**0.023 0.018** |  **0.096 0.014** | **0.311 0.471** | **0.041 0.044**| **0.191 0.109**|**0.301 0.251**| **0.153 0.253**|
 
From the results, it can be observed that our method performs better across all datasets. To quantify this more explicitly, we have calculated the average performance improvement （API） of our method compared to CSDI under both experimental settings.

| Dataset | Traffic |  Electricity | Wind | Wiki | Taxi | Solar | Mimic-IV |
| :-----| :---- | :---- |:---- | :---- |:-----|:---- |:-----|
| API (%) in CSDI Settings |17.39|14.28|20.35|17.91|19.19|18.35|13.41|
| API (%) in Our Settings |31.21|33.18|28.45|21.06|26.14|31.24|25.12|


It appears that our method will indeed achieve better performance on more complex datasets. Notably, our method exhibits higher performance gains in more complex experimental setups, aligning with our motivation to tackle more intricate time series forecasting tasks.
 
Additionally, we have supplemented results from disintegration experiments on six of these datasets. 
|Dataset | Exchange|  Electricity | Wind | Weather |Solar |Mimic-IV|
| :-----| :---- | :---- |:---- | :---- |:---- |:---- |
| CSDI| 0.077  |  0.379 | 1.066 |0.356|0.381|0.268|
| CSDI+Linear| 0.075 |0.316|0.932| 0.349|0.369|0.265|
| CSDI+Cross Attention| 0.028  |  0.173 |0.829 |0.291|0.340|0.183|
| CSDI+RMA|**0.013** |  **0.151**  | **0.784** | **0.281**|**0.327**|**0.172**|

As mentioned earlier, CSDI represents the basic network framework, CSDI+Linear denotes the approach where inputs and references are concatenated via a linear layer and fed into the network together, CSDI+CrossAttention signifies the use of cross attention to fuse features from inputs and references, and finally, CSDI+RMA, which incorporates an additional Reference Modulated Attention (RMA). 

For further analysis of RMA performance, we have also supplemented additional qualitative and quantitative experiments to compare the similarity between reference and prediction results. Due to our current inability to upload images or links, we present quantitative experimental results in the table below. Specifically, we use the Mean Squared Error (MSE) metric to measure the similarity between the reference and predicted results (where smaller values indicate greater similarity). Additionally, to assess whether the model effectively utilizes explicit information from the reference rather than noise, we employ the STL(
Seasonal-Trend decomposition using LOESS) method to evaluate the similarity of different components. The results are as follows: 

|Dataset: Solar | Overall Similarity| Trend Component Similarity |Seasonal Component Similarity | 
| :-----| :---- | :---- |:---- | 
| CSDI+Linear| 0.986 |0.401|0.392| 
| CSDI+Cross Attention| 0.533 | 0.201 |0.264 |
| CSDI+RMA|**0.471** |  **0.193**  | **0.241** | 

|Dataset: Mimic-IV | Overall Similarity| Trend Component Similarity |Seasonal Component Similarity | 
| :-----| :---- | :---- |:---- | 
| CSDI+Linear| 0.811 |0.381|0.202| 
| CSDI+Cross Attention| 0.401 | 0.191 |0.143 |
| CSDI+RMA|**0.305**|  **0.142**  |**0.091** |


In some complex time series datasets (Solar), trend components are more important for predictions, while for ECG sequences (Mimic-IV), periodicity is more crucial. Due to enhanced feature integration capabilities, our method adaptively captures the most important information for both cases, leading to better prediction results. In summary, our approach efficiently captures information from the reference and provides beneficial guidance on periodicity and trends for the generated time series.

Finally, we sincerely hope that the additional experiments we propose can address your concerns regarding the completeness of the experiments! If you have any further questions, please let us know!
We are happy to see your reply! 

First, we could not complete experiments on all datasets before the first author rebuttal deadline due to the time required for the preprocessing of the dataset (including the retrieval process) and training. Therefore, we only presented partial results and hope for your understanding.  Currently, training on the remaining datasets is still ongoing, and we are about to obtain comprehensive experimental results.

Second, we will follow your suggestion to conduct more comprehensive ablation experiments to demonstrate the effectiveness of the proposed new module. 

We expect to have all experimental results ready for presentation by this time tomorrow. Once again, thank you for your reply, and we will showcase the complete additional experiments tomorrow. Thank you！
I appreciate that the authors have done some additional experiments to address my questions.

This work looks interesting to me. But I do have some remaining concerns regarding insufficient experiments.

First, why some datasets are still missing, such as solar and wiki in CSDI [1]. According to a recent benchmark study [2], solar is pretty special as it includes complex data distribution to be captured.

[1] https://arxiv.org/pdf/2107.03502
[2] https://arxiv.org/pdf/2310.07446

Second, regarding main experiments and ablation tests, a comprehensive experiments covering diverse datasets (as mentioned above), varied horizons, and evaluation metrics would convince me about the solidness of this work.
- About the combation of datasets and horizons, the original paper covers some and additional experiments in response gives some but still do not deliver a comprehensive set of experiments
- Ablation tests should be aligned with the evaluation scenarios of main experiments. And some analysis to explain the performance gains would be helpful.

In summary, I like this work's idea but its current experimental results do not convince me sufficiently. So I keep my score.