SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In DiffTime, we leverage conditional diffusion models to support trend and fixed values for generating TS. However, just by conditioning the model generation is not possible to encode all the constraints. A common solution is to penalize the generative model proportionally to how much the generated TS violates the input constraint [36].
In this section, we propose Loss-DiffTime where a constraint penalty is applied to deal with any kind of constraint. The penalty function  $f_{c}:\mathcal{X}\rightarrow \mathbb{R}$  is added to the learning objective of the diffusion model, and it evaluates whether the generated TS  $\hat{\mathbf{x}}$  meets the input constraint. We discuss the penalty function  $f_{c}$  for constraints in Section 2 and in Equation 1. With  $f_{c}$  in the loss, the greater the constraint violation is, the greater the model loss during training will be. However, the optimization problem in Eq. 5 predicts the noise component for the sample  $\mathbf{x}_0$ , into which we cannot directly feed to our penalty function. Moreover, we cannot apply  $f_{c}(\mathbf{x})$  to a noisy sample  $\mathbf{x}_t$  as the constraints may be evaluated only on the final sample. Therefore, to apply our penalty function, we re-parametrize the optimization problem and force the diffusion model to explicitly model the final sample  $\hat{\mathbf{x}}_0$  at every step as follows:
$$
L (\theta) := \mathbb {E} _ {t, \mathbf {x} _ {0}, \epsilon} \left[ \| \epsilon - \epsilon_ {\theta} \left(\mathbf {x} _ {t}, t \mid s\right) \| ^ {2} + \rho f _ {c} \left(\hat {\mathbf {x}} _ {0}\right) \right] \tag {9}
$$
where  $\mathbf{x}_t = \sqrt{\hat{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \hat{\alpha}_t}\boldsymbol{\epsilon}$  and  $\hat{\mathbf{x}}_0 = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{1 - \alpha_t}{\sqrt{1 - \hat{\alpha}_t}}\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t,t)\right)$ . We consider that any constraint in  $C$  can be differentiable (as discussed in Section 2). So, we can train our diffusion model following Eq. 9 where  $\rho$  is a scale parameter used to adjust the importance of the constraint loss. The conditional information of the trend  $s$  can be removed if we do not need to enforce any trend constraint. Figure 1.d and Figure 1.e show two examples of more complex constraints with Loss-DiffTime.
3.4 Guided-DiffTime - Guided Diffusion models for constrained generation
The Loss-DiffTime model is now able to generate real TS while dealing with any constraint. However, we notice two major drawbacks: 1) since we translate constraints to penalty terms in the loss, we need to re-train the model for new constraints; and 2) the diffusion models usually require several iterative steps  $T$  which can make it slower and expensive for TS generation. Our final proposed approach, namely Guided-DiffTime, solves these two problems and can dramatically reduce the carbon footprint when using DGM for constrained TS generation. In particular, it adopts a Denoising Diffusion Implicit Model (DDIM) [37] which requires fewer diffusion steps at inference. Moreover, by following the groundbreaking work of [38, 39], which shows how to guide a diffusion model using a noisy classifier, we demonstrate how a pre-trained diffusion model can be guided (conditioned) using gradients from differentiable constraints.
DDIM is a class of non-Markovian diffusion processes with the same training objective of classic DDPMs [28], but fewer diffusion steps to generate high-quality samples. In particular, DDIMs keep the same training procedure as DDPMs defined in Section 3.2 while the sampling can be accelerated by using the following re-parametrization of the reverse process:
$$
\mathbf {x} _ {t - 1} = \sqrt {\hat {\alpha} _ {t - 1}} \left(\frac {\mathbf {x} _ {t} - \sqrt {1 - \hat {\alpha} _ {t}} \cdot \epsilon_ {\theta} (\mathbf {x} _ {t} , t)}{\sqrt {\hat {\alpha} _ {t}}}\right) + \sqrt {1 - \hat {\alpha} _ {t - 1} - \sigma_ {t} ^ {2}} \cdot \epsilon_ {\theta} (\mathbf {x} _ {t}, t) + \sigma_ {t} \epsilon \tag {10}
$$
where  $\hat{\alpha}_0\coloneqq 1$  and different parametrizations of  $\sigma_{t}$  lead to different generative processes. We set  $\sigma_t = 0,\forall t\in [0,T]$  to have a deterministic forward process from latent variables to the sample  $\mathbf{x}_0$  (since the noise term  $\sigma_t\epsilon$  is zeroed out). This deterministic forward process defines the DDIM which can use fewer diffusion steps to generate realistic samples. This diffusion steps are defined by a sequence  $\tau$  of length  $V$  which is a sub-sequence of  $[1,\dots ,T]$  with the last value as  $T$ , i.e.,  $\tau_V = T$  [37]. For example,  $\tau = [1,4,9,\ldots ,T]$ . Moreover, this parametrization is a generalization of DDPM as setting  $\sigma_t = \sqrt{(1 - \alpha_{t - 1}) / (1 - \alpha_t)}\sqrt{1 - \alpha_t / \alpha_{t - 1}}$  describes the original DDPM [37] and the DDIM work showed that re-training of the DDPM model is unnecessary when we change the value of  $\sigma$  or the diffusion steps  $\tau$ .
Given the DDIM, we can then apply the recent results from guided diffusion models [38, 39] to condition each sampling step with the information given by the gradients of the differentiable constraint  $f_{c}$  (see Section 3.3). Algorithm 1 shows the sampling procedure which computes the
Algorithm 1 Guided-DiffTime
Input: differentiable constraint  $f_{c}:\mathcal{X}\to \mathbb{R}$  , scale parameter  $\rho$    
Output: new TS,  $\mathbf{x}_0$ $\mathbf{x}_T\gets$  sample from  $\mathcal{N}(0,\mathbf{I})$    
for all  $t$  from  $T$  to 1 do   
 $\hat{\epsilon}\gets \epsilon_{\theta}(\mathbf{x}_{t},t)$ $\hat{\epsilon}\gets \hat{\epsilon} -\rho \sqrt{1 - \hat{\alpha}_{t}}\nabla_{\mathbf{x}_{t}}f_{c}(\frac{1}{\sqrt{\hat{\alpha}_{t}}} (\mathbf{x}_{t} - \hat{\epsilon}\sqrt{1 - \hat{\alpha}_{t}}))$ $\mathbf{x}_{t - 1}\gets \sqrt{\hat{\alpha}_{t - 1}}\left(\frac{\mathbf{x}_t - \sqrt{1 - \hat{\alpha}_t\hat{\epsilon}}}{\sqrt{\hat{\alpha}_t}}\right) + \sqrt{1 - \hat{\alpha}_{t - 1}}\hat{\epsilon}$    
end for   
return  $\mathbf{x}_0$
gradients w.r.t. to the input TS  $\mathbf{x}_t$ . We recall that the constraint is applied on the final sample  $\hat{\mathbf{x}}_0$ , computed according to the DDIM reverse process. Again, this approach does not require re-training of the original diffusion model to deal with new constraints, which can be applied just at inference time. Hence, we reduce the carbon footprint of the model, and get a faster time-series generation.
In this section we evaluate our approaches, showing their advantages and disadvantages when applied to different domains and constraints. In particular, we follow the five scenarios shown in Figure 1 while considering multiple real-world and synthetic datasets. For COP we use a subset of the original TS as starting solution (seed), we leave in the supplementary material the analysis of different seeds.
Baselines We compare our approaches against existing TS generative models, including GTGAN[12], TimeGAN [1], RCGAN [40], C-RNN-GAN [41], a Recurrent Neural Networks (RNN) [1] trained with T-Forcing and P-Forcing [42, 43], WaveNET [44], and WaveGAN [45]. For the constrained scenarios, we extend the benchmark architectures to cope with constraints, by introducing a penalty loss [18, 16] or by conditioning the generation process. We also employ rejection-sampling and fine-tuning with COP on their generated synthetic TS.
Datasets We consider three datasets with different characteristics such as periodicity, noise, correlation, and number of features: 1) daily stocks which uses daily historical Google stock data from 2004 to 2019 with open, high, low, close, adjusted close, and volume features [1]; 2) energy data from the UCI Appliances energy prediction dataset [46] containing 28 features with noisy periodicity and correlation; 3) sines a synthetic multivariate sinusoidal TS with different frequencies and phases [1].
Evaluation metrics For each experimental scenario, we evaluate the generative models and TS along different quantitative and qualitative dimensions. First, we evaluate the realism through a discriminative score [1], which measures how much the generated samples resemble (i.e., are indistinguishable from) the real data using a post-hoc RNN trained to distinguish between real and generated samples. We evaluate the distributional-similarity between the synthetic data and real data by applying t-SNE [47] on both real and synthetic samples; t-SNE shows (in a 2-dimensional space) how well the synthetic distribution covers the original input distribution. Then, we evaluate the usefulness of generated samples — how the synthetic data supports a downstream task such as prediction — by training an RNN on synthetic data and testing its prediction performance on real data (i.e., predictive-score [1]). To evaluate how the model satisfies different constraints, we introduce the following metrics: Perc. error distance which measures how much the synthetic data follows a trend constraint by evaluating the L2 distance between the TS and the trend; satisfaction rate which measures the percentage of time a synthetic TS meets the input constraints; the inference time measured as the average seconds required to generate a new sample with a given constraint; and finally the fine-tuning time which is the average time, in seconds, needed to enforce constraints over a generated sample, using COP to fine-tune it.
We provide further experiments, including details on the baselines, datasets, metrics, and algorithm hyperparameters in the supplementary material.
4.1 Unconstrained Generation
First, we compare the ability of DiffTime and  $COP$  to generate unconstrained TS against existing benchmark datasets and algorithms. In Figure 2 we evaluate the realism with respect to the distributional-similarity, where red dots represent the original TS and blue dots the generated TS. The figure shows that our approaches have significantly better performance with better overlap between red and blue samples. $^{1}$
(a) COP
(b) DiffTime
(c) GT-GAN
Figure 2: t-SNE visualizations on multivariate stock data, where a greater overlap of blue and red dots shows a better distributional-similarity between the generated data and original data. Our approaches show the best performance.
(d) TimeGAN
(e) RCGAN
(f) C-RNN-GAN
In Table 1 we measure the usefulness and realism through the predictive and discriminative scores, respectively. DiffTime consistently generates the most useful data with the best predictive score for both Sines and Stocks datasets, while keeping remarkable realism (i.e., discriminative score).  $COP$  generates excellent synthetic samples as well, indistinguishable from real data with the best discriminative score for both Stocks and Energy. However, we acknowledge COP is advantaged by the original time series as an input seed.
Table 1: Unconstrained Time-Series Generation (Bold indicates best performance).
Metric	Method	Sines	Stocks	Energy
Discriminative Score (Lower the Better)	DiffTime (Ours)	.013 ± .006	.097 ± .016	.445 ± .004
COP (Ours)	.020 ± .001	.050 ± .017	.101 ± .019
GT-GAN	.012 ± .014	.077 ± .031	.221 ± .068
TimeGAN	.011 ± .008	.102 ± .021	.236 ± .012
RCGAN	.022 ± .008	.196 ± .027	.336 ± .017
C-RNN-GAN	.229 ± .040	.399 ± .028	.449 ± .001
T-Forcing	.495 ± .001	.226 ± .035	.483 ± .004
P-Forcing	.430 ± .227	.257 ± .026	.412 ± .006
WaveNet	.158 ± .011	.232 ± .028	.397 ± .010
WaveGAN	.277 ± .013	.217 ± .022	.363 ± .012
Predictive Score (Lower the Better)	DiffTime (Ours)	.093 ± .000	.038 ± .001	.252 ± .000
COP (Ours)	.095 ± .002	.041 ± .001	.250 ± .003
GT-GAN	.097 ± .000	.040 ± .000	.312 ± .002
TimeGAN	.093 ± .019	.038 ± .001	.273 ± .004
RCGAN	.097 ± .001	.040 ± .001	.292 ± .004
C-RNN-GAN	.127 ± .004	.038 ± .000	.483 ± .005
T-Forcing	.150 ± .022	.038 ± .001	.315 ± .005
P-Forcing	.116 ± .004	.043 ± .001	.303 ± .005
WaveNet	.117 ± .008	.042 ± .001	.311 ± .006
WaveGAN	.134 ± .013	.041 ± .001	.307 ± .007
Original	.094 ± .001	.036 ± .001	.250 ± .003
4.2 Constrained Generation
We now evaluate the performance of our approaches against the constraints shown in Figure 1 using daily stock data. For univariate constraints (i.e., trend, fixed values, and global minimum) we consider only the Open value from the daily stock dataset. We consider as benchmarks the best three SoA approaches from Table 1 (i.e., GT-GAN, TimeGAN, and RCGAN). For trend- and fixed-values constraints we condition their generative process so that different trends and values can be used at inference time. For the other constraints, we add a penalty term in the training loss [18, 16].
<sup>1</sup>We report only the top 6 models, leaving the full evaluation to the Supplementary material.
Soft Constraints In table 2 we constrain the synthetic TS to follow a given trend, computed as a 3-degree polynomial approximation from the original samples. Our approaches generate synthetic data that are closer to the input trend, with the smallest relative distance (i.e., perc. error distance). Moreover, our approaches are among the best in terms of realism and usefulness. In the supplementary material we investigate the use of sinusoidal trends, including additional evaluation metrics.
Table 2: Soft Constraints (Trend) Time-Series Generation (Bold indicates best performance).
Algo	Discr-Score	Pred-Score	Inference-Time	Perc. Error Distance
COP (Ours)	0.01±0.01	0.20±0.00	0.73±0.05	0.015±0
DiffTime (Ours)	0.01±0.01	0.20±0.00	0.02±0.00	0.018±0
GT-GAN	0.04±0.03	0.22±0.00	0.00±0.00	1.378±2
TimeGAN	0.02±0.02	0.20±0.00	0.00±0.00	0.073±0
RCGAN	0.02±0.01	0.20±0.00	0.00±0.00	0.071±0
**Hard Constraints** In Table 3 we evaluate all the approaches against hard constraints (see Fixed Points, Global Min, and Multivariate in Figure 1). For Global Min almost all approaches have a great satisfaction rate. However, our approaches are above 0.90 and have the best discriminative and predictive score. Additionally, while  $100\%$  of the synthetic time-series generated by TimeGAN and RCGAN guarantee this type of constraint, they do not approximate the input distribution well (see Figure 3). For most complex constraints like the multivariate one, the satisfaction rate drops for most of the benchmarks while for our GuidedDiffTime and COP the satisfaction rate is still very high, with great realism and usefulness. Finally, when we employ the fixed point constraints, we fix the values of the points at index 6 and 18. All the benchmarks fail to satisfy these constraints, while we show instead that DiffTime is able to always guarantee this constraint, by enforcing it during the diffusion steps. Most importantly, it achieves very good discriminative and predictive scores with low inference time. To summarize our results: COP achieves almost always the best realism and usefulness scores, but with higher inference time and using original input TS as seed (which makes the generated TS very similar to the input data); diffusion models are also very powerful with lower inference time and use random noise as input seed as opposed to a real TS (this gives us better variety in TS compared to COP); and GuidedDiffTime is able to enforce new constraints without any re-training yet achieving excellent performance.
Table 3: Hard Constraints Time-Series Generation (Bold indicates best performance).
Constraint	Algo	Discr-Score	Pred-Score	Inference-Time	Satisfaction Rate	Fine-Tuning Time
Global Min	COP (Ours)	0.02±0.01	0.20±0.00	19.1±1.01	1.00±0.00	0.00±0.00
GuidedDiffTime (Ours)	0.03±0.02	0.21±0.00	0.03±0.00	0.90±0.01	3.01±0.10
LossDiffTime (Ours)	0.22±0.03	0.38±0.00	0.02±0.00	0.99±0.00	6.00±0.60
GT-GAN	0.04±0.02	0.22±0.00	0.00±0.00	0.87±0.02	9.30±1.30
TimeGAN	0.03±0.02	0.21±0.00	0.00±0.00	1.00±0.00	0.00±0.00
RCGAN	0.23±0.03	0.20±0.00	0.00±0.00	1.00±0.00	0.00±0.00
Multivariate (OHLC)	COP (Ours)	0.04±0.02	0.04±0.00	2.17±0.10	1.00±0.00	0.00±0.00
GuidedDiffTime (Ours)	0.08±0.00	0.04±0.10	0.15±0.00	0.72±0.02	31.0±1.50
LossDiffTime (Ours)	0.35±0.04	0.04±0.01	0.14±0.00	0.69±0.01	57.5±5.01
GT-GAN	0.22±0.07	0.05±0.00	0.00±0.00	0.05±0.01	44.5±3.01
TimeGAN	0.24±0.03	0.05±0.00	0.00±0.00	0.51±0.02	16.1±1.30
RCGAN	0.35±0.04	0.04±0.00	0.00±0.00	0.00±0.00	95.1±4.03
Two Fixed Points	COP (Ours)	0.02±0.02	0.20±0.00	0.56±0.11	1.00±0.00	0.00±0.00
DiffTime (Ours)	0.04±0.03	0.21±0.00	0.01±0.00	1.00±0.00	0.00±0.00
GT-GAN	0.04±0.03	0.21±0.00	0.00±0.00	0.00±0.00	0.99±0.10
TimeGAN	0.03±0.01	0.20±0.00	0.00±0.00	0.00±0.00	0.84±0.00
RCGAN	0.02±0.02	0.20±0.00	0.00±0.00	0.00±0.00	0.87±0.20
Figure 3: t-SNE visualizations on Global-Min constrained data, where a greater overlap of blue and red dots implies a better distributional-similarity between the generated data and original data. Our approaches show the best performance.
(b) Guided DiffTime
(c) LossDiffTime
(d) GT-GAN
(e) TimeGAN
(f) RCGAN
In summary, we defined the problem of generating synthetic TS data with both soft and hard constraints, and we presented a set of novel methods. We evaluated our approaches on different datasets and we compared their performance against existing state-of-art methods. We showed that our approaches outperform existing work both qualitatively and quantitatively. Most importantly, we introduced GuidedDiffTime to handle new constraints without re-training, and we showed that the COP-method can be used to fine-tune candidate solutions. Please refer to the supplementary material for more details on experiments comparing the methods presented herein.
This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates ("JP Morgan"), and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
[1] Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
[2] Eoin Brophy, Zhengwei Wang, Qi She, and Tomás Ward. Generative adversarial networks in time series: A systematic literature review. ACM Computing Surveys, 55(10):1-31, 2023.
[3] Yan Li, Xinjiang Lu, Yaqing Wang, and Dejing Dou. Generative time series forecasting with diffusion, denoise, and disentanglement. Advances in Neural Information Processing Systems, 35:23009-23022, 2022.
[4] James Jordan, Daniel Jarrett, Evgeny Saveliev, Jinsung Yoon, Paul Elbers, Patrick Thoral, Ari Ercole, Cheng Zhang, Danielle Belgrave, and Mihaela van der Schaar. Hide-and-seek privacy challenge: Synthetic data generation vs. patient re-identification. In NeurIPS 2020 Competition and Demonstration Track, pages 206–215. PMLR, 2021.
[5] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-24816, 2021.
[6] Andrea Coletta, Matteo Prata, Michele Conti, Emanuele Mercanti, Novella Bartolini, Aymeric Moulin, Svitlana Vyetrenko, and Tucker Balch. Towards realistic market simulations: a generative adversarial networks approach. In Proceedings of the Second ACM International Conference on AI in Finance, pages 1-9, 2021.
[7] Jean-Philippe Bouchaud, Julius Bonart, Jonathan Donier, and Martin Gould. Trades, quotes and prices: financial markets under the microscope. Cambridge University Press, 2018.
[8] Andrei Kirilenko, Albert S Kyle, Mehrdad Samadi, and Tugkan Tuzun. The flash crash: High-frequency trading in an electronic market. The Journal of Finance, 72(3):967-998, 2017.
[9] https://www.federalreserve.gov/publications/2023-stress-test-scenarios.htm.
[10] Sungyong Seo, Sercan Arik, Jinsung Yoon, Xiang Zhang, Kihyuk Sohn, and Tomas Pfister. Controlling neural networks with rule representations. Advances in Neural Information Processing Systems, 34:11196-11207, 2021.
[11] Jonathan Kinlay. Synthetic market data and its applications. Available at SSRN 4380552, 2023.