SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andrea Coletta
J.P. Morgan AI Research
Sriram Gopalakrishan
Daniel Borrajo
Madrid, ESP
Svitlana Vyetrenko
Synthetic time series are often used in practical applications to augment the historical time series dataset, amplify the occurrence of rare events and also create counterfactual scenarios. Distributional-similarity (which we refer to as realism) as well as the satisfaction of certain numerical constraints are common requirements for counterfactual time series generation. For instance, the US Federal Reserve publishes synthetic market stress scenarios given by the constrained time series for financial institutions to assess their performance in hypothetical recessions. Existing approaches for generating constrained time series usually penalize training loss to enforce constraints, and reject non-conforming samples. However, these approaches would require re-training if we change constraints, and rejection sampling can be computationally expensive, or impractical for complex constraints. In this paper, we propose a novel set of methods to tackle the constrained time series generation problem and provide efficient sampling while ensuring the realism of generated time series. In particular, we frame the problem using a constrained optimization framework and then we propose a set of generative methods including "GuidedDiffTime", a guided diffusion model. We empirically evaluate our work on several datasets for financial and energy data, where incorporating constraints is critical. We show that our approaches outperform existing work both qualitatively and quantitatively, and that "GuidedDiffTime" does not require re-training for new constraints, resulting in a significant carbon footprint reduction, up to  $92\%$  w.r.t. existing deep learning methods.
In recent years, synthetic time series (TS) have gained popularity in various applications such as data augmentation, forecasting, and imputation of missing values [1, 2, 3, 4, 5]. Additionally, synthetic TS are extremely useful to generate unseen and counterfactual scenarios, where we can test hypotheses and algorithms before employing them in real settings [6]. For example, in financial markets, it can be very useful to test trading strategies on unseen hypothetical markets scenarios, as poorly tested algorithms can lead to large losses for investors, as well as to overall market instability [7, 8]. In order to be useful, such hypothetical market stress scenarios need to be realistic - i.e., the synthetic market TS need to have statistical properties similar to the historical ones. They also need to satisfy certain constraints supplied by experts on how hypothetical market shock scenarios can potentially unfold. For instance, in order to ensure financial market stability, the US Federal Reserve annually assesses the market conditions and publishes a set of constrained market stress scenarios that financial institutions must subject their portfolios to, in order to estimate and adjust for their losses in case of market downturns [9].
(a) Unconstrained
(b) Trend
Figure 1: An example of synthetic stock market time-series under different constraints: (a) unconstrained generation; (b) a time-series following a trend constraint; (c) the final value of the TS has to hold a specific value; (d) the global minimum must be at a given time point; (e) multi-variate TS where the High and Low dimensions have the maximum and minimum values, respectively.
(c) Fixed value
(d) Global minimum
(e) Multivariate
Our work targets the problem of generating constrained TS that are both statistically similar to historical times series and match a given set of constraints. These constraints can be imposed by the underlying physical process that generates the data. For example, synthetic energy data should adhere to the principle of 'energy conservation' [10]. Or, as in the preceding example of the US Federal Reserve stress scenarios, constraints can be used to generate counterfactual synthetic TS with some given conditions, e.g., a stock market index decreases by  $5\%$  [9, 11].
Related work: Existing work employs deep generative models (DGMs) to capture the statistical data properties and temporal dynamics of TS [1, 12, 13, 14, 15], and additional constraints are usually introduced by penalizing the generative model proportionally to the mass it allocates to invalid data [16, 17, 18]; or by adding a regularization term to the loss function [19, 20]. Other approaches condition the generative process by encoding constraints and feeding them into the model [10]; or they reject and re-sample TS that do not match the constraints [21]. Finally, a different line of work proposes special-purpose solutions, with ad-hoc architectures or sampling methods, which however tackle specific applications (not TS generation) [22, 23, 24, 25]. In general, while most of these models are able to reproduce the real data statistics, complex constraints can still be challenging to be guaranteed. Most importantly, as DGMs incorporate constraints during training, a change to the constraints may require re-training since the learned distribution of a DGM may no longer cover the target distribution, and thus even rejection sampling would not be effective [26].
In this paper, we tackle the constrained TS generation problem for different constraint types, and we present and compare a set of methods that can incorporate constraints into TS generation. First, we present an optimization-based approach in which we compile the problem into a constrained optimization problem (COP) [27] and generate synthetic data from an explicit formulation of data properties and constraints. To overcome the need for an explicit definition of data properties, we employ a DGM that can implicitly capture data properties from training data. In particular, we introduce DiffTime, an approach based on conditional denoising diffusion models [28] where several constraints can condition the data generation. In addition, we show that any kind of constraint can be applied to diffusion models by penalizing the model proportionally to the constraint violation during training. This approach, called Loss-DiffTime, shows good performance with efficient sampling, but requires re-training upon a new constraint. Finally, to increase computational efficiency and reduce the carbon footprint of the model [29], we propose a guided diffusion model Guided-DiffTime that does not require re-training upon changes in the constraints — at inference, it can adjust the TS generation process based on the input constraints.
- We formally define the constrained TS problem, and characterize different types of constraints for synthetic TS.
- We propose and compare different approaches to generate synthetic TS. We evaluate their advantages and disadvantages with respect to different measures of performance and constraint types. We show how COP can also be used for post-hoc fine-tuning of TS, such that synthetic TS generated by any DGM can be adjusted to guarantee constraint satisfaction.
- We empirically demonstrate that our approaches outperform existing work in many TS domains, both qualitatively and quantitatively. We show that DiffTime outperforms existing state-of-art models in unconstrained TS generation, while Guided-DiffTime is the only solution where re-training for new constraints is not necessary, resulting in a significant carbon footprint reduction.
2 Definitions and Problem Formulation
The constrained TS generation problem requires generating synthetic TS data, where each TS is defined in the sample space  $\chi = \mathbb{R}^{L\times K}$  where  $L$  is the length of the TS and  $K$  is the number of features. Our goal is to generate synthetic data such that the synthetic distribution approximates the input data distribution, and each TS also conforms to user-specified constraints. The problem input is a tuple  $\langle \mathbb{D} = \{\mathbf{x}^i\}_{i = 1}^N,C\rangle$  consisting of a dataset  $\mathbb{D}$  of  $N$  time series  $\mathbf{x}^i\in \chi ,i\in [1..N]$  and a list of constraints  $C$  that a synthetic TS should conform to. The constraints include realism constraints (see Section 3.1). Henceforth, we will drop the sample index for  $\mathbf{x}$  unless needed, and only keep the position and feature indices. We also shortly denote  $[K]\triangleq \{0,\dots ,K\}$  and  $[L]\triangleq \{0,\dots ,L\}$ .
Constraints — like those in Figure 1 — are defined as tuples of the form  $\langle t, f \rangle \in C$ , where  $t$  can be either soft or hard and a differentiable function  $f$ . If the constraint type is hard, then  $f$  can be an inequality or an equality constraint. An inequality constraint is of the form  $f(\hat{\mathbf{x}}) \leq 0$  where  $\hat{\mathbf{x}}$  is the generated synthetic TS. An equality constraint is of the form  $f(\hat{\mathbf{x}}) = 0$ . Hard constraints are required to hold in a generated TS. Otherwise, the TS is rejected. Soft constraints are of the form  $f: \chi \to \mathbb{R}$  whose value we would like to optimize (minimize) for. Therefore, by definition, soft constraints do not require sample rejection. The constraints can be defined with respect to individual synthetic TS samples  $\hat{\mathbf{x}}$ , or at the dataset level (distribution-related constraints). As a type of soft constraint, we define trend-lines (see Figure 1.b) as a time series  $\mathbf{s} \in \chi$ . This constraint tells a generative method to minimize the L2 distance between the trend and the corresponding points of the synthetic TS. Formally, the synthetic TS  $\hat{\mathbf{x}}$  would be optimized as to minimize  $\| \mathbf{s} - \hat{\mathbf{x}} \|_2^2$ .
Additionally, both soft and hard constraints can be categorized into local and global constraints. Global constraints are those that compare across all the points in the TS. For example, we can enforce that  $x_{i,j} \leq x_{3,0}$ ,  $\forall (i,j) \in [L] \times [K]$  such that the maximum value is at  $x_{3,0}$ . Local constraints are those that only refer only to a subset of points. For instance, requiring  $(x_{i,j} = 2.5)$  for a given point  $(i,j) \in [L] \times [K]$ . We refer to this kind of constraint as fixed-point constraints (see Figure 1.c) since they require that the value of the TS is fixed at that point to a specific value. The set of all fixed-point constraints is  $\mathcal{R}$ , where each element  $r_{i,j} \in \mathbb{R}$  and  $(i,j) \in [L] \times [K]$ .
The aforementioned types of constraints are explicit. Additionally, the problem of synthetic data generation requires statistical similarity between the input and the synthetic datasets, which can either be built-in into the data generating method (e.g., by design GANs generate data that is distributionally similar to the input [2]) or specified explicitly as constraints in the model (e.g., autocorrelation similarity can be an explicit constraint). The methods presented herein assume that the constraints are differentiable. This is needed for deep generative methods, such as diffusion models, where constraints need to be incorporated into the training or inference process. If the functions are differentiable, then a straightforward approach [30] to incorporate them into the loss is:
$$
\operatorname {l o s s} (\mathbf {x}, \hat {\mathbf {x}}) = \text {o b j e c t i v e} _ {-} \operatorname {l o s s} (\mathbf {x}, \hat {\mathbf {x}}) + \lambda_ {g} \operatorname {R e L U} (g (\hat {\mathbf {x}})) + \lambda_ {h} h (\hat {\mathbf {x}}) ^ {2} \tag {1}
$$
where  $g(\hat{\mathbf{x}})$  and  $h(\hat{\mathbf{x}})$  are the inequality and an equality constraint respectively, which are added as soft constraints into the loss function with penalty terms  $\lambda_g$  and  $\lambda_h$ . However, incorporating constraints into the loss function may not guarantee constraint-conforming solutions, but good candidate or starting solutions that we can then fine-tune (i.e., adjust to guarantee constraints). If the constraints are not differentiable, one can use approaches that compute the loss for such a "rule" using perturbations [10].
3 Constrained Time-Series Generation - Proposed Approaches
In this section, we introduce several approaches to tackle the constrained TS generation problem. In particular, we discuss their advantages and disadvantages, and how they handle different scenarios and constraints.
3.1 Constrained Optimization Problem (COP)
Our first model tackles the synthetic TS generation problem as a Constrained Optimization Problem (COP) in which we treat each point  $x_{i,j} \in \mathbf{x}$  as a decision variable to be optimized. We will refer to this method simply as "COP-method". A COP problem is defined by an objective function that a solution is optimized for, and set of constraints that need to be satisfied by the solution.
We can use COP in two ways, as a generative method, and as a fine-tuning method. If COP is used for generating synthetic TS, then we take as input a real sample  $\mathbf{x}$  from  $\mathbb{D}$  as the starting TS (i.e., seed) for generation, and set the objective to maximize the difference between the seed and the synthetic TS. Formally, we maximize the L2 norm of the TS difference:  $obj(\mathbf{x},\hat{\mathbf{x}}) = \| (\mathbf{x} - \hat{\mathbf{x}})\| _2$ , where  $\mathbf{x}$  is the seed TS and  $\hat{\mathbf{x}}$  is the generated TS. COP can also be used for fine-tuning candidate solutions generated by other methods, such as by a diffusion model. When using COP as a fine-tuning method, the candidate solution generated by the other method becomes the seed TS, and the objective simply changes from maximizing the L2 difference to minimizing it; this is to preserve the information from the candidate TS and just search in the space of nearby solutions for one that satisfies all the constraints (if any failed). This can be helpful in fixing almost correct solutions, rather than using rejection sampling.
The constraints for the COP formulation come from  $C$ . Additionally, when using COP as a generative process we need to add constraints to satisfy the desired distributional properties that methods like GANs would implicitly handle, such as preserving the distribution of the autocorrelation of returns for stock data. This is done by constraining the COP solver to try and match the desired statistical properties of the seed TS; by matching the property at the sample level, we seek to match the distribution of that property at the dataset level. We do this by computing the target value from the seed TS and comparing it with the corresponding value from the synthetic TS. Specifically, we constrain the COP to limit the magnitude of the error between the property value computed for  $\hat{\mathbf{x}}$  and  $\mathbf{x}$  within an allowed amount (a budget for error tolerance). This is done by using an inequality constraint as follows:  $e(z(\hat{\mathbf{x}}), z(\mathbf{x})) - b \leq 0$ , where  $b$  is the budget we set ( $b = 0.1$  in our experiments),  $z(.)$  is the function that computes the desired property, and  $e(.)$  is the error function that measures the error between the target and generated values. For example, for autocorrelation of returns in stock data, the TS property is a vector, so the  $e(.)$  is the L2-norm of the difference. If the COP solver cannot find a solution within the allowed error tolerance, we double the budget and repeat the process for up to a fixed number of  $\eta$  repeats (we set  $\eta = 10$  in our experiments). In the supplementary material, we show how distributional constraints can be learned directly from the input data, by training a Wasserstein-GAN [31, 32] and using the GAN critic in the objective function.
Once we define the objective function and constraints for COP, we can employ one of the many solvers available to compute the synthetic TS. In our experiments, we use the Sequential Least Squares Programming (SLSQP) solver [27] in Scipy's optimization module [33].
3.2 DiffTime - Conditional Diffusion Model for Time Series generation
In the previous section, we introduced using COP to generate synthetic TS while guaranteeing the input constraints and data properties. However, such COP problems may be non-linear, and solving a non-linear problem is in general difficult and computationally expensive, especially with multi-variate and long time-series (see Section 4). In this section, we introduce a conditional diffusion model named DiffTime that leverages the latest advancements in score-based diffusion models [5, 34, 28, 35] to generate synthetic TS. Our model can generate realistic TS and cope with Trend and Fixed Points constraints by conditioning the generative model.
Denoising diffusion models Denoising Diffusion models are latent variable models which are trained to generate samples by gradually removing noise — denoising — from samples corrupted by Gaussian noise [28]. These models approximate a real data distribution  $q(\mathbf{x}_0)$  by learning a model distribution  $p_{\theta}(\mathbf{x}_0) \coloneqq \int p_{\theta}(\mathbf{x}_{0:T}) d\mathbf{x}_{1:T}$ , where the latent variables  $\mathbf{x}_{1:T}$  are defined in the same space  $\mathcal{X}$  of the sample  $\mathbf{x}_0$ . The training follows: a forward process that progressively adds noise to the sample  $\mathbf{x}_0$ ; and a reverse process where the generative process gradually denoises a noisy observation. The forward process is described with the following Markov chain with Gaussian transitions parameterized by  $\beta_{1:T}$ :
$$
q \left(\mathbf {x} _ {1: T} \mid \mathbf {x} _ {0}\right) := \prod_ {t = 1} ^ {T} q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right), \quad q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right) := \mathcal {N} \left(\sqrt {1 - \beta_ {t}} \mathbf {x} _ {t - 1}, \beta_ {t} \mathbf {I}\right) \tag {2}
$$
It admits the following close form  $q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}(\mathbf{x}_t; \sqrt{\hat{\alpha}_t} \mathbf{x}_0, (1 - \hat{\alpha}_t) \mathbf{I})$ , where  $\alpha_t := 1 - \beta_t$  and  $\hat{\alpha}_t := \prod_{i=1}^t \alpha_i$ , which allows sampling  $\mathbf{x}_t$  at any arbitrary diffusion step  $t$ . The generation is
performed by the reverse process defined as a Markov Chain starting at  $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{x}_T;\mathbf{0},\mathbf{I})$
$$
p _ {\theta} \left(\mathbf {x} _ {0: T}\right) := p \left(\mathbf {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right), \quad p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) := \mathcal {N} \left(\mathbf {x} _ {t - 1}; \boldsymbol {\mu} _ {\theta} \left(\mathbf {x} _ {t}, t\right), \boldsymbol {\Sigma} _ {\theta} \left(\mathbf {x} _ {t}, t\right)\right) \tag {3}
$$
Following the formulation of Denoising Diffusion Probabilistic Models (DDPM) [28] we parameterize the reverse process as follows:
$$
\boldsymbol {\mu} _ {\theta} \left(\mathbf {x} _ {t}, t\right) = \frac {1}{\sqrt {\hat {\alpha} _ {t}}} \left(\mathbf {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \hat {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\theta} \left(\mathbf {x} _ {t}, t\right)\right), \quad \boldsymbol {\Sigma} _ {\theta} \left(\mathbf {x} _ {t}, t\right) = \sigma^ {2} \mathbf {I}, \text {w h e r e} \sigma^ {2} = \sqrt {\beta_ {t}} \tag {4}
$$
where  $\epsilon_{\theta}$  is a trainable denoising function that predicts  $\epsilon$  from  $\mathbf{x}_t$ , and the choice of  $\beta$  corresponds to the upper bound on the reverse process entropy [35]. This function is approximated through a deep neural network trained according to the following objective:
$$
L (\theta) := \mathbb {E} _ {t, \mathbf {x} _ {0}, \boldsymbol {\epsilon}} \left\| \boldsymbol {\epsilon} - \boldsymbol {\epsilon} _ {\theta} \left(\sqrt {\hat {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \hat {\alpha} _ {t}} \boldsymbol {\epsilon}, t\right) \right\| ^ {2} \tag {5}
$$
where  $t$  is uniformly sampled between 1 and  $T$ , and the noise is Gaussian  $\epsilon \sim \mathcal{N}(0, I)$ . The diffusion steps  $T$  and variances  $\beta_{t}$  control the expressiveness of the diffusion process and they are important hyperparameters to guarantee that the forward and reverse processes have the same functional form [35].
Conditional diffusion models Our DiffTime model - which is a conditional diffusion model - supports both trend and fixed point constraints that were defined in Section 2. To constrain a particular trend, we condition the diffusion process using a trend TS  $s \in \chi$ . Following recent work on conditional diffusion models [5], we define the following model distribution:
$$
p _ {\theta} \left(\mathbf {x} _ {0: T} \mid \mathbf {s}\right) := p \left(\mathbf {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {s}\right), \quad p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {s}\right) := \mathcal {N} \left(\mathbf {x} _ {t - 1}; \boldsymbol {\mu} _ {\theta} \left(\mathbf {x} _ {t}, t \mid \mathbf {s}\right), \boldsymbol {\Sigma} _ {\theta} \left(\mathbf {x} _ {t}, t \mid \mathbf {s}\right)\right). \tag {6}
$$
which we learn by extending the parametrization in Eq. 4 with a conditional denoising function  $\epsilon_{\theta}$ :
$$
\boldsymbol {\mu} _ {\theta} \left(\mathbf {x} _ {t}, t \mid \mathbf {s}\right) = \frac {1}{\sqrt {\hat {\alpha} _ {t}}} \left(\mathbf {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \hat {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\theta} \left(\mathbf {x} _ {t}, t \mid \mathbf {s}\right)\right) \tag {7}
$$
where the  $\boldsymbol{\Sigma}_{\theta}(\mathbf{x}_t,t\mid \mathbf{s}) = \sigma^2\mathbf{I}$ . In this formulation, the trend is provided during each diffusion step  $t$  without any noise added to the conditioning trend. During the training, we extract the trend s directly from the input TS  $\mathbf{x}_0$ , which can be a simple linear or polynomial interpolation; during inference, the trend can be defined by the user at inference time. We recall that this is a soft constraint, meaning that we do not expect the generated TS to exactly retrace the trend. In particular, during training, we provide a trend that is a low-order polynomial approximation of  $\mathbf{x}_0$  to avoid the model from copying the trend s. Figure 1.b shows an example of the trend constraint.
Thus, the DiffTime training procedure minimizes the following revised loss function:
$$
L (\theta) := \mathbb {E} _ {t, \mathbf {x} _ {0}, \epsilon} \left\| \boldsymbol {\epsilon} - \boldsymbol {\epsilon} _ {\theta} \left(\sqrt {\hat {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \hat {\alpha} _ {t}} \boldsymbol {\epsilon}, t \mid \mathbf {s}\right) \right\| ^ {2} \tag {8}
$$
Fixed Points. To satisfy the fixed point constraints, which are hard constraints, we modify the reverse process of DiffTime to explicitly include them in the latent variables  $\mathbf{x}_{1:T}$ . We recall that  $\mathcal{R}$  is the set of fixed point constraints, such that a fixed point constraint  $r_{i,j} \in \mathcal{R}$  with  $(i,j) \in [L] \times [K]$ . Thus, at each diffusion step  $t$  we explicitly enforce the fixed-points values in the noisy time-series  $\mathbf{x}_t$ , such that  $\forall r_{i,j} \in \mathcal{R}$ ,  $x_{i,j} = r_{i,j}$  where  $x_{i,j} \in \mathbf{x}_t$ . This approach would guarantee that the generated TS have the desired fixed-point values. Most importantly, we experimentally validated that the forward process generates consistent neighboring points (around the constrained fixed-points) which means that the synthetic samples are conditioned by the fixed points, and preserve the realism of the original input data. During training, we randomly sample the fixed points from the input TS  $(\mathbf{x}_0)$  and require the diffusion process to conform to those fixed points. At inference, the fixed points can be provided by the user. Figure 1.c shows an example of a fixed point at the end of the TS, where the TS adapts to deal with the fixed point.
In the supplementary material, we provide additional details, network architecture, and the algorithm pseudo-codes.
3.3 Loss-DiffTime - Constrained generation with diffusion models