SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The asymmetric Laplace distribution is a generalization of the (symmetric) Laplace distribution and can be viewed as two exponential distributions with unequal rates glued about a location parameter. An alternative parameterization of this distribution has been employed for Bayesian quantile regression [58],
$$
p _ {\theta} (z) \propto \exp \left(- \frac {1}{b} \max \left\{\kappa \cdot (z - m), (\kappa - 1) \cdot (z - m) \right\}\right),
$$
where  $\kappa$  is the quantile level,  $m$  is the location parameter, and  $b$  is the scale parameter. Setting  $\kappa = 0.5$  yields the symmetric Laplace distribution. The negative log-likelihood of the asymmetric Laplace distribution corresponds to the pinball loss. An illustration of the probability density function (PDF) and the unnormalized negative log-likelihood (NLL) for different quantile levels is shown in Fig. 4.
(a) PDF
(b) NLL
Figure 4: Probability density function (PDF) and negative log-likelihood (NLL) of the unnormalized Asymmetric Laplace Distribution for different quantile levels  $(\kappa)$  with  $m = 0$  and  $b = 1$ .
Observation Self-Guidance. The observation self-guidance can be easily implemented using guided reverse diffusion [10]. The pseudo-code of observation self-guidance is provided in Alg. 1. Given an observation,  $\mathbf{y}_{\mathrm{obs}}$ , and guidance scale,  $s$ , the algorithm starts from Gaussian noise,  $\mathbf{x}^T$ , which is iteratively denoised while being guided towards the observation.
Algorithm 1 Observation Self-Guidance
Input: observation  $\mathbf{y}_{\mathrm{obs}}$  , scale  $s$ $\mathbf{x}^T\sim \mathcal{N}(\mathbf{0},\mathbf{I})$    
for  $t = T$  to 1 do   
 $\mathbf{x}^{t - 1}\sim \mathcal{N}(\mu_{\theta}(\mathbf{x}^t,t),\mathbf{I}) + s\sigma_t^2\nabla_{\mathbf{x}^t}\log p(\mathbf{y}_{\mathrm{obs}}|\mathbf{x}^t)$    
end for
Refinement. The energy-based refinement scheme requires an observation,  $\mathbf{y}_{\mathrm{obs}}$ , a base forecaster,  $g$ , a step size,  $\eta$ , and a noise factor,  $\gamma$ . The pseudo-code of the energy-based refinement scheme is shown in Alg. 2. First,  $\mathbf{y}_{\mathrm{obs}}$  and the prediction of the base forecaster,  $g(\mathbf{y}_{\mathrm{obs}})$ , are combined to  $\tilde{\mathbf{y}}$  and used as initial guess,  $\mathbf{y}_{(0)}$ . This initial guess is then iteratively refined using the score function of the conditional distribution,  $p_{\theta}(\mathbf{y}_{(i)}|\tilde{\mathbf{y}})$ . Setting  $\gamma = 0$  in Alg. 2 yields the maximum likelihood refinement scheme.
Algorithm 2 Energy-Based Refinement
Input: observation  $\mathbf{y}_{\mathrm{obs}}$  , base forecaster  $g$  , step size  $\eta$  noise factor  $\gamma$  , number of iterations  $N$ $\tilde{\mathbf{y}}\gets \mathrm{Combine}(\mathbf{y}_{\mathrm{obs}},g(\mathbf{y}_{\mathrm{obs}}))$ $\mathbf{y}_{(0)}\gets \tilde{\mathbf{y}}$    
for  $i = 0$  to  $N - 1$  do   
 $\xi_{i}\sim \mathcal{N}(0,\mathbf{I})$ $\mathbf{y}_{(i + 1)}\leftarrow \mathbf{y}_{(i)} - \eta \nabla_{\mathbf{y}_{(i)}}\log p_{\theta}(\mathbf{y}_{(i)}|\tilde{\mathbf{y}}) + \sqrt{2\eta\gamma}\xi_{i}$    
end for
We used eight popular univariate datasets from different domains in our experiments. Preprocessed versions of these datasets are available in GluonTS [2] with associated frequencies (daily or hourly) and prediction lengths. Table 5 shows an overview of the datasets. We used sequence lengths  $(L, \text{Context Length} + \text{Prediction Length})$  of 360 (i.e., 15 days) for the hourly datasets and 390 (i.e., approximately 13 months) for the daily datasets. These sequences were generated by slicing the original time series at random timesteps. In the following, we briefly describe the individual datasets.
- Solar [29] is a dataset of the photo-voltaic (i.e., solar) power production in the year 2006 from 137 solar power plants in the state of Alabama.5
- Electricity [29, 11] is a dataset of the electricity consumption of 370 customers.
- Traffic [29, 11] is a dataset of the hourly occupancy rates on the San Francisco Bay area freeways from 2015 to 2016.<sup>7</sup>
- Exchange [29] consists of daily exchange rates of eight countries including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore from 1990 to 2016.8
- M4-Hourly [37] is the hourly subset from the M4 competition.
- KDDCup [16] is a dataset of the air quality indices (AQIs) of Beijing and London used in the KDD Cup 2018.10
- UberTLC-Hourly [13] is a dataset of Uber pickups from Jan to Jun 2015 obtained from the NYC Taxi & Limousine Commission (TLC).<sup>11</sup>
- Wikipedia [14] is a dataset of daily count of the number of hits on 2000 Wikipedia pages.12
Table 5: Overview of the benchmark datasets used in our experiments.
Dataset	GluonTS Name	Train Size	Test Size	Domain	Freq.	Median Seq. Length	Context Length	Prediction Length
Solar	solar_nips	137	959	R+	H	7009	336	24
Electricity	electricity_nips	370	2590	R+	H	5833	336	24
Traffic	traffic_nips	963	6741	(0,1)	H	4001	336	24
Exchange	exchange_rate_nips	8	40	R+	D	6071	360	30
M4-Hourly	m4_hourly	414	414	N	H	960	312	48
KDDcup	kdd_cup_2018 Without MISSING	270	270	N	H	10850	312	48
UberTLC-Hourly	uber_tlc_hourly	262	262	N	H	4320	336	24
Wikipedia	wiki2000_nips	2000	10000	N	D	792	360	30
$^{5}$ Solar: https://www.nrel.gov/grid/solar-power-data.html
<sup>6</sup>Electricity: https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
7Traffic: https://zenodo.org/record/4656132
Exchange: https://github.com/laiguokun/multivariate-time-series-data
9M4-Hourly: https://github.com/Mcompetitions/M4-methods/tree/master/Dataset
10KDDCup: https://zenodo.org/record/4656756
11UberTLC-Hourly: https://github.com/fivethirtyeight/uber-tlc-foil-response
$^{12}$ Wikipedia: https://github.com/mbohlkeschneider/gluon-ts/tree/mv_release/datasets
B.2 Metrics
Continuous Ranked Probability Score (CRPS). The CRPS is a popular metric used to evaluate the quality of probabilistic forecasts. It is a proper scoring rule [15] and is defined as the integral of the pinball loss from 0 to 1, i.e.,
$$
\operatorname {C R P S} (F ^ {- 1}, y) = \int_ {0} ^ {1} 2 \Lambda_ {\kappa} (F ^ {- 1} (\kappa), y) d \kappa ,
$$
where  $\Lambda_{\kappa}(q,y) = (\kappa -\mathbb{1}_{\{y < q\}})(y - q)$  is the pinball loss for a specific quantile level  $\kappa$ . The CRPS measures the compatibility between the predicted inverse cumulative distribution function (also known as the quantile function),  $F^{-1}$ , and the observation,  $y$ . In practice, the quantile function is not analytically available and the CRPS is approximated using discrete quantile levels derived from samples. We used the implementation in GluonTS [2], which defaults to nine quantile levels,  $\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\}$ , estimated using 100 sample forecasts.
Linear Predictive Score (LPS). The LPS is our proposed metric to evaluate the predictive quality of synthetic time series samples from a generative model. We define the LPS as the test CRPS of a linear ridge regression model trained on the synthetic samples. We trained the ridge regression model using 10,000 synthetic samples and computed the CRPS on the corresponding real test set. Note that the CRPS of a point forecaster, such as the linear model, is equal to the 0.5-quantile loss (also known as the Normalized Deviation).
B.3 Hyperparameters and Training Details
We trained TSDiff using the Adam optimizer for 1,000 epochs with a learning rate of 1e-3. Every epoch is comprised of 128 batches constructed with 64 sequences each. We used 3 residual layers with 64 channels in the backbone of TSDiff. We also added a skip connection from input to output for some datasets as we observed improvements in the validation performance. The number of timesteps in the diffusion process was set to  $T = 100$  and we used a linear scheduler with  $\beta_{1} = 0.0001$  and  $\beta_{100} = 0.1$ . The diffusion timesteps were encoded into 128-dimensional positional embeddings [53]. Table 6 provides an overview of the hyperparameters.
Table 6: Hyperparameters of TSDiff.
Hyperparameter	Value
Learning rate	1e-3
Optimizer	Adam
Batch size	64
Epochs	1000
Gradient clipping threshold	0.5
Residual layers	3
Residual channels	64
Time Emb. Dim.	128
Diffusion steps T	100
β scheduler	Linear
β1	0.0001
β100	0.1
Normalization. The magnitudes of the time series can vary significantly, even within the same dataset. Training on raw values of such time series can lead to optimization instabilities. Therefore, we normalized the time series before training. Specifically, we used the mean scalar in GluonTS [2] which divides each time series individually by the mean of the absolute values of its observed context,
$$
\mathbf {y} _ {\mathrm {o b s}} ^ {\mathrm {n o r m}} = \frac {\mathbf {y} _ {\mathrm {o b s}}}{\mathrm {m e a n} (\mathrm {a b s} (\mathbf {y} _ {\mathrm {o b s}}))},
$$
where  $\mathrm{abs}(\cdot)$  is the element-wise absolute function and  $\mathrm{mean}(\cdot)$  denotes the mean aggregator. After inference, i.e., after computing  $\mathbf{y}_{\mathrm{ta}}^{\mathrm{norm}}$ , we rescale the target back to the original magnitude,
$$
\mathbf {y} _ {\mathrm {t a}} = \mathbf {y} _ {\mathrm {t a}} ^ {\text {n o r m}} \cdot \text {m e a n} (\text {a b s} (\mathbf {y} _ {\text {o b s}})).
$$
Guidance Scale Selection. The guidance scale in observation self-guidance (see Alg. 1) controls the strength of guidance and was selected on the basis of the validation performance. We tuned the scale from the set  $\{2/32, 3/32, 4/32, 5/32\}$  for mean square guidance and observed that it performs well with the scale  $4/32$  on every dataset. For quantile guidance, we tuned the scale from the set  $\{1, 2, 4, 8\}$ . Table 7 reports the best performing scale for quantile guidance on each dataset. The guidance scale also controls the alignment between the diffused time series and the observations. Fig. 5 shows that diffused time series from TSDiff-Q align closely with the observations.
Figure 5: Predictions generated by TSDiff-Q for time series in Electricity, KDDcup, and Exchange.
Table 7: Guidance scales,  $s$  ,for quantile self-guidance on each dataset.
Dataset	Quantile Guidance Scale
Exchange	8
Solar	8
Electricity	4
Traffic	4
M4-Hourly	2
KDDCup	1
UberTLC-Hourly	2
Wikipedia	2
Forecasting with Missing Values. As discussed in Sec. 4, we evaluated the forecasting performance of TSDiff under three missing value scenarios to demonstrate its flexibility. Specifically, we examined the ability of observation self-guidance to handle missing values during inference. For each dataset, we trained a single unconditional model for TSDiff-Q without any modifications to the training objective. During inference, we masked  $50\%$  of the timesteps in the context window during inference, e.g., 168 out of 336 timesteps on the Solar dataset. The selection of which timesteps to mask depended on the missing value scenario (see Fig. 6 for an illustration of the three scenarios). Observation self-guidance was then used to predict both the missing and the target values (i.e., the forecast). The CRPS was only computed for the forecast horizon. To ensure that TSDiff did not train on the timesteps that would be masked during inference, we excluded a window of length equal to  $L$  (context length + prediction length) from the end of the training time series. Consequently, the M4-Hourly and Wikipedia datasets were excluded from the missing value experiments due to the resulting time series being too short.
Figure 6: Illustration of the three missing value scenarios considered in our experiments.
On the other hand, the conditional models (TSDiff-Cond) were trained individually for each scenario, i.e., the missing values were masked and the model was optimized to predict them correctly.
Computational Cost of Observation Self-Guidance. Table 8 reports the inference time of observation self-guidance against the conditional diffusion model (TSDiff-Cond) on the Exchange dataset while controlling for aspects such as batch size, number of samples and GPU. The additional overhead for observation self-guidance (i.e., TSDiff-MS and TSDiff-Q) comes from the computation of the gradient of a neural network (see the Eq. 10).
Table 8: Comparison of inference times of TSDiff-Cond and observation self-guidance on the Exchange dataset.
Model	Inference Time (seconds)
TSDiff-Cond	162.97
TSDiff-MS	201.08
TSDiff-Q	201.67
Refinement Representative Step. As discussed in Sec. 3.2, we use a representative diffusion step to approximate the ELBO instead of randomly sampling multiple diffusion steps. The representative step corresponds to the diffusion step that best approximates the average loss. Fig. 7 shows the loss per diffusion step, the average loss and the representative step,  $\tau$ , for the eight datasets used in our experiments. We computed the loss per diffusion step on a randomly-sampled batch of 1024 datapoints, which took  $\approx 13$  seconds per dataset on a single Tesla T4 GPU.
Refinement Iterations Selection. To strike a balance between convergence and computational efficiency, our refinement scheme necessitates careful selection of the number of refinement iterations. The relationship between the relative CRPS and the number of iterations is visualized in Fig. 8 for the Solar dataset. Notably, immediate improvement is observed after just one iteration for point forecasters like Seasonal Naive and Linear. Conversely, DeepAR and Transformer exhibit gradual enhancements. This analysis was conducted solely on the Solar dataset, determining that 20 refinement iterations offer a favorable balance between performance and computational requirements. As a result, we set the number of refinement iterations to 20 for all other datasets and base forecasters.
We compared against eleven baselines, both from statistical literature and deep learning-based models, for the forecasting experiments. In the following, we briefly describe the individual baselines.
(a) Solar
(b) Electricity
(c) Traffic
(d) Exchange
(e) M4-Hourly
(f) KDDCup
(g) UberTLC-Hourly
(h) Wikipedia
Figure 7: Loss per diffusion step,  $t$ , and representative diffusion step,  $\tau$ , for the eight datasets used in our experiments.
Figure 8: Variation of the relative CRPS with the number of refinement iterations on the Solar dataset.
- Seasonal Naive is a naive forecaster that returns the value from the last season as the prediction, e.g., the value from 24 hours (1 day) ago for time series with hourly frequency.
- ARIMA [21] is a popular statistical model for analyzing and forecasting time series data, combining autoregressive (AR), differencing (I), and moving average (MA) components to capture underlying patterns and fluctuations. We used the implementation available in the forecast package [22] for R.
- ETS [21] is a forecasting method that uses exponential smoothing to capture trend, seasonality, and error components in the time series. We used the implementation available in the forecast package [22] for R.
Figure 9: A schematic of the architecture of the conditional model, TSDiff-Cond. The key difference from TSDiff is the incorporation of the conditioning input, c, through Conv1x1 layers.
- Linear is a linear ridge regression model across time, i.e., it takes the past context length timesteps as inputs and outputs the next prediction length timesteps. We used the implementation available in scikit-learn [40] with the regularization strength of 1 (its default value in scikit-learn). The model was fit using 10,000 randomly sampled sequences with the same context length for each dataset as for TSDiff.
- DeepAR [46] is an RNN-based autoregressive model that is conditioned on the history of the time series through lags and other relevant features, e.g., time features such as hour-of-day and day-of-week. The model autoregressively outputs the parameters of the next distribution (e.g., Student's-t and Gaussian) and is trained to maximize the (conditional) log-likelihood. We used the implementation available in GluonTS [2] with the recommended hyperparameters.
- MQ-CNN utilizes a convolutional neural network (CNN) architecture to capture patterns and dependencies within the time series data. The model is trained using the quantile loss to directly generate multi-horizon forecasts. We used the implementation available in GluonTS [2] with the recommended hyperparameters.
- DeepState [42] combines RNNs with linear dynamical systems (LDS). The RNN takes additional covariates (e.g., time features and item ID) as input and outputs the (diagonal) noise covariance matrices of the LDS. Other parameters of the LDS such as the transition and emission matrices are manually designed to model different time series components such as level, trend, and seasonality. The LDS and RNN are jointly trained via maximum likelihood estimation. We used the implementation available in GluonTS [2] with the recommended hyperparameters.
- Transformer is a sequence-to-sequence forecasting model based on the self-attention architecture [53]. The model takes time series lags and covariates as inputs and outputs the parameters of future distributions. We used the implementation available in GluonTS [2] with the recommended hyperparameters.
- TFT [32] is another sequence-to-sequence forecasting model based on the self-attention architecture that includes a variable selection mechanism to minimize the contributions of irrelevant input features. The model is trained to minimize the quantile loss at the selected quantiles. We used the implementation available in GluonTS [2] with the recommended hyperparameters.
- CSDI [52] is a conditional diffusion model for multivariate time series imputation and forecasting. Similar to TSDiff, CSDI is based on the DiffWave architecture [28], but utilizes transformer layers as fundamental building blocks. The original CSDI architecture was presented in the context of multivariate time series comprising temporal and feature transformer layers. We used the original implementation<sup>13</sup> in the univariate setting with the recommended hyperparameters and training setup in our experiments.
- TSDiff-Cond is a conditional version of TSDiff. It integrates the observed context together with an observation mask using a Conv1x1 layer followed by an addition after the S4 layer
$^{13}$ CSDI implementation: https://github.com/ermongroup/CSDI
in each residual block as shown in Fig 9. TSDiff-Cond's architecture resembles that of SSSD [1] with three key differences:
- TSDiff-Cond uses S4 layers in the residual blocks before addition with the conditioning vector whereas SSSD uses them both before and after addition.
- TSDiff-Cond is a univariate model where feature dimensions indicate lagged time series values whereas SSSD is a multivariate model with feature dimensions indicating the different features in the multidimensional time series.
- TSDiff-Cond computes the loss on the entire time series, unlike SSSD which only generates the unobserved timesteps. This led to better performance of the conditional model in our experiments.
We used the same hyperparameters (e.g., context lengths and lags) for TSDiff-Cond as for TSDiff (i.e., the unconditional model) to isolate the effect of conditional training.
For the train on synthetic-test on real experiments, we compared against two popular time series generative models.
- TimeVAE is based on variational autoencoders and includes network components specific to time series such as trend and seasonality blocks. We used the original implementation<sup>14</sup> and recommended hyperparameters in our experiments.
- TimeGAN employs an autoencoder to train a generative adversarial network in the latent space. The network components are trained using a combination of supervised, unsupervised, and discriminative loss functions. We used the original implementation<sup>15</sup> and recommended hyperparameters in our experiments.
C Synthetic Samples
Figs. 10 to 17 illustrate synthetic time series created by TimeVAE, TimeGAN, and TSDiff in comparison to real time series. The generated samples from TSDiff closely resemble the real samples across all datasets, exhibiting higher quality compared to the baselines. For instance, in the case of the solar dataset (Fig. 10), TSDiff accurately captures periods of zero solar energy production during nighttime and generates a wide range of peaks within individual time series. In contrast, samples from TimeVAE and TimeGAN appear more homogeneous and fail to capture the zero values precisely. TimeGAN also experiences mode collapse issues in certain datasets like Electricity, M4-Hourly, and UberTLC-Hourly, while TSDiff consistently generates diverse and realistic samples.
Figure 10: Real samples and synthetic samples generated by TSDiff, TimeGAN, and TimeVAE for the Solar dataset.
$^{14}$ TimeVAE implementation: https://github.com/abudesai/timeVAE
15TimeGAN implementation: https://github.com/jsyoon0823/TimeGAN
Figure 11: Real samples and synthetic samples generated by TSDiff, TimeGAN, and TimeVAE for the Electricity dataset.
Figure 12: Real samples and synthetic samples generated by TSDiff, TimeGAN, and TimeVAE for the Traffic dataset.
Figure 13: Real samples and synthetic samples generated by TSDiff, TimeGAN, and TimeVAE for the Exchange dataset.
Figure 14: Real samples and synthetic samples generated by TSDiff, TimeGAN, and TimeVAE for the M4-Hourly dataset.
Figure 15: Real samples and synthetic samples generated by TSDiff, TimeGAN, and TimeVAE for the UberTLC-Hourly dataset.
Figure 16: Real samples and synthetic samples generated by TSDiff, TimeGAN, and TimeVAE for the KDDCup dataset.
Figure 17: Real samples and synthetic samples generated by TSDiff, TimeGAN, and TimeVAE for the Wikipedia dataset.