Time-series Generation by Contrastive Imitation
Year: 2022

Abstract:
Consider learning a generative model for time-series data. The sequential setting poses a unique challenge: Not only should the generator capture the conditional dynamics of (stepwise) transitions, but its open-loop rollouts should also preserve the joint distribution of (multi-step) trajectories. On one hand, autoregressive models trained by MLE allow learning and computing explicit transition distributions, but suffer from compounding error during rollouts. On the other hand, adversarial models based on GAN training alleviate such exposure bias, but transitions are implicit and hard to assess. In this work, we study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy, where the reinforcement signal is provided by a global (but stepwise-decomposable) energy model trained by contrastive estimation. At training, the two components are learned cooperatively, avoiding the instabilities typical of adversarial objectives. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality. By expressly training a policy to imitate sequential behavior of time-series features in a dataset, this approach embodies “generation by imitation”. Theoretically, we illustrate the correctness of this formulation and the consistency of the algorithm. Empirically, we evaluate its ability to generate predictively useful samples from realworld datasets, verifying that it performs at the standard of existing benchmarks.

heading: 1 Introduction
text: Time-series data are ubiquitous in diverse machine learning applications, such as financial, industrial, and healthcare settings. At the same time, lack of public access to data is a recurring obstacle to the development and reproducibility of research in domains where datasets are proprietary [1]. Generating synthetic—but realistic—time-series data is a promising solution [2], and has received increasing attention in recent years, driven by advances in deep learning and generative adversarial networks [3,4].
Owing to the fact that time-series features are generated sequentially, generative modeling in the temporal setting faces a two-pronged challenge: First, a good generator should accurately capture the conditional dynamics of stepwise transitions p(xt|x1, ..., xt−1); this is important, as the faithfulness of any conceivable downstream time-series analysis depends on the learned correlations across both temporal and feature dimensions. Second, however, the recursive rollouts of the generator should also respect the joint distribution of multi-step trajectories p(x1, ..., xT ); this is equally important, as synthetic trajectories that inadvertently wander beyond the support of original data are useless at best.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Recent work falls into two main categories. On one hand, autoregressive models trained via MLE [5] explicitly factor the distribution of trajectories into a product of conditionals ∏ t p(xt|x1, ..., xt−1). While this allows directly learning and computing such transitions, with finite data this is prone to compounding errors during multi-step generation, due to the discrepancy between closed-loop training (i.e. conditioned on ground-truths as inputs) and open-loop sampling (i.e. conditioned on its own previous outputs) [6]. A variety of methods have sought to counteract this problem of exposure bias, employing auxiliary techniques from curriculum learning [7, 8] and adversarial domain adaptation [9, 10]; however, such remedies are not without biases [11], and empirical improvements have been mixed [12–14].
On the other hand, adversarial models based on GAN training and its relatives [15–17] directly model the distribution of trajectories p(x1, ..., xT ) [18–20]. To provide a more granular learning signal for the generator, a popular variant matches the induced distribution of sub-trajectories instead, providing stepwise feedback from the discriminator [21, 22]. TimeGAN [12] is the most recent incarnation of this, and operates within a jointly optimized latent space. GAN-based approaches alleviate the risk of compounding errors, and have been applied to banking [23], sensors [24], biosignals [25], and smartgrids [26]. However, the conditional dynamics are only implicitly learned, yielding no way of inspecting or assessing the quality of sampled transitions nor trajectories. Moreover, the adversarial objective leads to characteristically challenging optimization—exacerbated by the temporal dimension.
Three Operations Consider a probabilistic generative model p for some datasetD. We are generally interested in performing one or more of the following operations: (1) sampling a time series τ ∼ p, (2) evaluating the likelihood p(τ), and (3) learning the model p from a set of i.i.d. samples τ . In light of the preceding, we investigate a generative framework that attempts to fulfill the following criteria:
• Samples should respect both the stepwise conditional distributions of features, as well as the joint distribution of full trajectories; unlike pure MLE, we wish to avoid multi-step compounding error.
• Evaluating likelihoods should be possible as generic measures of sample quality for both transitions and trajectories—often desired for sample comparison, model auditing, or bias correction [27, 28].
• Unlike black-box GAN discriminators, we wish that the evaluator be decoupled from any specific sampler, such that the two components can be trained non-adversarially, thus may be more stable.
Contributions In the sequel, we explore an approach that seeks to satisfy these criteria. We first give precise treatment of the “compounding error” problem, thus motivating a specific trajectory-centric optimization objective from first principles (Section 2). To carry it out, we develop a general training framework and practical algorithm, along with its theoretical justification: We train a forward-looking transition policy to imitate the sequential behavior of time series using a stepwise-decomposable energy model as reinforcement, giving a method that embodies “generation by imitation” (Section 3). Importantly, to understand its strengths and limitations, we compare the method to existing generative models for time-series data, and relate it to imitation learning of sequential behavior (Section 4). Lastly, through experiments with application to real-world time-series datasets, we verify that it generates predictively useful samples that perform at the standard of comparable benchmarks (Section 5).

heading: 2 Synthetic Time Series

heading: 2.1 Problem Setup
text: We operate in the standard discrete-time setting for time series. Let feature vectors xt ∈ X be indexed by time steps t, and let a full trajectory of length T be denoted τ := (x1, ..., xT ) ∈ T := X T . Also, denote with ht := (x1, ..., xt−1) ∈ H := ∪Tt=1X t the history prior to time t. For ease of exposition we shall work with trajectories of fixed lengths T , but our results trivially generalize to the case where T itself is a random variable (for instance, by employing padding tokens up to some maximum length).
Consider a dataset D := {τn}Nn=1 of N trajectories sampled from some true source s. We assume the trajectories are generated sequentially by some unknown transition process πs ∈ ∆(X )H, such that features at each step t are sampled as xt ∼ πs(·|ht). In addition to this stepwise conditional, denote with µs(h) := 1T ∑ t p(ht = h|πs) the normalized occupancy measure—i.e. the distribution of histories induced by πs. Intuitively, this is the visitation distribution of “history states” encountered by a generator when navigating about the feature space X by rolling out policy πs. With slight abuse of notation, we may also write µs(h, x) := µs(h)πs(x|h) to indicate the marginal distribution of transitions. Finally, let the joint distribution of full trajectories be denoted by ps(τ) := ∏ t πs(xt|ht).
The goal is to learn a sequential generator πθ parameterized as θ using samples τ ∼ ps from D, such that pθ ≈ ps. Note here that we do not assume stationarity of the time-series data, nor stationarity of the transition conditionals; any influence of t is implicit through the dependence of πs (and πθ) on variable-length histories. In line with recent work [14, 20], for simplicity we do not consider static metadata as supplemental inputs or outputs, as these are commonly and easily incorporated via an additional conditioning layer or auxiliary generator [12, 19]. Lastly, note that much recent work on sequential modeling is devoted to domain-specific, architecture-level designs for generating audio [29, 30], text [31, 32], and video [33, 34]. In contrast, our work is closer in spirit to [12, 14] in being an agnostic, framework-level study applicable to generic tabular data in any time-series setting.
Measuring Sample Quality How do we determine the “quality” of a sample? In specialized domains, of course, we often have prior access to task-specific metrics such as BLEU or ROUGE scores in text generation [6, 35]—then, the generator can simply be optimized for such scores via standard methods in reinforcement learning [36]. In generic time-series settings, however, the challenge is that any such metric must necessarily be task-agnostic, and access to it must necessarily come from learning.
So, for any data source s, let us speak of some hypothetical function fs : H×X→ [−c, c] with c <∞, such that fs(h, x) gives the quality of any sampled transition—that is, any tuple (h, x). Intuitively, we may interpret this as quantifying how “typical” it is for the random process to be in state h and step towards x. Likewise, let as also speak of some function Fs : T → [−cT, cT ] such that Fs(τ) gives the quality of any sampled trajectory. Naturally, in time-series settings where the underlying process is causally-conditioned, it is reasonable to define this as the decomposition Fs(τ) := ∑ t fs(ht, xt). Now of course, we have no access to the true Fs. But clearly, in learning a generative model pθ of ps, we wish that the quality of samples τ drawn from pθ and ps be similar in expectation. More precisely:
Definition 1 (Expected Quality Difference) Let ∆F̄s :Θ→ [−2cT, 2cT ] denote the expected quality difference between ps and pθ, where Θ indicates the space of parameterizations for generator πθ:
∆F̄s(θ) := Eτ∼psFs(τ)− Eτ∼pθFs(τ) (1)
Our objective, then, is to learn a generator πθ that minimizes the expected quality difference ∆F̄s(θ). Two points bear emphasis. First, we know nothing about Fs—beyond it being the sequential aggregate of fs. This challenge uniquely differentiates this agnostic setting from more popular media-specific applications—for which various predefined measures are readily available for supervision. Second, in addition to matching this expectation over samples, we also wish to match the variety of samples in the original data. After all, we want pθ to mimic samples from ps of different degrees of “typicality”. So we should expect to incorporate some measure of entropy, e.g. the commonly used Shannon entropy.

heading: 2.2 Matching Local Moments
text: Recall the apparent tradeoff between autoregressive models and adversarial models. In the spirit of the former, suppose we seek to directly learn transition conditionals via supervised learning. That is,
argminθ Eh∼µsL(πs(·|h), πθ(·|h)) (2)
Consider the log likelihood loss L(πs(·|h), πθ(·|h)) := Ex∼πs(·|h) log πθ(x|h). In the case of exponential family models for πθ(·|h), a basic result is that this is dual to maximizing its conditional entropy subject to the constraint on feature expectations Eh∼µs;x∼πθ(·|h)T (x) = Eh∼µs;x∼πs(·|h)T (x), where T : X → R is some sufficient statistic [37–39]. More generally for deep energy-based models, we have (however, recall that strong duality does not generalize to the nonlinear case; see Appendix A):
argminθ ( E h∼µs x∼πθ(·|h) log πθ(x|h) + maxf∈RH×X ( E h∼µs x∼πs(·|h) f(h, x)− E h∼µs x∼πθ(·|h) f(h, x) )) (3)
Note that the moment-matching constraint is local—that is, at the level of individual transitions, and all conditioning is based on h from µs alone. This is precisely the “exposure bias”: The objective is only ever exposed to inputs h drawn from the (perfect) source distribution µs, and is thus unaware of the endogeneity of the (imperfect) synthetic distribution µθ induced by πθ. This is not desirable since πθ is rolled out by open-loop sampling at test time. Now, although at the global optimum the momentmatching discrepancy must be zero (i.e. the equality constraint is enforced), in practice there may be a variety of reasons why this is not perfectly achieved (e.g. error in estimating expectations, error in
function approximation, error in optimization, etc). Suppose we could bound how well we are able to enforce the moment-matching constraint; as it turns out, we cannot eliminate error compounding:1
Lemma 1 Let maxf∈RH×X ( E h∼µs x∼πs(·|h) f(h, x)−E h∼µs x∼πθ(·|h) f(h, x) ) ≤ . Then ∆F̄s(θ) ∈ O(T 2 ). Proof. Appendix A.
This reveals the problem with modeling conditionals per se: Not all mistakes are equal. An objective like Equation 2 penalizes unrealistic transitions (h, x) by treating all conditioning histories h equally— regardless of how realistic h is to begin with. Clearly, however, we care much less about how x looks like, if the current subsequence h is already highly unlikely (and vice versa). Intuitively, earlier mistakes in a trajectory should weigh more: Once πθ wanders into areas ofH with low support in µs, no amount of “good” transitions will bring the trajectory back to high-likelihood areas of T under ps.

heading: 2.3 Matching Global Moments
text: Now suppose instead that we seek to directly constrain the trajectory distribution pθ to be similar to ps:
argminθ L(ps, pθ) (4)
Consider the Kullback-Leibler divergenceL(ps, pθ) := DKL(ps‖pθ). Like before, we know that in the case of exponential family models for pθ, this is dual to maximizing its entropy subject to the constraint Eτ∼psT (τ) = Eτ∼pθT (τ), where T : T → R is some sufficient statistic [40]. More broadly for deep energy-based models, we have argminθ (Eτ∼pθ log pθ(τ) + maxF∈RT (Eτ∼psF (τ)−Eτ∼pθF (τ))) (but again, recall here that strong duality does not generalize to the nonlinear case; see Appendix A). Now, observe that by definition of occupancy measure µ, for any function f : H×X → R it must be the case that Eτ∼p ∑ t f(ht, xt) = TEh∼µ,x∼π(·|h)f(h, x). Therefore we may equivalently write
argminθ ( E h∼µθ x∼πθ(·|h) log πθ(x|h) + maxf∈RH×X ( E h∼µs x∼πs(·|h) f(h, x)− E h∼µθ x∼πθ(·|h) f(h, x) )) (5)
Importantly, note that the moment-matching constraint is now global—that is, at the level of trajectory rollouts, and πθ is now conditioned on histories h drawn from its own induced occupancy measure µθ. There is no longer any “exposure bias” here: In order to respect the constraint, not only does πθ(·|h) have to be close to πs(·|h) for any given h, but the occupancy measure µθ induced by πθ also has to be close to the occupancy measure µs induced by πs. As it turns out, this seemingly minor difference is sufficient to mitigate compounding errors. As before, although at the global optimum the momentmatching discrepancy must be zero, in practice this may not be perfectly achieved. Now, suppose we could bound how well we are able to enforce the moment-matching constraint; but we now have:
Lemma 2 Let maxf∈RH×X ( E h∼µs x∼πs(·|h) f(h, x) − E h∼µθ x∼πθ(·|h) f(h, x) ) ≤ . Then ∆F̄s(θ) ∈ O(T ). Proof. Appendix A.
This illustrates why even transition-centric adversarial models such as [12,21] have shown promise in generating realistic trajectories [23–26]. First, unlike trajectory-centric GANs [18, 19] which directly attempt to minimize some form of Equation 4, in transition-centric GANs the objective is to match the transition marginals µθ(h, x) and µs(h, x)—so the discriminator provides more granular feedback to the generator for training. At the same time, we see from Lemma 2 that matching transition marginals is already—indirectly—performing the sort of moment-matching that alleviates compounding error.
Can we be more direct? In Section 3, we shall start by tackling Equation 5 itself. As we shall see, this endeavor gives rise to a technique that trains a conditional policy (for sampling), an energy model (for evaluation), and a non-adversarial framework (for learning)—addressing our three initial criteria.

heading: 3 Generating by Imitating
text: First, consider the most straightforward implementation: Let us parameterize f ∈ RH×X as φ, and begin with the primal form of Equation 5, which yields the following adversarial learning objective:
1Lemmas 1 and 2 are similar in spirit to results for error accumulation in imitation by behavioral cloning and distribution matching. See Appendix A; this analogy with imitation learning is formally identified in Section 4.
L(θ, φ) := maxφ minθ ( E h∼µθ x∼πθ(·|h) log πθ(x|h) + E h∼µs x∼πs(·|h) fφ(h, x)− E h∼µθ x∼πθ(·|h) fφ(h, x) ) (6)
It is easy to see that this effectively describes variational training of the energy-based model pφ(τ) := exp(Fφ(τ)− logZφ)—where Fφ(τ) := ∑ t fφ(ht, xt)—to approximate the true ps(τ), using samples from the variational pθ. The (outer) energy player is the maximizing agent, and the (inner) policy player is the minimizing agent. The form of this objective naturally prescribes a bilevel optimization procedure in which we perform (gradient-based) updates of φwith nested (best-response) updates of θ.

heading: 3.1 Challenges of Learning
text: Abstractly, of course, training energy models using variational samplers is not new: Multiple works in static domains—such as image modeling—have investigated this approach as a means of bypassing the expense and variance of MCMC sampling [41, 42]. In our setting, however, there is the additional temporal dimension: The negative energy Fφ(τ) of any trajectory is computed as the sequential composition of stepwise qualities fφ(ht, xt), and each trajectory sampled from pθ must be generated as the sequential rollout of stepwise policies πθ(xt|ht). Consider the gradient update for the energy,
∇φL = E h∼µs x∼πs(·|h) ∇φfφ(h, x)− E h∼µθ x∼πθ(·|h) ∇φfφ(h, x) (7)
and the inner-loop update for the policy,
argminθ E h∼µθ x∼πθ(·|h) log πθ(x|h)− E h∼µθ x∼πθ(·|h) fφ(h, x) (8)
Note that the max-min optimization requires complete optimization within each inner update in order for the outer update to be correct. Otherwise the gradients will be biased, and there would be no guarantee the procedure converges to anything meaningful. Yet unlike in the static setting—for which there exists variety of standard approximations for the inner update [41–44]—here the policy update amounts to entropy-regularized reinforcement learning [45–47] using fφ(ht, xt) as reward function. Thus our first difficulty is computational: Repeatedly performing inner-loop RL is simply infeasible.
Now, an obvious alternative is to dispense with complete policy optimization at each step, and instead to employ importance sampling to ensure that the gradients for the energy updates are still unbiased:
∇φL = Eτ∼ps∇φFφ(τ)− 1
Zφ Eτ∼pθ
[ exp( ∑ t fφ(ht, xt))∏
t πθ(xt|ht) ∇φFφ(τ)
] (9)
where the partition function is computed as Zφ = Eτ∼pθ [exp( ∑ tfφ(ht, xt))/ ∑ tπθ(xt|ht)], and the sampling policy πθ is no longer required to be perfectly optimized with respect to fφ. Unfortunately, this strategy simply replaces the original difficulty with a statistical one: As soon as we consider time-series data of non-trivial lengths T , the multiplicative effect of each time step on the importance weights means the gradient estimates—albeit unbiased—will have impractically high variance [48,49].

heading: 3.2 Contrastive Imitation
text: We now investigate a generative framework that seeks to avoid these difficulties. The key idea is that instead of Equation 7, we shall learn pφ by contrasting (real) “positive” samples τ ∼ ps and (any) “negative” samples τ ∼ pθ, which—as we shall see—rids us of the requirement that πθ be fully optimized at each step for learning to be guaranteed. First, let us establish the notion of a “structured classifier”:2
Definition 2 (Structured Classification) Recall the πθ-induced distribution pθ(τ) := ∏ t πθ(xt|ht).
Denote with p̃φ the un-normalized energy-based model such that p̃φ(τ) := exp( ∑ t fφ(ht, xt)), and let Zφ be folded into φ as a learnable parameter. Define the structured classifier dθ,φ : T → [0, 1]:
dθ,φ(τ) :=
1 Zφ p̃φ(τ)
1 Zφ p̃φ(τ) + pθ(τ)
(10)
2The idea that density estimation can be performed by logistic regression goes back at least to [50], and formalized as negative sampling [51] and noise-contrastive estimation [52]. Structured classifiers have been studied in the context of imitation learning [53, 54] by analogy with GANs. In the time-series setting, however, we shall see that this approach is equivalent to noise-contrastive estimation with an adaptive noise distribution.
<latexit sha1_base64="tl4kFGgu9CCpudDg8URrpmHwRoI=">AAAB83icbVBNSwMxEM3Wr1q/qh69BIvgqeyKoseiF48V7Ad0l5JNs21okl2SiVCW/g0vHhTx6p/x5r8xbfegrQ8GHu/NMDMvzgQ34PvfXmltfWNzq7xd2dnd2z+oHh61TWo1ZS2ailR3Y2KY4Iq1gINg3UwzImPBOvH4buZ3npg2PFWPMMlYJMlQ8YRTAk4KQ2n7eQgjBmTar9b8uj8HXiVBQWqoQLNf/QoHKbWSKaCCGNML/AyinGjgVLBpJbSGZYSOyZD1HFVEMhPl85un+MwpA5yk2pUCPFd/T+REGjORseuUBEZm2ZuJ/3k9C8lNlHOVWWCKLhYlVmBI8SwAPOCaURATRwjV3N2K6YhoQsHFVHEhBMsvr5L2RT24rF89XNYat0UcZXSCTtE5CtA1aqB71EQtRFGGntErevOs9+K9ex+L1pJXzByjP/A+fwCDepIB</latexit>
<latexit sha1_base64="ltBEKEGmzWXf7MWdup2do/2ZSn0=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0WPRi8cKpi20oWy2k3bpZhN2N0IJ/Q1ePCji1R/kzX/jts1BWx8MPN6bYWZemAqujet+O6W19Y3NrfJ2ZWd3b/+genjU0kmmGPosEYnqhFSj4BJ9w43ATqqQxqHAdji+m/ntJ1SaJ/LRTFIMYjqUPOKMGiv5aT/X03615tbdOcgq8QpSgwLNfvWrN0hYFqM0TFCtu56bmiCnynAmcFrpZRpTysZ0iF1LJY1RB/n82Ck5s8qARImyJQ2Zq78nchprPYlD2xlTM9LL3kz8z+tmJroJci7TzKBki0VRJohJyOxzMuAKmRETSyhT3N5K2IgqyozNp2JD8JZfXiWti7p3Wb96uKw1bos4ynACp3AOHlxDA+6hCT4w4PAMr/DmSOfFeXc+Fq0lp5g5hj9wPn8AKveO7g==</latexit>
<latexit sha1_base64="NUd2qoKqbgBvuADu+Tfx0n/5fLA=">AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOOsmQmdllZlYISz7CiwdFvPo93vwbJ8keNLGgoajqprsrSgQ31ve/vcLa+sbmVnG7tLO7t39QPjxqmjjVDBssFrFuR9Sg4AoblluB7UQjlZHAVjS+m/mtJ9SGx+rRThIMJR0qPuCMWie1ujLtZWbaK1f8qj8HWSVBTiqQo94rf3X7MUslKssENaYT+IkNM6otZwKnpW5qMKFsTIfYcVRRiSbM5udOyZlT+mQQa1fKkrn6eyKj0piJjFynpHZklr2Z+J/XSe3gJsy4SlKLii0WDVJBbExmv5M+18ismDhCmebuVsJGVFNmXUIlF0Kw/PIqaV5Ug8vq1cNlpXabx1GEEziFcwjgGmpwD3VoAIMxPMMrvHmJ9+K9ex+L1oKXzxzDH3ifP7F1j9A=</latexit>
<latexit sha1_base64="tl4kFGgu9CCpudDg8URrpmHwRoI=">AAAB83icbVBNSwMxEM3Wr1q/qh69BIvgqeyKoseiF48V7Ad0l5JNs21okl2SiVCW/g0vHhTx6p/x5r8xbfegrQ8GHu/NMDMvzgQ34PvfXmltfWNzq7xd2dnd2z+oHh61TWo1ZS2ailR3Y2KY4Iq1gINg3UwzImPBOvH4buZ3npg2PFWPMMlYJMlQ8YRTAk4KQ2n7eQgjBmTar9b8uj8HXiVBQWqoQLNf/QoHKbWSKaCCGNML/AyinGjgVLBpJbSGZYSOyZD1HFVEMhPl85un+MwpA5yk2pUCPFd/T+REGjORseuUBEZm2ZuJ/3k9C8lNlHOVWWCKLhYlVmBI8SwAPOCaURATRwjV3N2K6YhoQsHFVHEhBMsvr5L2RT24rF89XNYat0UcZXSCTtE5CtA1aqB71EQtRFGGntErevOs9+K9ex+L1pJXzByjP/A+fwCDepIB</latexit>
<latexit sha1_base64="ltBEKEGmzWXf7MWdup2do/2ZSn0=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0WPRi8cKpi20oWy2k3bpZhN2N0IJ/Q1ePCji1R/kzX/jts1BWx8MPN6bYWZemAqujet+O6W19Y3NrfJ2ZWd3b/+genjU0kmmGPosEYnqhFSj4BJ9w43ATqqQxqHAdji+m/ntJ1SaJ/LRTFIMYjqUPOKMGiv5aT/X03615tbdOcgq8QpSgwLNfvWrN0hYFqM0TFCtu56bmiCnynAmcFrpZRpTysZ0iF1LJY1RB/n82Ck5s8qARImyJQ2Zq78nchprPYlD2xlTM9LL3kz8z+tmJroJci7TzKBki0VRJohJyOxzMuAKmRETSyhT3N5K2IgqyozNp2JD8JZfXiWti7p3Wb96uKw1bos4ynACp3AOHlxDA+6hCT4w4PAMr/DmSOfFeXc+Fq0lp5g5hj9wPn8AKveO7g==</latexit>
<latexit sha1_base64="sqYuxZYqdHzEbIbSfJs8zagSNqk=">AAAB73icbVBNSwMxEJ3Ur1q/qh69BIvgqexKRY9FLx4r2A9ol5JNs21oNhuTrFCW/gkvHhTx6t/x5r8xbfegrQ8GHu/NMDMvVIIb63nfqLC2vrG5Vdwu7ezu7R+UD49aJkk1ZU2aiER3QmKY4JI1LbeCdZRmJA4Fa4fj25nffmLa8EQ+2IliQUyGkkecEuukjupnPTXi03654lW9OfAq8XNSgRyNfvmrN0hoGjNpqSDGdH1P2SAj2nIq2LTUSw1ThI7JkHUdlSRmJsjm907xmVMGOEq0K2nxXP09kZHYmEkcus6Y2JFZ9mbif143tdF1kHGpUsskXSyKUoFtgmfP4wHXjFoxcYRQzd2tmI6IJtS6iEouBH/55VXSuqj6terlfa1Sv8njKMIJnMI5+HAFdbiDBjSBgoBneIU39Ihe0Dv6WLQWUD5zDH+APn8AZOuQNg==</latexit>
That is, unlike a black-box classifier that may be arbitrarily parameterized—such as a generic discriminator d in a GAN—here dθ,φ is “structured” in that it is modularly parameterized by the embedded energy and policy functions. Now, we shall train φ such that dθ,φ discriminates well between τ ∼ ps and τ ∼ pθ—that is, so that the output dθ,φ(τ) represents the (posterior) probability that τ is real,
Lenergy(φ; θ) := −Eτ∼ps log dθ,φ(τ)−Eτ∼pθ log ( 1−dθ,φ(τ) ) (11)
and as before, Lpolicy(θ;φ) := E h∼µθ
x∼πθ(·|h) log πθ(x|h)− E h∼µθ x∼πθ(·|h) fφ(h, x) (12)
Why is this better? As we now show formally, each gradient update no longer requires θ to be optimal for the current value of φ—nor does it require importance sampling—unlike the procedure described by Equation 6. The only requirement is that pθ can be sampled and evaluated efficiently, e.g. using learned Gaussian policies as usual, or—should more flexibility be required—with normalizing flowbased policies. As a practical result, this means policy updates can be interleaved with energy updates, instead of being nested within a repeated inner loop. Specifically, let us establish the following results:
Proposition 3 (Global Optimality) Let fφ ∈ RH×X , and let pθ ∈ ∆(T ) be any distribution satisfying positivity: ps(τ) > 0⇒ pθ(τ) > 0 (this does not require πθ be optimal for fφ). ThenLenergy(φ; θ) is globally minimized at Fφ(·)− logZφ = log ps(·), whence pφ is self-normalized with unit integral.
Proof. Appendix A.
This result is intuitive by analogy with noise-contrastive estimation [52, 55]: φ is learnable as long as negative samples τ ∼ pθ cover the support of the true ps. The positivity condition is mild (e.g. take Gaussian policies πθ), and so is the realizability condition (e.g. take neural-networks for fφ). Importantly, note that at optimality classifier dθ,φ is decoupled from any specific value of θ; contrast this with generic discriminators d in GANs, which are only ever optimal for the current generator. Now, in practice we must approximate ps and pθ using finite samples. In light of this, two questions are immediate: First, does the learned φ converge to the global optimum as the sample size increases? Second, what role does the “quality” of the policy’s samples play in how φ is learned? For the former:
Proposition 4 (Asymptotic Consistency) Let φ∗ denote the minimizer for Lenergy(φ; θ), and let φ̂∗M denote the minimizer for its finite-data approximation—that is, where the expectations over ps and pθ are approximated by M samples. Then under some mild conditions, as M increases φ̂∗M p−→ φ∗.
Proof. Appendix A.
Now for the second question: Clearly if pθ were too far from ps, learning would be slow—the job would be too easy for the classifier dθ,φ, and it may be able to distinguish samples via basic statistics alone. Indeed, in standard noise-contrastive estimation with a fixed noise distribution, learning is ineffective in the presence of many variables [56]. Precisely, however, that is why we continuously update the policy itself as an adaptive noise distribution: As pφ moves closer to ps, so does pθ—thus providing more “challenging” negative samples.3 In fact, should we insist on greedily taking each policy update to optimality, we recover a “weighted” version of the original max-min gradient from before:
3It is easy to see that minimizing Equation 12 equivalently minimizes the reverse KL div. between pφ and pθ .
Proposition 5 (Gradient Equality) Let φk be the value taken by φ after the k-th gradient update, and let θ∗k denote the associated minimizer for Lpolicy(θ;φk). Suppose pφ is already normalized; then
∇φLenergy(φ; θ∗k) = −T2∇φL(θ ∗ k, φ)
That is, at θ∗k the energy gradient (of Equation 11) recovers the original gradient (from Equation 7). In the general case, suppose pφ is un-normalized, such that pθ∗k = pφ/Kφ for some constant Kφ; then
∇φLenergy(φ; θ∗k) = TKφ Kφ+1 E h∼µθ∗ k x∼πθ∗ k (·|h) ∇φfφ(h, x)− TKφ+1E h∼µs x∼πs(·|h) ∇φfφ(h, x)
Proof. Appendix A.
This “weighting” is intuitive: If pφ were un-normalized such that Kφ > 1, the energy loss automatically places higher weights on negative samples h ∼ µθ∗k , x ∼ πθ∗k(·|h) to bring it down; conversely, if pφ were un-normalized such thatKφ < 1, the energy loss places higher weights on positive samples h ∼ µs, x ∼ πs(·|h) to bring it up. (If pφ were normalized, then Kφ = 1 and the weights are equal). In sum, we have arrived at a framework that learns an explicit sampling policy without exposure bias, a decoupled energy model without nested or saddle-point optimization, and is self-normalizing without importance sampling or estimating the partition function. Figure 1 gives a representative comparison.

heading: 3.3 Optimization Algorithm
text: Algorithm 1 Time-series Generation by Contrastive Imitation . Details in Appendix B 1: Input: source dataset D ≈ ps, mini-batch size M , regularization coefficient κ, learning rates λ 2: Initialize: replay buffer B, energy parameter φ, policy parameter θ, critic parameter ψ 3: for each iteration do 4: for each policy rollout do 5: B ← B ∪ {τ ∼ pθ} . Generate sample 6: for each gradient step do 7: θ ← θ − λactor ∇θ Lactor (θ;φ, ψ) + κ∇θLmle(θ) . Update policy 8: φ ← φ − λenergy∇φLenergy(φ; θ) . Update energy 9: ψ ← ψ − λcritic ∇ψLcritic (ψ;φ) . Update critic 10: Output: learned policy parameter θ∗ and energy parameter φ∗
The only remaining choice is the method of policy optimization. Here we employ soft actor-critic [57], although in principle any technique will do—the only requirement is that it performs reinforcement learning with entropy-regularization [45–47]. To optimize the policy per Equation 12, in addition to the policy “actor” itself, this trains a “critic” to estimate value functions. As usual, the actor takes soft policy improvement steps, minimizing Lactor(θ;φ, ψ) := Eh∼B Ex∼πθ(·|h)[log πθ(x|h)−Qψ(h, x)], where Qψ : H×X → R is the transition-wise soft value function parameterized by ψ, and B is a replay buffer of samples generated by πθ. For stability, the actor is regularized with the conditional MLE loss Lmle(θ) := Ex∼πs(·|h) log πθ(x|h). The critic is trained to minimize the soft Bellman residual: Lcritic(ψ;φ) := Eh,x∼B(Qψ(h, x)− fφ(h, x)− Vψ(h′))2, where the state-values are bootstrapped as Vψ(h′) := Ex′∼πθ(·|h′)[Qψ(h′, x′)− log πθ(x′|h′)]. By expressly training an imitation policy to mimic time-series behavior using rewards from an energy model trained by contrastive learning, we call this framework Time-series Generation by Contrastive Imitation (TimeGCI): See Algorithm 1.

heading: 4 Discussion
text: Our theoretical motivations are apparent (Sections 2.2–3.1), and the practical mechanics of optimization are straightforward (Section 3.2–3.3). To understand the strengths and limitations of TimeGCI, two questions remain: First, how does this relate to bread-and-butter imitation learning of sequential decision-making? Second, how does this compare with recent deep generative models for time series?
Imitation Perspective In sequential decision-making, imitation learning deals with training a policy purely on the basis of demonstrated behavior—that is, with no knowledge of the reward signals that induced the behavior in the first place [58–60]. Consider the standard Markov decision process setting, with states z ∈ Z , actions u ∈ U , dynamics ω ∈ ∆(Z)Z×U , and rewards ρ ∈ RZ×U . Classically, imitation learning seeks to minimize the regret Rs(θ):=Eπs [ ∑ tρ(zt, ut)]−Eπθ [ ∑ tρ(zt, ut)], with πs, πθ ∈ ∆(U)Z here being the demonstrator and imitator policies, and expectations are taken over episodes generated per ut ∼ π(·|zt) and zt+1 ∼ ω(·|zt, ut) [61,62]. First, observe that by interpreting h as “states” and x as “actions”, our problem setup bears a precise resemblance to imitation learning:
Corollary 6 (Generation as Imitation) Let state space Z := H, action space U := X , and reward function ρ := fs. In addition, let the dynamics be such that ω(·|ht, xt) is the Dirac delta centered at ht+1:=(x1, ..., xt). Then the regret exactly corresponds to the expected quality difference: Rs=∆F̄s.
Proof. Immediate from Definition 1.
Now, since we want low regret but have no knowledge of the true quality measure (i.e. “reward signal”), we may naturally learn it together. In this sense, TimeGCI is analogous to imitation by inverse reinforcement learning (IRL), which seeks to infer rewards that plausibly induced the demonstrated behavior, and to optimize imitating policies on that basis [63–66]. Further, in simultaneously optimizing for variety (cf. entropy) and typicality (cf. energy), TimeGCI is analogous to maximum-entropy IRL [67, 68]. Our contrastive approach also bears mild resemblance to stepwise discriminators studied in this vein [54,69], although our framework focuses on trajectory-wise modeling, and is not adversarial (see Appendix D for more discussion on how TimeGCI relates to popular imitation learning methods).
There are also crucial differences: In imitation learning, dynamics are generally Markovian; states are readily defined as discrete elements or real vectors, and action spaces are small/discrete. The practical challenge is sample efficiency—to reduce the cost of environment interactions [70,71]. In time-series generation, however, rollouts are free—generating a synthetic trajectory does not require interacting with the real world. But dynamics are never Markovian: The practical challenge is that representations of variable-length histories must be jointly learned. Moreover, actions are the full-dimensional feature vectors themselves, which renders policy optimization more demanding than usual (see Appendix B); beyond the tractable tabular settings we experiment in, higher-dimensional data may prove challenging.
Related Work Table 1 summarizes the key differentiators of TimeGCI from prevailing techniques. As discussed in Section 1, MLE-based autoregressive models [5, 10, 13] are easy to optimize, and learn explicit conditional distributions that can be used for inspection, resampling, or uncertainty estimation, but they suffer from exposure bias [8, 11, 12]. GAN-based adversarial models fall into two camps: For trajectory-centric methods [18, 19, 72], with only sequence-level signals to guide the generator, they often struggles to converge to the adversarial objective without extensive tuning [12]— with the exception of [72], which utilizes Sinkhorn divergences instead. Transition-centric methods [12,21,22] provide more granular signals to guide the generator, but this simply alters the objective of learning ps to one of learning µs, and still inherits the disadvantages of implicit, adversarial learning.
Our analysis is built on ideas from energy-based models (EBMs) [73–75] and reinforcement learning for sequence prediction [76–78]. In particular, our initial formulation (Section 3.1) can be viewed as a temporal extension of variational EBMs [41,42]. Moreover, by adaptively learning πθ to give negative samples for dθ,φ, the formulation we study (Section 3.2) is equivalent to a temporal analogue of noise-contrastive estimation (NCE) [55, 79]. More tangentially, conditional EBMs have been trained with NCE for text generation [80–82], and the strength of global normalization has been studied [83]; that said, these are confined to the case where external input tokens are available for conditioning at each step—and not free-running as in our time-series setting. Finally, note that viewing sequence generation as a decision-making problem is present in language modeling [6, 35] where task-specific metrics are available as signals. In the absence of predefined signals, GAN-based methods that jointly train discriminators to provide rewards for imitation have been studied [84–89], although they are adversarial, and all focus on the special case of generating discrete tokens for language modeling.

heading: 5 Experiments
text: Benchmarks We test Algorithm 1 (TimeGCI) against the following: The classic Teacher Forcing trains autoregressive networks using ground-truth conditioning (T-Forcing) [5]. Professor Forcing uses adversarial domain adaptation by training an auxiliary discriminator to encourage dynamics of the network’s free-running and teacher-forced states to be similar (P-Forcing) [10]. Trajectory-centric recurrent GANs (C-RNN-GAN) directly plug RNNs into the GAN framework as generators and discriminators for full sequences [18]. Causal Optimal Transport GAN (COT-GAN) is the latest variant of this [20], proposing to approximate Sinkkorn divergences instead of the standard JS divergence. For transition-centric recurrent GANs (RC-GAN), the adversarial loss is computed as the sum of log likelihoods for the stepwise feature vectors conditioned on histories [21], instead of directly as the log likelihood for the entire sequence. Finally, Time-series GAN (TimeGAN) is its latest incarnation [12], proposing to generate and discriminate within a jointly optimized embedding space for efficiency.
Datasets We employ five tabular time-series datasets with a variety of different characteristics, such as periodicity, noise level, and correlations: First, we use a synthetic dataset of multivariate sinusoids with different frequencies and phases (Sines) [12]. Second, we use a UCI dataset from the monitored energy usage of household appliances in a low-energy house (Energy) [90].
Third, we use a UCI dataset from temperature-modulated semiconductor gas sensors for chemical detection (Gas) [91]. Fourth, we use a UCI dataset of hourly interstate vehicle volume at a state traffic recording station (Metro) [92]. Fifth, we use a medical dataset of intensive-care patients from the Medical Information Mart for Intensive Care (MIMIC-III) [93]. All datasets are accessible from their sources, and we use the original source code for preprocessing sines and the UCI datasets by [12], publicly available at [94]. Table 2 shows summary statistics for the datasets used in the experiments.
Implementation Experiments for each dataset are arranged as follows: The real trajectories that constitute the original datasetD are fed as input to train all algorithms. Each algorithm is subsequently used in test mode to generate 10,000 synthetic trajectories. Then, the performance of each algorithm is evaluated on the basis of these generated trajectories. This process is then performed for a total of 10 repetitions, from which we compile the means and standard errors for each reported result. For fair comparison, analogous network components across all benchmarks share the same recurrent architecture: Wherever a generator, policy, discriminator, energy, or critic network applies, we use LSTMs with one hidden layer of 32 units to compute hidden states for representing histories h, and two fullyconnected hidden layers of 32 units each and ELU activations to compute task-specific output variables (i.e. the generator output, policy parameters, discriminator output, energy functions, or critic values). In other respects, we use the publicly available source code to construct the benchmark algorithms— accessible at [94–98]. See Appendix C for additional detail on hyperparameters and implementations.
Evaluation and Results In the tabular data setting, assessing synthetic data generation is inherently tricky [27, 99, 100]: Unlike in media-specific applications, we have no predefined measures such as music polyphony or BLEU scores, nor can we use human evaluation of realism as done for videos. For tabular time-series, the generally accepted standard for comparing synthetic data is to apply the Trainon-Synthetic, Test-on-Real (TSTR) framework, first proposed by [21] and employed by most recent work in synthetic time-series generation [12, 14, 21, 22, 26, 101], as well as more generally for tabular synthetic data of any kind [99, 100, 102]. Specifically, we apply the performance measure used by [12,14,101] to quantify how much the synthetic sequences inherit the predictive characteristics of the original dataset (Predictive Score): Using synthetic samples, a generic post-hoc sequence-prediction model is learned to forecast next-step feature vectors over training sequences. Then, the trained model is evaluated on the original data, and its predictive performance is quantified in terms of the mean absolute error. We use the original source code for computing this metric, publicly available at [94].
Further to prior works using this measure, we additionally believe that synthetic data evaluation should be more general than just next-step TSTR forecasting. After all, the distinguishing characteristic of sequential (vs. static) data generation is that we care about evolution of features over time. Hence we also compute TSTR metrics for horizons of other lengths (+3 Steps Ahead and +5 Steps Ahead). Importantly, note that a key strength of TSTR evaluation is in its sensitivity to mode collapse: If any generation scheme suffers from mode collapse (as GAN methods are prone to), TSTR scores would degrade due to the synthetic data failing to capture the diversity of the real data, which means any
prediction model trained on that basis would also fail to capture this variation). Finally, similar to some recent works [19, 20], we also compute the cross-correlations of real and synthetic feature vectors, and report the sum of the absolute differences between them, averaged over time (x-Corr. Score); this serves to verify if feature relationships are preserved well, in addition to temporal relationships. Table 3 shows the results: With respect to these metrics, we find that TimeGCI somewhat consistently produces synthetic samples that perform similarly or better than benchmark algorithms in all datasets. (Note that we do empirically observe several instances of mode collapse in GAN-based benchmarks).

heading: 6 Conclusion
text: In this work, we invite an explicit analogy between time-series generation and imitation learning, and explore a framework that fleshes out this connection. Two caveats are in order: First, while we began from the notion of moment-matching to address the error compounding problem, in practice there is no guarantee that this is accomplished well during optimization. In particular, scalability is a major limitation beyond the range of feature dimensions and sequence lengths considered in our experiments. Sample-based estimates could rapidly degrade with the horizon, especially if transitions are highly stochastic. A relevant question is whether or not training on fixed subsequence lengths could potentially alleviate this concern for longer sequences. In addition, while our approach seeks to dispense with the instabilities typical of adversarial training, we are instead left with the difficulties of policy optimization, which may prove a prohibitive challenge in higher-dimensional feature spaces. For the datasets we consider, we find that pre-training and regularizing the policy with maximum likelihood, combined with a small enough learning rate, had the most impact in promoting stability and learning. Second, we reiterate that a perennial challenge in modeling tabular data is in choosing the metric for evaluation. While we opted for the most commonly accepted method of TSTR, this may not be general enough to capture the range of downstream tasks that may be performed on the synthetic data. Future work will benefit from a deeper investigation into more sophisticated measures for time series, such as contrastive methods and how to evaluate different aspects of the “quality” of the generated trajectories.

heading: Acknowledgments
text: We would like to thank the reviewers for all their invaluable feedback. This work was supported by Alzheimer’s Research UK, The Alan Turing Institute under the EPSRC grant EP/N510129/1, the US Office of Naval Research, as well as the National Science Foundation under grant number 1722516.

Review summaries:
The paper proposes an imitation learning method that simultaneously learns a policy and an energy (“discriminator”) function. This is similar to a variational approach to learn a Gibbs distribution over trajectories, where the variational parameters consist of the policy and variational inference consists of running RL.   However, rather than alternating the normal Gibbs energy loss and the variational inference loss of the variational approach, the submission replaces the Gibbs energy loss with an alternate energy loss based on noise-contrastive estimation, which explicitly trains the energy to discriminate between “noise” samples (drawn from the current policy) and expert samples.   It is proved that under the noise-contrastive loss, the energy converges to the expert energy regardless of the noise distribution (under weak conditions), which justifies only partially optimizing the RL step in the alternating optimization. This results in a method potentially much faster than the “naive” variational approach, as it is not necessary to fully solve an inner-loop RL problem for each outer-loop iteration. 

The paper proposes a method for sequential data generation that tries to avoid the pitfalls of prior MLE- and GAN-based objectives. The proposed approach starts with the global moment-matching objective (that mitigates the compounding errors). The true data distribution is modeled as an EBM and the parameters of the energy function are learned with contrastive learning (NCE). The trajectory distribution induced by a (learned) generation policy is used as the “noise” for NCE. In the inner loop, the generation policy is optimized to reduce the reverse-KL to the EBM’s trajectory distribution – which is equivalent to entropy-regularized RL with the energy function as the reward. Experiments using 5 tabular time-series datasets pits TimeCGI favorably in comparison to prior methods on the metrics of predictive and correlation scores.

The paper proposes to perform time-series generation via a method inspired by imitation learning in a non-adversarial manner. The idea is to fit an energy-based model over the trajectories with noise contrastive estimation, with the noise being provided by a parametrized policy. The policy is also optimized to fit the energy-based model to provide a sound noise distribution. The method demonstrates superior empirical performance on various time-series datasets, on prediction error, and cross-correlation.