Time-series Generation by Contrastive Imitation

1 Introduction . Time-series data are ubiquitous in diverse machine learning applications , such as financial , industrial , and healthcare settings . At the same time , lack of public access to data is a recurring obstacle to the development and reproducibility of research in domains where datasets are proprietary [ 1 ] . Generating synthetic—but realistic—time-series data is a promising solution [ 2 ] , and has received increasing attention in recent years , driven by advances in deep learning and generative adversarial networks [ 3,4 ] . Owing to the fact that time-series features are generated sequentially , generative modeling in the temporal setting faces a two-pronged challenge : First , a good generator should accurately capture the conditional dynamics of stepwise transitions p ( xt|x1 , ... , xt−1 ) ; this is important , as the faithfulness of any conceivable downstream time-series analysis depends on the learned correlations across both temporal and feature dimensions . Second , however , the recursive rollouts of the generator should also respect the joint distribution of multi-step trajectories p ( x1 , ... , xT ) ; this is equally important , as synthetic trajectories that inadvertently wander beyond the support of original data are useless at best . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recent work falls into two main categories . On one hand , autoregressive models trained via MLE [ 5 ] explicitly factor the distribution of trajectories into a product of conditionals ∏ t p ( xt|x1 , ... , xt−1 ) . While this allows directly learning and computing such transitions , with finite data this is prone to compounding errors during multi-step generation , due to the discrepancy between closed-loop training ( i.e . conditioned on ground-truths as inputs ) and open-loop sampling ( i.e . conditioned on its own previous outputs ) [ 6 ] . A variety of methods have sought to counteract this problem of exposure bias , employing auxiliary techniques from curriculum learning [ 7 , 8 ] and adversarial domain adaptation [ 9 , 10 ] ; however , such remedies are not without biases [ 11 ] , and empirical improvements have been mixed [ 12–14 ] . On the other hand , adversarial models based on GAN training and its relatives [ 15–17 ] directly model the distribution of trajectories p ( x1 , ... , xT ) [ 18–20 ] . To provide a more granular learning signal for the generator , a popular variant matches the induced distribution of sub-trajectories instead , providing stepwise feedback from the discriminator [ 21 , 22 ] . TimeGAN [ 12 ] is the most recent incarnation of this , and operates within a jointly optimized latent space . GAN-based approaches alleviate the risk of compounding errors , and have been applied to banking [ 23 ] , sensors [ 24 ] , biosignals [ 25 ] , and smartgrids [ 26 ] . However , the conditional dynamics are only implicitly learned , yielding no way of inspecting or assessing the quality of sampled transitions nor trajectories . Moreover , the adversarial objective leads to characteristically challenging optimization—exacerbated by the temporal dimension . Three Operations Consider a probabilistic generative model p for some datasetD . We are generally interested in performing one or more of the following operations : ( 1 ) sampling a time series τ ∼ p , ( 2 ) evaluating the likelihood p ( τ ) , and ( 3 ) learning the model p from a set of i.i.d . samples τ . In light of the preceding , we investigate a generative framework that attempts to fulfill the following criteria : • Samples should respect both the stepwise conditional distributions of features , as well as the joint distribution of full trajectories ; unlike pure MLE , we wish to avoid multi-step compounding error . • Evaluating likelihoods should be possible as generic measures of sample quality for both transitions and trajectories—often desired for sample comparison , model auditing , or bias correction [ 27 , 28 ] . • Unlike black-box GAN discriminators , we wish that the evaluator be decoupled from any specific sampler , such that the two components can be trained non-adversarially , thus may be more stable . Contributions In the sequel , we explore an approach that seeks to satisfy these criteria . We first give precise treatment of the “ compounding error ” problem , thus motivating a specific trajectory-centric optimization objective from first principles ( Section 2 ) . To carry it out , we develop a general training framework and practical algorithm , along with its theoretical justification : We train a forward-looking transition policy to imitate the sequential behavior of time series using a stepwise-decomposable energy model as reinforcement , giving a method that embodies “ generation by imitation ” ( Section 3 ) . Importantly , to understand its strengths and limitations , we compare the method to existing generative models for time-series data , and relate it to imitation learning of sequential behavior ( Section 4 ) . Lastly , through experiments with application to real-world time-series datasets , we verify that it generates predictively useful samples that perform at the standard of comparable benchmarks ( Section 5 ) . 2 Synthetic Time Series . 2.1 Problem Setup . We operate in the standard discrete-time setting for time series . Let feature vectors xt ∈ X be indexed by time steps t , and let a full trajectory of length T be denoted τ : = ( x1 , ... , xT ) ∈ T : = X T . Also , denote with ht : = ( x1 , ... , xt−1 ) ∈ H : = ∪Tt=1X t the history prior to time t. For ease of exposition we shall work with trajectories of fixed lengths T , but our results trivially generalize to the case where T itself is a random variable ( for instance , by employing padding tokens up to some maximum length ) . Consider a dataset D : = { τn } Nn=1 of N trajectories sampled from some true source s. We assume the trajectories are generated sequentially by some unknown transition process πs ∈ ∆ ( X ) H , such that features at each step t are sampled as xt ∼ πs ( ·|ht ) . In addition to this stepwise conditional , denote with µs ( h ) : = 1T ∑ t p ( ht = h|πs ) the normalized occupancy measure—i.e . the distribution of histories induced by πs . Intuitively , this is the visitation distribution of “ history states ” encountered by a generator when navigating about the feature space X by rolling out policy πs . With slight abuse of notation , we may also write µs ( h , x ) : = µs ( h ) πs ( x|h ) to indicate the marginal distribution of transitions . Finally , let the joint distribution of full trajectories be denoted by ps ( τ ) : = ∏ t πs ( xt|ht ) . The goal is to learn a sequential generator πθ parameterized as θ using samples τ ∼ ps from D , such that pθ ≈ ps . Note here that we do not assume stationarity of the time-series data , nor stationarity of the transition conditionals ; any influence of t is implicit through the dependence of πs ( and πθ ) on variable-length histories . In line with recent work [ 14 , 20 ] , for simplicity we do not consider static metadata as supplemental inputs or outputs , as these are commonly and easily incorporated via an additional conditioning layer or auxiliary generator [ 12 , 19 ] . Lastly , note that much recent work on sequential modeling is devoted to domain-specific , architecture-level designs for generating audio [ 29 , 30 ] , text [ 31 , 32 ] , and video [ 33 , 34 ] . In contrast , our work is closer in spirit to [ 12 , 14 ] in being an agnostic , framework-level study applicable to generic tabular data in any time-series setting . Measuring Sample Quality How do we determine the “ quality ” of a sample ? In specialized domains , of course , we often have prior access to task-specific metrics such as BLEU or ROUGE scores in text generation [ 6 , 35 ] —then , the generator can simply be optimized for such scores via standard methods in reinforcement learning [ 36 ] . In generic time-series settings , however , the challenge is that any such metric must necessarily be task-agnostic , and access to it must necessarily come from learning . So , for any data source s , let us speak of some hypothetical function fs : H×X→ [ −c , c ] with c < ∞ , such that fs ( h , x ) gives the quality of any sampled transition—that is , any tuple ( h , x ) . Intuitively , we may interpret this as quantifying how “ typical ” it is for the random process to be in state h and step towards x . Likewise , let as also speak of some function Fs : T → [ −cT , cT ] such that Fs ( τ ) gives the quality of any sampled trajectory . Naturally , in time-series settings where the underlying process is causally-conditioned , it is reasonable to define this as the decomposition Fs ( τ ) : = ∑ t fs ( ht , xt ) . Now of course , we have no access to the true Fs . But clearly , in learning a generative model pθ of ps , we wish that the quality of samples τ drawn from pθ and ps be similar in expectation . More precisely : Definition 1 ( Expected Quality Difference ) Let ∆F̄s : Θ→ [ −2cT , 2cT ] denote the expected quality difference between ps and pθ , where Θ indicates the space of parameterizations for generator πθ : ∆F̄s ( θ ) : = Eτ∼psFs ( τ ) − Eτ∼pθFs ( τ ) ( 1 ) Our objective , then , is to learn a generator πθ that minimizes the expected quality difference ∆F̄s ( θ ) . Two points bear emphasis . First , we know nothing about Fs—beyond it being the sequential aggregate of fs . This challenge uniquely differentiates this agnostic setting from more popular media-specific applications—for which various predefined measures are readily available for supervision . Second , in addition to matching this expectation over samples , we also wish to match the variety of samples in the original data . After all , we want pθ to mimic samples from ps of different degrees of “ typicality ” . So we should expect to incorporate some measure of entropy , e.g . the commonly used Shannon entropy . 2.2 Matching Local Moments . Recall the apparent tradeoff between autoregressive models and adversarial models . In the spirit of the former , suppose we seek to directly learn transition conditionals via supervised learning . That is , argminθ Eh∼µsL ( πs ( ·|h ) , πθ ( ·|h ) ) ( 2 ) Consider the log likelihood loss L ( πs ( ·|h ) , πθ ( ·|h ) ) : = Ex∼πs ( ·|h ) log πθ ( x|h ) . In the case of exponential family models for πθ ( ·|h ) , a basic result is that this is dual to maximizing its conditional entropy subject to the constraint on feature expectations Eh∼µs ; x∼πθ ( ·|h ) T ( x ) = Eh∼µs ; x∼πs ( ·|h ) T ( x ) , where T : X → R is some sufficient statistic [ 37–39 ] . More generally for deep energy-based models , we have ( however , recall that strong duality does not generalize to the nonlinear case ; see Appendix A ) : argminθ ( E h∼µs x∼πθ ( ·|h ) log πθ ( x|h ) + maxf∈RH×X ( E h∼µs x∼πs ( ·|h ) f ( h , x ) − E h∼µs x∼πθ ( ·|h ) f ( h , x ) ) ) ( 3 ) Note that the moment-matching constraint is local—that is , at the level of individual transitions , and all conditioning is based on h from µs alone . This is precisely the “ exposure bias ” : The objective is only ever exposed to inputs h drawn from the ( perfect ) source distribution µs , and is thus unaware of the endogeneity of the ( imperfect ) synthetic distribution µθ induced by πθ . This is not desirable since πθ is rolled out by open-loop sampling at test time . Now , although at the global optimum the momentmatching discrepancy must be zero ( i.e . the equality constraint is enforced ) , in practice there may be a variety of reasons why this is not perfectly achieved ( e.g . error in estimating expectations , error in function approximation , error in optimization , etc ) . Suppose we could bound how well we are able to enforce the moment-matching constraint ; as it turns out , we can not eliminate error compounding:1 Lemma 1 Let maxf∈RH×X ( E h∼µs x∼πs ( ·|h ) f ( h , x ) −E h∼µs x∼πθ ( ·|h ) f ( h , x ) ) ≤ . Then ∆F̄s ( θ ) ∈ O ( T 2 ) . Proof . Appendix A . This reveals the problem with modeling conditionals per se : Not all mistakes are equal . An objective like Equation 2 penalizes unrealistic transitions ( h , x ) by treating all conditioning histories h equally— regardless of how realistic h is to begin with . Clearly , however , we care much less about how x looks like , if the current subsequence h is already highly unlikely ( and vice versa ) . Intuitively , earlier mistakes in a trajectory should weigh more : Once πθ wanders into areas ofH with low support in µs , no amount of “ good ” transitions will bring the trajectory back to high-likelihood areas of T under ps .

Summary: The paper proposes an imitation learning method that simultaneously learns a policy and an energy (“discriminator”) function. This is similar to a variational approach to learn a Gibbs distribution over trajectories, where the variational parameters consist of the policy and variational inference consists of running RL.   However, rather than alternating the normal Gibbs energy loss and the variational inference loss of the variational approach, the submission replaces the Gibbs energy loss with an alternate energy loss based on noise-contrastive estimation, which explicitly trains the energy to discriminate between “noise” samples (drawn from the current policy) and expert samples.   It is proved that under the noise-contrastive loss, the energy converges to the expert energy regardless of the noise distribution (under weak conditions), which justifies only partially optimizing the RL step in the alternating optimization. This results in a method potentially much faster than the “naive” variational approach, as it is not necessary to fully solve an inner-loop RL problem for each outer-loop iteration.