Time-series Generation by Contrastive Imitation

1 Introduction . Time-series data are ubiquitous in diverse machine learning applications , such as financial , industrial , and healthcare settings . At the same time , lack of public access to data is a recurring obstacle to the development and reproducibility of research in domains where datasets are proprietary [ 1 ] . Generating synthetic—but realistic—time-series data is a promising solution [ 2 ] , and has received increasing attention in recent years , driven by advances in deep learning and generative adversarial networks [ 3,4 ] . Owing to the fact that time-series features are generated sequentially , generative modeling in the temporal setting faces a two-pronged challenge : First , a good generator should accurately capture the conditional dynamics of stepwise transitions p ( xt|x1 , ... , xt−1 ) ; this is important , as the faithfulness of any conceivable downstream time-series analysis depends on the learned correlations across both temporal and feature dimensions . Second , however , the recursive rollouts of the generator should also respect the joint distribution of multi-step trajectories p ( x1 , ... , xT ) ; this is equally important , as synthetic trajectories that inadvertently wander beyond the support of original data are useless at best . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recent work falls into two main categories . On one hand , autoregressive models trained via MLE [ 5 ] explicitly factor the distribution of trajectories into a product of conditionals ∏ t p ( xt|x1 , ... , xt−1 ) . While this allows directly learning and computing such transitions , with finite data this is prone to compounding errors during multi-step generation , due to the discrepancy between closed-loop training ( i.e . conditioned on ground-truths as inputs ) and open-loop sampling ( i.e . conditioned on its own previous outputs ) [ 6 ] . A variety of methods have sought to counteract this problem of exposure bias , employing auxiliary techniques from curriculum learning [ 7 , 8 ] and adversarial domain adaptation [ 9 , 10 ] ; however , such remedies are not without biases [ 11 ] , and empirical improvements have been mixed [ 12–14 ] . On the other hand , adversarial models based on GAN training and its relatives [ 15–17 ] directly model the distribution of trajectories p ( x1 , ... , xT ) [ 18–20 ] . To provide a more granular learning signal for the generator , a popular variant matches the induced distribution of sub-trajectories instead , providing stepwise feedback from the discriminator [ 21 , 22 ] . TimeGAN [ 12 ] is the most recent incarnation of this , and operates within a jointly optimized latent space . GAN-based approaches alleviate the risk of compounding errors , and have been applied to banking [ 23 ] , sensors [ 24 ] , biosignals [ 25 ] , and smartgrids [ 26 ] . However , the conditional dynamics are only implicitly learned , yielding no way of inspecting or assessing the quality of sampled transitions nor trajectories . Moreover , the adversarial objective leads to characteristically challenging optimization—exacerbated by the temporal dimension . Three Operations Consider a probabilistic generative model p for some datasetD . We are generally interested in performing one or more of the following operations : ( 1 ) sampling a time series τ ∼ p , ( 2 ) evaluating the likelihood p ( τ ) , and ( 3 ) learning the model p from a set of i.i.d . samples τ . In light of the preceding , we investigate a generative framework that attempts to fulfill the following criteria : • Samples should respect both the stepwise conditional distributions of features , as well as the joint distribution of full trajectories ; unlike pure MLE , we wish to avoid multi-step compounding error . • Evaluating likelihoods should be possible as generic measures of sample quality for both transitions and trajectories—often desired for sample comparison , model auditing , or bias correction [ 27 , 28 ] . • Unlike black-box GAN discriminators , we wish that the evaluator be decoupled from any specific sampler , such that the two components can be trained non-adversarially , thus may be more stable . Contributions In the sequel , we explore an approach that seeks to satisfy these criteria . We first give precise treatment of the “ compounding error ” problem , thus motivating a specific trajectory-centric optimization objective from first principles ( Section 2 ) . To carry it out , we develop a general training framework and practical algorithm , along with its theoretical justification : We train a forward-looking transition policy to imitate the sequential behavior of time series using a stepwise-decomposable energy model as reinforcement , giving a method that embodies “ generation by imitation ” ( Section 3 ) . Importantly , to understand its strengths and limitations , we compare the method to existing generative models for time-series data , and relate it to imitation learning of sequential behavior ( Section 4 ) . Lastly , through experiments with application to real-world time-series datasets , we verify that it generates predictively useful samples that perform at the standard of comparable benchmarks ( Section 5 ) . 2 Synthetic Time Series . 2.1 Problem Setup . We operate in the standard discrete-time setting for time series . Let feature vectors xt ∈ X be indexed by time steps t , and let a full trajectory of length T be denoted τ : = ( x1 , ... , xT ) ∈ T : = X T . Also , denote with ht : = ( x1 , ... , xt−1 ) ∈ H : = ∪Tt=1X t the history prior to time t. For ease of exposition we shall work with trajectories of fixed lengths T , but our results trivially generalize to the case where T itself is a random variable ( for instance , by employing padding tokens up to some maximum length ) . Consider a dataset D : = { τn } Nn=1 of N trajectories sampled from some true source s. We assume the trajectories are generated sequentially by some unknown transition process πs ∈ ∆ ( X ) H , such that features at each step t are sampled as xt ∼ πs ( ·|ht ) . In addition to this stepwise conditional , denote with µs ( h ) : = 1T ∑ t p ( ht = h|πs ) the normalized occupancy measure—i.e . the distribution of histories induced by πs . Intuitively , this is the visitation distribution of “ history states ” encountered by a generator when navigating about the feature space X by rolling out policy πs . With slight abuse of notation , we may also write µs ( h , x ) : = µs ( h ) πs ( x|h ) to indicate the marginal distribution of transitions . Finally , let the joint distribution of full trajectories be denoted by ps ( τ ) : = ∏ t πs ( xt|ht ) . The goal is to learn a sequential generator πθ parameterized as θ using samples τ ∼ ps from D , such that pθ ≈ ps . Note here that we do not assume stationarity of the time-series data , nor stationarity of the transition conditionals ; any influence of t is implicit through the dependence of πs ( and πθ ) on variable-length histories . In line with recent work [ 14 , 20 ] , for simplicity we do not consider static metadata as supplemental inputs or outputs , as these are commonly and easily incorporated via an additional conditioning layer or auxiliary generator [ 12 , 19 ] . Lastly , note that much recent work on sequential modeling is devoted to domain-specific , architecture-level designs for generating audio [ 29 , 30 ] , text [ 31 , 32 ] , and video [ 33 , 34 ] . In contrast , our work is closer in spirit to [ 12 , 14 ] in being an agnostic , framework-level study applicable to generic tabular data in any time-series setting . Measuring Sample Quality How do we determine the “ quality ” of a sample ? In specialized domains , of course , we often have prior access to task-specific metrics such as BLEU or ROUGE scores in text generation [ 6 , 35 ] —then , the generator can simply be optimized for such scores via standard methods in reinforcement learning [ 36 ] . In generic time-series settings , however , the challenge is that any such metric must necessarily be task-agnostic , and access to it must necessarily come from learning . So , for any data source s , let us speak of some hypothetical function fs : H×X→ [ −c , c ] with c < ∞ , such that fs ( h , x ) gives the quality of any sampled transition—that is , any tuple ( h , x ) . Intuitively , we may interpret this as quantifying how “ typical ” it is for the random process to be in state h and step towards x . Likewise , let as also speak of some function Fs : T → [ −cT , cT ] such that Fs ( τ ) gives the quality of any sampled trajectory . Naturally , in time-series settings where the underlying process is causally-conditioned , it is reasonable to define this as the decomposition Fs ( τ ) : = ∑ t fs ( ht , xt ) . Now of course , we have no access to the true Fs . But clearly , in learning a generative model pθ of ps , we wish that the quality of samples τ drawn from pθ and ps be similar in expectation . More precisely : Definition 1 ( Expected Quality Difference ) Let ∆F̄s : Θ→ [ −2cT , 2cT ] denote the expected quality difference between ps and pθ , where Θ indicates the space of parameterizations for generator πθ : ∆F̄s ( θ ) : = Eτ∼psFs ( τ ) − Eτ∼pθFs ( τ ) ( 1 ) Our objective , then , is to learn a generator πθ that minimizes the expected quality difference ∆F̄s ( θ ) . Two points bear emphasis . First , we know nothing about Fs—beyond it being the sequential aggregate of fs . This challenge uniquely differentiates this agnostic setting from more popular media-specific applications—for which various predefined measures are readily available for supervision . Second , in addition to matching this expectation over samples , we also wish to match the variety of samples in the original data . After all , we want pθ to mimic samples from ps of different degrees of “ typicality ” . So we should expect to incorporate some measure of entropy , e.g . the commonly used Shannon entropy . 2.2 Matching Local Moments . Recall the apparent tradeoff between autoregressive models and adversarial models . In the spirit of the former , suppose we seek to directly learn transition conditionals via supervised learning . That is , argminθ Eh∼µsL ( πs ( ·|h ) , πθ ( ·|h ) ) ( 2 ) Consider the log likelihood loss L ( πs ( ·|h ) , πθ ( ·|h ) ) : = Ex∼πs ( ·|h ) log πθ ( x|h ) . In the case of exponential family models for πθ ( ·|h ) , a basic result is that this is dual to maximizing its conditional entropy subject to the constraint on feature expectations Eh∼µs ; x∼πθ ( ·|h ) T ( x ) = Eh∼µs ; x∼πs ( ·|h ) T ( x ) , where T : X → R is some sufficient statistic [ 37–39 ] . More generally for deep energy-based models , we have ( however , recall that strong duality does not generalize to the nonlinear case ; see Appendix A ) : argminθ ( E h∼µs x∼πθ ( ·|h ) log πθ ( x|h ) + maxf∈RH×X ( E h∼µs x∼πs ( ·|h ) f ( h , x ) − E h∼µs x∼πθ ( ·|h ) f ( h , x ) ) ) ( 3 ) Note that the moment-matching constraint is local—that is , at the level of individual transitions , and all conditioning is based on h from µs alone . This is precisely the “ exposure bias ” : The objective is only ever exposed to inputs h drawn from the ( perfect ) source distribution µs , and is thus unaware of the endogeneity of the ( imperfect ) synthetic distribution µθ induced by πθ . This is not desirable since πθ is rolled out by open-loop sampling at test time . Now , although at the global optimum the momentmatching discrepancy must be zero ( i.e . the equality constraint is enforced ) , in practice there may be a variety of reasons why this is not perfectly achieved ( e.g . error in estimating expectations , error in function approximation , error in optimization , etc ) . Suppose we could bound how well we are able to enforce the moment-matching constraint ; as it turns out , we can not eliminate error compounding:1 Lemma 1 Let maxf∈RH×X ( E h∼µs x∼πs ( ·|h ) f ( h , x ) −E h∼µs x∼πθ ( ·|h ) f ( h , x ) ) ≤ . Then ∆F̄s ( θ ) ∈ O ( T 2 ) . Proof . Appendix A . This reveals the problem with modeling conditionals per se : Not all mistakes are equal . An objective like Equation 2 penalizes unrealistic transitions ( h , x ) by treating all conditioning histories h equally— regardless of how realistic h is to begin with . Clearly , however , we care much less about how x looks like , if the current subsequence h is already highly unlikely ( and vice versa ) . Intuitively , earlier mistakes in a trajectory should weigh more : Once πθ wanders into areas ofH with low support in µs , no amount of “ good ” transitions will bring the trajectory back to high-likelihood areas of T under ps .

Summary: The paper proposes to perform time-series generation via a method inspired by imitation learning in a non-adversarial manner. The idea is to fit an energy-based model over the trajectories with noise contrastive estimation, with the noise being provided by a parametrized policy. The policy is also optimized to fit the energy-based model to provide a sound noise distribution. The method demonstrates superior empirical performance on various time-series datasets, on prediction error, and cross-correlation.