Time-series Generation by Contrastive Imitation
Authors: Daniel Jarrett
Ioana Bica
Mihaela van der Schaar
Venue: NeurIPS 2021 Poster (2021)
Keywords: Time Series Data
Synthetic Data Generation

Abstract: Consider learning a generative model for time-series data. The sequential setting poses a unique challenge: Not only should the generator capture the *conditional* dynamics of (stepwise) transitions, but its open-loop rollouts should also preserve the *joint* distribution of (multi-step) trajectories. On one hand, autoregressive models trained by MLE allow learning and computing explicit transition distributions, but suffer from compounding error during rollouts. On the other hand, adversarial models based on GAN training alleviate such exposure bias, but transitions are implicit and hard to assess. In this work, we study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) *transition policy*, where the reinforcement signal is provided by a global (but stepwise-decomposable) *energy model* trained by contrastive estimation. At training, the two components are learned cooperatively, avoiding the instabilities typical of adversarial objectives. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality. By expressly training a policy to imitate sequential behavior of time-series features in a dataset, this approach embodies "*generation by imitation*". Theoretically, we illustrate the correctness of this formulation and the consistency of the algorithm. Empirically, we evaluate its ability to generate predictively useful samples from real-world datasets, verifying that it performs at the standard of existing benchmarks.

Time-series Generation by Contrastive Imitation
Daniel Jarrett University Cambridge, UK daniel. parrett@mathes cam ac. uk
Ioana Bica
University of Oxford, UK
Alan Turing Institute, UK ioana. i.ana. bicabenn..e...... ox ac.
Mihaela van der Schaar
University of California, Los Angeles
University of Cambridge, UK
Alan Turing Institute, UK mv4720cam. ac.
Abstract
Consider learning a generative model for time-series data. The sequential setting poses unique challenge: Not only should the generator capture the conditional dy- namics of (stepwise) transitions, but its open-loop rollouts should also preserve the joint distribution of (multi-step) trajectories. On one hand, autoregressive models trained by MLE allow learning and computing explicit transition distributions, but suffer from compounding error during rollouts. On the other hand, adversarial mod- els based on GAN training alleviate such exposure bias, but transitions are implicit and hard assess. In this work, study generative framework that seeks to com- bine the strengths of both: Motivated by a moment moment-matching objective to mitigate compounding error, we optimize local (but forward-looking) transition policy, where the reinforcement signal is provided by global (but energy model trained by contrastive estimation. learned cooperatively. avoiding the instabilities typical of adversarial objectives. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves trajectory-level measure for evaluating sample quality. By expressly training policy to imitate sequential behavior of time-series features in a dataset, this approach embodies "generation by imitation". Theoretically, we illustrate the correctness of this formulation and the consistency of the algorithm. Empirically, we evaluate its ability to generate predictively useful samples from real- world datasets, verifying that it performs at the standard of existing benchmarks.
training, the two components are
Introduction
Time-series data are ubiquitous in diverse machine learning applications, such as financial, industrial, and healthcare settings. At the same time, lack of public access to data is recurring obstacle to the development reproducibility of research domains where datasets are proprietary [T]. Generating synthetic- but realistic-time-see time-series data is promising solution [2], and has received increasing attention in recent years, driven by advances in deep learning and generative adversarial networks (3[4).
Owing to the fact that time-series features are generated sequentially, generative modeling in the temporal setting faces two-pronged challenge: First, a good generator should accurately capture the conditional dynamics of stepwise transitions of any conceivable downstream time-series analysis depends on the learned correlations across both temporal and feature dimensions. Second, however, the recursive rollouts of the generator should also respect the joint distribution of multi-step trajectories p(#1,-.. JTT); this is equally important, as synthetic trajectories that inadvertently wander beyond the support of original data are useless at best.
2; -1) this is important, as the faithfulness
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Recent work falls into two main categories. On one hand, autoregressive models trained via MLE [ explicitly factor the distribution of trajectories into a product of conditionals [I. p(zz|21),...... -t While this allows directly learning and computing such transitions, with finite data this is prone to com-- pounding errors during multi-step generation, due to the discrepancy between closed-loop training (i.e. conditioned on ground-truths as inputs) and open-loop sampling (i.e. conditioned on its own previous outputs) [6]. A variety of methods have sought to counteract this problem of exposure bias, employing auxiliary techniques from curriculum learning 8 and adversarial domain adaptation (9 [0]; how- ever, such remedies are not without biases [ and empirical improvements have been mixed 12 14).
On the other hand, adversarial models based on GAN training and its relatives [15 17 directly model the distribution of trajectories p(z1,..., 2r) [18 20). To provide more granular learning signal for the generator, a popular variant matches the induced distribution of sub-trajectorier instead, providing
stepwise feedhack from the discriminator [2] TimeGAN (12) is the most recent incarnation of
this, and operates within a jointly optimized latent space. GAN-based approaches alleviate the risk
of compounding errors, and have been applied to banking (33), sensors (24), biosignals (25). and
smartgrids [26). However, the conditional dynamics only implicitly learned, yielding no way of
inspecting or assessing the quality of sampled transitions nor trajectories. Moreover, the adversarial objective leads to characteristically challenging
by the temporal dimension.
Three Operations Consider a probabilistic generative model p for some dataset D. We are generally interested in performing one or more of the following operations: (I) sampling a time series T ~ (2) evaluating the likelihood p(T), and (3) learning the model p from a set of i.i.d. samples т. In light of the preceding, we investigate generative framework that attempts to fulfill the following criteria:
Samples should respect both the stepwise conditional distributions of features, as well as the joint distribution of full trajectories; unlike pure MLE, we wish to avoid multi-step compounding error. Evaluating likelihoods should be possible as generic measures of sample quality for both transitions and trajectories desired for sample comparison, model auditing, or bias correction (27 28)
Unlike black-box GAN discriminators, we wish that the evaluator be decoupled from any specific sampler, such that the two components can be trained hon-adversarially. thus may be more stable.
Contributions In the sequel. we explore an approach that seeks to satisfy these criteria. We first give precise treatment of the "compounding error" problem, thus motivating a specific trajectory-centre optimization objective from first principles (Section 2) To carry it out, we develop a general training framework and practical algorithm, along with its theoretical justification: We train a forward-looking transition policy imitate the sequential behavior of time series using stepwise-decoomposbble energy model as reinforcement, giving a method that embodies "generation by imitation" (Section 3) Importantly, to understand its strengths and limitations, we compare the method to existing generative models for time-series data, and relate it to imitation learning of sequential behavior (Section 4 Lastly, through experiments with application real-world time-series datasets, we verify that it gen- erates predictively useful samples that perform at the standard of comparable benchmarks (Section
2 Synthetic Time Series
2.1 Problem Setup
We operate in the standard discrete-time setting for time series. Let feature vectors He E X' be indexed by time steps I, and let a full trajectory of length T denoted T == (TT) E T == X Also, denote with ha == (I1,..,, we shall work with trajectories of fixed lengths T, but our results trivially generalize to the case where T' itself is a random variable (for instance, by employing padding tokens up some maximum length).
E H == _f_ X' the history prior to time 1. For ease of exposition
Consider a dataset D (Tn of N trajectories sampled from some true source s. We assume the trajectories are generated sequentially by some unknown transition process n5 E A(x')* H such that features at each step are sampled as 2, ~ (-hh In addition to this stepwise conditional, denote with p. (h) = T Σ, p(h, = h (T.) the normalized occupancy measure -.e. the distribution of histories induced by π.. Intuitively, this the visitation distribution of "history states" encountered by a generator when navigating about the feature space X' by rolling out policy т. With slight abuse of notation, we may also write (r, x) = (1.(h)=, m, (ar(h) to indicate the marginal distribution of transitions. Finally, let the joint distribution of full trajectories be denoted by P. (T) == II. (z.))hh))

---

The goal is to learn sequential generator no parameterized as @ using samples Pa from D, such that po R Ps- Note here that we do not assume stationarity of the time-series data, nor stationarity of the transition conditionals; any influence of is implicit through the dependence of x (and TT)
on variable-length histories. In line with recent work [14 [1420]. for simplicity we do not consider static metadata as supplemental inputs or outputs, as these are commonly and easily incorporated via an additional conditioning layer or auxiliary generator [12 19]. Lastly, note that much recent work on sequential modeling is devoted to domain-specific, architecture-leve designs for generating audio [29 30). text 32] and video [33 34]. In contrast, our work is closer in spirit to [12 in study applicable generic tabular data in any time-series setting.
being an agnostic. framewwkkk
Measuring Sample Quality How do we determine the "quality" of a sample? In specialized domains, of course, we often have prior access to task-specific metrics such as BLEU or ROUGE scores in text generation 6 355 -ten, the generator can simply be optimized for such scores via standard methods in reinforcement learning [36]. In generic time-series settings, however, the challenge is that any such metric must necessarily be task-agnostic, and access to it must necessarily come from learning.
So, for any data source s, let us speak of some hypothetical function fa : H X c| with c < oc, such that fa (h,,2) x) gives the quality of any sampled transition- is, any tuple (h,z). Intuitively, we may interpret this as quantifying how "typical"" it is for the random process to be in state h and step towards x. Likewise, let as also speak of some function F; T 7 cTT] such that (T) gives the quality of any sampled trajectory. Naturally, in time-series settings where the underlying process sausally-conditione
it reasonable to define this as the decomposition FT(T) = E. fo (h,,,,), Now of course, we have no access to the true F.. But clearly, in learning generative model po of ps+ we wish that the quality of samples T drawn from po and ps be similar in expectation. More precisely:
Definition (Expected Quality Difference) Let AF, 0 + [ [-2,T 2c7T denote the expected qual- ity difference between p. and pg. where θ indicates the space of parameterizations for generator TTa
A (F) == F. - Br-po Fs (T)
Our objective, then, is learn a generator Ty that minimizes the expected quality difference AF. Two points bear emphasis. First, we know nothing about l'--beyond it being the sequential aggregate of fa. This challenge uniquely differentiates this agnostic setting from more popular media-specific applications- for which various predefined measures are readily available for supervision. Second, in addition to matching this expectation over samples, we also wish to match the variety of samples in the original data. After all, we want pe to mimic samples from pa of different degrees of "typicality". So we should expect incorporate some measure of entropy, e.g. the commonly used Shannon entropy.
2.2 Matching Local Moments
Recall the apparent tradeoff between autoregressive models and adversarial models. In the spirit of the former, suppose we seek to directly learn transition conditionals via supervised learning. That is,
argming En-y, (-1i)))), no (-11))
Consider the log likelihood loss C(T, (-(411)) = (e-,,(-1k) log To (x]h). In the case of expo-
nential family models for T- [h), basic result is that this is dual to maximizing its conditional en- tropy subject to the constraint on feature expectations EE, En-p,,-==a(( [(x)
(1)T))) tn>=,, (-1A) (11) (zz),
where T X² + R is some sufficient statistic (37 39). More generally for deep energy-based models, we have (however, recall that strong duality does not generalize to the nonlinear case; see Appendix (A):
argmin,
(E howp log ( (h) + max fERXx (E how, (-1h)
f(h,2) -E hops (n17,(()
f(h,2))
(3)
Note that the moment- matehing constraint is local that is, at the level of individual transitions, and all conditioning based on from H4 alone. This is precisely the "exposure bias": The objective is only ever exposed to inputs h drawn from the (perfect) source distribution Ns, and is thus unaware of the endogeneity of the (imperfect) synthetic distribution μ4g induced by πo. This is not desirable since my is rolled out by open-loop sampling at test time. Now, although at the global optimum the moment- matching discrepancy must be zero (i.e. the equality constraint is enforced), in practice there may be variety of reasons why this is not perfectly achieved (c.g. error in estimating expectations, error in
3

---

function approximation, error in optimization, etc). Suppose we could bound how well we are able to enforce the moment matthing constraint; as turns out, we cannot eliminate error compounding L
Lemma Let max JERMxx (E
(h, (-1)
I hop,, f(),a)) e. Then AF.(0) E e(T2)).
Proof. Appendix A
This reveals the problem with modeling conditionals per se: Not all mistakes are equal. An objective like Squation[peenaizeee unrealistic transitions (h, by treating all conditioning histories h equally- regardless of how realistic h is begin with. Clearly, however, we care much less about how looks like, if the current subsequence h is already highly unlikely (and vice versa). Intuitively, earlier mistakes in a trajectory should weigh more: Once Te wanders into areas of H with low support in Na no amount of "good" transitions will bring the trajectory back to
areas of 7 under ps-
2.3 Matching Global Moments
Now suppose instead that we seek to directly constrain the trajectory distribution pe to be similar to p,:
argming C(ssPo))
Consider the Kullback- Leibler divergence [(ps: pe) == DKL (p& pp)) Like before, we know that in the
case of exponential family models for pg. this is dual maximizing its entropy subject to the constraint
Er-p, (T)) = Ez-ps TTT), where T T - R is some sufficient statistic [40]. More broadly for deep energy-based models, (but again, recall here that strong duality does not generalize to the nonlinear case; see Appendix Aj.
have arg ming (Er-~pp log po(T) max FERT (Ky-p, F(T) ET F(T))
Now, observe that by definition of occupancy measure μ., for any function f f H X - it must be the case that Eyop E, [he, IT) TE (, x). Therefore we may equivalently write
argming (E hewuo log To hh) + max JE (E hww, z(h,x) - E heuu f(h, 22)) (-/ita-1hh))
(5)
Importantly, note that the moment-matching constraint is now global that is, at the level of trajectory rollouts, and Tyy is now conditioned on histories h drawn from its own induced occupancy measure μig There is no longer any "exposure bias" here: In order respect the constraint, not only does no (-1h) have to be close to n. (-[h) for any given h, but the occupancy measure bi induced by my also has to be close the occupancy measure ft induced by .. As it turns out, this seemingly minor difference is sufficient to mitigate compounding errors. As before, although at the global optimum the moment- matching discrepancy must be zero, in practice this may not be perfectly achieved. Now, suppose matching constraint; but we now have: we could bound how well we are able to enforce the moment- moment-mathhing
Lemma Let max
(E hww, x(, - EE R-uu f(h, x)) S e. Then (F, O(Te). x>n,((1))
Proof. Appendix A
This illustrates why even transifion-centric adversarial models such as 12 21 have shown promise in which directly attempt to minimize some form of Equation 4 in transition-centric GANs the objective is to match the transition marginals p (, .:) and μt, (h, x)- so the discriminator provides more granular feedback to 2 Ithat matching transition marginals the sort of moment-matchinn that alleviates compounding error.
generating realistic trajectories (23 First, unlike trajectory--emtric GANs [8 19
the generator for training. At the same time, we see from Lemma
we shall start by tackling Equation itself. As we shall see, this endeavor gives rise technique that trains a conditional policy (for sampling), an energy model (for evaluation), and a non-adversarial framework (for learning)aaddiee
Can we be more direct? In Section 3
our three initial criteria.
3 Generating by Imitating
First, consider the most straightforward implementation: Let us parameterize f E R as Φ. and begin with the primal form of Equation which yields the following adversarial learning objective:
'Lemmas I and 2 are similar in spirit to results for error accumulation in imitation by behavioral cloning and distribution matching. See Appendix A this analogy with imitation learning is formally identified in Section 4

---

((),6) == maxe ming
Aeru (-1R)
log (2)2)) + hop, fo(b, x) - E hempo fa(k, (6) 2~x,(11k)
It is easy to see that this effectively describes variational training of the energy-based model pT( = e log w (T((T)= = Σ, fh) approximate the true (T), using sam- ples from the variational po. The (outer) energy player is the maximizing agent, and the (inner) policy player is the minimizing agent. The form of this objective naturally prescribes bilevel optimization procedure in which we perform (gradient-base) based) updates of & with nested (best-response) updates of A.
3.1 Challenges of Learning
Abstractly, of course, training energy models using variational samplers is not new: Multiple works in static domains- as image modeling- have investigated this approach as means of bypassing the expense and variance of MCMC sampling 42] In our setting, however, there is the additional temporal dimension: The negative energy F (T) of any trajectory is computed as the sequential composition of stepwise qualities (h,,,,))) and each trajectory sampled from po must be generated as the sequential rollout of stepwise policies no (x (he) Consider the gradient update for the energy,
c = E
hapel Volo(h, x) - E hーpa Volo(h,2)) (-1b)(-1
am-To(-1h)
and the inner-loop update for the policy,
argming h-mp log (x]())) - happ So x)) (o160
(-170-11h)
Note that the max-min optimization requires complete optimization within each inner update in order for the outer update to be correct. Otherwise the gradients will be biased, and there would be no guarantee the procedure converges to anything meaningful. Yet unlike in the static setting-for which there exists variety of standard approximations for the inner update - amounts to entropy-regularize Thus our first difficulty is computational: Repeatedly performing inner-loop RL. is simply infeasible.
the policy update reinforcement learning [45|47] using so xx, ( as reward function.
Now, an obvious alternative is to dispense with complete policy optimization at each step, and instead to employ importance sampling to ensure that the gradients for the energy updates are still unbiased:
v Erop, Po((T)
1 Z
Er-pp
Ee Sa (he, 2x,
VoFo(T)
Ihe To (be)
where the partition function is computed as Ze Ey (exp (h,, x (og and the sampling policy To is no longer required to be perfectly optimized with respect to fa. Unfortunately. this strategy simply replaces the original difficulty with statistical one: As soon as we consider time-series data of non-trivial lengths T, the multiplicative effect of each time step on the importance weights means the gradient ciimmttesalbeii unbiased will have impractically high variance 48
3.2 Contrastive Imitation
We now investigate generative framework that seeks to avoid these difficulties. The key idea is that instead of Equation 7. we shall learn p& by contrasting (real) "positive" samples T ~ p. and (any) "neg- ative" samples 7 Pg. which-as we shall sec-rids us of the requirement that τa be fully optimized at each step for learning to be guaranteed. First, let us establish the notion of "structured classifier" D
Definition 2 (Structured Classification) Recall the Th rrdindduedd distribution pe (T) == IIe to (o(zz(h)) Denote with P the un-normalized energy-basce model such that (T( == exp( E. fo (hy, xc)). and let Zg be folded into ó as a Icarnable parameter. Define the structured classifier do + [0, 1]:
2g (T(()
(T)
(10)
2 (T) + Po(T)
"The idea that density estimation can be performed by logistic regression goes back at least to 50) and
formalized as negative sampling (51) and conts-contrasttve estimation (52). Structured classifiers have been studied in the context of imitation learning (53]54) by analogy with GANs. In the time-series setting, however, we shall see that this approach is equivalent contrasste estimation with an adaptive noise distribution.
5

---

(h.
Black Box Classification
Black Box Classification
Structured
Samples
Classificttion
Pa
P
Condinonnll MLE Grodient
Trajectory Samples
Adversaiial LL Grndient
(h.x) Samples
Advenarial LL Gradient
Trajectory Samples
Non advensrial Policy Gradient
Pe
W
Pe
g
Hs
g
Pe
W
Explieil Policy Rollout
Implicit
Implicit Geserttoo
Explaci
Geseraiio
Policy Rollout
(a) T-Forcing
1
(b) CARNN-GAN
18
(c) TimeGAN
12
(d) TimeGCI (Ours)
Figure 1: Comparison Time-series Generative Models. Examples of (a) conditional MI.F-hased autoregressive
model, (b) trajectory-centric GAN, and (c) transition-centric GAN. (d) Our proposed technique. See also Table
That is, unlike a black-box classifier that may be arbitrarily parameterized-suuce as a generic discrim- inator d in GeN-here dø, & is "structured" in that it modularly parameterized by the embedded energy and policy functions. Now, we shall train ø such that da discriminates well between T ~ P. and T pe that is, so that the output (T) represents the (posterior) probability that T is real,
Cenergy (6; = -EETo, log.o,o(T) (T)- log (1-da,((T)
(11)
and as before,
ppllicy (0: ④) = E
leppa log (z/hh- E hmpo (h) 2--00(-11)
(12)
Why is this better? As we now show formally, each gradient update no longer requires to be optimal for the current value of & nor does it require importance sampling uunlie the procedure described by Equation 6 The only requirement
é-nor
that po can be sampled and evaluated efficiently, e.g. using learned Gaussian policies as usual, or-should more flexibility be required-win normalizing flow- based policies. As a practical result, this means policy updates can be interleaved with energy updates, instead of being nested within repeated inner loop. Specifically, let establish the following results:
Proposition 3 (Global Optimality) Let fo E IR and let E A(T) be any distribution satisfy- ing positivity: Pa(T) > 0 + po(T) > (this does not require Ty be optimal for fo). Then Lenergy (ø; 0) is globally minimized at (-) - log Z log (.(.), whence Pş is self-normalized with unit integral.
Proof. Appendix A
This result is intuitive by analogy with noise-contrastive estimation 52 55
is learnable long as negative samples 7 ~ Pa cover the support of the true The positivity condition is mild (c.g. take Gaussian policies TT)) and so is the realizability condition (e.g. take neural-networks for fa). Importantly, note that at optimality classifier de a is decoupled from any specific value of 0; contrast this with generic discriminators d in GANs, which are only ever optimal for the current generator. Now, in practice we must approximate p. and pe using finite samples. In light of this, two questions are immediate: First, does the learned é converge to the global optimum the sample size increases? Second, what role does the "quality" of the policy's 's samples play in how á is learned? For the former:
Proposition 4 (Asymptotic Consistency) Let 6" denote the minimizer for Leneryy (c; f), and let d% denote the minimizer for its finite-data approximation- that is, where the expectations over p. and Po are approximated by M samples. Then under some mild conditions, as M increases 0i _ o².
Proof. Appendix A
Now for the second question: Clearly if po were too far from p.. learning would be slow-tth the job would be too easy for the classifier dg and it may be able to distinguish samples via basic statistics alone. Indeed, in standard noise-contrastive estimation with a fixed noise distribution, learning is inef- fective in the presence of many variables (56). Precisely, however, that is why we continuously update the policy itself as an adaptive noise distribution: As Ps moves closer Ps, does py- py-thus provid- ing more "challenging" negative samples In fact, should we insist greedily taking each policy update to optimality, we recover a "weighted" version of the original max-min gradient from before:
It is easy see that minimizing Equation 12 equivalently minimizes the reverse KL div. between pe and po.

---

Proposition 5 (Gradient Equality) Let Φb be the value taken by ó after the k-th gradient update, and let gg denote the associated minimizer for Ppoiicy (0: Φt). Suppose Pe is already normalized; then
V (6;; 0%) = V Vo ())
That is, at og the energy gradient (of Equation recovers the original gradient (from Equation 77 In the general case, suppose pa is un-normalized, such that pv: = Po/ K for some constant Ka; then 0%) Ko+1 anmeot
V Cenrry
(o: (o:
TKg E hoppa V vofa(h,x) Ko+I hww, Vofo(h.x) 2-0((1h)
T
Proof. Appendix A
This "weighting" is intuitive: If P were un-normalized such that K6 > 1. the energy loss automati- cally places higher weights on negative samples M 2 wa; (-14) to bring it down; conversely, if Pe were un-normalized such that Kg < L, the energy loss places higher weights on positive samples Px.2 ~ T. (-Ih) to bring up. (If pa were normalized, then Kg = 1 and the weights are equal). In sum, we have arrived at framework that learns an explicit sampling policy without exposure bias, a decoupled energy model without nested or saddle-point optimization, and is self- ooff-oomaliziig without importance sampling or estimating the partition function. Figure gives representative comparison.
3.3 Optimization Algorithm
Algorithm 1 Time-series Generation by Contrastive Imitation
b Details in Appendix B
I: Input: source dataset D p., mini-batch size M, regularization coefficient learning rates À
Initialize: replay buffer B, energy parameter Φ. policy parameter critic parameter A
3: for each iteration do
4: for cach policy rollout do
5: B - L {T ~ }
b Generate sample
6:
for each gradient step do
7:
Auctor V Eactor (0; ②b v/ + VoLo
D. Update policy
8:
d <- Avmergy
V Lmongy (o: 0)
D. Update energy
9:
*/b < db Acsiic V Lattic (w:: g)
D. Update critic
10:
Output: Iearned policy parameter 0" and energy parameter d"
The only remaining choice the method of policy optimization. Here we employ saft actor-critic [57 although in principle any technique will do-the only requirement is that it performs reinforcement learning with entropy-regularizatin (45 47 To optimize the policy per Equation 12 in addition to the policy "actor" itself, this trains a "critic" to estimate value functions. As usual, the actor takes soft policy improvement steps, minimizing Le vb) == En~B Ey (4) [log (1(2) h) - Q (h, x:]. where Qp H x X' + IR is the transition- wise soft value function parameterized by v. and B is a re- play buffer of samples generated by no. For stability, the actor regularized with the conditional MILE (owwn,(IM (-1N) log To(x))) The critic is trained to minimize the soft Bellman residual: ferini Φ) = B 2v (h, x) fo (h, x:) (h))), where the state-values are bootstrapped as (hh) Ey' mo( (1e) (h' x') log Ta To [f']. By expressly training an imitation policy to mimic time-series behavior using rewards from an energy model trained by contrastive learning, we call this framework Time-series Generation by Contrastive Imitation (TimeGCI): See Algorithm
loss L == E, Bz
4 Discussion
Our theoretical motivations are apparent (Sections 2.2 3.1. and the practical mechanics of optimiza- tion straightforward (Section 3.2 To understand the strengths and limitations of TimeGCI, two questions remain: First, how does this relate to bread-and sauudd--n-butteee imitation learning of sequential decision-making? Second, how does this compare with recent deep generative models for time series?
Imitation Perspective sequential decision-making imitation learning deals with training a policy purely on the basis of demonstrated behavior- -hat is, with no knowledge of the reward signals that induced the behavior the first place [58-60). Consider the standard Markov decision process setting, with states E 2, actions E U,, dynamics w E 2 xu and rewards E R x4 Classically, imitation learning seeks to minimize the regret R, (0) E,, [E, u. -E, Eip(z,, 4, ). with ne To ((2)² here being the demonstrator and imitator policies, and expectations are taken over episodes generated per ug ~ x(-12)) and žt+1 ~ w(-1zt, ut) [61 62) First, observe that by interpreting h as "states" and as "actions", our problem setup bears precise resemblance to imitation learning:
7

---

Table 1: Comparison of Time-series Generative Models, Examples of conditional MLE-based autoregressive models, trajectory-centric GANs, transition-centric GANs, as well as our proposed technique. See also Figure
Type
Examples
Optimization Objective(s)
Generator Discrim.
Signal
Signal
posure Bias Discrim. Adversarial Policy Energy
No Ex-
Decrupled
Non-
Explicit
Explicit
Condit.
MLE
I-Forcing
Data LL
Stepwise
(NIA)
(NIA)
Z- Forcing
Data LL (ELBO)
Stepwise
(NIA)
(NIA)
v
F-Frriing
T0
Data LL+ Class. LL. (pe v. Po)
Stepwise
Global
x
x
/
"Traject.
GAN
C-RNN-GAN
18
Classification L.L. (De v. Pa)
Global
Global
x
x
x
DoppelGANger
19
Classification LL (Po ps)
Global
Global
x
x
x
COT-GAN
20
Sinkhorn Divergence (ps V. ps)
Global
Global
/
x
x
x
x
Transit. GAN
RC-GAN
21
Classification LL (Jt Ps)
Stepwise Stepwise
x
x
x
x
T.GGAN
Classification L.I. (Jta V. (aa)
Stepwise Stepwise
x
x
x
x
TimeGAN
Class. L. (pa V. pa) + Data LL
Stepwise Stepwise
x
X
X
TImeGCI (Ours)
Discrim.: Class. LL (py v. p.)
Stepwisc Global
/
/
Generator: Policy Optimization
Corollary (Generation as Imitation) Let state space 2 == H. action space 4 == X, and reward function p = fs. In addition, let the dynamics be such that w (-IMe,,2 24) is the Dirac delta centered at he+1 ..., .e). Then the regret exactly corresponds to the expected quality difference: R, FF,
Proof. Immediate from Definition
T
Now, since we want low regret but have no knowledge of the true quality measure (i.e. "reward sign- al"), we may naturally learn it together. In this sense, TimeGCI is analogous to imitation by inverse reinforcement learning (IRL), which seeks to infer rewards that plausibly induced the demonstrated be- havior, and to optimize imitating policies on that basis (63]66). Further, in simultaneously optimizing for variety (cf. entropy) and typicality (cf. energy), TimeGCT is analogous to maaxiimm--nttrppa IRL [67]68). Our contrastive approach also bears mild resemblance to stepwise discriminators studied in this vein [54 69] although our framework focuses on trajectory- modeling, and is not adversarial (see Appendix for more discussion on how TimeGCI relates to popular imitation learning methods).
There are also crucial differences: In imitation learning, dynamics are generally Markovian; states are readily defined as discrete elements or real vectors, and action spaces are small/discrete. The practical challenge is sample efficiency to reduce the cost of environment interactions [T0. In time-series generation, however, rollouts are free- gererannna synthetic trajectory does not require interacting with the real world. But dynamics are never Markovian: The practical challenge is that representations of variable-length histories must be jointly learned. Moreover, actions are the mill-dimesiionll feature vectors themselves, which renders policy optimization more demanding than usual (see Appendix BB; beyond the tractable tabular settings we experiment in, higher-dimensional data may prove challenging.
Related Work Table I summarizes the key differentiators of TimeGCI from prevailing techniques. As discussed in Section 1 MLE-based autoregressive models [0 13 are easy to optimize, and learn explicit conditional distributions that can be used for inspection, resampling, or uncertainty estimation, but they suffer from exposure bias [8] 12] GAN-based adversarial models fall into two camps: For majectoory-centre generator, they often struggles to converge to the adversarial objective without extensive tuning [22 methods 12 22] provide more granular signals to guide the generator, but this simply alters the objective of learning p. to one of learning M.s and still inherits the disadvantages of implicit, adversarial learning.
methods [18] 19 72,, with only sequence-level signals to guide the
with the exception of (72]. which utilizes Sinkhorn divergences instead. Transition-cent
Our analysis is built on ideas from energy- based models (EBMs) 73 75 and reinforcement learning for sequence prediction 78). In particular, our initial formulation (Section 3.1 can be viewed as a temporal extension of variational EBMs (41 42). Moreover. by adaptively learning सe to give negative samples for da, d. the formulation we study (Section 3.2) is equivalent to temporal analogue of noise-contrastive estimation (NCE) (55 79. More tangentially, conditional EBMs have been trained with NCE for text generation (0-882 and the strength of global normalization has been studied (83); that said, these are confined to the case where external input tokens are available for conditioning at each step- and not free-runnnt free-running as in our time-series setting. Finally, note that viewing sequence generation as a decision-making problem is present in language modeling (5) where aaek-specific metrics are available as signals. In the absence of predefined signals, GAN-based methods that jointly train discriminators to provide rewards for imitation have been studied [84] although they are adversarial, and all focus on the special case of generating discrete tokens for language modeling.
8

---

5 Experiments
Benchmarks We test Algorithm| I (TimeGCI) against the following: The classic Teacher Forcing trains autoregressive networks using ground-truth conditioning (T-Forcing) [5]. Professor Foreing uses adversarial domain adaptation by training an auxiliary discriminator to encourage dynamics of the network's 's free-running and mocher-forced states to be similar (P-Forcing) (10). Trajectory- recurrent GANs (C-RNN-GAN) directly plug RNNs into the GAN framework as generators and dis- criminators for full sequences (18). Causal Optimal Transport GAN (COT-GAN) is the latest variant of this (20]. proposing to approximate Sinkkorn divergences instead of the standard JS divergence. For transition-centric recurrent GANs (RC-GAN), the adversarial loss is computed as the sum of log likelihoods for the stepwise feature vectors conditioned on histories [2I], instead of directly as the log likelihood for the entire sequence, Finally, Time-series GAN (TimeGAN) is its latest incarnation (12). proposing to generate and discriminate within a jointly optimized embedding space for efficiency.
Datasets We employ five tabular time-series datasets with a variety of different characteristics, such as pe- riodicity, noise level, and correlations: First, we use synthetic dataset of multivariate sinusoids with different frequencies and phases (Sines) [12). Second, we use a UCI dataset from the monitored energy usage of house- hold appliances in low-energy house (Energy) [90]. Third, we use a UCI dataset from
Table 2: Summary Statistics for Datasets Used.
Dataser
Dimeusion Length Autocor.
+3 Lag
Lag
Sines Metro
0.875
0.623
0.377
9
24
0.429
0.200
0.029
Gas
0.382
Energy
29
24
0.702
0.411
0.176
MIMIC-I11
0.532
0.212
0.059
semiconductor gas sensors for chemical
detection (Gas) [91). Fourth, we use a UCI dataset of hourly interstate vehicle volume at a state traffic recording station (Metro) [92]. Fifth, we use medical dataset of intensive-care patients from the Medical Information Mart for Intensive Care (MIMIC-III) (93). All datasets are accessible from their sources, and we use the original source code for preprocessing sines and the UCI datasets by (12), publicly available at (94). Table]ssoow summary statisties for the datasets used in the experiments.
Experiments for each dataset are arranged as follows: The real trajectories that constitute the original dataset ID are fed input to train all algorithms. Each algorithm is subsequently used in test mode to generate 10,000 synthetic trajectories. Then, the performance of cach algorithm evaluated on the basis of these generated trajectories. This process is then performed for a total of 10 repetitions, from which we compile the means and standard errors for each reported result. For fair comparison, analogous network components across all benchmarks share the same recurrent architec- ture: Wherever generator, policy, discriminator, energy, or critic network applies, we use LSTMs with one hidden layer of 32 units to compute hidden states for representing histories h, and two fully- connected hidden layers of 32 units each and ELU activations to compute task-specific output variables (i.e. the generator output, policy parameters, discriminator output, energy functions, or critie values). In other respects, we use the publicly available source code to construct the benchmark algorithms- accessible at (84. See Appendix C for additional detail on hyperparameters and implementations.
Implementation
Evaluation and Results In the tabular data setting, assessing synthetic data generation is inherently tricky (97 Unlike in media-specific applications, we have no predefined measures such as music polyphony or BLEU scores, nor can we use human evaluation of realism as done for videos. For tabular time-series, the generally accepted standard for comparing synthetic data to apply the Train- my-Shhhheii,, Test-on-Real (TSTR) framework, first proposed by (2T) and employed by most recent work in synthetic time-series generation [12 [2] I01], as well as more generally for tabular synthetic data of any kind [99 100 102). Specifically, we apply the performance measure used by [12 101] to quantify how much the synthetic sequences inherit the predictive characteristics of the original dataset (Predictive Score): Using synthetic samples, a generic post-hoc sequence-pre preecicciooniiioonnnn model is learned to forecast next-step feature vectors over training sequences. Then, the trained model is evaluated on the original data, and its predictive performance is quantified in terms of the mean absolute error. We use the original source code for computing this metric, publicly available at (94).
Further to prior works using this measure, we additionally believe that synthetic data evaluation should be more general than just next-step TSTR forecasting. After all, the distinguishing characteristic of sequential (vs. static) data generation is that we care about evolution of features over time. Hence we also compute TSTR metrics for horizons of other lengths (+3 Steps Ahead and +5 Steps Ahead). Importantly, note that key strength of TSTR evaluation is in its sensitivity to mode collapse: If any generation scheme suffers from mode collapse (as GAN methods are prone to), TSTR scores would degrade due to the synthetic data failing to capture the diversity of the real data, which means any

---

Table 3: Performance Comparison of TimeGCI and Benchmarks. Bold numbers indicate hest-performing results.
Benchmark
Metric
Sines
Energy
Gas
Metro
MIMIC-III
Predictive Score
0.108 + 0.002
0,310 + 0.001
0.035 = 0.003
0,242 = 0.001
0.017 = 0.001
T-Forcing
+3 Steps Ahead
0.115 + 0.001
0.281 + 0.001
0.080 0.001
0.244 + 0.001
0.024 0.007
+5 Steps Ahcad
0.122 + 0.003
0,270 + 0.002
0.111 = 0.001
0,248 = 0.001
0.018 0.003
C-Corr. Score
8.369 + 0.015
194.1 0.043
150.8 0.067
4.222 = 0.013
400.9 3.203
Predictive Score
0.105 + 0.001
0,303 + 0.002
0.037 + 0.001
0,241 + 0.001
0.023 0.006
Forcing
Steps Ahead
0.110 1: 0.001
0.268 0.002
0.086 0.002
0.241 0.001
0.001
+5 Steps Ahead
0.115 + 0.001
0.259 + 0.002
0.121 + 0.002
0.242 + 0.001
0.017 + 0.001
2. -Curr. Score
8.156 0.010
207.6 0.057
150.5 0.023
3.014 0.006
2.901
Predictive Score
0.751 +: 0.001
0,500 1: 0.001
0.242 1: 0.001
0.419 - 0.005
0.019 0.001
+3 Steps Ahead
0.769 + 0.001
0.500 = 0.001
0.243 = 0.001
0,416 = 0.002
0.020 = 0.001
C-RNN-GAN
+5 Steps Alead
0.786 1: 0.001
0,501 4 0.001
0,241 1: 0.001
0,416 4 0.003
0.019 0.001
x-Corr Score
10.76 + 0.012
6
266.4 + 0.008
18.39 = 0.003
1720. 0.339
Predictive Score
0.099 + 0.001
0,259 0.001
0,022 0.001
0.245 0.001
0,014 0.001
+3 Steps Ahead
0.109 + 0.001
0.261 + 0.001
0.050 0.001
0,246 + 0.001
0,013 + 0.001
COT-GAN
Steps Ahcad
0.110 + 0.001
0,262 0.001
0.072 0.001
0.245 0.001
0013 0.001
z- z-Corr. Score
3.114 + 0.038
67.93 + 0.227
25,56 +: 0.156
3.055 + 0.013
497.7 2.581
Predictive Score
+ 0.001
0.498
0.001
0.412 + 0.003
+3 Steps Aleaad
0.770 1 0.001
0,500 1 0.001
0.244 1 0.001
0.415 1 0.004
0.019 1 0.001
RANGAN
SSeppshheas Ahead
0.786 + 0.001
0.499
0.001
0,243 0.001
0,418 0.004
0.018 0.001
2- -Curr. Score
5.649 1 0.012
582.3 1 0.047
231.2 1 0.003
19.77 1 0.001
1592. 1 0.192
Predictive Score
0.196 +: 0.006
0.261 + 0.001
0.264 0.011
0,245 : 0.002
0.502 0.023
TimeGAN
+3 Steps Ahead
0.223 + 0.006
0,263 = 0.001
0,251 = 0.014
0.243 = 0.001
0.484 0.021
+5 SSeps Ahead
0.246 + 0.005
0,262 1 0.005
0.252 + 0.012
0,242 + 0.001
0.433 0.020
2-Con. Score
17.86 + 0.001
667.5
0.001
282.5 0.001
17.11 = 0.001
2140, 0,010
Predictive Score
0,097 + 0.001
0.251 = 0.001
0.018 + 0.000
0.239 = 0.001
0.002 = 0.000
TimeGCI (Ours)
Steps Ahead
0.104 + 0.001
0.251 0.001
0.042 0.001
0.239 0.001
0.001 0.000
+5 Steps Ahead
0.109 + 0.001
0,251 + 0.001
0.067 + 0.001
0.239 - 0.001
0.001 = 0.000
C-Corr Score
1.195 + 0.011
105.2 + 0.433
47.91 0.811
0.738
0.019
194.3 0.180
prediction model trained on that basis would also fail to capture this variation). Finally, similar to some recent works [19 20). we also compute the cross-correlane cosssscorelstionn of real and synthetic feature vectors, and report the sum of the absolute differences between them, averaged over time (æ-Corr. Score); this serves to verify feature relationships are preserved well, in addition to temporal relationships. Table the results: With respect to these metrics, we find that TimeGCI somewhat consistently produces synthetic samples that perform similarly or better than benchmark algorithms in all datasets. (Note that we do empirically observe several instances of mode collapse in GAN-base benchmarks).
6 Conclusion
In this work, we invite an explicit analogy between time-series generation and imitation learning, and explore framework that fleshes out this connection. Two caveats are order: First, while we began from the notion of moment-matching to address the error compounding problem, in practice there is no guarantee that this is accomplished well during optimization. In particular, scalability is a major limitation beyond the range of feature dimensions and sequence lengths considered in our experiments. Sample-based based estimates could rapidly degrade with the horizon, especially if transitions are highly stochastic. A relevant question is whether not training on fixed subsequence lengths could poten- tially alleviate this concern for longer sequences. In addition, while our approach seeks to dispense with the instabilities typical of adversarial training, we instead left with the diffieulties of policy optimization, which may prove a prohibitive challenge in higher-dimensional feature spaces. For the datasets we consider, we find that pre- pre-training and regularizing the policy with maximum likelihood, combined with a small enough learning rate, had the most impact in promoting stability and learning. Second, we reiterate that perennial challenge in modeling tabular data is in choosing the metric for evaluation. While we opted for the most commonly accepted method of TSTR, this may not be general enough to capture the range of downstream tasks that may be performed on the synthetic data. Future work will benefit from a deeper investigation into more sophisticated measures for time series, such as contrastive methods and how to evaluate different aspects of the "quality" of the generated trajectories.
Acknowledgments
We would like to thank the reviewers for all their invaluable feedback. This work was supported by Alzheimer's 's Research UK, The Alan Turing Institute under the EPSRC grant EP/N510129/1, the US Office of Naval Research, as well as the National Science Foundation under grant number 1722516.
10

---

References
[1] Jason Walonoski, Mark Kramer, Joseph Nichols, Andre Quina, Chris Moesel, Dylan Hall, Carlton Duffett, Kudakwashe Dube, Thomas Gallagher, and Scott McLachlan. Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association, 2018.
[2] Anna I. Buczak, Steven Babin, and Linda Moniz. Data-driven approach for creating synthetic electronic medical records. BMC medical informatics and decision making, 2010.
[3] Saloni Dash, Andrew Yale, Isabelle Guyon, and Kristin Bennett. Medical time-series data generation using generative adversarial networks. International Conference on Artificial Intelligence in Medicine (AIME), 2020.
[4] James Jordon, Daniel Jarrett, Evgeny Saveliev, Jinsung Yoon, Paul Elbers, Patrick Thoral, Ari Ercole, Cheng Zhang, Danielle Belgrave, and Mihaela van der Schaar. Hide-and-seek privacy challenge: Synthetic data generation vs. patient re-identification. NeurlPS 2020 Competition and Demonstration Track, 2021.
[5] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1989.
[6] Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. International Conference on Learning Representations (ICIR), 2016.
[7] Yoshua Bengio and Paolo Frasconi. An input output hmm architecture. Advances in neural information processing systems (NeurIPS). 1995.
[8] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. International Conference on Machine Learning (ICML), 2009.
[9] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Nomain-adversarias training of neural networks. Journal of Machine Learning Research (JMLR), 2016.
[10] Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[11] Ferene Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? International Conference on Learning Representations (ICLR), 2016.
[12] Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial networks. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[13] Anirudh Goyal, Alessandro Sordoni, Mare-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[14] Ahmed M Alaa, Alex Chan, and Mihaela der Schaar. Generative time-series modeling with fourier flows. International Conference on Learning Representations (ICLR), 2020.
[15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[16] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint, 2014.
[17] Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn divergences. International Conference on Artificial Intelligence and Statistics (AISTATS), 2018.
[18] Olof Mogren. C-rnnggnn Continuous recurrent neural networks with adversarial training. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[19] Zinan Lin, Alankar Jain, Chen Wang, Giulia Fanti, and Vyas Sekar. Generating high-fidelity. synthetic time series datasets with doppelganger. ACM Internet Measurement Conference (IMC), 2019.
11

---

[20] Tianlin Xu, Li K Wenliang, Michael Munn, and Beatrice Acciaio. Cot-gan: Generating
sequential data via causal optimal transport. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[21]
Cristóbal Esteban, Stephanie I. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint, 2017.
[22] Giorgia Ramponi, Pavlos Protopapas, Marco Brambilla, and Ryan Janssen. T-egan: Condi- tional generative adversarial network for data augmentation in noisy time series with irregllar sampling. arXiv preprint, 2018.
[23] Luca Simonetto. Generating spiking time series with generative adversarial networks: an application on banking transactions. 2018.
[24] Moustafa Alzantot, Supriyo Chakraborty, and Mani Srivastava. Sensegen: A deep learning architecture for synthetic sensor data generation. In 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pages 188-193.
IEEE, 2017.
[25] Shota Haradal, Hideaki Hayashi, and Seiichi Uchida. Biosignal data augmentation based on generative adversarial networks. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 368-371. IEEE, 2018.
[26] Chi Zhang, Sanmukh R Kuppannagari, Rajgopal Kamnan, and Viktor K Prasanna. Generative adversarial network for synthetic time series data generation in smart grids. In 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (Smar(GridCommm) pages 1-6. IEEE, 2018
[27] Ahmed M Alaa, Boris van Breugel, Evgeny Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. International Conference on Machine Learning (ICML), 2021.
[28] Aditya Grover, Jiaming Song, Alekh Agarwal, Kenneth Tran, Ashish Kapoor, Eric Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood-free importance weighting. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[29] Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. Interna- tional Conference on Learning Representations (ICLR), 2019.
[30] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. International Conference on Learning Representations (ICLR), 2019.
[31] Weili Nie. Nina Narodytska, and Ankit Patel. Relgan: Relational generative adversarial networks for text generation. International Conference on Learning Representations (ICLR), 2019.
[32] Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short, International Conference on Learning Representations (ICLR), 2020.
[33] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. IEEE International Conference on Computer Vision (ICCV), 2017.
[34] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[35] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. International Conference on Leurning Representations (ICLR), 2017.
[36] Vijay R Konda and John Tsitsiklis. Actor-critic algorithms. Advances in Neural Information P Processing Systems (NeurIPS), 2000.
[37] Peter D Grünwald, A Philip Dawid, et al. Game theory, maximum entropy, minimum discrep-
ancy and robust bayesian decision theory. Annals of Statistics, 2004.
[38] Farzan Farnia and David Tse. A minimax approach to supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 2016.
12

---

[39] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal
entropy. Dissertation, Carnegie Mellon University, 2010.
[40] Michael I Jordan. An introduction probabilistic graphical models. 2003.
[41] Taesup Kim and Yoshua Bengio. Deep directed generative models with energy-based probabil- ity estimation. International Conference on Learning Representations (ICLR), 2016.
[42] Shuangfei Zhai, Yu Cheng, Rogerio Feris, and Zhongfei Zhang. Generative adversarial
networks as variational training of energy based models. arXiv preprint, 2016.
[43] Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard Hovy, and Aaron Courville. Cali- brating energy- based generative adversarial networks. International Conference on Learning Representations (ICLR), 2017.
[44] Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint, 2019.
[45] Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. Conference on Uncertainly in Artificial Intelligence (UAI), 2016.
[46] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. International Conference on Machine Learning (ICML), 2017.
[47] Wenjie Shi, Shiji Song, and Cheng Wu. Soft policy gradient method for maximum entropy deep reinforcement learning. International Joint Conference on Artificial Intelligence (IJCAI). 2019.
[48] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[49] Josiah P Hanna and Peter Stone. Towards a data efficient off-policy policy gradient. AAAI Symposium on Data Efficient Reinforcement Learning (AAAI), 2018.
[50] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. Unsupervised as supervised learning. The Elements of Statistical Learning, 2009.
[51] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed repre- sentations of words and phrases and their compositionaliiy. Advances in Neural Information Processing Systems (NeurIPS), 2013.
[52] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research (JMLR), 2012.
[53] Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. NeurIPS Workshop on Adversarial Training, 2016.
[54] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. International Conference on Learning Representations (ICLR), 2018.
[55] lan J Goodfellow. On distinguishability criteria for estimating generative models. International Conference on Learning Representations (ICLR), 2015.
[56) Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT Press Cambridge, 2016.
[57] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), 2018.
[58] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. International conference on artificial intelligence and statistics (AISTATS), 2011.
[59] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Picter Abbeel, and Jan Peters. An algorithmic perspective imitation learning. Foundations and Trends in Robotics, 2018.
[60] Alexandre Attia and Sharone Dayan. Global overview of imitation learning. arXiv preprint, 2018.
13

---

[61] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. International conference on artificial intelligence and statistics (AISTATS), 2010.
[62] Umar Syed and Robert E Schapire. A reduction from apprenticeship learning to classification. Advances in neural information processing systems (NeurIPS), 2010.
[63] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. Interna- tional conference on Machine learning (ICML), 2000.
[64] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. International conference on Machine learning (ICML), 2004.
[65] Ioana Bica, Daniel Jarrett, Alihan Hüyük, and Mihaela van der Schaar. Learning what-if expla- nations for sequential recision-makingg International Conference on Learning Representations (ICLR), 2021.
[66] Daniel Jarrett, Alihan Hüyük, and Mihaela Van Der Schaar. Inverse decision modeling: Learning interpretable representations of behavior. International Conference on Machine Learning (ICML), 2021.
[67] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. AAAI Conference on Artificial Intelligence (AAAI), 2008.
[68] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. International conference on machine learning (ICML), 2016.
[69] Ahmed H Qureshi, Byron Boots, and Michael C Yip. Adversarial imitation via variational inverse reinforcement learning. International Conference on Learning Representations (ICLR), 2019.
[70] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Diseriminator--ctto--ritti Addressing sample inefficiency and reward bias in adversarial imitation. International Conference on Learning Representations (ICLR), 2019.
[71] Lionel Blondé and Alexandros Kalousis. Sample-efficient imitation learning via gans. Interna- tional conference on artificial intelligence and statistics (AISTATS), 2019.
[72] Huan Xu and Shie Mannor. Distributionally robust markov decision processes. Mathematics of Operations Research, 2012.
[73] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy- based learning. Predicting Structured Data, 2006.
[74] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. International Conference on Machine Learning (ICML), 2016.
[75] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[76] Philip Bachman and Doina Precup. Data generation sequential decision making, Advances in Neural Information Processing Systems (NeurIPS), 2015.
[77] Arun Venkatraman, Martial Hebert, and J Bagnell. Improving multi-step prediction of learned time series models. AAAI Conference on Artificial Intelligence (AAAI), 2015.
[78] Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K Reddy. Deep reinforcement learning for sequence-to-seqqence models. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2019.
[79] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation prin- ciple for unnormalized statistical models. International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
[80] Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. International Conference on Machine Learning (ICML), 2012.
[81] Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise- contrastive estimation. Advances in Neural Information Processing Systems (NeurIPS), 2013.
[82] Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018
14

---

[83] Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized transition- basedd neural networks. Annual Meeting of the Association for Compulational Linguistics (ACL.), 2016.
[84] Lantao Yu, Weinan Zhang, Jun Wang. and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. AAAI Conference on Artificial Intelligence (AAAI), 2017.
[85] Sidi Lu, Lantao Yu, Siyuan Feng, Yaoming Zhu, and Weinan Zhang. Cot: Cooperative training for generative modeling of discrete data. International Conference Machine Learning (ICML), 2019.
[86] Haiyan Yin, Dingcheng Li, Xu Li, and Ping Li. Meta-cotgan: A meta cooperative training paradigm for improving adversarial text generation. AAAI Conference on Artificial Intelligence (AAAI), 2020.
[87] William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the_. International Conference on Learning Representations (ICLR), 2018.
[88] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang. Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. AAAI Conference on Artificial Intelligence (AAAI), 2018.
[89] Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. International Conference on Machine Learning (ICML), 2017.
[90] Luis M Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in a low-energy house. Energy and buildings, 2017.
[91] Javier Burgués, Juan Manuel Jiménez-Soto, and Santiago Marco. Estimation of the limit of detection in semiconductor gas sensors through linearized calibration models. Analytica Chimica Acta, 2018
[92] John Hogue. Hourly interstate 94 westbound traffic volume for mn dot atr station 301. Minnesota Department of Transportatiin, 2018.
[93] Alistair EW Johnson, Tom Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Nature Scientific Data, 2016.
[94] Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial networks. https: /////gtttuub com http://aahbc.ccco/o///8///////// j 2019.
[95] Tianlin Xu, Li K Wenliang, Michael Munn, and Beatrice Acciaio. Cot-gan: Generating se- quential data via causal optimal transport. htttps://gitee http://hww...ooiiiaaaaaaaa/////a 2020.
[96] Olof Mogren. C-mn- gan: Continuous recurrent neural networks with adversarial training. https: htt///ggttttbbtt com/ www.ooooreooocoooooooooo c-rn -gan 2016.
[97] Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. htttps:// wwt.w//wwwwwa/c////ootooo RGAN 2017.
[98] Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. Professor forcing. https:://itth//: com http//anrruuh99//1.. http://tthubb.ccmmm.mmm/russ//// CANS 2016.
[99] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[100] Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. Data synthesis based on generative adversarial networks. International Conference on Very Large Data Bases (VLDB). 2018.
[101] Mohammad Navid Fekri, Ananda Mohon Ghosh, and Katarina Grolinger. Generating energy data for machine learning with recurrent generative adversarial networks. Energies, 2020.
[102] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. International Conference on Learning Representations (ICLR), 2019.
15


Reviews:
[{"review_id": "t-IAIUAB74-", "reviewer": "Reviewer_QAxC", "summary": "The paper proposes an imitation learning method that simultaneously learns a policy and an energy (“discriminator”) function. This is similar to a variational approach to learn a Gibbs distribution over trajectories, where the variational parameters consist of the policy and variational inference consists of running RL. \n\nHowever, rather than alternating the normal Gibbs energy loss and the variational inference loss of the variational approach, the submission replaces the Gibbs energy loss with an alternate energy loss based on noise-contrastive estimation, which explicitly trains the energy to discriminate between “noise” samples (drawn from the current policy) and expert samples. \n\nIt is proved that under the noise-contrastive loss, the energy converges to the expert energy regardless of the noise distribution (under weak conditions), which justifies only partially optimizing the RL step in the alternating optimization. This results in a method potentially much faster than the “naive” variational approach, as it is not necessary to fully solve an inner-loop RL problem for each outer-loop iteration.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The method seems easier to train than comparable adversarial imitation learning algorithms\n+ Like MaxEnt IRL, but sidesteps the difficulty of running full inference in the training loop\n+ Theoretical results provide intuition for why this is a reasonable thing to do\n+ Experimentally compared to a variety of other methods and datasets\n\nWeaknesses:\n- No experimental comparison to any adversarial imitation learning methods, which have similar properties\n- No evaluation on imitation learning scenarios\n- Presentation meanders a bit\n- Experimental gains are a bit modest\n\nImpact: \nOverall, the submission presents an interesting idea that may have the potential to make a significant impact on the community, with some caveats. Although the paper is framed as a generic time series prediction method, it does use terminology and ideas from the imitation learning (IL) literature, and I don’t see why it wouldn’t really be applicable to IL. So, I found it odd that the paper seemed to distance itself from IL.\n\nAnyhow, viewed as an IL method, the proposed method seems appealing in that it would seem to combine the mitigation of error compounding that is a feature of adversarial IL with a more stable and/or faster optimization algorithm. I think it has generally been believed anecdotally that adversarial IL and the closely related MaxEnt IRL fix the error compounding problem present in naive behavioral cloning, and I have seem some theoretical results of this nature (see, e.g., chapter 14 of this book: https://rltheorybook.github.io/), but most of these methods tend to be impractical to actually use, either due to unstable optimization, or slowness due to having to run RL in the inner loop. \n\nThis submission seems to propose a reasonable solution to those obstacles, with the caveat that it still requires running an RL step, which could be difficult to manage in practice. The main benefit of the method is that the method used to train the Gibbs energy (noise contrastive estimation) is relatively robust to inaccuracies in the “variational” distribution--i.e., the policy. This means that it is not strictly necessary to run the slow “inner loop” RL step to convergence for every outer loop iteration in order for the outer loop to converge. This should overall improve training stability relative to adversarial IL and should provide speed-ups relative to MaxEnt IRL because of the relaxation of the requirement to run RL to convergence. I believe this is the main takeaway of the paper, and the main reason why this work has the potential for impact.\n\nThe experimental results show a pretty significant improvement in the cross-correlation score, which is very encouraging. However, as mentioned elsewhere, trying the method in IL settings would probably be more interesting.\n\nRigor:\nThe submission scores some positive points in terms of theoretical rigor for including results that build intuition and support the main claims. The experiments also score points for rigor in that several datasets were tried, along with a variety of competing methods.\n\nOne negative aspect in this regard is the fact that the method was not compared to any adversarial imitation learning methods, or evaluated on imitation learning problems. Again, I found it a little perplexing that the work was framed as not being an IL method, despite it seeming very much like an IL method. Although I think this work has potential for impact, that potential is probably much reduced without comparisons to adversarial IL methods in IL settings. Also, the GAIL paper (https://arxiv.org/abs/1606.03476) should be cited, as that features some similar ideas, and is probably the most well-known adversarial IL method.\n\nAnother note about the experiments: the “predictive score” does not seem that intuitive or informative to me. For methods that produce a PDF for the policy, one could just compute the data likelihood (i.e., cross entropy) on a held-out dataset. \n\nI did not thoroughly check the theoretical results--in particular, I did not check the proofs of propositions 4 and 5. I skimmed the other results, and they seem reasonable. I did notice one minor issue with Lemma 2: its premise seems stronger than intended. Namely, the premise looks like $\\max_f E_{x \\sim p_s} f(x) - E_{x \\sim p_t} f(x) \\leq \\epsilon$, where $f$ is unrestricted. However, if $f$ is unrestricted, then the previous expression must equal either $0$ or $\\infty$--it can’t be equal to a finite value, because scaling $f$ always leads to a larger value, unless the value is $0$. So, I believe $f$ must be assumed bounded in order for the premise to make sense.\n\nAlso, section 3 casually mentions swapping the order of the min and max using the minimax theorem, but this is misleading, because “the” minimax theorem only applies for concave-convex games. This would therefore be inapplicable if the discriminator were parameterized as a nonlinear function.\n\nNovelty:\nAs far as I know, the main idea of essentially applying NCE in MaxEnt IRL is novel--although one could argue that the general idea of using different generator or discriminator objectives in adversarial training is pretty common. In terms of the theoretical results, Lemma 1 seems like a variation of a well-known result that behavioral cloning suffers from a compounding error phenomenon--this should probably be made clearer. I’m not really sure about the novelty of Lemma 2, but there is a very similar result in chapter 14 of this book: https://rltheorybook.github.io/rltheorybook_AJKS.pdf (sample complexity of DM-ST) as well as a similar result for GAIL here: https://arxiv.org/abs/2010.11876. Proposition 3 seems like a relatively straightforward application of the existing result for NCE. I did not really examine propositions 4 and 5 closely, so I can’t comment as to their novelty.\n\nClarity of presentation:\nThe presentation could use significant work. In particular, although I found Lemma 1 and Lemma 2 to be useful to build intuition, overall, sections 2.2 and 2.3 seem like a diversion from the main idea. I also thought that equations 3 and 5 were introduced in a slightly suspicious way, by implying that this is some sort of generalization of MaxEnt duality when the features are represented as nonlinear functions. I don’t think that’s really the case. So, I think more care should be taken to motivate these equations.\n\nAlso, as previously mentioned, the min/max swap in section 3 is also a bit misleading. Altogether, this gives the impression of a sloppy line of reasoning leading up to the main result. If I were to rewrite the paper, I would omit sections 2.2 and 2.3, and motivate equation 6 directly as a variational training objective, instead of motivating it through somewhat suspect analogies. I would then only swap the min and max to motivate the analysis, which shows that under some conditions, optimizing this objective leads to moment-matching and non-compounding error.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "oDlhubGd2as", "reviewer": "Reviewer_YqJQ", "summary": "The paper proposes to perform time-series generation via a method inspired by imitation learning in a non-adversarial manner. The idea is to fit an energy-based model over the trajectories with noise contrastive estimation, with the noise being provided by a parametrized policy. The policy is also optimized to fit the energy-based model to provide a sound noise distribution. The method demonstrates superior empirical performance on various time-series datasets, on prediction error, and cross-correlation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Update**\nI think the ablation studies address my concerns. More robustness of TImeGCI is shown, and may be a reason why we observe better performance (despite having the same \"amount\" of supervision). I am willing to raise the score because of this.\n\n**Originality and significance**\n\nWhile the paper discusses the relationship between time-series prediction and imitation learning, it seems that imitation learning methods are not given enough credit. For example, existing works in imitation learning [1,2] have already proposed methods to learn an energy-based model from the trajectories and then fir the policy to the energy function. \n\n[1] Kim, K., Jindal, A., Song, Y., Song, J., Sui, Y. and Ermon, S., 2020. Imitation with Neural Density Models. arXiv preprint arXiv:2010.09808.\n[2] Arenz, O. and Neumann, G., 2020. Non-Adversarial Imitation Learning and its Connections to Adversarial Methods. arXiv preprint arXiv:2008.03525.\n\nGiven these prior works, the novelty of the proposed method would take a hit, as time-series prediction problems can be posed as an imitation learning problem with the \"states\" in the MDP being the \"history\" of the previous timesteps, and \"action\" in the MDP transitions to the next \"state\" concatenated with \"history\".\n\nEven in the context of time series, the application of NCE is not entirely novel, such as https://www.cs.toronto.edu/~amnih/papers/ncelm.pdf\n\n**Clarity**\n\n**Explanation of empirical success** Another question that I have for the paper is why the method performs better and whether the success of \"trajectory-level\" imitation learning methods can be replicated here; one reason that GAIL-like methods outperform behavior cloning is that the RL algorithm can make use of external signals (e.g., \"done\") given by the environment. I have personally done experiments with GAIL and behavior cloning before and found it hard to outperform the latter without such information. The proposed method here does not appear to use any additional information than T-forcing or P-forcing, so the advantages must come from the loss function. However, there are no other experiments/ablation studies that show why the proposed method is better -- is it because of better first-step prediction, or is it because of better robustness when having small errors in the previous steps. I believe it is necessary to include such artificial ablation studies to see why the method is better (I am not entirely convinced by the theory alone, see below).\n\n**Explanation of non-adversarial training** The fact that the method is non-adversarial may seem confusing to some readers (especially if they are familar with GAN or GAIL), and I do not believe the current writing is clear enough to show this. It may seem easier for readers to understand why the method is non-adversarial through a variational interpretation: the goal is to minimize a divergence (seems in your case, reverse KL); adversarial methods minimize a variational lower bound (and thus have to be minimax optimized), whereas your case minimizes a variational upper bound (like the ELBO for log-likelihood).\n\n**Quality**\n\nOther than the aforementioned issues (insufficient credit to imitation learning methods, explanation of non-adversarial, etc.), the paper is written well.\n\n**Minor issues**\n\nWhile Lemma 2 appears to show that by minimizing the discrepancy at the trajectory level, one obtains an expected quality difference of O(T) instead of O(T^2), the $\\epsilon$ value bounds a different quantity (global discrepancy for T steps in Lemma 2, as opposed to local discrepancy for 1 step in Lemma 1). Thus, I do not think Lemma 2 fully explains why trajectory-centric methods are better (since the assumption is also \"stronger\"). Also, Lemma 1 seems to be quite relevant to the DAgger paper, which I don't think you have cited. \n\nThe concept of expected quality difference is highly relevant to integral probability metrics. As Lemma 1 and 2 uses the max \"expected quality difference\", that is essentially an IPM; there is no need to invent another wheel.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "7dspyq3Oelv", "reviewer": "Reviewer_WGYR", "summary": "The paper proposes a method for sequential data generation that tries to avoid the pitfalls of prior MLE- and GAN-based objectives. The proposed approach starts with the global moment-matching objective (that mitigates the compounding errors). The true data distribution is modeled as an EBM and the parameters of the energy function are learned with contrastive learning (NCE). The trajectory distribution induced by a (learned) generation policy is used as the “noise” for NCE. In the inner loop, the generation policy is optimized to reduce the reverse-KL to the EBM’s trajectory distribution – which is equivalent to entropy-regularized RL with the energy function as the reward. Experiments using 5 tabular time-series datasets pits TimeCGI favorably in comparison to prior methods on the metrics of predictive and correlation scores.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I find the paper to be well-written and nicely organized. The authors address an important problem of sequential data generation over long-horizon, which has many real-world use cases. I would like the authors to consider the following issues:\n1.\tComparison to [63]: Equations 10-12 in section 3.2 bear a lot of resemblance to the work in [63]. Specifically, the idea of replacing the black-box GAN discriminator with a structured classifier with decoupled generator density and EBM for the true data was first proposed in [63], to the best of my knowledge. Eq. 10 in this paper (after taking into account the errata from the appendix) is the same as the third equation in section 3.1 of [63]. Eq. 11-12 of this paper appear in [63] as well. I appreciate the important differences between RL/IRL and the time-series generation setting considered in this paper. Yet, I believe that the phrasing in Section 3.2 gives the impression that the structured classification and the NCE training are the new contributions of this paper. Could the authors contrast their optimization objectives (Eq. 10-12) with those used in [63]?\n\n2.\tProposition 5: This statement in its current form does not appear to be true. As the appendix shows, the gradient equality only holds when p_\\phi is normalized, i.e., at the global optimum of the energy function. I would recommend the text in the main section be revised to add the proper caveats.\n\n3.\tInformation on datasets: For the datasets considered, could you provide some information regarding: \n\n     a.\tThe trajectory-length (horizon) of the true and the generated sequences  \n\n     b.\tThe dimension of the features space\n\n     c.\tThe stochasticity and/or noise characteristics of the true data generation process\n\n\n4.\tImproving Scalability: TimeCGI is closest to [63] in terms of learning the structured classifier over the entire sequence, instead of stepwise transitions. However, [64] argues that [63] is difficult to apply to complex problem settings since it operates at the level of the entire trajectory. It would be informative to expand on the potential challenges when scaling TimeCGI to long-horizon sequences with stochastic transitions. Could the variance of the gradient-estimator for the energy function be a problem? Thoughts on possible directions to improve scalability would be interesting. \n\n\n$$\\newline$$\nPossible typos?\n1.\tLine 119: should there be a negative sign with the likelihood-loss?\n2.\tLine 186: should it be product over \\pi, rather than sum?\n\n$$\\newline$$\n[63] A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models, Finn et al.\n\n[64] Learning robust rewards with adversarial inverse reinforcement learning, Fu et al.\n\n\n&nbsp;\n&nbsp;\n\n---\n\n**Post-rebuttal update:** Thank you to the authors for a detailed rebuttal. I maintain my position that the paper is (marginally) above the acceptance threshold. What would make it a clear accept is evaluation on datasets that highlight the robustness of the method to transition stochasticity (i.e. high \"branching factor\") and trajectory horizon (current datasets generate sequences of 20-30 steps).\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is a good discussion on these issues in Section 6. I am further interested to know the directions to improve the scalability of TimeCGI (point 4. in the main review).", "review_text": "", "extra_scores": {"ethical_concerns": "I don't see any ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After very detailed replies by the authors all reviewers of the paper have recommended acceptance.
Decision: Accept (Poster)