Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to perform time-series generation via a method inspired by imitation learning in a non-adversarial manner. The idea is to fit an energy-based model over the trajectories with noise contrastive estimation, with the noise being provided by a parametrized policy. The policy is also optimized to fit the energy-based model to provide a sound noise distribution. The method demonstrates superior empirical performance on various time-series datasets, on prediction error, and cross-correlation.
main_review: **Update**
I think the ablation studies address my concerns. More robustness of TImeGCI is shown, and may be a reason why we observe better performance (despite having the same "amount" of supervision). I am willing to raise the score because of this.

**Originality and significance**

While the paper discusses the relationship between time-series prediction and imitation learning, it seems that imitation learning methods are not given enough credit. For example, existing works in imitation learning [1,2] have already proposed methods to learn an energy-based model from the trajectories and then fir the policy to the energy function. 

[1] Kim, K., Jindal, A., Song, Y., Song, J., Sui, Y. and Ermon, S., 2020. Imitation with Neural Density Models. arXiv preprint arXiv:2010.09808.
[2] Arenz, O. and Neumann, G., 2020. Non-Adversarial Imitation Learning and its Connections to Adversarial Methods. arXiv preprint arXiv:2008.03525.

Given these prior works, the novelty of the proposed method would take a hit, as time-series prediction problems can be posed as an imitation learning problem with the "states" in the MDP being the "history" of the previous timesteps, and "action" in the MDP transitions to the next "state" concatenated with "history".

Even in the context of time series, the application of NCE is not entirely novel, such as https://www.cs.toronto.edu/~amnih/papers/ncelm.pdf

**Clarity**

**Explanation of empirical success** Another question that I have for the paper is why the method performs better and whether the success of "trajectory-level" imitation learning methods can be replicated here; one reason that GAIL-like methods outperform behavior cloning is that the RL algorithm can make use of external signals (e.g., "done") given by the environment. I have personally done experiments with GAIL and behavior cloning before and found it hard to outperform the latter without such information. The proposed method here does not appear to use any additional information than T-forcing or P-forcing, so the advantages must come from the loss function. However, there are no other experiments/ablation studies that show why the proposed method is better -- is it because of better first-step prediction, or is it because of better robustness when having small errors in the previous steps. I believe it is necessary to include such artificial ablation studies to see why the method is better (I am not entirely convinced by the theory alone, see below).

**Explanation of non-adversarial training** The fact that the method is non-adversarial may seem confusing to some readers (especially if they are familar with GAN or GAIL), and I do not believe the current writing is clear enough to show this. It may seem easier for readers to understand why the method is non-adversarial through a variational interpretation: the goal is to minimize a divergence (seems in your case, reverse KL); adversarial methods minimize a variational lower bound (and thus have to be minimax optimized), whereas your case minimizes a variational upper bound (like the ELBO for log-likelihood).

**Quality**

Other than the aforementioned issues (insufficient credit to imitation learning methods, explanation of non-adversarial, etc.), the paper is written well.

**Minor issues**

While Lemma 2 appears to show that by minimizing the discrepancy at the trajectory level, one obtains an expected quality difference of O(T) instead of O(T^2), the $\epsilon$ value bounds a different quantity (global discrepancy for T steps in Lemma 2, as opposed to local discrepancy for 1 step in Lemma 1). Thus, I do not think Lemma 2 fully explains why trajectory-centric methods are better (since the assumption is also "stronger"). Also, Lemma 1 seems to be quite relevant to the DAgger paper, which I don't think you have cited. 

The concept of expected quality difference is highly relevant to integral probability metrics. As Lemma 1 and 2 uses the max "expected quality difference", that is essentially an IPM; there is no need to invent another wheel.


limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes an imitation learning method that simultaneously learns a policy and an energy (“discriminator”) function. This is similar to a variational approach to learn a Gibbs distribution over trajectories, where the variational parameters consist of the policy and variational inference consists of running RL. 

However, rather than alternating the normal Gibbs energy loss and the variational inference loss of the variational approach, the submission replaces the Gibbs energy loss with an alternate energy loss based on noise-contrastive estimation, which explicitly trains the energy to discriminate between “noise” samples (drawn from the current policy) and expert samples. 

It is proved that under the noise-contrastive loss, the energy converges to the expert energy regardless of the noise distribution (under weak conditions), which justifies only partially optimizing the RL step in the alternating optimization. This results in a method potentially much faster than the “naive” variational approach, as it is not necessary to fully solve an inner-loop RL problem for each outer-loop iteration.

main_review: Strengths:
+ The method seems easier to train than comparable adversarial imitation learning algorithms
+ Like MaxEnt IRL, but sidesteps the difficulty of running full inference in the training loop
+ Theoretical results provide intuition for why this is a reasonable thing to do
+ Experimentally compared to a variety of other methods and datasets

Weaknesses:
- No experimental comparison to any adversarial imitation learning methods, which have similar properties
- No evaluation on imitation learning scenarios
- Presentation meanders a bit
- Experimental gains are a bit modest

Impact: 
Overall, the submission presents an interesting idea that may have the potential to make a significant impact on the community, with some caveats. Although the paper is framed as a generic time series prediction method, it does use terminology and ideas from the imitation learning (IL) literature, and I don’t see why it wouldn’t really be applicable to IL. So, I found it odd that the paper seemed to distance itself from IL.

Anyhow, viewed as an IL method, the proposed method seems appealing in that it would seem to combine the mitigation of error compounding that is a feature of adversarial IL with a more stable and/or faster optimization algorithm. I think it has generally been believed anecdotally that adversarial IL and the closely related MaxEnt IRL fix the error compounding problem present in naive behavioral cloning, and I have seem some theoretical results of this nature (see, e.g., chapter 14 of this book: https://rltheorybook.github.io/), but most of these methods tend to be impractical to actually use, either due to unstable optimization, or slowness due to having to run RL in the inner loop. 

This submission seems to propose a reasonable solution to those obstacles, with the caveat that it still requires running an RL step, which could be difficult to manage in practice. The main benefit of the method is that the method used to train the Gibbs energy (noise contrastive estimation) is relatively robust to inaccuracies in the “variational” distribution--i.e., the policy. This means that it is not strictly necessary to run the slow “inner loop” RL step to convergence for every outer loop iteration in order for the outer loop to converge. This should overall improve training stability relative to adversarial IL and should provide speed-ups relative to MaxEnt IRL because of the relaxation of the requirement to run RL to convergence. I believe this is the main takeaway of the paper, and the main reason why this work has the potential for impact.

The experimental results show a pretty significant improvement in the cross-correlation score, which is very encouraging. However, as mentioned elsewhere, trying the method in IL settings would probably be more interesting.

Rigor:
The submission scores some positive points in terms of theoretical rigor for including results that build intuition and support the main claims. The experiments also score points for rigor in that several datasets were tried, along with a variety of competing methods.

One negative aspect in this regard is the fact that the method was not compared to any adversarial imitation learning methods, or evaluated on imitation learning problems. Again, I found it a little perplexing that the work was framed as not being an IL method, despite it seeming very much like an IL method. Although I think this work has potential for impact, that potential is probably much reduced without comparisons to adversarial IL methods in IL settings. Also, the GAIL paper (https://arxiv.org/abs/1606.03476) should be cited, as that features some similar ideas, and is probably the most well-known adversarial IL method.

Another note about the experiments: the “predictive score” does not seem that intuitive or informative to me. For methods that produce a PDF for the policy, one could just compute the data likelihood (i.e., cross entropy) on a held-out dataset. 

I did not thoroughly check the theoretical results--in particular, I did not check the proofs of propositions 4 and 5. I skimmed the other results, and they seem reasonable. I did notice one minor issue with Lemma 2: its premise seems stronger than intended. Namely, the premise looks like $\max_f E_{x \sim p_s} f(x) - E_{x \sim p_t} f(x) \leq \epsilon$, where $f$ is unrestricted. However, if $f$ is unrestricted, then the previous expression must equal either $0$ or $\infty$--it can’t be equal to a finite value, because scaling $f$ always leads to a larger value, unless the value is $0$. So, I believe $f$ must be assumed bounded in order for the premise to make sense.

Also, section 3 casually mentions swapping the order of the min and max using the minimax theorem, but this is misleading, because “the” minimax theorem only applies for concave-convex games. This would therefore be inapplicable if the discriminator were parameterized as a nonlinear function.

Novelty:
As far as I know, the main idea of essentially applying NCE in MaxEnt IRL is novel--although one could argue that the general idea of using different generator or discriminator objectives in adversarial training is pretty common. In terms of the theoretical results, Lemma 1 seems like a variation of a well-known result that behavioral cloning suffers from a compounding error phenomenon--this should probably be made clearer. I’m not really sure about the novelty of Lemma 2, but there is a very similar result in chapter 14 of this book: https://rltheorybook.github.io/rltheorybook_AJKS.pdf (sample complexity of DM-ST) as well as a similar result for GAIL here: https://arxiv.org/abs/2010.11876. Proposition 3 seems like a relatively straightforward application of the existing result for NCE. I did not really examine propositions 4 and 5 closely, so I can’t comment as to their novelty.

Clarity of presentation:
The presentation could use significant work. In particular, although I found Lemma 1 and Lemma 2 to be useful to build intuition, overall, sections 2.2 and 2.3 seem like a diversion from the main idea. I also thought that equations 3 and 5 were introduced in a slightly suspicious way, by implying that this is some sort of generalization of MaxEnt duality when the features are represented as nonlinear functions. I don’t think that’s really the case. So, I think more care should be taken to motivate these equations.

Also, as previously mentioned, the min/max swap in section 3 is also a bit misleading. Altogether, this gives the impression of a sloppy line of reasoning leading up to the main result. If I were to rewrite the paper, I would omit sections 2.2 and 2.3, and motivate equation 6 directly as a variational training objective, instead of motivating it through somewhat suspect analogies. I would then only swap the min and max to motivate the analysis, which shows that under some conditions, optimizing this objective leads to moment-matching and non-compounding error.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method for sequential data generation that tries to avoid the pitfalls of prior MLE- and GAN-based objectives. The proposed approach starts with the global moment-matching objective (that mitigates the compounding errors). The true data distribution is modeled as an EBM and the parameters of the energy function are learned with contrastive learning (NCE). The trajectory distribution induced by a (learned) generation policy is used as the “noise” for NCE. In the inner loop, the generation policy is optimized to reduce the reverse-KL to the EBM’s trajectory distribution – which is equivalent to entropy-regularized RL with the energy function as the reward. Experiments using 5 tabular time-series datasets pits TimeCGI favorably in comparison to prior methods on the metrics of predictive and correlation scores.
main_review: Overall, I find the paper to be well-written and nicely organized. The authors address an important problem of sequential data generation over long-horizon, which has many real-world use cases. I would like the authors to consider the following issues:
1.	Comparison to [63]: Equations 10-12 in section 3.2 bear a lot of resemblance to the work in [63]. Specifically, the idea of replacing the black-box GAN discriminator with a structured classifier with decoupled generator density and EBM for the true data was first proposed in [63], to the best of my knowledge. Eq. 10 in this paper (after taking into account the errata from the appendix) is the same as the third equation in section 3.1 of [63]. Eq. 11-12 of this paper appear in [63] as well. I appreciate the important differences between RL/IRL and the time-series generation setting considered in this paper. Yet, I believe that the phrasing in Section 3.2 gives the impression that the structured classification and the NCE training are the new contributions of this paper. Could the authors contrast their optimization objectives (Eq. 10-12) with those used in [63]?

2.	Proposition 5: This statement in its current form does not appear to be true. As the appendix shows, the gradient equality only holds when p_\phi is normalized, i.e., at the global optimum of the energy function. I would recommend the text in the main section be revised to add the proper caveats.

3.	Information on datasets: For the datasets considered, could you provide some information regarding: 

     a.	The trajectory-length (horizon) of the true and the generated sequences  

     b.	The dimension of the features space

     c.	The stochasticity and/or noise characteristics of the true data generation process


4.	Improving Scalability: TimeCGI is closest to [63] in terms of learning the structured classifier over the entire sequence, instead of stepwise transitions. However, [64] argues that [63] is difficult to apply to complex problem settings since it operates at the level of the entire trajectory. It would be informative to expand on the potential challenges when scaling TimeCGI to long-horizon sequences with stochastic transitions. Could the variance of the gradient-estimator for the energy function be a problem? Thoughts on possible directions to improve scalability would be interesting. 


$$\newline$$
Possible typos?
1.	Line 119: should there be a negative sign with the likelihood-loss?
2.	Line 186: should it be product over \pi, rather than sum?

$$\newline$$
[63] A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models, Finn et al.

[64] Learning robust rewards with adversarial inverse reinforcement learning, Fu et al.


&nbsp;
&nbsp;

---

**Post-rebuttal update:** Thank you to the authors for a detailed rebuttal. I maintain my position that the paper is (marginally) above the acceptance threshold. What would make it a clear accept is evaluation on datasets that highlight the robustness of the method to transition stochasticity (i.e. high "branching factor") and trajectory horizon (current datasets generate sequences of 20-30 steps).





limitations_and_societal_impact: There is a good discussion on these issues in Section 6. I am further interested to know the directions to improve the scalability of TimeCGI (point 4. in the main review).
review_title: null

Meta-review: After very detailed replies by the authors all reviewers of the paper have recommended acceptance.
Decision: Accept (Poster)