Title: LEARNING GIBBS-REGULARIZED GANS WITH VARIA-TIONAL DISCRIMINATOR REPARAMETERIZATION

Abstract: We propose a novel approach to regularizing generative adversarial networks (GANs) leveraging learned structured Gibbs distributions. Our method consists of reparameterizing the discriminator to be an explicit function of two densities: the generator PDF q and a structured Gibbs distribution ν. Leveraging recent work on invertible pushforward density estimators, this reparameterization is made possible by assuming the generator is invertible, which enables the analytic evaluation of the generator PDF q. We further propose optimizing the Jeffrey divergence, which balances mode coverage with sample quality. The combination of this loss and reparameterization allows us to effectively regularize the generator by imposing structure from domain knowledge on ν, as in classical graphical models. Applying our method to a vehicle trajectory forecasting task, we observe that we are able to obtain quantitatively superior mode coverage as well as better-quality samples compared to traditional methods.

1. INTRODUCTION:

Although recent progress in GANs and variational methods have significantly advanced the capabilities of generative models for high-dimensional data, many issues still limit the practical application of such methods. In practice, GANs typically suffer from mode loss, whereas VAEs suffer from poor sample quality compared to GANs [40, 14, 3] . Several factors may be identified as contributing to these issues: first, the training loss may optimize for sample quality at the expense of mode coverage or vice-versa; second, the variance of stochastic gradient estimates may be too high to admit efficient stochastic optimization; finally, the models may not admit a good regularization scheme via the imposition of appropriate inductive biases.

We advocate a novel approach to address these concerns, with a particular focus on the latter issue of regularization. Let p : R N → R + denote the PDF of a continuous data distribution and q : R N → R + denote the PDF of a learned model. Following recent work [31] , we advocate training a generative model to minimize the Jeffrey (symmetric KL) divergence min q KL(p, q) + KL(q, p), representing q in a way that enables it to be efficiently evaluated at any point (i.e., by representing q as a pushforward distribution induced by an invertible warp [29, 8, 18] ). In this work, we propose the key innovation of applying Fenchel-duality-based variational inference to KL(q, p), which allows the latter to be optimized without having to explicitly evaluate p. This yields the following variational approximation of the Jeffrey divergence, which constitutes the training loss for our method:

EQUATION

where the variational parameters consist of the function ν. We now assert and later elaborate the following properties of the Jeffrey divergence and its variational approximation (1): first, KL(p, q) essentially prevents mode loss, whereas KL(q, p) prevents the generation of any samples unlikely under the data [6, 14] ; second, an exactly unbiased stochastic gradient of KL(p, q) may be obtained without variational inference; finally, the optimal value of ν is p. The second key innovation of our work is to recognize that-in contrast to traditional GANs, which admit no comparable regularization principle-this model may be effectively regularized by imposing any domain-specific structure possessed by p on ν, which we thence interpret as a structured Gibbs distribution.

These concepts are illustrated in Fig. 1 . Figure 1a shows the result of training a model q to optimize KL(p, q), where q is represented as the pushforward of a Gaussian base distribution under an . Middle shows samples from current q (red), and true future path (cyan). Note that ν has learned to penalize regions with obstacles (V (x) = log ν(x), i.e. higher V (x) is lower traversal cost). Right: after incorporating the learned ν, q and its samples are shifted to avoid high-cost regions, corresponding to suppression of spurious modes.

autoregressive warp in two dimensions [29, 8, 18] . We observe that the generated samples effectively cover all the data modes, since a huge penalty is incurred if q(x) is low for any data point x ∼ p; however, the generator additionally places mass outside the support of the data distribution p, because shifting half of q's mass onto the support of p decreases KL(p, q) by no more than log 2; therefore, this objective effectively under-constrains q. We can resolve the ambiguity by adding the variationallyapproximated KL(q, p) term, which trains log ν to approximate log p while penalizing the generation of samples where log ν is low, as shown in Fig. 1b-d . By choosing different forms for log ν, we see that the ambiguity can be resolved in different ways: in Fig. 1b , log ν is chosen to have contours roughly matching the support of p, whereas Fig. 1c-d show the cases where log ν is represented as quadratic and as the output of a multi-layer perceptron, respectively. We see that matching the shape of log ν to the shape of p produces qualitatively good samples, whereas putting too little constraint on the shape of log ν (as in Fig. 1d ) yields qualitatively worse results. Estimated cross-entropy values are also shown for each model, which again demonstrates how KL(p, q) is largely insensitive to the overall shape of the model q, as long as q covers the modes of p.

Our method may also be viewed as incorporating a kind of f-GAN [26] optimizing a particular loss (Jeffrey divergence), using a (invertible) generator that admits exact PDF evaluation, and reparameterizing the discriminator in a certain way. The Fenchel-variational view espoused in [26] reveals the optimal discriminator T to be a function of the odds ratio: T * = h(q/p) for some function h determined by the particular f -divergence chosen. We observe that if q can be evaluated analytically, then we might as well represent the discriminator in terms of q and some function ν, where ν directly approximates p. This change of representation allows us to take any known regularities of p and impose them on ν; e.g., if p is known to be translation-invariant, then we can safely impose translational invariance on ν without imposing any undue constraints on our model. Note that we cannot similarly impose p's structure on T , which may be a complex function even if p and q are simple: for example, even if p and q are both bounded above, T may be unbounded, since it is a function of the ratio of the two.

Fig. 2 illustrates how ν is structured for our application domain of vehicle trajectory forecasting: ν is represented as a sum of learned spatial rewards, which penalizes trajectories according to a learned function over spatial positions. Intuitively, this prevents trajectories from colliding with obstacles, while simultaneously learning the concept of an obstacle.

2. PUSHFORWARD REPRESENTATION AND IMPLEMENTATION:

Optimizing Eq. ( 1) is straightforward except for one subtle point: we must be able to evaluate q pointwise and differentiate the expression E x∼q log(q(x)/ν(x)) with respect to the parameters of q.

Inspired by prior work [29, 8, 18] , we solve both these problems by representing q as the pushforward of a simple distribution under an invertible warp (also known as a normalizing flow). Suppose µ is a distribution over a set Z and g : Z → X is a function (the generator) with domain Z. Then we can define a measure on X as the distribution of g(z) sampling z from µ-this distribution, denoted here by g| µ , is referred to as the pushforward of µ under g. Now suppose g is parameterized by θ and differentiable in x and θ. By representing q as q θ = g θ | µ , for some simple distribution µ, we can move the derivative wrt. θ inside the expectation by exploiting the property

EQUATION

This is well-known as the reparameterization trick; it allows us to obtain a low-variance, unbiased estimate of the parameter derivatives for learning with SGD. However, one problem remains: evaluating q θ (x) = (g θ | µ )(x), which appears in both the first and last terms of (1). This is solved by assuming that g θ is invertible: ẑ := g -1 θ (x). Thus, we have an analytic formula for the pushforward density:

EQUATION

where |(dg θ ) z | represents the determinant of the Jacobian of g θ evaluated at the point z. This finally allows us to rewrite Eq. ( 1) in the following explicit form, after performing some simplifications:

-max

EQUATION

A pseudocode summary of our method is given in Algorithm 1. X and Z denote batches of observed and latent samples, respectively, while subscripts D and G denote either data or generated samples.

A few implementation issues are noted here. Applying the method to a particular problem requires the implementation of the invertible generator G(•; θ), the structured Gibbs energy log ν(•; φ), and the base distribution µ. In order to avoid numerical issues when q is not absolutely continuous wrt. ν (i.e., when ν is 0 but q is not), we reparameterize ν as ν ← αq + ν, where α is a small number. Given this assumption, the quantity q(x)/(αq(x) + ν(x)) can be rewritten in terms of the sigmoid σ.

Optimization proceeds by alternating between minimizing the loss (1) in the generator parameters θ and maximizing it in the energy parameters φ. However, as noted in Alg. 1, we may alternatively minimize a different objective for the generator: namely, -E x∼p log q(x) + E x∼q log(q(x)/ν(x)).

The rationale for the alternative objective is that, as noted in Sec. 3.1, the inner minimization may be viewed as fitting ν to p-in which case, we may approximate KL(q, p) as E x∼q log(q(x)/ν(x)). The alternate generator objective was used for the toy experiment in Fig. 1 , whereas the original objective was used for the trajectory forecasting experiments.

3. DERIVATIONS AND INTERPRETATIONS:

Equation ( 1) can be derived via a variational lower bound derived from Fenchel conjugacy, using a technique similar to [25, 26] . Our approach of pairing this convex conjugate with the pushforward Algorithm 1 Pseudocode for C3PO implementation Require: X D : a batch of training data, Z G : batch of generator noise samples from µ

1: Z D , det dg -1 X D ← G -1 (X D ; θ) G -1 (x) returns g -1 θ (x) and log det. of Jacobian of g -1 θ at x 2: X G , det dg Z G ← G(Z G ; θ)

G(z) returns g θ (z) and log det. of Jacobian of g θ at z

3: log q(X D ) ← log µ(Z D ) + log | det dg -1 X D | Generator PDF at data samples 4: log q(X G ) ← log µ(Z G ) + log | det dg Z G |

Generator PDF at generator samples 5: log q/ν(X D ) ← log α + log q(X D ) -log ν(X D ; φ)

6: log q/ν(X G ) ← log α + log q(X G ) -log ν(X G ; φ) 7: L ← BatchMean(-log q(X D ) -α -1 σ(log q/ν(X D )) + log σ(log q/ν(X G )) -log α) 8: for i ← 1 . . . N do 9: θ ← θ -β∇ θ L

Alternative generator loss: L := -log q(X D ) + log σ(log q/ν(X G ))

10:

for j ← 1 . . . M do 11:

φ ← φ + β∇ φ L

direct density estimation motivates our method's name: Convex Conjugate Coupled Pushforward Optimization (C3PO). Observe that the Jeffrey divergence can be written as min q -E x∼p log q(x) + E x∼q log q(x) -E x∼q log p(x). Since q can be sampled, evaluated, and differentiated (Sec. 2), but p can only be sampled, our goal in this section is to convert -E x∼q log p(x) to something expressable in terms of expectations wrt. p and q. This is achieved by applying the Fenchel-Young inequality to the function f (p) := -log p, which yields -log p ≥ sup λ<0 λp -(-1 -log(-λ)). Substituting this inequality in place of -log p(x) yields the following lower bound of -E x∼q log p(x):

EQUATION

where the sup on the right-hand side is taken over all functions λ : X → R -mapping the domain to a negative scalar. Observing that q(x)λ(x)p(x) dx can be written either as E x∼q λ(x)p(x) or E x∼p λ(x)q(x), and making the substitution ν = -1/λ, we have the equivalent bound

EQUATION

Substituting this the Jeffrey divergence yields Eq. (1).

3.1. INTERPRETATION AS LEARNING A GIBBS DISTRIBUTION:

Although we have so far viewed our goal as primarily learning q, we now observe that our method can also be interpreted primarily as a way of learning ν as a Gibbs distribution approximating p. Learning a Gibbs distribution, or a more general energy-based model [19] , is a very general and effective way to impose strong domain-specific regularization on probability distributions over high-dimensional data. Generally, this is achieved by structuring the energy function V φ := log ν φ to assign similar energies to similar examples; for example, a convolutional neural network might constitute a good energy for image generation, since the structure of a CNN encodes some degree of translational invariance.

Unfortunately, inference and learning with high-dimensional Gibbs distributions is difficult.

Our method can be viewed as a way to train a Gibbs distribution that circumvents some of these difficulties. In this view, the inner optimization in Eq. ( 1) is interpreted as minimizing a weighted divergence between p and ν; the weights are exactly q. Specifically, we observe that the inner optimization in Eq. ( 1) minimizes the following weighted Itakura-Saito divergence [16, 9] :

EQUATION

It would seem reasonable to choose the weights q as q = p; since p cannot be evaluated directly, alternating optimization is a sensible alternative. Minimizing Eq. ( 7) intuitively gives us the best Gibbs approximation of p over the support of q.

Intuition for the learning rule of ν can be obtained by computing the functional gradient of Eq. ( 1) with respect to ν (i.e., differentiating with respect to ν(x), ∀x. The functional gradient δ /δν expresses The input BEV map (roads in black), accompanied by 100 demonstrations of possible behavior (in blue). Each method is visualized with 60 of each samples. For the methods that learn a cost map, the learned cost map is blended with the input BEV feature map. the direction in which ν should be moved at each point x in order to optimally decrease the objective.

Observing that E x∼p q(x)/ν(x) = E x∼q p(x)/ν(x), we obtain the following for the functional gradient of the objective wrt. log ν: δC /δ log ν(x) = q(x) p(x) /ν(x) -1 . We therefore see that minimizing Eq. ( 1) in ν raises or lowers log ν at each point x according to whether it exceeds p(x), with a learning rate given by q(x), and a unique fixed point (assuming q > 0) of p = ν.

4. EXPERIMENTS:

We conducted experiments on four datasets of vehicle trajectories, where the objective is to forecast a distribution over the vehicle's future locations x ∈ R 20×2 given contextual information about each scene. The contextual information includes 2 seconds of the vehicle's previous position, as well as a Bird's Eye View (BEV) map of visual scene features. Our experiments are designed to quantify two key aspects of generative modeling: the learned model's likelihood of held-out test data (i.e. the negative forward cross-entropy -H(p, q)), and the quality of samples from the learned model (i.e. the negative reverse cross-entropy -H(q, p)). Our hypotheses are: 1) C3PO will achieve superior sample-quality performance to other methods 2) C3PO will learn a high-quality q, partially due to its ability to perform direct density evaluation and optimization of q 3) C3PO will learn an interpretable ν that penalizes bad samples.

Two of the four datasets are synthetic (BEVWORLD 1 and BEVWORLD1K), with known roads, enabling us to construct samples from a reasonable p distribution, approximate p via KDE, and measure the reverse cross-entropy H(q, p KDE ). We also calculate the percentage of trajectory points on the road as another measure of sample quality. Details of how we generate BEVWORLD are in the supplement. We also experiment with two real-world datasets: the KITTI dataset, and the CALIFORECASTING dataset [32] .

4.1. IMPLEMENTATION AND BASELINES:

Given our setting is that of forecasting future vehicle trajectories, we leverage ideas from Inverse Reinforcement Learning [24, 43, 41] , to structure our Gibbs energy V as a spatial cost map: where 

log ν φ = V φ (x) = T t=1 R φ (x t0 ,

x t1 ; BEV), and R φ (a, b; BEV) is the output of a CNN that can be interpolated at 2d positions of the form (a, b). This structure enables ν to penalize trajectories that travel to locations it perceives to be bad, e.g. locations with obstacles, or locations far from a perceived road.

We compare our method, C3PO, to several state-of-the-art approaches in imitation learning and generative modeling: Generative Adversarial Imitation Learning (GAIL) [12] , f-GAN [26] , the CVAE method of DESIRE [20] , and R2P2 [32] . In each baseline, we use architectures as similar to our own method as possible: the policy (generator) architecture of GAIL and the generator architecture of f-GAN are identical to the generator architecture of our own approach. The same architecture used for V φ in our method was also used for the discriminators in all baselines.

Our implementation of the q architecture is based on R2P2 [31] . One key difference between our method, C3PO, and R2P2, is that R2P2 does not have an adversarial component; its main focus is learning the forward KL term, the first component of C3PO's objective function. R2P2 starts from a similar objective: the symmetric sum of cross-entropies. However, it relies on a cruder approximation of H(q, p), because it learns an approximating p for H(q, p) offline, with no interaction from q.

4.2. SYNTHETIC EXPERIMENTS:

BEVWORLD 1contains a single scene with 300 samples from p. This setting is unconditional: the contextual information provided is identical, and not directly useful for modeling p. This setting also provides a fairer comparison to fGAN methods, which left the extension of fGAN to contextual settings as future work [26] . Table 1 shows the results of BEVWORLD 1 experiments. We observe that R2P2 and C3PO achieve the best -H(p, q) scores, with C3PO outperforming all nondegenerate methods in its sample quality. This evidence supports hypotheses 1) and 2): C3PO achieves superior sample quality and high-quality data density. We also observed C3PO to be very stable throughout training in terms of H(p, q): it was as easy to train it as R2P2 in all experiments.

We also observe that while fGAN KL indirectly learns a q with some support for p, its samples are quite poor. This matches expectations, as the forward KL divergence fails to impose much penalty on sample quality [14] . fGAN Jeffrey is the fGAN method most similar to our approach because it uses the Jeffrey divergence. We observe it to suffer mode collapse in all of our experiments, a similar result to [26] , in which fGAN Jeffrey had the worse test-set likelihood.

Next, we consider BEVWORLD1K, in which 100 training scenes, each with 100 samples from p, are used to learn conditional generative models. There are 1000 scenes in the test set. Table 1 shows the result of these experiments. We observe results similar to BEVWORLD1K, again supporting hypotheses 1) and 2). C3PO can learn in the conditional setting to produce high-quality samples from a distribution with good support of p. We display example results in Fig. 3 , and observe several methods, including C3PO , learn a cost-map representation that penalizes samples not on the road. This evidence of the learned intuitive perceptual measurement of ν supports hypothesis 3). (q) criterion. Each row corresponds to a method, and each column corresponds to the method's result on the item at a specific level of performance, from worst (left) to best (right) of results on 100 scenes. Each image is composed of the learned V KITTI φ (q) blended with the input BEV features, sample trajectories from each method (red), and the true future (blue). The learned V often penalizes samples that go off of the road or into obstacles inferred from the features. C3PO usually produces the best samples under this metric. Right: Evaluation of sample quality on test data from the KITTI dataset (a) and CALIFORECASTING dataset (b). Each approach generated 12 trajectory samples per scene, and the mean V φ score was calculated for each scene. The results are displayed as a normalized cumulative histogram, where the y-value at x is the percentage of scenes that received V ≤ x. At almost every given V , C3PO is likelier to have more samples above the given V than other methods. (q) blended with the input BEV features, 12 sample trajectories from each method (red), and the true future (blue). V learns to penalize samples that go off of the road or into obstacles.

4.3. REAL-WORLD EXPERIMENTS:

We now experiment with real-world data, in which each method is provided with noisy contextual information, and is evaluated by its ability to produce a generative model and high-quality samples. Evaluation of sample quality is more difficult in this scenario: we have only one sample of p in each scene, precluding construction of p KDE , and there are no labels of physical roads, precluding computation of on-road statistics. Fortunately, evaluation of -H(p, q) is still possible.

After observing the learned V φ of C3PO on both synthetic and real data, we found V φ to be generally interpretable and intuitively good: it assigns low cost to roads it learns to perceive and it assigns high cost to obstacles it learns to perceive. We therefore employed our learned model V φ to quantitatively evaluate samples from all methods. Fig. 4 illustrates results from several approaches visualized with V and point cloud features from the BEV, at various quantiles of each method's performance under V . Fig. 5 (left) illustrates each method on a set of randomly sampled scenes. In both figures, V perceives and assigns penalty to regions around obstacles in the point cloud data. Note that this energy is learned implicitly, because demonstrations from p avoid obstacles. These results provide further support for hypothesis 3).

Quantitative results are shown in Table 2 . We find that results are, overall, similar to results on the other datasets. C3PO produces a high-performing q in terms of both its likelihood, -H(p, q), as well as the quality of its samples, V . Additionally, we show evaluation of V for the top methods in Fig. 4 (right). Together, these results further strengthen evidence in support of all of our hypotheses.

5. RELATED WORK:

The initial motivation for our work was the apparent dichotomy between sample quality and mode coverage in existing deep generative models; this phenomenon has been noted and quantified in work such as [40, 23, 4, 13, 14] . Our work synthesizes several techniques from prior work to address these issues, including the Fenchel-variational principle from work such as [25, 26] , the well-known reparameterization trick [17, 35] , and analytic pushforward-based density estimators such as normalizing flows [29, 18] , RealNVP [8] , and related models [22, 10, 27, 38, 30] . Comparatively little work has explored combining these methods, with some exceptions. Combining a RealNVP [8] density estimator with a GAN objective was considered in [11] ; however, this is susceptible to the problems inherent with GANs mentioned in the introduction. A pushforward-based density estimator was employed in conjunction with variational inference to optimize the classical Bayesian evidence lower bound in [18] , but this cross-entropy objective suffers from the previously-mentioned problems with optimizing KL(p q) alone, as do all methods based on this objective, including [8, 10, 27, 38] .

Our work is also comparable to the extensive literature on learning deep graphical models, including variants of Boltzmann machines [1, 34] and various proposals for learning deep CRFs and structured energy-based models [19] based on inference techniques including convex optimization [28, 7, 2, 36] , various forms of unrolled inference [33, 42, 39] , pseudolikelihood [21] , and continuous relaxation [5] , among other techniques [13] . Viewed as a way to learn a deep graphical model, the most important distinguishing factor of our work is the fact that our method does not need to perform inference directly, as discussed in Sec. (3.1). To the extent that q is viewed as (indirectly) performing inference, our method and other deep generative models bears some similarity to the wake-sleep algorithm, which also alternates between optimizing a model distribution and an "inference" distribution using complementary divergences [13] . Score-matching [15] also learns a Gibbs distribution without inference; however, unlike our method, score-matching does not learn an inference distribution.

6. CONCLUSION:

We have demonstrated how GANs may be effectively regularized by reparameterizing the discriminator to be a function of the model density and a structured Gibbs density, showing how this may be achieved by optimizing a Jeffrey divergence loss, assuming the generator is invertible, and applying a variational bound based on Fenchel conjugacy. Applied to a trajectory forecasting problem, we observed superior mode coverage and qualitative results compared to traditional GANs. We hope to soon demonstrate the general applicability of our approach by applying it to the task of image generation as well.

:



Figures and Tables:

Figure 1: Different trained models for a 2D toy problem. The data distribution (samples shown as green dots) consists of a mixture of isotropic Gaussian distributions, arranged in a square grid. Four different trained models (generator samples shown as red dots) are shown: (a) result of training a model to optimize KL(p,q), (b-d) result of training models with our method (fine-tuning result of (a)), varying the form of the structured Gibbs distribution ν. Blue lines show contours of learned structured Gibbs distributions ν. In (b-c), the learned parameters of ν are µ 0 , σ 0 , and β .

Figure2: Our method applied to forecast ego-vehicle trajectories, showing input image and overhead views with the learned cost -log(ν) (overlaid on LIDAR map). Middle shows samples from current q (red), and true future path (cyan). Note that ν has learned to penalize regions with obstacles (V (x) = log ν(x), i.e. higher V (x) is lower traversal cost). Right: after incorporating the learned ν, q and its samples are shifted to avoid high-cost regions, corresponding to suppression of spurious modes.

Figure 3: Comparison of methods on a test scene from the BEVWORLD1K dataset. Left column:The input BEV map (roads in black), accompanied by 100 demonstrations of possible behavior (in blue). Each method is visualized with 60 of each samples. For the methods that learn a cost map, the learned cost map is blended with the input BEV feature map.

Figure 4: Left: Comparison of methods under the learned V KITTIφ

Figure 5: Comparison of methods on a set of random scenes from the CALIFORECASTING test set. Scene indices were selected uniformly at random (once) from the possible test indices. Each image shows the learned V CALIF

φ

Comparison of methods in two datasets. Left: Single BEVWorld Scene (identical train and test), 300 experts, 1800 policy samples. Right: BEVWorld 100 training scenes, 1000 test scenes, 100 experts/scene, 12 policy samples/scene. Means and their standard errors are reported. Bold indicates the best performing method among methods with nondegenerate -H(p, q) (i.e. fGAN Jeffrey is degenerate). CVAE's -H(p, q) is estimated via MC sampling 300 times per scene, see[37].

Table content: <table><tr><td/><td/><td>BEVWORLD 1</td><td/><td/><td>BEVWORLD1K</td></tr><tr><td>Method</td><td>-H(p, q)</td><td colspan="2">-H(q, pKDE) Road %</td><td>-H(p, q)</td><td>-H(q, pKDE) Road %</td></tr><tr><td>R2P2</td><td colspan="3">83.7 ± 0.2 -46.6 ± 0.5 0.988</td><td colspan="2">96.0 ± 0.03 -61.6 ± 0.6 0.922</td></tr><tr><td>C3PO (ours)</td><td>82.6 ± 0.3</td><td colspan="2">-44.8 ± 0.4 1.000</td><td>92.7 ± 0.1</td><td>-48.7 ± 0.3 0.989</td></tr><tr><td>R2P2 GAIL</td><td>64.6 ± 2.0</td><td colspan="2">-55.6 ± 1.0 0.952</td><td>62.0 ± 0.6</td><td>-74.6 ± 0.8 0.886</td></tr><tr><td>CVAE  *</td><td>18.6 ± 3.7</td><td colspan="2">-45.3 ± 1.7 0.990</td><td>12.5 ± 0.3</td><td>-71.3 ± 0.8 0.865</td></tr><tr><td>fGAN KL</td><td>16.6 ± 0.6</td><td colspan="2">-294.4 ± 22 0.568</td><td colspan="2">18.6 ± 0.03 -303 ± 2.6</td><td>0.698</td></tr><tr><td colspan="3">fGAN Reverse KL -17.3 ± 2.9 -171 ± 8.5</td><td>0.706</td><td colspan="2">-11.6 ± 0.2 -235 ± 1.8</td><td>0.709</td></tr><tr><td>fGAN Jeffrey</td><td colspan="3">-7e4 ± 6e3 -42.0 ± 0.05 1.000</td><td colspan="2">-5e3 ± 39 -46.1 ± 0.08 0.970</td></tr></table>

Comparison of methods in two real-world datasets: KITTI and CALIFORECASTING. Means and their standard errors are reported. Bold indicates the best performing method among methods with nondegenerate -H(p, q) (i.e. fGAN Jeffrey is degenerate).

Table content: <table><tr><td/><td colspan="2">KITTI</td><td/><td colspan="3">CALIFORECASTING</td></tr><tr><td>Method</td><td>-H(p, q)</td><td>V KITTI φ</td><td>(q)</td><td>-H(p, q)</td><td>V CALIF φ</td><td>(q)</td></tr><tr><td>R2P2</td><td colspan="3">63.7 ± 0.8 -744 ± 20</td><td colspan="3">74.1 ± 0.38 -6.50 ± 3.9</td></tr><tr><td>C3PO (ours)</td><td>61.5 ± 0.7</td><td colspan="2">-457 ± 13</td><td>73.5 ± 0.4</td><td colspan="2">57.3 ± 1.4</td></tr><tr><td>R2P2 GAIL</td><td>54.9 ± 0.7</td><td colspan="2">-693 ± 17</td><td>46.9 ± 0.3</td><td colspan="2">-61.1 ± 6.1</td></tr><tr><td>CVAE  *</td><td>9.22 ± 0.9</td><td colspan="2">-555 ± 9.9</td><td>10.1 ± 0.9</td><td colspan="2">48.3 ± 1.5</td></tr><tr><td>fGAN KL</td><td>32.9 ± 1.3</td><td colspan="2">-693 ± 10</td><td colspan="3">9.55 ± 0.02 -568 ± 20</td></tr><tr><td colspan="4">fGAN Reverse KL 12.8 ± 0.08 -1362 ± 33</td><td colspan="3">-89.7 ± 3.1 21.8 ± 2.6</td></tr><tr><td>fGAN Jeffrey</td><td colspan="3">-2e4 ± 2e3 -195 ± 4.0</td><td colspan="3">-2e4 ± 7e2 69.5 ± 0.1</td></tr></table>

Peer review (iclr2019):
 The paper proposes to reparameterize the discriminator to be an explicit function of two densities so that one could inject domain specific knowledge easily. As the authors say, that one way to inject domain specific information is by learning an energy function. Making use of this intuition, authors proposed to regularize the discriminator in (GANs) framework by leveraging structured Gibbs distributions. I found the introduction a bit hard to read. Otherwise paper is written in a readable way. Something which I like about this paper, is authors use the proposed method for actual RL problems as compared to just image generation. I think this is important as well as interesting. As a community we should be moving towards evaluating generative models for the problems where we actually want to use generative models for. Some questions: - I*m not sure if the paper is really novel as the authors themselves point out that it corresponds to adding adversarial component in R2P2. - I also did not find results very convincing. As I said, its important to evaluate on RL problems ONLY if it makes sense on toy problems first. Like in the paper, authors made a big claim about reducing spurious modes, but it has not been demonstrated any where per se. May be authors can construct a toy problem in which they can show that the spurious mode issue, and how the proposed method kills these spurious modes. This also reminds me of the literature in Boltzmann machines and more recently in Variational Walkback [1]. This could also be cited, and could be interesting to authors. [1] Variational Walkback, https://arxiv.org/abs/1711.02282. The authors in Variational walkback also make the assumption p == q. What would make the paper stronger ? - Constructing toy problems in order to illustrate the mode coverage and spurious modes issue would be interesting.