Year: 2018

Abstract:
We propose a novel approach to regularizing generative adversarial networks (GANs) leveraging learned structured Gibbs distributions. Our method consists of reparameterizing the discriminator to be an explicit function of two densities: the generator PDF q and a structured Gibbs distribution ν. Leveraging recent work on invertible pushforward density estimators, this reparameterization is made possible by assuming the generator is invertible, which enables the analytic evaluation of the generator PDF q. We further propose optimizing the Jeffrey divergence, which balances mode coverage with sample quality. The combination of this loss and reparameterization allows us to effectively regularize the generator by imposing structure from domain knowledge on ν, as in classical graphical models. Applying our method to a vehicle trajectory forecasting task, we observe that we are able to obtain quantitatively superior mode coverage as well as better-quality samples compared to traditional methods.

heading: 1 INTRODUCTION
text: Although recent progress in GANs and variational methods have significantly advanced the capabilities of generative models for high-dimensional data, many issues still limit the practical application of such methods. In practice, GANs typically suffer from mode loss, whereas VAEs suffer from poor sample quality compared to GANs [40, 14, 3]. Several factors may be identified as contributing to these issues: first, the training loss may optimize for sample quality at the expense of mode coverage or vice-versa; second, the variance of stochastic gradient estimates may be too high to admit efficient stochastic optimization; finally, the models may not admit a good regularization scheme via the imposition of appropriate inductive biases.
We advocate a novel approach to address these concerns, with a particular focus on the latter issue of regularization. Let p : RN → R+ denote the PDF of a continuous data distribution and q : RN → R+ denote the PDF of a learned model. Following recent work [31], we advocate training a generative model to minimize the Jeffrey (symmetric KL) divergence minq KL(p, q) + KL(q, p), representing q in a way that enables it to be efficiently evaluated at any point (i.e., by representing q as a pushforward distribution induced by an invertible warp [29, 8, 18]). In this work, we propose the key innovation of applying Fenchel-duality-based variational inference to KL(q, p), which allows the latter to be optimized without having to explicitly evaluate p. This yields the following variational approximation of the Jeffrey divergence, which constitutes the training loss for our method:
min q∈Q
Ex̂∼p log p(x̂)
q(x̂) + sup ν>0 −Ex̂∼p
q(x̂) ν(x̂) + Ex∼q log q(x) ν(x) , (1)
where the variational parameters consist of the function ν. We now assert and later elaborate the following properties of the Jeffrey divergence and its variational approximation (1): first, KL(p, q) essentially prevents mode loss, whereas KL(q, p) prevents the generation of any samples unlikely under the data [6, 14]; second, an exactly unbiased stochastic gradient of KL(p, q) may be obtained without variational inference; finally, the optimal value of ν is p. The second key innovation of our work is to recognize that—in contrast to traditional GANs, which admit no comparable regularization principle—this model may be effectively regularized by imposing any domain-specific structure possessed by p on ν, which we thence interpret as a structured Gibbs distribution.
These concepts are illustrated in Fig. 1. Figure 1a shows the result of training a model q to optimize KL(p, q), where q is represented as the pushforward of a Gaussian base distribution under an
.
autoregressive warp in two dimensions [29, 8, 18]. We observe that the generated samples effectively cover all the data modes, since a huge penalty is incurred if q(x) is low for any data point x ∼ p; however, the generator additionally places mass outside the support of the data distribution p, because shifting half of q’s mass onto the support of p decreases KL(p, q) by no more than log 2; therefore, this objective effectively under-constrains q. We can resolve the ambiguity by adding the variationallyapproximated KL(q, p) term, which trains log ν to approximate log p while penalizing the generation of samples where log ν is low, as shown in Fig. 1b-d. By choosing different forms for log ν, we see that the ambiguity can be resolved in different ways: in Fig. 1b, log ν is chosen to have contours roughly matching the support of p, whereas Fig. 1c-d show the cases where log ν is represented as quadratic and as the output of a multi-layer perceptron, respectively. We see that matching the shape of log ν to the shape of p produces qualitatively good samples, whereas putting too little constraint on the shape of log ν (as in Fig. 1d) yields qualitatively worse results. Estimated cross-entropy values are also shown for each model, which again demonstrates how KL(p, q) is largely insensitive to the overall shape of the model q, as long as q covers the modes of p.
Our method may also be viewed as incorporating a kind of f-GAN [26] optimizing a particular loss (Jeffrey divergence), using a (invertible) generator that admits exact PDF evaluation, and reparameterizing the discriminator in a certain way. The Fenchel-variational view espoused in [26] reveals the optimal discriminator T to be a function of the odds ratio: T ∗ = h(q/p) for some function h determined by the particular f -divergence chosen. We observe that if q can be evaluated analytically, then we might as well represent the discriminator in terms of q and some function ν, where ν directly approximates p. This change of representation allows us to take any known
regularities of p and impose them on ν; e.g., if p is known to be translation-invariant, then we can safely impose translational invariance on ν without imposing any undue constraints on our model. Note that we cannot similarly impose p’s structure on T , which may be a complex function even if p and q are simple: for example, even if p and q are both bounded above, T may be unbounded, since it is a function of the ratio of the two.
Fig. 2 illustrates how ν is structured for our application domain of vehicle trajectory forecasting: ν is represented as a sum of learned spatial rewards, which penalizes trajectories according to a learned function over spatial positions. Intuitively, this prevents trajectories from colliding with obstacles, while simultaneously learning the concept of an obstacle.

heading: 2 PUSHFORWARD REPRESENTATION AND IMPLEMENTATION
text: Optimizing Eq. (1) is straightforward except for one subtle point: we must be able to evaluate q pointwise and differentiate the expression Ex∼q log(q(x)/ν(x)) with respect to the parameters of q. Inspired by prior work [29, 8, 18], we solve both these problems by representing q as the pushforward of a simple distribution under an invertible warp (also known as a normalizing flow). Suppose µ is a distribution over a set Z and g : Z → X is a function (the generator) with domain Z. Then we can define a measure on X as the distribution of g(z) sampling z from µ—this distribution, denoted here by g|µ, is referred to as the pushforward of µ under g. Now suppose g is parameterized by θ and differentiable in x and θ. By representing q as qθ = gθ|µ, for some simple distribution µ, we can move the derivative wrt. θ inside the expectation by exploiting the property Ex∼gθ|µf(x) = Ez∼µf(gθ(z)) for all functions f :
d
dθ Ex∼q log
q(x) ν(x) = d dθ Ex∼qθ|µ log q(x) ν(x) = Ex∼µ d dθ log qθ(gθ(x)) ν(gθ(x)) . (2)
This is well-known as the reparameterization trick; it allows us to obtain a low-variance, unbiased estimate of the parameter derivatives for learning with SGD. However, one problem remains: evaluating qθ(x) = (gθ|µ)(x), which appears in both the first and last terms of (1). This is solved by assuming that gθ is invertible: ẑ := g−1θ (x̂). Thus, we have an analytic formula for the pushforward density:
qθ(gθ(z)) = (gθ|µ)(gθ(z)) = µ(z) ∣∣(dgθ)z∣∣−1 , (3)
where |(dgθ)z| represents the determinant of the Jacobian of gθ evaluated at the point z. This finally allows us to rewrite Eq. (1) in the following explicit form, after performing some simplifications:
−max θ inf φ
Ex̂∼p log µ(ẑ)
|(dgθ)ẑ| +
µ(ẑ)
|(dgθ)ẑ|νφ(x̂) + Ez∼µ log
µ(z)
|(dgθ)z|νφ(gθ(z)) . (4)
A pseudocode summary of our method is given in Algorithm 1. X and Z denote batches of observed and latent samples, respectively, while subscripts D and G denote either data or generated samples. A few implementation issues are noted here. Applying the method to a particular problem requires the implementation of the invertible generator G(·; θ), the structured Gibbs energy log ν(·;φ), and the base distribution µ. In order to avoid numerical issues when q is not absolutely continuous wrt. ν (i.e., when ν is 0 but q is not), we reparameterize ν as ν ← αq + ν, where α is a small number. Given this assumption, the quantity q(x)/(αq(x) + ν(x)) can be rewritten in terms of the sigmoid σ. Optimization proceeds by alternating between minimizing the loss (1) in the generator parameters θ and maximizing it in the energy parameters φ. However, as noted in Alg. 1, we may alternatively minimize a different objective for the generator: namely, −Ex∼p log q(x) + Ex∼q log(q(x)/ν(x)). The rationale for the alternative objective is that, as noted in Sec. 3.1, the inner minimization may be viewed as fitting ν to p—in which case, we may approximate KL(q, p) as Ex∼q log(q(x)/ν(x)). The alternate generator objective was used for the toy experiment in Fig. 1, whereas the original objective was used for the trajectory forecasting experiments.

heading: 3 DERIVATIONS AND INTERPRETATIONS
text: Equation (1) can be derived via a variational lower bound derived from Fenchel conjugacy, using a technique similar to [25, 26]. Our approach of pairing this convex conjugate with the pushforward
Algorithm 1 Pseudocode for C3PO implementation Require: XD: a batch of training data, ZG: batch of generator noise samples from µ
1: ZD, det dg −1 XD ← G−1(XD; θ) . G−1(x) returns g−1θ (x) and log det. of Jacobian of g −1 θ at x 2: XG, det dgZG ← G(ZG; θ) . G(z) returns gθ(z) and log det. of Jacobian of gθ at z 3: log q(XD)← logµ(ZD) + log |det dg−1XD | . Generator PDF at data samples 4: log q(XG)← logµ(ZG) + log |det dgZG | . Generator PDF at generator samples 5: log q/ν(XD)← logα+ log q(XD)− log ν(XD;φ) 6: log q/ν(XG)← logα+ log q(XG)− log ν(XG;φ) 7: L← BatchMean(− log q(XD)− α−1σ(log q/ν(XD)) + log σ(log q/ν(XG))− logα) 8: for i← 1 . . . N do 9: θ ← θ − β∇θL . Alternative generator loss: L := − log q(XD) + log σ(log q/ν(XG))
10: for j ← 1 . . .M do 11: φ← φ+ β∇φL
direct density estimation motivates our method’s name: Convex Conjugate Coupled Pushforward Optimization (C3PO). Observe that the Jeffrey divergence can be written as minq −Ex̂∼p log q(x̂) + Ex∼q log q(x)− Ex∼q log p(x). Since q can be sampled, evaluated, and differentiated (Sec. 2), but p can only be sampled, our goal in this section is to convert −Ex∼q log p(x) to something expressable in terms of expectations wrt. p and q. This is achieved by applying the Fenchel-Young inequality to the function f(p) := − log p, which yields − log p ≥ supλ<0 λp − (−1 − log(−λ)). Substituting this inequality in place of − log p(x) yields the following lower bound of −Ex∼q log p(x):∫
X q(x)(sup λ<0 λp(x) + log(−λ) + 1) dx = 1 + sup λ<0 ∫ X q(x) ( λ(x)p(x) + log(−λ(x)) ) dx, (5)
where the sup on the right-hand side is taken over all functions λ : X → R− mapping the domain to a negative scalar. Observing that ∫ q(x)λ(x)p(x) dx can be written either as Ex∼qλ(x)p(x) or Ex̂∼pλ(x̂)q(x̂), and making the substitution ν = −1/λ, we have the equivalent bound
−Ex∼q log p(x) ≥ 1 + sup ν>0
Ex̂∼p − q(x̂)
ν(x̂) − Ex∼q log ν(x). (6)
Substituting this the Jeffrey divergence yields Eq. (1).

heading: 3.1 INTERPRETATION AS LEARNING A GIBBS DISTRIBUTION
text: Although we have so far viewed our goal as primarily learning q, we now observe that our method can also be interpreted primarily as a way of learning ν as a Gibbs distribution approximating p. Learning a Gibbs distribution, or a more general energy-based model [19], is a very general and effective way to impose strong domain-specific regularization on probability distributions over high-dimensional data. Generally, this is achieved by structuring the energy function Vφ := log νφ to assign similar energies to similar examples; for example, a convolutional neural network might constitute a good energy for image generation, since the structure of a CNN encodes some degree of translational invariance. Unfortunately, inference and learning with high-dimensional Gibbs distributions is difficult.
Our method can be viewed as a way to train a Gibbs distribution that circumvents some of these difficulties. In this view, the inner optimization in Eq. (1) is interpreted as minimizing a weighted divergence between p and ν; the weights are exactly q. Specifically, we observe that the inner optimization in Eq. (1) minimizes the following weighted Itakura-Saito divergence [16, 9]:
min φ
∫ q(x) ( p(x)
νφ(x) − log p(x) νφ(x) − 1
) dx (7)
It would seem reasonable to choose the weights q as q = p; since p cannot be evaluated directly, alternating optimization is a sensible alternative. Minimizing Eq. (7) intuitively gives us the best Gibbs approximation of p over the support of q.
Intuition for the learning rule of ν can be obtained by computing the functional gradient of Eq. (1) with respect to ν (i.e., differentiating with respect to ν(x), ∀x. The functional gradient δ/δν expresses
the direction in which ν should be moved at each point x in order to optimally decrease the objective. Observing that Ex∼pq(x)/ν(x) = Ex∼qp(x)/ν(x), we obtain the following for the functional gradient of the objective wrt. log ν: δC/δ log ν(x) = q(x) ( p(x)/ν(x)− 1 ) . We therefore see that
minimizing Eq. (1) in ν raises or lowers log ν at each point x according to whether it exceeds p(x), with a learning rate given by q(x), and a unique fixed point (assuming q > 0) of p = ν.

heading: 4 EXPERIMENTS
text: We conducted experiments on four datasets of vehicle trajectories, where the objective is to forecast a distribution over the vehicle’s future locations x ∈ R20×2 given contextual information about each scene. The contextual information includes 2 seconds of the vehicle’s previous position, as well as a Bird’s Eye View (BEV) map of visual scene features. Our experiments are designed to quantify two key aspects of generative modeling: the learned model’s likelihood of held-out test data (i.e. the negative forward cross-entropy −H(p, q)), and the quality of samples from the learned model (i.e. the negative reverse cross-entropy −H(q, p)). Our hypotheses are: 1) C3PO will achieve superior sample-quality performance to other methods 2) C3PO will learn a high-quality q, partially due to its ability to perform direct density evaluation and optimization of q 3) C3PO will learn an interpretable ν that penalizes bad samples.
Two of the four datasets are synthetic (BEVWORLD 1 and BEVWORLD1K), with known roads, enabling us to construct samples from a reasonable p distribution, approximate p via KDE, and measure the reverse cross-entropy H(q, pKDE). We also calculate the percentage of trajectory points on the road as another measure of sample quality. Details of how we generate BEVWORLD are in the supplement. We also experiment with two real-world datasets: the KITTI dataset, and the CALIFORECASTING dataset [32].

heading: 4.1 IMPLEMENTATION AND BASELINES
text: Given our setting is that of forecasting future vehicle trajectories, we leverage ideas from Inverse Reinforcement Learning [24, 43, 41], to structure our Gibbs energy V as a spatial cost map: where
log νφ = Vφ(x) = ∑T t=1Rφ(xt0, xt1;BEV), and Rφ(a, b;BEV) is the output of a CNN that can be interpolated at 2d positions of the form (a, b). This structure enables ν to penalize trajectories that travel to locations it perceives to be bad, e.g. locations with obstacles, or locations far from a perceived road.
We compare our method, C3PO, to several state-of-the-art approaches in imitation learning and generative modeling: Generative Adversarial Imitation Learning (GAIL) [12], f-GAN [26], the CVAE method of DESIRE [20], and R2P2[32]. In each baseline, we use architectures as similar to our own method as possible: the policy (generator) architecture of GAIL and the generator architecture of f-GAN are identical to the generator architecture of our own approach. The same architecture used for Vφ in our method was also used for the discriminators in all baselines.
Our implementation of the q architecture is based on R2P2 [31]. One key difference between our method, C3PO, and R2P2, is that R2P2 does not have an adversarial component; its main focus is learning the forward KL term, the first component of C3PO’s objective function. R2P2 starts from a similar objective: the symmetric sum of cross-entropies. However, it relies on a cruder approximation of H(q, p), because it learns an approximating p̃ for H(q, p̃) offline, with no interaction from q.

heading: 4.2 SYNTHETIC EXPERIMENTS
text: BEVWORLD 1contains a single scene with 300 samples from p. This setting is unconditional: the contextual information provided is identical, and not directly useful for modeling p. This setting also provides a fairer comparison to fGAN methods, which left the extension of fGAN to contextual settings as future work [26]. Table 1 shows the results of BEVWORLD 1 experiments. We observe that R2P2 and C3PO achieve the best−H(p, q) scores, with C3PO outperforming all nondegenerate methods in its sample quality. This evidence supports hypotheses 1) and 2): C3PO achieves superior sample quality and high-quality data density. We also observed C3PO to be very stable throughout training in terms of H(p, q): it was as easy to train it as R2P2 in all experiments.
We also observe that while fGAN KL indirectly learns a q with some support for p, its samples are quite poor. This matches expectations, as the forward KL divergence fails to impose much penalty on sample quality [14]. fGAN Jeffrey is the fGAN method most similar to our approach because it uses the Jeffrey divergence. We observe it to suffer mode collapse in all of our experiments, a similar result to [26], in which fGAN Jeffrey had the worse test-set likelihood.
Next, we consider BEVWORLD1K, in which 100 training scenes, each with 100 samples from p, are used to learn conditional generative models. There are 1000 scenes in the test set. Table 1 shows the result of these experiments. We observe results similar to BEVWORLD1K, again supporting hypotheses 1) and 2). C3PO can learn in the conditional setting to produce high-quality samples from a distribution with good support of p. We display example results in Fig. 3, and observe several methods, including C3PO , learn a cost-map representation that penalizes samples not on the road. This evidence of the learned intuitive perceptual measurement of ν supports hypothesis 3).

heading: 4.3 REAL-WORLD EXPERIMENTS
text: We now experiment with real-world data, in which each method is provided with noisy contextual information, and is evaluated by its ability to produce a generative model and high-quality samples. Evaluation of sample quality is more difficult in this scenario: we have only one sample of p in each scene, precluding construction of pKDE, and there are no labels of physical roads, precluding computation of on-road statistics. Fortunately, evaluation of −H(p, q) is still possible. After observing the learned Vφ of C3PO on both synthetic and real data, we found Vφ to be generally interpretable and intuitively good: it assigns low cost to roads it learns to perceive and it assigns high cost to obstacles it learns to perceive. We therefore employed our learned model Vφ to quantitatively evaluate samples from all methods. Fig. 4 illustrates results from several approaches visualized with V and point cloud features from the BEV, at various quantiles of each method’s performance under V . Fig. 5 (left) illustrates each method on a set of randomly sampled scenes. In both figures, V perceives and assigns penalty to regions around obstacles in the point cloud data. Note that this energy is learned implicitly, because demonstrations from p avoid obstacles. These results provide further support for hypothesis 3).
Quantitative results are shown in Table 2. We find that results are, overall, similar to results on the other datasets. C3PO produces a high-performing q in terms of both its likelihood, −H(p, q), as well as the quality of its samples, V . Additionally, we show evaluation of V for the top methods in Fig. 4 (right). Together, these results further strengthen evidence in support of all of our hypotheses.

heading: 5 RELATED WORK
text: The initial motivation for our work was the apparent dichotomy between sample quality and mode coverage in existing deep generative models; this phenomenon has been noted and quantified in work such as [40, 23, 4, 13, 14]. Our work synthesizes several techniques from prior work to address these issues, including the Fenchel-variational principle from work such as [25, 26], the well-known reparameterization trick [17, 35], and analytic pushforward-based density estimators such as normalizing flows [29, 18], RealNVP [8], and related models [22, 10, 27, 38, 30]. Comparatively little work has explored combining these methods, with some exceptions. Combining a RealNVP [8] density estimator with a GAN objective was considered in [11]; however, this is susceptible to the problems inherent with GANs mentioned in the introduction. A pushforward-based density estimator was employed in conjunction with variational inference to optimize the classical Bayesian evidence lower bound in [18], but this cross-entropy objective suffers from the previously-mentioned problems with optimizing KL(p ‖ q) alone, as do all methods based on this objective, including [8, 10, 27, 38]. Our work is also comparable to the extensive literature on learning deep graphical models, including variants of Boltzmann machines [1, 34] and various proposals for learning deep CRFs and structured energy-based models [19] based on inference techniques including convex optimization [28, 7, 2, 36], various forms of unrolled inference [33, 42, 39], pseudolikelihood [21], and continuous relaxation [5], among other techniques [13]. Viewed as a way to learn a deep graphical model, the most important distinguishing factor of our work is the fact that our method does not need to perform inference directly, as discussed in Sec. (3.1). To the extent that q is viewed as (indirectly) performing inference, our method and other deep generative models bears some similarity to the wake-sleep algorithm, which also alternates between optimizing a model distribution and an “inference” distribution using complementary divergences [13]. Score-matching [15] also learns a Gibbs distribution without inference; however, unlike our method, score-matching does not learn an inference distribution.

heading: 6 CONCLUSION
text: We have demonstrated how GANs may be effectively regularized by reparameterizing the discriminator to be a function of the model density and a structured Gibbs density, showing how this may be achieved by optimizing a Jeffrey divergence loss, assuming the generator is invertible, and applying a variational bound based on Fenchel conjugacy. Applied to a trajectory forecasting problem, we observed superior mode coverage and qualitative results compared to traditional GANs. We hope to soon demonstrate the general applicability of our approach by applying it to the task of image generation as well.

Review summaries:
This paper combines a number of ideas to train generative models with (deep) structured constraints. The general idea is similar to Flow-GAN, which learns a normalizing flow-based generator by optimizing the negative loglikelihood with an augmented GAN loss. However, It’s difficult to impose prior structure information in the GAN framework. To address this problem, the authors proposed to minimize a so-called Gibbs-regularized variational bound of Jeffery divergence, which is the summation of KL and reverse KL divergence. The authors provide some justification that the Jeffery divergence works by yielding good mass-covering and mode-seeing properties.