Title: THE INFORMATION-AUTOENCODING FAMILY: A LAGRANGIAN PERSPECTIVE ON LATENT VARIABLE GENERATIVE MODELING

Abstract: A variety of learning objectives have been recently proposed for training generative models. We show that many of them, including InfoGAN, ALI/BiGAN, ALICE, CycleGAN, VAE, β-VAE, adversarial autoencoders, AVB, and InfoVAE, are Lagrangian duals of the same primal optimization problem. This generalization reveals the implicit modeling trade-offs between flexibility and computational requirements being made by these models. Furthermore, we characterize the class of all objectives that can be optimized under certain computational constraints. Finally, we show how this new Lagrangian perspective can explain undesirable behavior of existing methods and provide new principled solutions.

1. INTRODUCTION:

Deep generative models have been successfully utilized in a wide variety of tasks (Radford et al., 2015; Zhu et al., 2017; Yang et al., 2017; Li et al., 2017b) . Prominent examples include Variational Autoencoders (VAE, Kingma and Welling (2013) ; Rezende et al. (2014) ) with extensions such as β-VAE (Higgins et al., 2016) , Adversarial Autoencoders (Makhzani et al., 2015) , and InfoVAE (Zhao et al., 2017) ; adversarially trained models such as Generative Adversarial Networks (Goodfellow et al., 2014) and their extensions such as ALI/ BiGAN (Dumoulin et al., 2016a; Donahue et al., 2016) , InfoGAN (Chen et al., 2016a) and ALICE (Li et al., 2017a) ; hybrid objectives such as CycleGAN (Zhu et al., 2017) , DiscoGAN (Kim et al., 2017) and AVB (Mescheder et al., 2017) . All these models attempt to fit an empirical data distribution, but differ in the measure of similarity between distributions, whether or not they allow for efficient (amortized) inference, and the extent to which the latent variables should retain or discard information about the data. These methods also crucially differ in terms of the optimization techniques used, which can be likelihood-based or likelihoodfree (Mohamed and Lakshminarayanan, 2016) .

In this paper, we revisit the question of designing a training objective for a latent variable generative model. We formulate a constrained (variational) optimization problem that explicitly captures the various desired properties of a generative model, such as fit to the data distribution, tractable (amortized) inference, and lossy vs. lossless compression. Surprisingly, we show that all the previously mentioned models are different Lagrangian duals of the same primal optimization problem. They only differ in the value assigned to the Lagrange multipliers, corresponding to different prioritizations of the various (potentially conflicting) desiderata. In fact, by considering all possible Lagrangian duals, we obtain a strictly more general class of models, of which InfoGAN, ALI/BiGAN, ALICE, VAE, β-VAE, adversarial autoencoders, AVB, CycleGAN/DiscoGAN and InfoVAE are all special cases. This general formulation reveals the modeling trade-offs being made between flexibility and the tractability of computing and optimizing the resulting objective. We show that each of the previously mentioned models correspond to a special choice of the Lagrange multipliers that 'cancels out' certain intractable terms from the training objective, so that a particular computational constraint is satisfied (e.g., it can be optimized likelihood-based or likelihood-free). In addition, under some mild assumptions, we are able to characterize all the training objectives that can be optimized under certain computational constraints.

This generalization also provides new insights into the properties of these models, which are implied by particular choices of the Lagrange multipliers. First, solving the dual problem requires optimizaton over the Lagrange multipliers (to obtain the best approximation to the primal). However, existing models use fixed values for the Lagrangian multipliers. We explain the consequences of this approximation, and demonstrate scenarios where this approach fails to solve the primal. Furthermore we show that when the primal is infeasible (e.g., due to insufficient modeling capacity), the choice of Lagrange multipliers prioritizes different constraints, and show that appropriate prioritization can lead to improved log-likelihood score.

The paper is organized as follows. In Section 2, we start from an axiomatic derivation of VAEs from minimum desiderata, and realize that most generative model objectives can be written as the Lagrangian dual of a constrained variational optimization problem. In Section 3, we consider the more general case in which the Lagrangian multipliers can be arbitrarily selected, and enumerate over all possible objectives under different computation constraints. In Section 4, we introduce two applications of this formulation that allows us to diagnose potential problems, compare different approaches and design new models.

2. THE INFORMATION-AUTOENCODING FAMILY:

We consider a class of probabilistic generative models with two types of variables: observed 'data' variables (denoted x ∈ X ) and latent 'feature' variables (denoted z ∈ Z). For ease of exposition, we describe the case of discrete variables, but the results apply to continuous variables as well1 . The model family specifies a joint probability distribution p θ (x, z) parameterized by θ. We assume that the distribution can be factored as

p θ (x, z) = p(z)p θ (x|z)

where p(z) is a given, fixed distribution. We assume that it is efficient to sample from p(z) and evaluate p(z) for any z. Likewise, we assume it is easy to evaluate and to sample from p θ (x|z) for any x and z. In general, p θ (x) is intractable to compute but easy to sample from (by sampling (x, z) from p θ (x, z) and discarding z); p θ (z|x), on the other hand, is generally both difficult to evaluate and to sample from.

Given samples from a data distribution p data (x), the learning goal is to find θ such that p θ (x) ≈ p data (x). The hope is often to discover latent structure in the data through the latent variables z, evocatively called 'features'. Given a data point x, one would want to infer the corresponding latent variables using p θ (z|x). However, as noted above, that is generally intractable to evaluate and sample from. We consider the case of a separate amortized inference distribution q φ (z|x), and assume q φ (z|x) is easy to evaluate and to sample from. This results in the following approximation (lower bound) to the intractable p θ (x), known as the ELBO:

EQUATION

where D KL denotes the divergence:

D KL (p q) = E p [log p -log q].

2.1. AXIOMATIC DERIVATION OF VARIATIONAL AUTOENCODERS:

Instead of deriving a learning objective as an approximation to the intractable marginal likelihood p θ (x), we start from a list of basic properties that any learned model should satisfy. Intuitively, under ideal conditions, the generative model should model the true data distribution, while the amortized inference model q φ (z|x) should capture the true posterior p θ (z|x). We formalize this in the following definition: Definition 1 (Consistency Condition). A distribution p θ (x, z) over X × Z, a distribution p data (x) over X , and a family of conditional (inference) distributions {q φ (z|x)} x∈X are said to be consistent with each other if the following holds:

• Correct Data Marginal: ∀x, p θ (x) = p data (x).

• Correct Feature Inference: ∀x such that p data (x) = 0, q φ (z|x) = p θ (z|x).

We can alternatively express the consistency requirement using the following definition (Dumoulin et al., 2016b; Donahue et al., 2016) Definition 2 (Joint Inference Distribution). Given p data (x) over X , and a family of conditional (inference) distributions {q φ (z|x)} x∈X we define the joint inference distribution as

q φ (x, z) = p data (x)q φ (z|x)

For compactness, we denote q(x) = p data (x). Corollary 1. p θ (x, z), p data (x) and {q φ (z|x)} x∈X are consistent with each other if and only if

EQUATION

By noticing that the roles played by z and x are symmetric, we also have the following equivalent definition of consistency: Definition 3 (Feature Marginal and Posterior). Given the joint distribution q φ (x, z) defined in Definition 2 we define the following

q φ (z) = x q φ (x, z) q φ (x|z) = q φ (x, z)/q φ (z)

Corollary 2. p θ (x, z), p data (x) and {q φ (z|x)} x∈X are consistent with each other if and only if

• Correct Feature Marginal: ∀z, q φ (z) = p(z).

• Correct Data Inference: ∀z such that p(z) = 0, q φ (x|z) = p θ (x|z).

2.2. A CONSTRAINED OPTIMIZATION FORMULATION:

It is not difficult to see that for a given data distribution p data (x), there can be multiple distinct choices of θ, φ satisfying the stated consistency condition (Chen et al., 2016b; Zhao et al., 2017; Li et al., 2017a) 

Any f (θ, φ) can in principle be used. In this paper, we study a special but important case:

EQUATION

where I p (x; z), I q (x; z) are mutual information between x and z under distributions p(x, z) and q(x, z) respectively. The coefficients α 1 , α 2 can be either positive or negative, to maximize or minimize mutual . Both cases could be desirable, depending on the goal. Models like InfoGAN (Chen et al., 2016a) explicitly maximize mutual information with respect to a group of latent variables to ensure that this group of variables is used. Information maximization is also useful to avoid the uninformative latent code problem (Chen et al., 2016b; Zhao et al., 2017) , where latent features are not used when using certain training objectives. In other scenarios we may wish to minimize mutual information. For example, the information bottleneck (Tishby and Zaslavsky, 2015) approach minimizes mutual information between input and latent features under the constraint that the features can still predict some labels. Similar ideas are also widely used in compression (Shamir et al., 2010) where minimum information is retained while still being able to perform certain tasks.

2.3. VARIATIONAL AUTOENCODING AS CONSTRAINED OPTIMIZATION:

To solve the above optimization problem (3) constrained with the consistency condition we use the notion of a divergence between probability distributions. Definition 4. Let P(A) be the set of all probability distributions over a set A. A functional D :

P(A) × P(A) → R + is called a strict divergence if D(p q) = 0 if and only if p = q.

Examples of strict divergences include Kullback-Leibler (KL) divergence, maximum-mean discrepancy (with a universal kernel, see Gretton et al. (2007) ), Jensen-Shannon Divergence (Nowozin et al., 2016) , etc. In particular, the reverse KL divergence is defined as D ¬KL (p q) = D KL (q p).

Given strict divergences D, and D , and using the equivalent formulation of consistency in Definition 1 and Corollaries 1 and 2, we can define the following optimization problems with one of three possible constraints3 :

min θ,φ α 1 I p (x; z) + α 2 I q (x; z) s.t. D(p(x, z) q(x, z)) = 0 (5) or D(q(x) p(x)) = 0 , D (q(z|x) p(z|x)) = 0 ∀x ∈ X (6) or D(p(z) q(z)) = 0 , D (p(x|z) q(x|z)) = 0 ∀z ∈ Z (7)

It is easy to see that using any of the three constraints (5)(6)( 7) is equivalent to the original optimization problem in Eq.(3). In fact, any superset of the constraints (5)(6)(7) also leads to an equivalent optimization problem (with redundancy).

2.4. VARIATIONAL AUTOENCODING AS LAGRANGIAN DUAL:

A natural approach to solve constrained optimization problems is to consider their Lagrangian relaxation. For example, the Lagrangian corresponding to (5) would be f (θ, φ) + λD(p(x, z) q(x, z)) where λ is the Lagrange multiplier. More generally, we define a broad class of Lagrangian relaxations corresponding to the constraints in ( 5), (6), and (7) and their supersets. This leads to the following general class of training objectives, which we shall see encompasses a large number of existing probabilistic autoencoding models. Definition 5. The Information(Info)-Autoencoding family is the set of all objectives of the form

L(λ, α 1 , α 2 ) = α 1 I p (x; z) + α 2 I q (x; z) + n i=1 λ i D i (8)

where α 1 , α 2 ∈ R, λ i ∈ R, λ i > 0 are Lagrangian parameters, and the following are satisfied:

1) For all i = 1, • • • , n, each D i is one of the following D(p(x, z) q(x, z)), D(p(x) q(x)), D(p(z) q(z))

E r(x) [D(p(z|x) q(z|x))], E t(z) [D(p(x|z) q(x|z))]

where D is any strict divergence, and r, t are any distributions supported on X , Z respectively 2) There exists strict divergences D, D , and distributions r, t supported on X , Z respectively, so that one of the following sets

EQUATION

We call this family the KL Information-Autoencoding family if in the above definition all divergences D are either KL or reverse KL, r(x) is p(x) or q(x), and t(z) is p(z) or q(z).

In the above definition, r(x) is itself a set of Lagrangian parameters, corresponding to the set of constraints {D(q(z|x) p(z|x)) | x ∈ support(q(x))}. In this paper we assume r(x) is either p(x) or q(x). Similarly, we assume t(z) is either p(z) or q(z). We leave the study of more general choices for r, t as future work.

In general we must maximize over the Lagrangian dual parameters λ = (λ 1 , • • • , λ n ) to find the tightest relaxation to the primal. However we shall show in the following section that many existing models simply pick specific (suboptimal) values of λ. In Section 3, we will show that this is due to tractability reasons: the Lagrangian is easy to optimize (in a certain sense) only for some λ.

2.5. RELATIONSHIP WITH EXISTING MODELS:

The learning objectives used to train many existing latent variable models (such as ELBO, InfoGAN, etc.) can be rewritten in the Lagrangian form introduced in Definition 5. By rewriting them explicitly as in Eq.( 8), we uncover the implicit tradeoffs being made (by choosing the Lagrange multipliers, i.e., pricing constraint violations), revealing similarities and symmetries between existing models along the way.

1) The original evidence lower bound (ELBO) in Eq.( 1) can be rewritten into equivalent Lagrangian dual forms

EQUATION

The second form ( 13) is well known (Kingma and Welling, 2013) and is traditionally the explanation why optimizing ELBO leads to consistency condition. The same conclusion could also be derived symmetrically from Eq.( 12).

2) ELBO has been extended to include a β scaling parameter (Higgins et al., 2016 )

EQUATION

This can be written as

EQUATION

which is the Lagrangian for Eq.( 11) choosing α 1 = 0, α 2 = β -1. This shows clearly the information preference of the model. If β > 1 this objective is information minimizing; if β < 1 this objective is information maximizing; and neutral when β = 0 which is the original ELBO objective. This neutrality corresponds to the observations made in (Chen et al., 2016b; Zhao et al., 2017) .

3) The InfoGAN objective in (Chen et al., 2016a) 4 can be converted to Lagrangian form for Eq.( 10)

L InfoGAN = -I p (x; z) + E p(x) [D KL (p(z|x) q(z|x))] + D JS (p(x) q(x))

where α 1 = -1, α 2 = 0, and the Jensen-Shannon divergence D JS is approximately optimized in a likelihood-free way with adversarial training. This equivalent formulation highlights the mutual information maximization property of InfoGAN. It also reveals that the InfoGAN objective encourages correct inference because of the D KL (p(z|x) q(z|x)) term.

4) InfoVAE/AAE (Zhao et al., 2017; Makhzani et al., 2015) written in Lagrangian form becomes

L InfoVAE = -I q (x; z) + E q(z) [D KL (q(x|z) p(x|z))] + D(q(z) p(z))

where D is any divergence that can be optimized with likelihood-free approaches. This reveals an elegant symmetry between InfoGAN and InfoVAE. Both maximize mutual information, except InfoGAN is the Lagrangian of constraints in Eq.( 10) while InfoVAE is the Lagrangian of those in Eq.( 11).

5) ALI/BiGAN corresponds to direct minimization of D JS (q(x, z) p(x, z)), while ALICE (Li et al., 2017a) has the following Lagrangian form:

L ALICE = -I q (x; z) + E q(z) [D KL (q(x|z) p(x|z))] + D JS (q(x, z) p(x, z))

This is similar to InfoVAE except that the final divergence is taken with respect to the joint pair (x, z) rather than z only.

6) CycleGAN/DiscoGAN (Zhu et al., 2017; Kim et al., 2017) can be written as

L CycleGAN = -I q (x; z) -I p (x; z) + E q(z) [D KL (q(x|z) p(x|z))] + E p(x) [D KL (p(z|x) q(z|x))] + D JS (q(x) p(x)) + D JS (q(z) p(z))

This reveals clearly that the model is maximizing mutual information, and encouraging correct inference in both direction by both matching q(x|z) and p(x|z) and matching p(z|x) and q(z|x).

3. ENUMERATION OF TRACTABLE FAMILIES:

In the previous section, we introduced a very broad family of training objectives encompassing several widely used approaches. While all the objectives in the family are reasonable, the special cases we have identified have a very special property in common: they are (in a certain sense) 'easy' to optimize. This is a highly non-trivial property to have. In fact, mutual information and most divergences used in Definition 5 are generally not tractable to compute and/or optimize. Thus, the objectives will be 'easy' to optimize only if we carefully choose certain values of the Lagrange multipliers λ i , so that certain terms 'cancel out'.

In this section, we propose a general framework to formally characterize all training objectives in the family that satisfy certain tractability requirements. We formalize tractability by defining the notion of a tractable "tool box", corresponding to a set of terms that we assume can be computed/optimized reasonably well, such as the ones used in ELBO (Kingma and Welling, 2013; Rezende et al., 2014) , InfoGAN (Chen et al., 2016a) , InfoVAE (Zhao et al., 2017) , etc. Note that these objectives are generally not provably tractable to optimize, as they involve non-convexities, expectations over highdimensional spaces, etc. There is, however, significant empirical evidence that they can be optimized reasonably well in practice, and we therefore group them based on the techniques employed.

The main technical challenge is dealing with all the possible simplifications and transformations that leave an objective invariant. For example, we need to account for objectives like (15), which appear to involve components that are difficult to estimate such as I q (x; z), but can actually be simplified to a more manageable form ( 14) that can be optimized in a likelihood-based way.We therefore define a set of transformations that leave an objective equivalent according to the following definition. Definition 6. An objective L is equivalent to L when ∃C, so that for all parameters θ, φ, L(θ, φ) = L (θ, φ) + C. We denote this as L ≡ L .

We are then able to analyze all the objectives that contain elements from a given "tool box", and all objectives derived from them through equivalent transformations.

3.1. LIKELIHOOD-BASED AND LIKELIHOOD-FREE COMPONENTS:

We formalize the idea of "computation toolbox" by the following definition: Definition 7. For any divergence D, we define

(1) Likelihood-based terms as following set

T 1 = {E p [log p(x|z)], E p [log p(x, z)], E p [log p(z)], E p [log q(z|x)] E q [log p(x|z)], E q [log p(x, z)], E q [log p(z)], E q [log q(z|x)]}

(2) Unary likelihood-free terms as the following set

T 2 = {D(q(x) p(x)), D(q(z) p(z))}

(3) Binary likelihood-free terms as the following set

T 3 = {D(q(x, z) p(x, z))}

Define an objective L as likelihood-based computable if there exists equivalent transformations L ≡ L so that L only contains terms in T 1 ; unary likelihood-free computable if L only contains terms in T 1 ∪ T 2 ; binary likelihood-free computable if L only contains terms in T 1 ∪ T 2 ∪ T 3 .

Consider the set T 1 of likelihood-based terms. Because we can directly compute likelihoods for log p(x|z), log p(x, z), log p(z) and log q(z|x), optimizing over an expectation over these terms can be achieved by back-propagation using path-wise derivative methods (the reparameterization trick) (Kingma and Welling, 2013; Schulman et al., 2015; Jang et al., 2016; Maddison et al., 2016) , and by likelihood ratios when reparameterization is not applicable (Mnih and Gregor, 2014; Schulman et al., 2015) .

Likelihood-free methods have been extensively studied in recent years. Examples include Jensen Shannon divergence (Goodfellow et al., 2014) , f-divergences (Nowozin et al., 2016) , Wasserstein distance (Arjovsky et al., 2017; Gulrajani et al., 2017) and Maximum Mean Discrepancy (Gretton et al., 2007; Li et al., 2015) . Even though such models are still difficult to train, significant improvements have been made in terms of stability and efficiency. Here, we distinguish between the (unary) case, where we consider a single group of variable at a time (either x or z), and the more general binary case. The reason is that the binary case has been empirically shown to be more difficult to train, for example leading to inaccurate inference (Dumoulin et al., 2016b; Donahue et al., 2016) .

3.2. ELEMENTARY TRANSFORMATIONS:

Many elementary probabilistic identities can be used to rewrite an objective into another one which is equivalent according to Definition 6. An example is the chain rule identity

p(x, z) = p(x)p(z|x) = p(z)p(x|z)

In addition, terms that do not contain trainable parameters (φ, θ) can be freely added to the objective without affecting the optimization. Examples include E p [log p(z)] and E q [log q(x)]. We formalize the above intuition in the following definition.

Definition 8. Define the following set of transformations as the elementary transformations

E * [log p(x, z)] ≡ E * [log p(x) + log p(z|x)] ≡ E * [log p(z) + log p(x|z)] E * [log q(x, z)] ≡ E * [log q(x) + log q(z|x)] ≡ E * [log q(z) + log q(x|z)] E q [log q(x)] ≡ 0 E p [log p(z)] ≡ 0

where E * indicates that the equivalence holds for both E p and E q .

Note that this set does not contain all possible transformations, such as importance sampling

E q [log p(x)] ≡ E q(x) log E q(z|x) p(x, z) q(z|x)

We leave extensions to more general transformations to future work. Nevertheless, we are now in a position to study all the objectives that can be constructed using the building blocks from Definition 7, and any objective derived by applying elementary transformations. From now on, we restrict the definition of tractable family in Definition 7 to equivalence by elementary transformation only.

3.3. ENUMERATION OF KL INFORMATION-AUTOENCODING FAMILY:

In the previous sections we have formally defined a grammar for tractable models. In this section, we will use these definitions to find all possible objectives that can be optimized under given tractability assumptions. We shall write the largest family under each tractability constraint in two forms: a Lagrangian dual form, and an equivalent tractable form that only contains terms satisfying assumed computability constraints.

In this section we focus on the KL Information-Autoencoding family defined in Definition 5 where all divergences in the constraints are either D KL or D ¬KL . 5 We discuss KL divergence because it is the most interesting case to study under elementary transformations. KL divergence is a sum of expectations over log probabilities D KL (p||q) = E p [log p -log q] while the elementary transformations are all linear operations on log probabilities. We will discuss generalization to other divergences in the next section.

For likelihood based computable families, we show that variational mutual information maximization (Barber and Agakov, 2003) and β-ELBO (Higgins et al., 2016) are the only possible families (up to equivalences). Based on the set of transformations we considered, there do not exist other objectives satisfying the stated computational constraints. This is formally given by the following theorem.

Theorem 1. Define the following objectives 1) Variational mutual information maximization

L VMI = -I p (x; z) + D KL (p(z|x) q(z|x)) = E p [log q(z|x)]

2) β-ELBO, where β is any real number

L β-VAE = (β -1)I q (x, z) + D KL (q(x|z) p(x|z)) + βD KL (q(z) p(z)) = -E q [log p(x|z)] + βE q [log q(z|x)] -βE q [log p(z)]

Then any likelihood based computable objective of the KL Info-Autoencoding family is equivalent by elementary transformation to a linear combination of L β-ELBO and L VMI .

We formally prove this theorem in the appendix and give a brief sketch here. This theorem is proved by demonstrating that a set of terms such as E q [log p(x)] can be selected, so that each objective of the KL Information-Autoencoding family is a linear combination of these basic terms. Treating the set of basic terms as basis for a vector space (over functions), each training objective can be represented as a vector in this space. Each family of objectives can be represented as a subset and in some cases as a linear subspace. The key technical difficulty is to show that the following family of objectives correspond to the same linear subspace:

1. Linear combinations of L β-ELBO and L VMI , and their elementary transformations.

2. Likelihood-based objectives (as in Definition 7), and their elementary transformations.

For unary likelihood free computable objectives, we show that a slight generalization of InfoGAN and InfoVAE are the only possible families. This is formally given by the following theorem. Theorem 2. Define the following objectives

3) KL InfoGAN, where λ 1 , λ 2 are any real number

L InfoGAN = -I p (x; z) + D KL (p(z|x) q(z|x)) + λ 1 D KL (p(x) q(x)) + λ 2 D KL (q(x) p(x)) = -E p [log q(z|x)] + λ 1 D KL (p(x) q(x)) + λ 2 D KL (q(x) p(x))

4) KL InfoVAE, where α, λ 1 , λ 2 are any real number

L InfoVAE = αI q (x; z) + D KL (q(x|z) p(x|z)) + λ 1 D KL (q(z) p(z)) + λ 2 D KL (p(z) q(z)) = -E q [log p(x|z)] + (α + 1)E q [log q(z|x)] -(α + 1)E q [log p(z)]+ (λ 1 -α -1)D KL (q(z) p(z)) + λ 2 D KL (p(z) q(z))

Then any unary likelihood free computable objective of the KL Info-Autoencoding family is equivalent by elementary transformation to a linear combination of L InfoGAN and L InfoVAE .

Note that there is a slight asymmetry between InfoGAN and InfoVAE because log q(x) is assumed to be intractable, while log p(z) is assumed tractable. The consequence is that for InfoGAN, the coefficient for the mutual information term I p (x; z) must be tied together with the coefficient for D KL (q(x|z) p(x|z)) to meet the computational constraint. This means that InfoGAN can only maximize mutual information. On the other hand, InfoVAE can either maximize or minimize mutual information.

Finally for the most general case, we show that any binary likelihood free computable member of the KL Information-Autoencoding family can be written as a linear combination of InfoVAE and an information maximizing/minimizing form of BiGAN/ALI/ALICE (Donahue et al., 2016; Dumoulin et al., 2016b; Li et al., 2017a) , which we term InfoBiGAN. Theorem 3. Define the following objective 5) KL InfoBiGAN, where α, λ 1 , λ 2 are any real number

L InfoBiGAN = αI p (x; z) + D KL (p(z|x) q(z|x)) + λ 1 D KL (p(x) q(x)) + λ 2 D KL (q(x) p(x)) = (α + 1)D KL (p(x, z) q(x, z)) + (λ 1 -α -1)D KL (p(x) q(x))+ λ 2 D KL (q(x) p(x)) -αE p [log q(z|x)]

Then any binary likelihood free computable objective of the KL Info-Autoencoding family is equivalent by elementary transformation to a linear combination of L InfoVAE and L InfoBiGAN .

3.4. EXTENSIONS AND LIMITATIONS:

A simple extension beyond KL divergence is possible. For any KL Info-Autoencoding family under the given tractability assumptions, we may add terms in the same tractability category and the resulting objective will still satisfy the same tractability assumptions. For example, for a subset of the KL InfoVAE family, we can add the Maximum Mean Discrepancy (MMD) distance to derive the MMD InfoVAE (Zhao et al., 2017) in Lagrangian form.

L MMD-VAE = αI q (x; z) + D KL (q(x|z) p(x|z)) + λD MMD (p(z) q(z))

and this will still be unary likelihood free computable.

There are two limitations to our analysis. First we do not prove uniqueness results for families with arbitrary divergences. Second we use restricted computability toolboxes and elementary transformations. More general results could be derived by considering a broader family of divergences, computable terms, and transformations. For example, enumeration for models including Renyi divergence variational inference (Li and Turner, 2016) and Stein variational gradient (Liu and Wang, 2016) are interesting directions for future work.

4. APPLICATIONS:

The theory developed in the previous section reveals the tradeoffs between modeling flexibility and computational costs. We may gain additional insight into the behavior of the models by analyzing another trade-off: the Lagrangian dual parameters λ and the information preference parameters α 1 , α 2 specify how different terms are weighted or "priced" when it is not possible to optimally minimize all of them simultaneously. This gives us insight into the behavior of the models under finite capacity conditions.

4.1. CORRECTNESS OF LAGRANGIAN DUAL:

In theory, we should optimize L over both θ, φ (minimize) and λ (maximize) to solve the original primal optimization problem (assuming strong duality holds, or to get the tightest relaxation). However, as we have shown in Section 3, that this is not always possible, as only some choices of λ lead to a dual with a tractable equivalent form (so that some terms cancel out). In addition, many existing models optimize the Lagrangian dual over θ, φ only for a single choice of λ but still achieve impressive log likelihoods or sample quality, indicating that the primal is "approximately" solved.

Intuitively when f (θ, φ) is lower bounded, and when the model families {p θ } and {q φ } are highly flexible, we may achieve both near-optimality for the primal objective f (θ, φ), and near-satisfaction of the feasibility constraints. Therefore it might be acceptable to use a single value of λ, as long as λ > 0. On the other hand, if f (θ, φ) is not lower bounded, regardless of model capacity, for any fixed Lagrangian parameter λ, the Lagrangian can be minimized over θ, φ by only driving down f (θ, φ)

without lowering D(p(x, z)||q(x, z)). It is even possible that D(p(x, z)||q(x, z)) may increase if minimizing f (θ, φ) is in conflict with minimizing D(p(x, z)||q(x, z)).

We verify this intuition experimentally for a simple scenario of a mixture of two Gaussians, where it is clear that a reasonably large neural network is flexible enough to fit the distributions.6 

q(x) ∼ 1/2N (-1, 1/4) + 1/2N (1, 1/4)

We use the β-VAE model class (Higgins et al., 2016) . As discussed in Section 2 and Section 3, this is a Lagrangian dual to the following optimization problem

min (β -1)I q (x; z) s.t. D KL (q(z) p(z)) = 0, D KL (q(x|z) p(x|z)) = 0

When β < 1 the optimization objective (β -1)I q (x; z) is unbounded below for continuous distributions. So optimizing the Lagrangian dual with fixed Lagrangian multipliers may be problematic.

Figure 1 shows the mutual information and negative log likelihood during training. It can be observed that when β < 1 mutual information grows quickly initially (left panel), while at the same time the log-likelihood (right panel) actually gets worse. However, as soon as mutual information stops growing, either due to finite model capacity (solid line), or when we explicitly upper bound it (dashed line), log-likelihood of the data starts to improve (around iteration 20000 for dashed lines, and around iteration 40000 for solid lines). In the latter case, true log-likelihood actually becomes as good as the one for β = 1, which is the original ELBO. However, choosing β < 1 results in significantly larger mutual information I q (x; z), which might be desirable (Chen et al., 2016b; Zhao et al., 2017) .

Figure 1 : Estimated mutual information (Left) vs. true negative log likelihood (Right). Dashed lines correspond to the same objective but with an upper bound on mutual information term. When mutual information increases, log likelihood gets worse. As soon as mutual information stops increasing, log likelihood starts to improve.

This example illustrates what we believe is a general pattern. With highly flexible {p θ } and {q φ }, a fixed (suboptimal) λ can generally be used if the primal objective has an achievable lower bound. As soon as the model approaches this lower bound, it will focus on satisfying the consistency conditions.

4.2. TRADE-OFF FOR INFEASIBLE OPTIMIZATION PROBLEMS:

All objectives of the Info-Autoencoding family are Lagrangian duals of the same primal problem. However, different choices for λ may lead to very different behaviors when optimality cannot be achieved, either due to limited model capacity or optimization difficulties. The Lagrangian parameters λ give the cost of violating each constraint, so that when not all constraints can be satisfied, the model will prioritize constraints with larger Lagrangian coefficients. We use this perspective to analyze the benefit of the additional flexibility gained by giving up likelihood based computability.

For example, we consider the consistency condition D(q(z)||p(z)) = 0, D(q(x|z)||p(x|z)) = 0.

There is an asymmetry between the input space X and the latent feature space Z. X is usually high dimensional, while Z is low dimensional. Therefore any mistake fitting X is likely to be magnified. For example, consider fitting an n dimensional distribution N (0, I) with N ( , I) using KL divergence:

D KL (N (0, I), N ( , I)) = n 2 /2

As n increases with some fixed , the Euclidean distance between the mean of the two distributions is Θ( √ n), yet the corresponding D KL becomes Θ(n). For natural images, the dimensionality of X is often orders of magnitude larger than the dimensionality of Z. This means that the model will tend to sacrifice consistency between p(z) and q(z) in order to fit q(x|z) and p(x|z). We may wish to calibrate this with a larger multiplier for the D(q(z) p(z)) = 0 constraints. We show that doing so improves performance metrics such as log likelihood, but it is only feasible with likelihood free optimization.

We consider the following two classes of models written in Lagrangian dual form:

L β-ELBO = (β -1)I q (x; z) + βD KL (q(z) p(z)) + D KL (q(x|z) p(x|z)) L InfoVAE = L β-ELBO + λD(q(z) p(z))

We use a slightly different but equivalent definition from Theorem 2 to highlight the fact that In-foVAE has an additional free parameter λ. Note that the additional degree of freedom is not possible for β-ELBO as a result of the likelihood based requirement; if we increase the weighting for D KL (q(z) p(z)) we must penalize the mutual information term by the same amount to maintain likelihood-based tractability.

We compute an estimate of the true log likelihood over the binarized MNIST dataset, where the objectives are L β-ELBO and L InfoVAE respectively. Here L β-ELBO only depends on β, whereas L InfoVAE is allowed to select the λ parameter, allowing for an additional degree of freedom7 . We can choose λ > 0 which encourages q(z) and p(z) to be close. Interestingly, even for a very large λ = 5000 the InfoVAE objective is still able to perform reasonably well, which is due to the fact that z has a much smaller dimension than x (50 compared to 784 dimensions). As shown in Figure 2 , the additional degree of freedom provided by λ is able to result in better density estimation in both training and test sets, even when β takes suboptimal values indicating either too much or too little mutual information between x and z. This is surprising given that for λ > 0 or β < 1, we are no longer optimizing over a lower bound on the log-likelihood directly. 

5. CONCLUSION:

In this paper we explored a large family of learning objectives and models that generalize ELBO, β-ELBO, BiGAN/ALI/ALICE, InfoGAN, AVB, CycleGAN/DiscoGAN, AAE/InfoVAE. The Lagrangian form reveals several important properties of the model, while the corresponding "tractable" form tells us how we may optimize it. Under some assumptions, we were able to identify the most general class of objectives that can be optimized using likelihood-based and likelihood-free methods.

Future work includes exploring additional model classes, generalizing the analysis beyond two groups of variables, and extending the tractability analysis beyond KL divergence and elementary transformations.

Since the Information-Autoencoding Family introduces a wide range of valid objective configurations through α 1 , α 2 , λ parameters, it is also interesting to see scalable methods to optimize/search for the most effective set of parameters. Bayesian optimization methods Snoek et al. (2012) and recent advances in neural architecture search (Zoph and Le, 2016; Brock et al., 2017) 

A EXAMPLES OF UNDER-DETERMINATION OF CONSISTENCY CONDITION:

Example 1: Let X and Z be discrete sets with n distinct elements. Then p θ (x|z), q φ (z|x) are tables with n(n-1) degrees of freedom each, giving a total of 2n(n-1) free parameters. The requirement p θ (x) = p data (x) for all x ∈ X involves n -1 linear constraints, while q φ (z|x) = p θ (z|x) for all x adds additional n(n -1) linear constraints. Therefore, there are at least 2n(n -1) -(n + 1)(n -1) = (n -1) 2 free parameters satisfying consistency constraints. Therefore for all n ≥ 2, there are multiple consistent solutions.

Example 2: If for any z, p θ (x|z) is itself a sufficiently flexible distribution family (e.g. represented by PixelCNN), then multiple solutions may exists. For example, one solution could be

p θ (x|z) = q(x), ∀z ∈ support(q(x)) q φ (z|x) = p(z), ∀x ∈ support(p(z))

which means that input x and latent code z are completely independent. This may be an undesirable solution, and several models have been proposed to avoid this (Chen et al., 2016b; Zhao et al., 2017) .

B PROOFS:

Proof of Corollary 1. Assume they are consistent. Then for any x such that p data (x) = 0 and for any z, q φ (x, z) = p data (x)q φ (z|x) = p θ (x)q φ (z|x) = p θ (x)p θ (z|x)

For any x such that p data (x) = 0 and for any z, q φ (x, z) = p data (x)q φ (z|x) = 0

and because 0 = p data (x) = p θ (x) = z p θ (x, z), it follows that p(x, z) = 0 for all z.

Assuming q φ (x, z) = p θ (x, z) we have that

p θ (x) = z p θ (x, z) = z q φ (x, z) = p data (x)

Thus, the conditionals need to match when they are defined, i.e., when p data (x) = p θ (x) = 0.

Proof of Theorem 1 2 3. If all divergences are KL or reverse KL, then we can enumerate over every possible mutual information or divergence term in Definition 5.

EQUATION

Note that we do not consider the terms

EQUATION

because these terms are not in any tractability toolbox in Definition 7, nor can any elementary transformation be applied to them. So any KL Info-Autoencoding objective that contain any of these terms do not have a equivalent computable form. We only consider the 12 possibilities in (16).

To prove this theorem we first map all optimization objectives into a vector space V ⊆ (Θ × Φ → R) with standard function addition and scalar multiplication as operations. Here Θ is the domain for θ, Φ is the domain for φ, and f : Θ × Φ → R represents the objective function.

Define a set of basic terms

p atoms =      log p(x, z) log p(x|z) log p(z|x) log p(x) log p(z)      q atoms =      log q(x, z) log q(x|z) log q(z|x) log q(x) log q(z)     

Define the following set of basis vectors with 20 objective functions

E =    E p(x,z) [p atoms ] E p(x,z) [q atoms ] E q(x,z) [p atoms ] E q(x,z) [q atoms ]    T

It is easy to verify that all objectives of the KL Information Autoencoding family in ( 16), tractable terms in Definition 7 and transformations defined Definition 8 can be written as linear combinations of this set of basis, so they belong to the vector space spanned by these basis vectors. Therefore any optimization objective L in our family can be represented as a vector n ∈ R 20 , where

L = En

For any m ∈ N, the column space of any matrix N ∈ R n×m can be used to represent a family of objectives. Each column n i of N corresponds to an objective En i , and the family is defined by any linear combination of {En 1 , • • • , En m } Therefore we have embedded optimization objectives and linear model families into vectors and column spaces of matrices in this vector space. Now we write each objective and transformation we have discussed as vectors and matrices in this space.

The column space of this matrix form a basis where each column is one of the twelve terms defined in Eq.( 16). Denote the subspace spanned by the columns as R. This subspace represents all objectives that are a linear combination of Eq.( 16), or equivalently, every possible objective in Lagrangian dual form in Definition 5 where each divergence is either KL or reverse KL and does not include terms in (17).

2) Equivalent transformations. Note that all elementary transformations are log linear 

∈ R n = n + b(1, -1, 0, 0, -1, 0, • • • , 0) T      1 1 0 -1 0 0 0 -1 0 0 -1 0 -1 0 1 1 1 -1 0 0 -1 0 -1 -1 0 0 0 1 1 -1 0 0 -1 0 -1 -1 0 1 1 0 -1 0 0 0 -1 0 0 -1 1 -1 0 0      0 0 0 0 1 0 0 -1 0 0 0 0 0 0 -1 0 0 0 0 0 0 -1 0 0 0 -1      0 0 -1 0 1 0 1 1 0 0 0 0 0 0 -1 0 0 -1 0 0 0 0 0 0 -1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 -1 0 0 0 0 0 -1 0 0 0 0 0 0 0 0 0 -1 0 0 0 0 1 0 1 0 0 0 0 0 -1 0 1      0 0 0 1 0 0 0 1 0 -1 0 1 0 0 0 0 0 0 0 0 0 0 -1 0 0 0 -1 0 0 0 0 0 0 0 -1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 -1 0 0 0 0 0 -1 0 0 0 0 0 0 0 0 0 -1 0 0 0 0 1 0 1 0 0 0 0 0 -1 0 1

As before denote A + B as the sum of two subspaces A and B, or all vectors that are linear combinations of vectors in A and vectors in B. Consider first the L β-ELBO family and L V M I family.

EQUATION

Notice that any member of the L β-ELBO family and L V M I family can be transformed by elementary transformation to a Lagrangian dual form Eq.( 18)(20), or to a tractable form that only contains terms in T lb in Eq.( 19)21). This means that for any such member s, there is a r in the space of all Lagrangian dual forms R, and a p in space of all transformations P so that s = r + p. Therefore S β-ELBO + S V M I ⊂ R + P Similarly, for any such member s, there is a t ∈ T lb , and p ∈ P so that s = t + p. so the above subspaces must be equivalent. This is because if the two spaces are not equivalent, and there exists vector v ∈ (R + P) ∩ (T lb + P) but v ∈ S β-ELBO + S VMI + P. However 

To compute the above dimensions, we compute a basis for the each space under discussion. The rank of basis matrix is equal to the dimensionality of its column space. Let A, B be the column space of matrices A, B, then the basis matrix of A + B can be derived by ( A B ). the basis matrix of A ∩ B can be derived by the following process: Compute the null space for

( A B ) u v = 0

Let U, V be basis matrix for the null space of u, v respectively. Then AU = -BV is a basis matrix for A ∩ B.

We can perform this procedure for all the above mentioned subspaces, and derive the following table.

subspace dimension (S β-ELBO + S VMI ) + P 13 (T lb + P) ∩ (R + P) 13 (S InfoGAN ∪ S InfoVAE ) + P 17 (T ulf + P) ∩ (R + P) 17 (S InfoBiGAN ∪ S InfoVAE ) + P 18 (T blf + P) ∩ (R + P) 18

Therefore we have verified that dimensionality match for Eq. (22, 23, 24) . Therefore we have S β-ELBO + S VMI + P = (R + P) ∩ (T lb + P) S InfoGAN + S InfoVAE + P = (R + P) ∩ (T ulf + P) S InfoBiGAN + S InfoVAE + P = (R + P) ∩ (T blf + P) This implies that for any objective r ∈ R of the Info-Autoencoding family, if r can be converted into likelihood based tractable form by elementary transformations, or r ∈ T lb + P, then The latent space z is 1-dimensional, and both p(x|z) and q(z|x) are 1-dimensional Gaussians with arbitrary mean and variance. p(z) is the standard Gaussian, and the mapping z → p(x|z) and x → q(z|x) are both fully connected neural networks with two hidden layers each with 1024 units.

To estimate mutual information we can use -E q [log p(x|z)] because -E q [log p(x|z)] = I q (x; z) -D KL (q(x|z) p(x|z)) so we get a upper bound on the mutual information I q (x; z) that is tight if p(x|z) matches q(x|z) well. We also upper bound -E q [log p(x|z)] to place a explicit upper bound on the mutual information. To compute true log likelihood for this toy problem it is sufficient to estimate the integration by sampling We consider p(x|z) with one layer of latent code, and its corresponding inference network q(z|x).

Both p and q have two hidden layers with 1024 neurons each and tanh activations; the dimension of the latent variable z is 50. We obtain a binarized MNIST dataset with 50000 training examples and 10000 test examples by sampling each dimension from a Bernoulli distribution with probability given by the MNIST dataset. We use importance sampling (instead of AIS (Wu et al., 2016) )

log p(x) = log E p(z|x) p(x, z) q(z|x) ≈ log 1 N i=1,••• ,N,zi∼q(z|x) p(x, z i ) q(z i |x)

to evaluate the negative log-likelihood. Hence the results reported are slightly worse than the state of the art. This does not affect our arguments though, since we are comparing the relative performance of β-ELBO and InfoVAE.

:

under mild technical assumptions, such as the existence of the relevant densities.

Starting from this section we neglect the subscript p θ and q φ to simplify the notation.

To avoid cluttering the notation, we assume p(x, z) and q(x, z) are supported over the entire space X × Z.

The original InfoGAN applies information maximization only to a subset of latent variables z.

To avoid cluttering the notation, if a DKL term has free variables such as z in DKL(q(x|z) p(x|z)), then we take the expectation for that free variable with respect to p when the first distribution of the KL divergence is DKL(p •), and with respect to q otherwise, e.g. E q(z) [DKL(q(x|z) p(x|z))], E p(z)[DKL(p(x|z) q(x|z)].

Experimental setting provided in Appendix.

We include the exact training procedure in the Appendix.

Figures and Tables:

Figure 2: Estimated negative log likelihood under different β and λ values (lower is better). For both training set (left) and test set (right) the extra degree of freedom from λ improves performance (the curresponding curves are below the yellow one, for all values of β).

β-ELBO + S V M I ⊂ T lb + P Therefore S β-ELBO + S V M I ⊂ (R + P) ∩ (T lb + P) In addition we have P ⊂ (R + P) ∩ (T lb + P) so S β-ELBO + S VMI + P ⊂ (R + P) ∩ (T lb + P) (22) Therefore if we can further have dim(S β-ELBO + S VMI + P) = dim ((R + P) ∩ (T lb + P))

β-ELBO + S VMI + P + {v} ⊂ (R + P) ∩ (T lb + P) But dim (S β-ELBO + S VMI + P + {v}) > dim (S β-ELBO + S VMI + P) = dim ((R + P) ∩ (T lb + P)) leading to a contradiction. Similarly we have for the other families S InfoGAN + S InfoVAE + P ⊂ (R + P) ∩ (T ulf + P) (23) S InfoBiGAN + S InfoVAE + P ⊂ (R + P) ∩ (T blf + P)

∈ R ∩ (T lb + P) ⊂ (R + P) ∩ (T lb + P) This means r ∈ S β-ELBO + S VMI + P which implies that r can be converted by elementary transformation into a linear combination of elements in S β-ELBO and elements in S V M I . We can derive identical conclusion for the other two objective families. C EXPERIMENTAL SETUP C.1 EXPERIMENTAL SETUP FOR SECTION 4.1

log p(x) = log E p(z) [p(x|z)] ≈ log 1 N i=1,••• ,N,zi∼p(z) p(x|z i ) C.2 EXPERIMENTAL SETUP FOR SECTION 4.2

Chintala, and L. Bottou. Wasserstein GAN. ArXiv e-prints, January 2017. David Barber and Felix Agakov. The im algorithm: a variational approach to information maximization. In Proceedings of the 16th International Conference on Neural Information Processing Systems, pages 201-208. MIT Press, 2003.

Table content: <table><tr><td>might be useful</td></tr><tr><td>for this purpose.</td></tr><tr><td>REFERENCES</td></tr><tr><td>M. Arjovsky, S.</td></tr></table>

Peer review (iclr2018):
 Update after rebuttal ========== Thanks for your response on my questions. The stated usefulness of the method unfortunately do not answer my worry about the significance. It remains unclear to me how much "real" difference the presented results would make to advance the existing work on generative models. Also, the authors did not promised any major changes in the final version in this direction, which is why I have reduced my score. I do believe that this work could be useful and should be resubmitted. There are two main things to improve. First, the paper need more work on improving the clarity. Second, more work needs to be added to show that the paper will make a real difference to advance/improve existing methods. ========== Before rebuttal ========== This paper proposes an optimization problem whose Lagrangian duals contain many existing objective functions for generative models. Using this framework, the paper tries to generalize the optimization problems by defining computationally-tractable family which can be expressed in terms of existing objective functions. The paper has interesting elements and the results are original. The main issue is that the significance is unclear. The writing in Section 3 is unclear for me, which further made it challenging to understand the consequences of the theorems presented in that section. Here is a big-picture question that I would like to know answer for. Do the results of sec 3 help us identify a more useful/computationally tractable model than exiting approaches? Clarification on this will help me evaluate the significance of the paper. I have three main clarification points. First, what is the importance of T1, T2, and T3 classes defined in Def. 7, i.e., why are these classes useful in solving some problems? Second, is the opposite relationship in Theorem 1, 2, and 3 true as well, e.g., is every linear combination of beta-ELBO and VMI is equivalent to a likelihood-based computable-objective of KL info-encoding family? Is the same true for other theorems? Third, the objective of section 3 is to show that "only some choices of lambda lead to a dual with a tractable equivalent form". Could you rewrite the theorems so that they truly reflect this, rather than stating something which only indirectly imply the main claim of the paper. Some small comments: - Eq. 4. It might help to define MI to remind readers. - After Eq. 7, please add a proof (may be in the Appendix). It is not that straightforward to see this. Also, I suppose you are saying Eq. 3 but with f from Eq. 4. - Line after Eq. 8, D_i is "one" of the following... Is it always the same D_i for all i or it could be different? Make this more clear to avoid confusion. - Last line in Para after Eq. 15, "This neutrality corresponds to the observations made in.." It might be useful to add a line explaining that particular "observation" - Def. 7, the names did not make much sense to me. You can add a line explaining why this name is chosen. - Def. 8, the last equation is unclear. Does the first equivalence impy the next one? - Writing in Sec. 3.3 can be improved. e.g., "all linear operations on log prob." is very unclear, "stated computational constraints" which constraints?