Title: No title found

Abstract: should be indented 1/2 inch (3 picas) on both left and righthand margins. Use 10 point type, with a vertical spacing of 11 points.

:

The file iclr2020_conference.pdf contains these instructions and illustrates the various formatting requirements your ICLR paper must satisfy. Submissions must be made using L A T E X and the style files iclr2020_conference.sty and iclr2020_conference.bst (to be used with L A T E X2e). The file iclr2020_conference.tex may be used as a "shell" for writing your paper. All you have to do is replace the author, title, abstract, and text of the paper with your own. The formatting instructions contained in these style files are summarized in sections 2, 3, and 4 below.

2. GENERAL FORMATTING INSTRUCTIONS:

The text must be confined within a rectangle 5.5 inches (33 picas) wide and 9 inches (54 picas) long. The left margin is 1.5 inch (9 picas). Use 10 point type with a vertical spacing of 11 points. Times New Roman is the preferred typeface throughout. Paragraphs are separated by 1/2 line space, with no indentation.

Paper title is 17 point, in small caps and left-aligned. All pages should start at 1 inch (6 picas) from the top of the page. Authors' names are set in boldface, and each name is placed above its corresponding address. The lead author's name is to be listed first, and the co-authors' names are set to follow. Authors sharing the same address can be on the same line.

Please pay special attention to the instructions in section 4 regarding figures, tables, acknowledgments, and references.

The recommended paper length is 8 pages, with unlimited additional pages for citations. There will be a strict upper limit of 10 pages for the main text. Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages. Authors may use as many pages of appendices (after the bibliography) as they wish, but reviewers are not required to read these.

3. HEADINGS: FIRST LEVEL:

First level headings are in small caps, flush left and in point size 12. One line space before the first level heading and 1/2 line space after the first level heading.

3.1. HEADINGS: SECOND LEVEL:

Second level headings are in small caps, flush left and in point size 10. One line space before the second level heading and 1/2 line space after the second level heading.

3.1.1. HEADINGS: THIRD LEVEL:

Third level headings are in small caps, flush left and in point size 10. One line space before the third level heading and 1/2 line space after the third level heading.

4. CITATIONS, FIGURES, TABLES, REFERENCES:

These instructions apply to everyone, regardless of the formatter being used.

4.1. CITATIONS WITHIN THE TEXT:

Citations within the text should be based on the natbib package and include the authors' last names and year (with the "et al." construct for more than two authors). When the authors or the publication are included in the sentence, the citation should not be in parenthesis using \citet{} (as in "See Hinton et al. (2006) for more information."). Otherwise, the citation should be in parenthesis using \citep{} (as in "Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007) .").

The corresponding references are to be listed in alphabetical order of authors, in the REFERENCES section. As to the format of the references themselves, any style is acceptable as long as it is used consistently.

4.2. FOOTNOTES:

Indicate footnotes with a number1 in the text. Place the footnotes at the bottom of the page on which they appear. Precede the footnote with a horizontal rule of 2 inches (12 picas). 

matrix J ∈ R m×n of f : R n → R m ∇ 2 x f (x) or H(f )(x)

The Hessian matrix of f at input point x f (x)dx Definite integral over the entire domain of x S f (x)dx Definite integral with respect to x over the set S Probability and Information Theory

P (a):

A probability distribution over a discrete variable p(a)

A probability distribution over a continuous variable, or over a variable whose type has not been specified a ∼ P Random variable a has distribution P A number of width problems arise when LaTeX cannot properly hyphenate a line. Please give LaTeX hyphenation hints using the \-command.

E x∼P [f (x)] or Ef (x) Expectation of f (x) with respect to P (x) Var(f (x)) Variance of f (x) under P (x) Cov(f (x), g(x)) Covariance of f (x)

AUTHOR CONTRIBUTIONS:

If you'd like to, you may include a section for author contributions as is done in many journals. This is optional and at the discretion of the authors.

Sample of the first footnote

Sample of the second footnote

Figures and Tables:

2

All artwork must be neat, clean, and legible. Lines should be dark enough for purposes of reproduction; art work should not be hand-drawn. The figure number and caption always appear after the figure. Place one line space before the figure caption, and one line space after the figure. The figure caption is lower case (except for first word and proper nouns); figures are numbered consecutively.

Sample table title Make sure the figure caption does not get separated from the figure. Leave sufficient space to avoid splitting the figure and figure caption. You may use color figures. However, it is best for the figure captions and the paper body to make sense if the paper is printed either in black/white or in color. All tables must be centered, neat, clean and legible. Do not use hand-drawn tables. The table number and title always appear before the table. See Table 1. Place one line space before the table title, one line space after the table title, and one line space after the table. The table title must be lower case (except for first word and proper nouns); tables are numbered consecutively.

Table content: <table><tr><td>PART</td><td>DESCRIPTION</td></tr><tr><td colspan="2">Dendrite Input terminal</td></tr><tr><td>Axon</td><td>Output terminal</td></tr><tr><td>Soma</td><td>Cell body (contains cell nucleus)</td></tr></table>

Do not change any aspects of the formatting parameters in the style files. In particular, do not modify the width or length of the rectangle the text should fit into, and do not change font sizes (except perhaps in the REFERENCES section; see below). Please note that pages should be numbered.Most of the margin problems come from figures positioned by hand using \special or other commands. We suggest using the command \includegraphics from the graphicx package. Always specify the figure width as a multiple of the line width as in the example below using .eps graphics \usepackage[dvips]{graphicx} ... \includegraphics[width=0.8\linewidth]{myfile.eps} or \usepackage[pdftex]{graphicx} ... \includegraphics[width=0.8\linewidth]{myfile.pdf} for .pdf graphics. See section 4.4 in the graphics bundle documentation (http://www.ctan. org/tex-archive/macros/latex/required/graphics/grfguide.ps)

Table content: <table><tr><td colspan="2">6 FINAL INSTRUCTIONS</td><td/></tr><tr><td colspan="3">7 PREPARING POSTSCRIPT OR PDF FILES</td></tr><tr><td colspan="3">Please prepare PostScript or PDF files with paper size "US Letter", and not, for example, "A4". The</td></tr><tr><td colspan="3">-t letter option on dvips will produce US Letter files.</td></tr><tr><td colspan="3">Consider directly generating PDF files using pdflatex (especially if you are a MiKTeX user).</td></tr><tr><td colspan="3">PDF figures must be substituted for EPS figures, however.</td></tr><tr><td colspan="3">Otherwise, please generate your PostScript and PDF files with the following commands:</td></tr><tr><td colspan="3">dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps</td></tr><tr><td colspan="2">ps2pdf mypaper.ps mypaper.pdf</td><td/></tr><tr><td colspan="2">7.1 MARGINS IN LATEX</td><td/></tr><tr><td>H(x)</td><td colspan="2">Shannon entropy of the random variable x</td></tr><tr><td>D KL (P Q)</td><td colspan="2">Kullback-Leibler divergence of P and Q</td></tr><tr><td>N (x; µ, Σ)</td><td colspan="2">Gaussian distribution over x with mean µ and covariance</td></tr><tr><td/><td>Σ</td><td/></tr><tr><td/><td colspan="2">Functions</td></tr><tr><td>f : A → B</td><td colspan="2">The function f with domain A and range B</td></tr><tr><td>f • g</td><td colspan="2">Composition of the functions f and g</td></tr><tr><td>f (x; θ)</td><td colspan="2">A function of x parametrized by θ. (Sometimes we write</td></tr><tr><td/><td colspan="2">f (x) and omit the argument θ to lighten notation)</td></tr><tr><td>log x</td><td colspan="2">Natural logarithm of x</td></tr><tr><td>σ(x)</td><td>Logistic sigmoid,</td><td>1 1 + exp(-x)</td></tr><tr><td>ζ(x)</td><td colspan="2">Softplus, log(1 + exp(x))</td></tr><tr><td>||x|| p</td><td>L p norm of x</td><td/></tr><tr><td>||x||</td><td>L 2 norm of x</td><td/></tr><tr><td>x +</td><td colspan="2">Positive part of x, i.e., max(0, x)</td></tr><tr><td>1 condition</td><td colspan="2">is 1 if the condition is true, 0 otherwise</td></tr></table>

Peer review (iclr2020):
Mitigating Posterior Collapse in Strongly Conditioned Variational Autoencoders 1. Summary The paper considers strongly conditioned generative models -- a class of models in which the decoder can be powerful and the condition inputs are informative enough such that latent variables could be ignored. The paper proposes an objective function and a particular parameterisation of the variational distribution such that the latent variables explicitly depend on the input conditions. In particular, the paper proposes learning a vanilla generative model to explain the conditions, then using parameters of the variational distribution of this model to be in the prior for the latent variables of the data, as well as to constrain the variational distribution over the latent variables for the data. The proposed model + objective function are evaluated on a suite of tasks involving conditional motion generation and image captioning. The core evaluation metrics are to assess diversity of generated samples, as this indicates the latent variables are not switched off after training. 2. Though the proposed approach is sound and the presentation/writing is clear, I’m leaning toward *weak reject* because of the following reasons: a. I like maths and thus in general like to take people’s work and reinterpret/understand them by writing down the underlying probabilistic model, or the underlying variational distribution. First, the paper says that it considers conditional generative models -- I don’t think this is 100% precise, as the final objective is a variational bound to the log joint p(x, c), not just p(x | c), where x is the data and c is the conditions. Now let’s dig a bit deeper and write down the joint of everything, factorized as considered in this paper (where z_x and z_c are the latent variables for the data and conditions respectively): p(x, c, z_x, z_c) = p(x | z_x) p(c | z_c) p(z_x | z_c) p(z_c) Now consider the following variational approximation, which is what the paper implies in the explanation and uses as listed in the algorithm in the appendix: q(z_x, z_c) = q(z_c) q(z_x | z_c) The paper uses q(z_c) = N (z_c; mu_c, sigma_c^2) where mu_c and sigma_c are outputs of an encoder -- this is fine. The issue is that the paper uses a deterministic mapping z_x = mu + sigma z_c, or q(z_x | z_c) = delta (z_x = mu + sigma z_c). The paper uses this explicitly: “z is sampled given z_c” (to be precise, there is no sampling given z_c) and in algo 2 in the appendix. The paper correctly states the marginal density q(z_x) (which is another normal), but let’s just focus on this delta density for now. Now let’s us write down the variational objective (again, to deal with the joint, not the conditional): F = E_{q(z_c) q(z_x | z_c)} [ log p(x | z_x) ] + E_{q(z_c)} [ log p(c | z_c) ] - KL( q(z_c) || p(z_c) ) - E_{q(z_c)} [ KL( q(z_x | z_c) || p(z_x | z_c) ) ] Now the fourth terms is problematic as a density in the KL is a delta. So KL term presented in the paper is essentially approximating this by KL( q(z_x) || p(z_x | z_c) ) and ignoring the expectation wrt q(z_c). I don’t think this is bad or incorrect, just that I wish the paper presents their approach this way to make things clearly, rather than from presenting heuristics + designing losses. Now, here is my derivation of what the paper probably means in the first place (which is unfortunately not what the paper ends up doing because of the computational graph presented in the paper). So instead of choosing the variational distribution like above, the joint variational distribution is a mean-field one: q(z_c, z_x) = q(z_c) q(z_x) where q(z_c) = N (z_c; mu_c, sigma_c^2) and q(z_x) = N (z_x; mu_x + sigma mu_c, sigma^2 sigma_c^2). Note that there is no explicit conditioning, just the parameters are tied. This also means that the sampling of z_c and z_x should not be tied/sequential as in the paper. Now the paper also chooses p(z_c) = N (z_c; 0, I) and p(z_x) = N(z_x; mu_c, sigma_c^2). Now write down the variational bound, we will get the final bound in the paper in eq 8. So in short, there is a discrepancy in what the paper presents and the actual computation graph, with a valid variational bound in a generative model. As I said above, I don’t think this invalidates the results in the paper, just means that the theoretical foundation is not clean as I would like, in particular, as the paper seems to criticize previous methods: “Any annealing weight that does not become, and remain equal to one at some point during training yields an improper statistical model; weakening the decoder tends to degrade the quality of the generated samples; changing the objective does not optimize the true variational lower bound”. b. Having motivated the need of the proposed additional architecture and objective, the paper doesn’t include a comparison to vanilla conditional VAE, or VAE for both data and conditions in which the latent variables collapse to the prior, especially in the motion prediction example. The ELBO/log-likelihood reported is also not comparable (because of some models the conditional while this paper models the joint, as detailed above). It would also be interesting to see the performance of a conditional VAE with a variational distribution parameterized like the marginal considered here: q(z_x) = N (z_x; mu_x + sigma mu_c, sigma^2 sigma_c^2), i.e. the parameters of z_x explicitly depends on c and there is no concatenation of c and z_x to generate the data. 3. Additional points: just to be pedantic, what is the goal of the latent variables here given the observation likelihood can be explained by the condition inputs and that the objective is to maximise p( x | c )? Does this mean posterior collapse is a good thing here? Or are we essentially overfitting? Would early stopping help with diversity? It would be good to have some error bars for the numbers reported. Why z = z_c in the testing phase in algorithm 2 in the appendix? This seems strange.