Title: No title found

Abstract: should be indented 1/2 inch (3 picas) on both left and righthand margins. Use 10 point type, with a vertical spacing of 11 points.

:

The file iclr2020_conference.pdf contains these instructions and illustrates the various formatting requirements your ICLR paper must satisfy. Submissions must be made using L A T E X and the style files iclr2020_conference.sty and iclr2020_conference.bst (to be used with L A T E X2e). The file iclr2020_conference.tex may be used as a "shell" for writing your paper. All you have to do is replace the author, title, abstract, and text of the paper with your own. The formatting instructions contained in these style files are summarized in sections 2, 3, and 4 below.

2. GENERAL FORMATTING INSTRUCTIONS:

The text must be confined within a rectangle 5.5 inches (33 picas) wide and 9 inches (54 picas) long. The left margin is 1.5 inch (9 picas). Use 10 point type with a vertical spacing of 11 points. Times New Roman is the preferred typeface throughout. Paragraphs are separated by 1/2 line space, with no indentation.

Paper title is 17 point, in small caps and left-aligned. All pages should start at 1 inch (6 picas) from the top of the page. Authors' names are set in boldface, and each name is placed above its corresponding address. The lead author's name is to be listed first, and the co-authors' names are set to follow. Authors sharing the same address can be on the same line.

Please pay special attention to the instructions in section 4 regarding figures, tables, acknowledgments, and references.

The recommended paper length is 8 pages, with unlimited additional pages for citations. There will be a strict upper limit of 10 pages for the main text. Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages. Authors may use as many pages of appendices (after the bibliography) as they wish, but reviewers are not required to read these.

3. HEADINGS: FIRST LEVEL:

First level headings are in small caps, flush left and in point size 12. One line space before the first level heading and 1/2 line space after the first level heading.

3.1. HEADINGS: SECOND LEVEL:

Second level headings are in small caps, flush left and in point size 10. One line space before the second level heading and 1/2 line space after the second level heading.

3.1.1. HEADINGS: THIRD LEVEL:

Third level headings are in small caps, flush left and in point size 10. One line space before the third level heading and 1/2 line space after the third level heading.

4. CITATIONS, FIGURES, TABLES, REFERENCES:

These instructions apply to everyone, regardless of the formatter being used.

4.1. CITATIONS WITHIN THE TEXT:

Citations within the text should be based on the natbib package and include the authors' last names and year (with the "et al." construct for more than two authors). When the authors or the publication are included in the sentence, the citation should not be in parenthesis using \citet{} (as in "See Hinton et al. (2006) for more information."). Otherwise, the citation should be in parenthesis using \citep{} (as in "Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007) .").

The corresponding references are to be listed in alphabetical order of authors, in the REFERENCES section. As to the format of the references themselves, any style is acceptable as long as it is used consistently.

4.2. FOOTNOTES:

Indicate footnotes with a number1 in the text. Place the footnotes at the bottom of the page on which they appear. Precede the footnote with a horizontal rule of 2 inches (12 picas). 

matrix J ∈ R m×n of f : R n → R m ∇ 2 x f (x) or H(f )(x)

The Hessian matrix of f at input point x f (x)dx Definite integral over the entire domain of x S f (x)dx Definite integral with respect to x over the set S Probability and Information Theory

P (a):

A probability distribution over a discrete variable p(a)

A probability distribution over a continuous variable, or over a variable whose type has not been specified a ∼ P Random variable a has distribution P A number of width problems arise when LaTeX cannot properly hyphenate a line. Please give LaTeX hyphenation hints using the \-command.

E x∼P [f (x)] or Ef (x) Expectation of f (x) with respect to P (x) Var(f (x)) Variance of f (x) under P (x) Cov(f (x), g(x)) Covariance of f (x)

AUTHOR CONTRIBUTIONS:

If you'd like to, you may include a section for author contributions as is done in many journals. This is optional and at the discretion of the authors.

Sample of the first footnote

Sample of the second footnote

Figures and Tables:

2

All artwork must be neat, clean, and legible. Lines should be dark enough for purposes of reproduction; art work should not be hand-drawn. The figure number and caption always appear after the figure. Place one line space before the figure caption, and one line space after the figure. The figure caption is lower case (except for first word and proper nouns); figures are numbered consecutively.

Sample table title Make sure the figure caption does not get separated from the figure. Leave sufficient space to avoid splitting the figure and figure caption. You may use color figures. However, it is best for the figure captions and the paper body to make sense if the paper is printed either in black/white or in color. All tables must be centered, neat, clean and legible. Do not use hand-drawn tables. The table number and title always appear before the table. See Table 1. Place one line space before the table title, one line space after the table title, and one line space after the table. The table title must be lower case (except for first word and proper nouns); tables are numbered consecutively.

Table content: <table><tr><td>PART</td><td>DESCRIPTION</td></tr><tr><td colspan="2">Dendrite Input terminal</td></tr><tr><td>Axon</td><td>Output terminal</td></tr><tr><td>Soma</td><td>Cell body (contains cell nucleus)</td></tr></table>

Do not change any aspects of the formatting parameters in the style files. In particular, do not modify the width or length of the rectangle the text should fit into, and do not change font sizes (except perhaps in the REFERENCES section; see below). Please note that pages should be numbered.Most of the margin problems come from figures positioned by hand using \special or other commands. We suggest using the command \includegraphics from the graphicx package. Always specify the figure width as a multiple of the line width as in the example below using .eps graphics \usepackage[dvips]{graphicx} ... \includegraphics[width=0.8\linewidth]{myfile.eps} or \usepackage[pdftex]{graphicx} ... \includegraphics[width=0.8\linewidth]{myfile.pdf} for .pdf graphics. See section 4.4 in the graphics bundle documentation (http://www.ctan. org/tex-archive/macros/latex/required/graphics/grfguide.ps)

Table content: <table><tr><td colspan="2">6 FINAL INSTRUCTIONS</td><td/></tr><tr><td colspan="3">7 PREPARING POSTSCRIPT OR PDF FILES</td></tr><tr><td colspan="3">Please prepare PostScript or PDF files with paper size "US Letter", and not, for example, "A4". The</td></tr><tr><td colspan="3">-t letter option on dvips will produce US Letter files.</td></tr><tr><td colspan="3">Consider directly generating PDF files using pdflatex (especially if you are a MiKTeX user).</td></tr><tr><td colspan="3">PDF figures must be substituted for EPS figures, however.</td></tr><tr><td colspan="3">Otherwise, please generate your PostScript and PDF files with the following commands:</td></tr><tr><td colspan="3">dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps</td></tr><tr><td colspan="2">ps2pdf mypaper.ps mypaper.pdf</td><td/></tr><tr><td colspan="2">7.1 MARGINS IN LATEX</td><td/></tr><tr><td>H(x)</td><td colspan="2">Shannon entropy of the random variable x</td></tr><tr><td>D KL (P Q)</td><td colspan="2">Kullback-Leibler divergence of P and Q</td></tr><tr><td>N (x; µ, Σ)</td><td colspan="2">Gaussian distribution over x with mean µ and covariance</td></tr><tr><td/><td>Σ</td><td/></tr><tr><td/><td colspan="2">Functions</td></tr><tr><td>f : A → B</td><td colspan="2">The function f with domain A and range B</td></tr><tr><td>f • g</td><td colspan="2">Composition of the functions f and g</td></tr><tr><td>f (x; θ)</td><td colspan="2">A function of x parametrized by θ. (Sometimes we write</td></tr><tr><td/><td colspan="2">f (x) and omit the argument θ to lighten notation)</td></tr><tr><td>log x</td><td colspan="2">Natural logarithm of x</td></tr><tr><td>σ(x)</td><td>Logistic sigmoid,</td><td>1 1 + exp(-x)</td></tr><tr><td>ζ(x)</td><td colspan="2">Softplus, log(1 + exp(x))</td></tr><tr><td>||x|| p</td><td>L p norm of x</td><td/></tr><tr><td>||x||</td><td>L 2 norm of x</td><td/></tr><tr><td>x +</td><td colspan="2">Positive part of x, i.e., max(0, x)</td></tr><tr><td>1 condition</td><td colspan="2">is 1 if the condition is true, 0 otherwise</td></tr></table>

Peer review (iclr2020):
The paper tries to address a problem they call *posterior collapsed* in which conditional VAEs tend to ignore the latent variable and learn an almost-deterministic mapping from conditioning to samples. The paper poses a potentially algorithmic approach. However, the paper lacks evidence to validate most of its claims, in particular it fails at 1) Establishing that the problem actually exists, by theory or experiments (or carefully selected references), 2) Establishing the existence of a causal relationship between conditioning and latent variable being independent and posterior collapse, again either by theory and / or targeted experiments 3) Validating claims about the method*s performance empirically. Here go my detailed comments in these regards: Before going into the comments, I will use Q to denote the joint distribution (x, z, c) arising from the prior Q(z) ~ Gaussian, Q(c) ~ whatever the conditioning distribution is and independent of z, and Q(x |z, c) being the decoder. As well, I will use P to denote the joint (x, z, c) where P(x, c) is the joint of data and conditioning distributions, and P(z | x, c) is the encoder. I*m sorry this is different from the paper but I find it a lot easier to elaborate my review with this notation. I will refer to posterior collapse as in the abstract, namely Q(x | z, c) = Q(x | z*, c) = Q(x | c), or alternatively Q(x | c) being deterministic. 1) The authors state that Q(x | c) becomes deterministic in many tasks. However, this is not a priori a problem if there indeed P(x | c) is deterministic. The authors thus make a really bad job at motivating why this might be an important problem, and no experiments are provided show that this happens in tasks where P(x | c) is non-deterministic (aside from the unsubstantiated claim without a reference or experiment that this must happen in human motion prediction). 2) The authors imply many times that posterior colapse (Q(x | c) deterministic) is somewhat equivalent to the encoder P(z | x, c) ignoring x, i.e. P(z | x, c) = q( z ) [i.e. the KL in equation 2 is zero when c and z are independent for the prior]. The authors never justify this claim in any way, and it is essential since the method indirectly tries to address posterior collapse by preventing the encoder from satisfying P(z | x, c) = q(z). This to me is the biggest flaw of the paper. In the first paragraph when they discuss the non-conditional case and say *Training such models however may often result in posterior colapse: the posterior distribution q(z | x) of the latent variable carrying no information about the input. In other words, the model learns to ignore the latent variable*. First of all, it is unclear wether q(z | x) is the *approximate* posterior given by the encoder or the true posterior. If it is the approximate posterior given by the encoder, P(z | x) in my notation, then this claim seems false. This is a claim of equivalence between P(z | x) = P(z) for all x (the first sentence in this quote, i.e. the encoder ignoring x and matching the prior) and the second sentence in this paragraph, which according to the abstract should be Q(x | z) = Q(x) for all z, i.e. the decoder ignoring z. Surely this doesn*t follow from Bayes* rule, or at least I cannot see how. Namely, Q(x | z) = Q(z | x) Q(x) / Q(z). If it was the case that Q(z | x) = Q(z) then this would be true because the denominator would cancel with Q(z | x). However, this would require the posterior of the encoder matching the prior, not the decoder matching the prior! Unless we assume Q(z | x) = P(z | x) which is false unless the approximate posterior matches the true posterior, the two things (encoder ignoring x, decoder ignoring z) are far from equivalent or causally connected. This makes most of the paper fundamentally flawed in my opinion. In the same way, the link between the KL term in equation 2 being zero and Q(x | z, c) = Q(x | c) is false by the exact same argument. Q(x | z, c) = Q(x | c) would require Q(z | x, c) = Q(z | c), not P(z | x, c) = Q(z | c) which is the KL term being 0. 3) The authors employ very ad-hoc techniques for measuring quality and diversity of the samples. For instance, they use expected distance between samples to assess diversity, which fails to capture multimodal behaviour, and for quality they train a discriminator after training the generative model. The discriminator training after the generative model has been finalized tends to be meaningless, since the accuracy of the discriminator doesn*t depend on how close the generated samples are to the real data (or how close the fake and real data manifolds are), see https://arxiv.org/abs/1701.04862 . Furthermore, the authors fail to make experiments showing that these problems have indeed a non-deterministic P(x | c), thus the importance of posterior collapse is impossible to assess in these tasks.