Title: No title found

Abstract: should be indented 1/2 inch (3 picas) on both left and righthand margins. Use 10 point type, with a vertical spacing of 11 points.

:

The file iclr2020_conference.pdf contains these instructions and illustrates the various formatting requirements your ICLR paper must satisfy. Submissions must be made using L A T E X and the style files iclr2020_conference.sty and iclr2020_conference.bst (to be used with L A T E X2e). The file iclr2020_conference.tex may be used as a "shell" for writing your paper. All you have to do is replace the author, title, abstract, and text of the paper with your own. The formatting instructions contained in these style files are summarized in sections 2, 3, and 4 below.

2. GENERAL FORMATTING INSTRUCTIONS:

The text must be confined within a rectangle 5.5 inches (33 picas) wide and 9 inches (54 picas) long. The left margin is 1.5 inch (9 picas). Use 10 point type with a vertical spacing of 11 points. Times New Roman is the preferred typeface throughout. Paragraphs are separated by 1/2 line space, with no indentation.

Paper title is 17 point, in small caps and left-aligned. All pages should start at 1 inch (6 picas) from the top of the page. Authors' names are set in boldface, and each name is placed above its corresponding address. The lead author's name is to be listed first, and the co-authors' names are set to follow. Authors sharing the same address can be on the same line.

Please pay special attention to the instructions in section 4 regarding figures, tables, acknowledgments, and references.

The recommended paper length is 8 pages, with unlimited additional pages for citations. There will be a strict upper limit of 10 pages for the main text. Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages. Authors may use as many pages of appendices (after the bibliography) as they wish, but reviewers are not required to read these.

3. HEADINGS: FIRST LEVEL:

First level headings are in small caps, flush left and in point size 12. One line space before the first level heading and 1/2 line space after the first level heading.

3.1. HEADINGS: SECOND LEVEL:

Second level headings are in small caps, flush left and in point size 10. One line space before the second level heading and 1/2 line space after the second level heading.

3.1.1. HEADINGS: THIRD LEVEL:

Third level headings are in small caps, flush left and in point size 10. One line space before the third level heading and 1/2 line space after the third level heading.

4. CITATIONS, FIGURES, TABLES, REFERENCES:

These instructions apply to everyone, regardless of the formatter being used.

4.1. CITATIONS WITHIN THE TEXT:

Citations within the text should be based on the natbib package and include the authors' last names and year (with the "et al." construct for more than two authors). When the authors or the publication are included in the sentence, the citation should not be in parenthesis using \citet{} (as in "See Hinton et al. (2006) for more information."). Otherwise, the citation should be in parenthesis using \citep{} (as in "Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007) .").

The corresponding references are to be listed in alphabetical order of authors, in the REFERENCES section. As to the format of the references themselves, any style is acceptable as long as it is used consistently.

4.2. FOOTNOTES:

Indicate footnotes with a number1 in the text. Place the footnotes at the bottom of the page on which they appear. Precede the footnote with a horizontal rule of 2 inches (12 picas). 

matrix J ∈ R m×n of f : R n → R m ∇ 2 x f (x) or H(f )(x)

The Hessian matrix of f at input point x f (x)dx Definite integral over the entire domain of x S f (x)dx Definite integral with respect to x over the set S Probability and Information Theory

P (a):

A probability distribution over a discrete variable p(a)

A probability distribution over a continuous variable, or over a variable whose type has not been specified a ∼ P Random variable a has distribution P A number of width problems arise when LaTeX cannot properly hyphenate a line. Please give LaTeX hyphenation hints using the \-command.

E x∼P [f (x)] or Ef (x) Expectation of f (x) with respect to P (x) Var(f (x)) Variance of f (x) under P (x) Cov(f (x), g(x)) Covariance of f (x)

AUTHOR CONTRIBUTIONS:

If you'd like to, you may include a section for author contributions as is done in many journals. This is optional and at the discretion of the authors.

Sample of the first footnote

Sample of the second footnote

Figures and Tables:

2

All artwork must be neat, clean, and legible. Lines should be dark enough for purposes of reproduction; art work should not be hand-drawn. The figure number and caption always appear after the figure. Place one line space before the figure caption, and one line space after the figure. The figure caption is lower case (except for first word and proper nouns); figures are numbered consecutively.

Sample table title Make sure the figure caption does not get separated from the figure. Leave sufficient space to avoid splitting the figure and figure caption. You may use color figures. However, it is best for the figure captions and the paper body to make sense if the paper is printed either in black/white or in color. All tables must be centered, neat, clean and legible. Do not use hand-drawn tables. The table number and title always appear before the table. See Table 1. Place one line space before the table title, one line space after the table title, and one line space after the table. The table title must be lower case (except for first word and proper nouns); tables are numbered consecutively.

Table content: <table><tr><td>PART</td><td>DESCRIPTION</td></tr><tr><td colspan="2">Dendrite Input terminal</td></tr><tr><td>Axon</td><td>Output terminal</td></tr><tr><td>Soma</td><td>Cell body (contains cell nucleus)</td></tr></table>

Do not change any aspects of the formatting parameters in the style files. In particular, do not modify the width or length of the rectangle the text should fit into, and do not change font sizes (except perhaps in the REFERENCES section; see below). Please note that pages should be numbered.Most of the margin problems come from figures positioned by hand using \special or other commands. We suggest using the command \includegraphics from the graphicx package. Always specify the figure width as a multiple of the line width as in the example below using .eps graphics \usepackage[dvips]{graphicx} ... \includegraphics[width=0.8\linewidth]{myfile.eps} or \usepackage[pdftex]{graphicx} ... \includegraphics[width=0.8\linewidth]{myfile.pdf} for .pdf graphics. See section 4.4 in the graphics bundle documentation (http://www.ctan. org/tex-archive/macros/latex/required/graphics/grfguide.ps)

Table content: <table><tr><td colspan="2">6 FINAL INSTRUCTIONS</td><td/></tr><tr><td colspan="3">7 PREPARING POSTSCRIPT OR PDF FILES</td></tr><tr><td colspan="3">Please prepare PostScript or PDF files with paper size "US Letter", and not, for example, "A4". The</td></tr><tr><td colspan="3">-t letter option on dvips will produce US Letter files.</td></tr><tr><td colspan="3">Consider directly generating PDF files using pdflatex (especially if you are a MiKTeX user).</td></tr><tr><td colspan="3">PDF figures must be substituted for EPS figures, however.</td></tr><tr><td colspan="3">Otherwise, please generate your PostScript and PDF files with the following commands:</td></tr><tr><td colspan="3">dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps</td></tr><tr><td colspan="2">ps2pdf mypaper.ps mypaper.pdf</td><td/></tr><tr><td colspan="2">7.1 MARGINS IN LATEX</td><td/></tr><tr><td>H(x)</td><td colspan="2">Shannon entropy of the random variable x</td></tr><tr><td>D KL (P Q)</td><td colspan="2">Kullback-Leibler divergence of P and Q</td></tr><tr><td>N (x; µ, Σ)</td><td colspan="2">Gaussian distribution over x with mean µ and covariance</td></tr><tr><td/><td>Σ</td><td/></tr><tr><td/><td colspan="2">Functions</td></tr><tr><td>f : A → B</td><td colspan="2">The function f with domain A and range B</td></tr><tr><td>f • g</td><td colspan="2">Composition of the functions f and g</td></tr><tr><td>f (x; θ)</td><td colspan="2">A function of x parametrized by θ. (Sometimes we write</td></tr><tr><td/><td colspan="2">f (x) and omit the argument θ to lighten notation)</td></tr><tr><td>log x</td><td colspan="2">Natural logarithm of x</td></tr><tr><td>σ(x)</td><td>Logistic sigmoid,</td><td>1 1 + exp(-x)</td></tr><tr><td>ζ(x)</td><td colspan="2">Softplus, log(1 + exp(x))</td></tr><tr><td>||x|| p</td><td>L p norm of x</td><td/></tr><tr><td>||x||</td><td>L 2 norm of x</td><td/></tr><tr><td>x +</td><td colspan="2">Positive part of x, i.e., max(0, x)</td></tr><tr><td>1 condition</td><td colspan="2">is 1 if the condition is true, 0 otherwise</td></tr></table>

Peer review (iclr2020):
Summary This paper deals with the posterior collapse issue, i.e., when the posterior is set equal to the prior, which may affect variational autoencoders (VAEs) in practice. Compared to existing literature on this topic, the focus here is on conditional VAE, where observations “x” are modeled conditional on a latent variable “z” and some auxiliary information “c”. The authors argue that when the decoder is conditioned on the concatenation of “z” and “c”, then posterior collapse is even more likely than in vanilla VAE. Indeed, the decoder under such configuration is free to ignore the latent variable “z” as long as “c” carries enough information to explain observations. To alleviate this issue, the authors propose an alternative, which consists in making the latent variable “z” explicitly dependent on the auxiliary signals ‘’c”. In more details, they introduce a new VAE to represent the auxiliary data “c”, whose posterior q(z|c) is used as a prior in the original VAE over “x”. The posterior of the latter is defined by scaling and shifting the prior q(z|c). For fitting, the author alternate the optimization of the two VAEs and use annealing to down-weight the effect of the Kullback–Leibler (KL) terms in early training. The proposed method shows good performance compared to the retained baselines, on the tasks of conditional human motion prediction and image captioning. Main comments. Strengths. Overall, this paper is well written and easy to follow. Tackling posterior collapse in VAE and its variants is an important and current topic. The proposed idea, while simple, is interesting. Experiments show that, on the tasks of human motion prediction and image captioning, the proposed model CPP-VAE improves upon the performance of conditional VAE (CVAE), using concatenation, and some recent baselines introduced in the context of conditional human motion prediction. Weaknesses. The idea of making the latent variable “z” explicitly dependent on the auxiliary information “c” is not new, and it has been already explored, e.g., in the context of collaborative filtering [1]. Although the approach of [1] and that of the current paper are technically different, the former work should be discussed and its CVAE variant (section 4.2 in [1]) should be included as a baseline. This would allow assessing the importance of the proposed CPP-VAE. Experiments do not focus on posterior collapse making it hard to assess and understand the impact of the proposed approach on this phenomenon. Moreover, it is not clear to what extent the proposed model can mitigate the above issue. In more details, - The authors use annealing to down-weight the effect of the KL terms in their objective, eq. 8, during the first training iterations. It is therefore not clear whether CPP-VAE can mitigate posterior collapse because of annealing or because of conditioning of “z” on “c” a priori. - Experiments show that the KL of CPP-VAE does not drop to zero as with CVAE when concatenation is used. However, this is not enough to conclude that posterior collapse does not occur in CPP-VAE. In fact it may still occur partially, i.e., some dimensions may well have collapsed. Therefore, further experiments are required in order to assess to what extent the proposed approach can alleviate posterior collapse. For example, one possibility is to follow previous related work [2,3,4] and report quantities such as the number of “active-units”. - In addition to the CVAE variant explored in [1], comparisons with recent approaches to alleviate posterior collapse are lacking, e.g., KL-annealing, VampPrior [3], delta-VAE [5], etc. Although these have been introduced in the context of VAE, they can be easily considered/adapted to the conditional case, which would make it clear whether CVAE requires a different treatment. To sum up, I would strongly recommend revising the experiments according to the above comments and include more results regarding posterior collapse, which is the focus of the paper. Unfortunately, I cannot recommend acceptance with the current experiments. Additional comments/questions. - The proposed approach uses VAE to represent “c”, referred to as CS-VAE, whose posterior is used as a prior over “x”. What if posterior collapse happens in CS-VAE? CPP-VAE would be reduced to vanilla VAE and thus may in turn suffer from an over regularized latent space. Have you observed such behavior in practice or tried to investigate this potential limitation of the proposed method? - Please consider improving the readability of tables 3 and 4, the fonts are too small. References [1] Lee, Wonsung, Kyungwoo Song, and Il-Chul Moon. *Augmented variational autoencoders for collaborative filtering with auxiliary information.* Proceedings of the 2017 ACM CIKM, 2017. [2] Burda, Yuri, Roger Grosse, and Ruslan Salakhutdinov. *Importance weighted autoencoders.* ICLR (2016). [3] Tomczak, Jakub M., and Max Welling. *VAE with a VampPrior.* AISTATS (2018). [4] Dieng, Adji B., et al. *Avoiding latent variable collapse with generative skip models.* AISTATS (2019). [5] Razavi, Ali, et al. *Preventing posterior collapse with delta-vaes.* ICLR (2019).