Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a method for controlled text generation by using a new loss function (standard VAE loss with auxiliary losses added on). The method is tested on style transfer datasets: Yelp and Amazon. The central hypothesis is that when manipulating latent codes of a VAE, you can end up in low-density regions of the aggregated posterior. Such latent codes are rarely seen by the decoder so that quality of generation is low. To address this problem, they constrain the posterior mean to a learnt probability simplex and try to ensure that the simplex is densely filled. They do this by adding 2 regularizing losses to the VAE loss.

Some questions:-
- I had some trouble following section 4.2 where the derivation for L-reg is explained, this is the loss that constrains the posterior to a simplex. Is E = [e_1, ..., e_k] learnt or a hyperparameter? Most of the text seems to indicate that it is learnt, but you mention that alpha is a tunable hyperparameter, where alpha = e_i^2. Could you elaborate on how e_i^2 can be fixed to alpha is E is learnt?
- You state that mu^2 reaches a minimum at alpha/K. Could you please provide a proof? This could be placed in an Appendix.
- In section 6.3, is K still 3 or is it now >=4?

Some citations issues 
- You're missing a couple key citations. I think your work should cite Hu et al.'s (2018) Toward Controlled Generation of Text. 
- You should definitely cite Lample et al.'s (2019)  Multiple-Attribute Text Rewriting. The results in Lample et al. are definitely comparable, and often better than the results presented in this paper. While I think this paper makes an independent and useful contribution, some of the claims are overblown since the results from Lample et al. are not presented or discussed. For example, the Lample's model on Yelp achieves a higher accuracy and a lower perplexity than the CP-VAE model presented in this paper.
- Correction of citations: Kim et al. (2018), Adversarially regularized autoencoders, is in fact Zhao et al. (2018), you're just missing the first author.


Additionally, I'd like to see some more detail about the robustness of the method. Is the model robust to initialization and hyperparameter settings? Is posterior collapse always avoided? In Table 1 you show that without L-Reg the CP-VAE suffer with posterior collapse, but this would carry more weight if you showed some statistics on how often using L-Reg helps avoid posterior collapse. In general it looks like we're mostly presented with the best results, I'd like to know a little more about the mean and standard deviation of the model runs as well.

Overall, I think this paper makes a worthwhile contribution. I was unclear on a few parts of the derivation for the regularizing losses, but the intuition seems sensible. The presented results are impressive but have a few missing pieces: citing some key results (Lample et al. 2019) and including results about robustness. I think the questions I have, and any shortcomings this paper has, could be addressed in a camera-ready version.


Other bits,
- I do not understand figure 2. It's also never referred back to once we learn what v_p and v_n are. Some clarification here would be helpful.
- Section 5.1, line 3: achieve -> achieved
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper "On Variational Learning of Controllable Representations for Text without Supervision" tackles the problem of latent vacancy of text representation via variational text auto-encoders. Based on the observation that a single factor of the sentence encoding gathers most of relevant information for classifying the sentence as positive or negative (sentiment classification), authors study the impact of manipulating this factor in term of the corresponding decoded sentences. They reasonnably claim that if such a manipulation fails at decoding accurate sentences, it is because we fall in representation areas that the decoder never seen during training. Thus they propose a way to constrain the posterior mean to a
learned probability simplex and only perform manipulation within the probability simplex. 

The tackled problem is important. Variationnal text auto-encoding is a very challenging task, for which no perfect solution has been proposed yet. A important issue is that usually the posterior collapses, with the auto-regressive decoder eventually ignoring the codes. Another problem is that indeed the representation space can contain many holes, in codes have never been seen during training. The authors propose to cope with both problems by encouraging a part of the code mean to live in a simplex, which prevents from posterior collapsing. Next, to ensure that information is filled in this constrained part, they define a pairwise ranking loss which enforce the mean of each sentence to be more similar to the output of the encoding lstm than the mean of other sentences. For this part, more intuition justification is needed to well understand the effect of this additional loss. In what sense does it ensure that the space does not contain holes ? What ensures that the constrained part of the code is actually used by the decoder ? 

My main concern is with regards to the experiments, which are clearly not enough detailled. First, I cannot understand what NLL is considered in the preliminary experiments. Authors study the effect of code manipulation on an NLL. But the likelihood of what ? Of the encoded sentence ? If yes it is natural that the NLL is impacted since we move the representation so the manipulated representation encodes another sentence... Or maybe it is w.r.t. a generated sentence from the obtained code ? But what sense does it make to assess the nll of the generated sentence ? Ok if the distribution is to flat, the NLL would not be good, but is it really what we want to observe ? Also, authors compare the impact of modifications on the representations of $\beta$-VAE with modifications on their model, but these are not the same modifications. What ensure that they have the same magnitude ? I cannot understand the paragraph on vp and vn in the experimental setup. Comparisons with metrics on text style transfer are also difficult to understand to me. What is the reference sentence ? 

Minor questions:
    - z(1) is said to be parametrized by a MLP with sentences representations before eq2 and is said to be encoded by a LSTM after eq 2. What am I missing ? 
     - Please better detail fig2      

    
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper tries to pinpoint why sequence VAEs haven't worked well to provide disentangled representations. The authors posits that this happens due to the fact that perturbing the intermediate representation (codes) pushes them in regions which are not seen in training, and hence the model is not well equipped to perform well for those codes. To address this, they augment the VAE objective with terms to ensure that codes are present in a probability simplex and the entire simplex is uniformly filled by codes. I thought this was a very good paper. I found the observation very interesting, and the supporting experiments confirm the hypothesis. However, since I do not actively work in this area, I am not sure how exciting the result will be to other researchers. 

There's some related work on controlled text generation.
https://arxiv.org/pdf/1811.00552.pdf
https://arxiv.org/pdf/1811.01135.pdf
It will be good to mention them and possibly compare and contrast with the above work. 
review_title: Official Blind Review #2

Meta-review: This paper analyzes the behavior of VAE for learning controllable text representations and uses this insight to introduce a method to constrain the posterior space by introducing a regularization term and a structured reconstruction term to the standard VAE loss. Experiments show the proposed method improves over unsupervised baselines, although it still underperforms supervised approaches in text style transfer.

The paper had some issues with presentation, as pointed out by R1 and R3. In addition, it missed citations to many prior work. Some of these issues had been addressed after the rebuttal, but I still think it needs to be more self contained (e.g., include details of evaluation protocols in the appendix, instead of citing another paper). 

In an internal discussion, R1 still has some concerns regarding whether the negative log likelihood is less affected by manipulations in the constrained space compared to beta-VAE. In particular, the concern is about whether the magnitude of the manipulation is comparable across models, which is also shared by R3. R1 also think some of the generated samples are not very convincing. 

This is a borderline paper with some interesting insights that tackles an important problem. However, due to its shortcoming in the current state, I recommend to reject the paper.
Decision: Reject