Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies latent manipulations in text autoencoders. The authors propose that compared to random and coordinate directions, moving in the PCA directions of encodings of training examples will produce more interpretable text manipulations.

As the idea is straightforward, I'd like to see more in-depth analysis and more solid evaluations. The authors characterize the effects of PCA directions into four types (length, word change, word insertion, and structure enforcement), but for each type only one example is provided. What are the changed/inserted words and what are the enforced structures? Can you give a comprehensive list of them? When are these latent directions applicable and when are they not? For sentences that are not applicable, what effects will they bring?

The only evaluation in the paper is human evaluation of whether a latent direction shift produces interpretable generations. It's conducted on 20 sentences, which is too small to draw any conclusions. The results on the Wikipedia dataset are very poor. You may test the success rate of manipulations in a specific direction (such as word insertion) through automatic evaluation. This can also reveal which manipulations are easier to implement and which are more difficult.

I think with these changes, the paper will be more substantial, instead of spending 4 out of 8 pages on the background like in the current submission. Also, it's more suitable for NLP conferences than ICLR.
review_title: Straightforward idea, hasty  experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: The author propose to use PCA-like method on latent space of VAE models to unsupervisedly detect interpretable direction. The idea is reasonable and practically useful for large-scale pretrained VAE model, i.e. OPTIMUS. This paper has a clear idea and a thorough discussion with related works. 

I have some concerns about the model. The proposed model seems requiring a large-scale pretrained model. If the VAE model is just trained on SNIL level, is method still valid?  From the PCA side, it does not require a Gaussian space. So why specifically targeting on VAE model, not just AE model is another confusion. Since the direction is computed based on training data, I kind of feeling of no need of using VAE model.
review_title: An interesting examination and exploration of the OPTIMUS VAE model

final_score: rating: 3: Clear rejection
review_content: This paper presents a PCA-based latent variable language model for unsupervised latent variable interpretation.

Pros:
1. The authors propose to use PCA to extract the principal components of the results and claim them to be interpretable latent variables.

Cons:
1. The novelty is quite limited. Applying an existing well-known technique to obtain interpretable latent variables is not advancing this domain in the right direction.
2. The explanation of latent variable in this paper is self-justified. The self-defined baselines cannot be convincingly conveyed that latent variable are interpreted. And the baselines are quite weak.
3. In the quality evaluation, the authors do not show how clearly to modify the discovered latent variable to alter the sentences.

Question:
1. How do you encode a sentence in a two-dimensional space? Are both dimension probability?
2. Other than the current quantitative and qualitative analysis, do you think any other quantitative evaluation will be helpful?
review_title: Unsupervised Discovery of Interpretable Latent Manipulations in Language VAEs

final_score: rating: 3: Clear rejection
review_content: -------------------
Summary
-------------------
This paper proposes a simple approach to discover interpretable latent manipulations in trained text VAEs. The method essentially involves performing PCA on the latent representations to find directions that maximize variance. The authors argue that this results in more interpretable directions. The method is applied on top of a VAE model (OPTIMUS), and the authors argue that different directions discovered by PCA correspond to interpretable concepts.

-------------------
Strengths
-------------------
- The method is simple, and can be applied on top of existing text VAEs.
- Learning interpretable and controllable generative models of text is an important research area, and this paper contributes to this important field.

-------------------
Weaknesses
-------------------
- There are only mostly qualitative results presented. While I agree that performing quantitative results is difficult with this style of work, the authors could have (for example) adopted methods from the style transfer literature to show quantitative results. These metrics include perplexity (to see how fluent the generations are), reverse perplexity, and style transfer accuracy (this may not be applicable since there is no ground truth "style" in this work, but the ground truth style could be heuristically defined for some transformations, e.g. for singular/plural transformations).
- Human evaluation seems nonideal since it is only tested on 12 people.
- The generations are actually not so good in my opinion? E.g. many of the generations in the appendix are ungrammatical and/or semantically nonsensical.  Again, metrics such as perplexity could quantify the fluency of generated text.
- The method is only applied to one text VAE mode which specifically uses BERT/GPT-2 , so it is not clear if this will generalize to other models (e.g. models trained from scratch).

-------------------
Questions/Comments
-------------------
- In Figure 2, are these the top 4 principal directions? If not, how were these directions discovered?
- "It is known that variational autoencoders trained with a schedule for the KL weight parameter (equation 1) obtain disentangled representations (Higgins et al., 2016; Sikka et al., 2019; John et al., 2019). Since OPTIMUS is also trained with KL annealing, canonical coordinates in its latent space are likely to be disentangled." I believe this is only valid for beta > 1 so it is not really applicable here.
-----------------------
Edit after rebuttal: Thank you for the rebuttal and clarifying some of my questions. I have decided to keep the original score.
review_title: Simple method but unclear results

Meta-review: This paper proposes a simple method to discover latent manipulations in trained text VAEs. Compared to random and coordinate directions, the authors found that by performing PCA on the latent code to find directions that maximize variance, more interpretable text manipulations can be achieved. 

This paper receives 4 reject recommendations with an average score of 3.75. The reviewers have raised many concerns regarding the paper. (i) The idea is straightforward with limited novelty. (ii) There are only mostly qualitative results presented. More in-depth analysis and more solid evaluations are needed. (iii) Human evaluation is too small to draw any reliable conclusion. (iv) The proposed method is only tested on one text VAE, how well it can be generalized to other models remains unclear.

The rebuttal unfortunately did not address the reviewers' main concerns. Therefore, the AC regrets that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere. 
Decision: Reject