Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Summary:

This paper introduces a probabilistic generative model for
unsupervised style transfer of text. The approach introduced in
the paper does not require paired training data. An
encoder-decoder model is trained to transfer text from one style
to another and back.

Review:

This work is very well-written and easy to follow. The
contribution is clearly articulated as while there are
probabilistic generative models for transfer in the
literature (Shen et al does include one) they don't perform as
well. Ablation studies further confirm the need for the
particular kind of parameter sharing used in the model in the
paper. Great results are shown on 5 text transfer problems.

Clarifications and improvements:

Just for clarity, in the last paragraph on page 4. It says two encoder-decoder
models are learnt, but isn't the idea that there is effectively only one
encoder and one decoder learned that just put together in different ways
during training? I'm also curious why the baseline of BT+NLL was so strong? Is
having the loss of a language model work that much better than the regular
entropy term?

I would also like if possible if you could share some of the repetitive examples
created by BT-NLL which explain its low PPL.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a probabilistic framework for unsupervised text style transfer. Given two non-parallel corpora X,Y in different domains, the authors introduce unobserved corpora \bar{X}, \bar{Y}. These are used as latent variables that control the generation of the observed data. To train models, the paper proposes to optimize the evidence lower bound of the log marginal likelihood. To facilitate training, multiple techniques are suggested, such as parameter sharing, some gradient approximations and initialization with a reconstruction objective. The approach is evaluated on five style transfer tasks, as well as unsupervised machine translation. Models are evaluated with multiple metrics, and generally obtain reasonably strong performance.

I lean towards the acceptance of the paper because the approach is fairly simple and elegant, while obtaining promising results. The connections to back-translation and language models are also potentially interesting. However, while the paper aims to suggest a principled approach to style transfer, using greedy samples biases the reconstruction objective, and as such the method does not really optimize the ELBO.

Casting style transfer as data completion is a straight-forward idea that doesn't introduce unnecessary or too simplistic assumptions. Optimizing the ELBO follows naturally, and can lead to more diverse outputs than the BT+NLL approach, which misses the negative entropy term. Reference BLEU scores on all tasks are competitive, and sometimes clearly better, with strong baselines.

Greedily sampling latent sequences during training should ideally be justified more carefully as it biases the objective function. In particular, an experimental comparison to stochastic sampling, which should more closely approximate the expectation, would be appreciated. Additionally, detailing the similarities and differences between the proposed approach and current UNMT techniques could be helpful to some readers.

Questions:

Could you present the validation and test evidence lower bounds? If so, how is sampling performed?

In footnote 2, you mention tuning the strength of the KL regularizer. As the KL can be decomposed into 2 terms (Eq. 5), would it be beneficial to control each term separately?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The main contribution of this paper is a principled probabilistic framework of unsupervised sequence to sequence transfer (text to text in particular). 

However, I believe there is a large disconnect between the probabilistic formulation written it section 3 and whats actually happening experimentally in section 5. It is not clear whether the model is *actually* optimizing an ELBO because the gradients from sequence reconstruction loss are not backpropogated to the inference network as explained in paragraph on Approximating Gradients of ELBO. Moreover this restriction makes the authors method almost the same as the one used for unsupervised neural machine translation by Lample et al 2017 and Artetxe et al 2017. I would like to see a more detailed analysis from authors on how far the performance of Gumbel-softmax estimator and REINFORCE estimator is from simple stop-gradient estimator used in experiments.

In terms of experimental setup I like that the authors considered a large suite of experiments across various tasks. Although the evaluation metrics on text style transfer tasks like sentiment transfer, formality transfer, author imitation are in line with previous work ideally the human evaluation needs to be done to truly see how well each method performs. On unsupervised machine translation, authors show a large improvement on Serbian-Bostian translation. I am a bit skeptical since as I wrote above the proposed method is very similar to previously proposed unsupervised neural machine translation approaches and it is not clear why we are seeking such a large gain of 5 BLEU points.

Overall I think it is a well written paper with a large experimental suite, although I am skeptical of actual connection between probabilistic formulation and whats actually happening in practice.

================================================
Update: I have raised the score from 3 to 6.
review_title: Official Blind Review #3

Meta-review: This paper proposes an unsupervised text style transfer model which combines a language model prior with an encoder-decoder transducer. They use a deep generative model which hypothesises a latent sequence which generates the observed sequences. It is trained on non-parallel data and they report good results on unsupervised sentiment transfer, formality transfer, word decipherment, author imitation, and machine translation. The authors responded in depth to reviewer comments, and the reviewers took this into consideration. This is a well written paper, with an elegant model and I would like to see it accepted at ICLR. 
Decision: Accept (Spotlight)