Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: In this work, the authors propose a text reconstructing auto encoder which takes a sentence as the input sequence and an integrated text generator generates another version of the input text while a reconstructor determines how well this generated text reconstructs the original input sequence. The input to the discriminator (as real data) is a sentence that summarizes the ground truth sentences (rather than the ground truth sentences themselves). The experiments are conducted in two datasets of English and Chinese corpora.

Strengths:
The proposed idea of generating text using summary sentences is new.
The model overview in Figure 1 is informative.
The experiments are conducted on English and Chinese corpora, comparison with competitive baselines are provided.

Weaknesses:
The paper is poorly written which makes it difficult to understand. The second paragraph in the introduction is quite cryptic. Even after reading the entire paper a couple of times, it is not clear how the summary text is obtained, e.g. do the authors ask annotators to read sentences and summarize them? If so, based on which criteria do the annotators summarize text, how many annotators are there? Similarly, if so this would mean that the authors use additional supervision than the compared models. Please clarify how the summary text is obtained.

In footnote 1, the authors mention “seq2seq2seq2” term which they do not explain anywhere in the text.

No experiments that generate raw text (without using summaries) are provided. It would be interesting to see if GAN learns to memorize the ground truth sentences or generates sentences with enough variation. 

In the English Gigaword dataset the results consistently drop compared to WGAN. This behavior is observed for both the unsupervised setting and two versions of transfer learning settings. There are too few qualitative results: One positive qualitative result is provided in Figure 3 and one negative qualitative result is provided in Figure 4. Therefore, it is not easy for the reader to judge the behavior of the model well. 

The choice of the evaluation metric is not well motivated. The standard measures in the literature also include METEOR, CIDER and SPICE. It would be interesting to see how the proposed model performs in these additional criteria. Moreover, the results are not sufficiently discussed. 

As a general remark, although the idea presented in this paper is interesting, both in terms of writing and evaluation, this paper has not yet reached the maturity expected from an ICLR paper. Regarding writing, the definite and indefinite articles are sometimes missing and sometimes overused, similarly most of the times there is a singular/plural mismatch. This makes the paper very difficult to read. Often the reader needs to guess what is actually meant. Regarding the experiments, presenting results with multiple evaluation criteria and showing more qualitative results would improve the exposition.

Minor comments:
Page 5: real or false —> real or fake (true or false)
	     the lower loss it get —> ?
review_title: LEARN TO ENCODE TEXT AS COMPREHENSIBLE SUMMARY BY GENERATIVE ADVERSARIAL NETWORK

final_score: rating: 6: Marginally above acceptance threshold
review_content: TL;DR of paper: Generating summaries by using summaries as an intermediate representation for autoencoding the document. An encoder reads in the document to condition the generator which outputs a summary. The summary is then used to condition the decoder which is trained to output the original document. An additional GAN loss is used on the generator output to encourage the output to look like summaries -- this procedure only requires unpaired summaries. The results are that this procedure improves upon the trivial baseline  but still significantly underperforms supervised training.

This paper builds upon two recent trends:  a) cycle consistency, where f(g(x)) = x, which only requires unpaired data (i.e., CycleGAN), and (b) encoder-decoder models with a sequential latent representation (i.e., "Language as a latent variable" by Miao and Blunsom). A similar idea has also been explored by He et al. 2016 in "Dual Learning for Machine Translation". Both CycleGAN and He et al. 2016 are not cited. The key difference between this paper and He et al. 2016 is the use of GANs so only unpaired summaries are needed.

The idea is a simple but useful extension of these previous works. The problem set-up of unpaired summarization is not particularly compelling, since summaries are typically found paired with their original documents. It would be more interesting to see how well it can be used for other textual domains such as translation, where a lot of unpaired data exists (some other submissions to ICLR tackle this problem). Unsurprisingly, the proposed method requires a lot of twiddling to make it work since GANs, REINFORCE, and pretraining are necessary.

A key baseline that is missing is pretraining the generator as a language model over summaries. The pretraining baseline in the paper is over predicting the next sentence / reordering, but this is an unfair comparison since the next sentence baseline never sees summaries over the course of training. Without this baseline, it is hard to tell whether GAN training is even useful. Another experiment missing is seeing whether joint supervised-GAN-reconstruction training can outperform purely supervised training. What is the performance of the joint training as the size of the supervised dataset is varied?

This paper has numerous grammatical and spelling errors throughout the paper (worse, the same errors are copy-pasted everywhere). Please spend more time editing the paper.


review_title: Simple but useful extension of recent works, missing important experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a model for generating long text strings given shorter text strings, and for inferring suitable short text strings given longer strings. Intuitively, the inference step acts as a sort of abstractive summarization. The general gist of this paper is to take the idea from "Language as a Latent Variable" by Miao et al., and then change it from a VAE to an adversarial autoencoder. The authors should cite "Adversarial Autoencoders" by Makzhani et al. (ICLR 2016).

The experiment details are a bit murky, and seem to involve many ad-hoc decisions regarding preprocessing and dataset management. The vocabulary is surprisingly small. The reconstruction cost is not precisely explained, though I assume it's a teacher-forced conditional log-likelihood (conditioned on the "summary" sequence). The description of baselines for REINFORCE is a bit strange -- e.g., annealing a constant in the baseline may affect variance of the gradient estimator, but the estimator is still unbiased and shouldn't significantly impact exploration. Similar issues are present in the "Self-critical..." paper by Rennie et al. though, so this point isn't a big deal.

The results look decent, but I would be more impressed if the authors could show some benefit relative to the supervised model, e.g. in a reasonable semisupervised setting. Overall, the paper covers an interesting topic but could use extra editing to clarify details of the model and training procedure, and could use some redesign of the experiments to minimize the number of arbitrary (or arbitrary-seeming) decisions.
review_title: GANs for text, with text latent variables

Meta-review: As expressed by most reviewers, the idea of the paper is interesting:  using summarization as an intermediate representation for an auto encoder.  In addition, a GAN is used on the generator output to encourage the output to look like summaries.  They just need unpaired summaries.  Even if the idea is interesting, from the committee's perspective, important baselines are missing in the experimental section:  why would one choose to use this method if it is not competitive with other baselines that have proposed work in this vein?  One reviewer brings up the point that the method is significantly worse than a supervised baseline.  Moreover, the authors mention the work of Miao and Blunsom, but could have used one of their experimental setups to show that at least in the semi-supervised scenario, this work empirically performs as well or better than that baseline.
Decision: Reject