Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**
This paper proposed a new text GAN framework by combining non-autoregressive text generator based on transformer, straight-through gradient approximation, and various regularization techniques such as gradient penalty and dropout. The paper demonstrates the superiority of non-autoregressive generator in the context of text GANs through various experiments including unconditional text generation, latent space manipulation and unsupervised decipherment.

**Pros**
- This work narrows the gap between image GAN and text GAN by leveraging recent advances on non-autoregressive text generators and a straight-through gradient approximation. While these components are well studied in previous works, I think this work presents a neat combination of them in order to solve a well-known problem.

- The paper provides rich discussions in training text GAN and comprehensive experiments and ablations to demonstrate the usefulness of an implicit text generator in different contexts.

**Concerns & Questions to Answer during rebuttal**
- The original text GAN papers were mainly motivated to address the *exposure bias* problem in maximum likelihood estimation for autoregressive generators. In other words, when we use an autoregressive generator to sequentially generate tokens one by one, there is a distribution mismatch between training and test phase. In your case, now that you already have a non-autoregressive text generator, is there any *theoretical motivation/insights* for using the adversarial training framework? We know that MLE is statistically efficient (achieving Cram´er–Rao Lower Bound) and possesses many good properties, maybe training the non-autoregressive text generator with MLE (e.g., FlowSeq [1] or variational inference) is a better choice?

- The non-autoregressive (NA) text generator have been well studied recently so the novelty of this work is more on the integration of NA generator with adversarial training. Thus the main challenge here is how to solve the non-differentiability problem. The paper directly leverages a traditional workaround, the straight through estimator, which is a biased gradient approximation. Is the bias going to be an issue and is there any better strategy? I think the paper need to provide more discussions on this aspect. Overall the method section need to be polished with more details, as I feel this part is currently hard to follow.

- In figure 1, $z_1, \ldots, z_L$ are sampled independently, which are sent to a transformer and later produced the sample. Is the independence between $z_1, \ldots, z_L$ going to be a problem? When we use transformer to do neural machine translation, the attention mechanism will capture the dependence in the input sentence ($z_1, \ldots, z_L$ in this context) and then produce the output correspondingly. Hence will the independence in $z_1, \ldots, z_L$ lead to a less expressive sample distribution in your text generator (although this is not an issue in image GAN)?

- The paper also propose to use the Max Gradient Penalty from image GAN domain. The Max Gradient Penalty was introduced under the framework of Wasserstein GAN or Lipschitz GAN framework, which aims to constraint the function space to be Lipschitz smooth. However this work uses the vanilla GAN objective (eq 12 and eq 13), which is not the WGAN or LGAN framework. Thus the regularization may not be theoretically correct. Also why not instead use the WGAN objective which is empirically more stable and theoretically sound?

- Experiments: In table 2, all the results for NAGAN use the dropout with a positive ratio. How does NAGAN perform without dropout? Also I wonder if the comparison in the table and figures are fair, since most previous methods.baselines such as MLE or SeqGAN only use a vanilla RNN/LSTM, while NAGAN has a more complicated structure with transformers, and additional regularization such as gradient penalty and dropout. Perhaps we should control at least the number of parameters to be in the same level.

[1] FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

Update after rebuttal: 
After seeing the author response below, no change to my score.

review_title: Interesting work on integrating non-autoregressive generator to text GAN

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces the non-autoregressive generator in the GAN-based text generation, making textGAN can be trained without pre-training and better utilize latent variables to control the style of generated text.

Introducing non-autoregressive architectures into GAN-based text generator is a natural idea, and the modelling ability of non-autoregressive generator has been verified at BERT. I think that this paper is above the average because it provides comprehensive experiments (including unconditional text generation, unsupervised decipherment and sentence manipulation), showing the significant improvement in various evaluation metrics compared to the text model without NAR and baselines.

However, this paper should have more analysis of how non-autoregressive architectures work in the GAN-based text generation.
1.	How latent variables in the non-autoregressive generator influence the content of the generated text? Can you provide some analysis or examples about it? (e.g. change the value of some latent variable continually (from 0 to 1?) and give the generated text).
2.	Can you give some analysis about the generation process of the non-autoregressive generator, (e.g. attention map), which makes the generator more interpretable.
3.	The user study is absent.
4.	Is dropout necessary for the non-autoregressive generator? What if the dropout rate is 0, how the performance of generator changes?
5.    Can you give more details about experiments, such as model parameters, training time, inference time and GPU you used?
review_title: I think that this paper is above the average because it provides comprehensive experiments but lacks analysis about the generation process

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper creatively extends text GAN by introducing non-autoregressive generator, which is a well-known notion in translation and VAE like generation but not often applied in a GAN setting. The paper argues that a non-autoregressive generator brings more effective gradient-based optimization and also good latent representation learning capability. 

The comparison between NAGAN and other text GANS reads okay, but the reviewer concerns the limited scope and significance of this paper.

1, Given very strong text generation capability of MLE learning and pre-training, NAGAN makes little contribution to push the generation SOTA. Audiences of this approach are also limited. In this paper, given a very old baseline of MLE and a bunch of text GANs, the overall performance of NAGAN is still not much leading. Let alone compare it to other strong pre-trained generators.

2. When claiming good latent representation learning capability, there should be a big gap between NAGAN and text VAEs in this aspect. If the author adds more control and manipulation experiments in text VAE, NAGAN will be not as shining as now.

3. Non-autoregressive generator has difficulties in generalizing to long text generation and conditional generation. How does the author consider such settings, instead of simple unconditional generation in toy datasets like COCO?

Overall, the reviewer thinks this is a well-written paper, but a boardline one considering its limited significance for the venue.
review_title: Good extension of text GAN but limited significance for text generation

Meta-review: This paper proposes GAN-training of a non-autoregressive generator for text. To circumvent the usual problems with non-differentiability of text GANs, the authors turn to Gumbel-Softmax parameterisation and straight-through estimation. 

There are a number of aspects to this submission and they are not always clearly positioned. I will concentrate on the two aspects that seem most crucial:

1. The authors position their generator as an implicit generator, but it really isn't. If we take the continuous interpretation of the output distributions: the Gumbel-Softmax transformation does correspond to a tractable density, the Concrete density of Maddison et al, with known parameter. If we take the discrete interpretation of the output distribution: Gumbel-argmax is just an alternative to sampling from a Categorical distribution with known parameter. In either case, the generator maps the noise source to a collection of conditionally independent distributions each of which has a known parameter and analytical density/mass function. The authors do, however, train the architecture using a GAN-type objective *as if* the generator were implicit.

2. In the discussion phase the authors added that GAN training overcomes the independence assumptions made by the generator. Whereas that makes intuitive sense, it suddenly changes the emphasis of the contributions, from proposing an implicit generator (presumably powerful for it being implicit) to proposing a way to circumvent the strong independence assumptions of the generator with a mechanism other than more traditional approximate marginalisation of VAEs. In their rebuttal, the authors commented on the use of non-autoregressive VAEs in neural machine translation, and though those observations have indeed been made, they might well be specific to MT. The simplest and more satisfactory response would be to ablate the use of the GAN objective (that is, to train a non-autoregressive VAE, also note that, with the same choice of likelihood, posterior collapse is rather unlikely to happen).

Other problems raised by reviewers were addressed in the rebuttal, and I would like to thank the authors for that. For example, ablating the non-autoregressive generator and comparing to REINFORCE. I believe these improved the submission. 

Still, I cannot recommend this version for publication. I would suggest that the authors consider careful ablations of the components they see as precisely important for the results (that currently seems to be the GAN-like objective despite the model not, strictly speaking, requiring it). 





Decision: Reject