Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Summary:
 
The paper proposes an improvement to sequence generative adversarial networks (GAN) to cope with the common training issues of GANs. For the sake, the paper combines Gumbel-Softmax based GAN, relativistic discrimination  function with  the matching of mean representations of true and generated samples in a latent feature space. This feature statistics alignment allows to leak information from the discriminator to the generator as the used features are extracted from the discriminator network. Experimental evaluations on synthetic and real datasets show the improvement achieved by the proposed method over existing sequence generation networks.



Reasons for score: 
 
The paper straightforwardly combines existing procedures (relativistic discriminator, Gumbel-Softmax approximation for categorical distribution, features matching) to improved upon vanilla sequence generation networks and somehow lacks novelty. Although the ablation study is interesting and shows the improvement brought by each module, examples of lengthy generated sequences illustrate that the sentences produced by the GAN are not semantically meaningful. 
 

Pros:
- Overall, the paper is well written. In particular, the rationale behind the proposed method is justified. Empirical evaluations support these intuitions and show how they contribute to the observed quality of the generated sequences.  
- The paper aims at addressing a major issue in training GAN for sequence generation: how to strengthen the learning of the generator compared to the discrimination network which is easier to train? The approach promoted in the paper consists to align the mean statistics of true sequences and fake ones. Specifically, the statistics are computed over features extracted from discrimination network. The objective function of the generator is therefore composed of the usual GAN loss term and the distance between those mean representations. As such, this idea of guiding the generation network with information from the discriminator is interesting and plays a key role in the performances improvement. 
- In the same vein, the use of Gumbel-Softmax distribution (instead of the discrete distribution) and of the relativistic discrimination function (instead of the classical classification function) helps to learn a better generation model. However these ideas are not novel and were investigated separately in previous research works. 
- Experimental evaluations, including both qualitative analysis and quantitative results, are provided in the paper and in the supplementary to show the effectiveness of the proposed framework. The newly proposed GAN achieve superior performances. The comprehensive ablation study is interesting and helps to understand how each module (feature alignment, Gumbel-Softmax, batch size) contributes to the enhanced performances.
 
Cons: 
- Although the proposed method, according to the empirical results, show improved performances, it lacks novelty as features matching, relativistic discrimination or Gumbel-Softmax are not new ideas. The main contribution resides in the better quantitative results compared to existing sequence generation networks. However when one examines the generated sentences, it appears that they lack semantic meaningfulness especially for long sentences (see for instance Table 8). This shows that the proposed GAN (as well as the competitors) is not effective yet.  
- Features distribution alignment is an interesting way to measure how close are the marginal distributions of the real and fake sequences. The paper considers the Mean Distance Alignment (MDA) and the Mean Square alignment (MSA) which are respectively the distance and the squared distance between the mean latent representations of the real and generated sequences. Several comments can be made as hereafter.
     * MSA and MDA encode the same matching up  to a power 2. It’s unclear why they lead to different empirical results.
     * Instead of matching only the mean statistics, the overall distributions of the latent representations can be aligned by considering metrics such as MMD or Wasserstein distance. How would the results look like in that setting?
     * It should be clarified earlier in the paper that the used features are extracted from the discrimination network (as the weights between the discriminator and feature extractor are shared). Also the paper should make explicit from which layer of the discrimination network the features are extracted. 
- The findings of human evaluation (see Table 5) are not unequivocal. MSA and MDA achieve higher scores than the real sentences. The best model, the one with MSA, is not preferred because of a lack of diversity and quality. This raises the question of how reliable is the human evaluation score.
 
Other comments: 
- Page 3, definition of MSA: in the sentence “mean squared difference between the centroids...”, the term mean is over-used as Eq. (2) or (3) represents only the squared distance between centroids.
- Page 4: in “The FSA term on the RHS can also be regarded as a dynamic regularizer for the sequence generator” the notion of dynamic regularizer is unclear. In which sense FSA induces a dynamic regularization?
- In Equation (10) the function “one_hot” should be defined. Also in (10), I think $y_i$ should read the $|V|$-dimensional vector $y$.
- Equation (11) is to be checked carefully as the parameter $\tau$ simplifies in numerator and denominator. 
- Algorithm 1: the update of the generator $G_\theta$ requires a minibatch from real dataset in order to minimize $L_{RG}$ + $L_{FSA}$ as $L_{FSA}$ relies on the mean of real data latent representation.

After rebuttal
-  I read the response of the authors. The spotted typos are fixed in the revision. Some  questions/concerns  have been tentatively. However the novelty in the paper is still not blatant or how the use of distance such as MMD or Wasserstein to match the features is under-explored. Hence I intend to keep my rating.
review_title: The paper proposes an improvement to sequence generative adversarial networks (GAN) by combining Gumbel-Softmax based GAN with the matching of mean representations of true and generated samples in a latent feature space. Experimental evaluations on synthetic and real datasets show the effectiveness of the method. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Summary]
This paper proposes a new GAN-based text generation method that incorporates feature statistics alignment and gumbel-softmax for reparameterization to deal with mode collapse and unstable training. For feature statistics alignment, the authors design two methods such as mean square and mean distance alignments. They evaluate the proposed method on a synthetic dataset, MS COCO caption, and EMNLP2017 WMT news dataset, comparing them with RL-based and non RL-based models. With extensive experiments including ablation studies, the proposed method show promising results.

[Recommendation]
Overall, this paper is clear and well-written. So I lean to acceptance. But I have some concerns as well.

[Strength]
- Mode collapse is challenging issue in GAN training.
- Text generation is important problem.

[Weakness]
- The authors insist the use of Gumbel-softmax in GAN tranining is under-explored. But It is not clear. There are more method using Gumbel-softmax [Gu et al. 2019] and a similar softmax with temperature annealing. It is not clear for the authors to explicitly discriminate using Gumbel-softmax and other smoothed softmax methods. 
- Some related work  were missed such as DialogWAE [Gu et al. 2019] and ARAML [Ke et al. 2019]. In particular, DialogWAE uses GAN and Gumbel-softmax for text generation even if it focuses on dialog generation.
- For verifying mode collapse issues, how about using Self-BLUE in addition to BLUE scores as a metric to evaluate the diversity? 
- Novelty might be incremental. It seems that the novelty is from using feature statistics alignment. To emphasize the contribution of feature statistics, comparing between the latent feature visualization with and without FSA might be helpful in addition to ablation study. 

[Minor] 
In p3, the given real data is --> are


[Gu et al. 2019] DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder. ICLR 2019.
[Ke et al. 2019] ARAML: A Stable Adversarial Training Framework for Text Generation. EMNLP 2019. 
review_title: Interesting paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper addresses the task of improving GANs for sequence generation and proposed a method based on the relativistic discriminator. The proposed method employs a Feature Statistics Alignment (FSA) paradigm to reduce the gap between real and generated data distributions. It relies on the relativistic discriminator for "coarse" differences and FSA for "fine-grained" differences between real and generated data distributions. It is evaluated on synthetic and real datasets, and it significantly outperforms the baselines. It also outperforms baselines on human evaluation based on the acceptance, grammaticality, and meaningfulness of the generated sentences. 

Strengths:

The proposed approach is very effective, as demonstrated by significant performance improvements in the experiments across synthetic and real datasets. Also, it can generate better sentences compared to the baselines, as shown by human evaluation.

Weakness:

Although the proposed model is thoroughly evaluated and empirically effective, it is not very different from existing methods, except for FSA. The application of FSA in this context might be novel; however, the proposed approach seems to be a simple combination of two existing approaches. Therefore, the novelty of the model is limited.

Minor comments:
1. Correction:  Did you mean

In contrast, the lower τ could discourage exploration and tend to explore during training. -> In contrast, the lower τ could discourage exploration and tend to exploit during training. ?

2. It would have been good to see a head-on comparison of the generated samples (baseline vs. proposed approach) in the paper's main text.
review_title: Official Blind Review #2

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Main Claim:**

In this work, the authors propose to use the Feature Statistics Alignment paradigm to enrich the learning signal from the discriminator in a sentence generation GAN. The proposed model can generate sentences with better likelihood and BLEU on one synthetic and two real datasets.

**Contributions:**

This work introduces an novel and interesting idea of Feature Statistics Alignment in training GANs. 

The authors follow the convention in this domain, and evaluate the model on three datasets.

The experiment results show that the proposed model outperforms existing models. However, the authors need to clarify some details to make the results trustworthy (see weakness). 


**Strong points:**

The idea is novel and interesting. 

The model and training procedure is clearly explained. Related works are cited well. 


**Weak points:**

In Table 2: 

- The LSTM model gets NLL lower than the real data. This is a clear evidence of overfitting. 
- In SAL (Zhou et. al, 2020), NLL_{gen} is used to evaluate the diversity of the generator. But this metric is missing here without explanation. 

In Table 3:

- The BLEU metric in this paper is the BLEU(F) metric in SAL (Zhou et. al, 2020). This metric evaluates the generated sentences using the test set as a reference. Thus the BLEU(F) metric cannot show the diversity of examples. 
- The BLEU(B) (Zhou et. al, 2020) metric is missing. BLEU (B) metric evaluates the test set using the generated sentences as a reference, so it can detect mode collapse of a generative model. 

In section 4, although authors clearly cite previous works for experiment settings, I think it’s worthwhile to repeat the definition of each metric, and some other key points in the paper, so that readers can easily understand the notations and jargons in this section. 

**Recommendation:**
Reject. 

There’s a major flaw in the evaluation metrics. On both synthetic and real datasets, the evaluation metrics prefers overfitted models, i.e. if the model can remember one example from the training set, and repeat that sentence, it can get a very high score. 

I will reconsider my recommendation if (1) I miss interpret the metrics or (2) the authors provide more evidence on the diversity of the generated sentences, for example showing the NLL_{gen} metric on the synthetic dataset, and BLEU(B) metric on real datasets. 

**Questions:**

How is NLL_gen computed?

**After Rebuttal**
The author's reply partially resolved my concerns, although the diversity of models has not improved, nor has it significantly decreased. Thus I have increased my score from 3 to 4. 
review_title: Interesting idea but flawed experiments

Meta-review: The work introduces a method that uses the Feature Statistics Alignment paradigm to improve sequence generation with GANs. The contribution is interesting and novel (although marginally), clarity is also good.
However the reviewers raised several concerns calling for more comprehensive and thorough evaluation. Experiments show an improvement comparing to selected baselines and the revised paper addressed, at least partially, a serious evaluation concern of one reviewer.
Although the excellent revision work some important open questions still seem to remain, in particular the choose of alignment metrics and a thorough evaluation.

Decision: Reject