An Adversarial Non-Autoregressive Model for Text Generation with Incomplete Information
Authors: Da Ren, Yi Cai, Qing Li


Reviewer paper summaries: This paper presents an Adversarial Non-Autoregressive Transformer (ANT) designed to handle text generation tasks within the Incomplete Information Scenario (IIS). Traditional Non-Autoregressive (NAR) models exhibit limitations when working with incomplete input data, leading to significant challenges due to the multi-modality problem. ANT employs Generative Adversarial Networks (GANs) to bypass these issues and introduces two novel features: Position-Aware Self-Modulation for enhanced input representation and Dependency Feed Forward Network for better word dependency modeling. Experimental results demonstrate that ANT can achieve comparable performance to autoregressive (AR) models while maintaining lower decoding latency, making it suitable for applications like latent interpolation and semi-supervised learning.

This paper studies non-autoregressive models in the incomplete information scenario (IIS). Unfortunately, the multi-modality problem of existing non-autoregressive models is exacerbated in such a scenario. To address this, the authors propose adversarial non-autoregressive transformer based on GANs, which includes position-aware self-modulation and dependency feed-forward networks. Experimental results demonstrate that the proposed approach can achieve comparable performance with much fewer decoding iterations.

This work focuses on non-autoregressive (NAR) text generation in an Incomplete Information Scenario (IIS), where the information needed for the target sequence does not come solely from the source tokens. The authors hypothesize (and show proof) that the IIS setting poses more challenges for NAR models than the Complete Information Scenario setting (e.g., machine translation). The proposed approach is GAN based, and has three modules: 1) an aligner generates representations for words, 2) a generator takes as input from the aligner and generates the words, 2) a discriminator that tells if a sequence of representations is from the aligner or the discriminator. The training of the GAN system follows the Wasserstein GAN (Arjovsky et al., 2017). Empirically, the authors show their proposed system is able to achieve better performance than selected NAR baselines but behind the autoregressive (AR) counterpart.
Summary of strengths: 1. Well motivation. Though [1] has introduced REINFORCE into optimizing NAR models, the combination of GANs with NAR models is still innovative and well-motivated to address inherent multi-modality issues in standard NAR models for IIS. 



[1] [A Text GAN for Language Generation with Non-Autoregressive Generator](https://openreview.net/forum?id=wOI9hqkvu_)

Addressing the multi-modality problem of existing non-autoregressive models can benifit the ACL community.

---

After rebuttal, I will raise my score to 3.

---

1. The proposed GAN-based NAR approach is interesting. I like the idea (although it is proposed by Ren and Li (2022), a different work) of performing GAN on representations rather than tokens.
2. This work can be extended to encoder-decoder architecture or encoder with span-based masks (Su et al., 2021). Maybe it can have good potential in other tasks.
Summary of weaknesses: 1. The comparison of the experiment is not sufficient. A series of improved NAR models used to alleviate multi-modality issues were not included in the consideration. The author discussed DAT [2] in related work, but the results of DAT did not appear in the experimental comparison, which may reduce the persuasiveness of the effectiveness of the method in this paper.

[2] [https://proceedings.mlr.press/v162/huang22m.html](https://proceedings.mlr.press/v162/huang22m.html)

Please see the below questions.

- Figure 1: What is the second task? It seems to me that both are different tasks. I am wondering which machine translation tasks can belong to the incomplete information scenario (IIS)? In addition, have the authors ever tried MT tasks for experimental analysis?

- Line 12 - 13: GANs are free from the multi-modality problem. As this is the motivation of the paper, could you please elaborate on it in more detail?

- Line 319 - 322: An effective sampling method plays a key role in the success of GANs. How should the sampling method be understood in this context?

1. Some of the important details are not clear. 
    - For the generation tasks, it is not clear how the length for the NAR modules is obtained.
    - During inference, the generator gets input from the aligner. However, the input for the aligner is not clear.
2. The motivation for having position-aware self-modulation (PASM) and dependency feed-forward network (DFFN) is not clear.
    - Essentially, PASM needs additional parameters for each position within the maximum length. It is very similar to the learnable positional embedding used in many Transformer models. Further, adding these parameters creates unfair comparisons in the ablation study.
    - In Section 3.3, "builds word dependencies solely based on the attention mechanism by assigning weights dynamically," is not justified. Is the word dependency given by the autoregressive probability decomposition, i.e., $\boldsymbol y = \prod_i p(y_i | \boldsymbol y_{<i}$)? 
    - Additionally, DFFN adds more parameters that are not controlled in the experiments.
3. L307: "Casual" (typo, should be "causal") masks are added to the non-autoregressive models, which is unusual and lacks justification. 
4. It is not clear how the proposed method tackles/alleviates the fundamental problems of NAT, which fails at the "multi-modality" issue. My best understanding is the introduction of latent variables, but it is not clear how the GAN plays a role. Since the aligner module can generate text, it should also be added as a competing variational baseline (after controlling the number of model parameters). 
5. The CMLM-v model is not reasonable, as CMLM itself is able to generate from "all masks." I think SUNDAE (Savinov et al., 2022) is an important NAR baseline for this work. 

**References**
  - Step-unrolled Denoising Autoencoders for Text Generation. Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, Aaron van den Oord. ICLR 2022
  - Non-Autoregressive Text Generation with Pre-trained Language Models. Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, Nigel Collier. EACL 2021
Comments, suggestions and typos: **Questions**
1. Why not just use open-ended text generation, but introduce the new term of IIS? What's the difference?
2. In my opinion, introducing GAN can help solve the multi-modality problem of NAR, which is a general enhancement for NAR. Is its effectiveness not limited to IIS problems? Have the authors tried to conduct experimental analysis on MT tasks?

- Line 147: The definition on $l$ is missing.

**Typo/Writing**
- L289: "can not" -> "cannot" 
- L476: "Transformer" should be "autoregressive Transformer" as all the baselines are Transformer based.
- L474: "It shows the limitations of the biased straight-through estimator." This is questionable as many factors are not controlled.

**Questions**
1. Is the linear layer for the generator's word prediction fixed during training?
2. Lince 384, why is $c$ included as input to the discriminator? Wouldn't this leak the label?
3. L434, how is the classifier trained? Using what data?
4. Can you show the number of parameters for all the competing methods?
Overall assessment scores: 2.5

3

3