Self-Adversarial Learning with Comparative Discrimination for Text Generation
Authors: Wangchunshu Zhou
Tao Ge
Ke Xu
Furu Wei
Ming Zhou
Venue:  (2020)
Keywords: adversarial learning
text generation

Abstract: Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples. To address the issues, we propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. In contrast to standard GANs that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier for comparing the text quality between a pair of samples. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. This self-improvement reward mechanism allows the model to receive credits more easily and avoid collapsing towards the limited number of real samples, which not only helps alleviate the reward sparsity issue but also reduces the risk of mode collapse. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity, and yields more stable performance compared to the previous GANs for text generation.

Published as a conference paper at ICLR 2020
SELF-ADVERSARIAL LEARNING WITH COMPARATIVE DISCRIMINATION FOR TEXT GENERATION
Wangchunshu Zhou' Tao Ge
Ke Xu'
Furu Wei?
Ming Zhou?
Beihang University, Beijing, China 2 Microsoft Research Asia, Beijing, China
thouwangchunshu@bule edu. en, kexuênlsde buw.aaddddduaaadedeeoaaaaaaaaaae edu. cn (tage, fuwei, mingzhou ) @microsoft. com
ABSTRACT
Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples. To address the issues, we propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. In contrast to standard GANs that use binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier for comparing the text quality between a pair of samples. During training, SAL rewards the generator when its currently generated sentence is found be better than its previously generated samples. This self-improvement reward mechanism allows the model to receive credits more easily and avoid collapsing towards the limited number of real samples, which not only helps alleviate the reward sparsity issue but also reduces the risk of mode collapse. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity, and yields more stable performance compared the previous GANs for text generation.
INTRODUCTION
Generative Adversarial Networks (Goodfellow et al. 2014) (GANs) have achieved tremendous success for image generation and received much attention in computer vision. For text generation, however, the performance of GANs is severely limited due to reward sparsity and mode collapse: reward sparsity refers to the difficulty for the generator to receive reward signals when its generated samples can hardly fool the discriminator that is much easier to train; while mode collapse refers to the phenomenon that the generator only learns limited patterns from the real data. As result, both the quality and diversity of generated text samples are limited.
To address the above issues, we propose novel self- adversarial learning (SAL) paradigm for improving adversarial text generation. In contrast to standard GANs (Figure that use binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs comparative discriminator which is a pairwise classifier assessing whether the currently generated sample is better than its previously generated one, as shown in Figure [bb. During training, SAL rewards the generator when its currently generated samples are found to be better than its previously generated samples. In the earlier training stage when the quality of generated samples is far below the real data, this self-imprrvemment reward mechanism makes it easier for the generator to receive non-sparse rewards with informative learning signals, effectively alleviating the reward sparsity issue; while in the later training stage, SAL can prevent a sample from keeping receiving high reward the self-improvement for a popular mode will become more and more difficult, and therefore help the generator avoid collapsing toward the limited patterns of real data.
We comprehensively evaluate the proposed self-ad adersarial learning paradigm in both synthetic data and real data on the text generation benchmark platform (Zhu et al. 2018). Compared to the previous approaches for adversarial text generation Yu et al. 2017] 7 Che et al. 2017 Lin et al. 2017). our
This work was done during the first author's internship Microsoft Research Asia.

---

Published as a conference paper at ICLR 2020
leurning signal
learning signal
penerated sample
Generator
generated sample
Generator
Diseriminator
- real fake
Comparative Discriiinttrr
self-improvemmnnt
Memory
presious sample
previous samples
Standard GANs
Self-adversarial learning
Figure 1: (a) Conventional adversarial learning that uses binary real/fake classifier its discrimina- tor; (b): Self-adversarial learning that employs comparative discriminator to compare the currently generated sample to its previously generated samples for obtaining rewards through self-improvement.
approach shows a substantial improvement in terms of both the quality and the diversity of generated samples as well as better performance stability adversarial learning.
BACKGROUND: ADVERSARIAL TEXT TENERATION
Adversarial text generation has drawn much attention in recent years due to its advantages (e.g.. sequence-level guidance without the exposure bias issue (Bengio al., 2015)) over maximum likelihood estimation (MLE) for natural language generation. It formulates the learning process as minimax game between a generator Go parameterized by 0 and a discriminator Do parameterized by Φ: the discriminator is trained to distinguish between the samples drawn from the real data distribution Pdota and the samples generated by the generator; while the generator is trained to generate samples that can "fool" the discriminator. The adversarial learning objective of the generator and the discriminator can be formulated as:
min max Ez--pin [log Do + Re Dog(1 D&
where is sample from the real data, Gø(z) sample generated by the generator with the initialization that is drawn from the noise distribution Ps (e.g., standard normal distribution).
While GANs have shown some promising results (Yu al. 2017: Guo al. 2018), there two fundamental issues that impede their progress in text generation: (i) Reward sparsity, which is due to the fact that the discriminator tends to learn much better than the generator and thus easily recognizes generated samples as fake. In such cases, it will be difficult for the generator to receive rewards; (ii) Mode collapse, which arises from the intrinsic nature of GANs and leads the adversarial models to only learn the limited patterns from the real samples. These two issues limit the ability of GANs to generate high-quality and diverse text samples, which have not been well addressed yet.
3 SELF-ADVERSARIA LEARNING
To address the aforementioned issues, we propose a novel self-adversarial learning (SAL) paradigm. Inspired by self-play (Silver et al., 2017 Rennie et al., 2017) reinforcement learning, the core idea of SAL is to reward the generator if its currently generated sample is found to be better than its previously generated ones. Like AlphaGo (Silver et al. 2017), the generator in SAL struggles to learn generate better samples than its previously generated samples for passing the "self-improveemen" test by comparative discriminator, which a pairwise classifier trained to compare the quality of two samples, Figure [bb) shows.
Compared to conventional GANs (Figure [)), SAL has the following advantages: First, in the earlier training stage when the quality of generated samples is far below the real data, the self-improvement
2

---

Published as a conference paper at ICLR 2020
better
Real Data
real xentences
indistinguishable
Comparative
Discriminator
train
Generator
indistingusslabble
penerated sentenees
Figure 2: The training process of the comparative discriminator.
reward mechanism of SAL allows the generator to receive informative learning signals more easily as it makes the assessment of sample quality better adapted the current capability of the generator, making it less likely to suffer from the issue reward sparsity; Second, in the later training stage when the generated samples' quality is high, SAL can prevent a sample from keeping receiving high reward as will become more and more difficult to pass the "self-improvemmen" test, thus reducing the risk of the generator collapsing towards limited patterns. The ceff-improvement mechanism and the "tie" option in the comparative discriminator also provides reasonable baseline which corresponds to cases where newly generated samples are found be indistinguishable with previous ones, thus improving the training stability. We provide more detailed qualitative analysis of why the proposed adversaaelllial learning paradigm can alleviate these problems in Appendix.
3.1 COMPARATIVE DISCRIMINATOR
As introduced above, the component for SAL is the comparative discriminator, which is pairwise classifier comparing the quality of two samples. It learns total order of sample quality and encodes the inductive bias that one sample is better (>), worse (<), or indistinguishable (<) in terms of its quality compared to the other. For (text) sample, the comparative discriminator can offer more informative learning signals than the conventional binary (i.e., real/fake) classification based discriminator because the sample can receive multiple feedbacks from the comparative discriminator by comparing it with multiple other samples.
For training the comparative discriminator, we construct pairwise training examples from the real and generated samples, as Figure shows. For real sample 8., and generated sample s., we assign the label "better (>)" to the pair (84, s-) and "worse (<)" to (s_, 8.). For two samples both from real data or from the generated samples, we assign the label "indistinguishable (<e)" to such pairs (i.c., (si.. sf.)) and (s'.. s )). For a training set with n real samples and n generated samples, the comparative discrimination can construct (?r) pairwise training examples, allowing to enhance the generalization ability of the comparative discriminator.
Moreover, to improve the model's ability to distinguish between good generated samples and bad generated samples for self-play learning, we additionally select the samples generated during the later stage of MLE training pseudo-real samples, and select those generated in the earlier epochs when the generator does not fully converge as fake sentences. We then pair the pseudo-real samples with the fake samples to construct training instances to supervise the model to compare their qualities. In this way, the comparative discriminator is prevented from being taught to always recognize two generated samples as equally bad and assign reward to the generator. As result, it can become more sensitive to the quality difference in a pair of text samples and thus allow the generator to receive rewards more easily.
3

---

Published as a conference paper at ICLR 2020
3.2 TRAINING
Before we formally introduce the training procedure for SAL, we first define the learning objective for the comparative discriminator D3 and the generator Gg SAL:
Ln -B( (2M (x)) log (( m2)) (111, 222)
LG - (3)
W log D (g(Go(z),
(3)
gE(>,<R)
In Eq 2 and Eq (3) M is the set of previous generated samples by the generator, Q 22) E (>, < is the true label for the pair (ær, 22), 1 (2), 222) is the probability of the comparative discriminator's prediction being q (q E {>, <, <.+] for the pair (æ1, 22). Wgg the reward weight for the case q, which is hyperparameter for SAL. If the generator generates a sample Gø(z) that is better (>) than its previously generated sample ær, receives positive reward; Gz(z) is worse (<) than F,,, it receives negative reward; while if the quality of Gø(z) is classified as similar (R) to
Zr. receives zero credit. Therefore, we have (()) > 0 = ((=) > ((<).
Since LG can only be directly optimized in standard continuous GAN training, alternatively ploy the policy gradient algorithm (Sutton et al. 2000) to train the generator, as previous approaches for adversarial text generation. For SAL, define the reward for generated sample Ty compared with a reference sample I, which a previous generated sample by the generator as the weighted reward based on the probability distribution of the prediction by the comparative discriminator:
(Ig. No (= x,) D(>) (zy x.) w(<) (Ig 2,) + w(2) (zg.
In text generation, the generator Go obtains the reward only when one sample has been completely generated, which means no intermediate reward gained. To relieve this problem, following the practice in SeqGAN Yu et al. 2017). utilize the Monte Carlo rollout method to approximate intermediate rewards by sampling unknown tokens with generated prefix Y1 following generator policy Gg till sample completion. Empirically, found that the Monte Carlo rollout also helps to reduce the variance the reference sample utilized for comparison. We calculate the expected reward
(s Y1a-1,a== vt) (G æ.)]
The objective of the generator is to generate a sequence to maximize expected final reward. With likelihood ratio (Sutton al. 2000), we can formulate the gradient of the objective function for generator Go as:
VoJ(O)
Σ t=1
E Ey Vo VoggoGGe Re, Y1a-1, ye)l
(6)
To improve the training of self-adversarial learning, we borrow ideas from the field of deep reinforce- ment learning and propose two training techniques improve self-adversarial training.
Scheduled rewarding Similar to the peploitation-explrratti trade-off in reinforcement learn- ing (Langford & Zhang, 2007), the positive reward assigned for actions generating better samples encourage exploration while the penalty for generating worse samples makes the generator more conservative. Intuitively, in the earlier stage of self-adversarial learning, the generator should explore better policy by receiving higher rewards for relative progress; while in the later stage, the generator should be more conservative by penalizing more for worse samples to avoid performance degradation. We simply decrease ((>) and increase ((<) linearly with training iteration and refer this technique scheduled rewarding.
4

---

Published as a conference paper at ICLR 2020
Algorithm 1 Self-Adversarial Learning With Comparative Discriminator
Require: Generator Ga: comparative discriminator D&; samples of real sentences S4 self- adversarial learning step g; discriminator step memory buffer for the previous generated samples
1: Pretrain Go using MILE on S+
2: Generate samples with Gg and store them into M
3: repeat
4: for steps do
Collect a mini-batch of balanced sample pairs (#) from M US+
6: Update D3 via Eq
7: end for
8: for steps do
Generate a mini-bateh of samples 2, ~ G.
10: Collect a mini-hatch of reference samples at, from M
11:
Update Go via Eq (6)
12:
end for
13:
Update .M with Ge
until Convergence
Memory replay Continuously comparing the generator with its most recent stage may suffer from the correlation between generated samples and reference samples, which makes the training process unstable. Inspired by experience replay (Lillicrap et al. 2015), we construct a memory buffer which contains samples generated in the last K training steps. Reference samples are sampled from the memory buffer rather than samples generated in the most recent stage of the generator, which empirically helps stabilize the training process.
The training process of SAL is summarized in Algorithm 3 Self-adversarial learning with the proposed comparative discriminator achieves Nash Equilibrium when the generator models the distribution of real samples perfectly. In this case, the comparative discriminator cannot successfully distinguish generated samples from real samples and tends to recognize two samples as "indistin- guishable". The reward received by the generator is thus zero and training converges. However, how a non -en-oornoulli GAN converges to such an equilibrium is still an open problem (Goodfellow et al. Goodfellow 2014) and is beyond the scope of this work.
2014
4
EXPERIMENTS
4.1
EXPERIMENTAL SETTING
Following the experimental settings in previous work (Lin et al. 2017 Guo et al. 2018 Shi et al. 2018. Zhu al. 2018 Nie et al. 2018). evaluate approach in both synthetic and real datasets based on Texygen (Zhu et al. 2018), which a benchmark platform for evaluating adversarial text generation models. Table presents the basic information of the datasets used for evaluation.
Table 1: Description of the datasets used for evaluation
Synthetic
Image COCO
EMNI. EMNI.22017 P2017 WMT NEWS
Category Vocahulary size
simulated data
image description
article
5000
4682
5255
Sequence length
20/40
<37
<51
Sentence number (training)
10000
10000
270000
Sentence number (test)
10000
10000
10000
As SeqGAN (Yu al., 2017), our generator single-layer I.TTM (Hochreiter Schmidhuber 1997) and our discriminator almost based TexICNN (Kim) 2014] except that it concatenates the feature representation of two compared samples and outputs the probability for their comparative relations (i.e., >, <, z). We keep the most of the hyperparameters same with the SeqGAN except the hyperparameters introduced by our models (i.e., ([>)) W(<), W(,s) which are tuned based on the synthetic experiment and kept the same for the real data experiments.
We evaluate the adversarial text generation models terms of both quality and diversity. Following the prior work, in the synthetic dataset, use the oracle L.TTM to evaluate the negative log-likelihood
5

---

Published as a conference paper at ICLR 2020
Table 2: Performance comparison of different models in synthetic tests where sequence length is set to 20 and 40 respectively. For all the metrics presented, the lower, the better.
Method
NLLorade
(204g).(2040)
NLLovadte (2Lg)a(20040)
SAL
7.71 #0.17 / /9.11-00000
6.5820.15/ 6.97 20,05
14.29 #0.11 100.11/16.244 16.24::0008
SeqGAN
8.63 #0.19 20.119999... 9:63 #0.04
6.61#0.22/ (000:/2//66.00 6,98=0.08
15.00 16.35=00002
RankGAN
8.42 10.31 /9,52.0011
7.1410.34/7.05-0.1 12
15.01 16.37.10.02
MaliGAN
8.74:0.16/9.6 10.03
6.62::0.25
15.03=0.03
16.39:40.03
MLE
9.05 #0003/994400000
5.96:0.02 6.55=0002
15.02=0.03 / 16.39:#0.01
of our generated samples (denoted as NLLoracle) as the metric reflect quality; for the diversity metric, we use the negative log-likelihood of the synthetic dataset (denoted as NLLgen) evaluated by the generator with the best quality (i.e., the best NLLoracte score) during training. We also use the best NLLoracle + NLLgen obtained during training to evaluate the quality-diversity trade-off.
For the real data experiments, we follow the previous work to apply the commonly-use used BLEU scores (Papineni et al., 2002) (BLEU(F)) and the perplexity of generated samples evaluated by an open-sourced pretrained language model (Jozefowicz et al. (016) as quality metries since NLLoradde cannot be evaluated without an oracle language model. For evaluating diversity, we employ both backward BLEU (Shi et al. 2018) (BLEU(B)) which evaluates the test data using generated samples as reference and NLLgen as the metrics. To evaluate the generated samples in more aspects, calculate frechet distance (Heusel et al. (017) (FD) between generated samples and real data with sentence representation obtained by InferSent (Conneau et al. 2017) which is a pre-traaee pre-trained sentence embedding model.
We compare our approach to previous well-known adversarial text generation models including SeqGAN (Yu et al. (017), RankGAN (Li et al., 2017) and MaliGAN (Che et al.] (017). Leak- GAN (Guo al. 2018) and ReIGAN (Nie al.) 2018 focus on architecture-level modification, which is orthogonal to our work and the proposed self-adversarial learning paradigm can be applied to them as well. We provide results of the combination of LeakGAN with SAL in the Appendix.
In the following sections, we denote our proposed self-adversarial learning approach as SAL. Since adversarial training is very sensitive to random initialization and suffers from high variance, we conduct five individual runs with different random seeds for each model and report the mean and the standard deviation of the obtained results.
4.2
EXPERIMENTAL RESULTS
4.2.1 RESULTS IN SYNTHETIC DATA
Table 2 shows the results in the synthetic dataset. We can observe that SAL largely outperforms the previous GANs in all metrics in both cases of sequence length 20 and 40. Although its performance in NI.Lgen is worse than MLE as MI.E directly optimizes the metric, yields better quality- trade-off than MI_E training, which has not been achieved by the previous GANs, which is shown by the fact that the NLLoracte CetrracteeNLL.,. +NLLgen for SAL is lower than that yielded by MLE, while other GANs have the same after pretraining. This demonstrates SAL's advantage in alleviating the mode collapse problem. We also find that the training of SAL is more stable compared with other text GANs.
score with MLE, indicating that they fail to improve the quality-diversie trade-off
In addition, we find SAL learns faster and better than the other GANs by comparing their performance curves of NLLoracte during training in Figure 3, which is consistent with our intuition that the self- improvement reward mechanism in SAL can alleviate the reward sparsity issue in the earlier training stage and help the generator learn better.
4.2.2 RESULTS IN REAL DATA
The results for COCO image caption dataset are presented in Table and the performance curve of the perplexity during training is shown in Figure 4. As in the synthetic data, we observe that our SAL consistently yields better results in all the metrics with stable performance (i.e., low variance) compared to the previous GANs. According to Table and Figure 4, SeqGAN and our SAL can improve MLE in the quality metrics (i.e., BLEU (F) and Perplexity) while MaliGAN and RankGAN

---

Published as a conference paper at ICLR 2020
Leerning Curre
Lewniing Cures
10.0
peoplean
6.-
+50
8.5
303
5.0
Macel
253
103
350
230
253
70
100
110
120
Epocto
Epodas
Figure 3: The performance curves of NLL-oracle Figure 4: The performance curves of perplex- ity during training in the Image COCO dataset
during training in the synthetic dataset
Table 3: Performance comparison of different models in the COCO caption generation task. Metrics from top to bottom represent respectively the generation quality, the generation diversity, and the divergence between real data of generated sentences. For all the BLEU metrics, the higher, the better; for NLLggn and FD, the lower, the better.
Metries
MIE
SegGAN
MaliGAN
RankGAN
SAI
BLEU-2(F)
0.730 =0.01
0.748 +0.03
0.733 +0.03
0.727 =0.04
0.785 +:002
BLEU-3())
0.494 #0.01
0,514 +0.01
0.497 #0.01
0.491 =0.04
0,581 =0.03
BLEU-4(F)
0.303 +0.01
0,307 +.002
0.295 +0.03
0.291 +0.03
0.362 #0.02
BLEU-S(F)
0.187 10.01
0.187 10.02
0.178 20.02
0.175 =0.03
0.227 10.02
Perplexity
338.4 17.66
307.2 1:4.9
343.8 12113
391.2 135.1
231.3 110.8
BLFU--((B
0,759 +.0.2
0.694 4-0.03
0.676= 00
0.683++0.04
0.724 +-002
BLEU-3(B)
0.531 00.02
0.472++0.03
0.443=0.03
0.449:0004
0,503 +0.03
BLEU-K()
0,332 0.00
0,285:::0002
0.279=0.02
0.277:0.02
0.313
BLEU-S(B)
0.209
0.186:10.02
10.02
10002
0.198 1.002
NLLgen
0.721 00.02
1.035 +0.02
1.052 #0.02
1,145 =0.02
0.873 =0.02
FD
0.043 1:0.009
0.065 10.0188
0.076 1.0021
0,083 1:0027
0.051 0.0014
Table 4: Performance comparison of different models in the EMNLP2017 WMT news generation task. Metrics from top to bottom represent respectively the generation quality, the generation diversity, and the divergence between real and generated data. For all the BLEU metrics, the higher, the better. For NLLgen and FD, the lower, the better.
Metries
MLE
ScqGAN
MaliGAN
RankGAN
SAL
2FF)
0.769 10.02
0,761 1:003
0.764 10.03
0.736 10.02
0.788 10002
3LFU--(F)
0.475 +0.01
0.463 +-0.03
0.468 +0.03
0.441 +0.04
0.523 +-0.02
BLEU-4(F)
0.243 +0.02
0.228 +0.03
0.231 +0.03
0.204 +0.02
0.281
+:000
BLEU-S(F)
0.124 20.02
0.115 +0.02
0.113 20.03
0.095 20.02
0.149
10.02
Perplexity
702.6 18.66
743.8 +34.2
825.1 +-44.3
975.4 165.1
612.8 +:22.55
BLEU-2(B)
0.741 +0.02
0.693 1:003
0.684=0.03
0.67110.03
0.726 02
BLEU-3(B)
0.4761:0.01
0.413:10.03
0.391=0.044
0.373
0.431
B(B
0.245++0.01
0.216-40.03
0.197=0.03
0.191
0,232-40.02
BLEU-5(B)
0.129=0001
0,112
0,094=0.00 02
0.096+#0.03
0.123++0.00 02
NLLgen
2.38620.01
2.7324:0.01
2.862 .6
3.157=0.11
2.578:1:0.00 04.0
FD
0.079 +0.012
0.172 +0.032
0.194 +0.043
0,219 +0.052
0.137 -0.023
perform comparably to MLE. However, in the WMT NEWS dataset where text samples tend to be longer, we observe something different from Table 4 all the previous GANs fail to improve MLE, This is because long text generation makes the discrepancy between generated samples and real samples very large even after MLE pre-training. As result, previous GANs fail to stably enhance the sample quality due to the reward sparsity issue. In contrast, our SAL consistently performs well and largely improves the quality metrics over MLE. In addition, observe that the diversity of samples generated by SAL is much better than previous GANs and only marginally worse than MLE, indicating SAL is helpful in reducing the risk of mode collapse.
In addition to the automatic metrics, we also conduct human evaluation for the generated samples. As previous work (Nie et al. 2018), we randomly sample 20 sentences from each model and pool them
7

---

Published as a conference paper at ICLR 2020
Table 5:
Human evaluation results of different models in both datasets. Scores between 1-5, higher score indicates better quality.
Dataset
MLE
SegGAN
MaliGAN
RankGAN
SAL
COCO
2.96.10.51
3.26=0.56
3.14=0.57
2.91:0062
3.8410.56 (pc=0.01) 2.689:0..9 (pc=0.01)
WMT NEWS
2.35:10.86
2.19-0.88
2.24 0.87
0.05
Table 6: Results of the ablation tests in the synthetic data and the COCO dataset.
Dataset
SAL
CAL
who comparative
who "Re"
wlo acheduled revarding
we memory replay
Synthetic (NI..)
15.01 40.04
14.85 +0.16
14.46-+0.18 248.3 1142
14.41 +0.17 254.6 113.7
COCO (Perplexit))
231,3.10.8
276.7112.5
3418 113.4
291.5 116.3
with anonymizing the models' identity. We invite 20 graduate students with good English proficiency score each sentence on scale of 1-5 in terms of quality. According to Table 5 our SAL is consistently well rated in the human evaluation and outperforms all the baselines in both COCO and WMT NEWS datasets. We perform the Wilcoxon Rank Sum Test with the human evaluation results and find that samples generated by baseline models can be distinguished from samples generated by SAL with p < 0.01. Details of human evaluation procedure and samples generated by compared methods in two real-world datasets are presented in the Appendix.
4.3 DISCUSSION
To better understand SAL, we perform multiple ablation tests in both the synthetic and the real data. We employ NLLoracte + NLLgen score with sequence length 20 as the evaluation metric for the synthetic data, denoted as NLL.. For the real data, we use the perplexity of generated samples trained with COCO dataset as the evaluation metric. We compare SAL with the following reduced models:
CAL: Replacing the comparison between the generated samples (i.e., self-play) to the comparison between the real and generated samples.
w/o comparative: Using the binary discrimination scores of other generated samples as baseline for the policy gradient algorithm, which can be considered as a combination the self-critic training (Rennie et al. 2017) with RL-based text GANs.
w/o "Rs":
Replace the three-class comparative discriminator with binary comparative discriminator by removing the "x" class.
w/o scheduled rewarding and w/o memory replay
The results of the ablation tests are shown in Table 6 By observing the improvement by SAL over CAL, we confirm the importance of the self-play paradigm in SAL. It is notable that the proposed comparative discriminator alone (i.e., CAL) can yield good performance, demonstrating the effectiveness of learning by comparison. When replacing the comparative discriminator with the naive combination of self-critic baseline with text GANs, the performance largely decreases because the reward sparsity issue will be intensified when subtracting two already sparse rewards, this motivates the proposed pairwise comparative discriminator which makes self colf-oopaarsonn possible.
In addition, we find that the ">" option plays a critical role improving the result, without which the performance degrades significantly because it makes the task less trivial and provides a bascline for the policy gradient algorithm. Moreover, the training techniques (i.e., scheduled rewarding and memory replay) borrowed from deep reinforcement learning are also shown useful in improving the results but not so important as the core components (e.g., self-play and the comparative discriminator).
RELATED WORK
Many variants of GANs (including TexIGAN (Zhang et al. 2017). GSGAN (Kusner & Hernández,
Lobato 2016). SeqGAN (Yu et al. 2017), MaliGAN (Che al. (217)), RankGAN (Lin et al. 2017). FMGAN (Chen et al. 2018), LeakGAN (Guo et al. 2018), and RelGAN (Nie et al. (0188) have
been proposed for text generation as adversarial training has received increasing attention in recent
8

---

Published as a conference paper at ICLR 2020
years. Typically, they address the non-differentiable issue by making continuous approximation or reinforcement learning. These approaches introduce several different architectures and optimization objectives of both the generator and the discriminator for adversarial text generation. Among the previous studies for adversarial text generation, the most related work to ours is RankGAN (Lin al., 2017) which proposes ranker replace the conventional binary classifier its discriminator for allowing the discrimination process to involve richer information. Another work whose idea is similar to ours the relativistic discriminator (Jolicouu--Mardnn Marttneaul 2018) (RGAN). It compares binary scores assigned to generated samples and real samples by subtraction as the learning signal to implicitly represent the inductive bias that half of the samples received by the discriminator is fake. In contrast, our comparative discriminator directly encodes this inductive bias and assesses generated sentences by comparison with a pairwise classifier, which provides more informative learning signals than subtraction in RGAN (olicoeurMMattnnau (018) and normalized feature similarity in RankGAN (Lin et al. 2017). Our work is also related to the concurrent work (Zhou & Xu 2020) that learns comparative evaluator to evaluate open-domain natural language generation models.
6 CONCLUSION AND FUTURE WORK
We present a self-adversarial learning (SAL) paradigm for adversarial text generation. SAL rewards the generator when its comparative discriminator finds the generator becomes better than before. Through the self- improveenntt reward mechanism, the problem of reward sparsity and mode collapse can be alleviated and training of text GANs are more stable, which results in better performance in the text generation benchmarks in terms of both quality, diversity, and lower variance. In the future, we plan to generalize our approach to other domains and modals to explore the potential of SAL for adversarial learning. Generated samples are presented in the Appendix together with other details, including human evaluation details and qualitative analysis of the proposed SAL.
REFERENCES
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 1171-1179, 2015.
Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio.
augmented discrete generative adversarial networks. arXiv preprint arXiv:1702.07983 2017.
Liqun Chen, Shuyang Dai, Chenyang Tao, Haichao Zhang, Zhe Gan, Dinghan Shen, Yizhe Zhang, Guoyin Wang. Ruiyi Zhang, and Lawrence Carin. Adversarial text generation via ceature-movert distance. In Advances in Neural Information Processing Systems, pp. 4666-4677, 2018.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint TxXiv:7705002366 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Ian J Goodfellow. On distinguishability criteria for estimating generative models. arXiv preprint arXiv:1412.6515, 2014.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637 2017.

---

Published as a conference paper at ICLR 2020
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 7735-1780, 1997.
Alexia
The relativistic discriminator: key element missing from standard gan. arXiv preprint arXiv: 1807.00734, 2018.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv: 2016.
Yoon Kim. Convolutional neural networks for sentence classification arXiv preprint arXiv: 1408.5882. 2014.
Matt J Kusner and José Miguel Herándeez-oobtto Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint 1rXiv:1611.040333 2016.
John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi- armed bandits. In Proceedings of the 20th International Conference on Neural Information Processing Systems, pp. 817-824. Citeseer, 2007.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2157-2169, 2017.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint ww.Xiv:1099022.
2015.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. Adversarial ranking for language generation. In Advances in Neural Information Processing Systems, pp. 3155-3165, 2017.
Weili Nie, Nina Narodytska, and Ankit Patel. Relgan: Relational generative adversarial networks for text generation. 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10. 10.1109/cypr2201..131
httphttp:// dow..og//10000.
1109/CVPR. 2017.017..10 131
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Towards diverse text generation with inverse reinforcement learning, arXiv preprint arXiv: 1804. 2018.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in neural information processing systems, pp. 1057-1063, 2000.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Ad- versarial feature matching for text generation. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4000-4015. JMLR. org, 2017.
Wangchunshu Zhou and Ke Xu. Learning to compare for better training and evaluation of open domain text generation models. In WThirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097-1100. ACM, 2018
10

---

Published as a conference paper at ICLR 2020
A GENERATED SAMPLES
We present sentences generated by our proposed model and compared models to provide qualitative evaluation of different adversarial text generation models. From the presented generated samples, we can observe that samples generated by MLE training are less realistic compared with other samples. SeqGAN yield slightly better sample quality but the loss diversity is observable even within randomly sampled 15 sentences. Adversarial training with proposed comparator, when trained by comparing with real samples, yield better quality but still lack of diversity. Finally, with the proposed self-adversarial learning paradigm, both quality and diversity of generated samples improved.
A.1 GENERATED SAMPLES IMAGE COCO DATASET
Table 7: Samples generated by SAL in Image COCO dataset
a picture of a person 's umbrella in a cell phone.
man stands a green field.
young boy riding truck..
man on a motorcycle is flying on grassy field.
girl on a motorcycle parked on a city street.
a motorcycle parked in city street. group of bikers riding bikes on a city street.
kitchen with cat on the hood and street. bathroom containing toilet and sink.
a young woman in a kitchen with a smiley face jet plane the side of street.
a dish is sitting on a sidewalk next to baby giraffe.
dog on large green bike parked outside of the motor bike.
person on a kawasaki bike on a race track.
commercial aircraft is parked in front of a kitchen.
Table 8: Samples generated by CAL in Image COCO dataset
a man is on towel on table outside of a real kitchen.
group of lambs
tall building.
a young boy riding a truck.
man on motorcycle is flying on a grassy field.
a man with a computer desk next to white car
a cat is on the walls cat
plane on runway with a plane
an elegant, dilapidated plane are standing in front of a parking bag.
the woman is riding bike their way
a man wearing an old bathroom with a banana
plane is taking off from the ground.
a man holding a man in front herself.
woman walking across the road.
a kitchen with an island in green tiles.. a clean kitchen with two small appliances.
11

---

Published as a conference paper at ICLR 2020
Table 9: Samples generated by SeqGAN in Image COCO dataset
a large image of a herd of racing train.
man and woman on horse. a plane runway with a plane
a man preparing table with wood lid
view tiled floors and man prepares food.
a man wearing an old bathroom with a banana.
man is is with camera.
two people are parked on a street.
white and white black kitten eating on table.
a toilet is lit on the walls.
kitchen taking off from window
a man is wearing glasses wearing scarf.
kitchen with graffiti hanging off from an open plain.
two women playing with the orange.
a kitchen with an island clear glass
Table 10: Samples generated by MLE in Image COCO dataset
jet airplane flies flying through front from an airplane. a furry tub and overhead pot.
man in kitchen filled with dark lights green side a cross baby field dressed making cardboard a bathroom with small tub and oven
man above bathroom with oven room.
a jet airliner flying through the sky kitchen with dishwasher and plenty of pots pans
a person holding onto two red era arena sits on the street. bathroom with toilet and bath tub.
a cat perched on the phone and a baseball cap the view of the street filled with really parked the gates on the road a large hairy dog on a high bike with a cake
a man is riding a white back bench.
narrow bed and white spotted dark tiled walls.
A.2 GENERATED SAMPLES IN EMNLP2017 WMT DATASET
Table 11: Samples generated by SAL in EMNLP2017 WMT dataset
(1) it' likely to be egyptian and many of the canadian refugees, but for decade
(2) the ministry spokesperson also said it now significant connected to the mountain.
(3) it is the time they can more competitive where we have another 99. 100 per cent, and completely the alternative and that being affected.
(4) we expect 200 and 0. 3 percent for all you form other and, which then well. done
(5) so we wouldn t feel very large in the game, but you fail to fund, and and the paper that like its start.
(6) other countries made playoff cut with pages by mrs trump s eighth consecutive season
as a president.
12

---

Published as a conference paper at ICLR 2020
Table 12: Samples generated by CAL in EMNLP2017 WMT dataset
(I) i didn 't put relatively quiet, we have, his work right in the particular heat rate take steps traditionally clean.
(2) why the u. 8. then the table is our cabinet to do getting an vital company for the correct review
(3) those had trained for that, but no thin percentage of the nhs about being warned about the palestinian election before obama is not connected in israel.
(4) in course voters obama said torture is the outcome. the most powerful trade popularity is happening in as a success
(5) " in 2012, is nice remain- trump actor established this night scoring three films
(6) we kind of not listen to knowing my most one, ,only. for really good vote and where things fun, you know.
Table 13: Samples generated by SeqGAN in EMNLP2017 WMT dataset
(I) his missed 4, 000 the first 95 really 69 year olds
(2) but just things you want to thank it as my playing side has begun meeting with "and" the
score had to train up so he was tied for 11 years
(3) and when he got back doing fresh ties with his election, he will now step in january, back.
(4) when you know if saw her task find himself more responsibility ago
(5) his hold over up to nine hike in 2015, 13 percent of recently under suspects dead day.
and to the city..
(6) i look up by the city" vehicle on the day in meeting in november.
Table 14: Samples generated by MLE in EMNLP2017 WMT dataset
(1) you know that that is great for our ability to make thinking about how you know and you ?
(2) when it s real thing possible, is if you the first time in time here and get.
(3) u. now government spending at the second half of four years, country where the law
will join the region to leave japan in germany.
(4) deputy president, the issue of government and geneva probe threats and not backed trump
but well - changing violence for their islamic state militants were innocent people.
(5) he suggested in a presidential primary source and comment its size following protests conducted by 18. some in 2012 will be looked at tech energy hub
(6) " growing heavy hard, mr. romney said, he says matters that can again become
the asian player.
13

---

Published as a conference paper at ICLR 2020
B CASE STUDY: WHY IT WORKS
In this section, we present several qualitative case study examples to illustrate why comparative discrimination and self-adversarial learning helps mitigate the problem of reward sparsity and mode collapse. We extract a typical sentence generated during the initial stage of adversarial learning: "a man holding a suitcase holding phones." We see that this sentence is of limited quality and is easily recognized by binary discriminator in SeqGAN with high confidence, this makes the credit received by the generator very sparse and makes training difficult. Comparative adversarial learning (CAL) where we use the comparative discriminator to assess the quality of this sample by comparing it with a real sample helps as comparative discrimination have three catagories, which is less trivial. The improvement is not much as the discrepancy of generated samples and real samples is fairly large. However, with proposed self-adversarial learning paradigm, the comparative discriminator assesses this sentence by comparing with previous generated sentence which is also of poor quality. The self-improvement is easier to be recognized by the comparative discriminator and makes this sample get good rewards.
As the comparative discriminator has to learn a total order of sample quality which is more challenging than standard binary discrimination, the chance of the comparative discriminator he over-trained is reduced, which makes the model casier to achieve self-imprrvement, thus help to alleviate the reward sparsity problem.
We also extract a sentence generated in the late stages which is fairly good and fools the binary discriminator: woman sitting bench on a park." In standard adversarial text generation models, a sentence like this would keep receiving large rewards and result in mode collapse. In self-adversarial learning paradigm, this sentence is not much better than other sentences generated by the generator itself, so its reward is limited, which reduces the risk of mode collapse.
Table 15: Case study of comparative discrimination and self-adwersarial learning.
Generated sentence
Reference sentence
Reward
man holding a suitcase holding a phones.
(SeqGAN)
0.018
man holding suitcase holding phones.
student walks rain with green umbrella. (Real)
0.051
holding suitcase holding phones.
kitchen and cow. (Self)
0.561
woman sitting bench on a park.
(Se
0.825
woman sitting bench park.
young man rides his bicycle top cement bench. (Real)
0.438
sitting bench park.
silting on table watching television. (Self)
0.086
B.1 ABLATED MODELS
For the ablated model variants, SAL w/o self-play and w/o comparative discriminator is trained with the following algorithms. Specifically, the difference between SAL and CAL is that the reference sample which is compared with the currently generated sample is replaced by a real sample instead of previously generated one. For the variant without the comparative discriminator, binary discriminator trained in the same way with the vanilla GAN, as for the reward of generating Lg. we first sample previously generated sample H, reference and calculate the reward as D(æ,) - D(æ,).
employ
14

---

Published as a conference paper at ICLR 2020
Algorithm 2 Self-Adversarial Learning without self-play (i.e. CAL)
Require: Generator Ga: comparative discriminator D&; samples of real sentences S4 self-aadvesarial learning
step g; discriminator step memory buffer the previous generated samples
1: Pretrain Go using MI.E on S+
2: Generate samples with Gg and store them into M
3: repeat
4: for steps do
Collect a mini-batch of balanced sample pairs (#) from M US+
6: Update D3 via Eq
7: end for
8: for steps do
Generate a mini-batch of samples 2, ~ G 10: Collect mini-hatch of reference samples at, from S+
11:
Update Go via Eq (6)
12:
end for
13:
Update .M with Ge
until Convergence
Algorithm 3 Self-Aadersarial Learning without comparative discriminator
Require: Generator Go: binary discriminator Da: samples of real sentences S4; self-adversarial learning step discriminator step k; memory buffer M for the previous generated samples
1: Pretrain Go using MLE on S+
Generate samples with Gs and store them into M
3: repeat
4: for k steps do
5: Collect mini-hatch of generated samples from M.
6: Update D3 with conventional GAN discriminator loss
end for
8:
for steps do
9:
Generate mini-batch of samples ~ Gs
10: Collect mini-batch of reference samples æ. from M
Update Go Eq (6)
12:
end for
13:
Update M by Go with reward calculated by D(xg)= D(æ,)
14:
until Convergence
C TRAINING AND EVALUATION DETAILS
C. MODEL DETAILS
We follow most of the hyperparameters used the benchmark platform Texygen Zhu et al. (2018). Specifically, the generator is one layer LSTM with embedding size and hidden size 32. The discriminator is implemented as a TexICNN with a filter size of [2,3] and a filter number of [100,200]. The proposed self-at adversaril learning paradigm introduces the relative weights for credit assignment when a generated sample found to be better, indistinguishable or worse compared with another sample generated by the generator itself. We tuned it based on the performance in synthetic experiment and set Wp- 12 = -..1.
C.2 CHOICE & EXPLANATION METRICS
Note that many previous works use self-BLEU Zhu al. (2018) as a diversity metric. However, we find that there exist problem in the official implementation of the self-BLEU metric: Only in the first time evaluation that the reference and hypothesis come from the same "test data" (i.e. the set of generated sentences). After that, the hypothesis keeps updated but the reference remains unchanged (due to fistfisst=aals"",, which means hypothesis and reference are not from the same "test data" any and thus the scores obtained under this implementation end, we modified the implementation to make sure that the hypothesis and reference are always from the same "test data" (by simply removing the variables "self.reference and "self.is-first" and found
not self- BLE- scores. To this
15

---

Published as a conference paper at ICLR 2020
that the self-BLEU (2-5) scores always 1 when evaluating all the models. This problem is also discussed in the openreview of the RelGAN Paper|Nie et al. (2018).
Based on this consideration, (2018) and can also evaluate the diversity of generated samples. The forward and backward BLEU resemble precision and recall of generated samples with respect to the test data, which measures the generation quality and the generation diversity respectively.
decide to employ Bockward-BLED which is introduced in Shi et al.
For BLEU metric, there is sentence level alignment for unconditional generation, BLEU is evaluated by using the entire test set treated as a single reference, which contains 10000 sentences. We then generate the same number of sentences the prediction, and then compute n-gram overlap between the reference and the prediction. We did not apply brevity penalty following previous works. But we found the number of tokens generated are roughly the same across different compared models.
We briefly explain why NLLgen is able to measure the diversity of the generator: NLLgen measures the negative log- likelihood of the synthetic dataset evaluated by the generator. Intuitively. the generator is diverse and captures more patterns in the synthetic dataset, the NLL.gen will be lower. In contrast, if the generator suffers from severe mode collapse and is of low diversity, NLI will be higher as the generator fails to cover the diverse patterns in the training data.
As for the metric: N.L. our motivation that NLLorade measures the best quality of the generator throughout training, while NLLgen measures the best diversity attained by the generator during training. However, the best quality and diversity are generally not achieved at the same time as GAN-training generally sacrifices the diversity for better quality, Therefore, report +LLggen+NLLoog which can measure the diversity trade-off, as the previous work demonstrated, additional reference.
C.3 HYPERPARAMETERS
We follow most of the hyperparameters used the benchmark platform Texygen Zhu al. (2018). Specifically, we choose batch size to be 64, dropout keep prob to be 0.75, 12 regularization to be 0.2. pretrain all model for 120 epochs and fine-tune them until convergence.
The proposed self- aalf-alversarial learning paradigm introduces the relative weights for credit assignment when a generated sample found to be better, indistinguishable or worse compared with another sample generated by the generator itself. We tuned it based on the performance in synthetic experiment and find Wo W2 = 1 -0.1 to be a good choice for the initial weights and fixed wj to be O. We empirically find that the performance of SAL is not very sensitive to the choice of reward weights as long as the absolute value of w1 is larger enough than W2, which guarantees the training stability.
C.4 HUMAN EVALUATION
Nie Following the human evaluation setting in RelGAN Nie et al. (2018). The text quality evaluation
based on grammatical correctness and meaningfulness (i.e. sentence makes sense), text formatting problems (c.g., capitalization, punctuation, spelling errors, extra spaces between words and punctuations) are ignored. Detailed criteria is provided as follows:
Wi is linearly decreased 0.8 and wo increased -0.2
16

---

Published as a conference paper at ICLR 2020
Table 16:
The human evaluation scale from to 5 with corresponding criteria and example sentences.
Scale
Criterion & Example
5-Excellent
Its grammatically correct and makes sense.
For example, "a man standing on a skateboard the middle the street." It has small grammatical errors and mostly makes sense. For example, "two women is in a cafe look outside." It has major grammatical errors but the whole still conveys meanings. For example, "a man riding on on motor scooter and window." It has severe grammatical errors and the whole doesn't make sense, but some parts are meaningful. For example, "a blue bike on on dirt bike !" It is basically random collection of words. For example, "a motorcycle close close on and
4-Good
3-Fair
2-Poor
1-Unacceptable
C.5 ADDITIONAL RESULTS ON CAL AND LEAKGAN
In this section, present the performance comparison SAL CAL (comparative adversarial learning, which uses comparative discriminator but does not train the model with self-play.) and the application of SAL on LeakGAN. We find that SAL significantly outperforms CAL on all dataset. In addition, we find that the proposed self-adversarial learning paradigm can also be applied on other text GAN architectures, e.g. LeakGAN, and help improve its performance.
Table 17: Performance comparison of different models in synthetic tests where sequence length is set 20 and 40 respectively. For all metrics presented, lower value better.
Method NLLoradte
N4L)racte(40)
SAL
7.71 10.17
9..03
CAL
8.01 10.24
9.43=10.05
LeakGAN
7.04 10.37
7.19
LeakGAN SAL
6.69 10.29
6.91
Table 18: Performance comparison of different models in the COCO caption generation task. Metrics from top to bottom represent respectively the generation quality, the generation diversity, and the divergence between real data of generated sentences. For all BLEU metrics, higher value is better, for NLL and FD), lower is better.
Metrics
CAL
SAL
LeakGAN
LeakGAN + SAL
BIEU--((F)
0.767
+0.03
0.785
+0002
0.749 +0.03
0.798 +0003
BLEU-309
0.541 +0.03
0.581
+0.03
0.532 +0.04
0.592 +0.03
BLEU-4(F)
0.337 =0.03
0.362
+0.02
#0.03
=0002
BLEU--(()
0.211 +0.02
0.227
+0.02
0.233 +0.03
0.241 +0.02
Perplexity
276.7 12.55
231.3 10..8
291.4 #12.5
218.2 10.8
BLEU-2(B)
0.705 +0.03
0.724 +0002
0.733 +0.03
0.741 +0.02
BLEU-3(B)
0.479 +0.03
0.503 +0.03
0.512 +0.03
0.529 +0.03
BLEU-4(B)
0.296-10.03
0.31340.02
10.03
0.334-10.02
BLEU:5(()
0.191 0.03
0.198 +0.02
0.206 +0.03
0,211 +0.02
NLLgen
0.936 +0.03
0.873 +0.02
0.683 +.003
0.655 +0.02
FD
0.058 +0.016
0.051 10.014
0,048 10.016
0,044 10.014
17


Reviews:
[{"review_id": "SygZ4flAYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To alleviate the issues of reward sparsity and mode collapse in most text-generation GANs with a binary discriminator, this paper proposes a self-adversarial learning (SAL) framework with a novel comparative discriminator that takes pairs of text examples from real and generated examples and outputs better, worse, or indistinguishable. Inspired by self-play in reinforcement learning, SAL employs self-play to reward the generator to generate better samples than previous samples with self-improvement signals from the comparative discriminator. It is argued that, because the comparative discriminator always produces self-improvement signals during the training and the self-improvement signal will not be very strong when generated samples are already good enough, the issues of reward sparsity and mode collapses in conventional text GANs are reduced. Experimental results on synthetic data and benchmark datasets demonstrate that SAL outperforms SeqGAN, MaliGAN, and RankGAN both quantitatively and qualitatively.\n\nPros:\n\nThis paper is well-written. The studied problem is well-motivated and the method is clearly presented. The SAL framework with self-play and a comparative discriminator is novel and the results are convincing.\n\nCons:\n\n1) SAL has a new discriminator, which can be viewed as an architecture change. Although it has a new training strategy and is very different from recent text GANs such as LeakGAN and RelGAN, the latest results of these recent methods on the benchmark datasets should be included in Table 4 and 5 for reference.\n\n2) The comparative discriminator is novel and well-suited for comparing pairs of samples. However, the only informative signals come from pairs of data with one from a real sample and the other from a generated sample. During the self-play process, why the signals from the comparative discriminator comparing two generated samples are always trustworthy? Do the reward signals always help train a better generator? \n\n3) In section 4.3, since the self-play and the comparative discriminators are shown to be the most significant, it is better to clearly show the algorithms as in Algorithm 1 for training these two baselines either without self-play or without the new discriminator. Does replacing the binary classifier with WGAN classifier help here? All these details should be included in the appendix.\n\n4) Missing training details: It is unclear how the model architectures are chosen, and learning rate, optimizer, training epochs etc. are also missing. All these training details should be included in the appendix.\n\nIn summary, this paper proposes a novel discriminator with a new training strategy suited for the discriminator for text generation. Experimental results demonstrate the effectiveness of the proposed framework. I like the idea in the paper and am happy to vote for acceptance.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HklAibsRYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper describes a self-adversarial method to train a GAN for text generation that circumventes the problems of mode collapse and reward sparsity. They replace the traditional binary discriminator with a comparative discriminator, which provides the generator with more frequent rewards that are not always restricted to the limited number of real training examples. \n\nThey show gains on synthetic and real generation tasks. \n\nStrengths:\n\nThe paper is well written with adequate details about the training objectives and the learning algorithm. I appreciate the human analysis conducted by the authors. The SAL method performs better than many strong baselines consistently across 3 datasets and on various metrics.\n\nThe authors present strong ablations, that are very insightful, particularly, regarding the role played by classifying sentences as indistinguishable.\n\nWeaknesses and Questions: \n\nIn general, some more description about the motivation of each metric would be helpful, rather than just stating that its from previous work.  \n\nHow is BLEU evaluated for this text generation task? Is the entire test set treated as a single reference? Do you generate the same number of tokens as the reference and then compute n-gram overlap between the reference and the prediction? What happens to the brevity penalty of BLEU?\n\nIn Table 4, does BLEU-5(F) denote only 5-gram precision, or is it the geometric mean of 1-5 gram overlaps?\n\nHow does NLL_gen serve as a measure of diversity for the synthetic dataset?\n\nFor the human evaluation, does quality mean grammaticality? Can simple memorized sentences be scored higher?\n\nTypos in Section 4. The authors refer to tables 17 and 18. Please fix. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgVe1df9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a Self-Adversarial Learning (SAL) mechanism in GAN based text generation, aiming at tackling the problem of mode collapse and sparse rewards problem. Specifically, motivated by “self-play” mechanism in RL community, instead of using a binary classifier as discriminator in original GAN, SAL employs a comparative discriminator which is a pairwise classifier with three classes: “better”, “worse” and “indistinguishable”. The authors provide lots of experimental results and ablation study showing the efficiency of the proposed mechanism in comparison with previous GAN models.\n\nDecision: weak reject. \nThis paper is well motivated: clearly sparse rewards and mode collapse are two problems need to be solved in GAN based text generation, however, the following concerns prevent me from finding this paper acceptable in ICLR:\nThe “self-play” idea is widely used in RL. In RL, the comparison between policies can be determined directly by the game simulation results. In text generation, such comparison is more difficult to be judged. This paper appears to assume that the generated sentences are “worse” than the real samples, which is similar to the original definition of discriminator in GAN, and the generated sentences in earlier epochs are “worse” than that in later epochs, which needs further justified.\nAs a simple extension to GAN, I’m not convinced that the problem of mode collapse could be solved by proposed mechanism. If the generator falls into a local minimum, a collapsed mode, the proposed mechanism will never pull the generator out of that. Moreover, what is the theoretical foundation of the proposed evaluation metric on quality-diversity trade-off, NLL_{gen} + NLL_{oracle}?\nThe setting of the important set of hyper-parameters, reward weights, is unclear in the paper. The reward weights directly influence the reward in training, thus should play an important role of model performance. More discussion about this should be provided.\nMoreover, in the paper, only comparison between proposed mechanism with GAN based models are shown. Comparison with more recent models like RelGAN should be provided. And comparison with other state-of-the-art text generation model should be discussed.\n\n\n\nFeedbacks: \nReferences regarding experimental results in table are incorrect. For example, the results in synthetic data should be in “Table 2” and COCO image caption dataset should be in “Table 3”. \nSome imprecise parts, for example, in Equation (5) and (6), it should be G_\\theta(Y_{1:t-1} and G_\\theta(y_t | Y_{1:t-1} .\nI’m curious about what the performance would be like if the weakly supervision by regarding sentences generated in later training stage are “better” than the sentences generated in earlier training stage is removed in training comparative discriminator. This is different from the CAL model in the ablation study.\nWhat is the influence of different values of rewards weight? \nAlso about the rewards weight, in the description of Scheduled rewarding, the rewards weight is described to be linearly changed with training iteration, while in Appendix C.3, the rewards weight is described to be fixed. This is very confusing.\nA minor issue: for image captioning, a lot other metrics are widely used in measuring the model performance, e.g. CIDEr, SPICE and so on. Those metrics could be helpful for audience to understand how the model performs in comparison with other captioning models.\nAnother minor issue: for image captioning and WMT (conditional generation), the detailed model structures are not described in paper, which is not very friendly to audiences with relatively little knowledge in related areas.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for improving training of text generation with GANs by performing discrimination between different generated examples, instead of solely between real and generated examples.

R3 and R1 appreciated the general idea, and thought that while there are still concerns, overall the paper seems to be interesting enough to warrant publication at ICLR. R2 has a rating of "weak reject", but I tend to agree with the authors that comparison with other methods that use different model architectures is orthogonal to the contribution of this paper.

In sum, I think that this paper would likely make a good contribution to ICLR and recommend acceptance.
Decision: Accept (Poster)