Title: LEARNING TO ENCODE TEXT AS HUMAN-READABLE SUMMARIES USING GENERATIVE ADVERSARIAL NET-WORKS

Abstract: Auto-encoders compress input data into a latent-space representation and reconstruct the original data from the representation. This latent representation is not easily interpreted by humans. In this paper, we propose training an auto-encoder that encodes input text into human-readable sentences. The auto-encoder is composed of a generator and a reconstructor. The generator encodes the input text into a shorter word sequence, and the reconstructor recovers the generator input from the generator output. To make the generator output human-readable, a discriminator restricts the output of the generator to resemble human-written sentences. By taking the generator output as the summary of the input text, abstractive summarization is achieved without document-summary pairs as training data. Promising results are shown on both English and Chinese corpora.

1. INTRODUCTION:

When it comes to learning data representations, a popular approach involves the auto-encoder architecture, which compresses the data into a latent representation without supervision. In this paper we focus on learning text representations. Because text is a sequence of words, to encode a sequence, a sequence-to-sequence (seq2seq) auto-encoder (Li et al., 2015; Kiros et al., 2015) is usually used, in which a RNN is used to encode the input sequence into a fixed-length representation, after which another RNN is used to decode the original input sequence given this representation. Although the latent representation learned by the auto-encoder can be used in downstream applications, they are usually not human-readable. In this work, we use comprehensible natural language as a latent representation of the input source text in an auto-encoder model. This human-readable latent representation is shorter than the source text; in order to reconstruct the source text, it must reflect the core idea of the source text. Intuitively, the latent representation can be considered a summary of the text.

The idea that using human comprehensible representation as a latent representation has been explored on text summarization (Miao & Blunsom, 2016) , but only in a semi-supervised scenario. Previous work uses a prior distribution from a pre-trained language model to constrain the generated sequence to natural language. However, to teach the compressor network to generate text summaries, the model is trained using labeled data. In contrast, in this work we need no labeled data to learn the representations. The proposed model is inspired from cycle consistency (Zhu et al., 2017; He et al., 2016) . As shown in Fig. 1 , the proposed model is composed of three components: a generator, a discriminator, and a reconstructor. Together, the generator and reconstructor form a text auto-encoder. The generator acts as an encoder in generating the latent representation from the input text. Instead of using a vector as latent representation, however, the generator generates a word sequence much shorter than the input text. From the shorter text, the reconstructor reconstructs the original input of the generator. By minimizing the reconstruction errors, the generator learns to generate short text segments that contain the main information in the original input. We use the seq2seq model in modeling the generator and reconstructor because both have input and output sequences with different lengths. However, it is very possible that the generator's output word sequence can be processed by the reconstructor but is not readable by humans. Here, instead of regularizing the generator output with a pre-trained language model (Miao & Blunsom, 2016) , we borrow from adversarial autoencoders (Makhzani et al., 2015) and introduce a third component -the discriminator -to regularize the generator's output word sequence.

Figure 1 : Proposed model. Given long text, the generator produces a shorter text as a summary. The generator is learned by minimizing the reconstruction loss together with the reconstructor and making discriminator regard its output as human-written text.

The discriminator and the generator form a generative adversarial network (GAN) (Goodfellow et al., 2014) . GANs are generative models composed of a generator and a discriminator. The discriminator discriminates between the generator output and real data, and the generator produces output as similar as possible to real data to confuse the discriminator. Here, we only have to feed human-written sentences to the discriminator as real data. With the GAN framework, the discriminator teaches the generator how to create human-like summary sentences as a latent representation; however, this only guarantees that the generator produces grammatically correct sentences -not necessarily sentences that represent the input text. It is the reconstructor that teaches the generator how to produce a sentence that captures the core idea of the source text. However, generating discrete distributions with GAN is challenging, since it is difficult to evaluate the distance between the continuous distribution from the generator and the discrete distribution of the real sample. In addition, if we feed sampled words from the generator output distribution to the discriminator, the process of word selection is non-differentiable, which yields a discriminator gradient that precludes back-propagation to the generator. With GAN, there are two ways to generate language: (1) by training with a policy gradient, which regards words as actions, or (2) by directly feeding the generator's output layer to the discriminator, which yields a gradient suited to backpropagation to the generator. In this work, we propose new kind of method on training with policy gradient in which the discriminator evaluates the output of generator every time steps. On language generation with GAN, we conduct experiments using both (1) and ( 2) methods and evaluate their results. We evaluate the results on an abstractive text summarization task in which the machine generates a text summary in its own words. The model is learned from a set of unpaired documents and summaries 1 . We use the sentences in the summaries as real data for discriminator 2 . As the summaries can come from another set of documents not related to the training documents, training is unsupervised. We use the output word sequence of the generator as the summaries of the input text. The results show that the generator generates summaries with reasonable quality on both English and Chinese corpora.

2. RELATED WORK GAN FOR LANGUAGE GENERATION:

The major challenge in applying GAN to sentence generation is the discrete nature of natural language. To generate a word sequence, the generator usually has non-differential parts such as argmax 1 Here the titles of the documents are considered as the summaries. This is a typical setup in the study of summarization.

2 Instead of using general sentences as real data for discriminator, we here choose sentences from summaries because they have their own unique distribution. For example, the structure of a summary sentence is concise; they usually contain specific words such as country or person names, or key verbs from the source text. Because the following application is document summarization, using document summaries as real data is helpful. It is possible to use text from different styles as real data for the discriminator to produce output from the generator in different styles.

or other sample functions which cause the original GAN to fail. Therefore, new kinds of GANs have been proposed for sentence generation. SeqGAN (Yu et al., 2017) tackles the sequence generation problem with reinforcement learning. Here, we refer to this approach as adversarial REINFORCE, in which the generator is regarded as an agent, the generated sequence of words is viewed as a sequence of actions, and the current state is defined as the generated sequence to date and the prior input. However, the discriminator only measures the quality of whole sentences, and thus the rewards are extremely sparse and the rewards assigned to all actions in sequence are all the same. To tackle this problem, they propose MC search to evaluate approximate rewards at each time step, but this method suffers from high time complexity. Following this idea, (Li et al., 2017) proposes another approach to evaluate the expected reward at each time step. They break both the generated and real sequences into partial sequences, and the discriminator discriminates between the generated and real partial sequences. Inspired by this idea, we propose the self-critical adversarial REINFORCE algorithm as another way to evaluate the expected reward at each time step. In (Gulrajani et al., 2017) , instead of feeding a discrete word sequence, the authors directly feed the generator output layer to the discriminator. This method works because they use the earth mover's distance on GAN as proposed in (Arjovsky et al., 2017) , which is able to evaluate the distance between a discrete and a continuous distribution. In order to satisfy the requirement of the earth mover's distance, they use a gradient penalty trick to confine the complexity of discriminator function. Their method achieves an amazing result: it is the first work on GAN training that performs language generation without pre-training. In our work, we also conduct experiments on this method with discriminator settings almost the same as the original paper.

ABSTRACTIVE TEXT SUMMARIZATION:

Recent model architectures for abstractive text summarization basically use the sequence-tosequence (Sutskever et al., 2014) framework in combination with various novel mechanisms. One popular mechanism is attention (Bahdanau et al., 2015) , which has been shown helpful for summarization (Nallapati et al., 2016; Rush et al., 2015) . It is also possible to directly optimize evaluation metrics such as ROUGE (Lin, 2004) with reinforcement learning (Ranzato et al., 2016; Paulus et al., 2017; Bahdanau et al., 2016) . The hybrid pointer-generator network (See et al., 2017) selects words from the original text with a pointer (Vinyals et al., 2015) or from the whole vocabulary with a trained weight. In order to eliminate repetition, a coverage vector (Tu et al., 2016) can be used to keep track of attended words and coverage loss (See et al., 2017) can be used to encourage model focus on diverse words. While most papers focus on supervised learning with novel mechanisms, we explore unsupervised training models.

3. PROPOSED METHOD:

The overview of the proposed model is shown in Fig. 2 . The model is composed of three components: generator G, discriminator D, and reconstructor R. Both G and R are seq2seq hybrid pointer-generator networks (See et al., 2017) which can decide to copy words from encoder input text via pointing or generate from vocabulary.They both take a word sequence as input and output a sequence of word distributions. Discriminator D, on the other hand, takes a sequence as input and outputs a scalar. The model is learned from a set of documents x and human-written sentences y real . Although in real implementation, y real are the sentences in summaries, we note that the documents and summaries are unpaired. To train the model, a training document x = {x 1 , x 2 , ..., x t , ..., x T }, where x t represents a word, is fed to G, which outputs a sequence of word distributions G(x) = {y 1 , y 2 , ..., y n , ..., y N }, where y n is a distribution over all words in the lexicon. Then we sample a word y s n from each distribution y n , and a word sequence y s = {y s 1 , y s 2 , ..., y s N } is obtained according to G(x). We feed the sampled word sequence y s to reconstructor R, which outputs another sequence of word distributions x. The reconstructor R reconstructs the original text x from y s . That is, we seek an output of reconstructor x that is as close to the original text x as possible; hence the loss for training the reconstructor R, R loss , is defined as where the reconstruction loss l s (x, x) is the cross-entropy loss computed between the reconstructor output sequence x and the source text x, or the negative conditional log-likelihood of source text x given word sequence y s sampled from G(x). The reconstructor output sequence x is teacher-forced by source text x. The subscript s in l s (x, x) indicates that x is reconstructed from y s . K is the number of training examples (documents), and ( 1) is the summation of the cross-entropy loss over all the training documents x.

EQUATION

In the proposed model, the generator G and reconstructor R form an auto-encoder. However, the reconstructor R does not directly take the generator output distribution G(x) as input3 . Instead, the reconstructor takes a sampled discrete sequence y s as input. Due to the non-differentiable property of discrete sequences, we apply the REINFORCE algorithm, which is described in Section 4.

In addition to reconstruction, we need the discriminator D to discriminate between the real sequence y real and the generated sequence y s to regularize the generated sequence satisfying the summary distribution. D learns to give y real higher scores while giving y s lower scores. The loss for training the discriminator D is denoted as D loss ; this is further described in Section 5.

G learns to minimize the reconstruction error R loss , while maximizing the loss of the discriminator D by generating a summary sequence y s that cannot be differentiated by D from the real thing. The loss when training the generator G, G loss , is

G loss = αR loss -D loss (2)

where D loss is highly related to D loss -but not necessary the same -and α is a hyper-parameter.

After obtaining the optimal generator by minimizing (2), we use it to generate summaries.

Generator G and discriminator D together form a GAN. We use two different adversarial training methods to train D and G; as shown in Fig. 2 , these two methods have their own discriminators 1 and 2. Discriminator 1 takes the generator output layer G(x) as input, whereas discriminator 2 takes the sampled discrete word sequence y s as input. The two methods are described respectively in Sections 5.1 and 5.2.

4. MINIMIZING RECONSTRUCTION ERROR:

Because discrete sequences are non-differentiable, we use the REINFORCE algorithm. The generator is seen as an agent whose reward given the source text x is -l s (x, x). Maximizing the reward is equivalent to minimizing the reconstruction loss R loss in (1). However, the reconstruction loss varies widely from sample to sample, and thus the rewards to the generator are not stable either.

Hence we add a baseline to reduce their difference. We apply self-critical sequence training (Rennie et al., 2017) ; the modified reward r R (x, x) from reconstructor R with the baseline for the generator is

EQUATION

where -l a (x, x) -b is the baseline. l a (x, x) is also the same cross-entropy reconstruction loss as l s (x, x), except that x is obtained from y a instead of y s . y a is a word sequence {y a 1 , y a 2 , ..., y a n , ..., y a N }, where y a n is selected using the argmax function from the output distribution of generator y n . As in the early training stage, the sequence y s barely yields higher reward than sequence y a , to encourage exploration we introduce the second baseline score b, which gradually decreases to zero. Then, the generator is updated using the REINFORCE algorithm with reward r R (x, x) to minimize R loss .

5. GAN TRAINING:

With adversarial training, the generator learns to produce sentences as similar to the human-written sentences as possible. Here, we conduct experiments on two kinds of methods of language generation with GAN. In Section 5.1 we directly feed the generator output probability distributions to the discriminator and use a Wasserstein GAN (WGAN) with a gradient penalty. In Section 5.2, we explore adversarial REINFORCE, which feeds sampled discrete word sequences to the discriminator and evaluates the quality of the sequence from the discriminator for use as a reward signal to the generator.

5.1. DISCRIMINATOR 1: WASSERSTEIN GAN:

In the lower left of Fig. 2 , the discriminator model is shown as discriminator1. The discriminator loss D loss is

EQUATION

)

where K denotes the number of training examples in a batch, and k denotes the k-th example. The last term in (4) is the gradient penalty (Gulrajani et al., 2017) . We interpolate the generator output layer G(x) and the real sample y real , and apply the gradient penalty to the interpolated sequence y i . β determines the gradient penalty scale. In Equation (2), for WGAN, D loss is the score of the generated example:

D loss = 1 K K k=1 D(G(x (k) )).

5.2. SELF-CRITIC ADVERSARIAL REINFORCE:

In this section, we describe in detail the proposed adversarial REINFORCE method. The core idea is we use the LSTM discriminator to evaluate the current quality of the generated sequence {y s 1 , y s 2 , ..., y s i } at each time step i. Hence, the generator knows that compared to the last time step, as the generated sentence either improves or worsens, it can easily find the problematic generation step in a long sequence, and thus fix the problem easily.

5.2.1. DISCRIMINATOR 2:

As shown in Fig. 2 , the discriminator2 is a one-way LSTM network which takes a discrete word sequence as input. At time step i, given input word y s i it predicts the current score s i based on the sequence {y 1 , y 2 , ..., y i }. The score is viewed as the quality of the current sequence.

In order to compute the discriminator loss D loss , we sum the scores {s 1 , s 2 , ..., s N } of the whole sequence y s to yield

D(y s ) = 1 N N n=1 s n . D loss = 1 K K k=1 D(y s(k) ) - 1 K K k=1 D(y real(k) ),

where K and N denote the number of training examples and the generated sequence length respectively. With the loss mentioned above, the discriminator attempts to quickly determine whether the current sequence is real or fake. The earlier the timestep discriminator determines whether the current sequence is real or fake, the lower its loss. An example is shown in Fig. 3 .

Figure 3 : When the second arrested appears, the discriminator determines that this example came from the generator. Hence, after this time-step, it outputs low scores.

5.2.2. SELF-CRITICAL GENERATOR:

Since we feed a discrete sequence y s to the discriminator, the gradient from the discriminator cannot directly back-propagate to the generator. Here, we use the policy gradient method. At timestep i, we use the i -1 timestep score s i-1 from the discriminator as its self-critical baseline. The reward r D i evaluates whether the quality of sequence in timestep i is better or worse than that in timestep i -1. The generator reward r D i from D is

r D i = s i if i = 1 s i -s i-1

otherwise.

However, some sentences may be judged as bad sentences at the previous timestep, but at later timesteps judged as good sentences, and vice versa. Hence we use the discounted expected reward d with discount factor γ to calculate the discounted reward d i at time step i as

d i = N j=i γ j-i r D j .

The adversarial REINFORCE score related to discriminator D loss in ( 2) is

D loss = E y s i ∼p G (y s i |y s 1 ,...,y s i-1 ,x) [d i ].

We use the likelihood ratio trick to approximate the gradient.

6. IMPLEMENTATION:

Network Architecture. The model architecture of generator and reconstructor is almost same except the length of input and output sequence. We adapt model architecture for our generator and reconstructor from See et al. (2017) who used hybrid-pointer network with coverage vector for text summarization. The hybrid-pointer networks of generator and reconstructor are all composed of two one-layer LSTMs as its encoder and decoder, respectively, with a hidden layer size of 600. Since we use two kinds of methods on adversarial training, there are two discriminators with different model architecture. In the Section 5.1, the discriminator is composed of four residual blocks with 512 hidden dimensions. While in Section 5.2, we use only one hidden-layer one-way LSTM with a hidden size of 512 as our discriminator.

Details of Training.:

We set the weight α in (2) controlling R loss to 10 if not specified. We find that the if the value of α is too large, generator will start to generate output unlike human-written sentences. On the other hand, if the value of α is too small, the sentences generated by generator will sometimes become unrelated to input text of generator. For all the experiments, the baseline b in (3) gradually decreases from 0.25 to zero within 10000 updates on generator.

In Section 5.1, we set the weight β of the gradient penalty to 10, and used RMSPropOptimizer with a learning rate of 0.00001 and 0.001 on the generator and discriminator, respectively. In Section 5.2, we clip the value of the weights of discriminator to ±0.15, and used RMSPropOptimizer with a learning rate of 0.0001 and 0.001 on the generator and discriminator, respectively. It's also feasible to apply gradient penalty trick in this method to satisfy requirement of Wasserstein distance. However, in this method, the performance of gradient penalty trick and weights clipping trick is close.

7. EXPERIMENT:

We evaluate our model on the Chinese Gigaword and English Gigaword datasets. Before jointly training the whole model, we pre-trained the three major components -generator, discriminator, and reconstructor -separately. First, we pre-trained the generator in an unsupervised manner so that the generator would be able to somewhat grasp the semantic meaning of the source text. The details of the pre-training are in Appendix A. We pre-trained the discriminator and reconstructor respectively with the pre-trained generator's output to ensure that these two critic networks provide good feedback to the generator. During testing, when using the generator to generate summaries, we simply selected the words in a greedy fashion without beam-search, and we eliminated repetition. The Chinese Gigaword corpus is composed of 2.2M paired data of headlines and news. We preprocessed the raw data as following. First, we selected the 4000 most frequent Chinese characters as our vocabulary. We filtered out headline-news pairs with excessively long or short news segments, or that contained too many out-of-vocabulary Chinese characters, yielding 1.1M headline-news pairs from which we randomly selected 5K headline-news pairs as our testing set, 5K headline-news pairs as our validation set, and the remaining pairs as our training set. During training and testing, the generator took only the first 80 Chinese characters of the source text as input.

7.1. CHINESE GIGAWORD:



Methods:

The results are shown in Table 1 . Row (A) lists the results using 1.1 million document-summary pairs to directly train the generator without the reconstructor and discriminator: this is the upper bound of the proposed approach. In row (B), we simply took the first fifteen words in a document as its summary. The number of words was chosen to optimize the evaluation metrics. Part (C) are the results obtained in the unsupervised scenario without paired data. We show the results of the pre-trained generator in row (C-1); rows (C-2) and (C-3) are the results for the two GAN training methods respectively. We find that despite the performance gap between the unsupervised and supervised methods (rows (C-2), (C-3) v.s. (A)), the proposed method yielded much better performance than the trivial baselines (rows (C-2), (C-3) v.s. (B)).

7.2. ENGLISH GIGAWORD:

On English Gigaword, we set our vocabulary size to 15k, and used the dataset preprocessed by (Rush et al., 2015) for training and testing. We used 3.8M unpaired training data for our training, and we used the whole 200k filtered data in validation set for testing. The results on English Gigaword are shown in Table 2 . In row (B-1), we simply took the first eight words in a document as its summary. Row (B-2) is another trivial baseline. With unpaired documents and summaries, we matched documents to the most relevant summaries with unsupervised method. Table 2 : Results on English Gigaword: In row (B-1), we select the article's first eight words as its summary. In row (B-2), we match the documents to their most relevant summaries with unsupervised method. Part (C) are the results obtained without paired data. In part (D), we pre-trained the generator on CNN/Diary. In part (E), we not only pre-trained on CNN/Diary but also used the summaries from CNN/Diary as real data for the discriminator.

frequency) vectors. The summary whose vector maximized cosine similarity of a document vector was retrieved as summary of the document. With paired data from unsupervised matching, given documents as generator input, the generator was trained to predict retrieved summaries.

The results for the pre-trained generator is shown in row (C-1). Compared with the trivial baselines (part (B)), the proposed approach (rows (C-2) and (C-3)) showed good improvement in terms of ROUGE-1. As shown in Fig. 4 , the unsupervised method selects key words in the source text and generates the text summary. However, as shown in Table 2 , although both unsupervised methods yield ROUGE-1 scores close to that of supervised training, they achieve lower scores on ROUGE-2, especially when training GAN with reinforcement learning. This is because they are extracting the key words from the source text, despite sometimes failing to arrange these words in the correct order. As shown in part (C-3) of Fig. 5 , the words in the sentence generated in an unsupervised manner are not arranged correctly: the Italian prime minister, Berlusconi, should not be visiting himself.

7.3. TRANSFER LEARNING:

In this subsection, we study transfer learning. We used the CNN/Daily Mail dataset (Hermann et al., 2015; Nallapati et al., 2016) preprocessed by the script provided by (See et al., 2017) as our source domain S, and English Gigaword as our target domain T . The data distributions among the two datasets are quite different. In English Gigaword, the articles consist of 32 words on average and the summaries consist of one sentence with 8 words on average, whereas the CNN/Daily Mail dataset is composed of 790-word articles and multi-sentence summaries. We took only the first 30 to 45 words in the original source domain articles as our new source articles S a . There are 50K source articles in S a . The 50K source summaries of the source articles are S t . In contrast to English Gigaword, the summaries in CNN/Daily Mail dataset are composed of more than one sentence. We split the summary of each article into several sentences, and obtained 240K sentences S r in this way.

Transfer learning was applied in two directions. In the first direction, we pre-trained our generator on the data from the source dataset: we pre-trained the generator with S a as input, and the generator predicted S t . We pre-trained the generator in this manner in all of the transfer learning experiments. Then, after pre-training, the generator was further learned jointly with the reconstructor and discriminator on the data from the target domain. The results are shown in part (D) of Table 2 . We found that pre-training on the source data set did not degrade performance, and even improved performance in some cases (parts (D) v.s. (C)).

In the second direction, we used the summary from the source domain as our real data. The experiments conducted up to this point required unpaired summary and text data in the target domain.

In this experiment, the task was more challenging in that we used summaries S r from the source domain S as the real data for the discriminator; the generator took the target domain text as input. However, the summaries in each summarization task dataset had a different distribution in terms of the writing style, or in terms of the preferred summary words. To prevent overfitting to S r , we set the weight α to 50 which was larger than other experiments. With small weight of α, as training progressed, the generator summary diverged more and more from the article, and the ROUGE scores became lower and lower. The results without using summaries in the target domain are shown in part (E). We find that using sentences S r from another dataset yields lower ROUGE scores on the target testing set (parts (E) v.s. The two GAN training methods are not comparable as their settings are quite different, but we can still discuss the advantages and disadvantages of these two methods. When training with feeding output layer to discriminator, convergence is faster. This method sharpens the distribution at an early stage in training because it directly evaluates the distance between the generator's continuous distribution and the real data's discrete distribution data. However, this cause generator to converge to a not very good place.

In fact, adversarial REINFORCE is sensitive to initialization parameters. Adversarial REINFORCE requires better initialization for exploration; otherwise, with so many actions whose number is equal to vocabulary size to choose, it is extremely difficult to train generator from scratch. In semisupervised training, since we pre-trained generator with labeled data, the generator was better initialized, therefore adversarial REINFORCE performed better. To support this idea, in Fig. 6 , we compare the performance of two methods regarding labeled data. The result implies that with more labeled data, our proposed adversarial REINFORCE method performs better. In order to evaluate the performance of proposed self-critic baseline trick mentioned in Section 5.2.2, we compared the performance of our model with and without this baseline trick in Table 3 . For the experiments without the self-critic baseline trick, we replaced s i -s i-1 in Section 5.2.2 with s i . We found that the performance degraded without self-critic.

8. CONCLUSION AND FUTURE WORK:

Using GAN, we propose a model that encodes text as a human-readable summary, learned without document-summary pairs. Promising results are obtained on both Chinese and English corpora.

In future work, we hope to explore more techniques for natural language generation using GAN. Moreover, we hope to use extra discriminators to control the style and sentiment of the generated summaries.

A MODEL PRE-TRAINING:

As we found that the different pre-training methods for the generator influenced final performance dramatically in all of the experiments, we felt it was important to find a proper unsupervised pretraining method to help the machine grasp semantic meaning. We used the different pre-training strategies described below.

• Chinese Gigaword: Given the previous i -1 sentences sent 0 , sent 1 , ..., sent i-1 from the source text, the generator predicted the next sentence sent i in the source text as its pretraining target. If more than 50% of the words in sentence sent i did not appear in the given text, we filtered out this pre-training sample pair. This pre-training method allowed the generator to capture the important semantic meanings of the source text. • English Gigaword: As the length of the source texts in English Gigaword dataset is comparatively short, it is difficult to split the last sentence from the source text; hence the previous pre-training method on Chinese Gigaword is not appropriate for this dataset. To properly initialize the set, we randomly selected 6 to 11 consecutive words in the source text, after which we randomly swapped 70% of the words in the source text. Given text with incorrect word arrangements, the generator predicted the selected words in the correct arrangement. We pre-trained in this way because we expect the generator to initialize with a rough language model. In Chinese Gigaword we also conducted experiments on pre-training in this manner, but the results were not as good as those shown in the part (C) of Table 1 . We also used the retrieved paired data in row (B-1) in Table 2 to pre-train generator. However, pre-training generator with this method doesn't yield results better than those in Table 2 .

B SEMI-SUPERVISED LEARNING:

Semi-supervised learning experiments were conducted with 10K, 50K, 100K labeled data in both datasets. We conducted teacher-forcing on generator every 30, 12, 7 unsupervised updates with 10K, 50K, 100K labeled data respectively. The complete results for semi-supervised learning are shown in Tables 4 and 5 .

ROUGE 

:

We found that if the reconstructor R directly takes G(x) as input, the generator G learns to put the information about the input text in the distribution of G(x), making it difficult to sample meaningful sentences from G(x).

Figures and Tables:

Figure 2: Architecture of proposed model. The generator network and reconstructor network are a seq2seq hybrid pointer-generator network, but for simplicity, we omit the pointer and the attention parts.

(D)) due to the mismatch between the summaries of the source and target domains. However, the discriminator still roughly regularizes the language model of generated word sequence. After training, the model still greatly enhanced the ROUGE score of the pre-trained model (rows (E-1), (E-2) v.s. (D-1)).Although the results in part (E) are comparable with the trivial baselines in part (B), in inspecting the real examples, we found that the results in part (E) were in fact better; this is not reflected in the ROUGE scores. In Fig.4, the results in part (E) are better than the leading 8 words and the pre-trained generator results. To support this idea, we provide more examples in the Appendix C.

Figure 4: Real example from our model in English Gigaword. The proposed method generates summaries that capture the core idea of the article.

Figure 5: In part (C-3), some words in the summary sentences are arranged in incorrect order.

Figure 9: Real example from our model in English Gigaword.

Figure 10: Real example from our model in English Gigaword.

Figure 11: Real example from our model in English Gigaword.

Figure 12: Real example from our model in English Gigaword.



Table content: <table><tr><td>ROUGE-1 ROUGE-2 ROUGE-L</td></tr></table>

Adversarial REINFORCE with/without self-critic with unsupervised training.

Table content: <table><tr><td>Corpus</td><td>Method</td><td colspan="3">ROUGE-1 ROUGE-2 ROUGE-L</td></tr><tr><td>Chinese</td><td>with self-critic without self-critic</td><td>40.053 37.549</td><td>26.126 24.181</td><td>37.118 35.160</td></tr><tr><td>English</td><td>with self-critic without self-critic</td><td>32.826 31.104</td><td>9.332 9.249</td><td>28.727 28.592</td></tr></table>

Semi-supervised learning in Chinese Gigaword with different amounts of labeled data (10K, 50K, 100K).

Table content: <table><tr><td>-1 ROUGE-2 ROUGE-L</td></tr></table>

Semi-supervised learning in English Gigaword with different amounts of labeled data (10K, 50K, 100K).

Table content: <table/>

Peer review (iclr2018):
 TL;DR of paper: Generating summaries by using summaries as an intermediate representation for autoencoding the document. An encoder reads in the document to condition the generator which outputs a summary. The summary is then used to condition the decoder which is trained to output the original document. An additional GAN loss is used on the generator output to encourage the output to look like summaries -- this procedure only requires unpaired summaries. The results are that this procedure improves upon the trivial baseline but still significantly underperforms supervised training. This paper builds upon two recent trends: a) cycle consistency, where f(g(x)) = x, which only requires unpaired data (i.e., CycleGAN), and (b) encoder-decoder models with a sequential latent representation (i.e., "Language as a latent variable" by Miao and Blunsom). A similar idea has also been explored by He et al. 2016 in "Dual Learning for Machine Translation". Both CycleGAN and He et al. 2016 are not cited. The key difference between this paper and He et al. 2016 is the use of GANs so only unpaired summaries are needed. The idea is a simple but useful extension of these previous works. The problem set-up of unpaired summarization is not particularly compelling, since summaries are typically found paired with their original documents. It would be more interesting to see how well it can be used for other textual domains such as translation, where a lot of unpaired data exists (some other submissions to ICLR tackle this problem). Unsurprisingly, the proposed method requires a lot of twiddling to make it work since GANs, REINFORCE, and pretraining are necessary. A key baseline that is missing is pretraining the generator as a language model over summaries. The pretraining baseline in the paper is over predicting the next sentence / reordering, but this is an unfair comparison since the next sentence baseline never sees summaries over the course of training. Without this baseline, it is hard to tell whether GAN training is even useful. Another experiment missing is seeing whether joint supervised-GAN-reconstruction training can outperform purely supervised training. What is the performance of the joint training as the size of the supervised dataset is varied? This paper has numerous grammatical and spelling errors throughout the paper (worse, the same errors are copy-pasted everywhere). Please spend more time editing the paper.