for all the models described above, we report the results given each decoding
method previously described (Section 7.3): Beam Search, DAS<sub>Local</sub>, DAS<sub>Global</sub>, and
Coop-MCTS.

To the best of our knowledge, GANs and Cooperative decoding have never been
directly compared before this work. A fortiori, this is the first time that a GAN
model is tested with a Cooperative decoding method at inference. We investigate
possible distillation effects in Section 7.5.

7.4.3 Metrics

To compare the different models, we report two type of metrics: *n-gram based* and *discriminator*.

**N-gram based** We report the standard BLEU (Papineni et al. 2002) and ROUGE
(C.-Y. Lin 2004). Both measure an overlap of n-grams between the reference and
the evaluated text. They differ in that BLEU is precision oriented while ROUGE
is rather recall oriented.

**Discriminators** Both BLEU and ROUGE suffer from the aforementioned limitations. We therefore propose to consider discriminators for model evaluation. Intuitively, they measure how model outputs are similar to what a human would have written. We consider two different discriminators:

* **Base** is a discriminator trained on the MLE baseline outputs generated via beam search. It allows to measure the corresponding improvement from the MLE baseline. Note that it corresponds to the initial discriminator in all the GANs experiments, and the discriminator used in the cooperative search for the MLE baseline.

* **Base+** Since the *Base* discriminator plays a role in all our experiments (except MLE+Beam Search), it is possible that a model that makes use of this *Base* obtains better *Base* results, despite bringing new biases and *de*-generation behaviors. For this reason, we also report *Base+*, a discriminator fine-tuned on all the different model outputs together. *Base+* is never used by any model at training or inference time. It is thus more robust toward an undesirable adversarial generation mode, while still being comparable for the different experiments. We argue that a higher *Base+* score indicates a real improvement beyond potential bias.
<table><caption>Table 7.1: Results of our experiments on QG (left) and Summarization (right). For each generator, we report the results with the four different decoders. The reported metrics correspond to BLEU4 (B4), ROUGE-1 (R1), ROUGE-L (RL) and the discriminators Base and Base+ as described in Section 7.4.3. For Base and Base+ the scores correspond to the probability of being human, so higher is better for all the metrics. For Self-GAN<sub>MCTS</sub>, we experimented with 5 different seeds and the standard deviation is always inferior to 0.1 for BLEU4 and ROUGE, and inferior to 0.5% for Base and Base+.</caption><thead><tr><th rowspan="2">Generator<br>Decoder</th><th colspan="5">Question Generation</th><th colspan="5">Summarization</th></tr><tr><th>B4</th><th>R1</th><th>RL</th><th>Base</th><th>Base+</th><th>B4</th><th>R1</th><th>RL</th><th>Base</th><th>Base+</th></tr></thead><tbody><tr><td><strong>MLE</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>BeamSearch</td><td>19,7</td><td>45,2</td><td>41,1</td><td>15%</td><td>15%</td><td>15,9</td><td>42,3</td><td>40,4</td><td>9%</td><td>8%</td></tr><tr><td>DAS<sub>local</sub></td><td>19,9</td><td>45,2</td><td>41,1</td><td>28%</td><td>19%</td><td>16,6</td><td>43,8</td><td>40,9</td><td>17%</td><td>11%</td></tr><tr><td>DAS<sub>global</sub></td><td>20,0</td><td>45,2</td><td>41,2</td><td>20%</td><td>17%</td><td>16,2</td><td>44,1</td><td>41,9</td><td>12%</td><td>9%</td></tr><tr><td>Coop-MCTS</td><td>19,8</td><td>45,3</td><td>41,5</td><td>33%</td><td>21%</td><td>16,3</td><td>42,5</td><td>40,6</td><td>20%</td><td>12%</td></tr><tr><td><strong>ColdGAN</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>BeamSearch</td><td>19,9</td><td>45,2</td><td>41,4</td><td>26%</td><td>17.9%</td><td>16,3</td><td>42,8</td><td>40,7</td><td>15%</td><td>10%</td></tr><tr><td>DAS<sub>local</sub></td><td>19,8</td><td>45,3</td><td>41,1</td><td>31%</td><td>20%</td><td>15,9</td><td>42,5</td><td>42,0</td><td>19%</td><td>11%</td></tr><tr><td>DAS<sub>global</sub></td><td>20,2</td><td>45,6</td><td>41,5</td><td>26%</td><td>18%</td><td>16,6</td><td>44,6</td><td>41,2</td><td>16%</td><td>10%</td></tr><tr><td>Coop-MCTS</td><td>19,9</td><td>45,4</td><td>41,2</td><td>39%</td><td>22%</td><td>15,9</td><td>44,2</td><td>41,2</td><td>23%</td><td>12%</td></tr><tr><td><strong>SelfGAN<sub>DAS<sub>loc</sub></sub></strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>BeamSearch</td><td>20,2</td><td>45,4</td><td>41,6</td><td>27%</td><td>21%</td><td>16,9</td><td>44,2</td><td>42,5</td><td>16%</td><td>11%</td></tr><tr><td>DAS<sub>local</sub></td><td>20,5</td><td>45,5</td><td>41,7</td><td>30%</td><td>23%</td><td>16,9</td><td>44,4</td><td>41,9</td><td>18%</td><td>13%</td></tr><tr><td>DAS<sub>global</sub></td><td>20,1</td><td>45,4</td><td>41,7</td><td>33%</td><td>20%</td><td>16,6</td><td>44,0</td><td>42,3</td><td>19%</td><td>11%</td></tr><tr><td>Coop-MCTS</td><td>20,4</td><td>45,5</td><td>41,8</td><td>39%</td><td>23%</td><td>16,4</td><td>43,8</td><td>42,8</td><td>23%</td><td>13%</td></tr><tr><td><strong>SelfGAN<sub>DAS<sub>glob</sub></sub></strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>BeamSearch</td><td>20,4</td><td>45,5</td><td>41,7</td><td>24%</td><td>19%</td><td>16,9</td><td>43,0</td><td>41,5</td><td>14%</td><td>11%</td></tr><tr><td>DAS<sub>local</sub></td><td>19,9</td><td>45,4</td><td>41,3</td><td>32%</td><td>22%</td><td>15,9</td><td>42,7</td><td>40,6</td><td>18%</td><td>12%</td></tr><tr><td>DAS<sub>global</sub></td><td>20,7</td><td>45,6</td><td>41,9</td><td>29%</td><td>20%</td><td>17,0</td><td>43,7</td><td>42,6</td><td>17%</td><td>11%</td></tr><tr><td>Coop-MCTS</td><td>20,0</td><td>45,3</td><td>41,4</td><td>40%</td><td>24%</td><td>16,1</td><td>43,4</td><td>42,3</td><td>23%</td><td>13%</td></tr><tr><td><strong>SelfGAN<sub>Coop-MCTS</sub></strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>BeamSearch</td><td>20,5</td><td>46,6</td><td>42,6</td><td>34%</td><td>21%</td><td>17,0</td><td>42,8</td><td>41,5</td><td>20%</td><td>13%</td></tr><tr><td>DAS<sub>local</sub></td><td>20,6</td><td>46,7</td><td>41,7</td><td>42%</td><td>24%</td><td>16,6</td><td>43,7</td><td>42,8</td><td>25%</td><td>13%</td></tr><tr><td>DAS<sub>global</sub></td><td>20,5</td><td>46,6</td><td>41,7</td><td>39%</td><td>21%</td><td>16,5</td><td>42,8</td><td>40,9</td><td>23%</td><td>12%</td></tr><tr><td>Coop-MCTS</td><td>21,1</td><td>48,9</td><td>44,7</td><td>40%</td><td>26%</td><td>17,5</td><td>43,5</td><td>42,3</td><td>23%</td><td>15%</td></tr></tbody></table>
# 7.5 Results and discussion

## 7.5.1 Conditional Text Generation

In Table 7.1, we report the results for all the previously trained generators, with the different decoding algorithms presented in Section 7.3. By 'model', in the following, we refer to the couple composed by a trained generator and a decoding algorithm.

We report BLEU<sub>4</sub>, ROUGE-<sub>1</sub>, ROUGE-L along with scores for the discriminators *Base* and *Base+*, computed as the percentage of outputs considered as human by a given discriminator model. *Base* was only trained on *MLE+Beam Search* outputs. As expected, by further training on the outputs generated by all the different models, *Base+* has a higher accuracy, which consistently results in lower scores compared to *Base*.

We start by focusing on the MLE results to compare the different decoding mechanisms. We observe that all the cooperative searches outperform Beam Search. Regarding *Base* and *Base+* metrics, $DAS_{Local}$ compares favorably to $DAS_{Global}$. We hypothesize that invoking the discriminator to rank at each step can have more impact than using it only once on fully decoded sequences. Finally, our proposed *Coop-MCTS* obtains the best results by a large margin.

Regarding the different GANs, we first compare them given the default decoding mechanism, i.e. Beam Search. The three versions of *SelfGAN* compare favorably to MLE and ColdGAN on both n-gram based metrics and discriminators metrics. Among *SelfGANs*, *SelfGAN<sub>Coop-MCTS</sub>* obtains the best results: given a Beam Search decoding, it obtains the best BLEU, ROUGE-<sub>1</sub> and ROUGE-L on the two tasks (respectively 17.2; 44.3; 40.6 on QG and 12.3; 38.6; 36.7 on Summarization). The performance in term of *Base* and *Base+* for *SelfGAN<sub>Coop-MCTS</sub>* is even more important in comparison to the other models (34.1%; 21.9% on QG and 20.2%; 12.7% on Summarization).

Both GAN at training time and Cooperative decoding at inference time pursue the same objective: to obtain better outputs that look like human texts. Would a generator trained via GAN, coupled with a Cooperative Decoding mechanism for inference result into a cumulative improvement from the two methods? First, on both ColdGAN and three *SelfGANs*, we can observe that adding a Cooperative Decoding method allows to gain significant improvement on *Base* and *Base+*. In particular, it is interesting to note that for *SelfGAN* an additional pattern seems to emerge: using the same cooperative decoding algorithm both during training and inference seems to provide additional gains. The best performance is achieved with the generator *SelfGAN<sub>Coop-MCTS</sub>* paired with the decoding *Coop-MCTS*. Com-
<table><caption>Table 7.2: Results on Unconditional Text Generation for samples realized at three different temperatures, in terms of BLEU Vs Self-BLEU (higher better; lower better).</caption><thead><tr><th>Model</th><th>T=0.5</th><th>T=1</th><th>T=2</th></tr></thead><tbody><tr><td>MLE+Sample</td><td>0.42;0.29</td><td>0.31;0.11</td><td>0.18;0.07</td></tr><tr><td>ColdGAN+Sample</td><td>0.47;0.21</td><td>0.33;0.08</td><td>0.22;0.06</td></tr><tr><td>MLE+Coop-MCTS</td><td>0.45;0.22</td><td>0.34;0.10</td><td>0.21;0.06</td></tr><tr><td>SelfGANCoop-MCTS+Coop-MCTS</td><td>0.48;0.20</td><td>0.37;0.09</td><td>0.24;0.05</td></tr></tbody></table>

pared to MLE via Beam Search, it obtains a final improvement superior to 1 point in terms of ROUGE and BLEU. The relative improvement for Base+ is significant: from 15.2% to 26.2% on QG and from 8.6% to 15.3% on Summarization. This corresponds to almost twice more outputs that sound human according to the Discriminator metric.

### 7.5.2 Unconditional Text Generation

We follow the ColdGAN setup: we compared our proposed approaches on the EMNLP2017 News dataset. The evaluation takes into account both the quality and the diversity. Consistently with previous works (e.g. ColdGAN, ScratchGAN, LeakGAN), we use the following metrics: i) BLEU-5 for measuring the quality (higher better); ii) Self-BLEU-5 for measuring the diversity (lower better).

To obtain a finer comparison between models, Caccia et al. (2020) proposed to draw the curve of BLEU vs self-BLEU, by sampling with various temperatures at inference.

We denote as the standard method to generate text in this setup, as used in all the previous works we are comparing to. It is a simple left to right decoding where, at each step, a token is sampled among the Softmax probabilities scaled by the temperature.

In our Coop-MCTS, the probability of a token is given by its visit counts during the simulations. In Conditional generation, we select at each step the token with the maximum number of counts. In Unconditional generation, we sample from the tokens counts distribution.

Overall, the results are consistent with the experiments on Conditional Generation: the MLE generator decoded with our proposed MCTS (3rd row) obtains:

1. significantly better slightly lower results than ColdGAN decoded with Sample(2nd row)

2. results than when the same MLE decoded with Sample(1st row);
<figure><img src="image_31.png" /><figcaption>Figure 7.1: Average difference for Summarization between human references and model outputs for the Length (Left), the Novelty (Middle), and the 3-grams repetitions (Right) during training. The closer to 0 the less differences w.r.t. gold-references.</figcaption></figure>

3. SelfGAN decoded with MCTS (4th row) obtains the best results.

### 7.5.3 Discussion

**Human-like features during training** In NLG, various rules are often integrated into the Beam Search to improve the quality of the outputs, for instance a length penalty (See et al. 2017) or an interdiction for 3-grams repetitions (Paulus et al. 2017; Dong et al. 2019). Such a need to hard code these rules indicates a discrepancy between the human output characteristics and what the model has learned. In particular, in Chapter 5, we reported the difference between DAS and the human reference for: i) **Length**: the average number of tokens per output; ii) **Novelty**: percentage of tokens in the output that were not present in the source text; iii) **N-gram repetition**: percentage of N-grams that occur more than once in the output.

To measure how *SelfGAN* learns these features by itself, we report in Figure 7.1 the evolution of these statistics during training: we observe that *SelfGAN* constantly reach statistics more similar to human references than ColdGAN.

**Human Evaluation** We conduct a human evaluation to measure the models performances beyond automatic metrics. We limit the evaluation to three genera-
<table><caption>Table 7.3: Human Evaluation on Summarization. Two tailed t-test results are reported for each model compared to MLE+BeamSearch (*: p < .01, **: p < .001).</caption><thead><tr><th>Generator</th><th>Decoder</th><th>Consistency</th><th>Coherence</th><th>Fluency</th><th>Relevance</th></tr></thead><tbody><tr><td>MLE</td><td>BeamSearch</td><td>3.9</td><td>3.1</td><td>4.1</td><td>3.2</td></tr><tr><td>MLE</td><td>Coop-MCTS</td><td>3.4**</td><td>3.5**</td><td>3.8</td><td>3.6**</td></tr><tr><td>ColdGAN</td><td>BeamSearch</td><td>3.8</td><td>3.3</td><td>4.2</td><td>3.5</td></tr><tr><td>ColdGAN</td><td>Coop-MCTS</td><td>3.4**</td><td>3.6**</td><td>4.0</td><td>3.7**</td></tr><tr><td>SelfGAN<sub>Coop-MCTS</sub></td><td>BeamSearch</td><td><b>4.0</b></td><td>3.5**</td><td><b>4.3*</b></td><td>3.9**</td></tr><tr><td>SelfGAN<sub>Coop-MCTS</sub></td><td>Coop-MCTS</td><td>3.9</td><td><b>3.9**</b></td><td>4.0</td><td><b>4.2**</b></td></tr></tbody></table>

tors (MLE, ColdGAN, and *SelfGAN*<sub>Coop-MCTS</sub>) and two decoding methods (Beam Search and *Coop-MCTS*), for a total of 6 different models. Three professional English speakers rated 300 sampled summaries and followed the same protocol from Fabbri et al. (2020). Four dimensions are evaluated on a Likert scale from 1 to 5 (the higher the better):

1. **Consistency:** the proportion of facts in the summary correct w.r.t. the source text;

2. **Coherence:** how well-structured and well-organized is the summary;

3. **Fluency:** how fluent the summary is to read;

4. **Relevance:** the ratio between important and excess information in the summary.

From Table 7.3 we observe significantly better results for *SelfGAN*<sub>Coop-MCTS</sub> w.r.t. both MLE and ColdGAN. While *Coop-MCTS* decoding appears overall beneficial in terms of Coherence and Relevance, but scores lower on Consistency and Fluency, its combination with *SelfGAN*<sub>Coop-MCTS</sub> allows to obtain significant improvements on the former two dimensions while still maintaining comparable scores on the latter.

**Analysis** To further understand the benefits of selfGAN, we propose to analyze the evolution of the generator and discriminator networks through the learning process. In figure 7.2 (left), we first plot the average magnitude (L2 norm) of the discriminator gradients w.r.t. its parameters. We observe that *ColdGAN* induces important instabilities for its discriminator over time, with a highly fluctuating gradient magnitude. Conversely, thanks to its cooperative decoding process, *SelfGAN* produces sequences that form a more compact set for discriminator training, a variance of gradient magnitude twice lower than *ColdGAN*, for a comparable magnitude in average. This discriminator stability is a first explanation for the improvements of the proposed approach.

In a second plot, given on the right of Figure 7.2, we report the collinearity of generator gradients for the generated samples from the model with those for
<figure><img src="image_32.png" /><figcaption>Figure 7.2: Left: Moving Average of the magnitude of the *discriminators* gradients during training. Right: collinearity of the *generators* gradients between the sampled texts and their corresponding human reference for *Self-GAN*<sub>Coop-MCTS</sub>, *ColdGAN* and *SelfGAN*<sub>BeamSearch</sub>. Both on Summarization.</figcaption></figure>

the corresponding human references. Higher values indicate sampling strategies that induce a useful gradient flow for the generator. For ablation purposes, we first report values for a *"SelfGAN"* approach, where we used a standard Beam Search to generate the training examples: note that it has no discriminator, hence it is not a GAN anymore. We can observe its divergence, as opposed to *SelfGAN*<sub>Coop-MCTS</sub>, which emphasizes the importance of the cooperative decoding for producing the example used to train the model. For *SelfGAN*<sub>Coop-MCTS</sub> and *ColdGAN*, the gradients become more co-linear with human references through time, indicating a convergence of the process towards the human distribution. We observe that *SelfGAN*<sub>Coop-MCTS</sub> produces more useful sequences for achieving this convergence.

**Coop-MCTS as an alternative to the dead-end search** When analysing the behavior for the *Coop-MCTS* decoding, we observed in different examples that it provides an effective mean to revise generations that eventually ended up to be unlikely. To illustrate this, we report in Table 7.4 the different MCTS steps for an ambiguous example: the conditioned answer, *Super Bowl*, occurs at different places of the input. Therefore, the model has to decide which specific mention of *Super Bowl* to focus on: at step 17, it considers its current generation as a dead end and decides to start on new node (*How*). The final output is a question that arguably sounds better than the initial one.

## 7.6 Conclusion

In this Chapter we propose *SelfGAN*, a new framework to train Generative Adversarial Networks based on a cooperative decoding search. To overcome the left-to-right curse that limits standard search algorithms, we propose *Coop-MCTS*. We
**Conditioned Answer:** *Super Bowl*

**Context:**

Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals

**Step 01: What**

**Step 16: What was the name of the game that would have been known as "Super Bowl**

**Step 17: How**

**Step 46: How is called the American football game that determines the NFL champion?**

<caption>Table 7.4: Progressive results obtained by our Coop-MCTS decoding method on Question Generation during a simulation. Until the 16th step, the generation is left-to-right. Then, the cooperation mechanism kicks in, allowing the model to safely abort this beam, by restarting a new question with *How*. We report the cross-attention weights on the input context for step 16 (red) and 17 (blue).</caption>

conducted extensive experiments on two challenging tasks: Summarization and Question Generation, obtaining state-of-the-art performance for *SelfGAN* both in terms of automatic metrics and within a human evaluation. As the stability of the discriminator looks to be crucial for language GANs, we plan for future works to still focus on increasing it through the definition of dynamic regularization mechanisms. Finally, we will explore how reference-less metrics, e.g. QuestEval (presented in Chapter 4), can be combined to help the exploration during the decoding.
# CONCLUSION AND PERSPECTIVES

Fluent and reliable Natural Language Generation can have significant societal im-
pacts. On the one hand, we envision several applications beneficial for business,
research, or education: from automatic summarization of news articles, scientific
papers or books, to efficient information access; from automatic and personalized
student evaluation tests through question generation, to responsive conversational
interfaces. On the other hand, malicious actors can use the same technology to
build tools detrimental to society, e.g. for creation and propagation of mislead-
ing (fake) news as discussed in (Radford et al. 2019), impersonation, and deceit.
Nonetheless, keeping this research open and under public scrutiny is arguably
one of the best ways to defend against such actors (Zellers et al. 2019).

## 8.1 Beyond A Unique Reference

In this thesis, we have explored different axes to improve NLG, both on the generative and evaluation sides. One common trait among all these efforts is the need to work beyond a unique gold-reference.

In NLG, given an input, there are arguably many different possible outputs.
For both humans and machines, given the same input, it is highly likely that
several different – yet, all correct – sequences can be produced. In particular, the
probability to write the same exact sequence as the only gold-reference available
is very low. Should this behavior be penalized? Obviously not. And yet, this is
what happens under the standard NLG practices:

1) at training time with Teacher Forcing, any generated token that is different from the target increases the loss. The model can therefore be exposed to contradictory information, which might limit its effectiveness. Note that this issue does not apply to a discriminator, as only two output categories (machine or human) are possible.
2) **at inference time** models are evaluated with automatic metrics fed with only one gold-reference, since most large-scale datasets provide a single human example in their test set.

To address this limitation, and in contrast to other metrics, our QG/QA based metric *QuestEval* compares the evaluated text directly to the source, not the reference.

Regarding training, in the last chapter we proposed SelfGAN, which offers a theoretical solution to this multi-reference limitation. Lets denote $S_{human}$ the universe of possible correct outputs, where $s_{ref} \in S_{human}$. Then, given a perfect discriminator (optimal to distinguish real data distribution from a different distribution), and an infinite computational capacity, we have $s_{coop} \in S_{human}$. Indeed, given an infinite computational capacity, all the possible sequences can be explored. A perfect discriminator classifies a sequence $s$ as human only if $s \in S_{human}$. It results that $s_{coop} \in S_{human}$: the sequence generated via a cooperative mechanism is guaranteed to be indistinguishable from any human output, just like the reference.

In addition, since the generator probability is also taken into account in a cooperative decoding, we have $s_{coop} = \arg\max(P_{\pi}(S_{human}))$. We note that this is guaranteed only if all possible sequences are explored via an infinite computation. If we stop searching when one sequence is accepted by the decoder, it is pseudo-guaranteed since a Beam Search is only an approximation of the argmax.

$s_{coop}$ is the sequence among all the human sequences that maximise the likelihood according to the generator $\pi$. Therefore, if the generator outputs a human-level sequence (i.e. $s \in S_{human}$), it will actually correspond to $s_{coop}$. It results that considering $s_{coop}$ as the gold-reference in Teacher Forcing, the generator will not be subject to an undesirable loss.

In conclusion, SelfGAN can be interpreted as a generalisation of Teacher Forcing that takes into account the multiple possible references and trains the model on the reference the highest to its likelihood.

## 8.2 Future Directions

### 8.2.1 Metrics

In this thesis we have paved the way for a new kind of metrics based on Question Answering and Generation. We have shown how this method can be broadly adapted to various NLG tasks, systematically comparing favorably over the others metrics regarding human judgement.
Nonetheless, Question-based metrics have interesting properties by design. In particular, they provide a direct and fine-grained explanation w.r.t. the output hallucinations. This could open new ways toward automatic proofreading mechanisms, or being exploited within human-machine interactions. We note that such new directions are only made possible because our proposed metrics are reference-less, hence usable even when no ground-truth reference is available.

At the root of the reference-less property, lies the intuition that any objects can be compared by asking and answering questions. In this regard, QuestEval can be seen as a general similarity function, offering a way to compare the generated text with any object, in particular the source, and not necessarily a text. In contrast, metrics like ROUGE, BLEU or BERTScore operate at a token level. Hence, they require a certain alignment between the two compared texts. This prevents from using the source input, and limits to compare the evaluated text to a ground truth reference. Moreover, the NLG community is progressively moving toward more complex and longer documents. As several realizations of a correct generation are possible, the number of potentially correct gold-references exponentially rises w.r.t. the length of the input sequence. In this context, QuestEval becomes particularly interesting, as it loosens the dependence on a specific realization of the source text.

### 8.2.2 RL

Toward an unbiased reward, we have explored learned discriminators. First at inference time, as a way to select the words likely to sounds more human-like during the decoding process. Then at training time in a GAN. For the later, we have proposed different methods to stabilise the training process trough complex sampling strategies, such as MCTS. Overall, our empirical experiments obtain state-of-the-art results, under both automatic metrics and human evaluations.

Moving forward, investigating theoretical properties for our proposed approaches would be an important step. In particular, it would be interesting to introduce theoretical guarantees regarding discrete GANs, based on cooperative sampling strategies.

In addition, future works could study how our cooperative mechanisms might apply in the context of approaches based on density ratio estimators, such as the ones recently proposed in (lu2019cot; song2020improving). Hybrid approaches, based on ratio estimators between current densities and expected ones also constitute a promising research perspective.
<figure><img src="image_33.png" /></figure>
BIBLIOGRAPHY

Anthony, Thomas, Zheng Tian, and David Barber (2017). "Thinking fast and slow with deep learning and tree search". In: *Advances in Neural Information Processing Systems* 30 (cit. on p. 81).

Arjovsky, Martin and Léon Bottou (2017). "Towards principled methods for training generative adversarial networks". In: *arXiv preprint arXiv:1701.04862* (cit. on p. 82).

Arumae, Kristjan and Fei Liu (June 2019). "Guiding Extractive Summarization with Question-Answering Rewards". In: *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*. Minneapolis, Minnesota: Association for Computational Linguistics, pp. 2566–2577. URL: https://www.aclweb.org/anthology/N19-1264 (cit. on p. 21).

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio (2014). "Neural machine translation by jointly learning to align and translate". In: *arXiv preprint arXiv:1409.0473* (cit. on pp. 13, 14, 23).

Bengio, Samy, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer (2015). "Scheduled sampling for sequence prediction with recurrent neural networks". In: *Advances in Neural Information Processing Systems*, pp. 1171–1179 (cit. on pp. 2, 17).

Bengio, Yoshua, Patrice Simard, and Paolo Frasconi (1994). "Learning long-term dependencies with gradient descent is difficult". In: *IEEE transactions on neural networks* 5.2, pp. 157–166 (cit. on p. 11).

Bhandari, Manik, Pranav Gour, Atabak Ashfaq, and Pengfei Liu (Dec. 2020). "Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics". In: *Proceedings of COLING 2020, the 30th International Conference on Computational Linguistics: Technical Papers*. The COLING 2020 Organizing Committee (cit. on p. 2).

Brock, Andrew, Jeff Donahue, and Karen Simonyan (2018). "Large scale gan training for high fidelity natural image synthesis". In: *arXiv preprint arXiv:1809.11096* (cit. on p. 66).

Bubeck, Sébastien, Eric Price, and Ilya Razenshteyn (2018). "Adversarial examples from computational constraints". In: *arXiv preprint arXiv:1805.10204* (cit. on p. 70).

Caccia, Massimo, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin (2018). "Language gans falling short". In: *arXiv preprint arXiv:1811.02549* (cit. on p. 52).
Caccia, Massimo, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin (2020). "Language GANs Falling Short". In: *International Conference on Learning Representations*. URL: https://openreview.net/forum?id=BJgza6VtPB (cit. on pp. 19, 66, 73, 75, 79, 80, 88).

Che, Tong, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio (2017). "Maximum-likelihood augmented discrete generative adversarial networks". In: *arXiv preprint arXiv:1702.07983* (cit. on pp. 19, 73).

Chen, Ping, Fei Wu, Tong Wang, and Wei Ding (2018). "A Semantic QA-Based Approach for Text Summarization Evaluation". In: *Thirty-Second AAAI Conference on Artificial Intelligence* (cit. on p. 22).

Chen, Yen-Chun and Mohit Bansal (July 2018). "Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting". In: pp. 675–686. URL: https://www.aclweb.org/anthology/P18-1063 (cit. on p. 31).

Cho, Kyunghyun, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio (2014). "Learning phrase representations using RNN encoder-decoder for statistical machine translation". In: *arXiv preprint arXiv:1406.1078* (cit. on p. 12).

Chopra, Sumit, Michael Auli, and Alexander M Rush (2016). "Abstractive sentence summarization with attentive recurrent neural networks". In: *Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 93–98 (cit. on p. 12).

Clark, Kevin, Minh-Thang Luong, Quoc V Le, and Christopher D Manning (2019). "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators". In: *International Conference on Learning Representations* (cit. on pp. 19, 52).

Coulom, Rémi (2006). "Efficient selectivity and backup operators in Monte-Carlo tree search". In: *International conference on computers and games*. Springer, pp. 72–83 (cit. on p. 81).

Deng, Yuntian, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc'Aurelio Ranzato (2020). "Residual energy-based models for text generation". In: *arXiv preprint arXiv:2004.11714* (cit. on p. 80).

Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (2019). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". In: *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp. 4171–4186 (cit. on p. 19).

Dong, Li, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon (2019). "Unified language model pre-training for natural language understanding and generation". In: *Advances in*
*Neural Information Processing Systems*, pp. 13042–13054 (cit. on pp. 9, 54, 56, 57, 75, 76, 89).

Du, Xinya, Junru Shao, and Claire Cardie (2017). "Learning to Ask: Neural Question Generation for Reading Comprehension". In: *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 1342–1352 (cit. on p. 75).

Durmus, Esin, He He, and Mona Diab (2020). "FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization". In: *arXiv preprint arXiv:2005.03754* (cit. on p. 38).

Eyal, Matan, Tal Baumel, and Michael Elhadad (June 2019). "Question Answering as an Automatic Evaluation Metric for News Article Summarization". In: *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*. Association for Computational Linguistics (cit. on pp. 22, 24).

Fabbri, Alexander R, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev (2020). "SummEval: Re-evaluating Summarization Evaluation". In: *arXiv preprint arXiv:2007.12626* (cit. on pp. 17, 35, 40, 41, 90).

Fan, Angela, Mike Lewis, and Yann Dauphin (2018). "Hierarchical neural story generation". In: *arXiv preprint arXiv:1805.04833* (cit. on p. 8).

Gehrmann, Sebastian, Yuntian Deng, and Alexander Rush (2018). "Bottom-Up Abstractive Summarization". In: *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pp. 4098–4109 (cit. on pp. 9, 19, 23, 28, 31, 56, 60, 61).

Gilmer, Justin, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow (2018). "Adversarial Spheres". In: (cit. on p. 70).

Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio (2014). "Generative adversarial nets". In: *Advances in neural information processing systems*, pp. 2672–2680 (cit. on pp. 3, 51, 66).

Gunasekara, Chulaka, Guy Feigenblat, Benjamin Sznajder, Ranit Aharonov, and Sachindra Joshi (Nov. 2021). "Using Question Answering Rewards to Improve Abstractive Summarization". In: *Findings of the Association for Computational Linguistics: EMNLP 2021*. Punta Cana, Dominican Republic: Association for Computational Linguistics, pp. 518–526. URL: https://aclanthology.org/2021.findings-emnlp.47 (cit. on p. 49).

Guo, Jiaxian, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang (2018). "Long text generation via adversarial training with leaked information". In: *Thirty-Second AAAI Conference on Artificial Intelligence* (cit. on p. 73).
Hermann, Karl Moritz, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom (2015). "Teaching machines to read and comprehend". In: *Advances in neural information processing systems*, pp. 1693–1701 (cit. on pp. 24, 28, 56).

Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean (2015). "Distilling the knowledge in a neural network". In: *arXiv preprint arXiv:1503.02531* (cit. on p. 67).

Hochreiter, Sepp and Jürgen Schmidhuber (1997). "Long short-term memory". In: *Neural computation* 9.8, pp. 1735–1780 (cit. on p. 11).

Hokamp, Chris and Qun Liu (2017). "Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search". In: *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 1535–1546 (cit. on p. 9).

Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi (2019). "The curious case of neural text degeneration". In: *arXiv preprint arXiv:1904.09751* (cit. on pp. 1, 8, 37, 70, 72, 83).

Kober, Jens and Jan R Peters (2009). "Policy search for motor primitives in robotics". In: *Advances in neural information processing systems*, pp. 849–856 (cit. on p. 70).

Kocmi, Tom, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes (2021). "To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation". In: *arXiv preprint arXiv:2107.10821* (cit. on p. 15).

Kocsis, Levente and Csaba Szepesvári (2006). "Bandit based monte-carlo planning". In: *European conference on machine learning*. Springer, pp. 282–293 (cit. on p. 81).

Konda, Vijay and John Tsitsiklis (1999). "Actor-critic algorithms". In: *Advances in neural information processing systems 12* (cit. on p. 18).

Krause, Ben, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani (2020). "Gedi: Generative discriminator guided sequence generation". In: *arXiv preprint arXiv:2009.06367* (cit. on p. 63).

Kryscinski, Wojciech, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher (Nov. 2019a). "Neural Text Summarization: A Critical Evaluation". In: *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*. Hong Kong, China: Association for Computational Linguistics, pp. 540–551. URL: https://www.aclweb.org/anthology/D19-1051 (cit. on p. 35).

Kryscinski, Wojciech, Bryan McCann, Caiming Xiong, and Richard Socher (2019b). "Evaluating the factual consistency of abstractive text summarization". In: *arXiv preprint arXiv:1910.12840* (cit. on p. 19).
Kryściński, Wojciech, Romain Paulus, Caiming Xiong, and Richard Socher (Oct. 2018). "Improving Abstraction in Text Summarization". In: pp. 1808–1817. URL: https://www.aclweb.org/anthology/D18-1207 (cit. on pp. 19, 60).

Lamb, Alex M, Anirudh Goyal, Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio (2016). "Professor forcing: A new algorithm for training recurrent networks". In: *Advances In Neural Information Processing Systems*, pp. 4601–4609 (cit. on pp. 2, 17).

Lavie, Alon and Abhaya Agarwal (2007). "METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments". In: *Proceedings of the second workshop on statistical machine translation*, pp. 228–231 (cit. on p. 16).

Lee, Dongyub, Myeongcheol Shin, Taesun Whang, Seungwoo Cho, Byeongil Ko, Daniel Lee, Eunggyun Kim, and Jaechoon Jo (Dec. 2020). "Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization". In: *Proceedings of COLING 2020, the 30th International Conference on Computational Linguistics: Technical Papers*. The COLING 2020 Organizing Committee (cit. on p. 49).

Lee, Hwanhee, Thomas Scialom, Seunghyun Yoon, Franck Dernoncourt, and Kyomin Jung (2021). "QACE: Asking questions to evaluate an image caption". In: *arXiv preprint arXiv:2108.12560* (cit. on p. 48).

Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer (2019). "Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension". In: *arXiv preprint arXiv:1910.13461* (cit. on pp. 73, 76).

Li, Jiwei, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky (2017). "Adversarial Learning for Neural Dialogue Generation". In: *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*, pp. 2157–2169 (cit. on pp. 19, 66).

Lin, C-Y (2004). "ROUGE: A Package for Automatic Evaluation of Summaries". In: *Proc. of Workshop on Text Summarization Branches Out, Post Conference Workshop of ACL 2004*, pp. 74–81 (cit. on pp. 2, 16, 85).

Lin, Kevin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun (2017). "Adversarial ranking for language generation". In: *Advances in Neural Information Processing Systems*, pp. 3155–3165 (cit. on p. 73).

Liu, Yang and Mirella Lapata (2019). "Text Summarization with Pretrained Encoders". In: *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pp. 3721–3731 (cit. on p. 76).
Louis, Annie and Ani Nenkova (2013). "Automatically assessing machine summary content without a gold standard". In: *Computational Linguistics* 39.2, pp. 267–300 (cit. on p. 2).

Masson d'Autume, Cyprien de, Shakir Mohamed, Mihaela Rosca, and Jack Rae (2019a). "Training language gans from scratch". In: *Advances in Neural Information Processing Systems*, pp. 4302–4313 (cit. on pp. 19, 66, 73–75, 80).

Masson d'Autume, Cyprien de, Sebastian Ruder, Lingpeng Kong, and DaniYGogatama (2019b). "Episodic Memory in Lifelong Language Learning". In: *Advances in Neural Information Processing Systems*, pp. 13122–13131 (cit. on p. 73).

McKeown, Kathleen R (1980). "Generating Relevant Explanations: Natural Language Responses to Questions about Database Structure." In: AAAI, pp. 306–309 (cit. on p. 1).

Metz, Luke, Ben Poole, David Pfau, and Jascha Sohl-Dickstein (2016). "Unrolled generative adversarial networks". In: *arXiv preprint arXiv:1611.02163* (cit. on p. 72).

Mihalcea, Rada and Paul Tarau (2004). "Textrank: Bringing order into text". In: *Proceedings of the 2004 conference on empirical methods in natural language processing* (cit. on pp. 6, 26).

Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean (2013). "Efficient estimation of word representations in vector space". In: *arXiv preprint arXiv:1301.3781* (cit. on p. 6).

Mikolov, Tomas, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur (2010). "Recurrent neural network based language model." In: *Inter-speech. Vol. 2. 3*. Makuhari, pp. 1045–1048 (cit. on p. 10).

Mikolov, Tomáš, Stefan Kombrink, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur (2011). "Extensions of recurrent neural network language model". In: *2011 IEEE international conference on acoustics, speech and signal processing (ICASSP)*. IEEE, pp. 5528–5531 (cit. on p. 10).

Nallapati, Ramesh, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. (2016). "Abstractive text summarization using sequence-to-sequence rnns and beyond". In: *arXiv preprint arXiv:1602.06023*, pp. 280–290 (cit. on pp. 28, 40, 56, 75, 84).

Narayan, Shashi, Shay B Cohen, and Mirella Lapata (2018a). "Don't Give Me the Details, Just the Summary!" In: *Topic-aware Convolutional Neural Networks for Extreme Summarization*. In (cit. on p. 41).

Narayan, Shashi, Shay B. Cohen, and Mirella Lapata (June 2018b). "Ranking Sentences for Extractive Summarization with Reinforcement Learning". In: *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*. New Orleans, Louisiana: Association for Computational Linguistics, pp. 1747–1759. URL: https://www.aclweb.org/anthology/N18-1158 (cit. on p. 23).
Nenkova, Ani (2011). *Automatic Summarization*. Foundations and Trends in Information Retrieval. Now (cit. on pp. 22, 25).

Ng, Jun-Ping and Viktoria Abrecht (2015). "Better Summarization Evaluation with Word Embeddings for ROUGE". In: *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*, pp. 1925–1930 (cit. on p. 22).

Noble, William S (2006). "What is a support vector machine?" In: *Nature biotechnology* 24.12, pp. 1565–1567 (cit. on p. 6).

Novikova, Jekaterina, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser (Sept. 2017a). "Why We Need New Evaluation Metrics for NLG". In: *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*. Copenhagen, Denmark: Association for Computational Linguistics, pp. 2241–2252. URL: https://www.aclweb.org/anthology/D17-1238 (cit. on p. 16).

Novikova, Jekaterina, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser (2017b). "Why we need new evaluation metrics for NLG". In: *arXiv preprint arXiv:1707.06875* (cit. on p. 2).

Ott, Myle, Michael Auli, David Grangier, and Marc'Aurelio Ranzato (2018). "Analyzing uncertainty in neural machine translation". In: *arXiv preprint arXiv:1803.00047* (cit. on p. 9).

Page, Lawrence, Sergey Brin, Rajeev Motwani, and Terry Winograd (1999). *The PageRank citation ranking: Bringing order to the web*. Tech. rep. Stanford InfoLab (cit. on pp. 6, 26).

Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu (2002). "BLEU: a method for automatic evaluation of machine translation". In: *Proceedings of the 40th annual meeting on association for computational linguistics*. Association for Computational Linguistics, pp. 311–318 (cit. on pp. 15, 73, 85).

Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio (2013). "On the difficulty of training recurrent neural networks". In: *International conference on machine learning*. PMLR, pp. 1310–1318 (cit. on p. 11).

Pasunuru, Ramakanth and Mohit Bansal (June 2018). "Multi-Reward Reinforced Summarization with Saliency and Entailment". In: *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)*. New Orleans, Louisiana: Association for Computational Linguistics, pp. 646–653. URL: https://www.aclweb.org/anthology/N18-2102 (cit. on pp. 21, 24, 31).

Paulus, Romain, Caiming Xiong, and Richard Socher (2017). "A deep reinforced model for abstractive summarization". In: *arXiv preprint arXiv:1705.04304* (cit. on pp. 2, 9, 17, 18, 21, 23, 25, 27, 29–33, 79, 89).

Peyrard, Maxime (2019). "Studying summarization evaluation metrics in the appropriate scoring range". In: *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pp. 5093–5100 (cit. on p. 2).
Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. (2019). "Language models are unsupervised multitask learners". In: *Ope-nAI blog* 1.8, p. 9 (cit. on pp. 8, 16, 93).

Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu (2019). "Exploring the limits of transfer learning with a unified text-to-text transformer". In: *arXiv preprint* arXiv:1910.10683 (cit. on pp. 37, 67, 76, 84).

Rajpurkar, Pranav, Robin Jia, and Percy Liang (2018). "Know what you don't know: Unanswerable questions for SQuAD". In: *arXiv preprint arXiv:1806.03822* (cit. on p. 41).

Rajpurkar, Pranav, Jian Zhang, Konstantin Lopyrev, and Percy Liang (Nov. 2016). "Squad: 100,000+ questions for machine comprehension of text". In: *arXiv preprint arXiv:1606.05250*, pp. 2383-2392. URL: https://www.aclweb.org/anthology/D16-1264 (cit. on pp. 6, 24, 37, 38, 75, 84).

Ranzato, Marc'Aurelio, Sumit Chopra, Michael Auli, and Wojciech Zaremba (2015a). "Sequence level training with recurrent neural networks". In: *arXiv preprint arXiv:1511.06732* (cit. on pp. 1, 2, 18, 23).

Ranzato, Marc'Aurelio, Sumit Chopra, Michael Auli, and Wojciech Zaremba (2015b). "Sequence level training with recurrent neural networks". In: *arXiv preprint arXiv:1511.06732* (cit. on p. 17).

Rebuffel, Clément, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari (2021). "Data-QuestEval: A Referenceless Metric for Data to Text Semantic Evaluation". In: *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics (cit. on p. 48).

Reiter, Ehud and Robert Dale (1997). "Building applied natural language generation systems". In: *Natural Language Engineering* 3.1, pp. 57-87 (cit. on p. 1).

Riabi, Arij, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Staiano (2020). "Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering". In: *arXiv preprint arXiv:2010.12643* (cit. on p. 49).

Rosin, Christopher D (2011). "Multi-armed bandits with episode context". In: *Annals of Mathematics and Artificial Intelligence* 61.3, pp. 203-230 (cit. on p. 83).

Rückstieß, Thomas, Martin Felder, and Jürgen Schmidhuber (2008). "State-dependent exploration for policy gradient methods". In: *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*. Springer, pp. 234-249 (cit. on p. 70).

Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams (1986). "Learning representations by back-propagating errors". In: *nature* 323.6088, pp. 533-536 (cit. on p. 10).
Rush, Alexander M, Sumit Chopra, and Jason Weston (2015). "A neural attention model for abstractive sentence summarization". In: *arXiv preprint arXiv:1509.00685* (cit. on p. 14).

Sai, Ananya B, Akash Kumar Mohankumar, and Mitesh M Khapra (2020). "A survey of evaluation metrics used for NLG systems". In: *arXiv preprint arXiv:2008.12009* (cit. on p. 15).

Scialom, Thomas, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano (2020). "MLSUM: The Multilingual Summarization Corpus". In: *arXiv preprint arXiv:2004.14900* (cit. on p. 49).

Scialom, Thomas, Louis Martin, Jacopo Staiano, Éric Villemonte de la Clergerie, and Benoît Sagot (2021). "Rethinking Automatic Evaluation in Sentence Simplification". In: *arXiv preprint arXiv:2104.07560* (cit. on p. 48).

See, Abigail, Peter J Liu, and Christopher D Manning (2017). "Get To The Point: Summarization with Pointer-Generator Networks". In: *ACL (1)*, pp. 1073–1083 (cit. on pp. 23, 28, 31, 56, 60, 89).

Semeniuta, Stanislau, Aliaksei Severyn, and Sylvain Gelly (2018). "On accurate evaluation of gans for language generation". In: *arXiv preprint arXiv:1806.04936* (cit. on pp. 19, 66, 73, 75).

Silver, David, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. (2017). "Mastering the game of go without human knowledge". In: *nature* 550.7676, pp. 354–359 (cit. on p. 83).

Srivastava, Nitish, Elman Mansimov, and Ruslan Salakhudinov (2015). "Unsupervised learning of video representations using lstms". In: *International conference on machine learning*. PMLR, pp. 843–852 (cit. on p. 10).

Sulem, Elior, Omri Abend, and Ari Rappoport (Oct. 2018). "BLEU is Not Suitable for the Evaluation of Text Simplification". In: *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*. Brussels, Belgium: Association for Computational Linguistics, pp. 738–744. URL: https://www.aclweb.org/anthology/D18-1081 (cit. on p. 16).

Sutskever, Ilya, Oriol Vinyals, and Quoc V Le (2014). "Sequence to sequence learning with neural networks". In: *Advances in neural information processing systems*, pp. 3104–3112 (cit. on pp. 1, 7, 10, 12, 23, 67).

Sutton, Richard S and Andrew G Barto (2018). *Reinforcement learning: An introduction*. MIT press (cit. on p. 66).

Tevet, Guy, Gavriel Habib, Vered Shwartz, and Jonathan Berant (2018). "Evaluating text gans as language models". In: *arXiv preprint arXiv:1810.12686* (cit. on pp. 66, 75).

Trischler, Adam, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman (2016). "Newsqa: A machine comprehension dataset". In: *arXiv preprint arXiv:1611.09830* (cit. on p. 41).
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin (2017). "Attention is all you need". In: *Advances in neural information processing systems*, pp. 5998–6008 (cit. on pp. 14, 57, 75).

Venkatraman, Arun, Martial Hebert, and J Andrew Bagnell (2015). "Improving multi-step prediction of learned time series models". In: *Twenty-Ninth AAAI Conference on Artificial Intelligence* (cit. on p. 17).

Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly (2015). "Pointer networks". In: *Advances in Neural Information Processing Systems*, pp. 2692–2700 (cit. on pp. 19, 23).

Völske, Michael, Martin Potthast, Shahbaz Syed, and Benno Stein (Sept. 2017a). "TL;DR: Mining Reddit to Learn Automatic Summarization". In: *Proceedings of the Workshop on New Frontiers in Summarization*. Copenhagen, Denmark: Association for Computational Linguistics, pp. 59–63. URL: https://www.aclweb.org/anthology/W17-4508 (cit. on p. 56).

Völske, Michael, Martin Potthast, Shahbaz Syed, and Benno Stein (2017b). "Tl;dr: Mining reddit to learn automatic summarization". In: *Proceedings of the Workshop on New Frontiers in Summarization*, pp. 59–63 (cit. on p. 28).

Wang, Alex, Kyunghyun Cho, and Mike Lewis (2020). "Asking and answering questions to evaluate the factual consistency of summaries". In: *arXiv preprint arXiv:2004.04228* (cit. on pp. 35–39, 41, 47).

Wang, Alex, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman (2019). "Superglue: A stickier benchmark for general-purpose language understanding systems". In: *arXiv preprint arXiv:1905.00537*, pp. 3261–3275 (cit. on p. 67).

Wang, Alex, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman (Nov. 2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding". In: *Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP*. Brussels, Belgium: Association for Computational Linguistics, pp. 353–355. URL: https://www.aclweb.org/anthology/W18-5446 (cit. on p. 67).

Wang, Ziyu, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas (2016). "Sample Efficient Actor-Critic with Experience Replay". In: (cit. on p. 72).

Weizenbaum, Joseph (1966). "ELIZA—a computer program for the study of natural language communication between man and machine". In: *Communications of the ACM* 9.1, pp. 36–45 (cit. on p. 1).

Williams, Ronald J (1992). "Simple statistical gradient-following algorithms for connectionist reinforcement learning". In: *Machine learning* 8.3-4, pp. 229–256 (cit. on p. 23).
Williams, Ronald J and David Zipser (1989). "A learning algorithm for continually running fully recurrent neural networks". In: *Neural computation* 1.2, pp. 270–280 (cit. on pp. 1, 7, 17).

Williams, Ronald J and David Zipser (1995). "Gradient-based learning algorithms for recurrent". In: *Backpropagation: Theory, architectures, and applications* 433, p. 17 (cit. on p. 10).

Winograd, Terry (1980). "What does it mean to understand language?" In: *Cognitive science* 4.3, pp. 209–241 (cit. on p. 1).

Wu, Harris, Dragomir Radev, and Weiguo Fan (2002). "Towards answer-focused summarization". In: *Proceedings of the 1st International Conference on Information Technology and Applications* (cit. on p. 24).

Wu, Yonghui, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. (2016). "Google's neural machine translation system: Bridging the gap between human and machine translation". In: *arXiv preprint arXiv:1609.08144* (cit. on pp. 9, 79).

Xingjian, SHI, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo (2015). "Convolutional LSTM network: A machine learning approach for precipitation nowcasting". In: *Advances in neural information processing systems*, pp. 802–810 (cit. on p. 10).

Yoon, Wonjin, Yoon Sun Yeo, Minbyul Jeong, Bong-Jun Yi, and Jaewoo Kang (2020). "Learning by Semantic Similarity Makes Abstractive Summarization Better". In: *arXiv preprint arXiv:2002.07767* (cit. on p. 77).

You, Quanzeng, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo (2016). "Image captioning with semantic attention". In: *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 4651–4659 (cit. on p. 14).

Yu, Lantao, Weinan Zhang, Jun Wang, and Yong Yu SeqGAN (2016). "Sequence Generative Adversarial Nets with Policy Gradient". In: *arXiv preprint arXiv:1609.05473* (cit. on pp. 19, 66).

Zellers, Rowan, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi (2019). "Defending against neural fake news". In: *Advances in Neural Information Processing Systems*, pp. 9051–9062 (cit. on pp. 19, 93).

Zhang, Jingqing, Yao Zhao, Mohammad Saleh, and Peter J Liu (2019). "PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization". In: *arXiv preprint arXiv:1912.08777* (cit. on p. 76).

Zhang, Shiyue and Mohit Bansal (2019). "Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering". In: *arXiv preprint arXiv:1909.06356* (cit. on p. 76).

Zhang, Yizhe, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin (2017). "Adversarial feature matching for text generation". In:
*Proceedings of the 34th International Conference on Machine Learning-Volume 70.*
JMLR. org, pp. 4006–4015 (cit. on p. 66).

Zhou, Qingyu, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou (2017). "Neural question generation from text: A preliminary study". In: *National CCF Conference on Natural Language Processing and Chinese Computing*. Springer, pp. 662–671 (cit. on pp. 37, 75).

Zhou, Wangchunshu, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou (2020). "Self-Adversarial Learning with Comparative Discrimination for Text Generation". In: *International Conference on Learning Representations*. URL: https://openreview.net/forum?id=B118L6EtDS (cit. on pp. 19, 52).

Zhu, Yaoming, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu (2018). "Texxygen: A benchmarking platform for text generation models". In: *The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval*, pp. 1097–1100 (cit. on p. 73).

Ziegler, Daniel M, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving (2019). "Fine-tuning language models from human preferences". In: *arXiv preprint arXiv:1909.08593* (cit. on pp. 2, 66).