Joint Generator-Ranker Learning for Natural Language Generation
Year: 2023

Abstract:
Generate-then-rank is a widely used mechanism for text generation, where a generator produces multiple text candidates and a ranker chooses the best one among the text candidates. However, existing methods usually train the generator and the ranker individually, neglecting the mutual feedback that could further enhance the generation quality. To tackle this limitation, we propose JGR, a novel joint training algorithm that integrates the generator and the ranker in a single framework. JGR optimizes the generator with a hybrid objective that combines data likelihood and ranker reward, and trains the ranker with a contrastive loss that compares the generator outputs. By iteratively updating the generator and the ranker, JGR can effectively harmonize their learning and enhance their quality jointly. We evaluate JGR on various text generation tasks and demonstrate that it surpasses existing methods on four public datasets across three common generation scenarios. Our code and models are publicly available at https://github.com/ microsoft/ProphetNet/tree/master/JGR.

heading: 1 Introduction
text: The quality of the output texts produced by neural natural language generation (NLG) models, such as those for machine translation (Vaswani et al., 2017) and summarization (Lewis et al., 2019), depends largely on how they are trained and decoded. The conventional approach is to train them with loglikelihood objectives and decode them with greedy or beam search strategies. However, this approach often fails to select the best sample with the highest evaluation score among the generated candidates, as shown by previous studies (Cohen and Beck, 2019; Meister et al., 2020).
To overcome this limitation, some recent works (Liu and Liu, 2021; Liu et al., 2021; Li et al.,
∗Done during his internship at Microsoft Research Asia. † Corresponding authors.
2022b; Ravaut et al., 2022) proposed to use a separate ranker model to re-rank the output texts of the generator model, following a generate-then-rank pipeline. This pipeline can improve the quality of the output texts by exploiting the ranker model’s ability to evaluate and compare different candidates. However, this pipeline also has a drawback: it requires training the generator and ranker models in two separate phases, which may not fully exploit the generative ability of the generator model and the feedback from the ranker model.
In this paper, we propose a novel Joint training paradigm of both Generator and Ranker (JGR) for NLG tasks, which aims to overcome the drawback of the generate-then-rank pipeline. Unlike previous works, which train the generator and ranker models separately, we explore a joint and iterative training algorithm that updates both models in turn. Our main motivation for the joint and iterative training of the generator and ranker is twofold. First, the ranker model can provide valuable feedback to the generator model based on the ranking scores of the generated candidates. This encourages the generator model to produce better outputs. Second, the ranker model can also benefit from the outputs of a progressively better generator model, and improve its ranking performance gradually.
The JGR framework consists of a generator and a ranker. During training, the generator and ranker alternate to update their parameters, and each of them involves the other’s outputs in its own input signals. Specifically, the ranker model is trained to rank the outputs generated by the generator model for a given input text by assigning a ranking score. At the generator training phase, the generator model uses a combination of the ranker score and the matching score (e.g., BLEU) as the reward for each sample, and trains with policy gradients, which encourages the generator to produce candidates with higher rewards and mitigates the exposure bias issue in the teacher-forcing learning.
7681
To assess the effectiveness of JGR, we conduct experiments on four diverse NLG tasks from different domains, including abstractive summarization (Hermann et al., 2015), conversational summarization (Gliwa et al., 2019), question generation (Rajpurkar et al., 2016), and dialogue (Zhang et al., 2018). The experimental results demonstrate that JGR achieves remarkable performance gains over the conventional MLE training method, with a 3-point increase in ROUGE-2 score on the CNN/DailyMail dataset and a 3.5-point increase in BLEU-2 score on PersonaChat.
Furthermore, we make several interesting observations from the results. First, the rewards from the ranker are more effective than the rewards from the direct metrics, but combining them together stabilizes the training and produces a better generator. Second, training the ranker only on the candidates from the generator is better than using ground-truth as positive examples. Third, sampling more candidates during training leads to better performance within a certain range, which is consistent with data augmentation. Fourth, though trained with reinforcement learning aimed at optimizing automatic evaluation metrics, JGR still does not compromise on other aspects of generation quality. Finally, the joint training paradigm increases the diversity of the generator outputs, which in turn benefits the ranker training.

heading: 2 Related Work

heading: 2.1 Natural Language Generation
text: Natural language generation is a long-standing research topic. RNN-based methods for dialog systems (Wen et al., 2015) and convolutional methods for translation (Gehring et al., 2016) are some examples of earlier approaches. In the last few years, pre-trained transformer models have advanced the state of the art on many NLG tasks. These models, such as BART (Lewis et al., 2019), ProphetNet (Qi et al., 2020), and T5 (Raffel et al., 2020), use an encoder-decoder architecture and leverage large amounts of unlabeled data. Other models, such as GPT2 (Radford et al., 2019) and UniLM (Dong et al., 2019), use only a decoder or an encoder for natural language generation.
Reinforcement learning can assist the training of NLG models, as shown by several works. Rennie et al. (2017); Paulus et al. (2018) used selfcritical methods that measure the reward as the difference between the metric score and the baseline
score. Bahdanau et al. (2017); Le et al. (2022) introduced actor-critic frameworks (Konda and Tsitsiklis, 1999), which is also a joint training framework, while they have not considered the contrastive rewards between different candidates given one input. We provide a more detailed comparison in A.1.
Another common approach to NLG is to apply adversarial networks (Goodfellow et al., 2014). For example, SeqGAN (Yu et al., 2017), RankGAN (Lin et al., 2017), GCN (Lamprier et al., 2022) and SelfGAN (Scialom et al., 2021b). These methods also introduce a joint training framework, however, instead of training a ranker, they trained a discriminator, which distinguishes the ground-truth text and the generator outputs. In Appendix A.2, we detail the main distinctions between these methods and our JGR.

heading: 2.2 Generate-then-Rank Framework
text: The generate-then-rank framework generates some candidate texts with a generator and then ranks them with a ranker. SimCLS (Liu and Liu, 2021), RefSum (Liu et al., 2021), and SumRanker (Ravaut et al., 2022) train rankers separately to rank the outputs of summarization models such as BART (Lewis et al., 2019). In other domains, such as code generation and math problem solving, rankers are also used to evaluate the generated outputs, as shown by AlphaCode (Li et al., 2022b) and Verifier (Cobbe et al., 2021). There are also some works trying to compress the generate-then-rank pipeline to one single model using extra training objectives, for example, MATCHSUM (Zhong et al., 2020), CoLo (An et al., 2022), and BRIO (Liu et al., 2022) with contrastive learning, and Amortized Noisy-Channel NMT (Pang et al., 2021) with Q-learning. However, the above methods do not explore the joint training framework that optimizes both generators and rankers together.
In the retrieve-then-rank framework for dense retrieval (Karpukhin et al., 2020), a retriever first finds relevant documents from a large collection, then a ranker reorders them according to their scores. Our JGR is partially motivated by this framework, we think in the generate-then-rank framework, the generation can be viewed as a retrieval process. Therefore, during training and inference, the generator should sample enough candidates for the ranker to re-rank. Several works have proposed to jointly train the retriever and the ranker to improve retrieve-then-rank framework. Such as
RocketQA v2 (Ren et al., 2021) and AR2 (Zhang et al., 2021). However, to our knowledge, JGR is the first work applying the joint training paradigm to the generate-then-rank framework for NLG.

heading: 3 Methodology
text: The model architecture of our JGR, shown in Figure 1, has two components: a generator that outputs several text candidates for an input text using an encoder-decoder model, and a ranker that scores these text candidates. The JGR workflow works as follows: a) the generator generates multiple text candidates conditioned on the input text; b) the input text and the text candidates are combined and sent to the ranker; c) the ranker learns to rank the text candidates via a contrastive learning objective; d) the ranker gives a reward to each text candidate, which in turn is used to train the generator. In the following, we first introduce the basic elements of conditional text generation, including problem definition, model architecture, and model training.

heading: 3.1 Preliminaries
text: Given a text pair (x,y), x is the input text sequence, y is the target text sequence. The conditional text generation tasks ask the model to generate a high-quality output ŷ that close the ground-truth y based on the input x. We adopt the Transformer-based (Vaswani et al., 2017) encoderdecoder architecture as the general model for conditional text generation. The encoder part transforms x into a tensor representation He using the Transformer model, as shown in Eqn. 1.
He = Encoder(x), (1)
The decoder part uses He as input and produces a text sequence via the auto-regressive fashion.
ŷ ∼ Decoder(ŷ,He) = |ŷ|∏
t=1
p(ŷt|ŷ<t,He). (2)
To simplify the notation, we use Gθ(·) to denote the encoder-decoder generation model with parameters θ, and pGθ(ŷ|x) to denote the probability of generating ŷ given x. The standard way to train the encoder-decoder sequence generation model is to minimize the negative log-likelihood of the ground-truth target sequence:
LNLL = − |y|∑
t=1
log pGθ(yt|y<t,x). (3)
During inference of the generator, a decoding strategy such as beam search is usually adopted. However, previous studies (Cohen and Beck, 2019; Meister et al., 2020) observed that the top-scored candidate from decoding strategies is often not the optimal candidate regarding the evaluation metric. Therefore, we design JGR to alleviate this problem through joint training of the generator and a ranker.

heading: 3.2 Joint Generator-Ranker Training
text: We use Gθ(·) and Dϕ(·) to represent the generator model and ranker model respectively, where Gθ(·) is a text generation model with an encoder-decoder structure as explained in section 3.1, and Dϕ(·) works as a scoring model that takes the concatenation of input text x and generated text ŷ as the input, and outputs a scalar value sŷ reprensenting
the quality of the generated text:
sŷ = Dϕ([x, ŷ]) (4)
During the training stage, the generator and ranker are trained alternatively and iteratively. Algorithm 1 shows the training procedure of JGR. We first warm up the generator Gθ with a standard negative log-likelihood (NLL) loss according to Eqn 3. Then, we iteratively update the ranker and generator as follows.
Fix Gθ(·), Train Dϕ(·): the goal of the ranker model Dϕ(·) is to choose the best sample from a set of candidates generated by the generator model, which we denote as Ŷ = {ŷ1, ŷ2, ..., ŷC}
{ŷ1, ŷ2, ..., ŷC} ∼ pGθ(·|x), (5)
where C is the number of sampled candidates. For each ŷi, we calculate the matching score (e.g., BLEU or ROUGE) with the ground-truth text y, denoted as ∆(y, ŷi). Then, we pick up the positive and negative samples in the candidate set based on ∆(y, ŷi) for training the ranker. Specifically, we use ŷ+ to denote the text candidate with the highest matching score, and Ŷ−, whose size is a hyper-parameter, to denote the negative candidate set containing a certain number of candidates with the lowest scores. The ranker model is trained by minimizing contrastive loss:
Lϕ = − log pDϕ(ŷ+|Ŷ−,x), (6)
where pDϕ(ŷ +|Ŷ−,x) is the probability of selecting ŷ+ from {ŷ+} ∪ Y−, which is computed by applying softmax on the ranking scores:
pDϕ(ŷ +|Ŷ−,x) = exp
sŷ+
expsŷ+ + ∑
ŷ−∈Ŷ− exp sŷ−
,
(7) where sŷ+ and sŷ− are the ranking scores of positive candidate and negative candidate, respectively.
After several steps of updating the ranker, we fix the ranker and update the generator.
Fix Dϕ(·), Train Gθ(·): the generator model is trained in two ways. The first one is LNLL, which uses a teacher-forcing mechanism to minimize the negative log-likelihood loss function over the training instances as discussed in Section 3.1 ( Eqn. 3). The second one is LRL - a reinforcement learningbased approach in which the generator model acts as a policy network to produce a list of text samples Ŷ given the input x, and the ranker model gives a
Algorithm 1 Joint Training of Generator and Ranker (JGR) Require: Generator Gθ; Ranker Dϕ; Training
data D. 1: Initialize Gθ and Dϕ from the pre-trained lan-
guage models. 2: Train the warm-up generator G0θ on D. 3: while model has not converged do 4: for training steps A do 5: Sample candidates Ŷ ∼ pGθ(·|x) for each x in the mini-batch. 6: Select ŷ+ and Ŷ− from Ŷ 7: Update parameters of Dϕ with Eq 6. 8: end for 9: for training steps B do 10: Sample candidates Ŷ ∼ pGθ(·|x) for each x in the mini-batch. 11: Compute rewards R(ŷ) for each ŷ ∈ Ŷ . 12: Update parameters of Gθ with Eq 9. 13: end for 14: end while
reward to each text sample in Ŷ based on its ranking score. The generator model can be trained by maximizing (minimizing) the expected (negative) reward (Sutton et al., 1999):
LRL = − ∑
ŷ∈Ŷ (R(ŷ)− b)
∑
t
log pGθ(ŷt|ŷ<t,x),
(8) where R(ŷ) is the reward for sample ŷ, calculated by combining the matching score ∆(ŷ,y) and the ranking score sŷ: R(ŷ) = ∆(ŷ,y) + sŷ. A baseline b is used to reduce the variance in RL training, which is computed by averaging the rewards of all samples in the candidate set: b =
∑ ŷ∈Ŷ R(ŷ)/C.
We then combine LNLL and LRL to form the final objective function for generator model training :
Lθ = LNLL + LRL. (9)
After updating the generator for several steps, we go back to fixing the generator and updating the ranker. This iteration will continue until the entire JGR framework converges.

heading: 4 Experimental Settings

heading: 4.1 Datasets
text: We evaluate the proposed method on four publicly available benchmarks across four domains:
CNN/DailyMail (Hermann et al., 2015) for abstractive summarization, SAMSum (Gliwa et al., 2019) for conversational summarization, SQuAD 1.1 (Rajpurkar et al., 2016) for question generation, and PersonaChat (Zhang et al., 2018) for dialogue generation. The details of these benchmarks and the used evaluation metrics are given in Appendix F.

heading: 4.2 Implemention Details
text: We use BART-large (Lewis et al., 2019) as the backbone model for the generator. The backbone of the ranker is based on RoBERTa-large (Liu et al., 2019). The generator and ranker models are initialized with the off-the-shelf checkpoints1. On CNN/DailyMail , apart from initializing JGR with the language models, we also evaluate JGR that initializes the generator using the previous state-ofthe-art model BRIO (Liu et al., 2022).2
During training, the generator model adopts a nucleus sampling approach to generate the candidate set with temperature = 1.0 and top(p) = 1.0. In inference, we apply beam search decoding strategy with beam size = 16, and length penalty =
1RoBERTa: https://huggingface.co/ roberta-large, BART: https://huggingface.co/ facebook/bart-large
2BRIO: https://github.com/yixinL7/BRIO
1.0 for the generator, and we take the output text with the highest beam search score as the final output of the generator. We use the ranker to re-rank the total 16 beam search results and pick the one with the highest ranking order as the final output of the ranker. The details of other hyper-parameters (e.g., learning rate and training epochs, etc) are listed in Appendix G. JGR is implemented based on the open-source Huggingface Transformers framework (Wolf et al., 2020). We conduct experiments on a single node of 8 NVIDIA A100 GPUs.
It is worth noting that in order to initialize the ranker with a more general and reasonable ranking function, we increase the number of training steps and add a certain number of warm-up steps at the first ranker training iteration. The hyperparameters of the first ranker training iteration are also introduced in Appendix G.

heading: 5 Results and Analyses

heading: 5.1 Overall Results
text: Table 1 shows the results of JGR and other baseline methods on summarization tasks CNN/DailyMail and SAMSum. “Lead-3” is an ad-hoc summarization approach that uses the first three sentences in the article as the summary. “PTGEN” and
“PTGEN-COV” are sequence-to-sequence generation methods without pre-training. Other baselines are pre-trained language models fine-tuned on the benchmarks. “JGR-G” indicates the generator model in JGR, and “JGR-R” is using the ranker of JGR to re-rank the outputs of JGR-G. “JGR-G/Rinit w. BRIO” are our JGR with the generator initialized from BRIO. As shown in Table 1, the generator model (JGR-G) itself achieves a considerable performance gain compared with its backbone models on both the two benchmarks, which verifies the effectiveness of the proposed JGR training to obtain a better generator. On both CNN/DailyMail and SAMSum, the ranker (JGR-R) can further improve the performance of JGR-G. Both JGR-G and JGR-R can reach state-of-the-art on SAMSum. If initialized with BRIO, both our JGR-G and JGR-R can surpass the state-of-the-art on CNN/DailyMail with a considerable margin.
In Table 2, we compare the performance of JGR with four pre-trained language models (Song et al., 2019; Lewis et al., 2019; Dong et al., 2019; Qi et al., 2020) on SQuAD 1.1, since they have reported the results finetuned and evaluated in the same data split as in Liu et al. (2020). With a relatively weak backbone model, BART, our JGR-G can still outperform all the compared baselines. And JGR-R can also further improve the results of JGR-G.
Table 3 shows the results of compared methods in persona-based response generation. As shown in the results, our JGR-G and JGR-R can surpass the baselines significantly on the metrics of BLEU-1 and BLEU-2. However, both JGR-G and JGR-R can only perform the same level of the baselines on Distinct-1 and Distinct-2. It is noteworthy that PLATO and DialogVED are the only two language models that are pre-trained using a conversational corpus among these baselines. They achieved high scores on Distinct-1 and Distinct-2, showing the importance of pre-training corpus.

heading: 5.2 Performance of Generate-then-Rank Frameworks
text: Recently, several works adopt the generate-thenrank framework, especially on the summarization tasks (Liu and Liu, 2021; Liu et al., 2021; Ravaut et al., 2022; Liu et al., 2022; An et al., 2022). Different from JGR, these methods do not introduce the iterative training of the generator and ranker. We compare these methods with that our JGR-R on CNN/DailyMail. Since all the above methods train
the ranker separately with the fine-tuned BART as the generator on CNN/DailyMail, we only report their results in this setting.
The experimental results are shown in Table 4, where G0 denotes the base generator, i.e. BART, and D0 is the ranker after the first ranker training iteration, as described in Section 4.2. Several observations can be seen in the results. First, our JGR achieves the highest score with the inference pipeline. Second, on CNN/DailyMail, the performance gain brought by JGR-R is not as big as other related methods which introduced some extra modules to their models. Third, on CNN/DailyMail, after the joint training in JGR, the performance gain brought by the ranker drops. We think this is because as the generator’s performance grows, the quality of candidates rises, causing the ranker harder to pick the best among all candidates.

heading: 5.3 Impact of Rewards
text: In this section, we investigate the impact of rewards. We compare different reward settings on CNN/DailyMail. The compared methods are as follows: 1) Self-critic is the conventional selfcritical reinforcement-learning method where the rewards are the metric scores ∆(ŷ,y), and the greedy search output is used as baseline (Rennie et al., 2017; Paulus et al., 2018). 2) Actor-critic is the RL-based method that trains a critical model to fit the metric scores ∆(ŷ,y), and uses the critical score as the reward to train generator (Konda and Tsitsiklis, 1999; Bahdanau et al., 2017; Le et al., 2022). 3) JGR-Gonly mr/JGR-Gonly rr are our JGR where the generator is trained without the rewards from generator/metrics. The standard NLL loss is added in all the compared methods. The results are shown in Table 5.
According to the results, our JGR can outperform traditional RL significantly. Both JGR-
Gonly mr and JGR-Gonly rr suffer a performance decline compared to standard JGR-G, and the performance of JGR-Gonly mr is far worse than that of JGR-Gonly rr. In addition, the Actor-critical method outperforms the Self-critical method. The above two observations indicate that using rewards from a trained reward model contributes more than using rewards from metrics, and it is better to combine them. In Figure 2, we plot the curves of the dev scores under 3 random runs for the compared methods. As illustrated in the figure, although the standard Self-critical method appears to have a small variance under different random runs, its dev scores are hard to grow while training. The JGR-Gonly rr has a smaller variance than JGR-Gonly mr, however, it fails to achieve a high dev score. Our standard JGR, which combines metric rewards and ranker rewards, not only shows the relatively small variance in randomized trials but also can steadily improve the dev score during training.

heading: 5.4 Candidate Picking Strategies
text: We examine how different types and numbers of candidates can affect the performance of JGR. We first compare different methods of picking positive candidates and negative candidates when training the ranker. The results are shown in Table 6. The ŷ+=GT denotes the positive candidate ŷ+ being always the reference, not the generated samples. The result shows that if the best candidate is always
the reference, the performance of the generator is not as good as the standard JGR, and the ranker’s performance is even worse than the generator. This is because the ranker is misled by the reference, thus it may always misclassify the references as the positive candidates, while other candidates sampled by the generator as the negative candidates. As a result, neither the ranker is well-trained, nor it can pass proper rewards to train the generator.
The last four lines of Table 6 show the results of methods for picking negative samples, i.e., with the lowest matching scores (BOT(Ŷ), our standard setting), with the highest matching scores (TOP(Ŷ)), randomly pick (RAND(Ŷ)), and half has the highest matching scores and the second half has the lowest matching score (TOP-BOT(Ŷ)). From the results, we can see that our standard setting (BOT(Ŷ)) significantly outperforms other negative candidate picking strategies.
In Table 7, we show the performance of JGR with different numbers of sampled candidates when training the generator. According to the results, under a certain range (C = 2 ∼ 8), the performance of JGR goes up as the number of candidates increases. We attribute this to the fact that increasing the number of candidates means that the generator can be optimized on more probabilities from candidates, which is to some extent a way of data augmentation. However, the performance does not grow as desired when the number of candidates becomes too large.

heading: 5.5 Advanced Metrics and Human Evaluation
text: A model trained with RL objective may succeed in the metrics it uses as the reward function but perform poorly in other metrics. We hope to in-
vestigate whether JGR, which uses the RL objective to train its generator, suffers from the same problem. Firstly, we use three advanced metrics, namely BERTScore (Zhang* et al., 2020), FactCC (Kryscinski et al., 2020), and QuestEval (Scialom et al., 2021a), to evaluate JGR on CNN/DailyMail. BERTScore measures the semantic similarity of the predicted summary and ground-truth reference. FactCC and QuestEval use a trained language model to measure the factual consistency between the generated summary and input source document. According to the results shown in Table 8, JGR-G and JGR-R both achieve higher BERTScore than BART, indicating that they can generate summaries with better semantic quality. For FactCC and QuestEval, which measure factual consistency, JGR-G and JGR-R also surpass the BART baseline.
We also conduct a human evaluation on CNN/DailyMail3. Following Blenderbot v2 (Roller et al., 2021), we randomly picked 100 cases from the CNN/DailyMail test set and asked the annotators to explicitly compare which generated text is better for each pair of summaries generated by JGR-G and BART, rather than assign an evaluation score. This explicit comparison can avoid the per annotator bias in numerical scores (e.g., annotators who tend to give generous scores), and remedy many of the issues of sequential effects such as contrasting with a previous case. Three aspects corresponding to the generation quality are evaluated, namely informativeness (Inform.), factual consistency (Fact.), and readability (Read.). As shown in Table 9, JGR-G beats BART in 58 cases w.r.t informativeness and 61 cases w.r.t. factual consistency, indicating that JGR-G performs better than
3More details about human evaluation are in Appendix D.
BART on informativeness and factual consistency. For readability, JGR can generate summaries as readable as BART.
To conclude, though trained with reinforcement learning aimed at optimizing ROUGE score, JGR still does not compromise on other aspects of summary quality, including semantic similarity, factual consistency, informativeness, and readability.

heading: 5.6 Does Joint Training Matter?
text: To see how our proposed joint (iterative) training of the generator and ranker affects JGR, we compare the performance of our JGR and the variant that trains the generator in the same reinforcement learning paradigm as the JGR while fixing the ranker after fully training it (JGRw/o joint)4. As the results shown in Table 10, JGRw/o joint is far worse than JGR, and JGR-Rw/o joint achieves no performance gain over JGR-Gw/o joint, which indicates the importance of the iterative training. To take an indepth look, we analyze the distribution of rewards. We first draw the curves of the Wasserstein distance between ranker rewards and metrics rewards at each training interval for JGR and JGRw/o joint. As illustrated in Figure 3(a), the Wasserstein distances of JGR are hovering within a range, while the Wasserstein distances of JGRw/o joint are growing extremely high, which means the distribution of ranker rewards and metrics rewards are quite different in JGRw/o joint. Therefore we think that JGR-Rw/o joint might not assign the proper rewards to the sampled candidates, due to it not being jointly trained.
4More details are given in Appendix E.
We also analyze the diversity of sampled candidates for JGR-G and JGR-Gw/o joint. We use selfBLEU5 to measure the diversity of sampled candidates. A larger self-BLEU score means a lower diversity of the sampled candidates. We show the curves of the average self-BLEU score for generated candidates at each training interval in Figure 3(b). From the figure, we can see that the selfBLEU of JGRw/o joint increases rapidly after the generator is trained 1000 steps, while the same situation never happens in JGR. It indicates that if the ranker is not jointly trained with the generator, the rewards it feeds back to the generator will cause the generator to sample candidates that are more and more similar to each other, making the training of JGR harder. On the contrary, joint training can erase this phenomenon and help to keep a certain level of diversity in sampled candidates, thus leading to better training.

heading: 5.7 More Discussions
text: Due to the page limit, we show more discussions about JGR compared to reinforcement learning, GAN, data augmentation in Appendix A, the impact of decoding strategies in Appendix C.

heading: 6 Conclusion
text: In this paper, we propose a novel Joint training of Generator and Ranker framework, namely JGR, for natural language generation. Both the generator and ranker of our JGR can achieve state-of-the-art results on several benchmarks in the areas of summarization, question generation, and dialog. We also analyze our JGR in several aspects and find that: First, the rewards from the ranker work better than the rewards from the direct metrics such as BLEU, but combining them together helps the training become more stable. Second, during training, letting the ranker be trained on the candidates generated by the generator exclusively is even better than previous approaches using ground-truth as positive examples. Third, more candidates being sampled during training can lead to better performance, which is consistent with the findings from data augmentation. Fourth, though trained with reinforcement learning aimed at optimizing automatic evaluation metrics, JGR still does not compromise on other aspects of generation quality. Finally, the joint training paradigm helps the
5We introduce the computation of self-BLEU in Appendix B.
generator sample candidates with higher diversity, which in turn contribute to the training.

heading: Acknowledgements
text: Weizhou Shen and Xiaojun Quan were supported by the National Natural Science Foundation of China (No. 62176270), the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515012832), and the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (No. 2017ZT07X355).
Limitations
So far JGR has only been evaluated on the domains of summarization, conversational summarization, question generation, and dialog. It should be evaluated on a wider range of benchmarks, such as machine translation and code generation. And we have not explored JGR’s performance with extralarge language models such as GPT-3. We will evaluate JGR on the above points in the future.
Because the generator of JGR samples candidates using auto regressive sampling, it may occupy relatively longer computational time and larger memory then the conventional MLE training. Though the performance of JGR is satisfactory, we still want to improve its computational costs. We will try non-auto regressive sampling and other improvements such as parameter sharing in the future.
Ethics Statement
All the experiments are conducted on publicly available datasets, which don’t include any private information. Our work doesn’t involve identity characteristics or any gender and racial discrimination.

heading: A Discussion
text: In this section, we discuss the relations between our JGR and several popular methods, including reinforcement learning (RL), generative adversarial networks (GAN), and data augmentation.
A.1 JGR & RL
Some previous RL works, i.e., (Shen et al., 2015; Rennie et al., 2017; Paulus et al., 2018) proposed to use ∆(ŷ,y) to compute reward R(ŷ) directly which doesn’t combine ranking scores as feedback signals. However, we argue that the ranking score calculated by the ranker model can provide more semantic-relevant information than the matching scores (e.g., BLEU or ROUGE) which are purely based on the surface match. In the ablation study, we also demonstrate that the proposed approach is
superior to other configurations in terms of training stability and performance.
Some other RL works Bahdanau et al. (2017); Le et al. (2022) introduced actor-critic frameworks (Konda and Tsitsiklis, 1999), which jointly train an actor and a critic, are similar to our JGR framework. However, they have not considered the contrastive rewards between different candidates given one input. Different from these works, JGR allows the generator to sample several i.i.d. candidates and be optimized simultaneously on these candidates at each training step. This improvement makes the reward of a sampled candidate contain contrastive information from the candidates from the same candidate set. Furthermore, it effectively raises the number of diverse chains of probabilities on which the generator can be optimized. In Table 5, we compare our JGR-G with the simple self-critical that uses metric rewards, and the actorcritic baseline that the critic is trained to fit the metric score ∆(ŷ,y). The empirical results show that trained with the JGR framework, the generator model can surpass those trained with previous RL-based methods well used in the NLG area.
A.2 JGR & GAN
From the perspective of the composition of a framework, both JGR and GAN contain a generator and a critic. In GAN, the critic is the discriminator that aims at discriminating the real candidate from the candidate pool. While in JGR, the critic is the ranker that aims to re-rank the candidates generated by the generator.
The main difference between JGR and GAN comes from the training objective. Let the Gθ denotes the generator, and Dϕ denotes the discriminator/ranker. GAN trains Gθ and Dϕ with the min-max objective:
JGθDϕ = minθmaxϕEy+∼ptrue(·|x)[logpDϕ(y+, x)] +Eŷ−∼pGθ (·|x)[log(1− pDϕ(ŷ
−, x))] (10)
In Eq. 10, y+ is the ground-truth output of input x, and ŷ− is the candidate texts sampled by the generator. This is different from the setting of JGR, where both y+ (denoted as ŷ+ in JGR) and ŷ− are sampled from pGθ(·|x).
To implement GAN in NLG, according to (Yu et al., 2017), the policy gradient is used and the reward assigned to ŷ− is logpDϕ(ŷ −, x). Note that
the reward is always positive, therefore GAN essentially raises the probability of the generator outputs, regardless of the quality of the outputs. On contrary, as computed in Eq. 8, there are both positive and negative rewards in JGR, which means that JGR not only encourages the generator to generate good candidates but also punishes the generator when generating bad candidates.
Table 11 shows the performance of generators in JGR and GAN on CNN/DailyMail, where GANstd is the standard GAN setting that y+ is the groundtruth text and GANmod is our modified version of GAN that y+ is replaced by the best candidate sampled by the generator, i.e., ŷ+. As shown in the table, our JGR surpasses the GAN methods, and the performance of GANstd and GANmod can not even surpass the model trained on optimizing the standard NLL loss, indicating that the GAN methods are not suitable for all NLG tasks. The GANmod performs worse than GANstd, showing that for the min-max objective of GAN, it is not a good choice to letting ŷ+ as the positive sample, which is contrary to what we found in JGR.
A.3 JGR & Data Augmentation
Data augmentation methods aim to improve the models’ performance by adding modified or synthesized data to the existing training data (Li et al., 2022a). For natural language generation tasks, denote the augmented dataset as D̂, where D̂ contains several augmented samples (x̂, ŷ), the training object for model in the augmented data is:
LDA = − ∑
(x̂,ŷ)∈D̂
∑
t
log pGθ(ŷt|ŷ<t, x̂) (11)
The above equation is similar to JGR’s reinforcement learning loss in Eq. 8. Both of them optimize the generator by maximizing the log-likelihood of synthesized data. Therefore, from this perspective, we can regard our JGR as a way of data augmentation where the synthesized data is sampled from
the generator and the log-likelihood is re-scaled by the rewards.
We designed two simple but effective data augmentation methods named DAsep and DAmix. Both of DAsep and DAmix use a fine-tuned generator G0 to generate one summary ŷ for each input x in original training set D using beam search, the collection of all (x, ŷ) is treat as the augmented training data D̂. After that, 1) DAsep fine-tunes G0 firstly on D̂ and then on D, 2) DAmix further fine-tunes G0 on the mixture of D̂ and D. We compare the performance of DAsep and DAmix with our JGR on CNN/DailyMail, with BART as the generator, the results are shown in Table 12. As shown in the results, both DAsep and DAmix can further improve the performance of BART, verifying the effect of data augmentation. However, the performance gain brought by data augmentation is far less than that brought by JGR.

heading: B Computation of Self-BLEU
text: Given a candidate set Ŷ = {ŷ1, ŷ2, ..., ŷC} sampled from the generator, the self-BLEU score for Ŷ is computed as the average of mutual BLEU scores of all candidate pairs:
self-BLEU(Ŷ) =
∑ ŷi,ŷj∈Ŷ;i ̸=j BLEU(ŷi, ŷj)
C(C − 1) (12)
A higher self-BLEU score means the sampled candidates are more similar to each other, in other words, a lower diversity of the sampled candidates.
It is another way to assess the diversity of sampled candidates by computing the proportion of the number of distinct n-grams in the total number of tokens for the sampled candidates of an input sequence. We refer to this metric as selfDistinct-n where n refers to n-grams. The higher self-Distinct-n corresponds to the higher diversity of sampled candidates. Like Figure 3(b), we show
the curves of the average self-Distinct-2 for generated candidates at each training interval in Figure 4. From the figure, we can see that the self-Distinct-2 of JGRw/o joint drops rapidly after the generator is trained 1000 steps, while the self-Distinct-2 keeps hovering in a relatively high range for JGR. This phenomenon aligns with what we found when applying self-BLEU and further enhances our conclusion in Section 5.6.

heading: C Decoding Strategies
text: We study the impact of different decoding strategies during inference. Two decoding strategies are compared, namely beam search and group beam search (Vijayakumar et al., 2016). We also compare different beam sizes. The results of ROUGE-1 score with beam search on CNN/DailyMail are shown in Figure 5.
As shown in Figure 5, increasing the beam size does not contribute to the performance of JGR-G
when using the normal beam search. However, the performance of JGR-R can rise as the beam size increases. This indicates that increasing the beam size can raise the probability of JGR-R ranking a better candidate to the top among all the candidates decoded by JGR-G.
Figure 6 shows the results with diverse beam search. Firstly we can find that with diverse beam search the JGR system can not achieve comparable results with JGR using normal beam search, and the performance of JGR-G begins to drop when beam size exceeds 4. We can still observe that the performance of JGR-R rises as the beam size increases. However, since the performance of JGRG keeps declining, the performance ascent of JGRR is not as significant as that of JGR-R with the normal beam search.

heading: D Details of Human Evaluation
text: We conduct a human evaluation on CNN/DailyMail. Following Blenerbot v2 (Roller et al., 2021), we ask the annotators to explicitly compare which generated text is better for each pair of generated outputs, rather than assign an evaluation score. This explicit comparison can avoid the per annotator bias in numerical scores (e.g., annotators who tend to give generous scores), and remedy many of the issues of sequential effects such as contrasting with a previous case. We randomly picked 100 cases from the CNN/DailyMail test set, each case was organized as <Doc, Summary #1, Summary #2> where Doc means the source document, Summary #1 and Summary #2 mean the summaries generated by JGR and BART.
The annotators were asked to compare Summary #1 and Summary #2 on three aspects given at the end of each case. To avoid the stereotype of annotators that Summary #1 or Summary #2 is better according to previous cases, we randomly shuffle the summaries in each case, which means that Summary #1 is not necessarily from JGR or BART, and so as Summary #2.
Each picked case was annotated by 3 annotators, and they worked individually without communication. Given a certain human evaluation metric on one case, the comparison result is obtained by the following rules:
• If more than or equal to two annotators think JGR has won in that metric, then JGR wins.
• If more than or equal to two annotators think BART has won in that metric, then BART wins.
• Otherwise, the comparison result is marked as a tie.
We evaluate JGR and BART from three aspects, namely informativeness (Inform.), factual consistency (Fact.), and readability (Read.). The results are shown in Table 9. Note that since we use direct comparison, the number of “tie” cases may be fewer than some works that conduct human evaluation through assigning scores.

heading: E Details of JGRw/o joint
text: To implement JGRw/o joint, we first fully train the generator with the negative likelihood loss. Then we use this generator to generate candidates and fully train the ranker with the objective described in Eq. 6. Then we train the generator again using the same RL paradigm as JGR with the reward from the ranker. The only different between JGRw/o joint and JGR is that JGRw/o joint does not incorporate the iterative training.

heading: F Details of the Benchmarks and Evaluation Metrics
text: CNN/DailyMail (Hermann et al., 2015) is a benchmark for summarization. Both extractive and abstractive summarization models can be applied on CNN/DailyMail. Since our JGR focuses on text generation, we treat CNN/DailyMail as an abstractive summarization task. There are two versions:
anonymized and non-anonymized. We use the nonanonymized dataset See et al. (2017). The evaluation metrics are Rouge-1, Rouge-2, and Rouge-L.
SAMSum (Gliwa et al., 2019) is a benchmark for conversational summarization, whose inputs are the concatenation of dialog context. The evaluation metrics are Rouge-1, Rouge-2, and Rouge-L.
SQuAD 1.1 (Rajpurkar et al., 2016) is originally an machine reading comprehension dataset. We follow the data split and pre-processing as done by Du et al. (2017); Zhao et al. (2018); Liu et al. (2020), to make it a question generation dataset, which treats the concatenation of the answer span and article as the input, and the question as the target output. The evaluation metrics are Rouge-L, Bleu-4, and METEOR.
PersonaChat (Zhang et al., 2018) contains about 160K utterances. Given the multi-turn conversations and persona profile, the model learns to generate the response. The evaluation metrics are Bleu-1, Bleu-2, and the ratio of distinct unigrams and bigrams in the generated responses (Distinct-1 and Distinct-2).
The statistics of all benchmarks are shown in Table 13.
For evaluation on CNN/Daily and SAMSum, we use the python rouge score package: https://pypi.org/project/rouge-score/. For evaluation on SQuAD 1.1, we follow the evaluation scripts open-sourced by Liu et al. (2020) at https://github.com/microsoft/ ProphetNet/tree/master/GLGE_baselines/ script/script/evaluate/qg. For evaluation on PersonaChat, we follow the evaluation scripts open-sourced by Liu et al. (2020) at https://github.com/microsoft/ ProphetNet/tree/master/GLGE_baselines/ script/script/evaluate/personachat.

heading: G Hyper-parameters of Fine-tuning on Benchmarks.
text: The hyper-parameters for our JGR on each benchmark are shown in Table 14.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 and Appendix F

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 and Appendix F

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix F

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 and Appendix F
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We do not discuss them, but cite the original papers of these artifacts.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix F
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix F
C 3 Did you run computational experiments? Section 4 and Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.2 and Appendix G
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.2 and Appendix G
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We follow the previous works and only report the metrics scores

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4. and Appendix F
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.4 and Appendix E
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We have not reported the full text of instructions in the paper, but we provided them in our anonymous open-source link https://anonymous.4open.science/r/jgr-anonymous-F597.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? This is the business behavior at the company level.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We have not discussed this in the paper, but we provided it in our anonymous open-source link https://anonymous.4open.science/r/jgr-anonymous-F597.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. We did not provide data but conducted a human evaluation.

heading: Future Work
text: We will evaluate JGR on the above points in the future. Because the generator of JGR samples candidates using auto regressive sampling, it may occupy relatively longer computational time and larger memory then the conventional MLE training. Though the performance of JGR is satisfactory, we still want to improve its computational costs. We will try non-auto regressive sampling and other improvements such as parameter sharing in the future