Year: 2021

Abstract:
Natural language modeling with limited training data is challenging problem, and many algorithms make use of large-scale pretrained language models (PLMs) for this due to its great generalization ability. Among these transfer learning algorithms from PLMs, additive learning that incorporates a task-specific adapter on top of the fixed PLM has been popularly used to alleviate the severe overfitting problem in the few-shot setting. However, this added task-specific adapter is generally trained by maximum likelihood estimation that can easily suffer from the so-called exposure bias problem, especially in sequential text generation. Therefore, in this work, we develop a novel additive learning algorithm based on reinforcement learning (RL) for few-shot natural language generation (NLG) tasks. In particular, we propose to use a selective token generation between the transformerbased PLM and the task-specific adapter during both training and inference. This output token selection between the two generators allows the adapter to take into account only on the task-relevant parts in sequence generation, and therefore makes it more robust to overfitting as well as more stable in RL training. In addition, in order to obtain the complementary adapter from the PLM for each few-shot task, we exploit a separate selecting module that is also simultaneously trained using RL. Experimental results on various few-shot NLG tasks including data-to-text generation and text summarization demonstrate that the proposed selective token generation significantly outperforms the previous additive learning algorithms based on the PLMs.

heading: 1 INTRODUCTION
text: Natural language processing (NLP) have recently achieved great progress using advanced neural language models (Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Clark et al., 2020; Raffel et al., 2019; Lan et al., 2019; Lewis et al., 2020). However, these neural models typically require large-scale training data for each individual task, and solving a new NLP task that has only a few examples is still challenging problem (Yin, 2020). Especially, natural language generation (NLG) with limited training data is an important yet more difficult task due to its fast adaptation of sequential prediction models in a wide range of applications including text summarization, question answering, data-to-text generation, machine translation, etc (Peng et al., 2020; Chen et al., 2020; Xu et al., 2021; Schick & Schütze, 2020; Chang et al., 2021; Radford et al., 2019; Lewis et al., 2020; Brown et al., 2020).
More recently, pretrained language models (PLMs) have shown great generalization ability when combined with large-scale data and big transformer-based models (Devlin et al., 2019; Radford et al., 2019; Lewis et al., 2020; Brown et al., 2020; Subramanyam Kalyan et al., 2021). Therefore, transfer learning from transformer PLMs has been popularly used for few-shot NLG tasks with promising results. In specific, the use of PLM for few-shot NLG can be categorized into three approaches: 1) prompt-based, 2) finetuning, and 3) additive learning. Prompt-based approaches encode a task description and task-specific examples as a natural language prompt for few-shot text generation (Radford et al., 2019; Brown et al., 2020; Zheng & Huang, 2021; Schick & Schütze, 2020; Li & Liang, 2021). While these approaches can take full advantage of the universal natural language understanding and generation capabilities of large-scale PLMs without further training of the main model, these have some limitations in dealing with a large domain shift from the pretraining corpus data, tuning suitable task-specific prompts, and covering an increased size of conditioning examples.
On the other hand, finetuning of the PLM is able to explicitly impart task-specific knowledge to the model and hence lift the above limitations (Ziegler et al., 2019; Xu et al., 2021; Chen et al., 2020). However, these finetuned models are prone to overfitting when only a small amount of training data is available. In order to alleviate such an overfitting problem, additive learning has been extensively exploited by incorporating task-specific adapters into the PLM (Zeldes et al., 2020; Stickland & Murray, 2019).
In general, task-specialized adapters for few-shot NLG are trained by maximum likelihood estimation (MLE). While MLE is efficient in learning, it suffers from the exposure bias problem due to the difference in the training and inference mechanisms (He et al., 2019), and this problem can be severe with limited training data. Reinforcement learning (RL) is capable of resolving this exposure bias problem by sequential output sampling during training (Ranzato et al., 2015; Keneshloo et al., 2019; Shi et al., 2021). Moreover, it allows to leverage the target-specific sequence-level objectives such as BLEU and ROUGE (Wu et al., 2018; Guo et al., 2021). However, in the task of NLG, the exponentially large space of output sequences restricts the use of RL since it leads to high variance and unstable training which is more serious in the few-shot setting. In this work, we develop a novel RLbased additive learning algorithm on the transformer-based PLM to overcome these shortcomings and to improve the performance of few-shot NLG.
In particular, we first convert the NLG task to the sequential token generation task based on the transformer language model, and then propose a selective token generation between the PLM and the task-specific adapter, during both RL-based training and inference. The proposed output token selection enables to not only explicitly maintain a general prior knowledge from the frozen PLM but also focus only on the task-relevant parts in sequence generation. In addition, in few-shot learning this partial token generation makes the task-specific adapter more resilient to overfitting and furthermore reduces the overall output space which leads to stable RL training. Here, in order to make the two token generators (policies) complement each other as well as to realize the robust output selection at the token level on the fly, we exploit a separate token-level policy selector. It is noted that both the policy selector and the task-specific adapter are simultaneously learned by the RL algorithm. Experimental results on various few-shot NLG tasks show that the proposed selective token generation outperforms the previous PLM-based additive learning algorithms with the comprehensive (non-selective) token generation.
Our main contributions can be summarized as follows.
• A novel selective token generation between the PLM and the task-specific adapter is proposed for transformer-based few-shot NLG.
• A separate selecting module is exploited to adaptively determine each output token in a sequence both at training and testing time.
• RL is applied to train both the policy selector and the task-specific adapter that is complementary to the PLM in text generation.
• An extensive empirical validation on few-shot NLG tasks demonstrates that the proposed selective token generation performs better in comparison to the previous PLM-based additive learning algorithms.

heading: 2 BACKGROUND

heading: 2.1 NATURAL LANGUAGE GENERATION
text: The goal of NLG is to generate a text sequence y = [y0, ..., yT ] for a given task, where yt is the tth output token from a vocabulary V , and T is the output sequence length. For this generation, we aims to model the distribution of y that is autoregressively factorized as pθ(y) = ∏T t=0 pθ(yt|y<t), where θ denotes the model parameters and y<t = [y0, ..., yt−1]. Here, the conditional distribution to sample a token for each step, pθ(yt|y<t), is defined by the softmax function on the output logits fθ(yt|y<t). Note that in general, the language generation is conditioned on input context according to a given task. Here, we encode the conditioning context by the same sequential model for generating an output sequence, and for simplicity we omit it. In this work, we utilize the autoregressive transformer for our generative model.

heading: 2.2 ADDITIVE LEARNING FOR FEW-SHOT GENERATION
text: To effectively leverage the general linguistic knowledge, θ is first initialized by the PLM parameters, θLM , for NLG. Given N task-specific training instances, D = {yn∗}Nn=1, where yn∗ is the nth ground-truth output sequence, directly finetuning θLM usingD can incur the severe overfitting problem when N is small in the few-shot scenario. Therefore, we add the task-specific adapter, gθa parameterized by θa, on top of the PLM, and optimize only θa (Zeldes et al., 2020; Stickland & Murray, 2019). In specific, we reformulate f(·|y<t; θ) = WTh(y<t; θh) where W ∈ RH×|V| and h ∈ RH denote the weight matrix and the penultimate representations, respectively, and θ = {W, θh}. Then, we define the task-specific conditional distribution as follows:
p(yt|y<t; θLM , θa) = softmax ( WLM ThLM (y<t) +Wa T g ( hLM (y<t); θg )) , (1)
where hLM (y<t) = h(y<t; θh,LM ) and θa = {Wa, θg}. Here, the summation of the PLM logits and the adapter logits is motivated by auxiliary training1. It is noted that in our additive learning θa is updated while θLM is kept frozen. Hence, in the following we omit θLM such that pθa(yt|y<t) = p(yt|y<t; θLM , θa) for simplicity.

heading: 2.3 MAXIMUM LIKELIHOOD ESTIMATION (MLE)
text: Given a small amount of training data D = {yn∗}Nn=1, MLE optimizes θ by maximizing the data log-likelihood as follows:
θ̂ = argmax θ N∑ n=1 T∑ t=0 log pθ(y n∗ t |yn∗<t). (2)
Here, the output token at each step is conditioned on not the previous sampled tokens from the current model but the previous ground-truth tokens yn∗<t. Namely, tokens are drawn from the data distribution during training, which is opposed to that tokens are drawn from the model distribution at test time. This discrepancy, also known as the exposure bias, leads that the errors will be accumulated along the generated sequence at test time since the model is biased to only perform well on the ground-truth history distribution. Especially, this bias problem can be more severe in the few-shot training. In addition, the token-level cross-entropy loss in MLE training is different from the sequence-level test metrics such as BLEU and ROUGE that are commonly used in the tasks of NLG.

heading: 2.4 REINFORCEMENT LEARNING (RL)
text: As an alternative to MLE, RL is able to overcome the exposure bias problem of MLE by sequencelevel sampling from the model distribution during training (Ranzato et al., 2015). Also, RL can improve the performance by directly optimizing the evaluation metrics (Guo et al., 2021). In order to use RL for our additive learning, we reformulate our text generation as an RL problem: at each time step t, the agent takes the current state st = y<t as an input and performs an action at that outputs a token yt by a policy πθ(at|st) corresponding to pθ(yt|y<t). Then, the agent receives a reward rt = r(st, at) and deterministically transitions to the next state st+1. Here, note that the token-level intermediate reward rt = 0,∀t < T when we use the delayed reward associated with the sequencelevel evaluation metric between the two full sequences, y and y∗. Let τ = {(st, at, rt)}Tt=0 be the trajectory generated by πθ. The RL objective for the optimal agent is to maximize the expected sum of future discounted rewards:
J(πθ) = Eτ∼πθ [ T∑ t=0 γtrt ] , (3)
where γ ∈ [0, 1) is the discount factor. 1Although the auxiliary training is particularly designed for maximizing the likelihood of the target task output, it also can take an advantage for RL since the adapter logits are nearly zero before training is advanced. Namely, it lets the task-specific conditional distribution start learning from the distribution of PLM, not a uniform distribution.
Among a number of algorithms to approximately optimize the RL objective, we employ an actorcritic algorithm (Bahdanau et al., 2017) since it explicitly optimizes the policy network and it can also alleviate the delayed reward problem. The actor-critic algorithm requires the additional critic network to estimate the value of a state, V π(st) = Eπ[ ∑T t′=t γ t′−trt′ |st] = ∑ at π(at|st)Qπ(st, at)
where the state-action value function Qπ(st, at) = Eπ[ ∑T t′=t γ
t′−trt′ |st, at] = rt + V π(st+1). We use the following policy gradient loss to learn the policy parameters θ:
L = − T∑ t=0 Aπθ (st, at) log πθ(at|st), (4)
where Aπθ (st, at) = Qπθ (st, at)− V πθ (st) is the advantage function that quantifies how an action at is better than the average action in state st.
In few-shot text generation, the extremely large action space (|V|T ) as well as the small amount of training data often make it difficult to perform RL with degraded performances, even though we conduct the additive learning from the PLM. Furthermore, it commonly has a delayed reward function (e.g. BLEU) which is defined after an entire sequence generated. It is hard to decide which token and how much contributes to the reward. This problem is known as the credit assignment. Therefore, in this work, we propose a selective token generation for improving the RL-based additive learning.

heading: 3 SELECTIVE TOKEN GENERATION
text: Instead of generating all tokens in an output sequence from the single task-specific policy, πa = πθa(at|st), at each time step t, we sample an output token yt selectively from either the PLM policy πLM = πθLM (at|st) or the task-specific policy πa:
yt = at ∼ ( 1t[πLM is selected]πLM (at|st) + (1− 1t[πLM is selected])πa(at|st) ) , (5)
where 1t[·] is the indicator function (at t) that equals 1 if it is true and 0 otherwise. This output token selection allows to explicitly utilize a general linguistic knowledge from the PLM without catastrophic forgetting in few-shot learning. Also, the task-specific policy can focus on generating only the task-relevant parts, which enables more effective few-shot training with a reduced action space.
Now we need to determine how to select the proper policy at each step on the fly as well as to make the task-specific policy complementary to the PLM policy. For this, we exploit a separate
token-level policy selector. The proposed policy selector πs(it|st; θs) with the parameters θs, where it ∈ {0, 1}, is an another policy that stochastically decides a policy to generate at for st. Namely, a token sample yt is generated by the following process:
it ∼ πs(it|st), (6)
yt = { at ∼ πLM (at|st) if it = 0, at ∼ πa(at|st) if it = 1.
(7)
This process can be considered as a token generation from a hierarchical policy πh(at|st; θs, θLM , θa) where the policy selector represents the upper-level prior for the preference of the low-level policy. Therefore, the value function of this hierarchical policy can be formulated as
V πh(st) = Eπh [ T∑ t′=t γt ′−trt′ |st] = πs(it = 0|st) ∑ at πLM (at|st)Qπh(st, at)
+ πs(it = 1|st) ∑ at πa(at|st)Qπh(st, at), (8)
and Aπh(st, at) = Qπh(st, at) − V πh(st). Here, it is noted that a single critic network is used for the hierarchical policy since it does not affect st. Given a sample trajectory {(st, it, at, rt)}Tt=0, the loss for optimizing θs and θa is
L = − T∑ t=0 Aπh(st, at) [ 1[it = 0] ( log sg[πLM (at|st)] + log πs(it|st) ) + 1[it = 1] ( log πa(at|st) + log πs(it|st) )] , (9)
where sg stands for the stop-gradient operator. Similar to πa, πs makes use of the PLM representations and the task-specific adapter such that
πs(it|st; θs) = softmax ( m ( g ( hLM (st) ) ; θs )) , (10)
where m is the selector module that is implemented by a small neural network. Figure 1 depicts the overall text generation process by the proposed selective token generation (STG) in comparison to the previous non-selective token generation (Non-STG). Here, note that since all policies in STG share the same PLM representations, the increased computational cost by STG over Non-STG is negligible.
The use of the separated policy selector that is simultaneously trained with the task-specific policy allows the task-specific policy to be complementary to the PLM policy. Especially, this cooperative ensemble learning can be realized by our RL algorithm that performs sequential sampling from the model during training. In addition, the proposed STG model makes use of the PLM not at the feature level but the output distribution level in text generation. In our few-shot learning this is beneficial in explicitly retaining strong linguistic and world knowledge from the PLM.

heading: 4 EXPERIMENTS
text: In this section, we evaluate our method against baselines on Data-to-Text, Question Answering and Text Summarization tasks which are widely used in few-shot NLG. Since we claim the effectiveness of our method in additive learning, we compare our results against those obtained by other additive learning methods.

heading: 4.1 BASELINE
text: PLM. In our experiments, we assume that the PLM works to some extent for a given task. However, the naive PLM usually does not satisfy it for a new task unseen during training. Hence, we finetuned
GPT-22 (Radford et al., 2019) with MLE for few epochs and used it as the PLM. Fine-tuning the PLM with MLE is most commonly used for task adaptation and thus it can also be a strong baseline. This fine-tuning phase accelerates the learning of the adapter. This is particularly when the adaptation requires to cover the large domain shift. Severe performance degradation was observed for all the tasks when we skipped the fine-tuning.
Non-STG. This method stands for Non-Selective Token Generation which uses the above the PLM as an encoder (frozen) and the adapter (additional layer to be trained). We use two objectives, MLE and RL, for additive learning. These will be denoted as Non-STG-MLE and Non-STG-RL, respectively.
STG-Naive Ensemble. We believe that our proposed method encourages the task-specific policy (πa) to complement the PLM’s policy (πLM ) with a proper selection of the selector through joint training. To investigate this, we evaluate against two different naive ensembles of the policies, πa trained from Non-STG and πLM of the PLM. These ensemble schemes are as follows:
• NE(max): πmax = softmax(Max(πa, πLM )) • NE(mix): πmix = (πa + πLM )/2
We also evaluate another naive ensemble strategy NE(random) that randomly selects a token policy at each step between πa and πLM , however it shows lower performances than the others.

heading: 4.2 IMPLEMENTATION
text: Here, we explain the implementation set-up of our method.
Additional Layer: The task-specific adapter g in Section 2.2 is implemented by a LSTM to encode the dynamics of the representation vector hLM . We found that the use of MLP was not performed well.
Selector. We use a 2-layer MLP with ReLU activation for m.
Reinforcement Learning. We employ Actor-Critic method (Konda & Tsitsiklis, 2000; Fedus et al., 2018) for RL. The agents (i.e. selector and generator) receive a reward after generating a sentence. Here, we use different reward functions according to tasks. We use delexicalised BLEU for Data-toText following Peng et al. (2020), Averaged score of BLEU and ROUGE-L for Question Answering and ROUGE-L for Text Summarization following Paulus et al. (2017) as the reward function.
Token Sampling. During the training, it ∈ {0, 1} ∼ πs is first sampled, and then we use either πLM of the PLM for it = 0 or the task-specific policy πa for it = 1 to sample the tth token. During the evaluation, any decoding strategy, such as a beam search, can be used with the mixture of policies πh(·) = πs(it = 0)πLM (·) + πs(it = 1)πa(·). We use the beam search decoding with a sample size of k = 3 for Text Summarization and topp = 0.9 decoding for both Data-to-Text (k = 10) and Question Answering (k = 3).
2We make use of GPT-2 with 345M parameters as the initial checkpoint. We follow the training details in the previous works (Peng et al., 2020; Khandelwal et al., 2019) for each task.

heading: 4.3 DATA-TO-TEXT
text: Data-to-Text is a task that transforms structured data such as graphs or tables into natural language. Recent works (Mager et al., 2020; Peng et al., 2020; Kale, 2020) show that the PLM can be adapted successfully to this task by taking a serialized form of data as an input without a carefully designed model to encode the structured data. Here, we perform experiments on FewShotWOZ (Peng et al., 2020) dataset. The evaluation is conducted only on the topics of Restaurant, Hotel, Laptop, and TV since the evaluator has not published yet for the other topics (Taxi, Attraction, Train)3. There are 50 training instances for each topic and 129, 78, 1379, and 680 testing instances for Restaurant, Hotel, Laptop, and TV, respectively. The models are evaluated by measuring fluency and informativeness using BLEU score and ERR (slot ERror Rate), respectively. Table 1 shows the obtained results.

heading: 4.4 LONG ANSWER QUESTION ANSWERING
text: We consider Long Answer QA task on MS-MARCO (Nguyen et al., 2016) dataset. In this task, a passage and a query are given, and the model generates an answer with respect to the query by referring to the passage. Here, we randomly sample various sizes of (0.5% ≈ 500, 1%, and 2%) subset data from the train dataset. We also sample a validation and a test set, which contains 500 and 12,000 instances, respectively, from the dev dataset. We repeat this test three times with different random seeds. In other words, we collect three different random subsets for each size of few-shot. Thus, we perform experiments on total nine subsets. The models are evaluated by measuring BLEU, ROUGE-L (denoted as RL), and their average value (denoted as Avg). The performances of the models are reported in Table 2 and Appendix A.

heading: 4.5 TEXT SUMMARIZATION
text: We consider the problem of abstractive summarization: models aim to generate its summary (the “target” text) for a given piece of “source” text such that its meaning is intact. The model has to learn 1) building a semantic form internally from the source text and 2) converting the semantic form into a text. Here, we randomly sample various sizes of (0.5% ≈ 1, 500, 1%, and 2%) subset data from CNN/Daily Mail (See et al., 2017). We repeat this test three times for each size of fewshot as in above QA task. ROUGE (Lin, 2004) is commonly used to evaluate n-grams recall of the summaries with gold references. ROUGE-1, ROUGE-2, and ROUGE-L (measured based on the longest common sub-sequence) scores are reported (denoted as R1, R2, and RL, respectively) in Table 3 and Appendix B.

heading: 4.6 RESULT
text: We compare experimental results on various tasks in this subsection. In most cases, additive learning improves the performances over the PLM. However, they do not always guarantee a performance
3https://github.com/pengbaolin/SC-GPT
improvement. For example, the ERR score of the PLM on Laptop shows a better result except for STG and NE(mix)-RL (see Table 1) and the Non-STG models trained on 1% few-shot subset of MS-MARCO do not outperform the PLM (see Table 2 and Table 5 of Appendix A).
In Data-to-Text task, as shown in Table 1, we can observe that the Non-STG models do not outperform the PLM even though it has more neural units and takes more training time. The models trained on the RL objective show better performances for the ERR (lower is better). Interestingly, NE(mix) methods show strong improvements for the BLEU which measures the fluency of sentence but obvious degeneration for the ERR which measures the rate of missing information from the given data. These results suggest that the PLM is much more capable of task-general knowledge than the task-specific generator (i.e. πa) trained on few-shot dataset. It motivates us to jointly learn the policy selector and the task-specific generator. Here, while other methods show some trade-off between BLEU and ERR, only STG shows improvements on the both metrics for all topics in the dataset.
In Question Answering, NE(mix) and STG show significantly better performances than the other methods as shown in Table 2 and Appendix A. Notably, NE(mix) show good performances as much as STG. It obviously suggests that the PLM can be a complementary model to the additional model. Therefore, in this context, it can be lost of the prior knowledge of the PLM even the additional model has been built over the feature space of the PLM. We will discuss it in Section 4.7.
In Text Summarization (a problem of long-sequence generation), STG shows significantly larger gains than Non-STG-MLE, Non-STG-RL, and their naive ensembles with the PLM in every score metric and training data size as shown in Table 3 and Appendix B.

heading: 4.7 WHAT MAKES STG BETTER?
text: Knowledge Preservation. Although freezing the parameters of the PLM can mitigate the forgetting problem, it would be failed to do so when additional units having sufficient capacity is used since it is easy to memorize typical patterns of answers in small amount of training dataset and can degrade the generalization performance. An example that represent this issue is shown in Table 4 of Appendix A. In this case, a passage that contains two definitions (super-scripted and bolded) about conflict is given with a query which asks the psychological meaning of conflict. Without the knowledge of who Colman4 is, it can be a hard to answer since the word psychology is not appeared in the passage. Here, the PLM repeats the given query due to the imperfect domain adaptation. Non-STG models, both Non-STG-MLE and RL, generate the same answer, that is not the psychological meaning but the general meaning of conflict, with pretty low perplexity (≈ 0.35). The reason in producing this incorrect answer is that most queries in few-shot training data ask a general meaning of a concept, and Non-STG models are overfitted to this pattern. On the other hand, the answer of STG, which is close to the ground truth, is generated by the PLM policy πLM after some sequence of tokens (conflict is) that are sampled from the task-specific policy πa.
4A psychologist, https://en.wikipedia.org/wiki/Peter_T._Coleman_(academic)
We can find such examples for the other tasks in Appendix F: In Data-To-Text, as shown in the last example of Table 10, Non-STG generates nicam stereo which is not appeared in the given data. This is due to that nicam stereo was appeared 7 times (7/50, 14%) in training data. In Summarization, as shown in the first example of Table 14, Non-STG models only consider the forepart of the given article. Since the most of the major information is appeared in the forepart in News data, Non-STG models can be easily overfitted to generate the text according to such a pattern. Hence, we claim that Non-STG easily exposed learning patterns of typical answering and STG resolves this issue since it can be fully accessible to the knowledge of the PLM.
Resolving Disadvantages of RL. As described in Section 2.4, RL has some limitation with its application: 1) it suffers from exponentially large search space |V|T , and 2) it suffers from unstable training caused by credit assignment problem. STG resolves the first issue since the frozen PLM chooses a token when it is selected, and therefore the search space of the generator is approximately decreased from |V|T to |V|T−TPLM where TPLM is the average length of sequences generated by PLM. For the second issue, the loss function of STG (Equation 9) intuitively shows that the gradient to the task-specific policy πa associated with producing at will depend on the selector’s action (i.e. it = 1). Hence, unlike Non-STG, πa of STG knows which token is contributed to the reward (see Appendix D for an illustration).

heading: 5 RELATED WORK
text: Recently, prompt-based in-context learning with an extremely large transformer-based PLM shows impressive few-shot generation performances (Radford et al., 2019; Brown et al., 2020). Schick & Schütze (2020) propose manually designed natural language prompts for improved few-shot text summarization and headline generation. Elsahar et al. (2018) conduct zero-shot learning for question generation from knowledge graphs, however they require a large amount of in-domain training data for their transfer learning. Chen et al. (2020) directly finetune the pretrained GPT-2 with a small amount of serialized attribute-value pairs for table-to-text generation. Gong et al. (2020) further apply multiple tasks to effectively leverage the structured information of tables. In contrast to these approaches, our proposed method utilizes RL-based additive learning for few-shot text generation.
Applying RL for text generation has been widely used to mitigate the exposure bias problem of MLE as well as to directly optimize task-relevant evaluation metrics. Ranzato et al. (2015) use the REINFORCE algorithm for text summarization and machine translation while Bahdanau et al. (2017) use the actor-critic algorithm for machine translation. However, they require pretraining using MLE. Ding & Soricut (2017) propose softmax policy gradient to remove the MLE-based pretraining. However, it requires various techniques for effective training. Tan et al. (2018) propose an entropy-regularized policy optimization that subsumes many of the previous training algorithms. Our proposed method is different from these methods in that we apply RL for more difficult few-shot generative modeling.
Various methods take into account the RL tasks with large action spaces like NLG. Dulac-Arnold et al. (2015) consider only actions in a cluster around the latent state of action obtained from a given state. Chandak et al. (2019) define the action embedding as a distribution with semantic of action and use a deterministic policy to take an action. Even-Dar et al. (2003); Zahavy et al. (2018) devise a method of incorporating the process of directly removing unnecessary actions according to the state in the RL problem. Guo et al. (2021) propose to use soft Q-learning and path consistency learning to combine off- and on-policy updates. Unlike these approaches, we use the hierarchical policy that reduces the sequential action space.

heading: 6 CONCLUSION
text: In this work, we propose to exploit a selective token generation between the pretrained language model and the task-specific adapter with RL-based additive learning for the tasks of few-shot natural language generation. In particular, we devise a trainable policy selector at the token level and jointly learn it with the task-specific policy. The proposed policy selector and RL algorithm make the two policies complementary each other and lead to robust few-shot generative modeling. Experimental results on various tasks of few-shot text generation show that the proposed selective token generation along with RL-based additive learning consistently improves the performances with less
overfitting. For future work, we will investigate more general ensemble learning for few-shot learning and perform more study on the architectures of both the adapter and selector.

heading: A QUESTION ANSWERING RESULTS

heading: B SUMMARIZATION PERFORMANCE

heading: C TRAINING SETTINGS
text: In our experiments all the models of additive learning, Non-STG and STG, are used the same architecture and hyper-parameters (except whether to use pre-training) for training as described in Table 9. We found that pre-training the addtional layer of Non-STG-RL with MLE helps the performance improvements. On the other hand, STG without pre-training shows better performances. We use the training data for each topic of the task of Data-to-Text as their validation data.
D ILLUSTRATION OF STG

heading: E ADDITONAL STUDY
text: STG-MLE. Here, we evaluate the MLE version of STG (denoted as STG-MLE) which is trained by MLE for the mixture policy πh(·) = πs(it = 0)πLM (·) + πs(it = 1)πa(·) similar to copy mechanism (Gu et al., 2016). In few-shot training, the explicit use of PLM logits can efficiently reduce the fine-tuning loss especially when the adapter is light since the adapter can focus only on the task-relevant part in generation. STG-RL5 learns to do this naturally by stochastic policy sampling if the policy selector is initialized to perform uniform sampling. On the other hand, STGMLE can be easily collapsed to select only a task-specific policy (i.e. it = 1). This is because the gradient flows the additional model only and, unlike STG-RL, there is no chance to exploit diverse paths during training in the teacher forcing manner. As shown in Figure 4, STG-MLE starts from the same point of STG-RL but it collapsed to Non-STG-MLE.
Learning Curve. It is well known that the RL-tuning resolves the exposure bias of MLE-tuning. We can expect that an additive learner of MLE would be affected by the exposure bias as well, and the RL objective for additive learning resolves it. Here, we present some learning curves6 obtained from training in our experiments. As shown in Figure 4, the learners of MLE seem to have overfitting (in terms of Perplexity, PPL) and exposure bias (in terms of Score). On the other hand, the learners of RL were less effected by the problems. We can find that the STG models (denoted STG-RL) are superior to the others from the perspective of the score.
Effectiveness of Selector. Here, we investigate the effectiveness of the selector πs of the STG. We compare Fixed Selection against the Dynamic selection. In the fixed selection, the probability of selecting the PLM’s policy πLM is fixed to πs(0t|st) = 1−πs(1t|st). We measure the performance with respect to πs(1t|st) = c where c is a constant. The selection will be uniformly random when c = 0.5, and when c = 0, the performance will be equivalent to the performance of the PLM without additive learning. Figure 3 shows that the input-dependent dynamic selection by our STG
5We add ”-RL” to the STG to distinguish with STG-MLE in this context. 6The curve for Data-to-Text is not presented since there is no actual validation set.
outperforms the fixed selection with any c. We can find that how πs works for each task. For instance, in QA task, the first few tokens of an answer may decide the quality of generation (i.e. ”yes” or ”no” in binary QA). Therefore, an optimal strategy of the STG might be producing the first few tokens sampled from the task-specific πa and the remaining tokens from the PLM πLM . The curve supports this interpretation since the score is decreased as c is close to 1. Our STG learns such a strategy as shown from the generated answers in Table 11 and 12. In Data-to-Text, the BLEU score is increased as c is close to 1 while the ERR score is decreased. This fact supports the results of NE(mix) models as discussed in Section 4.6. The πs learns to balance between the BLEU and ERR.

heading: F GENERATED SENTENCE EXAMPLES
text: Here, we show generated sentence examples for each task (see Table 10 for Data-to-Text, Table 11 and Table 12 for Question Answering and Table 13 and Table 14 for Summarization.). The tokens sampled from the task-specific policy πa are presented in red.

Review summaries:
This paper proposes a selective token generation method for additive learning under pre-trained language models. Specifically, the authors introduce a selective module to decide the generation policy of tokens, either from the LM generation policy or the RL generation policy. The experiments are conducted on data-to-text task and text summarization task. Results show the improvements over the traditional baselines. 

The paper is interested in the problem of exposure bias of sequential text generation: tokens are drawn from the data during training, while during inference, the tokens are sampled from the model’s distribution. Specifically, the authors consider this problem in the context of few-shot learning, where only a few labeled examples are available. The authors introduce a method that combines task-specific adapters and reinforcement learning-based training: at each token prediction, the model chooses between the distribution of the original pre-trained LM and the distribution induced by the task-specific adapters on top of the pre-trained LM. The next token is sampled according to the selected distribution. The authors observed gains in two tasks (data-to-text generation and summarization). 

The authors proposes a method to improve large pre-trained language model's ability in few-shot language generation. The main idea is to freeze the large pre-trained language model (PLM), fine-tune a copy of this PLM on the task at hand, and use a selector to switch between generating the next token from the freezes PLM or the fine-tuned PLM. Instead of the commonly used maximum likelihood estimation, the authors cast the optimization problem as a reinforcement learning (RL) problem using the task evaluation metric such as BLEU or ROUGE as the reward. The authors perform experiments on few-shot data-to-text and text summarization settings, comparing to a few baselines (fine-tuned PLM and a few variants of the proposed method).