Selective Token Generation for Few-shot Language Modeling
Authors: Daejin Jo
Taehwan Kwon
Sungwoong Kim
Eun-Sol Kim
Venue: ICLR 2022 Submitted (2022)
Keywords: Natural Language Generation
Reinforcement Learning
Few-shot Learning
Deep Learning

Abstract: Natural language modeling with limited training data is challenging problem, and many algorithms make use of large-scale pretrained language models (PLMs) for this due to its great generalization ability. Among these transfer learning algorithms from PLMs, additive learning that incorporates a task-specific adapter on top of the fixed PLM has been popularly used to alleviate the severe overfitting problem in the few-shot setting. However, this added task-specific adapter is generally trained by maximum likelihood estimation that can easily suffer from the so-called exposure bias problem, especially in sequential text generation. Therefore, in this work, we develop a novel additive learning algorithm based on reinforcement learning (RL) for few-shot natural language generation (NLG) tasks. In particular, we propose to use a selective token generation between the transformer-based PLM and the task-specific adapter during both training and inference. This output token selection between the two generators allows the adapter to take into account only on the task-relevant parts in sequence generation, and therefore makes it more robust to overfitting as well as more stable in RL training. In addition, in order to obtain the complementary adapter from the PLM for each few-shot task, we exploit a separate selecting module that is also simultaneously trained using RL. Experimental results on various few-shot NLG tasks including data-to-text generation and text summarization demonstrate that the proposed selective token generation significantly outperforms the previous additive learning algorithms based on the PLMs.

Under review as a conference paper at ICLR 2022
SELECTIVE TOKEN GENERATION FOR FEW-SHOT LANGUAGE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Natural language modeling with limited training data is challenging problem, and many algorithms make use of large-scale pretrained language models (PLMs) for this due to its great generalization ability. Among these transfer learning algo- rithms from PLMs, additive learning that incorporates task-specific adapter on top of the fixed PLM has been popularly used to alleviate the severe overfitting problem in the few-shot setting. However, this added task-specific adapter is gen- erally trained by maximum likelihood estimation that can easily suffer from the so-called exposure bias problem, especially in sequential text generation. There- fore, in this work, we develop a novel additive learning algorithm based on rein- forcement learning (RL) for few-shot natural language generation (NLG) tasks. In particular, we propose to use a selective token generation between the transformer- based PLM and the task-specific adapter during both training and inference. This output token selection between the two generators allows the adapter to take into account only on the task-relevant parts in sequence generation, and there- fore makes it more robust to overfitting as well as more stable in RL training. In addition, in order obtain the complementary adapter from the PL.M for each few-shot task, we exploit separate selecting module that is also simultaneously trained using RL. Experimental results on various few-shot NLG tasks including data-to-text generation and text summarization demonstrate that the proposed se- lective token generation significantly outperforms the previous additive learning algorithms based on the PLMs.
INTRODUCTION
Natural language processing (NLP) have recently achieved great progress using advanced neural
language models (Radford et al. 2018 Devlin et al. 2019 Yang al. 2019 Liu al., 2019 Clark et al. 2020; Raffel et al.) 2019; Lan al. 2019 Lewis et al. 2020). However, these neural models typically require large-scale training data for each individual task, and solving new NLP task that has only few examples is still challenging problem Yin 2020). Especially, natural language generation (NLG) with limited training data is an important yet more difficult task due to its fast adaptation of sequential prediction models in wide range of applications including text summarization, question answering, data-to-text generation, machine translation, etc (Peng et al.,
2020
Chen al. 2020 Xu et al. [201 Schick & Schütze 2020 Chang al. 2021 Radford al. 2020 Brown et al. 2020).
2019
Lewis et al.
More recently, pretrained language models (PLMs) have shown great generalization ability when combined with large-scale data and big transformer-based models (Devlin et al. 2019: Radford et al.)
2019 Lewis et 2020: Brown al. 2020 [Subramanyam Kalyan al. 2021 Therefore. transfer learning from transformer PLMs has been popularly used for few-shot NLG tasks with promising results. In specific, the use of PLM for few-shot NLG can be categorized into three approaches:
1) prompt-based, 2) finetuning, and 3) additive learning. Prompt-based approaches encode task description and task-specific examples as a natural language prompt for few-shot text generation
(Radford et al.) 2019 Brown et al. 2020 Zheng & Huang 2021 Schick & Schütze 2020 Li &
Liang] 2021). While these approaches can take full advantage of the universal natural language understanding and generation capabilities of large-scale PLMs without further training of the main model, these have some limitations in dealing with a large domain shift from the pretraining corpus data, tuning suitable task-specific prompts, and covering increased size of conditioning examples.

---

Under review as a conference paper at ICLR 2022
On the other hand, finetuning of the PLM is able to explicitly impart task sesc-specific knowledge to the model and hence lift the above limitations (Ziegler et al. 2019 Xu et al. 2021: Chen et al. 2020 However, these finetuned models are prone to overfitting when only a small amount of training data is available. In order to alleviate such an overfitting problem, additive learning has been extensively
exploited by incorporating task-specific adapters into the PL.M (Zeldes al., 2020. Stickland & Murray 2019).
In general, task-specialized adapters for few-shot NLG are trained by maximum likelihood estima- tion (MLE). While MLE is efficient in learning, it suffers from the exposure bias problem due to the difference in the training and inference mechanisms (He et al., 2019). and this problem can be severe with limited training data. Reinforcement learning (RL) is capable of resolving this exposure bias problem by sequential output sampling during training (Ranzato al. 2015; Keneshloo al. 2019 Shi et al., 2021). Moreover, it allows to leverage the target-specific sequence-leve objectives such as BLEU and ROUGE (Wu et al.) 2018 Guo et al.) 2021). However, the task of NLGG, the expo- nentially large space of output sequences restricts the use of RL since it leads to high variance and unstable training which is more serious in the few-shot setting. In this work, we develop a novel RL- based additive learning algorithm the transformer-basse PLM overcome these shortcomings and to improve the performance of few-shot NLG.
In particular, first convert the NLG task to the sequential token generation task based on the transformer language model, and then propose a selective token generation between the PI.M and the task-specific adapter, during both RL-based training and inference. The proposed output token selection enables to not only explicitly maintain general prior knowledge from the frozen PLM but also focus only on the task-relevant parts in sequence generation. In addition, in few-shot learn- ing this partial token generation makes the task-specific adapter more resilient to overfitting and furthermore reduces the overall output space which leads to stable RL training. Here, in order to make the two token generators (policies) complement each other as well as to realize the robust output selection at the token level on the fly, we exploit a separate token-level policy selector. is noted that both the policy selector and the task-specific adapter are simultaneously learned by the RL algorithm. Experimental results on various few-shot NLG tasks show that the proposed selce- tive token generation outperforms the previous PLM-based additive learning algorithms with the comprehensive (non-selective) token generation.
Our main contributions can be summarized as follows.
A novel selective token generation between the PLM and the task-specific adapter is pro- posed for transformer- based few-shot NLG.
A separate selecting module is exploited to adaptively determine each output token in sequence both at training and testing time.
RL is applied to train both the policy selector and the task-specific adapter that is comple- mentary the PLM in text generation.
An extensive empirical validation on few-shot NLG tasks demonstrates that the proposed selective token generation performs better in comparison to the previous PLM-based addi- tive learning algorithms.
2 BACKGROUND
2.1 NATURAL LANGUAGE GENERATION
The goal of NLG is to generate text sequence y [y,o..., yT] for a given task, where ye is the 4th output token from a vocabulary V. and T is the output sequence length. For this generation, we aims to model the distribution of y that is autoregressively factorized as po(y) = =II Po (y Yet), where / denotes the model parameters and y<t = [yo, ..., 1.). Here, the conditional distribution to sample token for each step, (yyyly <x), is defined by the softmax function on the output logits <<),,
(Vely<e). Note that in general, the language generation is conditioned on input context according to given task. Here, we encode the conditioning context by the same sequential model for gener- ating an output sequence, and for simplicity we omit it. In this work, we utilize the autoregressive transformer for our generative model.
2

---

Under review as a conference paper at ICLR 2022
2.2 ADDITIVE LEARNING FOR FEW-SHOT GENERATION
To effectively leverage the general linguistic knowledge, 0 is first initialized by the PLM parame- ters, 0r.M, for NLG. Given N task-specific training instances, D = {{y"* ground-truth output sequence, directly finetuning OLM using DD can incur the severe overfitting prob- lem when N is small in the few-shot scenario. Therefore, we add the task-specific adapter, ge, pa- rameterized by l, on top of the PLM, and optimize only 6. (Zeldes et al. | Stickland & Murray 2019). In specific, we reformulate f(-ly=c <t; 0) = W h(y ex; 0n) where W E - and E R denote the weight matrix and the penultimate representations, respectively, and (W, 0, Then, we define the -ask-ipccciic conditional distribution as follows:
J==1 "y""">=
where y is the wihh
p OLAS Ba) = softmar WLM T (y + W. WaTg(han g(bLLM(<<):
(I)
where hy = h(y= ct: LR.LM) and 0g = {Wa: 0g). Here, the summation the PLM logits and the adapter logits is motivated by auxiliary training It is noted that in our additive learning Og is updated while OLM is kept frozen. Hence, in the following we omit Oras such that pa, (We m <c)) p(ylly 2x: LLM, 0a) for simplicity.
2.3 MAXIMUM LIKELIHOOD ESTIMATION (MLE)
Given a small amount of training data D {y"*) 1 log-likelihood as follows:
In=1 MLE optimizes 0 by maximizing the data
N
0 = arg MaxX
log Po(l ly
(2)
1=
Here, the output token at each step is conditioned on not the previous sampled tokens from the current model but the previous ground-truth tokens y Namely, tokens are drawn from the data distribution during training, which is opposed to that tokens are drawn from the model distribution test time. This discrepancy, also known as the exposure bias, leads that the errors will be accu- mulated along the generated sequence at test time since the model is biased to only perform well on the ground-truth history distribution. Especially, this bias problem sew-shot training. In addition, the token-level cross-entropy loss in MLE training is different from the sequence-level test metrics such as BLEU and ROUGE that are commonly used in the tasks of NLG.
be more severe in the
2.4 REINFOR LEARNING (RI.)
As an alternative to MLE, RL is able to overcome the exposure bias problem of MLE by sequence- level sampling from the model distribution during training (Ranzato et al.] 2015). Also, RL can improve the performance by directly optimizing the evaluation metrics (Guo et al. (021). In order to use RL. for our additive learning, we reformulate
RL problem: at each time step f, the agent takes the current state S, = yer as an input and performs an action a, that outputs token Yt by a policy ne (Sr|s:) corresponding
text generation
pe (ye y <r). Then, the agent receives reward = r(st. 0z) and deterministically transitions to the next state S:+1- Here, note that the token-level intermediate reward r, = 0, level evaluation metric between the two full sequences, y and y*. Let r [(St. a,, rx) rr) be the trajectory generated by wy. The RL objective for the optimal agent to maximize the expected sum of future discounted rewards:
T when we use the delayed reward associated with the sequence-
[(mo)= = BT
Σ
r
where y E (0,1) 1) the discount factor.
Although the auxiliary training is particularly designed for maximizing the likelihood of the target task output, it also can take an advantage for RL since the adapter logits are nearly zero before training is advanced. Namely, it lets the task-specific conditional distribution start learning from the distribution of PLM, not uniform distribution.
3

---

Under review as a conference paper at ICLR 2022
72
Yt
Yt
7
PLM
PLM
Y1a-1
Y1:1-1
Text generation processes of Non-STG and STG are described. In the Non-STG model, every token is sampled from the task-specific policy Ta (Left). On the other hand, the proposed STG model, each token is selectively sampled from either the PLM policy TLM or the test-specific policy Ta where the selection is performed by the selection policy vs (Right). Symbols with dashed line represent learnable models.
Figure 1:
Among number of algorithms to approximately optimize the RL objective, employ actor- critic algorithm (Bahdanau et al. 2017) since it explicitly optimizes the policy network and it can also alleviate the delayed reward problem. The actor-critic algorithm requires the additional critic network to estimate the value of state, V (T((s) E,, I
rrlss Ea, (aa]s:)) (st, at) where the state-action value function Q" 0²) E, v 7" 're Ss, ail + V ($(41)) We use the following policy gradient loss to learn the policy parameters 0:
C
Σ A (s;, az) log No (a))s))
where (St, at) = (s a.) - (s,) is the advantage function that quantifies how an action ag is better than the average action in state St.
In few-shot text generation, the extremely large action space (VIT) as well as the small amount of training data often make it difficult perform RL with degraded performances, even though we conduct the additive learning from the PLM. Furthermore, it commonly has a delayed reward function (e.g. BLEU) which defined after an entire sequence generated. It hard decide which token and how much contributes to the reward. This problem is known as the credit assignment. Therefore, in this work, we propose a selective token generation for improving the RL-based additive learning.
3 SELECTIVE TOKEN GENERATION
Instead of generating all tokens in an output sequence from the single task-specific policy, Ta = no, (st), each time step f, we sample an output token yt selectively from either the PLM policy TLM = To (0elst) or the task-specifer policy T.:
y/ = dt (1,[TTMM is selected]=r.M) (at (st) + (1 1e(FEM (a.
(5)
where L.[-] is the indicator function (at t) that equals if it is true and 0 otherwise. This output token selection allows to explicitly utilize general linguistic knowledge from the PLM without catastrophic forgetting in few-shot learning. Also, the task-specific policy focus generating only the task-relevant parts, which enables more effective fhw-shot training with a reduced action space.
Now we need to determine how to select the proper policy at each step on the fly as well as to make the task-specific policy complementary the PLM policy. For this exploit a separate
4

---

Under review as a conference paper at ICLR 2022
token-level policy selector. The proposed policy selector (4: with the parameters f... where & E {0, 1). is an another policy that stochastically decides a policy to generate a, for s,. Namely, a token sample yt is generated by the following process:
it
TT (ir(s)),
at ~ (arls) if O, if 1.
y1
aa ma(ess)
This process can be considered as token generation from hierarchical policy TA (araa/ss (5; 0.. LLMS 0o) where the policy selector represents the upper-level prior for the pref- erence of the low-level policy. Therefore, the value function of this hierarchical policy can be formulated as
TTh (s,) EEA. y [s/
T,(4 = 0|s,) Σ TL. (T (s, (s,
uz(in 1(s,)
naa (
and (st, ae) TT (st, a.) (s;). Here, it is noted that single critic network is used for the hierarchical policy since i, does not affect s,. Given sample trajectory {(St: by a,, Tt) H the loss for optimizing 0, and 0g is
T Σ 0=
-=
1ii = 0]( (log (g( + log (ie (s))) Th (St, az)
To + lie T. (ar(s)) + log x. (de (dels)))
(9)
where stands for the stop- coo-grraient operator. Similar to Ta. makes use of the PLM representa- tions and the task-specific adapter such that
T. (ie 0.) softmar (m(((HEM((ss):00))
(
(10)
where m is the selector module that is implemented by small neural network. Figure depicts the overall text generation process by the proposed selective token generation (STG) comparison to the previous non-selective token generation (Non-STG). Here, note that since all policies in STG share the same PL.M representations, the increased computational cost by STG over Non-STG negligible.
The use of the separated policy selector that is simultancously trained with the task-specific policy allows the task-specific policy to be complementary to the PLM policy. Especially, this cooperative ensemble learning can be realized by our RL algorithm that performs sequential sampling from the model during training. In addition, the proposed STG model makes use of the PLM not at the feature level but the output distribution level text generation, In our few-shot learning this is beneficial in explicitly retaining strong linguistic and world knowledge from the PL..
EXPERIMENTS
In this section, we evaluate our method against baselines on Data-to-Text, Question Answering and Text Summarization tasks which are widely used in few-shot NLG. Since we claim the effectiveness of our method in additive learning, we compare our results against those obtained by other additive learning methods.
4.1 BASELINE
PLM. our experiments, we assume that the PLM works to some extent for a given task. However, the naive PLM usually does not satisfy for new task unseen during training. Hence, finetuned
5

---

Under review as a conference paper at ICLR 2022
Table 1: Data-to-Text performance FewShotWOZ dataset.
Model
Restaurant
Hotel
TV
Laptop
BLEU t
ERR
BLEU
ERR
BLEU
ERR
BLEU
ERR
PLM
19.42
12.57
35.84
13.74
29.00
9.15
28.27
9.31
Non-STG-MLE
17.21
15.87
28.42
12.64
29.83
10.05
26.76
10.52
Non-STG-RL
18.01
11.98
36.72
12.64
28.66
9.19
28.59
9.21
NE(mnar)--LL.
14.12
15.27
31.32
14.29
28.23
10.21
26.93
10.02
NE(miz)-MLL
25.27
14.97
37.13
15.93
32.85
16.31
32.91
14.77
NE(mor)-RL
15.20
11.68
32.68
16.48
28.91
9.24
28.66
9,51
NE(miz)-Ro -RL
24.10
19.16
38.07
18.68
32.84
18.06
32.53
17.14
STG
21.28
10.78
38.09
11.54
30.24
9.03
30.41
8,91
GPT-2 Radford et al., 2019) with MLE for few epochs and used as the PLM. Fine-tuning the PLM with iLE most commonly used for task adaptation and thus it also be strong baseline. This fine-tuning phase accelerates the learning of the adapter. This is particularly when the adapta- tion requires to cover the large domain shift. Severe performance degradation was observed for all the tasks when we skipped the fine-tuning.
Non-STG. This method stands for Non-Selective Token Generation which uses the above the PLM as an encoder (frozen) and the adapter (additional layer to be trained). We use two objectives, MLE and RL, for additive learning. These will be denoted as Non-STG-MLE and Non-STG-RL. respectively.
STG-Naive Ensemble. We believe that our proposed method encourages the task-specific policy (m,) to complement the PLM's policy (TLM) with proper selection of the selector through joint training. To investigate this, we evaluate against two different naive ensembles of the policies, To trained from Non-STG and TIM of the PLM. These ensemble schemes are as follows:
NB(mar):: Tmax = Penftmax(aaxta lLM))
NE(miz): Tyii = (na + TLM)/2
We also evaluate another naive ensemble strategy NE(random) that randomly selects a token policy at each step between Ta and TLM. however it shows lower performances than the others.
4.2 IMPLEMENTATION
Here, explain the implementation set-up of our method.
Additional Layer: The task-specific adapter in Section 2.2 is implemented by a LSTM to encode
the dynamics of the representation vector hLM- We found that the use of MLP not performed well.
Selector. We 2-layer MLP with ReL.U activation for m.
Reinforcement Learning. We employ Actor- method (Konda Tsitsiklis 2000 Fedus et al., 2018) for RL. The agents (i.e. selector and generator) receive reward after generating sentence. Here, we use different reward functions according to tasks. We use delexicalised BLEU for Data-to- Text following Peng al. (2020), Averaged score of BLEU and ROUGE-L for Question Answering and ROUGE-L for Text Summarization following Paulus et al. (2017) as the reward function.
Token Sampling. During the training, is {0, 1) n. first sampled, and then we use either TLM of the PLM for i, or the task-specific policy Ta for in = 1 to sample the ith token. During the evaluation, any decoding strategy, such beam search, can be used with the mixture of policies (.).) = Ty (i4 ()TLM (-) (2+ = (.). We use the beam search decoding with sample size of 1 for Text Summarization and topp 0.9 decoding for both Data-to- Text (k 10) and Question Answering (k: 3).
We make use of GPT-2 with 345M parameters as the initial checkpoint. We follow the training details in the previous works (Peng 2020 Khandelwal al. 2019) for each task.

---

Under review as a conference paper at ICLR 2022
Table 2: Averaged performances and performance gains against the PLM for Question Answering on 1% few-shot subset data of MS-MARCO. The gain indicates the averaged performance gain against the PLM's one with the standard deviation of the gain in the blacket. See the results of the other few-shots in Appendix
Model
BLEU
BLEU gain
RL
RLgain
Avg
Auggain
PLM
41.49
0.00(+ 0.00)
49.76
0.00( 0.00)
45.63
0.00( 0.00)
MOn-STG-MLE
41.02
-0.47(+ 0.41)
50.14
0.37(# 0.40)
45.58
-0.05(+ 0.40)
Non-STG-RL
41.25
-0.24(+ 0.34)
49.97
0.20(+ 0.21)
45.61
-0.02(#: 0.27)
NE(mar)-MIA.
41.11
-0.38(+ 0.51)
50.77
1.01 0.57)
45.94
0.32( 0.54)
NE(mir)-Mo -MLE
42.26
0.77(+ 0.37)
51.14
1.38( 0.40)
46.70
1.08(# 0.38)
NE(maz)-Ro
41.51
0.02(# 0.47)
50.54
0.78(# 0.38)
46.03
0.40(# 0.41)
NE(mir)-RR
42.29
0.80(+ 0.24)
50.84
1.08(# 0.19)
46.57
0.94( 0.21)
STG
42.76
1.27(+ 0.38)
51.19
1.43(+ 0.37)
46.98
1.35( 0.37)
4.3
DATA-TO-TEXT
Data-to-Text is task that transforms structured data such as graphs or tables into natural language. Recent works (Mager et al.] 2020 Peng et al. 2020 Kale 2020) show that the PLM can be adapted successfully to this task by taking serialized form of data an input without carefully designed model to encode the structured data. Here, we perform experiments on FewShotWOZ (Peng et al,, 2020) dataset. The evaluation conducted only the topies of Restaurant, Hotel, Laptop, and TV since the evaluator has not published yet for the other topics (Taxi, Attraction, Train F There are 50 training instances for each topic and 129, 78, 1379, and 680 testing instances for Restaurant, Hotel, Laptop. and TV, respectively. The models are evaluated by measuring fluency and informativeness using BLEU score and ERR (slot ERror Rate), respectively. Table T the obtained results.
4.4 LONG ANSWER QUESTION ANSWERING
We consider Long Answer QA task on MS-MARCO (Nguyen et al. 2016) dataset. In this task, a passage and query are given, and the model generates an answer with respect to the query by referring to the passage. Here, randomly sample various sizes of (0.5% & 500, 1%. and 2%) subset data from the train dataset. We also sample validation and test set, which contains 500 and 12,000 instances, respectively, from the dev dataset. We repeat this test three times with different random seeds. In other words, we collect three different random subsets for each size of few-shot. Thus, perform experiments
total nine subsets. The models evaluated by measuring BLEU, ROUGE-L (denoted as RL),, and their average value (denoted as Aug). The performances of the models reported in Table 2 and Appendix A
4.5 TEXT SUMMARIZATION
We consider the problem of abstractive summarization: models aim to generate its summary (the "target" text) for a given piece of "source" text such that its meaning intact. The model has to learn 1) building semantic form internally from the source text and 2) converting the semantic form into text. Here, we randomly sample various sizes of (0.5% 1, 500, 1%, and 2%) subset data from CNN/Daily Mail (See al. 2017). We repeat this test three times for each size of few- shot as in above QA task. ROUGE (Lin 2004) is commonly used to evaluate n-grams recall of the summaries with gold references. ROUGE-I, ROUGE-2, and ROUGE-L (measured based on the longest common sub-sequence) scores are reported (denoted as R1, R2, and RL, respectively) in Table and Appendix B
4.6 RESULT
We compare experimental results on various tasks in this subsection. In most cases, additive learning improves the performances over the PLM. However, they do not always guarantee performance
GP

---

Under review as a conference paper at ICLR 2022
Table 3: Averaged performances and performance gains against the PLM for Text Summarization on 2% few-shot subset data of CNN/DM. The gain indicates the averaged performance gain against the PLM's one with the standard deviation of the gain in the blacket. See the results of the other few-shots in Appendix B
Model
RI
Rlgain
R2
R2gain
RL
Rogoon
PLM
33.05
0.00(+ 0.00)
12.96
0.00( 0.00)
23.39
0.00( 0.00)
Non-STG-MLE
33.19
0.14(: 0.08)
12.98
0.02(+ 0.05)
23.39
0.00(# 0.06)
Non-STG-RL
33.22
0.17(+ 0.09)
12.99
0.04(+ 0.05)
23.40
0.01 0.08)
NE(mar)-Mi -MLE
33.19
0.100
12.99
0.03(+00055)
23.40
0.01 0.06)
NE(mix)-MLL
33.11
0.04(+0.04)
12.99
0.03(+0.003)
23.41
0.02( 0.03)
NE(max)-RL
33.21
0.16(
12.99
0.04( 0.06)
23.41
0.02(=
NE(miz)-RR
33.14
0
13.00
0.05(#
23.42
0.03(# 0.03)
STG
33.45
0.40(+ 0.24)
13.14
0.18( 0.10)
23.66
0.27(+ 0.16)
improvement. For example, the ERR score of the PLM on Laptop shows better result except for STG and NE(miz))RL (see Table [ and the Non-STG models trained on 1% few-shot subset of MS-MARCO do not outperform the PLM (see Table 2 and Table of Appendix A).
In Data-to- Text task, as shown in Table D we can observe that the Non-STG models do not out- perform the PLM even though it has more neural units and takes more training time. The models trained on the RL objective show better performances for the ERR (lower is better). Interestingly, NE(mix) methods show strong improvements for the BLEU which measures the fluency of sen- tence but obvious degeneration for the ERR which measures the rate of missing information from the given data. These results suggest that the PLM much more capable of task- mesk-general knowledge than the task-specific generator (i.e. Ta) trained on few-shot dataset. It motivates us to jointly learn the policy selector and the task-specific generator. Here, while other methods show some trade-off between BLEU and ERR, only STG shows improvements on the both metrics for all topics in the dataset.
In Question Answering, NE(miz)) and STG show significantly better performances than the other methods as shown in Table and Appendix A Notably, NE(miar) show good performances as much as STG. It obviously suggests that the PLM can be a complementary model to the additional model. Therefore, in this context, it can be lost of the prior knowledge of the PLM even the additional model has been built over the feature space of the PLM. We will discuss it in Section 4.7
In Text Summarization (a problem of long-sequence generation). STG shows significantly larger gains than MLL-STG-MLE, Non-STG-RL, and their naive ensembles with the PLM in every score metric and training data size as shown in Table and Appendix B
4.7 WHAT MAKES STG BETTER?
Knowledge Preservation. Although freezing the parameters of the PLM can mitigate the forgetting problem, would be failed to do so when additional units having sufficient capacity is used since it is easy to memorize typical patterns of answers in small amount of training dataset and can degrade the generalization performance. An example that represent this issue is shown in Table 4 of Appendix A In this case, passage that contains two definitions (super-scripted and bolded) about conftict is given with query which asks the psychological meaning of conflict. Without the knowledge of who Colmar
fl is, it can be a hard to answer since the word psychology is not appeared in the passage. Here, the PLM repeats the given query due the imperfect domain adaptation. Non-STG models, both Non-STG-MLE and RL, generate the same answer, that is not the psychological meaning but the general meaning of conflict, with pretty low perplexity (Rs 0.35). The reason in producing this incorrect answer is that most queries in few-shot training data ask general meaning of a concept, and Non-STG models are overfitted to this pattern. On the other hand, the answer of STG, which is close the ground truth, is generated by the PLM policy WLM after some sequence of tokens (conflict is) that are sampled from the task-specific policy na-
AA psychologist, httph://////. //n. wtleenwwwkkkk. wiki wikipediee wtw.ia..rrgg/ii
www.eannnnnooccmmccooo..c (orn__(caaa (academ c)
8

---

Under review as a conference paper at ICLR 2022
We can find such examples for the other tasks in Appendix [ In Data-To- To-a-TooTeex,, shown in the last example of Table IO Non-STG generates nicam stereo which is not appeared in the given data. This is due that nicam stereo was appeared 7 times (7/50, 14%) in training data. In Summarization, as shown in the first example of Table 14 Non-STG models only consider the forepart of the given article. Since the most of the major information is appeared in the forepart in News data, Non-STG models can be easily overfitted to generate the text according to such pattern. Hence, we claim that Non-STG easily exposed learning patterns of typical answering and STG resolves this issue since can be fully accessible to the knowledge of the PLM.
Resolving vintdean of RL. As described in Section 2.4 RL has some limitation with its application: 1) it suffers from exponentially large search space D and 2) it suffers from unstable training caused by credit assignment problem. STG resolves the first issue since the frozen PLM chooses a token when it is selected, and therefore the search space of the generator is approximately decreased from |v[2 |VIT-TMLM where T PLA is the average length of sequences generated by PLM. For the second issue, the loss function of STG (Equation (9) intuitively shows that the gradient the task-specific policy Ta associated with producing a will depend on the selector's action (i.e. de = 1). Hence, unlike Non-STG, Ta of STG knows which token is contributed to the reward (see Appendix D for an illustrationn.
RELATED WORK
Recently, prompt-based in-context learning with extremely large transformer-based PLM shows impressive few-shot generation performances (Radford et al. 2019; Brown ct al., 2020). Schick & Schütze (2020) propose manually designed natural language prompts for improved few-shot text summarization and headline generation. Elsahar et al. (2018) conduct zero-shot learning for question generation from knowledge graphs, however they require large amount of in-domain training data for their transfer learning. [Chen et al.] (2020) directly finetune the pretrained GPT-2 with a small amount of serialized attribute-value pairs for table-to-text generation. Gong al. (2020) further apply multiple tasks to effectively leverage the structured information of tables. In contrast to these approaches, our proposed method utilizes RL-based additive learning for few-shot text generation.
Applying RL for text generation has been widely used to mitigate the exposure bias problem of MLE as well as to directly optimize task-relevant evaluation metries. Ranzato al. (2015) use the REINFORCE algorithm for text summarization and machine translation while Bahdanau et al. (2017) use the actor-critic algorithm for machine translation. However, they require pretraining using MLE. Ding & Soricut (017)) propose softmax policy gradient to remove the MLE-based pretraining. However, requires various techniques for effective training. TTan et al. (2018) propose an reguaiy-re policy optimization that subsumes many of the previous training algorithms. Our proposed method is different from these methods in that we apply RL for more difficult few-shot generative modeling.
Various methods take into account the RL tasks with large action spaces like NLG. Dulac-Arnold et al. (015)) consider only actions in cluster around the latent state of action obtained from given state. Chandak et al. (2019) define the action embedding as a distribution with semantic of action and use deterministic policy take action. EEven-Dar et al. (2003); (Zahavy et al. (2018) devise method of incorporating the process of directly removing unnecessary actions according to the state in the RL problem. Guo et al. (202T) propose to use soft Q-learning and path consistency learning to combine off- and on-policy updates. Unlike these approaches, we use the hierarchical policy that reduces the sequential action space.
CONCLUSION
In this work, we propose to exploit a selective token generation between the pretrained language model and the task-specific adapter with RL-based additive learning for the tasks of few-shot nat- ural language generation. In particular, we devise a trainable policy selector at the token level and jointly learn it with the task-specific policy. The proposed policy selector and RL algorithm make the two policies complementary each other and lead to robust few-shot generative modeling. Exper- imental results on various tasks of few-shot text generation show that the proposed selective token generation along with RL-based additive learning consistently improves the performances with less

---

Under review as a conference paper at ICLR 2022
overfitting, For future work, will investigate more general ensemble learning for few-shot learn- ing and perform more study on the architectures both the adapter and selector.
REFERENCES
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. International Conference on Learning Representations, 2017. URL https: www..iv..ogg/bb/11777.
07086
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.1416. 2020.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learn- ing action representations for reinforcement learning. In International Conference on Machine Learning, pp. 941-950 PMLR, 2019.
Ernie Chang, Xiaoyu Shen, Alex Marin, and Vera Demberg. The selectgen challenge: Finding the best training samples for few-shot neural text generation. arXiv preprint arXiv:2108.06614 2021.
Zhiyu Chen, Harini Favani, Wenhu Chen, Yinyin Liu, and William Yang Wang. Few-shot NLG with pre-trained language model. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 183-190, Online, July 2020. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/2222..ac- acl-main. 18. URL https: http///aaaatttttaa www.cnnnnnoooggo.. org/202c.. acl-main. 18
Kevin Clark, Minh- Mhah-Thang Luong, Quoc Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations, 2020.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 417141111444 14166, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1011863311/N11-1423. URL https:
//www. ac web.. org/ ant ho hnthology/hiii 1423
Nan Ding and Radu Soricut. Cold- start reinforcement learning with softmax policy gradient. arXiv preprint (tt0::::7000995500 2017.
Gabriel Dulac-Amold, Richard Evans, Hado Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep rein- forcement learning in large discrete action spaces. arXiv preprint 151.07751,.07679, 2015.
Hady Elsahar, Christophe Gravier, and Frederique Laforest. Zero-shot question generation from knowledge graphs for unseen predicates and entity types. arXiv preprint arXiv: arXiv:/802.0000 /802.06842, 2018.
Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action elimination and stopping conditions for reinforcement learning. In Proceedings of the 20th International Conference on Machine Learming (ICML-03), pp. 162-169, 2003.
William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the International Conference on Learning Representations, 2018.
Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. Tablegpt: Few-shot to-tex generation with table structure reconstruction and content matching. In International Conference Computational Linguistics, pp. 1978811988. 2020.
10

---

Under review as a conference paper at ICLR 2022
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. Incorporating copying mechanism in Sequence-to-sequueee sequence learning. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1631-1640 Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 18653vv1/P16-111 1154. URL https:
Han Guo, Bowen Tan, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Text generation with efficient (soft) q-learning. arXiv preprint arXiv:2106.077700 2021.
Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. Exposure bias versus self-
recovery: Are distortions really incremental for autoregressive text generation? arXiv preprint arXiv:1905.10617, 2019.
Mihir Kale. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005.104:: 2020.
Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K Reddy. Deep reinforcement learning for sequence-to-sequence models. IEEE transactions neural networks and learning systems, 3177):2499-2489 2019.
Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. Sample efficient text summa- rization using a single pre-trained transformer. arXiv preprint arXiv:1905.08833 2019.
Vijay Konda and John N Tsitsiklis, Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008-1014. Citeseer, 2000.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. International Conference on Learning Representations, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer pre-training
Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881- 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00/ 101.00/90, 2021.
Chin-Yer Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74 71-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. httpst htt///alaattthooo
org wwrg//0444 -
Yinhan Liu, Naman Goyal Myle Ott, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Manuel Mager, Ramón Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, and Salim Roukos. Gpt-too: language-model-finst approach for amr-to-text generation. arXiv preprint 20Xiv:2005.09123. 2020.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPS, 2016.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04300 2017.
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li. Jinchao Li, Michael Zeng, and Jianfeng Gao. Few-shot natural language generation for task-oriented dialog. arXiv preprint arXiv:2002. 2020.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018.
11

---

Under review as a conference paper at ICLR 2022
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 10683, 2019.
Marc' Aurelo Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint arXiv: 2015.
Timo Schick and Hinrich Schütze. Few-shot text generation with pattern -etttertxpploiiing training, arXiv preprint arXiv:20/2./1926, 2020.
Abigail See, Peter J Liu, and Christopher D Manning. Get the point: Summarization with pointer- generator networks. arXiv preprint arXiv:1704.04336,, 2017.
Tian Shi, Yaser Keneshloo, Naren Ramakrishnan, and Chandan K Reddy. Neural abstractive text summarization with sequence-to-sequence models. ACM Transactions on Data Science, 2(1):
2021.
Asa Cooper Stickland and lain Murray. Bert and pals: Projected attention layers for efficient adap- tation in multi-task learning. In International Conference on Machine Learning, pp. 5986-5995. PMLR, 2019.
Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. Ammus A survey of transformer-based pretrained models in natural language processing. arXiv preprint arXiv:2108.05544 05542, 2021.
Bowen Tan, Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric Xing. Connecting the dots between mle and rl for sequence prediction. arXiv preprint 1811.0881.099 2018.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie Yan Liu. A study of reinforcement learning for neural machine translation. arXiv preprint arXiv: wt..:100...0666 2018.
Xinnuo Xu, Guoyin Wang, Young Bum Kim, and Sungjin Lee. Augnlg: Few-shot natural language generation using self-trained data augmentation. arXiv preprint 05555,,,0600588 2021.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le.
Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5753-5763, 2019.
Wenpeng Yin. Meta-learning for few-shot natural language processing: A survey, arXiv preprint vrXiv:2007.09604, 2020.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Learn what to learn: Action elimination with deep reinforcement learning. arXiv preprint arXiv: 1009:22800002111 2018.
Yoel Zeldes, Dan Padnos, Or Sharir, and Barak Peleg. Technical report: Auxiliary tuning and its application to conditional text generation. arXiv preprint arXiv:2006.1687 2020.
Chujie Zheng and Minlie Huang. Exploring prompt-based few-shot learning for grounded dialog generation. arXiv preprint arXiv:2 109.06309.633. 2021.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 2019.
12

---

Under review as a conference paper at ICLR 2022
A QUESTION ANSWERING RESULTS
Table 4: Generated answers from instance of MS-MARCO QA dataset. Two definitions about conflict are presented in bold text in the passage. Tokens sampled from the task-specific policy ae are presented in red. The answers sampled from the models trained 0.5% few-shot subset data.
Passage
three types of conflicts are 1. intrapersonal conflicts 2. interpersonal con- flicts and unconscious conflicts the word conflict has been derived from a latin word "conflicts" which means "strike two things at the same time". conflict is aa opposition or a tug-of-war between contradictory impulses according colman "a conflict is the anticipated frustration entailed in the choice of either alternative".
Query
conflict definition psychology
Ground-truth
the anticipated frustration entailed the choice of either alternative.
PLM
conflict definition psychology.
Non-STG
conflict is opposition tug-of-war between contradictory impulses.
STG
conflict is the anticipated frustration entailed in the choice of either alternative.
Table 5: Averaged performances and performance gains against the PLM for Question Answering on 0.5% few-shot subset data of MS-MARCO.
Model
BLEU
BLEUgast gain
RL
RLgain
Aug
Auggain
PLM
35.64
0.00( 0.00)
43.10
0.00(= 0.00)
39.37
0.00( 0.00)
-OL-STG-MLE
34.53
-1.12(# 0.81)
43.08
-0.03(# 0.69)
38.80
-0.57(# 0.75)
Non-STG-RL
35.08
-0.56(= 0.24)
42.78
-0.32(# 0.14)
38.93
-0.44(= 0.17)
NE(maz)-Mi
34.69
-0.95(= 0.88)
43.93
0.83(= 0.88)
39.31
-0.06(= 0.87)
NE(mir))-ML.
36.26
0.62(+ 0.58)
44.43
1.32(= 0.64)
40.34
0.97( 0.61)
NE(maz)-Ro 35.14
-0.51 0.31)
42.94
-0.16(# 0.18)
39.04
-0.33(# 0.21)
NE(miz)-RL
35.93
0,29( 0.20)
43.52
0.42(= 0.29)
39.73
0.35( 0.24)
STG
37.37
1.72 0.34)
44.53
1.43== 0.61)
40.95
1.58( + 0,40)
Table 6: Averaged performances and performance gains against the PLM for Question Answering on 2% few-shot subset data of MS-MARCO.
Model
BLEU
BLEU gain
RL
RLgain
Aug
Auggain
PLM
47.72
0.00(= 0.00)
56.02
0.00(= 0.00)
51.87
0.00(= 0.00)
-On-STG-MLE
47.85
0.13(# 0.02)
56.81
0.79(# 0.21)
52.33
0.46(# 0.09)
Non-STG-RL.
48.00
0.28(+ 0.33)
56.83
0.82(+ 0.31)
52.42
0.55(+ 0.32)
NB(maz)-MM
47.65
0.07(#
57.22
1,20(# 0.35)
52.44
0.57(# 0.16)
NE(miz)-ML.F
48.44
0.72( 0.22)
57.30
1,28( 0.25)
52.87
1.00( 0.23)
NE(maz)-RL
47.58
-0.14( 0.57)
57.06
1.05( 0.57)
52.32
0.45(1 0.56)
NE(miz)-RL RL
48.28
0.56(# 0.08)
57.02
1.00(# 0.01)
52.65
0.78(# 0.04)
STG
48.42
0.70( 0.22)
57.30
1.28( 0.38)
52.86
0.99 0.30)
13

---

Under review as a conference paper at ICLR 2022
B SUMMARIZATION PERFORMANCE
Table 7: Averaged performances and performance gains against the PLM for Text Summarization on 0.5% few-shot subset data CNN/DM.
Model
R1
RI gain
R22
R2gain
RL
Rlgain
PLM
30.19
0.00(+ 0.00)
11.27
0.00(+ 0.00)
21.21
0.00(+ 0.00)
Bon-STG-MLE
30.34
0.14(# 0.13)
11.32
0.05(# 0.07)
21.20
-0.01 0.06)
Non-STG-RL.
30.35
0.15(+ 0.15)
11.34
0.07(+ 0.07)
21.22
0.02(+ 0.08)
NE(mar)-No -MLE
30.33
0.14(# 0.11)
11.31
0.04(# 0.06)
21.20
0.00(#: 0.04)
NE(mix)-Mo
30.32
0.13(+ 0.08)
11.31
0.04(+ 0.04)
21.23
0.02(+ 0.04)
NE(mar)-RL
30.37
0.18(# 0.11)
11.35
0.08(# 0.05)
21.26
0.06(# 0.02)
NE(miz)-RL
30.28
0.09(# 0.09)
11.30
0.03(# 0.05)
21.22
0.02(# 0.04)
STG
30.47
0.28( 0.15)
11.37
0.09(1 0.05)
21.36
0.15(1 0.09)
Table 8: Averaged performances and performance gains against the PLM for Text Summarization on 1 few-shot subset data of CNN/DM.
Model
R1
R1 Rigain
R2
R2 R2gain
RL
RLgain
PLM
31.03
0.00(# 0.00)
11.56
0.00(# 0.00)
21.62
0.00(# 0.00)
Non-STG-MLE
31.25
0.22(=
11.63
0.07(# 0.02)
21.72
0.10(1 0.07)
Non-STG-RL
31.24
0.21(=0013)
11.64
0.08(# 0.05)
21.76
0.13(# 0.12)
-MLE
31.25
0.22(=
11.63
0.07(# 0.03)
21.72
0.09(1 0.08)
NE(mir)-MMi MLL
31.20
0.17(=
11.61
0.04(+ 0.02)
21.69
0.06(# 0.04)
NE(max)-RR RR
31.26
+224+00.13)
11.66
0.10( 0.04)
21.78
0.15(+ 0.12)
NE(mix)
31.18
0.15(= 0.08)
11.63
0.06(# 0.02)
21.69
0.07(# 0.06)
STG
32.03
1.00(=: 0.48)
11.96
0.40(+ 0.15)
22.23
0.61 0.33)
C TRAINING SETTINGS
In our experiments all the models of additive learning, Non-STG and STG, are used the same archi- tecture and hyper-parameters (except whether to use pre-training) for training as described in Table 9 We found that pre-trale pre-training the addtional layer of -TG-ST with MLE helps the performance improvements. On the other hand, STG without pre-tree trainriiing shows better performances. We use the training data for each topic of the task of Pata-to-Text as their validation data.
Table 9: Hyper-parameter used for experiments
Hyper-parameter
Summarization
Data-to-Text
Question Answering
Num layer
RNN hidden size
512
256
256
Y
Optimizer
AdamW with belas (0.9, 0.999..) eps 10-8
Learning rate
2c-5
5e-5
Pre-train epoch
1
1
(NOR-STG-RIL)
Validation
500
50
500
Train epochs
16 (0.5%), (1%), 4 (2%)
30
20 (0.5%), (1%), (2%)
Batch size
16
10
16
14

---

Under review as a conference paper at ICLR 2022
D ILLUSTRATION OF STG
Target
ABCDEF
aa
Ta
Non-STG
ABDEFO
(not bad)
A B
H
I
(bad)
aTa
VTa
Ta
a
STG
D
F G
(not bad)
A B
G
(bad)
VT
VTA
simple schematic illustration of Non-STG and STG. Non-STG(RL); the whole se- quence of target is generated from the task-specific policy penalized from the delayed feedback. STG: the third token sampled from ma and the model lets the other tokens (highlighted with cyan) generated from the PLM's policy TLM which generates next letter the previous alphabet input. Here, Ta will be penalized at the third token.
Figure 2:
the right sub-sequence AB is also
E ADDITONAL STUDY
STG-MLE. Here, evaluate the MLE version of STG (denoted as STG-MLE) which is trained by MLE for the mixture policy (.(-) n T = () + Tz(i, = 1)Ta(-) similar to copy mechanism (Gu al. 2016). In few-shot training, the explicit reduce the fine-tuning loss especially when the adapter is light since the adapter can focus only on the task-relevant part in generation. STG-RI learns to do this naturally by stochastic policy sampling if the policy selector is initialized to perform uniform sampling. On the other hand, STG- MLE can be easily collapsed select only a task-specific policy (i.e. i, 1). This because the gradient flows the additional model only and, unlike STG-RL, there is no chance to exploit diverse paths during training in the teacher forcing manner. As shown Figure STG-MLE starts from the same point of STG-RI. but it collapsed to Non-STG-MILE.
of PLM logits efficiently
Learning Curve. It is well known that the RL-tuning resolves the exposure bias of ML.E-tuning We can expect that additive learner of MLE would be affected by the exposure bias well, and the RL objective for additive learning resolves it. Here, we present some learning curves training in experiments. As shown in Figure 4 the learners of MLE terms of Perplexity. PPL) and exposure bias (in terms of Score). On the other hand, the learners of RL were less effected by the problems. We can find that the STG models (denoted STG-RL) are superior to the others from the perspective of the score.
from to have overfitting (in
Effectiveness of Selector. Here, investigate the effectiveness of the selector n, of the STG. We compare Fixed Selection against the Dynamic selection. In the fixed selection, the probability of selecting the PLM's policy LLM is fixed to π. (Or(s:) = 1 v. (1elst). We measure the performance with respect to ( (1) = e where e is a constant. The selection will be uniformly random when 0. the performance will be equivalent the performance of the PLM without additive learning. Figure shows that the input-dependent dynamic selection by our STG
c 0.5. and when
WWe add "-RL." to the STG to distinguish with STG-MI.E in this context. 'The curve for Data-to-Text not presented since there actual validation set.
15

---

Under review as a conference paper at ICLR 2022
outperforms the fixed selection with any We find that how T. works for each task. For instance, in QA task, the first few tokens of an answer may decide the quality of generation (i.e. "yes" or "no" in binary QA). Therefore, optimal strategy of the STG might be producing the first few tokens sampled from the task-specific Ta and the remaining tokens from the PLM TLM- The curve supports this interpretation since the score decreased such a strategy as shown from the generated answers in Table and 12 In Data-to-Text, the BLEU score is increased is close while the ERR score decreased. This fact supports the results The T. learns to balance between the BLEU and
is close to Our STG learns
of NE(miz) models as discussed in Section 4.6 ERR.
QA
Summarization
0.4750
Fixed
STG
0.4725
random
0.22
Score
0.4700
PLM
Score
0.21
0.4675
0.4650
0.20
0.00
0.25
0.50
0.75
1.00
Fixed Selection Prob
0.00
0.25
0.50
0.75
1.00
Fixed Selection Prob
Data2Text (TV)
Data2Text (TV)
0.300
10.0
BLEU
ERR(%)
0.295
9.5
0.290
9.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Fixed Selection Prob
Fixed Selection Prob
Figure 3: Dynamic selection Fixed selection.
16

---

Under review as a conference paper at ICLR 2022
QA Train
QA Valid
14
PPL
PPL
Non-STG-MLE
Non-STG-RL
8
STG-MLE
STG-RL
12
200
600
0
200
400
600
0.38
SCORE
SCORE
0.36
0.45
0.40
0.32
400
600
training step
training step
Summarization Train
Summarization Valid
90
121
85
PPL
PPL
Non-STG-MLE
80
119
Non-STG-RL
STG-MLE
STG-RL
118
75
200
400
600
200
400
600
0.240
0.20
SCORE
0.235
SCORE
0.230
0.225
0.18
0.220
200
400
600
600
training step
training step
Figure 4: The learning curve. The Perplexity (PPL) and Score of each task (e.g. Rouge-L for Summarization) are measured the 0.5% sho- train and the valid set.
17

---

Under review as a conference paper at ICLR 2022
F GENERATED SENTENCE EXAMPLES
Here, we show generated sentence examples for each task (see Table 10 for Data-to-Text, Table and Table 12 for Question Answering and Table and Table [4 Summarization.). The tokens sampled from the task-specific policy To are presented in red.
Table 10: Data-to-Text examples from TV of FewShotWOZ dataset.
Data
reommenndnname-kratos 31; type=television; powerconssuptionn3] 1.4 watt; accessorics=remote control and european warranty; color=mat black bezel with mat black frame stand)
Ground-truth
the kratos 31 is great television. it only consumes 3 31.4 watts, comes with remote control and european warranty and has a stylish mat black bezel with mat black frame stand..
PLM
the kratos 31 features a mat black bezel with mat black frame stand and european warranty.
Non-STG-MLE
the kratos 31 features a mat black bezel with mat black frame stand and european warranty
NE(miz)-MLE
the kratos 31 television has 31.4 watt power consumption, remote control and european warranty.
STG
the kratos 31 television has 31 watt power consumption. it comes with remote control and european warranty with a color of mat black bezel with mat black frame stand.
Data
inform(name=triton 52; type=television; family=17; audio=cevo stereo; screensize=47 inch)
Ground-truth
the triton 52 47 inch sereen television from the product family and has cevo stereo audio.
PLM
the triton 52 television 47 inch television in the 17 family and with a cevo stereo.
Non-STG-MLE
the triton 52 television is in the 17 family, has 47 inches of screen, and uses cevo stereo sound capabilities.
NE(miz)-MLB
the triton 52 television is in the 17 family, it has 47 inches of screen, and 47 euro price.
STG
the triton 52 television is 47 inch television in the 17 family and has a cevo stereo.
Data
inform(name-atiis pricerange=cheap)
65;
type=television;
resolution=1080p:
Ground-truth
the attis 65 television is in the cheap price range and has nice resolution of 1080p.
PLM
the attis 65 television in the cheap price range has a 1080p resolution and a pricerange of cheap.
Non-STG-MLE
the attis television has a 1080p resolution, is in the cheap price range and has nicam stereo.
NE(miz)-MLE
the attis 65 television is 1080p television in the cheap price range and the cheap price range with nicam stereo.
STG
the attis 65 television is in the cheap price range and has 1080p resolution.
18

---

Under review as a conference paper at ICLR 2022
Table 11: Question Answering examples.
Passage
zygote: : this cell results from the union of an oocyte and a sperm a zygote is the beginning fertilized ovum refers to secondary oocyte that is impregnated by a sperm when fertilization is complete. the oocyte becomes zygote. (emphasis
new human being ( (i. an embryo the expression
added
Query
what a human zygote
Ground-truth
the beginning of a new human being.
PLM
the expression fertilized ovum refers secondary oocyte that is impregnated by a sperm
NE(miz)-MIA
the expression fertilized ovum refers to secondary oocyte that is impregnated
by a sperm when fertilization complete. the oocyte becomes zygote.
NE(miz) RL
the expression fertilized ovum refers to secondary oocyte that is impregnated by a sperm when fertilization complete the oocyte becomes zygote. it is the beginning a new human being..
STG
Passage
sometimes the terms reactive airway disease and asthma are used interchange- ably. but they are not the same thing often, the term reactive airway disease is used when asthma is suspected, but not yet confirmed. reactive airway disease children is general term that does n't indicate a specific diagnosis it may be used to describe history coughing, wheezing or shortness of breath triggered by infection. these signs and symptoms may or may not be caused by asthma
Query
symptoms of reactive airway discase in adults
Ground-truth
coughing. wheezing shortness of breath triggered by infection
PLM
symptoms reactive airway disease in children a general term that does nú indicate specific diagnosis.
NE(miz)-MLE
symptoms of reactive airway disease in adults are a general term that does ní indicate specific diagnosis.
NE(miz) RLL
yes, reactive airway disease in children is general term that does né indicate a specific diagnosis.
STG
coughing, wheezing or shortness of breath triggered by infection.
Passage
sulfate is an anion composed of a sulfur atom and four oxygen atoms overall charge of this anion is -2. the molecular formula for sulfate is so 4 2-. there. two oxygen atoms are bonded to the sulfur atom through double bonds and other two bonded to sulfur through single bonds. the oxidation state sulfur in sulfate is +6.
Query
is sulfate sulfur and oxygen bonded
Ground-truth
yes. sulfate sulfur and oxygen bonded. no answer present..
PLM
NE(mix)-ML.X
yes sulfate is sulfur and oxygen bonded the sulfur atom through double bonds and other two are bonded to sulfur through single bonds.
NE(miz)-RL
yes sulfate sulfur and oxygen bonded to the sulfur atom through double bonds and other two are bonded to sulfur through single bonds..
STG
yes, sulfate is sulfur and oxygen bonded to each other through double bonds and other two bonded to sulfur through single bonds.
19

---

Under review as a conference paper at ICLR 2022
Table 12: Question Answering examples.
Passage
gryffindor house is one of the four houses at hogwart into which the students are sorted when they start their first year . gryffindor is the hogwarts house of founder godric gryffindor gryffindor house places high value on traits such courage chivalry and nerve during harry time hogwarts the head of gryffindor house was minerva
Query
what kind of house gryffindor
Ground-truth
gryffindor house is one of the four houses at hogwart into which the students are sorted when they start their first year.
PLM
gryffindor house one of the four houses at hogwart into which the students are sorted when they start their first year.
NE(nia)-MLL
no answer present.
NE(miz)-Re RL
no answer present.
STG
gryffindor house is of the four houses hogwarts into which the students are sorted when they start their first year.
Passage
the median home value in louisville is $ 119,500 louisville home values have gone up 1.8 % over the past year and zillow predicts they will rise 3.0 % within the next year. the median list price per square foot in louisville is $ 125. the median price of homes currently listed in louisville is $ 180,000 the median rent price in louisville is $ 1,050 foreclosures will be factor impacting home values in the next several years in louisville 0. homes are foreclosed per 10,000)
Query
average per therm price in louisville ky
Ground-truth
the median home value in louisville is $ 119,500.
PLM
the average per therm price in louisville is $ 119,500.
NE(niz) MMLE
$ 125 per square foot
NE(miz)-RR RRL
the average per square foot in louisville is 125.
STG
the median home value in louisville is 119,500
Passage
the brain is made up of more than just nerve cells ( neurons ). although there are about 100 billion neurons in the brain, there may be about 10 50 times that many glial cells in the brain neurons have two processes called axons and dendrites glial cells have only one 2 neurons can generate action potentials ... glial cells not however glial cells do have resting potential neurons have synapses that use neurotransmiteers... glial cells do not have chemical synapses..
Query
number of glial cells vs neurons
Ground-truth
10 50 times.
PLM
no answer present.
NE(miz)-
$ 10 to 50 times that many glial cells in the brain...
NE(miz)-Ri
answer present.
STG
10 50 times
20

---

second deputy chief john roach all three deaths occurred between 9 and 9:20 a.m. man in his 60s fell hit his head two men others, ages 36 and 26, Under review as a conference paper at ICLR 2022
Table 13: Text Summarization examples.
Article
-Irb- cnn -rrb- three runners died sunday during the detroit free press/flagstar marathon in detroit, michigan, police told cnn. an emt vehicle is at the scene sunday detroit after three runners collapsed at a marathon. all three deaths occurred between 9 and 9:20 a.m. second deputy chief john roach said. man in his 60s fell and hit his head, roach said. the cause of the fall unknown. the man pital, where he was pronounced dead. two other men, ages 36 and 26, also collapsed during the race and were pronounced dead at the hospital, roach said. all three collapsed near the end the race, he said. witnesses describe scene the weather at the time was overcast, roach said, with temperatures in the low 40s. [...]
transported detroit receiving hos-
Ground-truth
collapsed. race was detroit free press/flagstar marathon detroit, michigan
PLM
three runners collapsed at marathon in detroit, police say. the cause the fall is unknown
Non-STG-MLE
three runners collapsed a marathon sunday, police say. the cause of the fall is unknown police say.
Non-STG-RL
three runners collapsed a marathon sunday, police say the cause of the fall is unknown police say.
STG
three runners collapsed at a marathon in detroit michigan all three deaths occurred between 9 and 9:20 a.m. et man in his 60s fell and hit his head, police say.
Article
london, england -Irb- enn -rrb- - up to 1,000 human rights campaigners demonstrated saturday in front of no. 10 downing street, the official resi- dence of british prime minister gordon brown, calling on the british govern- ment to demand that full democracy be restored in pakistan. jemima khan, center, ex-wife for former pakistani cricket star imran khan, joins protesters in london. protesters waved placards and chanted in support of the resigna- tion of pakistani president pervez. musharraf, a week after he imposed a state of emergency the country. the crowd of demonstrators massed behind barriers and included jemima khan, the ex-wife of former pakistani cricket star turned politician imran khan. the demonstrators carried placards saying free the innocent and end musharraf's regime and waved pakistani flags. imran khan, who heads the the movement for justice party, has been under house arrest since the emergency declaration. his ex-wife delivered a petition to doorman at downing street, calling on britain use its influ- ence to ensure that all institutions elections originally scheduled for early next year. the petition also [...] human rights campaigners demonstrate in front of 10 downing street protests urged uk government to demand full democracy restored in pakistan cricketer turned politician imran khan & ex wife jemima among protesters..
in place well in advance of pakistani
Ground-truth
PLM
pakistani president pervez musharraf has been under house arrest since the emergency declaration his ex-wife delivered petition doorman downing street.
Non-STG-MLE
pakistani president pervez musharraf has imposed state of emergency in the country he has been under house arrest since the emergency declaration pakistani president pervez musharraf has imposed state of emergency in the country he has imposed a state of emergency the country since last week
Non-STG-RL
STG
the official residence of british prime minister gordon brown. hundreds of protesters demonstrate in front of no 10 downing street. the petition calls for full democracy in pakistan.
21

---

Under review as a conference paper at ICLR 2022
Table 14:
Text Summarization examples.
Article
-Irb- enn -rrb- spoiler alert! it's not just women getting cloned. that was the big twist at the end of orphan black's second season. the kickoff to the new season leads the list six things to watch the week ahead. 1. orphan black, 9 p.m. saturday, april 18, bbe america. the cloning cult sci-fi series remains one of the most critically acclaimed shows on tv, thanks in large part the performance of tatiana maslany, who has taken on least six roles on the show so far, including a newly introduced transgender clone. maslany told reporters this week that we can expect even more impressive scenes with multiple clones, [..]] 2. turn: : washington's 's spies, " 9 p.m. et, monday, amc. the series about spies the early days of the revolutionary war returns with a new subtitle, washington's 's spies, and a new monday night time slot. series star jamie bell told cnn what we can expect in the second season. this year we have lot more battles [...] 3. ggame of thrones, p.m. et, sunday, hbo. the world westeros returns for a fifth season one of the biggest season premieres of the year. click here for more on what to expect. [...]
Ground-truth
critically acclaimed series orphan black returns turn washington & spies starts a second season game of thrones is back for season five. series returns with new subtitle. "washington spies, " "and monday night time slot series star jamie bell told enn what we can expect in the second season
PLM
Non-STG-MLE
orphan black returns with new subtitle and new monday night time slot series star tatiana maslany blown away by the response to the series so far
Non-STG-RL
orphan black returns with new subtitle and new monday night time slot. series star jamie bell says the series will bring more recognition to the culper spy ring.
STG
the new season of orphan black " premieres on saturday the 18th " game of thrones returns for a fifth season
Article
energy drinks can pose a real problem for children, according to a new study from the american heart association. researchers found that 40 % of the 5,156 calls to poison centers for energy drink exposure involved children under age most of the cases, the parents didn't 't know the children had gotten hold of an energy drink. many of the calls reported the children were expe- riencing serious side effects, such as an abnormal heart rhythm, or they were having seizure. the study is being presented at the american heart associa- tion's scientific sessions this week. study caffeine affects teen boys more. study author dr. steven lipshultz has handled cases involving children who became sick after consuming energy drinks. while studies about the impact of caffeine children are limited, lipshultz, [...]
Ground-truth
40 % of energy drink exposure calls to poison control centers are for children under study authors believe the number is under-counted. the fda does not set a caffeine limit for children.
PLM
drug calls poison centers for energy drink exposure in most cases the parents did n't know the children had gotten hold of an energy drink. many of the calls reported the children were experiencing serious side effects, such as an abnormal heart rhythm
Non-STG-MLE
drug calls to poison centers for energy drink exposure in most cases, the parents did n't know the children had gotten hold of an energy drink. many of the calls reported the children were experiencing serious side effects such as an abnormal heart rhythm
Non-STG-RL
this a very concerning finding. " "dr. laurence sperling says. the american academy of pediatrics recommends children consume no caffeine.
STG
drug calls poison centers for " energy drink exposure " nearly 40 % of calls to poison centers for energy drink exposure involved children under age 6. study caffeine affects teens more.
22


Reviews:
[{"review_id": "xGSeWcSWbqQ", "reviewer": "Reviewer_mErH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The reformulation of the proposed idea (selector to switch generation between a base PLM and a task-specific PLM) as a reinforcement learning problem is interesting. However, I have some major concerns on the technical clarify/rigor and experiment presentations that hopefully authors could clarify and help me better understand your contributions. \n\n---\n\nOn the technical side:\n- during optimization, the authors seem to be computing the reward at each time step (equation (9)) of the generation instead of on the whole-sequence level. However, as far as I know (I may be wrong), when RL methods are applied to NLG, the reward is typically computed on the whole-sequence instead of on the sub-sequence at each time of the generation. If the reward is computed on the partial generated sequence, how is it actually implemented? Why not compute the loss on the entire sequence which I think makes more sense for the BLEU and ROUGE metrics because both are operated on the whole-sequence level?\n- What are the instantiations of $Q$, $A$, and $V$? From my understanding, $Q$ and $V$ are actually not computed and $A$ equals the metric score such as BLEU score.\n- The selector module performs hard sampling, i.e., outputting either 1 or 0, in equations (5), (6), and (7). However, during _actual_ optimization, the selector performs soft sampling, i.e., outputting a probability between [0,1]. Why is there a mismatch? If the softmax is required as an approximation to the hard sampling, why do the authors need it? From my understanding, if the authors choose to use RL, then the loss function does not necessarily need to be differentiable, so approximating $\\pi_s$ as softmax may not be necessary? Which implementation (hard or soft sampling) did the authors use in the experiments? \n- What is the \"penultimate representations\" in the paper? Is it the last layer hidden states of all previously generated tokens, the last layer hidden state of the last generated token, or all layer hidden states of the last generated token?\n---\nOn the experiment side:\n- Overall I think more discussion and analyses are needed. Without these additional results, it is difficult to get insights into why and how the method works. The authors provide some discussions but I think it is insufficient; these claims should ideally be supported by empirical results. More below.\n- My main concern is Table 2, which does not seem to make sense to me. If I understand correctly, for the ROUGE metric, the higher number the better, and it cannot be a negative number. In Table 2, PLM, which is a baseline, achieves far better results than any other methods under comparison by a huge margin, including the authors own proposed method. Moreover, one ROUGE result is negative. What happened?\n- There is another baseline that I think the authors should compare but did not. How about setting $A$ in equation (9) to 1.0 and just run MLE using equation (9) as the loss? To me, $A$, which results from the authors' RL formulation of the optimization problem, is in practice simply a scaling factor to the usual MLE loss if they did not involve RL. This comparison would suggest how useful is RL training compared to MLE.\n- There are different BLEU metrics relying on unigram, bi-gram, etc, and similarly for ROUGE metric. How does the different choices of these variants of the metrics as the reward function impact results? \n- What does the selector do and when does it switch? Some more insights would be really useful. For example, the authors can provide some results on which word in the generation is generated by the frozen PLM $\\pi_{\\rm LM}$ and which word is generated by the fine-tuned policy $\\pi_a$.", "summary_of_the_paper": "The authors proposes a method to improve large pre-trained language model's ability in few-shot language generation. The main idea is to freeze the large pre-trained language model (PLM), fine-tune a copy of this PLM on the task at hand, and use a selector to switch between generating the next token from the freezes PLM or the fine-tuned PLM. Instead of the commonly used maximum likelihood estimation, the authors cast the optimization problem as a reinforcement learning (RL) problem using the task evaluation metric such as BLEU or ROUGE as the reward. The authors perform experiments on few-shot data-to-text and text summarization settings, comparing to a few baselines (fine-tuned PLM and a few variants of the proposed method).", "summary_of_the_review": "Because of my concerns in the rigor and clarify in both the technical approach and the experiments, the paper needs further development and is not yet ready for publication at ICLR. Therefore, I unfortunately cannot champion acceptance for this paper at this time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pzGqhE3Ecz8", "reviewer": "Reviewer_K8FA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall speaking, this paper is somehow not qualified enough. Below are the main points:\n1. The algorithm is the RL-based method, the difference is that the author introduces a policy selection policy to decide the generation policy. This is straightforward and not new to the NLP generation community. The novelty and technical contribution are truly limited. If this is not the case, the results are also not surprising to see the improvements. Therefore, it is hard to see the shining points.\n2. Even the method is not interesting, the experimental comparisons are also not enough. The authors do not give a comprehensive study of the results and the comparison with other papers or baselines. This is again, not qualified. \n3. The selective policy is introduced in a self-learning way, which is automatically learned by the same hidden states of generation tokens. Therefore, a study of the policy selection policy is required. However, it is missed, also for other parts in the model. \nThe authors are highly encouraged to make a major modification of the paper. ", "summary_of_the_paper": "This paper proposes a selective token generation method for additive learning under pre-trained language models. Specifically, the authors introduce a selective module to decide the generation policy of tokens, either from the LM generation policy or the RL generation policy. The experiments are conducted on data-to-text task and text summarization task. Results show the improvements over the traditional baselines. ", "summary_of_the_review": "The paper does not give enough information to claim the importance of this paper, neither the method nor the results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "dFTLrdJIQZZ", "reviewer": "Reviewer_3Ee3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well written, barring some unnecessary complexity in the description of the technique (details below), and the paper does a good job iterating through details of the method and considering some alternatives. However I find the results unconvincing, primarily in their improvement over the base method. There are also some questions I have with the proposed method and how it is supposed to solve the overfitting issue like the paper claims. As summary, I would say the paper is clear and the proposal is not something I am familiar with in the literature, but the improvement seems marginal.\n\n**Detailed Comments:**\n\n* **<Introduction>** You describe large-scale training data as needing high quality annotations, but much of the larger models are trained on unsupervised text. I think this should be clarified. \n\n* **<Section 2.2>** Can you explain the footnote?\n\n* **<Section 2.2>** How exactly is the approach motivated by auxiliary training?\n\n* **<Sections 2/3>** Equations 6 and 7 are clear and convey the key aspect of the proposed method well (absent the RL loss), however the majority of the other questions seem more dense and wordy than necessary, and are hard to follow at times (what is G in equation 1?). I think the sections introducing the method could benefit from more clear and concise mathematical formation (i.e. equation 9 can be re-written with less repetition).\n\n* **<Section 3>** The motivation for STG is to prevent overfitting., I understand why wanting to rely on the pretrained logits mostly and only sometimes use the finetuned ones would help ameliorate overfitting, what I don't understand is exactly how this instantiation helps. Why does the RL not learn to always use the finetuned logits downstream, since surely that would help on the finetuning loss?\n\n* **<Section 4>** Prompt engineering is brought up, how well does that compare to the other methods discussed in terms of finetune performance?\n\n* **<Section 4>** Why not try NE(sum)?\n\n* **<Section 5>** Why does the hierarchical policy reduce the action space?\n\n* **<Table 5>** STI should be replaced with STG.", "summary_of_the_paper": "This paper proposes STG, a method of adding a task-specific head to a pretrained language model, and finetuning it with RL along with a policy selector that chooses when to add this policy-specific logits to the pretrained logits for a given downstream task, targetting the low data regime.", "summary_of_the_review": "Overall the paper is well written and I am not aware of this exact method being used in the literature, but the improvement strikes me as marginal in results with unclear motivation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5s28ObMu1Hv", "reviewer": "Reviewer_PchS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The method is novel and brings knowledge of reinforcement learning to the challenges of NLP.\n- Paper is overall well written and easy to follow\n\nWeaknesses:\n- Gains with the method are limited, especially in summarization. Breath of results is also limited as the authors consider only two tasks\n- I would have appreciated seeing human evaluations. It is not obvious from the few samples in the appendix that the method leads to qualitatively superior generations.\n- Beyond the numerical gains, it is not clear why the method leads to better generations. For instance, it is not clear how the token predictions at each step differ between the baseline and the introduced method?\n\nQuestions:\n- The paper seems to be built on the hypothesis that fine-tuning PLMs in low data regimes leads to overfitting. Yet, it is not obvious from the numbers for PLM in line Table 1 and 2. Thoughts?\n- Another core hypothesis seems to be the intuition that there are “task-relevant parts in sequence generation”. I couldn’t find any definition of what “task relevant parts” are. How does it differ from learning the format of the task?\n- Why use the fine tuned model as an initialization for the model with adapters (non-stg). It seems related but I don’t understand this sentence: “this fine-tuning phase can be skipped when the advanced networks were used in the PLM such as External Encoders or Adapters which can cover the large domain shift”.\n- Table 2: are the numbers for non-stg, ne and stg rouge points or % difference with the PLM baseline?\n- Could you elaborate on the robustness of the method? For instance, what is the standard deviation between multiple runs?\n\nAdditional Feedback:\n- Typo - Equation (1) - What is \\theta_g? I think you mean \\theta_a.\n", "summary_of_the_paper": "The paper is interested in the problem of exposure bias of sequential text generation: tokens are drawn from the data during training, while during inference, the tokens are sampled from the model’s distribution.\nSpecifically, the authors consider this problem in the context of few-shot learning, where only a few labeled examples are available.\nThe authors introduce a method that combines task-specific adapters and reinforcement learning-based training: at each token prediction, the model chooses between the distribution of the original pre-trained LM and the distribution induced by the task-specific adapters on top of the pre-trained LM. The next token is sampled according to the selected distribution.\nThe authors observed gains in two tasks (data-to-text generation and summarization).\n", "summary_of_the_review": "The introduced method is somehow novel, but the experiments have very limited breadth and analysis don't go beyond discussion.\nGains are also limited, and it's not clear looking at the current results that the solutions proposed really answer the problems exposed in the introduction (exposure bias, task-relevant parts, overfitting in low-data regime).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a reinforcement learning inspired algorithm to train task-specific adapters to adapt pretrained language models for downstream tasks. The paper attempts to tackle an important problem. All reviewers have concerns about whether the results are strong enough to justify claims made in the paper. I appreciate revisions that have been done by the authors during the rebuttal period. However, I believe that the paper is still below the bar for ICLR. I recommend rejecting this paper.
Decision: Reject