Teacher Forcing Recovers Reward Functions for Text Generation
Authors: Yongchang Hao
Yuxin Liu
Lili Mou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Text Generation
Natural Language Processing
Reinforcement Learning

Abstract: Reinforcement learning (RL) has been widely used in text generation to alleviate the exposure bias issue or to utilize non-parallel datasets. The reward function plays an important role in making RL training successful. However, previous reward functions are typically task-specific and sparse, restricting the use of RL. In our work, we propose a task-agnostic approach that derives a step-wise reward function directly from a model trained with teacher forcing. We additionally propose a simple modification to stabilize the RL training on non-parallel datasets with our induced reward function. Empirical results show that our method outperforms self-training and reward regression methods on several text generation tasks, confirming the effectiveness of our reward function.


Teacher Forcing Recovers Reward Functions for Text Generation
Yongchang Haot, Yuxin Liu', Lili Moutt
Deptt. Computing Science, Alberta Machine Intelligence Institute (Amii)
University of Alberta, Canada
1Canada CIFAR AI Chair, Amii
kyongchal,yliul/n yliull)euallbetta ca, doublepower
wew.epowwer.mmuggaaa... www@mmaiumamm.m.mmmmmmmmmmm com
Abstract
Reinforcement learning (RL) has been widely used in text generation to alleviate the exposure bias issue to utilize non-parallel datasets. The reward function plays an important role in making RL training successful. However, previous reward functions are typically task-specific and sparse, restricting the use of RL. In our work, we propose a task-agnostic approach that derives step-wise reward function directly from model trained with teacher forcing. We additionally propose a simple modification to stabilize the RL training on non con-aaallll datasets with our induced reward function. Empirical results show that our method outperforms self- training and reward regression methods on several text generation tasks, confirming the effectiveness our reward function
Introduction
Teacher forcing the common training method for text generation models. Although this practice has been widely applied ? B 588, there are two main issues: 1) Teacher-forcing training data-hungry because parallel datasets are usually expensive to obtain. On the other hand, there numerous unlabeled, non-parallel datasets available. This poses urge to efficiently exploit non-parallel data. 2) Teacher forcing introduces a discrepancy between training and inference because the model learns predict the next word based the partial groundtruth reference during training, whereas in inference the model predicts the next word based on its self- self-generated previous words. This undesired discrepancy is known as exposure bias 44
59
To address the first problem, a straightforward method to generate pseudo-parallel sentences for data augmentation, such as self-training 22) sequence-level knowledge distillation 23 and back- tack-translation
However, the exposure bias remains such cases.
To address the second problem, the model should be trained on self self-generated sentences. Common solutions are often based on reinforcement learning (RL). text generation, however, there does not exist a naturally defined reward function for RI. Researchers have proposed various heuristic scores as the reward, such as BLEU 3 for translation and ROUGE [3 for summarization. These reward functions are task- speccfic and not generalizable to other tasks. Further, these rewards require parallel data, failing to address the first problem above; they are typically sparse (only non-zero the end of sentence), making RL training difficult.
The goal this paper address these two problems in one framework with learned, dense reward function. Our approach has two steps: we first train a sequence- suueenee-o-ssquence (seq2seq) model on the parallel dataset and induce a reward function from the model. Then, we apply RL on non-parallel data based on our induced reward function.
Our code is publicly available at https:://gittub..omm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Our method is task- task-agnostes agnostic and does not require handcrafted engineering or heuristics. Further, our reward function provides dense (step-wise) training signals, which makes RL training much easier than sparse rewards. Additionally, the reward function derived from the seq2seq model does not directly participate the generation, which allows the model to explore based on its own prediction and thus alleviates the exposure bias.
We conduct experiments on dialogue generation and paraphrase generation. The empirical results suggest that our method leads to better performance compared with several baselines, including self-training and task-specific heuristic reward learning, on both tasks. This confirms the effectiveness and generality of our framework.
2 Approach
Our approach trains the seq2seq model on non-parallel data with reinforcement learning, whose foundation is the Markov decision process (MDP). In this section, we first introduce the MDP formulation for text generation. Then we describe our method to derive the reward function from seq2seq model trained by teacher forcing. Finally, we describe the policy gradient method used in RL training with our induced reward function.
2.1 Reinforcement Learning Formulation of Text Generation
Text Generation Markov Decision Process (MDP). We formulate the text generation process as an (undiscounted) MDP, which can be represented as a tuple (S, A. T, r). At every step, decision E made based on its state E S. The transition dynamic T(s'ls, a) is the probability of the next state being 8', given the current state and the action A function : S x A - I defines the reward based on a state and an action.
Typically, the decision making is assisted by a policy æ, which is a predicted distribution over actions and is trained to maximize the expected total reward, also known as an action value function:
H Σ
9" as)
==
E as SENTEE (sc) (8e:a))
rs az)ls 01 =
where H is the number of steps. Theoretical results show that the optimal policy x' satisfies the Bellman optimality equation:
g" (+,,))=((saa)+
ET(''s max ot (s',a))
(2)
a'ES
For text generation, the MDP state can be defined as the partial generated sequence yer (y1, 1)),)), and the action as the next token yt in the vocabulary V. The transition dynamic T(Iss, a) here is deterministic, since every state-action pair (y we) leads to unique state y for the next step.
In previous RL-based text generation, there lacks a naturally defined reward function r(s, a). While researchers have applied various heuristics the reward 5500 they suffer from several shortcomings (e.g., sparsity and task specificity) as mentioned in Section To address these problems, we propose induce reward function for text generation tasks a principled approach by inverse reinforcement leamiing
Inverse Reinforcement Learning (IRL). The goal of IRL is to learn a reward function r(s, a). Especially, we wish the resulting action value function g computed by Eqn. ① could satisfy q a) > q(s, a') for every a' E A and every (s, a) pair in the training set D. In other words, the decisions in D are made greedily by argmax, q(s, a) given any state s. Unfortunately, Ng and Russell (6 show that this is an ill-posed problem since the desirable reward function is not unique. Therefore, we follow common assumption
39 to resolve the ambiguity:
Assumption 1. Given an action value function g, the policy æ takes the form of (a(s)) = exp(g(s, a))/ En exp(g(s,a'))

---

reward learning is difficult and this assumption does not directly yield reward function due the stochastic state transition [ a). However, our insight that the transition is deterministic for text generation tasks, and thus we may utilize Assumption to induce an action value function q. and then a reward function r, from some learned policy n, as explained in the next part.
In traditional IRL 4 4 69
2.2 Teacher Forcing Recovers IRL
One of main contributions is that show the seemingly complicated reward learning in Section 2.1 can be recovered by teacher forcing, the de facto common practice of supervised text generation. Our discovery leads convenient approach that derives a step-wise reward function simply from general seq2seq models, without the need for task-specific heuristics. This makes RI more general for text generation, and our step-wise reward largely simplifies RI. training.
Maximum Likelihood Estimation (MLE) for IRL. Following Assumption D we let the policy Ta (-1s) exp(g_(s..) where qu is parameterized action value function. Under such policy, the probability each trajectory s == (($1, a1)..... (5)= (i=))) in the dataset is given by the trajectory distribution p The likelihood of the dataset is given by
(D = II
PFaH (T). TED
(3)
Teacher-Foo Teacher-Forcing Training. minimize the loss:
For text generation, the standard forcerg seq2seq training is to
LTF(w;)) = lv1
Po Pw (311
(4)
BED 1=11
where the predicted probability of the next token being pa (ulp Evev fo (u for
exp( (xc(e((ee..))
the logit function for with parameters In seq2seq training, an additional input æ may be added to the conditional probabilities but is omitted here for simplicity.
The below theorem shows their equivalence up to an additional constant.
Theorem 1. Suppose the value function q in Eqn. and the seq2seq model f in Eqn. have the same parametrization w, we have
LTF(W; D) log PIRL + const.
(5)
Proof. For the MLE of IRL under Assumption D the lonescu-Tulcea theorem 22 asserts that there exists unique trajectory distribution PT satisfying
(s1) = u(s1);
P7 (51, ................................ St, a.) PT (1)((())))
P (s), [[ss,,a,..... St, a, St+1) = (s) (I.............................. St, a, ( $2, a.)
for any t 2 1, given the initial state distribution μr, transition probability T, and policy The likelihood can thus be factorized by the multiplication of μ. T. and m:
[r]
PIRt. (Dlw) II P (T II
((s)) (a1 (s1) II 2
T(sels at- a (uelst)
TED
TED
As mentioned, text generation has deterministic transition, i.e., T()1ls,)) for the next state = 8 + [a]. Taking the terms out, we have
log (D|w) = log II Tyw (a. (1) - log II p(s1): TED+=1
TED
where the second term constant in terms of w. Section 2. text generation modeled as
an MDP with yer and yt- Therefore, the first term of Eqn. is the same Eqn. 4 under the parametrization Fg.. = Pws concluding the equivalence between MLE for IRI. and teacher-forcing training of a seq2seq model.
3

---

Inducing the Reward Function. Theorem | shows that seq2seq training with teacher forcing actually learns an IRL model. Thus, we may derive reward function assuming the action value function is well trained:
r(s, a) = (s,
Σ T(s'ls, a) max gu (w( a'EAA
max fu(s + [a]: a')
(7)
d'EA
e'ES
where the first equality is due to the Bellman optimality condition
2); the second equality is due to the parametrization of 9u = fw and the deterministic transition T (s'ls, a) -1 for s' being the concatenation of the prefix and token a.
Remark 1. It is easy to notice that (s,-) may be arbitrarily shifted by constant without changing Tu. This also shifts the derived reward r8, a) by c, Cs+la]- However, it does not affect the optimal policy. We will prove this in Theorem 3 after introducing policy gradient methods.
Our use of the Bellman optimality condition is different from classic RI., where the reward is well- defined and the action value function is thus learned [55. Instead, we induce the underlying reward assuming the action value function is known (given by Assumption The following diagram shows the whole process of our derivation.
Teacher Forcing
Assumption
0
Eqn.
D
r
In real-world applications, the learned action value function might be imperfect; in this case, we may bound the eгrоr of our induced reward with the following theorem.
Theorem 2. Let r* be an underlying true reward function and q' be the corresponding optimal value function. Given approximate value function q, denote by r the reward function derived from Eqn. 7 Then, we must have ||| [ byr-"1, 1l bounded by O(llq - q" llo)) Here, || lloo takes the
maximum absolute value over all s Sand a A
Proof. See Appendix A
2.3 Periodically Synchronized Behavior Policy in Policy Gradient
In text generation, neural network can be viewed policy that predicts the word distribution given the state of a decoding step. The reward induced from Section 2.2 can be used to improve the policy through RL. To stabilize training, we propose variant of off-policy policy gradient methods 10) with a periodically synchronized behavior policy.
Our RL. training adopts the off-policy REINFORCE 633 as the backbone of our algorithm. Let T be the model policy (i.e., the model's prediction) to be optimized, and m. be the behavior policy (i.c., the sampling distribution during training). Through importance sampling, the gradient of the expected total reward with respect to can be obtained by the off-policy policy gradient theorem 10
Σ r(s ((s) = Σ (eg a log (a(((s))) Te
V E
(8)
where P: := TT (sr) To (ar(st) is the importance weight, and (5t, at) := Eize r(s;, a;) is the total reward the trajectory. In practice, off-policy REINFORCE (TT * Tb) is more exploratory than the on-policy one (TT #b). since the model policy The would become more concentrated during optimization and does not explore much, whereas no is typically chosen to cover more trajectories. However, Degris al. I1O adopt fixed behavior policy To. which does not perform exploitation according to the current model policy, The lack of exploitation might lead to less informative training.
To balance exploration and exploitation, we would like the behavior policy to be close to the model policy but stay exploratory at the same time. We thus propose periodically updating schedule, where the behavior policy is frozen for long period to encourage exploration but keeps track of the current model policy to enhance exploitation. Particularly, we synchronize the behavior policy with the model policy for every & gradient updates of the latter (e.g., k 5000). Our remedy is simple method overcoming the instability of REINFORCE. shares a common ground with number of policy gradient methods like the proximal policy optimization (PPO) 48), especially in that both

---

methods involve multiple updates with fixed behavior policy. As the main contribution of this paper reward induction, we resort to this simple fix and leave the mathematical connection as an interesting future direction.
Algorithm summarizes approach. Our implementation is able execute the loops in parallel, which speeds up the training process. Our periodically synchronized behavior policy further enables us parallelize sampling and model updates reduce the awaiting time.
2.4 Application to Semi-Supervised Learning
Our approach naturally aligns with the paradigm of semi-supervised learning, as involves training a seq2seq model to induce the reward function, which requires (at least small volume of) parallel data Dp. Additionally, we assume there a non- non-parallel dataset D,, containing input sentences only for RL training with the induced reward.
Our semi-supervised approach consists of two stages. We first train a seq2seq model fw on the parallel dataset Dp to induce the reward function by Eqn. (7). The procedure is described in Section 2.2
The reward function then facilitates RL training on the non-parallel dataset Da, which shown i Algorithm
Algorithm 1: Our Algorithm
Input: A non-parallel dataset Du, learned logit (value) function fw. policy Th with the initial parameter y, total update steps U, and synchronizing period k.
Output: A policy Ti parameterized by yp begin
for i 4 1...U do
if = (mod k) then
Tb <- T
Behavior policy update
Sample source sentence a E D,, Construct the initial state (ar, [BOS])
[BOS] is the beginning token
Sample trajectory T from the behavior policy T
Th+1 t and <- for & + | do if = r| then (8t,
Termination step, $141
else
Pt fu(ss; ar) max for fu(()11,a));
By Equ.
qi ri + Mi+1
Accumulating rewards Importance weight > By Eqn. 8
Pt n. (arls)))) / (arl:::;
g - g + p.g. V log n (
< "
Gradient ascent
return X.
As mentioned in Remark the reward r(s, a) can be arbitrarily shifted by c, - Cs+(a)- We show that this shift does not affect the optimal policy.
Theorem 3. Suppose r' r'(s, a) = r(s, a) c. Cx+(a) Then the learned policies under (s, a) and r(s, a) are the same.
Proof. By Eqn.
M. function & returns the expected total reward. In Algorithm D we sample it by To (st, 0z) : (st, at) + r' (s(411 a(+1) + ... +7'
=r(,,, u1) + Ca, - f ((541, + faTTT Ca.12 + r(0))) 4r1) + carr =cs, + r(s ax) + r( (41) + +... ((5) 911) =: q" (st, ai) Cer
The last line suggests the constant plays role as the baseline in policy gradient, which is shown to be irrelevant the optimal policy [55]
5

---

Table I: Main results. The higher/lower, the better. QQuoted from Wen et al. 62 on deduplicated dialogue datasets. 'Quoted from 2 *Qooted from For the paraphrase generation metric, we have iBLEU= = (1 a) BLEU aa SBLEU.
(b) Paraphrase generation. "Copy" refers to direclty copying
(a) Dialogue generation.
the input sentence.
Method
BLEU4 SBLEU4
iBLEU4"
Method
BLEU2 BLEU4
Parallel DailyDialog
Copy
29.88
100.0
16.89
Adalab
160
6.72
2.29
Parallel Quora Generation
DialogBERT 16
5,42
2.16
Dagger RL-NN
28,42
66.98
18.88
T5-Base
8.96
3.69
40
20.98
40.52
14.83
Parallel OpenSubtilles
T5-Base
30.83
44.77
23.27
[T5-Base Base] Fully Supervised 8.75
3.06
Non-Parallel Quora Generatoin
Non-Parallel OpenSubtitles
LTSL
|||
29.25
71.25
19.20
[T55 (15-Basc]aae Training
9.10
3.73
[T5- Self Seif-nraining 31.39
48.02
23.44
[T5- B R- R-Reeression
10.34
4.18
[T5- R-Regression
30.77
44.23
23.27
[T5-Base Ours
11.02
4.30
[T5- Ours
31.47
45.43
23.78
3 Experiments
3.1 Datasets and Metrics
Dialogue Generation. We adopt two widely used datasets, DailyDialog 30 and OpenSubti- tles [57] for the dialogue experiment. The DailyDialog dataset is constructed from English dialogues crawled from the Internet, whereas the OpenSubtitles dataset is constructed from movie subtitles based on IMDB identifiers. A dialogue session is split into single-turn context-response pairs in experiment. For semi-supervised learning. use the smaller dataset, DailyDialog, as the parallel corpus Dp. and the larger dataset, OpenSubtitles, as the non-parallel corpus Da (i.e., we only retain the context sentence in the OpenSubtitles dataset). This follows the common setup for semi-supervised learning, where the unlabeled dataset is larger than the labeled one.
It should be emphasized that a recent study 62 shows more than 20% of test samples identical to some training samples in both DailyDialog and OpenSubritles. This results meaningless comparison and inflated performance of previous methods, e.g., BLEU4 of 11.01 in AdaLabel [60] and 14.61 in DERT [6]. Therefore, we use the deduplicated datasets in [2), containing 60K/6.5 K/7KK samples for vaaiiing/alidaationness in DailyDialog and 1M non-parallel samples in OpenSubtitles. Although our scores will be lower than previous inflated ones, we follow the correct setting for research.
We use BLEU scores as main evaluation metrics, which are widely used in dialogue gener- ation [62] In particular, BLEU-n evaluates the geometric average of i-gram precision scores for i = 1, ... n. Following previous work 62) we lowercase all sentences and tokenize them with
NLTK library 33
and use the Quora Question Pair datase for the paraphrasing experiment. The Quora dataset originally designed for paraphrase classification, containing both paraphrase and cor-parap pairs. The paraphrase pairs naturally form parallel dataset for the generation purpose; following the common practice [KE], split into 124K/4K/20K samples for sianiing/vliddaionnessst The non-paraphrase pairs, containing 510K sentences, are discarded in previous work, but manner.
Paraphrase Generation. We follow previous studies I 32 4
able utilize them in semi-supervised
We use the standard iBLEU score 54 as the main evaluation metric. It involves penalty of Self- BLEU (SBLEU) between the generated and input sentences, as the paraphrasing task requires using
attpp://wwet http::/wwww.kkkeeeeoooo////eeeee

---

BLEU
75%
Regression
Ours
50%
95%
0%6
0.4
Figure I: The distributions of token-level estimation of future rewards (o) in Algorithm D on the DailyDialog validation set. The BLEU score of sentence is shared among tokens in the same sentence.
different lexicons. Specifically, it is calculated by iBLEU = (1 a.) BLEU -a SBLEU, where a is typically set to 0.1
For clarity, also report BLEU and S-BLEU scores in
experiment.
3.2 Settings and Competing Methods
For each task, we first fine-tune a T5-Base model 42 on the parallel data by Eqn. 4 Then we apply proposed method to induce the reward and further train the model by Algorthh T on the non-parallel data. We compare our approach with the following semi-supervised methods.
Self-Training We apply the supervised model to the non-parallel dataset and generate pseudo-target sentences, which are used to continue training the model. This a commonly used semi-supervised approach in text generation literature
21 68
R-Regression. Wu al. 666 propose reward regression (R- (E-Regression) approach, where the reward is defined as the BLEU score. Since their reward is the same the evaluation metric, such method may achieve higher BLEU scores without actually improving the generation quality. By contrast, our reward is induced in principled way and is agnostic to evaluation metrics. In our experiment, we replicate the R-regression method, which constitutes a controlled comparison to our approach, as the only difference is the reward function.
Appendix B provides implementation details and hyperparameters of approach.
3.3 Main Results
Results of Dialogue Generation. Table la shows the results of the dialogue generation task. We notice that our fine-tuned T5-Base model 12 has already outperformed dedicated methods, Adal.abll (60 and DialogBERT [T6]. This is consistent with the findings of [22 6l in that the
alleged "state-of-the dialogue systems do not outperform standard pretrained language models on deduplicated datasets, highlighting the importance working with the correct setting.
We then apply semi-supervised learning Seefn-Traiiin, R- R-Regression, and our approach) with the non-parallel OpenSubtitles dataset. We achieve higher performance than T5-Base trained only parallel DailyDialog. Interestingly. the fully supervised model trained on both parallel DailyDialog and parallel OpenSubtitles does not achieve high performance, DailyDialog only. It is noticed that the OpenSubtitles dataset noisy R which likely causes the performance degradation. This signifies the need of semi-supervised learning.
lower than the trained with
Among semi-supervised approaches, RL-based methods (R- (R-Regression and ours) are generally better than Self Telining This is within expectation because Self-Training learns from its own generation and may be overconfident, whereas RL approaches are able to explore different parts of the data space, being more effective way of semi-supervised learning.
Moreover, our approach outperforms RL with R- R-ggrressio, where the reward is the only difference. The controlled experiment confirms that the reward induced from models trained with teacher forcing effective for RL training. It is also worth noting that R-ggressioon uses evaluation metric as the reward, and thus may deliberately improve the metric rather than text quality. By contrast, our reward is induced in principled manner and agnostic to evaluation metrics, and our approach still achieves higher performance even with such a vinadvan

---

Table 2: Comparing sparse and dense reward functions.
Dialogue generation.
(b) Paraphrase generation.
Sparse
Method
BLEU2"
BLEU4
Sparse Method
BLEU4" SBLEU
iBLEU4
Self Training
F1
9.10
3.73
Training
23
31.39
48.11
23.44
Regrressio
66
9.45
3.73
R-Regression
66
30.78
44.32
23.27
Yes
Yes
Induced-R
9.75
3,99
Induced-R
31.28
45.22
23.63
R-Regression Induced-R
66
10.34
4.18
R-Regression
66
30.77
44.23
23.27
No
No
11.02
4.30
Induced-R
31.47
45.43
23.78
In general, our approach achieves the best performance in both metrics. In particular, it significantly improves DailyDDalog-traineed T5-Base by +2.06 (+23.0%) BLEU2 and +0.61 (+ (+16.5%) BLEU4. It also outperforms the second-bes method, R R-Regression, by 0.68 (+6.6%) in BLEU2 and 0.12 (+2.9%) in BLEU4, verifying the effectiveness of our approach.
Results of Paraphrase Generation. The results of para- phrase generation are shown in Table Tb As seen, directly copying the input already achieves high BLEU score against the reference. iBLEU addresses this by penalizing the Self- BLEU score (against input) and is considered the main metric.
BLEUZ
12
We consider another semi-supervised baseline LTSLL [11] It performs retrieval-based paraphrase expansion and meta op- timization, thus being task specific. We see that LTSL has an extremely high Self-BLEU, suggesting the generated para- phrase largely resembles the input. It achieves lower iBLEU score than other semi-supervised approaches.
BLEU4
4.4
We also that RL approaches generally achieve lower Self- BLEU than Self- TrainTrainnn. This is because Self- Training learns from its predictions, which overlap the input more than groundtruth paraphrases do (Self-BLEU of groundtruth: 29.87); result, Self-BLEU increases to 48.02 from 44.77 of T5- Base. By contrast, RL learns by exploring different possible paraphrases and is able retain low Self-BLEU.
3.8
3.6-
10
k-step training
The learning curves by choosing different values of Scores are measured on the valida- tion set of Diily.. Training terminated when the BLEUU score drops below 3.5.
Figure 2:
Overall, our approach achieves the highest BLEU and a rea- sonably low Self-BLEU, yielding the best iBLEU among all competing methods. The results are consistent with Table showing the generality of our approach.
3.4 Analyses
Step-Wiss Reward. In Figure 1 we show the distributions of different reward functions. As seen, the BLEU score is mostly concentrated at O, providing little information for training. R- Regression consequently suffers from similar problem, as it is trained by the groundtruth BLEU scores. The distribution of induced reward, on the other hand, has the lowest peak and is the most wide-spreading one.
We conduct another analysis to show the importance of step- wise rewards for RL training. We compare our approach with sparse reward function that defers all rewards to the end of sentence. In other words, the last step's ' reward is the sum of our step-wise rewards, whereas all previous steps have a reward of O. This constitutes a rigorous analysis, as the total reward and thus the training objective are the same in both cases. Results in Table 2 show that our step-wise reward outperforms the sparse reward in all cases. This suggests our approach serves as a meaningful credit assignment of the total reward, which is beneficial for RL training.

---

The Effect of the Synchronizing Period. We analyze the effect of the synchronizing period l: introduced in Section 2.3 In Figure 2 we see that the training is unstable if k 1 (on-policy). in which case the model generates uninformative and meaningless sentences (illustrated in Appendix D). When k = 1000, the performance increases quickly at the beginning, but starts to decrease with further training. We hypothesize that this is due to the lack of exploration (Section 2.3 When k: is infinitely large (the behavior policy is fixed), the performance grows slowly and stops improving after certain number of steps. Based on this analysis, we choose k = 5000 to balance exploitation and exploration. Although the experiment is conducted only on DailyDialog due to the limit of time and resources, we directly apply the setting to other experiments, showing the robustness of our approach.
Data Efficiency. In Figure 3 we analyze data efficiency by sampling different numbers of data points from the non- non-parallel corpus. As shown, our method consistently outperforms self- eeff-training, even with only 0.1% (the leftmost points) of the training set. Additionally, the performance of our method quickly increases with more data, whereas self-training grows slowly. This is expected because RL training explores different parts of the sentence space and learns from their rewards, whereas self-training only learns from the single generated sentence by the model itself given an input.
We also investigate how performance changes according to the size of the parallel dataset, which reflects the quality of the learned policy. Results are shown in Figure Appendix C
Related Work
Semi-Supervised Learning for Text Generation. In text generation, popular ways to utilize both parallel and non-parallel data include self-training 21 68 and back- translation 49]. Both methods first train model on the parallel data and then generate secudo-paralled pairs for the non-parae non-parallel sentences. The difference is that self-training generates pseudo- parallel pairs from source to target, whereas back mack-translation generates from target to source. We mainly consider self- training as a baseline because it does not require an additional model in the reversed direction, making the comparisons fairer. Our implementation of self-training is also similar to sequence- level knowledge distillation
BLEU-2
11
10-
9
reward self-training
0
10
20
30
40
50
BLEU-4
4.25
except that the latter augments the parallel data instead of the non-pan non-parallel ones. In Figure 3 we show that self-training cannot efficiently utilize the data because of the lack of exploration. Additionally, the exposure bias issue remains because they are trained with the feacher-foreing objective.
15
4.00-
3.75-
3.50
3.25-
3.00
6
10 20
30
40
50
Text Generation beyond Teacher Forcing, Teacher forcing known have the exposure bias issue, A line of work uses the generative adversarial network (GAN) [4] to alleviate the issue. For example, Yu et al.
% of non- connpaaallel data
Figure 3: Trends of self-training and our method given different sizes of the non-parallel data. Scores are measured on the Daily- Dialog test set.
[6]] and Guo al. [8] propose to use GAN-style training to generate text similar to the training set in an on-the-fly manner. This practice reduces the discrepancy between training and inference because GAN sends its own generation as inputs rather than using groundtruth sentences during training. Shi et al. [5T] further formulate the adversarial training using the IRL interpretation. These GAN-style methods are different from ours in two main ways. First, GAN-style training requires parallel corpora and thus cannot be directly applied to semi-supervised learning on non non-parallel datasets. Second, GAN-style training involves the optimization of an adversarial objective, making the training unstable, e.g., suffering from mode collapse [4].
Another paradigm to alleviate the exposure bias is RL. For instance, Sokolov et al. 5 and Kreutzer et al. (26) leverage the bandit-structured prediction framework for text generation with BLEU as the heuristically defined reward. Bahdanau et al. 1 and Shen et al. (50) utilize different variants of policy gradient for RI. training. However, these methods are task-speeific and suffer from the problem of sparse rewards, as mentioned in Section
4.4 More importantly, these approaches require

---

parallel data to calculate the reward and cannot utilize non paraalalll data either. To address this, Wu al. [66] propose to learn a reward regression model on the parallel dataset and perform RL on the non-parallel data with the learned reward. As mentioned, such method is still task-specific because requires the human heuristics of the task to define the proper reward function. Additionally, suffers from the reward-sparsity problem, as seen in Figure Tand Table
Search is also a popular way replace teacher forcing. The Learning to Search (L2S) framework enables the model to search for a better score during learning and is widely applied to text generation. For example, Wiseman and Rush [64] propose to optimize the beam search results through training. Li et al. 29 develop an unsupervised learning approach to text generation based on local search. In addition to the L2S framework, Leblond al. [27] leverage the Monte Carlo tree search 225 522 to
select better tokens in step from the sampled generation. These methods are different from ours since they need either heuristically defined scoring functions or parallel data, limiting their methods to certain tasks or to the supervised paradigm. However, given the success of these methods, we consider the search-based approach an interesting future extension of our work.
Imitation Learning. The intuition behind our work is also related to imitation learning methods in general. Typically, these methods aim to obtain a good policy given a dataset containing aate-nction pairs. The casiest approach is behavior cloning [39, which greedily imitates the demonstration. Similar to the exposure bias, behavior cloning also faces the problem of compounding errors [[6]. SMILe 4ol and DAgger [ mitigate the problem by querying an expert. In text generation, Du and Ji
empirically verify that imitation learning methods are helpful. Recently, Pang and He 37 frame the text generation task as an offlin reinforcement learning problem, which learns from a dataset containing tuples of state, action, and reward. Compared with our method, these approaches rely on parallel sentence pairs and cannot effectively make use of non-parallel datasets.
12
5 Conclusion
Summary. In this paper, we show that reward function can be derived from a model trained with teacher forcing. The derivation does not rely on human heuristics for certain tasks. Additionally, the derived reward function assigns step-wise scores and makes the RL training easier. Our approach leads to a training algorithm in semi-supervised manner and utilizes both parallel and non-parallel data. We conduct experiments on the dialogue and paraphrase generation tasks. The empirical results show that the performance of our approach is better compared with the baselines: self- self-training and reward regression. We further analyze our reward function and show the benefits of our approach.
Limitation and Future Work. First, the scale of the experiments in this paper is restricted by computational resources. It is interesting to see if our approach could obtain better performance with large models 3 4]
and larger datasets.
We also notice that Assumption has a deep connection with entropy-regularee RL Our approach can be easily extended to such cases in the future.
45
Another interesting direction would be using the reward as an interface between humans and the model to control the generation. Specifically, the current seq2seq models treat data as the ground truth, but the data may be contaminated with undesired or harmful information. We hope that our approach provides a way for humans to apply additional rules to the reward function to avoid the model generating harmful information.
Appendices
The full paper, including appendices, is available at hbttt::/aarxx..orr/b////////////
Acknowledgments
We thank all reviewers for their valuable comments. We also thank Guoqing Luo for early discussions. The research supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant No. RGPIN2020-04465, the Amii Fellow Program, the Canada CIFAR AI Chair Program, UAHJIC project, a donation from DeepMind, and the Digital Research Alliance of Canada (alliancecan.ca)
10

---

References
[1] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In ICLR, 2017. URL
[2] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training, In COLT, page 10-, 1998. URL https:://oi..orrreeeeeoo org 10 1145/ 1ee.111149999/1 279943.9994 279962
[3]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- In NeurlPS, 2020. URL https///p/////rreeeeddngg www.snenuriiscco.eeeoo met.rrpp..cc/pppee///////aaaa
[4] Alex J Chan and Mihaela vam der Schaar. Scalable Bayesian inverse reinforcement learning. http:///pwnreeeeeeeeeeeffrrrrrrr
ICLR, 2021. URL
[5] Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. In ICML, pages 2058-2066, 2015. URL http:
//proceddings.oo mlr. press/ vttesss//3/ccaagbib html
[6] Ting-Rui Chiang and Yun-Nung Chen. Relating neural text degeneration to exposure bias. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neu- ral Networks for NLP, pages 228-239 2021. URL
/201
1ackbboxllp-.166
[7] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. EMNLP, pages 1724-1734, 2014. URL https:
//aclanthology ortp//4-11////11/0/
[8] Richard Csaky and Gábor Recski. The Gutenberg dialogue dataset. In EACL, pages 138-159, org/ eac.1eeeccmmmii..
2021. URL A
[9] Hal Daumé, John Langford, and Daniel Marcu. Search-based structured prediction. Machine Learming, 7573)32373 2009. URL htttp::///iiii www.//iiii..iiiie.r....e.. com htt.:/cc//aaaaaaaa httpm//rrceeeee1110100 -19994-009-5166-
[10] Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. In ICMI, page 179-186, 2012. URL https:/:: acm.
[11] Kaize Ding, Dingcheng Li, Alexander Hanbo Li, Xing Fan, Chenlei Guo, Yang Liu, and Huan Liu. Learning to selectively learn for weakly-superrvised parapaaa generation. In EMNLP, pages 5930-5940, 2021. URL httts:://aclantthllllo//
/erg/2021.... emnlp-main.nnmmm.emmmmmmmmmm www..aiinn.m .80in.44000)
[12] Wanyu Du and Yangfeng Ji. An empirical comparison on imitation learning and reinforcement learning for paraphrase generation. In EMNLP, pages 6012-6018, 8, 2019. URL https://
aclanthology.o org/D19-1619
[13] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Con- volutional sequence to sequence learning. In ICML, pages 1243-1252 2017. URL http
//proceedings.min mlr wwtppp/w//sspppppppppppppppp gewringiriinn html
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pages 2672-2680, 2014. bttps htt.o/aafcccc---ttaartttttttataa
neurips httprripps.ccppppp//2///////////
[15] Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. In ICLR, 2018. URL http::://pperrreviee....
Forum2id=B1199t100
[16] Xiaodong Gu, Kang Min Yoo, and Jung-Woo Ha. DialogBERT: Discourse-aware response generation via learning to recover and rank utterances. In AAAI, pages 12911-12919, 2021. URL htttps://o/ aaai. org hwwg/index...e http://wwwppppaaattttaaaaatt///s
11

---

[17] Han Guo, Bowen Tan, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. Text generation with efficient (soft) q-learning. arXiv preprint arXiv:2006.07704, 2021. URL https://arxiv org/abs/2100 07704
[18] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In AAAI, pages 5141-5148, 2018. URL
https://www aaai wwg/oogg//./ooooo..oooooo ocB/ iin/xiiidxxiiiii
[19] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine, Reinforcemenntleaminng with 2017. URL /ttp::ttt//prroeeeding
deep energy-based policies. In ICML, pages
mlr. wwt.press//v7//haaaaan jan a17a. html
[20] Chenyang Huang. Hao Zhou, Osmar R Zaïane, Lili Mou, and Lei Li. Non-autoregressive translation with layer- wise prediction and deep supervision. In AAAI, pages 10776-10784, 2022.
/ttt://tp:://os aaai org/ index. php/ http//AAA/aartcclleccc /vte//vviw/////ee 21323
[21] Wenxiang Jiao, Xing Wang, Zhaopeng Tu, Shuming Shi, Michael Lyu, and Irwin King. Self- training sampling with monolingual data uncertainty for neural machine translation. In ACL, pages 2840-2850, 2021. URL
org/ org/2021/ 202110211112222 acl--ong. 221
[22] Olav Kallenberg, Foundations of Modern Probability. Springer, 2021. URL hwt.htttppp... pestps://1ite //1ink springer www.boww..ooooocoeooooooooooooo 10w.:0000000000000000000 http:///.....///...0///////3333. 6788-------000--1--111111-
[23] Yoon Kim and Alexander M. Rush. Sequence-level wwwp:/eeweeeeeoooooeeeeeeeeeeee distillation. In EMNLP, pages
1317-1327, 2016. URL htttpee///cclaabbbollogy.o..////
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR,
2015. URL fhtth::p////// http:::///xvvvvrea wtt.or/g//bbb/////////////////// 6980
[25] Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In ECML, pages 282-293, 2006. URL
[26] Julia Kreutzer, Artem Sokolov, and Stefan Riezler. Bandit structured prediction for neu- ral sequence-to--sqqunee learning. In ACL, pages 1513, 2017. URL httpp::/
oclaathology..or/PA77--
[27] Rémi Leblond, Jcan-Baptist Alayrac, Laurent Sifrre, Miruna Pislar, Lespiau Jean- Baptisee loannis Antonoglou, Karen Simonyan, and Oriol Vinyals. Machine translation decoding beyond beam search. In EMNLP, pages 8410-8434, 2021. httpshttss:/accaaattaa
org/2021 2021
mmnlp-main.6e 662
[28] Haoran Li and Wei Lu. Mixed cross entropy loss for neural machine translation. In ICML, pages 6425-6436, 2021. URL SRRp:ttt://proeeeding mlr. press/v139/1i2// html
[29] Jingjing Li, Zichao Li, Lili Mou, Xin Jiang, Michael R. Lyu, and Irwin King. Un- supervised text generation by learning from search. In NeurIPS, pages 10820- 10831, 2020. 1ate:3////ddcbdd11123333333 Ats.oa3tt-bttrat..ttt.
URL
wewpeeddinnnneuiiipiiee neurips. wwwurips.ccppppprr///eece hatppeer//20//ahhh
[30] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-tum dialogue dataset. In IJCNLP, pages 986 186-995, 2017. URL
https:://acaattholle
org/I17-1099
[31] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, 2004. URL /ttp/attps://cclantholoooy org. wrwp//4-100/11111111 W00-
[32] Xianggen Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song. Unsuper- vised paraphrasing by simulated annealing. In ACL, pages 302-312, 2020. URL https
//aclanthology-.. org/ 2200.2200..22 aal.aacl-a.. maii..8
[33] Edward Loper and Steven Bird. NLTK: The natural language toolkit. In Proceedings of the ACL-
02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 63-70, 2002. URL https://acllanthology org/ M02-0109
[34] Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. CGMH: Constrained sentence generation by Metropolis- Hastings sampling. AAAI, pages 6834-6842, 2019. URI.https httttt:pps/.oorrtttt/e////
10. 11609//aa.. v33101. 330116834
12

---

[35] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, (18(7750))555595538583 2015. 2015. htt.s//tts:///oi. org/ 10. 10t..11000//aatue14335555
[36] Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In ICML, pages 663-670, 2000. URL httpp:////d acm. wwtp///g//ooo// 6455299664599...00 657801
[37] Richard Yuanzhe Pang and He He. Text gemeration by learning from demonstrations. In ICLR.
2021. URL BRLLittps://ppereivier http:///wooooooooooooooooooooood
[38] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, BLEU: A method for automatic evaluation of machine translation. In ACL, pages 311-318, 2002. URL https haclannhologg..rg/// prg/ 1040
[39]
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. 1991199111//..1.999..... 1991.3
Neural Computation, 3(1):88-97, 1991. URL
1.88
[40]
Lihua Qian, Lin Qiu, Weinan Zhang, Xin Jiang, and Yong Yu. Exploring diverse expressions for paraphrase generation. EMNLP, pages 3173-3182, 2019. URL http:s:/aaalannioooogg
org org/D19-1313
[41] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv: 12. 1446,
2021. URL /ttpthtt////aaaa
www./orgg//////////e//..eeeeee/e abs/2/12 11446.
[42] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text Transformer. Journal of Machine Learning Research, (1110):1-677 2020. URL
https://jmlr httpp::/jjlr..orgppapeeeee wwwpor/w..pp..ooooooot.ot.eeeeep wvtperss//22////// 20111/20077444 htmml
[43] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning.... In I/CAI, pages
2007. URL https: / acm. org/ htg//oi/10..om
[44]
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016. URL 06732
[45] Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: Imitation learning via reinforce- ment learning with sparse rewards. In ICLR, 2020. URI htt/:t/t////oerrreieee net/ffrummf
id=S1xkd24twB
[46] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In A/STATS, pages 661-668, 2010. URL http:tt:: httt:://proceednngg .ww.hngg.........m..........e..e wttppp/eeesssssse////e html
[47] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, pages 627-635, 2011. URL httpp://proceediigg. mlr...r... pwwps/vi5...//p/p http///11//ros111e1/e11/1
ross wwtoosssaarrooo 1 pdf
[48] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347, 2017. URL https://rxxiv
org/abs/1707 06347
[49] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACL, pages 86-96, 2016. URL
o00/P16-1009
[50] Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In ACL, pages 1683-1692, 2016. URL
ftttp::/aacanttoo/e
www.ooggg........ 1ttg//P6-1155
[51] Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. In ISCAI, pages 4361-4367, 2018. URL Utt.hhtts:///oi..rg
10. N0.24963/ijcal 2018//0001666
[52] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 2016. URL https: //wp://ooooaaa org /0. 1038/ nature16911
13

---

[53] Artem Sokolov, Julia Kreutzer, Stefan Riezler, and Christopher Lo. Stochastic structured predic- tion under bandit feedback. In NIPS, pages 1489-1497 2016. http:///wowouottttttr//rr///eenr neurips.
html
[54]
Hong Sun and Ming Zhou. Joint leamning dual SMT system for paraphrase generation. In ACL, pages 38 38-42, 2012. URL htt.:ttteer///llmmmooooyr..////p
[55] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press,
2018.
wttp:deeeeeneeetteee www.:eee//bbeeee terrookkkhhe-ooo-- 2ndd html
[56) Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.
URL htt..tttp:///o...om org/ 10. 1109/CVPR. 2016 20166.008 308
[57] Jörg Tiedemann. News from OPUS- collection of multilingual parallel corpora with tools and interfaces. In Recent Advances in Natural Language Processing, pages 237-248,, 2009, URL http://dx. doi wttp://owo..oi...oo./o
10. 1075/ weiiittt0990 19tie
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS pages
2017. URL https::/prrcceeddiiee neurips httpppscccppppp//111111 2217//aa////
[59] Elena Voita, Rico Sennrich, and Ivan Titov. Analyzing the source and target contributions to predictions neural machine translation. In ACL, pages 1126-1140, 2021. URL https
/aclanthology . org/ www.eacc-22......... acl-long.
[60] Yida Wang, Yinhe Zheng, Yong Jiang, and Minlie Huang. Diversifying dialog generation via adaptive label smoothing. In ACL, pages 3507-3520, 2021. URL http:s//acclannhoooogy
htr/2021.acl1--.
272
[61] Yuqiao Wen, Yongchang Hao, Yanshuai Cao, and Lili Mou. An equal-size hard EM algorithm for diverse dialogue generation. arXiv preprint arXiv:2209.14022 2022. URL https://
arxiv. 2bs// 14627
[62] Yuqiao Wen, Guoqing Luo, and Lili Mou. An empirical study on the overlapping prob- lem of open-domain dialogue datasets. In LREC, pages 146-153, 2022. URL https wtwp://owy..rg//2...eeeee.
[63] Ronald Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine Learning, 8(3):229-256, 1992. URL
com/article/10, 1007 L007/BF00992696
[64] Sam Wiseman and Alexander M. Rush. soqueence-t--equeene learning as beam-scarch optimiza- 1137
tion. In EMNLP, pages 1296-1306, 2016. URL
[65] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020. URL https:// aclanthol 2020 emnlp-demos.
[66] Lijun Wu, Li Zhao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. Sequence prediction with unlabeled data by reward function learning. In I/CAI, pages 3098-31044 2017. URL https
/ org//0 24963/ i ijcai. 2017 /432
[67]
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial 2017. URL httph::::///// h/wp:///////ww///... aaai org/ 12- Yu-L- 1434-14444... pdf
nets with policy gradient. In AAAI, pages
[68]
Jiajun Zhang and Chengqing Zong Exploiting source-side monolingual data in neural ma- chine translation. In EMNLP, pages 1535-1545, 2016. URL
org/
D16-1160
[69] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, pages 1433-1438, 2008. URL hatt::///w/t - aaai
org brg/Librarr/MMM|/2003 20 200-227.22 phpl
14


Reviews:
[{"review_id": "osjVdAkr7BR", "reviewer": "Reviewer_BSmj", "summary": "The paper draws a connection between teacher-forcing and inverse reinforcement learning. The authors derives theoretically from the teacher-forcing objective to an equivalent reward function (i.e., a model trained with teacher-forcing recovers a reward function). With this insight, the authors leverage this learned reward function to train on non-parallel data. The reward can be applied step-wise, therefore mitigates the common instability issues when training seq2seq models.", "questions": "- In equation 7, the second term requires computing a $\\max_{a' \\in \\mathcal{A}} f_w(s + [a], a')$. Does this mean that at each generation step the model needs to run a forward path for every action, i.e., word in the vocabulary? If so it might render the method slow and hard to scale.\n- The Periodically Synchronized Behavior Policy is described as a contribution but to my knowledge similar issues (discrepancy between the behavior policy and the model policy) are long-standing and has been discussed as a simple fix such as in PPO (Schulman et al., 2017) or other off-line RL methods (update after K steps). Is there any main difference that I missed here?", "limitations": "The authors appropriately addressed potential limitations in a standalone section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The idea that links teacher-forcing to its inherently learned reward function is, to my knowledge, novel and the theoretical connection has not been explicitly drawn before\n- The paper is easy to follow and the methods are well motivated\n- The proposed method is simple and easy for the community to extend and build research upon\n- The experimental results and the analysis are illuminative\n\nWeaknesses phrased as questions below (if addressed or answered, I'm happy to change the score).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "npEK9Kw2vQK", "reviewer": "Reviewer_s2sH", "summary": "This paper presents an approach for training a text-generation model by transforming an existing conditioned language model into a reward function. The paper suggests that the pre-softmax logits of a pre-trained sequence-to-sequence language model can be used to compute a per-token reward for new input sequence data. The paper also proposes an algorithm for training a model with this reward function, including off-policy exploration. The experiments compare the proposed approach against several other methods for training with additional non-parallel data (e.g., self-training, and R-regression, which uses a reward function derived from BLEU score. Compared to existing approaches which use single-sentence reward (e.g., BLEU score), the proposed approach provides a denser reward which results in moderate improvement over the compared systems.", "questions": "* What is the intuition behind the derived reward function in Eq 7? It looks kind of like an advantage function.\n* What happens if r(s, a) < 0, if this ever happens (it seems like it should)? Does this cause problems with training, as this means Alg 1 is directly minimizing log \\pi?\n* Would training with teacher forcing basically be \\pi_b \\prop f_w, with k = \\inf?\n* How is \\pi_\\phi initiated? With f_w or just the original T5 base parameters?\n* Is the training for self-training using teacher forcing, just with sequences derived via inference from unpaired data?\n* How do you choose to stop training?\n* Eq 7 -- f_w returns unbounded logits, right? Does this cause issues for training?\n* Did you experiment with a discounting factor in Algorithm 1 that attenuates return backwards in time? If f_w is usually positive, this seems like it would give much higher rewards to actions at the beginning of a sequence rather than at the end.\n* Did you experiment with an entropy term to avoid collapse (similar to what you are seeing with k=1)?", "limitations": "I'd be interested in hearing a bit about computational efficiency for training and how the different methods compare. Self-training seems more efficient as it doesn't require sampling/inference at training time, except for a fixed number of examples. I believe R-regression requires running inference on a separate model, and obviously the proposed approach does as well, but it seems twice as many times, as in Eq 7 f_w is computed both for s and s + [a].", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nTo the best of my knowledge, I haven't seen any other work that trains a model with RL using rewards derived from another language model. It is a clever approach. Experiments were done on two different text generation datasets.\n\n---\n\nWeaknesses:\n\nThe improvements in performance reported in Table 1 seem quite small, except for BLEU2. Are they significant?\n\nThere are some experiments I would have liked to see: \n* The ability to compute the proposed reward function is general, but it still requires a domain-specific f_w to compute. Would have been nice to see experiments on deriving reward functions from other f_w besides in-domain, e.g., rewards from a general pre-trained language model.\n* How much does the quality of f_w matter? It would have been interesting to see performance relative to the number of training examples used to train f_w. How little parallel data do you need?\n* It would be nice to see learning curves / reward curves comparing the proposed approach with other methods. How many samples and gradient updates do you need for the proposed approach vs. self-training, R-regression, etc?\n\nAnd some evaluation I would have liked to see:\n* Some deeper analysis of model outputs after different training methods. E.g., more than a one or two pieces of generated text per approach.\n* Evaluation with a more recent text generation metric, e.g. BERTScore or similar\n\nSome claims / phrasing are a bit confusing:\n* Teacher-forcing isn't necessarily the only way to train a text generation model; masked training objectives are also effective and popular these days. Also, you don't need parallel data to use teacher-forcing based training objectives.\n* I'd also be interested to see non-conditional text generation with this proposed method. It doesn't seem like anything in the proposed method requires the task to be a sequence-to-sequence task (maybe this is what the authors are suggesting for future work with the reference to GANs in L153?).\n\nSome missing related work:\n* Kreutzer et al. 2016 / 2017 / 2018 use BLEU score for training text generation\n\n--- \n\nSome minor suggestions (e.g. readability):\n\n* In 2.1, I'd suggest moving lines 61 and 64 to closer to the top[ (e.g., after L56). I also am not sure what Eq 2 adds, besides it being standard in RL papers.\n* What does it mean for a reward function to be \"naturally defined\" in the context of text-generation; why would the suggested reward function be \"natural\"? (L65) \n* L100 -- policy q should be \\pi_q, right?\n* The discussion in the beginning of 2.3 is a bit confusing; the term \"behavior policy\" is unfamiliar to me (\"inference policy\" seems more natural)\n* The claim that any behavior policy can generate any trajectory seems broad to me. What if \\pi_b = \\pi_\\phi?\n* Nitpick on L130 -- the policy starts to generate fixed trajectories because the entropy is collapsing, not because it's becoming more optimal. Also in L254, this seems to be the case as well.\n* I was relatively confused about what was being plotted in Fig 1 -- what's the x-axis?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q6V_rvKBra", "reviewer": "Reviewer_bSdL", "summary": "This paper investigated the connection between supervised teacher forcing and inverse reinforcement learning. The authors showed that under the same parametrization and a deterministic transition function, the loss function of teacher forcing and maximum entropy IRL are equivalent. Then the authors proposed to use the teacher forcing model to replace the q function in the bellman optimality equation to derive a reward function. Empirical results show that the proposed methods outperform semi-supervised learning baselines on text generation tasks.\n\n\n-------\nPOST REBUTTAL:\n\nThe authors addressed my concerns. After reading the other reviews and the discussion, I increased the score to 6.", "questions": "In terms of experiments, since this paper is about IRL, why not include IRL baselines such as GAN-based IRL algorithms?\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed algorithm is interesting. It avoids estimating the partition function (e.g. using GAN or sampling methods) in maximum entropy IRL, but instead using the teacher forcing model to derive a reward function. The training in this case will be more stable.\n\nWeaknesses:\n\nI am not entirely convinced that the proposed reward function in equation (7) is valid.\n(1) Theorem 1 in the paper shows that, given the same parameterization, MaxEnt IRL and teacher forcing both maximize the likelihood of the trajectories in the data and hence are equivalent. However, the q function in MaxEnt IRL (in equation (3)) has to additionally satisfy bellman equation and cannot be optimized freely. Therefore, the q function q_w cannot be simply replaced by the logit function f_w, and f_w(s, a) does not equal to q_w(s, a) for a given (s, a) pair. Therefore, I am not entirely convinced that equation (7), which simply substitute q_w with f_w, is valid.\n\n(2) Even if we can replace q_w with f_w, f_w is learned from data and represents the behavior policy, while q_w in equation (7) should be the q function of the optimal policy.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HqcwkTBTln", "reviewer": "Reviewer_iPjS", "summary": "This work proposes a novel reward function for text generation models, which is derived from a pre-trained supervised model. Authors also suggest using the new reward function to utilize monolingual data for fine-tuning. This training framework aims to resolve two main weaknesses of text generation models, namely exposure bias and lack of parallel data. Empirical results on two different generation tasks show improvements upon the baselines.", "questions": "Comments and questions:\n- line 12: You say \"common training method\" and give one citation from 2014. It does not sound convincing. At least you can cite Transformer model/Bert/BART\n- While you speak about text generation training, do you mean the structure of the dataset or the input/output of the model by parallel dataset? Because the text generation model can be trained on monolingual data without any need for labels. \n- You claim that large models intensify the problem of creating parallel datasets. But these large models are usually trained on **monolingual data**, and can be fine-tuned on the smaller task-specific dataset\n- line 34: what objective do you use for training seq2seq?\n- line154: you do not say what seq2seq model: is it Transformer? or T5?\n- Table 1: Do I understand the training process correctly: T5 base pretrained --> train it on task-parallel data with teacher forcing --> to add additional parallel/non-parallel data using RL? Which of that does line 3 Table 1(a) (T5-Base) report?\n- You cite results from \"DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances\". Why can the numbers from your paper not be found in the source? Same for AdaLabel\n- According \"Exploring Diverse Expressions for Paraphrase Generation\", the quora paraphrase test set contains 30k, you report 20k on test set. Why such difference?\n- \"Exploring Diverse Expressions for Paraphrase Generation\" does not report iBLEU4, but you cite the number. What is this number?\n- \"Learning to Selectively Learn for Weakly-supervised Paraphrase Generation\" does not report SBLEU, how did you get the number?\n\n**UPD**: Questions are mostly answered by authors; Score is updated", "limitations": "Limitations are addressed in Section 5", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Building an interesting connection between training with teacher-forcing objective and RL objective\n- Significant improvements over the baseline (however, see weaknesses)\n \nWeaknesses:\n- The cited previous results in Table 1 need verification \n- Sloppy citations\n- Paper is hard to understand and needs clarification\n- Thus: the argumentation in the paper is not convincing\n\n**UPD**: Weaknesses are addressed by authors during discussion; Score is updated", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept