KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning
Authors: Xiao Yu
Qingyang Wu
Kun Qian
Zhou Yu
Venue: EMNLP 2023 Main (2023)
Keywords: task-oriented dialogues
reinforcement learning
TL;DR: We propose a new RL-based method to help model better learn keyword generation for task-oriented dialogues.

Abstract: In task-oriented dialogs (TOD), reinforcement learning (RL) algorithms train a model to directly optimize response for task-related metrics.
However, RL often needs to perform exploration, which can be time-consuming due to the slow auto-regressive sequence generation process. We investigate an approach to create a more efficient RL-based algorithm to improve TOD performance in an offline setting.
First, we use a faster generation procedure that samples from independent next-word distributions after training the language model (LM) with supervised learning. We then introduce a fine-grained reward function to help the model focus on learning key information in a dialog, by measuring the importance and semantic closeness of each generated token. Experiments on the MultiWoZ dataset show our new training algorithm, Keywords Reinforcement Learning with Next-word Sampling (KRLS), achieves state-of-the-art performance on the end-to-end response generation task, with a 15% training time reduction compared to a standard RL algorithm using auto-regressive generation.

KRLS:
Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning
Xiao Yu, Qingyang Wu, Kun Qian, Zhou Yu Columbia University
{ (xy2437, qw2345, kg2157, 2y2416 6 @ecolumbics edu
Abstract
Training dialog"
Sys Fleep.
User
DST
task-oriented dialogs (TOD), reinforcement learning (RL) algorithms train model to di- rectly optimize response for task-related met- rics. However, RL needs to perform explo- ration, which can be time-consuming due to the slow auto-regressive sequence generation process. We investigate an approach to create efficient RL-based algorithm improve TOD performance in an offline setting. First, we use a faster generation procedure that sam- ples from independent next-word distributions after training the language model (LM) with supervised learning. We then introduce fine- grained reward function to help the model fo- cus on learning key information a dialog. by measuring the importance and semantic close- ness of each generated token. Experiments on the MultiWoZ dataset show our new training algorithm. Keywords Reinforeement Learning with Next-word Sampling (KRLS), achieves state-of-the the-arr performance on end-to-end response generation task, with 15% training time reduction compared to a standard RL al- gorithm using auto-regressive generation'.
S
Sys
KRLS
Usar
DST
[nemnnel
Resp.
Sys Resp.
User
DS
Sys Hesp.
User
DST
(
Temoral
Gen. Reep
Supersited
Rennforcmmet Leaming
Overview of episodic training in KRLS. Each turn is treated as a separate episode, and the RL task is only generate/explore response at the end. The gold system resp. are also used in our reward computations.
Figure I:
et al., 2018). However, training RL-based dialog models often requires a good user simulator (Shi et al., 2019), and the training process could be time-consuming RL often needs to explore and auto-regressively generate many new responses per given input (Ramamurthy et al., 2022). For exam- ple, in our experiment, we found that this genera- tion process alone takes 172 minutes/epoch out of the total 362 minutes/epoch during training.
1 Introduction
In this work, we aim to create a more effi- cient RL procedure for TOD, which does not need user simulator nor use auto-regressive genera- tion during exploration. We propose ing algorithm, Keywords Reinforcement Learning with Next-word Sampling (KRLS), which com- bines a faster sequence generation procedure and a fine-graine fine-grained per-token reward an offline setting (Jaques et al., 2019; Pang and He, 2021). First, we treat each turn in a dialog as separate episode and consider an offline RI. task to generate response only at the end of each cpisode (sce Figure 1). Since this procedure only explores/generates sys- tem responses at the last turn, no interactive envi- ronment (e.g., user simulator) needed. During this RL process, KRLS generates new sequences by directly sampling from independent word
Task-oriented dialog systems help users complete pre-defined tasks such as booking a hotel or re- serving a table in restaurant. With advances in large-scale pre-trained generative models (Brown et al., 2020; Raffel et al., 2020; Zhang et al., 2020e; Peng et al., 2022), many recent approaches (Wu al., 2019; Hosseini-Ast et al., 2020; Yang al.,
train-
2021a; Lee, 2021; He et al., 2022) handle TOD as a holistic task of end-to-end (E2E) generation, as opposed to the traditional modular approach.
In this E2E setting, the dialog history is often used as input, and reinforcement learning (RL) algorithms can train a model to generate a re- sponse that directly optimizes for task-related met- rics, such the task success rate (Budzianowski Code availale at https//gtthah.com/sssooy////KK/

---

Supervised Leariing
Poward
Reward
Poward
baoked
plane
Inow
plane
booked
(value
Finetured Decoder
Ood
[PA0]
booktd
car
bppe
[saloe_cod
[PAD]
booked
cbr
typ
paalee car
car
tokkn:: Jalle car
Offline RL
Figure Overview of the KRLS algorithm. During traditional supervised training, the language model learns/imitates the gold response. During offline RI training, the fine-tuned model generates sequences by sampling from next-word distributions conditioned on the gold response and receives a per-token reward.
distributions, after training a language model with the traditional supervised learning (SL) technique (see Figure 2). This generation procedure is much faster than the traditional auto-regressive approach, as it only requires a single forward pass. Next, KRLS uses fine-grained per-token reward to help the model focus on learning key information in dialog, by measuring the importance and semantic closeness of each generated token. Experiments on the MultiWoZ dataset show that KRLS achieves state-of-the-ard performance the E2E response generation task, with 15% training time reduc- tion compared the standard RL approach using auto-regressive generation.
((0) is the context and x is response, the proba- bility of generating x(0) can be modeled as:
p(x(1)(()
p(a 1 cO))
where 2/! is the t-th token in the i-th response, and T is the length of the response. As men- tioned Ramamurthy et al. (2022); Lubis et al. (2020), this generation can be formulated as a MDP problem (S,A, R, P, 7). The input context c(i) would be the initial state E S, and the re- sponse x would represent the sequence of ac- a an episode, where a C A the 1-th token in the i-th re- sponse. The reward function R would represent the "utility" of each action contributing towards the overall performance, such task success in TOD. Typically, this is modeled by using R(s, a) 0 for non-terminal states, and R(sT,a) a) for termi- nal states which can be computed by combining scores such as task success and BLEU (Rama- murthy et al., 2022; Arora et al., 2022). The tran- sition function P S A + S would determin- istically append the action a₂ to the current state 8t that 8+41 (co...., Cra ao..... .:). Finally, [0, is the discount factor.
tions a {a]
(8)
This paper makes the following contributions:
(4)
We propose efficient offline RL algorithm that approximates auto-regressive generation by sampling from independent next-word dis- tributions conditioned on the gold response.
We introduce per-token reward function, which can be used our offline RL algorithm to promote keyword learning or to incorporate domain knowledge.
We show that our proposed KRLS algo- rithm can achieve state-of-the-art performance E2E response generation MultiWoZ (Budzianowski et al., 2018; Eric et al., 2019; Zang al., 2020).
3 Approach
In the MulitiWoZ. dataset (Budzianowski et al., 2018; Eric et al., 2019; Zang et al., 2020), observe that key information, such as restaurant "phone number" and "address", needs to be gen- erated correctly in a response to achieve a high task success/inform rate. However, traditional SL uses a negative lig-liielihood loss, which asks the
2 Background
To introduce RL. in NLP tasks, we begin by formu- lating the response generation process as an MDP. Given a supervised dataset = {(c x()} where

---

KRLS Reward Fundiion
closeness -0.3
closeness -1.0
importance x1
importance x5
sampled
booked plane piane
reward -0.3
reward -5.0
relurn
return
return
return
task score+0.0
Decodor
sampled
booked
type
plane
Gold
[pad]
type [value car]
gold
[value car)
importantkey token!
Decoder (
gen key quantitie: overall task score=0.0
Gold
[PAD]
booked
type
[value_comm _ar]
required key quanttlee; [value__co [arj
Figure 3: KRLS reward function. Immediate reward measures semantic closeness the generated token the gold token, scaled by its importance Return for each sampled token is combination of individual immediate reward and future rewards. Future rewards will help propagate final overall scores, such as overall task performance. Key tokens response are bolded and italicized.
language model to uniformly learn all correct to- kens .oold given input context without explicitly focusing on achieving task-related objectives:
that there is model po that generates sequences similar to the gold responses in the training set. Then, under this assumption, can approximate the MDP process of auto-regressive sequence gen- eration by sampling from the next-word distribu- tions conditioned on the gold response. This is because if previously generated tokens are similar to the gold context (e.g., after CSi. training), then conditioning on those generated tokens is similar to conditioning on the gold tokens. In this setting, the next- word sampling process could approximate regressgre generation used during RI., but it is much faster each token is generated in parallel. Specifically, we first perform forward pass obtain the next-word distributions for 2t given the context and the gold response up to
(0) - Σ p(zlc) log
= lo (o (c)
where the probability of p(rlc) 0 i goold We will refer to models fine-tuned with this objec- tive as Sisettinet..ed"..
We hypothesize that it can be beneficial to RL and a fine-grained per-token reward function help promote keyword learning and improve TOD performance. First, in Section 3.1 we pro- pose a fast sequence generation procedure that can be used during RL training/exploration, and uti- lize policy-gradient based methods (Sutton et al., 1999; Williams, 1992; Schulman et al., 2017) to optimize response generation for task-related met- rics. Then, in Section 3.2 we design a fine-gre fine-grained reward function Rx(c) according to how impor- tant each generated token is, but also how close it is from the gold token, so that the model can focus more on learning key information once can generate non-key tokens semantically close to the reference. Finally, we describe our KRLS algorithm in Section 3.3 and Algorithm 1, which utilizes RL combined with our proposed generation method and reward function.
po (a gen (
Po (17
(i) 1 (
(2)
Then, we generate each next-token at gen by sampling from pa(X x -e with temperature т. Finally, given some suitable re- ward function R(s,, ae) E [-1, 1, (see Section 3.2 for details), we can use policy gradient methods (Sutton et al., 1999; Williams, 1992) to perform a "weighted learning" on each generated token:
xt
C G.V log Po((cc))
(3)
Note that this procedure is much faster than auto- regressive generation as it only requires one for- ward pass. Moreover, it is suitable for training model to focus on generating key information. because we can use the gold response as an ora- cle to locate those key positions and penalize each generated token in the reward function accordingly.
3.1 RL with Next-word Sampling
To avoid the slow auto-regressive sequence gener- ation process during RI. training, we propose an alternative sequence generation mechanism that can be used for RL exploration. First, we assume

---

3.2 Per-Token Reward Function
Algorithm 1 KRLS Training Algorithm
To improve a model's keyword generation abil- ity, we design a reward function that measures the importance and semantic closeness of each generated token. This reward aims to prioritize accurate generation of key tokens, and also con- textually evaluate how far-off is each generated token from the ground truth. We draw inspiration from BERTScore (Zhang al., 2019), which uses a separate neural network to compute a contextual semantic score of the generated sequence by com- paring it against the gold reference. However, we found that directly adapting BERTScore to a per- token reward function is sub-optimal in our setting, our generated sequence is "sampled" from the gold response. Therefore, consider simpler approach, utilizing the fact that our generation pro- cedure provided a one-to-one mapping between each generated token and the gold token.
Require: generative network po
Require: semantic scoring network Pe Require: supervised language dataset D Require: empty buffer BL, Bs
1: Repeat for n epochs:
2: for batch b, in D = (b1,..., bm) do
3: Perform sup. learning on b; (Equation 1)
4: Update generative network pa
5: Append b; to buffer Bi.
if i% == then for each batched episode by in Bt do
Collect samples per episode
by sampling from Equation 2 Calculate per-token reward
10:
11:
using Pe and Equation Calculate per-token returns Ge Append all to replay buffer BR end for
12:
13:
14:
First, we use a LSL. fsnetfine decoder network Decoder(() to compute the probability Px(X; = -111:111 )() of generating any token x at time t, which can be done in a single forward pass.
15:
Perform RL Bi, (e.g., PPO) Update generative network pa Clear Bt, and BR
16:
17:
18:
end if
Then, we index into this probability distribution to find Px =
19: end for
) of our generated tokens, as a measure of the semantic appropriate- ness of 2=!h! in the given context. To ensure that R correctly reflects the gold tokens as the optimal choice, we manually set this semantic closeness score to 1 for any token that is correctly generated x-50n _ _.oold To emphasize keyword learning, we also strictly set this closeness score -1 a key token is incorrectly generated. Otherwise, we use the probability Ps produced by the decoder network as the closeness score. Finally, we adjust the reward for key tokens by an importance scale 1 > 1, which is syper-paraamter for specify- ing the relative importance between keywords and kewwods..... This gives per-token reward:
network ps is used to compute the reward function during RL phase (see Section 3.2).
For each epoch, we first perform SL. over sev- eral batches of training examples and update our network pg- This is because our generation pro- cedure is based on the assumption stated in Sec- tion 3.1, so we periodically train pe with the LSL objective to imitate the gold responses before pass- ing over to RL (see Appendix F for generated se- quences). Then, we store those SL-trained batches into buffer B1, and perform RL on this learned buffer. During this RL training, we first gener- ate l responses per trained episode by next-word sampling (see Section 3.1), calculate their rewards and returns using scoring network Pø, and ap- pend them replay buffer B Then, we utilize the clipped policy gradient objective from Proxi- mal Policy Optimization (Schulman et al., 2017) to learn from BR and update the generative policy model Po (see Appendix A and Appendix B).
R(st, az) = closeness(x;""
Fold c)
(4)
We standardize R to |-1, to be later compared with other related reward functions in Section 5.2.
3.3 KRLS Training Algorithm
Algorithm 1 describes our KRLS training al- gorithm. Given a supervised dataset D = {c,, N Ji=1 consisting of the dialog context c, and the gold response x55 gold at each turn, we train a neural network py to generate a response that satis- fies the user's need given by c, A separate neural
Note that Algorithm I only additionally requires prior definitions of keywords to compute the ward. This means that such an approach can be generic to many taik-oriented dialogues where key- words can be easily defined (c.g., using entities from a database). For instance, in movie recom-

---

mendation (Harper and Konstan, 2015), correctly generating key information such "movie ratings" and "movie_ aenvieegeenres" could be helpful to improve the system's 's response. Additionally, we believe that for other dialogue tasks such as QA/social chat, keywords can often be automatically processed and defined, such as using the entities mentioned in WikiQA answers (Yang et al., 2015) and the intent keywords in ESConv (Liu al., 2021).
with GODEL-base (Peng et al., 2022) as the back- bone, and report this as Baseline (MTTOD).
KRLS Since KRLS targets at improving re- sponse generation, we replace the SL objective dur- ing response training in MTTOD with the KRLS algorithm, which involves both SL and RL training. We report this result KRLS.
finetuneKKRLS As the generation procedure in KRLS is based on the assumption stated in Sec- tion 3.1, we first initialize the model with LSL- finetuned checkpoint, and then perform the same KRLS training procedure used in KRLS. We report this result as finetume
4
Experiments
4.1 Dataset and Preprocessing
We evaluate our algorithm on the MultiWoZ dataset (Budzianowski et al., 2018). MultiWoZ is large- scale multi-domain TOD dataset consisting of 8438, 1000, and 1000 dialogs for training, valida- tion, and test sets respectively. The dataset consists of seven domains: attraction, hotel, hospital, police, restaurant, taxi, and train. Each dialog consists of a sequence user utterances and system responses, all annotated with the corresponding dialog state and system action.
More details in training/hyperrarameters can be found in Appendix D.
4.4 Main Results
Table 1 summarizes the results of end-to-end sponse generation performance on MultiWoZ.. As shown in Table 1, when trained with KRLS directly from backbone (KRLS in Table we achieve an improvement 1.4 in Combined Score compared to the baseline, which mostly comes from increased inform and success rate. Since inform/success rate evaluates how often informableerequestable slot values (i.e. keywords) are generated correctly, this suggests that KRLS can help reinforce model's ability to generate key tokens (see Section 6.1).
We follow the preprocessing procedure from Zhang et al. (2020a) to delexicalize slot values for each system response, and use the standardized evaluation script released by Nekvinda and Dušek (2021), which has also been adopted by the official MultiWoZ dataset.
When trained from an LsL-finetuned checkpoint (finetume+ KRLS), KRLS further improves to com- bined score of 103.8, with major improvements again in the success and inform rate. We believe this is because, the model has already been fine- tuned on the entire training dataset, the assumption mentioned in Section 3.1 is better satisfied (see Ap- pendix FF for examples). Then, KRLS can better improve a model's keyword generation ability as compared to the case when trained from backbone.
4.2 Evaluation Metrics
In our experiments, primarily consider the task of end-to-end response generation. In Mul- tiWoZ, response generation performance is eval- uated by a combination of three metrics: In- form rate measures whether the system has pro- vided an appropriate entity; Success rate measures whether the system has answered all the requested attributes; BLEU measures the fluency as com- pared to the references, which are also delexical- ized. Finally, the Combined score is calculated as (Inform Success) 0.5 + BLEU.
Table compares the training speed of the KRLS algorithm with standard RL training which uses auto-regressive generation. During standard RL training, we removed the SL step in Algorithm 1, and only use a terminal reward during RL as newly generated sequences no longer have a one-to-one mapping to the tokens in gold response (more de- tails in Appendix J). We then measure the total wall-clock time per epoch spent by each algorithm during training and separately during experience collection (line 7-14 in Algorithm 1). While ad- ditionally initializing KRLS with a LSL -snettnnt checkpoint (finetnn+KRL.) achieves a better per-
4.3 Model Architecture and Baseline
In this work, we use GODEL-base (Peng et al., 2022) as a backbone, which is a T5-base model (Raffel et al., 2020) pretrained on both texts and dialog datasets (except MultiWoZ).
Baseline We use MTTOD (Lee, 2021), which achieves previous state-of-the-arr performance in response generation by performing SL with addi- tional multi-task training. We re-train MTTOD

---

Response Generation
Model
Backbone
Inform
Success
BLEU
Combined
SOLOIST (Peng et al., 2021)
GPT-2
82.3
72.4
13.6
90.9
DoTS (Jeon and Lee, 2021)
BERT-base
80.4
68.7
16.8
91.4
UBAR (Yang al., 2021b)
DPs2
83,4
70.3
17.6
94.4
PPTOD (Su al., 2022)
bS-base
83.1
72.7
18.2
96.1
BORT (Sun al., 2022a)
TS-small
85.5
77,4
17.9
99.4
MTTOD (Lee, 2021)
TS-base
85.9
76.5
19.0
100.2
GALAXY (He et al., 2022)
UniLM-base
85,4
75.7
19.0
100.2
Mars-Gi (Sun et al., 2022b)
TS-small
88.9
78.0
19,9
103.4
Baseline (MTTOD)
GODEL-base
86.0
77.4
18.9
100.6
KRLS
GODEL-base
87.3 (87.2+0.3)
78.3
(78.2400.5)
19.2
(19.1+:0.3)
102.0 (101.9#0.5)
linetun+KRRLS
GODEL-base 89.2 (89.24:0.3)
80.3 (80.0+0.4)
19.0 (19.0:::22)
103.8 (103.5+004)
Table I: MultiWoZ. 2.2 end-to-end response generation evaluation. Results are "best run (14, 7)" over three runs.
The results of previous works from the official leaderboard of MultiWOZ." indicates concurrent work.
Algo
Generation Time Training Time
Metric
Rt+KRLS Win
MTTOD Win
Tie
KRLS min/epoch
306 min/epoch
Fluency
34.7%
48.0%
17.3%
std. RL
172 min/epoch
362 min/epoch
Appropriateness
55.3
30.7%
14.0%
Informativeness
60.7 %
26.7%
12.7%
Overall
59.3%
29.3%
11.3%
Table 2: Training speed comparison between KRLS and RL. In standard RL (std. RL), ruto-regressive sequence generation used experience collection.
Table 3: Human evaluation on the generated responses. * indicates p < 0.01. Fluency result has no statistical significance due to large variances among annotators.
formance, note that the same procedure is often used for RL training in NLP (Ramamurthy al., 2022). Due the large exploration space for lan- guage models, RL algorithms may require many more epochs to train without initializing from a finetuned checkpoint (see Appendix H). Therefore, we focus our comparison solely on running the KRLS algorithm and the standard RL algorithm. As shown in Table the experience collection time (Generation Time) for KRLS is much shorter than RL using auto-regressive generation, as in KRLS only a single forward pass needed for sequence generation. However, since KRLS ad- ditionally includes SL (68 min/epoch) and a per- token reward computation, the total training time per epoch becomes 306 min/epoch, though still 15% faster than the 362 min/epoch with standard RL, which only includes experience collection and PPO training (190 min/epoch).
responses are rated in terms of their 1) appropriate- ness?, 2) fluency, 3) informativeness, and 4) overall quality given dialogue context. We randomly picked 50 turns in the test set, and provided the generated responses without delexicalization and the dialogue history up to that turn. For each metric, the croke-sorkers were choose which response is better, if it is tie. We collected preference from crowd-workers per sampled turn.
Table summarizes the human evaluation re- sults. Our method has been rated more appropriate, informative, and overall more preferred by Turk- ers. We believe this coincides with the results in Table that our method performs better in inform and success rate, by providing more relevant key in- formation. There no statistical significance in the fluency result (p > 0.05), which is expected given both models' comparable BLEU scores. Specifi- cally, only 16% of the dialogues have more than one annotator rating MTTOD believe this is because many pre-trained LMs already generate fluent texts, and is often chal- lenging for humans to notice the difference.
4.5 Human Evaluation
fluent. We
We consider the possibility that automatic metrics from MultiWoZ may not correlate well with human judgements (Liu et al., 2016; Lubis et al., 2022). Thus, we ask crokerworkers on Amazon Mechani- cal Turk to compare responses generated by base- line (MTTOD) and finetumeKKRI (ft+ (R+K). The
For appropriateness and fluency, we followed the defini- tions from prior work (Zhang et al., 2020b: Ramachandran et al., 2021; Jang et al., 2022; Feng et al., 2023).

---

5%
10%
20%
Model
Inform
Success
BLEU
Combined
Inform
Success
BLEU
Combined
Inform
Success
BLEU
Combined
MTTOD
51.1
20.7
9.5
46.6
63.1
44.4
13.8
67.7
75.0
61.0
84.8
ft-RLS
55.0
22.7
11.5
50.3
64.8
47.4
71.9
78,9
65.0
89.2
Table 4: MultiWoZ end-to-end response generation performance using 5%, 10%, and 20% of training data.
"ft+KRIS"" refers to f Both models use GODEL-base as backbone. Results are shown mean values over three runs.
4.6 Low Resource Experiment
Algo
Inform
Suceess
Bleu
Total
Since large and well-annotated dialogue datasets such as MultiWoZ (Budzianowski et al., 2018) is not easy to create in practice, we also investigate KRLS's performance under a low-resource regime. We use 5%, 10%, and 20% of training data to train both baseline (MTTOD) and finetune+ KRLS, and report their performance in Table 4. In Table 4 we find our method outperforms the baseline in all settings. We also find large contributions often from improving inform and success scores, which indicates the effectiveness of KRLS at key token learning without abundant training data.
SL Only
86.0
77.4
18.9
100.6
RL Only
84.2
72.2
17.5
95.7
SL4GOLD 86.1
77.1
18.8
100.4
KRLS(PG) 88.7
78.7
19.1
102.8
Table 5: KRLS Ablation Study. The first two are trained directly from the backbone, and the latter two are trained from a LSL. -in-ttint checkpoint (see more details in Appendix K).
5.2 Reward Function Ablation
In Table 6, we empirically compare our proposed reward with 4 different reward functions and show that: a) providing a per-token reward in addition to providing a terminal reward for the entire sequence is helpful, and b) a tme-grained, context- aware re- ward that correctly factors in our generation proce- dure can further improve performance.
5 Ablation Studies
5.1 KRLS Algorithm Ablation
Since KRLS effectively combines SL. and RI., we illustrate the contribution of each component in Ta- ble 5. During SL Only and RL Only, remove the RL training and SL training KRLS, respectively. In SL+GOLD, we replace our RL procedure with GOLD (Pang and He, 2021), which is an offline and off-policy RL algorithm that learns from the gold demonstrations without any generation (see Appendix K for more details), so that we can also isolate the impact of additionally generating se- quence in our approach. Since GOLD uses Policy Gradient (Sutton et al., 1999) without clipping, we do the same with KRLS here (denoted KRLS(PG)).
In this experiment, we replace our proposed re- ward in Section 3.2 (denoted as Prob. in Table 6) with the following alternatives: Zero, which assigns zero score all generated tokens, hence only uses the terminal reward for training; Error, which as- signs a hard penalty of = whenever the generated token is correct/incorrect; BERTS., which uses the core mechanism in BERTScore (Zhang et al., 2019) to measure the semantic similarity between the gen- erated tokens and the gold tokens (see Appendix G for more details); Static., which takes the static, context-unaware token embeddings from the em- bedding layer of GODEL-base and compute their cosine similarity as reward. In all cases, the same set of hyperparameters is used to make results more comparable.
As shown in Table 5, using SL+GOLD only achieved a similar performance as compared to SL Only. We believe that this is because GOLD only intends to learn from the gold responses, while KRLS(PG) also attempts to explore other sequences/tokens to reinforce its keyword leamn ing. Additionally, if the SL objective is removed from KRLS when directly trained from backbone, the performance degrades as the assumption men- tioned in Section 3.1 becomes harder to satisfy (c.f. KRLS in Table 1, and discussions in Appendix H).3
As shown in Table 6, our proposed teken-level
ing RL entirely. However, weighted SL only yields a minor improvement compared to the baselines, reaching a score of 101.2. KRLS using a similar reward function (Error, see Sec- tion 5.2) already achieves 102.5. We believe this is because KRLS rewards/penalizes generated tokens sampled from the model's 's distribution using RL. while SL only uses the gold tokens. This finding motivates KRLS S to use RI. with reward function emphasizing on keyword learning.
'In our prior study, we also experimented with simple alternative of only using weighted SL. on key tokens, remov-

---

Finetune+KRLS
Keyword Generation Accuracy
0.8
Reward
Inform
Success
BLEU
Total
baseline
None
86.0
77.4
18.9
100.6
KRLS
Zero
88.3
77.9
18.9
102.0
0.4
Eror
88.8
78.5
18.8
102.5
Static. 88.7
78.4
18.8
102.4
BERTS. 88.5
78.7
18.9
102.5
0.2
0.73
Prob.
89.2
80.3
19.0
103.8
0.0
Table KRLS using different per-token reward when trained from finetuned checkpoint. None refers the baseline of training only with supervised learning.
8
10
Validation Epoch
Figure 4: Keyword generation accuracy during valida- tion. Baseline trained with only supervised learning. CSL. Both models are trained directly from backbone to additionally demonstrate the difference during early training.
reward (Prob.) outperforms all other alternatives. Interestingly, all reward functions that specified a per-token reward (i.e. Error, BERTS., Static., Prob.) achieved improvements over Zero, which only relies on the terminal reward. This indicates that more fine-grained per-token reward function is helpful. Additionally, Proh. improves upon Er- ror, BERTS., and Static., because it additionally factors in our generation procedure that the gener- ated sequence conditioned on the gold response. Therefore, can also correctly capture the contex- tual relationship of the generated tokens.
Algo
Inform
Success
BLEU
Total
KRLS
89.2
80.3
19.0
103.8
+DST
93.1
83.7
19.1
107.5
+Both
93.5
90.9
29.8
122.0
Train 93.7
90.9
Table 7: Test performance of KRLS when generating with Gold Dialog State (+DST), and with both Gold Dialog State and Gold System Act (+Both), Train is the performance of the training dataset. Note that +DST is same as the "policy optimization" task.
6 Analysis
6.1 Keyword Learning
Since KRLS aims to improve the model's ability to generate key information correctly, we track the model's accuracy in generating key tokens during training and validation. In this experiment, we feed in the gold contexts up to the key tokens, and the model is tasked to generate the next token. We then calculate the accuracy by measuring how often the generated token matches the gold key token.
from incorrectly generated dialog states and sys- tem acts (see Appendix O for examples). This is understandable, as we only used KRLS to improve the response generation component.
To quantify these errors, additionally use our K KRLS-triieed model to generate responses when a) the gold dialog state is provided (+DST) and b) both the gold dialog state and the gold system action are provided (+Both). We present this result in Table 7, and found that +DST improved the overall score by nearly 4 points, and +Both further improved the overall score by 14.5 points, almost reaching the performance of the training dataser This shows that much error remains in the DST and system act generation process, so the overall performance can further increase if techniques to separately improve DST and system act generation (e.g., Sun et al. (2022b)) can be combined with KRLS. We leave this for future work.
As shown in Figure 4, only performing SL (base- line) leads to a slow increase in keyword generation accuracy during early training, as the model fo- cuses on learning other non-key tokens due to their abundance. On the other hand, KRLS periodically uses RL help the model focus on learning key tokens, which leads higher keyword generation accuracy throughout both training and validation (more details in Appendix I).
6.2 Error Analysis
Despite reaching a higher inform rate and success rate as more key tokens are generated correctly, we still observe responses that miss some key to- kens. We found that these errors often originate
'In MultWWZ, the training dataset includes human errors. hence does not have perfect inform/success score. Valida- tion/test datasets are hand-picked to only include successful dialogs, so that model performance can he fairly evaluated.

---

7 Related Work
(also see Section 5.1 for an empirical comparison). Other Notable Related Techniques Qian et al. (2021) utilizes a student-tcacher architecture and MAML (Finn et al., 2017) to improve domain adap- tation ability of the student model. Specifically, the teacher model provides weights to scale the NLL loss of each gold token when training the student model. In this aspect, this is similar to GOLD, per- forming a "weighted learning" on the gold demon- strations. KRLS aims to directly improve TOD performance and achieves this by utilizing RL to perform a "weighted learning" on generated tokens.
End-to-end dialog systems such as Lei et al. (2018);
Yang et al. (2021a); Lee (2021); He et al. (2022) have shown promising results in TOD benchmarks such as MultiWoZ. However, as the standard SL ob- jective does not directly account for TOD metrics such as task success rate, much recent work seeks to incorporate RL techniques to improve TOD per- formance. In this section, we discuss related ap- plications of RL in TOD, well as other non-RL- based approaches that have similarities in concept.
RL for Text Generation Ranzato et al. (2016); Li et al. (2016); Zhou et al. (2017); Ramamurthy et al. (2022) applies RL to text generation tasks by treating each word as an action and then uses auto-regressive generation explore high-reward sequences. This results in a large action space for exploration, and these work focuses on methods to stabilize the training process. In principle, these approaches can be modified for TOD tasks, but they would still generally use a user simulator and/or the slow auto-regressive generation step.
8 Conclusion
In this work, we explore an approach to utilize RL to improve a model's TOD performance, but also to avoid using a user-simulator or the slow auto- regressive generation process. We propose the Key- words Reinforcement with Next-word Sampling (KRLS) training algorithm, which combines offline RL with a fast sequence generation scheme that di- rectly samples from next-word distributions after supervised training, and a fine- fine-grained per-tokes tor-token re- ward function that measures the importance and se- mantic closeness of each generated token. We then evaluate KRLS on the MultiWoZ dataset and show that a) it can help improve E2E response generation performance, reaching new state-of state-of-the-ar in the inform rate, success rate, and combined score; b) it can be trained 15% faster than using standard RL algorithm that performs auto-regressive generation during training/exploration.
RL for Policy Planning Many direct applica- tions of RL in TOD focus on optimizing dialog pol- icy planning (Takanobu al., 2020; Tseng al., 2021; Lubis et al., 2020). Takanobu et al. (2020); Tseng et al. (2021) jointly optimize both a user sys- tem and a dialog system to improve a model's 's TOD task performance and/or domain adaptation ability, but could be resource intensive as additional user- side training is needed. Alternatively, Lubis et al. (2020); Zhao et al. (2019) uses RL optimize sys- tem action generation in a latent space, but tends to result in the model generating very short responses (i.c., a low BLEU score of 10.8 in MultiWoZ).
Limitations
Although KRLS is faster to train as avoids auto- regressive generation, it is difficult for the model to learn/generate sequences significantly different from the gold examples in the dataset. Therefore. this could limit the potential to achieve perfor- mance better than the training dataset itself.
Offline RL in TOD Many offline RL applica- tions in TOD consider an actor-critic type algo- rithm (Jang et al., 2022; Verma al., 2022), which involves using critic to choose better responses among several generated candidates. These ap- proaches tend to be vulnerable to errors made by the critic model (especially for OOD actions (Levine et al., 2020)), and is resource intensive multiple ruto-regressive generations are needed per episode. Alternatively, Pang and He (2021) proposes the GOLD algorithm, which uses policy- gradient based method in an off-policy setting. by learning solely from the gold demonstrations with out any generation/exploration. KRLS additionally performs sequence generations and utilizes gold demonstrations in computing the reward function
Additionally, since during training KRLS creates sequences by conditioning on the gold response, whereas at inference we use auto-regressive gen- eration, the problem of compounding generation error (exposure bias) is re- re-introduced (Bengio et al., 2015; Venkatraman et al., 2015; Ranzato et al., 2016). Therefore, in this aspect KRLS trades its faster training speed with certain benefits brought by standard RL. training in NLP. In the future, it would be worthwhile to explore if a more fine- grained trade-off can be found between an efficient

---

sequence exploration strategy and those benefits inferred by using auto-regressive generation.
potential unsafe/toiiclofiesiiee responses gener- ated from the models. Without safety guardrails such as Arora et al. (2022); Lu et al. (2022), we do not advocate using any of our trained models in production settings.
Next, to make KRLS have minimal require- ments of extra resources, we avoid using user sim- ulators and perform offline RL training at turn- level. As result, KRLS does not perform ex- ploration/planning on a dialog-level, which can be very useful for tasks that require long-horizon plan- ning to be successful (e.g., persuading person to donate to charity (Wang et al., 2019)). We believe one way to extend KRLS could be using a GPT-like model to learn from an entire dialog, and combine with safe policy improvement methods to avoid potentially large bias and poor sample efficiency during dialog-level RL learning (Ramachandran et al., 2022). We leave this for future work.
References
Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. Director: Generator-classifie for supervised language modeling. In Proceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 12th Internationaa Joint Conference on Natu- rul Language Processing (Volume I: Long Papers), pages 512-526, Online only. Association for Compu- tational Linguistics.
Finally, in our runtime experiments (Section 4.4) we found that performing PPO (as well as PG) is a significant bottleneck, taking up more than half of the total training time. Future work may wish to consider ways to improve the speed/memory efficiency of computing those RL objectives to further reduce training time.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Ad- vances in Neural Information Processing Systems. volume 28, Curran Associates, Inc.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vorr Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates. Inc.
10 Ethical Considerations
Our work describes an algorithm to improve a model's TOD performance and to expedite the training process. It aimed at making current TOD systems easier to train, and also better at helping users to achieve their goals.
Generally, while most algorithms are not de- signed for unethical usage, there is often potential for abuse in their applications. In our experiments, we apply KRLS on the MultiWoZ (Budzianowski et al., 2018) dataset, to improve performance on tasks such as restaurant booking and hotel reserva- tion. However, because TOD training algorithms are typically aggostice,, it is possible to use them for unethical tasks, such as scamming. We do not condone the use of KRLS for any unlawful or morally unjust purposes.
Pawel Budzianowski, Tsung- Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Ultes Stefan, Ramadan Os- man, and Milica Gašić, 2018. Multiwoz - a large- scale multi-domall wizard-of-oz dataset for task- oriented dialogue modelling, In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing (EMNLP).
Mihail Eric. Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani- Tur. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01660
Additionally, since our experiments use pre- trained language models, another concern is on their (in)ability to generate safe, respectful content (Welbl al., 2021: Gehman al., 2020). Our work specifically focuses on improving TOD per- formance, and hence we caution users against any
Yihao Feng, Shentao Yang. Shujian Zhang, Jianguo Zhang, Caiming Xiong, Mingyuan Zhou, and Huan Wang. 2023. Fantastic rewards and how to tame them: A case study reward learning for task-oriented di- alogue systems. In The Eleventh International Con- ference on Learning Representations.
'In our implementation, we noticed that certain compu- tations could be cached to save time. However, we found it infcasible in our setting due to limited GPU memory. Future work may also investigate ways to improve memory efficiency in our implementation, to allow for potential speedups.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of

---

deep networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR.
Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Sim- plifying task-oriented dialogue systems with single sequence-to-sequenee architectures, In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 1437-1447, Melbourne, Australia. Association for Computational Linguistics.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealTooi- cityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online. Association for Computational Linguistics.
Sergey Levine. Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643
F. Maxwell Harper and Joseph A. Konstan. 2015. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4).
Jiwei Li. Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep re- inforcement learning for dialogue generation. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1192- 1202, Austin, Texas. Association for Computational Linguistics.
Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao. Dermot Liu, Peng Jiang. Min Yang, Fei Huang, Luo Si, et al. 2022. Galaxy: generative pre-trained model for task-oriented dialog with semi- supervised learning and explicit policy injection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10749-10757.
Chia-Wos Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. arXiv preprint 1:X3v:660300800
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Ad- vances in Neural Information Processing Systems, volume 33, pages 20179-20191, Curran Associates, Inc.
Siyang Liu, Chujje Zheng, Orianna Demasi, Sahand Sabour, Yu Li. Zhou Yu, Yong Jiang, and Minlic Huang. 2021. Towards emotional support dialog systems.
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. 2022. GPT-critic: Offfine reinforcement learning for end-to- end task-oriented dialogue systems. In International Conference on Learning Representations.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. QUARK: Controllable text generation with reinforced unlearning. In Advances in Neural Information Processing Systems.
Natasha Jaques, Asma Ghandeharioun, Judy Han- wen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. 2019. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, htss11977.0056
Nurul Lubis, Christian Geishauser, Michael Heck, Hsien-Chin Lin, Marco Moresi, Carel van Nickerk. and Milica Gasic. 2020. LAVA: latent action spaces via variational auto-encoding for dialogue policy op- timization. CORR, abs/2011.09378.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandehari- oun. Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2020. Human- centric dialog training via offline reinforcement learn- ing. Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 3985-4003.
Nurul Lubis, Christian Geishauser, Hsien-chin Lin, Carel van Niekerk, Michael Heck, Shutong Feng. and Milica Gasic. 2022. Dialogue evaluation with offline reinforcement learning. In Proceedings of the 23rd Annual Meeting of the Special Interest Group Discourse and Dialogue, pages 478-489, Edinburgh, UK, Association for Computational Linguistics.
Hyunmin Jeon and Gary Geunbae Lee. 2021. Domain state tracking for simplified dialogue system.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Pruceedings.
Tomá's Nekvinda and Ondiej Dušek. 2021. Shades of BLEU, flavours of success: The case of MultiWOZ. In Proceedings of the 1st Workshop on Natural Lan- guage Generation, Evaluation, and Metries (GEM 2021). pages 34-46, Online. Association for Compu- tational Linguistics.
Yohan Lee. 2021. Improving end-to-end trientri dialog system with a simple auxiliary task. In Find- ings of the Association for Computational Linguis- fies: EMNLP 2021. pages 1296-1303, Punta Cana, Dominican Republic. ACL.
Long Ouyang, Jeff' Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller,

---

Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015b. high-dimensional continuous control using generalized advantage esti- mation.
Richard Yuanzhe Pang and He He. 2021. Text gener- ation by learning from demonstrations. In Interna- nional Conference on Learning Representations.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithmss CoRR, abs/ N1017777063447
Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. Godel: Large-scale pre-training for goal-directed dialog.
Weiyan Shi, Kun Qian, Xuewei Wang. and Zhou Yu. 2019. How to build user simulators to train rl-based dialog systems, In EMNLP/JCNNLP (1).
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022. Multi-task pre-training for plug-and suug-and-pay trsk-triented dialogue system.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2021. Soloist: Building task bots at scale with transfer learning and machine teaching.
Haipeng Sun, Junwei Bao, Youzheng Wu, and Xiaodong He. 2022a. BORT: Back and denoising reconstruc- tion for end-to-end task-oriented dialog. In Find- ings of the Association for Computational Linguis- rics: NAACL 2022, pages 2156-2170, 70, Scattle, United States. Association for Computational Linguistics.
Kun Qian, Wei Wei, and Zhou Yu. 2021. A seadent architecture for dialog domain adap- tation under the meta-lcarning setting. CoRR, abs/2104.02689.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits transfer learning with a unified text-to-texy to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Haipeng Junwei Bao, Youzheng Wu, and Xiaodong He. 2022b. Mars: Semantic aware contrastive learn- ing for end-to-end task-oriented dialog.
aware
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approxima- tion. In Proceedings of the 12th International Con- ference on Neural Information Processing Systems, NIPS 199S,99, page 1057-1063, Cambridge, MA, USA. MIT Press.
Govardana Sachithanandam Ramachandran, Kazuma Hashimoto, and Caiming Xiong. 2021. Causal-aware safe policy improvement for task-oriented dialogue. arXiv preprint arXiv:2103.06370.
Govardana Sachithanandam Ramachandran, Kazuma Hashimoto. and Caiming Xiong. 2022. [CASPI] causal-aware safe policy improvement for task- oriented dialogue. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pages 92-102, Dublin, Ireland. Association for Computational Lin- guistics.
Ryuichi Takanobu, Runze Liang, and Minlie Huang. 2020. Multi-agent task-oriented dialog policy learn- ing with role-aware reward decomposition. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 625-638, Online. Association for Computational Linguistics.
Bo-Hsiang Tseng, Yinpei Dai, Florian Kreyssig, and Bill Byrne 2021. Transferable dialogue systems and user simulators. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 152- Online. Association for Computational Linguistics.
Rajkumar Ramamurthy, Prithwiraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage. Hannanch Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion.
Arun Venkatraman, Martial Hebert, and J Andrew Bagnell. 2015. Improving multi-step prediction of learned time series models. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. 2016. Sequence level train- ing with recurrent neural networks. In 4th Inter- national Conference on Learning Representations. ICLR 2016.
Siddharth Verma, Justin Fu, Sherry Yang, and Sergey Levine. 2022. CHAI: A CHatbot Al for task-oriented dialogue with offline reinforcement learning. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4471 17744, Seattle, United States. Association for Computational Linguistics.
John Schulman, Sergey Levine. Philipp Moritz, Michael Jordan, and Pieter Abbeel. 2015a. Trust region pol- icy optimization. In Proceedings of the 32nd In- ternational Conference on International Conference on Machine Learning- Volume 37, ICML' 15, page 1889-1897. JoL..

---

Xuewei Wang, Weiyan Shi, Richard Kim. Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Per- suasion for good: Towards personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguisties, pages 5635-5649, Florence, Italy, Association for Computational Linguistics.
Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020b. Task- oriented dialog systems that consider multiple appro- priate responses under the same context. In Proceed- ings the AAAI Conference on Artificial Intelligence, volume 34, pages 9604-9611.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan, 2020c. DIALOGPT Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Cop- pin, and Po-Sen Huang. 2021. Challenges in detox- ifying language models. In Findings of the Associ- ation for Computational Linguisties: EMNLP 2021, pages 2447-2469, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent vari- able models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- mologies, Volume (Long and Short Papers), pages 1208-1218. Minneapolis, Minnesota. Association for Computational Linguistics.
Ronald J. Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4))229-256.
Qingyang Wu, Lei Li. and Zhou Yu. 2021. Textgail: Generative adversarial imitation learning for text gen- eration. AAAI, 35:14067-14075.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2019. Alternating recurrent dialog model with large-scale pre-trained language models. CoRR, abs/1910.03756.
Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. 2017. End-to-end offline goal-oriented di- alog policy learning via policy gradient. CoRR. abs/1772.02238..
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, pages 2013-2011 Lisbon, Portugal. As- sociation for Computational Linguistics.
Yunyi Yang. Yunhao Li, and Xiaojun Quan. 2021a. Ubar: Towards fully end-to-end orientre dialog system with gpt-2. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 14238.
Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2021 Ubar: Towards fully end-to-end task-oriented dia- log systems with gpt-2.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. Multiwoz 2.2: dialogue dataset with addi- tional annotation corrections and state tracking base- lines. In Proceedings of the 2nd Workshop Natu- ral Language Processing for Conversational AI, ACL 2020, pages 109-117.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Confer- ence Learning Representations.
Yichi Zhang. Zhijian Ou, and Zhou Yu. 2020a. Task- oriented dialog systems that consider multiple appro- priate responses under the same context. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9604-9611.

---

A KRLS RL Details
Finetune+KRLS
RL Algo
Inform
Success
BLEU
Total
To avoid high variance during policy gradient:
PG PPO
88.7
78.7
19.1
102.8
VC(0) - log po(alc))
89.2
80.3
19.0
103.8
we consider a clipped version of this objective, borrowing from proximal policy gradient (PPO) (Schulman et al., 2017) to provide more stable training and prevent the policy from moving too far away from the pretrained language model. Similar to Wu et al. (2021), we consider optimizing the following surrogate objective:
Table 8: Performance comparison when training KRLS with Policy Gradient (PG) and clipped version (PPO).
C Modeling Details
We use GODEL-base model as backbone, which is an encoder-decoder architecture like T5-base, with 220M parameters. Similar to MTTOD, an additional decoder is initialized for response gener- ation (the other decoder for DST), which results in an additional ~60M parameters. The encoder is shared for both DST and response generation. This results in a total model size of ~380M parameters.
A())), clip(((()) e, c,
CRL min
(5) where A, -V((+) r is the advantage function (Schulman et al., 2015b), is the clipping parameter, and r(0) is the ratio of the new policy the old policy pold
D Implementation and Training Details
Pa(x (c) pgg (z c))
r BBen pol(I))) (())
(6)
For all experiments (including (re)training MT- TOD). we use Adam (Kingma and Ba, 2015) for optimization, linear schedule with an initial learn ing rate of 5c and warm-up steps total training steps.
practice, we found that adding additional value function heads Vo would 1) increase model size, making it difficult to train under our GPU setting and 2) since KRLS training is performed over limited number of epochs (Appendix E), we found fitting small value function head result in high variance during training. As such, we fixed the value function to be zero and used the return as an estimate A ~ Σ, 7/ = GG, which also signals the model the correct tokens to generate. We note that according to Schulman et al. (2015b) page 4, G2 is 7-just advantage estimator for Áς- For simplicity, we refer "KRLS" to use this clipped policy gradient objective unless explicitly mentioned otherwise.
For Baseline, we re-train MTTOD (Lee, 2021) using publicly released code and the best set of hyperparameters reported by the author. For KRLS and KRLS, best set of hyperparameters (see Appendix E for details) using grid search. As task success and inform rate in MultiWoZ is highly corre- lated with correctly generating the predefined set of informableerequestable slot values such as "[value "dddress]dsss" in the response, we use = 5 for those key tokens and μ. = 1 for others. In ad- dition, we add a terminal reward by measuring the F1-score of generated key tokens compared to the gold key tokens (see Figure 3) measure overall performance in keywords generation. Note that we did not add a BLEU score for terminal reward, as we found the SL training in KRLS is sufficient.
pick the
B
KRLS using PPO v.s. PG
As shown in Table we found that using the clipped objective (Equation (5)) in KRLS can achieve better performance as compared to simple PG, which can cause high gradient variance (Schul- man et al., 2015a; Wu et al., 2021) during training. We believe that this is also due to the clipped ob- jective preventing the new policy Po to move too far away from the old policy, which is useful in approach as approximated sequence generation by "sampling" from the gold response. Therefore, we believe that being more pessimistic (Schulman et al., 2017 77) about each update can be beneficial in our setting.
E KRLS Hyperparameters
For the reported results of KRLS in MultiWoZ, use l: = 3, 0.5 x total steps per epoch, sampling temperature during generation
1.1, top-p during generation of 0.9, terminal reward scale of 5, learning rate of 5e 5, learning rate de- cay of 0.2 total steps in training, and batch size of When trained from a L finetuned checkpoint,

---

we additionally add regularization term using KL divergence (against the baseline model) with a weighting of 0.01 to reduce over-optimization on the reward function (Ouyang et al., 2022; Rama- murthy et al., 2022; Jaques et al., 2020). During testing, we used auto-regressive generation with greedy decoding (same as Lee (2021)).
viewing the sampled sequence standalone gen- crated sentence will distort each token's 's contextual meaning, leading to sub-optimal performance.
Validation Combined Score
100
All of our experiments GPU, NVIDIA RTX A4000. Running KRLS on -80M encoder-decoder model (see Appendix C) for epochs takes about one day, as consists of 306 min/epoch for training and 32 min/epoch for validation.
a single
only
KRLS
5
10
15
20
25
Epoch
F KRLS Sampled Response Examples
Figure 5: Validation performance of KRLS and RL Only over time. RL Only, we remove the SL objective from KRLS, and for each sampled episode we additionally append its corresponding gold response.
We provide example comparisons between our generated response using next-word sampling (SAMPLED), and responses produced with auto- regressive generation (GENERATED) in Figure Figure 10, and Figure 11, All examples are gener- ated from a finttinntteee checkpoint. For short re- sponses, we observe that GENERATED are similar to SAMPLED. When responses get longer, SAM- PLED responses become less similar to the GEN- ERATED ones.
Algo
Inform
Success
BLEU
Total
SL only 86.0
77.4
18.9
100.6
RL only
84.2
72.2
17.5
95.7
KRLS 87.3
78.3
19.2
102.0
Table 9: Performance of individual components of the KRLS Algorithm when trained directly from backbone.
When the model is not yet Lsn-finetuned on the dataset, SAMPLED responses tend to repeat
keywords and can hardly be interpreted as a se- quence. For example, given the input context in Fig- ure 9, a sampled response looks like: "[value_stay] [welcome] [value_stay] vvalue_arrvve [taxi] [train] [value_stay] vvalue_stay] [value_stay] [value_stay] [valuc_stay]".
H KRLS Directly from Backbone
In Table 9, present the results when training directly from from the backbone. SL only refers to the baseline of only training with SL objective. RL Only refers to the KRLS algorithm with SL objective removed. KRLS refers to the full KRLS algorithm. When trained directly from backbone, removing the SL objective (RL Only) degrades the performance as the assumption mentioned in Sec- tion 3.1 becomes harder satisfy especially during early training (see Figure 5). This is sensible be- cause, without prior SL training, many sequences generated from our sampling method can be highly different from the ruto-regressive generated ones. When SL training is included in KRLS, the overall performance improves by 1.4 points, as the addi- tional SL. training makes it easier to satisfy our assumption and also made training much smoother (see Figure 5).
G BERTScore for KRLS
To apply BERTScore in our setting, we first treat our sampled sequence as a standalone generated sequence, and use the cosine similarity between ...old the embedding of each pair of token 2.5en ..t
after passing through LSL fsnetuunt GODEL- base as rewards. Note that as we naturally have a one-to-one mapping between the generated and gold sequences, we can skip the maximal similarity matching step.
However, as shown in both Table 6 and Table 13, BERTS. does not perform well as Prob. This is because BERTScore is designed measure the semantic similarity between two standalone sen- tences, while in our setting the generated sequence is conditioned on the gold response. Therefore, in cases when many generated tokens are incorrect,
I Additional Keyword Learning Curves
In addition to the keyword generation accuracy dur- ing validation when trained directly from backbone

---

(see Section 6.1), in this section we also show: a) keyword generation accuracy when trained during both training and validation in Figure 6; b) overall generation accuracy learning curves in Figure 7; c) keyword and overall generation accuracy curves when trained from LsI.-finetin -st.tfine checkpoint in Figure 8. Overall generation accuracy is mea- sured by how often generated token xx 1 gold matches the ground truth goold whether zzz is key token or non-key token. Keyword genera- tion accuracy only performs the above comparison when the ground truth token is key token.
4 epochs. We believe that this is because, without fine-grained per-token reward, RL might need many more epochs to figure out the importance of those keywords.
Algo
Inform
Success
BLEU
Total
RL
88.5
79.3
18.8
103.1
gold
KRLS
89.2
80.3
19.0
103.8
Table 10: Test performance when trained from a LSL. finetuned checkpoint for 4 epochs. RL refers remov- ing the SL step in KRLS, and replacing sequence "sam- pling" step with auto-regressive generation.
As shown in Figure 6, KRLS can achieve higher keyword generation accuracy than baseline during both training and validation. We believe this is because the RL component in KRLS, especially during the early stages of training. can help the model also learn the less abundant but more im- portant keywords as it has higher reward. As result, in Figure 7, KRLS training can also achieve a higher overall generation accuracy.
K Ablation Study Setup: GOLD
GOLD (Pang and He, 2021) is an offline, off-policy RL algorithm that directly learns from the gold examples in the dataset without any generation step. As the RL component KRLS also an offline RL algorithm, we compared KRLS to GOLD Section 5.1.
checkpoint as shown in Figure 8, KRLS further increases its keyword generation accuracy. However, in Fig- ure 8(b), the overall generation accuracy is similar fsn--fnetuned baseline. We believe this is because, after the model has learned generate most of the tokens correctly, it needs to maintain a balance between ever-generating keywords (lower overall accuracy) and correctly generating the key- words (higher keyword generation accuracy).
When trained from a
To implement GOLD in our experiments, we fol- lowed the descriptions in Pang and He (2021), and replaced the RL component in KRLS with GOLD, We denoted this as SL+GOLD in Table 5. Addi- tionally, as GOLD uses the simple policy gradient (PG), for fair comparison we also replaced our PPO objective with PG KRLS in this experiment. Finally. we kept our per-token reward function R in SL+GOLD as the reward function proposed by Pang and He (2021) is aimed at optimizing other metrics such perplexity.
J RL with Auto-Regressive Gen. Setup
Section 4.4, we compared the training time be- tween normal RL with auto-regressive generation, and our KRLS algorithm. As KRLS has an ad- ditional SL step during training, we removed this component to provide a fairer comparison against the normal RL procedure, which usually only in- cludes an auto-regressive sequence generation step during experience collection and PPO training. However, after auto-regressive generation, the gen- erated tokens no longer have a one-to-one mapping to the gold response. Therefore, in this setting we used a zero reward for each token, and terminal reward of keywords Fi (same as KRLS) as well as a BLEU score (as SL training is removed).
L Proof of SL Equivalence
Applying our definition of R (see Section 3.2) in Equation we get, if zeen goood is generated correctly and corresponds to R = 1, and with the discount factor y = 0:
VL(0) & V log pa (2
this gives the same gradient as the traditional su- pervised learning for SL. in Equation 1.
M Effect of K, μ and l in KRLS
In addition in Section 4.4, we found that our approach can also reach a better overall performance in Table 10 when trained from a Lsr. finetuned checkpoint for
faster training speed as shown
We empirically tested a range of hyperparameters for KRL.S, including x E (0.1, 0.5, 1.0) x steps per epoch, μ E {2, 5, 10), and & E (1,3,5} 5). We present the results in Table and Table 12.

---

Finetunc+KRLS
direction for further improvement in lines of making dialog state and system action generation
μ 2.0
1 5.0
10.0
more robust (Sun al., 2022b).
0.1m
102.2
102.4
102.3
0.5n
103.3
103.8
103.8
1.0n
102.3
102.1
102.5
Table 11; Effect of different and j on the combined score in MultiWOZ. represents the number of training steps in epoch.
Kins k
k = 3 k =
K 0.1n,
102.9
103.8
102.8
/r 5.0
Table 12: Effect of different & on the combined score in MultiWOZ. represents the number of training steps in an epoch.
N Additional Reward Function Ablation
We additionally show the effect of several per-token reward functions in our KRLS algorithm when trained directly from backbone (hence the assump- tion mentioned in Section 3.1 is harder to satisfy). As shown in Table 13, all variants using KRLS still achieved improvement from baseline (also Figure 6 and Figure 7). Specifically, Zero reward and Prob. reward achieved the highest and second highest, with 102.2 and 102.0 Combined Score, respectively.
KRLS Success
Reward
Inform
BLEU
Total
None
86.0
77.4
18.9
100.6
Zero
87.7
78.6
19.0
102.2
Error
86.2
78.6
19.2
101.6
BERTS.
87.2
78.1
19.0
101.7
Static.
87.2
77.6
19.3
101.7
Prob.
87.3
78.3
19.2
102.0
Table 13: Performance of Training Directly from Back- bone with KRLS using Different per-token Reward
O KRLS Error Examples
We present three examples in Figure 12, Figure 13. and Figure 14 when KRLS trained model does not generate the required key tokens. We observe that in most cases, error originates from incorrectly generated system action and dialog state. This hints

---

Keyword Generation Accuracy
Keyword Generation Accuracy
0.8
KRLS
0.6
0.4
0.78
0.4
0.2
0.2
0.73
baseline
0.0
KRLS
0.0
200
400
600
800
1000
1200 1400
8
10
Training Step
Validation Epoch
(a) Keyword Learning during Training
(b) Keyword Learning Validation
Figure 6: Keyword Generation Accuracy during Training and Validation. Baseline is the standard SL training using
LSL. Both baseline and KRLS are directly trained from backbone.
Overall Generation Accuracy
0.8
Overall Generation Accuracy
0.74
0.72
baseline
0.6
KRLS
0.70
0.74
0.68
0.73
0.2
baseline
KRLS
0.72
0.0
200
400
600
800
1000
1200 1400
Training Step
Validation Epoch
(a) All Tokens Learning during Training
(b) All Tokens Learning Validation
Figure All Token Generation Accuracy during Training and Validation. Baseline the standard SL training using
CSL.- Both baseline and KRLS are directly trained from backbone.
Keyword Generation Accuracy
0.770
baseline
Overall Generation Accuracy
0.738
baseline
finetune+ KRLS
finetune+ KRLS
0.736
0.766
0.764
0.734
0.762
0.733
0.760
0.732
1.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Validation Epoch
Validation Epoch
(a) Key Tokens Learning Validation
(b) All Tokens Learning Validation
Figure 8: Key and All Token Generation Accuracy during Training and Validation. Baseline is the standard SL. training using LsL. finetunt+ is trained from Lst, finetuned checkpoint, i.e. Baseline.

---

can you help ne find a moderately priced place to stay?
sure. what area would you Like to stay in? sure, what area of town would like to stay in?
GOLD
GENERATED
sure, what area you Like stay
Figure 9: Example sequence generated by KRLS. Texts in black are the user's utterances. GOLD represents the gold response. GENERATED represents response produeed using ruto-segressive generation. SAMPLED represents response produced using the next-word sampling method in KRLS.
i need to go to avalon and could you give me the car type and contact number once booked?
have booked it. you will look for a [value_typer the GOLD contact number is [vatue_phone]
i have booked [value_car] for you. the contact number is @value_phone].
GENERATED
1 have booked for, the will be for [value _aree.carr.______ their SAMPLED
contact nunber is 23
Figure 10: Example sequence generated by KRLS. Texts in black are the user's utterances. GOLD represents the gold response. GENERATED represents response produced using auto-regressive generation. SAMPLED represents response produced using the next-word sampling method in KRLS.
would Like to the westside, at place with free parking and wifi.
there is the [aatue_name]] it is nice and is [ealue_star]]]] star [aalue_type].
GOLD
[vatue_ [aau]e [value_stars] star [value_tyee [ype] that GENERATED fits your needs.
paluue_nane" is the wvaluennamel it is and is [alue_stars]] [value_type].
SAMPLED
Figure II: Example sequence generated by KRLSS. Texts in black the user's utterances. GOLD represents the gold response. GENERATED represents response produced using auto-regressin generation. SAMPLED repsens response produced using next-word sampling method in KRLS.

---

Dialog: SNG01380
an looking For particutar restaurant its name is called restaurant alinentum
"restaurant":
"name": "restaurant Linentuum
[restaurant] [inforn] Food price area nane [offerbook]
ACT
[restaurant] [inforn] Food price area address name postcode phone
[value_ [value_name] is [vatue_ pricerange] [value_food] restaurant in the [value_aren) _areee you
[vatue_pricerange] reservation for you
RESP
[vatue_nane] offers [vadue_food] Found [value area] has ]
price range [vatue phone] (value postcode]
[value address]
Figure 12: KRLS Generation Error Example 1. Texts in black are the input context, in blue are the generated tokens, and yellow/gold are the ground truth. Texts highlighted in red incorrec//missing key tokens compared the ground truth.
Dialog: SNG8516
looking for place serving indian food in the moderate price range
have head [vatue_name] is fantastic what area are Looking for
the area does not natter curry prince sounds good please book table for people 19:30 on tuesday
"restaurant":
g
"Food": "indian",
"pricerange": "moderate",
"name": "curry prince",
"time": "19:38",
"day": "tuesday",
"people": "7"
[restaurant] [offerbooked] reference [generat] [reqmore]
ACT
[restaurant:] [offerbooked] people day Line reference
RESP
booking successful the table will reserved for 15 ninutes reference number [value_referencee] is there anything else help you with
ok have a table booked for [value_pennt people] at Ivauue time on Ivatue day
your reference number is [eatue_reference]]
Figure 13: KRLS Generation Error Example 2. Texts in black are the input context, in blue are the generated tokens. and in yellow/gold are the ground truth. Texts highlighted in red are incorrect/missing key tokens compared to the ground truth.

---

Dialog: MUL2177
am Looking for place in canbridge it does n't need to include internet and should be a hotel
not able to find [value_ type] type with internet there [value_type] with internet
okay that would be fine as long as has free parking
okay you any preference price range no how are available ?
there are (value_choice) different hotels around the city with free parking do you have preference for city area star rating
curry prince good please book
for 7 people at 19:39 tuesday
"hotel":
6
"type": "hotel",
"parking"; "yes",
"internet": "no"
"pricerange": "dont care"
"ares": "dont
"stars": "dont care"
[hotel] [recomnend] nane
[hotel] [recomeend] nane area
would suggest [value_ name]
hard say about the [value_nane] [value_ar _reaa
part of town
Figure 14: KRLS Generation Error Example 3. Texts in black are the input context, in blue the generated tokens, and in yellow/gold are the ground truth. Texts highlighted in red are incorrect/missing key tokens compared to the ground truth.


Reviews:
[{"review_id": "GCFt3Wussi", "reviewer": "Reviewer_GMA2", "summary": "", "questions": "1. In section 3.3, training the model with SL over several batches is enough for the assumption in section 3.1? Because the dialogue generation task is challenge. I think maybe the model can't generate a response that reflects the keywords yet.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and acknowledgement of our comprehensive experiments and analysis!\n\n1. a. “The final results of KRLS and baseline are close.”\n     \\\n     \\\n    We agree, but we believe improvements on the MultiWoZ dataset is very difficult. KRLS achieved more than 3 points of improvement compared to a strong baseline (MTTOD), reaching SOTA at a score of 103.8. Mars-G reaching 103.4 is a concurrent work with no public code or model weights available as of 08/24/2023. Additionally, we believe our approach is parallel to Mars-G, which focuses on improving DST and system act generation, while KRLS focuses on improving response generation.\n     \\\n     \\\n    b. “KRLS affects the Fluency of the generated responses (human evaluation)”\n     \\\n     \\\n     We note that in Table 3, human evaluation on fluency had **no statistical significance**. Specifically, only 16% of the dialogues has more than 1 annotator rating the MTTOD (baseline) as more fluent. We believe this is because many pretrained-LMs can already generate fluent texts, and it is often hard for humans to notice the difference. Additionally, in Table 1 we find both baseline and KRLS achieving a similar BLEU score of 19.0. We thus believe our method has no significant effect on the fluency of the generated responses. We will clarify this in our final manuscript!\n\n2. “the comparison in Table 2 is not fair enough” and “should take into account the time needed to fine-tune the model”\n     \\\n     \\\n     RL training in NLP often **also** initializes with a fine-tuned checkpoint (Ramamurthy et al, ICLR 2023). This is because actions in NLP tasks are often defined by the sequence of tokens in a response/text, hence it is difficult to train from scratch using RL as the exploration space is huge. We also showed this difficulty empirically in Appendix I and Figure 5: training with RL required 25 epochs to achieve a similar task performance compared to KRLS (w/o finetune) after only 5 epochs.\n     \\\n     \\\n    **However, we agree that this is a good point and should be mentioned in the main text**. We will add clarifications on this in our final manuscript.\n----\n#### Response to Questions:\n1. “Is training the model with SL over several batches enough for the assumption in section 3.1?”\n     \\\n     \\\n    KRLS alternates between SL and RL every epoch to help alleviate the assumption in section 3.1. In our experiments, we found that KRLS improved 1.4 points over baseline to 102.0, while finetune+KRLS further improved to a total score of 103.8. We believe initializing KRLS from a fine-tuned checkpoint helped to better satisfy the assumption and hence further improved the result, as the model has already been trained on the gold utterances several times (L318-237, Section 4.4).\n----\n[Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://arxiv.org/abs/2210.01241) (Ramamurthy et al, ICLR 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is clear and reasonable.\n\n2. The paper makes sufficient experiments and analyses to illustrate the effectiveness of the method.\n\n3. The paper is well-organized and clearly writing.", "reasons_to_reject": "1. Even though the paper demonstrates the improved effectiveness and faster training speed of this method, I think the authors should have explained more about the necessity of this method, considering the close final results of KRLS and baseline in Figure 4, and the fact that KRLS affects the Fluency of the generated responses (human evaluation). I agree that focusing on learning keywords will make it easier for the model to learn the message of the response, but the fluency of the response is still an important evaluation metric.\n\n2. The experimental results show that \"finetune + KRLS\" achieves significant results, which means that the effectiveness of this method is greatly influenced by the generative model. So I think the comparison in Table 2 is not fair enough. Since the comparison in Table 10 uses \"finetune + KRLS\", I think the comparison in Table 2 should take into account the time needed to fine-tune the model on the dataset first.", "questions_to_authors": "1. In section 3.3, training the model with SL over several batches is enough for the assumption in section 3.1? Because the dialogue generation task is challenge. I think maybe the model can't generate a response that reflects the keywords yet.", "paper_topic_and_main_contributions": "This work proposed a keywords reinforcement Learning framework with next-word sampling.  This method introduces a fine-grained reward function to learn more information about the keywords in a dialog, and reduces the training time compared with the standard Reinforcement Learning method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "upzTcW3v6L", "reviewer": "Reviewer_vXVC", "summary": "", "questions": "Please refer to the above “Paper Topic And Main Contributions” part.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We will update our final manuscript according to your questions/concerns to improve our clarity!\n\\\n\\\nOur main contributions are an efficient offline RL algorithm that 1) approximates auto-regressive generation with next-word sampling + SL, and 2) a per-token reward to improve a model’s keyword generation ability. We employed KRLS on MultiWoZ, and show that it can achieve both 15% training time reduction and improvements in test performance. We also presented various ablation studies (Section 5) and analysis (Section 6) to support our results.\n\n1. a. “The design of a fine-grained reward model is based on keywords, and defining keywords can be cumbersome”\n    \\\n    \\\n    We focused our study on task-oriented dialogues (as in dedicated in our title) because keywords in those tasks often come readily defined: e.g. restaurant name/address from restaurant booking tasks in the Multiwoz dataset, and movie name/ratings from movie recommendation datasets (L247-252). Additionally, we believe for other dialogue tasks such as QA/social chat, keywords can often be automatically processed and defined, such as the entities mentioned in WikiQA answers (Yang et al., EMNLP 2015), and the intent keywords in ESConv (Liu et al., ACL-IJCNLP 2021).\n    \\\n    \\\n    b. “reward definition in the paper is not very convincing and lacks empirical analysis”\n    \\\n    \\\n    In Section 5.2, we empirically compared our reward function with 4 other common reward functions. In Table 5, we find that our reward function (denoted as Prob., L415-416) performed the best (103.8 points), compared against alternatives such as only using terminal reward (denoted as Zero, 102.0 points), and using both terminal reward and BERTScore for a soft penalty on each generated token (denoted BERTS.,102.5 points). Comparisons between all reward functions in Table 5 also showed that a) using a per-token reward is beneficial and b) a context-aware reward can further improve performance (see Section 5.2, L408-414). Both of these aspects are included in our reward function design to achieve the best performance.\n    \\\n    \\\n    c. “explanation of the Reward Model is unclear”\n    \\\n    \\\n    We apologize. In sum, Section 3.2 and Equation 4 describes how our reward function is calculated for every token. Our per-token reward function combines two scores, the semantic closeness score of each generated token and its importance score. In general, the closeness score for each token is computed using a decoder network (L200-207), and the importance score $\\mu$ is a hyperparameter depending on whether the token is a keyword or not (L212-214). We mainly used $\\mu=5$ for key tokens and $\\mu=1$ otherwise (for experiments on hyperparameters see Appendix N).\n    \\\n    \\\n    d. “I do not understand how R is calculated when xgen != xgold”\n    \\\n    \\\n    In general, R is calculated using Equation 4. Under the special circumstance that a) a generated token is correct, the closeness score is fixed to 1.0, and b) a *keyword* is generated incorrectly, the closeness score is fixed to -1.0 to emphasize keyword learning (L207-211). In all other cases we use the closeness score computed by the decoder (see L200-207), i.e. the general procedure also described in our response above. We will make further clarification on this in our manuscript.\n\n2. “The next word sampling procedure does not seem special, as it uses the inherent capability of transformers”\n    \\\n    \\\n    The next-word sampling procedure (Section 3.1) is used by KRLS to approximate and replace the slow auto-regressive generation step used in traditional RL. In traditional RL algorithms, this auto-regressive process is used but is time-consuming as each token in a sequence is decoded **sequentially**. In KRLS, we introduce the next-word sampling procedure which effectively decodes each token **in parallel** by conditioning on the gold context, achieving significant speed improvement (Table 2, and L339-349). When performed after SL steps in KRLS, we find sequences decoded using next-word sampling can then be used as the exploration step (see Q1 from reviewer GMA2 for more details), replacing the traditional auto-regressive generation procedure.\n\n3. “Introduction of the baselines, KRLS, and finetune KRLS, is very confusing. Does KRLS not involve RL? What’s the difference between finetune+KRLS and KRLS?”\n    \\\n    \\\n    We are sorry for the confusion. In our main experiments (Section 4.3), the baseline is MTTOD which mainly performs SL. *KRLS* then replaces such SL training with the KRLS algorithm (L296-298), which involves RL. *finetune+KRLS* has identical training procedure as *KRLS*, but is initialized with a SL-finetuned checkpoint (L299-303). Performing RL from a SL-finetuned checkpoint is a commonly used technique for applications in NLP, see Ramamurthy et al, ICLR 2023 for example.\n\n4. “Why can the assumption (Section 3.1, L159-164) lead to next-word sampling approximating sequences generated by auto-regressive generation?”\n    \\\n    \\\n    Auto-regressive generates each token sequentially by conditioning on previously generated tokens. Next-word sampling generates each token in parallel by conditioning on the gold context. If we assume that the model “generates sequences similar to the gold responses in the training set” (L160-161), then previously generated tokens are similar to the gold context. This leads to sequences generated by next-word sampling approximating that generated by auto-regressive generation.\n\n5. a. “Equation 4 does not clarify how R is calculated when xgen != xgold”\n    \\\n    \\\n    Please see response 1.c and 1.d.\n    \\\n    \\\n    b. “definitions of xt and xgold are unclear, and they often seem to mean the same thing”\n    \\\n    \\\n    In L210-211, $x_t$ should be $x_t^{gold}$. We apologize for this typo, and thank you for pointing it out. In all other places, we believe we used superscripts such as $x_t^{gen}$ or $x_t^{gold}$ to differentiate when it is ambiguous. We will update this in our final manuscript.\n\n6. “In Table 1, methods in the leftmost column are not defined”\n    \\\n    \\\n    The first 7 methods (up to GALAXY) are official results reported in the MultiWoZ leaderboard as of submission. Mars-G relies on improving the DST/action generation module prior to response generation (L481-486). It is a concurrent work with results published recently but has no public code/model weights. The baseline and our methods (KRLS) in Table 1 are explained in Section 4.3 and our response to question 3. We will add short descriptions of those methods in our final manuscript for clarity.\n\n7. “What does ‘normal token’ mean in Figure 2?”\n    \\\n    \\\n    We refer to all non-key tokens as normal tokens. We will clarify this in our final manuscript.\n\n8. “The quotation marks used in line 298 are inconsistent with the rest of the paper”\n    \\\n    \\\n    Thank you for pointing this out! We will correct L296, L298, and L303 to use italics.\n---\n[WikiQA: A Challenge Dataset for Open-Domain Question Answering](https://aclanthology.org/D15-1237) (Yang et al., EMNLP 2015)\n\\\n[Towards Emotional Support Dialog Systems](https://aclanthology.org/2021.acl-long.269) (Liu et al., ACL-IJCNLP 2021)\n\\\n[Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://arxiv.org/abs/2210.01241) (Ramamurthy et al, ICLR 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Please refer to the above “Paper Topic And Main Contributions” part.", "reasons_to_reject": "Please refer to the above “Paper Topic And Main Contributions” part.", "questions_to_authors": "Please refer to the above “Paper Topic And Main Contributions” part.", "paper_topic_and_main_contributions": "This paper presents an offline RL-based algorithm to improve response generation for Task-Oriented Dialogue (TOD) systems. It employs a fine-grained reward function to guide the agent in focusing on key information, such as slot/slot-value-related words.\n\nI agree with the motivation of having the dialog agent focus on key information, and the use of a fine-grained approach is consistent with the analysis. However, there are several concerns:\n\n1、The core contribution of this paper seems to be the design of a fine-grained reward model based on keywords. However, these keywords need to be manually defined, which can be cumbersome. Additionally, the explanation of the Reward Model section is unclear, and I do not understand how R is calculated when xgen != xgold. The reward definition in the paper is not very convincing and lacks empirical analysis.\n\n2、The next word sampling section merely uses all gold tokens as input and leverages the next word distribution corresponding to each token. This capability is inherent to transformers and does not seem special.\n\n3、The introduction of the baselines, KRLS, and finetune KRLS, is very confusing. Lines 197-198 mention that KRLS replaces SL with the proposed KRLS algorithm, so does KRLS not involve RL? What is the difference between the SL part of finetune+KRLS and KRLS?\n\nBesides, several parts of the paper are hard to follow, such as:\n\n1、Lines 156-164 are confusing. Why can the “assumption” lead to the approximation of “sequence generation by sampling from the next-word distributions conditioned on the gold response? ”\n\n2、Equation 4 does not clarify how R is calculated when xgen != xgold. Moreover, the definitions of xt and xgold are unclear, and they often seem to mean the same thing.\n\n3、In Table 1, the methods in the leftmost column are not defined. What does “normal token” mean in Figure 2? \n\n4、What is the purpose of placing MDP in line 162?\n\n5、The quotation marks in line 298 are inconsistent with the rest of the paper.\n\nConsidering the issues mentioned above, I would recommend rejecting this paper.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please refer to the above “Paper Topic And Main Contributions” part."}, "extra_text": {}}, {"review_id": "hNeAEZx7zt", "reviewer": "Reviewer_uymB", "summary": "", "questions": "Since you have already has these kind of 'annotated' data from supervised model with weights, have you tried use normal supervise learning such as weighted cross entropy to finetune the model?\n\nIn next word sampling process, because each sampled token was based on golden response and it was not based on previous sample results, can I say the length of the trajectories is equal to 1?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and acknowledgement of our approach/experiments/analysis!\n\nResponse to Questions:\n1. “have you tried using weighted supervised learning”?\n    \\\n    \\\n    Yes! In our prior experiments, we performed a weighted supervised learning on key tokens and found a minor improvement of 0.6 points to a score of 101.2 compared to the baseline. KRLS using a similar reward function (see reward named Error in Section 5.2 L419-421) achieves 102.5. We believe this is because KRLS rewards/penalizes *generated* tokens sampled from the model's distribution, while SL only uses the *gold* tokens. However, this prior experiment did motivate our design of reward function emphasizing on keyword learning. We will add this discussion in our final manuscript.\n\n2. “Can I say the length of the trajectories is equal to 1” in the next word sampling process?\n    \\\n    \\\n    Yes. Given a gold sequence of length T, next-word sampling samples T tokens in parallel to construct a “generated” sequence. Thus, sampling each of the T tokens using this procedure can be seen as generating a trajectory of length 1 as you described.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well-written and easy to follow.\nThe proposed method is very straightforward and particle, the authors understand the importance of keywords in response for TOD and try to design a reward function with 'weighted' (importance) probabilities from another decoder models.\nThe experiments results are solid, the compared baselines, metrics, and ablation study makes sense to me. They also conduct human trials to valid the performance.", "reasons_to_reject": "None", "questions_to_authors": "Since you have already has these kind of 'annotated' data from supervised model with weights, have you tried use normal supervise learning such as weighted cross entropy to finetune the model?\n\nIn next word sampling process, because each sampled token was based on golden response and it was not based on previous sample results, can I say the length of the trajectories is equal to 1?", "paper_topic_and_main_contributions": "This paper proposes KRLS, an RL-based method to reinforce model generate responses with keywords. To reinforce keywords generation in from models, authors design a token-wised reward function by using a probability from a supervised fine-tuned decoder model given golden input, then scale the reward for keyword to ensure the keywords have higher probabilities to be generated.\nIn addition, since the typical RL methods relied on autoregressive generation process which takes time to sample trajectories, this paper introduces using next word probability with a supervised LM to speed up the sample process. \nIn their experiments, they used MTTOD as a supervised model before apply RL.\nThe authors evaluate the performance on well-known TOD metrics that includes BLEU, inform, success, and bleu. They also compare the several state of the art baselines and conduct the human evaluation.\nThe experimental results show that KRLS performs better than baseline on the metrics, the generation process is also faster than standard RL.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the three reviews, the majority of reviewers are positive about the paper's contributions and presentation. Reviewers appreciate the clear motivation and well-organized structure of the paper. The experimental results are considered solid, and the proposed method shows improvement over baselines. The use of a fine-grained reward function to focus on keywords is seen as a valuable contribution.

However, there are also some concerns raised by the reviewers. One reviewer questions the necessity of the proposed method, as the improvement over baselines is not substantial and the fluency of the generated responses is affected. Another reviewer points out the unclear explanation of the reward model and questions the uniqueness of the next word sampling approach. The confusion around the baselines and the comparison in Table 2 is also mentioned.

Overall, the paper is well-written and the proposed method is straightforward and valuable. The experimental results and analysis support the claims. However, there are some areas that need improvement, such as further explanation of the reward model and clearer comparison with baselines. The necessity and effectiveness of the proposed method should be addressed more thoroughly.
Decision: Accept-Main