Enhancing Task-oriented Dialogue Systems with Generative Post-processing Networks
Authors: Atsumoto Ohashi
Ryuichiro Higashinaka
Venue: EMNLP 2023 Main (2023)
Keywords: Task-oriented Dialogue System
Reinforcement Learning
Natural Language Generation
TL;DR: We propose generative post-processing networks (GenPPNs), a new component for improving task completion in dialogue systems by modifying the output of natural language generation modules via reinforcement learning.

Abstract: Recently, post-processing networks (PPNs), which modify the outputs of arbitrary modules including non-differentiable ones in task-oriented dialogue systems, have been proposed. PPNs have successfully improved the dialogue performance by post-processing natural language understanding (NLU), dialogue state tracking (DST), and dialogue policy (Policy) modules with a classification-based approach. However, they cannot be applied to natural language generation (NLG) modules because the post-processing of the utterance output by the NLG module requires a generative approach. In this study, we propose a new post-processing component for NLG, generative post-processing networks (GenPPNs). For optimizing GenPPNs via reinforcement learning, the reward function incorporates dialogue act contribution, a new measure to evaluate the contribution of GenPPN-generated utterances with regard to task completion in dialogue. Through simulation and human evaluation experiments based on the MultiWOZ dataset, we confirmed that GenPPNs improve the task completion performance of task-oriented dialogue systems.

Enhancing Task-oriented Dialogue Systems with Generative Post-processing Networks
Atsumoto Ohashi
Ryuichiro Higashinaka
Graduate School of Informatics, Nagoya University o atsumoto. coês. mail. nagoya-u. jp higashinaka@i nagoya-u. www.aaaaaiiinnaaauaaaaa jc.
Abstract
MLP-based PPN
MLP--ased
NLU
DST
PPN
Recently, post-processing networks (PPNs), which modify the outputs of arbitrary mod- ules including non-differentiable ones in task- oriented dialogue systems, have been pro- posed. PPNs have successfully improved the dialogue performance by post-processing nat- ural language understanding (NI.U), dialogue state tracking (DST), and dialogue policy (Pol- icy) modules with classification- -based ap- proach. However, they cannot be applied to natural language generation (NIG) modules because the post-processing utterances out- put by NLG modules requires a generative ap- proach. In this study, we propose new post- processing component for NLG, generative post-processing networks (GenPPNs). For op- timizing GenPPNs via reinforcement learning, the reward function incorporates dialogue act contribution, measure to evaluate the con- tribution of CeenPPN-generatie utterances with regard task completion in dialogue. Through simulation and human evaluation experiments based on the MultiWOZ dataset, we confirmed that GenPPNs improve the task completion per- formance of oriented dialogue systems'.
PLM-based GenPPN
MLP-based PPN
NLG
Policy
Diagrams of MLP-based PPN for NLU, DST, and Policy modules proposed by Ohashi and Hi- gashinaka (2022b) and PLM-based GenPPN for NLG proposed in this study.
Figure 1:
systems that consist of arbitrary modules including non-differentiable ones (e.g., rule-based and Web API-based). Their method uses RL optimize neural-based component called PPN, which modifies the output of each module. PPNs have been applied to three modules, namely NLU, DST, and Policy, and have been shown to improve the task completion performance of various pipelined systems. In PPNs, the post-processing of module outputs is treated as binary classification task, i.e., adding removing slots in the output of each module, and this classification task is modeled with multi-layer perceptron (MLP). This means that the current PPNs could not be applied to NLGs, where the unit output is not slots but a sequence of tokens, i.e., natural language.
1 Introduction
A typical task-oriented dialogue system has a pipelined structure consisting of four mod- ules (Young et al., 2013; Zhang et al., 2020): natural language understanding (NLU), dialogue state tracking (DST), dialogue policy (Policy), and natural language generation (NLG)). Many stud- ies have used reinforcement learning (RL. to im- prove the task completion performance of an entire pipelined dialogue system by fine-tuning modules directly (Lee et al.. 2021; Lin et al., 2021; Chen et al., 2023).
To overcome this limitation, we propose Pre- trained Language Model (PLM)-based Generative Post-processing Network (GenPPN) that can post- process the output of NLGs in manner similar to conventional PPNs (Figure 1). To optimize GenPPN via commonly used RL frameworks for PLMs (Ziegler et al., 2019; Stiennon et al., 2020), reward for each utterance at each turn (an utferance- level reward) is required. In task-oriented di- alogue, however, reward indicating success failure is obtained only at the end of multi-tur in- teraction (a dialogue-level reward). Therefore, we introduce a dialogue act (DA) contribution for dis- tributing the dialogue- reward to the utterance-
Recently, Ohashi and Higashinaka (2022b) pro- posed novel method using Post-Processing Net- works (PPNs) that can optimize pipelined dialogue
OOu code is publicly available at https: tttpgtttttttttt/t////////tt com/ nu-dialon Mi-dialoogu//GePPPy

---

level reward. Here, a DA is the meaning represen- tation of the information that NLG converts into an utterance, and DA contribution is measure of how much the DA each utterance contributes to the final task completion of the dialogue.
framework that simultaneously optimizes the Pol- icy module of the dialogue system and the user simulator. RL framework for online DST op- timization was also proposed to improve dialogue management performance (Chen et al., 2023).
Experiments on the MultiWOZ dataset (Budzianowski et al., 2018) confirm that GenPPNs improve the task performance of the entire dialogue system, regardless of the architecture of the NLG module. Furthermore, an ablation study reveals that the introduction DA contribution is effective for learning GenPPNs to improve task completion. The contributions of this study are threefold:
Instead of training each module, Ohashi and Hi- gashinaka (2022b) proposed generalized method that optimizes PPNs, classification-based models that modify the outputs of NLU, DST, and Policy, to enhance the task completion performance. this paper, we propose a new generative based PPN for post-processing NLG modules, which has not been supported by conventional PPNs.
2.2 Natural Language Generation Module for Task-oriented Dialogues
We propose the generative post-processing network (GenPPN) to modify the output utter- ances of the NL.G module, which was impos- sible with conventional PPNs.
Conventional NLGs for task-oriented dialogues used template-based basedd or carre-baaee methods (Walker et al.. 2002; Stent et al.. 2004). Later, data- driven methods using machine learning were pro- posed (Oh and Rudnicky, 2002; Angeli et al., 2010; Mairesse and Young, 2014) that do not require the cost of template and rule creation.
We introduce DA contribution to optimize the GenPPN by evaluating the impact of each ut- terance on dialogue task completion.
Simulation experiments on the MultiWOZ dataset confirm that the proposed GenPPN im- proves the task completion performance the entire dialogue system, regardless of the archi- tecture of the NLG module. We also validated that GenPPN optimized using simulation is effective in human evaluation experiments.
In recent years, many generative models based on deep learning have been proposed (Wen et al.. 2016; Tran and Nguyen, 2017; Su et al., 2018 8). Wen et al. (2015) proposed SC-LSTM that trols utterance generation using DA feature vectors and reading gates. SC-GPT (Peng et al., 2020) is the best NLG model of MultiWOZ, It achieves high performance by fine-tuning GPT-2 (Radford et al.. 2019) 9) on many task-oriented dialogue datasets such as MultiWOZ (Budzianowski et al., 2018) and the Schema- Dialogue Dataset (Rastogi et al., 2020).
2 Related Work
2.1 Optimization of Pipelined Task-oriented Dialogue Systems
Methods have been proposed to optimize the task completion performance of an entire pipelined sys- tem using RL. Zhao and Eskenazi (2016) and Li al. (2017) optimized a Policy network imple- mented in MLP using the Deep Q-Network algo- rithm (Mnih et al., 2013) to achieve robustness against errors that occur in real dialogues. Mehri et al. (2019) proposed a method to add additional parameters to NLU, Policy, and NLG and optimize them using RL. By expressing dialogue state out- put by DST probability distribution, Lee et al. (2021) made the entire system differentiable and jointly optimized it via RL. A method of fine-tuning an NLU while optimizing a Policy in dialogue sim- ulations was proposed (Lin et al., 2021). Tseng et al. (2021) proposed a domain admatin-daptive learning
For task-oriented dialogue, is crucial not only to generate natural utterances via maximum likeli- hood estimation (MLE) but also to accurately re- flect the input DA's content. To achieve this, Bal- akrishnan et al. (2019) introduced conditional decoding approach utilizing tree-shaped semantic representation, enhancing the slot content in gen- erated utterances. Furthermore, Li et al. (2020) offered a method to lower the slot error rate terances using an iterative RL framework for slot consistency. Ohashi and Higashinaka (2022a) pre- sented a fine-tuning method for utterance genera- tion that uses user's 's NLU model so that the NLU can understand the DA accurately.
In all of the above studies, the optimization was performed at the utterance level using a fixed cor-

---

Dialogue system
User simulator
Add to history 2
Post-processed
System utterance
GenPPN
(re-written) utterance
There are 21. They middle.
There are 21 hotels middle town.
NLU
Compute DA contribution
Reward r(x()
NLG
w(a, V 0.5
DAs a
(1.))
DAsa
Aggregate function
(Inform, Choice, 21) [Inform, Area, middle) X
(inform,Price)
0.2
[Inform, Choice, [Inform, Area, center)
X.0.9
(Inform,Choice)
0.5
(Inform,Area]
0.9
Figure 2: Schematic diagram GenPPN and how it is optimized. GenPPN M rewrites utterance u generated by NLG of dialogue system into u'. This rewriting of M is optimized using reward r(u') calculated the basis of DA contribution ((1,)): so system's ability to complete task improves.
pus of utterances. In this study, aim to optimize post-processing not only on the utterance level but also on the dialogue level improve the task com- pletion performance for multi-turn dialogues.
i.e., 0 means that the task failed, and 1 means that the task succeeded. The objective is obtain the optimal parameters 8* for the GenPPN Mg such that the expected value of e is maximized:
3 Method
0" = arg max
E U' U'mMa(X)
le(U")
(2)
3.1 Overview
Figure 2 shows schematic diagram of GenPPN and how it is optimized. At the beginning of each interaction, the user is given a user goal consisting list of constraints be informed the system and information to be obtained from the system through the dialogue. During the dialogue, from the user's utterance at a certain turn t, the NLU and DST modules of the system first estimate the current belief state of the user. Then, the basis of the belief state, the Policy module determines the next actions be taken by the system as DAs 0₁- The takes a structure containing one or more triples of intent I, slot s, and value v:
where U = {u/, ug ... 4/r are the system ut- terances sampled by M from prompts X [.1, 02, 2T) each turn.
Since the component to be trained is not the NLG itself but the GenPPN, our optimization is independent of the architecture and differentiability of NLG modules. In the following subsections, we describe the GenPPN, reward design for optimizing Eq. (2), and RL algorithm used.
3.2 Generative Post- Post-processing with PLM
We assume that the model M of GenPPN is lan- guage model based on the Transformer architec- ture (Vaswani et al., 2017). At turn t, the input prompt & is created from the previous dialogue context h2, DA Ot, and system utterances u, by using hand-crafted prompt (specific prompts are shown in Section A.2 in the appendix).
a (Li 8g. u4) I....,//al |a]}
(1)
The NLG module maps to the system's 's utterance u. Here, GenPPN M post-processes, i.e., rewrites uy. A prompt Ie Prompt(he, at, u)) created from the previous dialogue context h4, as and u, and M generates new utterance ut: u4 M(xx) (see Section 3.2 for details on GenPPN and the prompt format). The user then receives u/ esti- mates the system DAs âz, and the user's utterance is output for the next turn i + on the basis of the â and current status of the user goal. This exchange is repeated until the user goal is achieved or until the maximum number of turns is reached.
Through RL, M learns to generate a rewritten system utterance u' such that the dialogue result e is maximized. However, even the early stages of learning, if the system cannot produce at least reasonable system utterances, the dialogue always breaks down, and the training does not progress properly. For this reason, adopt instruction- tuned PLM (Wei al., 2022; Chung al., 2022: Taori al., 2023), which has shown high perfor- mance on many NL.P benchmarks including task- oriented dialogue modeling in few- and zero-shot settings (Pan et al., 2023; Hudeček and Dusek,
Let T be the number of turns the end dia- logue. addition, the dialogue result e E {0, 1} of whether the user goal was achieved is determined,

---

2023). Since the general PLM has a massive num- ber of parameters, we use Low-Rank Adaptation (LoRA) (Hu et al., 2022), which has good parame- ter efficiency. That is, a small number of parame- ters is added to each self-attention module in the PLM, and only 0 is optimized during training.
Ree,S the number of times that (I, has where "(,s) been correctly recognized by the user (subseript "Rec" for "recognized") in successful dialogue S:
Rec.S Σ E [1,,)) - (I',
(I,a)
(a.a)eS ("''s,"")Eaná
3.3 Reward with DA Contribution
Similarly, Unr.S indicates the number of times in ((I,s)
Eq. (2) implies that only e or a dialogue-leve reward at the end of multi-turn dialogue should be used to evaluate the generated utterances U for all T turns. However, since this reward is too sparse and it is feasible to optimize M, we approximate Eq. (2) by using the utterance-level reward r, which evaluates how much an effect each utterance has on the final task completion:
S that (I, was not recognized by the user (sub- script "Unr" for "unrecognized"", Rec.F indicates ((I,s) the number of times in F that (7,s) was recog- nized by the user, and (1.s) of times in F that (I, s) was not recognized by the user.
Unr.F indicates the number
Since C(rss) is the co-occurrence probability of (1, s) being recognized (or unrecognized) and the task being a success (or failure), this quantifies how accurately conveying (I, leads to task suc- cess. Also, ((T,s) is value that is updated the number of dialogue samples increases during train- ing. That is, G(Is)) can be computed adaptively the current performance of the GenPPN at each learning step. Note that v is not taken into account for each count because the possible values of v are so large (c.g., instances of phone number, address, etc.) that counting them separately would yield unreliable statistics.
0" arg max ('~Ma(()
Σ r(u)) LEEE
(3)
The role of NLG is to accurately convey DAs to the user, i.e., to generate utterances such that 0r âz. but this kind of evaluation based on the consistency of a and â does not take into account e, so it cannot be used for directly.
With this in mind, we design by combining e with the consistency evaluation for at and ât- Specifically, we statistically measure the contribu- tion of correctly conveying s,, v) E a, to the user in terms of the impact on on the basis of the dialogues we have sampled. Then, we give centribution-weighhee reward to the utterance u/ that succeeds in conveying (I, v).
Finally, the reward r(u() for the utterance gener- ated by GenPPN turn is calculated using this DA contribution C(Tss) as follows:
r(v() = Aggregate(w(aa. 0z))
(5)
w(a, âe) (T )(,s) 81) E n E E n
To realize this, first, the end of each dialogue, triples {(ar, âe, e){c E T} consisting of at and åz at each turn together with e are added to the DA history D. Here, e is determined retrospectively on the basis of the outcome at the end of the dia- logue; of all turns in a successful dialogue is 1, {aa,, ai, 1)]t E T}, and of all turns in failed dialogue is 0. {(ae, âi, 0)هt E T). Here, all (a, à) pairs in D sampled so far are split into S and F depending on whether the task was successful:
(6)
Uf-T ((.) (I,
Here, the DAs correctly recognized by the user (a,))) are given a default score T, and conversely, the DAs not recognized by the user (a n ā are given negative score -7, and w(r, âz) is the set of these scores weighted by each DA contribution C(I.s). The constant is hyperparameter, and "Aggregate" is a function for aggregating w(as, â;) into scalar value for the reward. We design the following two types of Aggregate functions and empirically determine which one is better:
S = ([a à) (a, a D F ((a à) (a, e) De 0}
mean Output the mean of uraa, āz)
Then, for each (I, 8, v) E an the contribution c(1,))
of (I, s) is caleulated follows:
absmax Output the value with the highest absolute
Rec.S + (((ss)
Unr.F
value among w(a, ât). This is emphasize the weighted score of [1 s, v) with the highest contribution.
((.s))
C(Iss)
Rec.S + (1,s)) (1,a))
Rec,F Unr.S Fnr (4)
+ ((.ss)
+ (..)))

---

At the beginning of learning, the number of DAs recorded in D is small, so is expected that an appropriate ((1,s) cannot be calculated. Therefore, prior to learning, (a, a) is sampled and recorded in D by conducting multiple dialogues between the dialogue system and the user simulator without using GenPPN.
that provide various modules for dialogue systems. user simulator, and an evaluation tool. The follow- ing describes the dialogue system, user simulator, and user goals used in this experiment.
Dialogue System To make it easier to assess changes in the performance of utterance genera- tion, the other modules (that is, NLU, DST, and Policy) should have stable performance. Therefore, used the best-performing BERT (Devlin et al., 2019)-based NLU (Chen et al., 2019), rule-based DST, and rule-based Policy available in Conv-a BERT- based NLU is model that uses representa- tions embedded by BERT classify user intentions in user utterances and extract slots by sequence la- beling. Rule-based DST and Policy are modules implemented using hand-crafted rules. Three mod- els were selected as NLG modules (see Section 4.2 for the NLG models used), and GenPPN was applied each of them to verify its generality.
3.4 Optimization via RL
Following Stiennon et al. (2020), the RL algo- rithm used in this method is Proximal Policy Op- timization (PPO) (Schulman et al., 2017). As a value network, a linear layer that outputs a scalar value randomly initialized with the parameter ø is added, and the overall trainable parameters are set to vi [0; ø). Also, following (Ziegler et al., 2019), prevent the probability distribution of My from deviating too far from that of the original M due to parameter updates and thus losing its naturalness, Kullback-Leible (KL) divergence penalty is added to r(4)) as the final reward R2 for utterance u/ at turn t:
User Simulator For the user simulator, we used combination of BERT-based NLU, agenda-based Policy (Schatzmann al., 2007), and template- based NLG. The agenda- based Policy models a user's behavior in MultiWOZ by using a stack-like agenda created using hand-crafted rules.
Rs re=r(uh) - Blogg Mo(u((as) M(ulI x4)
(7)
The clipped surrogate objective ((() (Schulman et al., 2017) is used to optimize v with the advan- tage estimated from the reward and value network. Since we need the user's subjective understand- ing results à, this study uses a user simulator to optimize the GenPPN. The learning algorithm is summarized in Algorithm 1 in the appendix.
User Goal user goal for each dialogue is ran- domly generated; the domains are randomly se- lected from one to three domains (out of all seven domains). The slots are also randomly selected on the basis of the slots' frequency in MultiWOZ.
4.2 NLG Baselines
4 Experiments
We applied GenPPN each of the three NLGs available in ConvLab-2 with different architectures in order demonstrate that it works for a variety of NLGs. In addition, one NLG optimized with only utterance-level rewards, without considering task success, was also evaluated for comparison.
In our experiments, we first evaluated the effec- tiveness of GenPPNs using a user simulator. Then, using the optimized GenPPN, we conducted dia- logue evaluation experiment using human subjects.
4.1 Dataset and Platform
We evaluated the effectiveness of our GenPPN using a dialogue system and a user simulator implemented on the basis of the MultiWOZ dataset (Budzianowski et al., 2018). MultiWOZ is a task-oriented dialogue dataset between clerk and customer at travel information center, col- lected in Wizard-of OZ. style. It contains variety of tasks across a total of seven domains (attraction, hotel, hospital, restaurant, taxi, train, and police). We used ConvLab-2 (Zhu et al., 2020), a plat- form for evaluating task oriented dialogue systems
Template NLG An NLG model that uses the template utterances representing each DA. Because each utterance is carefully de- signed by hand, this model has significantly higher performance than other NLG base- lines (Takanobu et al., 2020).
SC-LSTM (Wen et al., 2015) An LSTM-based model with a reading gate mechanism. This model takes binary feature vectors represent- ing DAs as context and decodes utterances.

---

Inform
NLG
Task Success
Book Rate
Tum
DA Fl
F1
Precision
Recall
Template NL.G
77.25
78.44
74.19
89.13
83.91
7.67
71.73
GenPPN
77.93
79.75
75.62
89,41
84.33
7.63
76.98
GenPPN absna
78.91
79.86
75.58
89.93
85.19
7.02
78.23
SC-LSTM
54.00
67.45
68.07 07
72.48
67.69
11.65
60.56
+ GenPPN 60.64**
75.40
75.42
81.01
78.74
9.42
79.80
GenPPNassax
72.95
79.46
77.38
86.16
78.46
7.21
79.08
SC-GPT 64.94
78.06
73.60
56.94
7.80
71.53
GenPPN
73.63"
76.54
71.81
87.79
82.08
8.03
73.07
GenPPNabomat
73.34*
77.34
72.29
89.03
80.79
7.50
73.72
GPT-2 RI. 72.36
76.70
73.50
85,99
76.81
7.47
81.17
GenPPNMemn
74.02
77.10
73.87
86
79.
7.54
80,98
GenPPN 75.20
78.79
75.58
88.02
79.80
7.15
80.08
Table Performance of dialogue system with each NLG and with GenPPN applied to them. Subscripts GenPPN indicate Aggregate function used reward calculation. Highest score for each NLG is in bold. and indicate Task Success for GenPPN and were significantly better than original NLGs at p < 0.05 and < 0.01, respectielly.. in McNemar test with Bonferroni correction.
SC-GPT (Peng et al., 2020) A GPT-2 based model that generates utterances from DA text sequences. This has been trained on seven task-oriented dialogue corpora including MultiWOZ and the Schema-Guided Dialogue Dataset (Rastogi et al., 2020), and it is SOTA on the MultiWOZ NLG benchmark.
Task Success is evaluated by calculating whether Inform Recall and Match Rate both became one within the maximum number of turns. To evaluate the utterance-Ievel performance of NLG, we addi- tionally used DA FI, which measures the match rate of a and à at each turn by Fl (Ohashi and Higashinaka, 2022a; Guo et al., 2023).
GPT-2 + RL (Ohashi and Higashinaka, 2022a) A GPT-2-based NLG model optimized with an utterance-level reward. This model was trained to maximize the accuracy of a and (as measured by FI) using the DA-system utterance pairs in the MultWOOZ corpus in an offline fashion.
4.4 Training GenPPN
Stanford Alpaca 7B (Taori et al., 2023) was used the instruction-tun PLM for the GenPPN. This model is fine-tuned version LLaMA-7B (Tou- vron et al., 2023) with a 52K instruction dataset. The input prompts include the dialogue history, the previous DA, the system utterances generated by NLG from that DA, and the instructions for rewrit- ing that system utterance (specific prompts are shown in Section 4.2 in the appendix). Throughout the experiment,
Note that, since the purpose of our experiment is verify whether GenPPN can enhance the per- formance of NLG irrespective of its architecture or base performance, the verification of methods that fine-tune NLG models themselves outside the scope of this study.
first sampled 1K dialogues initialize the DA history and then trained 200 itera- tions. See Scction A.3 for further training details. For the evaluation, the GenPPN at the highest- reward iteration was used, and dialogue simulations were performed using 1,024 user goals prepared specifically for the test. We report the average scores in this paper.
4.3 Evaluation Metrics
the evaluation of the dialogue system, we used common metrics for task-oriented dialogues: Turn, Inform C1/Precision/Recall, Book Rate, Task Suc- cess. Turn indicates the number of turns required for each dialogue; the smaller is, the more ef- ficiently the dialogue can be conducted. Inform F1/Precision/Recal indicate whether the system re- sponded appropriately the user's requests. Book Rate indicates whether the entity booked by the system correctly matched that of the user's goal.
4.5 Main Results
Table 1 shows the evaluation results?. For Tem- plate and SC-LSTM, GenPPN improved all evalua-
We used the latest version of ConyLab-2 as of June 2023: we could not reproduce the scores reported on the leaderbourd for some baseline systems. The main reason is probably due to the difference in the random seed and BERT NLU weights.

---

NLG
Success
Inf.
Book
DA
NLG
Success
Turns
Und.
App.
Sat.
SC-LSTM GenPPN
72.95
79.46
78.46
79.08
SC-LSTM 54
33.33
17.72
4.09
4.17
4.04
GenPPN 50
52.00'
15.08
4.04
4.14
3.98
w/o adaptive ((),))
62.11
74.80
78.15
70.12
w/o
59.47
77.02
71.06
76.00
w/o r(u')
63.67
77.58
80.83
62.69
Table 3: Human evaluation results. number sub-
w/o context
68.46
76.25
74.17
80.28
jects who interacted with each system. "Und.," "App." and "Sat." represent user's subjective evaluation of system's understanding. appropriateness of system ut- terance, and satisfaction with dialogue, respectively. Scores of SC-LSTM and our GenPPN were compared using the McNemar test; significant differences are indi- cated with (p < 0.05) and ** (p < 0.01).
w/o response
47.36
68.96
74.67
65.21
Table 2: Ablation results when applying GenPPN to
SC-LSTM. "absmax" Aggregate function is used in cal- culation of rewards using DA contributions.
tion metrics. The absmax aggregation function im- proved the final performance more than the mean. For SC-LSTM, Task Success was nearly 19 points better than the original SC-LSTM, especially when using absmax. These results indicate that a better strategy for evaluating each utterance is to prior- itize learning convey the DA with the highest contribution rather than to take the average. Con- sidering that the improvement in Template NL.G was only about 1.7%, the lower the performance of the original NLG, the more room there is for improvement by GenPPN.
4.6 Ablation Study
We analyzed the impact of each factor in the GenPPN optimization on the final performance. For the NLG model in this ablation study, we used SC-LSTM, which showed the greatest improve- ment in performance over the other NLGs in Table 1. Table 2 shows the results.
Adaptive DA Contribution "w/o adaptive c(sss) is a reward design that does not update the DA history, and in all steps, uses the DA contribution calculated only from the 1,000 dialogues sampled prior to training. This resulted in 10 points de- crease in Task Success. Therefore, found that it is important constantly update the DA contribu- tions as the GenPPN changes.
For SC-GPT, Task Success, Book Rate, and Turn steadily improved. However, Inform Preci- sion slightly decreased, unlike other cases where GenPPN was applied to other NLG modules. This suggests that GenPPN learned that improving Book Rate is more important for improving Task Success, even at the expense of Inform Precision.
Dialogue-level Reward "w/o e" does not use DA contribution, and only an utterance-level reward based on the accuracy (F1) between and á is used. That is, the reward design does not take into account the dialogue-leve reward e related task completion. As result, Task Success decreased significantly, while DA F1 did not decrease much. Therefore, it was shown that the DA contribution is a useful factor in improving the task success rate.
GPT-2 RL has higher DA F1 than any of the GenPPNs. This is reasonable, given that GPT-2 + RL is optimized using DA F1 as a reward. GenPPN was able to further improve GPT-2 + RL's Task Suc- cess while the utterance-level metric DA Fl slightly decreased from the original score. This suggests that not only utterance-level rewards but dialogue- level rewards must be introduced to improve the task completion performance of the entire system.
Utterance-level Reward "wlo r(u')" does not use the utterance-level reward r(u') but only the last dialogue evaluation result as reward. That is, all utterances were given the dialogue result E {0, 1} equally. Although Book Rate improved. Inform Fl did not, indicating that it is difficult to optimize GenPPN with only a sparse e reward, as described in Section 3.3.
Note that the performance GenPPN and SC-GPT GenPPN is lower than that of Temmlltee NLG baseline. However, the primary goal of our study is not to obtain an NLG model with SOTA performance but rather to enhance the performance of NLG irrespective of its architec- ture its base performance. In our experiments, the dialogue performance of NLG models like SC- LSTM and SC-GPT, in addition to Template NL.G, were improved using GenPPN, showing that our primary goal has been achieved.
SC-LSTM +
Dialogue Context the input of GeaPPN does not include the dialogue history. The slight decrease in Task Success, In- form F1, and Book Rate indicates that GenPPN should make use of the previous history to improve
"w/o context" indicates that

---

Context
User: Hello, can you help me find restaurant for my upcorning trip to Cambridge? Hmm, I 'lll try Asian oriental food. I'm looking for an expensive restaurant. also would like information on place to cat in the centre. System: kymmoy serves usian oriental food.
User: May I have the address? would like their phone number.
System DA
{[Inform, Attraction-Addr pool way, whitchill road), (Inform, Attraction-Phore 01223902088). (Inform, Restuurant-Addr 52 Mill Road City Centre), (Inform, Restuurant- Phone, 01223311911))
System Response
SC-LSTM:
GenPPN:
Their address is
Their address is 52 Mill Road City Centre phone number 01223311911.
DA predicted by user
0
[(Inform, Restaurant-Ade Addr, 52 Mill Road City Centre),
(Inform, Restaurant-Phone, 0122331191D);
Table 4: Response examples of systems using SC-LSTM and SC-LSTM + GenPPN in same context.
dialogue performance. Meanwhile, the improve- ment in DA FI suggests that dialogue history is not necessary if we only want optimize at the utterance level.
measures of task completion. However, the user's subjective ratings of understanding, appropriate- ness, and dialogue satisfaction showed no improve- ment, which is reasonable given that GenPPN is op- timized for task completion only. This also means that GenPPN did not generate unnatural utterances to improve Task Success and Turn, which is consid- ered positive result, We would like to evaluate the effectiveness of GenPPN for other NLG baselines than SC-LSTM by humans in the future.
System Response "w/o response" does not rewrite the system utterance, but GenPPN gener- ates it directly from the dialogue history and DA. can be seen that the performance of all the methods significantly degraded. The reason for this is that the untrained GenPPN, i.e., Alpaca-7B, has never seen the mapping from DA to system utterance in MultiWOZ and thus has difficulty generating sys- tem utterances in zero-shot setting, Therefore, it is considered more reasonable to learn rewrite ut- terances rather than to generate them from scratch.
5 Case Study
Table compares the behavior of two systems us- ing SC-LSTM and using SC-LSTM + GenPPN a dialogue. As a context, the user requested the address and phone number of the restaurant, but the Policy module of the dialogue system mistak- enly decided to answer with the address and phone number of an attraction well. Here, SC-LSTM was not able to generate sentences because there was no example in its training data, in which DAs in different domains occur at the same time. As a result, SC-LSTM failed to convey any informa- tion to the user. In contrast, GenPPN reflected the address and phone number of the restaurant in the post-processed utterance, and as result, the infor- mation was correctly conveyed as requested by the user, and the task was successful.
4.7 Human Evaluation
We tested whether GenPPN optimized by dialogue simulation is also effective for humans. In this ex- periment, we also used the system using SC-LSTM, which showed the best performance improvement over the other NLGs in Table 1. Over 50 crowd workers were recruited via Amazon Mechanical Turk, and each worker interacted once with one of the systems using SC-LSTM or SC-LSTM + GenPPN for up to 20 turns after being instructed about their user goal. Each worker had 20 turns judge whether or not the task was successful; after 20 turns, the task was forced to fail. Af- ter the dialogue was completed, each worker was asked to rate the system's 's ability to understand the language (Und.), the accuracy of the system's re- sponses (App.), and overall satisfaction with the dialogue (Sat.) on a 5-point Likert scale. See Sec- tion A.5 in the appendix for more details the human evaluation settings.
It is worth noting that the utterance generated by post- post-processing does not contain any informa- tion about the attraction. This is probably because GenPPN judged from the dialogue context that the information about the attraction was not what the requested, and that conveying this piece of in- formation would interfere with the task completion. These results indicate that GenPPN can correct er- rors propagated from preceding modules beyond the realm of NLG so that the entire system's 's per- formance improves.
Table shows the results of the evaluations of each system. GenPPN significantly improved in terms of both Task Success and Turn, which are

---

6 Summary and Future Work
GenPPN is required rewrite utterances, which expected increase the operation cost and gen- eration time. addition, more computational re- sources are required during learning. Therefore, it is necessary to study ways to reduce learning and inference costs.
Processecssng Networks (PPNs), which modify the outputs of arbitrary modules including non- diffcrentiable ones in task-oriented dialogue sys- tems, have been applied to NLU, DST, and Pol- icy modules. In this paper, we proposed Genera- tive Post-processing Networks (GenPPNs) for NLG modules. We optimized GenPPN toward the task completion performance of the entire system by us- ing DA contribution- based rewards. Experiments on the MultiWOZ dataset and dialogue simulations confirmed that our GenPPN could improve the task success rate of various dialogue systems, regardless of the NLG architecture. It was also confirmed that GenPPN optimized with dialogue simulation was effective in human evaluation experiment.
Ethics Statement
We used the publicly available MultiWOZ dataset in full compliance with its terms of use. Our use of the LLaMA language model was also entirely consistent with the prescribed usage guidelines. No private or confidential data was used at any stage of the rescarch. For our human evaluation experi- ment, we underwent and passed an ethical review from our institution and strictly adhered its rules and guidelines. The anonymity, privacy, and rights of the subjects were preserved throughout. We acknowledge that there are potential ethical con- cerns associated with the use of large pre-trained language models, such the risk of generating harmful discriminatory statements.
In the future, we would like to integrate our GenPPN with the existing PPNs of NLU, DST, and Policy to realize the post-processing of all mod- ules and further improve the systems performance. Since GenPPN is a generative model, we would like to extend it not only to text but also to the post-processing of speech recognition and speech synthesis modules in spoken dialogue systems.
A Acknowledgments
This work was supported by JSPS KAKENHI Grant Number 19H05692. We used the compu- tational resources of the supercomputer "Flow" at the Information Technology Center, Nagoya Uni- versity.
Limitations
Our work has several limitations, which we aim to address in our future work.
First, this study implemented, trained, and evalu- ated the GenPPN using only the MultiWOZ dataset. Therefore, the applicability of the GenPPN task- oriented dialogues other than MultiWOZ. has not been verificd. In particular, the DA-dependent re- ward design in this study needs to be improved for application to dialogues such as open-domain chats, where ontologies such as intent, slot, and value are not defined.
References
Gabor Angeli, Percy Liang, and Dan Klein. 2010. A simple domain demain-indeeenddent probabilistic approach to generation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Process- ing, pages 502-511 502-
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. Constrained decoding for neural NLG from compositional repre- sentations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 831 844.
Second, this study used a user simulator for train- ing the GenPPN. When applied to a new domain, the cost of implementing a user simulator is likely to be high. Therefore, a user simulator-free learn- ing method is nceded in this respect. In addition, a GenPPN optimized by a user simulator learns user- simulator-specific utterances, which may not be the best utterances for humans. To learn the optimal GenPPN for humans, it is essential to design re- ward model that takes humans into account, using human-in-tiee -imann-n-ttheloopp approach.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašié. 2018. MultiWOZ A Large-Scale Multi-Domain Wizard-of-Or Dataset for Maik-Orienttd Dialogue Modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5025
Third, this study required running a PLM with a relatively large parameter size. During inference, in addition to the system's response generation, the
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. BERT for Joint Intent Classification and Slot Filling, arXiv preprint arXiv: 100d. 10909.

---

Zhi Chen, Lu Chen, Xiang Zhou, and Kai Yu. 2023. Deep reinforcement learning for on-line dialogue state tracking. In Proceedings of Man-Machine Speech Communicattion pages 278-292.
François Mairesse and Steve Young. 2014. Stochastic Language Generation in Dialogue using Factored Language Models. Computational Linguistics, pages 763-799.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling catruction-finttune language models. arXiv preprint arXiv:2210. 1416..
Shikib Mehri, Tejas Srinivasan, and Maxine Eskenazi. 2019. Structured Fusion Networks for Dialog. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 165-177.
Volodymyr Mnih, Koray Kavukcuoglu, David Sil- ver, Alex Graves, Joannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. 2013. Playing Atari with Deep Reinforcement Learning. arXiv preprint arXiv:1312.5602.
Jacob Devlin, Ming-Wi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Alice H Oh and Alexander Rudnicky. 2002. Stochas- tic natural language generation for spoken dialog systems. Computer Speech & Language, pages 387- 407.
Ao Guo, Atsumoto Ohashi, Yuya Chiba, Yuiko Tsunomori, Ryu Hirai, and Ryuichiro Higashinaka. 2023. Personality- aware natural language genera- tion for task-oriented dialogue using reinforcement learning. In Proceedings of the 32nd IEEE Interna- tional Conference Robot and Human Interactive Communication.
Atsumoto Ohashi and Ryuichiro Higashinaka. 2022a. Adaptive natural language generation for task- oriented dialogue via reinforcement learning". Proceddings of the 29th International Conference on Computational Linguistics, pages 242-252.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations.
Atsumoto Ohashi and Ryuichiro Higashinaka. 2022b. Post-processing networks: Method for optimizing pipeline task-oriented dialogue systems using rein- forcement learning. In Proceedings of the 23rd An- nual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 1-13.
Vojtëch Hudeček and Ondrej Dusek. 2023. Are large language models all you need for task-oriented dia- logue? In Proceedings of the 24th Meeting of the Special Interest Group Discourse and Dialogue, pages 216-228 Prague, Czechia Association for Computational Linguistics.
Wenbo Pan, Qiguang Chen, Xiao Xu, Wanxiang Che. and Libo Qin. 2023. A preliminary evaluation of chatgpt for zero-shot dialogue understanding. arXiv preprint arXiv:2304.04233 044.555
Diederik P Kingma and Jimmy Ba. 2014, Adam: Method for Stochastic Optimization. arXiv preprint 14X2::4422.6900
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng. and Jianfeng Gao. 2020. Few-shot Natural Language Generation for Task-Oriente Dialog. In Findings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 172-182.
Hwaran Lee, Seokhwan Jo, Hyungjun Kim, Sangkeun Jung, and Tae- Yoon Kim. 2021. SUMBT+LaRL: Effective Multi-Domain End-to-End Neural Task- Oriented Dialog System. IEEE Access, pages 116133-116146.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-End Task- Completion Neural Dialogue Systems. In Proceed- ings of the Eighth International Joint Conference on Natural Language Processing, pages 733-743.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghay Gupta, and Pranav Khaitan. 2020. Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 8689-8696.
Yangming Li, Kaisheng Yao, Libo Qin, Wanxiang Che, Xiaolong Li, and Ting Liu. 2020. Slot-consistent nig for task- oriented dialogue systems with iterative rectification network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 97-
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda- Based User Simulation for Bootstrapping POMDP Dialogue System. In Proceedings of Human Language Tech- nologies 2007: The Conference of the North Amer- ican Chapter of the Association for Computational Linguistics, pages 149-152.
Zichuan Lin, Jing Huang, Bowen Zhou, Xiaodong He, and Tengyu Ma. 2021. Joint System-Wise Opti- mization for Pipeline Gral-Oriented Dialog System. arXiv preprini urXiv:2106.04833

---

John Schulman, Filip Wolski, Prafulla Dhariwal. Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv preprint
you Need. In Proceedings of Advances in Neural Information Processing Systems, pages 5998-69969 6608..
arXiv:1707.0633
Marilyn A. Walker, Owen C. Rambow, and Monica Rogati. 2002. Training sentence planner for spo- ken dialogue using boosting. Computer Speech & Language, 16(3):409-433.
Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable Sentence Planning for Complex In- formation Presentations in Spoken Dialog Systems. In Proceedings of the 42nd Anmual Meeting of the Association for Computational Linguistics, pages 79- 86.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language mod- cls are zero-shot learners. In Proceedings of Interna- tional Conference on Learning Representations.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. In Proceed- ings of Advances in Neural Information Processing Systems, pages 0008-3021.
Tsung- Hsinnn Wen, Milica Gašié, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain Neural Network Language Generation for Spoken Dialogue Systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 120-129.
Shang- Yu Su. Kai- Lo. Yi-Ting Yeh, and Yun-Nung Chen. 2018. Natural Language Generation by Hier- archical Decoding with Linguistic Patterns. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 61-66.
Tsung- Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei- Hao Su, David Vandyke, and Steve Young. 2015. Semantically Conditioned LSTM-based Natural Lan- guage Generation for Spoken Dialogue Systems. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721.
Ryuichi Takanobu, Qi Zhu, Jinchao Li, Baolin Peng, Jianfeng Gao, and Minlie Huang. 2020. Is Your Goal- Oriented Dialog Model Performing Really Well? Em- pirical Analysis of System-wise Evaluation. In Pro- ceedings of the 21st Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 297-310.
Steve Young, Milica Gašié, Blaise Thomson, and Ja- son D. Williams. 2013. Pomdp-bused statistical spo- ken dialog systems: A review. Proceedings of the IEFF, pages 1160-1179.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction LLaMA model. https: //github.ccm//tttsuubee Lab/ hacabb/stann /at.ra/aoraaaaaraaaaa.aaa.
Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020. Recent advances and challenges in task-oriented dialog systems. Sci- ence China Technological Sciences, pages 1-17.
Tiancheng Zhao and Maxine Eskenazi. 2016. Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 1-10.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne I Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and ef- ficient foundation language models. arXiv preprint 202i::2002.13971..
Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng. Jianfeng Gao. Xiaoyan Zhu, and Minlie Huang. 2020. CanyLa An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 142 142-149.
Van-Khanh Tran and Le-Minh Nguyen. 2017. Natural Language Generation for Spoken Dialogue System using RNN Sicoder-Decoder Networks. In Proceed- ings of the s1 Conference on Computational Natu- ral Language Learning, pages 442-451.
Bo-Hsiang Tseng. Yinpei Dai, Florian Kreyssig, and Bill Byrne. 2021. Transferable Dialogue Systems and User Simulators. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, pages 152- 166.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint arXiv: 1909.08533
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, L ukasz Kaiser, and Illia Polosukhin, 2017. Attention is All

---

2: Prepare Ma with randomly initialized LoRA parameters 0 and value network parameters ø A Appendix
Hyperparameter Name
Value
A.1 GenPPN Optimization Algorithm
Target projection matrix of self-attention module Rank
query, key value, oulput
LoRA
16
Algorithm Optimization of GenPPN PPO
Scaling factor
16
Require: Dialogue system A, User simulator U
Total iterations
200
Total hatch size Epoch
512
Require: GenPPN M
4
1: Initialize DA history D by sampling several
Total mini-batch size Learning Optimizer
32
le-5
dialogues using A and
PPO
Adam
Discount factor 7
1.0
GAE factor X
0.95
Clipping
0.2
3: for each training iteration do
Coef. of KL penalty
0.01
4: while #turns does not reach batch size do Sample dialogue and (ue: âz) for each turn using A, M your and U
5:
Table 5: Hyperparameter settings
6:
Obtain final evaluation result
7:
Add (at, dc,e) e) of each t to D> end while
Table shows the hyperparameters of LoRA and PPO during training. Throughout the experiment, we first sampled 1,000 dialogues to initialize the DA history and then trained 200 iterations, with a batch size of 512 turns per iteration (corresponding to about 50 dialogues) and fixed learning rate of le-5 with Adam Optimizer (Kingma and Ba, 2014). We used 16 1100 32-GB computational resources. Training of 200 iterations took about seven hours to complete.
8:
Calculate reward Rg using Eq. (7)
10:
Compute advantage estimates Optimize ((v) with a certain epoch and mini-batcd size
11:
Update tolld <- V
13: end for
A.2
Prompt for GenPPN
Figure 4 shows the training curves when GenPPN is applied to Template NLG, SC-LSTM. SC-GPT, and GPT-2 RL respectively. The figure shows that both the reward and Task Success are low at the start of training (i.c., before RL is ap- plied) and improve as RL progresses, which means that fine-tuning LLMs with RL is important for our GenPPN. We have yet to investigate the effects of instruction tuning. We plan to validate this by comparing the performance of instruction-tuned models other than Alpaca (e.g., Flan-T5 (Chung et al., 2022)) and non-instructiion-tneed models (e.g., LLaMA (Touvron et al., 2023)).
At given turn i, the input prompts to GenPPN include the DA a output by the system's 's Policy module and the original system utterance 4y gener- ated by the NLG from a, As the dialogue context, we also included user utterances at L- 1. DAs 41-1, system utterances u/ generated by GenPPN, and user utterances at f. The format of the instructional text in the prompts the Alpaca dataset (Taori et al., 2023). Examples of prompts are shown in Figure 3. In our experiments, due to the maximum input length of the model, the dialogue history was limited to the past one turn.
adapted from the phrases in
A.3 Details of GenPPN Training
As the model for GenPPN, we used Stanford Alpaca-7B trained weights for the prompts, which are publicly available on HuggingFace Hub The maximum number of input tokens for prompt was set to 512, and the maximum number of generated tokens was set to 128. When generating utterances, sampling was performed with the beam size set to I, temperature to 1.0, and top-p to 1.0. The 7 used in the reward calculation (Eq. (6)) was set to 1.
A.4 Additional Ablations
In addition to the study on SC-LSTM in Section 4.6, we conducted the same ablation study on two higher-performing NLGs: Template NL. and GPT- 2 RL. Table 6 shows the results. In Template NLG, the improvement with GenPPN was the great- when no context was used. Nevertheless, for both Template NL.G and GPT-2 w/ RL, there was a trend showing that adaptive DA contribution was important for performance improvement.
wttp:/aaa....ccaaeaaaaaaa co/tatsu-lab/ al alpaca-7b-wdif"

---

Below is instruction that describes task. Write response that appropriately completes the request.
### Instruction:
Begin by reading conversation between a customer and chatbot about travel information.
Customer: am looking for a hotel call kirkwood house. need a hotel as well
Chatbot Action: (Poorne moderate; Area north; guesthouss Parking yes; Stars 4; Name=kirkwood house; Interne==ccs)
Chatbot: kirkwood house is moderate 4 star guesthouse in the north.
Customer: Do they have free parking ?
Chatbot Action: (ofomm-Hooo(Nam==kiikve
house; Parking-yes)
Chatbot: kirkwood house is available.
Your task rephrase chatbot's utterance so customer can understand Make sure include the content of Chatbot Action:' in the utterance. If no rephrasing is necessary, repeat the original utterance.
### Response:
Chatbot:
Figure 3: Examples of prompts entered into GenPPN. "Customer:", "Chatbot Action:", and "Chatbot:" denote utterance, system DA, and system utterance at each turn, respectively.
NLG
Success
Inf. F1
Book DA Fl
NLG
Success
Inf. Fl
Book
DA Fl
Template NLG + GenPPN
78.91
79.86
85.19
78.23
GPT-2 + RL + GenPPN
75.20
78.79
79.80
80.08
adaptive e(r,a)
76.27
78.07
83,00
w/o adaptive c(1,))
67.68
76.75
79.31
81.04
77.15
79.29
83.30
76.89
72.56
76.57
77.36
80.63
w/o r(u')
71.09
78.14
80.89
63.68
w/o r(u')
72.85
78.52
81.82
71.15
w/o context
79.00
79.92
85.29
78.58
w/o context
73.24
77.18
77.14
(a) Template NLG
(b) GPT-2 RL
Table Ablation results when applying GenPPN to Template NLG and GPT-2 RL. "absmax" Aggregate function is used in calculation of rewards using DA contributions. Bold and underlined scores indicate top and second top, respectively.
A.5 Details of Human Evaluation
maximum number of turns. After 20 turns, the task was forced to fail. After the dialogue, the workers answered the three-question questionnaire described Section 4.7.
In collecting participants for the human evaluation via Amazon Mechanical Turk (AMT), we recruited workers who met the following four conditions: (1) had residence in English-speaking county, (2) had least 100 completed Human Intellienn Tasks (HIT) on AMT, (3) had an acceptance rate 95% or higher on the HIT on AMT, and (4) an- swered all five common sense questions correctly.
Considering that each HIT should take about 10 minutes to complete, the reward was set at $2 and the time limit was 30 minutes. The number of subjects not equal because the system randomfy selected each time, resulting in 54 evalua- tors for SC-LSTM and 50 evaluators for SC-LSTM + GenPPN.
In the experimental procedure, each worker was first instructed about a certain user goal. The user goal was randomly generated by a program as the dialogue simulation. Considering the high compre- hension ability of humans, the number of domains included in the user goal was set to 3, which is higher difficulty level than in the simulation. After reading the user goal, the workers interacted with either the system using SC-LSTM the system us- ing SC-LSTM + GenPPN. The maximum number turns in the dialogue was 20, and each worker judged whether the task was completed within the

---

Reward
Task Success
DA
1.0
1.0
1.0
0.8
0.8
0.5
0.6
0.6
mean
0.4
0.4
absmax
-0.5
50
100 Iterations
150
200
50
100 lterations
150
200
50
100
150
200
(a) Template NLG
Reward
Task Success
DA
1.0
1.0
1.0
0.5
0.0
0.4
0.4
absmax
-0.5
50
100
150
200
50
100 Iterations
150
200
50
100
150
200
Ibe natioons
Iterations
(b) SC-LSTM
Reward
Task Success
DA F1
0.8
0.8
0.6
0.6
0.0
mean absmax
0.4
0.4
-0.5
50
100
200
50
200
50
100
200
Rterations
lterations
lttrations
(c) SC-GPT
Reward
Task Success
DA
1.0
1.0
1.0
0.8
0.8
0.5
0.6
0.6
mean absmax
50
100
150
200
50
100 Iterations
150
200
50
100
150
200
Iterations
Iterations
(d) GPT-2 RL
Figure 4: Training curves when GenPPN is applied to each NLG. Either "mean" "absmax" is used as aggregation function. "Reward" indicates value calculated by Eq. (5).


Reviews:
[{"review_id": "30pxvBHTCk", "reviewer": "Reviewer_Bdnj", "summary": "", "questions": "I feel that these same results would be a lot more impactful and important if the authors could show improvements over Template that are not possible without GenPPN.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the thoughtful and encouraging review. We provide explanations and the new results of additional experiments to respond to your concerns and comments as follows.\n\n#### **Q1. Regarding the point that SC-LSTM and SC-GPT, even with GenPPN, cannot surpass the original template NLG. Is there a need to improve SC-LSTM and SC-GPT using GenPPN?**\n\nThe primary goal of our study is not to obtain an NLG model with state-of-the-art (SOTA) performance, but rather to enhance the performance of NLG irrespective of its architecture or its base performance. In our experiments, the dialogue performance of NLG models like SC-LSTM and SC-GPT, in addition to Template NLG, were improved using GenPPN, showing that our primary goal has been achieved.\n\nAlthough NLG baselines in our experiments included Template NLG, known for its high performance [1], in real-world situations, meticulously designed and high-performance NLG models like the template NLG are only sometimes available in other tasks or domains. We believe that GenPPN is a practical approach to improving NLGs in such scenarios.\n    \nWe also believe there is value in GenPPN when considering objectives other than task completion, such as enhancing naturalness and diversity. As additional experiments, we measured distinct-N of utterances generated by each NLG model during the test. The results are presented in the table below:\n\n| NLG | Task Success | Distinct-1 | Distinct-2 |\n| :--- | :---: | :---: | :---: |\n| Template | 77.25 | 1.91 | 5.37 |\n| Template w/ GenPPN | 78.91 | 1.99 | 5.97 |\n| SC-LSTM | 54.00 | 3.38 | 5.46 |\n| SC-LSTM w/ GenPPN | 72.95 | 3.10 | 7.76 |\n| SC-GPT | 64.94 | 4.39 | 15.89 |\n| SC-GPT w/ GenPPN | 73.63 | 4.62 | 16.08 |\n    \nThe distinct-N of Template NLG is low, which indicates fixed and mechanical utterances. In contrast, when applying GenPPN to SC-LSTM or SC-GPT, we were able to improve the Task Success score to a level comparable to Template NLG while maintaining the original high distinct-N of the model. This result suggests that we achieved dialogue performance comparable to Template NLG with more diverse utterances.\n\n#### **Q2. To what extent is the help of GenPPN, as confirmed for SC-LSTM, seen for Template NLG?**\nGiven that most metrics, including Task Success, have also shown improvements for the Template NLG, it can be inferred that GenPPN was similarly beneficial for the Template NLG as it was for SC-LSTM.\n\n#### **Q3. Can Template NLG be improved by additional work on the template language?**\nBy incorporating the insights from the analysis using SC-LSTM into the utterance generation rules of Template NLG, there is potential for improvements in dialogue performance.\n\n#### **References:**\n[1] Takanobu et al., 2020. Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation. In Proc. SIGDIAL2020, pages 297–310.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very thorough and explains each step of the process clearly. The results are clearly presented and the ablation study is informative. It provides an important final piece of postprocessing for the standard task-oriented dialogue pipeline. The most interesting result is the improvement over the very strong Template baseline.", "reasons_to_reject": "Looking at Table 2, GenPPN provides good relative improvements for SC-LSTM and SC-GPT. Yet, overall, the absolute performance of these models even with GenPPN does not exceed the performance of Template without GenPPN.  I don't think the authors explain why improving SC-LSTM and SC-GPT is worth doing when it's not better than Template, besides \"the lower the performance of the original NLG, the more room there is for improvement by GenPPN\", which is true by definition.\n\nWith performance worse than Template, I think that SC-LSTM and SC-GPT are weak baselines. The authors show an example in Table 4 where GenPPN helps the SC-LSTM model to produce text for a DA that would be impossible without GenPPN. But, I am curious if\n(1) there are similar cases where it improves over Template\n(2) the prevalence of such cases\n(3) whether it is possible that additional work on the template language can address these.\n\nWithout this context, it is very difficult to say whether it is worth all the CPU and GPU cycles that must have been used to produce these results.", "questions_to_authors": "I feel that these same results would be a lot more impactful and important if the authors could show improvements over Template that are not possible without GenPPN.", "paper_topic_and_main_contributions": "This paper introduced a method of post-processing that can be used in the NLG component of a task-oriented dialogue system. This is more challenging for the NLG component than for others (NLU, DST, Policy) as the output is a sequence of tokens rather than one or more slots. The authors use RL-based optimization and a transformer-based generative model to improve the generated output, resulting in improved performance on multiple datasets. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nkal26WL4r", "reviewer": "Reviewer_LAnm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the thoughtful and encouraging review. We provide discussions and explanations about your concerns as follows.\n\n#### **Q1. Is adding an additional component (i.e., PPN) to the complex TOD pipeline not a drawback?**\nAs you pointed out, the post-processing approach may increase complexity. However, we believe that the advantage of optimizing arbitrary modules for the dialogue performance of an entire system will be of more importance.\n\n#### **Q2. Would it not be more effective to fine-tune NLG models directly using reinforcement learning?**\nThe primary goal of our study is not to obtain an NLG model with state-of-the-art (SOTA) performance but rather to enhance the performance of NLG irrespective of its architecture or base performance. Therefore, we consider techniques that directly fine-tune the NLG model to be outside the scope of our study.\n\n#### **Q3. Considering the original PPN study [1], this study might seem incremental.**\nWe believe our study significantly advances the previous PPN study [1] in two main ways:\n1. We addressed the challenging but impactful and effective problem of NLG post-processing, which was impossible with conventional PPN.\n2. We optimized the performance of NLG at the dialogue level for the first time and demonstrated its utility.\n\n#### **References:**\n[1] Ohashi and Higashinaka, 2022. Post-processing networks: Method for optimizing pipeline task-oriented dialogue systems using reinforcement learning. In Proc. SIGDIAL2022, pages 1–13.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting approach to further improve NLG performance in TOD, by adding an RL-tuned network that post-processes the NLG model's output. \n\nThat the module is separate means it is somewhat agnostic to a variety of NLG models, increasing its applicability.\n\nThe utterance-level reward scheme is clever and could potentially be reused in other TOD work.\n\nResults on MultiWoZ are strong. \n\nThe paper is quite clearly written.", "reasons_to_reject": "It is somewhat hard to motivate this research, since the canonical TOD pipeline is already quite complex, and the motivation for adding another model therein is weak.\n\nApplying the proposed approach, or a variant thereof, directly to the NLG model may be more useful. The use of RL fine-tuning already permits this if the NLG model is a text-to-text LLM.\n\nViewed vis-a-vis the PPN paper (Ohashi and Higashinaka, 2022), this work may be considered relatively incremental by some.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper describes a module that could be added to a TOD pipeline, particularly to the NLG system or possibly other text-to-text systems in the pipeline, that rewrites its output. The module, GenPPN (generative post-processing network) is optimized using reinforcement learning, where the reward is estimated by the overall task success as well as utterance-level rewards based on which system dialogue acts are preserved in the rewrite and which ones are not. A key highlight of the paper was a heuristic to calculate the reward for recognizing/missing each dialogue act based on its importance for task completion. Results are shown for 3 models (template, SC-LSTM and SC-GPT) on the MultiWoZ benchmark, showing improved task success. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ouuWD1W7w8", "reviewer": "Reviewer_SfSx", "summary": "", "questions": "Question A: In the context of reward using DA contribution, whether the value of \"e\" can only be equal to 1 at the end of a dialogue? If this were the case, it raises the possibility of an imbalance between the Successful set and Failed set.\nQuestion B: Why was the main experiment, involving GPT-2 with RL, not conducted using GenPPNs?\nQuestion C: To what extent does the incorporation of task completion metrics in GenPPNs for RL optimization result in an excessive alignment of the final optimized model with the targeted metrics? More specifically, in the context of human evaluation experiments, why did the performance of GenPPNs decline across all three user subjectivity assessment metrics, and can this be attributed to the RL optimization objective?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the thoughtful and encouraging review. We provide explanations and the new results of additional experiments to respond to your concerns and comments as follows.\n\n#### **Q1. The task settings or baseline methods might not follow common practices. Why do template-based natural language generation (NLG) models outperform other baselines?**\nIn template-based NLG, high-quality system utterances that optimally represent each dialogue act (DA) are carefully designed manually. For this reason, it is known that template-based NLG performs significantly better than other NLG baselines [1], and the performance reported in this experiment is reasonable.\n\n#### **Q2. There is no study to determine whether the improvement by GenPPN is due to reinforcement learning or instruction tuning.**\nWe speculate that the improvement by GenPPN is attributed to reinforcement learning (RL). This is because, as Figure 4 reported in Section A.3 of the appendix, both the reward and Task Success are low at the start of training (i.e., before RL is applied) and improve as RL progresses.\n\nWe have yet to investigate the effects of instruction tuning. We plan to validate this by comparing the performance of instruction-tuned models other than Alpaca (e.g., Flan-T5 [2]) and non-instruction-tuned models (e.g., LLaMA [3]).\n\n#### **Q3. In the context of reward using DA contribution, can the value of \"e\" only be equal to 1 at the end of a dialogue? If this were the case, it raises the possibility of an imbalance between the Successful set and the Failed set.**\nWhether each turn’s “e” value in a given dialogue is 0 or 1 is determined retrospectively based on the outcome at the end of the dialogue, so we believe that the successful set and failed set are unlikely to be  unbalanced. Specifically, for a successful dialogue, “e” is 1 for all turns in that dialogue: $\\lbrace(a_t, \\hat{a}_t, 1) | t \\in T \\rbrace$. Conversely, for a failed dialogue, “e” is 0 for all turns in that dialogue: $\\lbrace (a_t, \\hat{a}_t, 0) | t \\in T \\rbrace$. Note that we did not consider the ratio of successful and failed dialogues, so there might be some potential for imbalance in that respect.\n\n#### **Q4. Why was GenPPN not applied to GPT-2 w/ RL?**\nWe did not apply GenPPN to GPT-2 w/ RL because we wanted to see only how the performance of GenPPN, which considers dialogue-level metrics, differs from NLG models optimized at utterance-level like GPT-2 w/ RL.\n\nFollowing your review, we applied GenPPN to GPT-2 w/ RL to test its effectiveness as an additional experiment. The results are shown in the following table:\n        \n        \n| NLG | Success | Inform F1 | Inform Precision | Inform Recall | Book Rate | Turn | DA F1 |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| GPT-2 w/ RL | 72.36 | 76.70 | 73.50 | 85.99 | 76.81 | 7.47 | **81.17** |\n| w/ GenPPN$_\\text{mean}$ | 74.02 | 77.10 | 73.87 | 86.82 | 79.19 | 7.54 | 80.98 |\n| w/ GenPPN$_\\text{absmax}$ | **75.20** | **78.79** | **75.58** | **88.02** | **79.80** | **7.15** | 80.08 |\n\nFrom the results, we observed that even for an NLG model optimized at the utterance-level, GenPPN was able to further improve its dialogue performance. However, the utterance-level metric DA F1 slightly decreased from the original GPT-2 w/ RL score, suggesting a limited correlation between dialogue-level and utterance-level performance.\n        \n#### **Q5. To what extent did the RL optimization cause GenPPN to over-align with the task completion metrics?**\nConsidering the human evaluation results, no significant differences were found in the subjective evaluation scores. Hence, we believe that there is no excessive alignment for task completion and no lack of naturalness. This may be due to the constraint on naturalness using KL divergence, as shown in Equation (7) in Section 3.4, which prevented excessive alignment to task completion only.\n\n#### **Q6. In human evaluations, the user subjectivity assessment metrics deteriorated; is this due to the RL objective?**\nAs you pointed out, the slight (but not significant) deterioration in subjective evaluation scores can be attributed to the RL objective, in which measures related primarily to task completion are considered. In fact, in the original PPN paper [4], rewards that only considered task completion did not improve the human subjective evaluation metrics. A similar phenomenon might have occurred in our study.\n\n#### **References:**\n[1] Takanobu et al., 2020. Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation. In Proc. SIGDIAL2020, pages 297–310.\n\n[2] Chung et al., 2022. Scaling Instruction-Finetuned Language Models. arXiv preprint arXiv:2210.11416.\n\n[3] Touvron et al., 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.\n\n[4] Ohashi and Higashinaka, 2022. Post-processing networks: Method for optimizing pipeline task-oriented dialogue systems using reinforcement learning. In Proc. SIGDIAL2022, pages 1–13.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " 1.  The utilization of reinforcement learning to improve the performance of the post-processing module is insightful, and the reward function design effectively integrates dialogue-level rewards into the utterance-level rewards.\n 2.  GenPPNs offer versatility by applying NLG models with arbitrary structures, making them adaptable and compatible with various systems.\n", "reasons_to_reject": " 1. The task settings or baseline methods might not follow common practices. Why do template-based natural language generation (NLG) models outperform other baselines across various metrics?\n 2.  The implementation of GenPPNs is based on the Alpaca model with instruction tuning. However, there seems to be a missing ablation experiment that would help determine whether the observed improvement in performance can be attributed to reinforcement learning or instruction tuning. \n", "questions_to_authors": "Question A: In the context of reward using DA contribution, whether the value of \"e\" can only be equal to 1 at the end of a dialogue? If this were the case, it raises the possibility of an imbalance between the Successful set and Failed set.\nQuestion B: Why was the main experiment, involving GPT-2 with RL, not conducted using GenPPNs?\nQuestion C: To what extent does the incorporation of task completion metrics in GenPPNs for RL optimization result in an excessive alignment of the final optimized model with the targeted metrics? More specifically, in the context of human evaluation experiments, why did the performance of GenPPNs decline across all three user subjectivity assessment metrics, and can this be attributed to the RL optimization objective?\n", "paper_topic_and_main_contributions": "The paper introduces a novel component called Generative Post-Processing Networks (GenPPNs) designed specifically for enhancing the quality of utterances generated by natural language generation (NLG) modules within task-oriented dialogue systems. GenPPNs leverage the previous dialogue context, dialogue action, and system utterance as input to generate a refined system output that is rewritten based on the original system utterance. To enhance the performance of GenPPNs using reinforcement learning, the paper proposes the incorporation of a dialogue act (DA) contribution, which facilitates the distribution of dialogue-level rewards to utterance-level rewards. This approach optimizes the learning process and improves the task completion performance of three distinct NLG modules, as evaluated on the MWOZ dataset. Moreover, the introduction of the DA contribution proves to be effective in facilitating the learning process of GenPPNs and further improving the overall task completion performance.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xf9nInCRwe", "reviewer": "Reviewer_zF36", "summary": "", "questions": "1. How did you run automatic evaluation in ConvLab-2? I wonder why the result of template-based NLG did not reach more than 82% of success rate? did you follow standard ConvLab-2 evaluation as mentioned in the DSTC-9 Track 2?\n2. Why did you consider only SC-LSTM for ablation study and human evaluation?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the thoughtful and encouraging review. We provide explanations and the new results of additional experiments to respond to your concerns and comments as follows.\n\n#### **Q1. The reward definition of reinforcement learning is mainly based on Task Success, but template-based NLG may be sufficient. Placing objectives on other criteria, such as fluency and naturalness, would be more meaningful.**\n    \nThe primary goal of our study is not to obtain an NLG model with state-of-the-art (SOTA) performance, but rather to enhance the performance of NLG irrespective of its architecture or its base performance. In our experiments, the dialogue performance of NLG models like SC-LSTM and SC-GPT, in addition to Template NLG, were improved using GenPPN, showing that our primary goal has been achieved.\n    \nAlthough NLG baselines in our experiments included Template NLG, known for its high performance [1], in real-world situations, meticulously designed and high-performance NLG models like the template NLG are only sometimes available in other tasks or domains. We believe that GenPPN is a practical approach to improving NLGs in such scenarios.\n    \nWe also believe there is value in GenPPN when considering objectives other than task completion, such as enhancing naturalness and diversity. As additional experiments, we measured distinct-N of utterances generated by each NLG model during the test. The results are presented in the table below:\n    \n| NLG | Success | Distinct-1 | Distinct-2 |\n| --- | :---: | :---: | :---: |\n| Template | 77.25 | 1.91 | 5.37 |\n| Template w/ GenPPN | 78.91 | 1.99 | 5.97 |\n| SC-LSTM | 54.00 | 3.38 | 5.46 |\n| SC-LSTM w/ GenPPN | 72.95 | 3.10 | 7.76 |\n| SC-GPT | 64.94 | 4.39 | 15.89 |\n| SC-GPT w/ GenPPN | 73.63 | 4.62 | 16.08 |\n    \nThe distinct-N of Template NLG is low, which indicates fixed and mechanical utterances. In contrast, when applying GenPPN to SC-LSTM or SC-GPT, we were able to improve the Task Success score to a level comparable to Template NLG while maintaining the original high distinct-N of the model. This result suggests that we achieved dialogue performance comparable to Template NLG with more diverse utterances.\n\n#### **Q2. In the evaluation, the focus is on comparing with the older SC-LSTM. It would be better if there were comparisons with SOTA methods like \"Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning\" [1] or template-based NLG.**\n\nAs you pointed out, a study with the SOTA method would be better. As additional experiments, we first applied GenPPN to GPT-2 w/ RL [1] to test its effectiveness. The results are shown in the following table:\n\n| NLG | Success | Inform F1 | Inform Precision | Inform Recall | Book Rate | Turn | DA F1 |\n| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| GPT-2 w/ RL | 72.36 | 76.70 | 73.50 | 85.99 | 76.81 | 7.47 | **81.17** |\n| w/ GenPPN$_\\text{mean}$ | 74.02 | 77.10 | 73.87 | 86.82 | 79.19 | 7.54 | 80.98 |\n| w/ GenPPN$_\\text{absmax}$ | **75.20** | **78.79** | **75.58** | **88.02** | **79.80** | **7.15** | 80.08 |\n        \nFrom the results, we observed that even for an NLG model optimized at the utterance-level, GenPPN was able to further improve its dialogue performance. However, the utterance-level metric DA F1 slightly decreased from the original GPT-2 w/ RL score, suggesting a limited correlation between dialogue-level and utterance-level performance.\n        \nWe also conducted ablation studies on SOTA models, GPT-2 w/ RL and Template NLG, and the results are as follows (**bold** and _italic_ values indicate the best and next best scores, respectively):\n\n| NLG | Success | Inform F1 | Book Rate | DA F1 |\n| --- | :---: | :---: | :---: | :---: |\n| GPT-2 w/ RL w/ GenPPN | **75.20** | **78.79** | _79.80_ | 80.08 |\n| w/o adaptive $c_{(I,s)}$ | 67.68 | 76.75 | 79.31 | _81.04_ |\n| w/o $e$ | 72.56 | 76.57 | 77.36 | 80.63 |\n| w/o $r(a, \\hat{a})$ | 72.85 | _78.52_ | **81.82** | 71.15 |\n| w/o context | _73.24_ | 77.18 | 77.14 | **85.11** |\n        \n| NLG | Success | Inform F1 | Book Rate | DA F1 |\n| --- | :---: | :---: | :---: | :---: |\n| Template w/ GenPPN | _78.91_ | _79.86_ | _85.19_ | _78.23_ |\n| w/o adaptive $c_{(I,s)}$ | 76.27 | 78.07 | 83.00 | 75.05 |\n| w/o $e$ | 77.15 | 79.29 | 83.30 | 76.89 |\n| w/o $r(a, \\hat{a})$ | 71.09 | 78.14 | 80.89 | 63.68 |\n| w/o context | **79.00** | **79.92** | **85.29** | **78.58** |\n\nFor GPT-2 w/ RL, trends similar to those of SC-LSTM were observed. Thus, it was evident that an adaptive DA contribution that considers both dialogue- and utterance-levels is crucial for Task Success.\n\n#### **Q3. Why does the performance of Template NLG not reach the 82% reported in the DSTC-9 Track 2 ConvLab-2 evaluation?**\nThe difference in Task Success between the template NLG presented on the ConvLab-2 leaderboard (which we will refer to as ConvLab-2's baseline) at 81.3% and our baseline at 77.3% can be mainly attributed to (1) the setting of the NLU module of the user simulator and (2) the setting of the template NLG.\n1. The NLU module of the user simulator in ConvLab-2’s baseline used the utterances in the last three turns as an additional context, while the NLU in our baseline did not. This setting without context is consistent with what was reported for the training of GPT-2 w/ RL [1], and we used this setting to ensure fair comparison in Section 4.5’s main result. This difference in configuration may have affected the performance.\n2. For the Template NLG, there are two modes: the “manual mode” that uses only hand-crafted utterances and the “retrieval mode” that constructs utterances by retrieving actual utterances from the MultiWOZ corpus. In our experiments, we employed the basic manual mode, whereas ConvLab-2’s baseline may be using the retrieval mode.\n\n#### **Q4. Why were only SC-LSTM considered for ablation study and human evaluation?**\nAs described in Sections 4.6 and 4.7, the improvement in Task Success of SC-LSTM w/ GenPPN is the largest compared to other NLG models, so we considered that using SC-LSTM is appropriate to analyze the impact of GenPPN.\n\n#### **References:**\n[1] Ohashi and Higashinaka, 2022. Adaptive Natural Language Generation for Task-Oriented Dialogue via Reinforcement Learning. In Proc. COLING2022, pages 242–252.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of proposing post-processing network to modify the output of the TOD system is useful in practical.\n- The experimental result shows that the proposed method can improve the performance of the TOD system", "reasons_to_reject": "- The motivation is still limited, although it is practical solution. As an example, the reward definition is mainly designed based on the success rate. In fact, using template-based NLG under this motivation is enough in my opinion. If the author considers other criteria for the main motivation such as fluency, naturalness of the generated sentences then it will be more meaningful.\n- The evaluations were highly intended to compare with SC-LSTM which is quite old method. If the evaluation is compared to the SOTA method such as \"Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning\" or template-based NLG then it will be better", "questions_to_authors": "1. How did you run automatic evaluation in ConvLab-2? I wonder why the result of template-based NLG did not reach more than 82% of success rate? did you follow standard ConvLab-2 evaluation as mentioned in the DSTC-9 Track 2?\n2. Why did you consider only SC-LSTM for ablation study and human evaluation?", "paper_topic_and_main_contributions": "This paper proposes a generative post-processing network to modify the output of NLG component in the pipeline task-oriented dialogue (TOD) system. The model was built using LLaMA language model and optimized using PPO with the predefined reward function.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents GenPPN, a generative post-processing components in dialogue systems to improve task completion. Experiments both through simulation and human evaluation on MultiWOZ dataset show improvement to the task completion performance of TOD system. The work is generally considered thorough, and the paper is well-written and easy to follow. One discussion point during rebuttal is around the high performance baseline Template NLG, which is sufficiently answered by author responses.
Decision: Accept-Main