Learning Retrieval Augmentation for Personalized Dialogue Generation
Authors: Qiushi Huang
Shuai Fu
Xubo Liu
Wenwu Wang
Tom Ko
Yu Zhang
Lilian Tang
Venue: EMNLP 2023 Main (2023)
Keywords: Personalized Dialogue Generation
Retrieval-augmented Dialogue Generation
Persona-Based Dialogue Generation

Abstract: Personalized dialogue generation, focusing on generating highly tailored responses by leveraging persona profiles and dialogue context, has gained significant attention in conversational AI applications. However, persona profiles, a prevalent setting in current personalized dialogue datasets, typically composed of merely four to five sentences, may not offer comprehensive descriptions of the persona about the agent, posing a challenge to generate truly personalized dialogues. To handle this problem, we propose $\textbf{L}$earning Retrieval $\textbf{A}$ugmentation for $\textbf{P}$ersonalized $\textbf{D}$ial$\textbf{O}$gue $\textbf{G}$eneration ($\textbf{LAPDOG}$), which studies the potential of leveraging external knowledge for persona dialogue generation. Specifically, the proposed LAPDOG model consists of a story retriever and a dialogue generator. The story retriever uses a given persona profile as queries to retrieve relevant information from the story document, which serves as a supplementary context to augment the persona profile. The dialogue generator utilizes both the dialogue history and the augmented persona profile to generate personalized responses. For optimization, we adopt a joint training framework that collaboratively learns the story retriever and dialogue generator, where the story retriever is optimized towards desired ultimate metrics (e.g., BLEU) to retrieve content for the dialogue generator to generate personalized responses. Experiments conducted on the CONVAI2 dataset with ROCStory as a supplementary data source show that the proposed LAPDOG method substantially outperforms the baselines, indicating the effectiveness of the proposed method. The LAPDOG model code is publicly available for further exploration.

Learning Retrieval Augmentation for Personalized Dialogue Generation
Qiushi Huang 12, Shuai Fu', Xubo Liu', ,Wewwu Wang', Tom Ko', Yu Zhang", Lilian Tang" Universiity of Surrey, 2Southern University of Science and Technology, 'ByteDance Al Lab fqiushi. huang, xubo.liu, w.wang, h.lang) @surrey.cc.uk akhas,, tomkoese, yu.zhang.ust, @ @gmail.com
Abstract
to generate consistent responses based on histor- ical dialogue context and given persona profiles, has recently drawn substantial attention in many applications. For instance, such an agent could ef- fectively adapt to different roles such as customer service representative by tailoring its responses to specific customer needs based on its persona and improving eustomer interaction and satisfac- tion. Besides, personalized responses can foster a sense of human-like interaction social platforms, thereby enriching the user experience.
Personalized dialogue generation, focusing on generating highly tailored responses by lever- aging persona profiles and dialogue context, has gained significant attention conversa- tional Al applications. However, persona pro- files, a prevalent setting in current personal- ized dialogue datasets. typically composed of merely four to five sentences, may not of- fer comprehensive descriptions of the persona about the agent, posing challenge to gener- ate truly personalized dialogues. To handle this problem, we propose Learning Retrieval Augmentation for Personalized DialOgue Generation (LAPDOG), which studies the po- tential leveraging external knowledge persona dialogue generation. Specifically, the proposed LAPDOG model consists of a story retriever and dialogue generator. The story retriever uses given persona profile queries to retrieve relevant information from the story document, which serves supplementary context to augment the persona profile. The dialogue generator utilizes both the dialogue history and the augmented persona profile to generate personalized responses. For optimiza- tion, we adopt a. joint training framework that collaboratively learns the story retriever and di- alogue generator, where the story retriever is op- timized towards desired ultimate metrics (e.g.. BLEU) to retrieve content for the dialogue gen- erator to generate personalized responses. periments conducted the CONVA12 dataset with ROCStory supplementary data source show that proposed LAPDOG method sub- stantially outperforms the baselines, indicating the effectiveness of the proposed method. The LAPDOG model code is publicly available for further exploration."
The persona profiles contain background sen- tences describing the agent (c.g., I like to go hunt- ing.) and play a crucial role in customizing the dialogue. Ideally, persona profile should be as comprehensive possible, containing diverse and detailed descriptions of an agent. However, these persona profiles, typically consisting of only four to five sentences, do not provide comprehensive descriptions for the persona of the agent. Such lack depth and diversity in the persona descrip- tions impedes existing methods (Liu et al., 2020; Song et al., 2021; Huang et al., 2022) from gen- erating highly personalized and contextually rich responses, though they have shown capabilities in producing grammatically correct and human responses. In essence, those models restricted by the static and limited persona profile. Hence, those models fail to dynamically incorporate more intensive extra personalized profiles when decod- ing the responses.
Though the given persona profile is limited, there are many external textual resources to describe per- sonality and daily life cireumstances. Hence, it is intuitive to ask: can we use other related datasets to enrich the details of the persona profile? This key question has not been thoroughly explored in exist- ing methods, which primarily rely on the persona profile and dialogue context alone. An immediate issue is which types of external datasets could be used. A promising source is story data since they
1 Introduction
Personalized dialogue generation (Zhang et al., 2018; Dinan al., 2019), which prompts agent
*Conresponding authors. hhanp:///ghub..commnsssiiiiiiila

---

encompass diverse life events, personality traits, motivations, and experiences, which can contribute to a more detailed and realistic persona. For exam- ple, given the persona sentence as "I like to work on vintage cars.", potential retrieved stories' titles can be "Antique Car Show" and "Mechanic", the details of the story content can be found in the appendix (Table 7). Furthermore, the clear and inherent structure in stories enhance the consis- tency of the persona. this work, we choose story data to facilitate the generation of more engaging and contextually mcaningful dialogues.
addition to the retrieved content, the persona in- formation and dialogue context are also integrated
into the dialogue generator. Furthermore. LAP- DOG adopts a cooperative framework wherein the retriever and generator are jointly trained. This pro- cess enables LAPDOG to generate personalized re- sponses that are coherent, contextually rich, and in line with the persona of the agent. Unlike other trieval models (Zhou et al., 2022; Santhanam al., 2022) that rely on annotated retrieval datasets, our method retrieves the supplementary context an end-to-end, unsupervised manner, which can be seamlessly extended to other suitable text sources.
Given the external knowledge (e.g., story data), how to infuse it into the process of personalized dialogue generation straightforwardly remains chal- lenging. The first hurdle is the lack of explicit anno- tations for retrieval, which are the key to selecting relevant and helpful content to augment persona profiles. In addition, the criterion for assessing the efficacy of these contents remains unclear. For instance, retrieval -aggeeetdd generation (RAG) (Lewis al., 2020) is based on predicted proba- bility distribution, which may not directly align with the objective of generating personalized re- sponses. Morcover, simply tuning dense retriever (Karpukhin et al., 2020) in RAG's paradigm may result in suboptimal retrieval outcomes the re- triever is inclined to consistently select similar pas- sages for all queries, which may impede the further expansion of the persona profile details.
We conduct experiments on the CONVA12 dataset (Dinan et al.. 2019), which is widely rec- ognized and extensively studied in the field of per- sonalized dialogue generation (Huang et al., 2022; Song et al., 2021; Liu et al., 2020), and the ROC- Story dataset (Mostafazadeh et al., 2016) acts external knowledge. Experiments demonstrate the positive impact of learnable retrieval augmentation on performance. Quantitatively, the proposed LAP- DOG method consistently yields improvements over the baseline models with varying model sizes. Moreover, the retrieved contents offer insights into the rationale behind the generation ability of the generator. Comprehensive ablation studies demon- strate that joint objective guidance outperforms each individual objective and provides insights into the size of retrieval candidates and the use of dif- ferent metrics.
In this paper, we give the first try to utilize the story data as external knowledge for the per- sonalized dialogue generation task and propose a Learning Retrieval Augmentation for Personalized DialOgue Generation (LAPDOG) framework. Specifically, the proposed model LAPDOG, con- sisting of retriever retrieve helpful informa- tion enrich the persona and generator gener- ate dialogues, an end-to-end learnable retrieval methodology for integrating additional contextual information into personalized dialogue generation. LAPDOG utilizes non-differentiable metrics (e.g., BLEU, F1, and ROUGE-L) to guide the training of the retriever by aligning the retriever scores to these desired metrics, thereby facilitating the generation of relevant and diverse personalized responses. To ensure diversity in the retrieval process, we design a retrieval candidate augmentation during training. which prevents consistently selecting similar pas- sages for all queries and provides a broader range contextual inputs for the dialogue generator. In
Overall, our contributions can be summarized follows.
We present novel LAPDOG model for per- sonalized dialogue generation retrieve rel- evant contents in external knowledge the persona using the non-differentiable objective.
We introduce candidate augmentation as a means to enhance learning retrieval augmen- tation, resulting in improved performance and increased diversity of candidate selections dur- ing the inference process.
The proposed LAPDOG framework signifi- cantly enhances the performance over base- lines, showing promising potential for learn- able retrieval augmentation on personalized dialogue generation. Our code and tre-trained model will be open-sourced.

---

2 Related Work
specifically for personalized dialogue generation with focus on optimizing desired objectives rather than the language model's probability distribution. Since RePlug, Toolformer, and FLARE are based on large language models or their API calls, we do not include them in the comparison to LAPDOG. Compared with other models, we do not rely on retrieval annotations bootstrapped retrieval an- notations. The training objectives are directly com- puted from comparison between the generated text and ground truth, rather than relying on train- ing probabilitics that are not always aligned with the desired metrics. Additionally, we introduce a candidate augmentation to avoid the limitations of a confined candidate set. This broadens the scope of potential dialogues and better captures the richness and diversity of an agent's persona.
2.1 Personalized Dialogue Generation
Based on the PersonaChat dataset (Zhang et al., 2018), Dinan al. curate the CONVAI2 dataset, which contains brief persona with four to five sentences for each interlocutor, This unique dataset has become standard benchmark for the person alized dialogue generation task and built on this dataset, there are numerous studies, each of which approaches personalized dialogue generation from diverse perspectives. For example, Wolf et al. proposes fine-tuning model based the GPT2 model (Radford et al., 2019). Song al. integrates three BERT models (Devlin et al., 2019) via rein- forcement learning to generate responses, Liu et al. propose a transmitter and receiver model, which utilizes reinforcement learning with manually de- signed rewards for further refinement, for the per- sonalized dialogue generation task. et al. adopt model-agnostic data augmentation to language models, such GPT2 and BERT (Devlin et al., 2019), augment the training set with pseudo training data. Huang al. devise an adaptive at- tention mechanism to integrate information from persona and context encoders seamlessly. In con- trast to the aforementioned models, the proposed LAPDOG method introduces an end-to-end dense retriever framework to simultancously augment the input of the generator from external data source and tune the retriever.
3 Methodology
In this section, we introduce the proposed LAP- DOG model.
3.1 Task Formulation
In a persona-based conversation session denoted by C - (P,U), the persona P = (Pi,.... Pe} con- sists of e profile sentences providing background information about machine interlocutor and the dialogue context U {uh,1; ; 1, .., (,,n) en- compasses the exchange of utterances between a human interlocutor and the machine interlocutor m7. the task of persona-based dialogue gener- ation, the persona P is used to characterize the machine interlocutor m, but it contains only four to five sentences. i.e., 4 S e S 5. The conver- sation always starts by the human interlocutor h. The primary objective of this task is to generate the response r Um.n based on the given persona P and dialogue context U.
2.2
Retrieval-Augmenneer Text Generation
There are some works to incorporate retrievers into their respective models via different integration strategies to enhance text generation tasks. For instance, DocPrompting (Zhou et al., 2022) cu- rated a retrieval annotation dataset train re- triever to retrieve and do input augmentation for code generation. Toolformer (Schick et al., 2023) bootstraps sarreval--nnotatte data, which performs fine-tuning on language models for the retrieval- augmentation ability. FLARE (Jiang et al., 2023) extends the Toolformer to large language models such as (Ouyang et al., 2022) with special- -esigned prompts. RePlug (Shi et al., 2023) further re- fines the retriever by distilling the knowledge from the language model's probability. RAG (Lewis al., 2020) jointly trains the retriever and lan- guage model, which updates the retriever by the language model's probability. Different from those models, the proposed LAPDOG model is designed
The persona short and hence cannot give a full characterization for the background informa- tion. To enrich the persona, we utilize a retrieval corpus D consisting of stories from story dataset (e.g., ROCStory (Mostafazadch et al., 2016)). Note that there is explicit annotation between D and P, necessitating an alternative approach evaluate the usefulness of the retrieval content.
3.2 The Architecture
shown Figure the architecture of the LAP- DOG model consists of generator, which adopts a transformer based recoder-decoder structure, to

---

Generator
Supervised Tuned Generator
Update
Ground Truth
Supervised Training
Dual-encoder Retriever
Generator
Generated
RROCBtory
Contewt
Generatedy
Metrics
Storyy
D2
D2
Dx
Story
Candidate Augmenlation
Cinerateda
Metticg
Denired Matrics
(s
l.
Dislogus History
Update
Figure I: An illustration of the two-stage training process in the proposed LAPDOG model.
generate dialogues and a dual-encoder retriever to efficiently obtain relevant information from ternal story corpus.
the generator (refer to Section 3.3.1). In the second stage, the framework starts tune the retriever and learn the retrieval augmentation jointly To learn retrieval augmentation (refer Section 3.3.2), the retriever's loss is computed from the evaluation metrics between the output the generator and the ground truth. During the process of learning re- trieval augmentation, to prevent the retriever from stagnating around limited set of candidates, we design the retrieval candidate augmentation (refer to Section 3.3.3). a method ensuring diversity in the retrieval process. Afterward, we enrich the input of the generator with retrieval-enhanced data and compute a generator loss based on the augmented input (refer to Section 3.3.4). Finally, we com- bine losses from both the generator and retriever to jointly train the two components (refer to Section 3.3.5), In the following sections, we introduce each part in detail.
Retriever Based on (Karpukhin et al., 2020), the retriever adopts transformer-bes encoder to em- bed the query and the story corpus, respectively. The retriever then calculates the dot product simi- larity score between the query and each story via their average pooled embeddings. Stories with the K' highest similarity scores are retrieved.
Generator The generator takes a transformer- based encoder-decodee architecture to generate the response from the persona, dialogue history, and re- trieved contents. To integrate the retrieved contents with the persona and dialogue history, we leverage the Fusion-in- in-Decoder (FiD) technique (Izacard and Grave, 2021). Specifically, each retrieved story is combined with the persona and context and indi- vidually encoded. The resulting encoded contexts are concatenated and cross-attended in the decoder to generate the final response.
3.3.1 Supervised Training
First, we train generator that accepts persona P. and context U input, and the ground-truth response r as the target without involving any re- trieval results. Hence, this stage is minimize the negative log- log-likelihood, which is formulated as
3.3 Training Process
It straightforward to directly train the genera- tor and retriever using the generator's s probability distribution in a way similar to the RAG method. However, this strategy does not work well since the retriever would trap into fixed candidate set and the predicted probability distribution is not always aligned with the desired objectives in the personalized dialogue generation task. Hence, as depicted in Figure 1, the LAPDOG model adopts a two-stage training procedure. In the first stage, the training process starts with supervised training for
1NL. = log(Go(r)P.U))
(1)
Σ (og(Ga(r)PP.UUraa)))
where ya denotes the /-th token in r. T<t denotes the sequence containing the first to (f- 1)-th tokens in r, Go(.) denotes the predicted probability distri- bution of the generator, and denotes parameters of the generator.

---

After supervised training, we obtain a supervised tuned generator denoted by Grap.
and PR as
CR KL(PRR o (Sg/Ta))
(2)
3.3.2 Learning Retrieval Augmentation
where (L(-.)) denotes the KL divergence, a(-) denotes the softmax function, and T, is temper- ature hyperparameter to control the sensitivity of the similarity scores. Combining LR with the trieval candidate augmentation introduced in the next section, we can update the retriever.
Intuitively, with the retrieval content as an augmen- tation, the goal to improve the generated content in terms of desired metrics. However, it is hard to build direct connections between retrieval con- tents and the quality of the final generated response to update the retriever. To achieve that, we use the trained generator Sup evaluator give feedback.
3.3.3 Retrieval Candidate Augmentation
During the training process, there is risk that the retriever gets stuck in local optimum and con- sistently retrieves a fixed set or a narrow range of candidates. Consequently, the generator fails to learn from the retriever and disregards the retrieved content. To address this challenge, we design trieval candidate augmentation to incorporate ran- domly sampled stories to encourage the framework to explore a wider range of candidates. Specifically, we first replace each d, with a randomly selected candidate d.u0 at a probability of as
Specifically, given the metric values from the trained generator Ssu.. we use those metric values as feedback to guide the update of the retriever. In other words, if the generator Guup finds that the retrieved story d; E Dy is useful to improve the per- formance in terms of the given metrics, we should encourage the retriever to rank the score of d; to be higher. In this way, we can let the model decide the usefulness of the retrieval content and avoid relying on the retrieval annotations between query 9 and story d, However, since the whole genera- tion and metric calculation process is hard or even impossible to be differentiate, we cannot directly perform gradient descent with respect to the caleu- lated metrics update the retriever. To solve this problem, instead we transform the metric values into a probability distribution as
وa CandAug(ds, p): E DDy
where D, denotes the set of retrieval stories, and forms perturbed set DT {d, "
K J==1- Then
we can compute the dot product similarity between the query q and each d:ng as the retrieval scores (00 Sq = (s )==1- where sgg denotes the dot product similarity between q and d.n0 Then we can apply the learning retrieval augmentation to ST and based on Eq. (2) minimize the following loss to update the retriever
MM,, Gen(Gsup: (di, P.U P,U))) Tg
exp
P
K DE exp
M(y, Gen(G snep: (de, P. P.U))
Tg
where M(y, 9) denotes a metric function to eval- uate the quality of the generated text á given the ground truth y, Gen(Srup (d,, P, P)) denotes the decoded text generated by Goup given (d,, U,U) as the input, and Ty is temperature hyperparame- ter to control the sensitivity of the metric. Here the metric function satisfies that higher value of M(..)) indicates better performance. If a smaller value of M(.,.) indicates better performance, we can replace :( with - M(; -) in the calculation of It is easy to see that a useful d; will have a large p4 and hence p; can be used as supervised signal to guide the learning of the retriever. That we could make the similarity score returned by the retriever close to PR - { Formally, sup- pose we have top- retrieval stories D, with its retrieval scores S,- E IRKK with respect to the query q. we can minimize the KL divergence between S.
aug R
KL( a (S0u /Tx))-
3.3.4 Training Retrieval-Augmented Generator
With the retrieval content obtained by the retriever, we hope to generate the response more accurately and hence we can further supervised train the gener- ator in way similar to the first stage (i.c., Section 3.3.1). Specifically,
minimize the negative log-likelihood of the response given the persona, dialogue context, and retrieval content
Lg = P, D2.9)
(3)
log(Go(r:IP,U,.))) r<t)).
It is casy to see that Eq. (3) is similar to Eq. (1)
with additionally inputting the retrieval content.

---

3.3.5 Reteraorr-Generator Joint Training At the final stage, aim to jointly train the retriever and generator to further improve them. Specifically, we minimize the sum of the losses of the two components as
existing personas within the dialogue. We have un- dertaken pre-processing of the ROCStory to align the narrative style more closely with persona rep- resentation, including changes like transforming 'he' to 'I' and 'does' to 'do'. The detailed pre- processing is listed in Appendix A.7. Statically, there are 98,161 stories within the corpus, and each story is composed of 5 sentences.
L L
Laug + Lg.
(4)
In Eq. (4), the two loss functions treated equally. Generally speaking, introducing and tun- ing a weighting hyperparameter between the two losses may result in better performance but it incurs computational costs when tuning it. For simplicity, we did not introduce this hyperparameter and this could be left for future study.
4.3 Experimental Settings
We employ T5 series models (Raffel et al., 2020) (small, base, XL) as the foundational model used for the generator. We initialize our generator with pre-trained weights from T5 and subsequently fine- tune it on the CONVAI2 dataset as Ssup- The dense retriever is initialized with Contriever (Izac- ard et al., 2021), a dual-encoder retriever that shares similar encoder structure to BERT (Devlin et al., 2019). We performed fine-tuning on both the re- triever and generator using Gsup. with learning rate of 5 10 and p 0.5. We further tune for learning retrieval augmentation based on the su- pervised foundation models one epoch. We use persona profile as the query to retrieve the relevant stories.
To summarize, Algorithm 2 in the appendix de- scribes the complete two-stage training process.
3.4 Inference Process
During inference, stories from the ROCStory dataset are fetched in alignment with the provided persona and then integrated into the dialogue con- text using the Fusion-in-Decodee (FiD) technique (Izacard and Grave, 2021). Each combination of story, persona, and context is individually encoded, These encoded contexts are concatenated and pro- cessed via cross-attention in the decoder to produce the final response in an auto-regressive fashion. Ad- ditional experiments evaluating the effects of vari- ous query combinations, such as persona+context and context alone, are detailed in Appendix A.2 to highlight their impact on performance.
4.4 Evaluation Metric
LAPDOG aims to optimize for some generation metrics to enhance the dialogue quality. The evalua- tion comprises three metrics. The first metric is F1, which computes the harmonic mean of precision and recall word level between the generated text and the ground truth. The second metric is BLEU (Papineni et al., 2002: Post, 2018), an n- gram precision-basee measure that quantifies the overlap between the generated text and the ground truth by penalizing for overly long or short outputs. The third metric is ROUGE-L (Lin, 2004), vari- ant of ROUGE that considers the longest common subsequence between the generated text and the ground truth, effectively measure sentence-level structural similarity. With those metrics, we ensure a comprehensive assessment of the quality of gen- crated dialogues. To enhance the aforementioned three metrics, we sum these three metrics together as the overall metric to train the LAPDOG model.
4
Experiment
In this section, we empirically evaluate the pro- posed LAPDOG model.
4.1 Dataset
ConvA12 (Dinan et al., 2019) is dialogue dataset collected from the crowd, featuring 8939/1000 multi-turn conversations that rely on 1155/100 per- sona descriptions for the train/dev splits. Each persona is succinctly depicted by approximately 5 profile sentences. Paired workers engaged in inter- active conversations based predefined personas.
4.2 Retrieval Corpus
4.5 Baseline
Given the absence of paired annotated retrieval corpus connected to ConvAl2 we employ ROC- Story (Mostafazadeh et al., 2016) as an auxiliary re- trieval dataset. Our aim is for the narratives within this dataset to serve as supplemental content to the
To compare the enhancement between different retrieval-augmentaiie approaches, we compared the results with the following baselines. First, we compare the LAPGOG with T5 S/B//XX models,

---

Model
F1
BLEU
ROUGE-L
F1,
BLEU,
ROUGE-L4
AVG
T5 Sap
13.90
3.03
14.15
0.00%
0.00%
0.00%
0.00%
T5
Rf
14.09
3.08
14.00
1.37%
1.59%
-1.03%
0.64%
Sup
T5
+R
RRL
2.50
0.21
5.49
-81.98%
-93.77%
-61.22%
-78.76%
Sup
T5
+RAG
14.20
2.94
14.10
2.14%
-2.84%
-0.35%
-0.35%
T5 Sup
+LAPDOG
14.62
3.23
14.44
5.17%
6.57%
2.07%
4.60%
T5
13.47
3.42
14.93
0.00%
0.00%
0.00%
0.00%
T5
+R
14.36
3.34
15.05
-7.18%
-2.31%
0.82%
-2.89%
Sup
TS +LAPDOG
16.08
3,53
15.33
3.94%
3.21%
2.67%
3.27% %
Sup
T5XL.
16.22
3.55
15.55
0.00%
0.00%
0.00%
0.00%
T5
Sap +R fir
16.27
3.36
15.02
0.31%
-5.32%
-3.41%
-2.81%
X +LAPDOG Sup
17.11
3.56
15.64
5.49%
0.30%
%
2.11% %
Table I: Experimental results of various methods based on language models with varying sizes. + denotes the relative improvement over the supervised foundation model. The best result under each setting shown in bold.
where S/B/XL indicates the model size small, base, XL respectively. Top T/ serves as the foundation models Suup. We also add fixed re- triever R, initialized from Contriever to validate the effectiveness of tuned and untuned retrievers. Meanwhile, utilized the reinforcement learning one baseline, where the reward set as the desired objective. Lastly, we introduce the RAG tuning that updates the retriever based on the generator's ' training output probabili- ties instead of the desired metric, which is to val- idate the direct and indirect metric tuning for the objective.
Method LAPDOG w/o BLEU
BLEU
ROUGE-L
F1
3.23
14.44
14.62
3.07
14.35
14.29
w/o FI
2.87
13.88
13.88
w/o ROUGE-L
2.96
14.12
13.99
Table 2: Ablation study with respect the of met- rics in the LAPDOG model.
tuning Sup +R ftx +RL) R(L)
of the generated text. In contrast, the LAPDOG- enhanced model, Sup *LAPDOG shows the highest improvements in all metrics among small models. This indicates that LAPDOG significantly enhances the ability to generate high-quality text and captures the desired metrics more effectively than other models.
4.6 Results
The T5$ Sup model forms our baseline. Augment- ing with a fixed retriever, T5 SuptR fi shows a slight improvement in FI and BLEU scores but a small decrease in the ROUGE-L score. This in- dicates moderate enhancement in both F1 and BLEU, though the decrease in ROUGE-L suggests trade-off in terms of capturing long-distance de- pendencies.
For larger models, similar phenomena are ob- served. LAPDOG consistently delivers the best im- provements over the base model, no matter whether is T Sup T5X Sup This suggests that the efficacy of LAPDOG is not confined smaller models and scales well with the model size.
Sup
Ablation Studies
The results for reinforcement learning tuning, exhibit a significant degradation across all metrics, indicating that this method might be not so effective for this task. This could be due to the challenge of setting an appropriate reward function for reinforcement learning.
We conduct ablation studies with respect to the metrics, the number of candidates, candidate aug- mentation, and training strategy, respectively.
5.1 Analysis on Metries
The T5% U model slightly outperforms the baseline model in terms of F1 but performs worse in terms of BLEU and ROUGE-L. This suggests that while the model seems generate more correct words, there may be a compromise on the overall grammatical and semantic quality
To understand the individual contribution of each metric, we perform ablation experiments by suc- cessively removing one metric from the combined optimization process with results shown in Table 2. When we remove BLEU, the performance experi- ences a slight drop across all metrics with BLEU

---

Retrieval Diversity
ROUCE-L
3.25
14.5
Feed Retriever
1353
14.4
CandAug
3.15
25
14.3
600
Unig Retrieved Stories
3.55
Figure Comparison of the unique number of stories
retrieved by different models.
2.50
14.0
Method
BLEU
ROUGE-L
F1
K-4
K-1
X-4
6-b
K-b
LAPDOG
3.23
14.44
14.62
Figure 2: Ablation study with respect to the number of candidates K.
w/o CandAug
3.14
14.19
14.43
LAPDOG scratch
2.95
13.90
14.23
decreasing to 3.07, ROUGE-L to 14.35, and Fl to 14.29, suggesting that BLEU contributes to more precise matching of response generation. More- over, when exclude F1, we more signifi- cant reduction in the performance, indicating the crucial role of F1 to ensure the overlap of words between the generated responses and the ground truth. Lastly, the removal of ROUGE-L also results in decrease in the performance across all three metrics, showing its essential role in evaluating the coherence of generated dialogues. In summary, each metric contributes uniquely to the optimiza- tion process, and their combination in LAPDOG provides a more comprehensive guide for the gen- eration high-quality, personalized dialogues.
Table 3: Ablation study the candidate augmentation and training strategy.
mentation. The incorporation of candidate augmen- tation leads superior performance across all three evaluation metrics. Specifically, LAPDOG with- out candidate augmentation attains slightly lower scores in all three metrics. This indicates that the inclusion of candidate augmentation enhances the overall performance of our model, confirming its crucial role in the proposed LAPDOG model.
To further investigate the impact candidate augmentation the retrieval diversity, we count the number of unique stories retrieved during test- ing. As shown in Figure 3, the LAPDOG method with candidate augmentation retrieves 1570 unique stories, whereas without it, the model only retrieves 738 unique stories. This result implies that can- didate augmentation significantly contributes the retrieval diversity. Other methods like RAG and fixed retriever manage to retrieve 25 and 1351 unique stories, respectively. This underlines the effectiveness the proposed candidate augmen- tation approach in enhancing the diversity of the retrieval process, which in turn can help generate more personalized and contextually rich responses.
5.2 Number of the Candidates
As shown in Figure 2, where increases from 2 to 6, we observe a consistent improvement in all met- rics (i.e., BLEU, ROUGE-L, and F1). Generally, increasing the number of retrieval candidates im- proves the performance of the model, as evidenced by the improvements in the BLEU, ROUGE-L, and Fl scores. Interestingly, is observed that the model performance does monotonically in- crease with the number candidates. The perfor- mance fluctuates as K varies, implying that the number of retrieval candidates needs to be care- fully selected. Too few candidates may not provide enough information for generating responses, while too many ones may introduce irrelevant informa- tion, which could potentially confuse the model.
5.4 Training Strategy
In contrast to the two-stage training process, we perform an ablation study by training LAPDOG from scratch, bypassing the first stage. As shown in Table 3, the results exhibit a significant decrease in all the metrics compared with the two-stage ap- proach. Additionally, training directly from scratch requires more time converge when compared with the two-stage training process. Overall, the two-stage training process essential from both performance and efficiency standpoints.
5.3 Candidate Augmentation
The influence of candidate augmentation is plored in two aspects: quantitative performance and the diversity of retrieved stories. Table pro- vides a comparison between the performance of the LAPDOG model with and without candidate aug-

---

6 Conclusion
when applying our model to real-world applica- tions, careful consideration should be given data privacy and consent. It is essential to ensure that all personal information used to generate personalized dialogues obtained ethically and used with the individuals' informed consent. Moreover, the gen- erated content should respect user privacy, dignity, and cultural sensitivities.
In this paper, we introduced LAPDOG, an end- to-end learnable retrieval augmentation personal- ized dialogue generation framework. We show that LAPDOG jointly tunes the retriever with the gen- erator to retrieve useful stories from the ROCStory dataset for enhancing the desired performance over the CONVA12 dataset. LAPDOG gains consistent performance enhancement over language models with varying sizes.
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. Henighan, R. Child, A. Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christo- pher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language mod- els are few-shot learners. arXiv:2005. 14165.
Acknowledgements
This work supported by NSFC general grant 620761 88 and Shenzhen fundamental research pro- gram JCY122021632410000000
Limitations
Given resource constraints, in this paper, we em- ploy language models such as T5 and do not con- duct experiments based on currently prevalent large language models (Brown al., 2020; Ouyang et al., 2022). Recognizing the enhanced reasoning capa- bilities of large language models, we posit that tuning the retrieval content with such models could yield significant advantages. Additionally, due to resource limitations, study small number of extracted passages (i.e., 2-6) and short context length (i.e., 512 tokens). Nevertheless, ipate that incorporating a larger set of integrated stories and a longer context would further enhance the performance. Also, a more diverse objective rather than the summation of FI, ROUGE, and BLEU might be more helpful to train an engaging conversational Al system. Also, the generator is simply conventional T5 model rather than explic- itly designed models, which could help improve the performance of the proposed LAPDOG model further.
Cao, Wei Bi, Meng Fang, Shuming Shi, and Dacheng Tao. 2022. A model-agnostic data manipu- lation method for persona-based dialogue generation. OrXiv:2204.09880 229660967.....
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In North American Chapter of the Asso- ciation for Computational Linguistics, pages 4171- 4186.
antic-
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur D. Szlam, lulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexan- der I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason Weston, 2019. The sec- ond conversational intelligence challenge (convai2). arXiv:
Qiushi Huang, Yu Zhang, Tom Ko, Xubo Liu, Bo Wu, Wenwu Wang, and Lilian Tang. 2022. Personalized dialogue generation with persona-adaptive attention. arXiv preprint arXiv:2210.15082
Ethics Statement
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Esduardd Grave 2021. Unsupervised dense infor- mation retrieval with contrastive learning.
This work proposes a novel LAPDOG model for personalized dialogue generation, focusing on gen- erating highly tailored responses by leveraging per- profiles and dialogue context. As with all machine learning applications, it is crucial to con- sider the ethical implications. The use of personal information our model is limited to fictional per- sona profiles, and we do not handle or store any real personal data in our experiments. However,
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.

---

Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang. Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation.
Kishore Papineni. Salim Roukos. Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Compu- national Linguistics, Association for Computational Linguisties, page 311-318, USA, Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguisties.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hcin- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Proceedings of the 34th Inter- national Conference on Neural Information Process- ing Systems, NIPS'20, Red Hook, NY, USA. Curran Associates Inc.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Zekang Li, Cheng Niu, Fandong Meng, Yang Feng, Qian Li, and Jie Zhou, 2019. Incremental transformer with deliberation decoder for document grounded conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 12-21, Florence, Italy. Association for Computational Linguistics.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3715-3734, Seat- tle. United States. Association for Computational Linguisties.
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessiy, Roherta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Tool- former: Language models can teach themselves to use tools.
Qian Liu, Yihong Chen, Bei Chen, GaanGG Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. 2020. You impress me: Dialogue generation via mutual per- sona perception. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisticy, pages 1417-1427, Online. Association for Computational Linguistics.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2023. Replug: Retrieval-augmeened black-box language models. ArXiv, abs/2301.
Haoyu Song, Yan Wang, Kaiyan Zhang, Weinan Zhang, and Ting Liu. 2021. Bob: Bert over bert for train- ing persona-based based dialogue models from limited per- sonalized data. In Association for Computational Linguistics.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruy Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 839-849, San Diego, Cal- ifornia. Association for Computational Linguisties.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conver- sational agents. /91.00999
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur D. Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: have a dog, do you have pets too? In Association for Computational Linguistics.
Long Ouyang, Jeff Wu. Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155 55.
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020. Low-resource nooweedge-gouunte dialogue generation. arXiv preprint arXiv:2002. 10348.

---

Shuyan Zhou, Uri Alon, Frank Xu, Zhengbao Jangg and Graham Neubig. 2022. Doccoder: Generating code by retrieving and reading does. arXiv preprint arXiv:2207.05987.

---

A Appendix
Query
BLEU
ROUGE-L FI
Persona
3.23
14.44
14.62
A.1
Detail Settings for Training
Dialoue
3.19
14.83
15.06
Generated
3.05
14.22
14.32
Setting
Value
One Persona
3.10
14.29
14.39
Tg
0.85
Ts
0.8
Table The evaluation on different queries.
dialogmazturus
Dropout
0.1
Method
BLEU
ROUGE-L F1
LR
5.00E-04
T5
2.01
12.48
11.56
Optimizer
Adam
T5 Tu LLAPDOG
2.21
13.44
12.82
WeightDecay
0.01
RetrievalCandidate
Table 6: Evaluation results on SENVIAI2-Revisee ver- sion, where the persona is paraphrased to more implicit background sentences. The revised version is consid- ered as more difficult task than the original task.
0.5
BatchSize
240
Table 4: Detail experimental settings for training LAP-
DOG.
truth. LAPDOG retrieves several stories about the bad days with his life and family, and this could be clue for him decide divorce. this conver- sation, T5XL sup complains about his life but does not mention anything about divorce. LAPDOG, rein- forced by the retrieval stories and persona. has a stronger intention to generate the divorce decision, which would be more aligned with the intention in the ground truth. The other retrieved stories describe the messes that happened during the work- ing time, which may reflect the persona "I hate my job."
A.2 Query Analysis
As shown in Table 5, we aim to analyze the per- formance impact of different retriever queries. As indicated in Table 5, using the Persona alone as a query achieves the highest BLEU score, albeit with slight trade-off in ROUGE-L and Fl scores. When combining Persona with the Dialogue, the ROUGE-L and F1 scores improve marginally, but at the expense of slightly decreasing in BLEU. Following the idea forward retrieval (Jiang et al., 2023), we experimented with using the generator's output (Generated) as a query, but observed less competitive performance. Lastly, a strategy of us- ing a single sentence from the persona profile cho- sen at random (One Persona) resulted in worse performance all metrics.
In the conversation mentioned in Table 12, the conversation is simple start with "How are you doing today?". T5XL gives standard, safe, but bland answer "I am good. How are you?", while LAPDOG incorporates stories and persona about the gym to answer with "I'm good. Just got back from the gym. How are you?", which would be more
A.3 Evaluation Results on CONVIA12 Revised Dataset
and engaging. Addi- tionally, the story "Lifestyle Change" provides a good clue for the agent about why he decided to go to the gym, and the story "Home Gym" describes the enthusiasm about the workout. These would potentially provide the model with enriched infor- mation on generating personalized responses.
As shown in Table 6, LAPDOG consistently en- hances the performance on the revised dataset, where the persona is paraphrased more implicit background sentences. The revised version is con- sidered a more difficult task than the original task.
As shown in Table 13, the sup model simply replied with "I hope so!" without further informa- tive content, while LAPDOG answers with richer information "I hope so! I I'm only in grade 3 I I'm hoping to go to Disney World soon!", which might consider the information from both persona and stories. Additionally, the retrieved stories like "Dream Job or "Disneyland" are aligned with the agent's favor.
A.4 Case Study
As shown in Tables 11 and 12, we present the case studies on two conversations to compare the gen- eration results among the www.rerreval-aggmeene results of X X +LADDL. and the ground truth.
sup retrieval augmented results of
For the conversation in Table 11, the agent is go- ing to talk about divorce as indicated in the ground

---

Story Examples
Antique Car Show
like fixing cars.
have just finished repairing and restoring an antique sports car.
proudly enter local car show for antique vehicles.
won a cash prize for my hard work! Now have enough money to buy another antique car to restore.
Mechanic
am mechanic and love to work on cars.
work in shop three days week.
In my spare time fix cars for people in my garage.
do great work fast pace for small fee. get two incomes doing what love.
Table Two examples of stories, pre- pre-processed from the ROCStory dataset.
Algorithm Pre-processing Procedure of Story Corpus
Input: A story corpus Dori
Output: A pre-processed story corpus D
1: Extract named entity using BERT-BASE-NES from Dori
2: Filter out cerson-related named entity with tag B-PER
3: Replace the person-related named entity with the first-person narratives within stories in Dori
4: Process the first-person stories over grammatical error correct model pec-t5_small
5: Output the corrected stories as 'D
Algorithm 2 Complete Training Process of LAPDOG
Input:
Persona sentences P, dialogue context U, ground truth response a generator G, a dense retriever R. and story corpus 'D
Output: A tuned retriever Rruned, tuned generator Gtuned
1: Construct query from query function Query(P,U)
2: Stagel:
3: Initialize Gsup with G
4: Train supervised tuned generator Gsup given input (P,U) and ground truth y
5: Train Gzup until converge
6: Stage2:
7: Retrieve top-K stories Dg given
8: Apply Candidate Augmentation Du CandAug(di, p); E Dy
9: Compute retriever scores Sqn0 between query q and Dau
for Retrieved story d; in Dou do
Construct augmented input by concatenation a; [d;; P.U]
12: Generate text by pred, (a;)
13:
Compute metrics as m, = M(pred,,, v)
14: end for
15: Gather M = (softmax((( (ectmmax((m:))555 k A = (a)) [=
16: Compute retriever's loss by C = KL.MM,, saug)
17: for augmented input a; in A do
18: Compute negative log- likelikeehood loss LG with input a, and ground truth target y
19: Update Goup and R by Laug + Lg as Gowned and Riuned
20: end for
21: Repeat the steps Stage2 until converge

---

A.5
Comparison to Traditional Knowledge Dialogue Approaches
A.8 Extended Evaluation Metries
The evaluation metrics for LAPDOG have been broadened to incorporate METEOR and BERT scores. This enhancement supplements the foun- dational assessment based on FI, BLEU, and ROUGE-L metrics, presenting a more diverse eval- uation landscape. The additional evaluation out- comes are tabulated in Table
Evaluating LAPDOG against traditional knowledge-grounded dialogue methods
crucial. A comparison with FLow-Resouree Showledge-Grounder Dialogue Generation" (Zhao et al., 2020) was considered but not possible due to the unavailability of its code. Instead, we selected ITDD (Incremental Transformer with Deliberation Decoder) (Li et al., 2019) for this purpose, given its proven effectiveness and wide recognition.
Based on the results in Table 9, LAPDOG excels in the METEOR score relative to baseline models, showcasing its capability in nuanced linguistic com- prehension. However, the variance in BERT scores minimal, likely due LAPDOG's optimization for traditional metrics. Enhancing performance by tailoring optimization for BERT scores represents promising area for future inquiry.
A.5.1 ITDD Experimental Setup
LAPDOG and ITDD are based on different prin- ciples. LAPDOG uses an unsupervised approach to train both retriever and generator for extract- ing relevant content from external corpus. On the other hand, ITDD is designed merge pre- annotated Documennt-convessteee a fair comparison, we used an off-the-shelf retriever (Izacard et al., 2021) to create paired data from the persona and ROC story corpus for the ITDD model.
A.9 Additional Related Work on Large Language Models (LLMs)
pairs. To ensure
Language models compute probability distribu- tions over text sequences. Recent advancements have escalated these models from millions of pa- rameters (Radford et al., 2019) to billions (Brown et al., 2020), extending the training corpus to en- compass web texts and instructional data (Ouyang et al., 2022). These strides have significantly en- hanced the performance of large language models (LLMs) across myriad of NLP tasks. Notably, in conversational tasks, the quality of generated text improves with the expansion of both the model size and training corpus. Our proposed approach, LAP- DOG, diverges from the prevailing trend of scaling; leverages curievall-uggeene more diverse and interpretable results, albeit with smaller model parameters and corpus size. Despite employing a smaller model in our experiments, we posit that our adaptive retrieval approach could complement existing LLMs, thereby potentially elevating their result-generation efficacy.
A.5.2 Comparison Results
We compared LAPDOG and ITDD using various metries, and the results presented in the table be- low, showcasing LAPDOG's superior performance.
Method ITDD
BLEU
ROUGE-L
9.71
0.66
10.90
T5-S+LAPDOG
14.62
3.23
14.44
generation to yield
T5-++LGPDOG
16.08
3.53
15.33
TS-L+LAPDOG
17.11
3.56
15.64
Table 8: Comparison of LAPDOG and ITDD key performance metrics.
These results highlight LAPDOG's effectiveness compared to the traditional ITDD method, enhanc- ing the paper's overall assessment of LAPDOG's performance.
A.10 Human Evaluation
We conducted a human evaluation to gauge the preference between the vetrieval-augmentt LAP- DOG and the fine-tuned T5 model. Evaluators were presented with a dialogue accompanied by two responses from each model and were asked to choose their preferred response.
A.6 Complete Training Procedure
The complete training procedure is described Algorithm 2.
Referring Table 10, while LAPDOG was pre- ferred by 56.32% of the evaluators, the difference between the two models is relatively narrow. This indicates that both models have their merits
A.7
Pre-process ROCStory Coprus
The pre-processing procedure is described at Algo- rithm

---

Model
METEOR
BERT-FI
BERT-PRECISION
BERT-RECALL.
T5
16.45
85.95
87.07
84.88
TSX +R, Sup X Sup
16.21
85.78
86.99
84.62
T5 +LADOG
17.76
85.99
87.35
84.70
Table 9: Additional evaluation metrics, METEOR and BERT scores.
LAPDOG Fune-tuned Model
Human Preferred 56.32%
43.68%
Table 10: Comparison of human preference for re- sponses generated by LAPDOG and fine-tuned T5 model.
certain conversational contexts. The slight edge for LAPDOG suggests that retrieval-augmentte re- sponses might align closer human expectations in some scenarios. Delving deeper into the nuances this preference, especially across varied dialogue topics, would provide more comprehensive un- derstanding.

---

Conversation #1
am 40 years old.
work as a car salesman.
Persona
My wife spends all my money.
T hate my job.
I'm planning getting a divorce.
Dialogue
Agent: know the struggle, I volunteer at homeless shelter.
Human: hope not either, would not wish it on my worse enemy.
T5XL sup
am 40, so think am done with it.
T5 L
LAADOOG
Yeah, I feel you. want to get divorce.
Ground Truth
Between wife, job with no money, I am thinking now of divoree.
Money
have a full-time job.
My wife stays at home.
One day, worked an over-time shift.
decided to spend all the extra money on myself.
This made my wife sad.
Randy leaves
am man who has kids. started arguing with my wife one day. got so upset promised to leave my wife. So that's what did.
left my children and wife.
Retrieval Stories
Cruel Boss
I have been dealing with extra expenses in the family. Since my wife is stay-at-home mom, pay for everything. tried get a raise by talking to my boss the next day. My boss understood my situation and sincerely apologized.
Although I have been my employee for years, the offer was rejected.
Paul Gets Fired
hate my job.
Every day at work I get angry and say mean things to people.
My boss gave me verbal warning about my attitude at work. Currently I am performance plan at work.
Next month I will be fired.
Table 11: The first case study.

---

Conversation #2
I like spend my money cars.
T go to the gym most days just socialize.
Persona
work from home.
watch too much tv in spare time. have never had steady relationship.
Dialogue
Human: Hi, How are you doing today?
T5XL
I am good. How you?
+LAPDOG
I'm good. Just got back from the gym. How are you? I'm pretty good. Just little lonely. Lifestyle Change
sup
Ground Truth
T was overweight and always tired of not having girlfriend.
I decided make some lifestyle changes.
I cancelled my cable TV and joined gym.
exercised and ate healthily for five months.
started to feel good about myself and started to date!
Home Gym
I am addicted to working out and I go every day.
T just started new job that has me working long hours.
These long hours prevent me from being able go to the gym.
By the time get off work the gym is closed.
I went and bought all the stuff needed work home. Mechanic
I am a mechanic and love to work on cars.
I work in a shop three days a week.
Retrieval Stories
In my spare time I fix cars for people in my garage.
I do great work at fast pace for a small fee.
I get two incomes doing what I love.
Car Mechanic
I love to work cars.
I work as mechanic local car repair shop. decided to work on cars in my spare time too.
I advertised on social media that would fix cars for chcap. Before long had a nice little side business going.
The Drive-Through
I ate out every day of my adult life.
I was 27-ycar-old man.
I loved going to drive troughs and preferred eating in my car. One day while I was ordering my favorite food I had heart attack.
I was rushed to the hospital by the employees and went into a coma. died due to my excessive and unhealthy lifestyle later that night.
Table 12: The second case study.

---

Conversation #3
love sing songs from the movie Frozen
in the third grade.
Persona
play with my friends on the playground.
love to go to Disney world every year.
Mickey Mouse is my favorite character.
Human: They're re about 5 years old.
Dialogue
Agent: Yeah Disney is their favorite! We always go to Disney land!
Human: That's great, maybe will meet there one day! hope so!
T sup
TS XE sup
+LAPDOG
hope so! I'm only in grade 3 so I'm hoping to go to Disney World soon!
Ground Truth
Mickey Mouse is my favorite, and love songs from the movie Frozen. Xena The Shy
My friend is very shy.
am always being teased because my name.
The boys at school call my warrior princess. People are so mean.
I wish I was warrior princess so could make them be quiet! Meeting Mickey
I went to Disney World for the first time when I was 7. My parents bought
book collect character signatures. didn't 't notice a line.
was so excited to get my signature ran to hug cut in front of many other kids.
The girl who was next in line yelled at me.
dream job
have always wanted to play character at Disney World. moved to Orlando and applied for a job.
Disney hired me customer service rep. worked very hard to achieve my goal. The other day got promotion to play Mickey Mouse. Pageant
Retrieval Stories
Little was a three-year-old beauty queen.
was very good at walking and smiling for the judges.
This week it was different.
I would have to sing.
The day of the show I looked beautiful.
Too bad butterflies in my tummy flew away with the words to the song.
Cade's Christmas Show
I excited about my Christmas show.
have been practicing Jingle Bells all week.
On the day of the show was so nervous.
On stage look out at the audience to find my mom.
I am in the first row, so now I know everything will be fine.
Disneyland
I have never been to Disneyland.
I love all the Disney characters.
My mom decided to take to Disneyland.
was so excited when got there.
was the best day my young life.
Table 13: The third case study.


Reviews:
[{"review_id": "JrrLY2uGXm", "reviewer": "Reviewer_jXtS", "summary": "", "questions": "The experiments conducted in the study are not solid enough, as most of the baseline models listed in Table 1 demonstrate negative performance. Furthermore, the paper neglects to compare the proposed approach with some traditional methods of knowledge dialogue that could be readily applied to the current scenario.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank you for your constructive comments. We address your concerns as follows.\n\n**********Q1: The experiments conducted in the study are not solid enough, as most of the baseline models listed in Table 1 demonstrate negative performance.**********\n\nWe acknowledge the reviewer's query about the baseline models' negative performance in Table 1. It's worth noting that personalized dialogue generation is not a straightforward task that can be easily tackled by directly training models on the available data. The need for nuanced contextual understanding, effective persona management, and the seamless integration of external knowledge makes it a challenging problem in the NLP landscape. Hence, a negative performance for baseline models is indicative of these challenges. The negative performance underscores the task's complexity rather than the study's limitations.\n\nThe chosen baselines were to isolate the impact of retrieval augmentation, which is a focal point of our study. Each baseline addresses a unique research question:\n\n1. A generator-only model gauges the retriever's impact.\n2. An untuned retriever assesses the value of joint tuning.\n3. Reinforcement learning evaluates the need for metric-based tuning.\n4. RAG serves as a comparative retrieval-augmented framework, a popular framework often used in retrieval-augmented text generation.\n\nThe study's experimental design aims to provide a nuanced understanding of retrieval augmentation. We use T5 as the generator for simplicity, allowing us to focus on the retrieval aspects. Our work adds a unique dimension to the existing solutions by emphasizing learning-based retrieval augmentation. We believe that our LAPDOG framework is promising for augmenting existing SOTA techniques (which usually focus on improving generator solely), a future research direction we are eager to explore.\n\n**Q2: Furthermore, the paper neglects to compare the proposed approach with some traditional methods of knowledge dialogue that could be readily applied to the current scenario.**\n\nWe recognize the reviewer's concerns regarding the need for comparison with traditional methods of knowledge-grounded dialogue generation. We aimed to compare LAPDOG with traditional methods, specifically “Low-Resource Knowledge-Grounded Dialogue Generation.” Unfortunately, the absence of publicly available code for this work posed a challenge. Instead, we compared with ITDD (Incremental Transformer with Deliberation Decoder) [1], the most effective baseline method in the above-mentioned paper and a widely used traditional approach, as an additional baseline method. This allows us to evaluate LAPDOG against traditional frameworks, enhancing the rigour of our experimental design and results. The detailed experimental setting and results are introduced as follows and we will add those results in the revision.\n\n- *ITDD Experimental Setup*\n    - Our LAPDOG model and traditional knowledge-grounded dialogue methods operate under different paradigms. Specifically, LAPDOG is designed to unsupervisedly train a retriever alongside a generator to adaptively extract valuable content from an external corpus, while ITDD learns to fuse the document where document-conversation pairs have already been annotated.\n    - To bridge this methodological difference and ensure a fair comparison, we employed an off-the-shelf retriever [2] to create paired data from the persona and the ROC story corpus, which was subsequently fed into the ITDD model.\n- *Comparison Results*\n    - We performed a meticulous comparison between LAPDOG and ITDD, examining their performance in terms of several key metrics. The results, which are shown in the following table, affirm the effectiveness of LAPDOG over ITDD.\n    \n    | Method | F1 | BLEU | ROUGE-L |\n    | --- | --- | --- | --- |\n    | ITDD | 9.71 | 0.66 | 10.90 |\n    | T5-S+LAPDOG | 14.62 | 3.23 | 14.44 |\n    | T5-B+LAPDOG | 16.08 | 3.53 | 15.33 |\n    | T5-L+LAPDOG | 17.11 | 3.56 | 15.64 |\n\n******************Reference******************\n\n[1] Zekang Li, Cheng Niu, Fandong Meng, Yang Feng, Qian Li, and Jie Zhou. Incremental transformer with deliberation decoder for document grounded conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 12–21, 2019.\n\n[2] Izacard, Gautier, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin and Edouard Grave. “Unsupervised Dense Information Retrieval with Contrastive Learning.” Trans. Mach. Learn. Res. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation behind this study is to investigate the possibilities of utilizing external textual resources, such as story data, to enhance the information within persona profiles and improve the generation of engaging and contextually relevant dialogues. The paper tackles the challenges associated with explicit annotations for retrieval and the selection of pertinent content for persona profile augmentation. The motivation is reasonable.", "reasons_to_reject": "The experiments conducted in the study are not solid enough, as most of the baseline models listed in Table 1 demonstrate negative performance. Furthermore, the paper neglects to compare the proposed approach with some traditional methods of knowledge dialogue [1] that could be readily applied to the current scenario. \n\n[1]. Low-Resource Knowledge-Grounded Dialogue Generation", "questions_to_authors": "The experiments conducted in the study are not solid enough, as most of the baseline models listed in Table 1 demonstrate negative performance. Furthermore, the paper neglects to compare the proposed approach with some traditional methods of knowledge dialogue that could be readily applied to the current scenario.", "paper_topic_and_main_contributions": "The paper introduces a method called Learning Retrieval Augmentation for Personalized Dialogue Generation (LAPDOG) to tackle the challenge of generating genuinely personalized dialogues with constrained persona profiles. LAPDOG utilizes external knowledge from story data to enhance the persona profiles and enhance the generation of personalized responses. The LAPDOG model consists of two stages: retrieval-augmented generation and joint training of the retriever and generator. In the retrieval-augmented generation stage, a dense retriever is trained to retrieve relevant stories based on a query. The generator is subsequently trained using the retrieved stories as additional input. In the joint training stage, both the retriever and generator are trained together to enhance performance. The training process involves minimizing loss functions and optimizing metrics such as F1, BLEU, and ROUGE-L.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j0ukgtRoXj", "reviewer": "Reviewer_d41o", "summary": "", "questions": "> Is the amount of stories you are taking in your dataset enough to cover every type of persona-specified dialogue?\n\n> Why don't you provide other evaluation metrics, BERT score, METEOR, and Human evaluation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank you for your constructive comments. We address your concerns as follows.\n\n**Q1: The model architecture needs to be discussed in detail, particularly the ROC story selection part.**\n\nThanks for your suggestion. In the revision, we will give more details on model architecture. We did not explicitly do selection on the ROCStory dataset, and we just let the retriever adaptively choose appropriate stories. We will introduce more details in the revision.\n\n**Q2: The related work section should include a discussion of the work in the context of LLMs.**\n\nWe appreciate the reviewer's suggestion to discuss our work in the context of Large Language Models (LLMs). Indeed, the rise of LLMs like ChatGPT and LLama1/2 has significantly impacted various tasks in natural language processing, including dialogue generation. We will add a subsection in the “Related Work” section to discuss how our approach both complements and diverges from techniques that utilize LLMs. Specifically, we will address how our model's capability to adaptively retrieve and incorporate external story-based context offers a unique value proposition compared to the more generalized capabilities of LLMs.\n\n**Q3: Is the amount of stories you are taking in your dataset enough to cover every type of persona-specified dialogue?**\n\nWe recognize the inherent limitations of any dataset in fully capturing the diversity of human personas and dialogues. Actually, we cannot guarantee that the story dataset can cover every type of persona-specified dialogue. However, our primary focus in this work is to introduce and validate a new methodological framework for enhancing persona-based dialogue systems by leveraging external story datasets. In light of your suggestion, we are interested in constructing large datasets to cover various types of persona-specific dialogues.\n\n**Q4: Why don't you provide other evaluation metrics, BERT score, METEOR, and Human evaluation?**\n\nWe appreciate the reviewer's suggestion to consider a more comprehensive suite of evaluation metrics, including BERT score, METEOR, and Human evaluation. Our framework, LAPDOG, is primarily designed to optimize a compound metric including F1, BLEU, and ROUGE-L, and naturally our experiments focus on evaluating models in terms of those metrics.\n\nIn light of your suggestion, we have extended our evaluation metrics to include METEOR and BERT scores. The following table summarizes the results.\n\n| Model | METEOR | BERT-SCORE-F1 | BERT-SCORE-PRECISION | BERT-SCORE-RECALL |\n| --- | --- | --- | --- | --- |\n| T5$^{XL}_{Sup}$ | 16.45 | 85.95 | 87.07 | 84.88 |\n| T5$^{XL}{Sup}$+$\\mathcal{R}{fix}$ | 16.21 | 85.78 | 86.99 | 84.62 |\n| T5$^{XL}_{Sup}$+LAPDOG | 17.76 | 85.99 | 87.35 | 84.70 |\n\nAccording to the results, LAPDOG demonstrates superior performance in terms of the METEOR score. This suggests that our framework may better capture some linguistic nuances, such as synonymy and word order, when compared with baseline models, since we optimized for F1, BLEU, and ROUGE-L. However, the BERT scores showed marginal improvement or even degradation. We attribute this to LAPDOG being optimized for traditional statistical metrics rather than neural model-based metrics. We believe that directly optimizing BERT scores in our framework could result in better performance, which could be a future direction of our work.\n\nDue to time constraints during the response period, human evaluations could not be conducted. We acknowledge the importance of such evaluations for a more rounded understanding of our model's capabilities and limitations. These will be included in the revision.\n\n**Q5: The paper contains punctual marks and formatting problems, including instances of bold font. Please conduct a thorough review to address these issues.**\n\nWe recognize the importance of clear presentation and have meticulously revised the manuscript to fix typos and formatting issues. Thank you for bringing this to our attention and we will polish the manuscript in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The work proposes a new differentiable probability distribution technique to remove the problem of using non-differentiable metric values (to train which story to retrieve for maximum information).\n\n2. A novel approach to generate persona-specific dialogues for a conversation.\n\n3. It solved the problem of getting stuck in local optima in a narrow range of candidates by candidate augmentation.", "reasons_to_reject": "No such potential reason to reject the paper.  \n\n> The model architecture needs to be discussed in detail, particularly the ROC story selection part. \n\n> The related work section should include a discussion of the work in the context of LLMs.", "questions_to_authors": "> Is the amount of stories you are taking in your dataset enough to cover every type of persona-specified dialogue?\n\n> Why don't you provide other evaluation metrics, BERT score, METEOR, and Human evaluation?", "paper_topic_and_main_contributions": "The paper introduces a model that utilizes context-based stories from an external story corpus to enhance the persona-specific dialogue system. The proposed algorithm comprises two sequential models. The initial model employs a vanilla generative approach with a transformer-based encoder-decoder using inputs such as the persona and user utterance. Subsequently, the second model adopts a retrieval-based generative architecture. It leverages the text generated by the first model to extract the top K retrieval stories. These stories, along with the persona and dialogue history, contribute to the generation of persona-specified text.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper contains punctual marks and formatting problems, including instances of bold font. Please conduct a thorough review to address these issues."}, "extra_text": {}}, {"review_id": "PvTIe4FMls", "reviewer": "Reviewer_Afr1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank you for your constructive comments. We address your concerns as follows.\n\n**Question: The baselines used in the experiments are not competitive enough. There is a lack of methods proposed by recent cutting-edge research, which makes the comparative experiments less convincing.**\n\nWe acknowledge the reviewer's point about the absence of cutting-edge baselines in our experiments. The unavailability of public code or too complex training stages for some of these cutting-edge approaches influenced our decision to use T5 as our generator architecture for a fair comparison. Our focus is to test the improvements coming specifically from our joint retriever-generator approach, embodied in LAPDOG, rather than the generator architecture itself.\n\nIt's important to note that LAPDOG's primary contribution lies in its ability to effectively fuse external story documents into persona dialogues. The framework is modular and not tied to a specific generator architecture, allowing it to be employed in conjunction with various state-of-the-art dialogue models.\n\nUnlike specialized architectures such as Persona-Adaptive Attention (PAA) [1], Coarse-to-Fine Persona-Aware Training Framework [2], and Bert-over-bert [3], LAPDOG offers a “plugin-like” functionality. This means it can be layered on top of these advanced methods, likely boosting their performance by providing additional contextual stories for persona profiles.\n\nIn future work, we plan to investigate how LAPDOG can synergize with these advanced architectures to substantiate its utility in enhancing the performance of personalized dialogue generation.\n\n**Reference**\n\n[1] Huang, Qiushi, Yu Zhang, Tom Ko, Xubo Liu, Bo Wu, Wenwu Wang, and H Tang. 2023. “Personalized Dialogue Generation With Persona-Adaptive Attention”. Proceedings of the AAAI Conference on Artificial Intelligence 37 (11):12916-23. https://doi.org/10.1609/aaai.v37i11.26518.\n\n[2] Li, Yunpeng, Yue Hu, Yajing Sun, Luxi Xing, Ping Guo, Yuqiang Xie, and Wei Peng. 2023. “Learning to Know Myself: A Coarse-to-Fine Persona-Aware Training Framework for Personalized Dialogue Generation”. Proceedings of the AAAI Conference on Artificial Intelligence 37 (11):13157-65. https://doi.org/10.1609/aaai.v37i11.26545.\n\n[3] Song, Haoyu, Yan Wang, Kaiyan Zhang, Wei-Nan Zhang, and Ting Liu. \"BoB: BERT over BERT for training persona-based dialogue models from limited personalized data.\" arXiv preprint arXiv:2106.06169 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors’ analysis of the difficulties in the personalized dialogue generation task is reasonable. The proposed method effectively addresses these issues through its well-designed approach.\n2. The paper is well-written, with a clear structure and easy-to-follow presentation of its contributions.\n3. The authors have committed to open-sourcing the code and pre-trained model.", "reasons_to_reject": "The baselines used in the experiments are not competitive enough. There is a lack of methods proposed by recent cutting-edge research, which makes the comparative experiments less convincing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on personalized dialogue generation. To address the issue of having too few samples in persona profiles, the authors propose LAPDOG, which leverages external story documents for persona dialogue generation. In the proposed method, the retriever and generator are jointly trained, eliminating the need for the proposed method to rely on annotated retrieval datasets. The authors also design a retrieval candidate augmentation during training to ensure diversity in the retrieval process.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on personalized dialogue generation by leveraging external story documents. In particular, a retriever and a generator are jointly trained, which eliminates the need for annotated retrieval datasets. The retriever is utilized to extract top-k relevant stories.

In general, the reviewers agree that the paper is well-motivated and the proposed model is well-designed. However, reviewers have pointed out that more recent cutting-edge baselines and traditional knowledge-grounding baselines should be included. Reviewer d41o also pointed out that the model architecture should be described in more detail and the paper has some minor formatting issues to fix.
Decision: Accept-Main