Venue: EMNLP 2023

# PK-ICR: Persona-Knowledge Interactive Multi-Context Retrieval for Grounded Dialogue

Minsik Oh1 Joosung Lee Jiwei Li Guoyin Wang

Please correspond to minisik@mskresearch.org.

###### Abstract

Identifying relevant persona or knowledge for conversational systems is critical to grounded dialogue response generation. However, each grounding has been mostly researched in isolation with more practical multi-context dialogue tasks introduced in recent works. We define Persona and Knowledge Dual Context Identification as the task to identify persona and knowledge jointly for a given dialogue, which could be of elevated importance in complex multi-context dialogue settings. We develop a novel grounding retrieval method that utilizes all contexts of dialogue simultaneously. Our method requires less computational power via utilizing neural QA retrieval models. We further introduce our novel null-positive rank test which measures ranking performance on semantically dissimilar samples (i.e. hard negatives) in relation to data augmentation.

## 1 Introduction

Effective conversation agents require external context as grounding information to enhance response generation. There has been much progress on each persona Majumder et al. (2020); Joshi et al. (2017); Shuster et al. (2018); Wu et al. (2019) and knowledge Li et al. (2022); Dinan et al. (2018); Zhao et al. (2020); Liu et al. (2021) grounded dialogue systems respectably. However, the combination of both and more unique contexts has not been studied, with limited interest in industry persona-based QA systems Byron et al. (2017); Ky and Joshi (2021).

Feng et al. (2020); Dinan et al. (2018); Moghe et al. (2018) have shown the importance of directly optimizing knowledge extraction in dialogue, while Zhang et al. (2018); Gu et al. (2021); Liu et al. (2020) have shown the importance of directly optimizing for concrete persona. We further argue that in practical settings, it is more realistic to assume the utility of multiple contexts, with an explicit use-case being travel assistance agent Jang et al. (2021).

Following the Knowledge Identification task in DIALKI Wu et al. (2021), we define _Persona and Knowledge Dual Context Identification_ as the task to identify persona and knowledge jointly for a given dialogue. The task is similar to personala based QA task in industry Byron et al. (2017); Ky and Joshi (2021), of creating a search engine based on persona, with exception of our study being in an interactive dialogue setting. We emphasize the specific interactions (Fig. 1) between persona, knowledge, and dialogue. We aim to formalize the nature of component-wise interactions via this research, resulting in enhanced multi-context retrieval methodology.

This separation of grounding retrieval tasks could be a particular benefit for multi-context dialogue, in which we can study complex context-wise interactions first, then apply the identified behavior as a sub-component of end-to-end systems. As a starting point, we re-purpose existing tasks and find that Question Answering (QA) is a good candidate Adolphs et al. (2021). This provides the

Figure 1: Illustration of dialogue component interactions regarding PK-ICR. In (1), dialogue is augmented with each persona to allow necessary interactions between persona and knowledge. (2) performs knowledge retrieval with (1). (3) performs precise persona scoring.



benefit of reduced computation and streamlined architecture by reusing powerful retrieval models previously developed for diverse tasks.

We develop a framework that exploits this relation, of which an interesting aspect is combining persona and dialogue1 as a form of component augmentation. This may be of further utility in complex systems as each pertains to attributes and actions of the human respectively. Interestingly, our suggested augmentation method creates positive and hard negative samples which could be applied to enhance retrieval (Appendix G). We introduce a novel evaluation methodology of the _Null-positive Rank Test_ (NRT) to quantify this trait.

Footnote 1: Persona-augmented Dialogue

Our contributions are summarized as follows.2

Footnote 2: Code for our experiments is available : [https://github.com/minsik-ai/PK-ICR](https://github.com/minsik-ai/PK-ICR)

1. **Persona and knowledge dual context retrieval methodology.** We enhance specific interactions between all components to successfully retrieve dialogue contexts. We achieve SOTA performance for both persona and knowledge retrieval.
2. **Framework for cross-task adaptation of dialogue context interactions.** We introduce a framework to benefit from existing performant retrieval models for complex dialogue grounding retrieval. Our zero-shot inference allows reduced computation (Table C1) and streamlined architecture.
3. **Evaluating the hard-negative trait of Persona-augmented Dialogue.** We augment dialogue with persona to form an enhanced retrieval input, in which we observe hard negative traits. We introduce a novel test to isolate this trait, applicable to scenarios where semantically dissimilar samples are produced via data augmentation.

## 2 Related Works

"Knowledge-enhanced text generation" Zhu et al. (2022); Yu et al. (2022) incorporates internal or external grounding contexts in tackling generative tasks such as dialogue or Q & A. Our research significantly contributes to the development of sophisticated "knowledge selection" for external knowledge systems. Our work is the first to model how to effectively select multiple distinct types of grounding contexts (persona & knowledge) for dialogue response generation.

To develop dialogue systems that rely on external knowledge information, open-domain dialogue datasets of Wizard of Wikipedia Dinan et al. (2018) and PersonaChat Zhang et al. (2018b) are most commonly studied. They consist of conversations that are grounded by Wikipedia knowledge and persona profile information, respectively. More recent datasets consist of conversations grounded by both persona & knowledge Jang et al. (2021) or blended scenarios Shuster et al. (2020); Smith et al. (2020). In line with the prior works, we treat persona and knowledge as different groundings with distinct characteristics and investigate semantic relations.

Integrating either persona or knowledge bases with dialogue agents in isolation has been actively studied. Zhang et al. (2018a); Majumder et al. (2020); Xu et al. (2020); Rashkin et al. (2019) for Persona, and Dinan et al. (2018); Zhao et al. (2020); Liu et al. (2021); Li et al. (2020); Ghazvininejad et al. (2017) for knowledge. Persona-only method prohibits elaboration with detailed knowledge. In contrast, relevant knowledge might depend on the persona of the user. We address the limitations by studying all dialogue component interactions.

_Knowledge Identification_Wu et al. (2021) task has been defined in recent papers stemming from knowledge-grounded dialogue. Our work aligns with the view in Wu et al. (2021) that context identification is a separately important task in an interactive dialogue setting, with similarities to open question answering Min et al. (2019); Chen et al. (2017) and industry persona-based QA systems Byron et al. (2017); Ky and Joshi (2021). Our research expands upon the Knowledge Identification task to specify persona & knowledge as dual contexts to be jointly retrieved from the dialogue.

## 3 Methodology

We maximize interactions between all components of a conversation turn for effective retrieval of dialogue groundings. Knowledge retrieval is a top-1 ranking task (Section 3.1), and persona retrieval is a point-wise scoring task with \(1\) or \(0\) true persona label (Section 3.2). We solve knowledge retrieval in a zero-shot manner, while we introduce _null-positive rank test_ to investigate the hard-negative traits of Persona-augmented Dialogue (Section 3.3). 3

Footnote 3: We note that each sequence - knowledge and persona retrieval - may be further optimized independently.

### Knowledge Retrieval

We introduce a novel adaptation of dialogue components as QA prompts (example in Fig. A1). This 

form is selected to infer relations between all inputs of the grounded dialogue and to replicate short question and descriptive answer pairs.

\[E:\{Q_{i},A_{j}\}=\{P_{i}+D,K_{j}\} \tag{1}\]

\(E\) is input to our model. \(Q_{i},A_{j},P_{i},K_{j}\) are specific QA candidates and persona & knowledge pairs. \(D\) is the dialogue for the pairs.

We then find the best knowledge for all pairs of \(i\) and \(j\) in a permutative manner and record the knowledge of the most aligned pair.

\[best_{i},best_{j}=\operatorname*{arg\,max}_{i\in 1...n,j\in 1..m}M_{q}\{ P_{i}+D,K_{j}\} \tag{2}\]

\[true_{j}=best_{j} \tag{3}\]

\(best_{i}\), \(best_{j}\) are indices from best-scoring persona / knowledge pair. \(true_{j}\) is the index of predicted knowledge \(K\). \(best_{i}\) is discarded. \(M_{q}\) is QA retrieval model for pair likelihood score. \(n\) / \(m\) is persona / knowledge count respectively.

### Persona Retrieval

Continuing from Section 3.1, we fine-tune the QA retrieval model using augmented persona and predicted true knowledge pairs only.4 We report that Persona-augmented Dialogue exhibits hard negative attributes (Section 3.3).

Footnote 4: This results in additional reduced computation of \(O(nm)\) to \(O(n)\) for both training and inference. In effect, this decreases negative pairs from 3M to 0.3M with 10x speedup.

\[E^{\prime}:\{Q_{i},A_{true}\}=\{P_{i}+D,K_{true_{j}}\} \tag{4}\]

\[M_{q}\xrightarrow{E^{\prime}_{train}}M_{f} \tag{5}\]

\(E^{\prime}\) is input to our model similar to \(E\), only difference being fixed true knowledge. \(E^{\prime}_{train}\) is data from a separate training set formulated in the same manner as \(E^{\prime}\) with labeled true knowledge. \(M_{f}\) is the fine-tuned model (Appendix D).

Next, we infer selected data pairs with \(M_{f}\) to obtain the persona likelihood score. We avoid retrieving unrelated persona via a threshold.

\[p_{i}=M_{f}\{P_{i}+D,K_{true_{j}}\} \tag{6}\]

\[true_{i}=\operatorname*{arg\,max}_{i}\begin{cases}p_{i},&\text{if $p_{ i}\geq p_{thres}$}\\ 0,&\text{otherwise}\\ &\text{for $i\in 1...n$}.\end{cases} \tag{7}\]

\(p_{i}\) is the likelihood score for \(P_{i}\). \(p_{thres}\) is the likelihood score threshold to remove persona that doesn't correspond to the dialogue turn. \(true_{i}\) is the index of the predicted true persona.

Finally, the retrieved grounding information is:

\[R:\{D,P,K\}=\{D,P_{true_{i}},K_{true_{j}}\} \tag{8}\]

\(R\) is the retrieved true persona & knowledge pair for the given dialogue turn.

### Null-positive Rank Test

We stress that fine-tuning model \(M_{q}\) with Persona-augmented Dialogue (\(P_{i}+D\)) to create model \(M_{f}\) is a specific choice. This is because the QA setup cannot be utilized without adjustments, due to the model scoring output skewing higher (Fig. E1). To analyze without inflated scores, we first interpret Persona-augmented Dialogue as hard negative sampling (Appendix G), in which the augmentation produces non-trivially hard-to-distinguish samples.5

Footnote 5: Note that positive samples are also created (Table A1). This augmentation is compatible with persona-only tasks.

To evaluate the above observation, we present a novel methodology of **null-positive rank test** to quantify the inherent difficulty of ranking \(P_{i}+D\) samples. Inspired by ranking metrics such as MRR, MAP, and NDCG, we utilize rank of a specific sample to compute model performance. This allows us to isolate the discriminative performance of the model corresponding to samples of interest, regardless of score output (Fig. 2, example in Table A1).

We designate null-positive6 (\(P_{o}\)) sample as a baseline for the model. We measure the following:

Figure 2: Null-positive Rank Test (NRT). \(P_{o}\), \(P_{pos}\), \(P_{neg_{i}}\) denote null-positive sample, positive and negative personas respectively. We omit augmentation \(+D\) in the figure for brevity. \(r_{min}=-1\) and \(r_{max}=+3\) in this figure. Arrows are possible positions for \(P_{o}\). Numbers on the rightmost side are the null-positive adjusted rank values, being \(0\) right below \(P_{pos}\) (example in Table A1).

 

_Can the model rank null-positive sample correctly in relation to non-trivially dissimilar augmented samples?_ The "non-triviality" metric which computes the average distance of null-positive sample's rank from the ideal rank7 is as follows:

Footnote 8: We verify that our cross-task adaptation of Q&A is significantly stronger than NLI, STS, and NSP (Table H1).

\[\neg T=\frac{\sum^{r_{\text{max}}}_{r=r_{\text{min}}}n_{r}*|r|}{\sum^{r_{\text {max}}}_{r=r_{\text{max}}}n_{r}} \tag{9}\]

Variants of the metric are in Appendix B. \(\neg T\) is non-triviality metric, with lower values of the metric meaning the model ranks better. \(n_{r}\) is the number of \(P_{o}\) samples with adjusted rank \(r\) (Fig. 2). We report "non-triviality" for each model \(M_{q}\), \(M_{f}\).

## 4 Experiment Setup

We utilize the Call For Customized Conversation Jang et al. (2021) dataset for which each conversation is built with both the user's persona and Wikipedia knowledge grounding information. We utilize multiple neural QA models trained on MS MARCO dataset Nguyen et al. (2016). More details in Appendix D.

## 5 Results

### Knowledge Retrieval

We experiment with ablations of dialogue / persona / knowledge interactions and find permutative evaluation of eq. 1 form yields best performance. Table 1 shows strong performance increase for our prompt input from dialogue-only model, confirming that all components of dialogue is important.9

Footnote 9: This is performance on persona retrieval free from score-related effect (Fig. E1).

### Persona Retrieval

Table 2 shows that fine-tuned \(P_{i}+D\) model has the best performance. However, we observe low performance for the non-fine-tuned \(P_{i}+D\) model. This is due to QA relationship of dialogue to true knowledge affecting the likelihood score (Fig. E1). Thus fine-tuning the model is a necessity to harness cross-domain adaptation.

### Null-positive Rank Test

To verify our observation of the effectiveness of \(P_{i}+D\), we perform null-positive rank test (Section 3.3). The performance of the model has increased in top-1 rank setting (0 threshold, 0-Acc)10 and all variants of non-triviality have improved for both models (Table 3, F1). We analyze sample count per rank (Fig. 3, F1).

Footnote 10: This is performance on persona retrieval free from score-related effect (Fig. E1).

## 6 Discussions and Conclusion

We introduce persona-knowledge dual context retrieval method PK-ICR in this paper. We perform QA-informed prompt-augmentations of data that

\begin{table}
\begin{tabular}{c c} \hline \hline Model Type & Accuracy (\%) \\ \hline Baseline & 65.06 \\ BERT-base & 11.78 \\ Proto-gen Saha et al. (2022) & 85.18 \\ \(D\) \& \(K_{j}\) & 79.26 \\ \(P_{i}\) \& \(K_{j}\) & 84.62 \\ \(P_{i}+D\) \& \(K_{j}\) & **94.69 (+29.63)** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Knowledge retrieval accuracy (cross-encoder) per asymmetric QA prompt. Zero-shot. Similar results for bi-encoder, which compares vectors (Table H1).

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Type & 0-Acc (\%) & \(\neg T^{2}\) & \(\neg T\) & \(\neg T_{+}\) & \(\neg T_{-}\) \\ \hline Z.S. & 79.30 & 1.84 & 1.02 & 1.04 & 0.62 \\ Ours & **86.81** & **1.66** & **0.97** & **0.96** & **0.56** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Null-positive rank test results for \(P_{i}+D\) & \(K_{true_{j}}\) cross-encoder models. Ours model is the fine-tuned variant, and Z.S. is Zero-Shot model. We report persona retrieval accuracy when \(p_{thres}=0\) (0-Acc) and variants of non-triviality (eq. 9, 12, 10, 11). Smaller non-triviality means superior ranking capability. Similar results for bi-encoder (Table F1).

 

successfully exploit the interactions between multiple dialogue components. We perform zero-shot top-1 knowledge retrieval and precise persona scoring. We present a novel evaluation method of null-positive rank test as to isolate the hard-negative effect of Persona-augmented Dialogue. We obtain SOTA results on both retrieval tasks of the Call For Customized Conversation benchmark and report the alignment of the non-triviality metric with threshold-free performance. With our research, we hope to stimulate readers to model dialogue context as an interactive whole of multiple components.

As the NLP community aims to tackle more complex dialogue systems, our methods may be further enhanced by sophisticated grounding contexts and interactions present in dialogue. Considering all components of dialogue, being persona, knowledge, and dialogue in a travel agent scenario, is crucial to obtain each grounding context required for accurate responses. We suggest two future directions for dialogue systems.

1. One possible future direction is incorporating different forms of grounding such as persona/knowledge summaries, web searches, Wikipedia documents, and extracted sentences Wu et al. (2021) for multi-context interactions.
2. Another would be extending our methodology to more sophisticated dialogue settings such as long-term memory Bae et al. (2022) or mutual persona Liu et al. (2020).

Our persona-aware dialogue augmentation (1 in Figure 1) is a form of modeling the human behavior in regards to Turing test Li et al. (2016); Vinyals and Le (2015). We isolate effectiveness of our augmentation with successful Null-Positive Rank Test (Section 3.3). We suggest two future directions regarding persona-dialogue augmentation.

1. One direction to pursue would be emphasizing different interactions, such as augmenting persona and knowledge first, or complex forms such as providing negative context in the augmentation themselves. Whether human-like behavioral modeling works better, or computationally tractable methods could be developed, is another interesting question.
2. Another direction to pursue would be advanced prompting, where specifying persona or knowledge via pre/post-fix prompts (i.e. "I'm thinking 'I like football'') provides more indicator information regarding each grounding and may improve performance.

Finally, our novel Null-positive Rank Test (NRT, Section 3.3) is widely applicable to information retrieval and ranking models. We only trained the models with positive and negative dialogue augmentations, but interestingly, we report that the model improves ranking correctness for "neutral" (non-augmented) dialogue. This evaluation method could be directly used to compute whether the model obtains sophisticated ranking capabilities. This would be especially relevant in the context of data augmentation or other ranking tasks, including personalized search / QA. We recommend performing our test for future works.

## 7 Limitations

Our cross-task adaptation of dialogue grounding retrieval to QA task is limited in terms of the target task and our prompt construction. In addition, retrieval models informed by inductive bias for multi-context scenarios could further improve our methodology.

We specifically study multi-context interactions and retrieval in dialogues, which is a relevant and novel problem for advancing broadly capable dialogue systems. As an extension to our research, future work could also report on modeling downstream generation tasks based on grounding interactions.

Figure 3: Analysis of null-positive rank data for \(P_{i}+D\)& \(K_{true_{j}}\) cross-encoder model. Delta value is the change between Zero-shot model and Ours model in terms of sample count (left axis). Ratio value is delta value divided by sample count for Zero-shot, in % (right axis). We report movements of delta in correct directions for rank \(-1\), \(0\) and ranks with long distance \(3\), \(4\). Similar results for bi-encoder (Fig. F1).

 



## Acknowledgement

We sincerely thank the ARR reviewers for their valuable insights.

## References

* Adolphs et al. (2021) Leonard Adolphs, Kurt Shuster, Jack Urbanek, Arthur Szlam, and Jason Weston. 2021. Reason first, then respond: Modular generation for knowledge-infused dialogue.
* Bae et al. (2022) Sanghwan Bae, Donghyun Kwak, Soyoung Kang, Min Young Lee, Sungdong Kim, Yuin Jeong, Hyeri Kim, Sang-Woo Lee, Woomyoung Park, and Nako Sung. 2022. Keep me updated! memory management in long-term conversations.
* Byron et al. (2017) Donna K. Byron, Alexander Pikovsky, David G. Radley, and Timothy P. Winkler. 2017. Answering questions via a persona-based natural language processing (nlp) system.
* Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Dinan et al. (2018) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. _arXiv preprint arXiv:1811.01241_.
* Feng et al. (2020) Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document-grounded dialogue dataset. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8118-8128, Online. Association for Computational Linguistics.
* Ghazvininejad et al. (2017) Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, William B. Dolan, Jianfeng Gao, Wen tau Yih, and Michel Galley. 2017. A knowledge-grounded neural conversation model. _AAAI_, abs/1702.01932.
* Gu et al. (2021) Jia-Chen Gu, Zhenhua Ling, Yu Wu, Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2021. Detecting speaker personas from conversational texts. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1126-1136, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Hofstatter et al. (2021) Sebastian Hofstatter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. In _Proc. of SIGIR_.
* Jang et al. (2021) Yoonna Jang, Jungwoo Lim, Yunu Hur, Dongsuk Oh, Suyhume Son, Yeonsoe Lee, Donghoon Shin, Seungryong Kim, and Heuisceok Lim. 2021. Call for customized conversation: Customized conversation grounding persona and knowledge.
* Joshi et al. (2017) Chaitanya K. Joshi, Fei Mi, and Boi Faltings. 2017. Personalization in goal-oriented dialog.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Dani Chen, and Wen tau Yih. 2020. Dense passage retrieval for open-domain question answering.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Ky and Joshi (2021) Piseth Ky and Anila Joshi. 2021. [link].
* Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model.
* Li et al. (2020) Lin-Xiao Li, Can Xu, Wei Wu, Yufan Zhao, Xueliang Zhao, and Chongyang Tao. 2020. Zero-resource knowledge-grounded dialogue generation. _NeurIPS_, abs/2008.12918.
* Li et al. (2022) Yu Li, Baolin Peng, Yelong Shen, Yi Mao, Lars Linden, Zhou Yu, and Jianfeng Gao. 2022. Knowledge-grounded dialogue generation with a unified knowledge representation.
* Lin et al. (2021) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021a. Pretrained transformers for text ranking: Bert and beyond.
* Lin et al. (2021) Sheng-Chib Lin, Jheng-Hong Yang, and Jimmy Lin. 2021b. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_, pages 163-173, Online. Association for Computational Linguistics.
* Liu et al. (2020) Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. 2020. You impress me: Dialogue generation via mutual persona perception. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1417-1427, Online. Association for Computational Linguistics.
* Liu et al. (2021) Shilei Liu, Xiaofeng Zhao, Bochao Li, Feiliang Ren, Longhui Zhang, and Shujuan Yin. 2021. A three-stage learning framework for low-resource knowledge-grounded dialogue generation. In _Conference on Empirical Methods in Natural Language Processing_

Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, dense, and attentional representations for text retrieval.
* Majumder et al. (2020) Bodhisattwa Prasad Majumder, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Julian McAuley. 2020. Like hiking? you probably enjoy nature: Persona-grounded dialog with commonsense expansions. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9194-9206, Online. Association for Computational Linguistics.
* Min et al. (2019) Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Knowledge guided text retrieval and reading for open domain question answering.
* Moghe et al. (2018) Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M Khapra. 2018. Towards exploiting background knowledge for building conversation systems. _arXiv preprint arXiv:1809.08205_.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset.
* Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with bert.
* Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.
* Saha et al. (2022) Sougata Saha, Souvik Das, and Rohini Srihari. 2022. Proto-gen: An end-to-end neural generator for persona and knowledge grounded response generation. In _Proceedings of the 1st Workshop on Customized Chat Grounding Persona and Knowledge_, pages 9-14, Gyeongju, Republic of Korea. Association for Computational Linguistics.
* Shuster et al. (2018) Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2018. Image chat: Engaging grounded conversations.
* Shuster et al. (2020) Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason Weston. 2020. The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2453-2470, Online. Association for Computational Linguistics.
* Smith et al. (2020) Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can you put it all together: Evaluating conversational agents' ability to blend skills. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2021-2030, Online. Association for Computational Linguistics.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model.
* Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minimlm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.
* Wu et al. (2019) Bowen Wu, Mengyuan Li, Zongsheng Wang, Yiftu Chen, Derek Wong, Qihang Feng, Junhong Huang, and Baoxun Wang. 2019. Guiding variational response generator to exploit persona.
* Wu et al. (2021) Zeqiu Wu, Bo-Ru Lu, Hannaneh Hajishirzi, and Mari Ostendorf. 2021. DIALKI: Knowledge identification in conversational systems through dialogue-document contextualization. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1852-1863, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval.
* Xu et al. (2020) Minghong Xu, Piji Li, Haoran Yang, Pengjie Ren, Zhaochun Ren, Zhumin Chen, and Jun Ma. 2020. A neural topical expansion framework for unstructured persona-oriented dialogue generation.
* Yu et al. (2022) Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge-enhanced text generation. _ACM Comput. Surv._, 54(11s).
* Zhang et al. (2018a) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a. Personalizing dialogue agents: I have a dog, do you have pets too? In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
* Zhang et al. (2018b) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018b. Personalizing dialogue agents: I have a dog, do you have pets too? _arXiv preprint arXiv:1801.07243_.
* Zhao et al. (2020) Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020. Low-resource knowledge-grounded dialogue generation. _ICLR_, abs/2002.10348.
* Zhu et al. (2022) Chenguang Zhu, Yichong Xu, Xiang Ren, Bill Lin, Meng Jiang, and Wenhao Yu. 2022. Knowledge-augmented methods for natural language processing. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_, pages 12-20 

Samples

### QA Cross-Task Adaptation Prompt Construction

[1.5cm] Question : "{I want to visit Seven Wonders of the Ancient World.} {Wow, what is this?}" Answer : "{The Great Pyramid of Giza ... of the Seven Wonders of the Ancient World, ...}"

Figure A1: Resulting QA cross-task adaptation prompt of persona & knowledge pair (eq. 1). Question form is ”{persona} {dialogue}” while answer is ”{knowledge}”.

 

Computational Efficiency

Saha et al. (2022) utilizes 2 BART models for input and persona & knowledge groundings.10 In contrast, PK-ICR utilizes 1 MiniLM (Wang et al., 2020) cross-encoder retrieval model for computing similarity scores. Saha et al. (2022) concatenates all groundings as one input, while PK-ICR groups them into pairs. Our zero-shot knowledge retrieval (Section 3.1) doesn't require any training, while Saha et al. (2022) trains for a maximum of 15 epochs.

Footnote 10: \(\mathrm{BART}\) encoders and 1 decoder for generation (may positively affect retrieval).

## Appendix D Experiment Setup

We utilize Call For Customized Conversation (Jang et al., 2021) dataset for evaluation and fine-tuning, which has 10 knowledge candidates and 5 persona candidates per dialogue. We utilize 12 layer MiniLM (Wang et al., 2020) (33M params) cross-encoder trained on MS MARCO11(Nguyen et al., 2016) from Sentence-BERT library (Reimers and Gurevych, 2019) and DistillBERT (66M params) TAS-B (Hofstatter et al., 2021) bi-encoder model trained on same data12. This data is for semantic search, with trained models evaluating short questions and long passages together. In addition, we report baseline performance on DPR (Karpukhin et al., 2020) models13(110M params) trained on NQ (Kwiatkowski et al., 2019) dataset14, with a dummy short segment of "Title", and treating Knowledge as long answer segment. For persona search (eq. 5, 7), we fine-tune for 2 epochs, 32 batch size, and sigmoid activation function with Binary Cross Entropy (cross-encoder) / Cosine Similarity (bi-encoder) Loss with \(p_{\textit{thres}}=0.5\). We list the official evaluation results on the test data. For MobileBERT (25M params) and BERT-base (110M params), we evaluate with Next Sentence Prediction task. We experiment with DistillRoBERTa (82M params) STS15 and NLI16 cross-encoder models. We work with RTX 3090 NVIDIA GPU.

Footnote 11: MRR@10 on MS MARCO Dev Set: 39.02

Footnote 12: MRR@10 on MS MARCO Dev Set: 34.43

Footnote 13: Bi-encoder model with separate question and answer encoders.

Footnote 14: NQ test set Accuracy@20: 78.4, Accuracy@100: 85.4

Footnote 15: STSbenchmark test performance: 87.92

Footnote 16: Accuracy on MNLI mismatched set: 83.98, we compare with entailment score.
 



## Appendix E Persona Retrieval Threshold Experiments

## Appendix F Null-Positive Rank Test for Bi-Encoder

Figure 1: Analysis of null-positive rank data for \(P_{i}+D\) & \(K_{true_{j}}\) bi-encoder model. Delta value is the change between Zero-shot model and Ours model in terms of sample count (left axis). Ratio value is delta value divided by sample count for Zero-shot, in % (right axis). We report large movements of delta in correct directions for rank 0 and ranks with long distance \(3\), \(4\).



Background: Ranking and Hard Negative Sampling

Text ranking is a task to generate an ordered list of texts in response to a query (Lin et al., 2021). It is a core task of information retrieval (IR) where you obtain a ranked list of samples ordered by estimated relevance to the query. We introduce widely accepted neural approaches, 'cross-encoder' and 'bi-encoder'. We will also describe 'hard negative sampling', a data-centric approach to improve retrieval models.

For cross-encoder (Nogueira et al., 2019), query and a single sample are concatenated by '[SEP]' token as an input to the model, resulting in a relevance score (FFN output of the '[CLS]' token representation) for the specific sample. We note that this setup is similar to sentence-wise classification settings presented in Devlin et al. (2018). Then, the samples are ordered by relevance score to produce the final ranked list.

For bi-encoder (Reimers and Gurevych, 2019), we generate dense vectors (sentence embeddings) per each query and each sample. This is obtained via '[CLS]' token representation of a specially fine-tuned model17, with a single query or sample input. The representations are then compared as pairs via cosine-similarity or dot-product to compute relevance scores. While original bi-encoder setup computes sentence-wise similarity (STS), we utilize models fine-tuned on QA data (Appendix D).

Footnote 17: Supervised learning via Siamese Network.

Hard negative sampling (also known as hard negative mining) is a technique to obtain specific samples (hard negatives) that are difficult to distinguish from positive samples, yet have a different label. The hard negative samples are then incorporated during model fine-tuning to improve model capabilities. For example, in the context of ranking, non-relevant texts scoring high by how many keywords match (Xiong et al., 2020) may be considered hard negatives. Xiong et al. (2020); Luan et al. (2021); Lin et al. (2021) have demonstrated that hard negative sampling improves ranking models considerably.

## Appendix H Detailed Results

More experiments are listed here. Our bi-encoder and cross-task experiments confirm our findings in Section 5. Explanation of the models in Appendix G.

 



\begin{table}
\begin{tabular}{c c} \hline \hline Model Type & Accuracy (\%) \\ \hline Baseline & 86.86 \\ MobileBERT & 86.86 \\ BERT-base & 71.82 \\ Proto-gen Saha et al. (2022) & 87.75 \\ \(D\) \& \(P_{i}\) (cross-encoder) & 86.78 \\ \hline \(P_{i}\) \& \(K_{true_{j}}\) (DPR) & 75.54 \\ \(P_{i}\) \& \(K_{true_{j}}\) (bi-encoder) & 78.64 \\ \(P_{i}\) \& \(K_{true_{j}}\) (cross-encoder) & 86.75 \\ \(P_{i}\) \& \(K_{true_{j}}\) (cross-encoder, fine-tuned) & 89.12 \\ \hline \(P_{i}+D\) \& \(K_{true_{j}}\) (DPR) & 74.76 \\ \(P_{i}+D\) \& \(K_{true_{j}}\) (bi-encoder) & 77.90 \\ \(P_{i}+D\) \& \(K_{true_{j}}\) (cross-encoder) & 83.83 \\ \(P_{i}+D\) \& \(K_{true_{j}}\) (bi-encoder, fine-tuned) & **85.55** \\ \(P_{i}+D\) \& \(K_{true_{j}}\) (cross-encoder, fine-tuned) & **91.57** \\ \hline \hline \end{tabular}
\end{table}
Table H2: Persona retrieval results, models are zero-shot unless fine-tuned. We report persona retrieval accuracy per asymmetric QA prompt. We do not fine-tune DPR model due to implementation limitations. D, K, P each refer to dialogue, knowledge and persona.



Retrieval Output Samples

We list some of the retrieved outputs with our best model in Table I1.

\begin{tabular}{l l l} \hline \hline dialogue \(D\) & persona \(P_{true}\) & knowledge \(K_{true}\) \\ \hline I think I've been there before but I don't remember the name of this place. & I am fond of Modernist architecture. & The Casa de les Punexes or Casa Terradas is a building designed by the Modernista architectet Josep Puig I Cadafalch. Located in the intersection between the streets of Rossello, Bruc and the Avinguda Diagonal in the Barcelona Eixample area. \\ \hline How much this railway line costed in those times? & I love railway. & Because of the difficult physical conditions of the route and state of technology, the construction was renowned as an international engineering achievement, one that cost US$8 million and the lives of an estimated 5,000 to 10,000 workers. \\ \hline Who built this rail line? & I love railway. & The line was built by the United States and the principal incentive was the vast increase in passenger and freight traffic from eastern USA to California following the 1849 California Gold Rush. \\ \hline What’s the highest point in the Mulanje Massif? & I like to climbing up the elevations on my neighborhood to take a look around. \\ \hline Who was the first explorer to find this mountain? & I have fantasies of being a Livingstone type explorer. & The first European to report seeing the Massif was David Livingstone in 1859, but archeological investigation reveals evidence of human visits to the Massif from the Stone Age onwards. \\ \hline Now I remember, can you tell me some characteristics of this channel? & N / A & And may be the oldest canal in England that is still in use. It is usually thought to have been built around AD 120 by the Romans, but there is no consensus among authors. \\ \hline \hline \end{tabular}