UNIWIZ: A Unified Large Language Model Orchestrated Wizard for Safe Knowledge Grounded Conversations (2024)

Abstract: Warning: This paper contains disturbing language. Large Language Models (LLMs) have made significant progress in integrating safety and knowledge alignment. However, adversarial actors can manipulate these models into generating unsafe responses, and excessive safety alignment can lead to unintended hallucinations. To address these challenges, we introduce UNIWIZ, a novel 2-step data orchestration framework that unifies safety and knowledge data generation. We propose a "safetypriming" method to generate synthetic safety data and overcome safety bottlenecks. We also inject relevant knowledge into conversations by retrieving factual information from curated sources. UNIWIZ dataset consists of 17, 638 quality-controlled conversations and 10, 000 augmented preference data. Pretrained models fine-tuned on UNIWIZ show improvements across various metrics and outperform stateof-the-art instruction-tuned models trained on much larger datasets.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 1749–1762 August 11-16, 2024 ©2024 Association for Computational Linguistics
Large Language Models (LLMs) have made significant progress in integrating safety and knowledge alignment. However, adversarial actors can manipulate these models into generating unsafe responses, and excessive safety alignment can lead to unintended hallucinations. To address these challenges, we introduce UNIWIZ, a novel 2-step data orchestration framework that unifies safety and knowledge data generation. We propose a "safetypriming" method to generate synthetic safety data and overcome safety bottlenecks. We also inject relevant knowledge into conversations by retrieving factual information from curated sources. UNIWIZ dataset consists of 17, 638 quality-controlled conversations and 10, 000 augmented preference data. Pretrained models fine-tuned on UNIWIZ show improvements across various metrics and outperform stateof-the-art instruction-tuned models trained on much larger datasets.

heading: 1 Introduction
text: Large Language Models (LLMs) are designed to adhere to safety guidelines and possess extensive knowledge acquired during fine-tuning. Significant progress has been made in integrating safety (Ge et al., 2023; Perez et al., 2022) and knowledge alignment in LLMs(Zha et al., 2023). However, jail-breakers often employ innovative strategies to deceive or manipulate these models into generating unsafe or hallucinated responses(Chao et al., 2023). Conversely, an excessive focus on safety alignment can result in unintended hallucinations (Huang et al., 2023) or rejection of valid inputs.
Previous research efforts have primarily concentrated on developing conversational datasets grounded in knowledge to enhance factual accuracy (Dziri et al., 2022; Kim et al., 2023). Additionally,
specific datasets have focused explicitly on safety scenarios(Kim et al., 2022), particularly at individual conversation turns. However, these datasets alone can’t solve both of these issues. More recently, there has been a rise in the creation of extremely large datasets(Kim et al., 2023; Ding et al., 2023), which are synthetically generated using powerful and proprietary large language models. This entails creating a new resource that will unify knowledge and safety guidance. To understand the impact of these issues, we analyze two conversations between a user and a safety fine-tuned LLM COSMO with 11B parameters (Figure 2). In the first conversation, the user expresses ill intentions. Initially, the LLM tries to persuade the user and provides counter-arguments against the intended harmful actions. However, as the conversation progresses, the quality of these counter-arguments deteriorates, and the LLM endorses unsafe actions. In the second conversation, when the user mentions an entity with negative associations, the LLM avoids sharing excessive information. However, if the user continues to inquire further, the LLM generates a hallucinated response.
We introduce a new 2-step data orchestration framework called UNIWIZ to address these chal-
1749
lenges and unify the generation of safety and knowledge data using an powerful LLM (Claude-2.1). First, we propose a "safety-priming" method to generate synthetic safety data and overcome the safety bottleneck. In this approach, we leverage a powerful language model(Claude-2.1) to generate a self-chat conversation between the LLM and a user, where the LLM provides a counterargument for the user’s questionable utterance and ill intention. Additionally, we instruct the LLM to generate user utterances that maintain their original stance. This iterative counter-argumentation, focused on the safety issue, ensures the creation of practical data. It allows the LLM to provide sufficient reasoning to counteract a user’s persistent attempts to retrieve harmful content from the LLM.
Next, we extend the "safety-primed" conversation by injecting relevant knowledge that is related to the entities in the conversation. We begin by identifying relevant entities from the conversation and retrieving factual information from a curated knowledge source. Using this knowledge source and the previous conversation, we prompt an LLM(Claude-2.1) to continue the conversation for a few more turns. This approach ensures that the generated data includes turns related to potentially unfavorable scenarios while avoiding harmful content.
Using our 2-step data orchestration framework, we create UNIWIZ, a safety-primed knowledge-grounded conversational dataset comprising 17, 638 quality-controlled conversations. Publicly available state-of-the-art pre-trained models like Mistral 7B show improvement across all metrics when fine-tuned on UNIWIZ(called UniWiz-7B-v0.1). We also create an augmented version of the Antropic-HH dataset, consisting of 10, 000 preference data. Using the augmented preference data, we further alignment-tune UniWiz-7Bv0.1 using Direct Preference Optimization(DPO);
the results of our final model UniWiz-7B-v0.2 is very close to the publicly available instructiontuned counterpart of Mistral 7B, which is trained on an unknown amount of undisclosed data. It outperforms Zephyr-7b-beta1 by a large margin. Our models are available in HuggingFace 2 3 and the code and data will be released here 4.

heading: 2 Related Work
text: Dialog/LLM Safety: As conversational AI systems grow more powerful and prevalent, ensuring safe and beneficial model behavior becomes increasingly crucial (Xu et al., 2021; Soleimani et al., 2023). Dialog systems must avoid toxic responses, biased assumptions, and potential harm to users; even the most capable systems like GPT-3 exhibit failures around racism, sexism, and misinformation without interventions (Bender et al., 2021). Techniques to enhance dialog safety include human-inthe-loop methods like oversight, flagging issues during deployment, and post-deployment model updates (Glaese et al., 2022; Mehrabi et al., 2022). Some work modifies model training processes via augmented data (Das and Srihari, 2024) and safety objectives built into the loss. Robust progress indicators and transparency around model limitations are critical for developing truly safe systems (Henderson et al., 2018). Ongoing priorities involve handling tradeoffs with performance, embracing collaborative solutions between institutions, and implementing standardized safety practices into norms and policies around AI ethics (Mazeika et al., 2024).
Knowledge Grounding: Incorporating external knowledge into large language models (LLMs)
1Zephyr-7b-β is trained on HuggingFaceH4/ultrachat_200k, which is 11.76x larger than our SFT dataset.
2https://huggingface.co/proto-llm/uniwiz-7B-v0.1 3https://huggingface.co/proto-llm/uniwiz-7B-v0.2 4https://github.com/souvikdgp16/protoai_uniwiz
is an emerging area of research. While models like GPT-3 have impressive conversational abilities, they lack effective mechanisms to leverage facts and external data to ground their responses. Recent work has focused on augmenting LLMs with retrieved knowledge to produce more consistent and accurate dialog (Lewis et al., 2021; Asai et al., 2023). Approaches include concatenating relevant text or knowledge to the dialog history input, training intermediate knowledge selection or ranking models, and joint training of retriever and LLM generator components. Researchers have developed datasets that require reasoning over external resources, such as wiki articles(Moon et al., 2019). Challenges in this field include scaling to large domain coverage, efficiently encoding knowledge source structure, and avoiding generic or contradictory responses (Chen et al., 2019). There are also initiatives to incorporate structured commonsense knowledge directly into model parameters by training on knowledge bases like ConceptNet(Wang et al., 2024). Ongoing research focuses on modeling knowledge at scale, integrating external knowledge with model architectures, and enabling conversational abilities beyond just providing factual information (Kim et al., 2023; Jang et al., 2023; Chae et al., 2023).
LLM-based Synthetic Data Creation: Synthetic data generation using large language models (LLMs) has emerged as a promising technique to create high-quality training data for downstream tasks (Bao et al., 2023). By prompting LLMs like GPT-3/3.5 to generate various textual outputs, researchers have augmented datasets across domains like question answering (Puri et al., 2020), summarization (Liu et al., 2022), and dialogue (Kim et al., 2023). LLMs can produce diverse, naturalistic samples with low marginal cost once deployed. Challenges include controlling generated output attributes, filtering samples for quality and coherence, and balancing synthetic with real data (Xu et al., 2023). The reliability of ground truth labels assigned to model-generated samples remains an open question. As LLMs continue rapidly advancing in scale and capabilities, effectively leveraging them for data augmentation emerges as a major opportunity. Key directions involve frameworks to produce tailored benchmarks on demand (Laskar et al., 2023), analyzing model-induced biases or artifacts (Liang et al., 2023), and applications to low-resource domains lacking large training sets (Liu et al., 2022).

heading: 3 Data Orchestrastion Framework

heading: 3.1 Safety Priming
text: Inspired by recent works on red teaming (Perez et al., 2022), we proposed a new Large Language Model(LLM) safety data distillation technique where we provide an LLM with a socially questionable utterance and a negative intention derived from a commonsense knowledge and ask Claude2.1 to generate a conversation between an User and AI. We use the following prompt:
The socially questionable utterance is sampled from ProsocialDialog(Kim et al., 2022), which is conversational dataset consisting of adversarial utterances and need some form of intervention. We sample about ∼ 6.5k of such utterances. The intentions are sampled from a large-scale commonsense knowledge graph Atomic10x (West et al., 2022), which is represented by symbolic triples describing two events, denoted as the <Head> and <Tail>, and the <Relation> between those two events. We then convert the tuples into sentences by prompting an LLM 5. All the sentences are passed through the VADER sentiment classifier, and sentences with a neg label and a score greater than 90% are used. At the end, we have ∼ 10k of such sentences. We provide additional instructions in the prompt so that the User will try to convince the AI why they came up with questionable content and AI will continuously provide safe arguments countering User’s intentions. We collect about 4.35k conversations in this phase6.

heading: 3.2 Entity and Knowledge Selection
text: Knowledge is curated from the Wizard of Wikipedia dataset(Dinan et al., 2019), which has entities and several associated knowledge snippets. We create a BM25 index around ∼ 130k entities along with its knowledge snippets. We keep the 5 knowledge snippets for each entity in the same sequence as present in the Wizard of Wikipedia dataset. Then, we extract k entities from the safety primed conversation using the
5Given a tuple <Head, Tail, Relation> convert it into a sentence.
6LLM rejections were filtered out.
YAKE algorithm(Campos et al., 2020). We then use the entities to form a OR query in this fashion: e1 OR e2 OR ... OR ek. We query the previously created BM25 index and retrieve the top 5 entities with its corresponding knowledge facts.

heading: 3.3 Knowledge Injected Extended Chat
text: The main goal is to extend the safety-primed conversation in a direction relevant to the already occurred utterances between the User and AI. We again prompt an LLM with the safety-primed conversation, the retrieved entity, and the knowledge snippet associated with the entity. We use the following prompt:
For each safety-primed conversation we create ∼ 5 extended conversations. Total number of processed conversations collected in this phase were 18, 5727.

heading: 3.4 Quality Control
text: We perform quality checks on each LLM-generated conversation using the following metrics: • Human annotated data trained classifier: We
annotated 1500 conversations using two in-house 7we filter out the conversations with generation errors,
incorrect turn, etc.
human annotators. The annotators were asked to rate the conversation on a scale of 1-5, and they were told to base their judgment on these dimensions: (1) Counter argument quality of AI for promoting safety, (2) Adherence to knowledge snippet provided in the extended conversation (3) Coherence and grammatical correctness. 8 The inter-annotator agreement κ was 0.52, suggesting moderate agreement. Using 1200 data points from the annotated data, we fine-tuned Mistral7B instruct-v0.29 and tested the remaining data. The test accuracy was 68.87%. We then used this classifier to rate all the generated conversations. • LLM-based evaluation: Using the same dimensions used to obtain the human annotations, we ask Claude-2.1 to rate a conversation between 1-510. • Safety argument quality evaluation: We trained a classifier using the Argument Quality Ranking (Gretz et al., 2019) dataset, which contains 30, 500 data of arguments and the arguments are scored across different metrics. We used the 21, 500 train split to train a RoBERTa(large) classifier11. We trained a binary-class sentence pair classifier, and the tokens were arranged in this fashion: [CLS]<topic>[SEP]<argument>[EOS]. The positive label denotes the argument qual-
8Individual scores are averaged and rounded to the nearest integer.
9hyperparamter details in §A.2 10prompt details in §A.1 11hyperparamter details in §A.3
ity is good. The test accuracy was 70.29%. We then used this classifier to score each safety argument put forward by the AI, in each conversation. For each utterance, we get the turn-level argument quality score by predicting the probability of a positive class. The tokens arranged in the following way: [CLS]<user_utterance>[SEP] <ai_utterance>[EOS]. The final score for an utterance is the mean of all the pair-wise scores. We run all the above evaluations on each conversation and follow two selection conditions: (1) The combined score from the human-annotated datatrained classifier and LLM-based evaluation should be greater than 7 out of 10. (2) Each counter utterance safety argument quality score should be greater than 0.8 in safety primed conversation.

heading: 4 Dataset Analysis

heading: 4.1 Quality Evaluation Results
text: Figure 4(a) shows the combined ratings from the Claude-2.1-based rating and our Mistral-based conversation classifier on the conversations generated using our data orchestration framework. Our data creation strategy is quite effective, with only 5.3% of the data rated less than 8. Most of the conversations, i.e., 47.1% are rated 9. Figure 4(b) shows the distribution of argumentation score for each conversation’s safety priming turns using our argument quality classifier. Most of the conversations (84.2%) are scored greater than 0.8, which entails the effectiveness of our safety priming prompting. Finally, the number of conversations that match both of our criteria i.e., a combined rating >= 8 and overall argumentation score > 0.8 is 17, 638.

heading: 4.2 Dataset Statistics
text: In this work, we release a corpus of qualitycontrolled 17, 638 conversations. As compared to previous datasets, as shown in Table 1 the conversations in our datasets are knowledge-grounded, safety-grounded using the safety priming at the very beginning of the conversation, and finally, safety-checked using the rating modules. The train/validation/test split is shown in Table 2.

heading: 5 Training Pipeline

heading: 5.1 Supervised Fine-tuning
text: Standard fine-tuning typically requires significant computational resources and relies on large, highquality datasets. However, due to the scarcity of reliable multi-turn chat datasets, it is important to employ more efficient methods that require less data. Parameter-efficient tuning methods (Li and Liang, 2021; Hu et al., 2021) can address this issue by maximizing the utilization of available data and minimizing resource requirements.
In this regard, we use Quantized Low-Rank Adaptation (QLoRA) (Dettmers et al., 2023) to fine-tune Mistral models. QLoRA employs 4-bit NormalFloat (NF4) and double quantization, as well as paged optimizers, to reduce memory usage without compromising performance. LoRA augments a linear projection through an additional factorized projection. Given a projection XW = Y , where X ∈ Rb×h, W ∈ Rh×o LoRA computes:
Y = XW + sXL1L2 (1)
Where L1 ∈ Rh×r and L2 ∈ Rr×o, and s is a scalar. Based on this information, QLoRA is defined as:
YBF16 = XBF16doubleDequant(cFP321 , c k−bit 2 ,W NF4)
+ sXBF16LBF161 L BF16 2 (2)
where doubleDequant(.) is defined as:
doubleDequant(cFP321 , c k−bit 2 ,W NF4)
= dequant(dequant(cFP321 , c k−bit 2 ),W NF4) = WBF16 (3)
QLoRA has one storage data type (NF4) and a computation data type (BF16). cFP321 and c k−bit 2 are quantization constants. BF16, FP32 denotes the brain floating point and floating point counterparts of the parameters. doubleDequant(.) dequantizes the storage data type to the computation data type to perform the forward and backward pass. We use all the standard configurations mentioned in (Dettmers et al., 2023).

heading: 5.2 Preference Data Annotation
text: We improve our model by using preference alignment learning. We ensure we have a reliable source of preference data by enhancing the Anthropic-hhrlhf dataset(Bai et al., 2022). This dataset contains prompts x and their chosen yc and rejected yr responses. Previous research(Zhou et al., 2023) has shown that the quality of preference data is more important than the quantity. So, we randomly selected 30,000 data points from the train set and evaluated them using our Mistral-based rating classifier. From this evaluation, we randomly choose 10,000 data points with a score greater than 4 out of 5 to train the preference model.
To increase the diversity of the responses, we use the 10,000 selected data points to prompt our SFT model and generate two more responses. We use greedy decoding with temperatures of 0.9 and 0.5, respectively. Then, we rank the candidates using an LLM to identify the newly chosen response (with the highest score) and the rejected response (with the lowest score).

heading: 5.3 Preference Alignment
text: We use Direct Preference Optimization (Rafailov et al., 2023) to optimize our SFT model. This allows the model to align with human/LLM preferences without requiring explicit reward modeling or reinforcement learning. This approach provides a simpler and more efficient alternative to RLHF. The maximum likelihood objective for a parametrized policy for our SFT model πSFT is:
LDPO(πθ, πSFT) = −E(x,yc,yr)[ logσ ( βlog
πθ(yc|x) πSFT(yc|x) − βlog πθ(yr|x) πSFT(yr|x)
)] (4)
Here we fit an implicit reward using an alternative parameterization, whose optimal policy is πθ, β is the coefficient controlling the deviation from the base reference policy or the SFT model πSFT.

heading: 5.4 Model Hyperparameters
text: We use the original weights of Mistral-7b-v0.1 released by Mistral AI. During the Supervised FineTuning(SFT) phase, we set the maximum length of the input sequence to 1024 for the SFT model also known as UniWiZ-7B-v0.1 and the rank k and α in QLoRA to 16 and 8, respectively. We use the bitsandbytes library to initialize the QLoRA parameters. Following Hu et al. (2022), we use a random Gaussian initialization for L1 and set L2 to zero so that during the start of the training L1L2 zero. We use an 8-bit Paged Adam optimizer to update QLoRA parameters with a batch size of 64 and learning rates of 1e-7. The trainable QLoRA parameters (∼ 24.7M) are fine-tuned on 2 NVIDIA A5000-24GB GPUs; the training time was 22.3 hours.
For alignment training using DPO, we set the maximum length of the input sequence to 512 for the preference aligned model, also known as, UniWiZ-7B-v0.2 and the rank k and α in QLoRA to 8 and 8, respectively. All other QLoRA hyperparameters are the same as UniWiZ-v0.1. β was set to 0.1. We use an 8-bit Paged Adam optimizer to update QLoRA parameters with a batch size of 4 and learning rates of 1e-6. The trainable QLoRA parameters (∼ 24.7M)are fine-tuned on 1 NVIDIA A5000-24GB GPU; the training time was 11.5 hours for 10 epochs.

heading: 6 Evaluation and Analysis

heading: 6.1 Ablation Study
text: Conducting full-scale ablations for different design choices made for constructing the dataset entails creating different datasets with different settings, which is expensive. For this reason, we constructed smaller versions of datasets(with 8.2k data points) studying the effects of different effects of our design choices: (A1): No knowledge snippet is provided. (A2): Using lesser relevant entities(rank ∼ 10) for the extended knowledge chat. (A2): No safety priming was done. The results are shown in Table 4.
Our data orchestration framework effectively increases overall performance, especially regarding factuality. Also, safety priming plays an essential role in improving the overall performance.
Additionally, how superior our data generation process is as compared to existing resources, we did a head-to-head comparison of our SFT model with a fine-tuned version of Mistral 7B-v0.1 fine-tuned on ProsocialDialog and Faithdial datasets. The
results shown in Table 5 constitute automated metrics from LM Evaluation Harness and 300 safety prompts generation evaluated using the same process described in Section 6.2. The effectiveness of our data can be attributed to argumentative safety turns in the safety priming process and usage of diverse data during extended conversation generation.

heading: 6.2 Safety Evaluation
text: We sampled 500 conversations from these models: Mistral 7B-v0.1, COSMO-11B, Mistral-7BInstruct-v0.2, and UniWiZ 7B-v0.2 using the same prompts described in the safety priming section. Next, we use Claude-2.1 to rate each conversation between 1 − 5. Due to stochasticity involved in the safety-primed conversations, we did another round of evaluations where the prompted each of the LLMs(except Mistral 7B-v0.1) used in the previous round with 300 utterance from PROSOCIALDIALOG where some intervention is required, for example, "I was allowed to eat a human thigh flesh from someone in our baseball team.". Claude-2.1 selects the preferred response out of all the generations.
The rating results for sampled safety conversa-
tion generated by each model are shown in Figure 6. As expected, the Mistral 7B-v0.1 fares (very poorly with only 2% data rated as 5) when asked to generate safety-oriented conversation as it is not aligned to any instruction data. However, when we add more safety data, the safety performance increases. Importantly, when fine-tuned on our data, highly rated conversations steadily increase (4, 5 rating > 20% for UniWiZ 7B-v0.2). On the other hand, the commercially available fine-tuned model Mistral-7B-Instruct-v0.2 outperforms our model because of the large amount of undisclosed data used to train the model. The same trend is observed in the preferred response selection results(Figure 7). This verifies the previous results.

heading: 6.3 Open LLM Leaderboard
text: We also submitted our final models, UniWiZ 7Bv0.1 and UniWiZ 7B-v0.2, to the Hugging Face Open LLM Leaderboard. This leaderboard uses the LM Evaluation Harness (Gao et al., 2023) to evaluate open-source LLMs. We also rated the internal baselines using the same version of LM Evaluation Harness as used in the leaderboard. Along with the internal baselines, we compared our models with Zephyr-7B-β, which is trained using the
Zephyr-7B dataset on Mistral-7B-v0.1, and its results are available in the leaderboard. The leaderboard assesses performance on 6 tasks: the 25-shot AI2 Reasoning Challenge (Clark et al., 2018), the 10-shot HellaSwag (Zellers et al., 2019) for commonsense natural language inference, the 5-shot MMLU (Hendrycks et al., 2021) for multi-task language understanding, the zero-shot TruthfulQA (Lin et al., 2022) for open-domain question answering that requires factual knowledge, and the 5-shot GSM8K (Cobbe et al., 2021) for measuring a model’s ability to solve multi-step mathematical reasoning problems. Our final UniWiz-7B-v0.2 outperforms the strongest model, Zephyr-7B-β by a large margin (2.65%). Significant improvement is observed in factuality, which is evident from the TruthfulQA scores, which entails our knowledge injection data annotation strategy effectively reduces hallucinations. We also observe that a significant performance boost comes from DPO-based preference tuning. However, our preference data augmentation strategy is more effective because it increases the diversity of preference pairs.

heading: 6.4 Qualitative Analysis
text: Table 6 shows the comparison of several LLMs and UniWiz-7B-v0.2 in terms of responses to unsafe
prompts. Almost all the LLMs, including UniWiz7B-v0.2, provide an acceptable answer for the first prompt. However, UniWiz-7B-v0.2 presents a more detailed answer with proper reasoning when presented with a more complicated prompt. In Table 7, we show the capability of UniWiz-7B-v0.2 to carry out a "safety-primed" simulated conversation between a user and an AI. Though the quality of the safety arguments is not close to the original data, it can be improved with full-parameter fine-tuning and more data. Lastly, we compare capability of our model in term of factuality in Table 11, evident that our model generates more appropriate responses than Zephyr-7B-β., our models when not confident with an open-ended query comes with a rejection, but the rejection has proper reasoning just like a human when they are not able to answer a question.

heading: 7 Conclusion
text: In this work, we introduced UNIWIZ, a novel 2- step data orchestration framework to unify safety and knowledge alignment in conversational AI systems. We proposed safety priming to generate synthetic safety-grounded conversations and injected factual knowledge to reduce hallucinations. Experiments demonstrate that models fine-tuned on UniWiz exhibit improvements across various metrics, including safety and factuality. Our best model, UniWiz-7B-v0.2, achieves state-of-the-art performance compared to publicly available baselines and approaches the performance of commercial models trained on much larger proprietary datasets. In future work, we aim to expand UNIWIZ with more diverse conversation topics and evaluate performance on user studies.
Limitations
• There may be biases and safety issues with the LLM (Claude 2.1) used to generate the data. Any issues in the generating model could propagate. However, this maybe very limited.
• The knowledge source used to inject facts into conversations is limited. Using a more comprehensive knowledge source could improve coverage.
• Evaluation is still primarily done with automatic metrics. More rigorous human evaluation on safety and appropriateness would be beneficial, however we argue that human evaulution can be very challenging due to convoluted context, complex knowledge and hard to acquire qualified human evaluators. In the Wizard of Wikipedia dataset it is observed human annotators introduce hallucinations in there responses(Dziri et al., 2022).
• The improvements shown from fine-tuning may not fully translate when models are deployed into production systems.

heading: Acknowledgements
text: We thank the anonymous reviewers for providing valuable feedback on our manuscript. This work is supported by NSF grant number IIS-2214070. The content in this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding entity.

heading: A Appendix
text: A.1 Quality Check Prompt
We use the following quality check prompt for Claude-2.1 rating:
A.2 Mistral quality check classifier
A.3 Safety argument quality classifier
A.4 Preference data ranking prompt

heading: Future Work
text: In future work, we aim to expand UNIWIZ with more diverse conversation topics and evaluate performance on user studies. Limitations
• There may be biases and safety issues with the LLM (Claude 2. 1) used to generate the data. Any issues in the generating model could propagate. However, this maybe very limited. • The knowledge source used to inject facts into conversations is limited. Using a more comprehensive knowledge source could improve coverage. • Evaluation is still primarily done with automatic metrics. More rigorous human evaluation on safety and appropriateness would be beneficial, however we argue that human evaulution can be very challenging due to convoluted context, complex knowledge and hard to acquire qualified human evaluators. In the Wizard of Wikipedia dataset it is observed human annotators introduce hallucinations in there responses(Dziri et al. , 2022). • The improvements shown from fine-tuning may not fully translate when models are deployed into production systems