Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering (2024)

Abstract: Deploying large language models (LLMs) to real scenarios for domain-specific question answering (QA) is a key thrust for LLM applications, which poses numerous challenges, especially in ensuring that responses are both accommodating to user requirements and appropriately leveraging domain-specific knowledge bases. They are the two major difficulties for LLM application as vanilla fine-tuning falls short of addressing. Combining these requirements, we conceive of them as the requirement for the model’s preference to be harmoniously aligned with humans’. Thus, we introduce Knowledgeable Preference AlignmenT (KnowPAT), which constructs two kinds of preference sets to tackle the two issues. Besides, we design a new alignment objective to align the LLM preference with different human preferences uniformly, aiming to optimize LLM performance in real-world, domainspecific QA settings. Adequate experiments and comprehensive comparisons with 15 baseline methods illustrate that our KnowPAT is a superior pipeline for real-scenario domainspecific QA with LLMs.

heading: 1 Introduction
text: In contemporary digital commerce platforms, the deployment of automated and intelligent questionanswering (QA) services is a pivotal task to augment service quality. These services are designed to furnish answers to domain-specific customer queries. Building such a domain-specific QA system, while highly sought after, remains a daunting challenge in practical scenarios.
Domain-specific QA necessitates a comprehensive understanding of a specific domain to answer specialized questions. However, traditional deep learning models (Devlin et al., 2019; Raffel et al.,
* Equal contribution. † Corresponding authors.
☹
"
#
Related Knowledge
2020) still have insufficient domain-specific expertise. This makes the domain knowledge bases (KBs) (Liang et al., 2022) a pivotal tool for the storage and querying of domain knowledge. KBs can store human knowledge in the triple form, offering a unified, maintainable, and extensible representation of the knowledge from heterogeneous sources. The utility of KBs has already been demonstrated across various application scenarios such as Ecommerce (Zhu et al., 2021), and health care (Li et al., 2020). Within the context of QA, incorporating external knowledge source represents a promising approach, which is known as KBQA (Jiang et al., 2023b).
Meanwhile, as large language models (LLMs) (West et al., 2023) achieve significant progress and exhibit substantial proficiency within numerous NLP fields (Zhu et al., 2023), applying LLMs into various downstream tasks have been a predominant trend in industry (Zhang et al., 2023a). Contrasting earlier pre-trained language models (Devlin et al., 2019; Raffel et al., 2020), LLMs trained on the
891
massive corpus have outperformed text generation capabilities which perform better when interacting with human (Ouyang et al., 2022). To adapt the LLMs for downstream usage, supervised finetuning (SFT) (Zhang et al., 2023e) is applied to fit the model with specific tasks and data. However, the LLM application for real-scenario QA with external KB remains an underexplored domain, with limited work addressing this intersection.
Our goal entails the resolution of a challenge in real-world applications: How can LLM be used to solve real-scenario QA problems supported by external knowledge bases? A generic pipeline for this problem is the retrieve-augmented generation (RAG) (Tian et al., 2023), which first retrieves relative knowledge triples for the question as reference data and subsequently fine-tunes the LLM with knowledge-enhanced prompt. However, this conventional approach often encounters obstacles in practical scenarios. Firstly, the LLM-produced responses must prioritize user-friendliness, avoiding any generation of inappropriate or unfriendly content. Secondly, the retrieved knowledge is not invariably useful, necessitating that LLMs develop the capacity to judiciously exploit knowledge. Figure 1 illustrates a simple case in which retrieved knowledge is not always desperately needed (e.g., MAC is a kind of lipstick), which requires the LLMs to selectively utilize the retrieved knowledge instead of generating answers without thoughtful consideration. These two issues can uniformly collectively constitute the preference problem of LLMs. LLMs have their style preference to generate contents and knowledge preference to selectively use the retrieved knowledge in the prompt. As a practical application, the preference of LLMs needs to align with human expectations and requirements for better service. This refers to preference alignment (PA) (Yuan et al., 2023), a burgeoning topic in the LLMs community, which would incorporate human preference to tune the LLMs during training. PA aims to control the model to generate human-preferred content and avoid unpreferred content. However, the scenarios faced by current PA works tend to be generic. No research has been explicitly directed towards domain-specific applications such as our scenario, providing impetus for further exploration.
In this paper, we propose a novel three-step Knowledgeable Preference AlignmenT (KnowPAT) pipeline to address the domain-specific QA task for a real-scenario LLM application. Know-
PAT propose knowledgeable preference set construction to incorporate domain KBs to construct knowledgeable preference data. Besides, a new alignment objective is designed to optimize the LLM with the knowledge preference. Our contribution can be summarized as three-folded: (1). We are the first work that introduces preference alignment for domain-specific QA with LLMs and domain KBs, which is an industrial practice with practical applications. (2). We propose a knowledgeable preference alignment (KnowPAT) framework to incorporate KBs into the preference alignment process of LLMs. We balanced the need for both style and knowledge preference and devised a new training objective to align the LLM with human preference. (3). We conduct comprehensive experiments to validate the effectiveness of our methods on two datasets, which shows that KnowPAT outperforms 15 existing baselines.

heading: 2 Problem Setting
text: In this section, we will first introduce our problem scenario and basic notations.
Our overall target is to fine-tune a LLM M with our QA datasets D = {(qi, ai) | i = 1, 2, . . . , N} where qi, ai represent a question and answer pair. The questions in the dataset are all about common usage issues with our cloud products while the questions and answers are manually collected and labeled, which are gloden answers with decent and knowledgeable responses. For vanilla fine-tuning (VFT), we first wrap the QA pair with a prompt template I and the model M is autogressively (Brown et al., 2020) optimized as:
Lft = − 1|ai|
|ai|∑
j=1
logPM(ai,j |I, qi, ai,<j) (1)
where ai,j is the j-th token of ai and PM denotes the token probability predicted by the model M. With such a training objective, the training QA data serves as the supervision information to tune the model M to the QA scenario. Besides, as a domain-specific task, we maintain a domain knowledge base (domain KB) B. The domain KB can take very many forms, such as domain knowledge graphs, or documents. We denote ki ∈ B as one support knowledge in the domain KB. By retrieving top-k knowledge k with higher relevance, the input prompt will incorporate the retrieved knowledge K. Thus, M can learn the
baichuan
Style
knowledge atom
Alignment Loss
relative knowledge during the VFT process, which is a general retrieve-augmented generation (RAG) pipeline for domain-specific LLM applications.
However, such a VFT approach can not achieve pretty good results for the domain-specific QA. On the one hand, applications in real scenarios should be user-friendly, otherwise, they will not bring commercial value. Thus, the text style of the generated response should be more acceptable for users. On the other hand, the knowledge retrieval process is unsupervised and the effectiveness of the retrieved knowledge is hard to guarantee, which means that the model M needs to acquire the ability to judge and selectively utilize the knowledge triples. Therefore, we should improve the basic VFT to solve these two problems.
Actually, both of these problems can be summarised as model preference. The LLM M has its style preference to generate texts and its knowledge preference to selectively utilize the retrieved knowledge. For the model to be practically applicable, the model preference should align with human preference, aiming to generate high-quality answers that humans prefer. Preference alignment (PA) is an important topic for LLMs. To apply PA during LLM fine-tuning, we sample a preference set P = {b1, b2, . . . , bl} with l different answers for each QA pair (q, a). We denote ri as the preference score of each answer bi where higher ri represents that humans prefer this answer. During training, we will define another objective Lalign to align the model M with the preference set P , aiming to increase the probability of a human preferred
answer appearing and simultaneously decrease the probability of an unpreferred answer. The human preference of each answer is the preference score r. The overall training objective then becomes L = Lft+Lalign.With such a multi-task objective, the LLM is fine-tuned to fit the golden answers while avoiding unpreferred results. The next question is how to generate a preference set to reflect both the style and knowledge preference.

heading: 3 Our KnowPAT Pipeline
text: In this section, we will present our pipeline of knowledgeable preference alignment (KnowPAT), which consists of three key parts: unsupervised knowledge retrieve, knowledgeable preference set construction, fine-tuning, and training. Figure 2 demonstrates an intuitive view of the three parts in our pipeline design.

heading: 3.1 Unsupervised Knowledge Retrieve
text: The first key parts is the unsupervised knowledge retrieve which aims to link the knowledge in the KB B to each question qi. We design a simple semantic similarity-based retriever H to achieve this goal. The similarity between the i-th question qi and the j-th knowledge kj is:
sim(i, j) = Cosine(H(qi),H(kj)) (2)
where the retriever H serves as a textual encoder and we treat both the question and knowledge as a text sequence to get their sentence representations. The similarity is based on the cosine similarity of the two representations. We retrieve the top-k
knowledge with the highest similarities for each question qi and denote the retrieved knowledge (RK) as K. RK will be added into the input prompt as the background knowledge for the current question.
This process is unsupervised as we have no manually labeled question-knowledge pairs. Besides, our model will be deployed for real scenario usage, so it also requires strong zero-shot generalization capabilities to new questions. For these two reasons, the retrieved knowledge K might be noisy and useless to provide background knowledge. We think that the LLM M should learn the knowledge preference to select helpful information from the retrieved knowledge K.

heading: 3.2 Knowledgeable Preference Set Construction
text: Motivated by such goal, we propose a knowledgeable preference set construction process to enable retrieved knowledge in the preference set construction, which consists of two parts: the style and the knowledge preference set.
For the style preference set (SPS) Ps, we select l−1 different LLMs denoted M1,M2, . . . ,Ml−1. These different LLMs Mi have different textual comprehension and expression skills, which can generate answers with different text styles. The ability and quality of these models to answer domain-specific questions are inferior compared to human-labeled golden answers. The l − 1 answers generated in this way and golden answers form a style preference set Ps = {b1, b2, . . . , bl} with length l. For the knowledge preference set (KPS), we assume that the knowledge that has high similarity but do not reach the top-k rank are more likely to be knowledge that is not useful for the input question. We can get preference sets with different quality by retrieving some relatively worse knowledge and prompting the model to generate responses with knowledge of different quality. In our design, we retrieve 3 groups of knowledge K1,K2,K3 from the CPKG. K1 represents the retrieved top-k knowledge, K2 = ∅ is an empty set with no retrieved knowledge. K3 represents the knowledge with top k + 1 to 2k similarities which we think are easily misused knowledge with relatively high semantic similarity. Then we wrap the different knowledge Ki with the input prompt I into the LLM M and generate different answers. These generated 3 answers and the golden answer form a knowledge preference set
Pk = {c1, c2, c3, c4}. By doing this, we can get two preference sets for each QA pair. To simplify the setting, we set l = 4 to let the two sets be of the same size. Besides, we design a rule-based strategy to decide the preference score r for each answer. For the style preference set Ps, the high-quality golden answer b1 is assigned with the highest score, and answers from other LLMs were determined by their general capabilities. In practice, we choose three different LLMs ChatGPT (b2) (Ouyang et al., 2022), ChatGLM-6B (b3) (Zeng et al., 2023), and Vicuna7B (b4). The results of several LLM ranking lists indicate that the three are ranked in order of ability as follows ChatGPT > ChatGLM > Vicuna. Besides, after verification by human experts, we also believe that the quality of the answers generated by these three models in our QA scenarios also conforms to this rule. Thus, the preference scores are assigned in this order: r1 > r2 > r3 > r4.
Meanwhile, for the knowledge preference set Pk, the golden answer c1 still has the highest preference score r1. The answer c2 generated with top-k knowledge K1 has the second highest preference. The answer c3 generated with no extra knowledge K2 has the third highest preference, and the answer c4 generated with knowledge K3 is the worst. We found in our actual tests that the mismatch rate between the retrieved knowledge K3 and the question q is very high and easily misleads the model M, so we set its score to be lower than the case of the empty knowledge K2. Thus, for the knowledge preference set Pk, the preference scores are still in the order: r1 > r2 > r3 > r4. For each QA pair, we can construct two preference sets and we finally get the whole preference data with 2N preference sets. The preference data will participate in the fine-tuning process to control the style preference and knowledge preference for the model M. Note that the size of the two preference sets need not be strictly same, and we have adopted the above formulation for the sake of uniformity of representation in our paper.

heading: 3.3 Fine-tuning and Preference Alignment
text: In addition to the vanilla fine-tuning loss Lft with the golden answer, the preference data will also participate in the training process. For each preference set, the preference score ri of the i-th answer represents our degree of preference. We expect the model M to align with our preference. Thus, we design another score to represent the preference of
the model, which is denoted as:
Si = 1|ai|
|ai|∑
j=1
logPM(ai,j |I, qi, ai,<j) (3)
This score Si is the average log-likelihood of each answer token conditioned on the given prompt template I and question qi. Higher scores represent a higher probability that the model considers the current answer to occur. To align the model preference with our envision, we designed a new alignment objective for our scenario. The alignment objective is denoted as:
Lalign = − |P|−1∑
i=1
 log σ(Si) + ∑
rj<ri
log σ(−Sj)   (4)
where σ is the sigmoid function. Such an objective is newly proposed by us to achieve the preference alignment process, which contrasts the preferred answer and the unpreferred answers. It is worth noting that the human preference scores ri will only determine the ordering corresponding to different answers and will not be directly involved in the computation and gradient accumulation. Existing methods like RRHF (Yuan et al., 2023) and SLiC-HF (Zhao et al., 2023) apply a margin-rank loss in the form
∑ rj<ri
max(0, λ − Si + Sj) to achieve preference alignment. But their design only optimizes the model preference when the model preference score S of a human preferred answer is lower than an unpreferred answer (a more formalized formulation would be Si < Sj when rj < ri). However, we think that the preference should still be optimized in this situation and propose such a training objective to continuously decrease the occurrence probability of the unpreferred answers. Meanwhile, as different answers have different text quality and preference degrees, we further design an adaptive weight to control the influence of each preferred answer, which is denoted as:
µi = Si − Smin
Smax − Smin (5)
where Smax and Smin are the max and min model preference scores in a preference set P . With such an adaptive weight, the influence of the answers with different preferences could be dynamically adjusted. The alignment loss then becomes:
Lalign = |P|−1∑
i=1
µi  log(1 + e−Si) + ∑
rj<ri
log(1 + eSj )
 
(6)
The final training objective is still in a multi-task manner and we add a hyper-parameter λ as the coefficient of the alignment loss:
L = Lft + λ|P| − 1Lalign (7)
where P−1 represents the count of prefer-unprefer contrast to normalize the alignment loss. For each preference set constructed in the previous section, the model is trained and optimized with such an objective.

heading: 4 Experiments and Analysis
text: In this section, we present the detailed experimental settings and analyze the experiment results to investigate the following four research questions: (i) RQ1: How does KnowPAT perform compared with the baseline methods? (ii) RQ2: Do the proposed modules in KnowPAT really benefit the performance of KnowPAT? (iii) RQ3: Are there some intuitive cases to demonstrate the effectiveness of KnowPAT. (iv) RQ4: Does the LLM still keep the general ability rather than catastrophic forgetting?
These four questions evaluate our approach on four dimensions: performance, design soundness, intuition, and usability in real scenarios. We will answer the four questions in the following sections.

heading: 4.1 Experiment Settings

heading: 4.1.1 Dataset Information
text: Our experiments are performed on both private and public datasets. The private CPQA dataset consists of a CPKG and QA pairs. The public dataset is RJUA-QA (Lyu et al., 2023), which is a urology domain QA dataset. The detailed dataset information is presented in Appendix B.1.

heading: 4.1.2 Baseline Methods
text: To make a comprehensive study, we select four types of different baseline methods to demonstrate the effectiveness of our preference alignment approach. We not only want to show that alignment is a better framework for LLM application compared to other paradigms (e.g. zero-shot reasoning, in-context learning (Dong et al., 2023), vanilla finetuning (Ouyang et al., 2022; Fang et al., 2023)), but also to show that our method is better than other preference alignment methods (Yuan et al., 2023; Zhao et al., 2023; Song et al., 2023; Wang et al., 2023b; Rafailov et al., 2023). The detailed information of the baselines are shown in Appendix B.2.

heading: 4.1.3 Evaluation Metrics
text: To make a comprehensive evaluation of the experimental results, we employ the different evaluation metrics from three aspects: traditional text generation metrics (BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015), and METEOR (Banerjee and Lavie, 2005)), model-based metrics (BERTScore (Zhang et al., 2020a), PPL), and manual evaluation. The detailed information of the evaluation metrics refers to Appendix B.3.

heading: 4.1.4 Implementation Details
text: In our experiment, we select Atom-7B 1 as the backbone LLM M, which is an open-source version of
1https://github.com/FlagAlpha/Llama2-Chinese
Llama2 (Touvron et al., 2023b,a) with Chinese vocabulary extension. As our dataset is mainly in Chinese, we choose Atom-7B-chat to be our backbone model for experiments. Another consideration for us is that using the open-source Llama architecture model enhances the generality of our method to maintain the community ecology of LLMs. For unsupervised triple linking, BGE-base-zh-v1.5 (Xiao et al., 2023) is applied as the retriever H to encode and retrieve relative knowledge candidates.
During training, we tune the backbone model with bf16 float precision. The training epoch is set to 3 and the gradient accumulation step is set to 8. We optimize the model using AdamW optimizer (Loshchilov and Hutter, 2019) while the learning rate is fixed to 3e−4. The coefficient hyperparameter λ is search in {1, 0.1, 0.01, 0.001}.

heading: 4.2 Main Results (Q1)
text: The main results of the traditional metrics are shown in Table 1. As we mentioned before, the traditional metrics can measure the similarity between the generated answer and the golden answer. From the results, we can observe that KnowPAT achieved obvious improvements compared with the baseline methods. We can conclude that KnowPAT achieves a more significant improvement in the
BLEU-3(42.03%)/BLEU-4(43.99%) than BLEU1(22.67%)/BLEU-2(34.79%), which means that KnowPAT makes more significant progress in capturing some complex phrases and discourse. Corresponding to our cloud product QA scenario, these complex phrase usages are usually specialized terms that have a critical impact on the quality of the answer.
Besides, we evaluate our methods with three model-based metrics BERTScore (Zhang et al., 2020a), reward score (Yuan et al., 2023), and PPL (Yuan et al., 2023), which is shown in Table 2. We can observe that KnowPAT still achieves good performance in the model-based metrics such as BERTScore and PPL, which means that the results generated by KnowPAT are more acceptable for the language models. For the reward score, relatively good results have also been achieved by KnowPAT.
Further, we conduct a human evaluation for our method and baseline methods. The two results from the two models are shown to the human evaluator anonymously so that the human evaluator can choose a better result. The model which generates that result will get one point and the competition results are shown in Figure 3. We can observe from the figure that our method generates answers that are more acceptable to humans compared to other baselines, maintaining a relatively high win rate in the competition. Only a small number of times does KnowPAT perform weaker than the baselines, and most of the time KnowPAT is equal or even better. Therefore, combining the above three different perspectives of evaluation, we can conclude that KnowPAT achieves outperforming results in the cloud product QA scenario.

heading: 4.3 Ablation Study (Q2)
text: We conduct Ablation experiments to verify the validity of each module design. We validated the effectiveness of the designed components in our
KnowPAT. We can find that the fine-tuning objective Lft and the alignment objective Lalign are both contributing to the model performance. Without fine-tuning (FT), the model performance can take a serious dip, as the LLM is not tuned to fit the golden answer. Besides, both two preference sets (SPS and KPS) in KnowPAT are contributing to the performance. The adaptive weights (AW) can control for the participation of different quality samples in the loss, which is also effective in KnowPAT.
Besides, we demonstrate the necessity of the CPKG with two groups of experiments. w/o RK denotes the experiment that removes the retrieved knowledge in the input prompt during the finetuning and preference alignment process. w/o KG denotes the experiment without KG in the whole process, which means the KPS and RK in the input prompt are all removed. For the results of these two groups of experiments, we can observe that the CPKG plays a remarkable role in KnowPAT. In the design of KnowPAT, the CPKG does not only serve as an external knowledge source during training but also participates in the preference set construction process, which is important to the model performance. In summary, each detailed design in our method KnowPAT has its unique role and contributes to the overall performance.

heading: 4.4 Case Study (Q3)
text: To make an intuition for the effectiveness of our method, we conduct a case study as shown in Table 4. We can observe that the answers generated by KnowPAT are more similar to the golden answer while keeping a user-friendly tone and providing sufficient information such as the host parameters in the second case. This suggests that the model
learns appropriate style preferences. Besides, the retrieved knowledge in the first case is (EIP, used for, IP Binding), (Select Box, belongs to, Alarm Management Component), etc., which are all helpless to answer this question. However, KnowPAT is not misled by this useless knowledge and generates the correct answer while RRHF falls into the trap.

heading: 4.5 Knowledge Retention Analysis (Q4)
text: As a project that needs to get off the ground in real-world scenarios, the general ability of the trained model should also be carefully evaluated, because the user may ask various kinds of questions if they like the model. We expect the model to keep their existing knowledge learned during pre-training and obtain new knowledge about our domain. Thus, we also conduct a commonsense evaluation on the trained models with the CMMLU
(Li et al., 2023) dataset, which is a benchmark for LLM’s Chinese ability evaluation. The evaluation result is shown in 4. We demonstrate the general ability on five distinctive commonsense regions (history, clinical, politics, computer science, and economics) for KnowPAT, vanilla Atom-7B (none), and other PA methods. As can be seen from the radargram, there is a relatively significant decline in the KnowPAT’s ability in medicine, but in the areas of politics, history, and economics it still maintains the ability of the original backbone model and even grows slightly. PRO, while unexpectedly showing a significant improvement in the economics problem, shows a more pronounced performance degradation than KnowPAT in several other areas. Taken together, such variations of KnowPAT in generalized ability are acceptable for our cloud product QA scenario.

heading: 5 Related Works
text: Preference alignment (PA) (Wang et al., 2023d; Cheng et al., 2023) seeks to tailor pre-trained LLMs to align with human preferences (feedbacks) (Ouyang et al., 2022). RLHF is a landmark work for PA, which leverages reinforcement learning (RL) (Schulman et al., 2017) to align human preference with LLMs. Due to the sensitivity of RL parameters and the intricate three-stage processes of RLHF, many PA approaches have been proposed to address these challenges. For example, RRHF (Yuan et al., 2023) propose a margin-rank loss to optimize the LLMs without the need for extra reward models. PRO (Song et al., 2023) optimizes complex preference data with a list-wise contrastive loss. DPO (Rafailov et al., 2023) propose a direct preference optimization method by treating the LLM itself as a reward model. AFT (Wang et al., 2023b) propose a ranking-feedback boundary-constrained alignment loss to optimize the preference data. Besides, our work also focuses on the large language model application and knowledge-enhanced QA. We give a brief introduction of these fields in Appendix A.1 and A.2.

heading: 6 Conlusion
text: In this paper, we introduce a novel framework, knowledgeable preference alignment (KnowPAT), for domain-specific QA tasks in cloud product services, leveraging LLMs and KGs in a practical application setting. Our approach constructs a knowledgeable preference set by retrieving and utilizing
knowledge triples to generate answers with different quantities. A new alignment objective is designed to unleash the power of the preference set. Comprehensive experiments demonstrate that our method surpasses existing solutions for this realworld challenge. Looking ahead, we aim to apply KnowPAT to more real scenarios such as enterpriseclass services and further investigate the potential of KG-enhanced LLM application in the future.

heading: Acknowledgment
text: This work is founded by National Natural Science Foundation of China ( NSFC62306276 / NSFCU23B2055 / NSFCU19B2027 / NSFC91846204 ), Zhejiang Provincial Natural Science Foundation of China (No. LQ23F020017), Ningbo Natural Science Foundation (2023J291), Yongjiang Talent Introduction Programme (2022A-238-G), Fundamental Research Funds for the Central Universities (226-2023-00138).
Limitations
In this paper, we mainly focuses on a real-world application problem to align LLMs with knowledge preference for better domain-specific QA. There are still some limitations in our work. Domain-specific scenario. Our approach is designed for specific domain (cloud product QA in our paper), and its effectiveness on general domains and open-source datasets is still subject to further validation. This will be the goal of our future endeavours. Forms of external knowledge. In our paper, we apply knowledge bases to store the external background knowledge for the QA tasks. This is a convenient and efficient way of storing knowledge for our scenario, but in more other scenarios, knowledge may be stored in other forms (e.g. unstructured text). Therefore, a more general framework to process the external knowledge with any format (KGs, unstructured text, documents) should be considered for better usage, which is also our future plan.
Ethical Considerations
In this paper, we employ the open-source LLM to validate the effectiveness of our approach. Besides, the dataset we used is manually labeled with golden answer from domain experts engaged legally with suitable work intensity and well above aversage wages. Their rights are well protected at work. The
content of the dataset is mainly questions about our cloud product usage, which do not involve private information and sensitive data of the target users. We promise that the content and collection steps of our dataset that are not against scientific ethics.

heading: A Related Works
text: A.1 KG-enhanced Question Answering
Knowledge graphs (KGs) (Wang et al., 2017; Liang et al., 2022) is a kind of complex semantic web that models world knowledge in terms of structural triples as (head entity, relation, tail entity). KGs serve as external knowledge source and benefit many AI tasks like language model pre-training (Liu et al., 2020), question answering (Yasunaga et al., 2021; Wang et al., 2023c), and recommendation systems (Wang et al., 2019; Sun et al., 2020). Besides, domain-specific KGs are the important
infrastructure of internet industry to provide exact factual knowledge, which is widely leveraged in E-commerce (Zhu et al., 2021; Zhang et al., 2021), telecom fault analysis (Chen et al., 2023b), health care (Li et al., 2020; Zhang et al., 2020b) and so on. It is a popular topic to utilize KGs in real industry applications. In our scenario, we construct a domain-specific KG for cloud service products to benefit our Question Answering (QA) task. QA stands as a cornerstone in NLP, aiming at equipping machines with the capability to autonomously respond to human queries (Su et al., 2019; Yoon et al., 2019). QA tasks can take on various forms. Some require the selection from multiple choices, as seen in certain knowledge base QA (KBQA) (Cui et al., 2017; Tian et al., 2023; Baek et al., 2023) and visual question answering (VQA) (Antol et al., 2015; Chen et al., 2021; Wang et al., 2018). Conversely, tasks like open-domain QA often challenge systems to directly produce textual responses without a set answer pool (Gao et al., 2021; Karpukhin et al., 2020). In the last few years, fine-tuning pre-trained language models has been a leading approach for QA tasks. Models like BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020) have previously achieved notable performance when adapted with question-answer pairs.
We hold that QA doesn’t just remain an academic pursuit; it acts as a bridge, facilitating the adoption of AI technologies in real-world applications. Numerous industrial efforts have been directed toward developing domain-specific QA systems to meet the needs of their users (Gao et al., 2021, 2019). Such systems often rely on domainspecific knowledge bases, like Knowledge Graphs (KGs), to provide relevant information for the posed questions. Our current investigation aligns with this trend, focusing on a domain-specific QA scenario for cloud service products. Moreover, our approach diverges from these recent KG-based QA systems (Jiang et al., 2023b,a; Luo et al., 2023; Zhang et al., 2023d; Chen et al., 2022) that utilize prompts for dialog with (large) language models to facilitate path reasoning and refine the scope of KG retrieval. We propose an innovative knowledgeable preference alignment framework that enhances KGaware QA with the knowledge preference.
A.2 Large Language Model Application Prominent large language models (LLMs) like GPT (OpenAI, 2023; Brown et al., 2020; Ouyang et al., 2022) and GLM (Zeng et al., 2023; Du et al., 2022)
are sparking a wave of research in the community due to their generalization ability in many NLP tasks such as relation extraction (Zhu et al., 2023), algebraic reasoning (Wei et al., 2022), and question answering (Dao et al., 2023; Nguyen and Nguyen, 2023). Most LLMs leverage the transformer (Vaswani et al., 2017) architecture, benefiting from training on vast corpora (Thakkar et al., 2023) through autoregressive tasks. Deploying and applying LLMs in real-life scenarios is also a major topic in industry today and several efforts have been made. For example, many works (Zhang et al., 2023a; Bao et al., 2023b; Zhang et al., 2023c; Bao et al., 2023a; Zhang et al., 2023b; Chen et al., 2023a) attempt to build recommendation systems with LLMs. Some work like Huatuo (Wang et al., 2023a) and LawyerLlama (Huang et al., 2023) have developed LLMs for domain-specific usage.
Our work proposes a knowledgeable preference alignment framework to incorporate the domainspecific KG into the preference alignment pipeline for the LLM application. By constructing a knowledgeable preference set, the LLMs are trained to align the knowledge preference with humans and select better factual knowledge in the input prompt to solve the QA task.

heading: B Experiment Details
text: B.1 Dataset Details The detailed information of our dataset are shown in this section. We evaluate the performance of KnowPAT on two dataset: one private dataset CPQA constructed by us, and one public dataset RJUA-QA (Lyu et al., 2023). They are both domain-specific datasets.
• CPQA is for the cloud product domain labeled by a team of human experts with 8909 QA pairs. CPQA employs a cloud product knowledge graph (CPKG) as the domain KB.
• RJUA-QA is a urological domain open-source dataset extracted from real-world medical records with 2132 QA pairs. RJUA-QA labels a series of medical context documents for each QA pair. We collect these contexts in the form of documents as the domain KB.
B.2 Baseline Details (i) Zero-shot approach, which directly prompts the LLM with the input question to get the answer without training.
(ii) In-context learning (Dong et al., 2023) approach, which would sample a few (k-shot) QA pairs as demonstrations from the training dataset as examples and get the answers from the LLM without training. (iii) Vanilla fine-tuning approach, which finetunes the LLM using the QA pairs w/ or w/o retrieved knowledge as Equation 1. The fine-tuning baseline with retrieved knowledge is also known as retrieve-augmented generation (RAG) method. (iv) Preference alignment approaches, which introduce additional preference alignment objectives during training to align with human preference. We select five existing state-of-the-art (SOTA) PA methods including RRHF (Yuan et al., 2023), SLiCHF (Zhao et al., 2023), DPO (Rafailov et al., 2023), PRO (Song et al., 2023), AFT (both AFT-BC and AFT-DC) (Wang et al., 2023b) as our baselines.
B.3 Evaluation Details We select three types of metrics to evaluate our method against baselines. The detailed information on the metrics is listed in the following:
(i) Traditional text generation metrics. We select several traditional text generation metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015), and METEOR (Banerjee and Lavie, 2005) to evaluate the generated answers. However, these evaluation metrics are mainly used to measure the text-level similarity between generated answers and real answers, which means they can not fully reflect the semantic relevance or depth of understanding of the text.
(ii) Model-based metrics. To evaluate the semantic similarity of the generated answers and the golden answers, we employ several model-based metrics such as BERTScore (Zhang et al., 2020a), perplexity (PPL), and preference score. These metrics evaluate the generated answers using various language models. BERTScore employs BERT (Devlin et al., 2019) to calculate the semantic similarity between two sentences. PPL measures the ability of the LLM to understand and predict the entire sentence. The preference score is S mentioned in Equation 3 to reflect the model’s preference degree of the current answer.
(iii) Manual evaluation metrics. We employ human labelers to evaluate the results from different methods. The labeler makes a judgment on two answers from unknown sources in a singleblind situation, chooses the better one, and counts
the results. The comparison result in each turn is recorded as win/tie/lose.
The three main categories of metrics respond to a certain part of the result’s characteristics at three levels: similarity at the textual level, similarity at the semantic level, and human preference.
B.4 Implemention Details For zero-shot baselines, we select several different LLMs (ChatGPT (Ouyang et al., 2022), ChatGLM6B (Zeng et al., 2023), Baichuan-7B 2, Vicuna-7B 3, and Atom-7B-CP) for the zeros-shot approach. For in-context learning, we sample 1,2,4,8-shot QA pairs as demonstrations to support the input question. For the PA methods, we leverage the official code of RRHF (Yuan et al., 2023) and implement other PA methods (SLiC-HF (Zhao et al., 2023), PRO (Song et al., 2023), AFT (Wang et al., 2023b)) based on the code to reproduce the results on our preference dataset. The selection of hyperparameters is based on the original paper. Atom7B-CP is employed as the backbone model for all the baseline methods such as in-context learning, vanilla fine-tuning, and PA methods.
2https://github.com/baichuan-inc/Baichuan-7B 3https://github.com/lm-sys/FastChat

heading: Future Work
text: This will be the goal of our future endeavours. Forms of external knowledge. In our paper, we apply knowledge bases to store the external background knowledge for the QA tasks. This is a convenient and efficient way of storing knowledge for our scenario, but in more other scenarios, knowledge may be stored in other forms (e. g. unstructured text). Therefore, a more general framework to process the external knowledge with any format (KGs, unstructured text, documents) should be considered for better usage, which is also our future plan. Ethical Considerations
In this paper, we employ the open-source LLM to validate the effectiveness of our approach. Besides, the dataset we used is manually labeled with golden answer from domain experts engaged legally with suitable work intensity and well above aversage wages. Their rights are well protected at work. The
content of the dataset is mainly questions about our cloud product usage, which do not involve private information and sensitive data of the target users