Improving Knowledge Production Efficiency With Question Answering on Conversation
Year: 2023

Abstract:
Through an online customer service application, we have collected many conversations between customer service agents and customers. Building a knowledge production system can help reduce the labor cost of maintaining the FAQ database for the customer service chatbot, whose core module is question answering (QA) on these conversations. However, most existing researches focus on documentbased QA tasks, and there is a lack of researches on conversation-based QA and related datasets, especially in Chinese language. The challenges of conversation-based QA include: 1) answers may be scattered among multiple dialogue turns; 2) understanding complex dialogue contexts is more complicated than documents. To address these challenges, we propose a multi-span extraction model on this task and introduce continual pre-training and multi-task learning schemes to further improve model performance. To validate our approach, we construct two Chinese datasets using dialogues as the knowledge source, namely ant-qaconv and kd-qaconv, respectively. Experimental results demonstrate that the proposed model outperforms the baseline on both datasets. The online application also verifies the effectiveness of our method. The dataset kd-qaconv1 will be released publicly for research purposes.

heading: 1 Introduction
text: With the rapid advance of Natural Language Processing (NLP), customer service chatbots have been widely applied in industries, as they can significantly reduce the cost of human customer service. Retrieval-based question answering model often plays an essential role in a chatbot system. However, building and maintaining a high-quality Frequently Asked Question (FAQ) database is laborintensive, which relies on human experts to produce the answers. In Alipay’s online customer ser-
1https://github.com/yclzju/kd-qaconv
vice system, there are many dialogues between customers and service agents collected daily, which contain customer questions and corresponding answers. To utilize these data, we design a knowledge production system, as depicted in Figure 1, to improve the efficiency of building the FAQ database. Our system extracts appropriate answers from the retrieved conversations for user questions that the chatbot cannot answer due to lacking relevant QA pairs in the FAQ database. After updating the FAQ database with produced QA pairs, the chatbot can answer the user questions correctly. The core of our system is the QA on conversations module, which extracts the answer from the candidate dialogues for a given question.
Current QA researches mainly focuses on document-based QA tasks, such as SQuAD (Rajpurkar et al., 2016), or conversational QA tasks (Reddy et al., 2019), which need to answer sequential dialogue-like questions based on the understanding of the given document, instead of QA on conversation. There are only a few relevant datasets, such as FriendsQA (Yang and Choi, 2019), Molweni (Li et al., 2020), QAConv (Wu et al., 2022), whose dialogues are in English and col-
225
lected from emails, TV shows, with the focus mainly on multi-party dialogues and speaker information. In contrast, our main objective is to extract knowledge from two-party dialogues, which are more common in the customer service industry.
Therefore, in this work, we construct ant-qaconv, a QA dataset consisting of human customer service dialogues. And to better validate our proposed method, we build another dataset, kd-qaconv, based on the public dataset kdconv (Zhou et al., 2020). The main challenges for our task, as reflected in the datasets, include: 1) knowledge information can be distributed in multiple turns rather than in a single sentence in a document, which means the answer can comprise multiple spans of text, 2) it’s difficult to model the hierarchical structure of a complex dialogue. Taking Figure 1 as an example, to answer the question "How long is the waiting period for the E-Life Medical Insurance?", the QA model must understand the context, including clarifications and co-references, then extract the answer from multiple turns of the dialogue. To address these challenges, based on a span-based machine reading comprehension model, we introduce a tag-based module to handle the multi-span challenge and propose a key utterance selection auxiliary task and continual pre-training to improve dialogue modeling. Experimental results show improved accuracy over baseline models from the proposed approach. Our main contributions can be summarized as follows:
• We design a knowledge production system based on the QA on conversation module, which improves the efficiency of maintaining the FAQ database for a chatbot.
• We introduce a construction pipeline and release the resulting dataset kd-qaconv, which, to the best of our knowledge, is the first public Chinese dataset for QA on conversation.
• We apply a multi-span extraction model and further improve its performance through continual pre-training and multi-task learning, and validate the approach’s effectiveness through extensive experiments on two datasets.

heading: 2 Related work

heading: 2.1 Pre-trained Language Models
text: Pre-trained language models (PLMs) such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019),
and AlBERT (Lan et al., 2019) have achieved stateof-the-art performance on many NLP tasks, including question answering (QA) tasks. These models are based on a self-attention mechanism and Transformer architecture (Vaswani et al., 2017) and are pre-trained on large corpora, which enables them to encode texts into contextualized representations.
However, most of these models are pre-trained on general domain textual corpora, such as Wikipedia, News, or Books, which can be notably different from the writing style and domain-specific language used in customer service conversations. As a result, many researchers (Gururangan et al.) have found that training PLMs on domain-related dialog corpora can help improve the model’s performance on dialogue-related and domain-specific downstream tasks.

heading: 2.2 Question answering
text: Question answering (QA) (Hirschman and Gaizauskas, 2001) is one of the most widely researched NLP tasks, which aims at providing correct answers to questions based on the given knowledge source.
Considering that dialogue is one of the primary forms of interaction and significantly different from the document, some researchers propose using dialogue as a knowledge source, named QA on conversation (Wu et al., 2022; Yang and Choi, 2019; Li et al., 2020). There are several unique challenges to QA on conversation: 1) information is scattered across multiple dialogue turns; 2) co-reference resolution is more difficult for understanding dialogues than documents. To alleviate such difficulties, Li and Zhao (2021) design self-supervised tasks on speaker prediction and key-utterance detection to capture salient information in long dialogues. Li and Choi (2020) propose several dialogue pre-training tasks, including utterance order prediction and mask language modeling, to learn both token and utterance embeddings to understand dialogue contexts better.

heading: 3 Dataset
text: The data collection pipeline (shown in Figure 2) includes three stages as follows:

heading: 3.1 Dataset Pre-processing
text: For ant-qaconv, we sample 4193 dialogues from the online customer service of our company and remove the personally identifiable information and
non-text elements such as emojis and pictures. For kd-qaconv, we choose kdconv, a public Chinese knowledge-driven conversation dataset about film, music, and travel, as the original dialogue data source.

heading: 3.2 Question Collection
text: We use three strategies to ensure the efficiency of the collection and quality of the candidate questions for each dialogue.
Question generation: Synthetic dataset construction has been proven effective in building robust and complex datasets (Feng et al., 2021). For ant-qaconv, we train the question generator (QG) with BART(Lewis et al., 2020)2 on Dureader (He et al., 2018) and select one candidate answer extracted by an internal extractive dialogue summary model for each dialogue to generate candidate questions. Since each conversation in kdconv is annotated with multiple knowledge graphs (KG) triples, we train a question generator on KgCLUE3 and randomly select two KG triples for each dialogue as the input of the QG model to generate candidate questions. Furthermore, we use a machine reading model trained from document datasets to filter out those generated questions with the predicted answer being used in QG since we suspect these questions may be too easy.
Question retrieval: For ant-qaconv, we also use BM25 (Robertson et al., 1995) to retrieve the most similar question from internal FAQ database as candidate question.
Human rewriting: For each dialogue, we ask the annotators to rewrite or remove the candidate questions with syntactic or semantic errors and try to write a new question that is different from the candidate question to ensure diversity of the questions.
2https://huggingface.co/fnlp/bart-base-chinese 3https://github.com/CLUEbenchmark/KgCLUE

heading: 3.3 Answer Labeling
text: For each sample, we ask internal annotators to read the questions and the dialogues and then label the answers, which can be nonexistent, a single text span, or multiple non-contiguous text spans in the dialogues.

heading: 3.4 Unanswerable questions
text: Previous work (Rajpurkar et al., 2018) show that unanswerable questions can force the model to decide whether a dialogue entails that a span of text can answer the question. To increase the number of unanswerable questions, we randomly sampled questions, and the annotators verified whether a text span was able to answer the selected question.

heading: 3.5 Dataset statistics
text: Both datasets have been divided into training, development, and testing sets in an 8:1:1 ratio. The data sample of the emphkd-qaconv dataset can be found in Section efsec:appendix, while detailed data statistics are presented in Table eftab:dataset. It is worth noting that the conversations in the emphant-qaconv dataset are not strictly structured as a question-answer format, and therefore, we consider one utterance as one turn for statistics of conversation turns. Compared to existing datasets, our proposed datasets offer a wider range of answer types, with larger mean and standard deviation of answer length, making them more challenging.

heading: 4 Method

heading: 4.1 Task Formulation
text: We define dataset as D = (Cm, qm, am)Mm=1, the dialogue including k turns is represented as Cm = (Sm,1, Um,1), (Sm,2, Um,2), ..., (Sm,k, Um,k), where Sm,i and Um,i represents the ith speaker and
utterance of the mth dialogue respectively. For a given dialogue C and question q, the model needs to predict the corresponding answer a, which could be a no-answer, single-span or multi-span answer. The task can be formulated as p(a|C, q).

heading: 4.2 Model
text: To better model the multiple types of answers, based on the span-based model, we apply the additional tag-based multi-span module and answer types classification modules. As shown in Figure 3, the model consists of the following components:
Transformer Encoder: The pre-trained transformer encoder aims to model the contextual feature representations of the question and dialogue. The input sequence of the encoder contains the question q and the flattened dialogue C, represented as X = [[CLS]; q; [SEP]; C], where X is the input token sequence, and C is the concatenation of each utterance Uk and corresponding speaker-id Sk of the kth utterance.
Answer types classification: The encoder representation H[CLS] of token [CLS] is treated as the dialogue level feature, which is used to predict the answerability and the multi-span classification by:
pa = sigmoid(MLP a(H[CLS])) (1)
pm = sigmoid(MLPm(H[CLS])) (2)
where MLP are multi-layer feed-forward networks, pa is the prediction score of answerability, which means whether the question can be answered. pm is the probability of whether the answer contains multiple spans. The loss function can be defined as follow, where ya, ym are the ground truths of answerability and multi-span classification, respectively:
La = −ya · log(pa)− (1− ya) · log(1− pa) (3)
Lm = −ym ·log(pm)−(1−ym)·log(1−pm) (4)
Answer Prediction: The answer prediction task is to predict the answer text from dialogue context, including the single-span prediction and multi-span prediction. HU = (H11, ...,Hij , ...,Hkn) is a sequence of contextualized representations for all utterance tokens. The single-span prediction task is to compute the probability of each token being the start or the end of an answer span. Formally, feed-forward networks are used to calculate a score for each token, then a softmax function computes the start and end probability distributions along all tokens in this sequence:
ps = softmax(MLP s(HU )) (5)
pe = softmax(MLP e(HU )) (6)
The loss of single-span prediction can be defined as follows, where ys and ye mean the labeled start and end position of the answer respectively:
Lspan = − 1
2 (log(psys) + log(p e ye)) (7)
The multi-span prediction task extracts a variable number of spans from the input dialogue utterances. We followed the method proposed by Segal et al. (2020), casting the task as a sequence tagging problem with IO tag schema, predicting for each token whether it should be part of the span or not. The probability of the model assigns to token ij having labeled tag Tij is:
pij(Tij) = softmax(MLP t(Hij)) (8)
and the loss function can be defined as follows, where N is the total count of tokens, ni is the count of tokens in ith utterance’s :
Ltag = − 1
N
k∑
i=1
ni∑
j=1
log(pij(Tij)) (9)

heading: 4.3 Key Utterance Selection
text: Since each answer span is a subsequence of utterance, modeling the key utterance selection(KUS) can help locate the answer spans. Formally, Hs = (H[s1], ...H[si]..., H[sk]) is the sequence of representations of each utterance, which is the corresponding contextualized representations of the speaker-id tokens, the probability that the i-th utterance contains the answer can be calculated as follows:
p[si] = sigmoid(MLPK(H[si])) (10)
The loss can be defined as:
LKUS = − 1
k
k∑
i=1
(y[si] · log(p[si])
+(1− y[si]) · (1− log(p[si])) (11)
y[si] represents the label for utterance [si], y[si] = 1 if it contains answer spans.
We adopt a multi-task learning scheme, which has been proven to be an effective way to improve model performance in NLP-related works (Chen et al., 2021). λa, λm, λspan, λtag, λKUS are hyperparameters to control the weights of each task, and the model loss is defined as:
Lmodel = λspan · Lspan + λtag · Ltag +λa · La + λm · Lm + λKUS · LKUS
(12)

heading: 4.4 Continual Pre-training
text: The public pre-trained transformer models such as BERT (Devlin et al., 2019), RoBERTa(Liu et al., 2019) have demonstrated state-of-the-art(SOTA) performance in various NLP tasks. However, they are primarily trained on general domain textual corpus such as Wikipedia, News or Books, notably different from discourse structure, writing style and domain in customer service conversations. Therefore, we introduce the dialogue continual pretraining approach to help better model the dialogue structure.
For ant-qaconv, we collect about one million unlabeled customer service dialogues from the online customer service chat log. For kd-qaconv, we download and process several publicly released Chinese dialogue data, including Duconv (Wu et al., 2019), Douban (Wu et al., 2017), Ecommerce (Zhang et al., 2018), DuRecDial (Liu et al., 2020) and LCCC (Wang et al., 2020). Considering the hierarchical structure of the dialogue, we design the following token and utterance level pre-training tasks.
Masked language model(MLM): MLM is an essential task to achieve better contextualized representations. For BERT (Devlin et al., 2019), 15% of tokens are picked randomly. 80% of these tokens are replaced with “[MASK]”, 10% are replaced with another random token, and 10% of the tokens are kept unchanged. For RoBERTa (Liu et al., 2019), we mask the whole word instead of the token (Cui et al., 2021). Then we compute the cross-entropy loss to predict the original token.
Response selection: Response selection is a classic dialogue pre-training task, which can enhance the contextual understanding of dialogue (He et al., 2022). The positive example (with label l = 1) is obtained by concatenating C with its corresponding response r in the original conversation. For negative samples, we randomly sample a response r− from other dialogue. We feed the concatenated sequence of C and r into the transformer encoder and a binary classification head on the token [CLS]:
p(l = 1|C, r) = sigmoid(H[CLS]) (13)
to classify whether r is the proper response for context C. The cross-entropy loss is defined as:
LRS = −log(p(l = 1|C, r) −log(p(l = 0|C, r−) (14)

heading: 5 Experiment

heading: 5.1 Experimental Setting
text: Following the standard evaluation metrics in the QA community, we choose word-level F1 and Exact Match(EM) accuracy as metrics to measure the overlap of the prediction and the ground truth answer(Rajpurkar et al., 2016). As the answer is long and descriptive in ant-qaconv, EM is unsuitable, so we choose F1 as our primary metric. To test our method, we choose bert-base-chinese4 and chineseroberta-wwm-ext5 as our PLMs, which are widely used in Chinese NLP tasks.
Due to the 512 positional embedding limit of RoBERTa and BERT, truncating inputs by removing overflowing tokens can result in loss of contextual information and samples. To address this issue, we utilize a sliding window mechanism to construct training samples. During prediction, we select the prediction with the answer position in the middle. In our experiments, the sliding window size is set to 128.
For training, we use the Adam optimizercitekingma2014adam with default parameters and learning rates of 1e-5, and a batch size of 16 for 15 epochs. We select the best model based on F1 score on the development set and evaluate on the test set.
The hyper-parameters setting is shown in Table 2. If we choose not to add some sub-tasks, we just set the weights to 0.

heading: 5.2 Experimental Results
text: Table 3 shows our experimental results on kdqaconv and ant-qaconv. As shown in Table 4, we further analyze the model’s F1 performance for different answer types in the kd-qaconv dataset. The tag-based models, such as bert-tagger and roberta-tagger, have been observed to perform well
4https://huggingface.co/bert-base-chinese 5https://huggingface.co/hfl/chinese-roberta-wwm-ext
in multi-span cases, but not as effectively in singlespan cases, which make up the majority of datasets. This may be due to the fact that the tag model needs to predict whether each token is part of the answer, which can be challenging to optimize, especially when the mean and variance of the length of answers in the datasets are large. In Table 4, the spanbased models, bert-span and roberta-span, were chosen as our base models owing to their superior overall performance. We then sequentially added proposed modules to further improve the model’s performance:
First, The tag-based multi-span module can help handle the multi-span answers, thus improving F1 by 0.5% in ant-qaconv, 3% in kd-qaconv. The latter improves more because there are more multi-span cases in the kd-qaconv dataset.
Second, continual pre-training on ant-qaconv leads to improvements of 2.15% and 3.1% over BERT and RoBERT, respectively. The gain for kd-qaconv is 1.84% for BERT and 0.55% for RoBERTa. We suspect the improvements come from improved dialogue representation and domain adaptation. Pre-training improves ant-qaconv even
more because there is a more significant gap between internal customer service dialog and general domain text corpus used by the original PLMs. So the domain knowledge and complicated dialogue structure introduced by continual pre-training can be of incredible help for downstream QA tasks.
Third, the key utterance selection(KUS) task improves the model accuracy in most places except for RoBERTa on ant-qaconv, probably because it can help the model to identify which turn contains information to answer a given question.
In all, our method is effective on both datasets and with both BERT and RoBERTa as PLMs.

heading: 5.3 Application in Knowledge Production
text: We have deployed the proposed model in realworld knowledge production for the FAQ database used by Alipay’s online customer service chatbot in the following workflow: 1) A cluster module and a classifier process the chatbot log data to identify incorrectly answered user questions; 2) A retrieval model based on BM25 and Simcse (Gao et al., 2021) retrieves the most relevant dialogues from human customer service data; 3) The proposed QA model extracts the answers for user questions from the retrieved dialogues; 4) human operators decide whether to adopt the QA pairs into the FAQ database and they can also refine them.
Online Evaluation: We choose the adoption rate as our end-to-end accuracy. Based on the statistics of three months’ data after the system deployment, the overall adoption rate is about 65%. We sample and analyze the QA pairs that are not adopted, only 8% of which are caused by inaccurate and incomplete extraction of the extraction model. The rest are caused by: retrieval mistakes, some queries or answers that are unclear or not suitable as the content of the FAQ database, etc. In the future, we will jointly optimize the knowledge production pipeline for better performance.
And we also choose knowledge production efficiency as our metric. The average time cost of producing a QA pair is reduced from 10 minutes to 2 minutes. When knowledge operators produce QA pairs directly, they must summarize answers from chat logs and related documents. However, with the assistance of our deployed system, they are only required to review and refine the recommended answers.

heading: 6 Conclusion
text: We design a knowledge production system including QA on conversations to help mine answers from human customer service dialogues. Based on the span-based extraction model, we add a multi-span extraction module trained with multi-task learning and continual pre-training schemes to extract incontiguous answers from conversational contexts. Experimental results show our approach outperforms the baseline models on both ant-qaconv and kd-qaconv datasets, the latter of which will be publicly released. Finally, the proposed method has been deployed to support the Alipay’s customer service chatbot system, which significantly saves the time cost of human operators’ producing new QA pairs.

heading: 7 Ethical Considerations
text: We present the following ethical considerations for data authorization, privacy, and deployments.
• We have obtained explicit permissions from the customer to collect and utilize customer service dialog data.
• We use desensitization tools to remove sensitive information in customer service conversations. Only a few annotators can access this data, and they will check again to ensure that there is no user personal information in the dataset. At the same time, the dataset antqaconv is only used for internal research.
• The QA pairs produced by the knowledge production system will be checked by human operators to make sure private message is removed.

heading: 8 Acknowledgments
text: Ant Group is China’s leading technology and financial services company that provides a range of digital financial solutions such as online payment and wealth management.
The authors would like to thank Changlin Yang, Siye Liu, Sen Hu, Wangshu Zhang, Teng Xu, Jing Zhen, and other members of Ant Group for helpful discussions and comments. We would also like to thank reviewers for their valuable comments.

heading: A Appendix
text: A.1 Data samples As shown in Figure 4, we sample a conversation and corresponding questions and answers from kdqaconv.

heading: Future Work
text: In the future, we will jointly optimize the knowledge production pipeline for better performance. And we also choose knowledge production efficiency as our metric. The average time cost of producing a QA pair is reduced from 10 minutes to 2 minutes. When knowledge operators produce QA pairs directly, they must summarize answers from chat logs and related documents. However, with the assistance of our deployed system, they are only required to review and refine the recommended answers