DialogQAE: N-to-N Question Answer Pair Extraction from Customer Service Chatlog
Authors: Xin Zheng
Tianyu Liu
Haoran Meng
Xu Wang
Yufan Jiang
Mengliang Rao
Binghuai Lin
Yunbo Cao
Zhifang Sui
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue QA Extraction

Abstract: Harvesting question-answer (QA) pairs from customer service chatlog in the wild is an efficient way to enrich the knowledge base for customer service chatbots in the cold start or continuous integration scenarios. Prior work attempts to obtain 1-to-1 QA pairs from growing customer service chatlog, which fails to integrate the incomplete utterances from the dialog context for composite QA retrieval. In this paper, we propose N-to-N QA extraction task in which the derived questions and corresponding answers might be separated across different utterances. We introduce a suite of generative/discriminative tagging based methods with end-to-end and two-stage variants that perform well on 5 customer service datasets and for the first time setup a benchmark for N-to-N DialogQAE with utterance and session level evaluation metrics. With a deep dive into extracted QA pairs, we find that the relations between and inside the QA pairs can be indicators to analyze the dialogue structure, e.g. information seeking, clarification, barge-in and elaboration. We also show that the proposed models can adapt to different domains and languages, and reduce the labor cost of knowledge accumulation in the real-world product dialogue platform.

Di:logQAE: N-to-N Question Answer Pair Extraction from Customer Service Chatlog
Xin Zheng 4+ Tianyu Liu
Haoran Mengl Xu Wang Yufan Jiang
Mengliang Rao Binghuai Lin Yunbo Cao² Zhifang Sui-t
National Key Laboratory for Multimedia Information Processing, Peking University 2 Institute of Software, Chinese Academy of Sciences, China 3 Tencent Cloud Al
4 University of Chinese Academy of Sciences, China 6 Alibaba Group zhengxin2020eiscan ac. haoranêstu, pku. edu. cn (tianyu8421, sepku edu. cn
Abstract
start continuous integration scenarios, harvest- ing QA pairs from existing or growing customer service chatlog is an efficient way to enrich knowl- edge bases. Besides the retrieved QA pairs can be valuable resources to improve dialogue summa- rization (Lin et al., 2021), gain insights into the prevalent customer concerns and figure out new customer intents or sales trends (Liang al., 2022), which are of vital importance business growth.
Harvesting question-answer (QA) pairs from customer service chatlog in the wild is an ef- ficient way to enrich the knowledge base for customer service chatbots the cold start or continuous integration scenarios. Prior work attempts to obtain 1-to-1 QA pairs from grow- ing customer service chatlog, which fails to integrate the incomplete utterances from the dialog context for composite QA retrieval. In this paper, propose N-to-N QA extraction task in which the derived questions and corre- sponding answers might be separated across different utterances. We introduce suite of generative/disriiinnatte tagging based meth- ods with end-to-end and two-stage variants that perform well on 5 customer service datasets and for first time setup a benchmark for N-to-N DA with utterance and session level evaluation metrics. With a deep dive into extracted QA pairs, we find that the relations between and inside the QA pairs can be indi- cators to analyze the dialogue structure, e.g. information seeking, clarification, barge-in and claboration. We also show that the proposed models can adapt to different domains and lan- guages, and reduce the labor cost of knowl- edge accumulation in the real-world product dialogue platform."
Prior work question-answer extraction fol- lows the utterance matching paradigm, e.g., match- ing the answers the designated questions in a dialogue session (Jia et al., 2020) in the offline setting or figuring out the best response to the spe- cific user query (Wu et al., 2017; Zhou et al., 2018; Yuan et al., 2019) in the online setting. Within this framework, 1-to-1 QA extraction has been explored by Jia et al. (2020), however, we argue that users might not cover all the details in a single query while interacting with the customer service agents, which means that a certain QA pair might involve multiple utterances in the dialogue session.
In this paper, we extend 1-to-1 QA matching to N-to-N QA extraction, where the challenges are two-fold: 1) cluster-to-cluster QA matching with no prior knowledge of the number of utterances in- volved in each QA pair and the number of QA pairs in each dialogue session, the question might be distributed in single or multiple user queries. 2) session-level representation learning with a longer context, as the paired questions and answers might be separated within the dialogue, and the model shall detect multiple same-topic questions and then check if the answer is related to any one of the questions. We propose session-level tagging- based methods to deal with the two challenges. Our method is not only compatible with the N-N QA
1 Introduction
The development of natural language processing and conversational intelligence has radically rede- fined the customer service landscape. The customer service chatbots empowered by knowledge bases or frequently asked questions (FAQs) drastically en- hance the efficiency of customer support (c.g. Cui et al., 2017; Ram et al., 2018; Burtsey et al., 2018; Liu et al., 2020; Paikens et al.. 2020) In the cold
extraction task setting but also 1-1 and 1-N, which is generic. Switching from matching to tagging, we feed the entire dialogue session into powerful pre-trained models, like BERT (Devlin et 2019)
"Our code and data are available at https://githhh: com/Mr hhtnthhh/nn/n/llggge "Equal contribution.
Cooresppnndigg authors.

---

UID=1
# When teeduury aritin?
UID-12
Mr $5
UID-23
Yourneed replace the previous tnvetco, rigne
UID-2
12 29 16?
#
UID=13
Giver me the phane number the hvery departmant.
UID-24
A, & Yes, itvos non- tnwolce.
At.
Do youuhhsee hollow quection bone foodbaci, other new questions?
UID=3
Mt $ F2- e5, I (E
UID-14
ml i. 57 16-6 fr. (1 = (thone)
the delivee
UID=25
Iived to changeit to special VAT inooise, qual boen uphoathe..
#R-F 3,, K 1:44 I
3f - R.T
Contect the deiveery person, Whln orivv?? hove been warting fora whole day.
UID-15
28 Tharks.
UID=26
$ 2-
UID=4
d: + Y you dona chocking?
12 d5
dit you need to send the oid iuciin bock, then fruntze will hancle ofeer moaids.
UID-16
UID=5
38 4 4
F..
there anything the lean helo with??
UID=27
#F 18 Himm, what's the ma ddiesss
Please wat socond am workingon
12 A. e 36, 25%% I He, iit tt
UID-17
$5:8% At 1i- T It 2, 2. (0) IT 14 Aluo need check when can anserbled.
[+ 1 [12 16) 44:58: (R: (AdtressPhone (Phone) Postode (Number
UID-6
anowered, ony phone
UID-28
UID=7
Fridouble clacking, ploase
di, a. & fi of, i 18
secand
UID-18
#i.E W 68 it 19
delvery person may heery.
UID-29
cic.
R7-19
&& 6- it [ A4] Pf 81 ki.. repactedd artie (nane].
13
UID-8
UID-19
& 49.
UID-30
donot have other phone numbeer
there www.eantthhgggg whnn can you my bant.
UID-9
tl Accemble for on [Dane].
1/ 15 SE
ik. Somry about that.
UID-20
23 Fl F2, it 25 Ut?? twent to know ronetly on what cleck in be defivere??
UID-31
32 F 7 No more qaestons
UID=10
1 EC H5
contact deluary person later.
R 1-54 1 &A H
JL Ab.0)
88 dy 12 19-5e ft delivery person
UID-11
You reeed to
UID=22
AL & & + 18 At- HI & A
Picczeggee give me thumbup. Thank for support itove good day.
the decalled time.
Laoo sppedaal
Q11
A1
A4
UID-1
UID=3
UID-10
UlD=B
UID-11
UID-6
UID-9
1-to-1 Mapping
A2
1-to-N Mapping
UID=17
UID-18
UID-19
UID=21
UID-13
UID-14
N-to-1 Mapping
A3
A6
UID=22
UID=23
UID-24
UID-25
UID-26
VID-27
UID=28
N-to-N Mapping
Figure 1: The task overview for DAE, Given a session of two-party conversation with 32 utterances (top), we aim at extracting six QA pairs (bottom) that characterize the dialogue structure and can serve valid resource to enrich the knowledge base. The task can be categorized into four types: 1-to-1, I-to-N, N-to-1, and N-to-N, according to the number of utterances involved in the extracted question or answer unions.
or mT5 (Xue et al., 2021), and design a set of QA tags that empowers N-to-N QA matching. Through careful analysis, we find that DialogQAE can serve powerful tool in the dialogue structure analysis. the relations between and within QA pairs implicit signals for dialogue actions like informa- tion seeking, clarification, barge-in and elaboration. From a pragmatic perspective, we show that the proposed models can be easily adapted to different domains and languages, and largely accelerate the knowledge acquisition on FAQS of real-world users in the product dialogue system. We summarize our contributions below:
paradigm for dialogue analysis by summariz- ing 5 between-Qua QA-pairs and in-QA-pain re- lations that characterize the dialogue structure in the conversation flow.
Through careful analysis of domain and lan- guage adaptation, as well as real-world ap- plications, we show that the proposed Di- alogQAE model effectively automates and ac- celerates the cold-start or upgrade of com- mercial dialogue system.
2 Task Overview
We setup benchmark for DAa with end-to-end and two-stage basclines that sup- port N-to-N QA extraction, as well as the ut- terance and session-level evaluation metrics.
A complete snippet of customer service conversa- tion between human service representatives and customers, which is canonically termed as di- alogue session S, consists of multiple, i.e. n, dialogue utterances. Formally we have = {(u ri), (uz, r2), ..) "a....,,,,,, (un, rw)]. in which r; sig-
We show that DiA is an effective

---

nifies the speaker role of the i-th utterance uf. In this paper we focus on two-party dialogue, more concretely we have r; E () in which 'C", 'A' represents the roles of speakers: customers and human agents respectively.
given questions in QA extraction (Jia et al., 2020), measuring the similarity of the user query and can- didate answers in the response selection (Wu et al., 2017; Henderson et al., 2019), or extracting re- lations with entity matching in the dialogue (Ti- gunova et al., 2021).
After feeding the dialogue session into the DialogQAE model, we expect the model to extract m QA pairs R {(VQ,, UA1); (U@,) UA2).... (UQm: Am,)]
The key to successfully excavating n-to-n QA pairs from customer service chatlog is to figure out the cluster-to-cluster mapping among utterances in a dialogue session.
UQ,- = ={(m) T (Uaa Ta)).... (Ugg: Fg.
3.2 End-to-end QA Extraction
(1)
UA, ((ua (Uaz) Tez)." (War ra:)
We convert QA extraction into" filll-in-iue in-the-blank" se- quence labeling task, hoping that the model would quickly learn to predict the label 4, E {0, Q, A based on the corresponding utterance U.. After label prediction, we collect the QA pairs R = [(Uq, UA,) {,,Q,,U,))UU
(2)
Uo UA, represent the unions of question and an- swer dialogue utterances in S respectively.
To better characterize the proposed n-to-n di- alogue QA extraction, we introduce two notions which are conceptually related to the mapping be- tween dialogue utterances and extracted QA pairs. 1) Exclusive dialogue utterance: each utterance in can only be exclusively mapped to one single question or answer union in R. i.e. the mapping between S and R is a one-to-one (injection) func- tion. 2) Speaker role consistency: a common as- sumption for most two-party conversations is that the customers raise questions while the agents an- swer the questions. Formally for each extracted {C).
[uglk = Qj S k S nA, = {ullk Aj, S k n).lss m] from the labels L. = (4 n).
As depicted in Figure 2, the input of the model is "ri: (1), [MASK] [SEP] ...In: uh [MASK] [SEP] ", where [MASK], [SEP] signifies mask to- ken, separation token and both of which are in the vocabulary of the masked language model. We for- mulate the QA sequence labeling task either a generative, i.e. mTS--ttyle (Xue et al., 2021), or discriminative, i.c. BERT-style (Devlin et al., 2019)), classifier.
QA pair, e.g. UQ, and UA in R, { [rai=] {A). In our setting, the rule of exclusive dialogue utterance strictly holds for all the datasets we used. However, although most datasets in this paper exhibit speaker role consistency, we still ob- serve the customer queries in the answer union or the agent responses in the question unions, for ex- ample in Fig 1 the 23-rd utterance from the agent is included in the question union Q3.
From the generative perspective, i.e., span- corruption model mT5, the [MASK] token sym- bolizes the <extra_id_i> for each utterance u, and we use the semicolon (:) as the replacement for the separation token [SEP]. The output of the QA extraction model is a list of Q/A labels L, where for the encoder-only model each predic- tion is exactly the masked position, and for encoder-decoder model mT5 the prediction is a sequence "<extra_id id____d______ 0> 4,.. <extra_id_nt 1> In For the discriminative tagging model (BERT- style), we use [unusedx] to denote the label set {0, Q1. QQ,....,,,... and for span-corruption incoder-decoder model, we just use their text form "O, Q1. .... 01, to represent the label.
As shown in Fig 1, depending on the sizes of question and answer unions, e.g., UQ UAi in the certain QA pair, we categorize the DialogQAE task into four types: 1 I-to-N, N-to-1 and N-to-N, in which the former and latter numbers indicate the size of question and answer unions respectively.
3 Methodology
3.3 Two-stage QA Extraction
3.1 Tagging as Extraction
Instead of predicting the label of utterance in single round, we could decouple the process into two steps (Moryossef et al., 2019; Liu et al., 2019, 2021), 1), which firstly figures out questions then ex- tract corresponding answers. We illustrate the two- stage workflow in Figure 6: in the first stage, the
The prior mainstream research on dialogue QA matching are based on the text segment alignment, such as matching the specific answers with the
In equation and 2, the upper indexes, i.c. j. of giax and Oie are omitted for simplicity.

---

Output
Dialog QA-Pair Extractor
Input
U,
[MASK]
[SEP]
U2
[MASK]
[SEP]
U3
[MASK]
[SEP]
U4
[MASK]
[SEP]
Figure 2: The model workflow for the 'fillin-the-blank style one-stage dialog QA pair extraction in the end-to-end fashion. U1 Qu.2, Ag and O represent dialogue utterances, questions, answers and not not-Q-or-A utterances.
model imput is the same as the end-to-end QA ex- traction (Sec 3.2), however the dialogue question extractor only predicts lstggel {0, Q1...., to determine whether each utterance the dialogue session is a question or not. In the second stage, we fill in the labels where the stage-1 model predicts questions. Then we feed the filled utterances sequence to the dialog answer extractor, which pre- diets the remaining utterances within the label set lstage2 {0, A.......
versations between doctors and patients. EduQA. CarsaleQA and ExpressQA, as indicated by their names, come from real-world conversations in the education, carsales and express delivery domains. As shown in Table 1, EduQA, CarsaleQA and Ex- pressQA are composed exclusively of 1-1 QA pairs while CSDS and MedQA involve 1-N, N-1 and N-N mappings in the extracted QA pairs.
decide whether they are the answer Aj to the question Qj-
4.2 Evaluation Metrics
To evaluate the performance at the utterance level, apply the traditional precision (P), recall (R) and Fi metrics, which ignore the non-QA label "O":
Moreover, the I-to-N (including 1-to-1) sce- nario, where question covers only single utter- ance, we could further break down the question labeling process in context-less way. As shown in Figure 7, at stage 1, separately perform binary classification {Q, O} for each utterance with the input format of [CLS] U.", where [CLS] is the classification token, and at stage we relabel those predicted as Q in the sequential order Q1, Q2.... to fill in the blank, and then apply the same dialog answer extracting strategy.
Ex Eg Pred P Oredd [Re
P
EnsTen Eieien Pred
EiE, "I-Inee Reef
to Preff Reff
(3)
R ΣiSIen Eisjsnt) Raf
F1 2 P
P+R
4 Experiments
where N the number of instances, and Pred Re denote the prediction and reference label sequences of the i-th instance.
4.1
Datasets
We conduct the experiments on 5 Chinese customer service multi-turn dialogue datasets, namely CSDS (Lin et al., 2021), MedQA (Jia et al., 2020), EduQA, CarsaleQA and ExpressQA. CSDS and MedQA two public dialogue chatlog datasets while the lat- ter three datasets are internal datasets which are accumulated through genuine customer-agent in- teractions in a commercial industrial dialogue plat- form. CSDS is derived from JDDC (Chen et al., 2020a) corpus and tailored for dialogue summa- rization where n-to-n QAs also provided as the clues for the summaries. MedQA is accumu- lated on medical QA platform² that covers con-
Similarly, for QA-pair level evaluation, we pro- pose adoption rate (AR), hit rate (HR) and session FI (S-FI):
AR Σ,E, |R (_preed n R_ref(i
EisIen R preddll
HR Σ. E, ( R
(4)
I
2 H AR
S-F1 = HR + AR
where (_pred()), R denote the prediction and reference QA-pair set of the i-th instance.
hhttps: /ww..//wwww.oo///////// 120ask.

---

Ratio of QA Pairs( %)
Dataset
#Sess
Avg Us
Avg_ Qs
Avg As
Dist QA
1-1
1-N
N-1
N-N
CSDS
9100
25.99
2.26
2.52
6.88
34.43
22.58
15.61
27.41
MedQA
700
36.46
9.73
10.78
2.19
70.95
29.05
0
EduQA
3000
10.63
1.29
1.27
2.23
100
0
0
CarsaleQA
3172
10.71
0.21
0.14
1.36
100
0
0
0
ExpressQA
5000
14.13
0.57
0.41
2.57
100
0
0
0
Table I: Dataset statistics. We list the number of sessions (#Sess), the average number of utterances (Avg_Us), questions (Avg_Qe Qs) and answers (Avg_As) in each session. We also figure out the average distances between the starting and ending utterance within QA pair (Dist_QAs), which signifies the (minimal) context required for successful QA extraction, as well as the ratio of 1-1, I-N, N-1, N-N QA pairs in each datasets.
From the perspective of FAQ database popula- tion by extracting QA pairs from the customer vice chatlog, the predicted QA pairs would serve as an automated module in the workflow, followed by the human verification. The adoption rate (AR) corresponds to the ratio of "accepted" QA pairs by human judges within the predicted QAs, which is analogous to the utterance-level precision. The hit rate (HR), the other hand, signifies the pro- portion of predicted QAs in all annotated QAs within the dialogue session which corresponds to the utterance-level recall.
Analysis and Discussions
5.1 Baseline Performance
Table and illustrate the utterance-level and session-level performance of QA extraction on the MedQA and CSDS datasets respectively. For both end-to-end and two-stage models, enlarging the model parameters leads to considerable per- formance gain, which indicates that the dialogue session encoders with higher capacity of vital importance for extracting QA pairs. In terms of the comparisons between the end-to-end and two- stage models, we observe that the two-stage mod- els outperform end-to-end models on the MedQA dataset while it is the other way around on the CSDS dataset, which shows that end-to-end meth- ods are more favorable in the N-to-N QA extraction that requires reasoning over longer dialogue con- text, such CSDS (6.88 in Dist QA and 65.57% non-l-to-1 QAs as shown in Fig 1), presum- ably complex Q-A mapping exaggerates the error propagation of aligning the potential answers to the given predicted questions the two-stage models. For the model performance on the N-to-N mapping shown in Fig 3, expect, the models get higher scores 1-to-1 mapping than N-to-N mapping,
4.3 Experimental Settings
We experiment with variety of pre-trained mod- cls via Hugging Face Transformers (Wolf et al., 2020), which encoder-decoder model mT5 (Xue et al., 2021) with three different parameter scales, namely T5-base (580M), T5-large (1.2B), T5-xl (3.7B), and encoder-only model including wewee-et (110M), chinese wwm-ext (1 10M), chineserrobera--wwm----llag (330M) (Cui et al., 2020), (304M)* 4, Bilaangshhen-MegtrnnnBBlee the backbones for the end-to-end and two-stage models. For contextless question classification and question-answe answer matching, we use wwm-ext-large (330M) (Cui et al., 2020). We use the Adam optimizer (Kingma and Ba, 2015) with the learning rate 3e-5 and train the models for at most 9 epochs on 4 Nvidia A100 GPUs.
-hinese--neeggee (1.3B) S, as
We also highlight the comparison between the generative (mT5-style, 'Gen') and the discrimina- tive (BERT-style, 'Tag") models in Table and 3, We observe that with comparable pre-trained model size, i.e. DeBERTaa-lrgee (304M) versus mT5-base (580M) and MegatronBERT (1.3B) versus mT5- large (1.5B), generative models perform better on the CSDS dataset while discriminative models win the MedQA dataset, showing that T5 models might be a promising option on the dialogue anal- ysis with long context (Meng et al., 2022). We believe that model size an important factor performance, since intuitively model with more
hhtpsss//hugiingaaceccoo// Saberta-Chinese-LLe
Co///// ww/.OO/--CCCCOA Erlangghen-Mgggarrneer---1-.1

---

Utterance Level (%)
Session Level %)
Training Strategy
Base Model
P
R
Fl
AR
HR
S-F1
End-to-Ead (Gen)
mT5--baee
79.00
86.19
82.44
48.11
50.21
49.13
End-to-End (Gen)
mT5--lare
87.90
91.69
89.75
66.77
68.99
67.86
End-to-End (Gen)
mT55xl
92.39
93.09
92.74
75.63
77.41
76.51
End-to-End (Tag)
BERT-base
79.85
81.58
80.70
48.85
48.10
48.47
End-to-End (Tag)
RoBERTa-base
80.73
83.45
82.07
$2.05
52.05
52.05
End-to-End (Tag)
RoBeRTa-large
88.32
89.17
88.75
65.10
65.50
65.30
End-to-End (Tag)
DeBERTa-large
88.52
89.91
89.21
66.55
67.51
67.02
End-to-End (Tag)
MegatronBERT
89.82
90.73
90.27
71.16
70.79
70.97
Two-Stage (G+G)
mTS-
82.74
89.62
86.04
56.81
58.88
57.83
Two-Stage (G+G)
mT5-larre
88.86
93.05
90.90
70.06
72.07
71.05
Two-Stage (G+G)
mT5-xl
92.86
92.97
92.91
77.62
78.70
78.15
Two-Stage (B+G)
mTS--base
82.24
89.64
85.78
57.83
58.21
58.02
Two-Stage (B+G)
mT5-large
88.32
92.13
90.19
70.63
71.10
70.86
Two-Stage (B+G) mT5-xl
92.85
91.96
92.40
77.21
77.72
77.46
Table 2: The benchmark for the QA extraction task on the MedQA dataset. The discriminatively (BERT-style) and generatively (mTS-style trained end-to-end models are abbreviated as "Tag' and 'Gen'. The 2 variants (B+G and G+G) of two-stage models differ in the model formulation of the first stage, i.e. binary classifier (Fig 6) versus mTS-style generative (Fig 7) model. We highlight the winner in each training strategy and the best scores with boldface and underlined marks.
Utterance Level (%)
Session Level %
Training Strategy
Base Model
P
R
F1
AR
HR
S-FI
End-to-End (Gen)
mT5-base
82.54
54.67
65.77
20.74
23.64
22.10
End-to-End (Gen)
mT5-large
84.38
57.55
68.43
22.04
26.86
24.22
End-to-End (Gen)
mT5-xl
84.61
57.15
68.22
22.98
26.53
24.63
End-to-End (Tag)
BERT-base
84.98
47.30
60.77
18.77
20.00
19.37
End-to-End (Tag)
RoBERTa-base
86.86
44.72
59.04
18.18
18.81
18.49
End-to-End (Tag)
RoBERTa-large
86.63
44.91
59.15
18.30
19.49
18.88
End-to-End (Tag)
DeBERTa-Large
83.95
45.91
59.36
19.61
19.41
19.51
End-to-End (Tag)
MegatronBERT
84.00
51.49
63.84
19.76
20.76
20.25
Two-Stage (G+G)
base
77.39
52.89
62.84
19.15
18.73
18.94
Two-Stage (G+G)
mT--large
80.45
56.17
66.15
20.32
22.71
21.45
Two-Stage (G+G)
mT5-xl
83.77
54.02
65.68
22.46
24.32
23.35
Table 3: The benchmark for the QA extraction task the CSDS dataset. Note that two-stage (B+G) models (Table 2) are incompatible with CSDS as the binary classifier tailored for 1-to-1 and I-to-N extraction (Sec 3.3).
parameters would fit the data better, and yet dis- criminative models with Masked Language Model pre-training task may not enjoy the same scaling law (Hoffmann et al., 2022) the generative mod- els do.
those methods are specific to the predefined se- mantic/information schema ontology, i.e.. dis- course, dependency, AMR parsing trees (Xu al., 2021, 2022), dialogue actions event/entity labels (Liang et al., 2022). Through the analysis of the extracted QA pairs of a dialogue session, we sum- marize a more general schema to categorize the dialogue structure according to the customer-agent interaction in the dialogue flow.
5.2 Dialogue Structure Analysis
Prior research tried to extract and analyze the struc- of given dialogue session through latent di- alogue states (Qiu et al., 2020), discourse pars- ing (Galitsky and Ilvovsky, 2018) or event ex- traction (Eisenberg and Sheriff, 2020). However,
Fig demonstrates the typical 'between-QA. pairs' relations based on the extracted QA map- pings. The most common case is Sequential

---

QA Flow, where Position(Al) < Pos) and Role(Q1) Role(Q2): in this case, one complete QA pair is after another. For Follow-up Informa- tion Seeking, here Position(A I) < (Q2) but Role(QI) + Ro)e(Q2), indicating the answer leads to new question. For claboration/Detallnng Posi- tion(Q2) < Position(AI) and Role(Q1) Ro)e(Q2). which means one person asked two questions in row, and in turn, the other answered consecutively. the example, the doctor sequentially answers the consecutive questions raised by the patients, with the second answer claborating on the first one. For Clurfication/Connfimtiio Position((2)) < Position(AA) and Role(Q1) + Role(Q2) and Po- sition(A2) < Position(A1) which implies the first question can not get the answer yet, and more in- formation is needed from the questioner; vided, the first question can finally be answered correctly. In the example, the doctor asked clari- fication question on when the symptoms occurred after the inquiry of the patient instead of answer- ing the inquiry instantly. For Barge-in/Interruttonn, which is not common, is the case of Position(Q2) < PositionnAN) 1) and Role(QI) = Role(Q2) and Po- (At) Position(A)) (A1), where the second ques- tion is answered first. As shown in Fig the QA extraction models perform better on SF, FIS, and BI than CC and ED, presumably the interleaving QA pairs pose a bigger challenge to the dialogue information extraction.
Dataset
Carsale
Express
Edu
Avg.
Carsales
70.89
40.78
60.73
57.47
Express
54.39
86.33
52.91
64.54
Edu
74.05
47.52
86.31
69.29
80.43
83.41
85.96
83.27
Table 4: The domain adaptation dialogQAE models on BduQA, CarsaleQA and ExpressQA. We report the utterance Fl scores (%) of the End-to-end (Gen, mT5- xl) models.
Domain
P
AR
MultDDOGO, _arline
91.30
86.40
MultiDOGO_ fastffoffoooodo
91.04
85.05
MuliDOGO finance
83.71
81.50
MultiDOGQ_inurri
93.17
90.00
MultiDOGO _acii
86.75
83.02
pro-
MultiDOGO_softwar
87.00
84.91
Table 5: The illustration for the language transfer of the End-to-end (Gen, m mT5xxl) model trained on MedQA. We abbreviate the precision and adoption rate scores in the utterance and session level as 'P' and 'AR'. The scores correspond to the accuracy of the predicted QA pairs, according to the human judges.
correlation between different domains is the key factor of the model performance the domain transfer, e.g. the bidirectional transfer between
the carsale and the education domains gets higher scores than other domain pairs.
We delve into the relative position of the ques- tion and answer utterances within an N-to-N QA pair in Fig 5. Most questions and answers are disjoint within a QA pair while the overlapping questions and answers account for 26.91% % in the CSDS dataset. We take further step to split the overlapping QAs into two circumstances: in-pair Q-A and in-pair Q-A-Q, depending the role (Q A) of the last utterance in the QA pair. As il- lustrated in Fig 3, all three QAE models perform better on the disjoint QA pairs than overlapping ones.
Thanks to the multilingual nature of mT55, the models trained on the Chinese datasets can be easily applied to datasets in other languages, e.g. English. We test the Chinese DialogQAE model different domains of the MultiDOGO dataset (Peskov al., 2019). As the MultiDOGO dataset does not have Q-A-pair annotations, we ask the human annotator to decide whether the recognized QA pairs by the MedQA-Dialog eligible according to the semantics in the dialogue flow. We use majority votes among 3 human judges and the inter-annotator agreement (the Krippen- dorf's alpha) is 0.89.
model are
5.3 Domain and Language Adaptation
Related Work
We illustrate the domain and language adaptation our dialogQAE models Table and respec- tively, which highlight the real-world utility of our models.
6.1 QA Extraction
For text-based QA extraction, Rajpurkar al. (2016) proposed the dataset SQuAD 1.1, in which the 100k+ questions were created by crowdwork- ers on 536 Wikipedia articles. Subsequently, Du and Cardie (2018) created IM+ paragraph-level
In Table 4, we observe that mixing the datasets from different domains is a simple but effective way to boost the overall performance. The potential

---

MedQA
CSDS
Model
90
95
G+G
Tag
Recall
90
70
50
FIS
B1
CC Type
ED
1-1
1-N QA
QAQA
QAQ
1-1 Type
1-N
N-1
N-N
Figure 3: Barplot of recall group by the dialog structure type, on the MedQA and CSDS datasets respectively. Here, we use End-to-Ente (Gennn,GGnn), mTT5-xl model. SF, FIS, BI, CC, ED refer to Sequential QA Flow, Follow-up Information Seeking, Barge-in/Inerrruttoon. refer to Disjoint In-pair Q-A, Overlap In-pair Q-A, Overlap In-pair Q-A-Q, and 1-1, I-N, N-1, N-N refer to 1-to-1, 1-to-N, N-to-1, N-to-N QA matching.
Surrfiaation/ooniimman
Elaboration/detailing QA, QAQA, QAQ
question-answer pairs over 10,000 Wikipedia arti- cles. For question generation, Yang et al. (2017) use a trained model to generate questions unla- beled data. Later, Wang al. (2019) proposed to identify key phrases first and then generate ques- tions accordingly. For machine reading comprehen- sion (MRC), research on dialogues MRC aims to teach machines to read dialogue contexts and make response Zeng et al. (2020) aims to answer the ques- tion based on a passage as context. Shinoda et al. (2021) leveraged variational question guestion-answer answer pair generation for better robustness MRC. However, extraction methods that can work 1-1, I-N, and N-N scenario is under-explored.
cuses on extracting relations between speakers and arguments in dialogue. DialogRE is an English dialogue relation extraction dataset, consisting of 1788 dialogues and 36 relations. MPDD (Chen et al., 2020b) is Multi-Party Dialogue Dataset built on five Chinese TV series, with both emo- tion and relation labels on each utterance. Long et al. (2021) proposed a consistent learning and inference method for dialogue relation extraction, which aims to minimize possible contradictions. Fei et al. (2022) introduced a dialogue- mixed dependency graph. Shi and Huang (2019) proposed deep sequential model for discourse parsing multi-party dialogues. The model predicts depen- deney relations and constructs a discourse structure jointly and alternately.
6.2 Dialogue Analysis
For dialogue information extraction (IE), in order to save the efforts of the assessor in the medical insurance industry, Peng et al. (2021) proposed a di- alogue IE system to extract keywords and generate insurance reports. To figure out the semantics in the dialogue flow, Galitsky and Ilvovsky (2018) pro- posed dialogue structure- building method from the discourse of questions. Qiu et al. (2020) incorporated structured attention into Variational Recurrent Neural Network for dialogue structure induction in an unsupervised way. Eisenberg and Sheriff (2020) introduced a new problem, extract- ing events from dialogue, annotated the dataset Personal Events in Dialogue Corpus, and trained support vector machine model.
Conclusion
In this paper, we propose N-to-N question and (QA) pair extraction from customer vice dialogue history, where each question or an- swer may involve more than one dialogue utterance. We introduce a suite end-to-end and two-stage tagging- based methods that perform well on 5 cus- tomer service datasets, as well as utterance and session level evaluation metrics for DialogQAE. With further analysis, find that the extracted QA pairs characterize the dialogue structure, e.g. information seeking, clarification, barge-in, and elaboration. Extensive experiments show that the proposed models can adapt to different domains and languages and largely accelerate knowledge accumulation the real-world dialogue platform.
sttucturebbildding
Relation Extraction over Dialogue is newly de- fined task by RE (Yu et al., 2020), which fo-

---

Limitations
Bowen Zhou. 2020a. The JDDC corpus: A large- scale multi-turn Chinese dialogue dataset for E- commerce customer service. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 459-466, Marseille, France. European Language Resources Association.
This work focuses on the N-to-N question and an- swer extraction from a dialogue session and does not touch the relevant tasks such as question gener- ation (e.g. Du et al., 2017; Duan et al., 2017) and dialogue summarization (Lin et al., 2021). The pro- posed task can be seen as a preparation for the sub- sequent tasks by decomposing the entire procedure of question generation and dialogue summariza- tion into two steps: extraction before generation. The extracted QA pairs can also be further pro- cessed in order to visualize some important factors for customer service, like common customer con- cerns about the products, winning sales scripts to persuade the customers and emerging or trending user intents (Liang et al., 2022), by a set of atomic natural language processing modules like keyword extraction, sentiment analysis and semantic parsing and clustering.
Yi-Ting Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020b. MPDD: multi-party dialogue dataset for analysis of emotions and interpersonal relationships, In Proceedings of the 12th Language Resources and Evaluation Conference, pages 610-614, Marseille, France. European Language Resources Association.
Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chao- qun Duan, and Ming Zhou. 2017. SuperAgent: A customer service chatbot for E-commerce websites. In Proceedings of ACL 2017, System Demonstrations, pages 97-102, Vancouver, Canada. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657-668, Online. Association for Computa- tional Linguistics.
Ethics Statement
The internal datasets we used in this paper, i.c. EduQA, CarsalesQA and ExpressQA, have gone through strict data desensitization process, with the guarantee that no user privacy any other sensitive data is being exposed by a hybrid of automatic and human verification. Human veri- fication also eliminates the dialogue sessions with gender/ethnic biases or profamitics. The other two datasets, CSDS and MedQA, are publicly available and we use them with any modification. The model for extracting questions and answers in the dia- logue paves the for N-to-N dialogue QA extraction, without any risk of violating the EMNLP ethics policy.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171 1171-4166. Minneapolis, Minnesota. Association for Computational Linguistics.
Xinya Du and Claire Cardie. 2018. Harvest- ing paragraph-level question-answer pairs from Wikipedia. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1907-1917, Mel- bourne, Australia. Association for Computational Linguistics.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume Long Papers). pages 1342 1352. Vancouver, Canada. Association for Computational Linguisties.
References
Mikhail Burtsev, Alexander Seliverstov, Rafael Airapetyan, Mikhail Arkhipov, Dilyara Baymurz- ina, Nickolay Bushkov, Olga Gureenkova, Taras Khakhulin, Yuri Kuratov, Denis Kuznetsov, Alexey Litinsky, Varvara Logacheva, Alexey Lymar, Valentin Malykh, Maxim Petrov, Vadim Polulyakh, Leonid Pu- gachev, Alexey Sorokin, Maria Vikhreva, and Marat Zaynutdinov. 2018. DeepPavlov: Open-source li- brary for dialogue systems. Proceedings of ACL 2018, System Demonstrations, pages 122-127, Mel- bourne, Australia. Association for Computational Linguistics.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 866-874, Copenhagen, Denmark. Association for Computational Linguistics.
Joshua Eisenberg and Michael Sheriff. 2020. Automatic extraction of personal events from dialogue. In Pro- ceedings of the First Joint Workshop on Narrative
Meng Chen, Ruixue Liu, Lei Shen, Shaozu Yuan, Jingyan Zhou, Youzheng Wu, Xiaodong He, and

---

Understanding, Storylines, and Events, pages 63-71, Online. Association for Computational Linguistics.
A ane-grained Chinese dataset for eustomer service dialogue summarization. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 4436-4451, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hao Fei, Jingye Li, Shengqiong Wu, Chenliang Li, Donghong Ji, and Fei Li. 2022. Global inference with explicit syntactic and discourse structures for dialogue-Ievel relation extraction. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4107-41 13. International Joint Conferences on Artificial Intelli- gence Organization. Main Track.
Ruixue Liu, Meng Chen, Hang Liu, Lei Shen, Yang Song, and Xiaodong He. 2020. Enhancing multi- turn dialogue modeling with intent information for e-commerce customer service In CCF International Conference on Natural Language Processing and Chinese Computing, pages 65-77. Springer.
Boris Galitsky and Dmitry Ilvovsky. 2018. Building dialogue structure from discourse tree of question. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on
Tianyu Liu, Fuli Luo, Pengeheng Yang, Wei Wu, Baobao Chang, and Zhifang Sui. 2019. Towards comprehensive description generation from factual attribute-value tables. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 5985-5996, Florence, Italy. Asso- ciation for Computational Linguistics.
Conversational AI, pages 17-23, Brussels, Belgium. Association for Computational L Linguistics.
Matthew Henderson, Ivan Vulié, Daniela Gerz, lñigo Casanueva, Pawet Budzianowski, Sam Coope, Geor- gios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić. and Pei-Hao Su. 2019. Training neural response selection for task-oriented dialogue systems. Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5392- 5404, Florence, Italy, Association for Computational Linguistics.
Tianyu Liu, Xin Zheng, Baobao Chang. and Zhifang Sui. 2021. Towards faithfulness in open domain table- to-text generation from an entity-centre Thirty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2021, Thirty- Third Conference on In- novative Applications of Artificial Intelligence, IAAI 2021. The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence FAAI 2021, Vir- tual Event, February 2-9, 2021, pages 13415-13423. AAAI Press.
view. In
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford. Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bog- dan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Frich Flsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. 2022. Au empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Sys- tems.
Xinwei Long, Shuzi Niu, and Yucheng Li. 2021. Con- sistent inference for dialogue relation extraction. In Proceedings of the Thirtieth International Joint Con- ference on Artificial Intelligence, IJCAI-21, pages International Joint Conferences on Arti- ficial Intelligence Organization. Main Track.
Haoran Meng, Zheng Xin, Tianyu Liu, Zizhen Wang. He Feng, Binghuai Lin, Xuemin Zhao, Yunbo Cao, and Zhifang Sui. 2022. Dialogusr: Complex dialogue utterance splitting and reformulation for multiple in- tent detection. arXiv preprint arXiv:2210.1 1279.
Qi Jia, Mengxue Zhang, Shengyao Zhang, and Kenny Q. Zhu. 2020. Matching questions and answers in di- alogues from online forums. In ECAI 2020 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela. Spuin, August 29- September 8. 2020- Including 10th Conference on Prestigious Applications of Arti- ficial Intelligence (PAIS 2020), volume 325 of Fron- tiers in Artificial Intelligence and Applications, pages 2046-2053. IOS Press
Amit Moryossef, Yoav Goldherg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pages 2267-2277. Minneapolis, Min- nesota. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Pêteris Paikens. Artürs Znotinš, and Guntis Bärzdinš. 2020. Human-in -n-tt----t-h--pppp conversation agent for cus- tomer service. In International Conference on Appli- cations of Natural I Language to Information Systems, pages 277-284. Springer.
Hua Liang, Tianyu Liu, Peiyi Wang, Mengliang Rao, and Yunbo Cao. 2022. Smartsales: Sales script extraction and analysis from sales chatlog. arXiv preprint 220.x2224.088811
Shuang Peng, Mengdi Zhou, Minghui Yang. Haitao Mi, Shaosheng Cao, Zujie Wen, Teng Xu, Hongbin Wang. and Lei Liu. 2021. A dialogue- information ex- traction system for medical insurance assessment. In
Haitao Lin, Liqun Ma, Junnan Zhu, Lu Xiang. Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2021. CSDS:

---

Findings of the Association for Computational Lin- guisties: ACL-IJCNLP 2021. pages 654-663 663, Online. Association for Computational Linguistics.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush, 2020. Trans- formers: State-of- -hee-----htheeeeee art natural language processing. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing: System Demonstrations, pages 38 38-45, Online. Association for Computational Linguisties.
Denis Peskov, Nancy Clarke, Jason Krone, Brigi Fodor, Yi Zhang, Adel Youssef, and Mona Diab. 2019. Multi-domain goal-oriented dialogues (MultiDoGO): Strategies toward curating and annotating large scale dialogue data. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4526-4536, Hong Kong, China. Association for Computational Linguistics.
Yu Wu, Wei Wu, Chen Xing. Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new archi- tecture for multi-turn response selection in retrieval- based chathots. Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Pupers). pages 496-505, Vancouver, Canada. Association for Computational Linguistics.
Liang Qiu, Yizhou Zhao, Weiyan Shi, Yuan Liang. Feng Shi, Tao Yuan. Zhou Yu, and Song-Chun Zhu. 2020. Structured attention for unsupervised dialogue struc- ture induction. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLPP), pages 1889-1899, Online. As- sociation for Computational Linguistics.
Runxin Xu, Tianyu Liu, Lei Li, and Baobao Chang- 2021. Document-level event extraction via heteroge- neous graph graph-based interaction model with tracker. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Pupers), pages -3436,546, Online. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages -3932,39,, Austin, Texas. Association for Computational Linguistics.
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, al. 2018. Conversational ai: The science behind the alexa prize. arXiv preprint 101.:880003604
Runxin Xu, Peiyi Wang, Tianyu Liu, Shuang Zeng. Baobao Chang, and Zhifang Sui. 2022. A two-stream MMM-eenhaneed model for document- gument extraction. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 5025-5036, Seattle, United States. Association for Computational Lin- guistics.
event ar-
Zhouxing Shi and Minlie Huang. 2019. A deep se- quential model for discourse parsing on multi-party dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7007-7014.
Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa. 2021. Improving the robustness QA models to challenge sets with variational question-answer pair generation. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Nat- ural Language Processing: Student Research Work- shop, pages 197-214, Online. Association for Com- putational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale. Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-traine trainedddd text- text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Anna Tigunova, Paramita Mirza, Andrew Yates, and Gerhard Weikum. 2021. PRIDE: Predicting Rela- tionships in Conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4636-4650, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-supervised QA with generative domain-adaptive nets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1040-1050, Vancouver, Canada. Association for Computational Linguistics.
Siyuan Wang, Zhongyu Wei, Zhihao Fan, Yang Liu, and Xuanjing Huang. 2019. agent communica- tion framework for question-worthy phrase extraction and question generation. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01)::166- 7175.
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020. Dialogue- relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4927-49400 Online. Association for Computational Linguistics.

---

Chunyuan Yuan, Wei Zhou, Mingming Li, Shangwen Lv, Fuqing Zhu, Jizhong Han, and Songlin Hu. 2019. Multi-hop selector network for multi-turn response selection in retrieval-based chatbots. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internu- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 111- 111-120, Hong Kong, China. Association for Computational Lin- guistics.
Changchang Zeng, Shaobo Li. Qin Li, Jie Hu, and Jian- jun Hu. 2020. A survey on machine reading compre- tasks, evaluation metries and benchmark datasets. Applied Sciences, 10(21):7640.
Xiangyang Zhou, Lu Daxiang Dong. Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-tun response selection for chatbots with deep attention matching network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1 1118-1127, Mclboume, Australia, Association for Computational Linguistics.

---

A Appendix
A.1 Model Performance on the Internal Datasets
We show the model performance on the internal datasets, i.e. EduQA, CarsalesQA and ExpressQA in Table 6. In terms of comparison between end- to-end and two-stage models, end-to-end models are clear winner with respect to session level on the Carsales and EndQA datasets, while two-stage models take the lead on the ExpressQA dataset. Ac- cording to the dataset statistics in Table 1, we guess this is because ExpressQA has longer dialogue ses- sion (14.13 13 for the average number of utterances) and require longer context (2.57 versus 2.23/1.36 in Dist QA) for extracting QA pairs.
The DialogQAE models have been deployed in a commercial platform for conversational intelli- gence. The module serves as an automatic dialogue information extractor, followed by human verifica- tion and modification on the extracted QA pairs so that they can serve standard and formal FAQs in the customer service. According the user feed- backs from the online customer service department an international express company, the assistance of dialogue QA extraction has largely accelerated the information enrichment for customer service FAQs, reducing from around 8 days per update to 2 days per update.
A.2 Between-QA-Pairy Relations Examples
We show more examples on the dialogue structure from the MedQA datasets below.

---

Fallow-up up Sneking (14.29%)
Barge aege-inn/lneen in/lnntrruuptiinn (1.80%)
Sequential (58.95%)
DIDeU
Foors snypiing wming with your nedd?
77 %
UID-u
Hollo, you have your liv landion
16-87 N:- da 15
UIDeU
+ 91% I- & H 10-0033
chockes dasing your phypacal nation?
UlDmu
UID=u
Wl & ly C. 7
Ag
UIDeu
& 6 there a stomach hally
Cencoa
UIDeU
Whhll deparment thoued go foriiin
E R 8-th H4-
UID-U
Gei my seoand shot lad wook.
J. 4 I =
UlD-uy
A 2 T-42 T
24 1/
UID-U
Mt &
UID+L
pannal Jud Moniee
Carifictton/Confimaatoo (12.34%)
Elaboration/Detaily (12. 62%)
&
Icon: Speaker//Speaken SpeakerB, can be ether customer
agent
A R. +- Bi
-T 10.00 8f
UIDEO
Now ihe chaal rightnes and weiting
UID-U
Can cored?
Box: questoon/anewerr
unions, might comprise =1 utterances.
DIDmu
prnco occur?
UlDmu
& Whee
UID=a
UID u/iu, We use the 1si ulterance in the union to represent its position the conversational
2f L. R When Woro noming
UlDEB
san cared
QA Tag: be tagged as ather A, no maller what speaker's role is. Explonation for the illustrations utterances
Q
utterance (union) can
UID-a
T F
fo dast and coicurr cupolomntttinn
Figure 4: The demonstration for the Sarveen-QA-piiee relations and their proportion in the MedQA dataset. Given snippet of consecutive dialogue utterances (for UIDs, < U2 Mg w4), we roughly categorize the dialogue flows into different types according to the between- QA interactions between customers and human agents.
Disjoint In-pair Q-A (83.09%)
Overlap In-pair Q-A (14.70%)
Overlap In-pair Q-A-Q (2.21%)
1E #1 Sill new or codoed?
QL
UlDeu
where the locdion of looal
UlD-u,
UllDeus
How opply?
UIDmu,
&
funy, con ck up mesott
it 42 95
A
UID-u:
UID:u
19 #- 22.8 + th onder
apsh
UIDeu,
Halli cantl the vursissan.
UID=u,
54-4-2 16 trsh??
UID-U,
Aller ft-te.tt
UlDeu,
12 A-E ti (2 t5 You cwn up delivery adke.
UID-u
the moneyy be returned mo??
Figure 5: The demonstration of the in QA-pair relations and their proportion in the CSDS dataset. According to the relative positions Q and utterances in n-to-n QA pair, we categorize the interleaving utterances into types.
Utterance Level %)
Session Level (%)
Dataset
Training Strategy
P
R
F1
AR
HR
S-F1
CarsalesQA
End-to-End (Gen)
83.67
61.50
70.89
42.11
44.44
43.24
EduQA
End-to-End (Gen)
88.96
83,81
86.31
55.50
65.98
60.29
Express
End-to-End (Gen)
96.26
78.26
86.33
63.46
71.74
67.35
CarsalesQA
Two-Stage (G+G)
85.96
73.13
79.03
43.31
53.54
47.89
EduQA
Two-Stage (G+G)
98.44
69.23
81.29
59.62
67.39
63.27
ExpressQA
Two-Stage (G+G)
92.99
76.54
83.97
59.66
67.31
63.25
Table 6: The performances of the end-to-end and two-stage models (mTS--large) for the QA extraction task on the internal datasets.

---

Role
Utterance
Patient
felt dizzy for three days in row. My whole body fluttered, but just fine after a night of sleep. What's going on?
Doctor
Stiff necks may cause this problem.
Doctor
Has there been an inspection? 2
Patient
No.
Doctor
Are there any other symptoms?
Patient 5 can't sleep well.
Patient
3 ni ?
Can wipe myself with alcohol?
Doctor
Wipe with warm water.
Doctor
?
Is it itchy?
Patient
T
Not itchy
Patient
How to treat it?
Doctor
The ( 1 13 HR #
RR. K S
- bb
Pay attention to your diet (Fat more vegetables and fruits. Drink more water. Eat less fatty, sweet and spicy food), and don't stay up late. Wear less makeup. Treatment: external application of fusidic acid cream (once during the day) and adapalene gel (once before bedtime night). Orally take tanshinone capsules.
Table 7: Follow-up Information Seeking examples MedQA
Stage Output
Dialog Answer Extractor
Stage 2
U1
[SEP]
U2
[SEP]
[MASK]
[SEP]
U4
[MASK]
[SEP]
Input
Dialog Question Extractor
[MASK]
[SEP]
[MASK]
[SEP]
[MASK]
[SEP]
[MASK] [SEP]
Input
Figure 6: The model workflow for the two-stage QA extraction. The first stage extract questions while the second stage corresponds to answer extraction given the predicted questions. Hereby we only illustrate the " style question extractor, where the binary question classifier shown Fig 7.

---

Role
Utterance
Doctor
What medicine have you taken?
Doctor
Are you still taking them?
Patient
ME'
Still taking.
Patient
Paracetamol, Pseudoephedrine Hydrochloride, Dextromethorphan Hydrobromide and Chlor- pheniramine Maleate Tablets
Doctor
x K T
How long has it been like this?
Doctor
$ How old are you?
Patient E+ET
I'm thirty-three.
Patient + It has been more than ten years.
Doctor
Does your baby have cough right now?
Doctor
Does have a fever?
Patient
No fever
Patient
Have cough.
Table 8: Barge-in/Interruppee examples in MedQA
Stage 2 Oulput
Dialog Answer Extractor
Stage Input
[SFP]
U2
[SFP]
Us
[MASK]
[SFP]
Uq
[MASK]
[SFP]
Dialog Question Classifier
Stuge Input
[CLS]
Uj
[CLS]
U2
[CLS]
Us
[CLS]
U4
Figure 7: The model workflow for the two-stage QA extraction where the first stage is the binary question classifier. The workflow only works for 1-to-1 1-to-N QA extraction.

---

Role
Utterance
Patient
2 Bi A 2f
Which department should go to the hospital check?
Doctor
Take cervical spine X-ray.
Patient
T & / 2
TM
Could it be a lack of sleep? felt OK after a good night of sleep, why?
Doctor
30 f |J 2 H ié Aig #51 Ifi @
A bad cervical spine compress the blood vessels of the brain.
Doctor
How is your appetite? Is there any noticeable change?
Patient
1f 2 [ Di
just have no appetite.
Doctor
TN H
Do you have taken a gastroscope?
Patient
Hi la - W
Previous examinations suggested chronic gastritis.
Doctor
AA $
What is the body temperature?
Patient
37
37 degree.
Doctor
* #
How is your mental statc?
Patient
ET
Not bad.
Table 9: Sequential QA Flow examples in MedQA

---

Role
Utterance
Patient
ft & ?
What the reason for the symptom?
Doctor
Which parts of your body have the symptom?
Patient
F. FAR 7
Hands and neck
Doctor
/ / Topically apply calamine lotion, and observe it.
Doctor
D & 2
Do you have any previous underlying diseases?
Patient
HE, 2 R H ?
Doctor, what is the underlying disease? tu it i Aa : 5 デ
Doctor
Cerebral infarction, chronic bronchitis, etc. A9, No, doctor.
Patient
Patient
I'm a mine-year-ee female. Why is my memory getting worse suddenly?
Doctor
i KHT π?
How long has it been like this?
Patient
1 # r
has been a week. i ス
Doctor
T
It's not big problem, don't worry too much.
Table 10: Clanfication/Connfmattiiin examples in MedQA

---

Role
Utterance
Doctor
Mii K [i] ??
How long have you had a stomach ache?
Doctor
- -
Is it constant pain bouts of pain?
Patient
ー
or three days
Patient
; Very uncomfortable pain.
Patient
- Does blood coagulation need to be checked? # Ee 7 fT lil A
Patient
Does it contain a coagulation test on these reports of mine?
Doctor
No need to check
Doctor
A No.
Doctor
Y E HÜ #
Have you ever had a backache before?
Doctor
Is the back pain related to the leg pain?
Patient
T # t Dj A
It doesn't hurt. Just after the ligament is strained, the waist and both sides hurt.
Patient
No leg pain.
Table 11: Elaboration/Detaaling examples in MedQA


Reviews:
[{"review_id": "cfjcEbDn9e", "reviewer": "Reviewer_VBi6", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and constructive feedback on our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tA well-structured and well-written paper.\n•\tThe topic is of research and applied interest.\n•\tThe experiments are well-designed and the results are analyzed.\n•\tA suite is proposed that set up a benchmark for N-to-N DialogQAE with utterance and session-level evaluation metrics \n•\tThe proposed models can adapt to different domains and languages, and reduce the labor cost of knowledge accumulation in the real-world product dialogue platform.\n", "reasons_to_reject": "-", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes an approach to deal with N-to-N QA extraction tasks in which the derived questions and corresponding answers might be separated across different utterances. The paper also introduces a suite of generative and discriminative tagging methods with end-to-end and two-stage architecture. The suitе is tested on 5 customer service datasets and set up a benchmark for N-to-N DialogQAE with utterance and session-level evaluation metrics. Analysis of extracted QA pairs shows that the relations between and inside the QA pairs can be indicators to analyze the dialogue structure, e.g. information seeking, clarification, barge-in, and elaboration. The proposed models are tested on dialogs from different domains and languages.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1FuYGvBtsm", "reviewer": "Reviewer_AycG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We would like to emphasize that:\n\n1. The technical contribution of our work\n\nWe leverage the feature of T5, span recovery, to address the task of QA extraction. Our method involves no extra modification of the original language model, making it easy to implement. It also echoes the current trend of converting most, if not all, Natural Language Processing tasks into Language Modeling. The success of our application stresses the importance and necessity of adding a span corruption / fill-in-the-blank task during pre-training, besides traditional pure left-to-right language modeling task, for better support of sequence-tagging NLU tasks. We will take your suggestion and fix the error in Figure 1.\n\nWe also would like to highlight that other main contributions include:\n\n- We set up a benchmark for DialogQAE with diverse baselines.\n- We show that DialogQAE is an effective paradigm for dialogue analysis.\n- We find that the proposed DialogQAE model effectively automates and accelerates the cold-start or upgrade of a commercial dialogue system.\n\n2. Regarding to the **Reproducibility of our work,** we would open-source the code, model and public-available dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Acquiring extra training data must be a common problem in the commercial chatbot world, and this paper provides a possibly useful method for acquiring it. \n\nThe paper is clearly written and easy to follow", "reasons_to_reject": "The paper does not offer anything very novel in terms of techniques, but it is nevertheless a useful contribution. I can't think of any strong reason why the paper should be rejected.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of acquiring question-answering dialog data from logs of customer service chatbots. The\nauthors propose to extend existing 1-1 question-answer pair extraction to a more general N-N setting. In the abstract,\nin two-party dialogue this involves going from a sequence of <utterance, role> pairs, to a set of <question, answer>\npairs where each member of the pair is a sequence of <utterance, role> pairs.\n\nThe paper experiments with two sequence labeling approaches to this task, one which operates end-to-end to label\nutterances as questions (Q), answers (A), or neither (O), and the second approach which does this in two stages, the\nfirst labeling as Q or O, and the second labeling the remaining utterances as A or O.\n\nIn both cases, a subsequent deterministic procedure groups the labeled utterances into larger <question, answer> clusters based on sequencing and contiguity.\n\nThe sequence labeling steps are carried out by various transformer-based systems, and the input is an entire\nconversational session.\n\nIn general, a large T5 based model was the strongest performer on the two datasets (of five real sets of commercially\ncollected data) they give detailed results for, on both the single and two-stage versions. Inspection of the results\nshowed that the extracted QA pairs exhibit different kinds of internal structure familiar from linguistic analysis of\ndialogues: clarification, barge-in, etc.\n\nThe paper concludes with a brief demonstration that the data acquired in this way improves QA performance on their datasets, and on others.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In figure 1 I think the symbol in A3 UID=26 should be the green system one."}, "extra_text": {}}, {"review_id": "UzqsegeEWT", "reviewer": "Reviewer_psJC", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review. We would like to clarify that:\n\n1. About N-to-N QA Extraction\n\nWe try to unify the question-answer pair extraction approaches, as previous work only focused on limited scenarios, i.e., 1-1 or 1-N, and N-to-N extraction has yet to be explored. Thus, one method may not be applied in other settings, and different settings would require different specific methods. As our method purely leverages the power of T5 model, training for 1-1, 1-N or N-N QA extraction involves no special modifications.\n\n2. About experimental results and baselines\n\nOur proposed unified QA Extraction is first-of-its-kind. It largely accelerates knowledge accumulation and updates in customer service applications, which has shown great impacts on real-world applications, including education and car sales.\n\nAs stated above, existing baselines in the prior work, which are also suitable for both 1-N and N-N scenarios, are unavailable. Therefore, our work provides diverse baselines, consisting of several variations and evaluates a wide range of encoder-only and encode-decoder models with different parameter sizes. We hope it can benefit future studies.\n\n3. About readability\n\nThe structure of the paper is as follows:\n\n1. Introduction: Provides an overview of the task of N-to-N question and answer (QA) pair extraction from customer service chatlogs. It discusses the importance of QA extraction for enriching knowledge bases and improving dialogue systems. It also highlights the challenges and contributions of the paper.\n2. Task Overview: Describes the task of N-to-N QA extraction and introduces the concept of exclusive dialogue utterances and speaker role consistency. It categorizes the task into four types based on the number of utterances involved in the extracted QA pairs.\n3. Methodology: Discusses the tagging-based methods for QA extraction. It explains the end-to-end and two-stage approaches and how they can be used for QA extraction. It also describes the input format and model architectures used in the experiments.\n4. Experiments: Provides details about the datasets used in the experiments and the evaluation metrics. It presents the experimental results for different models and discusses the performance of the models on different datasets.\n5. Analysis and Discussions: Analyzes the extracted QA pairs and discusses the dialogue structure and relations between QA pairs. It also explores the domain and language adaptation of the models.\n6. Related Work: Discusses related work in the areas of QA extraction and dialogue analysis, including research on text-based QA extraction, dialogue information extraction, and dialogue structure analysis.\n\nWe would revise our paper and clarify anything that is challenging to follow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset in this paper would be beneficial for researchers interested in conducting research on this particular problem.", "reasons_to_reject": "1.The N-to-N question answer pair extraction approach proposed in this paper is relatively limited. \n2.The experimental results show no significant improvement, and the paper lacks a comparison with other methods.\n3.This paper is not very clear to understand.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper primarily introduces the N-to-N question answering pair extraction approach and proposes the DialogQAE model to address this issue. The contribution is that the paper setups a benchmark for DialogQAE with end-to-end and two-stage baselines that support N-to-N QA extraction, as well as the utterance and session-level evaluation metrics.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the challenge of extracting question-answering dialogue data from customer service chatbot logs. The authors propose an extension of the existing 1-1 question-answer pair extraction to a more versatile N-N setting. Furthermore, the paper introduces a suite of generative and discriminative tagging methods, employing both end-to-end and two-stage architectures. These methods are evaluated on five customer service datasets. An in-depth analysis of the extracted QA pairs reveals valuable insights into the dialogue structure, including information-seeking, clarification, barge-in, and elaboration. The proposed models are subjected to testing on dialogues from diverse domains and languages. While reviewers acknowledged that the paper does not present groundbreaking techniques, it undeniably constitutes a valuable contribution to the field.
Decision: Accept-Findings