Towards Data Distillation for End-to-end Spoken Conversational Question Answering

1 INTRODUCTION . Conversational Machine Reading Comprehension ( CMRC ) has been studied extensively over the past few years within the natural language processing ( NLP ) communities ( Zhu et al. , 2018 ; Liu et al. , 2019 ; Yang et al. , 2019 ) . Different from traditional MRC tasks , CMRC aims to enable models to learn the representation of the context paragraph and multi-turn dialogues . Existing methods to the conversational question answering ( QA ) tasks ( Huang et al. , 2018a ; Devlin et al. , 2018 ; Xu et al. , 2019 ; Gong et al. , 2020 ) have achieved superior performances on several benchmark datasets , such as QuAC ( Choi et al. , 2018 ) and CoQA ( Elgohary et al. , 2018 ) . However , few studies have investigated CMRC in both spoken content and text documents . To incorporate spoken content into machine comprehension , there are few public datasets that evaluate the effectiveness of the model in spoken question answering ( SQA ) scenarios . TOEFL listening comprehension ( Tseng et al. , 2016 ) is one of the related corpus for this task , an English test designed to evaluate the English language proficiency of non-native speakers . But the multi-choice question answering setting and its scale is limited to train for robust SCQA models . The rest two spoken question answering datasets are Spoken-SQuAD ( Li et al. , 2018 ) and ODSQA ( Lee et al. , 2018 ) , respectively . However , there is usually no connection between a series of questions and answers within the same spoken passage among these datasets . More importantly , the most common way people seek or test their knowledge is via human conversations , which capture and maintain the common ground in spoken and text context from the dialogue flow . There are many real-world applications related to SCQA tasks , such as voice assistant and chat robot . In recent years , neural network based methods have achieved promising progress in speech processing domain . Most existing works first select a feature extractor ( Gao et al. , 2019 ) , and then enroll the feature embedding into the state-of-the-art learning framework , as used in single-turn spoken language processing tasks such as speech retrieval ( Lee et al. , 2015 ; Fan-Jiang et al. , 2020 ; Karakos et al. , 2020 ) , translation ( Bérard et al. , 2016 ; Serdyuk et al. , 2018 ; Di Gangi et al. , 2020 ; Tu et al. , 2020 ) and recognition ( Zhang et al. , 2017 ; Zhou et al. , 2018 ; Bruguier et al. , 2019 ; Siriwardhana et al. , 2020 ) . However , simply adopting existing methods to the SCQA tasks will cause several challenges . First , transforming speech signals into ASR transcriptions is inevitably associated with ASR errors ( See Table 2 ) . Previous work ( Lee et al. , 2019 ) shows that directly feed the ASR output as the input for the following down-stream modules usually cause significant performance loss , especially in SQA tasks . Second , speech corresponds to a multi-turn conversation ( e.g . lectures , interview , meetings ) , thus the discourse structure will have more complex correlations between questions and answers than that of a monologue . Third , additional information , such as audio recordings , contains potentially valuable information in spoken form . Many QA systems may leverage kind of orality to generate better representations . Fourth , existing QA models are tailored for a specific ( text ) domain . For our SCQA tasks , it is crucial to guide the system to learn kind of orality in documents . In this work , we propose a new spoken conversational question answering task - SCQA , and introduce Spoken-CoQA , a spoken conversational question answering dataset to evaluate a QA system whether necessary to tackle the task of question answering on noisy speech transcripts and text document . We compare Spoken-CoQA with existing SQA datasets ( See Table 1 ) . Unlike existing SQA datasets , Spoken-CoQA is a multi-turn conversational SQA dataset , which is more challenging than single-turn benchmarks . First , every question is dependent on the conversation history in the Spoken-CoQA dataset . It is thus difficult for the machine to parse . Second , errors in ASR modules also degrade the performance of machines in tackling contextual understanding with context paragraph . To mitigate the effects of speech recognition errors , we then present a novel knowledge distillation ( KD ) method for spoken conversational question answering tasks . Our first intuition is speech utterances and text contents share the dual nature property , and we can take advantage of this property to learn these two forms of the correspondences . We enroll this knowledge into the student model , and then guide the student to unveil the bottleneck in noisy ASR outputs to boost performance . Empirical results show that our proposed DDNet achieves remarkable performance gains in SCQA tasks . To the best of our knowledge , we are the first work in spoken conversational machine reading comprehension tasks . In summary , the main contributions of this work are as follows : • We propose a new task for machine comprehension of spoken question-answering style conversation to improve the network performance . To the best of our knowledge , our Spoken-CoQA is the first spoken conversational machine reading comprehension dataset . • We develop a novel end-to-end method based on data distillation to learn both from speech and text domain . Specifically , we use the model trained on clear syntax and close-distance recording to guide the model trained on noisy ASR transcriptions to achieve substantial performance gains in prediction accuracy . • We demonstrate the robustness of our DDNet on Spoken-CoQA , and demonstrates that the model can effectively alleviate ASR errors in noisy conditions . 2 RELATED WORK . Conversational Machine Reading Comprehension In recent years , the natural language processing research community has devoted substantial efforts to conversational machine reading comprehension tasks ( Huang et al. , 2018a ; Zhu et al. , 2018 ; Xu et al. , 2019 ; Zhang et al. , 2020 ; Gong et al. , 2020 ) . Within the growing body of work on conversational machine reading comprehension , two signature attributes have emerged : the availability of large benchmark datasets ( Choi et al. , 2018 ; Elgohary et al. , 2018 ; Reddy et al. , 2019 ) and pre-trained language models ( Devlin et al. , 2018 ; Liu et al. , 2019 ; Lan et al. , 2020 ) . However , these existing works typically focus on modeling the complicated context dependency in text form . In contrast , we focus on enabling the machine to build the capability of language recognition and dialogue modeling in both speech and text domains . Spoken Question Answering In parallel to the recent works in natural language processing , these trends have also been pronounced in the speech processing ( SP ) field , where spoken question answering , an extended form of Question Answering , have explored the prospect of machine comprehension in spoken form . Previous work on SQA typically includes two separate modules : automatic speech recognition and text question answering . It entails transferring spoken content to ASR transcriptions , and then employs natural language processing techniques to handle the speech language processing tasks . Prior to this point , the existing methods ( Tseng et al. , 2016 ; Serdyuk et al. , 2018 ; Su & Fung , 2020 ) focus on optimizing each module in a two-stage manner , where errors in the ASR module would suffer from severe performance loss . Concurrent with our research , Serdyuk et al . ( 2018 ) proposes an end-to-end approach for natural language understanding ( NLU ) tasks . SpeechBERT ( Chuang et al. , 2019 ) cascades the BERT-based models as a unified model and then trains it in an audio-and-text jointly learned manner . However , the existing SQA methods aim at solving a single question given the related passage without building and maintaining the connections of different questions within the human conversations . Knowledge Distillation Hinton et al . ( 2015 ) introduces the idea of Knowledge Distillation ( KD ) in a teacher-student scenario . In other words , we can distill the knowledge from one model ( massive or teacher model ) to another ( small or student model ) . Previous work has shown that KD can significantly boost prediction accuracy in natural language processing and speech processing ( Kim & Rush , 2016 ; Hu et al. , 2018 ; Huang et al. , 2018b ; Hahn & Choi , 2019 ) , while adopting KD-based methods for SCQA tasks has been less explored . Although we share the same research topic and application , our research direction and methods differ . Previous methods design a unified model to model the single-turn speech-language task . In contrast , our model explores the prospect of handling SQA tasks . More importantly , we focus the question of nature property in speech and text : do spoken conversational dialogues can further assist the model to boost the performance . Finally , we incorporate the knowledge distillation framework to distill reliable dialogue flow from the spoken contexts , and utilize the learned predictions to guide the student model to train well on the noisy input data . 3 TASK DEFINITION . 3.1 DATA FORMAT . We introduce Spoken-CoQA , a new spoken conversational machine reading comprehension dataset where the documents are in the spoken and text form . Given the spoken multi-turn dialogues and spoken documents , the task is to answer questions in multi-party conversations . Each example in this dataset is defined as follows : { Di , Qi , Ai } N1 , where Qi= { qi1 , qi2 , ... , qiL } and Ai= { ai1 , ai2 , ... , aiL } represent a passage with L-turn queries and corresponding answers , respectively . Given a passage Di , multi-turn history questions { qi1 , qi2 , ... , qiL−1 } and the reference answers { ai1 , ai2 , ... , aiL−1 } , our goal is to generate aiL for the given current question qiL . In this study , we use the spoken form of questions and documents as the network input for training . Note that questions and documents ( passages ) in Spoken-CoQA are in both text and spoken forms , and answers are in the text form . 3.2 DATA COLLECTION . We detail the procedures to build Spoken-CoQA as follows . First , we select the conversational question-answering dataset CoQA ( Reddy et al. , 2019 ) since it is one of the largest public CMRC datasets . CoQA contains around 8k stories ( documents ) and over 120k questions with answers . The average dialogue length of CoQA is about 15 turns , and the answer is in free-form text . In CoQA , the training set and the development set contain 7,199 and 500 conversations over the given stories , respectively . Therefore , we use the CoQA training set as our reference text of the training set and the CoQA development set as the test set in Spoken-CoQA . Then we employ the Google text-to-speech system to transform questions and documents in CoQA into the spoken form . Next , we adopt CMU Sphinx to transcribe the processed spoken content into ASR transcriptions . As such , we collect more than 40G audio data , and the data duration is around 300 hours . It is worth to note that since the constructed dataset does not update the answer spans based on the noisy ASR text and continues to assume answer-spans as per the actual text , we perform data filtering in our investigation by eliminating question-answer pairs from the corpus if answer spans to questions do not exist in the referenced ASR transcriptions . For clarity , we provide an example of our Spoken-CoQA dataset in Table 2 . Figure 4 compares spectrograms of samples from ASR modules . In this example , we observe that given the text document ( ASR-document ) , the conversation starts with the question Q1 ( ASR-Q1 ) , and then the system requires to answer Q1 ( ASR-Q1 ) with A1 based on a contiguous text span R1 . Compared to the existing benchmark datasets , ASR transcripts ( both the document and questions ) are much more difficult for the machine to comprehend questions , reason among the passage , and even predict the correct answer .

Summary: This paper proposes a new task: spoken conversational question answering, which combines conversational question answering (e.g. CoQA) with spoken question answering (e.g. Spoken-SQuAD). The task is to answer a question (in written text) given a question that is given in both audio form and text form. They create a dataset for this task by combining CoQA with some off-the-shelf text-to-speech and speech-to-text models. They then propose a new model, DDNet, which obtains improved performance on their dataset.