Towards Data Distillation for End-to-end Spoken Conversational Question Answering

1 INTRODUCTION . Conversational Machine Reading Comprehension ( CMRC ) has been studied extensively over the past few years within the natural language processing ( NLP ) communities ( Zhu et al. , 2018 ; Liu et al. , 2019 ; Yang et al. , 2019 ) . Different from traditional MRC tasks , CMRC aims to enable models to learn the representation of the context paragraph and multi-turn dialogues . Existing methods to the conversational question answering ( QA ) tasks ( Huang et al. , 2018a ; Devlin et al. , 2018 ; Xu et al. , 2019 ; Gong et al. , 2020 ) have achieved superior performances on several benchmark datasets , such as QuAC ( Choi et al. , 2018 ) and CoQA ( Elgohary et al. , 2018 ) . However , few studies have investigated CMRC in both spoken content and text documents . To incorporate spoken content into machine comprehension , there are few public datasets that evaluate the effectiveness of the model in spoken question answering ( SQA ) scenarios . TOEFL listening comprehension ( Tseng et al. , 2016 ) is one of the related corpus for this task , an English test designed to evaluate the English language proficiency of non-native speakers . But the multi-choice question answering setting and its scale is limited to train for robust SCQA models . The rest two spoken question answering datasets are Spoken-SQuAD ( Li et al. , 2018 ) and ODSQA ( Lee et al. , 2018 ) , respectively . However , there is usually no connection between a series of questions and answers within the same spoken passage among these datasets . More importantly , the most common way people seek or test their knowledge is via human conversations , which capture and maintain the common ground in spoken and text context from the dialogue flow . There are many real-world applications related to SCQA tasks , such as voice assistant and chat robot . In recent years , neural network based methods have achieved promising progress in speech processing domain . Most existing works first select a feature extractor ( Gao et al. , 2019 ) , and then enroll the feature embedding into the state-of-the-art learning framework , as used in single-turn spoken language processing tasks such as speech retrieval ( Lee et al. , 2015 ; Fan-Jiang et al. , 2020 ; Karakos et al. , 2020 ) , translation ( Bérard et al. , 2016 ; Serdyuk et al. , 2018 ; Di Gangi et al. , 2020 ; Tu et al. , 2020 ) and recognition ( Zhang et al. , 2017 ; Zhou et al. , 2018 ; Bruguier et al. , 2019 ; Siriwardhana et al. , 2020 ) . However , simply adopting existing methods to the SCQA tasks will cause several challenges . First , transforming speech signals into ASR transcriptions is inevitably associated with ASR errors ( See Table 2 ) . Previous work ( Lee et al. , 2019 ) shows that directly feed the ASR output as the input for the following down-stream modules usually cause significant performance loss , especially in SQA tasks . Second , speech corresponds to a multi-turn conversation ( e.g . lectures , interview , meetings ) , thus the discourse structure will have more complex correlations between questions and answers than that of a monologue . Third , additional information , such as audio recordings , contains potentially valuable information in spoken form . Many QA systems may leverage kind of orality to generate better representations . Fourth , existing QA models are tailored for a specific ( text ) domain . For our SCQA tasks , it is crucial to guide the system to learn kind of orality in documents . In this work , we propose a new spoken conversational question answering task - SCQA , and introduce Spoken-CoQA , a spoken conversational question answering dataset to evaluate a QA system whether necessary to tackle the task of question answering on noisy speech transcripts and text document . We compare Spoken-CoQA with existing SQA datasets ( See Table 1 ) . Unlike existing SQA datasets , Spoken-CoQA is a multi-turn conversational SQA dataset , which is more challenging than single-turn benchmarks . First , every question is dependent on the conversation history in the Spoken-CoQA dataset . It is thus difficult for the machine to parse . Second , errors in ASR modules also degrade the performance of machines in tackling contextual understanding with context paragraph . To mitigate the effects of speech recognition errors , we then present a novel knowledge distillation ( KD ) method for spoken conversational question answering tasks . Our first intuition is speech utterances and text contents share the dual nature property , and we can take advantage of this property to learn these two forms of the correspondences . We enroll this knowledge into the student model , and then guide the student to unveil the bottleneck in noisy ASR outputs to boost performance . Empirical results show that our proposed DDNet achieves remarkable performance gains in SCQA tasks . To the best of our knowledge , we are the first work in spoken conversational machine reading comprehension tasks . In summary , the main contributions of this work are as follows : • We propose a new task for machine comprehension of spoken question-answering style conversation to improve the network performance . To the best of our knowledge , our Spoken-CoQA is the first spoken conversational machine reading comprehension dataset . • We develop a novel end-to-end method based on data distillation to learn both from speech and text domain . Specifically , we use the model trained on clear syntax and close-distance recording to guide the model trained on noisy ASR transcriptions to achieve substantial performance gains in prediction accuracy . • We demonstrate the robustness of our DDNet on Spoken-CoQA , and demonstrates that the model can effectively alleviate ASR errors in noisy conditions . 2 RELATED WORK . Conversational Machine Reading Comprehension In recent years , the natural language processing research community has devoted substantial efforts to conversational machine reading comprehension tasks ( Huang et al. , 2018a ; Zhu et al. , 2018 ; Xu et al. , 2019 ; Zhang et al. , 2020 ; Gong et al. , 2020 ) . Within the growing body of work on conversational machine reading comprehension , two signature attributes have emerged : the availability of large benchmark datasets ( Choi et al. , 2018 ; Elgohary et al. , 2018 ; Reddy et al. , 2019 ) and pre-trained language models ( Devlin et al. , 2018 ; Liu et al. , 2019 ; Lan et al. , 2020 ) . However , these existing works typically focus on modeling the complicated context dependency in text form . In contrast , we focus on enabling the machine to build the capability of language recognition and dialogue modeling in both speech and text domains . Spoken Question Answering In parallel to the recent works in natural language processing , these trends have also been pronounced in the speech processing ( SP ) field , where spoken question answering , an extended form of Question Answering , have explored the prospect of machine comprehension in spoken form . Previous work on SQA typically includes two separate modules : automatic speech recognition and text question answering . It entails transferring spoken content to ASR transcriptions , and then employs natural language processing techniques to handle the speech language processing tasks . Prior to this point , the existing methods ( Tseng et al. , 2016 ; Serdyuk et al. , 2018 ; Su & Fung , 2020 ) focus on optimizing each module in a two-stage manner , where errors in the ASR module would suffer from severe performance loss . Concurrent with our research , Serdyuk et al . ( 2018 ) proposes an end-to-end approach for natural language understanding ( NLU ) tasks . SpeechBERT ( Chuang et al. , 2019 ) cascades the BERT-based models as a unified model and then trains it in an audio-and-text jointly learned manner . However , the existing SQA methods aim at solving a single question given the related passage without building and maintaining the connections of different questions within the human conversations . Knowledge Distillation Hinton et al . ( 2015 ) introduces the idea of Knowledge Distillation ( KD ) in a teacher-student scenario . In other words , we can distill the knowledge from one model ( massive or teacher model ) to another ( small or student model ) . Previous work has shown that KD can significantly boost prediction accuracy in natural language processing and speech processing ( Kim & Rush , 2016 ; Hu et al. , 2018 ; Huang et al. , 2018b ; Hahn & Choi , 2019 ) , while adopting KD-based methods for SCQA tasks has been less explored . Although we share the same research topic and application , our research direction and methods differ . Previous methods design a unified model to model the single-turn speech-language task . In contrast , our model explores the prospect of handling SQA tasks . More importantly , we focus the question of nature property in speech and text : do spoken conversational dialogues can further assist the model to boost the performance . Finally , we incorporate the knowledge distillation framework to distill reliable dialogue flow from the spoken contexts , and utilize the learned predictions to guide the student model to train well on the noisy input data . 3 TASK DEFINITION . 3.1 DATA FORMAT . We introduce Spoken-CoQA , a new spoken conversational machine reading comprehension dataset where the documents are in the spoken and text form . Given the spoken multi-turn dialogues and spoken documents , the task is to answer questions in multi-party conversations . Each example in this dataset is defined as follows : { Di , Qi , Ai } N1 , where Qi= { qi1 , qi2 , ... , qiL } and Ai= { ai1 , ai2 , ... , aiL } represent a passage with L-turn queries and corresponding answers , respectively . Given a passage Di , multi-turn history questions { qi1 , qi2 , ... , qiL−1 } and the reference answers { ai1 , ai2 , ... , aiL−1 } , our goal is to generate aiL for the given current question qiL . In this study , we use the spoken form of questions and documents as the network input for training . Note that questions and documents ( passages ) in Spoken-CoQA are in both text and spoken forms , and answers are in the text form . 3.2 DATA COLLECTION . We detail the procedures to build Spoken-CoQA as follows . First , we select the conversational question-answering dataset CoQA ( Reddy et al. , 2019 ) since it is one of the largest public CMRC datasets . CoQA contains around 8k stories ( documents ) and over 120k questions with answers . The average dialogue length of CoQA is about 15 turns , and the answer is in free-form text . In CoQA , the training set and the development set contain 7,199 and 500 conversations over the given stories , respectively . Therefore , we use the CoQA training set as our reference text of the training set and the CoQA development set as the test set in Spoken-CoQA . Then we employ the Google text-to-speech system to transform questions and documents in CoQA into the spoken form . Next , we adopt CMU Sphinx to transcribe the processed spoken content into ASR transcriptions . As such , we collect more than 40G audio data , and the data duration is around 300 hours . It is worth to note that since the constructed dataset does not update the answer spans based on the noisy ASR text and continues to assume answer-spans as per the actual text , we perform data filtering in our investigation by eliminating question-answer pairs from the corpus if answer spans to questions do not exist in the referenced ASR transcriptions . For clarity , we provide an example of our Spoken-CoQA dataset in Table 2 . Figure 4 compares spectrograms of samples from ASR modules . In this example , we observe that given the text document ( ASR-document ) , the conversation starts with the question Q1 ( ASR-Q1 ) , and then the system requires to answer Q1 ( ASR-Q1 ) with A1 based on a contiguous text span R1 . Compared to the existing benchmark datasets , ASR transcripts ( both the document and questions ) are much more difficult for the machine to comprehend questions , reason among the passage , and even predict the correct answer .

Summary: In this paper, the authors release a new dataset -  Spoken-CoQA which includes an ASR based version of the popular CoQA dataset. The dataset has been created by running the Google TTS system followed by ASR using CMU Sphinx, to create a speech-transcribed versions of the dataset. The dataset includes the corresponding TTS audio recordings. Since the transcribed dataset has transcription errors, existing reading comprehension models do not work well. Thus, the paper introduces a joint audio-textual model for QA on the Spoken-CoQA dataset that uses TTS recordings its corresponding ASR output.