Towards Data Distillation for End-to-end Spoken Conversational Question Answering
Authors: Chenyu You
Nuo Chen
Fenglin Liu
Dongchao Yang
Zhiyang Xu
Yuexian Zou
Venue:  (2021)
Keywords: spoken question answering
natural language processing
speech and language processing
knowledge distillation

Abstract: In spoken question answering, QA systems are designed to answer questions from contiguous text spans within the related speech transcripts. However, the most natural way that human seek or test their knowledge is via human conversations. Therefore, we propose a new Spoken Conversational Question Answering task (SCQA), aiming at enabling QA systems to model complex dialogues flow given the speech utterances and text corpora. In this task, our main objective is to build a QA system to deal with conversational questions both in spoken and text forms, and to explore the plausibility of providing more cues in spoken documents with systems in information gathering. To this end, instead of adopting automatically generated speech transcripts with highly noisy data, we propose a novel unified data distillation approach, DDNet, which directly fuse audio-text features to reduce the misalignment between automatic speech recognition hypotheses and the reference transcriptions. In addition, to evaluate the capacity of QA systems in a dialogue-style interaction, we assemble a Spoken Conversational Question Answering (Spoken-CoQA) dataset with more than 120k question-answer pairs. Experiments demonstrate that our proposed method achieves superior performance in spoken conversational question answering.


Under review as a conference paper at ICLR 2021
TOWARDS DATA DISTILLATION FOR END-TO-END SPOKEN CONVERSATIONAL QUESTION ANSWERING
Anonymous authors Paper under double-blind review
ABSTRACT
In spoken question answering, QA systems are designed to answer questions from contiguous text spans within the related speech transcripts. However, the most natural way that human seek or test their knowledge is via human conversations. Therefore, propose new Spoken Conversational Question Answering task (SCQA), aiming at enabling QA systems to model complex dialogues flow given the speech utterances and text corpora. In this task, our main objective is to build a QA system to deal with conversational questions both in spoken and text forms, and to explore the plausibility of providing more cues in spoken documents with systems in information gathering. To this end, instead of adopting automatically generated speech transcripts with highly noisy data, we propose a novel unified data distillation approach, DDNer, which directly fuse audio-text features to re- duce the misalignment between automatic speech recognition hypotheses and the reference transcriptions. In addition, to evaluate the capacity of QA systems in a dialogue-style interaction, we assemble Spoken Conversational Question Answering Spoken-CoQAi dataset with more than 120k question-answer pairs. Experiments demonstrate that our proposed method achieves superior perfor- mance in spoken conversational question answering.
INTRODUCTION
Conversational Machine Reading Comprehension (CMRC) has been studied extensively over the past few years within the natural language processing (NLP) communities (Zhu et al.. 2018 Liu al. 2019 Yang et al. 2019). Different from traditional MRC tasks, CMRC aims to enable models to learn the representation of the context paragraph and multi-tur turn dialogues. Existing methods to the conversational question answering (QA) tasks (Huang et al. 2018a Devlin et al. 2018 Xu al. 2019 Gong et al. 2020) have achieved superior performances on several benchmark datasets, such as QuAC (Choi et al., 2018) and CoQA (Elgohary et al. 2018). However, few studies have investigated CMRC in both spoken content and text documents.
To incorporate spoken content into machine comprehension, there are few public datasets that evalu- ate the effectiveness of the model in spoken question answering (SQA) scenarios. TOEFL listening comprehension (Tseng et al. 2016) is one of the related corpus for this task, an English test de- signed to evaluate the English language proficiency of non-native speakers. But the multi-choice question answering setting and its scale is limited to train for robust SCQA models. The rest two spoken question answering datasets are Spoken-SQuAD (Li et al. 2018) and ODSQA (Lee al.) 2018), respectively. However, there is usually no connection between series of questions and an- swers within the same spoken passage among these datasets. More importantly, the most common way people seek or test their knowledge is via human conversations, which capture and maintain the common ground in spoken and text context from the dialogue flow. There are many real-world applications related to SCQA tasks, such as voice assistant and chat robot.
In recent years, neural network based methods have achieved promising progress in speech process- ing domain. Most existing works first select a feature extractor (Gao et al. 2019), and then enroll the feature embedding into the state-of-the. art learning framework, as used in single-turp spoken
language processing tasks such speech retrieval (Lee al. [2055] Fan-Jiang et al.) 20200 Karakos
et al. 2020), translation (Bérard et al. 2016)] Serdyuk et al. 2018 Di Gangi et al. 2020 Tu et al.
2020 and recognition (Zhang et al. 2017 Zhou al. 2018 Bruguier al. 2019 Siriwardhana

---

Under review as a conference paper at ICLR 2021
Spoken Documents
Dialogue History
Spoken Questions
ASR
Q, WWatt color caught
A, whiie
S2
Q ==
{ Qq
42
Machine Reasoning
ASR System
laxsi: redi
Qut did live?
Koask mal
Ay in
Vitarbi Decoding
Viterbi Decoding
Conversational MRC models
ASR Context:
Onco spon Lane ber saw farm loosa (here leel litti captin raneel cotion. How sedd in rizs warn plic
ASR Q3 What color was caught in?
the
farwers
As white
Figure 1:
An illustration of flow diagram for spoken conversational question answering tasks with an example from our proposed Spoken-CoQA dataset.
Table 1: Comparison of Bonken-CooAA with existing spoken question answering datasets.
Dataset
Conversational
Spoken
Answer Type
TOFFL (Teeng al. 2016
X
Multi-choice
Booken-SQuall (L al
2018
X
Spans
ODSQA (Lee al. 2018
X
Spans
Spoken-CoQA
Free-form text, Unanswerable
et al. 2020). However, simply adopting existing methods to the SCQA tasks will cause several chal- lenges. First, transforming speech signals into ASR transcriptions inevitably associated with ASR errors (See Table Previous work (Lee et al. (2019) shows that directly feed the ASR output as the input for the following down-stream modules usually cause significant performance loss, especially in SQA tasks. Second, speech corresponds to a multi-turn conversation (e.g. lectures, interview, meetings), thus the discourse structure will have more complex correlations between questions and answers than that of monologue. Third, additional information, such audio recordings, contains potentially valuable information in spoken form. Many QA systems may leverage kind of orality to generate better representations. Fourth, existing QA models are tailored for specific (text) domain. For our SCQA tasks, it is crucial to guide the system to learn kind of orality in documents.
In this work, we propose duce Spoken-CoQA, a spoken conversational question answering dataset to evaluate a QA system whether necessary to tackle the task of question answering on noisy speech transcripts and text document. We compare Spoken-CoQA with existing SQA datasets (See Table D Unlike existing SQA datasets, Spoken-CoQA is multi-turn conversational SQA dataset, which more challenging than single-turn benchmarks. First, every question is dependent on the conversation history in the Spoken-CoOL CoQA dataset. It is thus difficult for the machine to parse. Second, errors in ASR mod- ules also degrade the performance of machines in tackling contextual understanding with context paragraph. To mitigate the effects of speech recognition errors, we then present a novel knowledge distillation (KD) method for spoken conversational question answering tasks. Our first intuition is speech utterances and text contents share the dual nature property, and we can take advantage of this property to learn these two forms of the correspondences. We enroll this knowledge into the student model, and then guide the student to unveil the bottleneck in noisy ASR outputs to boost perfor- mance. Empirical results show that our proposed DDNet achieves remarkable performance gains in SCQA tasks. To the best of our knowledge, we are the first work in spoken conversational machine reading comprehension tasks.
spoken conversational question answering task SCQA, and intro-
2

---

Under review as a conference paper at ICLR 2021
ASR Module
(Speech BERT)
Audio Decoder
transcriptions
Spoken Documents/Questionn
Speech Feature Emboddings
Orcee upon time, in tam near fimn to thene londs inte white fimm ramed Camon
Contextt:
Cross
Certon livect high up in- rise warn place shone whene
horee capp the home
(Text BERT)
Certon wannh slone bay
bar,
Teacher
Output Layer:
spon
high place Comon wonnt home staned her bad with her mommy and 5 other sittrr..
Attention Layer:
Encoding Layer:
Cross Attention:
CMRC Module
Student
Figure 2: An illustration of the architecture of DDNet.
In summary, the main contributions this work are follows:
We propose task for machine comprehension of spoken question-answeriigg style conversation to improve the network performance. To the best of our knowledge, our Spoken-CoQA is the first spoken conversational machine reading comprehension dataset.
We develop novel end-to-end method based on data distillation to learn both from speech and text domain. Specifically, we use the model trained on clear syntax and close-distance recording guide the model trained on noisy ASR transcriptions achieve substantial performance gains in prediction accuracy.
We demonstrate the robustness of our DDNet on Spoken-Cool CoQA, and demonstrates that the model can effectively alleviate ASR errors noisy conditions.
2 RELATED WORK
Conversational Machine Reading Comprehension In recent years, the natural language process- ing research community has devoted substantial efforts to conversational machine reading compre- hension tasks (Huang et al., 2018a 8a Zhu al. 2018 Xu et al. 2019 Zhhang et al.) 2020 Gong al., 2020). Within the growing body of work on conversational machine reading comprehension, two signature attributes have emerged: the availability of large benchmark datasets (Choi et 2018
Elgohary et al., 20088] Reddy et al.) 2019) and pre-trained language models (Devlin et al. 2018 Liu
al. 2019 Lan al. 2020). However, these existing works typically focus modeling the com- plicated context dependency in text form. In contrast, we focus on enabling the machine to build the capability of language recognition and dialogue modeling in both speech and text domains.
Spoken Question Answering In parallel to the recent works in natural language processing, these trends have also been pronounced in the speech processing (SP) field, where spoken question an- swering, an extended form of Question Answering, have explored the prospect of machine compre- hension in spoken form. Previous work on SQA typically includes two separate modules: automatic speech recognition and text question answering. It entails transferring spoken content to ASR tran- scriptions, and then employs natural language processing techniques to handle the speech language processing tasks. Prior to this point, the existing methods (Tseng al. 2016 Serdyuk et al. 2018 Su & Fung 2020) focus on optimizing each module in a two-stage manner, where errors in the ASR module would suffer from severe performance loss. Concurrent with research, Serdyuk et al.
3

---

Under review as a conference paper at ICLR 2021
proposes end-to-ent approach for natural language understanding (NLU) tasks. Speech- BERT Chuang et al., 2019) cascades the BERT-based models as a unified model and then trains it in an andio-and-eett jointly learned manner. However, the existing SQA methods aim at solving a sin- gle question given the related passage without building and maintaining the connections of different questions within the human conversations.
2018
Knowledge Distillation Hinton et al. (2015) introduces the idea of Knowledge Distillation (KD) in teacher-student scenario. In other words, we can distill the knowledge from one model (massive or teacher model) to another (small or student model). Previous work has shown that KD can significantly boost prediction accuracy natural language processing and speech processing (Kim & Rush, [2166] Hu et al. [2018] Huang et al. 20188; Hahn & Choi 2019), while adopting KD-based methods for SCQA tasks has been less explored. Although we share the same research topic and application, our research direction and methods differ. Previous methods design unified model to model the single-turn speech-language task. In contrast, our model explores the prospect of handling SQA tasks. More importantly, we focus the question of nature property in speech and text: do spoken conversational dialogues can further assist the model to boost the performance. Finally, we incorporate the knowledge distillation framework to distill reliable dialogue flow from the spoken contexts, and utilize the learned predictions to guide the student model to train well on the noisy input data.
3 TASK DEFINITION
3.1 DATA FORMAT
We introduce Spoken-CoQA, a new spoken conversational machine reading comprehension dataset where the documents are in the spoken and text form. Given the spoken multi-turn dialogues and spoken documents, the task is to answer questions in multi-party conversations. Each ex- ample in this dataset is defined as follows: {D,, Qs Ar where Q,=(%1, 9:...... ) ) and A;= {a.ı 022, ..., a&L) represent passage with L-turn queries and corresponding answers, respectively. Given passage D;, multi-turn history questions (9.1 9:2 .... I and the reference answers {aa 0:2, O.L--). our goal is to generate a.z. for the given current question liz. In this study, we the spoken form of questions and documents as the network input for training. Note that ques-
tions and documents (passages) in Spoken-CoQA are in both text and spoken forms, and answers in the text form.
3.2 DATA COLLECTION
We detail the procedures to build Spoken- -ocAA as follows. First, we select the conversational question-answering dataset CoQA (Reddy al. 2019) since it is one of the largest public CMRC datasets. CoQA contains around 8k stories (documents) and over 120k questions with answers. The average dialogue length of CoQA is about 15 turns, and the answer is in free-form text. In CoQA, the training set and the development set contain 7,199 and 500 conversations over the given stories, respectively. Therefore, set and the CoQA development set as the test set in Spoken-CoQA. Then we employ the Google text-to-speech system transform questions and documents in CoQA into the spoken form. Next, we adopt CMU Sphinx to transcribe the processed spoken content into ASR transcriptions. As such, collect more than 40G audio data, and the data duration around 300 hours. It is worth to note that since the constructed dataset does not update the answer spans based on the noisy ASR text and continues assume answer-spans as per the actual text, we perform data filtering in our investigation by eliminating question-answer pairs from the corpus if answer spans to questions do not exist the referenced ASR transcriptions.
the CoQA training set our reference text of the training
For clarity, we provide an example of our Spoken-CoQA dataset in Table 2 Figure 4 compares spectrograms of samples from ASR modules. this example, we observe that given the text docu- ment (ASR-document) the conversation starts with the question QI (ASR-Q1). ) and then the system requires to answer Q1 (ASR-Q1) with A₁ based on a contiguous text span R1- Compared to the existing benchmark datasets, ASR transcripts (both the document and questions) much more difficult for the machine to comprehend questions, reason among the passage, and even predict the correct answer.
4

---

Under review as a conference paper at ICLR 2021
Table 2: An example from Spoken-CoQA. We observe large misalignment between the manual transcripts and the corresponding ASR transcripts. Note that the misalignment is in bold font.
Manual Transcript
ASR Transcript
Once upon a time, a barn near a farm house, there lived little white kitten named Cotton. Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept. But Cotton wasn't I alone in her little home above the barn, oh no. She shared her hay bed with her mommy and 5 other sis- ters...
Once upon a time in a bar near farm house, there lived little like captain named cot- ton. How to live tied up in nice warm place above the bar and we're all of the farmers horses slapped. But cotton was not alone in her little home above the bar in now. She shared her hey bed with her mommy and 5 other sisters...
Q1: Did Cotton live alone?
ASR Qi: Did caught in live alone?
A1: no
A₁: no
R1: Cotton wasn't it alone.
R Cotton wasn't alone.
Q2: Who did she live with?
ASR-Q2: Who did she live with?
Ag: with her mommy and 5 sisters
A2: with her mommy and 5 sisters
R2: with her mommy and other sisters
R2: with her mommy and 5 other sisters
Q3: What color were her sisters?
ASR-Q3: What color were her sisters?
orrorange and white Ry: her sisters were all orange with beautiful
A3: orange and white
R3: her sisters were all orange with beautiful
white tiger stripes
white tiger stripes
DDNET
In this section, we detail our data distillation approach by leveraging the dual nature of speech and text domains boost the prediction accuracy in spoken dialogue system. An overview pipeline of this task is shown in Figure | We first introduce the multi-modality fusion mechanism. Then we present the major components of the CRMC module. Finally, we describe simple yet effective distillation strategy in the proposed DDNet to learn feature representation in the speech-text domain comprehensively.
Given spoken words S (s1, $2..... ...., Sn and corresponding text words X (x T2, Fn we uti- lize Speech-BERT and Beext-BERTio to generate speech feature embedding E. E.1, E.2. .... E.n) and context word embedding E
{ E. E.2.... E., respectively. Concretely, we first use vq- wav2vec (Baevski al. 2019) to transfer speech signals into a series of tokens, which is the standard tokenization procedure in natural language processing tasks, and then use Speech- BERT (Chuang cct al. 2201)), a variant of BERT-based models, to process the speech sequences for training. We re- train Speech- Speech-BERT similar with BERT-base (Devlin et al. 2018) model that contains 12 transformer layers with the residual structure and the embedding dimension is with 768. In parallel, embed the text context into a sequence of vectors via our text encoder TeXI-BERT. We adopt the same architecture of BERT-base (Devlin al. 2018) in our Text-BERT due to its superior performance.
BERT (Chuang et al.) 2019) our Spoken-CoQA dataset. The scale of Speech-BERT
Cross Attention Inspired by ViLBERT (Luu et al. 2019), we apply the co-attention transformer layersLu et al.] 2019). variant of Self-Attention Vaswani al., 2017], as the Cross Attention module for speech and text embedding fusion. We pass query, key, and value matrices (Q, K. V) as input to the Cross Attention module. We then compute the cross attention-pooled features by querying one modality with Q vector from another modality.
peross Cross E., E,, E.)
Eeross CrossAllention E,, E,, E.)
Finally, we obtain the aligned cross attention embedding Ecross by concatenating Éeross and ieross
5

---

Under review as a conference paper at ICLR 2021
4.1 KEY COMPONENTS
We build our CMRC module, based on recent works (Zhu et al. 2018 Huang al. 2017). We divide our CMRC module into three key components: Encoding Layer, Attention Layer and Output Layer.
Encoding Layer We encode documents and conversations (questions and answers) into the cor- responding feature embedding (e. .eg..charrtter embedding, word embedding, and contextual embed- ding), and then concatenate the output contextual embedding and the aligned cross attention embed- ding Ecross, and pass it input.
E Eenc-
[Eenci Ecross]
Attention Layer We compute the attention the context representations of the documents and questions, and extensively exploit correlations between them. Note that we adopt the default atten- tion layers in four baseline models.
Output Layer After obtaining attention pociiidd-pooe representations, the Output Layer computes the probability distribution of the start and end index within the entire documents and predicts an answer to the current question.
4.2 KNOWLEDGE DISTILLATION
For prior speech-language models, the only guidance is the standard training objective to measure the difference between the prediction and the reference answer. However, such criteria makes sense for noisy ASR transcriptions. To tackle this issue, we distill the knowledge from our teacher model, and use them guide the student model learn contextual features in our spoken CMRC task. Concretely, we set the model trained on the speech document and text corpus as the teacher model and trained the ASR transcripts the student model, respectively. Thus, the student trained on low-quality data learn to imbibe the knowledge that the teacher has discovered.
Concretely, given the zs and zr the prediction vectors by the student and teacher models, the objective is define as:
L ((rr + ()E v)),
(4)
NEN
where K.C(.) and X2E(+) denote the Kullback-Leibler divergence and cross entropy, respectively. represents the ground truth labels in the text training dataset X. p-(-) refers the softmax function with temperature and a is balancing factor.
5 EXPERIMENTS AND RESULTS
In this section, we first introduce several state-of-the-art language models as our baselines, and then evaluate the robustness of these models on proposed Spoken-CoQA dataset. Finally, we provide a thorough analysis of different components of our method. Note that we use the default setting in all the evaluated methods.
5.1 BASELINES
In principle, DDNet can utilize any backbone network for SCQA tasks. We choose several state-of- the-art language models (FlowQA (Huang al. 2018a). SDNet (Zhu al., 2018), BERT-base (De- vlin et al. 2018). ALBERT (Lan et al.) 2020) as our backbone network due to its superior perfor- mance. We also compare our proposed DDNet with several state-of-the-an SQA methods (Lee al.) 2018 Serdyuk et al. 2018 Lee et al. 2019 Kuo et al. 2020). To train the teacher-studenl pairs simultaneously. we first train baselines on the CoQA training set and then compare the performances of testing baselines CoQA dev set and Booken-CoQa dev set. Finally, train the baselines on the Spoken-CoQU CoQQ training set and evaluate the baselines on the CoQA dev set and Spoken-CoQA test set. We provide quantitative results in Table

---

Under review as a conference paper at ICLR 2021
Table 3: Comparison of four baselines (FlowQA, SIDNet, BERT, ALBERT). Note that denote
Spoken-CoQA test set as S-CoQA test for brevity.
CoQA
S-CoQA
CoQA
dev
S-CoQA
test
CoQA dev
S-CoQA
test
Methods
EM
F1
EM
EM
F1
EM
FlowQA QA (Huang et al. 2018a)
66.8
75.1
44.1
56.8
40.9
51.6
22.1
34.7
SDNet (Zhu ct al. 2018)
68.1
76.9
39.5
51.2
40.1
52.5
41.5
53.1
BERT-base (Devlin al.
2018)
67.7
77.7
41.8
54.7
42.3
55.8
40.6
54.1
ALBERT- -base (Lan al.
2020)
71.4
80.6
42.6
54.8
42.7
56.0
41.4
55.2
Average
68.5
77.6
42
54.4
41.5
54.0
36.4
49.3
5.2 EXPERIMENT SETTINGS
We use the official BERT (Devlin et al. 2018) and ALBERT (Lan et al. 2020) as our starting point for training. We BERT-base (Devlin et al. both include 12 transformer encoders, and the hidden size of each word vector is 768. BERT and ALBERT utilize BPE as the tokenizer. but FlowQA and SDNet use SpaCy (Honnibal Montani 2017) for tokenization. Specifically, in the case of tokens in spaCy (Honnibal & Montani| (017) correspond to more than one BPE sub-tokens, we average the BERT embeddings of these BPE sub-tokens the embedding for each token. To maintain the integrity of all evaluated model perfor- mance, we use standard implementations and hyper-paramettrs of four baselines for training. The balancing factor is set to 0.9, and the temperature is set to 2. For evaluation, Match (EM) and Pi score to compare the model performance on the test set. Note that this work, each baseline is trained in the local computing environment, which may results in different results compared with the ones on the CoQA leader board.
2018) and ALBERT-base (Lan al. 2020). which
use Exact
5.3 RESULTS
We compare several teacher-student pairs on CoQA and Spoken-CoQA dataset. Quantitative results are shown in Table 3 We can observe that the average F1 scores are 77.6% when training on CoQA (text document) and testing on the CoQA dev set. However, when training the models on Spoken- CoQA (ASR transcriptions) and testing on the Spoken-CoQA test set, average Fl scores are dropped to 49.3%. For FlowQA, the performance even dropped by 40.4% on score. This confirms the importance of mitigating ASR errors which severely degrade the model performance in our tasks.
As shown in Table 4 it demonstrates that our proposed Cross Attention block and knowledge dis- tillation strategy consistently boost the remarkable performance all baselines, respectively. More importantly, our distillation strategy works particularly well. For FlowQA, our method achieves 53.7% (vs.51.6%) and 39.2% 7%) in terms of F1 score over text document and ASR tran- scriptions, respectively. For SDNet, our method outperforms the baseline without distillation strat- egy, achieving 55.6% (vs.52.5%) and 56.7% (vs.53.1%) in terms of F1 score. For BERT-like models (BERT-base and ALBERT-base), our methods also improve Fl scores to 58.8% (vs.5588% and 57.7% (vs.54.1%% 59 59.6% (vs.56.0%) and 58.7% (vs.55.2% respectively. We also compare the combination of our distillation strategy and the cross attention mechanism. Our results suggest that such network notably improve prediction performance for spoken conversational question answering tasks. Such significant improvements demonstrate the effectiveness of DDNet.
QUANTITATIVE ANALYSIS
Speech Feature in ASR System To perform qualitative analysis of speech features, visualize the log-mel spectrogram features and the mel-frequency cepstral coefficients (MFCC) feature em- bedding learned by DDNer in Figure 4 We can observe how the spectrogram features respond to different sentence examples.
Temperature To study the effect of temperature 7 (See Section experiments of four baselines with the standard choice of the temperature 7 E {1,2, 4, 6, 8, 10). All models are trained Spoken-CoQA dataset, and validated on the CoQA dev and Spoken-CoQA
conduct the additional

---

Under review as a conference paper at ICLR 2021
Table Comparison of key components in DDNer. We set the model speech document and text
corpus as the teacher model, and the one on the ASR transcripts as the student model.
CoQA dev
S-CoQA test
Methods
EM
F1
EM
F1
FlowQA (Huang al. 2018a)
40.9
51.6
22.1
34.7
FlowQA + sub-word unit (Li et al. 2018
41.9
53.2
23.3
36,4
FlowQA+ SLU (Serdyuk al. 2018)
41.2
52.0
22.4
35.0
FlowQA + bacck-translation
Lee et a..
2018
40.5
52.1
22.9
35.8
FlowQA + domain adaptation (Lee et 2019
41.7
53.0
23.4
36.1
FlowQA + Cross Attention
41.1
52.2
22.5
35.5
FlowQA + Knowledge Distillation
42.5
53.7
23.9
39.2
FlowQA + Cross EttentionKKnwwldde Distillation
42.9
54.7
24.9
41.0
SDNet (Zhu et al. 2018)
40.1
52.5
41.5
53.1
SDNct sub-word unit (Li et al. 2018
41.2
53.7
41.9
54.7
SDNet+ SLU (Serdyuk ct al. 2018
40.2
52.9
41.7
53.2
SDNet back-translation (ee a.. 2018
40.5
53.1
42.4
54.0
SDNet domain adaptation (Lee et al. 2019)
41.0
53.9
41.7
54.6
SDNet Cross Attention
40.4
52.9
41.6
53.4
SDNet Knowledge Distillation
41.7
55.6
43.6
56.7
SDNet Cross datention-Knowledee Distillation
42.1
56.6
44.0
57.7
BERT-base (Devlin et al. 2018)
42.3
55.8
40.6
54.1
BERT-base + sub-word unit (Li al. 2018)
43.2
56.8
41.6
55.4
BERT-base+ SLU (Serdyuk et al. 2018)
42.5
56.1
41.0
54.6
BERT-base + back-translatuus (Lee et al.
2018
42.9
56.5
41.5
55.2
BERT-base domain adaptation (Lee al. 2019
43.1
57.0
41.7
55.7
aeBERT Kuo et al. 2020)
43.0
56.9
41.8
55.6
BERT-base Cross Attention
42.4
56.3
40.9
54.5
BERT-base Knowledge Distillation
44.1
58.8
42.8
57.7
BERT-base Cross Iitention-Knowledde Distillation
44.2
59.8
43.5
58.4
ALBERT-base (Lan al. 2020)
42.7
56.0
41.4
55.2
ALBERT-base sub-word unit (Li et al. 2018)
43.7
57.2
42.6
56.8
ALBERT-base SLU (Serdyuk et al. 2018)
42.8
56.3
41.7
55.7
ALBERT-base back-translatioon (Lee ct al. 2018
43.5
57.1
42.4
56.4
ALBERT-base domain adaptation (Lee et al. 2019)
43.5
57.0
42.7
56.7
ALBERT-basee Cross Attention
42.9
56.4
41.6
55.9
ALBERT-base Knowledge Distillation
44.8
59.6
43.9
58.7
ALBERTT-base Cross Attention+ Knowledge Distillation
45.2
60.2
44.4
60.1
test set, respectively. In Figure when T is to 2, four baselines all achieve their best performance in term of F1 and EM metrics.
Multi-Modality Fusion Mechanism To study the effect different modality fusion mechanisms, we introduce novel fusion mechanism Con Fusion: first, we directly concatenate two output em- bedding from speech-BERK following CMRC module. In Table 5 observe that Cross Attention fusion mechanism outper- form four baselines with Con Fusion in terms of EM and Fl scores. We further investigate the effect of uni-model input. Table 5 shows that text-only performs better than speech-only. One possible reason for this performance is that only using speech features can bring additional noise. Note that speech-only (text-only) represents that only feed the speech (text) embedding for speech-BERT (leXI-BERT) to the encoding layer in the CMRC module.
text-BERT models, and then pass it to encoding layer in the
CONCLUSION
In this paper, we have presented a new spoken conversational question answering task Spoken- CoQA, for enabling fomman-maching communication. Unlike the existing Spoken conversational machine reading comprehension datasets, Spoken-CoQA includes multi-turn conversations and pas- sages in both text and speech form. Furthermore, propose data distillation method, which
8

---

Under review as a conference paper at ICLR 2021
Table 5: Comparison of different fusion mechanisms in DDNet. We set the model trained speech document and text corpus as the teacher model, and the one trained on the ASR transcripts as the student model.
CoQA
dev
S-CoQA test
Models
FM
F1
FM
Fl
FAowQA
(Hang al.
201 Sa)
40.9
51.6
22.1
34.7
FlowQA Huang et al.
201 8a
speech-only
40.8
51.2
21.8
34.0
FA Huang al.
201
text-only
41.1
51.7
22.4
35.3
Flow QA
Huang ct al.
201 8a
+
Con Fusion
41.0
52.0
22.1
35,2
FlowQA Q
Huang al.
2018a
Cross Attention
41.1
52.2
22.5
35.5
SDNet Zhu et al.
2018
40.1
52.5
41.5
53.1
SDNet Zhu ct al.
2018
+ speech-only
39.3
51.6
40.9
52.28
SDNet Zhu et al.
2018
text-only
40.2
52.7
41.5
53.3
SDNet Zhu et al.
2018
Con Fusion
40.3
52.6
41.5
53.2
SDNet (Zhu al.
2018
Cross Attention
40.4
52.9
41.6
53.4
BERT-base (Devlin ct al.
2018
42.3
55.8
40.6
54.1
BERT-base
Devlin et al.
2018 8
+ speech-only
41.9
55.8
40.2
54.1
BERT-base
Devlin et al.
201
8
+ text-only
42.4
56.0
40.9
54.3
BERT-base
Devlin et al.
+ Con Fusion
42.3
56.0
40.8
54.1
BERT-base
DDvllin al.
201 8
Cross Attention
42.4
56.3
40.9
54.5
ALBERT-base
[Lan et al. 2020
42.7
56.0
41.4
55.2
ALBERT-base
Lan al. 2020
speech-only
41.8
55.9
41.1
54.8
ALBERTT-aase
LLan et al. 2020
text-only
42.9
56.3
41.4
55.7
ALBERT-base
LLnn al. 2020
Con Fusion
42.7
56.1
41.3
55.4
ALBERT-base
Lan ct al.
2020
Cross Attention
42.9
56.4
41.6
55.9
Figure 3: Ablation studies of temperature T DDNer performance (Flow QA, SDNet, BERT, AL- BERT). Red and blue denote the results on CoQA dev and Spoken-CoQA test set, respectively.
leverages audio-text features to reduce the misalignment between ASR hypotheses and the reference transcriptions. Experimental results show that DDNet achieves superior performance in prediction accuracy. For future work, we will further investigate different mechanism of integrating speech and text content, and propose novel machine learning based networks to migrate ASR recognition errors boost the performance of QA systems.
REFERENCES
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference Learning Representations, 2019.
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. Listen and translate: A proof of concept for end-to-end speech-to-text translation. arXiv preprint arXiv:1612.01144, 2016.
Antoine Bruguier, Rohit Prabhavalkar, Golan Pundak, and Tara N Sainath. Phoebe: Pronunciation- aware contextualization for end-to andtoo-end speech recognition. In 2019 IEEE International Confer- on Acoustics, Speech and Signal Processing (ICASSP), pp. 6171-6175. IEFE, 2019.

---

Under review as a conference paper at ICLR 2021
Eunsol Choi, He He, Mohit lyyer. Mark Yatskar, Wan-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2174-2184, 2018.
Yung-Sung Chuang, Chi-Liang Liu, and Hung-Yi Yi Lee. SpeechBERT: Cross-modal pre-trained lan- guage model for end-to-end spoken question answering. arXiv preprint arXiv: 19ti::/900.11599 1159,, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Mattia A Di Gangi, Viet-Nhat Nguyen, Matteo Negri, and Marco Turchi. Instance-based model adaptation for direct speech translation. In 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7914-7918. IEEE, 2020.
Ahmed Elgohary, Chen Zhao, and Jordan Boyd-Graber. Dataset and baselines for sequential open- domain question answering. In Empirical Methods in Natural Language Processing, 2018.
Shao-Wei Fan-Jiang, Tien-Hong Lo, and Berlin Chen. Spoken document retrieval leveraging bert- based modeling and query reformulation. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8144-8148. IEEE, 2020.
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6639- 6648, 2019.
Hongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen, and Dong Yu. Recurrent chunking mechanisms for long-text machine reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, 2020.
Sangchul Hahn and Heeyoul Choi. Self-knowledge distillation in natural language processing. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019). pp. 423 4303-4302200 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint wrXiv:1503.0253/ 2015.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear, 2017.
Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guide guided answer distillation for machine reading comprehension. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2077-2086, 2018.
Hsin Yuan Huang, Chenguang Zhu, Yelong Shen, and Weizhu Chen. Fusionnet: Fusing via fully- aware attention with application to machine comprehension, arXiv preprint arXiv: 111::1711.0034// 2017.
Hsin Yuan Huang, Eunsol Choi, and Wen-tau Yih. FlowQA: Grasping flow in history for conversa- tional machine comprehension. arXiv preprint arXiv: 1810006688, 2018 Sa.
Mingkun Huang, Yongbin You, Zhehuai Chen, Yanmin Qian, and Kai Yu. Knowledge distillation for sequence model. Proceedings Interspeech 2018, pp. 3703-3707, 2018b.
Damianos Karakos, Rabih Zbib, William Hartmann, Richard Schwartz. and John Makhoul. Refor- mulating information retrieval from speech and text as detection problem. In Proceedings of the workshop on Cross- LanguaLe Search and Sunmarization of Text and Speech (CLSSTS2020), pp- 38 43, 2020.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327, 2016.
Chia-Chih Kuo, Shang-Bal Bao Luo, and Kuan- Yu Chen. An spoken multiple-choice question answering. arXiv preprint arXiv:2005. /2/22, 2020.
based framework for
10

---

Under review as a conference paper at ICLR 2021
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: Lite BERT for Self-supervised Learning of Language Representations. In International Conference Learning Representations, 2020.
Lee. Shang-Ming Wang. Huan-Cheng Chang, and Hung-Y. Yi Lee. ODSQA: Open- domain spoken question answering dataset. In 2018 IEEE Spoken Language Technology Work- shop (SL.7), pp. 949-956. IEEE, 2018.
Chia-Hsuan Lee, Yun-Nung Chen, and Hung- Yi Lee. Mitigating the impact of speech recognition errors spoken question answering by adversarial domain adaptation. In 2019 IEEE Intera- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7300-7304, IEEE, 2019.
Lin-shan Lee, James Glass, Hung-yi Lee, and Chun-an Chan. Spoken content retrieval beyond cascading speech recognition with text retrieval. JEEE/ACM Trunsactions on Audio, Speech, and Language Processing, (3(9):1389111) 2015.
Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, and Hung-yi Lee. Spoken SQuAD: A study of mit- igating the impact of speech recognition errors listening comprehension. arXiv preprint 180.000000003300 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv preprint 1rXi:::907. 191.1997711999 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for assionaadd-anguaaa tasks. In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Siva Reddy, Danqi Chen, and Christopher D Manning. CoQA: A Conversational Question An- swering Challenge. Transactions of the Association for Computational Linguistics, 7:249-266,
2019.
Dmitriy Serdyuk, Yongqiang Wang, Christian Fuegen, Anuj Kumar, Baiyang Liu, and Yoshua Ben- gio. Towards end-to-end spoken language understanding. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5754-5755 IEEE, 2018.
Shamane Siriwardhana, Andrew Reis, Rivindu Weerasekera, and Suranga Nanayakkara. Jointly fine-tuning "bert-like" self supervised models to improve multimodal speech emotion recognition. arXiv preprint ar arXiv:2008.0660 066882 2020.
Dan Su and Pascale Fung. Improving spoken question answering using contextualized word repre- sentation. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8004 8008. IEEE, 2020.
Bo-Hsiang Tseng, Sheng-Syun Shen, Hung- Yi Lee, and Lin-Shan Lec. Towards machine com- prehension of spoken content: Initial TOEFL listening comprehension test by machine. arXiv preprint arxiv: 168:06608.06378. 2016.
Mei Tu, Fan Zhang, and Wei Liu. End-to-end speech translation with self-contained vocabulary ma- nipulation. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
IEEE, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. Review conversational reading comprehension. arXiv preprint arXiv: 000../92220022. 00821, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5753-5763, 2019.

---

Under review as a conference paper at ICLR 2021
Y. Zhang, W. Chan, and N. Jaitly. Very deep convolutional networks for end-to-end speech recog- nition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3845-4849, 2017.
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. SG-Net: Syntax-guided machine reading comprehension. In Proceedings of the Thirty-Fourth AAAI Con- ference on Artificial Intelligence, 2020.
Shiyu Zhou, Linhao Dong, Shuang Xu, and Bo Xu. Syllable-based sequence-to-sequenn speech recognition with the transformer in mandarin chinese. arXiv preprint arXiv: 10752, 2018.
Chenguang Zhu, Michael Zeng, and Xuedong Huang. SDNet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arxiv:18/2.03599, 2018.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 SPEECH FEATURES IN ASR SYSTEM
Due to the page limit, we present some examples of speech features here.
Once upon time, near house, lived little white kitten named Cotton.
Cotton lived high up in nice warn place above the barn where all of the farmer's horses slept.
But Cotton wasn't alone her little home above the barn, oh no.
She shared her hay bed with her mommy and 5 other sisters.
Figure 4: Examples of the log-mel spectrograms and the corresponding MFCC feature embedding. It can see that the log-mel spectrograms corresponds different example sentences from the Spoken- CoQA dataset.
13


Reviews:
[{"review_id": "zg8cWd3L90r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle the problem of spoken conversational question answering involving multiple turns of a dialogue, where the documents and questions are both in spoken and text form. The authors also compile a new dataset for spoken conversational QA with the help of text-to-speech systems.\n\nIn its current form, I think this work is fairly limited in its scope and is not yet ready to be published at ICLR. Other than ablations of the proposed technique and different choices of backbone networks for the proposed DDNet framework, since there are no comparisons made to prior approaches it is hard to assess the merits of the proposed approach. The authors also do not mention any plans of releasing the new dataset described in this work.\n\nThe draft will also benefit from a thorough editing pass; there are many typos (e.g., \"tailed for a specific domain\", \"nature language processing\", \"BRET-base\", etc).\n\nThree other comments:\n* It might be interesting to show how F1 scores vary on the test instances as a function of the number of turns in the conversation. Do the ASR errors compound to hurt performance on conversations with a large number of turns or is that not much of an issue? Similarly, showing how test F1 scores vary as a function of ASR accuracy of the spoken documents/questions would also be interestin\ng.\n* What is the error rate of the ASR system on the spoken documents and spoken questions? This will give the reader an idea of the accuracy of the transcriptions fed as input to the student model.\n* In Table 4, the F1 scores using SDNet are higher for S-CoQA compared to CoQA which is unexpected. Could the authors comment on why this might be?\n\n------------\n\nUpdate after author response: \n\nI've increased my score to 5. However, I still think this work is not ready to be published at ICLR in its current form. One of the other reviewers had raised an important point about the reliance of the proposed system on clean text which the authors should consider addressing in an updated version of this work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qFzbdVDzRh6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new task: spoken conversational question answering, which combines conversational question answering (e.g. CoQA) with spoken question answering (e.g. Spoken-SQuAD). The task is to answer a question (in written text) given a question that is given in both audio form and text form. They create a dataset for this task by combining CoQA with some off-the-shelf text-to-speech and speech-to-text models. They then propose a new model, DDNet, which obtains improved performance on their dataset.\n\nPros: \n\n-I buy the general argument made in the paper that relying solely on transcribed text from audio for answering the questions could be problematic, so I can see the value in the proposed dataset. I also thought the way the dataset is produced, by running CoQA through a text-to-speech model, was fairly clever. I’m not very familiar with the literature on spoken question answering to know if this is a common practice. \n\n-Another positive of the paper is that the DDNet architecture does seem to improve the performance on their dataset by a fairly large amount (though no error bars are given). \n\n-I appreciate that the paper conducts ablations on the different model components.  \n\nCons:\n-In my view, a drawback of this paper is that it is a bit difficult to read (partially due to grammatical errors). I found much of the motivation for this new task to not be clear from reading the paper. For example, I found the following excerpt difficult to parse:\n“Unlike existing SQA datasets, Spoken-CoQA is a multi-turn conversational SQA dataset, which is more challenging than single-turn benchmarks. First, every question is dependent on the conversation history in the Spoken-CoQA dataset. It is thus difficult for the machine to parse. Second, errors in ASR modules also degrade the performance of machines in tackling contextual understanding with context paragraph.“\n\n-Similarly, I found Figure 1 to be very confusing.\n\n\n-As far as I can tell, the paper doesn't compare to any other baselines that incorporate the audio information in a different way than DDNet. For example, the method from Serdyuk et al. (2018) could be considered. Strangely, the paper claims that this paper was ‘concurrent’, despite the fact that it was published in 2018, which is very confusing to me.\n\n-It’s unclear why the Spoken-CoQA dataset has to include text transcripts as well as the audio --- to me, it makes more sense for that to be part of the model solving the dataset.\n\n- Ideally the dataset would have natural speech, instead of synthetic speech, but I don’t consider this a major limitation.\n\nOverall:\nI think this is a borderline paper, erring on the side of rejection. My main concern is the lack of audio-based baselines other than DDNet, and the clarity of the paper.\n\n******* Update after reading rebuttal ********\nI appreciate the additional comparisons to prior work added by the authors. I've raised my score to a 6, but I still view this as a borderline paper due to concerns about clarity and impact, along with the  concerns raised by Reviewer 3.", "extra_scores": {}, "extra_text": {}}, {"review_id": "evkZQIWqsX1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies spoken conversational QA and its main contributions includes:\n\n- It proposes a task for end-to-end spoken QA. The new dataset Spoken-CoQA is derived from CoQA with additional features including audio data and ASR transcripts. \n\n- It proposes a method utilizing data distillation to learn from speech and text jointly. \n\nThe paper is technically sound and well structured. However, maybe I missed something, but I am not entirely sure if the contributions and its novelty warrant an accept. \n\nWhile the two methods, (1) the cross attention mechanism for speech and text embedding fusion and (2) knowledge distillation for combatting ASR errors, both bring improvement to the base models, they are existing, well-studied methods. The most important contribution of this paper, in my opinion, is the construction of the Spoken-CoQA dataset. But it seems the dataset isn't made available. \n\nBetween knowledge distillation and cross attention, it seems the former brings a larger improvement while the latter generally only has a marginal effect (Table 4). On this result, I have two questions: (1) Can these two mechanisms be combined (to reach an even better performance)? (2) Is it correct to say that the textual input is more useful than the audio input in this dataset? This might be an interesting question especially as the audio input is much larger and hence more difficult to process.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3swdK-jWmVG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---------------\nSummary\n---------------\n\nIn this paper, the authors release a new dataset -  Spoken-CoQA which includes an ASR based version of the popular CoQA dataset. The dataset has been created by running the Google TTS system followed by ASR using CMU Sphinx, to create a speech-transcribed versions of the dataset. The dataset includes the corresponding TTS audio recordings. Since the transcribed dataset has transcription errors, existing reading comprehension models do not work well. Thus, the paper introduces a joint audio-textual model for QA on the Spoken-CoQA dataset that uses TTS recordings its corresponding ASR output. \n\nThe model encodes audio data using the recent Speech-BERT model while the text is encoded using regular BERT. Similar to visio-linguistic models such as VilBERT, the model uses cross-attention on the intermediate audio and textual representations. This is done by using query matrix of a transformer block from one modality and applying it for attention using the key and value matrices on the other. That is, query keys from text are used to attend over keys and values from audio and vice-versa. The cross-attended representations are concatenated and used as input to any \"CMRC\" module. The authors experiment with two \"CMRC\" modules - SDNet and FlowQA. These are existing models developed for QA on CoQA, to return span-extracted answers from the input passage. Note that the dataset constructed does not update the answer spans based on the noisy ASR text and continues to assume answer-spans as per the actual text. Thus, to address this the authors include a knowledge distillation (KD) layer where the teacher network uses the gold speech transcriptions with audio, while the student layer uses the noisy transcriptions.  \n\nThe experiments have been presented using SDNeT, FlowQA as the CMRC modules and also by using BERT and ALBERT as the alternative CMRC modules. Models have been trained and cross-tested using CoQA and Spoken-CoQA. As expected, configurations with cross-testing report a deterioration in performance. For models trained using Spoken-CoQA, all models benefit from using audio-textual cross-attention as well as KD. \n\n--------------------------------------\nStrengths and Weaknesses\n---------------------------------------\nThe paper is interesting overall - while the model by itself is a trivial combination of existing multi-model methods, they result in upto 2pt improvements. I'd be willing to overlook this aspect, but to me the biggest weakness of this paper is in its data construction. It appears, when the ASR output is noisy -- the spans refer to ghost token positions (based on the clean text) -- example in Table 2 where the first question is unanswerable in span positions since the word \"white\" does not exist in the ASR text. The authors also allude to this when they motivate the KD layer, but my concern is what does one learn with this data? Any existing CMRC model trained on this data is obviously going to be bad -- it cant learn anything meaningful. Similarly, even when one employs the audio-textual model why should the model learn to predict the wrong span? I guess that is why the models also dont improve much - with the use of audio-textual encoding and KD. I'm not sure what can be done about this -- perhaps instead of using the text-based spans, one could return the audio segments  as answers? That might be more meaningful but will require annotation. In fact doing so may also remove the need of the KD layer which in some ways tries to (incorrectly) fix the problem by also showing it the gold clean transcription (original passage) . However, I'd argue that  relying on the clean text kind of defeats the purpose of speech-based conversational QA as motivated in this paper.\n\nSpeech based conversation QA is an important problem and the authors make a good first attempt, but unfortunately the paper falls short of delivering a usable dataset for this task. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a dataset and a method for the task of SpokenQA. The dataset is generated by using Google TTS to generate audio segments corresponding to the CoQA dataset and then using an ASR system to generate (noisy) transcripts of these speech segments. The authors then propose a method which uses a combination of various known techniques. 

Pros:
- A good first attempt at creating an interesting dataset for a useful task

Cons:
- Lack of clarity in writing
- Use of original clean text as input to the model which beats the purpose (in a natural setting such clean text will not be available)

All reviewers have appreciated the effort and attempt at creating a new dataset for this task. However, they have also pointed out that paper is not very clearly written and some important issues (use of clean text as input to the model) need to be adequately addressed before the paper is ready for acceptance. 
Decision: Reject