SlowGuess/NAACL_2022_OCR huggingface dataset:
Chenyu You\*, Nuo Chen\*, Fenglin Liu\*, Shen Ge\*, Xian Wu\*, Yuexian Zou
$\spadesuit$ Yale University  $\text{♥}$  Peking University  $\spadesuit$ Tencent
$\clubsuit$ chenyu.you@yale.edu  $\clubsuit$  {nuochen, fenglinliu98, zouyx} @pku.edu.cn  $\diamondsuit$  {shenge, kevinxwu} @tencent.com
In spoken question answering, the systems are designed to answer questions from contiguous text spans within the related speech transcripts. However, the most natural way that human seek or test their knowledge is via human conversations. Therefore, we propose a new Spoken Conversational Question Answering task (SCQA), aiming at enabling the systems to model complex dialogue flows given the speech documents. In this task, our main objective is to build the system to deal with conversational questions based on the audio recordings, and to explore the plausibility of providing more cues from different modalities with systems in information gathering. To this end, instead of directly adopting automatically generated speech transcripts with highly noisy data, we propose a novel unified data distillation approach, DDNET, which effectively ingests cross-modal information to achieve fine-grained representations of the speech and language modalities. Moreover, we propose a simple and novel mechanism, termed Dual Attention, by encouraging better alignments between audio and text to ease the process of knowledge transfer. To evaluate the capacity of SCQA systems in a dialogue-style interaction, we assemble a Spoken Conversational Question Answering (Spoken-CoQA) dataset with more than 40k question-answer pairs from 4k conversations. The performance of the existing state-of-the-art methods significantly degrade on our dataset, hence demonstrating the necessity of cross-modal information integration. Our experimental results demonstrate that our proposed method achieves superior performance in spoken conversational question answering tasks.
Conversational question answering (CQA) has been studied extensively over the past few years
within the natural language processing (NLP) communities (Zhu et al., 2018; Liu et al., 2019; Yang et al., 2019). Different from traditional question answering (QA) tasks, CQA aims to enable models to learn the representation of the context paragraph and multi-turn dialogues. Existing CQA methods (Huang et al., 2018a; Devlin et al., 2018; Xu et al., 2019; Gong et al., 2020) have achieved superior performances on several benchmark datasets, such as QuAC (Choi et al., 2018) and CoQA (Elgohary et al., 2018).
Current CQA research mainly focuses on leveraging written text sources in which the answer can be extracted from a large document collection. However, humans communicate with each other via spontaneous speech (e.g., meetings, lectures, online conversations), which convey rich information. Consider our multimodal experience, fine-grained representations of both audio recordings and text documents are considered to be of paramount importance. Thus, we learn to draw useful relations between modalities (speech and language), which enables us to form fine-grained multimodal representations for end-to-end speech-and-language learning problems in many real-world applications, such as voice assistant and chat robot.
In this paper, we propose a novel and challenging spoken conversational question answering task - SCQA. An overview pipeline of this task is shown in Figure 1. Collecting such a SCQA dataset is a non-trivial task, as in contrast to current CQA tasks, we build our SCQA with two main goals as follows: (1) SCQA is a multi-turn conversational spoken question answering task, which is more challenging than only text-based task; (2) existing CQA methods rely on a single modality (text) as the context source. However, plainly leveraging uni-modality information is naturally undesirable for end-to-end speech-and-language learning problems since the useful connections between speech and text are elusive. Thus, employing data from
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1219 - 1232 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: An illustration of flow diagram for spoken conversational question answering tasks with an example from our proposed Spoken-CoQA dataset.
Manual Transcript	ASR Transcript
Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton. Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept. But Cotton wasn't alone in her little home above the barn, oh no. She shared her hay bed with her mommy and 5 other sisters...	Once upon a time in a bar near farm house, there lived a little like captain named cotton. How to live tied up in a nice warm place above the bar and we're all of the farmers horses slapped. But caught in was not alone in her little home above the bar in now. She shared her hey bed with her mommy and 5 other sisters...
Q1: Did Cotton live alone?	ASR-Q1: Did caught in live alone?
A1: no	A1: no
R1: Cotton wasn't alone.	R1:Caught in wasn't alone.
Q2: Who did she live with?	ASR-Q2: Who did she live with?
A2: with her mommy and 5 sisters	A2: with her mommy and 5 sisters
R2: with her mommy and 5 other sisters	R2: with her mommy and 5 other sisters
Q3: What color were her sisters?	ASR-Q3: What color were her sisters?
A3: orange and white	A3: orange and white
R3: her sisters were all orange with beautiful white tiger stripes	R3: her sisters were all orange with beautiful white tiger stripes
Table 1: An example from Spoken-CoQA. We can observe large misalignment between the manual transcripts and the corresponding ASR transcripts. Note that the misalignment is in bold font and the example is the extreme case. For more dataset information, please see Section 5 and Appendix Section "More Information about Spoken-CoQA".
the context of another modality (speech) can allow us to form fine-grained multimodal representations for the downstream speech-and-language tasks; and (3) considering the speech features are based on regions and are not corresponding to the actual words, this indicates that the semantic inconsistencies between the two domains can be considered as the semantic gap, which requires to be resolved by the downstream systems themselves.
In order to provide a strong baseline for this challenging multi-modal spoken conversational question answering task, we first present a novel knowledge distillation (KD) method for the proposed SCQA task. Our intuition is that speech utterances and text contents share the dual nature property, and we can take advantage of this property to learn the correspondences between these two forms. Specifically, we enroll multi-modal knowledge into the teacher model, and then guide the student (only trained on noisy speech documents) to boost network performance. Moreover, considering that the
semantics of the speech features and the textual representations are usually inconsistent, we introduce a novel mechanism, termed Dual Attention, to encourage fine-grained alignments between audio and text to close the cross-modal semantic gap between speech and language. One example of cross-modal gap is shown in Table 1. The experimental results show that our proposed DDNET achieves remarkable performance gains in the SCQA task. To the best of our knowledge, we are the first work in spoken conversational question answering task.
- We propose Spoken Conversational Question Answering task (SCQA), and comprise Spoken-CoQA dataset for machine comprehension of spoken question-answering style conversations. To the best of our knowledge, our Spoken-CoQA is the first spoken conversational question answering dataset.
- We develop a novel end-to-end method based
Figure 2: An illustration of the architecture of DDNET. In training stage, we adopt the teacher-student paradigm to enable the student model (only trained on speech documents) to achieve good performance. As for test, we only use student model for inference.
on data distillation to learn both from speech and language domain. Specifically, we use the model trained on clear texts as well as recordings to guide the model trained on noisy speech transcriptions. Moreover, we propose a novel Dual Attention mechanism to align the speech features and textual representations in each domain.
- We demonstrate that, by applying our proposed DDNET on several previous baselines, we can obtain considerable performance gains on our proposed Spoken-CoQA dataset.
Text Question Answering. In recent years, the natural language processing research community has devoted substantial efforts to text question answering tasks (Huang et al., 2018a; Zhu et al., 2018; Xu et al., 2019; Zhang et al., 2020; Gong et al., 2020; Chen et al., 2020, 2021a). Within the growing body of work on machine reading comprehension, an important sub-task of text question answering, two signature attributes have emerged: the availability of large benchmark datasets (Choi et al., 2018; Elgohary et al., 2018; Reddy et al., 2019) and pre-trained language models (Devlin et al., 2018; Liu et al., 2019; Lan et al., 2020). However, these existing works typically focus on modeling the complicated context dependency in text form. In contrast, we focus on enabling the machine to build the capability of language recognition and dialogue modeling in both speech and text domains.
Spoken Question Answering. In parallel to the recent works in natural language processing (Huang et al., 2018a; Zhu et al., 2018), these trends have also been pronounced in the speech field (Chen et al., 2018; Haghani et al., 2018; Lugasch et al., 2019; Palogiannidi et al., 2020; You et al., 2021a,b,c,d, 2020a; Chen et al., 2021b; Xu et al., 2021; Su et al., 2020, 2021), where spoken question answering (SQA), an extended form of QA, has explored the prospect of machine comprehension in spoken form. Previous work on SQA typically includes two separate modules: automatic speech recognition (ASR) and text question answering. It involves transferring spoken content to ASR transcriptions, and then employs NLP techniques to handle speech tasks.
Existing methods (Tseng et al., 2016; Serdyuk et al., 2018; Su and Fung, 2020) focus on optimizing each module in a two-stage manner, where errors in the ASR module would result in severe performance loss. Lee et al. (2019) proved that utilizing clean texts can help model trained on the ASR transcriptions to boost the performance via domain adaptation. Chuang et al. (2019) cascaded the BERT-based models as a unified model, and then trained it in a joint manner of audio and text. However, the existing SQA methods aimed at solving a single question given the related passage, without building and maintaining the connections of different questions in the human conversations. In addition, we compare our Spoken-CoQA with existing SQA datasets (See Table 2). Unlike existing SQA datasets, Spoken-CoQA is a multi-turn con
Dataset	Conversational	Spoken	Answer Type
TOEFL (Tseng et al., 2016)	×	✓	Multi-choice
S-SQuAD (Li et al., 2018)	×	✓	Spans
ODSQA (Lee et al., 2018)	×	✓	Spans
S-CoQA	✓	✓	Free-form
Table 2: Comparison of Spoken-CoQA with existing spoken question answering datasets. S-SQuAD and S-CoQA denote Spoken-SQuAD and Spoken-CoQA, respectively.
versational SQA dataset, which is more challenging than single-turn benchmarks.
Knowledge Distillation. Hinton et al. (2015) introduced the idea of Knowledge Distillation (KD) in a teacher-student scenario. In other words, we can distill the knowledge from one model (massive or teacher model) to another (small or student model). Previous work has shown that KD can significantly boost prediction accuracy in natural language processing and speech processing (Kim and Rush, 2016; Hu et al., 2018; Huang et al., 2018b; Hahn and Choi, 2019; Liu et al., 2021b,a; Cheng et al., 2016b; Cheng and You, 2016; Cheng et al., 2016a; You et al., 2020b, 2021e, 2022b,a, 2018, 2019a,b; Lyu et al., 2018, 2019; Guha et al., 2020; Yang et al., 2020; Ma et al., 2021a,b), while adopting KD-based methods for SQA tasks has been less explored. In this work, our goal is to handle the SCQA tasks. More importantly, we focus the core nature property in speech and text: Can spoken conversational dialogues further assist the model to boost the performance? Finally, we incorporate the knowledge distillation framework to distill reliable dialogue flow from the spoken contexts, and utilize the learned predictions to guide the student model to train well on the noisy input data.
In this section, we propose the novel SCQA task and collect a Spoken-CoQA (S-CoQA) dataset, which uses the spoken form of multi-turn dialogues and spoken documents to answer questions in multi-turn conversations.
Given a spoken document  $D^{s}$ , we use  $D^{t}$  to denote the clean original text and  $D^{a}$  to denote the ASR transcribed document. We also have  $Q_{1:L}^{a} = \{q_{1}^{a}, q_{2}^{a}, \dots, q_{L}^{a}\}$ , which is a collection of  $L$ -turn ASR transcribed spoken questions  $Q_{1:L}^{s}$ , as well as  $A_{1:L}^{t} = \{a_{1}^{t}, a_{2}^{t}, \dots, a_{L}^{t}\}$  which are the corresponding answers to the questions in clean texts. The objective of SCQA task is
then to generate the answer  $a_{L}^{t}$  for question  $q_{L}^{a}$ , given document  $D^{a}$ , multi-turn history questions  $Q_{1:L-1}^{a} = \{q_{1}^{a}, q_{2}^{a}, \dots, q_{L-1}^{a}\}$ , and reference answers  $A_{1:L-1}^{t} = \{a_{1}^{t}, a_{2}^{t}, \dots, a_{L-1}^{t}\}$ . In other words, our task in the testing phase can be formulated as
$$
\left\{D ^ {s}, Q _ {1: L} ^ {s} \right\} \xrightarrow {\mathrm {A S R}} \left\{q _ {L} ^ {a}, D ^ {a}, Q _ {1: L - 1} ^ {a}, a _ {1: L - 1} ^ {t} \right\}\rightarrow a _ {L} ^ {t} \tag {1}
$$
Please note that in order to improve the performance, in the training phase, we make use of auxiliary information which are the clean texts of document  $D^{t}$  and dialogue questions  $Q^{t} = \{q_{1}^{t}, q_{2}^{t}, \ldots, q_{L}^{t}\}$ , to guide the training of student model. As a result, the training process could be formulated as below:
$$
\left. \begin{array}{c} \text {s t u d e n t :} \left\{D ^ {s}, Q _ {1: L} ^ {s} \right\} \xrightarrow {\text {A S R}} \left\{q _ {L} ^ {a}, D ^ {a}, Q _ {1: L - 1} ^ {a}, a _ {1: L - 1} ^ {t} \right\} \\ \text {t e a c h e r :} \left\{D ^ {t}, Q _ {1: L} ^ {t} \right\} \end{array} \right\} \to a _ {L} ^ {t}
$$
However, in the inference stage, these additional information of  $D^t$  and  $Q_{1:L}^{t}$  are not needed.
4 DDNet
In this section, we propose DDNET to deal with the SCQA task, which is illustrated in Figure 2. We first describe the embedding generation process for both audio and text data. Next, we propose Dual Attention to fuse the speech and textual modalities. After that, we present the major components of the DDNET module. Finally we describe a simple yet effective distillation strategy in the proposed DDNET to learn enriched representations in the speech-text domain comprehensively.
4.1 Embedding
Given spoken words  $S = \{s_1, s_2, \dots, s_m\}$  and corresponding clean text words  $T = \{t_1, t_2, \dots, t_n\}$ , we utilize Speech-BERT and Text-BERT to generate speech feature embedding  $\mathbf{E}_s = \{\mathbf{E}_{s1}, \mathbf{E}_{s2}, \dots, \mathbf{E}_{sm}\}$  and context word embedding  $\mathbf{E}_t = \{\mathbf{E}_{t1}, \mathbf{E}_{t2}, \dots, \mathbf{E}_{tn}\}^1$ , respectively. Concretely, for speech input, we first use vqvav2vec (Baevski et al., 2019) to transfer speech signals into a series of tokens, which is the standard tokenization procedure in speech related tasks. Next, use Speech-BERT (Chuang et al., 2019), a variant of BERT-based models retrained on our Spoken-CoQA dataset, to process the speech sequences for training. The text contents are embedded into a sequence of vectors via our text encoder - Text-BERT, with the same architecture of BERT-base (Devlin et al., 2018).
In our implementation, the padding strategy is used to keep m and n to be the same as the max sequence length.
4.2 Dual Attention
Dual Attention (DA) is proposed to optimize the alignment between speech and language domains by capturing useful information from the two domains. In particular, we first use cross attention to align speech and text representations in the initial stage. After that, we utilize contextualized attention to further align the cross-modal representations in the contextualized word-level. Finally, we employ the self-attention mechanism to form fine-grained audio-text representations.
Cross Attention. Inspired by ViLBERT (Lu et al., 2019), we apply the co-attention transformer layer, a variant of Self-Attention (Vaswani et al., 2017), as the Cross Attention module for the fusing of speech and text embeddings. The Cross Attention is implemented by the standard Attention module involving Multi-Head Attention (MHA) and Feed-Forward Network (FFN) (Vaswani et al., 2017) as below:
$$
\operatorname {A t t e n t i o n} (Q, K, V) = \operatorname {F F N} (\operatorname {M H A} (Q, K, V))
$$
$$
\operatorname {C r o s s A t t e n t i o n} \left(F _ {1}, F _ {2}\right) = \operatorname {A t t e n t i o n} \left(F _ {1}, F _ {2}, F _ {2}\right) \tag {2}
$$
where  $Q, K, V$  denote query, key, and value matrices, and  $F_{1}, F_{2}$  denote features from difference modalities, respectively. The co-attention module then use the Cross Attention function to compute the cross attention-pooled features, by querying one modality using the query vector of another modality.
$$
\begin{array}{l} \hat {\mathbf {E}} _ {s} ^ {\text {c r o s s}} = \text {C r o s s A t t e n t i o n} (\mathbf {E} _ {s}, \mathbf {E} _ {t}) \\ = \operatorname {A t t e n t i o n} \left(\mathbf {E} _ {s}, \mathbf {E} _ {t}, \mathbf {E} _ {t}\right), \tag {3} \\ \end{array}
$$
$$
\begin{array}{l} \hat {\mathbf {E}} _ {t} ^ {\mathrm {c r o s s}} = \operatorname {C r o s s A t t e n t i o n} (\mathbf {E} _ {t}, \mathbf {E} _ {s}) \\ = \operatorname {A t t e n t i o n} \left(\mathbf {E} _ {t}, \mathbf {E} _ {s}, \mathbf {E} _ {s}\right), \\ \end{array}
$$
where  $\hat{\mathbf{E}}_s^{\mathrm{cross}}\in \mathbb{R}^{n\times d}$ $\hat{\mathbf{E}}_t^{\mathrm{cross}}\in \mathbb{R}^{n\times d}$  and  $d$  is the dimension of feature vectors.
Contextualized Attention (CA). After obtaining speech-aware representation  $\hat{\mathbf{E}}_s^{\mathrm{cross}}$  and text-aware representation  $\hat{\mathbf{E}}_t^{\mathrm{cross}}$ , our next goal is to construct more robust contextualized cross-modal representations by integrating features from both modalities. The features with fused modalities are computed as follows:
$$
\mathbf {H} _ {\mathrm {C A}} = \operatorname {R e L U} \left(\hat {\mathbf {E}} _ {s} ^ {\text {c r o s s}} W _ {1} ^ {T}\right) \operatorname {R e L U} \left(\hat {\mathbf {E}} _ {x} ^ {\text {c r o s s}} W _ {1} ^ {T}\right) W _ {2} ^ {T}, \tag {4}
$$
where  $W_{1}, W_{2} \in \mathbb{R}^{n \times d}$  are trainable weights.
Self-Attention. To build a robust SCQA system, special attention needs to be paid on the sequential order of the dialogue, since the changes in utterances order may cause severely low-quality and in-coherent corpora. As a result, to capture the long-range dependencies such as co-references for the downstream speech-and-language tasks, similar to (Li et al., 2016; Zhu et al., 2018), we introduce a self-attention layer to obtain the final Dual Attention (DA) representations.
$$
\begin{array}{l} \mathbf {E} _ {\mathrm {D A}} = \text {S e l f A t t e n t i o n} \left(\mathbf {H} _ {\mathrm {C A}}\right) \tag {5} \\ = \operatorname {A t t e n t i o n} \left(\mathbf {H} _ {\mathrm {C A}}, \mathbf {H} _ {\mathrm {C A}}, \mathbf {H} _ {\mathrm {C A}}\right). \\ \end{array}
$$
4.3 Key Components
The framework of our SCQA module is similar to recent works (Zhu et al., 2018; Huang et al., 2017), which is divided into three key components: Encoding Layer, Attention Layer and Output Layer.
Encoding Layer. Then documents and conversations (questions and answers) are first converted into the corresponding feature embeddings (i.e., character embeddings, word embeddings, and contextual embedding). The output contextual embeddings are then concatenated by the aligned cross-modal embedding  $\mathbf{E}_{\mathrm{DA}}$  to form the encoded input features:
$$
\mathbf {E} _ {e n c} = \left[ \mathbf {E} _ {t}; \mathbf {E} _ {\mathrm {D A}} \right]. \tag {6}
$$
Attention Layer. We compute the attention on the context representations of the documents and questions, and extensively exploit correlations between them. Note that we adopt the default attention layers in four baseline models.
Output Layer. After obtaining attention-pooled representations, the Output Layer computes the probability distributions of the start and end index within the entire documents and predicts an answer to current question:
$$
\begin{array}{l} \mathcal {L} = - \log \mathcal {P} (\mathrm {s t} = a _ {L, \mathrm {s t}} | \mathbf {X}) \tag {7} \\ - \log \mathcal {P} (\mathrm {e d} = a _ {L, \mathrm {e d}} | \mathbf {X}) \\ \end{array}
$$
where  $\mathbf{X}$  denotes the input document  $D$  and  $Q^L$ , and "st", "ed" denote the start and end positions.
4.4 Knowledge Distillation
In previous speech-language models, the only guidance is the standard training objective to measure the difference between the prediction and the reference answer. However, for noisy ASR transcriptions, such criteria may not be suitable enough. To overcome such problem, we distill the knowledge
Domain	Passages	QA-Pairs	Passage Length	Avg.TURNS
Children	357	3.5k	212	9.8
Literature	898	8.7k	275	9.7
Mid./High School	878	9.5k	308	10.8
News	967	9.5k	271	9.8
Wikipedia	864	8.9k	249	10.3
Overall	3964	40.1k	270	10.1
Table 3: Statistical analysis on Spoken-CoQA.
from our teacher model, and use them to guide the student model to learn contextual features in our SCQA task. Concretely, we set the model trained on the speech document and the clean text corpus as the teacher model and trained on the ASR transcripts as the student model, respectively. Thus, the student trained on low-quality data learns to absorb the knowledge that the teacher has discovered. Given the  $z_{S}$  and  $z_{T}$  as the prediction vectors by the student and teacher models, the objective is defined as:
$$
\mathcal {L} _ {\mathrm {S C Q A}} = \sum_ {x \in \mathcal {X}} \left(\alpha \tau^ {2} \mathrm {K L} \left(p _ {\tau} \left(z _ {S}\right), p _ {\tau} \left(z _ {T}\right)\right) + (1 - \alpha) \mathcal {L}\right), \tag {8}
$$
where  $\mathrm{KL}(\cdot)$  denotes the Kullback-Leibler divergence.  $p_{\tau}(\cdot)$  is the softmax function with temperature  $\tau$  , and  $\alpha$  is a balancing factor.
In this section, we first describe the collection and filtering process of our proposed Spoken-CoQA dataset in detail. Next, we introduce several state-of-the-art language models as our baselines, and then evaluate the robustness of these models on our proposed Spoken-CoQA dataset. Finally, we provide a thorough analysis of different components of our method. Note that we use the default settings in all evaluated methods.
Data Collection. We detail the procedures to build Spoken-CoQA as follows. First, we select the conversational question-answering dataset CoQA (Reddy et al., 2019) $^2$  as our basis data since it is one of the largest public CQA datasets. CoQA contains around 8k stories (documents) and over 120k questions with answers. The average dialogue length of CoQA is about 15 turns, and the answers are in free-form texts. In CoQA, the training set and the development set contain 7,199 and 500 conversations over the given stories, respectively. Therefore, we use the CoQA training set as our reference text
of the training set and the CoQA development set as the test set in Spoken-CoQA.