SlowGuess/ACL_2020_OCR huggingface dataset:
Jakob D. Havtorn Jan Latko Joakim Edin Lasse Borgholt Lars Maaloe
Lorenzo Belgrano Nicolai F. Jacobsen Regitze Sdun Željko Agić
Corti
Store Strandstraße 21, 4
1255 Copenhagen K, Denmark
jdh@corti.ai
We address a challenging and practical task of labeling questions in speech in real time during telephone calls to emergency medical services in English, which embeds within a broader decision support system for emergency call-takers. We propose a novel multimodal approach to real-time sequence labeling in speech. Our model treats speech and its own textual representation as two separate modalities or views, as it jointly learns from streamed audio and its noisy transcription into text via automatic speech recognition. Our results show significant gains of jointly learning from the two modalities when compared to text or audio only, under adverse noise and limited volume of training data. The results generalize to medical symptoms detection where we observe a similar pattern of improvements with multimodal learning.
Our paper addresses the challenge of learning to discover and label questions in telephone calls to emergency medical services in English. The task is demanding in two key aspects:
1. Noise: A typical phone call to an emergency medical service differs significantly from data within most standard speech datasets. Most importantly, emergency calls are noisy by nature due to very stressful conversations conveyed over poor telephone lines. Automatic speech recognition (ASR) and subsequent text processing quickly becomes prohibitive in such noisy environments, where word error rates (WER) are significantly higher than for standard benchmark data (Han et al., 2017). For this reason, we propose a sequence labeler that makes use of two modalities of a phone call: audio and its transcription into text by utilizing an ASR model. Hereby we create a multimodal
Figure 1: A speech sequence from our phone call dataset. Two audio segments are highlighted: a question (in blue) and a reported symptom (in yellow).
architecture that is more robust to the adverse conditions of an emergency call.
2. Real-time processing: Our model is required to work incrementally to discover questions in real time within incoming streams of audio in order to work as a live decision support system. At runtime, no segmentation into sub-call utterances such as phrases or sentences is easily available. The lack of segmentation coupled with the real-time processing constraint makes it computationally prohibitive to discover alignments between speech and its automatic transcription. For these reasons, we cannot utilize standard approaches to multimodal learning which typically rely on near-perfect cross-modal alignments between short and well-defined segments (Baltrusaitis et al., 2018).
Context and relevance. Learning to label sequences of text is one of the more thoroughly explored topics in natural language processing. In recent times, neural networks are applied not only to sequential labeling like part-of-speech tagging (Plank et al., 2016) or named entity recognition (Ma and Hovy, 2016), but also to cast into a labeling framework otherwise non-sequential tasks such as syntactic parsing (Gómez-Rodríguez and Vilares, 2018; Strzyz et al., 2019).
By contrast, assigning labels to audio sequences
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2370-2380 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
of human speech is comparatively less charted out. When addressed, speech labeling typically adopts a solution by proxy, which is to automatically transcribe speech into text, and then apply a text-only model (Surdeanu et al., 2005; Mollá et al., 2007; Eidelman et al., 2010). The challenge then becomes not to natively label speech, but to adapt the model to adverse conditions of speech recognition error rates. Such models typically feature in end-to-end applications such as dialogue state tracking (Henderson et al., 2014; Ram et al., 2018). Recent advances in end-to-end neural network learning offer promise to directly label linguistic categories from speech alone (Ghannay et al., 2018). From another viewpoint, multimodal learning is successfully applied to multimedia processing where the modalities such as text, speech, and video are closely aligned. However, contributions there typically feature classification tasks such as sentiment analysis and not finer-grained multimedia sequence labeling (Zadeh et al., 2017).
Our contributions. We propose a novel neural architecture to incrementally label questions in speech by learning from its two modalities or views: the native audio signal itself and its transcription into noisy text via ASR.
1. Our model utilizes the online temporal alignment between the input audio signal and its raw ASR transcription. By taking advantage of this fortuitous real-time coupling, we avoid having to learn the multimodal alignment over the entire phone call and its transcript, which would violate the real-time processing constraint that is crucial for decision support.
2. We achieve consistent and significant improvements from learning jointly from the two modalities compared to ASR transcriptions and audio only. The improvements hold across two inherently different audio sequence labeling tasks.
3. Our evaluation framework features a challenging real-world task with noisy inputs and real-time processing requirements. Under this adversity, we find questions and medical symptoms in emergency phone calls with high accuracy. Our task is illustrated in Figure 1.
2 Multimodal speech labeling
We define the multimodal speech labeler MultiQT as a combination of three neural networks that we apply to a number of temporal input modalities.
In our case, we consider speech and associated machine transcripts as the separate modalities or views. The model is illustrated in Figure 2.
To obtain temporal alignment between speech and text, we propose a simple approach that uses the output of an ASR system as the textual representation. Here, we take the ASR to be a neural network trained with the connectionist temporal classification (CTC) loss function (Graves et al., 2006). Given audio, it produces a temporal softmax of length  $T_{s}$  with a feature dimension defined as a categorical distribution, typically over characters, words or subword units, per timestep.
We refer to a sequence of input representations of the audio modality as  $(\mathbf{x}_a^{(t)})_{t\in [1..T_a]}$  and of the textual modality as  $(\mathbf{x}_s^{(t)})_{t\in [1..T_s]}$ . From the input sequences we compute independent unimodal representations denoted by  $\mathbf{z}_a^{(t)}$  and  $\mathbf{z}_s^{(t)}$  by applying two unimodal transformations denoted by  $f_{a}$  and  $f_{s}$ , respectively. Each of these transformations is parameterized by a convolutional neural network with overall temporal strides  $s_a$  and  $s_s$  and receptive fields  $r_a$  and  $r_s$ . With  $T_{m}$  as length of the resulting unimodal representations:
$$
\mathbf {z} _ {a} ^ {(t)} = f _ {a} \left(\left(\mathbf {x} _ {a} ^ {(i)}\right) _ {i = s _ {a} t - r _ {a, l}} ^ {s _ {a} t + r _ {a, r}}\right) \tag {1}
$$
$$
\mathbf {z} _ {s} ^ {(t)} = f _ {s} \left(\left(\mathbf {x} _ {s} ^ {(i)}\right) _ {i = s _ {s} t - r _ {s, l}} ^ {s _ {s} t + r _ {s, r}}\right),
$$
for  $t \in [1..T_m]$ , where  $r_{a,l}, r_{a,r}, r_{s,l}$  and  $r_{s,r}$  are the left and right half receptive fields of  $f_a$  and  $f_s$ , respectively. For  $f_a, r_{a,l} = \lfloor (r_a - 1)/2 \rfloor$  and  $r_{a,r} = \lceil (r_a - 1)/2 \rceil$  and similarly for  $f_s$ . For  $i < 1$  and  $i > T_a$  we define  $\mathbf{x}_a^{(i)}$  and  $\mathbf{x}_s^{(i)}$  by zero padding, effectively padding with half the receptive field on the left and right sides of the input. This then implies that  $T_m = \lfloor T_a / s_a \rfloor = \lfloor T_s / s_s \rfloor$  which constrains the strides according to  $T_a$  and  $T_s$  and functions as "same padding". This lets us do convolutions without padding the internal representations for each layer in the neural networks, which in turn allows for online streaming.
To form a joint multimodal representation from  $\mathbf{z}_a^{(t)}$  and  $\mathbf{z}_s^{(t)}$  we join the representations along the feature dimension. In the multimodal learning literature such an operation is sometimes called fusion (Zadeh et al., 2017). We denote the combined multimodal representation by  $\mathbf{z}_m^{(t)}$  and obtain it in a time-binded manner such that for a certain timestep
Figure 2: MultiQT model illustration for two timesteps  $i$  and  $j$ . We depict the convolutional transformations  $f_{a}$  and  $f_{s}$  of the audio and character temporal softmax inputs into the respective modality encodings  $\mathbf{z}_a^{(i)}$  and  $\mathbf{z}_s^{(i)}$ , along with the corresponding receptive fields and strides:  $r_a, s_a$  and  $r_s, s_s$ . The convolutions are followed by multimodal fusion and finally dense layers  $g$  and  $h$  to predict the question labels  $\hat{\mathbf{y}}^{(i)}$  and  $\hat{\mathbf{y}}^{(j)}$ .
$\mathbf{z}_m^{(t)}$  only depends on  $\mathbf{z}_a^{(t)}$  and  $\mathbf{z}_s^{(t)}$
$$
\mathbf {z} _ {m} ^ {(t)} = \operatorname {f u s i o n} \left(\mathbf {z} _ {a} ^ {(t)}, \mathbf {z} _ {s} ^ {(t)}\right). \tag {2}
$$
In our experiments fusion  $(\cdot)$  either denotes a simple concatenation,  $[\mathbf{z}_a^{(t)};\mathbf{z}_s^{(t)}]$ , or a flattened outer product,  $[1\mathbf{z}_a^{(t)}]\otimes [1\mathbf{z}_s^{(t)}]$ . The latter is similar to the fusion introduced by Zadeh et al. (2017), but we do not collapse the time dimension since our model predicts sequential labels.
Finally,  $\mathbf{z}_m^{(t)}$  is transformed before projection into the output space:
$$
\mathbf {z} _ {y} ^ {(t)} = g \left(\mathbf {z} _ {m} ^ {(t)}\right), \tag {3}
$$
$$
\hat {\mathbf {y}} ^ {(t)} = h \left(\mathbf {z} _ {y} ^ {(t)}\right), \tag {4}
$$
where  $g$  is a fully connected neural network and  $h$  is a single dense layer followed by a softmax activation such that  $\hat{\mathbf{y}}^{(t)}\in \mathbb{R}^K$  is a vector of probabilities summing to one for each of the  $K$  output categories. The predicted class is  $\arg \max (\hat{\mathbf{y}}^{(t)})$ .
2.1 Objective functions
In general, the loss is defined as a function of all learnable parameters  $\Theta$  and is computed as the average loss on  $M$  examples in a mini-batch. We denote by  $\{\mathcal{X}_a,\mathcal{X}_s\}$  a dataset consisting of  $N$  pairs of input sequences of each of the two modalities. As short-hand notation, let  $\mathbf{X}_a^{(n)}$  refer to the  $n$ 'th audio sequence example in  $\mathcal{X}_a$  and similarly for
$\mathbf{X}_s^{(n)}$ . The mini-batch loss is then
$$
\begin{array}{r l} \mathcal {L} \left(\Theta ; \left\{\mathbf {X} _ {a} ^ {(n)}, \mathbf {X} _ {s} ^ {(n)} \right\} _ {n \in \mathcal {B} _ {i}}\right) = & \\ \frac {1}{M} \sum_ {n \in \mathcal {B} _ {i}} \mathcal {L} ^ {(n)} \left(\Theta ; \mathbf {X} _ {a} ^ {(n)}, \mathbf {X} _ {s} ^ {(n)}\right), & \end{array} \tag {5}
$$
where  $\mathcal{B}_i$  is an index set uniformly sampled from  $[1..N]$  which defines the  $i$ 'th batch of size  $|\mathcal{B}_i| = M$ .
The loss for each example,  $\mathcal{L}^{(n)}$ , is computed as the time-average of the loss per timestep,
$$
\begin{array}{c} \mathcal {L} ^ {(n)} \left(\Theta ; \mathbf {X} _ {a} ^ {(n)}, \mathbf {X} _ {s} ^ {(n)}\right) = \\ \frac {1}{T} \sum_ {t = 1} ^ {T} \mathcal {L} ^ {(n, t)} \left(\Theta ; \mathbf {X} _ {a} ^ {(n, \mathbf {t} _ {a})}, \mathbf {X} _ {s} ^ {(n, \mathbf {t} _ {s})}\right), \end{array} \tag {6}
$$
where  $\mathbf{t}_a = [s_a t - r_{a,l} \ldots s_a t + r_{a,r}]$  and similarly for  $\mathbf{t}_s$  since the dependency of the loss per timestep is only on a limited timespan of the input. The loss per timestep is defined as the categorical cross-entropy loss between the softmax prediction  $\hat{\mathbf{y}}^{(t)}$  and the one-hot encoded ground truth target  $\mathbf{y}^{(t)}$ ,
$$
\mathcal {L} ^ {(n, t)} \left(\Theta ; \mathbf {X} _ {a} ^ {(n, \mathbf {t} _ {a})}, \mathbf {X} _ {s} ^ {(n, \mathbf {t} _ {s})}\right) = \sum_ {k = 1} ^ {K} y _ {k} ^ {(t)} \log (\hat {y} _ {k} ^ {(t)}).
$$
The full set of learnable parameters  $\Theta$  is jointly optimized by mini-batch stochastic gradient descent.
2.2 Multitask objective
In addition to the loss functions defined above, we also consider multitask training. This has been reported to improve performance in many different domains by including a suitably related auxiliary task (Bingel and Søgaard, 2017; Martínez Alonso and Plank, 2017).
For the task of labelling segments in the input sequences as pertaining to annotations from among a set of  $K - 1$  positive classes and one negative class, we propose the auxiliary task of binary labelling of segments as pertaining to either the negative class or any of the  $K - 1$  positive classes. For question tracking, this amounts to doing binary labelling of segments that are questions of any kind. The hope is that this will make the training signal stronger since the sparsity of each of the classes, e.g. questions, is reduced by collapsing them into one shared class.
We use the same loss function as above, but with the number of classes reduced to  $K = 2$ . The total
Label	Description	Example	Count	Fraction
Q1	Question about the address of the incident.	What's the address?	663	26.3%
Q2	Initial question of the call-taker to begin assessing the situation.	What's the problem?	546	21.6%
Q3	Question about the age of the patient.	How old is she?	537	21.3%
Q4	All questions related to patient's quality of breathing.	Is she breathing in a normal pattern?	293	11.6%
Q5	All question about patient's consciousness or responsiveness.	Is he conscious and awake?	484	19.2%
Table 1: Explanation and prevalence of the questions used for the experiments.
multitask loss is a weighted sum of the  $K$ -class loss and the binary loss:
$$
\mathcal {L} _ {\mathrm {M T}} ^ {(n, t)} = \beta \mathcal {L} _ {\text {b i n a r y}} ^ {(n, t)} + (1 - \beta) \mathcal {L} ^ {(n, t)}. \tag {7}
$$
The tunable hyperparameter  $\beta \in [0,1]$  interpolates the task between regular  $K$ -class labeling for  $\beta = 0$  and binary classification for  $\beta = 1$ .
Our dataset consists of 525 phone calls to an English-speaking medical emergency service. The call audio is mono-channel, PCM-encoded and sampled at  $8000\mathrm{Hz}$ . The duration of the calls has the mean of  $166\mathrm{s}$  (st. dev.  $65\mathrm{s}$ , IQR  $52\mathrm{s}$ ). All calls are manually annotated for questions by trained native English speakers. Each question is annotated with its start and stop time and assigned with one of 13 predefined question labels or an additional label for any question that falls outside of the 13 categories. Figure 1 illustrates these annotations. We observe an initial inter-annotator agreement of  $\alpha = 0.8$  (Krippendorff, 2018). Each call has been additionally corrected at least once by a different annotator to improve the quality of the data. On average it took roughly 30 minutes to annotate a single call. For our experiments, we choose the five most frequent questions classes, which are explained in Table 1. Out of 24 hours of calls, the questions alone account for only 30 minutes (roughly  $2\%$ ) of audio. For the experiments we use 5-fold cross-validation stratified by the number of questions in each call, such that calls of different lengths and contents are included in all folds.
We test our model on an additional speech sequence labeling challenge: tracking mentions of medical symptoms in incoming audio. By using another task we gauge the robustness of MultiQT as a general sequence labeling model and not only a question tracker, since symptom utterances in speech carry inherently different linguistic features than questions. As our question-tracking data was not manually labeled for symptoms, we created silver-standard training and test sets automatically
by propagating a list of textual keywords from the ground truth human transcripts back onto the audio signal as time stamps with a rule-based algorithm. The initial list contained over 40 medical symptoms, but in the experiment we retain the most frequent five: state of consciousness, breathing, pain, trauma, and hemorrhage.
The utterances that we track are complex phrases with a high variance: There are many different ways to express a question or a medical symptom in conversation. This linguistic complexity sets our research apart from most work in speech labeling which is much closer to exact pattern matching (Salamon and Bello, 2017).
Inputs. The audio modality is encoded using 40 log-mel features computed with a window of  $0.02\mathrm{s}$  and stride  $0.01\mathrm{s}$ .
The textual modality is formed by application of an ASR system to the audio modality. In all reported experiments, only ASR outputs are used and never human transcriptions, both in training and evaluation. The audio input to the ASR is encoded in the same way as described above. The ASR available to us has a purely convolutional architecture similar to the one in (Collobert et al., 2016) with an overall stride of 2. For MultiQT, this means that  $T_{a} = 2T_{s}$ . The ASR is trained on 600 hours of phone calls to medical emergency services in English from the same emergency service provider as the question and symptoms tracking datasets. Both of these are contained in the ASR test set. The ASR is trained using the connectionist temporal classification (CTC) loss function (Graves et al., 2006) and has a character error rate of  $14\%$  and a word error rate of  $31\%$ . Its feature dimension is 29 which corresponds to the English alphabet including apostrophe, space and a blank token for the CTC loss.
Systems. The basic version of MultiQT uses a single softmax cross-entropy loss function and forms a time-bound multimodal representation by concatenating the unimodal representations. We then augment this model in three ways:
1. MultiQT-TF: tensor fusion instead of concatenation following Zadeh et al. (2017),
2. MultiQT-MT: auxiliary binary classification with  $\beta = 0.5$
3. MultiQT-TF-MT: combination of 1 and 2.
Baselines. MultiQT can easily be adapted to a single modality by excluding the respective convolutional transformation  $f_{a}$  or  $f_{s}$ . For example, MultiQT can be trained unimodally on audio by removing  $f_{s}$  and then defining  $\mathbf{z}_{m}^{(t)} = \mathbf{z}_{a}^{(t)}$  instead of concatenation or tensor fusion. We baseline the multimodal MultiQT models against versions trained unimodally on audio and text. We also compare MultiQT to two distinct baseline models:
1. Random forest (RF)
2. Fully connected neural network (FNN)
Contrary to MultiQT, the baselines are trained to classify an input sequence into a single categorical distribution over the labels. At training, the models are presented with short segments of call transcripts in which all timesteps share the same label such that a single prediction can be made. The baselines are trained exclusively on text and both models represent the windowed transcript as a TF-IDF-normalized bag of words similar to Zhang et al. (2015). The bag of words uses word uni- and bigrams, and character tri-, four- and five-grams with 500 of each selected by  $\chi^2$ -scoring between labels and transcripts on the training set.
Hyperparameters. We use 1D convolutions for  $f_{a}$  and  $f_{s}$ . For  $f_{a}$  we use three layers with kernel sizes of 10, 20 and 40, filters of 64, 128 and 128 units and strides of 2, 2 and 2 in the first, second and third layer, respectively. For  $f_{s}$  we use two layers with kernel sizes of 20 and 40, filters of 128 and 128 units and strides of 2 and 2. Before each nonlinear transformation in both  $f_{a}$  and  $f_{s}$  we use batch normalization (Ioffe and Szegedy, 2015) with momentum 0.99 and trainable scale and bias, and we apply dropout (Srivastava et al., 2014) with a dropout rate of 0.2. For  $g$  we use three fully connected layers of 256 units each and before each nonlinear transformation we use batch normalization as above and apply dropout with a
dropout rate of 0.4. We  $l_{2}$  regularize all learnable parameters with a weighting of 0.1.
The FNN model uses the same classifier as is used for  $g$  in MultiQT with a dropout rate of 0.3 and an  $l_{2}$  regularization factor of 0.05.
All neural models are trained with the Adam optimizer (Kingma and Ba, 2015) using a learning rate of  $1 \times 10^{-4}$ ,  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  and batch size 6 except for those with tensor fusion which use a batch size of 1 due to memory constraints. Larger batch sizes were prohibitive since we use entire calls as single examples but results were generally consistent across different batch sizes. All hyperparameters were tuned manually and heuristically. It takes approximately one hour to train the base MultiQT model on one NVIDIA GeForce GTX 1080 Ti GPU card.
Evaluation. For each model we report two F1 scores with respective precisions and recalls macroaveraged over the classes.
- TIMESTEP: For each timestep, the model prediction is compared to the gold label. The metrics are computed per timestep and micro-averaged over the examples. This metric captures the model performance in finding and correctly classifying entire audio segments that represent questions and is sensitive to any misalignment.
-INSTANCE: A more forgiving metric which captures if sequences of the same label are found and correctly classified with acceptance of misalignment. Here, the prediction counts as correct if there are at least five consecutive correctly labeled time steps within the sequence, as a heuristic to avoid ambiguity between classes. This metric also excludes the non-question label.
The baseline models are evaluated per TIMESTEP by labeling segments from the test set in a sliding window fashion. The size of the window varies from 3 to 9 seconds to encompass all possible lengths of a question with the stride set to one word. Defining the stride in terms of words is possible because the ASR produces timestamps for the resulting transcript per word.
Labeling accuracy. The results are presented in Table 2. They show that for any model variation, the best performance is achieved when using both audio and text. The model performs the worst when using only audio which we hypothesize to be due
Model	Modality	INSTANCE	TIMESTEP
P	R	F1	P	R	F1
RF-BOW	T	61.8±3.5	88.5±0.9	72.2±2.2	39.3±1.1	70.4±1.0	48.1±1.0
FNN-BOW	T	42.2±1.4	92.8±0.6	57.5±1.3	38.1±0.7	71.0±1.7	46.9±0.8
MultiQT	A	87.4±1.9	60.6±4.0	70.3±3.1	79.2±1.3	57.8±3.3	65.0±2.4
MultiQT	T	84.2±1.6	78.5±2.8	81.1±2.0	78.8±1.2	69.4±2.0	73.5±1.3
MultiQT	A+T	83.6±2.2	83.3±2.5	83.3±1.6	75.7±2.2	73.8±2.3	74.5±1.3
MultiQT-MT	A	84.6±5.1	57.4±3.9	66.2±2.9	77.7±5.6	56.0±2.8	62.8±2.0
MultiQT-MT	T	81.9±1.1	80.6±2.8	81.0±1.8	75.9±1.5	71.2±2.4	73.3±1.7
MultiQT-MT	A+T	85.2±2.7	83.2±1.2	84.1±2.0	78.5±2.5	74.0±0.7	76.0±1.1
MultiQT-TF	A+T	85.0±1.8	83.3±2.6	83.9±1.7	78.9±2.1	75.2±2.3	76.7±1.2
MultiQT-TF-MT	A+T	85.1±3.2	83.1±1.6	83.8±1.7	78.7±3.7	75.0±1.6	76.5±1.4
Table 2: Question tracking results on audio (A) and text (T) modalities with variations of MultiQT using modality concatenation (MultiQT) or tensor fusion (MultiQT-TF) and the auxiliary task (MultiQT-MT). The evaluation metrics are precision (P), recall (R), and (F1) at the macro level per TIMESTEP or INSTANCE. We report means and standard deviations for five-fold cross-validation runs. All F1 differences are statistically significant at  $p < 0.001$ , save for between MultitQT [T] & MultitQT-MT [T], and MultitQT [A+T] & MultitQT-TF-MT [A+T] ( $p \approx 0.64$ ). We employ the approximate randomization test with  $R = 1000$  and Bonferroni correction (Dror et al., 2018). Bold face indicates the highest F1 score within each metric and MultiQT model group.
to the increased difficulty of the task: While speech intonation may be a significant feature for detecting questions in general, discerning between specific questions is easier with access to transcribed keywords.