SlowGuess/ACL_2020_OCR huggingface dataset:
Including the auxiliary binary classification task (MultiQT-MT) shows no significant improvement over MultiQT. We hypothesize that this may be due to training on a subset of all questions such that there are unlabelled questions in the training data which add noise to the binary task.
Applying tensor fusion instead of concatenating the unimodal representations also does not yield significant improvements to MultiQT contrary to the findings by Zadeh et al. (2017). Since tensorfusion subsumes the concatenated unimodal representations by definition and appends all elementwise products, we must conclude that the multimodal interactions represented by the element-wise products either already exist in the unimodal representations, by correlation, are easily learnable from them or are too difficult to learn for MultiQT. We believe that the interactions are most likely to be easily learnable from the unimodal representations.
Comparing any MultiQT variant with instances and TIMESTEP F1 clearly shows that instances is more forgiving, with models generally achieving higher values in this metric. The difference in performance between different combinations of the modalities is generally higher when measured per
INSTANCE as compared to per TIMESTEP.
The RF and FNN baseline models clearly underperform compared to MultiQT. It should be noted that both RF and FNN achieve F1-scores of around 85 when evaluated per input utterance, corresponding to the input they receive during training. On this metric, FNN also outperforms RF. However, both models suffer significantly from the discrepancy between the training and streaming settings as measured per the instances and TIMESTEP metrics; this effect is largest for the FNN model.
Real-time tracking. One important use case of MultiQT is real-time labelling of streamed audio sequences and associated transcripts. For this reason, MultiQT must be able to process a piece of audio in a shorter time than that spanned by the audio itself. For instance, given a 1 s chunk of audio, MultiQT must process this in less than 1 s in order to maintain a constant latency from the time that the audio is ready to be processed to when it has been processed. To assess the real-time capability of MultiQT, we test it on an average emergency call using an NVIDIA GTX 1080 Ti GPU card. In our data, the average duration of an emergency call is 166 s.
To simulate real-time streaming, we first process the call in 166 distinct one-second chunks using 166 sequential forward passes. This benchmark includes all overhead, such as the PCIe transfer of data to and from the GPU for each of the for
ward passes. The choice of 1 s chunk duration matches our production setting but is otherwise arbitrary with smaller chunks giving lower latency and larger chunks giving less computational overhead. In this streaming setting, the  $166\mathrm{s}$  of audio are processed in  $1.03\mathrm{s}$  yielding a real-time factor of approximately 161 with a processing time of  $6.2\mathrm{ms}$  per 1 s of audio. This satisfies the real-time constraint by a comfortable margin, theoretically leaving room for up to 161 parallel audio streams to be processed on the same GPU before the real-time constraint is violated.
When a single model serves multiple ongoing calls in parallel, we can batch the incoming audio chunks. Batching further increases the real-time factor and enables a larger number of ongoing calls to be processed in parallel on a single GPU. This efficiency gain comes at the cost of additional, but still constant, latency since we must wait for a batch of chunks to form. For any call, the expected additional latency is half the chunk duration. We perform the same experiment as above but with different batch sizes. We maintain super real-time processing for batches of up 256 one-second chunks, almost doubling the number of calls that can be handled by a single model.
In the offline setting, for instance for on-demand processing of historical recordings, an entire call can be processed in one forward pass. Here, MultiQT can process a single average call of  $166\mathrm{s}$  in  $10.9\mathrm{ms}$  yielding an offline real-time factor of 15,000. Although batched processing in this setting requires padding, batches can be constructed with calls of similar length to reduce the relative amount of padding and achieve higher efficiency yet.
Label confusion. We analyze the label confusion of the basic MultiQT model using both modalities on the TIMESTEP metric. Less than  $1\%$  of all incorrect timestamps correspond to question-to-question confusions while the two primary sources of confusion are incorrect labelings of 1) "None" class for a question and 2) of a question with the "None" class. The single highest confusion is between the "None" class and "Q4" which is the least frequent question. Here the model has a tendency to both over-predict and miss: ca  $40\%$  of predicted "Q4" are labeled as "None" and  $40\%$  of "Q4" are predicted as "None". In summary, when our model makes an error, it will most likely 1) falsely predict a non-question to
Figure 3: Error margin distributions for start and stop timestamps of question sequences. The dotted lines depict the ground truth start and stop timestamps.
be a question or 2) falsely predict a question to be a non-question; once it discovers a question, it is much less likely to assign it the wrong label.
Model disagreement. We examined the inter-model agreement between MultiQT trained on the different modes. The highest agreement of  $\sim 90\%$  is achieved between the unimodal text and the multimodal models whereas the lowest agreement was generally between the unimodal audio and any other model at  $\sim 80\%$ . The lower agreement with the unimodal audio model can be attributed to the generally slightly lower performance of this model compared to the other models as per Table 2.
Question margins. In Figure 3, we visualize the distribution of the errors made by the model per TIMESTEP. For each question regarded as matching according to the instance metric we compute the number of seconds by which the model mismatched the label sequence on the left and right side of the label sequence, respectively. We see that the model errors are normally distributed around a center value that is shifted towards the outside of the question by slightly less than  $100\mathrm{ms}$ . The practical consequence is that the model tends to make predictions on the safe side by extending question segments slightly into the outside of the question.
Modality ablation. To evaluate the model's robustness to noise in the modalities, we remove all information from one of the modalities in turn and report the results in Table 3. We remove the information in a modality by randomly permuting the entire temporal axis. This way we retain the numerical properties of the signal which is not the case when replacing a modality by zeros or noise. To increase MultiQT's robustness to this modality ablation, we apply it at training so that for each batch example we permute the temporal axis of the
Modality	Permuted	INSTANCE	TIMESTEP
Training	Test	P	R	F1	P	R	F1
A+T	Yes	T	82.2±4.9	60.1±5.6	68.6±5.7	79.0±4.7	58.4±3.7	64.7±3.5
A+T	Yes	A	82.6±3.2	75.9±2.9	78.7±1.6	78.3±2.4	68.3±2.7	72.3±1.1
A+T	Yes	-	86.3±1.6	83.8±2.8	84.8±2.0	80.4±1.0	74.1±2.2	76.9±1.3
A+T	No	T	0.0±0.0	0.0±0.0	0.0±0.0	16.2±0.0	16.7±0.0	16.4±0.0
A+T	No	A	89.5±3.1	69.2±4.4	77.0±2.5	84.3±2.6	63.7±3.5	71.0±2.0
A+T	No	-	83.6±2.2	83.3±2.5	83.3±1.6	75.7±2.2	73.8±2.3	74.5±1.3
A	No	-	87.4±1.9	60.6±4.0	70.3±3.1	79.2±1.3	57.8±3.3	65.0±2.4
T	No	-	84.2±1.6	78.5±2.8	81.1±2.0	78.8±1.2	69.4±2.0	73.5±1.3
Table 3: Results from the modality ablation on the MultiQT model. We compare multimodal MultiQT trained with the audio (A) and text (T) modalities temporally permuted in turn during training with probability  $p_a = 0.1$  and  $p_s = 0.5$  to MultiQT trained without modality permutation, unimodally and multimodally (some results copied from Table 2). We can obtain robustness to loosing a modality while maintaining (or even slightly improving) the multimodal performance. All results are based on five-fold cross-validation as in Table 2.
Figure 4: Relation between TIMESTEP F1 and WER on call-taker utterances without the "None" label.
audio or text modality with some probability  $p_a$  or  $p_s$ . We choose  $p_a = 0.1$  and  $p_s = 0.5$  since the model more easily develops an over-reliance on the text-modality supposedly due to higher signal-to-noise ratio. The results are listed in Table 3 along with results for MultiQT from Table 2 for easy reference. We observe that the basic MultiQT model suffers significantly from permutation of the text modality and less so for audio which suggests that it relies on the audio only for supportive features. Training MultiQT with the random temporal permutation forces learning of robustness to loosing all information in a modality. We see that the results when removing a modality almost reach the level achieved when training exclusively on that modality while still maintaining the same (or better) performance of the basic MultiQT model.
Relation to ASR. In Figure 4, we plot the performance of the multimodal model on different
subsets of the test split by the maximum WER of the ASR (measured only on the call-taker utterances). This evaluation compares the microaveraged model F1-score when increasing the noise on the textual input. We see that regardless of the modality, the performance is the highest for calls with very low WER. We observe that the performance improvement of using both modalities over unimodal text or unimodal audio increases as we include noisy samples. This implies that multi modality increases robustness. Training on permuted inputs additionally improves the performance on noisy data.
The evaluation of MultiQT in our paper has thus far been only in relation to one particular ASR model with CTC loss (Graves et al., 2006), where our system displays significant gains from multimodal learning. Yet, do these results hold with another ASR system, and in particular, are the multimodal gains still significant if WER decreases and produced text quality increases? For an initial probing of these questions, we replace the fully convolutional ASR with a densely-connected recurrent architecture with convolutional heads. This model is similar to the one in (Amodei et al., 2015) but also uses dense bottleneck layers. With this model the transcription quality improves by around  $+4\%$  in WER, while the F1-scores of MultiQT still strongly favor the multimodal approach, by  $+6.15$  points absolute over text-only. We argue that in a real-world scenario with high WER and limited in-domain training data, the gains warrant learning from joining the text and audio views on the input speech when learning a question tracker. Alterna-
tively, the ASR model itself could be extended into a multitask learning setup to jointly track questions and transcribe speech; we defer that line of work for future research. On a practical note, for this multitask approach, the data must be fully transcribed by human annotators in addition to the question annotations. This is generally more time consuming and expensive than exclusively annotating questions.
Qualitative analysis. We analyze the model predictions on a subset of 21 calls to identify the most likely reasons for incorrect labeling. We find that in over half of the analysed cases the incorrect prediction is triggered either by a question-related keyword uttered in a non-question sentence or by a question asked in the background by a caller that was not assigned a label. We also encounter undetected questions that have a very noisy ASR transcript or are asked in an unusual way.
Symptom labeling. The experiment with our silver-standard symptoms data shows a trend that is similar to question tracking: The dual-modality MultiQT scores an instance F1 score of 76.9 for a +1.8 absolute improvement over the best single modality. Text-only is the runner up (-1.8 F1) while audio-only lags behind with a significant -23.6 decrease in F1. At the same time, a simple text-only keyword matching baseline scores at 73.7. We argue that symptom tracking strongly favors text over audio because the distinctive audio features of questions, such as changes in intonation, are not present when communicating symptoms in speech.
The broader context of our work is to track the dialogue state in calls to emergency medical services, where conversations are typically formed as sequences of questions and answers that pertain to various medical symptoms. The predominant approach to dialogue state tracking (DST) in speech is to first transcribe the speech by using ASR (Henderson et al., 2014; Henderson, 2015; Mrkšić et al., 2017). In our specific context, to entirely rely on ASR is prohibitive because of significantly higher WER in comparison to standard datasets. To exemplify, while WER is normally distributed with a mean of  $37.6\%$  in our data, the noisiest DST challenge datasets rarely involve with WER above  $30\%$  (Jagfeld and Vu, 2017) while standard ASR benchmarks offer even lower WER (Park et al.,
2019). None of the standard ASR scenarios thus directly apply to a real-life ASR noise scenario.
From another viewpoint, work in audio recognition mainly involves with detecting simple single-word commands or keyword spotting (de Andrade et al., 2018), recognizing acoustic events such as environmental or urban sounds (Salamon et al., 2014; Piczak, 2015; Xu et al., 2016) or music patterns, or document-level classification of entire audio sequences (Liu et al., 2017). McMahan and Rao (2018) provide a more extensive overview. While approaches in this line of work relate to ours, e.g. in the use of convolutional networks over audio (Sainath and Parada, 2015; Salamon and Bello, 2017), our challenge features questions as linguistic units of significantly greater complexity.
Finally, research into multimodal or multi-view deep learning (Ngiam et al., 2011; Li et al., 2018) offers insights to effectively combine multiple data modalities or views on the same learning problem. However, most work does not directly apply to our problem: i) the audio-text modality is significantly under-represented, ii) the models are typically not required to work online, and iii) most tasks are cast as document-level classification and not sequence labeling (Zadeh et al., 2018).
We proposed a novel approach to speech sequence labeling by learning a multimodal representation from the temporal binding of the audio signal and its automatic transcription. This way we learn a model to identify questions in real time with a high accuracy while trained on a small annotated dataset. We show the multimodal representation to be more accurate and more robust to noise than the unimodal approaches. Our findings generalize to a medical symptoms labeling task, suggesting that our model is applicable as a general-purpose speech tagger wherever the speech modality is coupled in real time to ASR output.
The authors are grateful to the anonymous reviewers and area chairs for the incisive and thoughtful treatment of our work.
Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong
Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse H. Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. 2015. Deep speech 2: End-to-end speech recognition in english and mandarin. CoRR, abs/1512.02595.
Douglas Coimbra de Andrade, Sabato Leo, Martin Loesener Da Silva Viana, and Christoph Bernkopf. 2018. A neural attention model for speech command recognition. arXiv preprint arXiv:1808.08929.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423-443.
Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 164-169, Valencia, Spain. Association for Computational Linguistics.
Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve. 2016. Wav2letter: an end-to-end convnet-based speech recognition system. CoRR, abs/1609.03193.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics.
Vladimir Eidelman, Zhongqiang Huang, and Mary Harper. 2010. Lessons learned in part-of-speech tagging of conversational speech. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 821-831, Cambridge, MA. Association for Computational Linguistics.
Sahar Ghannay, Antoine Caubrière, Yannick Esteve, Antoine Laurent, and Emmanuel Morin. 2018. End-to-end named entity extraction from speech. arXiv preprint arXiv:1805.12045.
Carlos Gómez-Rodríguez and David Vilares. 2018. Constituent parsing as sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1314-1324, Brussels, Belgium. Association for Computational Linguistics.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist
temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pages 369-376, New York, NY, USA. ACM.
Kyu J Han, Seongjun Hahm, Byung-Hak Kim, Jungsuk Kim, and Ian R Lane. 2017. Deep learning-based telephony speech recognition in the wild. In INTER-SPEECH, pages 1323-1327.
Matthew Henderson. 2015. Machine learning for dialog state tracking: A review. Technical report.
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272.
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448-456, Lille, France. PMLR.
Glorianna Jagfeld and Ngoc Thang Vu. 2017. Encoding word confusion networks with recurrent neural networks for dialog state tracking. In Proceedings of the Workshop on Speech-Centric Natural Language Processing, pages 10-17, Copenhagen, Denmark. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications.
Yingming Li, Ming Yang, and Zhongfei Mark Zhang. 2018. A survey of multi-view representation learning. IEEE Transactions on Knowledge and Data Engineering.
Chunxi Liu, Jan Trmal, Matthew Wiesner, Craig Harman, and Sanjeev Khudanpur. 2017. Topic identification for speech without asr.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
Héctor Martínez Alonso and Barbara Plank. 2017. When is multitask learning effective? semantic sequence prediction under varying data conditions. In Proceedings of the 15th Conference of the European
Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 44-53, Valencia, Spain. Association for Computational Linguistics.
Brian McMahan and Delip Rao. 2018. Listening to the world improves speech command recognition. In Thirty-Second AAAI Conference on Artificial Intelligence.
Diego Mollá, Menno van Zaanen, and Steve Cassidy. 2007. Named entity recognition in question answering of speech data. In Proceedings of the Australasian Language Technology Workshop 2007, pages 57-65, Melbourne, Australia.
Nikola Mrkšić, Diarmuid Žáeghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788, Vancouver, Canada. Association for Computational Linguistics.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689-696.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. Interspeech 2019.
Karol J Piczak. 2015. Environmental sound classification with convolutional neural networks. In 2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6. IEEE.
Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412-418, Berlin, Germany. Association for Computational Linguistics.
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, and Art Pettigre. 2018. Conversational AI: the science behind the alexa prize. CoRR, abs/1801.03604.
Tara Sainath and Carolina Parada. 2015. Convolutional neural networks for small-footprint keyword spotting. Technical report.
Justin Salamon and Juan Pablo Bello. 2017. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Processing Letters, 24(3):279-283.