GenAI4ELab/papercli-papers huggingface dataset:
Thai Dialect Corpus and Transfer-based Curriculum Learning Investigation for Dialect Automatic Speech Recognition
Authors: Artit Suwanbandit; Burin Naowarat; Orathai Sangpetch; Ekapol Chuangsuwanich
Venue: Interspeech 2023 (main)

Abstract: We release 840 hours of read speech multi-dialect ASR corpora consisting of 700 hours of main Thai dialect, named Thai-central, and 40 hours for each local dialect , named Thai-dialect, with transcripts and their translations to Thai. The dialects, selected to represent different regions of Thailand, are Khummuang, Korat, and Pattani. We also release the baseline dialectal ASR models trained using the curriculum learning approach. We found that the pre-training with the high-resource main dialect and target dialect generally yields the best performance. We believe that the availability of our corpora would contribute to the problem of low-resource Thai dialects. The corpus data will be available on Github.
HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translation
Authors: Cihan Xiao; Henry Li Xinyuan; Jinyi Yang; Dongji Gao; Matthew Wiesner; Kevin Duh; Sanjeev Khudanpur
Venue: Interspeech 2023 (main)

Abstract: We introduce HK-LegiCoST, a new three-way parallel corpus of Cantonese-English translations, containing 600+ hours of Cantonese audio, its standard traditional Chinese transcript, and English translation, segmented and aligned at the sentence level. We describe the notable challenges in corpus preparation: segmentation, alignment of long audio recordings, and sentence-level alignment with non-verbatim transcripts. Such transcripts make the corpus suitable for speech translation research when there are significant differences between the spoken and written forms of the source language. Due to its large size, we are able to demonstrate competitive speech translation baselines on HK-LegiCoST and extend them to promising cross-corpus results on the FLEURS Cantonese subset. These results deliver insights into speech recognition and translation research in languages for which non-verbatim or ''noisy'' transcription is common due to various factors, including vernacular and dialectal speech.
A Metric-Driven Approach to Conformer Layer Pruning for Efficient ASR Inference
Authors: Dhanush Bekal; Karthik Gopalakrishnan; Karel Mundnich; Srikanth Ronanki; Sravan Bodapati; Katrin Kirchhoff
Venue: Interspeech 2023 (main)

Abstract: Conformer-based end-to-end automatic speech recognition (ASR) models have gained popularity in recent years due to their exceptional performance at scale. However, there are significant computation, memory and latency costs associated with running inference on such models. With the aim of mitigating these issues, we evaluate the efficacy of pruning Conformer layers while fine-tuning only on 20% of the data used for the pre-trained model. We score Conformer layers using correlation, energy, and gradient-based metrics and rank them to identify candidate layers for pruning. We also propose an iterative pruning strategy which monitors and prunes layers that are consistently ranked low by the metrics during training. Using our methods, we prune large pre-trained offline and online (streaming) models by 20% and 40% with little impact on performance, while outperforming a strong knowledge distillation baseline.
Distillation Strategies for Discriminative Speech Recognition Rescoring
Authors: Prashanth Gurunath Shivakumar; Jari Kolehmainen; Yile Gu; Ankur Gandhe; Ariya Rastrow; Ivan Bulyko
Venue: Interspeech 2023 (main)

Abstract: Second-pass rescoring is employed in most state-of-the-art speech recognition systems. Recently, BERT based models have gained popularity for re-ranking the n-best hypothesis by exploiting the knowledge from masked language model pre-training. Further, fine-tuning with discriminative loss such as minimum word error rate (MWER) has shown to perform better than likelihood-based loss. Streaming applications with low latency requirements impose significant constraints on the size of the models, thereby limiting the word error rate (WER) performance gains. In this paper, we propose effective strategies for distilling from large models discriminatively trained with the MWER objective. We experiment on Librispeech and production scale internal dataset for voice-assistant. Our results demonstrate relative improvements of upto 7% WER over student models trained with MWER. We also show that the proposed distillation can reduce the WER gap between the student and the teacher by 62% upto 100%.
Another Point of View on Visual Speech Recognition
Authors: Baptiste Pouthier; Laurent Pilati; Giacomo Valenti; Charles Bouveyron; Frederic Precioso
Venue: Interspeech 2023 (main)

Abstract: Standard Visual Speech Recognition (VSR) systems directly process images as input features without any apriori link between raw pixel data and facial traits. Pixel information is smartly sieved when facial landmarks are extracted from pictures and repurposed as graph nodes. Their evolution through time is thus modeled by a Graph Convolutional Network. However, with graph-based VSR being in its infancy, the selection of points and their correlation are still ill-defined and often bound to aprioristic knowledge and handcrafted techniques. In this paper, we investigate the graph approach for VSR and its ability to learn the correlation between points beyond the mouth region. We also study the different contributions that each facial region brings to the system accuracy, proving that more scattered but better connected graphs can be both computationally light and accurate.
RASR2: The RWTH ASR Toolkit for Generic Sequence-to-sequence Speech Recognition
Authors: Wei Zhou; Eugen Beck; Simon Berger; Ralf SchlÃ¼ter; Hermann Ney
Venue: Interspeech 2023 (main)

Abstract: Modern public ASR tools usually provide rich support for training various sequence-to-sequence (S2S) models, but rather simple support for decoding open-vocabulary scenarios only. For closed-vocabulary scenarios, public tools supporting lexical-constrained decoding are usually only for classical ASR, or do not support all S2S models. To eliminate this restriction on research possibilities such as modeling unit choice, we present RASR2 in this work, a research-oriented generic S2S decoder implemented in C++. It offers a strong flexibility/compatibility for various S2S models, language models, label units/topologies and neural network architectures. It provides efficient decoding for both open- and closed-vocabulary scenarios based on a generalized search framework with rich support for different search modes and settings. We evaluate RASR2 with a wide range of experiments on both switchboard and Librispeech corpora. Our source code is public online.
Streaming Speech-to-Confusion Network Speech Recognition
Authors: Denis Filimonov; Prabhat Pandey; Ariya Rastrow; Ankur Gandhe; Andreas Stolcke
Venue: Interspeech 2023 (main)

Abstract: In interactive automatic speech recognition (ASR) systems, low-latency requirements limit the amount of search space that can be explored during decoding, particularly in end-to-end neural ASR. In this paper, we present a novel streaming ASR architecture that outputs a confusion network while maintaining limited latency, as needed for interactive applications. We show that 1-best results of our model are on par with a comparable RNN-T system, while the richer hypothesis set allows second-pass rescoring to achieve 10-20% lower word error rate on the LibriSpeech task. We also show that our model outperforms a strong RNN-T baseline on a far-field voice assistant task.
Accurate and Structured Pruning for Efficient Automatic Speech Recognition
Authors: Huiqiang Jiang; Li Lyna Zhang; Yuang Li; Yu Wu; Shijie Cao; Ting Cao; Yuqing Yang; Jinyu Li; Mao Yang; Lili Qiu
Venue: Interspeech 2023 (main)

Abstract: Automatic Speech Recognition (ASR) has seen remarkable advancements with deep neural networks, such as Transformer and Conformer. However, these models typically have large model sizes and high inference costs, posing a challenge to deploy on resource-limited devices. In this paper, we propose a novel compression strategy that leverages structured pruning and knowledge distillation to reduce the model size and inference cost of the Conformer model while preserving high recognition performance. Our approach utilizes a set of binary masks to indicate whether to retain or prune each Conformer module, and employs L0 regularization to learn the optimal mask values. To further enhance pruning performance, we use a layerwise distillation strategy to transfer knowledge from unpruned to pruned models. Our method outperforms all pruning baselines on the widely used LibriSpeech benchmark, achieving a 50% reduction in model size and a 28% reduction in inference cost with minimal performance loss.
MERLIon CCS Challenge: A English-Mandarin code-switching child-directed speech corpus for language identification and diarization
Authors: Victoria Y. H. Chua; Hexin Liu; Leibny Paola Garcia; Fei Ting Woon; Jinyi Wong; Xiangyu Zhang; Sanjeev Khudanpur; Andy W. H. Khong; Justin Dauwels; Suzy J. Styles
Venue: Interspeech 2023 (main)

Abstract: To enhance the reliability and robustness of language identification(LID) and language diarization(LD) systems for heterogeneous populations and scenarios, there is a need for speech processing models to be trained on datasets that feature diverse language registers and speech patterns. We present the MERLIon CCS challenge, featuring a first-of-its-kind Zoom video call dataset of parent-child shared book reading, of over 30 hours with over 300 recordings, annotated by multilingual transcribers using a high-fidelity linguistic transcription protocol. The audio corpus features spontaneous and in-the-wild English-Mandarin code-switching, child-directed speech in non-standard accents with diverse language-mixing patterns recorded in a variety of home environments. This report describes the corpus, as well as LID and LD results for our baseline and several systems submitted to the MERLIon CCS challenge using the corpus.
Spoken Language Identification System for English-Mandarin Code-Switching Child-Directed Speech
Authors: Shashi Kant Gupta; Sushant Hiray; Prashant Kukde
Venue: Interspeech 2023 (main)

Abstract: This work focuses on improving the Spoken Language Identification (LangId) system for a challenge that focuses on developing robust language identification systems that are reliable for non-standard, accented (Singaporean accent), spontaneous code-switched, and child-directed speech collected via Zoom. We propose a two-stage Encoder-Decoder-based E2E model. The encoder module consists of 1D depth-wise separable convolutions with Squeeze-and-Excitation (SE) layers with a global context. The decoder module uses an attentive temporal pooling mechanism to get fixed length time-independent feature representation. The total number of parameters in the model is around 22.1 M, which is relatively light compared to using some large-scale pre-trained speech models. We achieved an EER of 15.6% in the closed track and 11.1% in the open track (baseline system 22.1%). We also curated additional LangId data from YouTube videos (having Singaporean speakers), which will be released for public use.
Improving wav2vec2-based Spoken Language Identification by Learning Phonological Features
Authors: Mostafa Shahin; Zheng Nan; Vidhyasaharan Sethu; Beena Ahmed
Venue: Interspeech 2023 (main)

Abstract: Spoken language identification (SLI) is a key component in speech-processing tools such as spoken language understanding. In code-switching conversational speech, speakers change languages for short durations posing an additional challenge to language identification techniques. In this work, we investigate the ability of a wav2vec2-based SLI method in identifying the spoken language of English/Mandarin code-switching child-directed conversational speech recorded via Zoom. The proposed system allows the pre-trained wav2vec2-based model to learn language-dependent phonological features by fine-tuning first on detecting manners and places of articulation, then on classifying between English and Mandarin speech segments. The proposed system was tested against parent-child Zoom recordings provided as a part of the MERLIon CCS challenge of language identification. The system achieved the best balanced accuracy of 81.3% and the second-lowest equal error rate of 10.6%.
Language Identification Networks for Multilingual Everyday Recordings
Authors: Kiran Praveen; Balaji Radhakrishnan; Kamini Sabu; Abhishek Pandey; Mahaboob Ali Basha Shaik
Venue: Interspeech 2023 (main)

Abstract: This paper describes the systems SRIB has proposed for task-1 of the inaugural MERLion CCS challenge in the closed domain and open domain. Our system for the closed task is based on an end-to-end conformer architecture trained for the task of automatic speech recognition using RNN-T loss, which is then transfer learned for the task of language classification. We train the ASR model initially to ease the task of learning the right features for the classification task. This system achieves a 13.9% Equal Error Rate (EER) and 81.7% Balanced Accuracy (BAC) on the evaluation set. For the open track, we use an ensemble of Open AI's Whisper model and one of the ASR models used our closed track. This system achieves 9.5% EER and 78.9% BAC on the evaluation set. Compared to the challenge baseline we observe relative improvements for EER of 35.9% in the closed track and 56.2% in the open track. We achieve 1st position on both the closed and the open track leaderboards.
Investigating model performance in language identification: beyond simple error statistics
Authors: Suzy J. Styles; Victoria Y. H. Chua; Fei Ting Woon; Hexin Liu; Leibny Paola Garcia; Sanjeev Khudanpur; Andy W. H. Khong; Justin Dauwels
Venue: Interspeech 2023 (main)

Abstract: Language development experts need tools that can automatically identify languages from fluent, conversational speech and provide reliable estimates of usage rates at the level of an individual recording. However, LID systems are typically evaluated on metrics such as equal error rate and balanced accuracy, applied at the level of an entire speech corpus. These overview metrics do not provide information about model performance at the level of individual speakers, recordings, or units of speech with different linguistic characteristics. Overview statistics may mask systematic errors in model performance for some subsets of the data, and consequently, have worse performance on data derived from some subsets of human speakers, creating a kind of algorithmic bias. Here, we investigate how well a number of LID systems perform on individual recordings and speech units with different linguistic properties in the MERLIon CCS Challenge featuring accented code-switched child-directed speech.
Classification of Vocal Intensity Category from Speech using the Wav2vec2 and Whisper Embeddings
Authors: Manila Kodali; Sudarsana Reddy Kadiri; Paavo Alku
Venue: Interspeech 2023 (main)

Abstract: In speech communication, talkers regulate vocal intensity resulting in speech signals of different intensity categories (e.g., soft, loud). Intensity category carries important information about the speaker's health and emotions. However, many speech databases lack calibration information, and therefore sound pressure level cannot be measured from the recorded data. Machine learning, however, can be used in intensity category classification even though calibration information is not available. This study investigates pre-trained model embeddings (Wav2vec2 and Whisper) in classification of vocal intensity category (soft, normal, loud, and very loud) from speech signals expressed using arbitrary amplitude scales. We use a new database consisting of two speaking tasks (sentence and paragraph). Support vector machine is used as a classifier. Our results show that the pre-trained model embeddings outperformed three baseline features, providing improvements of up to 7%(absolute) in accuracy.
The effect of clinical intervention on the speech of individuals with PTSD: features and recognition performances
Authors: Alexander Kathan; Andreas Triantafyllopoulos; Shahin Amiriparian; Sabrina Milkus; Alexander Gebhard; Jonas Hohmann; Pauline Muderlak; JÃ¼rgen Schottdorf; BjÃ¶rn W. Schuller; Richard Musil
Venue: Interspeech 2023 (main)

Abstract: Post-traumatic stress disorder (PTSD) is an anxiety disorder that can occur as a response to traumatic experiences, such as catastrophic events, and can have a detrimental influence on mental wellbeing. Furthermore, PTSD is present in 5-10 % of the population, making it a prevalent disorder in our time, thus necessitating a timely diagnosis and proper treatment. In this paper, we present results for PTSD detection based on speech recordings on a newly collected dataset consisting of 15 participants, including speakers with PTSD and a control group. Moreover, the dataset includes speech data immediately before and after a clinical intervention (i.e., acupuncture-supported psychotherapy), allowing us to examine the effect of a single treatment session. In our experiments, we achieve a best area under the curve (AUC) of 82 % using solely pre-treatment data. Finally, we analyse prominent acoustic patterns of individuals with PTSD compared to the control group.
Analysis and automatic prediction of exertion from speech: Contrasting objective and subjective measures collected while running
Authors: Andreas Triantafyllopoulos; Alexander Gebhard; Alexander Kathan; Maurice Gerczuk; Shahin Amiriparian; BjÃ¶rn W. Schuller
Venue: Interspeech 2023 (main)

Abstract: Monitoring runner exertion in real-time can provide unique insights that help improve training and reduce injuries. Most existing methods use heart rate (HR) as a physiological proxy of it, but this does not always correspond to self-perceived exertion. This is an additional factor in determining overall strain and is typically evaluated with the Borg rating of perceived exertion (RPE) scale. In recent years, speech has been one of the many modalities used to monitor exertion; however, mostly used to predict physiological measures using speech collected after a physical task. In this work, we contrast the manifestation of subjective vs objective exertion on speech signals obtained while running in real-life environments. We identify and interpret a set of prosodic and spectral features related to both markers, and proceed to train deep learning models that directly predict RPE and HR from speech, obtaining an average Pearson correlation of .341 and .418, respectively.
The Androids Corpus: A New Publicly Available Benchmark for Speech Based Depression Detection
Authors: Fuxiang Tao; Anna Esposito; Alessandro Vinciarelli
Venue: Interspeech 2023 (main)

Abstract: This paper presents the Androids Corpus, a new benchmark for speech-based automatic depression detection. The corpus is a collection of 228 recordings uttered by 118 native Italian speakers, including 64 who were diagnosed with depression by professional psychiatrists. Out of the 228 recordings, 112 contain read speech (all speakers read the same text) and 116 contain spontaneous speech (all speakers answer the same questions posed by an interviewer). For 110 speakers, including 58 diagnosed with depression, the corpus includes both read and spontaneous speech samples. Overall, the total duration of the material is 1 hour, 33 minutes and 49 seconds for read speech and 7 hours, 24 minutes and 22 seconds for spontaneous speech. Besides the data, the corpus includes experimental protocols that can be replicated, thus ensuring reproducibility of the experiments performed over it and comparison of the results.
Comparing Hand-Crafted Features to Spectrograms for Autism Severity Estimation
Authors: Marina Eni; Ilan Dinstein; Yaniv Zigel
Venue: Interspeech 2023 (main)

Abstract: In this work, we compared two different input approaches to estimate autism severity using speech signals. We analyzed 127 audio recordings of young children obtained during the Autism Diagnostic Observation Schedule 2nd edition (ADOS-2) administration. Two different sets of features were extracted from each recording: 1) hand-crafted features, which included acoustic and prosodic features, and 2) log-mel spectrograms, which give the time-frequency representation. We examined two different Convolutional Neural Network (CNN) architectures for each of the two inputs and compared the autism severity estimation performance. We showed that the hand-crafted features yielded lower prediction error (normalized RMSE) in most examined configurations than the log-mel spectrograms. Moreover, fusing the estimated autism severity scores of the two feature extraction methods yielded the best results, where both architectures exhibited similar performance (Pearson R=0.66, normalized RMSE=0.24).
Acoustic characteristics of depression in older adults' speech: the role of covariates
Authors: Carmen Mijnders; Esther Janse; Paul Naarding; Khiet P. Truong
Venue: Interspeech 2023 (main)

Abstract: Depression in older adults is often associated with various physical conditions and is hence different from depression at a younger age. Ageing may come with cognitive decline, medication use, and frailty, which are known to be predictors of late-life depression. One common symptom of depression is psychomotor retardation, that may also affect speech production. Most speech studies on depression so far have focused on younger or middle-aged adults. In this study, we used speech data from a large longitudinal Dutch study on late-life depression and its comorbid symptoms to compare speech acoustics in persons with depression (PWD) and controls. We investigated whether groups differed by taking several covariates into account (e.g., frailty, slowness, and medication use). Group differences were found in within-vowel F2 range, speech rate and mean pause duration. These data indicate that speech acoustics can be used to differentiate PWDs and controls, even with low-quality speech data.
Dual Transformer Decoder based Features Fusion Network for Automated Audio Captioning
Authors: Jianyuan Sun; Xubo Liu; Xinhao Mei; Volkan KÄ±lÄ±Ã§; Mark D. Plumbley; Wenwu Wang
Venue: Interspeech 2023 (main)

Abstract: Automated audio captioning (AAC) which generates textual descriptions of audio content. Existing AAC models achieve good results but only use the high-dimensional representation of the encoder. There is always insufficient information learning of high-dimensional methods owing to high-dimensional representations having a large amount of information. In this paper, a new encoder-decoder model called the Low- and High-Dimensional Feature Fusion (LHDFF) is proposed. LHDFF uses a new PANNs encoder called Residual PANNs (RPANNs) to fuse low- and high-dimensional features. Low-dimensional features contain limited information about specific audio scenes. The fusion of low- and high-dimensional features can improve model performance by repeatedly emphasizing specific audio scene information. To fully exploit the fused features, LHDFF uses a dual transformer decoder structure to generate captions in parallel. Experimental results show that LHDFF outperforms existing audio captioning models.