Sensitivity of Syllable-Based ASR Predictions to Token Frequency and Lexical Stress
Authors: Alessandro Vietti, Domenico De Cristofaro, Sara Picciau (Libera Università di Bolzano, 2024)
Topic: Multimodal

Abstract
Automatic Speech Recognition systems (ASR) based on neural networks achieve great results, but it remains unclear which are the linguistic features and representations that the models leverage to perform the recognition. In our study, we used phonological syllables as tokens to fine-tune an end-to-end ASR model due to their relevance as linguistic units. Furthermore, this strategy allowed us to keep track of different types of linguistic features characterizing the tokens. The analysis of the transcriptions generated by the model reveals that factors such as token frequency and lexical stress have a variable impact on the prediction strategies adopted by the ASR system.

Introduction
The syllable is crucial in the process of spoken word recognition. It serves as an integral component within the prosodic system because it encompasses both traditional segmental and suprasegmental levels, facilitating the extraction of lexical and syntactic structures from acoustic information AUTHOR. Specifically, the syllable serves as the linguistic unit where crucial information for speech segmentation, rhythmic patterns, and lexical access is encoded AUTHOR. In the field of Automatic Speech Recognition (ASR), graphemic segment has traditionally been the primary unit of processing. However, recent studies endorse the use of syllables or phonetic units of similar duration as an alternative strategy AUTHOR. In latest ASR research employing Transformer-based neural models, the role of syllables is investigated both as tokens for word recognition and as components influencing internal speech representations within neural networks AUTHOR. In our study, a neural ASR model was trained to process and recognize phonological syllables, integrating them into word structures. Our goal is to conduct a linguistic analysis on the output of syllabic processing by the speech recognition system. Through fine-tuning a large acoustic model, the study mapped speech signals onto phonological transcriptions segmented into syllables and words. The primary objective of our linguistic analysis is to test the effect of syllable token frequency and lexical stress on the accuracy of output neural representation. To understand how the ASR processes syllables and words differently, we developed a fine-grained linguistic annotation system. This approach was essential to move beyond the limitations of purely numerical metrics like Word-Error-Rate or, in our context, Token-Error-Rate. By employing this system, we could accurately categorize prediction types and link them with specific linguistic aspects of speech. We utilized Multiple Correspondence Analysis and Multinomial Logistic Regression to explore and uncover patterns that relate the neural network's output behavior to the linguistic factors.

Conclusion
This study provides insights into the role of syllables in ASR performance, particularly when integrating phonological information into the recognition process. By fine-tuning a neural ASR model to process and recognize phonological syllables, we were able to conduct a detailed linguistic analysis of its output. Our findings indicate that syllable frequency and lexical stress significantly impact ASR accuracy. Specifically, stressed syllables are more accurately recognized than unstressed ones, especially as frequency increases. Contrary to our expectation, among the low-frequency syllables, stressed tokens are more prone to substitution, whereas mid-frequency unstressed ones are more susceptible to deletion. This demonstrates the neural model's sensitivity to both distributional information in the dataset and phonological information and highlights the model's ability to detect varying syllabic prominence at the lexical level within the signal. As future work, we plan to include other linguistic factors as independent variables to refine our analysis. An interesting approach is to evaluate the impact of unstressed syllables and specific parts of speech by conducting an analysis exclusively on content words. Furthermore, we aim to investigate in detail syllable substitution in relation to token frequency and phonetic similarity to compare the weight of each factor whenever this strategy is adopted to deal with low-frequency tokens. In conclusion, our study showed the influence of token frequency and prominence in ASR predictions while demonstrating that complex computational tools, like modern neural networks, can be effectively utilized by linguists to simulate and test linguistically relevant hypotheses. \clearpage \appendix % Use to avoid numbering