# Using Whisper to Investigate Learner Pronunciations of English: comparing LLM transcriptions with human perception of VOT

Tori Fullerton, Nicolas Ballier

<figure><img src="image_1.png" /></figure>

## To cite this version:

Tori Fullerton, Nicolas Ballier. Using Whisper to Investigate Learner Pronunciations of English: comparing LLM transcriptions with human perception of VOT. 21e colloque d'anglais oral de Villetaneuse, ALOES; Université Sorbonne Paris Nord (Pléiade – UR 7338), Mar 2024, Villetaneuse (Université Paris 13), France. hal-04911801

HAL Id: hal-04911801

https://hal.science/hal-04911801v1

Submitted on 25 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Using Whisper to Investigate Learner Pronunciations of English: comparing LLM transcriptions with human perception of VOT

Tori Fullerton & Nicolas Ballier (Université Paris Cité / CLILLAC-ARP)

This talk will show that Whisper large Language Model (LLM) is a tool for providing automatic pronunciation feedback and phonetic diagnoses to L2 learners using Automatic Speech Recognition (ASR) transcriptions of audio files and that the tiny model is an accurate representation of human interpretation as evidenced by the transcriptions of bilabial plosives according to VOT variability.

Whisper is an audio Pretrained Large Language Model (PLLM) that can be used for both transcription and translation tasks (Radford et al., 2023). There are seven main models within Whisper that provide varying transcriptions for the same audio input. The difference between these models is the number of parameters as well as how much data each model has been trained on. These models also include multilingual models that perform a language detection task before the transcription task (tiny, small, base, medium, large, large-v1, and large-v2) as well as native .en models (base.en, medium.en, small.en, and tiny.en) that assume the input is in English.

The main goal of this research is to investigate if the ASR output of Whisper is consistent with the human judgements produced by Native English speakers and L2 learners. More specifically, we wish to determine if the tiny models can be trusted to produce transcriptions consistent with human interpretations.

Previous research has shown that phonological features can be used in ASR systems for pronunciation training for L2 speakers (Arora, Lahiri, & Reetz, 2018). By investigating the accuracy of the outputs of these systems, we can determine which areas need to be improved in order to incorporate these models into training materials for L2 learners (Choe et al., 2023). Past research on Whisper specifically has also shown that Whisper's tiny model produces output that captures errors in phonetic features and that this model can be used as a simulation to explore misunderstandings involving native speakers (Ballier, Méli, Amand, & Yunès, 2023).

This paper will analyse whether the tiny and tiny.en Whisper model outputs are plausible ASR interpretations of speech signals compared to human interpretations. We chose to investigate VOT in word initial plosives as the current literature has defined this as an area of importance in phoneme comprehension that ASR models may interpret differently than human listeners. The word < pigtail > was chosen as a follow-up to a previous experiment where < pigail > was transcribed as <big tail> by one of the Whisper models for some of the 38 recordings of French learners of English (Ballier et al., 2023). In this experiment, we reused the original recordings as a baseline and synthetically modified the VOTs using a Praat script to modify its duration, varying from -10ms to +10 using 1ms intervals. We ran the modified files through each Whisper model and compared the Whisper transcriptions.

**Preliminary Results** Figure 1 in the appendix shows how Whisper reacts to VOT variability according to its different models. The horizontal axis corresponds to VOT duration (in milliseconds) and the vertical axis corresponds to the percentage of stimuli transcribed as <p> or <P>. The tiny.en model response to VOT is consistent with previous studies: higher values of VOT are transcribed as voiceless plosives. We will then discuss the different responses to VOT and compare them for our findings in our perception tests. Based on the results of previous SLA research involving exposure to phonetic variability, our hypothesis is that Whisper and Native English speakers will perform similarly on the perception tests, whereas L2 learners of English will produce different results as they have been exposed to less phonetic variability (Colantoni & Steele, 2023). Based on existing SLA research involving VOT in bilinguals, we can hypothesise that French-English bilinguals have shorter VOTs than native English speakers and therefore may incorrectly attribute the VOT of a /b/ realisation to the /p/ realisation due to the L1 influence of French VOT and the lack of phonetic variability within their current exposure to English (Kehoe & Kannathasan, 2021). Using this same reasoning, we can hypothesise that Native English speaker and the Whisper LLM tiny model that has been trained on a variety of different English data should both perform at similar levels and produce similar results.

## References

Arora, V., Lahiri, A., & Reetz, H. (2018). Phonological feature-based speech recognition system for pronunciation training in non-native language learning. , 143(1), 98–108. Retrieved 2023-11-20, from https://pubs.aip.org/jasa/article/143/1/98/616116/Phonological-feature-based-speech-recognition doi: 10.1121/1.5017834
Ballier, N., Méli, A., Amand, M., & Yunès, J.-B. (2023). Using Whisper LLM for Automatic Phonetic Diagnosis of L2 Speech: A Case Study with French Learners of English. In *6th International Conference on Natural Language and Speech Processing (ICNLSP 2023)* (Vol. 6, p. 282-292).

Choe, J., Chen, Y., Chan, M. P. Y., Li, A., Gao, X., & Holliday, N. (2023). Language-specific effects on automatic speech recognition errors for world englishes. In N. Calzolari et al. (Eds.), *Proceedings of the 29th international conference on computational linguistics* (pp. 7177–7186). International Committee on Computational Linguistics. Retrieved 2023-11-20, from https://aclanthology.org/2022.coling-1.628

Colantoni, L., & Steele, J. (2023). The mixed effects of phonetic input variability on relative ease of l2 learning: Evidence from english learners' production of french and spanish stop-rhotic clusters. , 3(2), 12. Retrieved 2023-12-13, from https://www.mdpi.com/2226-471X/3/2/12 (Number: 2 Publisher: Multidisciplinary Digital Publishing Institute) doi: 10.3390/languages3020012

Kehoe, M., & Kannathasan, K. (2021). Development of voice onset time in monolingual and bilingual french-speaking children. , 252, 102937. Retrieved 2023-12-13, from https://www.sciencedirect.com/science/article/pii/S0024384120301455 doi: 10.1016/j.lingua.2020.102937

Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In *International conference on machine learning* (pp. 28492–28518). PMLR. Retrieved 2023-10-24, from https://proceedings.mlr.press/v202/radford23a.html

# Appendix

<figure><img src="image_3.png" /><figcaption>Figure 1: Whisper transcription response (percentage of < p >) to VOT Variability (duration in ms)</figcaption></figure>