# Audio LLM subtokens as encapsulated "knowledge": the case of Persian subtoken graphemic representations in Whisper

Behnoosh Namdarzadeh, Nicolas Ballier

► To cite this version:

Behnoosh Namdarzadeh, Nicolas Ballier. Audio LLM subtokens as encapsulated "knowledge": the case of Persian subtoken graphemic representations in Whisper. Grapholinguistics in the 21st century. From graphemes to knowledge, Yannis Haralambous, IMT Atlantique & CNRS Lab-STICC, Brest, France; Sveva Elti di Rodeano, Dipartimento di Studi umanistici, Università Ca' Foscari, Venice, Italy, Oct 2024, Venice, Italy. [https://hal.science/hal-04927138](https://hal.science/hal-04927138) [hal-04927138](https://hal.science/hal-04927138)

HAL Id: hal-04927138

https://hal.science/hal-04927138v1

Submitted on 3 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Audio LLM subtokens as encapsulated "knowledge": the case of Persian subtoken graphemic representations in Whisper

Behnoosh Namdarzadeh · Nicolas Ballier

Received: date / Accepted: date

**Abstract** Whisper is a widely used, open-access Large Language Model (LLM) trained using a multilingual paradigm. As such, it represents an important opportunity for researchers to study how multilingual LLMs function across languages. In this paper, we probe the grapholinguistic representations of Whisper for an under-resourced language with a Perso-Arabic script: Persian [11, 16, 17, 19, 28, 33, 36]. Persian, with its abjad writing system, has a consonant-heavy inventory of letters and does not indicate vowels, making it a good candidate to test an audio LLM such as Whisper [22]. Large Language Models do not deal with tokens, morphemes or graphemes as linguists know them but subtokens, fragments of words generated by an algorithm. This subtokenisation process creates fracto-morphemes, clusters of letters, that continue to represent language (this is literally “natural language processing”) but the relationship of the letters to morphology, semantics and phonology is completely changed. We want to investigate this technical paradigm shift for writing systems such as Persian, by analysing the sounds to letters mapping as mediated by acoustic representations and LLM subtokens. Using a C++ implementation [12] of Whisper [30], we have extracted the subtoken dictionary of this model. For the transcription task of spoken Persian, we also extracted the subtokens and their probabilities using a customised C++ implementation of Whisper.

In this work, we probe the phonotactic and graphotactic status of the subtokens representing Persian in order to understand how knowledge (representation) is distributed in the Whisper subtokens used to transcribe Per-

<footnote>F. Author
first address
Tel.: +123-45-678910
Fax: +123-45-678910
E-mail: fauthor@example.com</footnote>

<footnote>S. Author
second address</footnote>
sian. Understanding the subtokenisation process of large language models like
Whisper is crucial for advancing both NLP techniques and linguistic theories.
Investigating how these models represent languages such as Persian sheds light
on the intricate relationship between the acoustic signal, subtoken structures,
and linguistic properties. The corresponding research questions are universal
and others are more language-specific. We address them in this order.

- RQ1 Do LLM subtokens follow Zipf's law and other related large number of rare events (LNRE) models?

- RQ2 Are there “morpho-phonological” rules that potentially map sub tokens to phonological representations?

- RQ3 Can an architectural bias be observed in the knowledge encoded for Persian?

- RQ4 How are special phenomena in Persian script such as the connector ezafe encoded and processed with Whisper?

We will show that the Whisper models distributions of subtokens do not follow this Zipf's law. Our findings suggest that only a limited amount of subtokens is used to represent Persian as illustrated in Figure . Our results tend to show that the decoder is used to post-process the data, as evidenced by the fact that the internal representations of the Whisper models are only the isolated forms of graphemes and limited digraphs.

We demonstrate that Whisper models tend to select the formal variant in their transcription. For example, in a read speech of 100 sentences. Whisper produces the formal variant of the object marker *rā* in the transcriptions three times more than its informal variant *ro*, even though the speaker only produces the informal variant in the read speech. Last, we show how a *silent hé* is added across Whisper models for the *ezafe* connector and how that can be attributed to its unstressed ’-e’ sound.

The core of our work is based on the study of Whisper's ability to auto-
matically discover associations between speech and text segments, based on
training data and without any prior linguistic knowledge or annotation. More
specifically, we propose to study not only the predictions made at each posi-
tion, but also the alternatives considered by Whisper. To do this, we exploit
a modification to the whisper.cpp implementation, in order to retrieve the
*n*-best list (the *n* tokens with the highest probabilities at each stage of the
prediction) and analyse their phonetic properties.

**Keywords** audio Multilingual Large Language Model · Subtokenisation · Dictionary of subtokens · Graphemic Status

## 2 Introduction

Whisper is an automatic speech recognition (ASR) system developed using large-scale weak supervision. It was trained to predict transcripts of audio from 680,000 hours of labeled audio data, with 117,000 hours covering 96 languages other than English. The audio was broken into 30-second segments and
paired with transcripts from different sources. Audio was resampled to 16,000 Hz, and an 80-channel log-magnitude Mel spectrogram was computed. Whisper was trained for several tasks such as language identification, transcription, translation, and timestamps prediction. Whisper's training included data for 96 languages besides English, including Persian, but with only 392 hours for the translation task and 24 hours for the transcription task. The performance for transcription of Persian is reported with a word error rate of 44.8 for the large model (compared to 6.3 for English), and the impact of performance of the amount of training data is consistent with the expectations [3]. The training data transcripts were tokenised at subword level using a byte pair encoding algorithm [34] that was applied to the 96 languages of the training data, resulting in a multilingual vocabulary of units we call "subtokens" in the rest of the paper.

Whisper is a neural network that learns from a training corpus made up of audio recordings (690,000 hours for the largest model), combined with their transcription or translation. The aim is to match frames (windows of the sound signal) with (sub)tokens, i.e. arbitrary sequences of characters. It is therefore a supervised learning algorithm. The (sub)tokens are generated automatically from the training data using tokenisation algorithms. These algorithms divide the text into units based on the frequency with which sequences appear in the corpus. The main algorithms used are Byte Pair Encoding (BPE) [34], Word-Piece, initially developed for BERT [8], and SentencePiece [20]. Depending on their frequency of appearance, (sub)tokens can correspond to whole words, sequences of 2 or 3 characters, or even a single character.

Whisper is an auto-regressive algorithm: given an audio input and the vector representation it constructs, it predicts the tokens one after the other to generate the final output. More specifically, based on the input audio and the prefix already generated (the previously predicted tokens), Whisper's decoder determines a probability distribution for all the tokens in its vocabulary. By choosing the token with the highest probability at each stage (or by sampling according to this distribution), it gradually builds the output sequence.

We wish to explore the graphophonemic status of these subtokens as applied to the transcription task (ASR) of Persian. These LLM subtokens are a paradoxical object that correspond to morphemes (splinters) but that are not morphological entities. We want to elaborate the inventory of subtokens used by the multilingual Whisper for Persian transcription by investigating the mapping between the acoustic realisations of the Persian data used as input and the Whisper prediction. In this sense, we address the presumed encapsulated "knowledge" in the subtoken dictionary of LLM. An important aspect to bear in mind is the potential misleading metaphor that if LLMs have representations they "understand" speech or "recognize" words. Even though the main type of encoded representation is a dictionary of subtokens that serves as a repository for the corresponding numbers of vectors of numerical figures, that serves as a form of representation, if any, alternatively for every 30 second speech window. The different attention weights corresponding to the internal representations can also be extracted and potentially investigated (see [25] as
an example). It still remains to appreciate as linguists what become of the
different languages that are processed using these multilingual models. In
this respect, it is likely that the subtokenization phase is not innocuous [29].
We partially discuss the phonology of the Whisper dictionary of subtokens <fnref n="1" />,
the subtokens used to represent Persian writing in the LLM. Persian offers a
unique perspective for this type of investigation for three reasons. First, the
scarcity of the training data entailed a scarcity of subtokens in the Whisper
dictionary used to transcribe Persian. Second, we have a privileged window
on the phonographematics of Whisper since the acoustic space is transcribed
by a limited amount of subtokens, mostly Persian isolated characters, whereas
Persian has graphotactic constraints for final, medial or initial letters. Third,
Persian shares many characters with Arabic, which has more training data.

As an abjad, rather than an alphabet, the Persian written language does
not note vowels; this frequently results in an error in the transcription of vow-
els. 28 letters of the Arabic script have been supplemented with 4 additional
consonants in order to preserve consonantal distinctions that do not exist in
Arabic. However, Arabic is also encoded in the Whisper dictionary of subto-
kens. Previous studies address the challenge of adapting multilingual language
models to specific languages for different NLP tasks [9].

The rest of the paper is structured as follows: Section 3 presents the tenets
of the Persian writing system and the variety of Persian we describe. Section
4 presents the phonological features of Persian and subtokenisation process
in audio LLMs. Section 5 details our methodology and the datasets we used
in our experiments. Section 6 presents our experiments with the probing of
the subtoken dictionary and discusses the distribution of graphemes encapsu-
lated in the Whisper subtokens used to transcribe Whisper. We analyse our
results in Section 7 and Section 8 discusses them. Finally, we conclude and
recommend areas for further research in Section 9.

## 3 Background on Persian Writing system

This section provides an overview of Persian graphematics and examines the
graphotactic constraints that may influence the performance of audio-based
LLMs. The history of the Persian language is divided into three periods:
Old Persian (c. 525 BC - 300 BC), Middle Persian (c. 300 BC - 800 AD) and
Modern Persian (800 AD to the present). After the advent of Islam in 642 AD,
Persians adapted the Arabic alphabet to develop the modern Persian alphabet.
The Arabic alphabet has 28 letters, but Iranians added four additional letters
to create the current 32 Persian letters. Modern Persian emerged in the 9th
century after this adoption of the alphabet and the borrowing of many Arabic
words.

The Persian script, technically called “abjad”, consists of 32 letters, 28 of which are shared with Arabic, while the remaining four are unique to Persian.

<footnote n="1">¹ We do not discuss the special subtokens, see [3].</footnote>
Persian adopts a non-Latin script written from right to left [4]. It comprises six vowels: three short vowels, marked with diacritical marks, and three long vowels, marked by certain consonants [4]. Therefore, Persian writing can be called a consonantal alphabet. Diacritics for short vowels are mainly used for elementary readers, disappearing from textbooks after the first grade [31], and are generally absent from standard written texts. Long vowels, in contrast, are consistently represented in writing.

The relationship between graphemes and phonemes in Persian is highly consistent. Each grapheme corresponds to a single phoneme [2]. Therefore, Persian is classified as having a shallow orthography when short vowels are represented or when only long vowels and consonants are present [4]. However, the omission of short vowels results in a deep and opaque orthography where the relationships between graphemes and phonemes are inconsistent.

This study aims to analyse the mapping of sounds to letters within a criti-
cal pipeline of audio large language models (LLMs)—subtokenisation—for an
under-resourced language like Persian, which is represented by only 14 hours
of training data in Whisper. By examining the phonotactic and graphotac-
tic properties of the subtokens representing Persian, we seek to uncover the
extent of linguistic knowledge encapsulated in the model for transcribing this
under-resourced language.

1 Graphotactic constraints for Persian

This subsection explores the graphotactic constraints of Persian, which may
affect the performance of LLMs such as Whisper. Persian is a language written
with Arabic characters and several glyphs supplementing them. [23]. Ninety
different forms, referred to as “allographs”, are needed to represent the 28 let-
ters of the Persian alphabet. Persian characters can have different graphematic
forms depending on the place of occurrence. The isolated form of characters
(28 letters), depending on their position within a word. The isolated form
of each letter (28 letters) can vary depending on its contextual placement,
taking on different forms (allographs) in the initial, medial, or final positions
[23]. The training data may not include all the required cases to ensure an
ideal distribution of these contextual character forms, specifically Segment-
Initial, Segment-Medial, Segment-Final, and Isolated forms. The dictionary
of subtokens in Whisper will be comprehensively analysed in Section 2.

It is likely that consonants which have a single form used to represent the
letter are more likely to be well transcribed. Graphophonemics is not only a
matter of letter correspondence, it is also a matter of representation within
the word.

It should be mentioned that the dialect we are examining for the sets of
our experiments is a variant of Farsi, called Tehrani, the standard colloquial
dialect spoken in Iran [18]. We use the term “Persian” to refer to this variant
in this paper.
In Persian, a single phoneme can be represented by multiple letters, re-
sulting in graphematic inconsistencies. This variation requires practice and
familiarity to determine which specific grapheme is used in a given word. For
instance, the phoneme /t/ is represented by two letters: <ت> and <ط>; the
phoneme /h/ by two letters: <ه> and <ح>; the phoneme /s/ by three letters:
<س>, <ص>, and <ث>; and finally, the phoneme /z/ by four letters: <ز>,
<ژ>, <ض>, and <ظ>. This is particularly problematic when transliterating
foreign words and selecting the appropriate grapheme for a given phoneme.
For example, there is controversy regarding the graphematic representation
of the French-origin word *billet* and the two variants of /t/ phoneme. This
has resulted in two different graphematic forms: <بلیت> and <بلیط>. Table 3
illustrates the various letters that correspond to the same phoneme. However,
in Arabic, each of these letters corresponds to a distinct phoneme.

4 Phonological System of Persian and the speech-to-subtoken
mapping

We first recall the phonological system of Persian before explaining how sounds
are mapped to the inventory of subtokens. In this paper, we follow the stan-
dard phonological convention that indicates graphemes (letters) with angled
brackets( < > ), realisations in square brackets and phonemes (or targets) with
slanted bars (//).

1 vowels and consonants

The standard Persian of Iran has six vowel sounds as shown in 1. The vowels
□/a, e, o/□ are short vowels and □/, i u/□ are long vowels. Since Persian writ-
ing system is abjad (consonantal alphabet), short vowels do not have specific
letters and are not written in written Persian, and only long vowels appear
in texts. Long vowels are always marked with consonant letters. The short
vowels, as mentioned earlier, do not mark in standard written Persian, but
for learners of Persian, they can be precisely marked with diacritics. Inserting
diacritics is not allowed in the early years of primary school.

The three long vowels, /د, i, u/, are frequently found word-finally. Less
commonly occurring word-finally are the short vowels, /a, e, o/. While /e/ is
present in some high-frequency words, word-final /o/ is rare (still appears in
some high frequency words.)

Any vowel may initiate a word, though some occur more frequently than
others. Word-initial /a/ is highly prevalent, while word-initial /i/ is less com-
mon. Word-initial /u/ is rare. In summary, lax vowels such as /e/ and /a/ are
the most common in word-initial positions, while word-initial /ow/ is seldom
encountered.

Due to the complexities of inserting or omitting vowels in written Persian,
significant challenges arise in accurately transcribing vowels. There are some
<table><caption>Table 1: Tehrani Persian Vowel Chart</caption><thead><tr><td>Height</td><td>Front</td><td>Back</td></tr></thead><tbody><tr><td>Close</td><td>i</td><td>u</td></tr><tr><td>Mid</td><td>e</td><td>o</td></tr><tr><td>Open</td><td>a</td><td>ɔ</td></tr></tbody></table>

factors such as the neighbouring graphemes and the type of syllable containing the vowel which affect the choice of grapheme (allographs) for the vowel [23]. Persian script often omits short vowels, as seen in /kətəb/ (<u>کتاب</u>) where the short vowel /e/ is not represented, reflecting the inherent complexity of vowel representation in the language. Additionally, vowel representation is influenced by neighbouring graphemes, as in /χphər/ (<u>خواهر</u>) where the character <u>خ</u> does not denote a /v/ or /u/ sound but forms part of a digraph for /χ/ followed by /n/.

This subsection explores the constraints that govern the distribution and arrangement of consonants in written Persian. Persian includes 23 consonant sounds, each represented by a dedicated letter. The table of International Phonetic Alphabet (IPA) symbols for these consonants is presented in 2. Notably, the shape of a letter in Persian script varies depending on its position within a word. A single letter may assume different forms when appearing in the initial, medial, or final position of a word. This positional variability is a defining characteristic of the Persian writing system.

Word-initial consonant clusters are absent in Persian, and Persian speakers tend to insert a vowel between consonants when pronouncing borrowed words with initial clusters from the source language, thus avoiding word-initial consonant clusters. Alternatively, a vowel may be inserted before the cluster, leading to a VC+C sequence [21]. Moreover, word-initial clusters can be broken up, resulting in a CV+C sequence. Various types of two-consonant clusters are permissible in syllable-final positions (e.g., nasal+plosive: /omq/ meaning 'depth').

<table><caption>Table 2: Persian Consonants Organized by Manner and Place of Articulation</caption><thead><tr><td></td><td>Labial</td><td>Alv.</td><td>Post-alv.</td><td>Velar</td><td>Uvular</td><td>Glottal</td></tr></thead><tbody><tr><td>Nasal</td><td>m</td><td>n</td><td></td><td></td><td></td><td></td></tr><tr><td>Stop</td><td>p, b</td><td>t, d</td><td>tʃ, dʒ</td><td>k, g</td><td>(q)</td><td>?</td></tr><tr><td>Fricative</td><td>f, v</td><td>s, z</td><td>ʃ, z</td><td>x, ɣ</td><td></td><td>h</td></tr><tr><td>Tap</td><td></td><td>ʃ</td><td></td><td></td><td></td><td></td></tr><tr><td>Approx</td><td></td><td>l</td><td>j</td><td></td><td></td><td></td></tr></tbody></table>
## 2 Syllable structure

In terms of syllable structure, Persian features three forms: CV (consonant-vowel), CVC (consonant-vowel-consonant), and CVCC (consonant-vowel-consonant-consonant). Notably, consonant clusters are only found at the end of words [35], as exemplified by the word mard ('man'). Persian thus exhibits a less complex syllable structure than languages such as Swedish, which allows consonant clusters of up to three consecutive consonants.

## 3 The semiotics of subtokenisation

In this subsection we explain how the Persian representation between graphemes and meaning is mediated with the subtokenisation phase [34]. Subtokens are a hybrid representation that bear testimony to the frequency of tokens in the training data of the subtokenisation algorithm (and of Whisper's training).

Subword units have emerged as a crucial component in language modeling. They play a foundational role in the tokenisation processes of advanced transformer-based language models, such as those utilising Byte Pair Encoding (BPE) [34]. This subword-based tokenization method underpins the architecture of many LLMs [24]. The adoption of subword tokenization strategies, like BPE, has significantly contributed to the success of pre-trained transformer models in natural language processing, enabling them to effectively handle a diverse array of languages and linguistic structures [5]. Subtokenisation can be challenging for Persian due to its graphematic and phonographematic constraints, as outlined in 3. In this context, the implementation that predicts on a sub-token basis may differ significantly from a representation based on entire sentences. Typically, isolated tokens are expected to correspond systematically to individual letters. Figure 1 illustrates the subtokenisation in three languages—French, English, and Persian—using the BERT tokeniser <fnref n="2" />.

<figure><img src="image_2.png" /><figcaption>Figure 1: Subtokenisation in French, English, and Persian sentences Using Bert</figcaption></figure>

To validate this, it is necessary to empirically examine the issue by testing the Persian FLEURS dataset [7] and conducting a distributional analysis. This

<footnote n="2">² https://huggingface.co/docs/transformers/model doc/ berttransformers.BertTokenizer</footnote>
analysis would help assess whether the subtokenisation aligns with the priors of the Whisper representation.

hallucinations One way to analyse the evolution of hallucinations across the size of the models is to observe that the hallucinations in the outputs belong to different language groups, and, in that respect, they in a way follow the pecking order of the hierarchy of languages used in the training data. The Chinese characters are overwhelmingly present in the transcriptions of the tiny models and that is consistent with the role of the size of the Chinese training data. One must imagine the number of times these types of subtokens were seen during the training phase. Incidentally, we may report the expected number of tokens observed for one hour of speech because the training data for whisper is expressed in duration in hours of speech. We may approximate the expected number of subtokens that were seen during the training phase. In the largest models, hallucination is only observed within a person and, for example, a verb might be repeated several times at the end of a sentence. So that is still a hallucination but within the domain of a single language.

# 5 Material and Methods

This section outlines the materials and methods used in the study.

## 1 Materials

Here are the three datasets used in our experiments:

- YouTube Audio File: A 69-minute audio recording extracted from YouTube <fnref n="3" /> was selected for analysis. The recording features three male speakers engaging in a discussion on social issues. Due to the spontaneous and unstructured nature of the conversation, manual transcription would be both challenging and time-consuming. By utilising OpenAI's Whisper for transcription, we gain two significant advantages: the generation of an accurate transcription and its accompanying translation. This output provides a valuable foundation for developing a bilingual corpus.

- Common Voice Mozilla: Additional Persian data for detailed experiments was sourced from the Common Voice Mozilla project <fnref n="4" />, a diverse dataset of speech recordings designed for speech recognition tasks.

- FLEURS Dataset: The FLEURS dataset <fnref n="5" />, which focuses on multilingual speech recognition and understanding, was also used to supplement our analysis and broaden the scope of our Persian corpus experiments.

<footnote n="3">³ https://www.youtube.com/watch?v=YsMDE0EH42s</footnote>

<footnote n="4">⁴ https://commonvoice.mozilla.org/fa</footnote>

<footnote n="5">⁵ https://huggingface.co/datasets/google/fleurs</footnote>
## 2 Transcription Method with OpenAI's Whisper

Our approach focused on leveraging a customized C++ implementation of Whisper <fnref n="6" />, building upon the foundational work by Radford [30]. Whisper offers six model variants, ranging from tiny to large-v1, which provide valuable insights into the mapping of spoken Persian to written Persian. The Whisper.cpp implementation by Gerganov <fnref n="7" /> facilitates efficient processing of Whisper's parameters and includes tools for visualizing subtokens along with their associated probabilities. This functionality enables access to probability scores for each subtoken generated by Whisper, forming the basis for developing a calibration model to evaluate the accuracy of the Whisper models.

Calibration refers to the process of aligning a model's predicted probabilities with the true likelihood of outcomes, ensuring that the confidence scores it produces are interpretable and reliable. In the context of Whisper, calibration is essential to assess and improve the accuracy of the generated transcriptions. By analysing the probability scores assigned to subtokens, calibration allows us to quantify the model's uncertainty and identify patterns of overconfidence or underconfidence. This process involves statistical techniques such as calibration models, which adjust probability outputs to better reflect true accuracy. Effective calibration enhances trust in the model's predictions and enables more informed downstream decision-making, particularly in critical applications requiring high reliability.

## 3 Documenting Whisper's Dictionary of subtokens used for Persian

We compared two methods. We first compiled the prediction at subtoken levels extracted for the different models. Then we run the tokenisation algorithms on the Persian component of the FLEURS training data to compare the distribution of subtokens. The FLEURS dataset <fnref n="8" /> (FLores Enhanced Universal Representation for Speech) is a groundbreaking multilingual speech dataset, serving as the speech-focused counterpart to the FLoRes machine translation benchmark. It builds on the FLoRes dev and devtest datasets, comprising 2009 n-way parallel sentences translated into 102 languages. This extensive coverage makes FLEURS one of the most valuable resources for advancing multilingual speech research.

A key strength of FLEURS lies in its use of multilingual fine-tuning, which allows models to harness shared linguistic features across diverse languages. The dataset's results are thoughtfully organized into seven geographical regions, showcasing the dataset's cultural and linguistic breadth. Among the languages included is Persian, highlighting the dataset's inclusion of languages from the Central Asia/Middle East/North Africa region.

<footnote n="6">⁶ https://github.com/jbyunes</footnote>

<footnote n="7">⁷ https://github.com/ggerganov/whisper.cpp</footnote>

<footnote n="8">⁸ https://huggingface.co/datasets/google/fleurs</footnote>
# 6 Experiments

This section describes the experiments conducted in this study on written Persian. To better understand the distribution of graphemes in Persian, we used the FLEURS training dataset [6], which is publicly available in 102 languages and published by Google. We analysed the distribution of Persian graphemes across this dataset. The dataset contains a total of 280,786 graphemes, with 34 unique graphemes. Figure 4 shows the distribution of graphemes in written Persian. We hypothesize that the more frequently a subtoken appears in publicly available online corpora, the fewer challenges LLMs like Whisper encounter in accurately processing it. Frequent exposure to common subtokens during training allows the model to develop a stronger representation and mapping for these elements, reducing errors in transcription and improving overall reliability. Conversely, subtokens that are rare or less represented in the training data are more likely to lead to inaccuracies, reflecting gaps in the model's learned patterns.

<figure><img src="image_3.png" /><figcaption>Figure 2: Distribution of Persian Graphemes in the FLEURS dataset</figcaption></figure>

## 1 Distributional constraints

To improve the performance of Natural Language Processing (NLP) tasks, the quantity of training data play a crucial role [3]. In general, an increase in the volume of training data directly correlates with improved task outcomes, as measured by various performance metrics. For instance, in Automatic Speech Recognition (ASR) tasks, an increase in the availability of training data for a particular language tends to lower the Word Error Rate (WER), indicating enhanced model accuracy and reliability.
Figure 4 illustrates the distribution of graphemes in the FLEURS dataset. This subsection examines the frequency of consonants and vowels that appear at the beginning and end of words in written Persian, providing insight into the analysis of subtokenisation output produced by Whisper. As shown in figure 3, the characters <د>, <م>, <ب>, and <ک> typically appear at the beginning of words, while the consonants <ر>, <د>, <ه>, and <ن> typically appear at the end of words. Regarding vowels, the vowel <ی> frequently appears both at the beginning and at the end of words. If we consider the FLEURS corpus as a foundational dataset for Persian, it is reasonable to assume that Whisper, when trained on similar data, may exhibit comparable patterns in transcription accuracy. Specifically, Whisper may be more likely to accurately transcribe graphemes that are highly represented in the corpus. Conversely, graphemes that are under-represented in the dataset might pose challenges, potentially leading to transcription inaccuracies due to limited exposure during training.

<figure><img src="image_4.png" /><figcaption>Figure 3: Positional distribution of consonants and vowels at the beginning and end of words</figcaption></figure>

## 2 Positional constraints in Whisper for Persian

This subsection analyses the positional constraints of Persian graphemes in the Whisper dictionary. By accessing to the dictionary of subtokens for all 99 languages supported by Whisper, we extracted the subtokens associated with the Perso-Arabic script. We identified four possible positions for Persian characters within a word: initial, medial, final, and isolated forms <fnref n="9" />.

Table 3 summarises the findings of this analysis. The first column presents the IPA representations of Persian characters. The second column lists the

<footnote n="9">⁹ Some entries in the columns of graphematic representation for Persian contain a value of 0, indicating that these forms do not exist in the Persian script.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 4: Distribution of common consonants and vowels at the beginning and at the end of words</figcaption></figure>

isolated forms of Persian graphemes, accompanied by a binary code (0-1) indicating their presence in Whisper's subtoken dictionary. A value of 1 signifies that the grapheme is included in the dictionary, while a value of 0 indicates its absence. For the isolated representation of graphemes, it was observed that پ, which belongs exclusively to the Persian alphabet and not the Arabic script, is not present in the dictionary.

The fourth column shows the final graphematic representations of characters. Notably, the characters <گ غ ز ژ چ پ> are absent from Whisper's subtoken dictionary in their final forms. It is important to highlight that پ ژ چ are Persian-specific graphemes.

The sixth column represents the medial forms of graphemes in Persian. In Whisper's subtoken dictionary, we did not observe any Persian characters in their medial forms, except for <ی ن ل ع ت >.

The eighth column reports the initial forms of characters within Persian tokens. As shown in Table 3, the following characters are absent in their initial forms: <گ غ ظ ض چ ج پ>.
<table><caption>Table 3: Whisper coding of Persian Letters in its subtoken dictionary</caption><thead><tr><th>IPA</th><th>Isolated form</th><th>Whisper isolated</th><th>Final form</th><th>Whisper final</th><th>Medial form</th><th>Whisper medial</th><th>Initial form</th><th>Whisper initial</th></tr></thead><tbody><tr><td>[b]</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td></tr><tr><td>[b]</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td><td>ب</td></tr><tr><td>[p]</td><td>پ</td><td>پ</td><td>پ</td><td>پ</td><td>پ</td><td>پ</td><td>پ</td><td>پ</td></tr><tr><td>[t]</td><td>ت</td><td>ت</td><td>ت</td><td>ت</td><td>ت</td><td>ت</td><td>ت</td><td>ت</td></tr><tr><td>[s]</td><td>س</td><td>س</td><td>س</td><td>س</td><td>س</td><td>س</td><td>س</td><td>س</td></tr><tr><td>[dʒ]</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td></tr><tr><td>[tʃ]</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td><td>ج</td></tr><tr><td>[h]</td><td>ح</td><td>ح</td><td>ح</td><td>ح</td><td>ح</td><td>ح</td><td>ح</td><td>ح</td></tr><tr><td>[χ]</td><td>خ</td><td>خ</td><td>خ</td><td>خ</td><td>خ</td><td>خ</td><td>خ</td><td>خ</td></tr><tr><td>[d]</td><td>د</td><td>د</td><td>د</td><td>د</td><td>د</td><td>د</td><td>د</td><td>د</td></tr><tr><td>[z]</td><td>ذ</td><td>ذ</td><td>ذ</td><td>ذ</td><td>ذ</td><td>ذ</td><td>ذ</td><td>ذ</td></tr><tr><td>[r]</td><td>ر</td><td>ر</td><td>ر</td><td>ر</td><td>ر</td><td>ر</td><td>ر</td><td>ر</td></tr><tr><td>[z]</td><td>ز</td><td>ز</td><td>ز</td><td>ز</td><td>ز</td><td>ز</td><td>ز</td><td>ز</td></tr><tr><td>[ʒ]</td><td>ژ</td><td>ژ</td><td>ژ</td><td>ژ</td><td>ژ</td><td>ژ</td><td>ژ</td><td>ژ</td></tr><tr><td>[s]</td><td>س</td><td>س</td><td>س</td><td>س</td><td>س</td><td>س</td><td>س</td><td>س</td></tr><tr><td>[ʃ]</td><td>ش</td><td>ش</td><td>ش</td><td>ش</td><td>ش</td><td>ش</td><td>ش</td><td>ش</td></tr><tr><td>[s]</td><td>ص</td><td>ص</td><td>ص</td><td>ص</td><td>ص</td><td>ص</td><td>ص</td><td>ص</td></tr><tr><td>[z]</td><td>ض</td><td>ض</td><td>ض</td><td>ض</td><td>ض</td><td>ض</td><td>ض</td><td>ض</td></tr><tr><td>[t]</td><td>ط</td><td>ط</td><td>ط</td><td>ط</td><td>ط</td><td>ط</td><td>ط</td><td>ط</td></tr><tr><td>[z]</td><td>ظ</td><td>ظ</td><td>ظ</td><td>ظ</td><td>ظ</td><td>ظ</td><td>ظ</td><td>ظ</td></tr><tr><td>[ʔ]</td><td>ع</td><td>ع</td><td>ع</td><td>ع</td><td>ع</td><td>ع</td><td>ع</td><td>ع</td></tr><tr><td>[q]</td><td>غ</td><td>غ</td><td>غ</td><td>غ</td><td>غ</td><td>غ</td><td>غ</td><td>غ</td></tr><tr><td>[f]</td><td>ف</td><td>ف</td><td>ف</td><td>ف</td><td>ف</td><td>ف</td><td>ف</td><td>ف</td></tr><tr><td>[g]</td><td>ق</td><td>ق</td><td>ق</td><td>ق</td><td>ق</td><td>ق</td><td>ق</td><td>ق</td></tr><tr><td>[k]</td><td>ک</td><td>ک</td><td>ک</td><td>ک</td><td>ک</td><td>ک</td><td>ک</td><td>ک</td></tr><tr><td>[g]</td><td>گ</td><td>گ</td><td>گ</td><td>گ</td><td>گ</td><td>گ</td><td>گ</td><td>گ</td></tr><tr><td>[l]</td><td>ل</td><td>ل</td><td>ل</td><td>ل</td><td>ل</td><td>ل</td><td>ل</td><td>ل</td></tr><tr><td>[m]</td><td>م</td><td>م</td><td>م</td><td>م</td><td>م</td><td>م</td><td>م</td><td>م</td></tr><tr><td>[n]</td><td>ن</td><td>ن</td><td>ن</td><td>ن</td><td>ن</td><td>ن</td><td>ن</td><td>ن</td></tr><tr><td>[v]</td><td>و</td><td>و</td><td>و</td><td>و</td><td>و</td><td>و</td><td>و</td><td>و</td></tr><tr><td>[h]</td><td>ه</td><td>ه</td><td>ه</td><td>ه</td><td>ه</td><td>ه</td><td>ه</td><td>ه</td></tr><tr><td>[j]</td><td>ی</td><td>ی</td><td>ی</td><td>ی</td><td>ی</td><td>ی</td><td>ی</td><td>ی</td></tr></tbody></table>
## 3 Experiment on Spontaneous Data

To understand the phonographemic representation of Whisper, we analysed a 69-minute spontaneous video from YouTube. The context is a studio where three young men (in their 30s) discuss various social and cultural topics. The session is mostly informal, with frequent laughter, repetition, and other linguistic and non-linguistic discourse markers, which could pose challenges for ASR systems to transcribe.

This reverse engineering helps us understand how Persian graphemes are encoded in a multilingual model like Whisper. Furthermore, we aim to comprehend the architectural bias and properties of LLM subtokens and explore how the linguistic features of Persian are encoded in LLMs. Additionally, we hope to gain a better understanding of how languages related to Persian may influence subtokenisation.

To assess the accuracy of Whisper in transcribing this audio file, we report the Word Error Rate (WER) and Character Error Rate (CER). The WER of 0.403 suggests that approximately 40 % of the words in the Whisper's prediction are incorrect compared to the gold standard. A WER of 40.32 % indicates a significant discrepancy between Whisper's prediction and the gold standard reference. This is generally considered poor performance for many tasks, as nearly half of the words are incorrect. The CER of 0.222 suggests that 22.27 % of the characters in Whisper's prediction are incorrect when compared to the gold standard, reflecting a relatively high error rate in Whisper's transcription.

<table><caption>Figure 5: Performance Metrics for Prediction Accuracy of Whisper</caption><thead><tr><td>Metric</td><td>Value</td></tr></thead><tbody><tr><td>Precision</td><td>0.0004</td></tr><tr><td>Recall</td><td>0.0003</td></tr><tr><td>F1-Score</td><td>0.0003</td></tr></tbody></table>

The results we have obtained from the Precision, Recall, and F1-Score 5 are extremely low, indicating poor alignment between the Whisper's predictions and the gold standard at the word level. Here is what each of these metrics suggests:

Precision measures how many of the predicted words are actually correct. A precision of 0.0354 % means that out of all the words Whisper predicted, almost none of them were correct. This indicates a high number of false positives (words predicted incorrectly). Recall measures how many of the actual correct words were successfully predicted. A recall of 0.029 % means the system is missing nearly all the correct words, suggesting a very high
number of false negatives. F1-Score combines both precision and recall into a single metric. With such a low score, it reflects that both the precision and recall are very low, indicating a significant mismatch between the prediction and the reference.

## 4 The speech to subtoken mapping

In our analysis of the Persian subtokens derived from the dataset, we observe a notable distribution of token frequencies, with certain characters and sequences appearing much more frequently than others illustrated in figure 6. The most frequent subtokens are key characters that are integral parts of many common Persian words. The dominance of these components reflects both the linguistic characteristics of Persian and the tokenization strategy employed by the Whisper model.

<figure><img src="image_6.png" /><figcaption>Figure 6: Distribution of Persian Graphemes in our YouTube Dataset</figcaption></figure>

Another important aspect of subtokenisation is the length of the subtokens across different languages. In the case of Persian, the tokenisation process tends to generate shorter subtokens. Based on our analysis of the subtokenisation of Persian YouTube data, out of a total of 30,544 subtokens generated by Whisper, 24,190 consist of only a single Persian character. Subtokens containing two Persian characters account for 4,325, while those with three Persian
characters are limited to just 11. This observation suggests that subtokenisation for Persian, as an under-resourced language, may differ significantly from languages with larger training datasets, potentially reflecting the challenges and limitations in processing languages with less representation in model training.

## 5 Experiment on Whisper's Alternate Prediction

We ran another experiment based on the experimentally customised implementation of C++ Whisper that extracts the 10 first predictions of the model and the corresponding probabilities<fnref n="10" />. Do we get subtokens reserved for Persian or character-based representations of corresponding sounds for the alternate predictions?

<figure><img src="image_7.png" /><figcaption>Figure 7: Top 10 alternate predictions for the sentence 001-A.wav file from the Common Voice corpus (large model)</figcaption></figure>

On Figure 7, one can observe the prediction used in the final transcription (prediction 1) and the alternate predictions. A bit like with the beam search for machine translation, we have access to the alternative transcriptions of the model. For instance, the first line, which predicts the special token, lists the alternate candidates for the transcription, all of which correspond to times-tamps close to the initial starting point of the file. If we take the highlighted line of predictions (1.16), the first prediction (with a high level of certainty, 0.9969) corresponds to the unique letter that corresponds to the bilabial nasal /m/. Predictions 2 and 4 correspond to combinations with the vowel <م> and the consonant <مر>, respectively, while predictions 3 and 5 correspond to the Latin letter encoding the bilabial nasal. We also observe an unknown character in prediction 6. The seventh and eighth generated subtokens are <ب>, a voiced bilabial plosive, and <ن>, a voiced alveolar nasal. These

<footnote n="10">¹⁰ https://github.com/ggerganov/whisper.cpp</footnote>
cases are particularly intriguing as they highlight challenges faced by audio LLMs like Whisper in distinguishing between phonemes with similar places or manners of articulation. This comparison of the Whisper transcription (prediction 1) with the alternate predictions allows us to investigate the subtokens that are the potential graphemic candidates that correspond to a portion of the acoustic signal. This technique allows us to probe the subtokens that may correspond to the corresponding speech interval. This method allows us to answer this phonographemic question: What are the competing subtokens for a given representation of the signal for a given time slice of the signal and the competing languages or the competing subtokens for a given portion of the signal?

# 7 Results

This section presents the key findings, including the qualitative and quantitative analyses from the transcription task using various Whisper models, as discussed in 1 and 2. Additionally, it highlights the findings on subtokenisation phase in Whisper for Persian, detailed in 3. In 5, we address the four main research questions.

## 1 Qualitative Analysis of the Transcriptions

This subsection evaluates the performance of various Whisper models in transcribing Persian. The base model showcases robust multilingual outputs. We have observed tokens from other languages like Serbian, Chinese, Arabic, and even mixed-language phrases, as seen in examples like *Après (=after) getting* (a blend of French and English). Furthermore, it often produces hallucinations in the form of meaningless, repetitive phrases—particularly in the middle of transcriptions—which affect the coherence and reliability of the output [14].

The tiny model exhibits a higher frequency of hallucinations than the base model, often repeating clusters of characters or sentences. These hallucinations frequently contain words from other languages, predominantly English. Hallucinations are found in multilingual models in various translation contexts. Hallucinations are most common in low-resource languages and non-English output, sometimes exposing toxic patterns linked to training data [14]. The small model reduces the presence of foreign tokens significantly, with only isolated English words occasionally appearing within Persian transcriptions. This improvement indicates better language isolation, allowing for clearer and more coherent transcriptions with fewer instances of language interference.

In the medium model, Persian transcription accuracy improves further; however, the model fails to handle the same phoneme, i.e., /z/, which has multiple graphemic representations in Persian: <ژ>, <ذ>, <ض>, and <
ط >. Additionally, there are instances where the sequence of Persian characters is rendered incorrectly, resulting in absurd or invalid forms. For example, the noun < لطف > (/lotf/, meaning *grace*) is transcribed incorrectly as < لوف > (/lutf/), which is not a valid sequence in Persian. There are two issues with this transcription. First, the short vowel /o/, which is not explicitly written in Persian, is incorrectly transcribed as the long vowel /u/ in the erroneous version of the word. Second, the transcription selects the wrong grapheme to represent the phoneme /t/ in Persian, choosing < ت > instead of < ط >. Overall, while the model achieves greater accuracy in transcribing Persian, issues with hallucinations persist.

Finally, the large models offer the highest level of transcription quality, virtually eliminating hallucinations and foreign language tokens. Transcriptions are consistently accurate and free from unrelated phrases. Named Entity Recognition (NER) issues are acknowledged in [30]; proper nouns can undergo what we can call a *retranscription* [27], i.e., that differs from the original text in terms of choosing the correct grapheme, but that is phonetically consistent with the speech input. For example, the proper noun < رضا > (Reza), is written with the grapheme < ض >, representing the /z/ sound. However, the model consistently uses the wrong grapheme < ژ >, which is not licit.

## 2 Quantitative Analysis of the Transcriptions

This subsection presents a statistical analysis of the transcriptions generated by Whisper, focusing on Word Error Rate (WER) and Character Error Rate (CER) as key metrics to evaluate transcription accuracy [26]. For Persian, CER is particularly valuable, as it accounts for variations in spacing, which are common in the language and can impact transcription evaluation. Unlike WER, CER effectively ignores inconsistencies in how spaces are placed between letters, providing a more precise measure of accuracy for Persian text. For Persian, CER is very close to a subtoken error rate.

<table><caption>Table 4: WER and CER results for different models. The best-performing model (Large) is shown in bold characters</caption><thead><tr><th>Model</th><th>WER</th><th>CER</th></tr></thead><tbody><tr><td>Base</td><td>0.97</td><td>0.84</td></tr><tr><td>Tiny</td><td>2.66</td><td>1.22</td></tr><tr><td>Small</td><td>0.89</td><td>0.58</td></tr><tr><td>Medium</td><td>0.83</td><td>0.56</td></tr><tr><td>Large</td><td><strong>0.42</strong></td><td><strong>0.20</strong></td></tr><tr><td>Large-v2</td><td>0.57</td><td>0.24</td></tr></tbody></table>

The large Whisper model performed best, with the lowest WER (0.42) and CER (0.20). It captures both word-level and character-level accuracy
<figure><img src="image_8.png" /><figcaption>Figure 8: Whisper's Vocabulary growth curves in our transcription data</figcaption></figure>

effectively. The medium model also performed well, with WER (0.83) and CER (0.56), though it is not as accurate as the large model. Base and small models showed higher WER and CER compared to medium and large models. Tiny model performed the worst, with extremely high WER (2.66) and CER (1.22), indicating significant transcription errors. Model large-v2 strikes a balance but does not outperform the large model.

To compare the transcriptions produced by different Whisper models, ranging from base to large-v2, we employed a vocabulary growth method, as illustrated in Figure 8. The analysis of vocabulary growth patterns across Whisper model variants reveals distinctive logarithmic scaling behaviors when examined on a log-log scale. The data demonstrates that all models - from tiny to large-v2 - exhibit an initial phase of rapid vocabulary acquisition, followed by a more gradual growth trajectory. This pattern is particularly evident in the steep ascent observed between $10^0$ and $10^2$, where the learning curve is most pronounced. The convergence of multiple model trajectories around the $10^2$ word mark suggests a common developmental threshold across model architectures.

## 3 Distribution of subtokens in transcription across models

Our comparative analysis between model variants reveals hierarchical performance patterns aligned with model size. The larger models, particularly large
and large-v2, demonstrate marginally superior vocabulary acquisition capabilities, especially beyond the 10³ word threshold. The medium and small variants maintain remarkably similar growth trajectories through the middle ranges, while the tiny and base models show distinct divergence patterns at various intervals. This divergence becomes more pronounced in the upper word count ranges, suggesting that model capacity plays an increasingly significant role in vocabulary acquisition as the lexical complexity increases. Notably, several models exhibit plateau phases at different points in their growth curves, indicating potential capacity limitations or optimization boundaries specific to each model architecture.

In this subsection, we report our findings on the transcription of the data. Figure 9 displays the frequency of different Persian subtokens produced by the Whisper tiny, medium, and large-v1 models. The x-axis represents different Persian subtokens. The y-axis shows the frequency values, ranging up to 5000. As for the tiny model, the subtoken on the far left □ has the highest frequency, exceeding 5000. The remaining subtokens have lower frequencies, ranging approximately between 1000 and 2000. The pattern shows an interesting distribution, with one dominant token and several others at lower frequencies. Looking at the bar chart for the Medium Whisper model's Persian subtoken distribution, we observe that the highest frequency is 4100 occurrences for the subtoken <c>. The second most frequent subtoken <o> has approximately 3750 occurrences. There is a significant drop after the second token, with the next group clustering around 1500-2000 occurrences. The lowest frequencies, <ρ> and <o>, are seen in the last two subtokens, with around 300-400 occurrences. In sum, the graph illustrates a clear non-uniform distribution of Persian subtokens. This distribution may affect how well the model handles different types of Persian text.

Figure 9 also shows the frequency of different Persian subtokens produced by the large-v1 model of Whisper. There appears to be a descending pattern in frequency from left to right like the other models. The most frequent subtoken <c> has approximately 5000 occurrences, and the second most common subtoken <o> has around 3300 occurrences. The remaining subtokens show relatively consistent frequencies between 1500-2000 occurrences. As for the analysis of the pattern, we observed that the distribution is not uniform. There is a sharp decline between the two first frequent subtokens. After the second subtoken, the frequency stabilises with a more gradual decline.

<figure><img src="image_9.png" /><figcaption>Figure 9: Comparison of the distribution of subtokens across models</figcaption></figure>
By analysing the distribution of subtokens in Whisper outputs across dif-
ferent Whisper models, we can compare these distributions to those of Persian
subtokens within the FLEURS dataset, as illustrated in 4. All Whisper mod-
els, with the exception of smaller models (e.g., tiny), demonstrate consistency
with the distribution of graphemes in the FLEURS dataset. The graphemes
<ς> and <l> are the most frequent in the FLEURS dataset, a pattern
also reflected in the outputs of the larger Whisper models, namely medium
and large-v1.

4 Probing of the 10 Best Predictions

In this experiment, we considered 15 minutes of speech from the Mozilla Foun-
dation and we analysed the ten best predictions for the large model. We first
show the validity of the results of our method probing the alternative predic-
tions for the special subtoken used to indicate the beginning of transcriptions
[_BEG_] . We used the density probability representation (Figure 10), which
is why the probabilities of the unique occurrences of [ _TT_30 ] [ _TT_6 ] are
not represented. The most frequent second alternate predictions are [ _TT_1 ]
(5 occurrences) and [ _TT_2 ] (23), where these special subtokens correspond
to time stamps [3]. Even if the time alignment is far from perfect with Whis-
per [3], the results are consistent with the corresponding portion of the signal:
these time stamps point to the very beginning of the signal (the bigger the
[ _TT_ ] value, the further from the beginning).

We similarly illustrate the consistency of the Whiper alternate predictions
for the voiced alveolar fricative and for the bilabial nasal as case studies.

For the transcription of the voiced fricative /z/ for the word <همزمان>,
the expected letter would be <j> but alternate predictions include subtokens
belonging to other languages, like <зам>/zam/, which is a Russian word possibly
meaning “deputy” or “assistant” or the subtoken in Latin characters zaman
(see Figure 7).

The first prediction corresponds to the correct letter in Persian for the
voiced alveolar fricative. Nevertheless, the alternative letters are used in the
predictions three and five and six and eight, but they are mostly used in
Arabic. Alternatively, the full subtoken zaman is used in the fifth prediction
and corresponds to the full realization of the following sounds. The alternate
candidates for the same portion of the speech signal are nevertheless consistent
rendering and transcriptions of the sibilant so that we can see the competition
of the subtokens of the different languages in the prediction. What this probing
of the alternate predictions confirms is the correspondence between the speech
signal and the acoustic properties of the corresponding speech signal.

A similar phenomenon can be observed with nasals. They all correspond
to subtokens in different languages that correspond to the same phonological
sounds like m and M with the Latin script.

On Figure 11, one can see the alternate predictions for the bilabial nasal
min, where the following vowel can be observed as well as the alveolar nasal <
<figure><img src="image_10.png" /><figcaption>Figure 10: The second best predictions for the [BEG] Whisper special token for transcription beginning (large model)</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 11: The second best predictions for the bilabial nasal (large model)</figcaption></figure>
ی >. Probing Whisper at the prediction phase does not capture the grapho-
tactics of Whisper but is a vantage point for the characterisation of the phono-
graphematics of an audio LLM and efficiently summarises the dictionary biases
of the subtokenisation: languages with a limited number of subtokens in the
Whisper dictionary are more likely to be transcribed with subtokens used to
represent other languages. Since all the Whisper models were trained with the
same subtoken dictionary, this probably explains the hallucinations observed
with the lower models: the transcriptions use ‘foreign’ subtokens that are
deemed to be more probable because the association of the foreign subtoken
with the acoustic signal is more present in the training data of other languages.

5 Answers to our Research Questions

**RQ1 Do LLM subtokens follow Zipf's law and other related large number of rare events (LNRE) models?**

We show that the Whisper models distributions of subtokens do not follow
Zipf's law as indicated in 9. Realising that the Whisper predictions were
only based on the isolated form of Persian letters came as a shock to the
authors. It could be construed that this is the consequence of the small amount
of training hours. We hypothesise that those under-represented languages
are likely to be treated on a character-based scale, namely only the abjad
(alphabet) is represented in the Whisper subtoken dictionary, perhaps only in
its citational form or its isolated form, as opposed to languages which are much
more represented in the training data. For the latter, subtokens corresponding
to full words are much more likely to be found, as is the case with English and
French.

**RQ2 Are there morpho-phonological rules that potentially map subtokens to phonological representations?**

An intriguing observation is the decoder’s “knowledge” to accurately post-
process transcriptions of written Persian into their correct positional variants.
This is particularly significant given that the subtoken dictionary in Whisper
contains only the isolated forms of Persian graphemes, without including their
initial and medial variants.

**RQ3 Can an architectural bias be observed in the knowledge encoded for Persian?**

The transcription outputs from Whisper demonstrate a notable orientation
toward selecting the formal variants of words, reflecting a bias in the lan-
guage model's processing. For instance, the Persian direct object marker /rd/
is consistently transcribed in its formal written form rather than its collo-
quial variant /ro/, which is more common in spoken Persian. This preference
suggests that Whisper's underlying linguistic framework is influenced by the
conventions of written Persian, potentially prioritising the standard or literary
register over spoken forms.
Another significant factor influencing transcription is the presence of loan-
words from languages such as Arabic and Urdu. These borrowed words, which
are integrated into Persian vocabulary, may exhibit unique phonetic and mor-
phological patterns that affect how the model processes and transcribes them.
The handling of loanwords can highlight the model's adaptability to cross-
linguistic influences, but it also raises questions about the comprehensive-
ness of its training data in capturing these nuances. Moreover, the language
frequency distribution within the Whisper dictionary plays a crucial role in
shaping transcription outputs. The frequency of subtokens directly impacts
the likelihood of their selection during the transcription process. If a subto-
ken appears more frequently in the training data, the model is more inclined
to produce it, even in contexts where a less frequent variant might be more
appropriate. This distribution can influence both the selection of formal ver-
sus colloquial variants and the representation of loanwords, underscoring the
interplay between token frequency and linguistic accuracy in Whisper's tran-
scriptions.

**RQ4** *How are special phenomena in Persian script such as the connector ezafe encoded and processed with Whisper?*

A noteworthy observation across Whisper models is the addition of a letter referred to as the silent /he/ to represent the *ezafe* construction, which corresponds to an unstressed /e/ sound. In standard Persian orthography, this [e] sound is not explicitly transcribed. For example, Whisper transcribes <دوسته من> (/dusteh man/, meaning “my friend”), whereas the gold standard transcription in Persian is <

8 Discussion

1 Distribution of subtokens in Whisper Dictionary: the case of Arabic

To analyse the distribution of Perso-Arabic written characters, we extracted the corresponding subtokens from the subtoken dictionary of Whisper. Figure 12 shows that 345 subtokens are associated with Perso-Arabic characters. Among these, 215 subtokens correspond to both Persian and Arabic (labeled as “Both”), 88 subtokens are specific to Arabic (labeled as “ARunique”), and only 12 subtokens are exclusive to Persian, representing characters that are uncommon in Arabic but unique to Persian (labeled as “FAunique”).

Figure 12 highlights the over-representation of Arabic, which shares the
same script as Persian, in Whisper's subtoken dictionary. This imbalance ex-
plains the predominance of Arabic outputs, particularly in Whisper's smaller
models (e.g., base, tiny, and small). Other languages are also present in the
dictionary, notably Chinese, which reflects the challenges of multilingual mod-
els with uneven language distributions. This issue has been studied, and prior
<figure><img src="image_12.png" /><figcaption>Figure 12: Language Frequency Distribution in the Whisper Dictionary</figcaption></figure>

research suggests that fine-tuning can significantly improve performance for under-represented languages by allocating more subtokens to them[9].

The lexical status of digraphs for low-resourced languages We discuss the number of subtokens used to reconstruct Persian texts in Whisper, compare the status of digraphs and trigraphs to other languages like French. Are there any trigraphs in the predictions? Are digraphs all lexemes or morphemes?

## 2 Sensitivity to speaker variability

Building on previous experiments, we report our findings on the variability of Whisper outputs for the same text read by different native Tehrani speakers. Based on this experiment, Despite both female speakers employing the Tehrani dialect, commonly spoken in Iran's capital, variations in the articulation of certain consonants and vowels were observed when the same word was read. Specifically, phonemic confusions were noted, including between /m/ and /b/, as well as /ʃ/ and /tʃ/. Additionally, inconsistencies arose in the differentiation between short and long vowels, such as /e/ (close-mid front unrounded) and /a/ (open front unrounded). These findings suggest that subtle differences in speakers' manner of articulation can influence the transcription outputs.

## 3 Sensitivity to Gender, Social Level, Accent, Register

Understanding how speaker attributes—such as gender, social background, accent, and register—affect ASR system performance is a critical area of investi-
gation. [13] focused on English, evaluates the impact of these variables on Ope-
nAI's Whisper ASR system. The results reveal Whisper's strong recognition
capabilities for North American English accents but highlight challenges with
British, Australian, and certain non-native accents, particularly Vietnamese
and Thai. Speaker-specific characteristics, including English proficiency, vowel
inventory, and language typology, significantly influence recognition accuracy.
Notably, male speakers generally experience higher error rates. Whisper also
demonstrates superior performance with read speech compared to conversa-
tional speech, though the gender-related accuracy gap diminishes in conver-
sational contexts. These findings underscore the importance of developing
robust and inclusive ASR systems that can effectively handle diverse accents
and speech patterns. Future research should prioritise reducing biases, uti-
lizing more diverse datasets, and advancing personalized, context-aware ASR
technologies to enhance real-world applicability.

4 The Subtokenisation Architectural Bias of audio LLMs

An important factor to further investigate for the phono-graphematics of
Whisper is the role of the distribution of languages and of the subtokenisation
algorithm. Under-resourced languages seem to be doomed to a character-
based representation : whereas languages like English are represented by sev-
eral thousand subtokens, some of them being actual words, less represented
language only get a small fraction of the subtoken dictionary. The unequal
distribution of the languages in the training data lead to an architectural bias
in the Whisper models [3]. Previous research has pointed out some architec-
tural biases, effects of the subtokenisation on the process and on the tasks. For
example, for translation, [37] have shown that the more subtokens are needed
for occupational nouns, the more gender bias errors are likely to be found in
translations. Therefore, it is likely that subtokenisation raises issues that are
still to be uncovered, whether in the representation of the written language or
for the spoken language. Our paper is an initial investigation of these issues
using a language that is under-represented in the lists of languages used for
the training data, suggesting some of the dominant effects that can also be
pointed out when investigating the transcriptions from Whisper. Clearly, the
model size, therefore the number of representations, plays a crucial role, as
well as, predictably enough, the size of the training data. Compared to the
larger model where very few representations were spotted when transcribing
12 hours of speech of Persian, errors of that type could be spotted in the
different models.

It may be the case that alternate (sub)tokenisation algorithms may be
preferable, a point already made for the morphological tokenisation of So-
ranî Kurdish [1], which is written in a modified Perso-Arabic script. Be-
cause of its specific orthographic and structural properties, the Perso-Arabic
script presents specific challenges and opportunities for subword tokenization.
Subtokenisation models need to account for these differences to achieve effec-
tive morphological segmentation, and the Unigram model tends to outperform BPE for in this task. It should be noted that this issue has been raised for LLMs [29] and mitigating strategies [10, 15] have been proposed for written models, and comparable experiments should be carried out with audio LLMs.

## 9 Conclusion and Future Research

We have probed the graphemic outputs of Whisper and some of its internal representations for the transcriptions of spontaneous speech. Our reverse engineering perspective has enabled us to document how graphemes of Persian are encoded in a multilingual LLM model like Whisper. Our methodology could be replicated for other languages that have a writing system potentially biased in the (sub)tokenisation phase. We have been able to discuss architectural bias and properties of LLM subtokens and explore how linguistic properties (like the *ezafe*) are encoded in LLMs.

One of the questions is the generalisability of our reverse engineering method. We can suggest that it can be extended to all the languages which are treated by Whisper as a character-based model, namely languages for whose the Whisper dictionary of subtokens mostly comprises their alphabet. As a result of which, the decoder has to apply most of the graphotactic constraints of the said language. This is what we showed with the automatic substitution of initial and final letters which were required in the final transcript as opposed to the grapheme in isolation that are predicted internally, see Table 3 that summarizes this point. The list of languages likely to be treated on a character-based system are likely to be deducted from the small number of hours in the training data as published in the [30] appendix.

The demographic of the speakers in the Whisper training data is only partially documented in [30] since the training data is not really known, even for the small number of hours (24) of the Persian training data. We can nevertheless surmise that similar effects that have been diagnosed for English [13, 32] can be observed, for example, a different sensitivity to accents and gender has been documented for English [13]. Future research will need to see if the fine-tuned model for Persian using the Mozilla Foundation Common Voice data<fnref n="11" /> has similar gender bias for speech recognition.

Our strategy has been to treat the Whisper dictionary of subtokens as a repository of graphemic units mapped to acoustic representations of varying scope (from phonemic, syllabic to word-based representation for languages like English) correspondences We do not detail here the fraction of special subtokens used to represent tasks, task position of time stamps, see for instance [3]. We have shown that Persian corresponds the character-based representation : a limited number of subtokens in the dictionary was used in the encoder for the prediction phase and graphotactic rules are applied by the decoder.

More generally, our roadmap for investigating the phonological ‘knowl-edge’ of audio LLMs consists of investigating the existence of phonographemic

<footnote n="11">¹¹ https://huggingface.co/speechbrain/asr-whisper-large-v2-commonvoice-fa</footnote>
mapping rules for subtokens. For this phonology of subtokens, if we assume
that subtokens are associated to similar pronunciations, what is the acoustic
granularity of this similarity? In the name of robustness, we have to as-
sume some plasticity for the acoustic input matching the subtoken, a unique
subtoken corresponding to several phonetic realisations. This grapho-centric
strategy, when applied to languages at the end of the spectrum like English,
corresponds to a sort of exemplar representation: what are the constraints for
a word to be represented by the subtoken?

To further our understanding, we propose experimental protocols that
leverage the Whisper tokeniser to generate a gold-standard tokenisation using
datasets such as the ISLE corpus. This approach would allow us to observe the
variation introduced by native speakers and investigate whether all subtokens
share the same degree of plasticity. By analysing the phonographematics of
audio LLMs, we can explore how robustness is encoded and how relevant subto-
kens map to the variability of audio inputs. Specifically, we aim to investigate
how Whisper enforces phonological contrasts in its transcriptions and whether
its mechanisms for handling acoustic variability differ across languages with
disparate levels of representation in the training data. For example, the sub-
stantial difference in training data—438,218 hours for English transcriptions
compared to 24 hours for Persian or 9,752 hours for French—likely impacts the
system's ability to represent phonetic contrasts and its robustness to speaker
or contextual variability. Future research should investigate the decoder's
role, the effects of training data discrepancies, and the dual functions of time-
segment subtokens in audio LLMs, aiming to deepen our understanding of
their phonological principles.

10 Acknowledgements

The authors would like to express their gratitude to Prof. Richard Wright for
his assistance with our revised phonological analysis of Persian and the pho-
netic correlates potentially associated with Persian graphemes. Additionally,
they are grateful to Dr. Jean-Baptiste Yunès for providing the C++ imple-
mentation of OpenAI's Whisper, which made this work possible. The authors
would like to extend their appreciation to the reviewers for their insightful
comments and constructive feedback, which have significantly enhanced the
quality of this work. We also thank Guillaume Wisniewski for comments on
an earlier version of this paper.

References

1. Ahmadi S (2020) A tokenization system for the Kurdish language. In: Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pp 114–127
2. Arab-Moghaddam N, Sénéchal M (2001) Orthographic and phonological processing skills in reading and spelling in Persian/English bilinguals. International Journal of Behavioral Development - INT J BEHAV DEV 25:140-147, DOI 10.1080/01650250042000320

3. Ballier N, Burin L, Namdarzadeh B, Ng SB, Wright R, Yunès JB (2024) Probing whisper predictions for French, English and Persian transcriptions. In: Abbas M, Freihat AA (eds) Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024), Association for Computational Linguistics, Trento, pp 129-138, URL https://aclanthology.org/2024.icnlsp-1.15

4. Baluch B, Choudhury S (2011) Spelling transparency and its impact on short-term memory: Evidence from persian and english. Spelling Skills: Acquisition, Abilities, and Reading Connection pp 93-106

5. Bostrom K, Durrett G (2020) Byte pair encoding is suboptimal for language model pretraining. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp 4617-4624

6. Conneau A, Ma M, Khanuja S, Zhang Y, Axelrod V, Dalmia S, Riesa J, Rivera C, Bapna A (2022) Fleurs: Few-shot learning evaluation of universal representations of speech. URL https://arxiv.org/abs/2205.12446, 2205.12446

7. Conneau A, Ma M, Khanuja S, Zhang Y, Axelrod V, Dalmia S, Riesa J, Rivera C, Bapna A (2023) Fleurs: Few-shot learning evaluation of universal representations of speech. In: 2022 IEEE Spoken Language Technology Workshop (SLT), IEEE, pp 798-805

8. Devlin J (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805

9. Downey C, Blevins T, Goldfine N, Steinert-Threlkeld S (2023) Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages. In: Ataman D (ed) Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), Association for Computational Linguistics, Singapore, pp 268-281, DOI 10.18653/v1/2023.mrl-1.20, URL https://aclanthology.org/2023.mrl-1.20

10. Downey C, Blevins T, Goldfine N, Steinert-Threlkeld S (2023) Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages. In: Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pp 268-281

11. Freihat AA, Abbas M (eds) (2021) Proceedings of the Second International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2021) co-located with ICNLSP 2021, Association for Computational Linguistics, Trento, Italy, URL https://aclanthology.org/2021.nsurl-1.0

12. Gerganov G (2003) whisper.cpp : A high-performance inference of OpenAI's Whisper automatic speech recognition (ASR) model. https://github.com/ggerganov/whispercpp

13. Graham C, Roll N (2024) Evaluating openai's whisper asr: Performance analysis across diverse accents and speaker traits. JASA Express Letters 4(2):025206
14. Guerreiro NM, Alves DM, Waldendorf J, Haddow B, Birch A, Colombo P, Martins AFT (2023) Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics 11:1500-1517, DOI 10.1162/tacl_a_00615, URL https://aclanthology.org/2023.tacl-1.85

15. Han Y, Zou D (2024) Understanding and mitigating tokenization bias in language models. In: Salakhutdinov R, Kolter Z, Heller K, Weller A, Oliver N, Scarlett J, Berkenkamp F (eds) Proceedings of the 41st International Conference on Machine Learning, PMLR, Proceedings of Machine Learning Research, vol 235, pp 17480-17504, URL https://proceedings.mlr.press/v235/han24g.html

16. Hosseini Fs, Kashef S, Shabaninia E, Nezamabadi-pour H (2021) IDPL-PFOD: An image dataset of printed Farsi text for OCR research. In: Freihat AA, Abbas M (eds) Proceedings of the Second International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2021) co-located with ICNLSP 2021, Association for Computational Linguistics, Trento, Italy, pp 22-31, URL https://aclanthology.org/2021.nsurl-1.4

17. Jafari MM, Behmanesh S, Talebpour A, Ghomsheh AN (2021) Improving pre-trained language model for relation extraction using syntactic information in Persian. In: Freihat AA, Abbas M (eds) Proceedings of the Second International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2021) co-located with ICNLSP 2021, Association for Computational Linguistics, Trento, Italy, pp 38-44, URL https://aclanthology.org/2021.nsurl-1.6

18. Karimi S (2005) A Minimalist Approach to Scrambling. De Gruyter Mouton, Berlin, New York, DOI doi:10.1515/9783110199796, URL https://doi.org/10.1515/9783110199796

19. Khalilia H, Freihat AA, Giunchiglia F (2021) The dimensions of lexical semantic resource quality. In: Freihat AA, Abbas M (eds) Proceedings of the Second International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2021) co-located with ICNLSP 2021, Association for Computational Linguistics, Trento, Italy, pp 15-21, URL https://aclanthology.org/2021.nsurl-1.3

20. Kudo T (2018) Sentencepiece: A simple and language independent sub-word tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:180806226

21. Mahootian S, Gebhardt L (1997) Persian. Descriptive grammars, Routledge, London, includes index

22. Maleki J, Ahrenberg L (2008) Converting Romanized Persian to the Arabic writing systems. In: Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), European Language Resources Association (ELRA), Marrakech, Morocco, URL http://www.lrec-conf.org/proceedings/lrec2008/pdf/738paper.pdf

23. Maleki J, Ahrenberg L (2008) Converting romanized persian to the arabic writing systems.
24. Manova S (2023) Chatgpt, n-grams and the power of subword units: The future of research in morphology. In: Fourth International Workshop on Resources and Tools for Derivational Morphology, vol 5, p 1

25. Mohebbi H, Chrupała G, Zuidema W, Alishahi A (2023) Homophone disambiguation reveals patterns of context mixing in speech transformers. In: Bouamor H, Pino J, Bali K (eds) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, pp 8249–8260, DOI 10.18653/v1/2023.emnlp-main.513, URL https://aclanthology.org/2023.emnlp-main.513

26. Morris AC, Maier V, Green PD (2004) From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. In: Interspeech, pp 2765–2768

27. Nicolas B, Adrien M, Maelle A, Yunès JB (2023) Using whisper LLM for automatic phonetic diagnosis of L2 speech, a case study with French learners of English. In: Abbas M, Freihat AA (eds) Proceedings of the 6th International Conference on Natural Language and Speech Processing (ICNLSP 2023), Association for Computational Linguistics, Online, pp 282–292, URL https://aclanthology.org/2023.icnlsp-1.30

28. Oji R, Taghizadeh N, Faili H (2021) Perspeldata: An exhaustive parallel spell dataset for Persian. In: Freihat AA, Abbas M (eds) Proceedings of the Second International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2021) co-located with ICNLSP 2021, Association for Computational Linguistics, Trento, Italy, pp 8–14, URL https://aclanthology.org/2021.nsurl-1.2

29. Petrov A, La Malfa E, Torr P, Bibi A (2024) Language model tokenizers introduce unfairness between languages. Advances in Neural Information Processing Systems 36

30. Radford A, Kim JW, Xu T, Brockman G, Mcleavey C, Sutskever I (2023) Robust speech recognition via large-scale weak supervision. In: Krause A, Brunskill E, Cho K, Engelhardt B, Sabato S, Scarlett J (eds) Proceedings of the 40th International Conference on Machine Learning, PMLR, Proceedings of Machine Learning Research, vol 202, pp 28492–28518, URL https://proceedings.mlr.press/v202/radford23a.html

31. Rahbari N, Sénéchal M (2009) Lexical and nonlexical processes in the skilled reading and spelling of Persian. Reading and Writing 22(5):511–530, DOI 10.1007/s11145-008-9122-1

32. Sanabria R, Bogoychev N, Markl N, Carmantini A, Klejch O, Bell P (2023) The Edinburgh international accents of English corpus: Towards the democratization of English ASR. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 1–5

33. Sartakhti MS, Etezadi R, Shamsfard M (2021) Improving Persian relation extraction models by data augmentation. In: Freihat AA, Abbas M (eds) Proceedings of the Second International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2021) co-located with ICNLSP
2021, Association for Computational Linguistics, Trento, Italy, pp 32–37, URL https://aclanthology.org/2021.nsurl-1.5

34. Sennrich R, Haddow B, Birch A (2016) Neural machine translation of rare words with subword units. In: Erk K, Smith NA (eds) Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Berlin, Germany, pp 1715–1725, DOI 10.18653/v1/P16-1162, URL https://aclanthology.org/P16-1162

35. Shakeri N, Soleymani Z, Zarifian T, Kamali M (2015) Investigating Phonological Awareness in Persian-Speaking Children With Phonological Disorders. Middle East Journal of Rehabilitation and Health 2(4):e32200, DOI 10.17795/mejrh-32200, URL https://brieflands.com/articles/mejrh-21516

36. Taghizadeh N, Ebrahimi A, Faili H (2021) Nsurl-2021 shared task 1: Semantic relation extraction in Persian. In: Freihat AA, Abbas M (eds) Proceedings of the Second International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2021) co-located with ICNLSP 2021, Association for Computational Linguistics, Trento, Italy, pp 1–7, URL https://aclanthology.org/2021.nsurl-1.1

37. Wisniewski G, Zhu L, Ballier N, Yvon F (2021) Screening gender transfer in neural machine translation. In: Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp 311–321