PhonologyBench: Evaluating Phonological Skills of Large Language Models (2024)

Abstract: Phonology, the study of speech’s structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. LLMs are widely used in various downstream applications that leverage phonology such as educational tools and poetry generation. Moreover, LLMs can potentially learn imperfect associations between orthographic and phonological forms from the training data. Thus, it is imperative to benchmark the phonological skills of LLMs. To this end, we present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. Despite having no access to speech data, LLMs showcased notable performance on the PhonologyBench tasks. However, we observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans. Our findings underscore the importance of studying LLM performance on phonological tasks that inadvertently impact real-world applications. Furthermore, we encourage researchers to choose LLMs that perform well on the phonological task that is closely related to the downstream application since we find that no single model consistently outperforms the others on all the tasks. 1

heading: 1 Introduction
text: Large language models (LLMs) are a class of deep generative models (Ruthotto and Haber, 2021; Bond-Taylor et al., 2021) that are trained on an internet-scale text corpora (e.g., two trillion tokens). To assess their usefulness in the real-world scenarios, prior work has focused on holistic evaluation of LLMs (Liang et al., 2022). For instance, they
1We will release the dataset and code in the camera-ready version.
are evaluated on tasks that require syntactic and semantic understanding of the language such as summarization, rationale generation, story generation and question answering (Liang et al., 2022; Zheng et al., 2023; Valmeekam et al., 2023; Bang et al., 2023; Beeching et al., 2023; Qin et al., 2023; Kocoń et al., 2023; Sun et al., 2023). However, there are various text-based tasks of practical importance that require joint understanding of the written and spoken language such as poetry generation (Ormazabal et al., 2022; Henderson, 1965; Suzuki, 1985). In particular, these tasks require the model to have phonological skills i.e., understanding the patterns of speech units and rules governing pronunciation in language. While LLMs have been adopted to perform text-based tasks that require phonological skills (Ding et al., 2024; Kwon, 2023), it is unclear to what extent they acquire phonological skills by training on large-scale text data, without access to speech data.
Despite being trained solely on textual data, LLMs have been applied to many tasks that benefit from a deep understanding of phonology. These applications include poetry generation, song writing, machine translation, and language learning (Ding et al., 2024; Kwon, 2023; Yu et al., 2024). For example, the generation of poetry and lyrics leverages the models’ knowledge of rhyme and meter to produce rhythmically engaging content. Similarly, phonetic transcriptions from related Dravidian languages can enhance the accuracy of multilingual machine translations (Chakravarthi et al., 2019). We argue that due to their extensive training and further alignment, LLMs may leverage the learnt associations between written and spoken forms to accomplish these tasks. However, due to the lack of deep phonological understanding, they fail to perform phonological reasoning tasks in open-world scenarios (Peng et al., 2023). For example, although LLMs are popular for composing poetry, machine-generated poetry lacks diverse rhyming
1
schemes and is highly criticised by experts (Vice, 2023).
While earlier studies (Jiao et al., 2023; Sawicki et al., 2023; Peng et al., 2023), have evaluated LLM performance on poetry generation, machine translation and language learning. These studies were designed to analyse LLM performance on the downstream task rather than probe the model’s inherent understanding of phonology. To this end, we devise three diagnostic tasks that benchmark various LLMs on fine-grained tasks that explicitly apply phonological concepts in English. Our benchmark, PhonologyBench, comprises of three tasks - grapheme-to-phoneme (G2P) conversion, syllable counting and rhyme word generation, is shown in Figure 1. G2P is a widely adopted task for speech recognition applications that leverages pronunciation information to produce phonemes. Additionally, syllable counting and rhyme word generation are educational exercises used to enhance early language skills in children (Bruck and Genesee, 1995). Thus, PhonologyBench provides a systematic framework to analyse the phonological skills of LLMs that inadvertently affects their performance in real-world applications (writing assistants, translators) Figure 1.
We conduct a comprehensive study across 6 widely used LLMs, including three closed-source models - GPT-4 (OpenAI, 2023), Claude-3Sonnet (Anthropic, 2024), GPT-3.5-Turbo (OpenAI, 2022), and three open-source models - , Mistral-7B (Jiang et al., 2023) and Mixtral-8X7B (Jiang et al., 2024). Furthermore, we study the impact of word frequency, tokenization and sentence complexity across all three tasks. We find that overall LLMs exhibit surprisingly good performance on the three evaluated tasks while being trained solely on textual data. However, we observe a gap of 45% between human performance and LLM per-
formance on syllable counting whereas the gap is nearly 17% for rhyme word generation. We also note that, based on our evaluation, no single model consistently outperforms the others across all evaluated tasks. For example, GPT-4 is the best model in Rhyme Generation while Claude-3-Sonnet outperforms GPT-4 by a large margin in Syllable Counting. This highlights that models are good at different phonological tasks and researchers should choose LLMs based on their performance on the core task related to the downstream applications.
Our main contributions are:
1. We introduce PhonologyBench, a benchmark to evaluate the phonological awareness of LLMs on three diagnostic tasks in English - grapheme-to-phoneme conversion, syllable counting and rhyme word generation. PhonologyBench offers 4k datapoints along various axes to holistically evaluate LLMs.
2. We benchmark six LLMs and study the impact of word frequency, tokenization and sentence complexity on LLM performance.
3. Our findings suggest that LLMs perform suboptimally in comparison to human. In addtion, we show that no single model is consistently good at all the tasks and researchers should choose LLMs that perform well on the tasks that affect the related downstream application.

heading: 2 Related Work
text: Evaluation of LLM Linguistics. Hu and Levy (2023) assesses the efficacy of prompting as a way of probing the model’s metalinguistic ability, i.e., the ability to perform linguistic analyses given a natural language input. Contrary to our work, (Beguš et al., 2023) presents qualitative case studies on phonology, syntax and semantics of GPT-4 by generating theoretical analyses
of linguistic phenomenon on toy languages. Additionally, (Basmov et al., 2023) evaluates the performance of LLMs on linguistic inferences such as grammatically-specified entailments and monotonicity entailments using natural language inputs. More recently, (Peng et al., 2023) proposes a multichoice question answer dataset to evaluate the spoken language knowledge of LLMs. The proposed dataset comrpises of spoken language questions from an academic source (linguistics 101) whereas PhonologyBench comprises of tasks that closely align with real-world applications.
Grapheme-to-Phoneme. Many studies have been conducted on G2P conversion. Early works have proposed joint n-gram models (Galescu and Allen, 2002) and joint-sequence models (Bisani and Ney, 2008) for G2P conversion. Recent developments in G2P studies have shown LSTM (Toshniwal and Livescu, 2016) and Transformer (Yolchuyeva et al., 2019) to be powerful G2P models. These models have access to various pronunciation dictionaries and are trained to explicitly accomplish the grapheme-phoneme conversion for speech applications like Automatic Speech Recognition(ASR) and Text-To-Speech (TTS) (Masumura et al., 2020). Finally, (Park and Kim, 2019) combines the CMUdict 2 corpus with a neural-network model to convert graphemes to phonemes. However, previous works have not analysed the ability of LLMs as G2P models.
Syllable Counting. Syllable counting is used for testing the phonological awareness of children at an early age (Bruck and Genesee, 1995; Ukrainetz et al., 2011). It is also useful in second language acquisition and commonly used as an educational tool. Additionally, counting of syllables is crucial for composing songs, poems and haikus (Tian and Peng, 2022; Henderson, 1965; Suzuki, 1985). Several works have studied the syllable structure of haikus and poetry in English, haikus are particularly popular for they 5-7-5 syllable structure (Henderson, 1965; Suzuki, 1985). Recently, (Tian and Peng, 2022), (Tian et al., 2023) and (Ormazabal et al., 2022) have utilized syllable counts as decoding contraints or metre descriptors to generate formatted sonnets, lyrics and poems. (Sun et al., 2023) evaluates the ability of LLMs to generate syllable-controlled texts (e.g. - Complete a sentence in 5 syllables). In this work, we focus
2http://www.speech.cs.cmu.edu/cgi-bin/cmudict
on evaluating the ability of LLMs to recognise and count the number of syllables in a sentence.
Rhyming and Rhymes. Rhyming words are pivotal in early age evaluation of language development in children and popularly used as an education tool to teach languages (Bruck and Genesee, 1995). Additionally, rhyming words are critical components of creative writing tasks such as poetry and song writing (Caplan, 2014). Prior works in automatic poetry and sonnet generation rely on external rhyming dictionaries sch as CMUdict to induce rhyme during generation (Tian and Peng, 2022; Tian et al., 2023). Recently, several works have studied LLM-generated poetry and creative artifacts with a focus on rhyme schemes, style and creativity (Sawicki et al., 2023). However, previous works have not evaluated the ability of LLMs to generate corresponding rhyming words as a phonological skill assessment.

heading: 3 PhonologyBench
text: Here, we present our benchmark, PhonologyBench, that evaluates the English phonological skills of LLMs across three tasks: Grapheme-to-Phoneme Conversion (§3.1), Syllable Counting (§3.2), and Rhyme Word Generation (§3.3). An overview of these tasks, along with their significance in a variety of downstream applications, is provided in Figure 1. Starting with Grapheme-to-Phoneme Conversion, this task is pivotal for enhancing speechrelated applications, including Automatic Speech Recognition and Text-to-Speech systems, by ensuring accurate phonetic interpretations of text. Furthermore, the incorporation of phonemic and phonetic transcriptions into machine translation models introduces additional layers of understanding through phonetic embeddings, thereby improving translations (Liu et al., 2019; Chakravarthi et al., 2019). In addition, Syllable Counting and Rhyme Word Generation are traditionally popular as educational tools. Beyond facilitating language learning, they are essential in the creative processes involved in poetry and song composition, illustrating their versatility. Overall, we find that the tasks curated in PhonologyBench shed light on a wide range of real-world applications for LLMs, and benchmarking LLM performance on these tasks can advace research in LLM development for improved linguistic and creative generations. We provide the dataset statistics in Table 1 and outline the experimental methodology in § 3.1- 3.3.

heading: 3.1 Grapheme-to-Phoneme Generation
text: Task Description. The grapheme-to-phoneme (G2P) task involves converting the orthographic representation of a word into its phonemic representation. Writing systems often do not have a oneto-one mapping with spoken forms in English. For example, the element o in ‘olive(//)’ differs phonetically from the o in ‘rose(/o/)’. Therefore, inferring phonetic transcriptions solely from orthography is challenging. In this task, we prompt LLMs with a word to predict the correct phonemic form in International Phonetic Alphabet (IPA).
Dataset and Evaluation Metric. We sample 3,000 words and their corresponding pronunciations from the SIGMORPHON 2021 G2P task (Ashby et al., 2021) for American English to curate our dataset. Then, we probe the LLMs with the prompt (zero-shot) shown in Table 6 and report the accuracy for each model. Accuracy is the percentage of words whose predicted phoneme sequences were identical to the gold references.
Baseline. We report the performance of the G2P (Park and Kim, 2019) library on our dataset as the baseline. This library is a combination of a dictionary look-up and a neural network and serves as a reasonable baseline with access to pronunciation information.

heading: 3.2 Syllable Counting
text: Task Description. In syllable counting, individuals must identify the vowel peaks and the consonants that may precede or follow these peaks to determine the number of syllables in a word (Bruck and Genesee, 1995). Syllable counting is an educational tool used to improve a child’s phonemic awareness. Therefore, assessing the performance of LLMs can provide insights into the model’s phonemic awareness. Here, we prompt
LLMs with a sentence to count the number of syllables in the sentence.
Dataset and Evaluation Metric. Following (Sun et al., 2023), we curate 1,000 sentences from the datasets of Romance Books and Reddit Short Stories 3. After that, we employ a grapheme-tophoneme library to count the syllables in the sentences (Park and Kim, 2019). Finally, we probe LLMs with the prompt shown in Table 6 and report the accuracy for each model. Accuracy is the percentage of sentences whose predicted syllable counts are correct.
Baseline. We implement a naive approach by counting the number of vowels (a,e,i,o,u) in the sentence (text) as a baseline. Additionally, we ask an annotator with graduate-level engineering education and working proficiency in English (US) to perform the task and treat it as the human performance baseline.

heading: 3.3 Rhyme Generation
text: Task Description. In this task, we aim to analyse the LLMs’ capability to generate correct rhyming words for a given word. Rhyme words are a crucial aspect of creative writing and rely on pronunciation knowledge. Since, LLMs are used popularly as creative writing assitants, LLM performance on this task provides insights about creative generation.
Dataset and Evaluation Metric. We collect 300 words from the Spelling Bee Study List (Maguire, 2006) and the Google One Trillion corpus 4 to curate our dataset. We retrieve all the rhyming words (slant and strict rhymes) for a given word from an online rhyming dictionary, WordHippo5 and treat these as the gold reference. We obtain an average of 1200 rhyming words for a word to ensure a good coverage in our gold references. Similar to subsection 3.1, we categorized the dataset into two segments - high-frequency words and low-frequency words by using WIMBD (Elazar et al., 2024) index. Finally, we probe LLMs with the prompt shown in Table 6 and report the Success Rate (SR). We compute the word-specific success rate as the number of generated rhyming candidates that belong to the ground-truth set of rhyming words. SR is the average success rate for all the words.
3https://www.kaggle.com/datasets/trevordu/reddit-shortstories
4https://github.com/first20hours/ google-10000-english
5https://www.wordhippo.com/
Baseline. We present human performance as a baseline. Two human annotators with college-level education and native proficiency in English (US) performed the task. All human annotators were paid an $18 per hour and we spent $100 to acquire all annotations. We finetune a LLaMa-2-13B-Chat model (2 epochs, 2e-6 learning rate) on the common words and evaluate its performance on the rare words as a task-specific baseline.

heading: 4 Experimental Results
text: Here, we aim to benchmark the phonological skills of LLMs across the three tasks - G2P (§ 4.1), Syllable Counting (§ 4.2) and Rhyme Word Generation (§ 4.3). We further study the impact of word frequency in the pretraining dataset, tokenization strategy, and sentence complexity on LLM performance across all the tasks. Figure 7 shows human as well as model responses on the three tasks along with the gold references. We prompt all models in a zero-shot setting.

heading: 4.1 Grapheme-to-Phoneme
text: Overall Model Performance. Our results in Table 2 show that LLMs are worse than the phonologically trained baseline, G2P by ∼10% with Claude3-Sonnet and GPT-4 achieving the highest performance. We observe that the best performing opensource model (with <15B params), is ∼34% behind GPT-4. We provide qualitative examples in Table 5.
Frequently Used Words. Usually, words that are more commonly used in real-world scenarios are more prevalent in the training dataset. Highly frequent words may also afford LLMs greater opportunities to learn their correct pronunciations during training as shown in numerical reasoning tasks (Razeghi et al., 2022). To explore LLM performance on different frequency words, we categorized the dataset into two segments - highfrequency words and low-frequency words. We count the frequency of all the words in our dataset using the index in WIMBD (Elazar et al., 2024) 6. Our dataset comprises 1,000 high-frequency words with more than 10M occurrences and 2,000 lowfrequency words that occur less than 1M in pretraining corpora. Results in Table 2 indicate that models (including the baseline) generally have higher
6WIMBD returns the number of documents where a word occurs in the C4 (Raffel et al., 2023) dataset. We use this as a proxy for occurrence frequency.
performance on high-frequency words than on lowfrequency words. This could be attributed to wider availability of pronunciation information for highfrequency words in the training dataset as well as online dictionaries. However, our baseline model still outperforms the best LLM (Claude-3-Sonnet) on both high-frequency and low-frequency words highlighting the need for curating better phonologyrich datasets.
Tokenization in LLMs. Tokenization, refers to the division of input sequence of bytes into discrete tokens. LLMs primarily use Byte Pair Encoding (BPE) (Shibata et al., 1999) or its variants (Kudo and Richardson, 2018) for tokenization, leading to a generative process predominantly based on subword generation. We argue that this tokenization process can result in the loss of a word’s phonological structure, which in turn, may impede the phoneme generation process. To investigate this further, we divided high-frequency words into two categories: whole-word tokens and split-word tokens. We utilize the tiktoken library, which provides open access to the tokenizer used in OpenAI’s models, for word tokenization and subsequent data segmentation. Our analysis revealed that 30% of the high frequency words were tokenized as whole words by the OpenAI tokenizer. Results in Table 3 show that LLMs achieve higher accuracy with whole-word tokens compared to split-word tokens. We also observe that GPT-4 outperforms Claude3-Sonnet on split-words. This highlights that subword tokenization may lead to loss of phonological information that eventually affects the phonological skills of LLMs.

heading: 4.2 Syllable Counting
text: Overall Model Performance. We find that Claude-3-Sonnet achieves the best performance of 55% which is far behind the human baseline at 90%. All the closed-source models beat the vowel baseline. Open-source models like and Mistral-7B perform worse than the vowel baseline indicating that these models do not have an innate understanding of syllable structure and their relation with vowels. Surprisingly, GPT-4 falls behind Claude-3-Sonnet by ∼22% while being at-par in G2P task. We provide qualitative examples in Table 4.
Sentence Complexity and Sentence Length. A syntactically complex sentence is hard to comprehend due to the increased cognitive load required to grasp the syntax and semantics of the sentence
(Mikk, 2008). We posit that LLMs face similar difficulties in performing reasoning tasks like syllable counting over syntactically complex sentences. Therefore, we group our dataset into two categories : simple and complex sentences 7. We utilize the spacy library 8 to identify the number of clauses in a sentence. Our dataset comprises 74% simple sentences and 26% complex sentences. Our results are shown in Table 4. We observe that with increasing sentence complexity model performance drops significantly. Usually, syntactically complex sentences are positively correlated with sentence length, hence, we show the performance of LLMs on differing sentence lengths in Appendix A. This highlights that LLMs can leverage the statistical correlations they learned from orthography to accomplish simple tasks. However, they tend to degrade in performance when faced with complex task and increased ambiguity. This shows that
7We overload the term ’complex’ sentence to refer to sentences with more than one clause to simplify our analysis
8https://spacy.io/
LLMs are accomplishing phonology-rich tasks by utilizing side evidence from the training data instead of phonological concepts and reasoning as humans do. To our surprise, for Claude-3-Sonnet the performance is higher for complex sentences than simple sentences. This could be due to training data bias where models have seen many complex and longer sentences found in literature, academic papers, and professional communications (Elazar et al., 2024). Moreover, Claude-3 models have shown near perfect in long context evaluations indicating that they are good at processing longer inputs (Anthropic, 2024).

heading: 4.3 Rhyme Word Generation
text: Overall Model Performance. We find that GPT-4 achieves the best overall performance of 57.6%. In comparison, open-source models such as LLaMA and Mixtral-8X7B lag behind their closedsource counterparts, with Mixtral-8X7B securing the highest success rate among them at 27.9%. Overall, humans beat all the evaluated models by
Rhyme Generation Grapheme2Phoneme Syllable Counting PhonologyBench Tasks
0 5
10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 M od el P er fo rm an ce (% )
LLaMA-2-13B-chat Mistral-7B-Instruct-v1.0 Mixtral-8X7B-Instruct
GPT-3.5-Turbo GPT-4
Claude-3-Sonnet Baseline
Figure 2: Performance of the 6 evaluated models on PhonologyBench. We consider human performance on Rhyme Word Generation and Syllable Counting as the baseline. For G2P, we consider a dictionary look-up based approach as the baseline.
Model Overall Simple Complex Open-Sourced Models 6.9 7.5 5.1 Mistral-7B-Instruct 6.8 7.0 6.2 Mixtral-8X7B-Instruct 6.6 7.0 5.5 Closed-Sourced Models GPT-3.5-Turbo 19.6 20.2 14.0 GPT-4 23.3 24.2 15.3 Claude-3-Sonnet 55.3 53.4 65.0 Baseline Vowel Baseline 12.1 12.4 12.1 Human 90.0 93.3 86.7
Table 4: Results for syllable counting by LLMs. Here, we report the Accuracy as the percentage of correct syllable counts predicted by LLMs. Simple sentences contain only one clause while Complex sentences contain more than one clause.
17%. Intriguingly, supervised fine-tuning of a - Chat model does not yield any performance enhancements. This suggests that additional training focused on orthography does not contribute to improving the model’s capabilities in generating rhyme. We provide qualitative examples in Table 6.
Impact of Word Frequency Similar to § 3.1, we study LLM performance on words with varying frequency of usage. Our dataset comprises 200 common words with more than 10M appearances and 100 rare words that occur less than 10K in in the proxy pretraining dataset (C4). Rare words are less commonly used by English speakers and their pronunciations may not be widely known though available in large datasets. Results in Table 5 indi-
cate that LLMs exhibit higher accuracy in rhyme generation for common words compared to their rare counterparts. This could be due to less exposure to rare words and their pronunciations during training and highlights the importance of phonological information for improved LLM performance.

heading: 4.4 Discussion
text: In summary, our findings suggest that despite being trained on orthographic form LLMs perform surprisingly well on English phonological tasks. This could be attributed to the imperfect associations between text and speech learned by LLMs during to their training. Additionally, in Figure 2 we show that despite notable performance on different tasks, LLMs fall behind relevant base-
lines across all tasks. We find that the evaluated open-source models are consistently worse than the closed-source models. However, we also observe that no single model consistently outperforms the others in these tasks. For example, while outperforms Mistral-Instruct-7B on rhyme generation and G2P, it has the same performance on Syllable Counting. This indicates that higher performance in one task does not signify higher performance across all tasks, thus researchers should choose LLMs based on their performance on the core task for related downstream applications.

heading: 5 Conclusion
text: In this work, we present, PhonologyBench that consists of 4k data samples to evaluate the phonological skills of LLMs across G2P conversion, syllable counting and rhyme word generation in English. We show that LLMs fail to outperform human performance in syllable counting and rhyme word generation. Our findings thus highlight that there is scope of improving LLM performance on these core phonological task and encourages future research in training phonologically aware models. A straightforward approach to improving model performance across these phonological tasks is to add more phonological data during pre-training. Prior work by Liu et al. (2019) have shown the efficacy of joint textual and phonetic embedding in neural machine translation, thus, future work can focus on augmenting LLMs with phonetic representations. Overall, we show that no single model consistently performs well on all the tasks. Thus, we encourage researchers to develop downstream applications
that rely on these phonological tasks by carefully selecting LLMs that perform well on the particular task.

heading: 6 Limitations
text: Our work analyses LLM performance on various phonological tasks. The limitations of our work are two-fold, including concept coverage and data quality. While we present a comprehensive study on the various facets of LLM performance on G2P, syllable counting and rhyme word generation, our work only covers a limited portion of the variety of phonological phenomenon such as blending, homonyms and homographs. Additionally, our work focuses on American English and does not extend to various dialects. Future studies can focus on multilingual phonological phenomenon as well as LLM performance on different dialects. We encourage research that addresses these various languages, dialects and phenomenon to further the understanding of LLM language skills. Finally, our benchmark though extensive is limited by the quality of the curated gold references. For example, despite a large coverage of rhyming words we cannot be certain that our gold references are exhaustive. Similarly, LLMs can process syllable counts differently for clinical data or scientific communications that is not covered in PhonologyBench. Therefore, we encourage further contributions from the research community to help develop high-quality evaluation sets.

heading: 7 Acknowledgements
text: We thank Hritik Bansal, Xiao Liu and other UCLANLP/PLUS lab members for their constructive comments. We thank the anonymous reviewers for their helpful discussions and suggestions.

heading: A Syllable Counting at Varying Sentence Lengths
text: In this section, we study the impact of sentence length on LLMs on the syllable counting task. Figure 3 shows that overall models have higher accuracy for shorter sentence (length=6) than longer sentences(length=13). This highlights the relationship between sentence complexity and sentence length with similar trends as § 4.2.

heading: B Qualitative Examples
text: In this section, we present the qualitative examples to show the varied model responses on different

heading: Future Work
text: Future studies can focus on multilingual phonological phenomenon as well as LLM performance on different dialects. We encourage research that addresses these various languages, dialects and phenomenon to further the understanding of LLM language skills. Finally, our benchmark though extensive is limited by the quality of the curated gold references. For example, despite a large coverage of rhyming words we cannot be certain that our gold references are exhaustive. Similarly, LLMs can process syllable counts differently for clinical data or scientific communications that is not covered in PhonologyBench. Therefore, we encourage further contributions from the research community to help develop high-quality evaluation sets