PhonATe: Impact of Type-Written Phonological Features of African American Language on Generative Language Modeling Tasks

Authors: nicholas  deas
jessi  grieser
xinmeng  hou
shana  kleiner
tajh  martin
sreya  nandanampati
desmond  patton
kathleen  mckeown
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 06:47:21

Abstract: Current Large Language Models perform poorly on African American Language (AAL) texts in tasks like toxicity detection and sentiment analysis. AAL is underrepresented in both pre-training data and existing benchmarks for these tasks, hindering thorough evaluation and understanding of these biases. We introduce a novel approach to synthetically introduce type-written phonological features of AAL into text, a class of AAL features that has been overlooked in prior work. Our goal is to better understand how these features affect generative language models' performance on three tasks: toxicity detection, sentiment analysis, and masked span prediction. We find that fine-tuning with synthetic type-written phonological features lowers perceived biases on downstream tasks and our ablations reveal which features have particularly large negative impacts on model performance. Our results suggest that phonological features are vital to consider when designing bias mitigation techniques.

Main paper: 
 introduction: 
 Existing text resources used in the development and evaluation of Large Language Models (LLMs) lack sufficient representation of linguistic diversity (Dodge et al., 2021) . This presents a major obstacle to reliably evaluating and mitigating language model biases against minoritized language varieties like African American Language (AAL), the dialects of English primarily associated with the African American community in the United States (Grieser, 2022) . AAL, in particular, has minimal representation in both pre-training corpora for developing LLMs and finetuning data for tasks such as toxicity detection and sentiment analysis (e.g., see Xia et al. 2020) . In contrast, text resources overwhelmingly reflect White Mainstream English (WME) , the variety of English primarily used by White Americans and reinforced as the norm of English (Baker-Bell, 2020)2 . Reliable evaluation is vital to measuring and mitigating potential linguistic biases that can pose allocational (e.g., accessibility of LLMs) and representational (e.g., disproportionately labeling AAL as toxic) harms to historically marginalized language communities3 .To account for this representation disparity, recent works have proposed data augmentation techniques through morphosyntactic and lexical transformations (Tan et al., 2020; Ziems et al., 2022; 2023) . Other work has also traced biased predictions on tasks like toxicity detection to lexical features (Zhou et al., 2021) . While these approaches have contributed to understanding of language model biases, they do not fully capture one particular class of these lexical features, which we refer to as type-written phonological features.Type-written phonological features are spelling variants found in written text that reflect phonetic variation, such as writing goin in place of going. Past work has found that such features are primarily associated with texting, social media, and other digital contexts (Hillewaert, 2015; Cunningham, 2014; Ali et al., 2022) . Furthermore, Eisenstein (2013) finds significant correlations on Twitter between the use of type-written phonological features commonly associated with AAL and county-level demographic data.To improve understanding of models' behavior with AAL text found online and inform bias mitigation approaches, we focus on type-written phonological features of AAL and how these features impact model predictions. To do so, we introduce Phonological Augmentations for Text (PHONATE), employing models to convert between graphemes (characters in normal text) and phonemes (phonetic symbols) as well as a set of 10 rulebased phoneme transformations derived from sociolinguistic literature (Green, 2009; Bailey & Thomas, 2021; Thomas, 2007; Pollock et al., 1998) . We then use this approach to augment existing datasets for tasks with type-written phonological features of AAL and study these features' effect on generative language models' behavior in a controlled manner. Table 1 : Example toxicity and sentiment predictions of Flan-T5 on WME text and text with synthetic type-written phonological features. Synthetic type-written phonological features (bolded) of AAL using PHONATE lead the model to predict a non-toxic text as toxic and a neutral text as expressing negative sentiment.We emphasize that we develop this data augmentation approach strictly to evaluate and analyze model behavior and we do not advocate for developing models by training on synthetically generated AAL text (ethical implications are discussed further in section 6). We summarize our contributions as follows:1. We introduce a novel approach, PHONATE, to generate synthetic typewritten phonological features for use in evaluating language model performance on dialectal variation.2. We show that finetuning on text with PHONATE transformations reduces measured AAL biases across most models and tasks, suggesting that type-written phonological features are a vital consideration for bias mitigation approaches. 3. We find that particular phonological features can drastically alter model predictions on toxicity and sentiment tasks. 
phonate approach: 
 While some orthographic variants may be predictable given a specific phonological feature, such as the use of g-dropping in spelling going as goin, many other features have more complex effects on orthography, such as the reduced diphthong in spelling my as mah. Impacts on orthography may be complicated further by the lack of a one-to-one correspondence between phonemes and graphemes in English (Fry, 2004; Thorndike & Lorge, 1944; Hanna et al., 1966) . For example, the voicing distinction between the interdental fricatives (th) in thing (/θIN/) and the (/D@/) have different corresponding stops (/d/ and /t/ respectively) that may be realized in spelling as da and ting, capturing this distinction despite originally represented with the same grapheme.The PHONATE augmentation process involves transcribing each text into a phoneme sequence, applying sociolinguistics-informed phonological rules, and transcribing the augmented phoneme sequence back into a new grapheme sequence. To better control the frequency of each feature, we incorporate hyperparameters designating the probability that each feature is applied.Phoneme/Grapheme Models. To transcribe sentences between graphemes and phonemes, we employ an existing multilingual grapheme-to-phoneme (G2P) model and finetune a custom phoneme-to-grapheme (P2G) model on AAL data. The multilingual G2P model released by Zhu et al. (2022) transcribes text into IPA phonetic symbols. The model is based on byT5 (Xue et al., 2022) , which processes and generates individual bytes rather than relying on a tokenizer, making it more suitable for character-level tasks. Compared to a pronunciation dictionary, this approach can also predict phoneme sequences for previously unseen words such as those that may appear on social media but are unlikely to be included in existing dictionaries.To ensure that the P2G model used in PHONATE is able to generate words with type-written phonological features of AAL, we finetune a custom model. Phonological Rules. We use a set of 10 phoneme transformation rules5 derived from phonological features studied in sociolinguistics literature (Green, 2009; Bailey & Thomas, 2021; Thomas, 2007; Pollock et al., 1998) . Each phonological feature is encoded using phonemic regex patterns, deleting phonemes or replacing phonemes with the appropriate alternative based on descriptions in literature.For example, g-dropping is the orthographic feature 6 corresponding to the pronunciation of the velar nasal (e.g., the -ng in running) as an alveolar nasal (e.g., the -n in runnin) while monophthongization describes the simplification of diphthongs, which are two adjacent vowel sounds (e.g., the y in my) to a single vowel sound (e.g., the -a in ma). Based on these features, the corresponding regex-based transformations identify "η"s at word boundaries and replace them with "n"s to simulate g-dropping as well as remove the second vowel from sequences of two vowels to simulate monophthongization. In addition to gdropping and monophthongization, we create transformation rules for r-lessness, l-deletion, thsubstitutions, consonant cluster reduction, morpheme-final devoicing, str-backing, and stress-dropping. These are among the most common phonological features of AAL as well as among the most common to be realized as type-written phonological features. Descriptions of each phonological feature studied and the accompanying orthographic transformations are included in Appendix A and Appendix B respectively.Filtering. Because the transformations operate on phoneme sequences, the transformations may inadvertently alter the meaning of the sentence such as converting past tense verbs to present tense. For example, passed in the original text may undergo the transformation for Monophthongization and become pass. To avoid these transformations, we use the spaCy part-of-speech (POS) tagger to label all original and augmented texts, and revert any transformations that result in a distinct recognized term or a term with a different predicted part of speech tag. 
tasks: 
 Models are evaluated on three tasks: toxicity detection, sentiment analysis, and masked span prediction. Biases against AAL have been identified for both toxicity (e.g., Sap et al. 2019) and sentiment (e.g., Resende et al. 2024) classification tasks, but no studies to the authors' knowledge have examined these tasks in the context of AAL in generative language models. Additionally, prior work evaluating biases in generative language models have been limited to primarily intrinsic evaluations (Deas et al., 2023; Groenwold et al., 2020) , while our work also includes extrinsic evaluations of model performance. We evaluate models on a masked span prediction (MSP) task following these prior works on generative models as well. To finetune models, we use existing datasets primarily composed of WME and filter out potential AAL texts to form baselines with no additional exposure to AAL and to remove potentially biased annotations of AAL in the original datasets (Sap et al., 2022) 7 . We then augment these datasets using PHONATE to introduce synthetic type-written phonological features and compare against other data augmentations. To evaluate downstream performance, we use naturally occurring AAL to measure impacts on model predictions. All datasets used are summarized in Table 2 .Toxicity Detection. For toxicity detection experiments, DWMW17 (Davidson et al., 2017) and Jigsaw8 are used for finetuning all model variants. The datasets include samples of Tweets and Wikipedia comments respectively labelled for toxicity. We reduce the combined dataset to two labels, conserving "non-toxic" labels and collapsing all other to "toxic" (toxic, severely toxic, obscene, threat, insult, identity hate, offensive, hate speech, or abusive).Sentiment Analysis. For the sentiment analysis experiments, we use the sentiment analysis subset of the TweetEval corpus (Barbieri et al., 2020) to finetune models. The corpus includes Tweets labeled for a variety of tasks including emotion detection, stance detection, and sentiment analysis. The sentiment analysis subset includes labels for each Tweet as either negative, neutral, or positive sentiment.Masked Span Prediction. Finally, masked span prediction experiments use the BookCorpus dataset (Zhu et al., 2015) for finetuning. The BookCorpus includes short sentences from thousands of books by authors that were unpublished at the time.Evaluation. To evaluate finetuned models on genuine AAL text on social media, we use a corpus of Tweets reflecting AAL: the TwitterAAE corpus Blodgett et al. (2018) . Prior work has found that an extremely small percentage of randomly sampled Tweets are toxic (Founta et al., 2018) . Based on this and following Xia et al. (2020) , we also assume that all texts in TwitterAAE are non-toxic. In particular, we use the subset of TwitterAAE texts and the associated WME counterparts introduced in (Groenwold et al., 2020) .Augmentation Quality. To evaluate the quality of the rule-based phonological transformations, we conduct a human evaluation of augmented texts. We create the evaluation data by first sampling 60 WME texts from the data introduced in Groenwold et al. ( 2020) and then apply PHONATE to the WME counterparts to generate pairs of WME and corresponding texts with synthetic type-written phonological features of AAL. As baselines, we also collect judgments for the original AAL counterparts in the dataset to compare the quality of PHONATE transformations to naturally occurring AAL texts. We ask annotators to rate each text on two dimensions using 5-point Likert scales. Naturalness asks how likely the text would be written by a human on social media; and following Deas et al. (2023) , Meaning Preservation asks how well the augmented text conserves the meaning of the original text.Each annotator is asked to rate 50 texts such that 40 texts are shared between them. Ratings for these shared texts are averaged. Both annotators are self-identified AAL speakers studying linguistics.Toxicity. To evaluate the impact of different data variants on toxicity bias measures, we examine the False Positive Rate (FPR) on the AAL test set. Using the assumption that a small fraction of the TwitterAAE (Blodgett et al., 2018) dataset is toxic, we consider any text predicted to be toxic as a false positive (Founta et al., 2018; Xia et al., 2020) . In the experiments, FPR is calculated as FPR = # Toxic Predictions # Texts, and we consider lower FPR percentages to indicate less biased model toxicity predictions.Sentiment Detection. To evaluate how each data augmentation technique affects perceived sentiment biases, we again examine how often models trained on each dataset variant label a given text with negative sentiment. Unlike the toxicity task, sentiment expressed on social media is often mixed preventing the assumption that most texts express positive sentiment. Instead, we extract the pairs of AAL and WME text for which the model labels the WME text as positive sentiment and calculate the percentage of AAL texts in the subset that are labeled as negative sentiment. Within the set of positively labeled WME texts, the rate of negative predictions is calculated as # Negative AAL Predictions # Positive W ME Predictions . We consider lower percentages to illustrate less biased predictions as it suggests that models are less likely to label a positive sentiment text as negative due to the presence of AAL features.Masked Span Prediction. Finally, we evaluate models in a generation setting where models are tasked with predicting a word or phrase masked from the input as in Deas et al. (2023) . Performance on this task is measured with perplexity of the generated span to reflect how likely the original span is from the model's perspective, and top-k entropy to reflect the model's overall confidence in the predicted spans. Lower perplexities are considered to reflect less biased model behavior as lower scores suggest that the model gives higher probability to AAL texts.We evaluate the effect of type-written phonological features on four generative language models, expanding on prior work identifying AAL biases in generative language models (Deas et al., 2023; Groenwold et al., 2020) . T5 (large) is an encoder-decoder model with 770 million parameters and is pre-trained on the C4 dataset as well as several generation tasks such as summarization (Raffel et al., 2020) . With the same architecture, Flan-T5 (large) has 780 million parameters and is instruction-tuned on a larger set of tasks as well as chain-ofthought data (Chung et al., 2022) . To represent larger, open-source models, we evaluate Mistral-7B, a 7.3 Billion parameter decoder-only model (Jiang et al., 2023) . Because Mistral is a decoder-only model and is not pretrained on the same masked span prediction task as T5 and Flan-T5, we restrict its evaluation to the toxicity and sentiment tasks. Finally, we evaluate the robustness of ChatGPT, a commercially available, chat-based model trained on human preferences. Model checkpoints and prompts are included in Appendix D and details on finetuning and generation hyperparameters are included in Appendix E.First, we evaluate the effect of finetuning on synthetic data. Measuring the impact of synthetic finetuning on real AAL data performance can suggest whether better representation of type-written phonological features or knowledge of these features can aid in debiasing language models on these tasks. To establish a baseline, models are finetuned on the original WME data with no augmentations (Base) to reflect models with no additional exposure to AAL. To represent the impact of morphosyntactic features alone, a set of models are finetuned on the data augmented using VALUE (Ziems et al., 2022) . In this case, all features are applied to the text except the lexical features in order to isolate the impact of PHONATE on model predictions. Finally, a set of models are finetuned on data with morphosyntactic and random phonological augmentations to isolate the effect of systematic augmentations inspired by phonological features of AAL against random perturbations9 . Finetuning hyperparameters for T5, Flan-T5, and Mistral are included in Appendix F.We compare all baseline data variants to data augmented with both VALUE and PHONATE to reflect the added benefit of phonological features in finetuning data. Across all tasks, phonological transformations are performed where applicable with probability 20%. We stress that this does not accurately reflect the frequency or use of type-written phonological features online as our intent is not necessarily to produce realistic AAL text, but to study and evaluate models' behavior with these features. Examples of paired original texts and PHONATE-augmented texts are included in Table 1 .In a second set of experiments, we leverage PHONATE to investigate and better isolate the impact of each feature on model predictions. 5,000 texts with non-toxic labels for toxicity task and 5,000 texts with positive labels for the sentiment task are randomly sampled from the original dataset to analyze T5, Flan-T5, and Mistral. We examine 3 variants of the sample for each transformation: the original, VALUE-augmented, and PHONATE-augmented variants. Unlike in the finetuning experiments, each phonological feature is applied individually to the sample in order to isolate the effect of each feature10 . We examine WME-finetuned models predictions on each variant measuring incorrect predictions. Due to the large number of texts for each feature, ChatGPT is evaluated on a smaller sample of 500 candidate texts. 
phonate quality: 
 Table 4 shows the average ratings for the original AAL texts in (Groenwold et al., 2020) and WME texts augmented with PHONATE. Most PHONATE-augmented texts (61.9%) are rated 3 or higher for naturalness, suggesting that a majority of transformations reasonably reflect natural type-written phonological features. The Naturalness score for PHONATE falls below that of the human-written texts, but this is expected for synthetic augmentations, particularly as the judged PHONATE texts include type-written phonological features alone with no other features. Additionally, PHONATE receives a high rating for Meaning Preservation, suggesting that observed changes to model performance in the presence of PHONATE features are likely due to the orthographic changes rather than incidental meaning changes. Toxicity. Figure 1 presents the FPRs for models finetuned on each dataset variant. First, T5 and Flan-T5 finetuned on random augmentations (Random) increase the FPR over the WMEfinetuned baselines (Original), suggesting that random augmentations do not consistently improve model understanding or robustness to phonological features. Morphosyntactic augmentations (VALUE) also slightly raise the FPR for T5 and Mistral, but lower the FPR for Flan-T5, suggesting that morphosyntactic features alone do not consistently aid in toxicity detection tasks. Both morphosyntactic and phonological features (+PHONATE), however, lower the FPR for all models, and lower the FPR the most for T5 and Flan-T5. This supports our hypothesis that phonological features are significant in toxicity detection models, as the use of systematic phonological features rather than random or morphosyntactic augmentations improves the FPR most.Sentiment Detection. Figure 1 also includes the rate of negative sentiment labels on AAL texts where the WME counterparts are labeled with positive sentiment across finetuning dataset variants. As with the toxicity task, random augmentation (Random) raises the rate of negative sentiment labels for T5 and Flan-T5 compared to the WME-finetuned baseline (Original), while morphosyntactic augmentations (VALUE) alone slightly raise the rate for T5 and slightly lower the rate for Flan-T5 and Mistral. Also similar, finetuning with the combination of morphosyntactic and phonological augmentations (+PHONATE) results in the lowest rate of negative sentiment predictions across models, suggesting again that representation of phonological features is significant for sentiment analysis models predicting on AAL text as well. Masked Span Prediction. Unlike the toxicity and sentiment tasks, the results for the masked span prediction task present less consistent trends, shown in Figure 2 (full results included in Appendix G). In this case, for T5, the WME dataset (Original) and morphosyntax-augmented dataset (VALUE) result in the lowest perplexity scores. At the same time, the same variants of Flan-T5 yield the highest perplexity scores and the lowest entropy scores, suggesting that they confidently predict incorrect spans on the AAL data. Despite not having the lowest perplexity for T5, the combination of morphosyntactic and phonological augmentations (+PHONATE) does yield the lowest perplexity for Flan-T5, consistent with the toxicity and sentiment tasks. Additionally, the addition of phonological transformations still exhibits a lower perplexity than random augmentations (Random) in both cases.Toxicity. Figure 3 presents the FPRs when phonological features are applied in isolation alongside the average FPRs on the original and VALUE-augmented data (FPRs on original and VALUE-augmented data within each subset are shown in Appendix H). Overall, while certain features lead to minor changes in performance, others result in notably higher FPRs.In particular, consonant cluster reduction and final devoicing cause large increases in FPR for T5, Flan-T5, and Mistral, as well as monophthongization for T5 and Flan-T5. Interestingly, the most and least impactful features are highly similar between T5 and Flan-T5, possibly because instruction finetuning includes little or no AAL to impact performance. The most impactful features for ChatGPT are largely different from the other models, with predictions primarily impacted by features such as th-substitutions, r-lessness, and l-lessness. This may signal differences in the distribution of AAL features among models' pre-training corpora. Sentiment. A similar analysis for the sentiment task is shown in Figure 4 , presenting the rate of Neutral or Negative sentiment labels for each phonological transformation alongside original and VALUE-augmented data variants. In contrast to the toxicity task, all phonological transformations cause notable increases in the rate of negative or neutral labels for T5 and Flan-T5. Again, the order of the most impactful features is largely similar between the two models, though the order largely differs from that of Mistral, ChatGPT, and the results in the toxicity analysis.Between both the toxicity and sentiment robustness results, consonant cluster reduction, final devoicing, and monophthongization appear to be responsible for some of the largest increases in the rate of toxic or neutral/negative sentiment predictions for T5, Flan-T5, and Mistral. ChatGPT, however, appears to be more consistently impacted by stress dropping and th-substitutions in both tasks. 
related works: 
 AAL Bias in NLP Tasks. Models and datasets across a variety of tasks have been shown to be biased or less performant with AAL texts. One of the most studied settings for AAL biases is toxicity detection (Sap et al., 2019; 2022; Chuang et al., 2021; Xia et al., 2020) .Particularly related to the impact of phonological features and lexical variants, Zhou et al. (2021) examines how lexical items and dialect markers of AAL may trigger toxicity systems to falsely label a text as toxic. In addition to toxicity detection, Resende et al. ( 2024) also identifies how sentiment analysis systems are disproportionately likely to label African American English expressions with negative sentiment. Beyond these tasks, performance disparities have also been uncovered in Automatic Speech Recognition systems (Martin & Tang, 2020; Koenecke et al., 2020; Mengesha et al., 2021) , language identification systems (Blodgett et al., 2016; Blodgett & O'Connor, 2017) , and models for dependency-parsing and POS-tagging (Blodgett et al., 2018; Jørgensen et al., 2016) . Biases in generative language models, however, have received little attention and been constrained to intrinsic evaluations (Deas et al., 2023; Groenwold et al., 2020) . In contrast, we include extrinsic evaluations of models on toxicity detection and sentiment analysis tasks while also giving focus to type-written phonological features.Linguistic Data Augmentation. Several prior works have used data augmentation for both evaluating and mitigating societal biases in language models. Maudslay et al. ( 2019) augments data with counterfactuals and name substitutions to mitigate gender biases, while Qian et al. (2022) uses neural perturbations to augment data along multiple demographic axes. Similar approaches have been applied to linguistic biases as well, using augmentations to approximate linguistic features of a language variety or dialect. Morpheus (Tan et al., 2020) augments word inflections adversarial during training to make models more robust to L2 English and AAL speakers. Alternatively, to better reflect the targeted language variety, Ziems et al. (2022) and Ziems et al. (2023) use surface-level transformations to synthetically introduce morphosyntactic and lexical features of a variety of dialects. In contrast to these approaches, we study linguistic biases against AAL focusing on synthetically introducing type-written phonological features without relying on limited dictionaries. 
conclusion: 
 We introduce a novel approach, PHONATE, to augment WME texts with synthetic typewritten phonological features of AAL. We use PHONATE to conduct two sets of experiments evaluating the significance of type-written phonological features on generative language models in three tasks. First, we finetune models for toxicity, sentiment, and masked span prediction using datasets with different sets of augmentations. We find that finetuning with both morphosyntactic and phonological features typically results in the best or least biased performance on toxicity, sentiment, and masked span prediction tasks, outperforming finetuning with morphosyntactic features alone and random phonological augmentations. Additionally, we evaluate how often morphosyntactic and individual phonological features alter model predictions on the toxicity and sentiment tasks, finding that particular phonological features have large impacts on model predictions. 
 
 Appendix: We recognize several limitations accompanying the results of our experiments. First, the augmentations applied by PHONATE are not exhaustive and do not perfectly capture the natural use of type-written phonological features. The aim in using the PHONATE approach is to improve the recall of type-written phonological features by avoiding reliance on a finite list of features, and we combat potential inaccuracies by filtering transformations with POS-tagging. Additionally, both PHONATE and dictionary approaches to incorporating type-written phonological features are limited in that they do not consider larger context in applying transformations (e.g., some features depend on the preceding or following word).As there are few studies of the use of type-written phonological features, we leave improved augmentations and study of the differences between spoken and written phonological features to future work.Additionally, we limit our evaluation to generative models on three tasks-toxicity, sentiment, and masked span prediction-which may not reflect the same patterns as with other tasks. At the same time, in the masked span prediction experiments, we are unable to reliably mask only AAL features when evaluating models, leading the scores to also reflect performance on spans without features. We examine these tasks based on prior studies that have uncovered biases in both encoder-only and generative language models as well as to include both an extrinsic and intrinsic evaluation of the included models. We leave further analysis, evaluation of classification model architectures, and evaluation on other tasks to future work.We recognize that in developing an approach to simulate features of African American Language, it could be used to finetune a model to generate AAL and mimic AAL speakers online. To avoid this in our own experiments, we refrain from finetuning any model to generate AAL using the synthetic augmentations: in the toxicity and sentiment experiments, only the inputs are modified, and in the masked span prediction task, we avoid masking augmented tokens. Additionally, we emphasize that finetuning on data augmented to introduce synthetic features of AAL is not intended as an approach to mitigate, and our primary aim is to show that phonological features in online texts have a notable impact on model performance. We are committed to openness and transparency in our research on the impact of AAL features on LLM performance and our ultimate aim is the development of core mitigation strategies that can address problems in current practice. Given the historical linguistic subordination of AAL that derives from prejudices against it speakers, we feel it is imperative to center AAL speakers in decision-making processes, and thus, we recruit AAE-speakers to validate our approach. The research must come out of Black communities in order to develop best practices around the use of AAL data and the development of applications with social impact that must be made available to all communities.Beyond morphosyntactic and lexical features, AAL also exhibits phonological and phonetic features that are realized as variation in the pronunciation of consonants and vowels in particular contexts (Thomas, 2007) . Frequent features of AAL phonology considered in this work are detailed below, compiled from linguistic studies of AAL phonology (Green, 2009; Bailey & Thomas, 2021; Thomas, 2007; Pollock et al., 1998) .r-lessness, non-rhoticity, or r-deletion describes cases where the syllable and often word-final /r/ may not be pronounced or vocalized as /w/. In non-rhotic dialects, a word such as teacher (/tiÙ@/) may be pronounced as teacha (/tiÙ@r/), or store (/stOr/) as sto (/stOw/).l-deletion similarly involves deletion of /l/ sounds, in word-final positions or preceding labial consonants. The /l/ may be deleted entirely, or vocalized as /o/, /w/, or /@/. For example, cool (/kul/) may be pronounced as coo (/ku/).g-dropping occurs when the word-final -ing (-/Iη/) may be pronounced as -in (/In/) in non-monosyllabic words. For example, running (/r2nIη/) may be pronounced as runnin (/r2nIn/)th-fronting alters the pronunciation of /θ/ (voiceless in bath) and /δ/ (voiced in that), with differing realizations depending on the position in the word and voicing. /θ/ may be pronounced as /t/ in word-initial positions or /f/ in word-final positions while /δ/ may be pronounced as /d/ in word-initial positions or /v/ in word-final positions. For example, bath (/baθ/) may be pronounced as baf (/baf/) while that (/δat/) may be pronounced dat (/dat/).Consonant cluster reduction simplifies clusters of two or more adjacent consonants in wordfinal positions. In most cases, the final consonant sound is deleted, such as how first (/f3rst/) may be pronounced firs (/f3rs/).Final devoicing replaces morpheme-final voiced consonants with their voiceless counterparts. For example, kids (/kIds/) may be pronounced as kits (/kIts/).A feature unique to AAL among North American dialects is str-backing, where /str/ in word-initial positions may be pronounced /skr/. For example, street (/strit/) may be pronounced skreet (/skrit/).Beyond consonantal features, vowels may be altered through monophthongization or diphthong simplification, where diphthongs, two successive vowel sounds, are simplified to a monophthong, a single vowel sound. For example, my (/mai/) may be pronounced as mah (/ma/).Finally, stress-dropping describes how word-initial, unstressed syllables may be deleted. A common example is realized when about (/@'boUt/) is pronounced 'bout (/boUt/).Table 5 shows each feature with a description of the associated regex transformation. We employ this approach rather than an end-to-end approach to ensure adherence to the features as identified by sociolinguistics studies. This approach also better ensures that the meaning of the original text is conserved, given that prior work finds that style transfer approaches often fail to conserve meaning (Ziems et al., 2022) .While we use an existing Grapheme-to-Phoneme model for transcribing text into phoneme sequences, PHONATE requires a Phoneme-to-Grapheme model capable of producing terms reflecting type-written phonological features of AAL. To do this, we create a dataset consisting of 1) the CMU English Pronouncing Dictionary11 with added syllable markers12 , and 2) predicted pronunciations of AAL text on social media from Blodgett et al. (2018) .We use the multilingual Grapheme-to-Phoneme model Zhu et al. (2022) to generate predicted pronunciations of all terms in the TwitterAAE corpus. We filter out terms containing 3 or more sequential repeated characters (e.g., yesssss) or non-Latin characters before generating the paired AAL terms and predicted pronunciations. For T5, Flan-T5, and Mistral, we use the google-t5/t5-large, google/flan-t5, and mistralai/Mistral-7B-v0.1 checkpoints respectively made available through Huggingface. For querying Chat-GPT, we use the gpt-3.5-turbo-1106 checkpoint for reproducibility and a temperature of 1 in generations.Prompts used for each task and model are shown in Table 6 . All prompts are drawn from (Si et al., 2023) and (Zhang et al., 2024) . Mistral prompts are appended with an end token ("⟨/s⟩").For toxicity and sentiment tasks, models are restricted to generating 1 and 3 tokens as expected outputs are ("Yes", "No") and ("Negative", "Neutral", "Positive") respectively. For the masked span prediction task, models can generate up to 64 tokens. All generation is performed with greedy decoding.For reproducibility, we query ChatGPT with temperature 0 and seed 1. Only user messages are provided through the API with no system message.Figure 5 and Figure 6 present the robustness results including the raw scores on the original and VALUE-augmented texts for each subset. As the percentages for each feature are calculated based on the subset of texts where a given feature may be applied, there is slight variance in the scores on the original and VALUE-augmented texts. Our code is available at https://github.com/NickDeas/PhonATeWe acknowledge the different terms used to refer to these varieties and follow those used by Baker-Bell (2020) andAlim & Smitherman (2012).We followBlodgett et al. (2020) in identifying harms. Allocational harms refer to the unfair distribution of resources among social groups, while representational harms refer to the inaccurate representation or unfair treatment of social groups. The harms we identify follow those ofDeas et al. (2023), particularly in applications with high social impact.https://github.com/open-dict-data/ipa-dictEach feature may be represented by multiple transformations depending on context, such as th-substitutions which depend on the location in the word as well as the original voicing.While the phonetic feature is not referred to in this way, we use g-dropping to refer to the orthographic realization.Following(Xia et al., 2020), we use the classifier introduced in Blodgett et al. (2016) to remove all texts for which the predicted probability of representing AAL is higher than 0.8.https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/ dataVowels are only replaced with other vowels, and consonants only with other consonants. For a fair comparison, we restrict random augmentations such that the same counts of insertions, substitutions, and deletions align with the transformations applied using PHONATE for each text.Samples are subset to only the texts where it is possible to synthetically introduce each feature.http://www.speech.cs.cmu.edu/cgi-bin/cmudicthttps://github.com/open-dict-data/ipa-dict 


Decision: This work aims to improve known issues of biases based on African American Language text in areas such as sentiment and toxicity (they also include masked span prediction) via synthetic data augmentation that transforms SAE text into AAL text. 

The reviewers bring up a number of concerns around presentation and clarity, that I agree with -- the graphs could be easier to parse, arguably some of the tables should be graphs, e.g. Fig 1 is a table despite being called a figure and in all cases it should be more clearly indicated what the metrics are and whether higher or lower is better.
There are insufficient methodological details regarding fine tuning, which harms transparency and replicability, please add those at least to the appendix.

However, these are quite minor changes (though important) and can easily be done before publication in another revision, which I recommend that the authors do. none of these complaints are about soundness of scientific rigor.

Given that the work is sound, I believe there is real value in this approach to the research community, which has struggled with what to do about dialects beyond SAE for some time and has really done very little about it. Synthetic data augmentation is a commonly used approach in LLM post training, and this is an interesting linguistically motivated one that is impelementable and works quite well at addressing a real problem. 

The only remaining concerns about publication from the reviews are wanting them to have used models larger than Mistral and Chat GPT, which is frankly a ridiculous complaint, everyone does experiments on smaller models and they are of sufficient size to generalise. And a few concerns about validity of using sentiment with no ground truth labels (which the authors address in a rebuttal) and the engagement with the AAL community, which they also address. So I am discounting these concerns as solved.

Reviews: review: paper_summary: paper_summary: The study introduces Phonological Augmentations for Text (PhonATe), a method that converts between graphemes and phonemes, as well as rule-based phoneme transformations derived from sociolinguistic literature. By augmenting datasets with AAL phonological features, the researchers aim to understand how these features influence model predictions in tasks such as toxicity detection and sentiment analysis. The study also discusses the potential implications of incorporating AAL features into language models for bias mitigation and highlights the importance of developing more inclusive and accurate models for diverse linguistic contexts.
strength_weakness: strength_weakness: While the paper states that fine-tuning on data augmented with synthetic AAL features is not intended as an approach to mitigate biases , there may be ambiguity in how the fine-tuning process was conducted and its implications on model behavior. Clearer explanations or guidelines on the fine-tuning methodology and its impact on model performance are necessary for transparency and reproducibility.

The paper emphasizes the importance of centering AAL speakers in decision-making processes . However, there is a lack of clarity on how the research engages with or involves the AAL community in the development and evaluation of the PhonATe approach. Involving relevant communities in the research process can provide valuable insights and ensure that the study is culturally sensitive and beneficial to those it aims to represent.

Previous research has highlighted how annotator beliefs and identities can bias toxic language detection tasks . Considering the sensitive nature of the tasks evaluated in the study (toxicity detection, sentiment analysis), there should be a thorough discussion on how potential biases in annotation or data collection were addressed to ensure the reliability and fairness of the results.
paper_summary: paper_summary: The authors study if introducing synthetic data mimicking African-American English dialects into model training can narrow the gap in performance on AAE evaluation sets. They first develop a framework to transform text into AAE based on typed phonological features of AAE derived from sociolinguistics literature. They then present results showing that finetuning on this synthetic language lowers false positive rates in toxicity detection with AAE and leads to fewer model predictions of negative sentiment on AAE text. They also conduct a masked span prediction experiment where they find mixed results in terms of perplexity after synthetic AAE finetuning, though it can lower perplexity in some cases.
strength_weakness: strength_weakness: - The writing is unclear in many places and should be revised before publication. For example, in the section on augmentation quality it states “both annotators are self-identified AAL speakers,” but it isn't stated explicitly that there are two annotators judging each example and how the final rating is derived. The description of the data subsets used on pg 6 ("5000 non-toxic texts and 5000 texts for the sentiment dataset") is also confusing. 

- The sentiment detection eval seems very odd, given that there is no ground-truth for the sentiment. Presumably if an example is labeled positive in WME, it shouldn't become negative just from a transformation into AAE, however this experiment could be conducted in a cleaner way to make it clear the outcome is a result of bias. 

-   The paper never really motivates the use of synthetic AAE instead of finetuning on a larger corpus of real AAE
questions: questions: 1. What was the error rate in the phonological transformations?

2. Why do you think finetuning works for lowering perplexity with Flan-T5 and not T5?

3. Figures 2/3 are hard to decipher
paper_summary: paper_summary: The paper presents an approach to enhance the performance of LLMs on texts with African American Language (AAL) features particularly focusing on type-written phonological features. They introduce an approach of generating synthetic typewritten phonological features that generates text with phonological transformations from a standard text. The paper investigates how injecting this synthetic AAL text into training datasets impacts the performance of LLMs on tasks such as toxicity detection, sentiment analysis, and masked span prediction.
strength_weakness: strength_weakness: Perhaps the authors should have explained better why these phonological features make their way to text, no where in the paper transcription is mentioned, so I assume (from the cited papers) most of the time people write these phonologically transformed text on social media.
I understand that the authors have resource constraints but the two main LLMs used in the experiments are quite old and even Mistral is considered a small model. One cannot generalize these results and assume larger language models would behave similarly.  
Apart from that, I really think this paper fits better in a conference like *CL as the linguistic content is quite rich but the LLM part is quite limited.

Comments: title: Rebuttal Response
comment: Thank you the authors for responding to my concerns and questions. I look forward to the revised version and my review is unchanged.