Do transformer models do phonology like a linguist?
Year: 2023

Abstract:
Neural sequence-to-sequence models have been very successful at tasks in phonology and morphology that seemingly require a capacity for intricate linguistic generalisations. In this paper, we perform a detailed breakdown of the power of such models to capture various phonological generalisations and to benefit from exposure to one phonological rule to infer the behaviour of another similar rule. We present two types of experiments, one of which establishes the efficacy of the transformer model on 29 different processes. The second experiment type follows a priming and held-out case split where our model is exposed to two (or more) phenomena; one which is used as a primer to make the model aware of a linguistic category (e.g. voiceless stops) and a second one which contains a rule with a withheld case that the model is expected to infer (e.g. word-final devoicing with a missing training example such as b→p). Our results show that the transformer model can successfully model all 29 phonological phenomena considered, regardless of perceived process difficulty. We also show that the model can generalise linguistic categories and structures, such as vowels and syllables, through priming processes.

heading: 1 Introduction
text: In computational linguistics, neural networks have occupied much of recent work. One prime driver is adaptability to multiple facets of linguistic phenomena. As an example, sequence-to-sequence models have been shown to capture inflection patterns across numerous languages (Kodner et al., 2022). While their performance represents significant advances, the abstractions generated during the modelling process warrant further investigation. We experiment with phonological processes on a constructed language to compare the generalisations learned by transformer models with widespread linguistic phenomena.
In particular, we address the following questions:
• Learning specific phonological processes (are some more difficult than others?)
• Categorisation (can the model generalise a category, vowels, consonants, specific consonant groups, e.g. plosives?)
• Is word structure (syllables) implicitly learned?
We establish that the transformer model successfully models all 29 phonological phenomena we consider, regardless of linguistic complexity. Our results show that the model can generalise to linguistic categories with some caveats. By examining the transformer model’s generalisation of haplology, we show that the model appears to learn syllables; the model can recognise the difference between VC and CV and generate previously unseen CV sequences.

heading: 2 Related Work
text: Investigating the cognitive reality of linguistic categories defined within phonology has long been of interest to linguistics. Does the natural class of phonemes bear any significance to a cognitive reality? For example, a series of experiments (Finley and Badecker, 2009; Chambers et al., 2010; Skoruppa and Peperkamp, 2011) examine the natural class of vowels and whether phonological patterns can be extended to previously unseen vowels. The studies suggest that participants were mostly able to generalise. In a similar vein, Finley (2011) presents a study on consonant harmony. The results suggest that learners (human learners) can generalise to novel consonants when the phonological pattern is general. However, the learners failed to generalise when the rule triggering the consonant harmony pattern was highly specific.
We adapt this long-standing linguistic question to ask whether Transformer-based abstractions are
8529
linguistically informed. Our experiment setup swaps the human learner with the Transformer architecture. Previous studies investigating phonological phenomena with Transformers include Elsner (2021), where Transformers can handle reduplication and gemination. To an extent,1 the SIGMORPHON shared tasks (Kodner et al., 2022) also demonstrate the capacity of Transformers to represent phonological processes through capturing allomorphs conditioned by phonological environments.
There have been extensive studies on various phonological processes and RNNs. Haley and Wilson (2021) shows that encoder-decoder networks (specifically LSTM and GRU architectures) can learn infixation and reduplication. Mirea and Bicknell (2019) explores whether phonological distinctive feature information is required for learning word-level phonotactic generalisations using LSTMs. The authors find that information about phonological features hinders model performance, and phonotactic patterns are learnable from the distributional characteristics of each segment alone. Moreover, distributional information proves to be integral in recovering phonological categories (Mayer, 2020).
Another way to investigate neural architecture abstractions is to probe the model internally. Silfverberg et al. (2021) examines whether RNN states encode phonological alternations through experiments on Finnish consonant gradation. The authors show that the models often encode consonant gradation in a select number of dimensions. Rodd (1997) probes the hidden states of an RNN model which controls Turkish vowel harmony. Similarly, Silfverberg et al. (2018) establish a correlation between embedding representations and distinctive phonological features for Finnish, Spanish and Turkish. This paper focuses on a model-external interrogation of Transformer generalisations by studying the predictions produced.

heading: 3 Language Design
text: The phonological phenomena in question are tested on a constructed language. The primary motivation for this is to allow for a controlled experiment and ensure that we can generate enough samples of the required phonological environments for rules to be triggered and thus observed. With this in
1This largely depends on the language considered and the phonological processes it exhibits.
mind, we require the constructed language to be as representative as possible of natural language. Therefore, key features were chosen based on the condition of being the most typologically common ones (Maddieson, 1984; Ladefoged and Maddieson, 1996; Maddieson, 2013). The main characteristics are listed in Table. 1.
Generating a lexicon The most complex syllable structure possible in the language is CCVVCC and the simplest one is V. Since our language design aims to generate a synthetic lexicon, we also control for word length distribution. Previous works have shown that word length over word types exhibits a roughly Gaussian distribution with a mean in the range [7, 10], depending on the language (Smith, 2012). We have chosen a mean word length of 8.
An additional constraint when generating a lexicon is the sonority sequencing principle (SSP) (Selkirk, 1984; Clements, 1990). Syllable structures tend to be highly influenced by the sonority scale, with the general rule that more sonorous elements are internal (i.e., close to the nucleus) and less sonorous elements are closer to the syllable edge. Therefore, we use a sonority metric to avoid generating implausible consonant clusters, with the onset and coda requiring opposite values on the metric, i.e. increasing sonority in the onset and decreasing in the coda.
4 Data2
Our data preparation follows three steps: lexicon generation, triplet (lemma, tag, surface form) formation via the finite-state tool foma (Hulden, 2009) and, finally, sampling of these triplets ac-
2All data and code is available at https://github.com/ smuradoglu/phon-proc
cording to the experiment at hand and formatting for Fairseq.(Ott et al., 2019)3
We train the model as a standard ‘inflection’ task (Kodner et al., 2022), but with tags being identifiers of the processes that are to be triggered instead of morphosyntactic information. For example, the input sequence moupi#GEMINATION would be paired with the output mouppi. More example triplets are shown in Table 2.4
Lexicon generation entails generating viable syllable structures and filling these abstract structures using vowel and consonant inventories. The syllables are concatenated n times, where n is an integer between 1 and 10. We sample from this uniform distribution to produce a Gaussian distribution for word length with a mean of 8 symbols.
We include a COPY tag, where the input is copied to the output, to negate any performance drop by the model when unseen lemmata are encountered (Liu and Hulden, 2022). In other words, the model, at test time, will never encounter a completely unseen lemma on which to perform a phonological change, since it will always have witnessed at least an input-output pair of any lemma used that is simply copied to the output.
3See B for model details. 4Our nomenclature of sound changes follows Campbell
(2013).

heading: 5 Modelling common phonological processes with varying degrees of complexity
text: In this experiment, we establish that seq2seq models can successfully capture a range of phonological processes, including more complex rules such as metathesis. As seen in Figure 1, the transformer model performs reasonably well across all phonological phenomena, with little distinction between the complexity of the process considered.

heading: 6 Linguistic Category generalisation
text: We examine whether the transformer model can generalise linguistic categories such as vowels or syllables from examples of alternations. During training, we expose the model to two phenomena at once (priming/held-out cases) of processes where the model could potentially infer relevant
categories and extend this knowledge to withheld cases. The first set of experiments focuses on the generalisation of vowels, and the second centres on categorising consonants.

heading: 6.1 Vowel Experiments

heading: 6.1.1 Apocope/Aphaeresis
text: In this experiment, Aphaeresis–deleting wordinitial vowels—is the priming process and Apocope—deleting word-final vowels—is the heldout case. The training set consists of aphaeresis cases with all five vowels. In other words, lexicon beginning with a,e,i,o,u are included. Apocope examples exclude cases where u occurs word-finally. The u-final words with the Apocope tag are present only at test time. Table. 3 summarizes the results. From these results, it is clear that the model extends the Apocope rule to the unseen u-vowel. There are only 8 instances within the 10 errors where ‘u’ is not deleted. The remaining 2 errors are other modelling errors (such as repeating characters): outputting sou instead of the gold so with input sou.

heading: 6.1.2 Vowel shortening/lengthening
text: Following a similar setup to the Apocope/Aphaeresis experiment, the vowel shortening (priming) and lengthening (withheld case u) case involves training a model with all vowel cases for shortening, and all vowels except u for the vowel lengthening process. The results show a 100% accuracy for the previously unseen u-cases for vowel lengthening. The two errors observed are from other categories (i.e., vowel shortening and non-u lengthening).

heading: 6.2 Consonant Experiments

heading: 6.2.1 Gemination/Degemination
text: This experiment involves training a model for Degemination (priming) and Gemination (withheld case p) processes. The results show that the transformer model has successfully extended the consonant category to include the unseen p. Out of the 453 test cases, only 12 were incorrect p cases, with the remaining five non-target errors. Incorrectly predicted instances follow the pattern of outputting lup with input lup instead of the gold lupp.

heading: 6.2.2 Devoicing/Intervocalic voicing
text: This experiment involves final stop Devoicing (priming) and Intervocalic Voicing (with-held-case
p). The training set is comprised of all word-final devoicing cases (b>p,d>t,g>k) and all intervocalic cases except the p case (where p>b).
The results show that p is transformed to a b 77.6% of the instances. Where the conversion does not take place, errors typically follow the pattern of, e.g. outputting epeiSe instead of ebeiSe with the input epeiSe
To investigate the comparatively low performance. We compare word-initial devoicing with word-initial voicing as a priming process. The results are summarised in Table. 4. The accuracy of the predictions for the unseen p was substantially lower in the case of word-initial voicing (40%) compared with the word-initial devoicing (74.8%). Interestingly, word-initial voicing
involves the same process as intervocalic voicing (p>b), with only different environments triggering the process.

heading: 7 Word-internal representations
text: To test whether seq2seq models can learn a representation of word-internal structures, such as syllables, we experiment with examples of haplology. Haplology (tatasa > tasa) is the process in which a repeated sequence of sounds is simplified to a single occurrence. For example, if the word haplology were to undergo haplology, it would reduce the sequence lolo to lo, haplology > haplogy.
In this experiment, we include two additional processes so the model can witness the contrast between vowels and consonants separately: (1) wordfinal vowel deletion and (2) word-final consonant deletion.
To test the generalisation capacity of the model, at test time, we include the following withheld cases: unseen CVCV structures—i.e. cases where haplology should apply, but the specific CVCVsequence is never seen in the training data; words where haplology occurs more than once; and VCVC structures to see if the model (erroneously) learns to delete any repeating sequence of symbols. In our experiment, we withhold from the training set the following CVCV-sequences: dede, fofo, kuku, wowo, baba, vivi, papa, titi, soso, momo, nene, rere, lili, SuSu, jiji, ÙuÙu, NaNa, gugu.
Note that haplology includes both cases where haplology applies and does not since the input word
may or may not contain a CVCV-sequence where the two CVs are identical.
Table 7 summarises the results obtained. The model shows high accuracy for the supplementary word-final vowel and consonant deletion processes. We separate the haplology cases further into specific test cases. Our results from the unseen CVCV category show strong evidence for model generalisation of CV structures. We further tested the same model on a separate test set consisting of VCVC structures. We see that for approximately 78% of the set, it correctly recognises these cases as incorrect conditions for haplology. In the remaining instances, the model does show a rare over-generalisation to sometimes delete repeating sequences regardless of the characteristics of the sequence.
The largest source of error within the haplology cases is the scenario in which haplology can be applied twice within the same word. In these cases, typically, the first case of repeating CV is deleted, and the second instance remains untouched, as when outputting fuejaja with input fufuejaja, instead of the gold fueja.

heading: 8 Conclusion
text: The transformer model successfully models all 29 phonological phenomena with slight variation across phenomenon complexity. Our results show that the model can generalize linguistic categories and structures. Through haplology, we show that the model appears to learn to recognize and generalize syllabic structure and is capable of recognizing the difference between VC and CV and can also generalize the transformation triggered by haplology to unseen CV sequences.
Limitations
One drawback of the experiments presented here is the reliance on a constructed language. While we have tried to design a language that is as representative of natural language as possible, there may be additional statistical effects that are not taken into account. For example, it is unlikely that one language would capture all 29 phenomena presented here and that the process would be triggered enough times to produce a large enough corpus. How these findings extended to existing language corpora is an open question for future studies.

heading: A Summary of phonological processes
text: Affrication a process where either a stop, or fricative, becomes an affricate.
Anaptyxis (VCCV > VCVCV) a kind of epenthesis where an extra vowel is inserted between two consonants.
Aphaeresis (atata > tata) the deletion of word initial vowels.
Apocope (tata > tat) the loss of a sound, usually a vowel, at the end of a word.
Deaffrication an affricate becomes a fricative.
Degemination (CC > C) a sequence of two identical consonants is reduced to a single occurrence.
Devoicing the devoicing of stops word-finally.
Diphthongization an original single vowel changes into a sequence of two vowels.
Excrescence (amra > ambra; anra > andra; ansa > antsa) the insertion of a consonant. In our case, the insertion of b, d, or t.
Gemination (C > CC) produces a sequence of two identical consonants from a single consonant.
Hiatus glide (puo > pujo) a semi-vowel/glide is inserted between falling vowel pair.
Hiatus stop (pia -> pika) the insertion of a stop which breaks up a falling vowel pair.
Intervocalic Voicing various sounds become voiced between vowels, in this case we focus on stops.
Lengthening (tast > ta:t) a vowel lengthens subsequent to the loss of a following consonant, also called compensatory lengthening.
Metathesis (asta > atsa; asata > atasa) a change in which sounds exchange positions with one another within a word.
Monophthongization a diphthong changes into a single vowel.
Nasal Assimilation (np > mp) the change of nasal sounds to agree with the place of articulation of following stops.
Nasalization (ana > ãna) vowels become nasalized before nasal consonants.
Palatalization (k -> Ù, or d -> j) involves the change of a velar/alveolar sound to palato-alveolar, this often takes place before or after i or e.
Paragoge (tat > tata) adds a vowel to the end of a word.
Prothesis (tata > atata) a kind of epenthesis in which a sound is inserted at the beginning of a word.
Rhotacism (ase > are) s becomes r; this takes place between vowels or glides.
Shortening (ta: -> ta) vowels shorten in a variety of contexts, e.g. word-finally.
Spirantization an affricate is weakened to a fricative, or a stop to a fricative.
Strengthening fortition of sounds; an affricate becomes a stop, or a fricative becomes an affricate.
Syncope (atata > atta) the loss of a vowel from the interior of a word (not initially or finally)
Vowel lowering results in high vowels becoming mid or low vowels, or mid vowels becoming low.
Vowel raising is where low vowels raise to mid (or high) vowels, or mid vowels to high vowels).

heading: B Model details
text: Hyperparameter Value
Encoder/Decoder layers 4 Encoder/Decoder attention heads 4
Optimization Adam Embedding size 256 Hidden layer size 1024 Learning rate 0.001 Batch Size 400 Label Smoothing 0.1
Gradient clip threshold 1.0 Warmup updates 1000
Max updates 6000
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: How these findings extended to existing language corpora is an open question for future studies