The Hidden Folk: Linguistic Properties encoded in Multilingual Contextual Character Representations
Year: 2023

Abstract:
To gain a better understanding of the linguistic information encoded in character-based language models, we probe the multilingual contextual CANINE model. We design a range of phonetic probing tasks in six Nordic languages, including Faroese as an additional zero-shot instance. We observe that some phonetic information is indeed encoded in the character representations, as consonants and vowels can be well distinguished using a linear classifier. Furthermore, results for the Danish and Norwegian language seem to be worse for the consonant/vowel distinction in comparison to other languages. The information encoded in these representations can also be learned in a zero-shot scenario, as Faroese shows a reasonably good performance in the same vowel/consonant distinction task.

heading: 1 Introduction
text: Subword and character sequence information is crucial in state-of-the-art neural language models (Sennrich et al., 2016; Kudo, 2018) because it improves their generalization and robustness capabilities (Xue et al., 2022; Tay et al., 2021). Additionally, character-level features are beneficial for morphologically rich and low-resource languages (Papay et al., 2018; Riabi et al., 2021). However, there is a lack of interpretability methods for character-based models. Only a few approaches have tried to understand the linguistic information encoded in character embeddings and cross-lingual approaches must be evaluated more rigorously by considering typology and linguistic distance (Artetxe et al., 2020). Therefore, in this work, we analyze how much phonetic information is encoded in contextualized multilingual character embeddings from the CANINE language model (Clark et al., 2022).
Based on six Nordic languages, we extract phonetic features for characters in context through unsupervised grapheme-to-phoneme alignment
and design a set of probing tasks. We explore the character representations in two different evaluation scenarios, a traditional train/test split scenario and a leave-one-letter-out scenario. We find that phonetic information on a global level (e.g., vowel and consonant detection) is encoded accurately in the character representations and more mixed results are achieved on lower-level probing tasks such as consonant voicing and manner, or vowel height and roundness. We also see that this information is transferred to the related zero-shot language Faroese. Our code is available online1.

heading: 2 Related Work
text: We discuss previous work in this area by focusing on existing multilingual character language models and the interpretability of these models.
Building Character-Level Models Subword and character-level information is exhibiting great benefits for computational language models (Bostrom and Durrett, 2020; Zhang et al., 2021). With the rise of multilingual models, pre-trained simultaneously on 100+ languages, these are also being adapted for and augmented with character-level information (Xue et al., 2022; Tay et al., 2021). We choose to work with the CANINE model by Clark et al. (2022), because it is a strongly performing neural encoder which operates directly on character sequences, i.e., without explicit tokenization or vocabulary, and incorporates a pre-training strategy operating directly on characters.
Understanding Character-Level Models Because of their popularity and lack of interpretability, many researchers have attempted to understand what is behind word and letter representations. A number of probing tasks have been employed to extract knowledge from contextualized word representations (e.g. Liu et al. 2019). For character
1https://github.com/manexagirrezabal/ hidden_folk_repository
6
representations, there are works in which the authors attempt to interpret the individual characterlevel hidden state contributions (Pinter et al., 2019; Kementchedjhieva and Lopez, 2018). In our case, though, we employ probing as the mechanism to interpret the representations. Recent work probes word-level representation with respect to their knowledge about characters. For instance, Kaushal and Mahowald (2022) predict the presence of a particular character in a token showing that large models robustly encode this information across various scripts. Additionally, Itzhak and Levy (2021) test the "spelling abilities" of language models showing that the embedding layers of RoBERTa and GPT-2 learn the internal character composition of whole words to a surprising extent, without seeing the characters coupled with the tokens during training.
Specifically on character-level models, Boldsen et al. (2022) compare perceptual representations to character embeddings. Their cross-lingual analysis shows that character representations correlate with phonological representations for languages using an alphabetic script and implies a relationship between the information encoded in the embeddings and the orthographic transparency of the languages. Furthermore, Hahn and Baroni (2019) probe character models in a cognitively realistic task on data with removed word boundaries showing that recurrent LMs learn morphological, syntactic and semantic aspects even on unsegmented text. These findings encourage the exploration of character and phoneme-level learning.

heading: 3 Contextualized Character Embeddings
text: We extract character embeddings (in the context of full words) from the CANINE model. In this section, we present the multilingual data and the embedding extraction.
Data Since character-level features are important for morphologically rich and low-resource languages (Lauscher et al., 2020; Garrette and Baldridge, 2013), we choose a set of six Nordic languages for our experiments: Danish (da), Swedish (sv), Norwegian (nb), Finnish (fi), Icelandic (is) and Faroese (fo). Five of the languages are included in the training data of the character language model (da sv, nb, fi and is). Additionally, we use Faroese to test performance of multilingual zero-shot embeddings. The starting point for extracting character embeddings is a frequency list for each language (see Table 1). We select the 10000 most frequent words of every language and then randomly sample 3000 of these words and retrieve embeddings for all characters in these 3000 words. This implies that more frequent characters in a given language will be better represented in the embeddings. Note that word length will affect the number of character embeddings extracted.
Model We extract contextualized embeddings from the CANINE model (Clark et al., 2022). CANINE is a neural encoder which operates directly on character-level without requiring an explicit tokenization strategy or a pre-defined vocabulary. We choose this model since it showed superior performance on multilingual downstream tasks. CANINE has been trained on data from 104 languages.2 While it performs well on NLP tasks, it has not yet been explored which type of linguistic information is encoded in these pre-trained character representations. We use the HuggingFace checkpoint of the CANINE model with autoregressive character loss.3 We input words to the model, and use the last hidden state of a character in the context of the word it occurs in as a contextualized character embedding (d = 768).

heading: 4 Phonetic Feature Extraction
text: In order to extract phonetic features from characters, we need to know how a specific letter should be
2The CANINE model is pretrained on on the multilingual Wikipedia data of mBERT.
3https://huggingface.co/google/ canine-c
pronounced. We do that by aligning the characters with the string of phones as given by Wikipron. As the number of letters and phones may not match, we align them using m2m-aligner (Jiampojamarn et al., 2007), an unsupervised model that is based on Expectation-Maximization. Then, we use the ipapy toolkit to obtain phonetic features for each phone. We describe this process below.
Pipeline We use the aligner to obtain phonetic features for characters in all six languages. First, we obtain a dictionary for each language from Wikipron (Lee et al., 2020),4 and then, we align graphemes to phones using m2m-aligner.5 Wikipron includes both phonemic and phonetic representations of words, which they refer to broad and narrow, respectively. As the model for extracting features works at a phonetic level, we use the phonetic representations (narrow). In the next step, we use the ipapy6 toolkit to extract phonetic features for each phone. Finally, the IPA features are merged with the CANINE character representations. This process results in one dataset per language, consisting of 6067 characters in 899 words for Danish, 700 characters in 135 words for Faroese, 4698 characterss in 745 words for Finnish, 268 characters in 43 words for Icelandic, 302 characters in 57 words for Norwegian, and 312 character in 58 words for Swedish. The reason for the drastic decrease in samples for some of the language is the small size of the pronunciation dictionaries.

heading: 5 Probing
text: In this section, we describe the probing tasks and the evaluation scenarios that we devise to test the extracted character embeddings.
We design 23 tasks to investigate the phonological knowledge encoded in the multilingual character representations. The tasks are split into three categories: global features (e.g., is this character a vowel or not?), consonant features (e.g., is the manner of articulation of this consonant plosive or not?7, and vowel features (e.g., is this vowel pronounced as a rounded vowel?8 The probing tasks have the struc-
4Please find the size of the dictionaries in Appendix A.2. 5We evaluated the alignments of m2maligner by using a manually aligned dictionary. This is available for the Danish language (Juul, 2010), where ∼ 42, 000 words and their phonetic transcriptions are aligned. Results show that the word error rate is below < 2.5%.
6https://github.com/pettarin/ipapy 7In Danish, "b" in peber ( [’phew5]) vs. åben ( [’O:b
˚ m " ])
8In Danish, "o" as in ballon ([b ˚ a’l2N]) (unrounded) vs. blod
tureF : X → Y , where given a set of character representations X , we want to find the best mapping F that relates X to a set of target features Y using a supervised Logistic Regression classification model.
As discussed by Hewitt and Liang (2019), it is important to take into account the expressivity of a probing task, since overly expressive probes, i.e., too many possible mappings for F : X → Y , does not reveal much about the internal feature representations. Therefore, we test the all probing classifiers in two evaluation scenarios: (i) 80/20, a random 80% training and 20% test split of the data, and (ii) LOLO, a leave-one-letter-out training and test split. The 80/20 setup implies that the same characters (in different contexts) can appear in the train and test split, resulting in a simpler probing task, whereas the LOLO setup ensures zero-shot learning for the specific character of which all representations in all contexts are held out during training.

heading: 6 Results
text: To understand the implicit phonetic information encoded in contextual representations, we present weighted F1-scores for a selection of features.9 Most of the individual probing tasks show F1-scores above 0.5, which means that the models generally perform better than a random baseline.
In Figure 2, we observe the results for two global features, where we predict whether a character is a vowel (top) and whether a letter is a consonant (bottom). In each plot, we report the results of both evaluation scenarios. Faroese performs very similar
([’b ˚ loDP])(rounded) 9See Appendix A.4 for the full results.
to Danish, even though the CANINE training data does not include any Faroese data. Furthermore, Norwegian and Swedish are the languages with the smallest difference between different validation methodologies (80/20 vs. LOLO).
Figure 3 shows the LOLO performance of each model for all characters and languages. The models were trained to predict whether the contextual character embedding had the label global_type_vowel, meaning that the character in this specific context is pronounced as a vowel. Taking a closer look at Danish and Norwegian, for instance, the F1-scores are relatively similar. This was expected given the similarities between the two languages in the written form. Although the results are similar, if we zoom in to specific character, we observe that the performance for vowels is systematically worse for Danish, which is reasonable given the complex nature of the Danish vowel system (Trecca et al., 2018).

heading: 7 Conclusions & Future Work
text: In this work, we design a probing mechanism to better understand information encoded in contextual character representations, for which we use two validation mechanisms. The first one, where the training data is divided into training and testing set, and the second one where we train one model for each character. The reasoning behind this is that a letter representation, even though it occurs in a different context, would have a similar representation and therefore, it would involve a kind of data leakage to the testset. We can imagine the example of the letter "a" in the words "tram" and "gas" in English. The contexts are rather different, but we would expect the representation of the letter "a" to be relatively similar, as it is pronounced
similarly. Therefore, the LOLO evaluation allows us to test the representations of a character in any context in a zero-shot scenario.
Following this methodology, we observe that contextual character representations encode some phonetic information, supported by the probing classifier performances. This result is in line with similar works performed on less complex representations, such as word2vec, PPMI+SVD or an RNN encoder-decoder (Silfverberg et al., 2018), where they found that in some languages these representations showed distributional properties.
We use Wikipron data as linguistic knowledge and we align graphs and phones using an automatic aligning mechanism. We validate the aligner for Danish on manually aligned data, but other languages may have their own specific challenges. We use a wide range of phonetic probing tasks to accommodate language-specific particularities.
It is crucial that the number of positive and negative instances is checked when probing each phonetic feature. As expected, many pronunciationrelated features do not occur in specific languages, and thus, this results in phonetic features with only one class, which will not shed any light on such phonetic properties. Hence, future research should address the design of these probing tasks as much as the results. Finally, since some of the datasets were relatively small, using a grapheme-to-phoneme model could increase the number of instances. Besides, these analyses could be extended to more languages to cover the full spectrum of orthographic depth.

heading: A Appendix
text: A.1 Frequency Lists Table 1 contains the details about the frequency lists used for each language.
A.2 Pronunciation Dictionaries Table 2 shows the size (word count) of the pronunciation dictionaries used in this work to train the m2n-aligner in all six languages.
A.3 Average word length and standard deviation Table 3 shows the average word length and standard deviation for each language in the CANINE embeddings.
A.4 Full Results Table 4 presents the LOLO results (F1 scores) for all probing tasks and all languages.

heading: Future Work
text: Hence, future research should address the design of these probing tasks as much as the results. Finally, since some of the datasets were relatively small, using a grapheme-to-phoneme model could increase the number of instances. Besides, these analyses could be extended to more languages to cover the full spectrum of orthographic depth