Deep Active Learning for Morphophonological Processing
Year: 2023

Abstract:
Building a system for morphological processing is a challenging task in morphologically complex languages like Arabic. Although there are some deep learning based models that achieve successful results, these models rely on a large amount of annotated data. Building such datasets, specially for some of the lower-resource Arabic dialects, is very difficult, time-consuming, and expensive. In addition, some parts of the annotated data do not contain useful information for training machine learning models. Active learning strategies allow the learner algorithm to select the most informative samples for annotation. There has been little research that focuses on applying active learning for morphological inflection and morphophonological processing. In this paper, we have proposed a deep active learning method for this task. Our experiments on Egyptian Arabic show that with only about 30% of annotated data, we achieve the same results as does the state-of-the-art model on the whole dataset.

heading: 1 Introduction
text: Recently, there has been lots of interest in morphological (re-)inflection processing (Narasimhan et al., 2015; Kirov and Cotterell, 2018; Belth et al., 2021). Having an acceptable model for morphological processing will help improve the performance of different natural language processing (NLP) tasks like speech synthesis (Halabi, 2016), morphological disambiguation (Khalifa et al., 2020; Inoue et al., 2021), and machine translation (Sennrich and Haddow, 2016; Erdmann et al., 2019; Alhafni et al., 2020). Despite recent progress in this field of study, there are lots of challenges for low-resource languages. Especially for utilizing successful but data-hungry deep learning (DL) models, the need for annotated data is vital. However, data annotation is a hard,
expensive, and time-consuming task. In addition, lots of annotated data does not contain useful information for improving the quality of a learning algorithm. In this paper, we propose a deep active learning (DAL) algorithm for morphophonological processing that is able to decrease the need for annotated data, by using only informative samples. In our experiments, we have chosen Arabic, a morphologically rich language. In addition, lots of Arabic dialects are very low-resource and the results from this study can help in building required datasets in a smarter way. Among Arabic dialects, Cairene Egyptian Arabic has been selected, because it is well-studied and has many resources and it is appropriate for our DAL simulation experiments. It should be noted that the proposed method is not specific to Arabic and it can be utilized on other languages or dialects.
As our baseline, we have chosen a very successful transformer model for character-level transduction tasks (Wu et al., 2021). We propose a pool-based DAL method in this study. To find the most uncertain (informative) samples, we combine an entropy strategy with a clustering method to keep an acceptable balance between the uncertainty and diversity of the chosen samples. The results of our experiments on the selected dataset show the success of the proposed DAL method.

heading: 2 Previous Work
text: In this section, we give a brief review of morphological inflection processing methods. Some recent DAL methods will also be reviewed.
There are several approaches for applying DL models for the morphological inflection problem (Yang et al., 2022; Wehrli et al., 2022; Batsuren et al., 2022; Wu et al., 2021; Dankers et al., 2021), which achieve successful results on different languages. Most of these models use character-level neural transducers using transformers, data augmentation, and recurrent
793
neural network (RNN)s. In Arabic, there is also much non-neural research on morphological modeling, including finite state technology (Habash and Rambow, 2006), precompiled tabular morphological analyzers (Buckwalter, 2002, 2004; Graff et al., 2009; Taji et al., 2018), and allomorphy modeling through linguistically descriptive rules (Habash et al., 2022).
In recent years, DAL has been used in some sub-fields of NLP. Zhang et al. (2017) and Ru et al. (2020) achieved satisfactory results in text classification using active learning (AL) with convolutional neural networks and adversarial uncertainty sampling, respectively. Different acquisition functions using the conditional random field model have been applied in named entity recognition (Shen et al., 2017; Prabhu et al., 2019; Liu et al., 2020). In neural machine translation, Peris and Casacuberta (2018) used an attention-based function, and Liu et al. (2018) applied a reinforcement learning method. In another study, Zhao et al. (2020) proposed a word frequency-based acquisition function to train neural machine translation actively. More recently, Muradoglu and Hulden (2022) introduced a DAL method for lemma inflection in written modern standard Arabic and some other languages. Their method uses entropy at the word level, while we find that the max of character-level entropy for a word performs best.

heading: 3 Background and Problem Definition
text: Morphophonology involves studying the relationship between morphology and phonology. The goal is to analyze data in order to discover the underlying forms and ordered rules that explain the observed data (Hayes, 2008). Arabic morphophonology is particularly interesting due to its complex templatic and concatenative morphology. Changes in morphophonology can happen on the stem pattern and also on word boundaries and stem. In addition, phonological alterations can be triggered by the addition of morphemes in concatenative morphology cases. The main problem of this paper is morphophonological generation, in which an underlying representation (UR) is transformed into a surface form (SF). We also investigate analysis, i.e., learning to transform a SF to a UR.

heading: UR SF SF Arabic

heading: 4 Dataset
text: We use the Arabic morphophonology dataset created by Khalifa et al. (2022). It uses a broad phonemic transcription. They generated URs from the CALIMAEGY morphological analyzer (Habash et al., 2012) for every SF extracted from the ECAL dataset (Kilany et al., 2002). They also added the analyzer’s segmentation to the UR part, delimiting word boundaries with #, prefixes with −, and suffixes with =. The dataset contains pairs of (UR, SF) as shown in Table 1. The split of this dataset is based on ECAL’s split, which contains TRAIN, DEV, and EVAL subsets. Due to the fact that ECAL’s splits are based on running texts, some words can occur in more than one split. Therefore Khalifa et al. (2022) also created subsets of the DEV and EVAL sets, called DEV-OOV and EVAL-OOV, which only contain non-overlapping words with the TRAIN split. The sizes of these splits are given in Table 2.

heading: 5 Proposed Method
text: In this section, we give a brief description of the baseline network. Then, the proposed DAL method will be explained in more detail.

heading: 5.1 Baseline Network
text: We have done several experiments to choose the most successful model for our AL experiments. Among different existing successful approaches (e.g., (Wehrli et al., 2022) and (Wu et al., 2021)), we chose Wu et al. (2021)’s system as our baseline system for conducting the DAL experiments because of its successful results on the utilized dataset. This is a transformer-based model that outperformed existing sequence-to-sequence models based on RNN for character-level transduction tasks, such as
morphological inflection generation, and achieved state-of-the-art results. Due to the fact that in character-level transduction tasks, the dataset sizes are significantly smaller than other tasks like machine translation, they have proposed a smaller transformer. In the next subsection, we will describe our proposed algorithm.

heading: 5.2 Active Learning Method
text: In this research, we have used the pool-based AL method in combination with the entropy strategy and clustering to determine uncertain samples based on the model’s prediction preserving the diversity of chosen samples. The AL method considers all the available data of the pool, U , which contains 13,170 samples, unannotated. Initially, about 10% of the samples (i.e., 1,400 samples) are chosen randomly from U for the data annotation process. In the first cycle of training the model, 500 samples from these 1,400 annotated samples are used for tuning, T , and the rest of the labeled samples (i.e., 900 samples) are used as the initial training dataset, L. The tune dataset is fixed throughout the procedure and determines the model with the highest accuracy during each AL training cycle. However, the training dataset, L, is increased by δ samples (i.e., 250 samples) per training cycle.
After training the model on the L dataset for the first time, all the pool samples are passed to the model for prediction. For each UR, the probability values are determined by computing the softmax of the model’s output logits. Most data sampling strategies in the AL method are based on some uncertainty criteria. In the case of sequence-to-sequence models focusing on character-level tasks, the sampling method based on entropy criteria is a suitable choice for uncertainty detection.
The output logits for each character of the predicted SF word, wSF, corresponds to the elements of a character vocabulary generated according to the predicted SF words. Using Equation (1), each set of logits, ch, is used to calculate the probability vector of a character in wSF. Here, Pi(ch) is the probability value of the ith element in the probability vector, chi is the logit of the ith element, and N is the vocabulary size.
Pi(ch) = echi
∑N i=1 e chi (1)
The entropy of a character ch, E′(ch), is calculated by Equation (2) based on the probability values of all possible generated characters vocabulary.
E′(ch) = − N∑
i=1
Pi(ch) logPi(ch) (2)
Equation (3) determines the entropy of the word wSF by choosing the maximum value among all its characters’ entropy values. That is, predicted labels with the lowest confidence have the highest entropy.
E(wSF) = max ch∈wSF
E′(ch) (3)
In each AL cycle, the trained model selects the next cycle’s additional (informative) samples. δ number of UR words with the highest wSF entropy are sampled without replacement. According to Equation (4), these most informative samples, w∗, are annotated and combined with the current L dataset to be utilized by the baseline system in the next AL cycle for training.
w∗ = argmax w∈U E(w) (4)
We augment this approach with a clustering technique to maintain diversity during the sampling process. In this approach, α number of UR words, α > δ, with the highest wSF entropy are selected for clustering. The optimal number of clusters is determined by computing the sum of the squared error (SSE) for various cluster counts, k. In each observation, Si, centroids are determined by taking the mean value, µi, of all the points, w, in each cluster. The sum of the deviation of the points from centroids for all clusters combined determines SSE. An observation’s cluster count, k, is optimal only if its SSE is minimal among all observations.
SSE = k∑
i=1
∑
w∈Si ||w − µi||2 (5)
The best cluster count in each AL cycle is used for clustering. The ultimate goal is to find δ most informative UR words with an acceptable diversity of total α (i.e., 1,000) samples.
A character-based word embedding model based on RNN is trained on datasets L and T to extract features for performing efficient clustering. Using a word embedding model enables us to have a sense
of UR words and their semantic relations with their corresponding SF words, considering L and T data points are labeled.
A word embedding model is used to vectorize α samples selected from U . Two sequences of one-hot vectors, representing a pair of UR and SF, pass through two long short-term memory (LSTM) networks, and the norm of the difference between output embeddings is fed to a sigmoid activation function. A sample from the dataset is a related pair, and samples with the highest Levenshtein distance from each other are non-related. These samples are combined and used to train the network. The model converts a word to a vector based on a character vocabulary generated from the unique characters of the training set.
After standardizing the vectors, they are fed to principal component analysis (PCA) in order to retain the most significant features and minimize the computational costs by reducing dimensions. The features dimension is reduced to 3 components for clustering. α samples were divided into clusters using the k-means method. Then, proportional to the size of clusters, δ samples are selected from each group. These samples will be annotated and moved to L for the next AL training cycle.

heading: 6 Experimental Results
text: The details of the model’s parameters and other variables in our experiments can be seen in Appendix A. To evaluate our proposed DAL method, we have run all experiments 5 times and the average and standard deviation of accuracy are visualized in Figure 1. We have reported random training (passive learning), AL with entropy, and AL with combined entropy and clustering method for each cycle of training.
As can be seen in Figure 1, the proposed DAL method (with and without clustering) grows much faster than the random curve and presents an asymptotical shape which shows that it has extracted all the useful information present in U when it reaches the asymptote using only 4,000 samples (i.e., about 30% of the training set). In contrast, the random (passive) learner requires the entire training set to achieve maximum performance. This is true for all evaluation sets, except for EVAL-OOV, where the random (passive) learner reaches maximum performance after 6,000 samples. We have no explanation for this unusual behavior of EVAL-OOV, but we do observe that
the overall top performance on EVAL-OOV is 2 percentage points lower than on DEV-OOV, while EVAL and DEV have similar results. This supports the hypothesis that EVAL-OOV contains some very difficult cases which the underlying system (with active or passive learning) cannot learn, so that active and passive learning converge earlier.
Unfortunately, we could not obtain evidence from this dataset that the proposed hybrid method (combining entropy and clustering) is effective.
Our model is designed to be language and model independent. It can be applied on different languages and it can be used on top of different character-based DL models. However, the proposed method should be applied on new languages and dialects in further studies. We expect that further studies will show that clustering can be effective, especially if many diverse phenomena are present in the data.

heading: 7 Error Analysis
text: We performed an error analysis on 100 randomly chosen errors from the DEV-OOV set, generated by models trained on 1,900 samples using AL with entropy and random selection methods. On this sample size, the random selection approach achieves an error rate of 12.9%, which reduces to 7.8% through AL (a 40% error reduction). We distinguish three basic types of errors: the system deletes a letter (i.e., phoneme) it should not; the system changes a letter it should not; the system adds a letter it should not, or does not delete a letter it should. We summarize results in Table 3. Starting at the top, we see that letter deletion is greatly reduced by moving from random selection to AL; affix deletion is a special case, where an entire affix is not realized, and this problem is eliminated. Letter change is also greatly reduced. However, a special case of letter change becomes more prominent: vowel length changes. This is a common effect in Arabic phonology due to changes in syllabification resulting from adding affixes. Finally, we see that letter addition remains a problem, with the special case of the system failing to delete a letter in fact increasing. The only case which is reduced is the special case of i-deletion in the active participle, which the AL setting appears to learn much better. We then have a fairly small category with multiple errors, which remains about the same. As we expect when the error rate goes down, the proportion of problem cases in the
corpus goes up. We distinguish three cases. First, foreign words have lexically idiosyncratic rules which cannot be learned. Second, almost all Arabic dialects replace the /l/ of the definite determiner /Al/ with the first letter of the following noun if it is coronal (“sun-letter assimilation”). However, Egyptian optionally also does this for /j/ and /k/, and the optionality is reflected in the training data which makes consistent learning impossible. Third, the corpus has a number of actual errors in the gold standard, usually in the UR. So in summary, the AL system has improved in all error types except for the letter addition categories.

heading: 8 Conclusion
text: In this paper, we have proposed a deep active learning method for the morphological inflection processing task. The proposed method can be used on different languages; however, as a case study, we have focused on the Egyptian Arabic dialect. The results of our experiment demonstrate the outstanding efficiency of the proposed method: With only 30% of the total training dataset, we achieve the same accuracy as the state-of-the-art model trained on whole dataset.
Future research includes applying this method to different low-resources Arabic dialects and other languages for building datasets, using other baseline algorithms, working on new uncertainty measures, and exploring for which datasets the clustering method can be helpful. We also intend to investigate how we can exploit our insight from the error analysis that the letter addition cases remain high (or even increase).

heading: Acknowledgements
text: We would like to thank three anonymous reviewers for their comments. Experiments were performed on the SeaWulf HPC cluster maintained by RCC and the Institute for Advanced Computational Science (IACS) at Stony Brook University and made possible by National Science Foundation (NSF) grant No. 1531492.
Limitations
Like lots of deep learning algorithms, our work also needs GPU resources. In common learning problems, models will be trained once on the existing training datasets, using dev sets for tuning the models. Then the trained model would be ready for use. In contrast, in active learning, we need to train the model several times (i.e., whenever new annotated samples are added to the current training set, the model should be re-trained), which increases the need for GPU resources. However, the need for GPU, is not related to our proposed method and it is due to the nature of active learning. In addition, one can run the active learning method once (rather than iteratively) for building an acceptable dataset.
It should be noted that we have designed the algorithm in a way to be independent of the target language and utilized model. However, we only tested our method on Egyptian Arabic dialect and the accuracy of the model should be investigated on other languages and dialects using different learning models in further studies.
Ethics Statement
The current work is a fundamental research and it is not essentially related to a particular application. We do not predict any ethical concerns from the algorithms and technologies proposed in this work. We have utilized publicly available dataset and open source libraries, which have been published before.

heading: B Additional Results
text: Employing the proposed method, we conducted the experiments in the reversed direction (i.e., SF to UR) for morphophonological analysis. As demonstrated in Figure 2, the DAL methods outperformed random training by extracting all the informative samples of the pool set, U , when reaching the asymptote using 8,000 samples. Since the current baseline is designed for morphophonological generation tasks, its performance is diminished for SF to UR. As our proposed method is model-agnostic, a more suitable baseline model for this task would achieve higher accuracies in morphophonological analysis for both passive and active learning.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the Limitation section after Paper’s conclusion.

heading: 3 A2. Did you discuss any potential risks of your work?
text: In Ethics Statement section after Limitation section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Please refer to the abstract and introduction (Section 1) of the paper.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sections 3 and 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 3, 4, and appendix A.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sections 3 and 4.
C 3 Did you run computational experiments?

heading: Appendix A.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5, Appendix A, and Appendix B.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future research includes applying this method to different low-resources Arabic dialects and other languages for building datasets, using other baseline algorithms, working on new uncertainty measures, and exploring for which datasets the clustering method can be helpful. We also intend to investigate how we can exploit our insight from the error analysis that the letter addition cases remain high (or even increase)