## 3 Results
### Named Entity Recognition  
Named entity recognition (NER) is a process of locating and categorizing proper nouns in text documents into predefined classes like a person, organization, location, time, and numeral expressions [52]. In this regard, one of the early attempts for Amharic NER is the work by [PERSON] [53], which is conducted on a corpus of 10,405 tokens. Using the CRF classifier, the research indicated that POS tags, suffixes, and prefixes are important features to detect Amharic named entities. The work by [PERSON] [54] also conducted similar work on a manually developed corpus of 13,538 words with the Stanford tagging scheme. The work by [PERSON] [55], an approach for a hybrid Amharic named entity recognition employed a combination of machine learning (decision trees and support vector machines) and rule-based methods. They have reported that the pure machine learning approaches with POS and nominal flag features outperformed the hybrid approach. Another work by [PERSON] and [PERSON] [56] also developed language-independent features to extract Amharic named entities using a bi-directional LSTM deep learning neural network model and merged the different feature vectors with word embedding for better performance. [PERSON] and [PERSON] [57] later employed a stack-based deep learning approach incorporating various semantic information sources that are built using an unsupervised learning algorithm with word2 Vec, and a CRF classifier trained with language-independent features. They have reported that the stack-based approach outperformed other deep learning algorithms. The main challenges with Amharic NER are: (1) As there are no capitalization rules in the language, it is very difficult to build a simple rule or pattern to extract named entities. (2) POS tags might help in discriminating named entities from other tokens, however, there is no publicly-available POS tagger for Amharic that can be integrated into NER systems. (3) Most of the research is carried out as part of academic requirements for a Bachelor's and Master's thesis where the research output was not well documented. Moreover, there are no benchmark dataset or tools that could advance future research in Amharic. In this work, we explore the effectiveness of semantic models for Amharic NER and release both the NER models and the benchmark datasets publicly (in Table 7). \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Model** & **Precision** & **Recall** & **F1** \\ \hline \multicolumn{4}{c}{CRFsuite Word Features} \\ \hline CRFsuite & **94.78** & **94.81** & **94.74** \\ \hline \multicolumn{4}{c}{Word Embeddings} \\ \hline AmWord2 Vec & 81.41 & 81.94 & 81.05 \\ fastText & 84.18 & 84.46 & 83.94 \\ \hline \multicolumn{4}{c}{Contextual Embeddings} \\ \hline AmRoBERTa & 94.08 & 94.13 & 94.08 \\ AmFlair & 91.75 & 91.71 & 91.69 \\ MultFlairFT & 91.19 & 91.07 & 91.06 \\ XLMR & 94.20 & 94.17 & 94.16 \\ MultFlair & 89.65 & 89.53 & 89.48 \\ \hline \multicolumn{4}{c}{Graph Embeddings} \\ \hline DeepWalk & 82.27 & 82.71 & 81.81 \\ Role2 Vec & 81.45 & 81.95 & 80.91 \\ \hline \multicolumn{4}{c}{State-of-the-art result} \\ \hline \hline \multicolumn{4}{c}{92.27 accuracy *} \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Experimental results for POS tagging (macro-averages). * indicated the stat-of-the-art result of the ELRCQB dataset but the results are not comparable as: (1) The reported result shows only the accuracy, and (2) they are not benchmark datasets. For this experiment, the Amharic dataset annotated within the SAY project at New Mexico State University's Computing Research Laboratory was used. The data is annotated with six classes, namely person, location, organization, time, title, and others. There are a total of 4237 sentences where 5480 tokens out of 109,676 tokens are annotated as named entities.