dataesr/scientific-paragraphs-categorization huggingface dataset:
These embedding models learn each word representation considering the window size. For example, given the sentence "ጽሁፍ የተጻፈው የህክምና እርዳታ ለመስጠት አይደለም /tsʼɨhufɨ jətətsʼafəwɨ jəhɨkɨmɨna ʔɨrɨdata ləməsɨt ətɨ ʔəjɨdələmɨ/" along with its corresponding stem and root representations and the window size of 3, the word2vec model learns the representation of each word using the pair of input and output training datasets as presented in Table  1. The size of the window in the case of word2vec, GloVe, and fastText is presented in Table  2. Stem (ር-ድ, ጽ-ህ-ፍ), (ር-ድ, ጽ-ህ-ፍ), (ር-ድ, ህ-ክ-ም), (ር-ድ, ስ-ጥ) Root ለመስጠት (ለመስጠት, የተጻፈው), (ለመስጠት, የህክምና), (ለመስጠት, እርዳታ), (ለመስጠት, አይደለም) Word (ስጥ, ጻፍ), (ስጥ, ህክም), (ስጥ, ርድ) Stem (ስ-ጥ, ጽ-ህ-ፍ), (ስ-ጥ, ህ-ክ-ም), (ስ-ጥ, ር-ድ) Root Table  1. Word-based, stem-based, and root-based Skip-gram training samples of word2vec.
Word Training Sample Type ጽሁፍ (ጽሁፍ, የተጻፈው), (ጽሁፍ, የህክምና), (ጽሁፍ, እርዳታ) Word (ጽሁፍ, ጻፍ), (ጽሁፍ, ህክም), (ጽሁፍ, ርድ) We built three types of word embedding models, namely word2vec, GloVe, and fastText. The number of hidden layers between the input layer and the output layer is 1, as the word embeddings learn only one vector representation for each word. These embedding models learn each word representation considering the window size. For example, given the sentence "ጽሁፍ የተጻፈው የህክምና እርዳታ ለመስጠት አይደለም /tsʼɨhufɨ jətətsʼafəwɨ jəhɨkɨmɨna ʔɨrɨdata ləməsɨt ətɨ ʔəjɨdələmɨ/" along with its corresponding stem and root representations and the window size of 3, the word2vec model learns the representation of each word using the pair of input and output training datasets as presented in Table  1. The size of the window in the case of word2vec, GloVe, and fastText is presented in Table  2.
Each Amharic word is represented as a bag of sub-words and the word itself. For example, consider the Amharic word  However, in the case of fastText, for a given word, n-grams are taken from 3 to 5 g and a word is represented by the sum of the vector representations of its n-grams. Each Amharic word is represented as a bag of sub-words and the word itself. For example, consider the Amharic word ለምስክርነት /ləmɨsɨkɨrɨnətɨ 'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Stem (ር-ድ, ጽ-ህ-ፍ), (ር-ድ, ጽ-ህ-ፍ), (ር-ድ, ህ-ክ-ም), (ር-ድ, ስ-ጥ) Root ለመስጠት (ለመስጠት, የተጻፈው), (ለመስጠት, የህክምና), (ለመስጠት, እርዳታ), (ለመስጠት, አይደለም) Word (ስጥ, ጻፍ), (ስጥ, ህክም), (ስጥ, ርድ) Stem (ስ-ጥ, ጽ-ህ-ፍ), (ስ-ጥ, ህ-ክ-ም), (ስ-ጥ, ር-ድ) Root Word Stem Root Word-based:<ለም, ለምስ, ምስክ, ስክር, ክርነ, ርነት, ነት>, and ለምስክርነት Stem-based:<ምስ, ምስክ, ስክር, ክር>, and ምስክር Root-based:<ም-, ም-ስ,-ስ-, ስ-ክ, -ክ-¸ ክ-ር, -ር>, and ም-ስ-ክ-ር Using such self-supervised datasets, each vocabulary entry is represented by a real-valued vector which is learned from the corpora.
'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Using such self-supervised datasets, each vocabulary entry is represented by a realvalued vector which is learned from the corpora. The vector dimension of a word (or token) is different in the case of word2vec, GloVe, and fastText models. Word-based:  However, in the case of fastText, for a given word, n-grams are taken from 3 to 5 g and a word is represented by the sum of the vector representations of its n-grams. Each Amharic word is represented as a bag of sub-words and the word itself. For example, consider the Amharic word ለምስክርነት /ləmɨsɨkɨrɨnətɨ 'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Word-based:<ለም, ለምስ, ምስክ, ስክር, ክርነ, ርነት, ነት>, and ለምስክርነት Stem-based:<ምስ, ምስክ, ስክር, ክር>, and ምስክር Root-based:<ም-, ም-ስ,-ስ-, ስ-ክ, -ክ-¸ ክ-ር, -ር>, and ም-ስ-ክ-ር Using such self-supervised datasets, each vocabulary entry is represented by a real-valued vector which is learned from the corpora.
Stem-based: Information 2023, 14, x FOR PEER REVIEW 8 of 23 However, in the case of fastText, for a given word, n-grams are taken from 3 to 5 g and a word is represented by the sum of the vector representations of its n-grams. Each Amharic word is represented as a bag of sub-words and the word itself. For example, consider the Amharic word ለምስክርነት /ləmɨsɨkɨrɨnətɨ 'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Word-based:<ለም, ለምስ, ምስክ, ስክር, ክርነ, ርነት, ነት>, and ለምስክርነት Stem-based:<ምስ, ምስክ, ስክር, ክር>, and ምስክር Root-based:<ም-, ም-ስ,-ስ-, ስ-ክ, -ክ-¸ ክ-ር, -ር>, and ም-ስ-ክ-ር Using such self-supervised datasets, each vocabulary entry is represented by a real-valued vector which is learned from the corpora.
Root-based: Information 2023, 14, x FOR PEER REVIEW 8 of 23 However, in the case of fastText, for a given word, n-grams are taken from 3 to 5 g and a word is represented by the sum of the vector representations of its n-grams. Each Amharic word is represented as a bag of sub-words and the word itself. For example, consider the Amharic word ለምስክርነት /ləmɨsɨkɨrɨnətɨ 'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Word-based:<ለም, ለምስ, ምስክ, ስክር, ክርነ, ርነት, ነት>, and ለምስክርነት Stem-based:<ምስ, ምስክ, ስክር, ክር>, and ምስክር Root-based:<ም-, ም-ስ,-ስ-, ስ-ክ, -ክ-¸ ክ-ር, -ር>, and ም-ስ-ክ-ር Using such self-supervised datasets, each vocabulary entry is represented by a real-valued vector which is learned from the corpora.
For example, inputting the sentence  However, in the case of fastText, for a given word, n-grams are taken from 3 to 5 g and a word is represented by the sum of the vector representations of its n-grams. Each Amharic word is represented as a bag of sub-words and the word itself. For example, consider the Amharic word ለምስክርነት /ləmɨsɨkɨrɨnətɨ 'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Word-based: <ለም, ለምስ, ምስክ, ስክር, ክርነ, ርነት, ነት>, and ለምስክርነት Stem-based: <ምስ, ምስክ, ስክር, ክር>, and ምስክር Root-based: <ም-, ም-ስ,-ስ-, ስ-ክ, -ክ-¸ ክ-ር, -ር>, and ም-ስ-ክ-ር Using such self-supervised datasets, each vocabulary entry is represented by a real-valued vector which is learned from the corpora. The vector dimension of a word (or token) is different in the case of word2vec, GloVe, and fastText models.
For example, consider the Amharic word ለምስክርነት /ləmɨsɨkɨrɨnətɨ 'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Word-based: <ለም, ለምስ, ምስክ, ስክር, ክርነ, ርነት, ነት>, and ለምስክርነት Stem-based: <ምስ, ምስክ, ስክር, ክር>, and ምስክር Root-based: <ም-, ም-ስ,-ስ-, ስ-ክ, -ክ-¸ ክ-ር, -ር>, and ም-ስ-ክ-ር Using such self-supervised datasets, each vocabulary entry is represented by a real-valued vector which is learned from the corpora. The vector dimension of a word (or token) is different in the case of word2vec, GloVe, and fastText models.
Each Amharic word is represented as a bag of sub-words and the word itself. For example, consider the Amharic word ለምስክርነት /ləmɨsɨkɨrɨnətɨ 'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Word-based: <ለም, ለምስ, ምስክ, ስክር, ክርነ, ርነት, ነት>, and ለምስክርነት Stem-based: <ምስ, ምስክ, ስክር, ክር>, and ምስክር Root-based: <ም-, ም-ስ,-ስ-, ስ-ክ, -ክ-¸ ክ-ር, -ር>, and ም-ስ-ክ-ር Using such self-supervised datasets, each vocabulary entry is represented by a real-valued vector which is learned from the corpora. The vector dimension of a word (or token) is different in the case of word2vec, GloVe, and fastText models.
For example, consider the Amharic word ለምስክርነት /ləmɨsɨkɨrɨnətɨ 'for witness'/ and the 3 g characters. Its sub-word representations in the case of word-based, stem-based, and root-based are presented as follows: Word-based:<ለም, ለምስ, ምስክ, ስክር, ክርነ, ርነት, ነት>, and ለምስክርነት Stem-based:<ምስ, ምስክ, ስክር, ክር>, and ምስክር Root-based:<ም-, ም-ስ,-ስ-, ስ-ክ, -ክ-¸ ክ-ር, -ር>, and ም-ስ-ክ-ር Using such self-supervised datasets, each vocabulary entry is represented by a real-valued vector which is learned from the corpora. The vector dimension of a word (or token) is different in the case of word2vec, GloVe, and fastText models.
Pre-Trained Amharic Models for Text Classification We also investigated the usability of our pre-trained BERT models for text classification. The common practice for using a pre-trained BERT model is to update (or retrain) the original output layer with a task-specific layer and fine-tune the whole model. Fine-tuning is the task of creating application on top of pre-trained models using labeled data. Finetuning BERT is one of the most popular and effective methods to tackle NLP tasks  [48]. Here, we fine-tuned our pre-trained Amharic BERT models for text documents classification tasks. The pre-trained Amharic models were fine-tuned using manually labeled training datasets. The training datasets passed through the same data processing used for pre-training models. Furthermore, the same tokenizer and vocabulary were used.
The pre-trained Amharic MLM model was retrained and evaluated using these datasets. For the classification task, we added a classification head on top of the pre-trained Amharic BERT model and then trained the entire model on our training datasets. All parameters were fine-tuned from end-to-end. The softmax function was used to estimate the probability of the label c at output layer o using Equation (  6 ). We fine-tuned all of the parameters from BERT and W. P(c|o) = softmax(W) ( 6 ) where W is the task-specific parameter matrix. Text classification is one of the core tasks of NLP, and it is commonly considered to show the effectiveness of a pre-trained BERT model. Our pre-trained Amharic models were fine-tuned for Amharic document classification based on subject and relevance to a query.
For document classification based on subject, the documents in our datasets were classified as technology, politics, religion, sport, justice, entertainment, social, agriculture, economy, education, and health. Hence, the number of labels was 11. Moreover, we applied the pre-trained Amharic BERT models to check the relevance of a document for a query. Documents were classified as relevant or irrelevant to a query, and thus it was a binary text document classification. The fine-tuned relevance classification model was trained by inputting a pair of texts: query and document. Since the input of a BERT is a sequence, the two inputs are separated by a special token  [SEP]. We transformed a query sQ and a document sD into the format compatible with BERT sequence-pair classification tasks as: [CLS] sQ [SEP] sD [SEP].
Datasets To train the word embeddings (word2vec, GloVe and fastText), BERT models (NSP and MLM), and fine-tuned models, we used three datasets: word-based, prepared in 2AIRTC  [50] , stem-based, and root-based corpora built by  [51]. Each of these datasets has 6069 documents and 240 queries. All documents were used to train the pre-trained word embedding and BERT models. The documents consisted of 72,814 sentences constructed from 1,592,351 words. Furthermore, the retrieval effectiveness of Amharic IR system after query expansion using word embedding models was tested using these test collections. The test collections had corpus, topic set, and relevance judgment. The datasets are publicly available freely at https://www. irit. fr/AmharicResources/, accessed on 7 August 2022. For relevance and non-relevance classification, we used only 50 word-based queries and 5514 documents, as well as their relevance judgment from the 2AIRTC collection.
Pre-trained Models and Training Configurations The word2vec, GloVe, and fastText models were trained using the configurations presented in Table  2. The MLM model was trained to predict randomly masked tokens, whereas the NSP model learned to predict whether sentence B is the next sentence of A, given a pair of input sentences A and B. We used the standard BERT model architecture. The two BERT models' trainings were performed using sentences as input. The maximum number of tokens of each input sentence for each model was 512. The two models were trained using the same hyper-parameters over the three datasets using epochs 5 and 10 (see Table  3 ). We used Adam optimizer  [52]  with a learning rate of 10 -5 to optimize the objective functions of MLM, NSP, and classification models.
Evaluation Metrics The performance of the constructed Amharic BERT models was evaluated based on loss, F-score, and accuracy metrics. The performance of the NSP-BERT model was evaluated using 50 similar pairs of sentences but in word-based, stem-based, and root-based forms. These 50 pairs of sentences were selected randomly from the three datasets; 25 pairs were adjacent and the remaining 25 pairs of sentences were non-adjacent sentences. The accuracy of the NSP model is the percentage of corrected predicted pairs of sentences over the total sentence pairs, and it is computed using Equation  (7). Accuracy = TP + FP TP + FP + TN + FN ( 7 ) where TP is true positive, TN is true negative, FP is false positive, and FN is false negative. The retrieval effectiveness of our proposed IR system using the pre-trained word embedding models was evaluated using precision, recall, MAP, bpref, and normalized discounted cumulative gain (NDCG).
Results and Discussion We carried out various experiments to create Amharic pre-trained word embedding and BERT models, and evaluated them in NLP and retrieval tasks. In this section, we demonstrate the essence of BERT on Amharic language by fine-tuning the pre-trained models and investigating the retrieval effect of query expansion using word embeddings. Therefore, we present the intrinsic and extrinsic evaluations with detailed analysis in the following sections. Three NSP-BERT models were trained using word-based, stem-based, and root-based corpora to predict whether two sentences appear consecutively within a document or not. After training, we manually selected 50 pairs of sentences from each corpus in order to evaluate the accuracy of the models. Each pair of sentences was tokenized using the same WordPiece tokenizer, which is also used for the corpora.
The training loss, validation loss and accuracy are presented in Table  5. As shown in the table, the testing loss and accuracy of the word-based NSP model was better than stem-based and root-based datasets on the same evaluation dataset at epoch 5. However, the model selects pairs of sentences randomly during the training phase. As a result of this, the training loss values of wordbased, stem-based, and root-based were based on different samples, making it difficult to directly compare training loss values. The Amharic pre-trained MLM model was fine-tuned for document classification tasks. Our pre-trained MLM was retrained using labeled datasets to create two fine-tuned models: a fine-tuned model to classify documents as relevant or non-relevant for 50 queries, and a fine-tuned model to classify 1189 labeled documents based on their subjects.
These two fine-tuned models were created on top of the pre-trained word-based MLM model. The labeled datasets included both training and validating datasets used for training and evaluating the fine-tuned models. The labeled training and validating documents in the datasets were preprocessed using the same technique used for document processing during training MLM and NSP models. Our labeled datasets had two columns: the document and label columns. In the case of document relevance identification, the values of the label column were relevant and non-relevant, whereas in the case of document classification based on subject, the values of the label columns were education, religion, politics, sport, etc. Each of the queries were trained with both relevant and non-relevant documents. Since there was no usable Amharic stemmer and morphological analyzer to extract stems and roots from labeled dataset, we could not evaluate the stem-based and root-based versions of the pre-trained MLM models.
Since the fine-tuned models have consistent training and validation loss values after epoch 14, we trained them for the first 17 epochs and reported the results. Their training and validation losses were decreased from epoch 0 to 17 (see Figure  5 ). Their F1-score and accuracy values were improved from epoch 0 to 17. On the graph legend, R represents relevance, and S represents subject of a document. As depicted in Figure  5 , after epoch 14, the training, validation, and F1 scores of the two fine-tuned models did not show significant change. The optimal loss, F1-score (weighted), and accuracy of the fine-tuned models are reported in Table  6. As shown in Table  6 , the performance of document classification models was high because the training dataset was of a moderate size and balanced, and the size of almost all training and testing As depicted in Figure  5 , after epoch 14, the training, validation, and F1 scores of the two fine-tuned models did not show significant change.
Dataset Retrieval Effectiveness CBOW Skip-gram P@5 P@10 ndcg R-prec bpref P@5 P@10 ndcg R-prec bpref Word-based 0. As indicated in Table  7 , the retrieval performance of the word-based approach was better than the stem-based and root-base ones. On the three datasets, the CBOW and skipgram models performed equally or closely on the same dataset. As presented in Table  8 , word-based retrieval outperformed the stem-based and root-based, with a large difference. As shown in Table  9 , as with word2vec and GloVe, retrieving using a word-based approach was better than the others. The skip-gram model outperformed CBOW. This indicates that skip-gram is more suitable for obtaining or grouping similar Amharic words. For the three word embeddings, the retrieval effectiveness of the word-based approach was better than the root-based approach, which, in turn, was better than the stembased one.
Query Expansion Using Word Embedding The retrieval effectiveness of query expansion using the three word embedding models was evaluated. We ran 240 queries on word-based, stem-based, and root-based index files. In the case of stem-based and root-based retrieval, documents and user information needs were morphologically processed before indexing and query expansion. The retrieval results of each query were processed to convert them into TREC format, which can be used by Trec-Eval toolkit. The retrieval effectiveness of query expansion using word2vec, GloVe, and fastText is presented in Tables  7 8 9 , respectively. As indicated in Table  7 , the retrieval performance of the word-based approach was better than the stem-based and root-base ones. On the three datasets, the CBOW and skipgram models performed equally or closely on the same dataset.
The resources we share on this link are: i. Three BERT WordPiece tokenizers (word-based, stem-based, and root-based); ii. Three BERT byte-pair encoding (BPE) tokenizers (word-based, stem-based, and root-based); iii. Three BERT masked language models (word-based, stem-based, and root-based); iv. Three BERT next sentence prediction models (word-based, stem-based, and root-based); v. Nine word embedding models (word2vec, GloVe and fastText; each has word-based, stem-based and, root-based versions). The constructed BERT model achieves remarkable results on verified Amharic NLP and IR down streaming tasks. However, the effect of word embedding to the improve retrieval effectiveness of IR systems in many languages is low. The performance of our models can be compared to some other languages' models using Tables  6  and 13.
In this study, an investigation was made to build Amharic word embedding and BERT models and verify their use for Amharic NLP and IR tasks. Due to some challenges, such as experimental resource and computational machine scarcity, we have not explored the following things in our experiments. i. Since our datasets are small relation to the requirement of BERT model, we tested only the effect of BERT at the base case. However, recently, BERT, at a large scale, has performed better than the base case on huge datasets. Therefore, the performance of the BERT model can be further enhanced with the use of a large corpus. ii. Recently, the use of the BERT model for ranking retrieval results has provided better performance. In our study, we evaluated the performance of the BERT model to identify relevant and non-relevant documents for a query rather than ranking.
Application of the Pre-Trained Word Embedding and BERT Models One of the aims of this research was to create pre-trained Amharic word embedding and BERT models and make them publicly accessible. We trained and released our pre-trained Amharic word2vec, GloVe, fastText, and BERT models, and Amharic BERT tokenizers, which can motivate the research community to carry out experiments on the pre-trained Amharic language models. Pre-training is fairly expensive, but it is a one-time procedure. Our pre-trained models can be used to reduce the need for many heavilyengineered tasks. The extracted features can be fed into a machine learning model so as to work with text data and preserve the semantic and syntactic information. The constructed Amharic models can be used in different applications. For example, our word embedding vectors can be used for word analogy tasks, namely entity recognition and chunking.
Conclusions Amharic exhibits complex morphological characteristics. It is one of the underresourced languages. There is lack of usable and publicly accessible pre-trained language models for this language. Because of this, the effect of learned text representation on NLP tasks has not yet been investigated. In this study, we explored the impact of word embedding (word2vec, GloVe, fastText) and BERT models on Amharic text. Wordbased, stem-based, and root-based corpora were used to investigate the effect of word embedding models on query expansion. Moreover, we built Amharic pre-trained BERT models and fine-tuned them for text classification. The usability of our pre-trained BERT models were evaluated in IR and NLP tasks. The training and validating losses of the fine-tuned BERT model decreased as number of epochs increased.
In contrast, their F1-score and accuracy improved from the first epoch to the next. We also made the pre-trained word embeddings, BERT models, and BERT tokenizers publicly accessible at https://www. irit. fr/AmharicResources/amharic-pre-trained-language-models/, (accessed on 19 May 2022). From the experiments we performed on the constructed models, we found that learned text representation using a word-based approach outperformed stem-based and root-based approaches across all tasks considered in this research. Promising results were obtained from the BERT and fastText models. As we created pre-trained models at base scale, our work can be further enhanced by increasing the size of the corpus. Therefore, this work can be expanded with the aim of investigating the significance of model construction at large scale using a huge dataset.
Information 2023 , 2023 14, x FOR PEER REVIEW 15 of 23during training MLM and NSP models. Our labeled datasets had two columns: the document and label columns. In the case of document relevance identification, the values of the label column were relevant and non-relevant, whereas in the case of document classification based on subject, the values of the label columns were education, religion, politics, sport, etc. Each of the queries were trained with both relevant and non-relevant documents. Since there was no usable Amharic stemmer and morphological analyzer to extract stems and roots from labeled dataset, we could not evaluate the stem-based and rootbased versions of the pre-trained MLM models. Since the fine-tuned models have consistent training and validation loss values after epoch 14, we trained them for the first 17 epochs and reported the results.