# Meta-Learner for Amharic Sentiment Classification
## 3 Materials and Methods
### Evaluation Metrics  
To measure the performance of a classification system, evaluation metrics are required. The most popular metrics include accuracy, precision, recall, and F1-score. Before describing the evaluation metrics, let us define important terms. The terms such as True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) are defined as follows:  
* (i) True Positive (TP) is the number of samples belonging to the positive class that are correctly predicted by the model. * (ii) True Negative (TN) is the number of samples belonging to the negative class that are correctly predicted by the model. * (iii) False Positive (FP) is the number of samples belonging to the positive classes that are wrongly predicted by the model. This is also called Type I Error. * (iv) False Negative (FN) is the number of samples belonging to the negative classes that are wrongly predicted by the model. This is also called Type II Error. The model's performance evaluation metrics which are used in this study are described as follows:  
* (i) Accuracy (A) is the percentage of correctly predicted samples, i.e., \[A=\frac{TP+TN}{TP+TN+FP+FN}.\] (4)
* (ii) Precision (P) is the evaluation metric that measures the correctly predicted samples actually turned out to be positive, i.e., \[P=\frac{TP}{TP+FP}.\] (5) Note that precision is a metric that measures the reliability of the model. * (iii) Recall (R) is a measure of the number of actual positive samples which are correctly predicted by the model, i.e., \[R=\frac{TP}{TP+FN}.\] (6) Recall is also called sensitivity. Note that precision is more important to tell when the model predicted more false positive samples than false negative samples. In contrast, recall is a more important metric to tell if the model predicts more false negative than false positive samples. * (iv) F1-Score (F1) is the harmonic mean between precision and recall. It is calculated from precision and recall, i.e., \[F1=2\cdot\frac{Precision\cdot Recall}{Precision+Recall}.\] (7)
* (v) Area Under Curve(AUC) is the most popular evaluation metric for binary classification problem. AUC is a measure of the probability of a classifier that will rank a randomly chosen positive example higher than a randomly chosen negative example. AUC ranges [0,1], the higher the AUC implies, the better the model. * (vi) Logarithmic Loss (LogLoss) is used to measure how close or far model's predicted value from actual value. For the binary classification problem, LogLoss is also called binary cross-entropy, which is the negative average of the log of corrected predicted probabilities. For \(N\) samples, LogLoss is given by \[LogLoss=\frac{1}{N}\cdot\sum_{i=1}^{N}-(y_{i}\cdot log(p_{i})+(1-y_{i})\cdot log (1-p_{i}))\cdot\] (8) where \(y_{i}\) is the actual value, \(p_{i}\) is the probability of class 1, and \(1-p_{i}\) is the probability of class 0. * (vii) Cross-Validation (CV): K-fold Cross-Validation is the model evaluation technique where all data set samples are randomly divided into k folds of equal size. As there is no standard rule for selecting \(k\), usually the value of \(K\) is 5 or 10. We chose \(K=5\). For each run, \(K\)-1 folds are used for training the model and the remaining are used as the testing set. This is repeated k times so that each of the folds is used once for testing. The average accuracy of the 5 models and standard deviation is returned. This is illustrated in Figure 1.  
The benefits of using cross-validation include (i) its capability of avoiding overfitting/underfitting, (ii) it can evaluate model consistency by producing accuracy/error rate for all the test sets, and (iii) it has also capability of training/testing on a small data by using the total datasets [47]. * (viii) Mean of Accuracy and Standard Deviation (SD): Note that from statistics,'mean' is defined as the average of accuracy of the models in five folds using testing sets, whereas 'variance' is the sum of the squares of the model's accuracy deviated from the mean divided by the total number of models.