# Automatic Era Identification in Classical Arabic Poetry
## 4 Experiments and Results
### Experiment 4: Era Classification by Poet  
In the experiments described thus far, the poems by the different poets were randomly split between the training and test sets. The fact that the lexical features account for most of the classification success suggests that the classifier might have learned word choices that characterize specific poets, having observed similar poems by the poets during the training and test phases. In a more challenging experiment, we aimed to identify the period of the poems, where testing is performed on the poems of poets whose poems were seen during training. Namely, we performed 'leave-one-poet-out' experiments, where a classifier is trained using the era-labeled poems by all poets except one poet, whose poems comprise the test set. We repeated this setup for each poet in turn, using the best-performing SVM as the classifier of choice. The results of this experiment are presented in Table 8. The table includes the resulting recall and F1 scores for each poet, after the classifier learned the model using the poems of the other poets. We note that in this evaluation setup, the precision is always 1.0 (Precision evaluates the proportion of poems assigned to the correct era out of the poems in the held-out set that belong to that era, where in this experiment all of the evaluated examples homogenously belong to the same poet, and hence to the target era)7 For each poet, the table also details the number of poems in our dataset and the respective era. The table also shows the average performances, both as a macro average, assigning equal importance to each poet, and as a weighted average, where the performance is weighted by the number of poems \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Feature Set Elimination** & **P** & **R** & **F1** \\ \hline \multicolumn{4}{c}{**No statistical features**} \\ \hline \multicolumn{4}{c}{‘Abbsäd} & 0.875 & 0.869 & 0.872 \\ \hline Pre- ‘Abbsäd & 0.871 & 0.877 & 0.874 \\ \hline Weighted AVG & 0.873 & 0.873 & 0.873 \\ \hline \multicolumn{4}{c}{**No lexical features**} \\ \hline \multicolumn{4}{c}{‘Abbsäd} & 0.827 & 0.839 & 0.833 \\ \hline Pre- ‘Abbsäd & 0.838 & 0.827 & 0.833 \\ \hline \multicolumn{4}{c}{**Weighted AVG**} & 0.833 & 0.833 \\ \hline \multicolumn{4}{c}{**No syntactic features**} \\ \hline \multicolumn{4}{c}{‘Abbsäd} & 0.886 & **0.889** & **0.887** \\ \hline \multicolumn{4}{c}{‘Pre- ‘Abbsäd} & **0.89** & 0.887 & **0.888** \\ \hline \multicolumn{4}{c}{‘Weighted AVG**} & **0.888** & **0.888** & **0.888** \\ \hline \multicolumn{4}{c}{**No domain-specific features**} \\ \hline \multicolumn{4}{c}{‘Abbsäd} & **0.89** & 0.881 & 0.885 \\ \hline Pre- ‘Abbsäd & 0.883 & **0.892** & 0.887 \\ \hline \multicolumn{4}{c}{‘Weighted AVG**} & 0.886 & 0.886 & 0.886 \\ \hline \multicolumn{4}{c}{**No topical features**} \\ \hline \multicolumn{4}{c}{‘Abbsäd} & 0.883 & 0.884 & 0.884 \\ \hline Pre- ‘Abbsäd & 0.885 & 0.884 & 0.885 \\ \hline \multicolumn{4}{c}{Weighted AVG} & 0.884 & 0.884 & 0.884 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results of Experiment 3a—feature ablation (the best results are marked in a bold font). available per poet. As shown, the overall performance is high, measured by F1 scores of 0.858 and 0.840 using the macro and weighted averages, respectively. We remind the reader that learning using a random split of the poems yielded an F1 of 0.887. This gap can be explained by the classifier learning the stylistic and lexical clues that characterize specific poets. Nevertheless, we deduce that there exists much evidence that is era-related rather than merely poet-related, which leads to high era classification results, even when evaluated on newly introduced poets. It is interesting to note that while this may seem counterintuitive at first, for poets with a small number of poems, the era identification is still high. In fact, this implies that almost all of the poems in the dataset (by the other poets) are utilized for training a model, providing it ample evidence from which to learn.