# Automatic Era Identification in Classical Arabic Poetry
## 3 Dataset and Methods
### Methods
#### 3.2.2 Feature Extraction  
Table 3 presents the feature types that we use to represent each poem. As shown, these features encode multiple different aspects of the text: statistical, lexical, syntactic, domain-specific, and topical. Statistical and lexical information is often used to quantify an author's writing style and has been proven effective in past studies [60; 61; 62; 15; 28; 63; 64]. We adapted these features and further encoded syntactic, domain-specific, and topical features to model the unique characteristics of classical Arabic poetry. Statistical features contain general quantitative descriptors of the poem, including the poem's length in terms of words, the average length of each line (_bayt_), and the average word length. Lexical features contain features that pertain to the content of the poem and the poet's word choices. A main feature encodes the multinominal distribution of words that comprise the poem. Considering that Arabic is a morphologically rich language, where word forms often represent similar meanings in different inflection forms, we also encode the distribution of word stems in the poem as well as stem sequences of two consecutive words (bigrams) and the total number of unique stems that appear in the poem. In addition, we consider the character distribution within the content and three-letter character sequences, which model word roots as well as morphological prefixes and suffixes. Syntactic features require deeper linguistic analysis. We consider the POS tags that are assigned to each word unit and represent the distribution of POS tags as well as POS sequences of word bigrams. Following the literature on AA, we also represent the multinomial distribution of negation words and function words in the poem. We distinguish between these word categories based on their POS analysis, using the POS tags of 'Neg' (negation) and 'Prep' (preposition), respectively. The domain-specific features contain features related to the writing style and characteristics of classical Arabic poetry. We focus on various properties that correspond to rhyming patterns and literary elements that we automatically identify and encode as features. These include the _kafiya_, the letter that ends each line of the poem; the proportion of lines in the poems in which the end of the first hemistich matches the end of the verse; the proportion of lines in which two hemistiches in a line end with words derived from the same stem; and the literary element where the two hemistiches in a line end with two words derived from stems that share the same letters with a different order (the encodings take the form of an indication and stem pairs). We also represent the lexical diversity of the content as a stylistic marker, measuring the ratio of unique content words out of all content words. Finally, the topical features reflect the prospect of identifying an era or a poet according to the distribution of topics that are represented in the poem. As a preliminary step, we used Latent Dirichlet Allocation (LDA) [65], a generative probabilistic model, to identify the latent topics in our collection of poems. The LDA model assumes that documents contain multiple topics, where each topic is characterized by a distribution over words that tend to co-occur in the text. For example, words like '[PERSON]', 'prayer', and 'belief' may cluster together based on co-occurrence statistics into a single topic. While LDA does not assign names to topics, such a word cluster would correspond to the topic of religion. Given a corpus of documents (poems, here), and a pre-specified number of topics, LDA identifies these topic distributions from the word co-occurrences in the corpus. We applied a sampling-based variant of LDA as implemented in the MALLET learning toolkit [65; 66] to our corpus. The number of topics, K, to be discovered using this classic variant of LDA must be pre-specified.