SlowGuess/ACL_2020_OCR huggingface dataset:
Tim Van de Cruys
Institut de Recherche en Informatique de Toulouse (IRIT)
Artificial and Natural Intelligence Toulouse Institute (ANITI)
CNRS, Toulouse
tim.vandecruys@irit.fr
In the last few years, a number of successful approaches have emerged that are able to adequately model various aspects of natural language. In particular, language models based on neural networks have improved the state of the art with regard to predictive language modeling, while topic models are successful at capturing clear-cut, semantic dimensions. In this paper, we explore how these approaches can be adapted and combined to model the linguistic and literary aspects needed for poetry generation. The system is exclusively trained on standard, non-poetic text, and its output is constrained in order to confer a poetic character to the generated verse. The framework is applied to the generation of poems in both English and French, and is equally evaluated for both languages. Even though it only uses standard, non-poetic text as input, the system yields state of the art results for poetry generation.
Automatic poetry generation is a challenging task for a computational system. For a poem to be meaningful, both linguistic and literary aspects need to be taken into account. First of all, a poetry generation system needs to properly model language phenomena, such as syntactic well-formedness and topical coherence. Furthermore, the system needs to incorporate various constraints (such as form and rhyme) that are related to a particular poetic genre. And finally, the system needs to exhibit a certain amount of literary creativity, which makes the poem interesting and worthwhile to read.
In recent years, a number of fruitful NLP approaches have emerged that are able to adequately model various aspects of natural language. In particular, neural network language models have improved the state of the art in language modeling, while topic models are successful at capturing clear-cut, semantic dimensions. In this paper, we explore
how these approaches can be adapted and combined in order to model both the linguistic and literary aspects that are required for poetry generation. More specifically, we make use of recurrent neural networks in an encoder-decoder configuration. The encoder first constructs a representation of an entire sentence by sequentially incorporating each word of the sentence into a fixed-size hidden state vector. The final representation is then given to the decoder, which emits a sequence of words according to a probability distribution derived from the hidden state of the input sentence. By training the network to predict the next sentence with the current sentence as input, the network learns to generate plain text with a certain discourse coherence. By modifying the probability distribution yielded by the decoder, we enforce the incorporation of poetic constraints, such that the network can be exploited for the generation of poetic verse. It is important to note that the poetry system is not trained on poetic texts; rather, the system is trained on a corpus of standard, prosaic texts extracted from the web, and it will be the constraints applied to the network's probability distribution that confer a poetic character to the generated verse.
The rest of this article is structured as follows. In section 2, we present an overview of related work on automatic poetry generation. Section 3 describes the different components of our model. In section 4, we present an extensive human evaluation of our model, as well as a number of examples generated by the system. Section 5, then, concludes and discusses some future research directions.
Early computational implementations that go beyond mere mechanical creativity have often relied on rule-based or template-based methods. One of the first examples is the ASPERA system (Gervás,
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2471-2480 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2001) for Spanish, which relies on a complex knowledge base, a set of rules, and case-based reasoning. Other approaches include Manurung et al. (2012), which combines rule-based generation with genetic algorithms, Gonçalo Oliveira (2012)'s PoeTryMe generation system, which relies on chart generation and various optimization strategies, and Veale (2013), which exploits metaphorical expressions using a pattern-based approach.
Whereas poetry generation with rule-based and template-based models has an inherent tendency to be somewhat rigid in structure, advances in statistical methods for language generation have opened up new avenues for a more varied and heterogeneous approach to creative language generation. Greene et al. (2010), for example, use an  $n$ -gram language model in combination with a rhythmic model implemented with finite-state transducers. And more recently, recurrent neural networks (RNNs) have been exploited for poetry generation; Zhang and Lapata (2014) use an encoder-decoder RNN for Chinese poetry generation, in which one RNN builds up a hidden representation of the current line in a poem, and another RNN predicts the next line word by word, based on the hidden representation of the current line. The system is trained on a corpus of Chinese poems. Yan (2016) tries to improve upon the encoder-decoder approach by incorporating a method of iterative improvement: the network constructs a candidate poem in each iteration, and the representation of the former iteration is used in the creation of the next one. And Wang et al. (2016) extend the method using an attention mechanism.
Ghazvininejad et al. (2016) combine RNNs (for syntactic fluency) with distributional similarity (for the modeling of semantic coherence) and finite state automata (for imposing literary constraints such as meter and rhyme). Their system, Hafez, is able to produce well-formed poems with a reasonable degree of semantic coherence, based on a user-defined topic. Hopkins and Kiela (2017) focus on rhythmic verse; they combine an RNN, trained on a phonetic representation of poems, with a cascade of weighted finite state transducers. Lau et al. (2018) present a joint neural network model for the generation of sonnets, called Deep-spare, that incorporates the training of rhyme and rhythm into the neural network; the network learns iambic stress patterns from data, while rhyming word pairs are
separated from non-rhyming ones using a margin-based loss. And a number of recent papers extend neural poetry generation for Chinese with various improvements, such as unsupervised style disentanglement (Yang et al., 2018), reinforcement learning (Yi et al., 2018), and rhetorical control (Liu et al., 2019).
Note that all existing statistical models are trained on or otherwise make use of a corpus of poetry; to our knowledge, our system is the first to generate poetry with a model that is exclusively trained on a generic corpus, which means the poetic character is endowed by the model itself. Secondly, we make use of a latent semantic model in order to model topical coherence, which is equally novel.
3.1 Neural architecture
The core of the poetry system is a neural network architecture, trained to predict the next sentence  $S_{i + 1}$  given the current sentence  $S_{i}$ . The architecture is made up of gated recurrent units (GRUs; Cho et al., 2014) that are linked together in an encoder-decoder setup. The encoder sequentially reads in each word  $w_{1,\dots,N}^{i}$  of sentence  $S_{i}$  (represented by its word embedding  $\mathbf{x}$ ) such that, at each time step  $t_i$ , a hidden state  $\hat{\mathbf{h}}_t$  is computed based on the current word's embedding  $\mathbf{x}_t$  and the previous time step's hidden state  $\hat{\mathbf{h}}_{t - 1}$ . For each time step, the hidden state  $\hat{\mathbf{h}}_t$  is computed according to the following equations:
$$
\mathbf {r} _ {t} = \sigma \left(\mathbf {W} _ {r} \mathbf {x} _ {t} + \mathbf {U} _ {r} \hat {\mathbf {h}} _ {t - 1}\right) \tag {1}
$$
$$
\mathbf {z} _ {t} = \sigma \left(\mathbf {W} _ {z} \mathbf {x} _ {t} + \mathbf {U} _ {z} \hat {\mathbf {h}} _ {t - 1}\right) \tag {2}
$$
$$
\bar {\mathbf {h}} _ {t} = \quad \tanh  (\mathbf {W} \mathbf {x} _ {t} + \mathbf {U} (\mathbf {r} _ {t} \odot \hat {\mathbf {h}} _ {t - 1})) \tag {3}
$$
$$
\hat {\mathbf {h}} _ {t} = \quad (1 - \mathbf {z} _ {t}) \odot \hat {\mathbf {h}} _ {t - 1} + \mathbf {z} _ {t} \odot \bar {\mathbf {h}} _ {t} \tag {4}
$$
where  $\mathbf{r}_t$  represents the GRU's reset gate,  $\mathbf{z}_t$  represents the update gate,  $\bar{\mathbf{h}}_t$  represents the candidate update state, and  $\odot$  represents pointwise multiplication.
$\hat{\mathbf{h}}_t$  can be interpreted as a representation of the sequence  $w_{1},\ldots ,w_{t}$  and the final hidden state  $\hat{\mathbf{h}}_N$  will therefore be a representation of the entire sentence. This final hidden encoder state is transferred to the decoder. The decoder then sequentially predicts the next sentence word by word, conditioned on the encoder's final hidden representation; at each time step  $t_{i + 1}$  the decoder equally computes a hidden state  $\mathbf{h}_t$  based on the current word's embedding  $\mathbf{x}_t$  (which was predicted by the decoder
Figure 1: Graphical representation of the poetry generation model. The encoder encodes the current verse, and the final representation is given to the decoder, which predicts the next verse word by word in reverse. The attention mechanism is represented for the first time step. The rhyme prior is applied to the first time step, and the topic prior is optionally applied to all time steps, mediated by the entropy threshold of the network's output distribution.
in the previous time step) and the previous time step's hidden state  $\mathbf{h}_{t - 1}$  (the first hidden state of the decoder is initialized by  $\hat{\mathbf{h}}_N$  and the first word is a symbolic start token). The computations for each time step  $\mathbf{h}_t$  of the decoder are equal to the ones used in the encoder (equations 1 to 4).
In order to fully exploit the entire sequence of representations yielded by the encoder, we augment the base architecture with an attention mechanism, known as general attention (Luong et al., 2015). The attention mechanism allows the decoder to consult the entire set of hidden states computed by the encoder; at each time-step—for the generation of each word in sentence  $S_{i+1}$ —the decoder determines which words in sentence  $S_i$  are relevant, and accordingly selects a linear combination of the entire set of hidden states. In order to do so, we first compute an attention vector  $\mathbf{a}_t$ , which attributes a weight to each hidden state  $\hat{\mathbf{h}}_i$  yielded by the encoder (based on the decoder's current hidden state  $\mathbf{h}_t$ ). According to equation 5:
$$
\mathbf {a} _ {t} (i) = \frac {\exp (\operatorname {s c o r e} \left(\mathbf {h} _ {t} , \hat {\mathbf {h}} _ {i}\right))}{\sum_ {i ^ {\prime}} \exp (\operatorname {s c o r e} \left(\mathbf {h} _ {t} , \hat {\mathbf {h}} _ {i ^ {\prime}}\right))} \tag {5}
$$
$$
\operatorname {s c o r e} \left(\mathbf {h} _ {t}, \hat {\mathbf {h}} _ {i}\right) = \mathbf {h} _ {t} ^ {T} \mathbf {W} _ {\mathbf {a}} \hat {\mathbf {h}} _ {i} \tag {6}
$$
The next step is to compute a global context vector  $\mathbf{c}_t$ , which is a weighted average (based on attention
vector  $\mathbf{a}_t$ ) of all of the encoder's hidden states. The resulting context vector is then combined with the original decoder hidden state in order to compute a new, attention-enhanced hidden state  $\tilde{\mathbf{h}}_t$ .
$$
\tilde {\mathbf {h}} _ {t} = \tanh  \left(\mathbf {W} _ {\mathbf {c}} \left[ \mathbf {c} _ {t}; \mathbf {h} _ {t} \right]\right) \tag {7}
$$
where  $[\cdot ;\cdot ]$  represents vector concatenation. Finally, this resulting hidden state  $\tilde{\mathbf{h}}_t$  is transformed into a probability distribution  $p(\mathbf{w}^t |w^{<  t},S_i)$  over the entire vocabulary using a softmax layer.
$$
p (\mathbf {w} ^ {t} | w ^ {<   t}, S _ {i}) = \operatorname {s o f t m a x} (\mathbf {W _ {s}} \tilde {\mathbf {h}} _ {t}) \qquad (8)
$$
As an objective function, the sum of the logprobabilities of the next sentence is optimized, conditioned on the hidden state representation of the current sentence.
$$
J _ {t} = \sum_ {\left(S _ {i}, S _ {i + 1}\right) \in C} - \log p \left(S _ {i} \mid S _ {i + 1}\right) \tag {9}
$$
At inference time, for the generation of a verse, each word is then sampled randomly according to the output probability distribution. Crucially, the decoder is trained to predict the next sentence in reverse, such that the last word of the verse is the first one that is generated. This reverse operation is important for an effective incorporation of rhyme, as will be explained in the next section. A graphical representation of the architecture, which includes the constraints discussed below, is given in Figure 1.
3.2 Poetic constraints as a priori distributions
As the neural architecture described above is trained on generic text, its output will in no way resemble poetic verse. In order to endow the generated output with a certain poetic character, we modify the neural network's output probability distribution through the application of a prior probability distribution, that constrains the standard output probability distribution, and boosts the probability of words that are a good fit within the defined constraints. We will consider two kinds of constraints: a rhyme constraint and a topical constraint.
3.2.1 Rhyme constraint
In order to adequately model the rhyme constraint, we make use of a phonetic representation of words, extracted from the online dictionary Wiktionary. For each word of the vocabulary, we determine its rhyme sound (i.e. the final group of vowels, optionally followed by a group of consonants), as well as the group of consonants that precedes the group of vowels. A sample of rhymes that are thus extracted is represented in Table 1.
word	rhyme
embrace	(mbr, eis)
suitcase	(tk, eis)
sacrifice	(f, ais)
paradise	(d, ais)
reproduit	(dq, i)
thérapie	(p, i)
examen	(m, ð)
canadien	(dj, ð)
Table 1: A number of rhyme examples extracted from Wiktionary, for both English and French.
The next step then consists in creating a probability distribution for a particular rhyme sound that we want the verse to adhere to:
$$
p _ {r h y m e} (\mathbf {w}) = \frac {1}{Z} \mathbf {x} \text {w i t h} \left\{ \begin{array}{l l} x _ {i} = 1 & \text {i f} i \in R \\ x _ {i} = \epsilon & \text {o t h e r w i s e} \end{array} \right. \tag {10}
$$
where  $R$  is the set of words that contain the required rhyme sound,  $\epsilon$  is a small value close to zero, used for numerical stability, and  $Z$  is a normalization factor in order to ensure a probability distribution. We can now use  $p_{\text{rhyme}}(\mathbf{w})$  as a prior probability distribution in order to reweight the neural network's standard output probability distribution—according to Equation 11—each time the rhyme
scheme demands it:
$$
p _ {\text {o u t}} (\mathbf {w}) = \frac {1}{Z} \left(p \left(\mathbf {w} ^ {t} \mid w ^ {<   t}, S _ {i}\right) \odot p _ {\text {r h y m e}} (\mathbf {w})\right) \tag {11}
$$
where  $\odot$  represents pointwise multiplication.2 As we noted before, each verse is generated in reverse; the reweighting of rhyme words is applied at the first step of the decoding process, and the rhyme word is generated first. This prevents the generation of an ill-chosen rhyme word that does not fit well with the rest of the verse.
3.2.2 Topical constraint
For the modeling of topical coherence, we make use of a latent semantic model based on nonnegative matrix factorization (NMF; Lee & Seung, 2001). Previous research has shown that nonnegative factorization methods are able to induce clear-cut, interpretable topical dimensions (Murphy et al., 2012). As input to the method, we construct a frequency matrix  $\mathbf{A}$ , which captures cooccurrence frequencies of vocabulary words and context words. This matrix is then factorized into two non-negative matrices  $\mathbf{W}$  and  $\mathbf{H}$ ,
$$
\mathbf {A} _ {i \times j} \approx \mathbf {W} _ {i \times k} \mathbf {H} _ {k \times j} \tag {12}
$$
where  $k$  is much smaller than  $i, j$  so that both instances and features are expressed in terms of a few components. Non-negative matrix factorization enforces the constraint that all three matrices must be non-negative, so all elements must be greater than or equal to zero. Using the minimization of the Kullback-Leibler divergence as an objective function, we want to find the matrices  $\mathbf{W}$  and  $\mathbf{H}$  for which the divergence between  $\mathbf{A}$  and  $\mathbf{W} \mathbf{H}$  (the multiplication of  $\mathbf{W}$  and  $\mathbf{H}$ ) is the smallest. The factorization is carried out through the iterative application of update rules. Matrices  $\mathbf{W}$  and  $\mathbf{H}$  are randomly initialized, and the rules in 13 and 14 are iteratively applied—alternating between them. In each iteration, each vector is adequately normalized, so that all dimension values sum to 1.
$$
\mathbf {H} _ {a \mu} \leftarrow \mathbf {H} _ {a \mu} \frac {\sum_ {i} \mathbf {W} _ {i a} \frac {\mathbf {A} _ {i \mu}}{(\mathbf {W H}) _ {i \mu}}}{\sum_ {k} \mathbf {W} _ {k a}} \tag {13}
$$
$$
\mathbf {W} _ {i a} \leftarrow \mathbf {W} _ {i a} \frac {\sum_ {\mu} \mathbf {H} _ {a \mu} \frac {\mathbf {A} _ {i \mu}}{(\mathbf {W H}) _ {i \mu}}}{\sum_ {v} \mathbf {H} _ {a v}} \tag {14}
$$
2Such a multiplicative combination of probability distributions is also known as a Product of Experts (Hinton, 2002).
3The raw frequencies are weighted using pointwise mutual information (Turney and Pantel, 2010).
1www.wiktionary.org
Tables 2 and 3 present a number of example dimensions induced by the model, for both English and French.
dim 13	dim 22	dim 28
sorrow	railway	planets
longing	trains	planet
admiration	rail	cosmic
earnest	station	universe
Table 2: Three example dimensions from the NMF model for English (4 words with highest probability)
dim 1	dim 20	dim 25
tendresse	gare	hypocrisie
joie	bus	mensonge
bonheur	métro	accuser
sourires	rer	hypocrite
Table 3: Three example dimensions from the NMF model for French (4 words with highest probability)
The factorization that comes out of the NMF model can be interpreted probabilistically (Gaussier and Goutte, 2005; Ding et al., 2008): matrix  $\mathbf{W}$  can be considered as  $p(\mathbf{w}|k)$ , i.e. the probability of a word given a latent dimension  $k$ . In order to constrain the network's output to a certain topic, it would be straightforward to simply use  $p(\mathbf{w}|k)$  as another prior probability distribution applied to each output. Initial experiments, however, indicated that such a blind modification of the output probability distribution for every word of the output sequence is detrimental to syntactic fluency. In order to combine syntactic fluency with topical consistency, we therefore condition the weighting of the output probability distribution on the entropy of that distribution: when the output distribution's entropy is low, the neural network is certain of its choice for the next word in order to generate a well-formed sentence, so we will not change it. On the other hand, when the entropy is high, we will modify the distribution by using the topical distribution  $p(\mathbf{w}|k)$  for a particular latent dimension as prior probability distribution—analogous to Equation 11—in order to inject the desired topic. The entropy threshold, above which the modified distribution is used, is set experimentally.
Note that the rhyme constraint and the topical constraint can straightforwardly be combined in order to generate a topical rhyme word, through pairwise multiplication of the three relevant distributions, and subsequent normalization in order to
ensure a probability distribution.
3.3 A global optimization framework
The generation of a verse is embedded within a global optimization framework. There are two reasons to integrate the generation of a verse within an optimization procedure. First of all, the generation of a verse is a sampling process, which is subject to chance. The optimization framework allows us to choose the best sample according to the constraints presented above. Secondly, the optimization allows us to define a number of additional criteria, that assist in the selection of the best verse. For each final verse, the model generates a considerable number of candidates; each candidate verse is then scored according to the following criteria:
- the log-probability score of the generated verse, according to the encoder-decoder architecture (section 3.1);
- compliance with the rhyme constraint (section 3.2.1); additionally, the extraction of the preceding group of consonants (cf. Table 1) allows us to give a higher score to rhyme words with disparate preceding consonant groups, which elicits more interesting rhymes;
- compliance with the topical constraint (section 3.2.2); the score is modeled as the sum of the probabilities of all words for the defined dimension;
- the optimal number of syllables, modeled as a Gaussian distribution with mean  $\mu$  and standard deviation  $\sigma$ ;
- the log-probability score of a standard  $n$ -gram model.
The score for each criterion is normalized to the interval [0, 1] using min-max normalization, and the harmonic mean of all scores is taken as the final score for each candidate. After generation of a predefined number of candidates, we keep the candidate with the highest score, and append it to the poem.
4We equally experimented with rhythmic constraints based on meter and stress, but initial experiments indicated that the system had a tendency to output very rigid verse. Simple syllable counting tends to yield more interesting variation.
5The harmonic mean is computed as  $\frac{n}{\sum_{i = 1}^{n}\frac{1}{x_i}}$  we choose this measure in order to balance the different scores.
4 Results and evaluation
4.1 Implementational details
We train two different models for the generation of poetry in both English and French. The neural architecture is trained on a large corpus of generic web texts, constructed on the basis of the CommonCrawl corpus.<sup>6</sup> In order to filter out noise and retain clean, orderly training data, we apply the following filtering steps:
- we only keep sentences written in the relevant language;
we only keep sentences of up to 20 words;
- we only keep sentences that contain at least one function word from a predefined list—the idea again is to filter out noisy sentences, and only keep well-formed, grammatical ones; we create a list of about 10 highly frequent function words, specific to each language;
- of all the sentences that remain after these filtering steps, we only keep the ones that appear successively within a document.
Using the filtering steps laid out above, we construct a training corpus of 500 million words for each language. We employ a vocabulary of  $15\mathrm{K}$  words (those with highest frequency throughout the corpus); less frequent words are replaced by an  $\langle \mathrm{unk}\rangle$  token, the probability of which is set to zero during generation.
Both encoder and decoder are made up of two GRU layers with a hidden state of size 2048, and the word embeddings are of size 512. Encoder, decoder, and output embeddings are all shared (Press and Wolf, 2017). Model parameters are optimized using stochastic gradient descent with an initial learning rate of 0.2, which is divided by 4 when the loss does no longer improve on a held-out validation set. We use a batch size of 64, and we apply gradient clipping. The neural architecture has been implemented using PyTorch (Paszke et al., 2017), with substantial reliance on the OpenNMT module (Klein et al., 2017). For the application of the topical constraint, we use an entropy threshold of 2.70.
The  $n$ -gram model is a standard Kneser-Ney smoothed trigram model implemented using KenLM (Heafield, 2011), and the NMF model is factorized to 100 dimensions. Both the  $n$ -gram