SlowGuess/ACL_2020_OCR huggingface dataset:
model and the NMF model are trained on a large, 10 billion word corpus, equally constructed from web texts without any filtering steps except for language identification. For syllable length, we use  $\mu = 12$ ,  $\sigma = 2$ .
We generate about 2000 candidates for each verse, according to a fixed rhyme scheme (ABAB CDCD). Note that no human selection whatsoever has been applied to the poems used in the evaluation; all poems have been generated in a single run, without cherry picking the best examples. Four representative examples of poems generated by the system are given in Figure 2.
4.2 Evaluation procedure
Quantitatively evaluating creativity is far from straightforward, and this is no less true for creative artefacts that are automatically generated. Automatic evaluation measures that compute the overlap of system output with gold reference texts (such as BLEU or ROUGE), and which might be used for the evaluation of standard generation tasks, are of little use when it comes to creative language generation. The majority of research into creative language generation therefore makes use of some form of human evaluation, even though one needs to keep in mind that the evaluation of textual creativity is an inherently subjective task, especially with regard to poetic value. For a discussion of the subject, see Gonçalo Oliveira (2017).
We adopt the evaluation framework by Zhang and Lapata (2014), in which human annotators are asked to evaluate poems on a five point scale with regard to a number of characteristics, viz.
- fluency: is the poem grammatical and syntactically well-formed?
- coherence: is the poem thematically structured?
meaningfulness: does the poem convey a meaningful message to the reader?
- *poeticness: does the text display the features of a poem?
Additionally, we ask annotators to judge if the poem is written by a human or a computer.
In total, we evaluate four different sets of poems, yielded by different model instantiations. The different sets of poems considered during evaluation are:
$^{6}$ commoncrawl.org
At the moment it seems almost impossible  
Yet life is neither good nor evil  
The divine mind and soul is immortal  
In other words, the soul is never ill
So far, it has barely lost its youthful look. But no man is ever too young for the rest. He thought deeply, and yet his heart shook. At that moment he seemed utterly possessed
Figure 2: Four representative examples of poems generated by the system; the left-hand poems, in English, are respectively generated using dimensions 13 and 28 (cf. Table 2); the right-hand poems, in French, are generated using dimensions 1 and 25 (cf. Table 3).
The moon represents unity and brotherhood  
The earth stands in awe and disbelief  
Other planets orbit the earth as they should  
The universe is infinite and brief
The sky has been so bright and beautiful so far  
See the moon shining through the cosmic flame  
See the stars in the depths of the earth you are  
The planet the planet we can all see the same
Malgre mon enthousiasme, le chagrin s'allonge  
Le bonheur est toujours superbe  
Toi, tu es un merveilleuxSONge  
Je te vois rver de bonheur dans l'herbe
Tu trouveraslebonheur de tesrèves Je t'aime comme tout le monde Je t'aime mon amour,je me lève Je ressens pourtoi une joie profonde
Rien ne prouve qu'il s'indigne Dans le cas contraire, ce n'est pas grave Si la ✓erite est fausse, c'est très mauvais signe Il est vrai que lesgens le savent
Et cela est faux, mais qu'importe  
En fait, le mensonge, c'est l'effroi  
La négation de l'homme en quellesorte  
Le tort n'est pas de penser cela, il est magistrat
-  $rnn$ : poems generated by the neural architecture defined in section 3.1, without any added constraints;
- rhyme: poems generated by the neural architecture, augmented with the rhyme constraint;
-  $nmf_{rand}$ : poems generated by the neural architecture, augmented with both the rhyme constraint and the topical constraint, where one of the automatically induced NMF dimensions is selected randomly;
-  $nmf_{spec}$ : poems generated by the neural architecture, augmented with both the rhyme constraint and the topical constraint, where one of the automatically induced NMF dimensions is specified manually.
For a proper comparison of our system, we equally include:
- random: poems yielded by a baseline model where, for each verse, we select a random sentence (that contains between 7 and 15 words) from a large corpus; the idea is that the lines selected by the baseline model should be fairly fluent (as they come from an actual corpus), but lacking in coherence (due to their random selection);
- human: poems written by human poets; the scores on this set of poems function as an upper bound;
- Hafez and Deep-spare: poems generated by two state of the art poetry generation systems for English, respectively by Ghazvininejad et al. (2016) and Lau et al. (2018); we use the code made available by the respective authors. Note that we only compare to other poetry generation systems for English, as no other readily available systems exist for French.
4.3 Results for English
For English, 22 annotators evaluated 40 poems in total (5 poems for each of the different sets considered in the evaluation; each poem was evaluated by at least 4 annotators). The annotators consist of native speakers of English, as well as master students in English linguistics and literature. For the human set, we select five poems by well-established English poets that follow the same rhyme scheme as the generated ones. For  $nmf_{spec}$ , we select dimension 13 of Table 2. The results of the evaluation for English are presented in the upper part of Table 4.
First of all, note that all our model instantiations score better than the random baseline model,
This can be regarded as manually defining the theme of the generated poem. The specified dimension is selected for its poetic character.
Hafez needs to be initialized with user-defined topics; for a fair comparison, we seed the system with the top words of the NMF dimension used for our best performing model.
The selected poets are W.H. Auden, E.E. Cummings, Philip Larkin, Sarojini Naidu, and Sylvia Plath.
English
model	fluency	coherence	meaningfulness	poeticness	written by human (%)
rnn	2.95	2.50	2.45	2.55	0.18
rhyme	3.41	2.77	2.82	2.95	0.59
nmfrand	3.32	3.09	2.86	2.95	0.32
nmfspec	3.64	3.41	3.27	3.86	0.55
random	2.68	2.09	1.91	2.41	0.14
Deep-spare	2.11	2.00	2.00	3.00	0.22
Hafez	3.44	3.11	3.11	3.50	0.53
human	3.73	3.73	3.68	4.00	0.73
French
model	fluency	coherence	meaningfulness	poeticness	written by human (%)
rnn	3.45	2.73	2.59	2.55	0.27
rhyme	3.82	2.55	2.18	3.23	0.14
nmfrand	3.64	3.32	3.09	2.86	0.27
nmfspec	3.82	3.82	3.55	3.95	0.45
random	2.95	1.86	1.68	2.18	0.00
human	4.59	4.59	4.50	4.81	0.95
Table 4: Results of the human evaluation (mean score of all annotators) for English and French; values in bold indicate best performance of all generation models
even with regard to grammatical fluency. The good scores on fluency for the constrained models indicate that the applied constraints do not disrupt the grammaticality of the generated verse (rhyme is significantly different<sup>10</sup> with  $p < 0.05$ ;  $nmf_{rand}$  and  $nmf_{spec}$  with  $p < 0.01$ ; recall that the baseline consists of actual sentences from a corpus). Secondly, we note that the rhyme constraint seems to improve poeticness (though not significantly), while the topical constraint seems to improve both coherence ( $p < 0.01$  for  $nmf_{spec}$ ) and meaningfulness (not significantly). Interestingly, a large proportion of the poems produced by the rhyme model are labeled as human, even though the other scores are fairly low. The score for poeticism is considerably higher ( $p < 0.01$ ) for  $nmf_{spec}$  (with a manually specified theme selected for its poeticism) than for  $nmf_{rand}$  (with a randomly selected topic, which will often be more mundane). And the best scores on all criteria are obtained with the  $nmf_{spec}$  model, for which more than half of the poems are judged to be written by a human; moreover, the difference between  $nmf_{spec}$  and human poetry is not significant. Finally, our poetry generation compares favourably to previous work:  $nmf_{spec}$  scores markedly and significantly better than  $Deep-spare$  (which does not differ significantly from the random baseline), and it equally attains better scores than  $Hafez$  on all
four criteria (though not significantly so).
4.4 Results for French
The setup of the French evaluation is analogous to the English one: an equal number of 22 annotators have evaluated a total of 30 poems (5 poems for each of the six sets considered in the evaluation; each poem was evaluated by at least 4 annotators). The annotators are all native speakers of French. For the human poems, we select five poems with the same rhyme scheme as the generated ones, among the highest ranked ones on short-edition.com—a website with submissions by amateur poets. For  $nmf_{spec}$ , we select dimension 1 of Table 3. The results for French are presented in the lower part of Table 4.
Generally speaking, we see that the results for French confirm those for English. First of all, all model instantiations obtain better scores than the random baseline model, even with regard to fluency  $(p < 0.01)$ , again confirming that the application of the rhyme constraint and topical constraint are not detrimental to the grammaticality of the verse. Secondly, the rhyme constraint significantly improves the score for poeticness  $(p < 0.05$  compared to  $rnn)$ , while the topical constraint improves both coherence  $(p < 0.05)$  and meaningfulness  $(p < 0.01)$ . Contrary to the English results, only a small proportion of poems from the rhyme model are thought to be human. We do again see that the score for poeticness is considerably higher  $(p < 0.01)$  for
${}^{10}$  Significance testing is carried out using a two-tailed permutation test.
$nmf_{spec}$  than for  $nmf_{rand}$ , which seems to indicate that the topic of a poem is an important factor in people's judgements on poeticness. Finally, we again see that the best scores on all criteria are obtained with  $nmf_{spec}$ , for which almost half of the poems are judged to be written by a human.
We presented a system for automatic poetry generation that is trained exclusively on standard, non-poetic text. The system uses a recurrent neural encoder-decoder architecture in order to generate candidate verses, incorporating poetic and topical constraints by modifying the output probability distribution of the neural network. The best verse is then selected for inclusion in the poem, using a global optimization framework. We trained the system on both English and French, and equally carried out a human evaluation for both languages. The results indicate that the system is able to generate credible poetry, that scores well with regard to fluency and coherence, as well as meaningfulness and poeticness. Compared to previous systems, our model achieves state of the art performance, even though it is trained on standard, non-poetic text. In our best setup, about half of the generated poems are judged to be written by a human.
We conclude with a number of future research avenues. First of all, we would like to experiment with different neural network architectures. Specifically, we believe hierarchical approaches (Serban et al., 2017) as well as the Transformer network (Vaswani et al., 2017) would be particularly suitable to poetry generation. Secondly, we would like to incorporate further poetic devices, especially those based on meaning. Gripping poetry often relies on figurative language use, such as symbolism and metaphor. A successful incorporation of such devices would mean a significant step towards truly inspired poetry generation. And finally, we would like to adapt the model for automatic poetry translation—as we feel that the constraint-based approach lends itself perfectly to a poetry translation model that is able to adhere to an original poem in both form and meaning.
In order to facilitate reproduction of the results and encourage further research, the poetry generation system is made available as open source software. The current version can be downloaded at https://github.com/timvdc/poetry.
This work is supported by a grant overseen by the French National Research Agency ANR (project QUANTUM - ANR-19-CE23-0025); it has equally benefited from a GPU donated by NVIDIA Corporation.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734. Association for Computational Linguistics.
Chris Ding, Tao Li, and Wei Peng. 2008. On the equivalence between non-negative matrix factorization and probabilistic latent semantic indexing. Computational Statistics & Data Analysis, 52(8):3913-3927.
Eric Gaussier and Cyril Goutte. 2005. Relation between plsa and nmf and implications. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 601-602. ACM.
Pablo Gervás. 2001. An expert system for the composition of formal Spanish poetry. In Applications and Innovations in Intelligent Systems VIII, pages 19-32, London. Springer.
Marjan Ghazvininejad, Xing Shi, Yejin Choi, and Kevin Knight. 2016. Generating topical poetry. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1183-1191, Austin, Texas. Association for Computational Linguistics.
Hugo Gonçalo Oliveira. 2012. Poetryme: a versatile platform for poetry generation. Computational Creativity, Concept Invention, and General Intelligence, 1:21.
Hugo Gonçalo Oliveira. 2017. A survey on intelligent poetry generation: Languages, features, techniques, reutilisation and evaluation. In Proceedings of the 10th International Conference on Natural Language Generation, pages 11-20.
Erica Greene, Tugba Bodrumlu, and Kevin Knight. 2010. Automatic analysis of rhythmic poetry with applications to generation and translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 524-533. Association for Computational Linguistics.
Kenneth Heafield. 2011. KenLM: faster and smaller language model queries. In Proceedings of the
EMNLP 2011 Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland, United Kingdom.
Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. *Neural computation*, 14(8):1771-1800.
Jack Hopkins and Douwe Kiela. 2017. Automatically generating rhythmic verse with neural networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 168-178. Association for Computational Linguistics.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72. Association for Computational Linguistics.
Jey Han Lau, Trevor Cohn, Timothy Baldwin, Julian Brooke, and Adam Hammond. 2018. Deep-spare: A joint neural model of poetic language, meter and rhyme. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1948-1958. Association for Computational Linguistics.
Daniel D Lee and H Sebastian Seung. 2001. Algorithms for non-negative matrix factorization. In Advances in neural information processing systems, pages 556-562.
Zhiqiang Liu, Zuohui Fu, Jie Cao, Gerard de Melo, Yik-Cheung Tam, Cheng Niu, and Jie Zhou. 2019. Rhetorically controlled encoder-decoder for modern Chinese poetry generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1992-2001, Florence, Italy. Association for Computational Linguistics.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421. Association for Computational Linguistics.
Ruli Manurung, Graeme Ritchie, and Henry Thompson. 2012. Using genetic algorithms to create meaningful poetic text. Journal of Experimental & Theoretical Artificial Intelligence, 24(1):43-64.
Brian Murphy, Partha Talukdar, and Tom Mitchell. 2012. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proceedings of COLING 2012, pages 1933-1950. The COLING 2012 Organizing Committee.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163. Association for Computational Linguistics.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
Peter D Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research, 37:141-188.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Tony Veale. 2013. Less rhyme, more reason: Knowledge-based poetry generation with feeling, insight and wit. In Proceedings of the international conference on computational creativity, pages 152-159.
Qixin Wang, Tianyi Luo, Dong Wang, and Chao Xing. 2016. Chinese song iambics generation with neural attention-based model. In Proceedings of International Joint Conference on Artificial Intelligence, pages 2943-2949.
Rui Yan. 2016. i, poet: Automatic poetry composition through recurrent neural networks with iterative polishing schema. In Proceedings of International Joint Conference on Artificial Intelligence, pages 2238-2244.
Cheng Yang, Maosong Sun, Xiaoyuan Yi, and Wenhao Li. 2018. Stylistic Chinese poetry generation via unsupervised style disentanglement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3960-3969, Brussels, Belgium. Association for Computational Linguistics.
Xiaoyuan Yi, Maosong Sun, Ruoyu Li, and Wenhao Li. 2018. Automatic poetry generation with mutual reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3143-3153, Brussels, Belgium. Association for Computational Linguistics.
Xingxing Zhang and Mirella Lapata. 2014. Chinese poetry generation with recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 670-680. Association for Computational Linguistics.