SlowGuess/ACL_2020_OCR huggingface dataset:
pinyin $^4$  to get the pronunciations (PinYin) of the words in the rhyming positions, and then conduct the evaluation. For Shakespeare's Sonnets corpus, the rhyming rule is clear "ABAB CDCD EFEF GG" and there are 7 groups of rhyming tokens. For the generated samples, we employ the CMU Pronouncing Dictionary $^5$  (Speech@CMU, 1998) to obtain the phonemes of the words in the rhyming positions. For example, the phonemes for word "asleep" and "steep" are ['AH0', 'S', 'L', 'IY1', 'P'] and ['S', 'T', 'IY1', 'P'] respectively. And then we can conduct the evaluation by counting the overlapping units from both the original words and the extracted phonemes group by group. We report the Macro-F1 and Micro-F1 numbers in the results tables as well. Integrity Since the format in our task is strict and
<sup>4</sup> http://github.com/mozillazg/python-pinyin
<sup>5</sup>http://wwwspeech.cs.cmu.edu/cgi-bin/cmudict
Model	PPL↓	Diversity (Distinct) ↑
VAL	TEST	MA-D-1	MI-D-1	MA-D-2	MI-D-2
SongNet	12.75	14.73	75.96	2.69	97.59	37.26
SongNet-GRU	16.52	20.49	74.73	1.77	98.30	28.98
SongNet w/o C	13.51	15.38	75.42	2.48	97.36	34.85
SongNet w/o P	14.16	17.16	73.73	2.56	97.52	34.82
SongNet w/ inverse-P	13.40	15.13	74.95	2.54	97.76	35.65
SongNet w/o S	13.23	15.44	75.38	2.74	97.31	37.50
Model	Format↑	Rhyme↑	Integrity↓
MA-F1	MI-F1	MA-F1	MI-F1
SongNet	99.81	99.83	79.23	78.63	2.14±0.10
SongNet-GRU	98.99	98.99	52.13	50.93	3.28±1.67
SongNet w/o C	84.73	85.39	78.59	78.24	1.77±0.53
SongNet w/o P	99.61	99.59	67.85	67.29	3.33±0.18
SongNet w/ inverse-P	99.68	99.69	65.89	65.43	2.24±0.21
SongNet w/o S	99.84	99.86	80.43	80.13	1.99±0.10
Table 4: Ablation analysis on SongCi
rigid, thus the number of words to be predicted is also pre-defined. Our model must organize the language using the limited positions, thus sentence integrity may become a serious issue. For example, the integrity of "love is not love.  $\langle /s\rangle$  is much better than"love is not the.  $\langle /s\rangle$  . To conduct the evaluation of sentence integrity, we design a straightforward method by calculating the prediction probability of the punctuation characters before  $\langle /s\rangle$  given the prefix tokens:
$$
I n t e g r i t y = 2 ^ {- \frac {1}{| Y |} \sum_ {i = 1} ^ {| Y |} \log \left(P \left(y _ {p u n c} ^ {i} \mid y _ {0} ^ {i}, y _ {1} ^ {i}, \dots , y _ {<   p u n c} ^ {i}\right)\right)} \tag {11}
$$
where  $Y$  is the generated sequence of sentences. Smaller integrity metric value indicates higher sentence quality. To achieve this goal, we conduct pre-trainings for two GPT2 (Radford et al., 2019) models on the large scale Chinese corpus and English corpus respectively. Then we utilize the GPT2 models to conduct the evaluation for sentence integrity.
Human Evaluations For SongCi, we sampled 50 samples for 25 CiPais. For Sonnet, the whole 27 samples in the test set are selected for human evaluation. We recruit three helpers to score the Relevance, Fluency, and Style. The rating criteria are as follows: Relevance: +2: all the sentences are relevant to the same topic; +1: partial sentences are relevant; 0: not relevant at all. Fluency: +2: fluent; +1: readable but with some grammar mistakes; 0: unreadable. Style: +2: match with SongCi or
Sonnet genres;  $+1$ : partially match; 0: mismatch.
4.4 Comparison Methods
S2S Sequence-to-sequence framework with attention mechanism (Bahdanau et al., 2014). We regard the format and rhyme symbols  $C$  as the input sequence, and the target as the output sequence.
GPT2 We fine-tune the GPT2 models (the pretraining versions are used for sentence integrity evaluation) on SongCi and Sonnet respectively.
SongNet Out proposed framework with both the per-training and fine-tuning stages.
We also conduct ablation analysis to verify the performance of the defined symbols as well as the variants of model structures.
- SongNet (only pre-tuning) Without the fine-tuning stage.
- SongNet (only fine-tuning) Without the pretraining stage.
- SongNet-GRU Employ GRU (Cho et al., 2014) to replace Transformer as the core structure.
- SongNet w/o C Remove the format and rhyme symbols  $C$ .
- SongNet w/o P Remove the intra-position symbols  $P$ .
- SongNet w/o S Remove the sentence segment symbols  $S$ .
- SongNet w/ inverse-P Arrange the intra-position indices in ascending order instead of the descending order.
Figure 3: Parameter tuning of  $k$  on the metrics of Rhyme, Integrity, and Micro-Dist-2.
Model	Cases of Generated Results
SongNet-SongCi	CiPai: Zhe Gu Tian, Format: 7.7.7, 7.3, 3.7.7, 7.
南枝欲破春寒浅(qian)。东风一霎春犹暖(nuan)。海棠零乱暗香残(fan),露颗未消红泪满(man)。春不曾(guan),人何远(yuan)。江头旧日清溪畔(ban)。欲凭楼上数行书,却为梦魂无觅畔(ban)。
CiPai: Bu Suan Zi, Format: 5, 5.7, 5.5, 5.7, 5.
新月挂帘旌,暗草蚤凄楚(chu)。夜半银潢十二阑,画幕灯花吐(tu)。何处笛声残,晓窗风雨(yu)。独许寒梅伴小鬓,香在秋千桂(zhu)。
CiPai: Self-Defined, Format: 3, 3, 5.3, 3, 5.7, 7.
翠岩中,流水外,别有小壶天(tian)。自霜清,独斗芳,谁敢并飞仙(xian)。夜久凉生一曲愁,月如花影似花圆(yuan)。
CiPai: Self-Defined, Format: 9.9.9.9.
雨洒黄泉粼粼细作枣(han)。越水西来时节自清风(xian)。歌笑一杯长唱醉醒问(jian)。春去不堪寻梦绕舍山(shan)。
SongNet-Sonnet	how do you hold such a thing like this, \ when my eyes are so not black? \ but how can i show myself, so strange, \ that all this black is white?
where am i to hide this from my eyes, \ from this white mine eyes all fals, \ where is the good fortune, in me, \ that hath no excuse, no excuse?
what is that which can mask the true love \ and for whom is this true love more? \ the one, which shall save the poor my eye, \ from the false truth of my judgment?
what lies, for when you are not that, \ no one in this and that can see me lies!
Table 5: Cases of the generated results for SongCi and Sonnet respectively. For SongCi, the number in Format (e.g., 3,5,7) denotes the number of tokens in one sentence. The rhyming words are labeled in red color and italic font following is the Pinyin. (Since cases are provided to confirm the format consistency, thus we did not conduct translation for the Chinese samples. Translation for Chinese poetry is also a challenging task.)
Model	Cases of Generated Results Given the Formats with Partial Content
SongNet-SongCi	CiPai: Bu Suan Zi, Format: 5, 5.7, 5.5, 5.7, 5. Format C: _ to _ . 俏. _ . 报. _ . 笑。 (1)风暖莺声苦,又是年时到。长爱柳色三分里,多少为春俏。共少年时节,惯得花时报。争似赊径入廛来,步步随春笑。 (2)春来春又去,花影几波到。门外绿杨柳潇洒,溪桥人人俏。门外东风马,似旧时香报。但怪桃李晚时羞,冷落无人笑。 Format C: _ 到 _ 冰, 俏. _ 春, 春报. _ 山花, _ 笑。 (1)春雨碧连营,柳暗门巷到。宫殿步辇踏寒冰,百媚生春俏。正日烘晴春,花雾春风报。一点山花醉眼明,半倚阑干笑。 (2)雪雨弄轻暖,新涨绿波到。楼上对面湿翠冰,一夜风花俏。晓日烘长春,睡起春宵报。枕畔山花落叶声,不是梦魂笑。
SongNet-Sonnet	_ with _ hearts , _ lacking _ dead ; _ love _ parts ,and _ buried . _ many _ tear, hath _ eye , _ now appear , _ thee lie ! _ buried _ live , _ of _ gone , _ parts _ give , _ thine alone : _ view _ thee , _ all _ me .	though all thy love with thy hearts , thou still are lacking of my dead ; if thy love love is lost to your love and parts , and yet mine own heart can be buried . so many are ill or in tear, hath not this time that we will make their eye , for that which lies not well hath now appear, no longer nor the world that holds thee lie ! for if it would be buried in my live , or by the earth of mine was gone , then my own parts as my body and mine give , may not be so far beyond thine alone : so far as thee and this world view find thee , then mine life be far enough from all thee and no me .
Table 6: Cases of the generated results given the formats with partial pre-defined content. Format token “_” needs to be translated to real word token.
Please note that we mainly employ top- $k$  sampling method (Fan et al., 2018; Radford et al., 2019) to conduct the generation, and we let  $k = 32$  here. The parameter tuning of  $k$  is described in Section 5.3.
Table 1 and Table 2 depict the experimental results of SongNet as well as the baseline methods S2S and GPT2 on corpus SongCi and Sonnet respectively. It is obvious that our pre-training and fine-tuning framework SongNet obtain the best per
formance on most of the automatic metrics. Especially on the metric of Format accuracy, SongNet can even obtain a  $98\%+$  value which means that our framework can conduct the generation rigidly matching with the pre-defined formats. On the metric of PPL, Rhyme accuracy, and sentence integrity, SongNet also performs significantly better in a large gap than the baseline methods such as S2S and GPT2 as well as the model variants only with the pre-training or fine-tuning stage.
Another observation is that some of the results on corpus Sonnet are not as good as the results
Model	Relevance	Fluency	Style
SongNet-SongCi	1.36	1.45	2.00
SongNet-Sonnet	0.58	0.42	0.83
on SongCi. The main reason is that Sonnet only contains 100 samples in the training set as shown in Table 3. Therefore, the model cannot capture sufficient useful features especially for the rhyming issue.
We conduct ablation study on corpus SongCi and the experimental results are depicted in Table 4. It should note that all the models are purely trained on SongCi corpus without any pre-training stages. From the results we can conclude that the introduced symbols  $C$ ,  $P$ , and  $S$  indeed play crucial roles in improving the overall performance especially on the metrics of format, rhyme, and sentence integrity. Even though some of the components can not improve the performance simultaneously on all the metrics, the combination of them can obtain the best performance.
5.3 Parameter Tuning
Since we employ top- $k$  sampling as our main decoding strategy, thus we design several experiments to conduct the parameter tuning on  $k$ . We let  $k$  to be 1, 5, 10, 20, 50, 500 respectively. We also provide the beam-search (beam=5) results for comparing and reference.
The parameter tuning results are depicted in Figure 3. From the results we can observe that large  $k$  can increase the diversity of the results significantly. But the Rhyme accuracy and the sentence integrity will drop simultaneously. Therefore, in the experiments we let  $k = 32$  to obtain a trade-off between the diversity and the general quality.
5.4 Human Evaluation
For human evaluation, we just conduct the judging on the results generated by our final model SongNet. From the result we can observe that the results on corpus SongCi is much better than the ones on corpus Sonnet, which is because the corpus scale is different. And the small scale also lead to dramatically dropping on all the metrics.
5.5 Case Analysis
Table 5 depicts several generated cases for SongCi and Sonnet respectively. For SongCi, the formats (CiPai) are all cold-start samples which are not in the training set or even newly defined. Our model can still generate high quality results on the aspects of format, rhyme as well as integrity. However, for corpus Sonnet, even though the model can generate 14 lines text, the quality is not as good as SongCi due to the insufficient training-set (only 100 samples). We will address this interesting and challenging few-shot issue in the future.
In addition, we mentioned that our model has the ability of refining and polishing given the format  $C$  which contains some fixed text information. The examples of the generated results under this setting are shown in Table 6, which show that our model SongNet can generate satisfying results especially on SongCi.
We propose to tackle a challenging task called rigid formats controlled text generation. A pre-training and fine-tuning framework SongNet is designed to address the problem. Sets of symbols are tailor-designed to improve the modeling performance for format, rhyme, and sentence integrity. Extensive experiments conducted on two collected corpora demonstrate that our framework generates significantly better results in terms of both automatic metrics and human evaluations given arbitrary cold start formats.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1243-1252. JMLR.org.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Philipp Koehn. 2004. Pharaoh: a beam search decoder for phrase-based statistical machine translation models. In Conference of the Association for Machine Translation in the Americas, pages 115-124. Springer.
Jey Han Lau, Trevor Cohn, Timothy Baldwin, Julian Brooke, and Adam Hammond. 2018. Deep-spare: A joint neural model of poetic language, meter and rhyme. arXiv preprint arXiv:1807.03491.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Piji Li. 2020. An empirical investigation of pre-trained transformer language models for open-domain dialogue generation. arXiv preprint arXiv:2003.04195.
Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. 2017. Deep recurrent generative decoder for abstractive text summarization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2091-2100.
Yi Liao, Yasheng Wang, Qun Liu, and Xin Jiang. 2019. Gpt-based generation for classical chinese poetry. arXiv preprint arXiv:1907.00151.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? arXiv preprint arXiv:1909.10705.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
William Shakespeare. 2000. Shakespeare's sonnets. Yale University Press.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577-1586.
Speech@CMU. 1998. Carnegie-mellon university pronouncing dictionary for american english. Version 0.7b. Available at [http://www.speech.cs.cmu.edu/cgi-bin/cmudict].
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Xingxing Zhang and Mirella Lapata. 2014. Chinese poetry generation with recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 670-680.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27.