SlowGuess/ACL_2020_OCR huggingface dataset:
Piji Li Haisong Zhang Xiaojiang Liu Shuming Shi
{pijili,hansonzhang,kieranliu,shumingshi}@tencent.com
Neural text generation has made tremendous progress in various tasks. One common characteristic of most of the tasks is that the texts are not restricted to some rigid formats when generating. However, we may confront some special text paradigms such as Lyrics (assume the music score is given), Sonnet, SongCi (classical Chinese poetry of the Song dynasty), etc. The typical characteristics of these texts are in three folds: (1) They must comply fully with the rigid predefined formats. (2) They must obey some rhyming schemes. (3) Although they are restricted to some formats, the sentence integrity must be guaranteed. To the best of our knowledge, text generation based on the predefined rigid formats has not been well investigated. Therefore, we propose a simple and elegant framework named SongNet to tackle this problem. The backbone of the framework is a Transformer-based auto-regressive language model. Sets of symbols are tailor-designed to improve the modeling performance especially on format, rhyme, and sentence integrity. We improve the attention mechanism to impel the model to capture some future information on the format. A pre-training and fine-tuning framework is designed to further improve the generation quality. Extensive experiments conducted on two collected corpora demonstrate that our proposed framework generates significantly better results in terms of both automatic metrics and the human evaluation.<sup>1</sup>
Recent years have seen the tremendous progress in the area of natural language generation especially benefiting by the neural network models such as Recurrent Neural Networks (RNN) or Convolutional Neural Networks (CNN) based sequence-to-sequence (seq2seq) frameworks (Bahdanau et al.,
Figure 1: Examples of text with rigid formats. In lyrics, the syllables of the lyric words must align with the tones of the notation. In SongCi and Sonnet, there are strict rhyming schemes and the rhyming words are labeled in red color and italic font.
2014; Gehring et al., 2017), Transformer and its variants (Vaswani et al., 2017; Dai et al., 2019), pre-trained auto-regressive language models such as XLNet (Yang et al., 2019) and GPT2 (Radford et al., 2019), etc. Performance has been improved significantly in lots of tasks such as machine translation (Bahdanau et al., 2014; Vaswani et al., 2017), dialogue systems (Vinyals and Le, 2015; Shang et al., 2015; Li, 2020), text summarization (Rush et al., 2015; Li et al., 2017; See et al., 2017), story telling (Fan et al., 2018; See et al., 2019), poetry writing (Zhang and Lapata, 2014; Lau et al., 2018; Liao et al., 2019), etc.
Generally, most of the above mentioned tasks can be regarded as free text generation, which means that no constraints on the format and structure, say the number of words and rhyming rules. Note that tasks of dialogue generation and story telling are almost in an open-ending generation style as long as the generated content is relevant with the conditional input text. Although there are
'Code: http://github.com/lipiji/SongNet
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 742-751 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
format constraints on the poetry text, the proposed models just treat the formats as kind of latent information and let the model capture this feature implicitly during training (Liao et al., 2019). The model trained on the five-character quatrain corpus cannot generate seven-character verses. Moreover, it is impossible to trigger these models to generate satisfying results according to arbitrary new defined formats.
In practice we will confront some special text paradigms such as Lyrics (assume the music score is given), Sonnet (say Shakespeare's Sonnets (Shakespeare, 2000)), SongCi (a kind of Ci. Ci is a type of lyric poetry in the tradition of Classical Chinese poetry. $^{2}$ , SongCi is the Ci created during Song dynasty), etc., and some examples are illustrated in Figure 1. The typical characteristics of these text can be categorized into three folds: (1) The assembling of text must comply fully with the predefined rigid formats. Assume that the music score is composed, then the lyricist must fill the lyric content strictly tally with the schemes lie in the notation. Take partial of song "Edelweiss" as shown in the first row of Figure 1 as example, the syllables of the lyric words must align with the tones of the notation. The second row of Figure 1 depicts the content of a SongCi created based on the CiPai of "Bu Suan Zi". Given the CiPai, the number of characters and the syntactical structure of the content are also defined (e.g., the number of characters of each clause: 5, 5. 7, 5. 5, 5. 7, 5.). (2) The arrangement of the content must obey the defined rhyming schemes. For example, all the final words (words in red color and italic font) of the SongCi content in Figure1 are rhyming (the spelling of each word is: "zhu", "yu", "du", and "gu"). The example in the third row of Figure 1 comes from Shakespeare's "Sonnet 116" (Shakespeare, 2000), the first four sentences. Usually, the rhyming schemes of Shakespeare's Sonnets is "ABAB CDCD EEFEGG". In the example, the rhyming words in scheme "ABAB" are "minds", "love", "finds", and "remove". (3) Even though the format is rigid, the sentence integrity must always be guaranteed. Incomplete sentence such as "love is not the" is inappropriate.
To the best of our knowledge, text generation based on the predefined rigid formats constraints has not been well investigated yet. In this work,
we propose a simple and elegant framework named SongNet to address this challenging problem. The backbone of the framework is a Transformer-based auto-regressive language model. Considering the three folds characteristics mentioned above, we introduce sets of tailor-designed indicating symbols to improve the modeling performance, especially for the robustness of the format, rhyme, as well as sentence integrity. We improve the attention mechanism to impel the model to capture the future information on the format to further enhance sentence integrity. Inspired by BERT (Devlin et al., 2019) and GPT (Radford et al., 2018, 2019), a pretraining and fine-tuning framework is designed to further improve the generation quality. To verify the performance of our framework, we collect two corpora, SongCi and Sonnet, in Chinese and English respectively. Extensive experiments on the collected datasets demonstrate that our proposed framework can generate satisfying results in terms of both the tailor-designed automatic metrics including format accuracy, rhyming accuracy, sentence integrity, as well as the human evaluation results on relevance, fluency, and style.
- We propose to tackle a new challenging task: rigid formats controlled text generation. A pre-training and fine-tuning framework named SongNet is designed to address the problem.
- Sets of symbols are tailor-designed to improve the modeling performance. We improve the attention mechanism to impel the model to capture the future information to further enhance the sentence integrity.
- To verify the performance of our framework SongNet, we collect two corpora, SongCi and Sonnet, in Chinese and English respectively. We design several automatic evaluation metrics and human evaluation metrics to conduct the performance evaluation.
- Extensive experiments conducted on two collected corpora demonstrate that our proposed framework generates significantly better results given arbitrary formats, including the cold-start formats or even the formats newly defined by ourselves.
The task of rigid formats controlled text generation is defined as follows:
<sup>2</sup> http://en.wikipedia.org/wiki/Ci_(poetry)
<sup>3</sup> http://en.wikipedia.org/wiki/Shakespeare%27s_sonnets
Figure 2: The framework of our proposed model.
Input: a rigid format  $C \in \mathcal{C}$ :
$$
C = \left\{c _ {0} c _ {1} c _ {2} c _ {3}, c _ {0} c _ {1} c _ {2} c _ {3} c _ {4} c _ {5}. \right\} \tag {1}
$$
where  $\mathcal{C}$  is the set of all possible formats. Note that we can define arbitrary new formats not restricted to the ones pre-defined in the corpus, thus  $|\mathcal{C}| \to \infty$ . Format token  $c_{i}$  denotes a place-holder symbol of  $C$  which need to be translated into a real word token. Format  $C$  contains 10 words plus two extra punctuation characters “,” and “.”
Output: a natural language sentence  $Y \in \mathcal{V}$  which tally with the defined format  $C$ :
$$
Y = \text {l o v e}
$$
bends with the remover to remove.
where the example sentences are extracted from the Shakespeare's Sonnets (Shakespeare, 2000). From the result  $Y$  we can observe that the count of words is 10 which is consistent with the format  $C$ . The punctuation characters "," and ":" are also correct. Thus, we claim that it is a  $100\%$  format accuracy result. Also, since the two clause sentences are complete, we can get a good sentence integrity score. If  $\mathcal{C}$  is defined on the literary genres of SongCi or Sonnet which have rhyming constraints, the rhyming performance should be evaluated as well. Recall that  $\mathcal{C}$  can be arbitrary and flexible, thus we can rebuild a new format  $C'$  based on the generated result  $Y$  by masking partial content, say  $C' = \{c_0 c_1 c_2 \text{ love}, c_0 c_1 c_2 c_3 c_4 \text{ remove}\}$ , then we may obtain better results by re-generating based on  $C'$ . We name this operation as polishing.
Finally, the target of this problem is to find a mapping function  $G$  to conduct the rigid formats controlled text generation:
$$
Y = G (C) \tag {2}
$$
3 Framework Description
As shown in Figure 2, the backbone of our framework is a Transformer-based auto-regressive language model. The input can be the whole token sequences of samples from SongCi or Sonnet. We tailor-design several sets of indicating symbols to enhance the performance in terms of accuracy on format, rhyme, and sentence integrity. Specifically, symbols  $C = \{c_i\}$  are introduced for format and rhyming modeling; Intra-position symbols  $P = \{p_i\}$  are designed to represent the local positions of the tokens within each sentence aiming to improve the rhyming performance and the sentence integrity. Segment symbols  $S = \{s_i\}$  are employed to identify the sentence border to further improve the sentence quality. Attention mechanism is improved to impel the model to capture the future format information such as the sentence ending markers. Similar to BERT (Devlin et al., 2019) and GPT (Radford et al., 2018, 2019), pre-training and fine-tuning paradigm is utilized to boost the performance of the original models.
3.2 Details
We use two sentences (as shown in Figure 1) "love is not love, ..., bends with the remover to remove" extracted from the Shakespeare's Sonnets (Shakespeare, 2000) as examples to describe the details of our framework SongNet. Since our basic model is a Transformer-based auto-regressive language model, during training, the input is "  $\langle bos\rangle$  love is not love,  $\langle /s\rangle$  ..., bends with the remover to remove.  $\langle /s\rangle$  ", and the corresponding output is a left-shifting version of the input (tokenized, and we
ignore "... for convenience and clarity):
love is not love,  $\langle /s\rangle$
bends with the remover to remove.  $\langle /s\rangle$  (eos)
where  $\langle /s\rangle$  denotes the clause or sentence separator, and  $\langle eos\rangle$  is the ending marker of the whole sequence. The target of our framework is to conduct the formats controlled text generation. Therefore, the indicating symbols for format and rhyme as well as the sentence integrity are designed based on the target output sequence.
Format and Rhyme Symbols:
$$
\begin{array}{l} C = \left\{c _ {0}, c _ {0}, c _ {0}, c _ {2}, c _ {1}, \langle / s \rangle \right. \tag {3} \\ \left. \left. c _ {0}, c _ {0}, c _ {0}, c _ {0}, c _ {0}, c _ {2}, c _ {1}, \langle / s \rangle , \langle e o s \rangle \right\} \right. \\ \end{array}
$$
where we use  $\{c_0\}$  to represent the general tokens;  $\{c_1\}$  depict the punctuation characters;  $\{c_2\}$  represent the rhyming tokens "love" and "remove".  $\langle /s\rangle$  and  $\langle eos\rangle$  are kept.
Intra-Position Symbols:
$$
\begin{array}{l} P = \left\{p _ {4}, p _ {3}, p _ {2}, p _ {1}, p _ {0}, \langle / s \rangle \right. \tag {4} \\ \left. p _ {6}, p _ {5}, p _ {4}, p _ {3}, p _ {2}, p _ {1}, p _ {0}, \langle / s \rangle , \langle e o s \rangle \right\} \\ \end{array}
$$
$\{p_i\}$  denote the local positions of tokens within the same clause or sentence. Note that we align the position symbol indices in a descending order. The aim is to improve the sentence integrity by impelling the symbols capture the sentence dynamic information, precisely, the sense to end a sequence. For example,  $\{p_0\}$  usually denote punctuation characters, thus  $\{p_1\}$  should be the ending words of sentences.
Segment Symbols:
$$
\begin{array}{l} S = \left\{s _ {0}, s _ {0}, s _ {0}, s _ {0}, s _ {0}, \langle / s \rangle \right. \tag {5} \\ \left. \left. s _ {1}, s _ {1}, s _ {1}, s _ {1}, s _ {1}, s _ {1}, s _ {1}, \langle / s \rangle , \langle e o s \rangle \right\} \right. \\ \end{array}
$$
where  $s_i$  is the symbol index for sentence  $i$ . The purpose is to enhance the interactions between different sentences in different positions by defining the sentence index features.
During training, all the symbols as well as the input tokens are fed into the transformer-based language model. Contrast to Transformer (Vaswani et al., 2017), BERT (Devlin et al., 2019), and GPT2 (Radford et al., 2019), we modify the traditional attention strategies slightly to fit our problem.
Specifically, for the input, we first obtain the representations by summing all the embeddings of the input tokens and symbols, as shown in the red solid box of Figure 2:
$$
\mathbf {H} _ {t} ^ {0} = \mathbf {E} _ {w _ {t}} + \mathbf {E} _ {c _ {t}} + \mathbf {E} _ {p _ {t}} + \mathbf {E} _ {s _ {t}} + \mathbf {E} _ {g _ {t}} \tag {6}
$$
where 0 is the layer index and  $t$  is the state index.  $\mathbf{E}_{*}$  is the embedding vector for input  $*$ .  $w_{t}$  is the real token at position  $t$ .  $c$ ,  $p$ , and  $s$  are three pre-defined symbols.  $g$  is the global position index same as position symbols used in Transformer (Vaswani et al., 2017).
Moreover, the state at time  $t$  need to know some future information to grasp the global sequence dynamic information. For example, the model may want to know if it should close the decoding progress by generating the last word and a punctuation character to end the sentence. To represent the global dynamic information, we introduce another variable  $\mathbf{F}^0$  by only summing the pre-defined symbols as shown in the blue dash box of Figure 2:
$$
\mathbf {F} _ {t} ^ {0} = \mathbf {E} _ {c _ {t}} + \mathbf {E} _ {p _ {t}} + \mathbf {E} _ {s _ {t}} \tag {7}
$$
After processing the input, two blocks of attention mechanisms are introduced to conduct the feature learning procedure. The first block is a masking multi-head self-attention component, and the second block is named global multi-head attention.
Masking Multi-Head Self-Attention:
$$
\mathbf {C} _ {t} ^ {1} = \operatorname {L N} \left(\operatorname {F F N} \left(\mathbf {C} _ {t} ^ {1}\right) + \mathbf {C} _ {t} ^ {1}\right)
$$
$$
\mathbf {C} _ {t} ^ {1} = \operatorname {L N} \left(\operatorname {S L F - A T T} \left(\mathbf {Q} _ {t} ^ {0}, \mathbf {K} _ {\leq t} ^ {0}, \mathbf {V} _ {\leq t} ^ {0}\right) + \mathbf {H} _ {t} ^ {0}\right)
$$
$$
\mathbf {Q} ^ {0} = \mathbf {H} ^ {0} \mathbf {W} ^ {Q}
$$
$$
\mathbf {K} ^ {0}, \mathbf {V} ^ {0} = \mathbf {H} ^ {0} \mathbf {W} ^ {K}, \mathbf {H} ^ {0} \mathbf {W} ^ {V} \tag {8}
$$
where  $\mathrm{SLF - ATT}(\cdot)$ ,  $\mathrm{LN}(\cdot)$ , and  $\mathrm{FFN}(\cdot)$  represent self-attention mechanism, layer normalization, and feed-forward network respectively. Note that we only use the states whose indices  $\leq t$  as the attention context.
After obtaining  $\mathbf{C}_{\mathbf{t}}^{\mathbf{1}}$  from Equation (8), we feed it into the second attention block to capture the global dynamic information from  $\mathbf{F}^0$ .
Global Multi-Head Attention:
$$
\mathbf {H} _ {t} ^ {1} = \operatorname {L N} \left(\operatorname {F F N} \left(\mathbf {H} _ {t} ^ {1}\right) + \mathbf {H} _ {t} ^ {1}\right)
$$
$$
\mathbf {H} _ {t} ^ {1} = \operatorname {L N} \left(\operatorname {G L O B A L - A T T} \left(\mathbf {Q} _ {t} ^ {1}, \mathbf {K} ^ {1}, \mathbf {V} ^ {1}\right) + \mathbf {C} _ {t} ^ {1}\right)
$$
$$
\mathbf {Q} ^ {1} = \mathbf {C} ^ {1} \mathbf {W} ^ {Q}
$$
$$
\mathbf {K} ^ {1}, \mathbf {V} ^ {1} = \mathbf {F} ^ {0} \mathbf {W} ^ {K}, \mathbf {F} ^ {0} \mathbf {W} ^ {V} \tag {9}
$$
We can observe that all the context information from  $\mathbf{F}^0$  are considered. This is the reason why we name it as "global attention" and why the input real token information  $\mathbf{E}_{w_t}$  is NOT considered. Then
the calculation of the unified first model layer is finished. We can iteratively apply these two attention blocks on the whole  $L$  model layers until obtain the final representations  $\mathbf{H}^L$ . Note that  $\mathbf{H}$  is renewed layerly, however the global variable  $\mathbf{F}^0$  is fixed.
Finally, the training objective is to minimize the negative log-likelihood over the whole sequence:
$$
\mathcal {L} ^ {\text {n l l}} = - \sum_ {t = 1} ^ {n} \log P \left(\mathbf {y} _ {t} \mid \mathbf {y} _ {<   t}\right) \tag {10}
$$
3.3 Pre-training and Fine-tuning
Although our framework can be trained purely on the training dataset of the target corpus, usually the scale of the corpus is limited. For example, there are only about 150 samples in the corpus of Shakespeare's Sonnets (Shakespeare, 2000). Therefore, we also design a pre-training and fine-tuning framework to further improve the generation quality.
Recall that in the task definition in Section 2, we claim that our model owns the ability of refining and polishing. To achieve this goal, we adjust the masking strategy used in BERT (Devlin et al., 2019) to our framework according to our definitions. Specifically, we randomly (say  $20\%$ ) select partial of the original content and keep them not changed when building the format symbols  $C$ . For example, we will get a new symbol set  $C'$  for the example sentences:
$$
\begin{array}{l} C ^ {\prime} = \left\{c _ {0}, c _ {0}, c _ {0}, l o v e, c _ {1}, \langle / s \rangle \right. \\ b e n d s, c _ {0}, c _ {0}, c _ {0}, c _ {0}, r e m o v e, c _ {1}, \langle / s \rangle , \langle e o s \rangle \} \\ \end{array}
$$
where "love", "bends" and "remove" are kept in the format  $C'$ .
After the pre-training stage, we can conduct the fine-tuning procedure directly on the target corpus without adjusting any model structure.
3.4 Generation
We can assign any format and rhyming symbols  $C$  to control the generation. Given  $C$ , we will obtain  $P$  and  $S$  automatically. And the model can conduct generation starting from the special token  $\langle bos\rangle$  iteratively until meet the ending marker  $\langle eos\rangle$ . Both beam-search algorithm (Koehn, 2004) and truncated top-k sampling (Fan et al., 2018; Radford et al., 2019) method are utilized to conduct the decoding.
The parameter size of our model are fixed in both the pre-training stage and the fine-tuning stage. The number of layers  $L = 12$ , and hidden size is 768. We employ 12 heads in both the masking multi-head self-attention block and the global attention block. Adam (Kingma and Ba, 2014) optimization method with Noam learning-rate decay strategy and 10,000 warmup steps is employed to conduct the pre-training.
We conduct all the experiments on two collected corpus with different literary genres: SongCi and Sonnet, in Chinese and English respectively. The statistic number are shown in Table 3. We can see that Sonnet is in small size since we only utilize the samples from the Shakespeare's Sonnets (Shakespeare, 2000). Since SongCi and Sonnet are in different languages, thus we conduct the pre-training procedure on two large scale corpus in the corresponding languages respectively. For Chinese, we collect Chinese Wikipedia (1700M Characters) and a merged Chinese News (9200M Characters) corpus from the Internet. We did not conduct the word segmenting operations on the Chinese datasets, which means that we just use the characters to build the vocabulary, and the size is 27681. For English, same as BERT, we employ English Wikipedia (2400M words) and BooksCorpus (980M words) (Zhu et al., 2015) to conduct the pre-training. We did not use BPE operation (Sennrich et al., 2015) on this corpus considering the format controlling purpose. We keep the most frequent 50,000 words to build the vocabulary.
Besides PPL and Distinct (Li et al., 2016), we also tailor-design several metrics for our task to conduct the evaluation for format, rhyme, and sentence integrity.
Format Assume that there are  $m$  sentences defined in the format  $C = \{C_1^s, C_2^s, \dots, C_m^s\}$ , and the generated results  $Y$  contains  $n$  sentences  $Y = \{Y_1^s, Y_2^s, \dots, Y_n^s\}$ . Without loss of generality, we align  $C$  and  $Y$  from the beginning, and calculate the format quality according to the following rules: (1) the length difference  $||C_i^s| - |Y_i^s|| \leq \delta$ ; (2) the punctuation characters must be same. For SongCi, we let  $\delta = 0$  and rule (2) must be conforming.
Model	PPL↓	Diversity (Distinct) ↑
VAL	TEST	MA-D-1	MI-D-1	MA-D-2	MI-D-2
S2S	19.61	20.43	75.35	2.48	98.35	36.23
GPT2	148.11	104.99	-	-	-	-
GPT2 w/ Fine-tuning	18.25	17.00	73.87	2.57	96.07	33.92
SongNet (only Pre-training)	24.41	16.23	74.84	4.59	95.09	54.98
SongNet (only Fine-tuning)	12.75	14.73	75.96	2.69	97.59	37.26
SongNet	11.56	12.64	75.04	2.66	97.29	36.78
Model	Format↑	Rhyme↑	Integrity↓
MA-F1	MI-F1	MA-F1	MI-F1
S2S	44.32	38.16	53.80	52.27	8.30±2.06
GPT2 w/ Fine-tuning	35.70	35.20	53.48	52.50	45.92±20.12
SongNet (only Pre-training)	29.12	29.46	53.77	53.13	30.98±14.06
SongNet (only Fine-tuning)	99.81	99.83	79.23	78.63	2.14±0.10
SongNet	99.88	99.89	73.21	72.59	1.77±0.16
Table 1: Automatic evaluation results on SongCi
Model	PPL↓	Diversity (Distinct) ↑
VAL	TEST	MA-D-1	MI-D-1	MA-D-2	MI-D-2
GPT2 w/ Fine-tuning	31.47	31.03	73.87	2.57	96.07	33.92
SongNet (only Pre-training)	28.56	28.07	49.92	25.14	85.35	65.70
SongNet (only Fine-tuning)	34.62	34.53	42.31	4.96	90.76	47.26
SongNet	27.46	27.63	43.01	10.43	80.06	56.14
Model	Format↑	Rhyme↑	Integrity↓
MA-F1	MI-F1	MA-F1	MI-F1
GPT2 w/ Fine-tuning	2.03	1.91	5.20	6.24	15.77±3.63
SongNet (only Pre-training)	99.99	99.99	3.93	4.01	15.28±2.04
SongNet (only Fine-tuning)	99.25	99.99	7.50	7.41	18.86±2.59
SongNet	98.73	98.73	11.46	11.41	11.86±3.01
Table 2: Automatic evaluation results on Sonnet
Corpus	#Train	#Dev	#Test	#Vocab
SongCi	19,244	847	962	5310
Sonnet	100	27	27	2801
Table 3: Statistics of the datasets SongCi and Sonnet.
For Sonnet, we relax the condition where we let  $\delta = 1$  and ignore rule (2). Assume that the number of format-correct sentences is  $n'$ , then we can obtain Precision  $p = n' / n$ , Recall  $r = n' / m$ , and F1-measure. We report both the Macro-F1 and Micro-F1 in the results tables.
Rhyme For SongCi, usually, there is only one group of rhyming words in one sample. As the example shown in Table 1, the pronunciation of the red rhyming words are "zhu", "yu", "du", and "gu" respectively, and the rhyming phoneme is "u". For the generated samples, we first use the tool