Structure Controllable Text Generation
Authors: Liming DENG
Long WANG
Binzhu WANG
Jiang Qian
Bojin Zhuang
Shaojun Wang
Jing Xiao
Venue:  (2021)
Keywords: Natural language generation
structure representation
structure controlling
conditional language model
structure aware transformer

Abstract: Controlling the presented forms (or structures) of generated text are as important as controlling the generated contents during neural text generation. It helps to reduce the uncertainty and improve the interpretability of generated text. However, the structures and contents are entangled together and realized simultaneously during text generation, which is challenging for the structure controlling. In this paper, we propose an efficient, straightforward generation framework to control the structure of generated text. A structure-aware transformer (SAT) is proposed to explicitly incorporate multiple types of multi-granularity structure information to guide the text generation with corresponding structure. The structure information is extracted from given sequence template by auxiliary model, and the type of structure for the given template can be learned, represented and imitated. Extensive experiments have been conducted on both Chinese lyrics corpus and English Penn Treebank dataset. Both automatic evaluation metrics and human judgement demonstrate the superior capability of our model in controlling the structure of generated text, and the quality ( like Fluency and Meaningfulness) of the generated text is even better than the state-of-the-arts model.


Under review as a conference paper at ICLR 2021
STRUCTURE CONTROLLABLE TEXT GENERATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Controlling the presented forms (or structures) of generated text are as importan- as controlling the generated contents during neural text generation. helps to reduce the uncertainty and improve the interpretability of generated text. Howev- er, the structures and contents are entangled together and realized simultancously during text generation, which challenging for the structure controlling. In this paper, we propose efficient, straightforward generation framework to control the structure generated text. A structure-aware transformer (SAT) is proposed to explicitly incorporate multiple types of multi-granularity structure information to guide the text generation with corresponding structure. The structure informa- tion is extracted from given sequence template by auxiliary model, and the type of structure for the given template can be learned, represented and imitated. Exten- sive experiments have been conducted on both Chinese lyrics corpus and English Penn Treebank dataset. Both automatic evaluation metrics and human judgement demonstrate the superior capability of our model in controlling the structure of generated text, and the quality like Fluency and Meaningfulness) of the gener- ated text is even better than the state-of ofethhearts model.
INTRODUCTION
Natural language is not just a sequence collections of tokens but a structure well-organized sequence expressing understandable information. The structure of language usually obeys set of grammat- ical rules, which helps beginners grasp the language with less efforts. Similarly, incorporating the structure into neural language model can obtain an increasing abstract level of representation and improves the generalization which may potentially reduce the need of large amount of training data (Shen et al., 2019b). The incorporations of structure information demonstrates considerable im- provements in many language understanding tasks (Zhang et al. 2019 Hao et al. 2019 Wang et al.) 2019).
In text generation, it cares about not only the generated contents (i.e., what to say) but also the p- resented structure forms (i.e., how to say) (Peng et al., 2019). Similar contents or meanings can be presented with different structure forms. The structures and contents can be considered and planned separately to achieve a highly informative generated text. From an empirical view, controlling or planning the generated structure may be helpful in several aspects: i) reducing the uncertainty of the generated contents with specific structure conditions, which may contribute to a good quality of gen- erated text; ii) enhancing the interpretability of the generated text since more controlling attributes can be realized during the generation; iii) improving the structure, format or style consistence in spe- cific structure-consstriin generation task or specific domain generation with particular formats, such as style or paraphrase generation (Chen et al. 2019 Ficler & Goldberg. 2017), poetry generation (Deng al. 2020: Li et 2020). and lyric generation (Watanabe et 2018 Lu al., 2019).
The language structures determined by the set of grammatical rules vary from different granularity levels, such as participial construction (pc) is character-level, part of speech (pos) word/phrase level, and sequence length is sentence level. These kinds of structure are coupled and nested togeth- er, which are realized with the contents simultaneously in most of the token by token generation. is difficult to disentangle the contents and the text structure, and even harder to discriminate and con- trol the different granularity level of structure during text generation. Individually controlling some specific types of structure like sequence length (Kikuchi al. 2016), verbal predicate (Tu et al.) 2019) have been investigated in text generation. These works design specific structure representa- tion and are inappropriately for controlling other types of structure, let alone controlling multiple

---

Under review as a conference paper at ICLR 2021
types of structure simultaneously. Directly embedding the structure and adding them into the word embeddings can achieve considerable controlling capability in character-level structure during text generation, such as tone level and rhyme (Deng al., 2020) controlling in Chinese poctry gener- ation. While this method may fail when the controlled structure (such as phrase level or sentence level) needs aware the subsequent structure during the generation process. In addition sum- marizing the structure embeddings and word embeddings, SongNet (Li et al. (202)) designs another structure embeddings which are queried and incorporated globally by the summarized embeddings to renew the representation. With vee-raining and fine- fine-tuning, the SongNet (Li et al. 2020) can also achieve good controllability designde formats I for this format are particular designed and may not applicable for other type of structure.
(sentence level structure). The symbol sets
Contrast to the above works, this paper, we are not focus on controlling specific type of struc- ture or format, instead we propose framework to control more general types of structure in text generation. This framework allows for controlling individual type of structure, multiple or multi- granularity types of structure during text generation. The controlled types of structure are extracted from sequence templates (any valid sentence a valid template) by one or several auxiliary models. The extracted structure information are regarded conditions, and the auxiliary model can be any credible model or tool that can extract soundable structure information from template. Since we want the generation of the current token word aware the global structures, the bi-directional transformer encoder is adopted for structure representation and learning. The learned structure rep- resentations are further incorporated into the decoder to guide the realization of the controlled struc- ture. The main contributions of this work are summarized as follows:
A straightforward. interpretable structure controlling text generation framework is pro- posed, which is capable of controlling multi- caattiigraularitty sequence structure from character- level to sentence-level structure by explicitly incorporating the corresponding structure in- formation.
A simple alignment method and structure embedding, representation and learning method are proposed, which are utilized for representing the multi-granularigg and multiple types of structure.
A structure-aware transformer language model is proposed, and the structure representation and token representation can be learned simultaneously. The structure information are queried globally and incorporated into the token representation with attention mechanism, which contribute to controlling the generated structure.
Extensive experiments in controlling different individual type of structure and multi- granularity types of structure have been conducted on Chinese lyrics corpus. The struc- ture controllability is effective and the quality of the generated lyrics favorable. We also conduct controlling experiments on English Penn Treebank dataset, which demonstrates similar structure controlling capability with this proposed framework.
2 RELATED WORKS
Controllable text generation has received much attention recently. Many efforts are devoted to con- trolling the content of the generated text (Kiddon et al.) 2016 Lebret et al. 2016]] Shen et al. [019a). Based on conditioned RNN language model, stylistic parameters are further incorporated as condi- tioning context to control stylistic aspects of the generated text (Ficler & Goldberg 2017). Basing generator on VAEs, Hu et al. (2017) proposes a generative model to generate plausible sentences with designated semanties. A simple plug and play language model is proposed in Dathathri et al. 2019) to guide controlling attributes (e.g. topic or sentiment) in text generation, without further training of the pre-trained language model. None of these work attempts control the structure of the generated text. A similar approach, exemplar-basee (2019), where for each input text, an exemplar text is retrieved from the training data and is then used to construct customized decoder for outputting target. It is ambiguously to discriminate how much the exemplar contributes to the generated structure or contents. Another similar work SongNet (Li et al. 2020), which are proposed to control the so called rigid formats. The rigid for-
text generation, is proposed in Peng et al.
TThis format structure is more about the length of each sentence within one paragraph or passage.
2

---

Under review as a conference paper at ICLR 2021
mats specifically designed with sequence of placeholder symbols, which are utilized to control the sentence (or sub-sentence) length.
Our method is different from all the previous methods in fourfold: 1) we focus general structure controlling framework in text generation instead of controlling a specific type structure;2) both individual type of structure and multiple multi-granularity types of structure can be controlled:3) instead of designing the structure symbols by ourself, we adopt the most representative structure symbols extracted by external models increase the applicability of framework:4) the tracted structure information decoupled from the sequence information are learned and represented fully before them are incorporated into word information to guide the text generation.
MODEL DESCRIPTION
3.1 STRUCTURE CONDITIONAL LANGUAGE MODEL.
Given a natural language sequence denoted by æ = "r.....,,,, each word denoted as Zt,t I..., The sequence joint distribution p(æ) can be factorized into the product of conditional distributions p<=, as follows:
p(x) =(
T
II p(=()
(I)
A standard language model is modeling the above distribution and maximizing the corresponding likelihood accordingly (Bengio et al.) 2003, Peters et al. 2018, [hhnn et al. (019b)). The above distribution considers the order structure of natural language sequence explicitly, and the conditional distribution are based on the previous word tokens.
Although the standard language model can generate sentence with high quality, the generated struc- ture is inexplicable and cannot be controlled to satisfy specific generation task. Therefore, we incor- porate the structure information explicitly into language model, and guide the structure generation. The joint distribution of sequence & can be reformulated as shown in Equation equation 2
p(x) =p(x1..... ..... IT)
T
=p(s) II [(
where, s represents the global structure of the natural language sequence æ, the global structure can be any of the structure information like pos tags or semantic roles the sequence, and p(s) prior distribution of the global structure. We extract the structure information with auxiliary model, and this structure information considered as prior knowledge, which will not be optimized by the language model.
The model parameters are learned by maximizing the objective function SCLM, which to maximize the likelihood as shown in Equation equation
mor (z(
Po (c)
(3)
We utilize the Transformer (Vaswani et al. 2017) as the backbone for implementing our SCLM. The structure information is first extracted by auxiliary model and then encoded into transformer encoder. The structure information can learned and represented fully, which can further corporated to contribute the aware of the structure for sequence token representation with attention mechanism. The reason why both the transformer encoder and decoder are adopted here is that we want each token in sequence aware its local and global structure information. Only the Trans- former decoder, like GPT (Radford et al. 2018) ignores the subsequent structure information of the token. The Transformer architecture is well designed and suitable for the implementation of the structure conditional language model. We only modified the input representation and few parameters of transformer.

---

Under review as a conference paper at ICLR 2021
3.2 STRUCTURE EXTRACTION
We auxiliary model (such lexical tool) g(+) to extract the structure information from natural language sequence a as shown in Equation equation The auxiliary model can be regarded as prior knowledge and will not be optimized.
8 = (
(4)
The structure can be any sounded structure information of language sequence vary from character- level structure (like participial construction), word-level structure (like part of speech) to sentence- level structure (positions for example).
The multi guanliigaaularity types of sequence structure 81; 82, 8; can be extracted by different auxiliary models 91 ( 92 92("), ........ 9a (+) respectively. Since each structure unit (especially for word-level and sentence-level structure) may contain several characters, we assign these characters with the same symbol of this kind structure. We keep the length of the structure the same with the sequence tokens.
To be specific, we use the part of speech (pos) and participial construction (pc) as examples to illustrate the alignment of multi-granularity types of structure. The pos information tracted by many lexical analyzer tools like Jieba analyzer and Stanza (Qi et al. (2020) for Chi- nese and English sequence respectively. In Chinese, the pos is type of word-level structure, and the participial construction is the character-level structure for each segmented word. We u- tilize the symbol collections Cpoz the pos for each word. The symbol collections Epe S, B, M, E3} E are utilized repre- sent the for each character within each word. Suppose we have two levels (word-level and character-level) structure information for sequence the word-level form of the sequence with w [W....., Wj. Wn.,, n. S n, and the pos struc- ture can be represented with 8 = [pos),... .... ,:.;;,,,................ ...; m. posj E Cposi each word con- tains several characters Wij (...,.J..As J...:...... E [1,m]] and the pc structure for each word are Sc.j = "..,,p ...]. PCg.k E Cpe where Σj=1 my = n. Therefore, we can obtain the word-level structure (pos) and character-level structure (pc) with the same length with the original sequence as can be shown in the following expressions:
be
r.... from lexical analyzer (like Jieba) to represent
[aI, Fs...., Fnl. we can also present
Su [..., posj, ... posj ..... E nol
(5)
8. PCj ony PCj PCj.m
(6)
m
The sentence level structure like positions have unique representation for each token and do not need any further processing for the aligmment. With the alignment process, guuti-gramuulrrty and multi-type of sequence structure can be incorporated and controlled in the generation.
An illustration of multi cunti--raanularity structure information for natural language sentence can be shown in Fig.
3.3 STRUCTURE AWARE TRANSFORMER
We propose a Structure Aware Transformer (SAT) to implement the multi-granularity structure controlling in text generation. The encoder stacks multi-layer Transformer encoder (Vaswani al. 2017) with Multi-Head Self Attention in each layer to represent the extracted structure. The ex- tracted structure information are first embedded and then summarized together as the structure input representation Ho A which allows for controlling multiple types of structure in text generation simultaneously. The structure representation for each layer Hi la 1, ... N₂ can be obtained
n, v. represent the noun, verb, pronoun respoctively; for complete symbols can refer to http- s:// ss/ggitub..om/fxiiyjjieeee
P represent the pe structure special token, S represent word only contains a single character, B, M, F. represent the beginning, middle and ending of the word respectively. "positions regarded as sentence-level structure and are also added into the structure representation.
4

---

Under review as a conference paper at ICLR 2021
/ love
chips x
Sequence
X /
X
pc
S
BE
B B
pos
r
Figure 1: An alignment illustration of word-level and character-level structure for a Chinese sequence, The final pe structure is Spc [S, B, B, E., and the final pos structure is 8por = [r, u, U,
according to the following formulas:
Ho
Es, (s;)
1=0
Qa K
W W
= H
(8)
V
W
QuK3² T
A. softmax
VV
Vd
H. LN(A, + H1,-1)
(10)
H, (H(F )()
(11)
where E. is the structure embedding matrix, m is the number of structure types, Le is the number of encoder layers, and Hi_ is the output structure representation for layer le- seftmax()) LN(.). FFN(.) represent the softmax function, layer normalization and feed-forward network respectively.
The final layer output of structure encoder H N. is then utilized the decoder, and the decoder is similar to the Transformer decoder (Vaswani et al.) (017) with two attention blocks in each layer. The below attention block is Masked Multi-Head Self Attention, which obtains the token æt representation without considering the information from its subsequent tokens E>*- The upper attention block the Structure-Aware Attention, which incorporates the structure information (HN.) ) into the token representation.
Fg = Ex(x) + E, F Mask-Att(Ei Fiz-1)
(12)
(13)
Q Fú, W
(14)
K, V = HN. W
HN. W QKT
(15)
A softmax
V
(16)
Vd
F LN(A.s. + F
(17)
F. LN(FFN(F")) F
(18)
where E, is the token embedding matrix, E, is the position embedding matrix, Mask-Aft represents the Masked Multi-Head Head Self Attention mechanism, la E [1] is the number of decoder layer, FI, is regarded as the structure-aware token representation.
The final output of the decoder FN, be utilized to calculate the probabilities ( s), and the parameters in the architecture can be learned by maximizing the likelihood in Equation equation 3
3.4 STRUCTURE CONTROLLABLA GENERATION
With our proposed SAT, we can controlling multi multi-graauulaityy and multiple types of structure text generation simultaneously. Both the specified structure information and the template sequence
5

---

Under review as a conference paper at ICLR 2021
can be utilized to control the structure of the generated text. The input context ze, which can be any precede words for continue generation (or topic words for topic related text generation). is utilized to guide the content generation. If no input context is specified, the model will start the generation from start token until the end token is generated.
4 EXPERIMENTS AND EVALUATIONS
4.1 SETUP
We follow the GPT2 source code from /uggingface repository (Wolf et al. 2019) and add an addi- tional structure encoder with multi-head self attention to implement our proposed SAT. The number of encoder layer No is 2 and the number of decoder layer Na is 6. The other configurations are the same with the GPT2 except vocabulary size and structure size for embedding matrix. The structure information is extracted by Jieba 0 and Stanza (Qi et al. 2020) for Chinese and English text respec- tively. The extracted structure information reagrded as the conditional structure, which are not optimized by our language model. However, the structure embeddings (E,) or representation vector (Hi.) can learned by the proposed SCLM.
4.2 DATASETS
We conduct the experiments on both Chinese lyrics corpus and English Penn Treebank (PTB) dataset. Over 80,000 Chinese lyrics are crawled from set of online music websites, and the number of lyries sentences without repetition is about 1.38 million. Every two adjacent lyric lines within one song concatenated with comma increase the structure complexity, which is prepared for the generation task. We randomly split them into three parts for model training(90%) validation(5%) and testing(5%). The statistics of data corpus for Chinese Lyries and PTB dataset shown in Appendix.
4.3 MODEL COMPARISONS
We conduct the model comparisons on both Chinese lyrics corpus and English PTB dataset. The pos structure is considered as the mainly structure for the structure conditional language model, and we compare the GPT2 and SAT-pos on the continue text generation with both Chinese lyrics corpus and PTB dataset. The continue text generation utilizes the prompt words guide the following sequence generation. The length each prompt is randomly varied from 0 to the half length of the whole template sequence.
We also investigate multi-granularity types of structure individually and simultancously for the S- CLM Chinese lyrics corpus. The additional structure the participial construction (pc), which can also be extracted by Jieba analyzer. Two other models SAT-pc (conditioned with structure) and SAT-p² (conditioned with both pc and pos structure) are also compared on Chinese lyrics cor- pus. To better compare the generation capability of these language models, a topic related generation task are also performed based on Chinese lyrics corpus. The topic words are extracted by Jieba with TF-IDF method. For fair comparisons, we train the SAT and GPT2 from scratch without utilizing any pre-trained model.
4.4 EVALUATION METRICS
Both automatic evaluation metrics and human evaluations are adopted for model comparisons. The PPL is to evaluate the performance of language model, and the BLEU score (Papineni et al.) (2022) utilized to measure the content similarity of the generated text with its referred sequence text.
The structure controlling capability, like the sentence length, the and participial construction are also compared. The length controllability is measured by the prediction accuracy. Assume the length
'We have conduct experiments on different number of encoder layers, and the gain of larger number of layer trivial, plcase refer Appendix for the result and analysis. "https//gitub..omm/ff com/fxsjj//jcchh Indicators prompt word is specified, and the generation starts from the start token.

---

Under review as a conference paper at ICLR 2021
of the input template is l and the predicted sequence length is l'. If the length difference ō = ||-1' is within specified threshold, we regard the predicted length is accurate with this tolerance. We report the Accuracy of length control with tolerance 8 I 0, ó < 2 and å < 4.
The BLEU score can also be utilized to mcasure the pos and pe controllability. We extract the pos and pe structure from both test template and predicted sequence with the same lexical tool (Jieba or Stanza), and the BLEU score of pos or pc can be calculated accordingly.
Human evaluation is inevitable for evaluating the quality of the generated text, especially in the meaningfulness and fluency. However, human evaluation is time-consuming and costing. We con- duct the human evaluation for model comparisons on the continue generation task of Chinese lyries. Four well educated annotators are recruited to evaluate the continue generation of Chinese lyric- s sentence in three dimensions, namely Fluency, Meaningfulneess and Structure Matching. The Fluency and Meaningfulness are easy to understand and have been utilized by many previous works Deng et al. (2020). The Structure Matching is evaluate the matching degree of generated text structure and template structure in several aspects, which considers the global structure (like subjec- tive, predicates and objective structure) matching constitute structure matching and pos matching for local words. The rating scores are I to 5 to represent the quality from bad to excellent for all the criteria. Each model generates 500 lyric lines and with the same random length prompt. Total 1000 lyric lines are generated and randomly shuffled, and the four annotators rated on the shuffled lyrics lines. Therefore, we can obtain 4000 (4 x 2 x 500) ratings.
4.5 RESULTS& & DISCUSSIONS
Table [ shows the perplexity of the language models on both Chinese lyrics corpus and English PTB dataset. The results demonstrates that the pos structure can improve the language modeling performance on both Chinese and English sequence. And the language model performance can be further improved when additional structures are also incorporated, as shown in the table that the PPL of SAT-p" -p with the lowest scores. We can observe that the pos condition gains more improvements than the pc structure condition when compared SAT-pos model with SAT-pe model on Chinese lyries corpus. The probably reason is that the pos structure (with dictionary size 58) contains richer structure information than pc structure (with dictionary size 5).
Table I: Perplexity scores for model comparisons.
Chinese Lyrics
PTB
Model
Val.
Test
Val.
Test
GPT2
10.57
11.24
8.60
8.12
SAT-pc
7.51
8.01
SAT-pos
4.07
4.34
3.56
3.39
SAT-p²
3.92
4.19
The text generation performance can also be improved by proposed model, demonstrated in Table [] and [ The generation performance of our proposed structure conditional models obtains obvious improvements on the BLEU scores of text sequence. The improvements of text BLEU scores are with similar improvement paradigms as the PPL scores, which are 1) the prior structure information is useful for the modeling and generation; 2) the more the structure information incorpo- rated, the better the modeling performance and generation results. Our proposed model SAT shows the superior structure controllability as demonstrated by the BLEU scores on pc and pos structure. The BLEU scores of structure can be significantly improved when the corresponding structures are conditioned and incorporated into the language model.
It is interesting to observe that the pos structure can improve the BLEU scores on pC significantly (SAT-pos versus GPT2), while the pc structure only slightly improves the BLEU scores on pos (SAT-pc versus GPT2). These phenomena are consistent with the fact that the pos structure can reflect the segmentation border of words. The pe structure is more coarse structure information than pos. We also observe that the pos structure not improve the BLEU scores pc when the pc structure is already incorporated (SAT-p² versus SAT-po), while the pe structure can further improve BLEU scores when the poy structure is already incorporated (SAT-p² versus SAT-pos))
7

---

Under review as a conference paper at ICLR 2021
Table 2: The BLEU scores for model comparisons the continue generation of Chinese lyrics.
Task
Model
Text
pc
BL-1
BI.-2
BL-1
BL--
BI.-1
BL-2
GPT2
0.144
0.015
0.653
0.608
0.396
0.241
Sp
0.174
0.028
0.98
0.975
0.486
0.323
Continue
SoT-
0,268
0.115
0.939
0.919
0.949
0.939
SAT-p²
0.269
0.115
0.97
0.962
0.952
0.941
GPT2 0.247
0.133
0.676
0.643
0.449
0.309
Sp
0.279
0.151
0.981
0.976
0,553
0,395
Topic
SAT-pos
0.356
0.226
0.936
0.915
0.941
0.925
SAT-p²
0.36
0.231
0.966
0.957
0.948
0.935
The probably explanation is that the pos structure is type of fine-grained (or miero scale) structure compares to the pc structure, and the fine-grained information is too details for clarifying coarse information
The length controllability of our proposed model is demonstrated by Table 6 (in Appendix). Al- though the text length not explicitly incorporated as the condition, the generated text length is controlled effectively by the sequence length of conditioned and pc.
The Human evaluation results, as shown in Table also demonstrate that the proposed model is superior in controlling the structure of the generated text. Although the strict structure constraints, our model can also achieve even better performance in terms of Fluency and Meaningfulness. As for the case and ablation studies please refer to the Appendix.
Table The Human evaluation results for continue generation of Chinese lyrics. Flu., Mea., Mat. represent the Fluency, Meaningfulness and Structure Matching.
Model
Flu.
Mea.
Mat.
GPT2
3.12
3.25
2.01
SAT-p²
3.59
3.82
4.01
5 CONCLUSION
In this paper, we propose a straightforward, interpretability and effective framework to control wide range of language structures from character-level, word-level sentence-level structure text generation. These kinds of structure regarded prior knowledge are explicitly extracted by external models and aligned together, which allows for both individually and simultancously controlled in text generation. The structures decoupled from word information and the structure represen- tations are learned by bi-directional transformer encoder, which is powerful to learn the structure representations sufficiently. Subsequently, the structure representations globally queried by the transformer decoder and are incorporated into contextualized word representations to guide the text generation with corresponding types of structure.
Extensive experiments on both Chinese lyrics corpus and English Penn Treebank dataset have been conducted. Without pre-training large amount of dataset, the results demonstrate the powerful structure controllability of our method in terms of the sequence length, pos, and pc. The superior performance of quality with respect to fluency and meaningfulness are also achieved significant improvements than the free text generation model. Our method can be easily applied to control other kinds of structure in text generation and may even reduce the uncertainty and improve the quality of the generated text.
Let''s 's analogy this with example, the micro-scale shape of an object is not helpful or may even disturbing the identification of macro-scale shape of the object.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1 137-1155, 2003.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. Controllable paraphrase genera- tion with a syntactic exemplar. arXiv preprint arXiv:I906. 00005, 2019.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: simple approach to controlled text generation. arXiv preprint arXiv: 164, 2019.
Liming Deng, Jie Wang, Hang-Ming Liang. Hui Chen, Zhiqiang Xie, Bojin Zhuang, Shaojun Wang, and Jing Xiao. An iterative polishing framework based on quality aware masked language model " Association for the Advancement of Artificial for chinese poetry generation. pp. http:////oo.r/////../ wrg//o.gg/ooooo... 1609 aaai. v34105. 6265" Intelligence", 2020. URL wttpt:::://////c
Jessica Ficler and Yoav Goldberg, Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.0266 2017.
Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. Towards better modeling hierarchical structure for self-attention with ordered neurons. arXiv preprint 1xXiv:990.00166 2019.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Toward con- trolled generation of text. In Proceedings of the 34th International Conference Machine Learn- ing- - Volume 70, ICML17, pp. 15871596. JMLR.org. 2017.
Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. Globally coherent text generation with neural checklist models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, 2016.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Con- trolling output length in neural encoder-decoders. In Proceedings of the 2016 Conference on Empirical Methods
Natural Language Processing, pp. 1928-1338, Austin, Texas, Novem- ber 2016. Association for Computational Linguistics. doi: (0.18653/v//11-11 1140. URL https: h wwww www.wwwwlllcc..... www.ww.wwwbb...om www.:ee.....g..ooogoooo/aaeoeeee
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. pp. 1203-1213, 01 2016. doi:
Piji Li, Haisong Zhang, Xiaojiang Liu, and Shuming Shi. Rigid formats controlled text generation. arXiv preprint arXiv:2004.0800 2020.
Xu Lu, Jie Wang, Bojin Zhuang. Shaojun Wang, and Jing Xiao. A syllable-structured, contextually- based conditionally generation of chinese lyrics. arXiv preprint vrXivvv906.000 2019.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei Jing Zhu. Bleu: a method for automatic eval- uation of machine translation. In Proc Meeting of the Association for Computational Linguisties, 2002.
Hao Peng, Ankur P Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. Text generation with exemplar-based adaptive decoding. arXiv preprint arXiv: 04428, 2019.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2018
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL https://s3-us-wess amazonaws. com/openai- www./rreeabooeeeeuuuuueeevvveeee understanding paper. pdf. 2018

---

Under review as a conference paper at ICLR 2021
Xiaoyu Shen, Jun Suzuki, Kentaro Inui, Hui Su, Dietrich Klakow, and Satoshi Sekine. Select and at- tend: Towards controllable content selection in text generation. arXiv preprint arXiv:1909.044: 04453, 2019a.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. International Conference on Learning Representa- tions, 2019b.
Lifu Tu, Xiaoan Ding, Dong Yu, and Kevin Gimpel. Generating diverse story continuations with controllable semantics. In Proceedings of the 3rd Workshop on Neural Generation and Transla- lion, pp. 44-58, Hong Kong, November 2019. Association for Computational I Linguistics, doi: 10.18653/v1/D11111
UtI.https::/
acl web. wwt.b..rg//rrcboolo////11------
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser. and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-69988 2017.
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. Self-attention with structural position representations. arXiv preprint urXiv:/909. 1909000383, 2019.
Kento Watanabe, Yuichiroh Matsubayashi, Satoru Fukayama, Masataka Goto, Kentaro Inui, and Tomoyasu Nakano. A melody-conditionee lyrics language model. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 163-172, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrie Cistac, Tim Rault, Rmi Louf, Morgan Funtowicz, and Jamie Brew. Transformers: State- of-the-art natural language processing, 2019.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang. Xi Zhou, and Xiang Zhou. Semantics-aware bert for language understanding. arXiv preprint
2019.
A DATA DESCRIPTION
The statistic of utilized data summarized Table The size pos structure for Chinese lyrics datasets extracted from Jieba is 58, and the size of participial construction for lyries words is 5. The vocabulary size of the lyrics dataset including the special tokens (like [PAD]. [START), [END], [TOPIC]) is 4102, and some low frequency characters are replaced with /UNKJ. The special tokens that indicate the start, end or pad of the sentence are regarded as a special structure. The vocabulary size of PTB is 10005 (with some special tokens), and the structure size pos extracted by Stanza is 43.
Table The Statistics of the utilized data corpus.
Corpus
# #Train
#Validation
#Test
Chinese Lyries
6088,90
33830
33902
Penn Treebank
42068
3370
3761
B SUPPLEMENTARY RESULTS
Table 5: The BLEU scores for model comparisons on the continue generation of PTB dataset.
Model
Text
pos
BL-1
BL-2
BL-1
BL-2
GPT2
0.093
0.028
0.277
0.128
SAT-pos
0.309
0.138
0.903
0.788
10

---

Under review as a conference paper at ICLR 2021
Table The length accuracy results on both Chinese lyrics test corpus and PTB test datasets. Topic in the bracket represent the topic related generation task, and Continue in the bracket represents the continue generation task.
Lyries (Topic)
Lyrics (Continue)
PTB (Continue)
Model
<0
6 <2
<4
<0
6<2
8 <4
8 <0
82
8 <4
GPT2
0.08
0.37
0.61
0.10
0.47
0.73
0.11
0.36
0.56
1.00
1.00
1.00
1.00
1.00
1.00
SAT-pos
1.00
1.00
1.00
1.00
1.00
1.00
0.91
1.00
1.00
SAT-p²
1.00
1.00
1.00
1.00
1.00
1.00
C CASE STUDY
Figure 2[nd [ compare several cases generated by GPT2 and our proposed SAT Chinese lyrics corpus and PTB dataset. We can notice that our model is capable of controlling the sentence-level structure (length), word-level structure (pos) (and character-level structure pc for lyrics generation) simultaneously. and the quality of the generated texts are also qualified and understandable. It is should be noted that the auxiliary model tool utilized for extracting the structure is not optimized by our language model, and the accuracy of these tool will affect the quality of the generated text.
Prompt
Pc
pos ns/n/d/x//pps
Template:
88M0 S 1ü ESB @. / 752
BMME/S/BE/S/BMMENIE
Suddenly looking back the love is can't help mysall.
SAT-pc
# Mi 8) 1188 7 #7 / 9686 love song for looking back,
BMME/S/BE/S/EMME/SSE
ns me/u////x////m//
thousands of words are spoken sinceraly.
SAT-pos
#8 iil 7: / / tE 1.Dkk
BMMEES//BE/S/EMEE/S/EE
n6 ra/n/d/x/1/pss
Suddenly looking
infinite love.
mousands of words in my heart.
SAT-p²
# //700/ 17 75 /bbl
BAME/S/BE////5///////8
ms/n/d/x/l/p/s
Suddenly looking back in tears.
thousands words my heart.
# Maa s ( /// / # /0 /830///////////// BIMMESSSSB1SSSS5557335MMEEEBBESS htt.:////.////////// m/t/////rrr/////////////
Suddenly looking
your face,
you will never be separated my heart.
Figure 2: Cases generated by different models with the same input prompt.
Prompt
Prediction
Template:
a dog was running in room
(DT NN VBD VBG DT NN)
SAT-pos
the market was falling at this point
(DT
VBG
NN)
GPT
the company sald the new faclity
will begin to eamiigs for fiscal N
Template
a
a room dag is walking (NN VBZ VBG
DT
DT NN)
SAT-pos
problem coming that
DT
(NN vBZ VBG IN DT NN)
unk spokesman said The company trying sell millinn australian dollars 5 N billion of assets
Template:
Is walking in the room [DT VBZ
NN)
SAT-pos
the is working the problem NN VBZ VBG DT
GPT the woman says my bush is nt wornied whether
unk
way about taking
Figure Cases comparisons on different models with different length of prompt and different templates. The template is utilized to provide pos information for SAT-pos, and the corresponding pos are in the bracket. The notations the pos are provided by Stanza.
11

---

Under review as a conference paper at ICLR 2021
D ABLATION STUDY
We conduct ablation study experiments on Chinese lyrics corpus to investigate the effects of encoder layer number. The PPL scores compared for layer 4, and 6 for different types of structure information. As shown in table 7 we cannot observe the obvious improvement due to the larger number of encoder layer. The probably explanation is that the structure information comparative small (with dictionary size 5 for pe structure and 58 for pos, while vocabulary size is 4102) and layer encoder enough to process nd represent the information.
Table 7:
Perplexity scores of different encoder layer number for different structure information.
layer
4 layer
6 layer
Model
Val.
Test
Val.
Test
Val.
Test
SPT-pc
7.508
8.013
7.491
8.606
8.105
8,686
SAT-pos
4.068
4.342
4.058
4.765
4.554
4.847
SAT-p²
3.923
4.190
3.912
4.563
4.310
4.604
12


Reviews:
[{"review_id": "vlVBbgRebL_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a text generation method which can utilize the language structures from character-level, word-level to sentence-level structure.  The proposed model, structure-aware transformer (SAT), explicitly incorporates multiple types of multi-granularity structure information to guide the text generation with corresponding structure. \n\nPros:\n1. The paper is clearly written. The experiments show the effectiveness of the proposed method.\n2. The  proposed method can explicitly incorporate multiple types of multi-granularity structure information to guide the text generation.\n3. The proposed method shows its advantages in structure control.\n\nCons:\n\n1. The method just incorporates structure information in the encoder part rather than the decoder. It's hard to guarantee the quality of structure control. Moreover, this way to incorporate structure information is just adding extra features and not a new framework.\n2. The structure information used in the paper is only segmentation and POS information. Other information, such as syntax, also should be considered.  Segmentation and POS information are just shallow structure information.\n3. BLEU score for POS or BMES is not suitable. Since the structure information is given, the generated text should be evaluated with other measures, such as F1 score.\n4. The fluency could be due to the pre-trained language model GPT2. An experiment should be performed without PLMs.\n5. Some related references are missing.\n\nQuestions:\n1. The title of paper is \"Structure Controllable Text Generation\", but the proposed method is just to infuse structure information as features. Therefore, the proposed method is more like \"structure-infused\" rather than \"structure controllable\".\n\nMissing References:\n1. Zhang X, Yang Y, Yuan S, et al. Syntax-infused variational autoencoder for text generation[J]. arXiv preprint arXiv:1906.02181, 2019.\n2. Casas N, Fonollosa J A R, Costa-jussà M R. Syntax-driven Iterative Expansion Language Models for Controllable Text Generation[J]. arXiv preprint arXiv:2004.02211, 2020.\n3. Wu S, Zhou M, Zhang D. Improved Neural Machine Translation with Source Syntax[C]//IJCAI. 2017: 4179-4185.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "W3-NBJteV86", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes a variation of the conditioned Transformer-based language model. The authors use POS labels (for English and Chinese) and participial construction labels (only for Chinese) to control the output of the decoder and show the results are better than an unconditioned generation with GPT-2 in terms of several metrics.\n\nCons and questions:\n- The paper describes a generic conditioned language model approach, I can see no novelty of methods or results here.\n- No definition of \"structures\" given. Instead, the authors use fuzzy formulations like \"multiple types of multi-granularity structure information\" or \"the auxiliary model can be any credible model or tool that can extract soundable structure information from the template.\" In fact, the paper provides the results of experiments with POS labels and participial construction labels (only for Chinese).\n- No comparisons with other known syntax-aware language models are done (consider [arXiv:1909.02273], DOI:10.18653/v1/N19-1118, DOI:10.1109/IALP48816.2019.9037672, etc).\n- No exact details on the structure encoder or decoder pre-training or finetuning process are given, although they can represent some practical interest.\n- No clear description of the resulting architecture is provided (\"We only modified the input representation and few parameters of transformer\").\n- The reason for using a Transformer encoder is ambiguous (\"both the transformer encoder and decoder are adopted here is that we want each token in sequence to aware its local and global structure information\"), the logic of the number of Encoder layers selection is hard to check without details of the training process.\n- The Structure Matching evaluation was done through human assessment. However, no clear rules of the process are given.\n\nMinor comments:\nThe paper has a lot of typos and grammar issues. Here are some samples:\n(1) \"importan-t\" -- an incorrect word-wrap.\n(2) \"quality ( like Fluency\" -- a space after the open parenthesis\n(3) \"extracted structure information are regarded\"\n(4) \"word can aware the global structures\"\n(5) \"which can be any precede words for continue generation\"\n(6)\t\"reagrded\"\n\nThere are many more, I suggest at least to use some automatic spellchecking tools.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T4qv5ktf3g1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY\n\nThis paper presents a text generation model conditioned on desired structures. The proposed method is essentially a translation model from structure information (represented with multiple sequences of tokens) to a text. This study converts a text into structure information such as part of speech (POS) and participial construction (PC). Then, this paper proposes Structure Aware Transformer (SAT), which is essentially the same as the Transformer architecture. The experiments use datasets of Chinese lyrics and English Penn Treebank. This paper reports that giving structure information improved the performance in PPL and BLEU compared with GPT-2.\n\nPROS\n\nIt was nice to confirm that we can control language generation from POS and PC.\n\nCONS\n\nThe proposed method presented in Section 3.3 is identical to Transformer except that:\n\n+ An input consists of multiple sequences of structure information (e.g., pos and pc)\n\n+ Input embeddings are sums of structure embeddings (Equation 7)\n\nFor this reason, I do not think Structure Aware Transformer (SAT) is a novel proposal. If this explanation is sufficient, I think that the descriptions in Section 3.1 and 3.3 are redundant.\n\nBecause structure sequences (POS and PC) are obtained from sentences in the test set, it is not surprising to see performance improvements of language models. In other words, predicting word sequences with some hints (POS and PC) is much easier than doing without any hint (GPT-2). For this reason, the findings in this paper are not convincing.\n\nQUESTIONS\n\nCould you explain the major difference between the proposed method and Transformer (excluding minor differences in how input embedings are computed and hyper-parameters such as the number of layers)?", "extra_scores": {}, "extra_text": {}}, {"review_id": "27Qv8GUONiK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review of Paper: Structure Controllable Text Generation\n\nSummary:\nThis paper proposes a structure-aware Transformer (SAT) by incorporating multiple types of multi-granularity structure information to control the text generation. Their method can extract structure information given sequence template by auxiliary model and learn the structure representations.  \n\nStrengths:\n•\tThe proposed model is able to control multiple or multi-granularity types sequence structure from character-level to sentence-level structure by explicitly incorporating the corresponding structure information. \n•\tThe model can simultaneously learn the structure representation and token representation.\n\nWeakness:\n•\tThe paper does not clearly define the input content of this task. \nWhat’s the input of the task? It seems the input should be the “prompt” (prefix of the poem), according to figure 2 in appendix C.\n\nIf so, why not use the content of “prompt” to predict the following tokens, which is more important for generation rather than the structure information? \n\nIf the input of this task is the structure information of the whole poem, the model can access future information (the structure of the sequence to be generated) during inference. It is unfair to compare this model with GPT-2, because it’s not a standard language modeling task. Instead, it should compare with structure controlled text generation task like SongNet (Li, P.), but the paper does not conduct such a comparison. \n\n•\tThe baseline is not solid and suitable for the datasets they use. \nGPT-2 is suitable for large-scale training corpus. Training GPT-2 on a small-scale dataset may result in lower performance, so GPT-2 cannot act as a strong baseline on a small-scale dataset (e.g. Chinese lyrics corpus). However, GPT-2 is the only one baseline in this paper. \n\nAs a large-scale text generation dataset is not hard to acquire, why not try it on a large-scale dataset?\n\n\n\n\n\nReference:\n[Li, P.] Li, P., Zhang, H., Liu, X., & Shi, S. (2020). Rigid Formats Controlled Text Generation. arXiv preprint arXiv:2004.08022.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a controllable text generation model conditioned on desired structures, converting a text into structure information such as part of speech (POS) and participial construction (PC). It proposes a “Structure Aware Transformer” (SAT) to generate text and claims better  PPL and BLEU compared with GPT-2. Reviewers pointed out that limited novelty of this paper - SAT is essentially a transformer run on multiple sequences of structure information, with sums of structure embeddings as input embeddings -  the proposed method essentially infuses structure information as features, rather than “controlling” text generation. Some references are also missing, most prominently: 

1. Zhang X, Yang Y, Yuan S, et al. Syntax-infused variational autoencoder for text generation[J]. arXiv preprint arXiv:1906.02181, 2019.
2. Casas N, Fonollosa J A R, Costa-jussà M R. Syntax-driven Iterative Expansion Language Models for Controllable Text Generation[J]. arXiv preprint arXiv:2004.02211, 2020.
3. Wu S, Zhou M, Zhang D. Improved Neural Machine Translation with Source Syntax[C]//IJCAI. 2017: 4179-4185.

Unfortunately, no answers are provided by the authors to the questions asked by the reviewers, which makes me recommend rejection.
Decision: Reject