Structure Controllable Text Generation

1 INTRODUCTION . Natural language is not just a sequence collections of tokens but a structure well-organized sequence expressing understandable information . The structure of language usually obeys a set of grammatical rules , which helps beginners grasp the language with less efforts . Similarly , incorporating the structure into neural language model can obtain an increasing abstract level of representation and improves the generalization which may potentially reduce the need of large amount of training data ( Shen et al. , 2019b ) . The incorporations of structure information demonstrates considerable improvements in many language understanding tasks ( Zhang et al. , 2019 ; Hao et al. , 2019 ; Wang et al. , 2019 ) . In text generation , it cares about not only the generated contents ( i.e. , what to say ) but also the presented structure forms ( i.e. , how to say ) ( Peng et al. , 2019 ) . Similar contents or meanings can be presented with different structure forms . The structures and contents can be considered and planned separately to achieve a highly informative generated text . From an empirical view , controlling or planning the generated structure may be helpful in several aspects : i ) reducing the uncertainty of the generated contents with specific structure conditions , which may contribute to a good quality of generated text ; ii ) enhancing the interpretability of the generated text since more controlling attributes can be realized during the generation ; iii ) improving the structure , format or style consistence in specific structure-constraint generation task or specific domain generation with particular formats , such as style or paraphrase generation ( Chen et al. , 2019 ; Ficler & Goldberg , 2017 ) , poetry generation ( Deng et al. , 2020 ; Li et al. , 2020 ) , and lyric generation ( Watanabe et al. , 2018 ; Lu et al. , 2019 ) . The language structures determined by the set of grammatical rules vary from different granularity levels , such as participial construction ( pc ) is character-level , part of speech ( pos ) is word/phrase level , and sequence length is sentence level . These kinds of structure are coupled and nested together , which are realized with the contents simultaneously in most of the token by token generation . It is difficult to disentangle the contents and the text structure , and even harder to discriminate and control the different granularity level of structure during text generation . Individually controlling some specific types of structure like sequence length ( Kikuchi et al. , 2016 ) , verbal predicate ( Tu et al. , 2019 ) have been investigated in text generation . These works design specific structure representation and are inappropriately for controlling other types of structure , let alone controlling multiple types of structure simultaneously . Directly embedding the structure and adding them into the word embeddings can achieve considerable controlling capability in character-level structure during text generation , such as tone level and rhyme ( Deng et al. , 2020 ) controlling in Chinese poetry generation . While this method may fail when the controlled structure ( such as phrase level or sentence level ) needs to aware the subsequent structure during the generation process . In addition to summarizing the structure embeddings and word embeddings , SongNet ( Li et al. , 2020 ) designs another structure embeddings which are queried and incorporated globally by the summarized embeddings to renew the representation . With pre-training and fine-tuning , the SongNet ( Li et al. , 2020 ) can also achieve good controllability in tailor-designed formats 1 ( sentence level structure ) . The symbol sets for this format are particular designed and may not applicable for other type of structure . Contrast to the above works , in this paper , we are not focus on controlling specific type of structure or format , instead we propose a framework to control more general types of structure in text generation . This framework allows for controlling individual type of structure , multiple or multigranularity types of structure during text generation . The controlled types of structure are extracted from sequence templates ( any valid sentence is a valid template ) by one or several auxiliary models . The extracted structure information are regarded as conditions , and the auxiliary model can be any credible model or tool that can extract soundable structure information from template . Since we want the generation of the current token or word can aware the global structures , the bi-directional transformer encoder is adopted for structure representation and learning . The learned structure representations are further incorporated into the decoder to guide the realization of the controlled structure . The main contributions of this work are summarized as follows : • A straightforward , interpretable structure controlling text generation framework is proposed , which is capable of controlling multi-granularity sequence structure from characterlevel to sentence-level structure by explicitly incorporating the corresponding structure information . • A simple alignment method and structure embedding , representation and learning method are proposed , which are utilized for representing the multi-granularity and multiple types of structure . • A structure-aware transformer language model is proposed , and the structure representation and token representation can be learned simultaneously . The structure information are queried globally and incorporated into the token representation with attention mechanism , which contribute to controlling the generated structure . • Extensive experiments in controlling different individual type of structure and multigranularity types of structure have been conducted on Chinese lyrics corpus . The structure controllability is effective and the quality of the generated lyrics is favorable . We also conduct controlling experiments on English Penn Treebank dataset , which demonstrates similar structure controlling capability with this proposed framework . 2 RELATED WORKS . Controllable text generation has received much attention recently . Many efforts are devoted to controlling the content of the generated text ( Kiddon et al. , 2016 ; Lebret et al. , 2016 ; Shen et al. , 2019a ) . Based on conditioned RNN language model , stylistic parameters are further incorporated as conditioning context to control stylistic aspects of the generated text ( Ficler & Goldberg , 2017 ) . Basing generator on VAEs , Hu et al . ( 2017 ) proposes a generative model to generate plausible sentences with designated semantics . A simple plug and play language model is proposed in Dathathri et al . ( 2019 ) to guide controlling attributes ( e.g . topic or sentiment ) in text generation , without further training of the pre-trained language model . None of these work attempts to control the structure of the generated text . A similar approach , exemplar-based text generation , is proposed in Peng et al . ( 2019 ) , where for each input text , an exemplar text is retrieved from the training data and is then used to construct a customized decoder for outputting a target . It is ambiguously to discriminate how much the exemplar contributes to the generated structure or contents . Another similar work is SongNet ( Li et al. , 2020 ) , which are proposed to control the so called rigid formats . The rigid for- 1This format or structure is more about the length of each sentence within one paragraph or passage . mats are specifically designed with a sequence of placeholder symbols , which are utilized to control the sentence ( or sub-sentence ) length . Our method is different from all the previous methods in fourfold : 1 ) we focus on a general structure controlling framework in text generation instead of controlling a specific type of structure ; 2 ) both individual type of structure and multiple or multi-granularity types of structure can be controlled ; 3 ) instead of designing the structure symbols by ourself , we adopt the most representative structure symbols as extracted by external models to increase the applicability of our framework ; 4 ) the extracted structure information decoupled from the sequence information are learned and represented fully before them are incorporated into word information to guide the text generation . 3 MODEL DESCRIPTION . 3.1 STRUCTURE CONDITIONAL LANGUAGE MODEL . Given a natural language sequence denoted by x = [ x1 , ... , xT ] , each word denoted as xt , t = 1 , ... , T . The sequence joint distribution p ( x ) can be factorized into the product of conditional distributions p ( xt|x < t ) as follows : p ( x ) =p ( x1 , ... , xT ) = T∏ t=1 p ( xt|x < t ) . ( 1 ) A standard language model is modeling the above distribution and maximizing the corresponding likelihood accordingly ( Bengio et al. , 2003 ; Peters et al. , 2018 ; Shen et al. , 2019b ) . The above distribution considers the order structure of natural language sequence explicitly , and the conditional distribution are based on the previous word tokens . Although the standard language model can generate sentence with high quality , the generated structure is inexplicable and can not be controlled to satisfy specific generation task . Therefore , we incorporate the structure information explicitly into language model , and guide the structure generation . The joint distribution of sequence x can be reformulated as shown in Equation equation 2 : p ( x ) =p ( x1 , ... , xT ) =p ( s ) T∏ t=1 p ( xt|x < t , s ) ( 2 ) where , s represents the global structure of the natural language sequence x , the global structure can be any of the structure information like pos tags or semantic roles of the sequence , and p ( s ) is the prior distribution of the global structure . We extract the structure information with auxiliary model , and this structure information is considered as prior knowledge , which will not be optimized by the language model . The model parameters are learned by maximizing the objective function of SCLM , which is to maximize the likelihood as shown in Equation equation 3 : max θ log pθ ( x ) = T∑ t=1 log pθ ( xt|x < t , s ) ( 3 ) We utilize the Transformer ( Vaswani et al. , 2017 ) as the backbone for implementing our SCLM . The structure information is first extracted by auxiliary model and then encoded into transformer encoder . The structure information can be learned and represented fully , which can be further incorporated to contribute the aware of the structure for sequence token representation with attention mechanism . The reason why both the transformer encoder and decoder are adopted here is that we want each token in sequence to aware its local and global structure information . Only the Transformer decoder , like GPT ( Radford et al. , 2018 ) ignores the subsequent structure information of the token . The Transformer architecture is well designed and suitable for the implementation of the structure conditional language model . We only modified the input representation and few parameters of transformer . 3.2 STRUCTURE EXTRACTION . We use auxiliary model ( such as lexical tool ) g ( • ) to extract the structure information s from natural language sequence x as shown in Equation equation 4 . The auxiliary model can be regarded as prior knowledge and will not be optimized . s = g ( x ) . ( 4 ) The structure can be any sounded structure information of language sequence vary from characterlevel structure ( like participial construction ) , word-level structure ( like part of speech ) to sentencelevel structure ( positions for example ) . The multi-granularity types of sequence structure s1 , s2 , .. , si can be extracted by different auxiliary models g1 ( • ) , g2 ( • ) , ... , gi ( • ) respectively . Since each structure unit ( especially for word-level and sentence-level structure ) may contain several characters , we assign these characters with the same symbol of this kind structure . We keep the length of the structure the same with the sequence tokens . To be specific , we use the part of speech ( pos ) and participial construction ( pc ) as examples to illustrate the alignment of multi-granularity types of structure . The pos information can be extracted by many lexical analyzer tools like Jieba analyzer and Stanza ( Qi et al. , 2020 ) for Chinese and English sequence respectively . In Chinese , the pos is a type of word-level structure , and the participial construction is the character-level structure for each segmented word . We utilize the symbol collections Cpos = { n , v , r , ... } 2 from lexical analyzer ( like Jieba ) to represent the pos for each word . The symbol collections Cpc = { P , S , B , M , E } 3 are utilized to represent the pc for each character within each word . Suppose we have two levels ( word-level and character-level ) structure information for a sequence x = [ x1 , ... , xi , ... , xn ] , we can also present the word-level form of the sequence with w = [ w1 , ... , wj , ... , wnw ] , nw ≤ n , and the pos structure can be represented with s′w = [ pos1 , ... , posj , ... , posnw ] , posj ∈ Cpos ; each word contains several characters wj = [ ... , xj , k , ... ] , k ∈ [ 1 , mj ] , and the pc structure for each word are sc , j = [ ... , pcj , k , ... ] , pcj , k ∈ Cpc where ∑nw j=1mj = n. Therefore , we can obtain the word-level structure ( pos ) and character-level structure ( pc ) with the same length with the original sequence as can be shown in the following expressions : sw = [ ... , posj , .. , posj︸ ︷︷ ︸ mj , ... ] , j ∈ [ 1 , nw ] ( 5 ) sc = [ ... , pcj,1 , ... , pcj , k , ... , pcj , mj︸ ︷︷ ︸ mj , ... ] ( 6 ) The sentence level structure like positions have unique representation for each token and do not need any further processing for the alignment . With the alignment process , multi-granularity and multi-type of sequence structure can be incorporated and controlled in the generation . An illustration of multi-granularity structure information for a natural language sentence can be shown in Fig . 1 .

Summary: This paper presents a text generation model conditioned on desired structures. The proposed method is essentially a translation model from structure information (represented with multiple sequences of tokens) to a text. This study converts a text into structure information such as part of speech (POS) and participial construction (PC). Then, this paper proposes Structure Aware Transformer (SAT), which is essentially the same as the Transformer architecture. The experiments use datasets of Chinese lyrics and English Penn Treebank. This paper reports that giving structure information improved the performance in PPL and BLEU compared with GPT-2.