Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation

1 Introduction . Transformer-based neural language models ( LMs ) , such as GPT/BART [ 1–3 ] , have led a wave of new trends in natural language generation , producing texts of prominent quality . They are trained roughly on huge amounts of text corpora to reconstruct the full sentences ( i.e. , next coming tokens and missing text fragments ) . Despite their success in varieties of NLP tasks , we argue that the black-box nature of these models leads to inefficiently learning to follow constraints and incorporating prior knowledge . In controllable text generation , most relevant studies [ 4–6 ] focus on controlling high-level text attributes ( e.g. , topic , sentiment ) or simply keyword/phrase . More complex fine-grained control constraints such as “ generate a sequence of tokens with ‘ apple ’ in the first sentence which has 15 words and ‘ orange ’ or ‘ oranges ’ in the fourth sentence ” are less explored . A very recent work [ 7 ] reveals that large-scale LMs do not learn to obey the underlying constraints reliably , even in a quite simple constrained generation task ( cover all the given keywords without hallucinating new ones ) . In general text generation , existing works on various tasks reveal the benefit of incorporating task-specific prior knowledge : machine translation [ 8 ] ( e.g. , each source phrase should be translated ∗Work done during the internship at Microsoft STCA . †Corresponding author : Daxin Jiang ( djiang @ microsoft.com ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . into exactly one target phrase ) , text summarization [ 9 ] ( e.g. , the lead bias : front loading the most salient information ) , dialogue generation [ 10 ] ( e.g. , humans tend to repeat entity names or even long phrases in conversation ) . However , they either need designing specific model architectures ( e.g. , Coverage Mechanism and Copy Mechanism ) or devising well-designed learning objectives ( e.g. , GSG [ 11 ] ) . These methods require careful design case-by-case and are difficult to combine multiple arbitrary constraints or prior knowledge simultaneously . Motivated by the above research dilemma , we take the first step towards building an unified framework to handle Fine-grained Control and Prior Knowledge Integration and propose a novel module Neural Rule-Execution Tracking Machine ( NRETM ) 3 Specifically , NRETM is a trainable neural module that can be equipped with transformer-based sequence-to-sequence pre-trained LMs . It can handle constraints in any Predicate Logic Formula , which crucially includes the arbitrarily complicated relations among different control tasks . For example , the above fine-grained constraint can be written as : ( InSen ( apple , 1 ) ∧ Len ( 1 , 15 ) ) ∧ ( InSen ( orange , 4 ) ∨ InSen ( oranges , 4 ) ) To build NRETM , we combat three major challenges : i ) modeling the complicated relationships among control tasks and the logic operators ( i.e. , ∧ , ∨ ) in the constraint expressions ; ii ) an unified control system is required to execute different control tasks simultaneously and iii ) , the control signals for different control tasks should be properly aligned with the constraint expressions . NRETM uses the encoder of transformer-based pre-trained sequence-to-sequence LMs to model the relationship between control tasks and the logic operators . NRETM completes different control tasks via nondifferential Logic Trackers ( empowered by executable programs ) in an unified control progress system during the decoding process . Finally , the encoded constraint expressions and control progress signals are combined together in the transformer decoder . NRETM is fine-tuned with the pre-trained LMs ( except logical trackers ) to follow the control progress signal and predicate logic formula . NRETM reconciles symbolic computing ( that has precise logic and numerical calculation capabilities from logic trackers ) with neural language generation ( that has an exceptional ability of wording and phrasing ) , which results in both the accurate controllability and the superior generation performance . For evaluation , we select three representative benchmarks because all of them involve constraints or prior knowledge , allowing us to verify the effectiveness of our proposed NRETM model : ROCStories [ 12 ] are five-sentence stories with complicated predicate constraints over the story structure ; Commonsense Generation task [ 13 ] with the constraints of mentioning all input concepts ; TED15 Zh-En document-level machine translation benchmark [ 14 ] with prior knowledge of translating input sentences one by one . Our contributions in this work are three-fold : ( 1 ) To the best of our knowledge , we are the first to propose a general framework that incorporates control signal and prior knowledge , formulated as predicate logic constraints , into transformer-based seq2seq text generation models ; ( 2 ) We train ( or fine-tune ) the transformer-based seq2seq text generation models to follow the predicate logic constraints ( i.e. , control signal or prior knowledge ) by dynamically updating the rule execution intermediate progress value to the text decoder ; and ( 3 ) Empirical verification of the effectiveness of the proposed approach on three benchmarks . 2 Approach . This section first formalizes fine-grained content control task , then introduces an overview of proposed NRETM model , followed by diving into details of each component . 2.1 Fine-Grained Content Control . In this work , we focus on fine-grained content control task where the model input consists of predicate logic constraints x = [ x1 , . . . , xlx ] ∈ X that should be satisfied in the outputs and optional context input c = [ c1 , . . . , clc ] . The encoder takes concatenation of x and c ( i.e. , [ c ; x ] ) as input . At decoding step t , the decoder take y : t = [ y1 , · · · , yt ] ∈ Y as input and generate yt+1 . 3Our Source Code can be found in https : //github.com/GaryYufei/NRETM 2.2 Predicate Logic Constraint . We define predicate U ( a , y ) as a boolean function that indicates whether output y has satisfied control task a which could be values ( e.g. , status , total length , stop word counts ) or lexicons ( e.g. , copying particular words/phrases ) . In this paper , NRETM accepts predicate logic constraints in Conjunctive Normal Form ( CNF ) : ( U1 · · · ∨ Ui ) ∧ · · · ∧ ( Uk · · · ∨ Un ) . Each predicate logic constraint includes multiple predicates Ui and basic logic operators ( e.g. , ∨ , ∧ and brackets ) . 2.3 Neural Rule-Execution Tracking Machine . NRETM can be equipped into transformer-based sequence-to-sequence LMs . Figure 1 illustrates an overview of our neural rule-execution tracking machine ( NRETM ) . To enable LMs to follow predicate logic constraints , it is essential to 1 ) model the complicated relationships among predicates and basic logic operators ; 2 ) control multiple predicates ( i.e. , control tasks ) in the constraints simultaneously ; 3 ) combine the control signals with the predicate logic constraint expressions . For 1 ) , we treat the whole constraint expressions as natural language sentences and feed it into the transformer encoder . For 2 ) , we propose a set of unified control signals that can be used to dynamically describe the step-wise execution progress of different predicates . For 3 ) , we represent the control signals as relative position embedding and align them with encoded constraints expressions in the transformer decoder . 2.3.1 Encoding Predicate Logic Constraints . Given predicate logic constraint expression x = [ x1 , . . . , xlx ] where xi either corresponds to a predicate Ui or a basic logic operator , we feed x into the transformer encoder . Due to the tokenization strategies of pre-trained LMs , each xi may be tokenized into a continuous token sequence . x is tokenized into t = [ t1 , · · · , tlt ] where lt ≥ lx and there exists one-to-one mapping m ( ti ) = xj . We use he to denote the encoder output of x . As pre-trained LMs is trained with significant amount of natural language sentences , it should encode complicated sequential relationships within the constraints expressions . 2.3.2 Mentoring Control Progress . Specialized controlling components ( e.g. , Constrained Beam Search [ 15 ] and Copy Mechanism [ 10 ] ) can only be used for limited control tasks . To enable unified controlling system , we propose to complete control by mentoring control progress . We describe the control progress of different predicates using an unified progress state system . Each predicate Ui has a corresponding Logic Tracker QUi ( y ) , which is a non-differentiable executable program ( i.e. , written by Python ) and takes current generated outputs and returns one progress state at each generation step , formulated as follows : QUi ( y ) = S0 Ui is ∅ S1 Ui is not triggered in y ( S2 , V ) Ui is in progress in y S3 Ui is satisfied in y ( 1 ) where State S0 always is assigned to non-predicate ∅ ( i.e. , basic logic operators in the constraint expression ) ; State S1 means the tracking for predicate Ui is not triggered in y . For example , when controlling the stop word counts of the second sentence , the Logic Tracker returns S1 when the LMs are generating the first sentence ; State S2 means predicate Ui is in progress and V is the optimal intermediate value that allows fine-grained tracking . For example , in generation length control , V could be total target length minus the current length informing pre-trained LMs the number of words left to satisfy the constraint ; State S3 means Ui is satisfied in y . In short , Logic Tracker unifies different predicates by returning the same set of control signals . Global Or-Clause Update : Each Logic Tracker traces the execution progress of its corresponding predicate Ui independently . This independent tracing strategy works well in the And-Clause because all involved predicates are required to reach State S3 . However , only a subset of predicates are required to reach State S3 in the Or-Clause . Our preliminary experiment shows that the independent tracing strategy trains the model not to complete the constraints . To solve this issue , we propose to update the status of all predicates in the same Or-Clause to State S3 when one of the predicates reach State S3 . This forces all predicates finish themselves in State S3 and improves the constraint satisfaction ratio in the Or-Clause . Control Progress Matrix : Given the predicate logic constraint expressions t = [ t1 · · · , tlt ] , we further define Control Progress Matrix S to align the predicates with their control progress signals returned by Logic Trackers : S = [ C ( t , ε ) ; C ( t , y:1 ) ; · · · ; C ( t , y : t ) ] ( 2 ) C ( t , y : t ) = [ v ( t1 , y : t ) , · · · , v ( tlt , y : t ) ] ( 3 ) where ε is the empty string at first decoding step . S is a two-dimensional matrix where each row describes the control progress of all tokens in t at a single decoding step and each column describes the control progress of a single token in t along all decoding steps . Recall that basic logic operators in predicate logic constraint expressions do not require control progress tracking . Each cell Si , j in S is formulated as : Si , j = v ( ti , y : j−1 ) = { Q∅ ( y ) m ( ti ) = xk and xk is a basic logic operator QUq ( y ) m ( ti ) = xk and xk is a predicate Uq ( 4 ) Example : In Figure 2 , we are given three logic constraints , a ) copy “ car ” ; b ) the stop word ratio of the output should be 0.5 and c ) the length of second sentence should be 6 . The basic logic operators & are assigned with S0 . Length control and Stop Word Ratio maintain intermediate values ( e.g. , the residual Length and Stop Word Ratio ) . The length control is assigned with S1 when generating the first sentence because it will only be triggered in the second sentence . Copy control does not have intermediate values and its State are updated from S2 to S3 only when the corresponding words ( at step 10 in our example ) appear in the y : t. Control Progress Matrix Encoder : Control Progress Matrix S aligns the results from Logic Tracker with the encoded predicate logic constraint expressions . However , S is a non-differentiable symbolic matrix with each cell Si , j being discrete symbol S0 to S3 combined with additional numbers ( i.e. , V ) . As the encoder has already captured the inter-relationship in the predicate logic constraints , we only model each cell Si , j independently . To support various types of predicates , we treat Si , j as a string and encode it using a single-layer transformer-based encoder ShallowEncoder which shares the same vocabulary and word embeddings as the pre-trained LMs : hsij = ShallowEncoder ( Si , j ) ( 5 ) h̄sij = MeanPooling ( h s ij ) ( 6 ) where hsij ∈ Rl s ij×d , h̄sij ∈ Rd and lsij is the length of the tokenized Si , j and d is the hidden size of ShallowEncoder . We use h̄s to denote the neural representation of whole S .

Summary: The authors propose an approach aimed at encouraging transformer-based sequence-to-sequence models to respect constraints on the text they generate. In particular, the authors propose a particular syntactic representation of whether propositional logic-style constraints have been satisfied in some (partially) generated text, which is tracked throughout decoding. This syntactic representation is a function both of the current decoding step as well as the index of the particular constraint, and is updated dynamically as decoding progresses. The authors tokenize and encode these representations, and use them inside a relative-position-style attention mechanism. The authors fine-tune their models using these dynamic representations and they show that this allows for better control over various aspects of the generated text on RocStories, Commongen, and Zh-En document level translation tasks, with comparable or better quality.