Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation

1 Introduction . Transformer-based neural language models ( LMs ) , such as GPT/BART [ 1–3 ] , have led a wave of new trends in natural language generation , producing texts of prominent quality . They are trained roughly on huge amounts of text corpora to reconstruct the full sentences ( i.e. , next coming tokens and missing text fragments ) . Despite their success in varieties of NLP tasks , we argue that the black-box nature of these models leads to inefficiently learning to follow constraints and incorporating prior knowledge . In controllable text generation , most relevant studies [ 4–6 ] focus on controlling high-level text attributes ( e.g. , topic , sentiment ) or simply keyword/phrase . More complex fine-grained control constraints such as “ generate a sequence of tokens with ‘ apple ’ in the first sentence which has 15 words and ‘ orange ’ or ‘ oranges ’ in the fourth sentence ” are less explored . A very recent work [ 7 ] reveals that large-scale LMs do not learn to obey the underlying constraints reliably , even in a quite simple constrained generation task ( cover all the given keywords without hallucinating new ones ) . In general text generation , existing works on various tasks reveal the benefit of incorporating task-specific prior knowledge : machine translation [ 8 ] ( e.g. , each source phrase should be translated ∗Work done during the internship at Microsoft STCA . †Corresponding author : Daxin Jiang ( djiang @ microsoft.com ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . into exactly one target phrase ) , text summarization [ 9 ] ( e.g. , the lead bias : front loading the most salient information ) , dialogue generation [ 10 ] ( e.g. , humans tend to repeat entity names or even long phrases in conversation ) . However , they either need designing specific model architectures ( e.g. , Coverage Mechanism and Copy Mechanism ) or devising well-designed learning objectives ( e.g. , GSG [ 11 ] ) . These methods require careful design case-by-case and are difficult to combine multiple arbitrary constraints or prior knowledge simultaneously . Motivated by the above research dilemma , we take the first step towards building an unified framework to handle Fine-grained Control and Prior Knowledge Integration and propose a novel module Neural Rule-Execution Tracking Machine ( NRETM ) 3 Specifically , NRETM is a trainable neural module that can be equipped with transformer-based sequence-to-sequence pre-trained LMs . It can handle constraints in any Predicate Logic Formula , which crucially includes the arbitrarily complicated relations among different control tasks . For example , the above fine-grained constraint can be written as : ( InSen ( apple , 1 ) ∧ Len ( 1 , 15 ) ) ∧ ( InSen ( orange , 4 ) ∨ InSen ( oranges , 4 ) ) To build NRETM , we combat three major challenges : i ) modeling the complicated relationships among control tasks and the logic operators ( i.e. , ∧ , ∨ ) in the constraint expressions ; ii ) an unified control system is required to execute different control tasks simultaneously and iii ) , the control signals for different control tasks should be properly aligned with the constraint expressions . NRETM uses the encoder of transformer-based pre-trained sequence-to-sequence LMs to model the relationship between control tasks and the logic operators . NRETM completes different control tasks via nondifferential Logic Trackers ( empowered by executable programs ) in an unified control progress system during the decoding process . Finally , the encoded constraint expressions and control progress signals are combined together in the transformer decoder . NRETM is fine-tuned with the pre-trained LMs ( except logical trackers ) to follow the control progress signal and predicate logic formula . NRETM reconciles symbolic computing ( that has precise logic and numerical calculation capabilities from logic trackers ) with neural language generation ( that has an exceptional ability of wording and phrasing ) , which results in both the accurate controllability and the superior generation performance . For evaluation , we select three representative benchmarks because all of them involve constraints or prior knowledge , allowing us to verify the effectiveness of our proposed NRETM model : ROCStories [ 12 ] are five-sentence stories with complicated predicate constraints over the story structure ; Commonsense Generation task [ 13 ] with the constraints of mentioning all input concepts ; TED15 Zh-En document-level machine translation benchmark [ 14 ] with prior knowledge of translating input sentences one by one . Our contributions in this work are three-fold : ( 1 ) To the best of our knowledge , we are the first to propose a general framework that incorporates control signal and prior knowledge , formulated as predicate logic constraints , into transformer-based seq2seq text generation models ; ( 2 ) We train ( or fine-tune ) the transformer-based seq2seq text generation models to follow the predicate logic constraints ( i.e. , control signal or prior knowledge ) by dynamically updating the rule execution intermediate progress value to the text decoder ; and ( 3 ) Empirical verification of the effectiveness of the proposed approach on three benchmarks . 2 Approach . This section first formalizes fine-grained content control task , then introduces an overview of proposed NRETM model , followed by diving into details of each component . 2.1 Fine-Grained Content Control . In this work , we focus on fine-grained content control task where the model input consists of predicate logic constraints x = [ x1 , . . . , xlx ] ∈ X that should be satisfied in the outputs and optional context input c = [ c1 , . . . , clc ] . The encoder takes concatenation of x and c ( i.e. , [ c ; x ] ) as input . At decoding step t , the decoder take y : t = [ y1 , · · · , yt ] ∈ Y as input and generate yt+1 . 3Our Source Code can be found in https : //github.com/GaryYufei/NRETM 2.2 Predicate Logic Constraint . We define predicate U ( a , y ) as a boolean function that indicates whether output y has satisfied control task a which could be values ( e.g. , status , total length , stop word counts ) or lexicons ( e.g. , copying particular words/phrases ) . In this paper , NRETM accepts predicate logic constraints in Conjunctive Normal Form ( CNF ) : ( U1 · · · ∨ Ui ) ∧ · · · ∧ ( Uk · · · ∨ Un ) . Each predicate logic constraint includes multiple predicates Ui and basic logic operators ( e.g. , ∨ , ∧ and brackets ) . 2.3 Neural Rule-Execution Tracking Machine . NRETM can be equipped into transformer-based sequence-to-sequence LMs . Figure 1 illustrates an overview of our neural rule-execution tracking machine ( NRETM ) . To enable LMs to follow predicate logic constraints , it is essential to 1 ) model the complicated relationships among predicates and basic logic operators ; 2 ) control multiple predicates ( i.e. , control tasks ) in the constraints simultaneously ; 3 ) combine the control signals with the predicate logic constraint expressions . For 1 ) , we treat the whole constraint expressions as natural language sentences and feed it into the transformer encoder . For 2 ) , we propose a set of unified control signals that can be used to dynamically describe the step-wise execution progress of different predicates . For 3 ) , we represent the control signals as relative position embedding and align them with encoded constraints expressions in the transformer decoder . 2.3.1 Encoding Predicate Logic Constraints . Given predicate logic constraint expression x = [ x1 , . . . , xlx ] where xi either corresponds to a predicate Ui or a basic logic operator , we feed x into the transformer encoder . Due to the tokenization strategies of pre-trained LMs , each xi may be tokenized into a continuous token sequence . x is tokenized into t = [ t1 , · · · , tlt ] where lt ≥ lx and there exists one-to-one mapping m ( ti ) = xj . We use he to denote the encoder output of x . As pre-trained LMs is trained with significant amount of natural language sentences , it should encode complicated sequential relationships within the constraints expressions . 2.3.2 Mentoring Control Progress . Specialized controlling components ( e.g. , Constrained Beam Search [ 15 ] and Copy Mechanism [ 10 ] ) can only be used for limited control tasks . To enable unified controlling system , we propose to complete control by mentoring control progress . We describe the control progress of different predicates using an unified progress state system . Each predicate Ui has a corresponding Logic Tracker QUi ( y ) , which is a non-differentiable executable program ( i.e. , written by Python ) and takes current generated outputs and returns one progress state at each generation step , formulated as follows : QUi ( y ) = S0 Ui is ∅ S1 Ui is not triggered in y ( S2 , V ) Ui is in progress in y S3 Ui is satisfied in y ( 1 ) where State S0 always is assigned to non-predicate ∅ ( i.e. , basic logic operators in the constraint expression ) ; State S1 means the tracking for predicate Ui is not triggered in y . For example , when controlling the stop word counts of the second sentence , the Logic Tracker returns S1 when the LMs are generating the first sentence ; State S2 means predicate Ui is in progress and V is the optimal intermediate value that allows fine-grained tracking . For example , in generation length control , V could be total target length minus the current length informing pre-trained LMs the number of words left to satisfy the constraint ; State S3 means Ui is satisfied in y . In short , Logic Tracker unifies different predicates by returning the same set of control signals . Global Or-Clause Update : Each Logic Tracker traces the execution progress of its corresponding predicate Ui independently . This independent tracing strategy works well in the And-Clause because all involved predicates are required to reach State S3 . However , only a subset of predicates are required to reach State S3 in the Or-Clause . Our preliminary experiment shows that the independent tracing strategy trains the model not to complete the constraints . To solve this issue , we propose to update the status of all predicates in the same Or-Clause to State S3 when one of the predicates reach State S3 . This forces all predicates finish themselves in State S3 and improves the constraint satisfaction ratio in the Or-Clause . Control Progress Matrix : Given the predicate logic constraint expressions t = [ t1 · · · , tlt ] , we further define Control Progress Matrix S to align the predicates with their control progress signals returned by Logic Trackers : S = [ C ( t , ε ) ; C ( t , y:1 ) ; · · · ; C ( t , y : t ) ] ( 2 ) C ( t , y : t ) = [ v ( t1 , y : t ) , · · · , v ( tlt , y : t ) ] ( 3 ) where ε is the empty string at first decoding step . S is a two-dimensional matrix where each row describes the control progress of all tokens in t at a single decoding step and each column describes the control progress of a single token in t along all decoding steps . Recall that basic logic operators in predicate logic constraint expressions do not require control progress tracking . Each cell Si , j in S is formulated as : Si , j = v ( ti , y : j−1 ) = { Q∅ ( y ) m ( ti ) = xk and xk is a basic logic operator QUq ( y ) m ( ti ) = xk and xk is a predicate Uq ( 4 ) Example : In Figure 2 , we are given three logic constraints , a ) copy “ car ” ; b ) the stop word ratio of the output should be 0.5 and c ) the length of second sentence should be 6 . The basic logic operators & are assigned with S0 . Length control and Stop Word Ratio maintain intermediate values ( e.g. , the residual Length and Stop Word Ratio ) . The length control is assigned with S1 when generating the first sentence because it will only be triggered in the second sentence . Copy control does not have intermediate values and its State are updated from S2 to S3 only when the corresponding words ( at step 10 in our example ) appear in the y : t. Control Progress Matrix Encoder : Control Progress Matrix S aligns the results from Logic Tracker with the encoded predicate logic constraint expressions . However , S is a non-differentiable symbolic matrix with each cell Si , j being discrete symbol S0 to S3 combined with additional numbers ( i.e. , V ) . As the encoder has already captured the inter-relationship in the predicate logic constraints , we only model each cell Si , j independently . To support various types of predicates , we treat Si , j as a string and encode it using a single-layer transformer-based encoder ShallowEncoder which shares the same vocabulary and word embeddings as the pre-trained LMs : hsij = ShallowEncoder ( Si , j ) ( 5 ) h̄sij = MeanPooling ( h s ij ) ( 6 ) where hsij ∈ Rl s ij×d , h̄sij ∈ Rd and lsij is the length of the tokenized Si , j and d is the hidden size of ShallowEncoder . We use h̄s to denote the neural representation of whole S .

Summary: This paper proposes a general framework, NRETM, for the conditional transformer-based seq2seq model. The paper first uses a state flag to indicate the expression progress of each predicate in the constraints set as a concatenation of logic trackers. The paper uses a one-layer transformer encoder to encode the state flag.  The paper utilizes the state matrix as a relative position to integrate it into the model. The results in three experiments show that the NRETM improves the transformer performance.