Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation
Year: 2022

Abstract:
Sequence-to-Sequence (Seq2Seq) neural text generation models, especially the pre-trained ones (e.g., BART and T5), have exhibited compelling performance on various natural language generation tasks. However, the black-box nature of these models limits their application in tasks where specific rules (e.g., controllable constraints, prior knowledge) need to be executed. Previous works either design specific model structures (e.g., Copy Mechanism corresponding to the rule “the generated output should include certain words in the source input”) or implement specialized inference algorithms (e.g., Constrained Beam Search) to execute particular rules through the text generation. These methods require the careful design case-by-case and are difficult to support multiple rules concurrently. In this paper, we propose a novel module named Neural Rule-Execution Tracking Machine (NRETM) that can be equipped into various transformer-based generators to leverage multiple rules simultaneously to guide the neural generation model for superior generation performance in an unified and scalable way. Extensive experiments on several benchmarks verify the effectiveness of our proposed model in both controllable and general text generation tasks.

heading: 1 Introduction
text: Transformer-based neural language models (LMs), such as GPT/BART [1–3], have led a wave of new trends in natural language generation, producing texts of prominent quality. They are trained roughly on huge amounts of text corpora to reconstruct the full sentences (i.e., next coming tokens and missing text fragments). Despite their success in varieties of NLP tasks, we argue that the black-box nature of these models leads to inefficiently learning to follow constraints and incorporating prior knowledge.
In controllable text generation, most relevant studies [4–6] focus on controlling high-level text attributes (e.g., topic, sentiment) or simply keyword/phrase. More complex fine-grained control constraints such as “generate a sequence of tokens with ‘apple’ in the first sentence which has 15 words and ‘orange’ or ‘oranges’ in the fourth sentence” are less explored. A very recent work [7] reveals that large-scale LMs do not learn to obey the underlying constraints reliably, even in a quite simple constrained generation task (cover all the given keywords without hallucinating new ones). In general text generation, existing works on various tasks reveal the benefit of incorporating task-specific prior knowledge: machine translation [8] (e.g., each source phrase should be translated ∗Work done during the internship at Microsoft STCA. †Corresponding author: Daxin Jiang (djiang@microsoft.com).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
into exactly one target phrase), text summarization [9] (e.g., the lead bias: front loading the most salient information), dialogue generation [10] (e.g., humans tend to repeat entity names or even long phrases in conversation). However, they either need designing specific model architectures (e.g., Coverage Mechanism and Copy Mechanism) or devising well-designed learning objectives (e.g., GSG [11]). These methods require careful design case-by-case and are difficult to combine multiple arbitrary constraints or prior knowledge simultaneously.
Motivated by the above research dilemma, we take the first step towards building an unified framework to handle Fine-grained Control and Prior Knowledge Integration and propose a novel module Neural Rule-Execution Tracking Machine (NRETM) 3 Specifically, NRETM is a trainable neural module that can be equipped with transformer-based sequence-to-sequence pre-trained LMs. It can handle constraints in any Predicate Logic Formula, which crucially includes the arbitrarily complicated relations among different control tasks. For example, the above fine-grained constraint can be written as:(
InSen(apple, 1 ) ∧ Len(1 , 15 ) ) ∧ ( InSen(orange, 4 ) ∨ InSen(oranges, 4 ) ) To build NRETM, we combat three major challenges: i) modeling the complicated relationships among control tasks and the logic operators (i.e., ∧, ∨) in the constraint expressions; ii) an unified control system is required to execute different control tasks simultaneously and iii), the control signals for different control tasks should be properly aligned with the constraint expressions. NRETM uses the encoder of transformer-based pre-trained sequence-to-sequence LMs to model the relationship between control tasks and the logic operators. NRETM completes different control tasks via nondifferential Logic Trackers (empowered by executable programs) in an unified control progress system during the decoding process. Finally, the encoded constraint expressions and control progress signals are combined together in the transformer decoder. NRETM is fine-tuned with the pre-trained LMs (except logical trackers) to follow the control progress signal and predicate logic formula. NRETM reconciles symbolic computing (that has precise logic and numerical calculation capabilities from logic trackers) with neural language generation (that has an exceptional ability of wording and phrasing), which results in both the accurate controllability and the superior generation performance.
For evaluation, we select three representative benchmarks because all of them involve constraints or prior knowledge, allowing us to verify the effectiveness of our proposed NRETM model: ROCStories [12] are five-sentence stories with complicated predicate constraints over the story structure; Commonsense Generation task [13] with the constraints of mentioning all input concepts; TED15 Zh-En document-level machine translation benchmark [14] with prior knowledge of translating input sentences one by one.
Our contributions in this work are three-fold: (1) To the best of our knowledge, we are the first to propose a general framework that incorporates control signal and prior knowledge, formulated as predicate logic constraints, into transformer-based seq2seq text generation models; (2) We train (or fine-tune) the transformer-based seq2seq text generation models to follow the predicate logic constraints(i.e., control signal or prior knowledge) by dynamically updating the rule execution intermediate progress value to the text decoder; and (3) Empirical verification of the effectiveness of the proposed approach on three benchmarks.

heading: 2 Approach
text: This section first formalizes fine-grained content control task, then introduces an overview of proposed NRETM model, followed by diving into details of each component.

heading: 2.1 Fine-Grained Content Control
text: In this work, we focus on fine-grained content control task where the model input consists of predicate logic constraints x = [x1, . . . , xlx ] ∈ X that should be satisfied in the outputs and optional context input c = [c1, . . . , clc ]. The encoder takes concatenation of x and c (i.e., [c;x]) as input. At decoding step t, the decoder take y:t = [y1, · · · , yt] ∈ Y as input and generate yt+1.
3Our Source Code can be found in https://github.com/GaryYufei/NRETM

heading: 2.2 Predicate Logic Constraint
text: We define predicate U(a, y) as a boolean function that indicates whether output y has satisfied control task a which could be values (e.g., status, total length, stop word counts) or lexicons (e.g., copying particular words/phrases). In this paper, NRETM accepts predicate logic constraints in Conjunctive Normal Form (CNF): ( U1 · · · ∨ Ui ) ∧ · · · ∧ ( Uk · · · ∨ Un ) . Each predicate logic constraint includes multiple predicates Ui and basic logic operators (e.g., ∨, ∧ and brackets).

heading: 2.3 Neural Rule-Execution Tracking Machine
text: NRETM can be equipped into transformer-based sequence-to-sequence LMs. Figure 1 illustrates an overview of our neural rule-execution tracking machine (NRETM). To enable LMs to follow predicate logic constraints, it is essential to 1) model the complicated relationships among predicates and basic logic operators; 2) control multiple predicates (i.e., control tasks) in the constraints simultaneously; 3) combine the control signals with the predicate logic constraint expressions. For 1), we treat the whole constraint expressions as natural language sentences and feed it into the transformer encoder. For 2), we propose a set of unified control signals that can be used to dynamically describe the step-wise execution progress of different predicates. For 3), we represent the control signals as relative position embedding and align them with encoded constraints expressions in the transformer decoder.

heading: 2.3.1 Encoding Predicate Logic Constraints
text: Given predicate logic constraint expression x = [x1, . . . , xlx ] where xi either corresponds to a predicate Ui or a basic logic operator, we feed x into the transformer encoder. Due to the tokenization strategies of pre-trained LMs, each xi may be tokenized into a continuous token sequence. x is tokenized into t = [t1, · · · , tlt ] where lt ≥ lx and there exists one-to-one mapping m(ti) = xj . We use he to denote the encoder output of x. As pre-trained LMs is trained with significant amount of natural language sentences, it should encode complicated sequential relationships within the constraints expressions.

heading: 2.3.2 Mentoring Control Progress
text: Specialized controlling components (e.g., Constrained Beam Search [15] and Copy Mechanism [10]) can only be used for limited control tasks. To enable unified controlling system, we propose to complete control by mentoring control progress. We describe the control progress of different predicates using an unified progress state system. Each predicate Ui has a corresponding Logic Tracker QUi(y), which is a non-differentiable executable program (i.e., written by Python) and takes
current generated outputs and returns one progress state at each generation step, formulated as follows:
QUi(y) =  S0 Ui is ∅ S1 Ui is not triggered in y
(S2,V) Ui is in progress in y S3 Ui is satisfied in y
(1)
where State S0 always is assigned to non-predicate ∅ (i.e., basic logic operators in the constraint expression); State S1 means the tracking for predicate Ui is not triggered in y. For example, when controlling the stop word counts of the second sentence, the Logic Tracker returns S1 when the LMs are generating the first sentence; State S2 means predicate Ui is in progress and V is the optimal intermediate value that allows fine-grained tracking. For example, in generation length control, V could be total target length minus the current length informing pre-trained LMs the number of words left to satisfy the constraint; State S3 means Ui is satisfied in y. In short, Logic Tracker unifies different predicates by returning the same set of control signals.
Global Or-Clause Update: Each Logic Tracker traces the execution progress of its corresponding predicate Ui independently. This independent tracing strategy works well in the And-Clause because all involved predicates are required to reach State S3. However, only a subset of predicates are required to reach State S3 in the Or-Clause. Our preliminary experiment shows that the independent tracing strategy trains the model not to complete the constraints. To solve this issue, we propose to update the status of all predicates in the same Or-Clause to State S3 when one of the predicates reach State S3. This forces all predicates finish themselves in State S3 and improves the constraint satisfaction ratio in the Or-Clause.
Control Progress Matrix: Given the predicate logic constraint expressions t = [t1 · · · , tlt ], we further define Control Progress Matrix S to align the predicates with their control progress signals returned by Logic Trackers:
S = [C(t, ε); C(t,y:1); · · · ; C(t,y:t)] (2) C(t,y:t) = [v(t1,y:t), · · · , v(tlt ,y:t)] (3)
where ε is the empty string at first decoding step. S is a two-dimensional matrix where each row describes the control progress of all tokens in t at a single decoding step and each column describes the control progress of a single token in t along all decoding steps. Recall that basic logic operators in predicate logic constraint expressions do not require control progress tracking. Each cell Si,j in S is formulated as:
Si,j = v(ti,y:j−1) = {
Q∅(y) m(ti) = xk and xk is a basic logic operator QUq (y) m(ti) = xk and xk is a predicate Uq
(4)
Example: In Figure 2, we are given three logic constraints, a) copy “car”; b) the stop word ratio of the output should be 0.5 and c) the length of second sentence should be 6. The basic logic operators & are assigned with S0. Length control and Stop Word Ratio maintain intermediate values (e.g., the residual Length and Stop Word Ratio). The length control is assigned with S1 when generating the first sentence because it will only be triggered in the second sentence. Copy control does not have intermediate values and its State are updated from S2 to S3 only when the corresponding words (at step 10 in our example) appear in the y:t.
Control Progress Matrix Encoder: Control Progress Matrix S aligns the results from Logic Tracker with the encoded predicate logic constraint expressions. However, S is a non-differentiable symbolic matrix with each cell Si,j being discrete symbol S0 to S3 combined with additional numbers (i.e., V ). As the encoder has already captured the inter-relationship in the predicate logic constraints, we only model each cell Si,j independently. To support various types of predicates, we treat Si,j as a string and encode it using a single-layer transformer-based encoder ShallowEncoder which shares the same vocabulary and word embeddings as the pre-trained LMs:
hsij = ShallowEncoder(Si,j) (5) h̄sij = MeanPooling(h s ij) (6)
where hsij ∈ Rl s ij×d, h̄sij ∈ Rd and lsij is the length of the tokenized Si,j and d is the hidden size of ShallowEncoder. We use h̄s to denote the neural representation of whole S.

heading: 2.3.3 Combining Predicate Logic Constraint with Control Progress Matrix
text: Finally, we combine the encoded Predicate Logic Constraints he with the encoded Control Progress Matrix h̄s in the transformer-based pre-trained LMs. Injecting h̄s into the transformer encoder would result in encoder content re-computation at each decoding step and stop the standard parallel training for transformer-based decoders. In addition, as Control Progress Matrix incrementally increases as the decoding goes on, it is reasonable to equip h̄s into the transformer decoder. Given the encoder output he, decoder input y:t, the probability of the next token yt+1 can be calculated by:
hdt = KV(W s qy:t,W s ky:t,W s vy:t (7)
ot+1 = CrossKV(W c qh d t ,W c kh e,Wcvh e) (8)
p(yt+1|x1:lx , y1:t) = softmax(Wo ot+1) (9)
where ot+1 ∈ Rdc is the hidden state at step t with dc the hidden size, and Wo ∈ R|V |×dc , Both KV and CrossKV are the standard key-value self-attention described in [16]. In the CrossKV which takes hdt and h
e as input, the resulting attention score matrix has the same size as S, making CrossKV suitable to incorporate our Control Progress Matrix.
Control Progress Matrix as Relative Position: Inspired by [17] which incorporates token relative positions into the self-attention module, we propose to inject Control Progress Matrix as the “relative positions” between encoder output he and current decoder input y:t in the cross-attention (Eq. 8) module. Following this approach, we linearly project each h̄ij into Control Progress Matrix key hkij = W f k · h̄sij + b f k and Control Progress Matrix Value h v ij = W f v · h̄sij + bfv . All transformer decoder layers share the same representations. Eq. 8 is changed to:
ot+1 = R(W c qH d t ,W c kH e,WcvH e,hk,hv) (10)
where Rlx×t×d and R is the Self-Attention function with relative position, defined as follows:
R(q,k,v,mk,mv)j = lx∑ i=1 ai,j(vi + m v i,j) (11)
where a∗,j = Softmax (e∗,j) and ei,j = qj(ki + mki,j) T d−1/2.
2.4 Why NRETM Could Satisfy Constraints
A powerful implicit compulsion comes from the combined force of two aspects: 1) before generating the EOS token (i.e., End-Of-Sequence Token), all the predicate constraints should be satisfied. As demonstrated in Fig 2, all elements in Control Progress Matrix are set to “satisfied” (i.e., S3) at EOS position; 2) The pre-trained LMs are trained to generate text with limited length. Such a soft way of combining symbolic operators (good at logical and mathematical calculations ) and neural operators (good at wording and phrasing) can retain their respective strengths to the utmost extent.
2.5 What If NRETM Fails to Satisfy Constraints
NRETM does not forces the pre-trained LMs to execute the hard constraints on the text decoder explicitly, but instead, provides Control Progress Matrix as input features describing rule execution
intermediate values to the text decoder. That is, no explicit effect when NRETM fails to satisfy the constraints. It is possible that our text generators decide to stop the generation before completing all constraints. In our experiments, NRETM has less than 1% chance not to complete all constraints.
2.6 The Generalization Ability of NRETM
The generalization ability of NRETM comes from two aspects: 1) NRETM can construct new constraints via combining pre-trained predicates with basic logic operators in arbitrarily complicated ways; 2) To expand a new predicate, users only need to implement the corresponding Logic Trackers, which returns S1-S3 and intermediate values, via executable programs.

heading: 3 Experiment
text: We test our proposed NRETM on the controllable text generation and general text generation tasks. For controllable text generation, we verify NRETM on the complex fine-grained control instructions in the ROCStories Benchmark [12]. Further, we test NRETM on the general text generation tasks, commonsense generation and document-level machine translation, to show that NRETM can efficiently integrate prior knowledge into seq2seq models towards superior generation performance.

heading: 3.1 Controllable ROC Stories
text: ROCStories is a corpus of five-sentence stories that capture a rich set of causal and temporal commonsense relations between daily events. Following [18], we extract key phrases from the ground-truth stories. In this experiment, we design multiple predicate logic constraints to inform NRETM about the stories to be generated and verify if NRETM can follow these constraints exactly.
Predicate Logic Formulation As shown in table 1, five constraints with increasing difficulties are used: (1) Generate a story with storyline wi in the pith sentence. (2) Generate a story with an ordered storyline w1, · · · , w4 (3) Generate a story with storyline wi in the pith sentence which has lpi words (i = 1, 2). (4) Generate a storyline w1 in the p1th sentence which has lp1 words or sp1 stop words and w2 in the p2th sentence that does not mention w3 (5) Generate a storyline wi in the pith sentence which has lpi words or spi stop words (i = 1, 2).
Baselines and Metrics Both baseline and NRETM use T5-Base model [19]. We report Constraints Success Ratio (CSR), the ratio of stories that completely satisfy the given constraints. We additionally report ROUGE-L (RL), BERT-Score (BS), BLEU-1/4 (B1/4) to show the generated stories quality.
Main Results As shown in Table 1, in all five predicate logic constraints, compared to the T5 model, the NRETM model achieves higher Constraint Success Ratio and maintains a similar level of ROUGHL, showing that the NRETM model can be flexibly controlled without loss of generated text quality.
The gap in CSR between the T5 and NRETM model is moderate in the first two constraints with simple token permutations. However, the success ratio of T5 model drops significantly given constraints that requires long-range numerical tracking (e.g., sentence length and the count of stop words).

heading: 3.2 Commonsense Generation
text: COMMONGEN is a generation benchmark dataset target explicitly test machines for the ability of generative commonsense reasoning. Given a set of common concepts the task is to generate a coherent sentence describing an everyday scenario using these concepts.
Predicate Logic Formulation The input is an unordered set of n concepts x = {xi}ni=1. From the expectation of COMMONGEN, one easily obtained prior knowledge is that each xi must appear in output y. The corresponding predicate logic constraint Pc is:
Pc = ∧ni=1 ( Copy(xi) ) where y will appear by default, for the sake of brevity, we have omitted y in predicate Copy. Another prior knowledge comes from the observation that generating y requires giving the correct morphological inflections of the concept word rather than copy its original form. Let x̃i = {x̃ik} |x̃i| k=1 denote all inflections of xi. y covers concept xi, if at least one of {x̃ik} |x̃i| k=1 appears. The constraint P̂c is:
P̂c = ∧ni=1 ( ∨|x̃ i| j=1 Copy(x̃ i j) )
Baselines and Metrics We experiment with T5-Base and T5-Large. We equip NRETM into the T5Large and T5-Base model to incorporate Pc and P̂c respectively (+ NRETM Pc) (+ NRETM P̂c). Grid Beam Search (GBS) [20] (+ G) is a well-designed decoding method that ensures the generation model satisfies the lexical constraints. We only apply GBS to the T5-Base model due to the memory constraint. Following the suggestions in [13], we use CIDEr [21] and SPICE [22] to automatically assess the quality of generated texts. We calculate constraint satisfaction for all constraints (ALL), novel constraints (Novel) and seen constraints (Seen).
Main Results Table 2 shows that the NRETM model improves the constraint satisfaction over the baselines for all cases, achieving close to 100% (i.e., 99.5% and 99.2%). While GBS achieves perfect constraint satisfaction (i.e., 100%), doing so significantly degrades the output text quality (more than 50 CIDEr), indicating the necessity integrating prior knowledge in training rather than inference. In addition, both prior knowledge Pc and P̂c have a positive effect on our model, improving our T5-large baseline by 3.1 and 5.0 CIDEr score, respectively. Finally, our T5-Large + NRETM P̂c model outperforms the previous state-of-the-art result [23], which integrates the ConceptNet [24] into the BART model, suggesting that our incorporated task-specific prior knowledge could be as powerful as knowledge from large-scale hand-crafted corpus. All of the above shows how potential it is to find a method that could execute multiple rules effectively.

heading: 3.3 Document-Level Machine Translation
text: Document-level machine translation tasks is a general text generation task, where the goal is to translate segments of text (up to an entire document). Following [14], we use TED15 Zh-En (from IWSLT 2014 and 2015 [25, 26]) as training and validation set and 2010-2013 TED as the test set.
Predicate Logic Formulation The input is an ordered set of n sentences in the source language that form a document x = {xi}ni=1, the expected output is a translated document y = {yi}ni=1 in the target language. We observed that neural model is prone to sentence correspondence confusion (the ith sentence in source document is translated as the jth sentence in target document) when doing document-level translation. To alleviate this problem, we propose incorporating Doc-mBART25 with prior knowledge: each source sentence should be translated only once. It is formulated as:
TranslatedOnce(xi) = { S3 θ(y:t) > i S2 θ(y:t) = i S1 θ(y:t) < i
(12)
where θ(·) returns the line number of yt in y, as t is monotonic during generation, the status only set to be 2 once. To trace the sentence translation progress, we add an additional End-Of-Sentence token at the end of each sentence to the training data. Once NRETM finishes the ith sentence (generating an end-of-sentence token) in the decoder, we assume that the ith sentence in the encoder has been translated. The predicate logic constraint Pc of this task can be formulated as:
Pc = ∧ni=1 ( TranslatedOnce(xi) ) Baselines and Metrics We combine our NRETM Pc component with the Doc-mBART25 model proposed in [3] which is a state-of-the-art multilingual pre-trained language model. We compare this model with the state-of-the-art non-pretraining and pretraining approaches, including HAN (Hierarchical Attention Networks) [14], Doc-mBART25 and Sen-mBART25 proposed in [3]. When implementing our model, we use the same pre-processing method, blocks segmentation strategy and beam search setting as [3]. TED15 Zh-En provides sentence-to-sentence translation from Chinese to English. We use both document-level (d-BLEU) and sentence-level (s-BLEU) to measure the similarities between generated target document and the source document. We also report Sentence Aligned Ratio (SAR), the ratio of source and target documents with the same sentence count, to show the effectiveness of our control over this translation prior knowledge.
Main Results Table 3 shows that the NRETM Pc component helps the Doc-mBART25 model to better capture the sentence-level corresponding relationship between the source and target documents. In particular, sentence-level alignment ratio is improved from 98.7% to 100%. The improvement in s-BLEU (+ 1.1 BLEU) also confirms that our final Doc-mBART25 + NRETM Pc model learns to translate sentences based on the sentence order in source documents.

heading: 3.4 Discussion
text: Updating Progress in Encoder In Sec 2.3.3, we incorporate the Control Progress Matrix as relative position embeddings in the decoder. To show the importance of this design choice, we conduct an ablation study in Table 4 where the row of Control Progress Matrix is concatenated with the encoder output. We find that updating the rule execution progress information with the encoder output contributes little to improve the CSR. This shows that simply extracting rule execution intermediate values is not enough. This could be because the encoder that encodes the rule execution intermediate values cannot effectively broadcast this information into text decoders.
NRETM Robustness The above experiment results are based on the perfect training data. In this section, we explore the effect of training data noise. We corrupt the training data by replacing the input commonsense keywords with a random sampled one under the probability 5%, 10%, 15%, 25%, and 50% (Validation and Test Split remain unchanged). As shown in Table 5, in all noise levels,
NRETM successfully achieves higher constraint coverage (i.e, Cons) and CIDEr score than the T5 baseline model, showing that NRETM is robust to the training data noise. It is worthwhile to note that the main goal of NRETM is to incorporate constraints that are satisfied by the training data into transformer-based seq2seq text generators. It is reasonable to assume that in practice, the noise level should be relatively low (e.g., 0% - 10%).
Zero-Shot Execution In Table 1, we show that the pre-trained language model T5 cannot handle complicated and fine-grained constraints even after fine-tuning. Here, we further demonstrate that NRETM model is capable to handle zero-shot rule execution. We train the T5 and NRETM model to only mention keywords in the 3rd, 4th and 5th sentence and test these models to mention keywords in the first and second sentence of the whole story. As shown in Table 6, although both T5 and NRETM model mention most of the keywords (95.7% and 98.3% respectively) in the generated story, the T5 model only mention 19.7% of keywords
in the correct sentence and the NRETM model makes 97.7% of keywords correct. This is becuase the T5 model cannot recognize the novel sentence index (i.e., the first and second) during the generation. The logic tracker helps the NRETM model to generalize to handle these cases.
Running Efficiency We compare the inference time (in minutes) for NRETM on the test split of commonsense generation task in Table 7. All models use the beam search decoding algorithm with beam size 5. Adding NRETM components to T5-Base and T5-Large approximately double the inference time. While the Grid Beam Search (GBS) algorithm uses a much longer inference time. Compared to existing constrained decoding approaches, NRETM uses much less computational costs.

heading: 4 Related Work
text: NRETM is mainly related to two lines of research work in text generation: constrained decoding and prior knowledge integration.
Constrained Decoding NEUROLOGICEarly work in constrained decoding can be traced back to dual decomposition and lagrangian relaxation [27, 28]. These works focus on sequence labelling and parsing problems where the solution space is relatively small, compared to text generation tasks. Research efforts in text generation tasks [29–31] involve controllable generation methods where the generators are trained on text data with the labeled target attributes. CTRL [4], PPLM [6] and CoCon [32] are recent approaches that built on the transformer-based large-scale pretrained LMs,
they pay more attention on controlling high-level attributes, phrases and keywords. [33, 34] propose to trace the control task progress in the text generation decoder. [33] treats the control signal as training loss in memory network and [34] treats the control signal as additional input features. [35] controls the text generation outputs via mentoring the output gradient. However, these work only focus on specific controlling tasks such as phrases copying and generation length. While NRETM focuses on controlling text generation to follow arbitrary logical constraints, leading to a fine-grained control. They can be seen as special cases of NRETM. Recently, GDC [36] permits to specify both pointwise and distributional constraints over the target LMs. Very recently, NEUROLOGIC [7] was proposed to generate fluent text while satisfying complex lexical constraints (in a predicate logic form). There are three main differences between NRETM and NEUROLOGIC: 1) NEUROLOGIC only provides control constraints over the text generators. Instead, NRETM is a general framework that provides control constraints (e.g., copy or not copy words) and prior knowledge (e.g., translating sentences one by one). NEUROLOGIC can be viewed as a special case of NRETM; 2) NEUROLOGIC is an inference-only algorithm that only controls the model to generate or avoid specific words or phrases at decoding time; while NRETM fine-tunes the pre-trained transformer-based seq2seq text generators with the predicate logic constraints; 3) NEUROLOGIC only supports the “copy” predicate (i.e., to generate or not to generate specific words or phrases), while NRETM is a general framework that supports various control predicates. NRETM supports 6 kinds of logic operators in this paper, and it is also possible for users to expand new logic operators.
Prior Knowledge Integration Existing efforts [37–41] to incorporate prior knowledge into sequence-to-sequence framework either resort to modifying model architectures, including adding external memory components, specialized decoding method or designing training objectives, including minimum risk training. These methods usually can only support to inject one narrow type of knowledge into the neural models. To the best of our knowledge, we first attempt to formalize the prior knowledge integration in seq2seq generation as text generation that conforms to predicate logic constraints.

heading: 5 Conclusion and Future Work
text: In this paper, we propose a unified controllable generation framework that leverages predicate logic constraints to implement efficient complex fine-grained control and scalable prior knowledge integration. We explore and compare two controllable strategies: dynamic tracking and static strategy, and show that the proposed dynamic tracking mechanism significantly outperforms the static ones. Empirical results on three benchmarks indicate that NRETM could achieve accurate control and exhibits a superior generation ability over different tasks. Pre-trained models have been the dominant paradigm in natural language processing, and researchers resort to massive data and large-scale models to improve performance. We unify the rules used in various tasks into the form of predicate logic, provide the possibility to pretrain models on massive rules. In the future, we will explore pre-training large-scale neural rule-execution machine with massive rules and data.
Broader Impact
Our work proposes a unified and scalable approach to efficiently perform fine-grained controllable text generation and incorporate multiple prior knowledge for superior text generation performance. This work uses story generation, machine translation, commonsense generation as applications to verify the effectiveness. However, while our proposed method achieves promise performance on several benchmarks, deployment of our method in the real world requires a careful analysis of potential societal benefits and harms (e.g., the harms associated with furthering negative stereotypes against certain vulnerable groups). The potential ethical issues include: powerful language models might be used to generate abuse, faked or misleading content in the news or on social media; they might pose safety concerns if they are used to generate harassing or hateful materials. In order to mitigate these risks, it is possible to use AI systems to fight against misleading content and harassing material. However, as discussed in previous work [42, 43], mitigating these risks could be an extremely complex socio-technical problem that many are working to understand and solve.

heading: Acknowledgement
text: We thank anonymous reviewers for their insightful suggestions to improve this paper.
Funding Transparency Statement
Yufei Wang, Can Xu, Huang Hu, Chongyang Tao and Daxin Jiang are supported by Microsoft Software Technology Center at Asia (STCA). Yufei Wang also receives a MQ Research Excellence Scholarship and a CSIRO’s DATA61 Top-up Scholarship.

Review summaries:
The paper proposes a unified framework to incorporate prior knowledge into the text generation process via predicate logic constraints. The text generation model is a transformer and constraints are also encoded by a separate transformer. The encoded constraints then become part of the attention mechanism, therefore it can be considered as an approach that uses “soft” constraints.   Contributions - A general framework of incorporating prior knowledge as (soft) constraints into the decoding process for text generation. 

This paper proposes a general framework, NRETM, for the conditional transformer-based seq2seq model. The paper first uses a state flag to indicate the expression progress of each predicate in the constraints set as a concatenation of logic trackers. The paper uses a one-layer transformer encoder to encode the state flag.  The paper utilizes the state matrix as a relative position to integrate it into the model. The results in three experiments show that the NRETM improves the transformer performance. 

The authors propose an approach aimed at encouraging transformer-based sequence-to-sequence models to respect constraints on the text they generate. In particular, the authors propose a particular syntactic representation of whether propositional logic-style constraints have been satisfied in some (partially) generated text, which is tracked throughout decoding. This syntactic representation is a function both of the current decoding step as well as the index of the particular constraint, and is updated dynamically as decoding progresses. The authors tokenize and encode these representations, and use them inside a relative-position-style attention mechanism. The authors fine-tune their models using these dynamic representations and they show that this allows for better control over various aspects of the generated text on RocStories, Commongen, and Zh-En document level translation tasks, with comparable or better quality.