Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a general framework, NRETM, for the conditional transformer-based seq2seq model. The paper first uses a state flag to indicate the expression progress of each predicate in the constraints set as a concatenation of logic trackers. The paper uses a one-layer transformer encoder to encode the state flag.  The paper utilizes the state matrix as a relative position to integrate it into the model. The results in three experiments show that the NRETM improves the transformer performance. 
main_review: Strengths:
1. The paper focus on an interesting problem about constraint-based generation tasks. The paper solves this problem by proposing a new Neural Rule-execution tracking machine that tries to incorporate rules into current transformer-based seq2seq generation models. The paper utilizes the logic tracker as relative position information to guide the model in the generation The experiment results over three different tasks seem to be promising. 

2. The paper lists detailed experiment settings for three different tasks. The paper also discusses the ability of the model in a zero-shot setting. The paper includes code as a supplement and put more generation results as case studies for three different tasks. Those examples show that the model captures those constraints through the proposed framework,

Weaknesses:
1. Section 2.4 is a little bit confusing. Even though the paper provides Figure 2 as a running example of the NRETM model with three logic constraints, it still takes me some time to understand all the concepts introduced in this section. It would be better to reorganize the content. For example, a logic tracker should be introduced before the state flag which can help readers to better understand the concepts. 

2. In section 3, the paper only compares their model to a limited baseline. The paper needs to add some baseline mentioned in related work such as NEUROLOGIC DECODING (Liu et al., 2020). In section 3.2, the constraint satisfaction metric is not clearly defined. The evaluation metrics for all/novel/seen constraints are also absent. The evaluation metrics are also limited. For story generation, rouge L is not able to cover all the details. Similarly, CIDEr can also show some limited aspect of generation results. It would be better to include more automatic metrics to show a more comprehensive view of the generation quality. such as Bertscore(Zhang et al., 2019), BLEU, etc. 

3. The generalizability of the framework is unclear. It seems that the model can only be applied with several pre-defined logic operators. Those logic operators

Lu, X., West, P., Zellers, R., Bras, R. L., Bhagavatula, C., & Choi, Y. (2020). Neurologic decoding:(un) supervised neural text generation with predicate logic constraints. arXiv preprint arXiv:2010.12884.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
limitations_and_societal_impact: The paper describes the limitation and social impacts in the last section. Some limitations are addressed. The potential misuse of this framework is also stated.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose an approach aimed at encouraging transformer-based sequence-to-sequence models to respect constraints on the text they generate. In particular, the authors propose a particular syntactic representation of whether propositional logic-style constraints have been satisfied in some (partially) generated text, which is tracked throughout decoding. This syntactic representation is a function both of the current decoding step as well as the index of the particular constraint, and is updated dynamically as decoding progresses. The authors tokenize and encode these representations, and use them inside a relative-position-style attention mechanism. The authors fine-tune their models using these dynamic representations and they show that this allows for better control over various aspects of the generated text on RocStories, Commongen, and Zh-En document level translation tasks, with comparable or better quality.
main_review: This paper tackles an interesting problem, and it obtains good results. Another nice thing about the paper is the authors largely consider constraints that are *not* easy to implement with just a simple constrained beam search. In particular, rather than constraints which disallow certain tokens or alignments, the authors consider constraints that require the presence or a particular number of certain words, which are much more challenging to guarantee with standard decoding algorithms.

In terms of contribution, the core contribution of the paper appears to be the proposal of an approach to dynamically and explicitly updating the decoder in response to whether or how much the text generated so far conforms with some pre-specified constraints.  While this sort of dynamic updating seems quite reasonable, similar ideas have been proposed before (as the authors note), such as in the case of coverage attention (Tu et al., 2016). Accordingly, I think the proposed approach would feel more compelling if the authors could argue that it is superior to other approaches to updating the decoder in response to how well constraints have been satisfied in the generated text so far. However, I don't think there are any such comparisons. In particular, it seems the main baselines the authors consider are either a baseline seq2seq model with the constraints encoded statically on the source side (Table 1) or a baseline seq2seq model with a constrained decoding algorithm (Table 2). As such, it's hard to tell how crucial the various modeling/encoding choices the authors have made in implementing their approach are. (A natural baseline not considered, for instance, might involve dynamically updating the constraint tokens consumed by the encoder in the baseline in Table 1 instead of using the authors' proposed attention model).

In terms of presentation, I think the paper is largely clear, although I would encourage the authors to emphasize earlier on that their method involves fine-tuning the model rather than simply implementing constraints at decoding time; I think the fact that fine-tuning is involved only really becomes clear on the bottom of page 5. 

Minor:
Are the script 'U's on like 116 different from the non-script upper case 'U's on line 109?

Update after response from authors: thanks for your response; I'm increasing my score in view of the new baseline results.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Interesting problem, but many issues along the way
main_review: This work aims to find a way to impose the symbolic rules into powerful neural networks. To this end, it proposes a unified and scalable approach, named NRETM, to perform fine-grained controllable text generation and incorporate multiple prior knowledge. 

Strength:

1. This paper proposes a framework to unify the constrained generation and general text generation with prior knowledge incorporation.

2. Experimental results on story generation, machine translation, and commonsense generation partially demonstrate the effectiveness of the proposed method. 

Drawbacks:

1. The novelty of this work is not clear. It seems to extract the intermediate values of the rules to the model, which is helpful for the model to learn the relationship between the rules and the final outputs. If my understanding is correct, is it not surprising that NRETM can make better results since additional prior knowledge can usually benefit the generation process.

2. The experiments lack rigor. The author should add some competitive baselines for comparison. This paper mainly compares NRETM with pre-trained model T5. But T5 can not utilize prior knowledge efficiently. Feeding the predicate logic constraints as inputs directly is not a suitable way to impose prior knowledge.



The manuscript has some typos: 
1) eq. 1 P(y|x) is not defined.
2) line 103, in 1 -> in Figure 1.
3) line 116, consist -> consists.
limitations_and_societal_impact: The applications of the proposed methods are limited. The paper states that "because in the training data at the position of EOS token in the ground-truth output, all elements in the state matrix must be set to the satisfied, NRETM can learn the rules by itself." However, if the training data contains some noise, can the proposed methods work as well? More analyses about this should be conducted.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a unified framework to incorporate prior knowledge into the text generation process via predicate logic constraints. The text generation model is a transformer and constraints are also encoded by a separate transformer. The encoded constraints then become part of the attention mechanism, therefore it can be considered as an approach that uses “soft” constraints. 

Contributions
- A general framework of incorporating prior knowledge as (soft) constraints into the decoding process for text generation.

main_review: After Authors' Response
The authors' response was very thorough and addressed all my raised concerns, therefore I have increased my score.

Strength
-	The idea makes sense on a high level and having an efficient manner of incorporating constraints into the decoding process for text generation would be very beneficial for many tasks and applications.

Weaknesses
-	Some important points about the method and the experiments are left unclear (see also questions below).
-	The writing could be improved (see also Typos & Additional Questions below)
-	Multiple runs and significance tests are missing. This makes it hard to judge the improvements (Table 2 & 3).

Most Important Questions 
-	Line 156: What is q_ij^k here exactly? I thought q_ij was a state flag, such as “2” or “0”. But you tokenize it and encode it, so it sounds more like it is something like “Copy(snow)”? (If it is the latter, then what is the meaning of tokenizing and encoding something like “Len(9)”?)
-	192: What exactly is storyline and what do you need it for?
-	The baseline takes the predicate logic constraints as input: How does T6 know what to do with these inputs? Was the model trained on this but without the NRETM module? Can you give an example of what the input looks likes? How do these inputs  guide which sentences should be generated? Looking at the datsset, it feels like one would need at least the first 2 sentences or so to know how to continue. Maybe this information is now in your constraints but it would be important to understand what they look like and how they were created. Is there no other suitable baseline for this experiment?
-	What is the overhead of your method compared to standard decoding approaches? (you mention GBS can only be used with T5-Base, so your method is more efficient? That would be important to point out)
-	What happens if the decoding process cannot find a sequence that satisfies all constraint? 
-	Document-level MT: How do you know at test time whether the system translates a particular sentence or not?
-	How many sentences are misaligned by Doc-mBART25? What are the s-BLEU and d-BLEU values on the subset that NRETM aligns correctly and Doc does not?
-	Why was NEUROLOGIC not used as a comparison baseline?
-	What is dynamic vs static strategy? In which experiment did you show that dynamic works better than static (from conclusion)?


Typos & Additional Questions
-	Line 40: you could mention here that the examples will be translated into logic forms in the next section.
-	Paragraph starting at line 53: Why did you choose these datasets? How will they help evaluate the proposed approach?
-	Line 75: a and b should be bold faced?
-	83: “that used” -> “that are used”
-	83: “details” -> “for details”
-	Paragraph at line 86: At this point, the state matrix is unclear. What are the initial values? How can the state matrix be used to understand if a constraint is satisfied or not?
-	98: “take[s]” & “generate[s]”
-	108: “be all” -> “all be”
-	Paragraph at line 101: What is dynamic vs static strategy?
-	Paragraph at line 109: The state flag explanation would greatly benefit from an example. Does q_i refer to whether a particular U_i is satisfied?
-	Eq 2: What is the meaning of N? Can it change depending on the definition of U_k? Does it mean this constraint is not relevant for x_i?
-	133: Figure 1 should be Figure 2
-	Figure 2: What exactly do the “&” rows track?
-	Figure 2: Is the state flag matrix equal to the state matrix? If not, how do you go from one to the other?
-	Line 146: What does the inf in the superscript signify?
-	177: What is the symbolic operator?
-	Paragraph at line 194: Without understanding what a storyline is, it is not clear what the constraints are. An example might be helpful here.
-	Line 204: what is the ROUGH-L metric? Do you mean ROUGE-L?
-	Line 223: How do you obtain the morphological inflections for the concepts?
-	237: @necessity [of] integrating”
-	3.3: How exactly is the document-level MT done? Is the entire input document the input to T5?
-	293: “because” typo
-	3.4 where/how exactly is the sentence index used?

limitations_and_societal_impact: The paper's broader impact section discusses general potential benefits and issues of text generation (from large language models). It could maybe be tailored a bit better by discussing what effect this proposed work would have on the potential benefits and issues.
review_title: null

Meta-review: This paper proposes a technique for including global constraints during decoding, and letting these be softly violated during decoding. Although the technique seems to be reasonable (there were a few concerns about the terseness of the approach), and reasonably evaluated, the problem of constrained optimisation is one of (if not the most) widely studied problem in optimisation, and indeed, constrained optimisation has come up numerous times even in "decoding problems" (e.g., frameworks for decoding like dual decomposition, Rush and Collins, 2012, and lagrangian relaxation, Martins et al., 2012 et passim), and using first order predicate logic to express constraints has a long history (the constrained conditional models of Rizzolo and Roth, 2007, and Markov Logic Networks of Poon and Domingos). I would encourage the authors to explicate how this work relates to other work in constrained optimisation in the context of decoding algorithms, and in particular, to be more precise in the motivation and nature of the constraints used in the experiments (for instance, are they designed on a validation set and then tested on a held-out test set?).
Decision: Accept (Poster)