Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper describes a model for text generation, based on target dependency relations that should be in the output.
The word-level output probabilties are modified to increase the likelihood of generating words that match the target relation.
During beam decoding, the candidate construction method also takes the target relations into account.
Evaluation is performed on several datasets, formulating the task as text generation based on dependency relations.
strengths_and_weaknesses: The model is interesting.
The paper is well written and clear.
The specific method of modifying the output probabilities based on the given relations can be useful.
Results are demonstrating the benefit of the proposed model over the chosen baselines.

The work is presented as a completely novel task. However, it does not sufficiently address existing work on data-to-text generation. 
In data-to-text, the task is very similar - generation text based on relation tuples. 
While data-to-text systems do not typically restrict themselves to only dependency relations, they could definitely be applied on this task.
Therefore, work in this area should be covered under related research and relevant baselines should be used for comparison.
There is currently a very minimal section for evaluation on a data-to-text dataset, but only a very generic model (not designed for data-to-text) is used as a baseline.

There is a nice section on dependency-guided generation, which has the same goal as the proposed task. However, none of the listed previous work seem to be reported as a baseline.

The proposed evaluation metrics are interesting for diagnostics but overall not very convincing.
It is claimed that overlap-based metrics are not appropriate and parser-based metrics are used instead. 
This doesn't really reflect how readable the sentence is at all. 
For a high score, the model could just linearise the triplets that are provided as input, without producing a coherent sentence. 
The UC and LC metrics only measure recall, not precision, so there is no penalty for generating loads of different relations in the output.
Similarly, the Word% metric could just copy over the model input into the output in order to get a 100% score.
BLEU 1-4 should be reported, along with METEOR. One BLEU is currently reported but it is unclear which one it is. BLEU-1 would not be sufficient to measure sentence quality.
questions: It is currently unclear how many dependencies are given as input to the model. And if they are not all the dependencies, then how are they chosen.

Table 8 in the appendix seems to have some issues. The dependencies in the first row do not match the sentences.

limitations: There is a very brief discussion of computational complexity in the appendix. There could be a much more thorough discussion on the limitations of the system, the task, the evaluation metrics and text generation based on dependencies in general.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new paradigm in constrained text generation called relation-constrained decoding (RCD). RCD ensures that the tokens in the constraint set are adhered to (lexically constrained decoding) while decoding from NLG models without compromising on the underlying "relation" between the tokens. To address RCD, the paper presents a decoding time algorithm called RESEAL that can handle different types of relations (syntactic, factual, or entity based). On top of the standard evaluation, the paper also highlights results on 3 NLG tasks that help understand the RCD paradigm's applicability. 

Implementation of RESEAL involves two main strategies, probability surgery, and relation-guided lexical decoding. 
strengths_and_weaknesses: Strengths:
1. The paper is well written and easy to understand.
2. Comprehensive evaluation of a variety of NLG tasks.
3. Good results against competitive baselines.
4. Fact-based editing is an elegant application of this paradigm.

Weaknesses:
1. Dependence upon a suitable relation identifier is a prerequisite. In most cases, it won't be easy to obtain/train. Thereby limiting the applicability of RCD to very simplistic tasks. In most cases, a lexically constrained decoding paradigm is sufficient.
2. Evaluation on newer and robust NLG evaluators like BERTScore, and BLEURT is missing.
3. No human evaluation of results.

questions: 1. Line 102: Based on the formulation the candidates are just getting re-ranked. How is the signal guaranteeing the presence of relation constraint?
2. Line 130: Why is 2 needed in the denominator?
3. Table 2: Shouldn't w/o prob = DBA? Why is the result different from DBA (Table 1)?
4. Line 225: What is the word+rel strategy?

Typo: 
Line 111: Line 1-7 -> Line 1-6
limitations: The paper highlights some of the limitations in section 5.1(results) and societal impacts in the appendix.

Limitation: The model needs performance depends on access to a good relation identifier. The errors in the relation-identifier propagate into the model.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a way to incorporate "relation constraints" in decoding. For example, given the constraint triplet: (phone, amod, fancy), the decoding process will attempt to generate a sentence with that includes words "phone" and "fancy" while also satisfying an "amod" relation based on a relation predictor (e.g. "This is a fancy phone"). This is an extension of the lexically-constrained decoding problem, though it seems that the relation constraints here are only satisfied probabilistically. 
strengths_and_weaknesses: 
Strength
- This is an interesting problem. Techniques that give us more control of the decoding process is generally a nice contribution to the field. I am not aware of previous work that incorporates relations. 

Weakness: 
- While relation contraints are interesting, it is not immediately clear in what applications they are helpful. Various tasks and results are shown, but I don't see intuitively why the relation constraint is important. For example, given the constraint triplet: (phone, amod, fancy), would a simple lexically-constraint decoder that uses (phone, fancy) produce those words in the right order anyway (due to a strong language model)? In contrast, would your relation-constrained decoder produce these words in the wrong order but still labeled with the right head-child dependency relation (due to a poor relation predictor)? Similarly, for the summarization problem, why do relations help? Perhaps the Data-to-Text and Fact-based Text editing tasks are easier to motivate, but the paper doesn't dedicate enough prose explaining these tasks. 

- The proposed method is difficult to understand. I think I got it after several readings, but there may be ways to improve especially the understanding of the general purpose of line 4 and 5 in Algo 1. One suggestion is to elaborate on the left part of Figure 1 (which explains line 4 at least). 
questions: - In Table 1, it seems that some of the lexically-constrained decoding methods (e.g. DBA) are worse than Base. Is that expected? 
- Are there speed comparisons between proposed method and the simple Rerank baseline? 
- Can you provide some example sentences of your decoder vs DBA in the main part of the paper? This may help motivate the use of relations. 
- It seems to me that the relations in the decoder are always predicted. It is a hidden variable, so it is somewhat different in nature to the words in the lexical constraints, which are observed. If you have a poor relation predictor, can it fool your decoder into believing that the constraints are satifised? 
limitations: Appendix E writes: "... our proposed methods may be used to generate more human-like fake text. However, the impacts are more apparent when considering deployed applications, while our proposed methods as the methodologies can not have any direct negative societal impacts." 

I am actually not sure what you are saying here. Do you mean that as a general technique, this does not have direct negative social impact, because the fault is with the application that uses this technique? I guess I see your point partly but I think the paragraph can be modified to say things in a better way. If your method is so good it makes it more difficult to detect fake text, isn't that a direct negative impact compared to other methods?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper is under the theme of constrained decoding, which incorporates predefined rules into a generated sentence. Compared with previous approaches mainly applied to words or phrases, the authors focus on the relations among them, which are called as Relation-Constrained Decoding (RCD). In terms of this setting, they introduce a model, RESEAL, that contains two special modules: RElation-guided probability Surgery and bEam ALlocation, which respectively modify the emission probabilities from the decoder and utilize the constraint information to guide the candidate selection of beam search.
strengths_and_weaknesses: Pros: 
1) well-defined problem that is worth investigation;
2) experiments on many tasks, which seems to be solid.

Cons:
1) the writing of Sec. 3 needs to be polished. Also please don't use python codes in your Algorithm 2;
2) the model seems to be incremental to DBA and is not flexible since it resorts to external knowledge (e.g., a well-trained dependency parser); I also think the authors should even clarify the external dependence issue in the introduction, which is the main problem we are facing in the NLP industry.

questions: 1, May you put some case studies to show how your model works in practice?
2, Eq. (3) only considers past information; What about the future? Does this scoring style have a bias in this sense?
3, It seems that applying p_{rel) in every decoding step is not reasonable. Could you explain more about this, especially the intuition of g?
limitations: I've mentioned previously in 'Strengths And Weaknesses': 1) No big difference between the proposed model and DBA except that the former applies external knowledge to change the emission probabilities of the decoder, which is a bit trivial; 2) the reliance of the introduced model on external knowledge.

Will improve my score if the authors can address my concerns.
review_title: null

Meta-review: metareview: The paper describes a model for text generation, based on target dependency relations that should be in the output. The word-level output probabilities are modified to increase the likelihood of generating words that match the target relation. Evaluation is performed on several datasets, formulating the task as text generation based on dependency relations.
The empirical gains are OK but not particularly large. What I find more compelling is the ability to control the output of the model, which is currently lacking in most approaches.
The reviewer scores straddle the decision boundary and it was unfortunately not possible to get the reviewers to engage in a discussion but the authors did a good job addressing all initial comments/questions.
confidence: Less certain
Decision: Accept