Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper is a novel and profound contribution to sequence-to-sequence graph parsing of natural language, quantifying and evaluating compositional uncertainty in seq2seq graph parsing through a newly-proposed probabilistically interpretable framework and a new metric which measures a model's calibration in predicting graph structures. A large battery of tests together with human evaluations shows strong empirical support for the framework.
strength_and_weaknesses: Strengths:
- A novel and creative contribution to modelling and evaluation for uncertainty in seq2seq parsing.
- Thorough experiments, high-quality writeup, convincing results.

Weaknesses:
- The selection of datasets is not thoroughly elaborated from the viewpoint of availability vs. linguistic breadth. English is dominant in graph parsing, but care should be taken to qualify how the approach would scale up (with the number of languages covered) and down (with diminished resource availability in low-resource languages). A Limitations section addressing this linguistic breadth angle, and an honest account of this contribution being by and large for English only, would suffice.
clarity,_quality,_novelty_and_reproducibility: The work is of exceptional quality and clarity, and it is both original and important to the (sub-)field.
summary_of_the_review: Contained above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors do seq2seq parsing with various semantic parsing datasets, and employ a specialized method called GAP that is meant to incorporate "compositional uncertainty" (UC). My interpretation is that UC incorporates probabilities over graph construction (i.e. edge and node creation) rather than token predictions when training seq2seq models with max-likelihood objectives (although there is a fine line between these two, and perhaps the former is achieved simply by using the right action space). Also, their training involves approximate normalization over subgraphs via importance sampling. Because of uncertainty measures that GAP implements, it is suitable for human-in-the-loop graph editing (sec 3.2). This last idea is quite interesting --- maybe your initial graph is fine and a human can slightly fix it up to drastically improve performance. That being said, the eval is somewhat limited and the main text is lacking details.
strength_and_weaknesses: # Strengths

1. The paper is focused on seq2seq parsing with T5. This is a challenging task and active research area. They evaluate on many semantic parsing tasks.

2. The authors explicitly model the graph rather than solely treat it as a string-to-string task.

3. The authors evaluate in a creative scenario where humans can edit noisy graphs to drastically improve performance, although this section is limited.

# Weaknesses

1. There are many missing related work, and it is not right to say they are first to explore local probabilities for seq2seq parsing.

2. The evaluation is very hard to interpret. In the main table 1, there are no baselines and only T5 w/ GAP is evaluated as far as I can tell. In figure 3 there are many methods but I am not sure how it ties into the main storyline regarding T5 w/ GAP. In addition the evaluation is not very standard, which alone is not a major problem, but warrants further explanation in the text.

3. To me the most exciting part of the work is that noisy graphs can potentially be edited to improve performance. This section leads to some concerns... I am surprised RANDOM can be so effective. There are not really any details on how human edits are done. There is not much analysis on the subgraph editing, and it seems like most of editing can be done by editing one or two labels.
clarity,_quality,_novelty_and_reproducibility: 1. Probably should mention Vinyals et al, 2014 "Grammar as a Foreign Language", one of the earliest successful attempts to do seq2seq parsing.

2. Also, for parsing specifically there are probabilistic models that represent uncertainty well, i.e. chart parsers.

3. Since you are using PENMAN, you may be interested in the SPRING model from Bevilacqua et al. 2021 and the stack-based transformer work from Ramon Astudillo and others (starting in 2020 with multiple followups). From those works, the paper from Drozdov et al. 2022 is particularly relevant as it explicitly models uncertainty with respect to alignments between graph nodes and input text tokens using importance sampling. The earlier RNNG work from Dyer et al 2016 also uses importance sampling in seq2seq setting.

4. The compositional uncertainty is measured locally. Although convenient, it's well know this can be lead to propagating errors (i.e. label bias problem), which was motivation to introduce CRFs to combat this problem in MEMMs. This is not a major issue per say, since contextual encoders blur lines between local and global decisions, but should be aware of this when using graphical models. Some related discussion in Andor et al 2016 "Globally Normalized Transition-based Neural Nets" and also Buy and Blunsom 2018 "Neural Syntactic Generative Models with Exact Marginalization".

5. About this claim "We are the first work to study compositional uncertainty for seq2seq graph parsing.", certainly others have addressed this issue of P(v | pa(v), x) before even if they have used different terminology, so this is not the first.

6. Does "long-range" in "long-range parent-child conditional probability" refer to distance in graph or distance in sequence? This usage was a bit confusing to me. My understanding is that neighboring words can be arbitrarily far away in the graph and vice versa. Later in the text suggests this means long-range in both spaces.

7. In sec 2.1.1 I think "global likelihood" is really "cumulative likelihood". One other thing I am confused about here is what is being integrated over... is it graphs of a certain size? Is there an ordering over graph creation actions that is important to know?

8. Regarding "sequence accuracy (ACCseq) does not necessarily correlate to the SMATCH score,": is sequence accuracy the same as "exact match"? Doesn't SMATCH assign partial credit? So clearly SMATCH would be different.

9. In general, I think it is neat to use PENMAN here, but I am also a little confused why it would be useful. Is it because of top-down representation? Does the data employ non-tree graphs? If so, then wouldn't variables be necessary (it says you use the non-variable PENMAN).

10. The idea about editing subgraphs based on uncertainty is neat, but I have some questions that I was confused about. Can any arbitrary subgraph be edited, or only certain types (e.g. must include all nodes between root and leaf)? Do authors believe subgraph editing is sufficient --- when subgraph is wrong, perhaps the entire graph has mistakes? Does low model probability really equate to uncertainty --- wouldn't ambiguity (near random) more align with concept of uncertainty? It seems intuitive that locally low probability could be an unreliable indicator in out-of-domain --- predicting a node for a rare or unusual token for example could give low local probability, so maybe there is other low hanging fruit for identifying subgraphs to edit.

11. Assuming the GAP formulation is desirable, I'm still confused on the details of the parameterization. Can it really be derived solely from beam-search probabilities? If so, is this achieved simply by choosing the correct action space? I would think this is not enough, since the same graph can be potentially created from different actions and I am not sure it would have the same probability.

12. The authors may be interesting in existing work on retrieve-and-edit (Hashimoto et al.) or parse editing in general, which is loosely relevant.

typo: "Here to The model"

UPDATE: Numbered for convenience.

summary_of_the_review: The core idea of the paper, that uncertainty can be used to inform graph editing, seems very interesting but I am not sure it is well executed. Parts of the presentation are confusing, and there could be more analysis on the graph editing portion. By fixing up writing, adding appropriate baselines, and providing more emphasis on graph editing part, then I think this could be a stronger paper. Right now I do not think it meets standards of the conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper propose a method to calculate the compositional uncertainty of sequence to sequence model for semantic parsing tasks, that is in tasks where the input is an utterance and the output is a graph.
For doing so the authors propose to marginalize the probability of output sequences so to calculate the expected probability of a node, give its parent node in a generated graph. Since marginalizing over all possible subsequences between two symbols is intractable in general, the authors propose to approximate this process using beam decoding.
The authors use the compositional uncertainty to calculate the expected calibration error, that is how well the uncertainty matches the performances of the model. The authors compare standard ECE and compositional ECE, showing that the second one is a better choice.

Finally the model is used in a collaborative semantic parsing where parts of the predictions (for example where the models is more uncertain on) can be send to human validation. Results show that the proposed measure of uncertainty finds more errors than random, and it is close to an oracle selection.
strength_and_weaknesses: The paper is well written and seems to be a missing piece in the semantic parsing literature. Calibration error is a standard metric to understand how trustworthy the predictions of a model are, and apparently this is the first attempt at calculating it in the setting of compositional sequence to sequence models.

The paper is pretty clear and goes step by step explaining the various concepts used in the paper. it is equation-heavy but easy to understand.

The premises for section 3.2 are a bit vague, to me modelling compositional uncertainty would be very helpful in an active learning framework, where subgraph on which the model is uncertain are asked to be reannotated. The collaborative semantic parsing scenario is not very realistic to me.
It is not clear to me also how the oracle is defined.
clarity,_quality,_novelty_and_reproducibility: First sentence of the intro, typo, graph paring -> graph parsing
Table 1 and fig 3 are extremely small

I find the paper technically sound and clear.
summary_of_the_review: This work introduce a technique that allows practitioner to calculate compositional uncertainty in seq 2 seq model for graph parsing.
I am not an expert in graph parsing but it seems that this paper fills a void in that space.
I am not able to judge if more experiments on different datasets are required to prove if the approach is general enough to be used in any kind of text 2 graph task.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies uncertainty quantification for sequence-to-sequence semantic graph parsers. The paper proposes a framework for mapping the probability of elements in the graph linearization to graph elements, together with a beam search approach to approximate marginalizing over subsequences. Compositional uncertainty is then quantified by a proposed compositional expected calibration error (CECE), which is compared to expected calibration error (ECE). Evaluation on 3 semantic graph parsing tasks and 10 domains shows that CECE correlates better with out of domain generalization than ECE under various uncertainty quantification methods. The results show structural differences between uncertainty in graph parsing compared to classification. The method is then applied as a proof of concept for collaborative semantic parsing, where performance can be improved by replacing a small number of uncertain subgraphs through human intervention (simulated here through an oracle). The proposed approach is a valuable contribution to the semantic parsing literature in terms of measuring a model’s calibration in predicting graph structures. Evaluating on more kinds of graph structures would be useful, and the experiments on graph editing could be extended, but the experiments are strong enough to support to claims of the paper and the overall contribution is strong. 
justification_for_why_not_higher_score: The paper makes a strong contribution to the problem of quantifying uncertainty in sequential models of structured prediction. While this could have a wider impact than just semantic graph parsing, the paper's audience will likely be more limited and therefore is most suited as a poster presentation.
justification_for_why_not_lower_score: The paper makes novel contributions to the problem of quantifying uncertainty in sequential models for graph parsing, which is an important problem due to the challenges in approaching structured prediction as a sequential prediction problem.
Decision: Accept: poster