Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work focuses on the update regression problem for structured prediction. The authors explore two standard structured prediction approaches (graph-based and transition-based). They conduct a set of experiments and introduce a noveal approach to mitigate the update regression problem.

**EDIT after discussion with the authors:**
I still disagree with authors, but they made a good point in our discussion, so I upgraded my grades. I strongly recommend the authors to soften some claim made in the paper, refer that in some cases KL div technique for KD can be used and give proper citations about how one could do that.
strengths_and_weaknesses: **Strenghts:**
- this work focuses on an important and under-studied problem
- experimental results are interesting

**Weaknesses**

Unfortunately, this paper has many weaknesses. First, and importantly, the main contribution of this paper (section 5.3) is not well motivated and the explanation of the contribution itself is really handwavy. Second, explanation of the different baseline approaches is also too handwavy, making the overall paper difficult to understand and not self-contained.

On the motivations: the authors argue that previous approaches cannot be applied to structured prediction. I strongly disagree with this.

- l 227, "the output distributiion of global prediction is often intractable": in the non-projective dependency parsing case, the distribution can be computed using the Matrix Tree Theorem [1, 2, 3]. If we restrict the problem to projective trees, this can be computed via dynamic programming [4, 5]. For the transition based model, approximated approaches have been explored in the litterature [6]
- l. 222, "the instantiations of KD are usually based on a loss function computing the cross-entropy of output distributions [...] however existing methods are no directly applicable to structure prediction tasks": I don't understand why. First, the next sentence is false (see previous point), but also KL divergence between structured distributions has been studied in the litterature. For non-projective trees, see [7, Section 6.3], for methods based on dynamic programming see [8]
- l. 263, "furthermore, it is unclear how to adapt existing sampling methods to solutions that are not formalized as sequence generation": Recent work has considered sampling from the non-projective tree distributions, including sampling without replacement [9, 10]. Moreover, previous work has also considered perturb-and-MAP approaches [11, 12, 13]. Finally, in the case of dynamic programming algorithms, it is well known that it is possible to sample from the associated exponential familly distributions, see e.g. [14]

**Related work**

As suggested by the comment above, the literature is not properly explored or cited by the authors.
There are similar problems in the introduction and related work section. For example:
- l. 54: authors cite Dozan and Manning (2017) for graph-based parsers, whereas the correct citations is more than 10 years older [15]
- l. 21: for transition-based parsers, they cite Ma et al. (2018), better citations would be [16, 17]

[1] Structured Prediction Models via the Matrix-Tree Theorem (Koo et al.)

[2] Probabilistic Models of Nonprojective Dependency Trees (Smith and Smith)

[3] On the complexity of non-projective data-driven dependency parsing (McDonald and Satta)

[4] Semiring parsing (Goodman)

[5] Differentiable Dynamic Programming for Structured Prediction and Attention (Mensch and Blondel)

[6] Globally Normalized Transition-Based Neural Networks (Andor et al.)

[7] Efficient Computation of Expectations under Spanning Tree Distributions (Zmigrod et al.)

[8] First- and Second-Order Expectation Semirings with Applications to Minimum-Risk Training on Translation Forests (Li and Eisner)

[9] Efficient Sampling of Dependency Structures (Zmigrod et al.)

[10] Unbiased and Efficient Sampling of Dependency Trees (Stanojevi)

[11] Perturb-and-MAP random fields: Using discrete optimization to learn and sample from energy models (Papandreou and Yuille)

[12] Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder (Corro and Titov)

[13] Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming (Corro and Titov)

[14] See section 17.4.5 of  Machine Learning: A Probabilistic Perspective  (Muprhy) for the idea and Latent Template Induction with Gumbel-CRFs (Fu et al.) for application to CRF-like distribution

[15] Non-projective Dependency Parsing using Spanning Tree Algorithms (McDonald et al.)

[16] A Classifier-Based Parser with Linear Run-Time Complexity (Sagae, Lavie)

[17] Algorithms for Deterministic Incremental Dependency Parsing (Nivre)
questions: - is there any formal motivation for the contribution in section 5.3?
- is there any reason why the previous work listed in my comment above has not been considered to adapt preivous work in the non-structured case to the structured case?
limitations: nothing to report
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper examines model update regression in NLP structured prediction tasks. Model update regression is an issue appearing when a model is updated and the new model classifies part of the test examples negatively, while the old model classifies them positively.
The novelty of the paper is that it explores the problem in structured prediction, while it has been previously explored for classification.
The work proposes a new method for correcting the new model from the old one - backward-congruent re-ranking and experimentally shows that it performs better than other previously used methods, such as model ensemble and knowledge distillation. 
The work also defines a new measure related to the model update regression problem in structured prediction - negative flip impact.
strengths_and_weaknesses: Strengths:
- Studying the model update regression in relation to structured prediction is interesting and important. The contributions are novel enough.
- The paper is very well-written and clear. The problem, previous work, and the proposed solution are clearly described.
- The proposed improvements are meaningful and the experiments show that the proposed method for correction - backward-congruent re-ranking works better than the alternatives on the explored tasks.

Weaknesses:
I did not find any major issues with this work. I am suggesting some things I found that could improve it further.
- Line 120: maybe add a citation for UD 2.2. On the next line only the POS-tagging dataset is cited.
- Line 182: an example could be beneficial for understanding how the global prediction is thus decomposed into a sequence of next-word predictions.
- Line 217: from the description it is not really clear how checkpoint averaging and prediction ensembling differ.
- Table 5: It is not clear what is the difference between rows 5 and 8 (for dropout-p=0.3)

questions: n/a
limitations: n/a
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper studied a problem called model update regression in the field of structured prediction for NLP, which means a new model that achieves higher scores may perform worse than its baselines on some cases. The authors proposed an approach, BCR, to handle this issue. The basic idea is to use an old model to filter predicted candidates from the new model. A trick called dropout-p sampling is also applied to improve the diversity and quality of produced candidates. The experiments had been done on Dependency Parsing and Conversational Semantic Parsing, which were aimed to show the effectiveness of BCR.
strengths_and_weaknesses: Pros:
1) the writing is good and clear;
2) the problem is well defined and the solution is straightforward.

Cons:
1) syntactic parsing is indeed very classical for computational linguistics but only stands for a tiny part of tasks in the field of structured prediction. Some other tasks like NER and POS Tagging are far more common and have much more applications in the NLP industry. Plus, a lot of space in the paper is used for introducing syntactic parsing, which causes a bar even for researchers from NLP and is not necessary for its future applications to common scenarios. Therefore, The authors should investigate more representative tasks in structured prediction such that the formulation is more clear and the work has a broad impact on the whole community.
2) knowledge distillation for structured prediction is a mature field in NLP. Please at least see this work, Structure-Level Knowledge Distillation For Multilingual Sequence Labeling, and check the recent literature. Hence, the authors shouldn't say like 'existing knowledge distillation-based solutions cannot be directly applied to structured prediction models' in the paper, and more solid experiments are expected for comparisons.
3) the problem mentioned in the paper is formulated in a quite intuitive manner, which might lead to many explanations that can't be proved false. For example, from my view, the problem may be a cause of randomness in the premise that neural networks can't fit the whole dataset, which may explain why the ensemble model works. Therefore, I believe a theoretical foundation is expected.
questions: 1) Can you put some case studies in the Experiment section to show where your gains are from?
2) Since more than one model is used in your approach, does this strategy cause some of your improvements are in fact overlapping with the ensemble approach?
limitations: Besides my concerns raised in 'Strengths And Weaknesses', I think the authors should do more experiments to empirically confirm the problem and describe it in a more acceptable way. I have seen that the authors took a lot of space to show the performance numbers of their models, which I believe should not be the focus of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of model update regression: when a model with better aggregate performance is deployed, it may make mistakes that a previous model didn't make.  How can we minimize these "regressions"?  Specifically, this paper is the first to look at this problem in a structured prediction setting. On both dependency parsing and on a conversational dialogue / semantic parsing task, the paper evaluates new models to see how many examples they regress on. The paper describes several techniques to mitigate regression, including a new one, backward-congruent reranking, that reranks the new model's output using the old model. This is combined with a form of dropout at test time to yield diverse samples. The paper shows that BCR achieves good task accuracy while minimizing the number of model regressions.
strengths_and_weaknesses: STRENGTHS

- The problem setting in this paper is novel. While a somewhat niche problem, I am convinced of the core argument that this could be useful (with some reservations; see below under "Weaknesses").

- The exploration of "new" models is quite thorough, and it's nice to see the contrasts between some different types of model updates, including changing the mode paradigm.

- The BCR technique is simple and I could see it being widely adopted for this task

- The dropout-p sampling technique is interesting and a novel way of getting diverse samples from these kinds of generation models.

WEAKNESSES

The main conceptual weakness I see in this paper is the notion of a regression as it ties to model confidence.

It seems to me that if you take a model achieving 90% accuracy and perturb its weights slightly, it may still get 90% accuracy but make different predictions.  Many of the flipped predictions will be those where the 2nd-best option had nearly the same probability as the 1-best option; that is, cases where the confidence was low / entropy of the label distribution was high. It's not clear to me in these cases that the first model is getting things "right" and the second is getting them "wrong", more that the model had high uncertainty and it broke one way (correct) for some examples and another way (incorrect) for other examples.

Considering that BCR's performance is always lower than that of the ensemble, it seems to me that what's happening here is that this is really an ensemble that weights more heavily towards the earlier model. We aren't ensembling the two models symmetrically, but by reranking outputs from the second using the first, we're really only making changes where the second model is significantly more confident and doesn't even return what the first model initially preferred, resulting in a smaller performance gain but also fewer changes. So basically this is a clever way of combining the models but mostly has the effect of preferring the first.

I also wonder about the NFR/NFI metrics as they relate to the motivation of this paper. The idea of a system changing and then breaking UX or downstream modules that were depending on certain things working is a problem that resonates with me. However, do NFI and NFR really correlate with this motivation?  Again, many of the differences seem like they could most heavily depend on examples that were basically half-working to begin with, which were probably not consistent features a user was very attached to.

As a result, while this paper explores a different point in the design space of how to ensemble these two models, I'm not sure what it presents is really groundbreaking in terms of results.
questions: How is RoBERTa used for the seq2seq semantic parser? Is this the encoder and a separate transformer decoder model is learned? Please clarify since this is not a standard use of the encoder-only RoBERTa model.
limitations: There is little discussion of limitations and societal impact, but I don't see this as a big problem for this paper.
review_title: null

Meta-review: metareview: This is a good paper with a topic that is very important in practical scenarios but does not have many off the shelf solutions, and I find that this paper makes an attempt to this end.  In addition, I am happy to see the discussion with the reviewers, most of whom suggest acceptance.
confidence: Certain
Decision: Accept