Measuring and Reducing Model Update Regression in Structured Prediction for NLP
Authors: Deng Cai
Elman Mansimov
Yi-An Lai
Yixuan Su
Lei Shu
Yi Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Model Update Regression
Structured Prediction
Backward Compatibility

Abstract: Recent advance in deep learning has led to rapid adoption of machine learning based NLP models in a wide range of applications. Despite the continuous gain in accuracy, backward compatibility is also an important aspect for industrial applications, yet it received little research attention. Backward compatibility requires that the new model does not regress on cases that were correctly handled by its predecessor. This work studies model update regression in structured prediction tasks. We choose syntactic dependency parsing and conversational semantic parsing as representative examples of structured prediction tasks in NLP. First, we measure and analyze model update regression in different model update settings. Next, we explore and benchmark existing techniques for reducing model update regression including model ensemble and knowledge distillation. We further propose a simple and effective method, Backward-Congruent Re-ranking (BCR), by taking into account the characteristics of structured output. Experiments show that BCR can better mitigate model update regression than model ensemble and knowledge distillation approaches.

Measuring and Reducing Model Update Regression in Structured Prediction for NLP
Deng Cai* The Chinese University of Hong Kong chisisjcykcd@@mail com
Elman Mansimov Amazon AWS Al Labs
Yi-An Lai Amazon AWS AI Labs yianl@amazon, com
mansimov@amazon. com
Yixuan Su"
Lei Shu
Yi Zhang
University of Cambridge ys4840cam.c ac. ac.uk
Amazon AWS Labs leishu@amazon. com
Amazon AWS Labs
yizhngn@amazon.com com
Abstract
Recent advance in deep learning has led to the rapid adoption of machine learning- based NLP models in a wide range of applications. Despite the continuous gain in accuracy, backward compatibility is also important aspect for industrial applications, yet it received little research attention. Backward compatibility requires that the new model does regress cases that correctly handled by its predecessor. This work studies model update regression in structured prediction tasks. We choose syntactic dependency parsing and conversational semantic parsing as representative examples of structured prediction tasks in NLP. First, we measure and analyze model update regression different model update settings. Next, we explore and benchmark existing techniques for reducing model update regression including model ensemble and knowledge distillation. We further propose a simple and effective method, Beekward-Congnene account the characteristics of structured prediction. Experiments show that BCR can better mitigate model update regression than model ensemble and knowledge distillation approaches.
Re-ranking (BCR), by taking into
Introduction
Model update regression refers the deterioration of performance in update, even if the new model has on average better performance than the old model. For example, on model may achieve 1% absolute improvement in accuracy. However, it does
cases after model
fixed test set, the not necessarily mean that the new model gets 1 %% extra test cases correct. Instead, it may correct 6% mistakes made by its predecessor but also introduce 5% new errors. The occasional worse behavior of the new model can overshadow the benefits of overall performance gains and hinder its adoption. Imagine newly updated virtual assistant stops understanding user's favorite ways inquire about the local traffic. This can severely degrade the user experience even if the assistant has been improved in other aspects.
Prior research (Shen et al. 2020 Yan et al. 2021 Xie et al. 2021 has investigated model update regression in classification problems of computer vision and natural language processing (NLP). However, the formalization of classification only has limited coverage of real-world NLP applications. For instance, the natural language understanding component behind modern virtual assistants aims to transform user utterances into semantic graphs, a task known as conversational semantic paring
Most of the work was done during internship at Amazon AWS AT Labs.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(Figure D. In contrast to classification tasks, the global prediction of structured prediction (e.g., graph or tree) often composed of many local predictions (e.g., nodes and edges). Some local predictions can be correct while the global prediction is imperfect. Therefore, model update regression can happen at fine-grate tne-grained levels. Also, the output space is input-dependent and can be extremely large. To our knowledge, the model update regression problem has not been studied in structured prediction tasks.
In this work, we measure and analyze model update regression in two typical NLP structured prediction tasks: one ceneral-purpose task (i.e., syntactic de- pendency parsing) and one application-oriented task (i.e.. conversational semantic parsing). We set up evaluation protocols and test range of model up- dates rooted in different aspects, including changes in model design, training data, and optimization pro- cess. Experiments show that model update regression is prevalent and evident across various model update settings.
Input:
Directions David's party.
Old Model
New Model
Output:
model update
X EVENRT
DIRECTONN
Aivoniig in aUDGSTTATTION
Arvesting Iy aLDESTTAATION
DRFIND EVENT
X SATTED TIME
X
SLLOCIATON
BLCATESCOTE
SL BLOGGAMEEE
CATESORY
Dandd
part/
Davii
partyy
Figure 1: Model update regression: When up- dating an old model to a new one, old errors are corrected (X-> v) but new errors are in- troduced X).x).
The above finding shows general and critical de- mand for techniques to reduce model update regres- sion in structured prediction. Prior work for classifi- cation problems (Shen et al. 2020 Yan et al. 2021
Xie et al. 202T) has borrowed the idea of knowledge distillation (Hinton et al. 2015). where the new model (student) is trained to fit the output distribution of the old model (teacher). However, vanilla knowledge distillation (Hinton et al. 2013) cannot be directly applied to structured prediction models due to the intractability of the exact distribution of global prediction. Moreover, the goal prediction of structured prediction can be factorized in distinct ways. For example, a graph-based dependency parser (e.g.. Dozat and Manning 2017) scores every possible edge and searches for maximum spanning tree, while a transition based dependency parser (e.g., Ma et al., 2018) builds the tree incrementally through series of actions. We refer to a model update between models with different factorizations as a heterogeneous model update. For heterogencous model updates, existing fictorzatiion-speeinf approximation (e.g.. Zmiigrd et al. 2021a Wang et al. 2021 even knowledge distillation at the level of local predictions is inapplicable. To develop generic solutions that do not assume any specific factorizations, we instead resort to the sequence-level knowledge distillation (Kim and Rush 2016), which is originally proposed for machine translation. Previous work also finds model ensemble, though not as practical due to high computational cost, strong baseline for reducing model update regression. We also include it for comparison purpose.
We further propose novel and generic method named Backward- Congruent Re-ranking (BCR). BCR takes into account the variety of structured prediction output. That is, a new model can produce set of different predictions achieving similar accuracies. The key is to pick the one with the highest backward compatibility. In a nutshell, BCR uses the old model as a re-ranker to select a top structure from set of candidates predicted by the new model. We further propose dropout-p sampling, simple and generic sampling method, to improve the diversity and quality of candidates. Despite the simplicity of BCR, it is generic and surprisingly effective approach for mitigating model update regression, substantially outperforming knowledge distillation and ensemble methods across all studied model update settings. More surprisingly, we find that BCR can also improve the accuracy of the new model.
2 Related Work
Forgetting in Machine Learning Model update regression is related to a spectrum of research related to forgetting in machine learning, including continual learning (Chen and Liu 2018 Par Parisi
et al. 2019 Biesialska et al. 2020), incremental learning (Polikar et al. 2001 Gepperth and Hammer 2016 Masana et al. 2020). which are concerned with the learning of new tasks or streaming examples and the forgetting of previously learned ones. The model update regression problem differs in that models tested on the same task and have full access previous data.

---

Prior Research in Model Update Regression The model update regression problem was first studied (Shen et al. 2020) on learning buckward-compatible visual embeddings for image retrieval. YYan et al. 2021) suggested mitigating regression in image classification with positive-congruent training, a specialized variant of knowledge distillation. (Xie et al, 202T) extended the study of backward compatibility to NLP classification and explored several knowledge distillation variants for regression reduction. Träuble et al. 202T) introduced probabilistic approach to switch between the legacy and updated systems, bringing down regression in image classification. Our work investigates the model update regression NLP structured prediction tasks, which can be particularly challenging due to the enormous output space and the variety of different factorizations.
Re-ranking in Structure Prediction The re-ranking technique has a long history in NLP for improving not only accuracy (Shen al. 2004 Collins and Koo 2005 Salazar et al. 2019 Yee et al. 2019), but also output diversity (Li et al. (016). fluency (Kriz et al. 2019; Yin and Neubig 2019).
fairness Geeyik et al. 2019), and factual consistency (Falke et al., 2019p. Different from prior work, we show that re-ranking can be adopted for improving backward compatibility in model updates.
3 Preliminaries
We first lay out the basic concepts for the quantitative measurement of model update regression in structured prediction. Then we introduce two representative structured prediction tasks in NLP: dependency parsing and conversational semantic parsing, and set up the evaluation benchmarks.
3.1 Prediction Flips in Structured Prediction
Model update regression happens after a model update, where an old model is replaced by new model. For typical NLP structured prediction tasks (Smith 20IT), the input is a piece of natural language text and the output is a graph G (V, E)) with multiple nodes V and edges E. Unlike classification problems where the output is label in a pre-defined label set, the output graph G is decomposable and we may be interested in the correctness of some meaningful sub-graph G' (V' C V. C E) even though the graph not completely correct. We call the graphs (or sub- -rgggsrpphh) that are correctly predicted by both the new and the old models as positive-congruen predictions. On the other hand, negative flips are the cases where the graphs (or sub- mub-graphs) are correctly predicted by the old model but incorrectly predicted by the new one. There are also positive flip,, where the new model corrects the mistakes that the old one makes, which are desirable accuracy improvements.
Negative Flip Rate (NFR) To measure the severity of model update regression, we count negative flips and compute their fraction of the total number of graphs (or sub-graphs). We term the fraction as negative flip rate (NFR) (Yan et al. 202T).
Negative Flip Impact (NFI) Note that NFR capped by the overall error rate (ER) of the new model, comparison is challenging across cases with different ERs. For this reason, we introduce the negative flip impact (NFT): NFI = NFR neR Intuitively, the NFI computes the proportion of errors caused by negative flips. In other words, the proportion of errors that could have been avoided without the model update.
3.2 Dependency Parsing
The task of dependency parsing (Kübler et al. (009) seeks to find the syntactic head word for each word in a sentence and their syntactic relation. The overall output has tree structure where nodes
are words in the sentence and edges are their relations. The importance of dependency parsing is
widely recognized the NLP community, with it benefiting a wide range of NLP applications (Qi et al. 2020
We use the English EWT treebank from the Universal Dependency (UD2.2) Treebanks We adopt
the standard training/dev/test splits and use the universal POS tags (Petrov et al. 2012) provided in the
treebank. The parsing performance measured by word-level accuracies: unlabeled/labeled attach- ment score (UAS/LAS), and tree- accuracies: unlabeled/labeled complete match (UCM/LCM).
http:://usieerraaddppdeeeceeeeee
3

---

We compute NFR and NFI under each accuracy metric. We find that the results of labeled metrics are always consistent with unlabeled metrics. For clarity, we only show the results of unlabeled metrics, the results of labeled metrics are presented in Appendix A Following conventions (Dozat al. 2018), we report results excluding punctuations.
and Manning 2017
3.3 Conversational Semantic Parsing
Conversational semantic parsing is an important component for intelligent dialog systems such as Amazon Alexa, Apple Siri, and Google Assistant, which transform user utterances into semantic frames comprised of intents and slots.
We the TOP dataset (Gupta al. 2018) for our experiments, which uses hierarchical intent-slot annotations for utterances in navigation and event domains F The output is a constituency- parse tree, where leaf nodes words and the interior nodes are labeled with slots and intents. We use the dataset version with the noisy UNSUUPPPPR * intents excluded (Einolghozati et al.) 2019). The parsing performance is conventionally measured by tree-level Exact Match accuracy (EM). We also report span-level EM accuracy (span EM): We represent the semantic annotations on sentence as of labeled spans (i, L) where i, mark the start and end position of span and L is the corresponding label For example, the gold tree in Figure can be viewed as four labeled spans including (0, 4. IN: GET DIRECTION), (2, 4, SL: DELDDIIINATION IN: FIND LIND_EVENT), (2, 2, SL: ORGANIZER), and (4, 4, SL: CATEGORY. We calculate the label accuracy on such spans. For example, the new model's ' output correctly predicted three of them, thus the span-level EM accuracy is 3/4 75%. We calculate NFR and NFI under both metrics.
4 Measuring Model Update Regression
We carry out a series of experiments to examine the severity of model update regression under various model update scenarios. In this section, we report the empirical results and conclude the findings.
4.1 Model Update Settings
In total, we consider four different causes for model updates.
Change the training configurations (e.g., random seed, learning rate, etc). Despite this setting being less likely in practice, we consider it synthetic evaluation to understand the severity of model update regression.
Scale up model sizes (e.g., increasing the depth or width of neural network). is common that using a larger pre-trained language model brings in a larger performance gain.
Train on more data. Practitioners frequently collect more and more labeled data for training.
Take another factorization choice. It happens when new algorithms or models are invented and adopted.
4.2 Dependency Parsing
Setup For dependency parsing, we consider two popular models, namely the biaffine parser (Dozat and Manning 2017) (deepbiaf) and the stack-pointer parser (M et al. 2018) (stackptr). The biaffine parser a first-order graph-based parser, where the global prediction is decomposed to arc/edge predictions between any two words. The stack-pointer parser is transition-based auto-regressive parser, where the global prediction decomposed into a series of building action predictions. These two represent two distinct factorizations of the dependency parsing problem. Table I (top) presents their accuracies (mean and standard deviation). We can see that in terms of word-level accuracy, the performances of deepbiaf and stackptr are very close. However, in terms of complete match, stackptr slightly better than deephiaf, likely due to its auto-regressive nature.
We study model update regression in three different model update settings deepbigf=deeppiice stackprr= and Shephiaa-=stackppt In the first two settings, the old and new models have
http://fb. http://ff...eeeeeeeeee mee/sematti ceamanttpaarsinddialog
The label can be chain of slot and/or intent labels.

---

Model
EM
spen 1
Model
UCM |
UAS f
s2s-base-part
85.04=0.22
89.83=0.15
s25-large-part
86.86=0.25
90.94=0.15
deephiaf
63.88:0.56
91.76:10.10
s2r-buse
85.67-0.10
90.34-0.10
stuckpir
65.83=0.44
91.1110.09
225-large
86.97=0.11
91.11=0112
Model Update
NFR L
NFI
NFR L
NFI
Model Update
NFR L
NFI
NFR
NFL
deephiuf deeppiuuf
3.67
10.17
1.55
18.80
s2r-hase- 22r-base-part-ec22 par---aatt- base
2.98
21.12
2.25
23.68
stackptr-s strckppr
3.44
10.07
1.64
19.99
Purge parts large
20.03
1.99
deepbiaf -y stackpir
3.84
11.24
2.14
25.57
(2x-haxe-ss2c-argg
3.32
25.45
2.61
29.39
Table I: Accuracy (top) and model update re- gression (bottom) results on dependency parsing. 1: higher is better and lower is better.
Table 2: Accuracy (top) and model update regres- sion (bottom) results on conversational semantic parsing. -part indicates the models are trained with only 4/5 of the available training data.
identical model architecture and training data. The only difference is that they are trained with different random seeds. We present these controlled experiments to demonstrate that even small changes such as altering random seeds can introduce significant regression.
Results Table I (bottomm shows that model update regression is severe across all model update settings, including ddeepiaf= and stackptr==sacckpr; This implies that it could be funda- mentally difficult to reduce model upgrade regression as initialization and optimization processes naturally introduce prediction inconsistencies. On the other hand, model update regression in het- erogeneous model update
<rrepbiaa=saccpppi is even account for up to 25% of the errors made by new models Notably, despite that stackptr improves over deepbiaf by 1.95% on UCM, both NFR and NFI in deepbiaf stackptr is higher than those in homogeneous model updates. This demonstrates that reducing the error rate is not sufficient reduce regression.
severe. Particularly, negative flips
according to UAS).
4.3 Conversational Semantic Parsing
Setup
For conversational semantic parsing, we use the seq2seq parser (Rongali et al. (0200) in our experiments. It formulates the semantic parsing task seq2seq generation of the linearized parse tree given the source text. The global prediction is thus decomposed into a sequence of next-word predictions.
The seq2seq parsers can initialized with different pre-trained language models. We experiment with two model variants: seq2seq parser with (1) roberta-bart (Liu et al. 2019) (s2s-base) and (2) with Saberta-large
(s2s-large). In order to simulate data-driven model updates, we also train parsers with only 4/5 of the available training data, denoted by s2s-base-par and s2s-large- pargte Table 2 (top) shows that s2s-large performs better than s2s- 2a-base and training on more data leads to better performance, in terms of both EM and span EM. We test three kinds of model updates: 22--large-part>>>2s----
22--base-arrt>>s2--bn
and s2s-buxe=>s2s-large
Results As shown in Table (bottom), model update regression is prevalent across different model update settings. The most severe model update regression comes from different pre-trained language models (22s-bue=x255-large new model, while the absolute performance gains are merely 1 1.3% EM points and 0. 0.77% span EM points.
accounts for 25.45% (EM) and 29.33 99.39% (span EM) errors made by the
4.4 Discussions
The key findings observed across all studied tasks and model updates are the following: (1) Model update regression is prevalent on two typical structured prediction tasks and various model update settings. (2) Even small changes in training configuration such as changing the random seed can introduce significant regression (a minimum of 1.55% NFR is observed). (3) Heterogeneous model updates lead to severer regression compared to homogeneous model updates. (4) Updating pre-trained language models leads to higher regressions than those caused by training more data. As much as out of 10 test errors are regression errors (NF1 (5) Improving accuracy alone does not necessarily reduce regression (in fact, NFR can be larger than accuracy gain).
5

---

5 Reducing Model Update Regression
First, we describe the adaptation of the methods explored in reducing model update regression in classification problems (Shen et al. 2020 Yan et al., 2021; Xie et al. (021). Then, we introduce a
novel method, backward-congruent re-ranking that is better suited to structured prediction tasks.
5.1 Model Ensemble
In classification problems, Yan et al. 2021 Xie al. 2021 found that model ensemble can reduce model update regression without any information about the old model. This can be attributed to the reduction of variance by ensembling: Every single model may capture the training data from distinct aspect. The ensemble aggregates different aspects and has a shorter average distance to other train different models with different random seeds and average the
single models. Concretely, local prediction scores from all the models during decoding. Nevertheless, ensembles may not be practical due to the high cost of training and inference.
We note that parameter averaging (i.c., averaging the parameters of different models and using the resulting model for prediction) is an alternative to prediction ensembling. However, preliminary experiments show that this method gives worse performance than prediction ensembling.
5.2 Knowledge Distillation
Knowledge distillation (KD) is a technique originally proposed for model compression (Buciluá et al.
2006 [Ba and Caruana 2014 Hinton et al.] 2015), where a (smaller) student model is trained to mimic a (larger) teacher model. By treating the old model as the teacher and the new model as the student, KD has been proved promising approach for reducing model update regression both vision and language classification tasks YYan et al. 2021 Xie et al. 202T). The instantiations are usually based on loss function computing the cross- entropyy between the output distributions predicted by the teacher and the student. However, for structured prediction problems, the output space is exponential in size, making the exact output distribution computationally intractable. Although some efficient methods for approximating the cross-entrope have been proposed (Zmigrod et al.) [202 Wang et al. 202T). they assume the models follow certain factorizations and thus have limited application scenarios (e.g., the techniques proposed in (Ziggrd et al., 2021a) a) can only be used for edge-factored parsers). One may think of performing KD on local predictions or hidden representations instead. However, such distillation can also be infeasible between models with different factorizations or structures (e.g.,
in dependency parsing).
To tackle the above problem, we borrow the idea from sequence-level KD (Kim and Rush 2016), which approximates the teacher distribution with its mode. Specifically. sequence-level KD suggests to (1) run the teacher model over the training set to create pseudo training data, (2) then train the student model on this new dataset. The idea is well-suited to the problem of model update regression because it is model-agnostic; it does not assume any specific factorizations for the old and new models.
5.3 Backward-Congruer Re-ranking
Unlike model ensemble, knowledge distillation attempts to explicitly align the behaviors the new model to the old model during training. Alternatively, we propose Backward-Congrae Re-ranking (BCR), which does not impose any constraint on the training of the new model and only takes effect during inference.
Re-ranking is popular approach in structured prediction to combine the strengths of two different models (Collins and Kool 2005 Socher et al. 2013] Le and Zuidemal 2014 Do and Rehbein, 2020). suggests use one model as the candidate generator for creating a candidate pool and use the other model as the re-ranker for picking the best candidate. While previous work has been focused on developing powerful re-rankers for overall performance improvement, the purpose BCR is to reduce model update regression. BCR treats the new model as the candidate generator and the old model as the re-ranker. Our motivations are two-fold. First, structured prediction models can produce many possible predictions. Second, different predictions may achieve similar error rates but differ by

---

the mistakes they made. Among them, the most likely one according to the old model should have the least prediction flips. These make re-ranking particularly useful for structured prediction.
Formally, we have the old model and the new model parameterized by Onew and Dold respectively. For given input æ. the new model first generates set of candidates @ (z:). Then we choose the prediction y² with the highest score computed by the old model.
arg max Pou (y]zr),
DEGEN (a)
where Pows (y/=) is the old model's generation probability (score) of y given the input x and GEN can be implemented by various decoding methods, including maximization-basee search such as beam search and k-best MST algorithm (Zmigrod et al. 2020 2021b), and stochastic sampling such as
top-k sampling (Fan al. 2018. Radford et
2019) and top-p sampling (Holtzman al. 2019).
Dropout-p Sampling Maximization-basel search attempts to find the top outputs that received the highest scores from the model. This inherently leads a set of similar candidates and re-ranking may suffer from the lack of diversity. On the other hand, stochastic sampling introduces more variances among candidates by randomized choices during decoding. However, traditional stochastic sampling may suffer from the low quality of sampled outputs due to the deterministic nature of structured prediction. Furthermore, it is unclear how to adapt popular sampling methods, such as top-k and top-p sampling, solutions that not formalized as sequence generation (e.g., the biaffine parser).
In this work, we explore a special sampling method, dropout-p sampling, that has attracted little attention in the literature. Dropout (Srivastava et al. 2014) is a regularization technique used in almost all modern neural network-based models. The key idea to randomly drop some neurons from the neural network during training. Normally, dropout is turned off during inference. However, in dropout-p sampling, we keep using dropout with dropout rate p. Compared to traditional sampling methods, dropout-p sampling has the following advantages: (I) It directly changes the scoring function and keeps any default decoding algorithm unchanged; (2) It can be regarded as conducting global sampling instead of series of local sampling at each decoding step, potentially improving the formality of output structure; (3) also has a broader applicable scope that is not limited to sequence generation models.
Discussion Following previous work (Seen et al. 2004; Yan et al. 2021 Xie et al. 2021). our discussion has been focused on handling one model update. However, BCR can be extended to handle multiple turns of model updates. To do so, one can keep the most recent & models and use a weighted combination of their scores the re-ranking metric. In practice, can be set to trade-off between performance and runtime cost.
One downside of BCR is that must maintain and deploy both the old and new models. Because the re-ranking step has less time complexity compared to the decoding algorithms and the computation is fully parallelizable, this does not create much additional inference latency. However, the increase in memory footprint does entail an increase in the inference hosting cost. One remedy could be to use knowledge distillation to distill the old model(s) into a smaller one, which leave for future work.
Experiments
6.1
Setup
For the ensembles of biaffine parsers, we first average the edge scores from each model, then run MST the average scores. For stack-pointer and seq2xeq parsers, we average the local prediction scores from each model at each decoding step. We combine 5 models in model ensemble. The impact of ensemble size is discussed in Appendix B For knowledge distillation, as mentioned in Section 5..2 we simply replace the ground-truth output in the original training set with the old model's predictions when training the new model For BCR, various decoding methods are explored for candidate generation. Specifically, we use bebest spanning trees algorithm Zmigrod et al. 2220 (021b) with biaffine parsers, and beam search with stock-pointer parsers and seq2seq parsers. We also explore
We also tried the combination of pseudo and original training data but found little difference in performance and regression.
7

---

deepbiaf deeppiaa
stackptr > stackptr
deepbiaf > stackpir
UCM
UAS
UCM
UAS
UCM
UAS
ACC
NFR
NFI
ACC NER
NFI
ACC NER
NFI
ACC NER
NFI
ACC NFR
NEI ACC NFR
NFI
Old
63.88
91.76
65.83
91.81
63.88
91.76
Untreated
63.97 3.69
10.25
91.64 1.66 19.85
66.03
3.43
10.10
91.73 1.67 20.20
66.03
3.73
10.98
91.73 2.10 25.37
Distillation
64.00 3.82
10.62
91.67 1.62 19.45
65.66
3.57
10:40
91.70
1.70 20.49
66.11
3.62
10.68
91.78
2.03
24.71
Ensemble
64.81
2.51
7.14
92.10
1.02
12.97
67.21
2.21
6.74
92.21
1.11
14.30
67.21
2.83
8.62
92.21
1.62
20.75
BCR
64.36 1.12
3.14
91.78
0.84
10.21
66.45
1.05
3.12
91.88 0.84
10.30
66.76
1.20
3.60
92.01
1.11 13.87
Table 3: Comparison of different regression reduction methods on dependency parsing (NFR & NFI, + ACC f) using unlabeled metrics (results of labeled metrics are shown in Appendix A). Old indicates the old model's performance before model update. Untreated denotes the results of new models without any treatment. BCR denotes the results with hackward-congrueet re-ranking.
sampling- based decoding methods such top-k sampling (A {5. 10, 50, 100)) (Fan al., 2018 Radford et al. 2019). top-p sampling (p E {0.95, 0.90, 0.85, 0.80)) (Holtzman et al., 2019), and dropout-p sampling (p E {0.1, 0.2, 0.3, 0.4]). The number of candidates in BCR is set to T0. Our experiments show that the best results are always achieved using dropout-p sampling. For brevity, we only report the best results and defer the detailed analysis of the candidate generation methods to Section 6.4 We include the model update regression results without any treatment in Section 4 (denoted Untreated) for reference (i.e., the new models are trained normally with a different set of random seeds).
For dependency parsing, we adapt the implementation of the biaffine parser Dozat and Manning (2017) and the stack- pointkrr parser Ma et al. (2018) in NeuroNLP2 We use the default configurations suggested in (Ma et al. 2018) for training and testing models. For conversational semantic parsing, we re-implement the seq2seg parser Rongali et al. (2000) using Fairseq Ont et al. (2019) and the pre-trained language models Liu al. (2019) are downloaded from HuggingFacd
6.2 Results on Dependency Parsing
shows the results of different methods for reducing model update regression on the dependency parsing task. We can see that model ensemble brings down negative flips in all model update settings compared with the untreated baseline (on average 28% relative reduction in NFI), On the other hand, knowledge distillation shows little effect on reducing model update regression. We hypothesize that it is due to the almost perfect word-level accuracy (UAS) of the old model when running on the training set. As a result, there is little difference between the original training set and the synthetic training set used for distillation, leading to performance close to the untreated baseline.
For BCR, substantially reduces NFR and NFI across all model update settings (an average of 58% relative reduction in NFI), greatly outperforming model ensemble and knowledge distillation. This is very encouraging given that model ensemble is often the most effective approach to reduce regression in classification problems (Yan et al. 202T, Xie et al., 202T) or even considered as a paragon (Yan et al. 202T). Interestingly, BCR delivers more pronounced gains over model ensemble In particular, BCR reduces NFI by 67% while model ensemble only obtains 21% relative reduction, according to UCM. The reason might be that model ensemble can only reduce the variance in models of the same kind, while the intrinsic gap between different kinds of models cannot be reduced. In contrast, BCR selects the most compaaibiee prediction regardless of the source of candidates. Another interesting observation is that BCR can also improve the overall accuracies though the improvements slightly lag behind model ensemble (e.g., +0.19 +0.37 UAS points on average). We regard it side benefit since our primary goal here is to reduce model update regression rather than to boost accuracy.
in the heterogeneous model update setting (deepbiaf=stackpptt
6.3 Results on Conversational Semantic Parsing
Table shows the results of different methods for reducing model update regression on the conver- sational semantic parsing task. The empirical results generally support the main findings on the
httpae//ggthub..oo/xx/uzeeeaaa/a attps://huggingface.co
8

---

22s-brse-part Sourt->$2. >sss-be
s2s-large-pau s2s-large
s2s-base > >s2s-large
EM
span EM
EM
span EM
EM
span EM
ACC NFR
NFI
ACC NER
NFI
ACC
NER
NFI
ACC NER
NFI
ACC NFR
NFI
ACC NFR
NE
Old
85.04
89.83
86.86
90.94
85.67
90.34
Untreated
85.88
2.98 21.12
90.50
2.25
23.68
86.98
2.61
20.03
91.10
1.99 22.37
87.04
3.16 24.40
90.96
2.68 29.64
Distillation
85.54 2.61 18.06
90.23
2.07
21.21
86.64
2.04
15.25
90.88
1.63 17.90
87.69
1.97 15.97
91.53 1.79
21.16
Ensemble
86.90
1.73
13.24
91.35
1.28
14.80
87.65
1.17
9.47
91.62
1.00
11.78
87.65
2.70
21.88
91.62
2.15
25.60
BCR
86.16
0.75
5.39 90.69
0.69
7.42
87.41
0.54
4.30
91.24
0.67
7.61
87.49
0.69
5.51
91.53
0.74 8.78
Table 4: Comparison of different methods on conversational semantic parsing (NFR J. NFI L ACC 1):
30
Method
FM
span FM
NFI
ACC NFR
NFI
ACC NFR
NET
ER
Untreated
87.04
3.16
24.40
90.96
2.68
29.64
NFR
Beam, bm10
87.61
1.63
13:20
91.52
1.49
17.57
top-k=5
87.71
2.16 17.55
91.62
1.81
21
top-p=0.95
87.73
2.18
17.77
91.66
1.81
21.65
12
dropout-p=O..1
87.49
0.69
5.51
91.53
0.74
8.78
6
dropout-p=0.11
18.01
1.26
10.54
91.98
1.06
13.23
dropout--=0.22
87.87
0.88
7.27
91.88
0.81
10.03
0
dropout-y=0.3
87.49
0.69
5.51
91.53
0.74
8.78
2
3
4
dropout-p=0.44
87.26 0.72
5.63
91.31
0.81
9.36
Number candidates
Table 5: Comparison of different decoding methods with selected parameters of each method (22x-baxe=s<3-largee)
Figure 2: Effect of increasing number of candi- dates NFI (L). NFR (L), and error rate (ER ))
(625-bax-=s2>-large))
dependency parsing task: (1) Knowledge distillation has little effect on reducing NFR and NFI (except for the s2s-buse-=ss-llarg setting). (2) Model ensemble brings moderate reduction (28% relative NFI reduction on average) across all model update settings. (3) BCR achieves substantial reduction (73% relative NFI reduction on average), greatly outperforming model ensemble and knowledge distillation. BCR not only reduces model update regression but also improves accuracies as compared to the untreated baseline. This is particularly intriguing since we are using a weaker model (the old model) is used to re-rank the candidates generated by a stronger model (the new model). We speculate that the reason is the complementary effect different models in diffcrentiating distractors.
6.4 Analysis of BCR
Next, study the two key components of BCR, namely candidate generation and the re-ranking metric. We analyze the results different decoding methods described in Section 5.3 and empirically reveal the performance upper- hooerr-buunds of BCR.
Candidates for Re-ranking We present the results of BCR with different decoding methods in Table [] For clarity, we only report the best result obtained using each decoding method together with the optimal
setting (except for dropout-p sampling). We can see that dropout-p sampling obtains the largest reduction in both NFR and NFI, substantially outperforming beam search and traditional truncated sampling (the gaps in NFI are larger than 10 points). top-p and top-k sampling are the least effective approaches. We hypothesize that traditional truncated sampling is not suitable for our structured prediction tasks. Unlike open-ended generation tasks, there are only a few valid local predictions at each decoding step. Therefore, the candidates generated by truncated sampling may lack diversity. For dropout-p sampling, as p increases from 0.1 to 0.3, the improvements in accuracies shrink while the reductions NFR and NFT increase. This is expected since larger p leads to lower quality of individual candidates but more diverse candidate pool.
We plot the curves of NFR, NFI, and error rate (ER) with the number of candidates in Figure 2 As seen, all three metrics decrease as the number of candidates increases. However, they converge at different points. NFI and NFR drop significantly even after the reduction in ER reaches its limit.
Oracle Re-ranker To shed light on the effectiveness of our choice of the re-ranking metric (i.e., the old model's 's prediction score P we compare to two kinds oracle re-rankers: one using span-level EM (w/ ACC) and one using span-level NFR (w/ NFR). The ACC and NFR re-rankers

---

represent the upper bounds of ACC and NFR that re-ranking methods can achieve. The results are presented in Table 6 It can be observed that the NFI and NFR of using Pau are very close to the upper bounds for reducing model update regression (w/ NFR). Another notable observation is that the NFIs of using Pawu and NFR re-ranking are much lower than the NFI of ACC re-ranking. This demonstrates that improving accuracy is not equivalent to reducing model update regression.
For dropout-p sampling, the over- all computation overhead of the decoding step grows linearly with the number of candidates. Nevertheless, different runs of sampling can be done in parallel. With the same inference hardware (one Nvidia V100 GPU) and the same batch size of 32, the decoding and re-ranking speeds of deephiaf are 171 and 244 sentences per second, and 64 and 221 sentences per second for stackptr. For seq2seq parsers, we observe that the speed of re-ranking is about 5x of the de- coding speed. In practice, the re-ranking step can be made even faster it generally allows for larger batch sizes.
Inference Speed
Re-ranker
EM
span EM
ACC
NFR
NET
ACC
NFR
NEI
Untreated
87.04
3.16
24.40 90,96 2.68
29.64
w/ Péua
87.49
0.69
5.51 91.53 0.74
8.78
w/ NFR
89.50
0.65
6.21 92.90
0.58
8.13
ACC
92.68 0.65
8.90 95.04
0.59 11.82
Table 6: Comparison of BCR (P to ora- cle re-rankers (NFR, ACC) (s2s-base=s2s- large).
Conclusions
This paper presented the first study on the model update regression issue in NLP structured prediction tasks. Experiments on two representative structured prediction tasks showed that model update regression severe and widespread across different model update settings. To reduce model update regression, we explored knowledge distillation, model ensemble, and soccward-congreett re-ranking. Backward-congruen re-ranking consistently achieves much more significant regression reduction than distillation and ensemble methods. In the future, model update regression should be examined in other structured prediction tasks such as text generation in NLP and image segmentation in computer vision, and scenarios of multiple rounds of model updates.
References
Lei Jimmy Ba and Rich Caruana. 2014. Do deep nets really need be deep? In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, pages 2654-2660
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523-6541.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu- Mieulescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541.
Zhiyuan Chen and Bing Liu. 2018. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):12077
Michael Collins and Terry Koo. 2005. Discriminative reranking for natural language parsing. Computational Linguistics, 31(1)225-70.
Bich-Ngoc Do and Ines Rehbein. 2020. Neural reranking for dependency parsing: An evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4123-4133.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In International Conference Learning Representations, 2017.
Arash Einolghozati, Panupong Pasupat, Sonal Gupta, Rushin Shah, Mrinal Mohit, Mike Lewis, and Luke Zetrlemoyer. 2019. Improving semantic parsing for task oriented dialog. arXiv preprint 17Xiv:1902.06000.
10

---

Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computeational Linguistics, pages 2214-2220.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 889-898.
Alexander Gepperth and Barbara Hammer. 2016. Incremental learning algorithms and applications. In European symposium on artificial neural networks.
Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi. 2019. Fairness-aware ranking in search & recommendation systems with application to linkedin talent search. In Proceedings of the 25th acm sigkad international conference on knowledge discovery & data mining, pages 2221-2233
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787-2792.
Geoffrey Hinton, Oriol Vinyals. and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprini urXiv:/503.0255:
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327.
Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Buree 2019. Complexity-wcighted loss and diverse reranking for sentence simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3137-3147.
Sandra Kübler, Ryan McDonald, and Joakim Nivre. 2009. Dependency parsing. Synthesis lectures on human language technologies, 1(1):1-127.
Phong Le and Willem Zuidema. 2014. The inside-outside recursive neural network model for dependency parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 729-739.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techhologies, pages 110-119.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692.
Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1403-1414.
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, and J. Weijer. 2020. Class-incremental learning: survey and performance evaluation. ArXiv, abs/2010.15277.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (emonstrations)) pages 48-53.
11

---

G. I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and S. Wermter. 2019. Continual lifelong learning with neural networks: review. Neural networks the official journal of the International Neural Network Society, 113:54-71.
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal purt-of-speed tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, pages 2089 2099-2096.
Robi Polikar, Lalita Upda, Satish S Upda, and Vasant Honavar. 2001. Learn++: An incremental learning algorithm for supervised neural networks. IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 3 31(4):497-508.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. 2020. Don't parse, generate! sequence to sequence architecture for task-oriented semantic parsing. In Proceedings The Web Conference 2020, pages 2962-2968.
Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2019. Masked language model scoring. arXiv preprint arXiv: 1910. 1910.14699,
Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In Proceedings of the Human Language Technology Conference the North American Chapter of the Association for Computational Linguistics: HL.T-AACLL 2004, pages 177-184.
Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. 2020. Towards backward-compatin representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3368-6377,
Noah A Smith. 2011. Linguistic structure prediction. Synthesis lectures on human language technologies, 4(2):1-274.
Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pages 455-465.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 1929911929-998.
Frederik Träuble, Julius von Kügelgen, Matthäus Kleindessner, Francesco Locatello, Bernhard Schölkopf, and Peter V. Gehler. 2021. Backward-compatibl prediction updates: A probabilistic approach. arXiv preprint trXiv:2107.001
Xinyu Wang, Yong Jiang, Zhaohui Yan, Zixia Jia, Nguyen Bach, Tao Wang. Zhonggiang Huang. Fei Huang, and Kewei Tu. 2021. Structural knowledge distillation: Tractably distilling information for structured predictor In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 550-564, Online. Association for Computational Linguisties.
Yuqing Xie, Yi-An Lai, Yuanjun Xiong, Yi Zhang, and Stefano Soatto. 2021. Regression bugs are in your model! measuring, reducing and analyzing regressions NLP model updates. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 6589-6602.
12

---

Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang. Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto. 2021. Positive-congee congruunt training: Towards regression model updates. Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pages 14299-14308,
Kyra Yee, Yann Dauphin, and Michael Auli. 2019. Simple and effective noisy channel modeling for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5696-5701.
Pengcheng Yin and Graham Neubig. 2019. Reranking for neural semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4553-4559,
Ran Zmigrod, Tim Vieira, and Ryan Cotterell. 2020. Please mind the root: Decoding arborescences for dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4809 4819.
Ran Zmigrod, Tim Vieira, and Ryan Cotterell. 2021a. Efficient computation of expectations under spanning tree distributions, Transactions of the Association for Computational Linguistics, 9:675- 690.
Ran Zmigrod, Tim Vieira, and Ryan Cotterell. 2021b. On finding the k-best non-projective depen- dency trees. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1 1324 1337.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section f
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [No] The data used in
this paper is publicly available. We will release our code upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 6.1 and Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section [
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 6
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "sOxxbl35MQE", "reviewer": "Reviewer_TTdd", "summary": "This work focuses on the update regression problem for structured prediction. The authors explore two standard structured prediction approaches (graph-based and transition-based). They conduct a set of experiments and introduce a noveal approach to mitigate the update regression problem.\n\n**EDIT after discussion with the authors:**\nI still disagree with authors, but they made a good point in our discussion, so I upgraded my grades. I strongly recommend the authors to soften some claim made in the paper, refer that in some cases KL div technique for KD can be used and give proper citations about how one could do that.", "questions": "- is there any formal motivation for the contribution in section 5.3?\n- is there any reason why the previous work listed in my comment above has not been considered to adapt preivous work in the non-structured case to the structured case?", "limitations": "nothing to report", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts:**\n- this work focuses on an important and under-studied problem\n- experimental results are interesting\n\n**Weaknesses**\n\nUnfortunately, this paper has many weaknesses. First, and importantly, the main contribution of this paper (section 5.3) is not well motivated and the explanation of the contribution itself is really handwavy. Second, explanation of the different baseline approaches is also too handwavy, making the overall paper difficult to understand and not self-contained.\n\nOn the motivations: the authors argue that previous approaches cannot be applied to structured prediction. I strongly disagree with this.\n\n- l 227, \"the output distributiion of global prediction is often intractable\": in the non-projective dependency parsing case, the distribution can be computed using the Matrix Tree Theorem [1, 2, 3]. If we restrict the problem to projective trees, this can be computed via dynamic programming [4, 5]. For the transition based model, approximated approaches have been explored in the litterature [6]\n- l. 222, \"the instantiations of KD are usually based on a loss function computing the cross-entropy of output distributions [...] however existing methods are no directly applicable to structure prediction tasks\": I don't understand why. First, the next sentence is false (see previous point), but also KL divergence between structured distributions has been studied in the litterature. For non-projective trees, see [7, Section 6.3], for methods based on dynamic programming see [8]\n- l. 263, \"furthermore, it is unclear how to adapt existing sampling methods to solutions that are not formalized as sequence generation\": Recent work has considered sampling from the non-projective tree distributions, including sampling without replacement [9, 10]. Moreover, previous work has also considered perturb-and-MAP approaches [11, 12, 13]. Finally, in the case of dynamic programming algorithms, it is well known that it is possible to sample from the associated exponential familly distributions, see e.g. [14]\n\n**Related work**\n\nAs suggested by the comment above, the literature is not properly explored or cited by the authors.\nThere are similar problems in the introduction and related work section. For example:\n- l. 54: authors cite Dozan and Manning (2017) for graph-based parsers, whereas the correct citations is more than 10 years older [15]\n- l. 21: for transition-based parsers, they cite Ma et al. (2018), better citations would be [16, 17]\n\n[1] Structured Prediction Models via the Matrix-Tree Theorem (Koo et al.)\n\n[2] Probabilistic Models of Nonprojective Dependency Trees (Smith and Smith)\n\n[3] On the complexity of non-projective data-driven dependency parsing (McDonald and Satta)\n\n[4] Semiring parsing (Goodman)\n\n[5] Differentiable Dynamic Programming for Structured Prediction and Attention (Mensch and Blondel)\n\n[6] Globally Normalized Transition-Based Neural Networks (Andor et al.)\n\n[7] Efficient Computation of Expectations under Spanning Tree Distributions (Zmigrod et al.)\n\n[8] First- and Second-Order Expectation Semirings with Applications to Minimum-Risk Training on Translation Forests (Li and Eisner)\n\n[9] Efficient Sampling of Dependency Structures (Zmigrod et al.)\n\n[10] Unbiased and Efficient Sampling of Dependency Trees (Stanojevi)\n\n[11] Perturb-and-MAP random fields: Using discrete optimization to learn and sample from energy models (Papandreou and Yuille)\n\n[12] Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder (Corro and Titov)\n\n[13] Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming (Corro and Titov)\n\n[14] See section 17.4.5 of  Machine Learning: A Probabilistic Perspective  (Muprhy) for the idea and Latent Template Induction with Gumbel-CRFs (Fu et al.) for application to CRF-like distribution\n\n[15] Non-projective Dependency Parsing using Spanning Tree Algorithms (McDonald et al.)\n\n[16] A Classifier-Based Parser with Linear Run-Time Complexity (Sagae, Lavie)\n\n[17] Algorithms for Deterministic Incremental Dependency Parsing (Nivre)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "khR1SVPGf0G", "reviewer": "Reviewer_uHkD", "summary": "The paper examines model update regression in NLP structured prediction tasks. Model update regression is an issue appearing when a model is updated and the new model classifies part of the test examples negatively, while the old model classifies them positively.\nThe novelty of the paper is that it explores the problem in structured prediction, while it has been previously explored for classification.\nThe work proposes a new method for correcting the new model from the old one - backward-congruent re-ranking and experimentally shows that it performs better than other previously used methods, such as model ensemble and knowledge distillation. \nThe work also defines a new measure related to the model update regression problem in structured prediction - negative flip impact.", "questions": "n/a", "limitations": "n/a", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Studying the model update regression in relation to structured prediction is interesting and important. The contributions are novel enough.\n- The paper is very well-written and clear. The problem, previous work, and the proposed solution are clearly described.\n- The proposed improvements are meaningful and the experiments show that the proposed method for correction - backward-congruent re-ranking works better than the alternatives on the explored tasks.\n\nWeaknesses:\nI did not find any major issues with this work. I am suggesting some things I found that could improve it further.\n- Line 120: maybe add a citation for UD 2.2. On the next line only the POS-tagging dataset is cited.\n- Line 182: an example could be beneficial for understanding how the global prediction is thus decomposed into a sequence of next-word predictions.\n- Line 217: from the description it is not really clear how checkpoint averaging and prediction ensembling differ.\n- Table 5: It is not clear what is the difference between rows 5 and 8 (for dropout-p=0.3)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M4eVSm9WT-A", "reviewer": "Reviewer_XYXe", "summary": "The paper studied a problem called model update regression in the field of structured prediction for NLP, which means a new model that achieves higher scores may perform worse than its baselines on some cases. The authors proposed an approach, BCR, to handle this issue. The basic idea is to use an old model to filter predicted candidates from the new model. A trick called dropout-p sampling is also applied to improve the diversity and quality of produced candidates. The experiments had been done on Dependency Parsing and Conversational Semantic Parsing, which were aimed to show the effectiveness of BCR.", "questions": "1) Can you put some case studies in the Experiment section to show where your gains are from?\n2) Since more than one model is used in your approach, does this strategy cause some of your improvements are in fact overlapping with the ensemble approach?", "limitations": "Besides my concerns raised in 'Strengths And Weaknesses', I think the authors should do more experiments to empirically confirm the problem and describe it in a more acceptable way. I have seen that the authors took a lot of space to show the performance numbers of their models, which I believe should not be the focus of this paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1) the writing is good and clear;\n2) the problem is well defined and the solution is straightforward.\n\nCons:\n1) syntactic parsing is indeed very classical for computational linguistics but only stands for a tiny part of tasks in the field of structured prediction. Some other tasks like NER and POS Tagging are far more common and have much more applications in the NLP industry. Plus, a lot of space in the paper is used for introducing syntactic parsing, which causes a bar even for researchers from NLP and is not necessary for its future applications to common scenarios. Therefore, The authors should investigate more representative tasks in structured prediction such that the formulation is more clear and the work has a broad impact on the whole community.\n2) knowledge distillation for structured prediction is a mature field in NLP. Please at least see this work, Structure-Level Knowledge Distillation For Multilingual Sequence Labeling, and check the recent literature. Hence, the authors shouldn't say like 'existing knowledge distillation-based solutions cannot be directly applied to structured prediction models' in the paper, and more solid experiments are expected for comparisons.\n3) the problem mentioned in the paper is formulated in a quite intuitive manner, which might lead to many explanations that can't be proved false. For example, from my view, the problem may be a cause of randomness in the premise that neural networks can't fit the whole dataset, which may explain why the ensemble model works. Therefore, I believe a theoretical foundation is expected.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AzAEgCTGhmW", "reviewer": "Reviewer_EpNN", "summary": "This paper tackles the problem of model update regression: when a model with better aggregate performance is deployed, it may make mistakes that a previous model didn't make.  How can we minimize these \"regressions\"?  Specifically, this paper is the first to look at this problem in a structured prediction setting. On both dependency parsing and on a conversational dialogue / semantic parsing task, the paper evaluates new models to see how many examples they regress on. The paper describes several techniques to mitigate regression, including a new one, backward-congruent reranking, that reranks the new model's output using the old model. This is combined with a form of dropout at test time to yield diverse samples. The paper shows that BCR achieves good task accuracy while minimizing the number of model regressions.", "questions": "How is RoBERTa used for the seq2seq semantic parser? Is this the encoder and a separate transformer decoder model is learned? Please clarify since this is not a standard use of the encoder-only RoBERTa model.", "limitations": "There is little discussion of limitations and societal impact, but I don't see this as a big problem for this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\n- The problem setting in this paper is novel. While a somewhat niche problem, I am convinced of the core argument that this could be useful (with some reservations; see below under \"Weaknesses\").\n\n- The exploration of \"new\" models is quite thorough, and it's nice to see the contrasts between some different types of model updates, including changing the mode paradigm.\n\n- The BCR technique is simple and I could see it being widely adopted for this task\n\n- The dropout-p sampling technique is interesting and a novel way of getting diverse samples from these kinds of generation models.\n\nWEAKNESSES\n\nThe main conceptual weakness I see in this paper is the notion of a regression as it ties to model confidence.\n\nIt seems to me that if you take a model achieving 90% accuracy and perturb its weights slightly, it may still get 90% accuracy but make different predictions.  Many of the flipped predictions will be those where the 2nd-best option had nearly the same probability as the 1-best option; that is, cases where the confidence was low / entropy of the label distribution was high. It's not clear to me in these cases that the first model is getting things \"right\" and the second is getting them \"wrong\", more that the model had high uncertainty and it broke one way (correct) for some examples and another way (incorrect) for other examples.\n\nConsidering that BCR's performance is always lower than that of the ensemble, it seems to me that what's happening here is that this is really an ensemble that weights more heavily towards the earlier model. We aren't ensembling the two models symmetrically, but by reranking outputs from the second using the first, we're really only making changes where the second model is significantly more confident and doesn't even return what the first model initially preferred, resulting in a smaller performance gain but also fewer changes. So basically this is a clever way of combining the models but mostly has the effect of preferring the first.\n\nI also wonder about the NFR/NFI metrics as they relate to the motivation of this paper. The idea of a system changing and then breaking UX or downstream modules that were depending on certain things working is a problem that resonates with me. However, do NFI and NFR really correlate with this motivation?  Again, many of the differences seem like they could most heavily depend on examples that were basically half-working to begin with, which were probably not consistent features a user was very attached to.\n\nAs a result, while this paper explores a different point in the design space of how to ensemble these two models, I'm not sure what it presents is really groundbreaking in terms of results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept