GROOT: Corrective Reward Optimization for Generative Sequential Labeling
Authors: Kazuma Hashimoto
Karthik Raman
Venue: Submitted to ICLR 2023 (2023)
Keywords: sequential labeling
reward optimization
natural language processing

Abstract: Sequential labeling is a fundamental NLP task, forming the backbone of many applications.
Supervised learning of Seq2Seq models (like T5) has shown great success on these problems.
However there remains a significant disconnect between the training objectives of these models vs the metrics and desiderata we care about in practical applications.
For example, a practical sequence tagging application may want to optimize for a certain precision-recall trade-off (of the top-k predictions) which is quite different from the standard objective of maximizing the likelihood of the gold labeled sequence.
Thus to bridge this gap, we propose GROOT -- a simple yet effective framework for Generative Reward Optimization Of Text sequences.
GROOT works by training a generative sequential labeling model to match the decoder output distribution with that of the (black-box) reward function.
Using an iterative training regime, we first generate prediction candidates, then correct errors in them, and finally contrast those candidates (based on their reward values).
As demonstrated via extensive experiments on four public benchmarks, GROOT significantly improves all reward metrics.
Furthermore, GROOT also leads to improvements of the overall decoder distribution as evidenced by the quality gains of the top-k candidates.

Under review as a conference paper at ICLR 2023
GROOT: CORRECTIVE REWARD OPTIMIZATION FOR GENERATIVE SEQUENTIAL LABELING
Anonymous authors
Paper under double-blind review
ABSTRACT
Sequential labeling is a fundamental NLP task, forming the backbone of many applications. Supervised learning of Seq2Seq models has shown great success on these problems. However, the training objectives are still significantly dis- connected with the metrics and desiderata ple, a practical sequence tagging application may want to optimize for a certain precision-recall trade-off (of the top-k predictions) which is quite different from the standard objective of maximizing the likelihood of the gold labeled sequence. Thus bridge this gap, we propose GROOT a simple yet effective framework for Generative Reward Optimization Of Text sequences. GROOT works by train- ing generative sequential labeling model match the decoder output distribution with that of the (black-box) reward function. Using an iterative training regime, we first generate prediction candidates, then correct errors in them, and finally contrast those candidates (based on their reward values). As demonstrated via ex- tensive experiments on four public benchmarks, GROOT significantly improves all reward metrics. Furthermore, GROOT leads to improvements of the overall decoder distribution as evidenced by the quality gains of the top-k candidates.
about in practice. For exam-
INTRODUCTION
ATIS
SNIPS
Sequential labeling tasks are ubiquitous among NLP applications. Tasks ranging from syntactic analysis (e.g., POS tag- ging and phrase chunking) to semantic analysis (e.g., named entity recognition, slot filling, and query segmentation), critical components
GROOT
GROOT
NLL baseline
NLL baseline
Reward
Reward
2
end-to-end appli- cations, such search engines and goal- oriented dialog systems.
1st 2nd 3rd Rank Beam Search
1st 2nd 3rd 4th 5th Rank in Beam Search
Figure/ Results for our model (GROOT) the NLL baseline demonstrating the precipitous drop-off in qual- ity of NLL model predictions outside the top-1.
Advances in pretraining of generative language models (LMs) like T5 (Raffel
al. 2020) and mT5 (Xue et al. 20221)
have enabled us to use the same training
strategy seamlessly across these diverse sequence labeling tasks. We can fine-tune a pretrained LM by maximizing the likelihood of generating the ground-truth (human annotated) labeled data.
However, in practice, the metrics and constraints we may care about remain fairly disconnected from the standard Negative Log- Ligellhoodood (NLL) objective used to train these models. To understand this better, consider an example an entity recognition model within an e-commerce system. This model would be typically trained data of the following form:
Input:
black & decker blender under 100
Label: [BRAND black & decker] [PRODUCT blender] [PRICE under 100]
While this e-commerce pipeline could utilize the model's predictions in different ways, likely use is in retrieving candidates that match the predicted annotations. However, with models being imperfect, even well-trained models may make errors like:

---

Under review as a conference paper at ICLR 2023
Input text:
Original training data
Generated training data for reward optimization
XY W U Ground-truth annotation: [e w [C
L
2
Lt+1
Top-k predictions:
train"
NLL
data generation
GROOT oea
[ W V] - reward value: -8.00
(
reward value: -3.66
Selected candidate by the best reward
"Replace"
((0)
(())
A
[AX]Y [C
reward value: 0.667
"Drop" [A Y ( V]
Pre-trained model
NLL baseline
GROOT model
reward value: 3,333
Corrected candidate with better reward
(a) Overall pipeline of GROOT
(b) Data generation process by correction operations
Figure 2: (a) An overview of the GROOT pipeline and (b) training data generation process.
NLL
(a) Model predictions for example from SNIPS validation
0.213
book [party aize number seven] in [spatial relation neighboring] (geographic poi moorpark]
0,564
book betaacyy-iieennliee seven] in [spat wettial.relatiin neighboring] [city moorpark] (*)
seven] in [spaLial.reld] wel.leeeeiiiinn neighboring] [poi moorpark]
1.311
book (pacty-s wtt.yyssiee.mumme..
1,443
book [party aize number seven] in [spatial relation neighboring) lobject name moorpark]
1,878
book (pacly.sise ppaaclysiie_nuabeee ieubbeeeerrrrrrre.eeeeeeeeee scven] [spatial relation neighboring moorpark]
NLL
(b) Model predictions for example from the SNIPS training set
0.0003
add (artist stephen menally) to [p-aylist confidence boost] (*)
3,9628
add (entity name stephen menally] to [playlise confidence boost]
4.0684
(art st stephen menally] [playlist confidence] boost
4.6491
add (artist stephen menally] [entity name confidence boost]
5,3953
add [artist stephen menally] to [album confidence boost]
Table 1: Top-5 predictions of NLL baseline model (with errors bolded, and perfect prediction marked with (*)).
Incorrect prediction: [COLOR black] & decker [PRODUCT blender] [PRICE under 100]
Thus such a retrieval usecase may require a desired precision-recall balance, since precision errors (i.e., incorrectly annotated spans) could lead to catastrophic failures downstream perhaps incor- rectly filtering only "black" colored blenders in the above example. Unfortunately, current models do not allow to optimize for incorporate such complex metrics or trade-offs.
Such errors are in fact commonplace as seen Table D and empirically in our results. The issue further exacerbated when we go beyond the top-1 prediction drop-off in the quality of predictions outside the top-1.
in Figure D with drastic
In addition to identifying this shortcoming of NLL-based models, we make the following contribu- tions in this paper:
We propose GROOT simple yet effective framework for training sequence labeling models to optimize (black-box) reward metrics. GROOT takes a generative sequential labeling model learn the reward metries associated with the output space of sequences. We propose CML - new Corrective Margin Loss function - that contrasts candidates with differing reward metrics, to enable the model to better understand the reward space.
We show that simply relying on candidates sampled from the decoder output distribution as proposed prior work for machine translation (Shu al. 2021) does not work (and often worsens reward scores significantly).
To enable principled and targeted exploration of the output reward space, we introduce correction function-based based approach correcting errors in predictions to explore the space. Extensive experiments over 4 public datasets demonstrate that GROOT significantly im-
proves rewards over competitive baselines.
Furthermore, we demonstrate that GROOT learns a better overall decoder distribution with significant gains in correlation with reward scores.
With extensive experiments aimed at understanding the value of each component of GROOT, we believe work can help significantly inffuence practical applications of sequence labeling models.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1 GENERATIVE SEQUENTIAL LABELING
Sequential labeling typically involves tagging spans of the input with their corresponding label. The emergence of Seq2Seq learning (Sutskever et al. 2014) has seen common NLP tasks like POS tagging, chunking. NER and segmentation cast Seq2Seq text generation tasks (Vinyals et al. 2015 Fitzlerald 2020 Raffel et al. 2020). More formally, given an input text x, we can generate the sequential text label as arg (1 |2:), where the output sequence probability (1 (x:)
'/' computed by Seq2Seq model, denoted by the model parameters 0. Arbitrary text formats acceptable for x: and as long as we can interpret the output. For example, [Raman et al. (2022)
investigated various formats, and showed that such generative approach using pre-trained Seq2Seq models outperforms word-level classification models (c.g., mBERT (Devlin et al. 2018)).
To train the Seq2Seq models, we typically use a dluman-annotatedd ground-truth dataset: set of training examples, Dtrain where each example is a pair of an input text æ and its ground-truth annotation y e (z, y²). The most common supervised learning approach (Williams & Zipser 1989) would then optimize the NLL loss:
NLLo - Po
While the NLL loss-based training offers many advantages and promising results (Athiwaratkun al. 2020 Raffel et al., 2020; Yan et 202T), also comes with significant drawbacks. Con- sider the top-5 predictions of an NLL loss based model (Table [ on the SNIPS dataset (Coucke jet al. 2018). Aside from the clear overfitting issue (commonly observed in these models (Bishop & Nasrabadi 2006)), we also find that the top-5 predictions have numerous different errors. While any ML model is bound to make prediction errors in practice, we should recognize that different errors affect downstream applications differently. For example, some applications may prefer recall emrors / false negatives (i.e., unannotated spans) over precision errors false positives (i.e., incorrectly an- notated spans) or vice-versa. However the NLL loss loss-based training does not provide for easy way to incorporate such desiderata or tradeoffs.
2.2 REWARD OPTIMIZATION AND CONTRASTIVE Loss
Recent attempts have been made to more closely correlate the output distribution of text generation models (esp. for machine translation) with desired reward metric (Ranzato et al. 2016 Shen et al.) 2016 Edunov et al. 2018 Choshen et al. 2020). Most notably, Shu et al. (2021) investigated an effi- cient and effective method of reward optimization for large neural machine translation models. They have proposed a contrastive loss to compare the best and worst prediction candidates among top-k predictions, while previous related studies investigated different combinations of the candidates to define contrastive losses. However, the underlying assumption is that sampling from the decoder distribution allows for effective exploration (and learning) of the metric space. Unfortunately this assumption does hold true in most scenarios as the output distribution of training examples is highly skewed and often contains low-quality predictions outside the top-1 (see example in Table Hence we need alternative ways to achieve meaningful exploration of the metric space.
3 PROPOSED FRAMEWORK: GROOT
As alluded to above, solely optimizing the NI.L loss may not provide us sufficient exploration of the output metric space. Given a predefined (blackbox) reward function, we want the Seq2Seq model to learn output distribution that more closely correlates with and maximizes reward values.
3.1 OVERVIEW
To tackle this issue, we propose GROOT (visualized in Figure p. GROOT works by iteratively training and improving the reward of decoder outputs. More specifically, starting from the NLL- based initial model ((1), in each iteration 2 < T we
3

---

Under review as a conference paper at ICLR 2023
training set Lę by correcting (top-k) predictions (on subset of the training set) of the previous model ((t-1), As we show empirically, leveraging this cor- rection function is key to efficiently explore the output space and significantly outperforms exploration based solely
(Section 3.2
Create
current predictions (Shu et al., 2021).
(Section 3.3) Train new improved model g(t) using Lu by applying our proposed Correc- live Margin Loss that contrasts and orders different candidates based on their rewards.
Henceforth we will denote the predefined (blackbox) reward R, where R(y: v²) denotes the reward for an example prediction (optionally computed using the gold label y²).
3.2 DATA GENERATION
For the 1-th step of GROOT, we create a new training set L, from a (random) subset of the training data C. In particular, for each example (z, v²) C) we do the following:
Get top-k predictions Using the previous timestep's model ((t-1), perform inference L using a beam search to generate the top-k output candidates for each example e.
Select correctable candidate Using the input reward function R y²) we compute a reward value for each of the top-k candidates y. Based on the rewards we select ÿ. using fixed criterion (e.g. non-perfect candidate with highest reward in top-k). To maximize learning of the output- reward relation, we only select candidates that can be "corrected" - as described below.
Based on prior work (Shu et al., 2021p, one may assume that simply comparing the top-k allows for sufficient learning of the output space. However, using the top-k predictions alone is unreliable and does not allow for sufficient exploration of the output space and at times may even worsen results (Table 3). Thus to help us better explore the output space in a guided manner, we introduce a key additional step:
show empirically (in Section 6). simply
CCorrect erroneous candidate To effectively learn what predictions lead to better reward values, we introduce correction functions to rectify erroneous predictions. More specifically, a correction function Cyy. y*) is designed return new prediction y+, such that r+ R(y+. y²) is expected be greater than (or equal to) r = R(y, y²).
In other words, given an imperfect prediction y and optionally the gold label y* the correction function aims to find prediction that improves upon or fixes errors in y. The goal of the correction function is not to find the best possible prediction y- (which would just be the gold label y²) but rather to expose the model to novel (improved) prediction candidates potentially quite different from the existing model's output distribution. Since correction functions will be used to improve predictions and explore the output space, they could be either algorithmic or themselves model- based. Note that in some applications with complex reward function, we may not be able to easily devise a single correction function that always finds improved prediction. However in such cases, we can instead combine multiple correction functions. We require that the functions together can produce Y+ (s.t., r+ r) with high probability. If we cannot find improved y+ from any correction functions, then we can choose to drop the example instead.
Once we have this improved y+. each example is expanded to form new tuple (Ir, y" r", D. F, Dt, F4), where r" = R(y", y²), RRG, y²), and F4 R(T+, y*).
3.3 TRAINING WITH CORRECTIVE MARGIN Loss
To help the model best learn the output space of rewards we train the model by contrasting pairs of y and 7.. In particular, for each ē E L. we use the following corrective margin loss:
CM U. F, J+. F+)= max(0, m log (z) + p()l=))
(2)
where m is margin computed with hyperparameter a: m = a= f).
The form of this loss function is motivated by previous work (Edunov et al. 2018 Shu al. 202 where they compare candidates among the model-generand ones. By contrast, we use CML to teach the model learn prefer the corrected ÿ+ over the original ÿ.
4

---

Under review as a conference paper at ICLR 2023
An astute reader may notice that the above formulation of CML does not directly use the gold label y" While this raises interesting possibilities of unsupervised variants, for the purpose of brevity largely leave this to future work. Instead
incorporate y" by also teaching the model to explicitly prefer y" over ÿ+ since we would still like the model to ideally produce y" as the best candidate. Thus we add another CML term define the following corrective ranking loss:
CRL.ole') (2,, 0. f, D+. f4) + CMLo U+. E+, V² r").
(3)
To further prioritize the gold label we can still include NLL in our training loss as:
(e) + (1.0 A) CRLg(e'),
where À E [0, 1.0] is a hyperparameter to control the regularization effect by NLLa(e). In practice we found that a small value of À helped stabilize learning and reduce variance (See Figure
4 REWARD AND CORRECTION FUNCTIONS
4.1 REWARD FUNCTION
Practical applications for generative model may span variety of different complex reward fune- tions which we would like to optimize for directly. Unfortunately public benchmarks do not come up with any such provided complex reward function. Thus we look create realistic reward func- tion - for the running example of a sequence tagger entity recognizer. Our reward function should be realistic enough reflect the complexities of real-world metries (and thus more complex than what traditional techniques can optimize) while still being understandable for us and readers.
As motivated in the introduction, practical application one may weight precision and recall differ- ently. With 3 controlling the importance of precision vs. recall, consider reward of the form:
Rly, y²) = 3 x precision(y: y²) + recall(y, y y²),
(5)
To further challenge the model, let look into the definition of precision for sequence labeling and highlight a practical issue that needs addressing. Consider this synthetic example:
ground truth: y" [A X] Y Z W U] V] prediction (1): = [D XI Y JEZW [F VJ
(a completely imprecise prediction) (an empty prediction)
prediction (2): y/2
Y SEWUT U
The typical definition of precision would lead to precision(11, y²) procision y²) = 0, since the number of true positives is However practice these two predictions would have very different behaviors. When used to influence search engine behavior the first prediction would lead to far worse results (due to false positives) unlike the second prediction. This boils down to the precision metric not differentiating between false positives and empty predictions which is valid choice in sequential tagging problems.
To more closely reflect this distinction, let modify our definition of the span-level precision to be:
precision' y²) = TP -
(6)
TP
where TP and FP stand for true positive and false positive, respectively, and (2 0) is a hyperpa- rameter explicitly penalize false positives. When c 0 we recover the classical precision metric. This small change now allows us to easily distinguish the above two predictions and yz:
0- x
precision' (y11 v²)
- precision' (y2, y²) 0.
0 043
As an illustrative reward, we set (c. B) = (2, 4) for our reward function in the rest of the paper leading to reward scores between -8 (completely wrong) to +5 (perfect). We would like to reinforce that the above is just one example of complex but realistic reward function. Experiments with other reward functions yielded similar- if not stronger-
findings (see Appendix G).
5

---

Under review as a conference paper at ICLR 2023
Domain (Task)
Train
Valid
Test
ATIS SNIPS
Travel (SH)
4,478
500
893
Algorithm Correction fn
Assistant (SF)
13,084
700
700
1: function Cly. )²)
MIT-R
Dining (SP)
6,845
789
1,516
2: yR REPLACE(y, v²)
en hi
13,667
2,235
4,386
3: yn DROP(y,, 8²)
МТОP
Assistant (SP)
11,330
2,012
2,789
4: if yp = y then
11,814
1,577
3,193
5:
// = DROP (y. v²)
CoNLLO3 News (NER)
14,987
3,466
3,684
6:
else
CoNLLOO
News (CHU)
8,936
1,844
2,012
7:
3/+ YD
8:
end
9: return y+
Table 2: Characteristics of datasets used in our experiments (SP=Semantic Parsing, SF Slot Filling, NER Named- Entity Recognition, CHU Chunking).
10:
end function
4.2
CORRECTION FUNCTION
The goal of the correction function is help explore the output space by yielding improved predic- tions y+ Cly, y²) such that R(y+. y²) 2 R(y. y²) w.h.p. While the suitability of a correction function may depend on the desired reward function, this section we define three correction op- erations that we found widely effective across a slew of sequential tagging reward functions: Drop, Replace and Annotate.
Is designed to fix precision errors by dropping tagged spans in the prediction y that do not match with the ground truth y" In addition to improving precision, this also often enables exploring new candidates since NLL training does not explicitly teach the model drop uncertain tags. For the example y11 in Section 4.1 this would result all predicted spans being dropped.
Drop:
Replace: Works by replacing incorrect tags in with the correct ground-truth tags. For yi this would result fixing "[D XJ" and "[F VJ" to produce "[A XJ Y [E Z W [c VJ".
Annotate: Improves recall by annotating untagged spans with the correct ground-truth tags.
We can also combine these operations. For example combining all three can lead to perfectly fixing any predictions. Given our example reward's focus on precision, we combine the first two (which improve precision) to give us our proposed correction function (see Algorithm [T). This function not only fixes most possible errors in predictions, but also helps efficiently explore the output space.
EXPERIMENTAL SETTINGS
5.1 DATASETS
For comprehensive evaluation, used six different datasets spanning different domains and tasks as detailed in Table 2 ATIS (Price 1990) and SNIPS (Coucke et al., 2018) are two slot-filling tasks covering the travel (e.g.. booking flight ticket) and virtual assistant domains, respectively. MIT-restaurant (MIT-R I and MTOP et al. 2021) are semantic parsing datasets from the dining (c.g., ordering dish) and voice assistant domains, respectively. The CoNLL 2003 dataset (Tjong Kim Sang & De Meulder 2003) is from news domain (i.e., Reuters) for the NER task. Lastly, the CoNLL 2000 dataset (Tjong Kim Sang Buchholz, 2000) is also from news domain (i.e., WSJ) for the syntactic chunking task. For space reasons, we report results for the latter two sets in the Appendix [ Additionally, to verify our findings hold across languages we also evaluated on the French and Hindi MTOP datasets with results are reported in Appendix A
5.2 METHODS COMPARED
Our primary comparison against the current standard an model NLLg trained solely using the NLL loss. For the most fair and competitive baseline, we train the NLL model using the entire training set. This also forms the initialization point for all iterative models. Note that this means our iterative models do not benefit from any data specifically kept for iterative training.
While previous work (Raaan al. 2022) has shown that generative NLL--based approach out- performs token-classification methods, for the sake of completeness we evaluate three additional
https://groups. /

---

Under review as a conference paper at ICLR 2023
token classification baselines and report results in Appendix H We also compare against the best existing (and most relevant) approach for reward optimization denoted as Shu++ (Shu et al. 2021). Shu++ relies solely the decoder top-k for exploring the reward space, and defines contrastive loss between best- and worst-reward candidates; the comparison with Shu++ thus helps understand benefits of our correction function for better exploration in the reward optimization framework.
In addition to GROOT, we also evaluate a variant GROOT (NG: No Gold) which does not use the gold label in the corrective loss i.e. using CMLs(a, D: F, D+ F4) instead of CRL (in Equation (4)
5.3
MODEL
As an instance of a competitive gequence-to-sequnng model, we use pre-trained mT5 (Xue et al. 2021] and build the T5X code base (Roberts et al. (022). We a Base-sized model maximize experimentation but verify results hold larger models (XXL-sized) in the Appendix.
We use the Adafactor optimizer (Shazeer & Stern 2018) train all models, along with Z-loss regularization (de Brébisson & Vincent (016)). A constant learning rate of 0.001 is used for the NLL-only training stage, and 0.0001 otherwise.
NLL training is run for upto 2500 steps (evaluating checkpoints after every 100 steps), while iterative reward optimization runs for upto 200 steps (eval every 10 steps). For both NLL and iterative reward optimization select the best checkpoint ()()) per the reward metric of the top-1 prediction the validation set Dvval, Test set metrics are reported from the best checkpoint from the last iteration.
5.4 DEFAULT PARAMETER SETTINGS OF PROPOSED FRAMEWORK
Below are the default (and recommended) settings of our proposed framework. The effect these different parameters are analyzed in Section 6.1 onwards and the Appendix.
Text format: We use the "sentinel+tag (SI)" format to represent the input and output texts as recent studies (FitzGerald) 2020) Raman al. 2022) have found that to be most effective and ideal to minimize hallucinations in text generation.
Data preparation: We set the number of reward optimization iterations T (Section 3.2) and create L, from the entire training set at each iteration.
Beam size: We set k evaluation.
beam size for the top-k prediction stage in both training and
Candidate selection: To select the candidate to improve ÿ from the top-k, we default to selecting the highest reward candidate.
Candidate correction:
We use the correction function described in Section
Loss function: We set (a, (0.5, 0.1) ) for our loss function in Section 3.3
Note: avoid any "human overfitting", all development of the methodology (incl. refinements and ablations) was conducted using only the validation sets. Test sets were only used to report final results for the paper (in a completely human-blind fashion).
5.5 EVALUATION METRICS
We evaluate each example by the following four metrics, and report macro averaged scores. All the reported scores are based on mean and standard deviation across five different runs. Results for additional metrics are reported in the Appendix.
Top-1 reward: Equation ( for the top-1 candidate by the beam search. (Maximum value: 5.0)
Average reward: Reward averaged across the top-k candidates. (Maximum value: 5.0)
Max reward: The highest reward value among the top-k candidates. (Maximum value: 5.0)
Rank correlation: Spearman's 's rank correlation coefficient to measure if the model has correctly predicted the top-k candidates in order of their reward values. (Maximum value: 1.0)
RESULTS AND DISCUSSIONS
As seen from the test results Tables and g both GROOT variants significantly outperform the NL.. and Shu++ baselines across (nearly) all metrics all six datasets. GROOT not only improves

---

Under review as a conference paper at ICLR 2023
ATIS
SNIPS
IITOP
KITEE
XFa++
NFL.
NEL.
KARCOT
STL
CIRUITY
Bipd moad
117.
LXIT
1.007
1166
19P8
12311
1315
1.315
L/19
7.60
1213
XTI
0.58
2:39
1297
2159"
-1366
9.330
6.38
11457
-388'
133X
10315
1.3766
1.380
0014
19316
1.0
0331
1011
8.773
1.000
8335
1..10
1-0.66
1.000
Max Masionad
4,397
4355
4.50
4759
4.0
4250
402
4917
4
4833
4380
2.99F
4837
4687
5,500
Bank corrilaton
0,59
1166
18.035
6:9
10182
675
10073
-0000
1922
1355
101000
43495
10.00
409966
10003
40/07
355
101003
R651
-0V09
4.88
0.55
Table 3: Test set results on the four datasets. Standard deviation values are over runs. Bolded
values and the symbol indicate 99% significance (via z-test) vs. NLL and Shu++ respectively.
Top-1 reward (ATIS)
Top-1 reward (SNIPS)
Top-1 reward (MTOP)
Top-1 reward (MIT-R)
3.4
Reward
Reward
4.4
Reword
Reward
3.2
4.6
GROOT
GROOT
4.2
3.8
3.0
4.4
NLL Easetine
2.8
Shuee
2
4
5
6
2
3
4
7
8
1
2
3
4
6
7
8
1
2
5
6
7
lterations
Iterations
Iterations
Iterations
Avcrage reward (ATIS)
Average reward (SNIPS)
Average reward (MTOP)
Average reward (MIT-R)
Reward
Reward
Reward
Reward
o
1 2 5 6
3
9
2
8
1
6
Iterations
Iterations
Iterations
Iterations
Figure 3: Performance of different loss functions on the top-1 and average reward metries. For all figures (here and below) error bars denote standard deviations across runs.
the reward of the top-1 prediction consistently, but also significantly improves the quality of the top-k predictions evidenced by the large increases in the Average Reward scores. GROOT meth- ods also score highest when looking at the highest reward scores among the top-k. The improved rank correlations provide the final confirmation that the GROOT models do indeed gain better understanding of the output space in regards to the provided (blackbox) reward function.
These results also demonstrate the importance of efficiently exploring the output space. As in Table 3 the Shu++ model is often no better and in many cases significantly worse - than the NLL baseline. This clearly illustrates that simply relying on the top-k may not suffice. More specifically the lower quality of the top-k of the NL.L baseline provides very little room for hill- climbing towards better reward scores, which in turn leads the poor scores observed. In contrast the GROOT methods outperform Shu++ on every metric, proving that the correction -ouniiitoonnooonnnnnooooo based exploration coupled with the contrastive loss functions leads to a significantly better model.
6.1 EFFECT OF LOSS FUNCTIONS
Figure 3 shows how the top-1 and average reward values change over the course of the iterative training process. We can clearly see that GROOT methods smoothly improve the reward scores and improve the decoder output distribution. Between the GROOT variants, we find that GROOT NG tends more aggressively optimize the intended reward (since it not anchored to also optimize the gold label probability). This does come up with drawback as other metrics different than the provided reward may degrade (as shown in Table 7 in Appendix), due to the focus on the provided reward. Regardless the strong performance of GROOTNG hints at the promise of such techniques even being used in low supervision unsupervised settings. However we leave this to be explored in future work and instead focus on the vanilla GROOT model in the rest of the analyses.
6.2 EFFECT OF CORRECTION FUNCTION
Figure shows the effect of the correction function used. While "Drop=replacemenn" corresponds to our default correction function (Algorithm D. "Only drop" only uses the drop operation. We can see that the drop operation by itself is quite effective, due to the increased exploration it offers relative to NLL-only training.
Coupled with the earlier results, we clearly see how correction functions can help break the limits of exploration in Seq2Seq models. While alternatives such decoder distribution sampling (Ran-
izato et al., 2016; Ackley et al. 1985) Fieler & Goldberg, 2017) do exist, the highly skewed output distributions of the NLL. model (as seen in Table often constrain exploration and need significant
8

---

Under review as a conference paper at ICLR 2023
Top-1 reward (ATIS)
Top-1 reward (SNIPS)
Top-1 reward (MTOP)
1bp-1 reward (MIT-R)
4.0
Reward
4.8
Reward
4.4
Reward
Reward
3,0
3.8
4.7
4.2
2.8
2
6
lterations
Iterations
Iterations
Iterations
Figure 4: Comparison between different correction functions.
Top-1 reward (ATIS)
Top-1 reward (SNIPS)
Top-1 reward (MTOP)
Top-1 reward (MIT-R)
3.25
Reward
4.8
Reward
4.4
Reward
3.9
3.00
3,8
2.75
4.2
4.7
3.7
2.50
6
3
4
2
8
7
lterations
Iterations
Iterations
Iterations
Figure Comparison between different candidate selection strategies.
Top-1 reward (ATIS)
reward (SNIPS)
reward (MTOP)
Top-1 reward (MIT-R)
3.2
Reward
Reward
1.4
Reward
Reward
3.0
4.6
4.2
3.6
2.8
4.4
1 2
2
4
7
lterations
Iterations
Iterations
Iterations
Figure 6: Comparison between different values of in Equation
tuning of temperature parameters to smooth out the distribution. Thus we conclude that correction functions provide far simpler and more effective way of exploring the output space.
6.3 EFFECT OF CANDIDATE SELECTION STRATEGY
Figure 5 shows how different candidate selection strategies affect the results. While our default choice corresponding to "Best" is often the most consistent, other strategies such as "Worst" (lowest reward candidate) or "Random" are also effective at helping the model learn reward contours. Thus we believe that our framework not sensitive to the selection strategies, and could even incorporate multiple candidates via different ranking objectives (Jagerman et al. 2022).
6.4 EFFECT OF À
Figure shows how the NLL loss contribution (controlled by in Equation affects results. In general we find that small contribution of NLL as seen in the À = 0.1 curves generally help to stabilize learning, while larger values still do benefit from iterative improvements of reward (albeit slightly less). λ = 0.0 though was fairly unstable training, and thus we recommend setting À > 0.
7 CONCLUSION
We have presented a simple yet effective framework, GROOT, and empirically verified the effec- tiveness on sequential labeling tasks across multiple tasks and languages. Via the use of a correction function based exploration and contrastive loss, GROOT is able to directly and effectively optimize a Seq2Seq model for the given blackbox reward function. Furthermore, unlike standard maximum likelihood training, the output distribution of GROOT also significantly improves and is well corre- lated with the desired reward metric.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 91):1477-16, 1985. ISSN 0364-0213. doi: https:
URL https: www. sciencedirect. com/
Ben Athiwaratkun, Cicero Nogueira dos Santos, Jason Krone, and Bing Xiang. Augmented natural language for generative sequence labeling. arXiv preprint Ntt::2009.132777 2020.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the Weaknesses of Reinforce- ment Learning for Neural Machine Translation. In International Conference on Learning Repre- sentations, 2020.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, al. Snips voice plat- form: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv: 1805. 10190, 2018.
Alexandre de Brébisson and Pascal Vincent. The Z-loss: a shift and scale invariant classification loss belonging the Spherical Family. arXiv preprint arXiv: 1604.0660.00800 08859, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1xXiv:811000480, 2018.
Lee Dice. Measures of the amount of ecologic association between species. Ecology, 26(3): 297-302, 1945.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Classical Structured Prediction Losses for Sequence to Sequence Learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume (Long Papers), pp. 355-364, 2018. URL https:
7/aclantho www.aathoooo....oooo 001/118-1033
Jessica Ficler and Yoav Goldberg. Controlling Linguistic Style Aspects in Neural Language Generation. In Proceedings of the Workshop Stylistic Variation, pp. 94-104, 2017. doi: 10.18653/v 1/W55///1/11-4912 URL httpt: http://aaaattoooooorooooo/
Jack FitzGerald. STIL Simultancous Slot Filling, Translation, Intent Classification, and Lan- guage Identification: Initial Results using mBART on MultiATIS+++. In Proceedings of the Ist Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 576-581, 2020. URL
www.aanttwwoooooooooo/oeo//e/ aacl ma ma.nn.7
Paul Jaccard. The distribution of the flora in the alpine zone. New phytologist, 11(2):37-50, 1912.
Rolf Jagerman, Xuanhui Wang, Honglei Zhuang, Zhen Qin, Mike Bendersky, and Mare Najork. Rax: Composable Learning-to-RR -o-iin using JAX. In KDD 2022, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequencc pre- training for natural language generation, translation, and comprehension. arXiv preprint 1111v:1910.13461, 2019.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. Mtop: A comprehensive multilingual task lask-oriented semantic parsing benchmark, 2021.
Patti Price. Evaluation of spoken language systems: The atis domain. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990, 1990.
10

---

Under review as a conference paper at ICLR 2023
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text- Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. hRLhttp:
7/jml r. httpp/ppppsss///1/
html
Karthik Raman, Iftekhar Naim, Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi, and Kr- ishna Srinivasan. Transforming Sequence Tagging Into A Seq2Seq Task. arXiv preprini arXiv:2203.08337,, 2022.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence Level Train- ing with Recurrent Neural Networks. In International Conference on Learning Representations, 2016.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bu- lian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy. Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepart Noah Fiedel, Mark Omernick, Brennan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling Up Models and Data with 5xx and seqio. arXiv preprint arXiv:2203.17189 2022. URL httts:://axxiv org/ abs////////// /2000/2220...2 17189
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory
Cost. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4596-4604. PMIR, 10-15 Jul 2018.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum Risk Training for Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1683-1692, 2016. URL https: //pss///cc ac antho ogy. wrw.yoo-gp11611110
Raphael Shu, Kang Min Yoo, and Jung-Woo Ha. Reward Optimization for Neural Machine Trans- lation with Learned Metrics. arXiv preprint arXiv:2 104.02210 2021.
Thorvald A Sorensen. A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application analyses of the vegetation on danish com- mons. Biol. Skar. 5:1-34, 1948.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to Sequence Learning with Neu- ral Networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL https: htttpp::/pprcceedi.. newriis
hctpppaappe//211//ieeee a 2Llac33ce2277225588eec.....e3 ott.pp/eee........o.o......ooorr
pdf
Erik F. Tjong Kim Sang and Sabine Buchholz. Introduction to the CoNI.L-2000 Shared Task Chunk- ing. In Fourth Conference on Computational Natural Language Learning and the Second Learn- ing Language in Logic Workshop. 2000.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CONLL-2003 Shared Task: prgguage-Indeeendenn Named Entity Recognition. In Proceedings of the Seventh Conference Natural Language Learning at HLT-NAACI 2003, pp. 142-147, 2003. URL https://
ac aclanthology wttg//3-----0000 0419
Amos Tversky. Features of similarity. Psychological review, 84(4):327, 1977.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: htttt::://ppoceeeiiigg / hew.eeip..iii..iiiiie.ii.... http:///ww...aaooaoooo////////// 3 T
pdf

---

Under review as a conference paper at ICLR 2023
Oriol Vinyals, E ukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. Grammar as a Foreign Language. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gameett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015. URL
neurii pc/ppaaprr/201///11el 27733306622225533344442244466282
Ronald Williams and David Zipser. A Learning Algorithm for Continually Running Fully Recur- rent Neural Networks. Neural Computation, 1(2):270-200, 1989. doi: 10.1162/neco.191 1989.1..2 270.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A Massively Multilingual Pre-trained Eext-to-Txx Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, 2021. doi: 10.18653/
v1/2021.naacl-e maiacl- URL.https. /etetps://aclaaee http:htttp:://ccattotoolo www.clannoollyy.ooooo org 2021 naacl -ma n.. 41
000
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. A unified gen- erative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 5808 5822, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v12001 lcl-long.441. URL httpe://eelaallooooo..o////0....
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A RESULTS IN NON-ENGLISH LANGUAGES
Table@showe results on the MTOP dataset in Hindi and French. The effectiveness of our method is consistently observed in Hindi and French as well.
MTOPP (hi)
MTOP (ff)
NEL Easeline
GROUT
NLL baseline
GROOT
Top-1
Average reward
-15524 2: 0.068
00240 =: 0.125
11394 = 0.031
0C377 # 0.047
Mas reward
4.444 + 0.029
4,756 0.19
4,551 + 0.17
4.796 + 0.017
comolasinn
0.768 20012
0003
0.000 E 0.002
Table 4: Test set results on the Hindi and French MTOP dataset.
B RESULTS WITH MT5 XXL
We mainly used mT5 BASE to maximize experimentation and analyze the effectiveness different aspects of our proposed framework. While this is already large transformer model (Vaswani et al.) 2017]. a natural question one may ask is whether the observed gains carry over to even larger models.
To answer this question, we verify the effectiveness using mT5 XXL With 13B+ parameters, the XXL. variant significantly larger than mT5 BASE as described in Xue et al. (202T). For ex- ample, mT5 BASE consists of 12 transformer layers with 768-dimensional embeddings and 2048- dimensional Multi-Layer Perceptrons (MLPs), while mT5 XXL consists 24 layers with 4096- dimensional embeddings and 10240-dimensional MLPs.
Table 5) shows the results. When compared with the results Table 3 it is unsurprising to see significant gains for all methods using mT5 XXL over mT5 BASE. However still observe the same trends with GROOT improving reward- based metrics. It is interesting to note that while the average reward scores improve significantly by simply using the larger model (except ATIS), GROOT still helps maintain a sizeable gap. Thus we believe our reward optimization framework will help even when larger and larger models are invented in the future.
ATIS
SNIPS
MTOP (cn)
MIT-R
NIL
GROOT
NEL
GROOT
NEL
GROOT
NEL
GROOT
Top-
4,300
4775
AGerage
22366 0.5
0.
Mix rewaadd
2,616 2: 0.022
4788 = 0.018
4857 - 0.013
43005 0L033
47764 0.032
4.863 #000
0.099
4..29' HE 0L042
Rank correlation
0.444 a: 0.028
0.698 a: 0.105
0.629 = 0.011
0.849 = 0.028
0.634 # 0,006
0.785 0.04
0.621
0,007
0.655 +: 0,000
Table 5: Test set results with mT5 XXL.
C EXAMPLE PREDICTIONS
Table 6 shows how the prediction examples in Table not confident enough about the tag of "moorpark." the top- prediction leaves it untagged.
! (a) are modified by GROOT. If the model is
NLL
Model predictions for an example from the SNIPS validation set
0.154
book (party_alee zeunumber seven] [apatial poaataalreelatinn neighboring moorpark
0.712
book (party at numbar seven] [apatiial ralat neighhoring! Igeographic poi moorpark]
1.303
book [party-sit ze-numbe sevea] in [spatial Sspatial-reeatte ion neighboring) [poi moorpark]
1.383
book [party_ai ze_nembor seven] [spatiall_re aatlaallraaallaeaae neighhoring/ [cicy moorpark] (*)
2.004
book [party size number sevea] in [apatial relation neighboring] [object name moorpark]
Table Top-5 predictions refined by GROOT. This can be contrasted with Table (a).
base.gin
https: htap::///githin http:///hh.hiiioooooooooooaoa/aa h
gin
13

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL EVALUATION METRICS
In addition to the evaluation metrics introduced in Section 5.5 we also report the results with the following metrics for reference:
-Top- precision': Equation 6 for the top-1 candidate. (Maximum value: 1.0)
Top-1 precision: The standard precision metric for the top-1 candidate. (Maximum value: 1.0)
-Top-1 recall: The standard recall metric for the top-1 candidate. (Maximum value: 1.0)
Top-1 EM: binary score check if the top-1 candidate perfectly matches the ground-truth annotation. (Maximum value: 1.0)
Table 7 shows the results, corresponding to those in Table 3
As expected, the precision' scores significantly improved, while slightly degrading recall (and hence EM).
SNIPS
STTP oz)
MITS
RLL
GROOT
G8O0T cm
Skar
MLE
CROOT
EROOT kg
St.t3
NELL
CROOT
Skg
NLL
GROOT
UROOT
Skace
-000
5:0006
2:000
1:900
2.00432
2-9.05
0.00
+00952
60015
2-0:00
1.0.62
60051
=6/2
2-001
=0400
01.27
03066
0359
Tep mail
Tep EM
0.8
0006
8000
-.
336
038
5.00
9990
8909
9984
2931
139
3357
-
-
20011
8.67
6.82
1035
10352
333
0000
3367
2.002
380
335
29.93
1099
1083
335
682
0005
665
C36
2583
2:00
330
326
=010
2283
Table 7: Additional evaluation metrics for the results in Table 3
E MORE ANALYSIS RESULTS
and II show how the average reward scores are improved the course the iterative reward optimization. The average reward scores smoothly improved in all the cases. We can also see similar trends that are observed in the cases the top-1 reward metric.
Figures
Average reward (ATIS)
Average reward (SNIPS)
Average reward (MTOP)
Average reward (MIT-R)
0.0
0.5
Reward
Reward
Reward
-0.5
0.0
0.0
1-00
0.0
x-0.1 A-05
A-01
2-0.1
A-011
0-
2-05
00.
A 05
2
4 5 6
3
4
6 Iterations
1
2
Iterations
Iterations
Iterations
Figure 7: Comparison between different values of À in Equation for the average reward metric.
Average reward (ATIS)
Average reward (SNIP)
Average reward (MTOP)
Average reward (MIT-R)
Reward w
0.0
0.5
Reward
Reward
Reward
0.55
0.0
Only
Only Stop
Only drop
-0.5
2
Iterations
Iterations
Iterations
Iterations
Figure 8: Comparison between different correction functions for the average reward metric.
Average reward (ATIS)
Average reward (SNIPS)
Average reward (MTOP)
Average reward (MIT-R)
0.5
Reward
w
Reward
Reward
Reward
2
6
2
7
Iterations
lterations
Iterations
Iterations
Figure 9: Comparison between different candidate selection strategies for the average reward metric.
14

---

Under review as a conference paper at ICLR 2023
E.1 DOES A MORE TEST-LIKE DATA SPLIT WORK?
As mentioned in Section 5.2 we use "all" of the training examples for NLL and iterative training stages. However as seen in Table [] NLL training tends to overfit heavily leading to very skewed distributions. Instead more test-like split would use only part of the data for NLL and then new batch for the first stage training and so on. This setup more closely simulates test-time behavior since the model would not have seen any of the examples before. However given the limited training set size such splitting may seem likely to degrade performance. Thus we compared against a variant where the training was "split" into pieces which are eyclically used for each iteration.
As seen in Figure TO despite starting from a much worse initialization, the GROOT model quickly recovers outperforming NLL and nearly catching upto the "all" performance on all datasets. For reference, we also show Figure [I] We believe with a more careful data strategy and/or a larger training set, we could see further gains.
Top-1 reward (ATIS)
Top-1 reward (SNIPS)
Top-1 reward (MTOP)
Top-1 1 reward (MIT-R)
4.0
4.8
4.4
3.0
Reward
Reward
Reward
3.8
Reward
0.7
4.2
GROOT (eplit
GROCT (epiloo
3.6
GROOT Tegil:
2.5
GROOT TepR; ERCOT Tall NLL baseline
4.6
GROOT (al)
GROOT (al)
BROOT [all
NILL base
4.0
KLL baseline
NLL baseline
5
6
1 2
7
3.4
1
2
7
1
6 Iterations
7
lterations
Iterations
Iterations
Figure 10: Comparison between different data preparation strategies.
Average reward (ATIS)
Average reward (SNIPS)
Average Reward (MTOP)
Average reward (MIT-R R)
0.0
Reward
GP001 Spll
Reward
GROOT ispirt
Reward
GROO1 (spli))
Reward
GRO01 Espit)
NLL buscline
GROOT
GROOT lall
bracire
-0.5
GROCT (al))
beacline
NLL besclig
DROOT (all
-1.0
Ieeations
lterations
lterations
Iterations
Figure 11: Comparison between different data preparation strategies for the average reward metric.
F EXPERIMENTS ON DIFFERENT TASKS
This section shows how GROOT works on the NER and chunking tasks, where all the experimental settings are consistent with those on the ATIS, SNIPS, mTOP, and MIT-R datasets. Table & shows the results. We can see that GROOT consistently improves upon the NLL baseline in all the reward- based metrics on the two tasks, while Shu++ does not. Those results show the generalization ability of our proposed framework across different tasks.
CONILO03 (NER)
CONI.000 (Chunking)
NLL
Shu++
GROOT
NLL
Shu++
GROOT
Top-1 reward
0.193
3,670 = 0.070
3,720 0.108
4.469 4 0.057
4,423 0.048
4.500 = 0.035
Average reward
-1.837 0.171
0,581 + 1.057
-0.045 + 0.507
3.357 + 0.051
2.261 + 0.151
4.131 + 0.055
Max reward
0.05
4.352 1 0.121
1.79640.000
4,842 0001
4.754 1 0.034
4.865 - 0.011
Rank correlation"
0.015
0.707 0,0766
0.786
0.543 4 0.000
0.600 0.019
0.642 0.001
Table 8: Test set results on the NER and chunking tasks.
G EXPERIMENTS WITH DIFFERENT REWARD FUNCTIONS
As discussed in Section 4. limited our exposition in the main body of the paper to single reward function for simplicity. However we found similar - if not stronger empirical results with other reward functions described in this section.
15

---

Under review as a conference paper at ICLR 2023
G.1 DEFINITIONS: VARIANTS OF REWARD FUNCTIONS
Our default reward function defined by Equations ( and 66) which we denote as RpR linearly interpolated variant of precision and recall. This reward function inherently favors small number of highly confident spans. The results in Table 7 show such a tradeoff. To explore a different set of rewards with potentially different characteristics we took inspiration from the popular Jaccard similarity (Jaccard 1912) and the Tversky index (Tversky 1977).
G.1.1 MODIFIED JACCARD SIMILARITY
The Jaccard similarity can be defined as:
YYNA JAC(y. v²) YUU- TP +
TP
(8)
where the set of the predicted spans, Y is the set of the ground-truth spans, and FN stands for false negative. Compared with precision' in Equation (6). this already takes recall into account by false negatives.
As another example of more complex reward function, we can extend this with similar penalty coefficient to obtain:
RJAC v²) |
TP -X TP PP-PP+ FN
YUY
G.1.2 MODIFIED TVERSKY INDEX
The Tversky index is generalization of the Jaccard similarity and Sorensen-Diee coeffi cient (Sorensen) 1948 Dice 1945) which allows for fine-grained control of the balance between different error classes:
TP
TVE(y, 8²) = YnY x Y
TP x FP + x FN (10)
where y and w are hyperparameters. We can further extend this in analogous manner above to define the following reward function:
- cx Y
TP - CX
RTVE(V, v²)
TP IP+7xIP#WXE (11)
+
G.2 SETTINGS
We use the MTOP (en) and MIT-R datasets to verify that GROOT effectively works with these reward functions. We simply replace the original reward function with the new reward functions and did not change any other settings including the hyperparameters. The only change we change we make is to re-scale the value of RJAC and RrvE by a factor of (33 + 1.0) in the loss function to align the value ranges of rewards similar to what we had previously.
We report evaluation scores for the associated reward function that was used for training in each setup. For example, we use RJAC for evaluation if RJAC is used for training. We report results (averaged over 5 runs) for [RAA,, e = 1.0) and { (RTVE, C = 1.0, = 0.5, 1.0) We note that we have observed consistent results with other setups.
G.3 RESULTS
Tables and 10 show the results. can see that GROOT consistently improves upon the NLL baseline in all the reward-based metrics, while Shu++ does not. These results empirically verify the robustness of our proposed reward optimization framework.
H TOKEN CLASSIFICATION RESULTS
Previous works (Raman al. 2022 has shown that generative NLL-based approach outperforms other cokss-llasiiiication approaches. However, for the sake of completeness we implemented and evaluated three additional baselines:
16

---

Under review as a conference paper at ICLR 2023
MTOP (en)
MIT-R
NLL
Shu++
GROOT
NLL
Shu++
GROOT
Top-1 reward
0.836 +: 0.012
0.845
0.001
0.837
0.009
0.621 1 0.027
0.618 = 0.014
0.651 1: 0.005
Average reward
0.133 0.003
00.32
0.033
-0005
0.772+0001
0.100
0.062
T.263 6.000
Max reward
0,958
4: 0.00
0.944
0.004
0,963
0.002
0.164
0.860
0.005
0.890 E 0.066
Rank correlation
0.701 + 0.009
0.708 0.002
0.81 0000
0.614
0.646 0.020
0.673 0.004
Table 9: Test results with RJAC the MTOP (en) and MIT-R datasets.
MTOP (cn)
MIT-R
NLL
Shu++
GROOT
NLL
Shu++
GROOT
Top-1 reward
1 0.011
0.816
0.003
0.835
0.007
0.615 0.018
0.605 0.018
1 0.008
Average roward
0.648 + 0.017
0.229 0.027
0.047 0.140 700077
6.093 = 0.034
0.239 F 0.005
Max reward
0.953 + 0.003
0.937
0.003
0.965 0.00
0.887 0.007
0.862 0.009
0.891 + 0.002
Rank correlation
0.700 + 0.002
0.679 = 0.003
0.818 0.000
0.614 00008
0.645 = 0.017
0.666 = 0.007
Table 10: Test set results with RTVE on the MTOP (en) and MIT-R datasets.
This uses mBERT (encoder-only) model (Devlin al., 2018) to predict the per-token labels. We used the Base-sized mBERT model as the most comparable with the Base-sized mT5 backbone used in most of experiments.
mBERT:
mT5: As shown in prior work (Lewis et al. 2019 Raman et al. 2022 generative models (like T5) can also be used to predict per-token labels. Thus we used the same mT5-Base model predict the sequence of BIO token labels. As recommended by |Raman et al. (2022), we use a single Inside label rather than per-class label - to improve performance.
mT5 (+Input): The results from Raman et al. (2022) showed that learning to generate the input tokens along with the BIO token labels, leads to better performance using mT5 models. Thus we evaluated this as an additional baseline.
The results for these additional baselines are provided in Table 11 When compared with the other methods (Table 3 for Rpn. Table 9 for RJAC and Table 10 for RrVE). we find that the GROOT significantly outperforms all three token classification approaches proach. Note that GROOT may potentially be helpful in the token classification setting well. However we leave this for future work to explore, given the general potency of the generative ap- proach.
further validation of our ap-
mBERT
mTS
mlT5 (-Input)
mBERT 4.11
mTT
miT5 (-Inpot)
mBERT 3.31
WTS
miT5 (-Input)
mBERT
WTT
miTS (- (-Inpot)
Apg TEAC Tove
40
4127
417
4T6
110
346
161
1.37
2.37
132
0,500
103
1587
10876
D.8
D374
DN6
066 0.79
035
0371
688
G87
0.879
0.87
0.755
0.79
0,7
0.156
6618
Table 11: Test set results for the top-1 rewards of different token-classiffcation baselines (results averaged over three runs).
17


Reviews:
[{"review_id": "winlJZ9Wdud", "reviewer": "Reviewer_oMZW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Seq2seq based is a popular approach for generative sequence labelling. However, the popoular log likelihood training doesn't always model well the loss each practical problem speicfies. \n\nIn this paper, the authors first pose such a setting, and then proposes a framework (GROOT) that allows to incoporate those information in generative sequence modelling frameworks. Authors propose two novel techniques to solve this issue: 1) correction function which incrementally improves over the current predicion model 2) Margin loss that can be applied to leverage current predicitons, gold annotations as well as the candidates generated by correction function. Through empirical studies, the authors show that their method improves over baseline as well as recently proposed methods.", "summary_of_the_review": "For generative sequence labelling, authors propose to model reward to better guide seq2seq learning. Authors propose two novel techniques to solve this issue: 1) correction function which incrementally improves over the current predicion model 2) Margin loss that can be applied to leverage current predicitons, gold annotations as well as the candidates generated by correction function. Through empirical studies, the authors show that their method improves over baseline as well as recently proposed methods.\n\nMy main concern is that I can't find support that the algorithms authors propose generalize to more scenarios than the ones posed in 4.1. ", "strength_and_weaknesses": "Strength:\n- The paper exposes a clear motivation and proposes novel correction function to better explore the space.\n- The paper shows strong empirical results\n- The paper has done quite thorough ablation studies showing the effectivenss of their methods as well being stable and not very senstible to various settings\n\nWeakness:\n- The paper exposes results in one particular reward function setting (section 4.1) which shows convincing results. However, to know that the proposed algorithm is effective, one should at least know the performance of other settings; it is mentioned in the paper that other settings yielded similar empirical findings, however I can not find those findings including appendix. Without this, I can't know if the paper has proposed an effective algorithm or not.\n- (Question) I don't' understand why the paper proposes to incorporate these methods only once the model is fined tuned. The method is general and can be applied as a fine tuning method. Not sure why there is no experiment as well as no mentioning on this.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "USxrerK1eX1", "reviewer": "Reviewer_cdc3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new framework for training sequence labeling models to optimize reward metrics, and a CML loss to help the model better understand the reward space. The experimental results show superiority compared with recent baselines. ", "summary_of_the_review": "I'd like to assign 5 (marginally below the acceptance threshold). I hope the authors could make the clarification clearer and do more analysis on the influences of different modules/hyperparameters.", "strength_and_weaknesses": "### Pros\n1. The proposed GROOT is simple, effective, and easy to implement.\n2. The whole framework is clear.\n3. The experimental results show great improvement in reward function and precision.\n\n### Cons\n1. The relationship and difference with *Shu++* should be discussed more clearly.\n2. Only one machine translation baseline *Shu++* was compared. It will be more convincing if it is compared with more baselines.\n3. Sufficient experiments and figures are provided, but there are no consistent performance gains across different datasets. \n4. There should be more descriptions of the datasets.\n4. Some statements are difficult to read and may contain some grammatical errors, e.g, *One may assume that just the above may be sufﬁcient for models to be learn the output space sufﬁciently as done in prior work (Shu et al., 2021)* on page 4, and *Correct erroneous candidate* paragraph on page 4, and *Works by replace incorrect tags in y with the correct ground-truth tags* on page 6. The authors should pay more attention to the writing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0V63doFIQi9", "reviewer": "Reviewer_PagF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes GROOT,  a novel and general reward optimization framework for training sequence labeling models toward reward metrics, via construct corrective candidates as contrastive examples to guide the model learning.", "summary_of_the_review": "Overall, the proposed method is novel and general, and paper is well-written, but the experiments cannot support the general claim because they lack task and domain diversity, and important baselines are missing.", "strength_and_weaknesses": "Strengths:\n* The paper is well-written and easy to understand.\n* The proposed approach is novel and general.\n\nWeaknesses:\n* Although the authors use 4 datasets and include languages more than English, they don't provide details of their specific domains and tasks, and looks all of the 4 datasets are for semantic parsing task in the spoken language domain. As the paper mentions, the sequence labeling tasks range from NER to QA. Therefore, the experiments are too narrow to support their claims generally on top of sequence labeling tasks.\n* As a paper on sequence labeling tasks, the authors don't include the basic solution, token classification, as one of the baselines. Although this paper focuses on the generative manner, the performance of token classification should be still considered as important reference.\n* Moreover, the token classification can also be performed using the same sequence-to-sequence backbone model, just like how BART (https://arxiv.org/pdf/1910.13461.pdf) does in the QA task (SQuAD), which should be also considered as an important baseline.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject