SlowGuess/ACL_2020_OCR huggingface dataset:
tli@cs.utah.edu
Parth Anand Jawale
Parth.Jawale@colorado.edu
Martha Palmer
Martha.Palmer@colorado.edu
Vivek Srikumar
svivek@cs.utah.edu
Recent neural network-driven semantic role labeling (SRL) systems have shown impressive improvements in F1 scores. These improvements are due to expressive input representations, which, at least at the surface, are orthogonal to knowledge-rich constrained decoding mechanisms that helped linear SRL models. Introducing the benefits of structure to inform neural models presents a methodological challenge. In this paper, we present a structured tuning framework to improve models using softened constraints only at training time. Our framework leverages the expressiveness of neural networks and provides supervision with structured loss components. We start with a strong baseline (RoBERTa) to validate the impact of our approach, and show that our framework outperforms the baseline by learning to comply with declarative constraints. Additionally, our experiments with smaller training sizes show that we can achieve consistent improvements under low-resource scenarios.
Semantic Role Labeling (SRL, Palmer et al., 2010) is the task of labeling semantic arguments of predicates in sentences to identify who does what to whom. Such representations can come in handy in tasks involving text understanding, such as coreference resolution (Ponzetto and Strube, 2006) and reading comprehension (e.g., Berant et al., 2014; Zhang et al., 2020). This paper focuses on the question of how knowledge can influence modern semantic role labeling models.
Linguistic knowledge can help SRL models in several ways. For example, syntax can drive feature design (e.g., Punyakanok et al., 2005; Toutanova et al., 2005; Kshirsagar et al., 2015; Johansson and Nugues, 2008, and others), and can also be embedded into neural network architectures (Strubell et al., 2018).
In addition to such influences on input representations, knowledge about the nature of semantic roles can inform structured decoding algorithms used to construct the outputs. The SRL literature is witness to a rich array of techniques for structured inference, including integer linear programs (e.g., Punyakanok et al., 2005, 2008), bespoke inference algorithms (e.g., Tackström et al., 2015), A* decoding (e.g., He et al., 2017), greedy heuristics (e.g., Ouchi et al., 2018), or simple Viterbi decoding to ensure that token tags are BIO-consistent.
By virtue of being constrained by the definition of the task, global inference promises semantically meaningful outputs, and could provide valuable signal when models are being trained. However, beyond Viterbi decoding, it may impose prohibitive computational costs, thus ruling out using inference during training. Indeed, optimal inference may be intractable, and inference-driven training may require ignoring certain constraints that render inference difficult.
While global inference was a mainstay of SRL models until recently, today's end-to-end trained neural architectures have shown remarkable successes without needing decoding. These successes can be attributed to the expressive input and internal representations learned by neural networks. The only structured component used with such models, if at all, involves sequential dependencies between labels that admit efficient decoding.
In this paper, we ask: Can we train neural network models for semantic roles in the presence of general output constraints, without paying the high computational cost of inference? We propose a structured tuning approach that exposes a neural SRL model to differentiable constraints during the finetuning step. To do so, we first write the output space constraints as logic rules. Next, we relax such statements into differentiable forms that serve as regularizers to inform the model at training time.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8402-8412 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Finally, during inference, our structure-tuned models are free to make their own judgments about labels without any inference algorithms beyond a simple linear sequence decoder.
We evaluate our structured tuning on the CoNLL-05 (Carreras and Marquez, 2005) and CoNLL-12 English SRL (Pradhan et al., 2013) shared task datasets, and show that by learning to comply with declarative constraints, trained models can make more consistent and more accurate predictions. We instantiate our framework on top of a strong baseline system based on the RoBERTa (Liu et al., 2019) encoder, which by itself performs on par with previous best SRL models that are not ensembled. We evaluate the impact of three different types of constraints. Our experiments on the CoNLL-05 data show that our constrained models outperform the baseline system by 0.2 F1 on the WSJ section and 1.2 F1 on the Brown test set. Even with the larger and cleaner CoNLL-12 data, our constrained models show improvements without introducing any additional trainable parameters. Finally, we also evaluate the effectiveness of our approach on low training data scenarios, and show that constraints can be more impactful when we do not have large training sets.
1. We present a structured tuning framework for SRL which uses soft constraints to improve models without introducing additional trainable parameters. $^{1}$
2. Our framework outperforms strong baseline systems, and shows especially large improvements in low data regimes.
2 Model & Constraints
In this section, we will introduce our structured tuning framework for semantic role labeling. In §2.1, we will briefly cover the baseline system. To that, we will add three constraints, all treated as combinatorial constraints requiring inference algorithms in past work: Unique Core Roles in §2.3, Exclusively Overlapping Roles in §2.4, and Frame Core Roles in §2.5. For each constraint, we will discuss how to use its softened version during training.
We should point out that the specific constraints chosen serve as a proof-of-concept for the general methodology of tuning with declarative knowledge.
For simplicity, for all our experiments, we use the ground truth predicates and their senses.
2.1 Baseline
We use RoBERTa (Liu et al., 2019) base version to develop our baseline SRL system. The large number of parameters not only allows it to make fast and accurate predictions, but also offers the capacity to learn from the rich output structure, including the constraints from the subsequent sections.
Our base system is a standard BIO tagger, briefly outlined below. Given a sentence  $s$ , the goal is to assign a label of the form B-X, I-X or O for each word  $i$  being an argument with label X for a predicate at word  $u$ . These unary decisions are scored as follows:
$$
e = \operatorname {m a p} (\operatorname {R o B E R T a} (s)) \tag {1}
$$
$$
v _ {u}, a _ {i} = f _ {v} \left(e _ {u}\right), f _ {a} \left(e _ {i}\right) \tag {2}
$$
$$
\phi_ {u, i} = f _ {v a} \left(\left[ v _ {u}, a _ {i} \right]\right) \tag {3}
$$
$$
y _ {u, i} = g \left(\phi_ {u, i}\right) \tag {4}
$$
Here, map converts the wordpiece embeddings  $e$  to whole word embeddings by summation,  $f_v$  and  $f_a$  are linear transformations of the predicate and argument embeddings respectively,  $f_{va}$  is a two-layer ReLU with concatenated inputs, and finally  $g$  is a linear layer followed by softmax activation that predicts a probability distribution over labels for each word  $i$  when  $u$  is a predicate. In addition, we also have a standard first-order sequence model over label sequences for each predicate in the form of a CRF layer that is Viterbi decoded. We use the standard cross-entropy loss to train the model.
2.2 Designing Constraints
Before looking at the specifics of individual constraints, let us first look at a broad overview of our methodology. We will see concrete examples in the subsequent sections.
Output space constraints serve as prior domain knowledge for the SRL task. We will design our constraints as invariants at the training stage. To do so, we will first define constraints as statements in logic. Then we will systematically relax these Boolean statements into differentiable forms using concepts borrowed from the study of triangular norms (t-norms, Klement et al., 2013). Finally, we will treat these relaxations as regularizers in addition to the standard cross-entropy loss.
$^{1}$ Our code to replay our experiments is archived at https://github.com/utahnlp/structured_tuning_srl.
8403
All the constraints we consider are conditional statements of the form:
$$
\forall x, L (x) \rightarrow R (x) \tag {5}
$$
where the left- and the right-hand sides— $L(x), R(x)$  respectively—can be either disjunctive or conjunctive expressions. The literals that constitute these expressions are associated with classification neurons, i.e., the predicted output probabilities are soft versions of these literals.
What we want is that model predictions satisfy our constraints. To teach a model to do so, we transform conditional statements into regularizers, such that during training, the model receives a penalty if the rule is not satisfied for an example.2
To soften logic, we use the conversions shown in Table 1 that combine the product and Gödel t-norms. We use this combination because it offers cleaner derivatives make learning easier. A similar combination of t-norms was also used in prior work (Minervini and Riedel, 2018). Finally, we will transform the derived losses into log space to be consistent with cross-entropy loss. Li et al. (2019) outlines this relationship between the cross-entropy loss and constraint-derived regularizers in more detail.
Logic	∧i ai	∨i ai	¬a	a → b
Gödel	min (ai)	max (ai)	1 - a	-
Product	Πai	-	1 - a	min (1, b/a)
Table 1: Converting logical operations to differentiable forms. For literals inside of  $L(s)$  and  $R(s)$ , we use the Gödel t-norm. For the top-level conditional statement, we use the product t-norm. Operations not used in this paper are marked as  $^{-}$ .
2.3 Unique Core Roles  $(U)$
Our first constraint captures the idea that, in a frame, there can be at most one core participant of a given type. Operationally, this means that for every predicate in an input sentence  $s$ , there can be no more than one occurrence of each core argument (i.e,  $\mathcal{A}_{core} = \{\mathtt{A0},\mathtt{A1},\mathtt{A2},\mathtt{A3},\mathtt{A4},\mathtt{A5}\}$ ). In
first-order logic, we have:
$$
\begin{array}{l} \forall u, i \in s, X \in \mathcal {A} _ {\text {c o r e}}, \\ B _ {\mathrm {X}} (u, i) \rightarrow \bigwedge_ {j \in s, j \neq i} \neg B _ {\mathrm {X}} (u, j) \tag {6} \\ \end{array}
$$
which says, for a predicate  $u$ , if a model tags the  $i$ -th word as the beginning of the core argument span, then it should not predict that any other token is the beginning of the same label.
In the above rule, the literal  $B_{\mathbb{X}}$  is associated with the predicted probability for the label  $\mathbb{B} - \mathbb{X}^3$ . This association is the cornerstone for deriving constraint-driven regularizers. Using the conversion in Table 1 and taking the natural log of the resulting expression, we can convert the implication in (6) as  $l(u,i,\mathbb{X})$ :
$$
\left. \max  \left(\log B _ {\mathrm {X}} (u, i) - \min  _ {j \in s, j \neq i} \log \left(1 - B _ {\mathrm {X}} (u, j)\right)\right). \right.
$$
Adding up the terms for all tokens and labels, we get the final regularizer  $L_{U}(s)$ :
$$
L _ {U} (s) = \sum_ {(u, i) \in s, \mathrm {X} \in \mathcal {A} _ {\text {c o r e}}} l (u, i, \mathrm {X}). \tag {7}
$$
Our constraint is universally applied to all words and predicates (i.e.,  $i, u$  respectively) in the given sentence  $s$ . Whenever there is a pair of predicted labels for tokens  $i, j$  that violate the rule (6), our loss will yield a positive penalty.
Error Measurement  $\rho_{u}$  To measure the violation rate of this constraint, we will report the percentages of propositions that have duplicate core arguments. We will refer to this error rate as  $\rho_{u}$ .
2.4 Exclusively Overlapping Roles  $(\mathcal{O})$
We adopt this constraint from Punyakanok et al. (2008) and related work. In any sentence, an argument for one predicate can either be contained in or entirely outside another argument for any other predicate. We illustrate the intuition of this constraint in Table 2, assuming core argument spans are unique and tags are BIO-consistent.
Based on Table 2, we design a constraint that says: if an argument has boundary  $[i,j]$ , then no other argument span can cross the boundary at  $j$ .
3 We will use  $B_{\mathrm{X}}(u,i)$  to represent both the literal that the token  $i$  is labeled with  $\mathbb{B} - \mathbb{X}$  for predicate  $u$  and also the probability for this event. We follow a similar convention for the  $\mathbb{I} - \mathbb{X}$  labels.
${}^{2}$  Constraint-derived regularizers are dependent on examples, but not necessarily labeled ones. For simplicity, in this paper, we work with sentences from the labeled corpus. However, the methodology described here can be extended to use unlabeled examples as well.
8404
Token index	i	···	j	j+1
[i-j] has label X	BX	···	IX	¬IX
Not allowed	-	-	BY	Iy
Not allowed	¬BY ∧ ¬Iy	-	Iy	Iy
Table 2: Formalizing the exclusively overlapping role constraint in terms of the  $B$  and  $I$  literals. For every possible span  $\left\lbrack  {i - j}\right\rbrack$  in a sentence,whenever it has a label  $\mathrm{X}$  for some predicate (first row),token labels as in the subsequent rows are not allowed for any other predicate for any other argument  $\Upsilon$  . Note that this constraint does not affect the cells marked with a - .
This constraint applies to all argument labels in the task, denoted by the set  $\mathcal{A}$ .
$$
\forall u, i, j \in s \text {s u c h t h a t} j > i, \text {a n d} \forall X \in \mathcal {A},
$$
$$
P (u, i, j, \mathrm {X}) \rightarrow \bigwedge_ {\substack {v \in s, \mathrm {Y} \in \mathcal {A}\\(u, \mathrm {X}) \neq (v, \mathrm {Y})}} Q (v, i, j, \mathrm {Y}) \tag{8}
$$
$$
\begin{array}{l} P (u, i, j, \mathrm {X}) = B _ {\mathrm {X}} (u, i) \wedge I _ {\mathrm {X}} (u, j) \wedge \neg I _ {\mathrm {X}} (u, j + 1) \\ Q (v, i, j, \Upsilon) = Q _ {1} (v, i, j, \Upsilon) \wedge Q _ {2} (v, i, j, \eta) \\ Q _ {1} (v, i, j, \Upsilon) = \neg B _ {\Upsilon} (v, j) \vee \neg I _ {\Upsilon} (v, j + 1) \\ Q _ {2} (v, i, j, \Upsilon) = \\ B _ {\Upsilon} (v, i) \vee I _ {\Upsilon} (v, i) \vee \neg I _ {\Upsilon} (v, j) \vee \neg I _ {\Upsilon} (v, j + 1) \\ \end{array}
$$
Here, the term  $P(u,i,j,\mathbb{X})$  denotes the indicator for the argument span  $[i,j]$  having the label  $\mathbb{X}$  for a predicate  $u$  and corresponds to the first row of Table 2. The terms  $Q_{1}(v,i,j,\mathbb{Y})$  and  $Q_{2}(v,i,j,\mathbb{Y})$  each correspond to prohibitions of the type described in the second and third rows respectively.
As before, the literals  $B_{\mathrm{X}}$ , etc are relaxed as model probabilities to define the loss. By combining the Gödel and product t-norms, we translate Rule (8) into:
$$
L_{O}(s) = \sum_{\substack{(u,i,j)\in s\\ j > i,\mathrm{X}\in \mathcal{A}}}l(u,i,j,\mathrm{X}). \tag{9}
$$
$$
\begin{array}{l} l (u, i, j, \mathrm {X}) = \max  \left(0, \log P (u, i, j, \mathrm {X}) \right. \\ -\min_{\substack{v\in s,\Upsilon \in \mathcal{A}\\ (u,\Upsilon)\neq (v,\Upsilon)}}\log Q(v,i,j,\Upsilon)) \\ \end{array}
$$
$$
\begin{array}{l} P (u, i, j, \mathrm {X}) = \\ \min  \left(B _ {\mathrm {X}} (u, i), I _ {\mathrm {X}} (u, j), 1 - I _ {\mathrm {X}} (u, j + 1)\right) \\ \end{array}
$$
$$
Q (v, i, j, \Upsilon) = \min  \left(Q _ {1} (v, i, j, \Upsilon), Q _ {2} (v, i, j, \Upsilon)\right)
$$
$$
\begin{array}{l} Q _ {1} (v, i, j, \Upsilon) = 1 - \min  \left(B _ {\Upsilon} (v, j), I _ {\Upsilon} (v, j + 1)\right) \\ Q _ {2} (v, i, j, \Upsilon) = \\ \max  \left(B _ {\Upsilon} (v, i), I _ {\Upsilon} (v, i), 1 - I _ {\Upsilon} (v, j), 1 - I _ {\Upsilon} (v, j + 1)\right) \\ \end{array}
$$
Again, our constraint applies to all predicted probabilities. However, doing so requires scanning over 6 axes defined by  $(u,v,i,j,\mathrm{X},\mathrm{Y})$ , which is computationally expensive. To get around this, we observe that, since we have a conditional statement, the higher the probability of  $P(u,i,j,\mathrm{X})$ , the more likely it yields non-zero penalty. These cases are precisely the ones we hope the constraint helps. Thus, for faster training and ease of implementation, we modify Equation 8 by squeezing the  $(i,j)$  dimensions using top-k to redefine  $L_{O}$  above as:
$$
\mathcal {T} (u, \mathrm {X}) = \arg \operatorname {t o p} - \mathrm {k} _ {(i, j) \in s} P (u, i, j, \mathrm {X}) \tag {10}
$$
$$
L _ {O} (s) = \sum_ {u \in s, \mathrm {X} \in \mathcal {A}} \sum_ {(i, j) \in \mathcal {T} (v, \mathrm {X})} l (u, i, j, \mathrm {X}). \tag {11}
$$
where  $\mathcal{T}$  denotes the set of the top-k span boundaries for predicate  $u$  and argument label X. This change results in a constraint defined by  $u, v, \mathrm{X}, \mathrm{Y}$  and the  $k$  elements of  $\mathcal{T}$ .
Error Measurement  $\rho_{o}$  We will refer to the error of the overlap constraint as  $\rho_{o}$ , which describes the total number of non-exclusively overlapped pairs of arguments. In practice, we found that models rarely make such observed mistakes. In §3, we will see that using this constraint during training helps models generalize better with other constraints. In §4, we will analyze the impact of the parameter  $k$  in the optimization described above.
2.5 Frame Core Roles  $(F)$
The task of semantic role labeling is defined using the PropBank frame definitions. That is, for any predicate lemma of a given sense, PropBank defines which core arguments it can take and what they mean. The definitions allow for natural constraints that can teach models to avoid predicting core arguments outside of the predefined set.
$$
\forall u \in s, k \in \mathcal {S} (u),
$$
$$
\operatorname {Sense}(u,k)\to \bigwedge_{\substack{i\in s\\ \mathrm{X}\not\in\mathcal{R}(u,k)}}\neg (B_{\mathrm{X}}(u,i)\wedge I_{\mathrm{X}}(u,i))
$$
where  $S(u)$  denotes the set of senses for a predicate  $u$ , and  $\mathcal{R}(u,k)$  denotes the set of acceptable core arguments when the predicate  $u$  has sense  $k$ .
As noted in §2.2, literals in the above statement can to be associated with classification neurons. Thus the  $\operatorname{Sense}(u,k)$  corresponds to either model prediction or ground truth. Since our focus is to
8405
validate the approach of using relaxed constraints for SRL, we will use the latter.
This constraint can be also converted into regularizer following previous examples, giving us a loss term  $L_{F}(s)$ .
Error Measurement  $\rho_{f}$  We will use  $\rho_{f}$  to denote the violation rate. It represents the percentage of propositions that have predicted core arguments outside the role sets of PropBank frames.
Loss Our final loss is defined as:
$$
L _ {E} (s) + \lambda_ {U} L _ {U} (s) + \lambda_ {O} L _ {O} (s) + \lambda_ {F} L _ {F} (s) \tag {12}
$$
Here,  $L_{E}(s)$  is the standard cross entropy loss over the BIO labels, and the  $\lambda$ 's are hyperparameters.
In this section, we study the question: In what scenarios can we inform an end-to-end trained neural model with declarative knowledge? To this end, we experiment with the CoNLL-05 and CoNLL-12 datasets, using standard splits and the official evaluation script for measuring performance. To empirically verify our framework in various data regimes, we consider scenarios ranging from where only limited training data is available, to ones where large amounts of clean data are available.
Our baseline (described in §2.1) is based on RoBERTa. We used the pre-trained base version released by Wolf et al. (2019). Before the final linear layer, we added a dropout layer (Srivastava et al., 2014) with probability 0.5. To capture the sequential dependencies between labels, we added a standard CRF layer. At testing time, Viterbi decoding with hard transition constraints was employed across all settings. In all experiments, we used the gold predicate and gold frame senses.
Model training proceeded in two stages:
1. We use the finetuned the pre-trained RoBERTa model on SRL with only cross-entropy loss for 30 epochs with learning rate  $3 \times 10^{-5}$ .
2. Then we continued finetuning with the combined loss in Equation 12 for another 5 epochs with a lowered learning rate of  $1 \times 10^{-5}$ .
During both stages, learning rates were warmed up linearly for the first  $10\%$  updates.
For fair comparison, we finetuned our baseline twice (as with the constrained models); we found that it consistently outperformed the singly finetuned baseline in terms of both error rates and role F1. We grid-searched the  $\lambda$ 's by incrementally adding regularizers. The combination of  $\lambda$ 's with good balance between F1 and error  $\rho$ 's on the dev set were selected for testing. We refer readers to the appendix for the values of  $\lambda$ 's.
For models trained on the CoNLL-05 data, we report performance on the dev set, and the WSJ and Brown test sets. For CoNLL-12 models, we report performance on the dev and the test splits.
3.2 Scenario 1: Low Training Data
Creating SRL datasets requires expert annotation, which is expensive. While there are some efforts on semi-automatic annotation targeting low-resource languages (e.g., Akbik et al., 2016), achieving high neural network performance with small or unlabeled datasets remains a challenge (e.g., Fürstenau and Lapata, 2009, 2012; Titov and Klementiev, 2012; Gormley et al., 2014; Abend et al., 2009).
In this paper, we study the scenario where we have small amounts of fully labeled training data. We sample  $3\%$  of the training data and an equivalent amount of development examples. The same training/dev subsets are used across all models.
Table 3 reports the performances of using  $3\%$  training data from CoNLL-05 and CoNLL-12 (top and bottom respectively). We compare our strong baseline model with structure-tuned models using all three constraints. Note that for all these evaluations, while we use subsamples of the dev set for model selection, the evaluations are reported using the full dev and test sets.
We see that training with constraints greatly improves precision with low training data, while recall reduces. This trade-off is accompanied by a reduction in the violation rates  $\rho_{u}$  and  $\rho_{f}$ . As noted in §2.4, models rarely predict label sequences that violate the exclusively overlapping roles constraint. As a result, the error rate  $\rho_{o}$  (the number of violations) only slightly fluctuates.
3.3 Scenario 2: Large Training Data
Table 4 reports the performance of models trained with our framework using the full training set of the CoNLL-05 dataset which consists of  $35\mathrm{k}$  sentences with 91k propositions. Again, we compare RoBERTa (twice finetuned) with our structure-tuned models. We see that the constrained models
8406
CoNLL-05  $(3\%)$ $1.1\mathrm{k})$
Dev	P	R	F1	δF1	ρu	ρo	ρf
RoBERTa2	67.79	72.69	70.15	14.56	23	6.19
+U,F,O	70.40	71.91	71.15	1.0	8.56	20	5.82
WSJ	P	R	F1	δF1	ρu	ρo	ρf
RoBERTa2	70.48	74.96	72.65	13.35	37	NA
+U,F,O	72.60	74.13	73.36	0.7	7.46	49	NA
Brown	P	R	F1	δF1	ρu	ρo	ρf
RoBERTa2	62.16	66.93	64.45	12.94	6	NA
+U,F,O	64.31	65.64	64.97	0.5	5.47	6	NA
CoNLL-12 (3%, 2.7k)
Dev	P	R	F1	δF1	ρu	ρo	ρf
RoBERTa2	74.39	76.88	75.62	7.43	294	3.23
+U,F,O	75.99	76.80	76.39	0.8	4.37	245	3.01
Test	P	R	F1	δF1	ρu	ρo	ρf
RoBERTa2	74.79	77.17	75.96	6.92	156	2.67
+U,F,O	76.31	76.88	76.59	0.6	4.12	171	2.41
Table 3: Results on low training data (3% of CoNLL-05 and CoNLL-12). RoBERTa $^2$ : Baseline finetuned twice. U: Unique core roles. F: Frame core roles. O: Exclusively overlapping roles.  $\delta$ F1: improvement over baseline.  $\rho_f$  is marked NA for the CoNLL-05 test results because ground truth sense is unavailable on the CoNLL-05 shared task page.
CoNLL-05  $(100\%)$ $36\mathrm{k})$
Dev	P	R	F1	δF1	ρu	ρf
RoBERTa2	86.74	87.24	86.99	1.97	3.23
+U,F,O	87.24	87.26	87.25	0.3	1.35	2.99
Oracle	0.40	2.34
WSJ	P	R	F1	δF1	ρu	ρf
RoBERTa2	87.75	87.94	87.85	1.71	NA
+U,F,O	88.05	88.00	88.03	0.2	0.85	NA
Oracle	0.30	NA
Brown	P	R	F1	δF1	ρu	ρf
RoBERTa2	79.38	78.92	78.64	3.36	NA
+U,F,O	80.04	79.56	79.80	1.2	1.24	NA
Oracle	0.30	NA