SlowGuess/ACL_2020_OCR huggingface dataset:
Table 4: Results on the full CoNLL-05 data. Oracle: Errors of oracle.  $\rho_o$  is in [0,6] across all settings.
consistently outperform baselines on the dev, WSJ, and Brown sets. With all three constraints, the constrained model reaches 88 F1 on the WSJ. It also generalizes well on new domain by outperforming the baseline by 1.2 points on the Brown test set.
As in the low training data experiments, we observe improved precision due to the constraints.
This suggests that even with large training data, direct label supervision might not be enough for neural models to pick up the rich output space structure. Our framework helps neural networks, even as strong as RoBERTa, to make more correct predictions from differentiable constraints.
Surprisingly, the development ground truth has a  $2.34\%$  error rate on the frame role constraint, and  $0.40\%$  on the unique role constraint. Similar percentages of unique role errors also appear in WSJ and Brown test sets. For  $\rho_{o}$ , the oracle has no violations on the CoNLL-05 dataset.
The exclusively overlapping constraint (i.e.  $\rho_{o}$ ) is omitted as we found models rarely make such prediction errors. After adding constraints, the error rate of our model approached the lower bound. Note that our framework focuses on the learning stage without any specialized decoding algorithms in the prediction phase except the Viterbi algorithm to guarantee that there will be no BIO violations.
What about even larger and cleaner data? The ideal scenario, of course, is when we have the luxury of massive and clean data to power neural network training. In Table 5, we present results on CoNLL-12 which is about 3 times as large as CoNLL-05. It consists of 90k sentences and 253k propositions. The dataset is also less noisy with respect to the constraints. For instance, the oracle development set has no violations for both the unique core and the exclusively overlapping constraints.
We see that, while adding constraints reduced error rates of  $\rho_{u}$  and  $\rho_{f}$ , the improvements on label consistency do not affect F1 much. As a result, our best constrained model performs on a par with the baseline on the dev set, and is slightly better than the baseline (by 0.1) on the test set. Thus we believe when we have the luxury of data, learning with constraints would become optional. This observation is in line with recent results in Li and Srikumar (2019) and Li et al. (2019).
But is it due to the large data or the strong baseline? To investigate whether the seemingly saturated performance is from data or from the model, we also evaluate our framework on the original BERT (Devlin et al., 2019) which is relatively less powerful. We follow the same model setup for experiments and report the performances in Table 5 and Table 9. We see that compared to RoBERTa, BERT obtains similar F1 gains on the test set, sug
8407
gesting performance ceiling is due to the train size.
CoNLL-12  $(100\% ,90\mathrm{k})$
Dev	P	R	F1	δF1	ρu	ρf
RoBERTa2	86.62	86.91	86.76	0.86	1.18
+U,F,O	86.60	86.89	86.74	0	0.59	1.04
Oracle	0	0.38
Test	P	R	F1	δF1	ρu	ρf
RoBERTa2	86.28	86.67	86.47	0.91	0.97
+U,F,O	86.40	86.83	86.61	0.1	0.50	0.93
Oracle	0	0.42
Dev	P	R	F1	δF1	ρu	ρf
BERT2	85.62	86.22	85.92	1.41	1.12
+U,F,O	85.97	86.38	86.18	0.3	0.78	1.07
Test	P	R	F1	δF1	ρu	ρf
BERT2	85.52	86.24	85.88	1.32	0.94
+U,F,O	85.82	86.36	86.09	0.2	0.79	0.90
Table 5: Results on CoNLL-12. BERT<sup>2</sup>: The original BERT finetuned twice.  $\rho_{o}$  is around 50 across all settings. With the luxury of large and clean data, constrained learning becomes less effective.
4 Ablations & Analysis
In §3, we saw that constraints not just improve model performance, but also make outputs more structurally consistent. In this section, we will show the results of an ablation study that adds one constraint at a time. Then, we will examine the sources of improved F-score by looking at individual labels, and also the effect of the top-k relaxation for the constraint  $O$ . Furthermore, we will examine the robustness of our method against randomness involved during training. We will end this section with a discussion about the ability of constrained neural models to handle structured outputs.
Constraint Ablations We present the ablation analysis on our constraints in Table 6. We see that as models become more constrained, precision improves. Furthermore, one class of constraints do not necessarily reduce the violation rate for the others. Combining all three constraints offers a balance between precision, recall, and constraint violation.
One interesting observation that adding the  $O$  constraints improve F-scores even though the  $\rho_{o}$  values were already close to zero. As noted in §2.4, our constraints apply to the predicted scores of all labels for a given argument, while the actual decoded label sequence is just the highest scoring
sequence using the Viterbi algorithm. Seen this way, our regularizers increase the decision margins on affected labels. As a result, the model predicts scores that help Viterbi decoding, and, also generalizes better to new domains i.e., the Brown set.
CoNLL-05  $(100\% ,36\mathrm{k})$
Dev	P	R	F1	ρu	ρf
RoBERTa2	86.74	87.24	86.99	1.97	3.23
+U	87.21	87.32	87.27	1.29	3.23
+U,F	87.19	87.54	87.37	1.20	3.11
+U,F,O	87.24	87.26	87.25	1.35	2.99
WSJ	P	R	F1	ρu	ρf
RoBERTa2	87.75	87.94	87.85	1.71	NA
+U	87.88	88.01	87.95	1.18	NA
+U,F	88.05	88.09	88.07	0.89	NA
+U,F,O	88.05	88.00	88.03	0.85	NA
Brown	P	R	F1	ρu	ρf
RoBERTa2	79.38	78.92	78.64	3.36	NA
+U	79.36	79.15	79.25	1.74	NA
+U,F	79.60	79.24	79.42	1.00	NA
+U,F,O	80.04	79.56	79.80	1.24	NA
Table 6: Ablation tests on CoNLL-05.
Sources of Improvement Table 7 shows labelwise F1 scores for each argument. Under low training data conditions, our constrained models gained improvements primarily from the frequent labels, e.g., A0-A2. On CoNLL-05 dataset, we found the location modifier (AM-LOC) posed challenges to our constrained models which significantly performed worse than the baseline. Another challenge is the negation modifier (AM-NEG), where our models underperformed on both datasets, particularly with small training data. When using the CoNLL-12 training set, our models performed on par with the baseline even on frequent labels, confirming that the performance of soft-structured learning is nearly saturated on the larger, cleaner dataset.
Impact of Top-  $k$  Beam Size As noted in  $\S 2.4$ , we used the top-  $k$  strategy to implement the constraint  $O$ . As a result, there is a certain chance for predicted label sequences to have non-exclusive overlap without our regularizer penalizing them. What we want instead is a good balance between coverage and runtime cost. To this end, we analyze the CoNLL-12 development set using the baseline trained on  $3\%$  of CoNLL-12 data. Specifically, we count the examples which have such overlap but the regularization loss is  $\leq 0.001$ . In Table 8, we
8408
CoNLL-05 3%	CoNLL-05 100%	CoNLL-12 3%	CoNLL-12 100%
RoBERTa2	+U,F,O	RoBERTa2	+U,F,O	RoBERTa2	+U,F,O	RoBERTa2	+U,F,O
A0	81.28	82.11	93.43	93.52	84.99	85.73	92.78	92.81
A1	72.12	73.59	89.23	89.80	78.36	79.67	89.88	89.75
A2	46.50	47.52	79.53	79.73	68.24	69.20	84.93	84.90
A3	39.58	42.11	81.45	81.86	33.26	34.47	72.96	73.24
A4	51.61	51.56	74.60	75.59	56.29	58.38	80.80	80.33
AM-ADV	44.07	47.56	66.67	66.91	55.26	54.93	66.37	66.92
AM-DIR	16.39	18.92	55.26	55.56	36.51	35.81	64.92	64.95
AM-DIS	71.07	70.84	80.20	80.50	76.35	76.40	82.86	82.71
AM-LOC	53.08	51.60	69.02	66.50	59.74	59.94	72.74	73.21
AM-MNR	44.30	44.18	68.63	69.87	56.14	55.67	70.89	71.13
AM-MOD	91.88	91.60	98.27	98.60	95.50	95.76	97.88	98.04
AM-NEG	91.18	88.35	94.06	93.60	93.29	93.05	95.93	95.83
AM-TMP	74.05	74.13	88.24	88.08	79.00	78.78	87.58	87.56
Overall	70.48	71.55	87.33	87.61	76.66	77.45	87.60	87.58
see that  $k = 4$  yields good coverage.
Table 7: Label-wise F1 scores for the CoNLL-05 and CoNLL-12 development sets.
k	1	2	4	6
# Ex.	10	8	3	2
Robustness to random initialization We observed that model performance with structured tuning is generally robust to random initialization. As an illustration, we show the performance of models trained on the full CoNLL-12 dataset with different random initializations in Table 9.
Table 8: Impact of  $k$  for the top-  $k$  strategy,showing the number of missed examples for different  $k$  . We set  $k = 4$  across all experiments.
CoNLL-12 (100%, 90k)
Test F1	Seed1	Seed2	Seed3	avg δF1
BERT2	85.88	85.91	86.13
+U,F,O	86.09	86.07	86.19	0.1
Test F1	Seed1	Seed2	Seed3	avg δF1
RoBERTa2	86.47	86.33	86.45
+U,F,O	86.61	86.48	86.57	0.1
Table 9: F1 scores models trained on the CoNLL-12 data with different random seeds. The randomness affects the initialization of the classification layers and the batch ordering during training.
Can Constrained Networks Handle Structured Prediction? Larger, cleaner data may presumably be better for training constrained neural models. But it is not that simple. We will approach the above question by looking at how good the
transformer models are at dealing with two classes of constraints, namely: 1) structural constraints that rely only on available decisions (constraint  $U$ ), 2) constraints involving external knowledge (constraint  $F$ ).
For the former, we expected neural models to perform very well since the constraint  $U$  represents a simple local pattern. From Tables 4 and 5, we see that the constrained models indeed reduced violations  $\rho_{u}$  substantially. However, when the training data is limited, i.e., comparing CoNLL-05  $3\%$  and  $100\%$ , the constrained models, while reducing the number of errors, still make many invalid predictions. We conjecture this is because networks learn with constraints mostly by memorization. Thus the ability to generalize learned patterns on unseen examples relies on training size.
The constraint  $F$  requires external knowledge from the PropBank frames. We see that even with large training data, constrained models were only able to reduce error rate  $\rho_f$  by a small margin. In our development experiments, having larger  $\lambda_F$  tends to strongly sacrifice argument F1, yet still does not to improve development error rate substantially. Without additional training signal in the form of such background knowledge, constrained inference becomes a necessity, even with strong neural network models.
Semantic Role Labeling & Constraints The SRL task is inherently knowledge rich; the outputs are defined in terms of an external ontology of frames. The work presented here can be gener-
8409
alized to several different flavors of the task, and indeed, constraints could be used to model the interplay between them. For example, we could revisit the analysis of Yi et al. (2007), who showed that the PropBank A2 label takes on multiple meanings, but by mapping them to VerbNet, they can be disambiguated. Such mappings naturally define constraints that link semantic ontologies.
Constraints have long been a cornerstone in the SRL models. Several early linear models for SRL (e.g. Punyakanok et al., 2004, 2008; Surdeanu et al., 2007) modeled inference for PropBank SRL using integer linear programming. Riedel and Meza-Ruiz (2008) used Markov Logic Networks to learn and predict semantic roles with declarative constraints. The work of (Tackström et al., 2015) showed that certain SRL constraints admit efficient decoding, leading to a neural model that used this framework (FitzGerald et al., 2015). Learning with constraints has also been widely adopted in semi-supervised SRL (e.g., Fürstenau and Lapata, 2012).
With the increasing influence of neural networks in NLP, however, the role of declarative constraints seem to have decreased in favor of fully end-to-end training (e.g., He et al., 2017; Strubell et al., 2018, and others). In this paper, we show that even in the world of neural networks with contextual embeddings, there is still room for systematically introducing knowledge in the form of constraints, without sacrificing the benefits of end-to-end learning.
Structured Losses Chang et al. (2012) and Ganchev et al. (2010) developed models for structured learning with declarative constraints. Our work is in the same spirit of training models that attempts to maintain output consistency.
There are some recent works on the design of models and loss functions by relaxing Boolean formulas. Kimmig et al. (2012) used the Lukasiewicz t-norm for probabilistic soft logic. Li and Srikumar (2019) augment the neural network architecture itself using such soft logic. Xu et al. (2018) present a general framework for loss design that does not rely on soft logic. Introducing extra regularization terms to a downstream task have been shown to be beneficial in terms of both output structure consistency and prediction accuracy (e.g., Minervini and Riedel, 2018; Hsu et al., 2018; Mehta et al., 2018; Du et al., 2019; Li et al., 2019).
Final words In this work, we have presented a framework that seeks to predict structurally consistent outputs without extensive model redesign, or any expensive decoding at prediction time. Our experiments on the semantic role labeling task show that such an approach can be especially helpful in scenarios where we do not have the luxury of massive annotated datasets.
We thank members of the NLP group at the University of Utah for their valuable insights and suggestions; and reviewers for pointers to related works, corrections, and helpful comments. We also acknowledge the support of NSF Cyberlearning-1822877, SaTC-1801446, U.S. DARPA KAIROS Program No. FA8750-19-2-1004, DARPA Communicating with Computers DARPA 15-18-CwC-FP032, HDTRA1-16-1-0002, and gifts from Google and NVIDIA.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Omri Abend, Roi Reichart, and Ari Rappoport. 2009. Unsupervised argument identification for semantic role labeling. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP.
Alan Akbik, Vishwajeet Kumar, and Yunyao Li. 2016. Towards semi-automatic generation of proposition Banks for low-resource languages. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Xavier Carreras and Lluís Márquez. 2005. Introduc- tion to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Confer- ence on Computational Natural Language Learning (CoNLL-2005).
Ming-Wei Chang, Lev Ratinov, and Dan Roth. 2012. Structured Learning with Constrained Conditional Models. Machine learning, 88(3):399-431.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Xinya Du, Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen tau Yih, Peter Clark, and Claire Cardie. 2019. Be Consistent! Improving Procedural Text Comprehension using Label Consistency. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Nicholas FitzGerald, Oscar Tackström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 960-970.
Hagen Fürstenau and Mirella Lapata. 2009. Graph alignment for semi-supervised semantic role labeling. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing.
Hagen Fürstenau and Mirella Lapata. 2012. Semi-supervised semantic role labeling via structural alignment. Computational Linguistics, 38(1):135-171.
Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. 2010. Posterior Regularization for Structured Latent Variable Models. Journal of Machine Learning Research.
Matthew R. Gormley, Margaret Mitchell, Benjamin Van Durme, and Mark Dredze. 2014. Low-resource semantic role labeling. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what's next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Richard Johansson and Pierre Nugues. 2008. Dependency-based semantic role labeling of PropBank. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing.
Angelika Kimmig, Stephen Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. 2012. A short Introduction to Probabilistic Soft Logic. In Proceedings of the NIPS Workshop on Probabilistic Programming: Foundations and Applications.
Erich Peter Klement, Radko Mesiar, and Endre Pap. 2013. Triangular Norms. Springer Science & Business Media.
Meghana Kshirsagar, Sam Thomson, Nathan Schneider, Jaime Carbonell, Noah A. Smith, and Chris Dyer. 2015. Frame-semantic role labeling with heterogeneous annotations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Tao Li and Vivek Srikumar. 2019. Augmenting Neural Networks with First-order Logic. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Sanket Vaibhav Mehta, Jay Yoon Lee, and Jaime Carbonell. 2018. Towards Semi-Supervised Learning for Deep Semantic Role Labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Pasquale Minervini and Sebastian Riedel. 2018. Adversarily Regularising Neural NLI Models to Integrate Logical Background Knowledge. In Proceedings of the 22nd Conference on Computational Natural Language Learning.
Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018. A span selection model for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Martha Palmer, Daniel Gildea, and Nianwen Xue. 2010. Semantic role labeling. Synthesis Lectures on Human Language Technologies, 3(1):1-103.
Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina,
8411
Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning.
Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2005. The necessity of syntactic parsing for semantic role labeling. In IJCAI.
Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics.
Vasin Punyakanok, Dan Roth, Wen-tau Yih, and Dav Zimak. 2004. Semantic role labeling via integer linear programming inference. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics.
Sebastian Riedel and Ivan Meza-Ruiz. 2008. Collective semantic role labelling with Markov logic. In CoNLL 2008: Proceedings of the Twelfth Conference on Computational Natural Language Learning.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-Informed Self-Attention for Semantic Role Labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Mihai Surdeanu, Lluis Márquez, Xavier Carreras, and Pere R Comas. 2007. Combination strategies for semantic role labeling. Journal of Artificial Intelligence Research, 29:105-151.
Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Efficient inference and structured learning for semantic role labeling. Transactions of the Association for Computational Linguistics.
Ivan Titov and Alexandre Klementiev. 2012. Semisupervised semantic role labeling: Approaching from an unsupervised perspective. In Proceedings of COLING 2012.
Kristina Toutanova, Aria Haghighi, and Christopher Manning. 2005. Joint learning improves semantic role labeling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtopicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. 2018. A Semantic Loss Function for Deep Learning with Symbolic Knowledge. In International Conference on Machine Learning.
Szu-ting Yi, Edward Loper, and Martha Palmer. 2007. Can Semantic Roles Generalize Across Genres? In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 548-555, Rochester, New York. Association for Computational Linguistics.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020. Semantics-aware bert for language understanding. AAAI Conference on Artificial Intelligence (AAAI).
A.1 Hyperparameters
We show the hyperparameters of  $\lambda$ 's in Table 10. We conducted grid search on the combinations of  $\lambda$ 's for each setting and the best one on development set is selected for reporting.
Model	λU	λO	λF
RoBERTa CoNLL-05 (3%) +U,F,O	2	0.5	0.5
RoBERTa CoNLL-2012 (3%) +U,F,O	1	2	1
RoBERTa CoNLL-05 (100%) +U	1
+U,F	1	0.5
+U,F,O	1	0.5	0.1
RoBERTa CoNLL-2012 (100%) +U,F,O	1	1	0.1
BERT CoNLL-2012 (100%) +U,F,O	0.5	1	0.1
Table 10: Values of hyperparameter  ${\lambda }^{\prime }\mathrm{s}$  .
8412