SlowGuess/ACL_2020_OCR huggingface dataset:
Chang et al. (2007) showed that hand-crafted constraints specific to this domain can vastly help models to correctly identify citation fields. We show that constraints learned from the training data can improve a trained model without the need for manual effort.
Dataset and baseline. We use the dataset from Chang et al. (2007, 2012) whose training, development and test splits have 300, 100 and 100 examples, respectively. We train a first-order Markov model using structured SVM (Tsochantaridis et al., 2004) on the training set with the same raw text features as in the original work.
Constraint features. We explore multiple simple constraint features  $\psi (\mathbf{x},\mathbf{y})$  in the citation field extraction experiments as shown in Table 2. Detailed descriptions of these features, including how to develop negative examples for each feature, and experiment settings are in the supplementary material.
Feature	Description
Label existence	Indicates which labels exist in a citation
Label counts	Counts the number of occurrences of a label
Bigram labels	Indicators for adjacent labels
Trigram labels	Indicators for 3 adjacent labels
Part-of-speech	Indicator for the part-of-speech of a token
Punctuation	Indicator for whether a token is a punctuation
Table 2: Constraint feature templates for the citation field extraction task
5.1 Experiments and Results
For each constraint feature template, we trained a rectifier network with 10 ReLUs in the hidden layer. We then use Theorem 1 to convert the resulting network to a system of  $2^{10} - 1$ , or 1023 linear inequalities. We used beam search with beam size 50 to combine the learned inequalities with the original sequence model to predict on the test set. States in the search space correspond to partial assignments to a prefix of the sequence. Each step we predict the label for the next token in the sequence. The pretrained sequence model (i.e., the baseline) ranks search nodes based on transition and emission scores, and the learned inequality prunes the search space accordingly<sup>4</sup>. Table 3 shows the token level accuracies of various methods.
The results show that all versions of constrained search outperform the baselines, indicating that the learned constraints are effective in the citation field extraction task. Furthermore, different constraints learned with different features can be combined. We observe that combining different constraint features generally improves accuracy.
It is worth pointing out that the label existence and label counts features are global in nature and cannot be directly used to train a sequence model. Even if some constraint features can be used in training the original model, it is still beneficial to learn constraints from them. For example, the bigram label feature is captured in the original first order model, but adding constraints learned from them still improves performance. As another test, we trained a model with POS features, which also contains punctuation information. This model achieves  $91.8\%$  accuracy. Adding constraints learned with POS improves the accuracy to  $92.6\%$ ; adding constraints learned with punctuation features further improves it to  $93.8\%$ .
We also observed that our method for learning constraints is robust to the choice of the number of hidden ReLUs. For example, for punctuation, learning using 5, 8 and 10 hidden ReLUs results in accuracy of  $90.1\%$ ,  $90.3\%$ , and  $90.2\%$ , respectively. We observed similar behavior for other constraint features as well. Since the number of constraints learned is exponential in the number of hidden units, these results show that learning redundant constraints will not hurt performance.
<sup>4</sup>Since the label-existence and label-counts features are global, pruning by learned inequalities is possible only at the last step of search. The other four features admit pruning at each step of the search process.
Baselines	Search with learned constraints	Combine constraints
Exact	Search	L.E.	L.C.	B.L.	T.L.	POS	Punc.	C1	C2	C3
86.2	87.3	88.0	87.7	87.9	88.1	89.8	90.2	88.6	90.1	90.6
Table 3: Token level accuracies (in percentage) of baseline models and constrained-search models, for the citation field extraction task. Exact is our trained first-order Markov model. It uses exact inference (dynamic programming) for prediction. Search is our search baseline, it uses the same model as Exact, but with beam search for inexact inference. L.E., L.C., B.L., T.L., POS, Punc. use search with different constraint features: label existence, label counts, bigram labels, trigram labels, part-of-speech, and punctuation features. C1 to C3 are search with combined constraints. C1 combines L.E. and T.L. C2 combines L.E., T.L. and POS. Finally C3 combines all constraints.
Note that carefully hand-crafted constraints may achieve higher accuracy than the learned ones. Chang et al. (2007) report an accuracy of  $92.5\%$  with constraints specifically designed for this domain. In contrast, our method for learning constraints uses general constraint features, and does not rely on domain knowledge. Therefore, our method is suited to tasks where little is known about the underlying domain.
6 Chunking Experiments
Chunking is the task of clustering text into groups of syntactically correlated tokens or phrases. In the instance below, the phrase labels are underlined:
[NP An A.P. Green official] [VP declined to comment] [PP on] [NP the filing] [O.]
We treat the chunking problem as a sequence labeling problem by using the popular IOB tagging scheme. For each phrase label, the first token in the phrase is labeled with a "B-" prefixed to phrase label while the other tokens are labeled with an "I-" prefixed to the phrase label. Hence,
[NP An A.P. Green official]
is represented as
[ \underline{[\mathrm{B} - \mathrm{NP}\mathrm{An}]} \underline{[\mathrm{I} - \mathrm{NP}\mathrm{A.P.}]} \underline{[\mathrm{I} - \mathrm{NP}\mathrm{Green}]} ]  
[ \underline{[\mathrm{I} - \mathrm{NP}\mathrm{official}]} ]
This is done for all phrase labels except "O".
Dataset and Baselines. We use the CoNLL2000 dataset (Tjong Kim Sang and Buchholz, 2000) which contains 8936 training sentences and 2012 test sentences. For our experiments, we consider 8000 sentences out of 8936 training sentences as our training set and the remaining 936 sentences as our development set. Chunking is a well-studied problem and showing performance improvements on full
training dataset is difficult. However, we use this task to illustrate the interplay of learned constraints with neural network models, and the impact of learned constraints in the low training data regime.
We use the BiLSTM-CRF (Huang et al., 2015) for this sequence tagging task. We use GloVe for word embeddings. We do not use the BERT (Devlin et al., 2019) family of models since tokens are broken down into sub-words during pre-processing, which introduces modeling and evaluation choices that are orthogonal to our study of label dependencies. As with the citation task, all our constrained models use beam search, and we compare our results to both exact decoding and beam search baselines. We use two kinds of constraint features: (i)  $n$ -gram label existence, and (ii)  $n$ -gram part of speech. Details of the constraint features and construction of negative samples are given in the supplementary material.
6.1 Experiments and Results
We train the rectifier network with 10 hidden units. The beam size of 10 was chosen for our experiments based on preliminary experiments. We report the average results on two different random seeds for learning each constraint. Note that the  $n$ -gram label existence is a global constraint while the  $n$ -gram POS constraint is a local constraint which checks for validity of label assignments at each token. In essence, the latter constraint reranks the beam at each step by ensuring that states that satisfy the constraint are preferred over states that violate the constraint. Since the  $n$ -gram label existence is a global constraint, we check the validity of the tag assignments only at the last token. In the case where none of the states in the beam satisfy the constraint, the original beams are used.
The results for this set of experiments are presented in Table 4. We observe that the POS constraint improves the performance of the base
4849
Constraint	n	Percentage of training data used
1%	5%	10%	25%	50%	100%
Label existence	2	81.28	88.30	89.73	91.24	90.40	92.48
3	80.98	88.20	90.58	91.20	92.37	93.12
Part-of-speech	3	86.52	90.74	91.80	92.41	93.07	93.84
4	84.21	90.99	92.17	92.46	93.08	93.93
Search without constraints	81.29	88.27	90.62	91.33	92.51	93.44
Exact decoding	82.11	88.70	90.49	92.57	93.94	94.75
Table 4: Token level accuracies (in percentage) for the chunking baseline and constrained model. The results are shown on  $n$ -gram Label Existence and  $n$ -gram Part of Speech constraints with  $n = \{2,3\}$  and  $n = \{3,4\}$  respectively. The results are shown on  $\{1\%, 5\%, 10\%, 25\%, 50\%, 100\% \}$  of training data. Exact decoding with Viterbi algorithm and Search w/o constraint are baseline models which do not incorporate constraints during inference.
line models significantly, outperforming the beam search baseline on all training ratios. More importantly, the results show sizable improvements in accuracy for smaller training ratios (e.g,  $4.41\%$  and  $5.23\%$  improvements on exact and search baselines respectively with  $1\%$  training data). When the training ratios get bigger, we expect the models to learn these properties and hence the impact of the constraints decreases.
These results (along with the experiments in the previous sections) indicate that our constraints can significantly boost performance in the low data regime. Another way to improve performance in low resource settings is to use better pretrained input representations. When we replaced GloVe embeddings with ELMo, we observed a  $87.09\%$  accuracy on 0.01 ratio of training data using exact decoding. However, this improvement comes at a cost: the number of parameters increases from 3M (190k trainable) to 94M (561k trainable). In contrast, our method instead introduces a smaller rectifier network with  $\approx 1000$  additional parameters while still producing similar improvements. In other words, using trained constraints is computationally more efficient.
We observe that the label existence constraints, however, do not help. We conjecture that this may be due to one of the following three conditions: (i) The label existence constraint might not exist for the task; (ii) The constraint exists but the learner is not able to find it; (iii) The input representations are expressive enough to represent the constraints. Disentangling these three factors is a future research challenge.
Structured prediction is an active field in machine learning and has numerous applications, including various kinds of sequence labeling tasks, parsing (e.g., Martins et al., 2009), image segmentation (e.g., Lam et al., 2015), and information extraction (e.g., Anzaroot et al., 2014). The work of Roth and Yih (2004) introduced the idea of using explicitly stated constraints in an integer programming framework. That constraints and knowledge can improve models has been highlighted by several lines of work (e.g., Ganchev et al., 2010; Chang et al., 2012; Hu et al., 2016).
The interplay between constraints and representations has been sharply highlighted by recent work on integrating neural networks with structured outputs (e.g., Rocktäschel and Riedel, 2017; Niculae et al., 2018; Manhaeve et al., 2018; Xu et al., 2018; Li and Srikumar, 2019; Li et al., 2019, and others). We expect that constraints learned as described in this work can be integrated into these formalisms, presenting an avenue for future research.
While our paper focuses on learning explicit constraints directly from examples, it is also possible to use indirect supervision from these examples to learn a structural classifier (Chang et al., 2010), with an objective function penalizing invalid structures.
Related to our goal of learning constraints is rule learning, as studied in various subfields of artificial intelligence. Quinlan (1986) describes the ID3 algorithm, which extracts rules as a decision tree. First order logic rules can be learned from examples using inductive logic programming (Muggleton and de Raedt, 1994; Lavrac and Dzeroski, 1994;
Page and Srinivasan, 2003). Notable algorithms for inductive logic programming include FOIL (Quinlan, 1990) and Progol (Muggleton, 1995).
Statistical relation learning addresses learning constraints with uncertainty (Friedman et al., 1999; Getoor and Mihalkova, 2001). Markov logic networks (Richardson and Domingos, 2006) combines probabilistic models with first order logic knowledge, whose weighted formulas are soft constraints and the weights can be learned from data. In contrast to these directions, in this paper, we exploit a novel representational result about rectifier networks to learn polytopes that represent constraints with off-the-shelf neural network tools.
We presented a systematic way for discovering constraints as linear inequalities for structured prediction problems. The proposed approach is built upon a novel transformation from two layer rectifier networks to linear inequality constraints and does not rely on domain expertise for any specific problem. Instead, it only uses general constraint features as inputs to rectifier networks. Our approach is particularly suited to tasks where designing constraints manually is hard, and/or the number of training examples is small. The learned constraints can be used for structured prediction problems in two ways: (1) combining them with an existing model to improve prediction performance, or (2) incorporating them into the training process to train a better model. We demonstrated the effectiveness of our approach on three NLP tasks, each with different original models.
We thank members of the NLP group at the University of Utah, especially Jie Cao, for their valuable insights and suggestions; and reviewers for pointers to related works, corrections, and helpful comments. We also acknowledge the support of NSF Cyberlearning-1822877, SaTC-1801446 and gifts from Google and NVIDIA.
Sam Anzaroot, Alexandre Passos, David Belanger, and Andrew McCallum. 2014. Learning Soft Linear Constraints with Application to Citation Field Extraction. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daume, and John Langford. 2015. Learning to Search Better than Your Teacher. In Proceedings of The 32nd International Conference on Machine Learning.
Ming-Wei Chang, Lev Ratinov, and Dan Roth. 2007. Guiding semi-supervision with constraint-driven learning. In Proceedings of the 45th annual meeting of the association of computational linguistics.
Ming-Wei Chang, Lev Ratinov, and Dan Roth. 2012. Structured Learning with Constrained Conditional Models. Machine Learning.
Ming-wei Chang, Vivek Srikumar, Dan Goldwasser, and Dan Roth. 2010. Structured Output Learning with Indirect Supervision. Proceedings of the 27th International Conference on Machine Learning (ICML-10).
Michael Collins and Brian Roark. 2004. Incremental Parsing with the Perceptron Algorithm. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics.
Hal Daumé, John Langford, and Daniel Marcu. 2009. Search-based Structured Prediction. Machine Learning Journal (MLJ).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Janardhan Rao Doppa, Alan Fern, and Prasad Tadepalli. 2014. Structured Prediction via Output Space Search. The Journal of Machine Learning Research.
Nir Friedman, Lise Getoor, Daphne Koller, and Avi Pfeffer. 1999. Learning Probabilistic Relational Models. Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence.
Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. 2010. Posterior Regularization for Structured Latent Variable Models. The Journal of Machine Learning Research.
Lise Getoor and Lilyana Mihalkova. 2001. Learning Statistical Models from Relational Data. Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. *Generative Adversarial Nets*. In Advances in Neural Information Processing Systems 27.
Gurobi Optimization LLC. 2019. Gurobi optimizer reference manual.
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016. Harnessing Deep Neural Networks with Logic Rules. In "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)".
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv preprint arXiv:1508.01991.
Nikos Komodakis, Nikos Paragios, and Georgios Tziritas. 2007. MRF Optimization via Dual Decomposition: Message-passing Revisited. Proceedings of the IEEE International Conference on Computer Vision.
Michael Lam, Janardhan Rao Doppa, Sinisa Todorovic, and Thomas G Dietterich. 2015. HC-Search for Structured Prediction in Computer Vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Nada Lavrac and Saso Dzeroski. 1994. Inductive Logic Programming: Techniques and Applications. Ellis Horwood.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019. A Logic-Driven Framework for Consistency of Neural Models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Tao Li and Vivek Srikumar. 2019. Augmenting Neural Networks with First-order Logic. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. 2018. Deepproblog: Neural probabilistic logic programming. In Advances in Neural Information Processing Systems.
Andre FT Martins, Mario AT Figueiredo, Pedro MQ Aguiar, Noah A Smith, and Eric P Xing. 2011. An Augmented Lagrangian Approach to Constrained MAP Inference. In Proceedings of the 28th International Conference on International Conference on Machine Learning.
Andre FT Martins, Noah A Smith, and Eric P Xing. 2009. Concise Integer Linear Programming Formulations for Dependency Parsing. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1.
Stephen Muggleton. 1995. Inverse Entailment and Progol. New Generation Computing.
Stephen Muggleton and Luc de Raedt. 1994. Inductive Logic Programming: Theory and Methods. The Journal of Logic Programming.
Vlad Niculae, Andre Martins, Mathieu Blondel, and Claire Cardie. 2018. SparseMAP: Differentiable Sparse Structured Inference. In International Conference on Machine Learning.
David Page and Ashwin Srinivasan. 2003. ILP: A Short Look Back and a Longer Look Forward. Journal of Machine Learning Research.
Xingyuan Pan and Vivek Srikumar. 2016. Expressiveness of Rectifier Networks. In Proceedings of the 33rd International Conference on Machine Learning.
J. R. Quinlan. 1986. Induction of Decision Trees. Machine Learning.
J. R. Quinlan. 1990. Learning Logical Definitions from Relations. Machine Learning.
Matthew Richardson and Pedro Domingos. 2006. Markov Logic Networks. Machine Learning.
Sebastian Riedel and James Clarke. 2006. Incremental Integer Linear Programming for Non-projective Dependency Parsing. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing.
Tim Rocktäschel and Sebastian Riedel. 2017. End-to-end differentiable proving. In Advances in Neural Information Processing Systems.
Dan Roth and Wen-tau Yih. 2004. A Linear Programming Formulation for Global Inference in Natural Language Tasks. Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004.
Dan Roth and Wen-tau Yih. 2005. Integer Linear Programming Inference for Conditional Random Fields. Proceedings of the 22nd International Conference on Machine Learning.
Alexander M Rush, David Sontag, Michael Collins, and Tommi Jaakkola. 2010. On dual decomposition and linear programming relaxations for natural language processing. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing.
Noah A. Smith. 2011. Linguistic Structure Prediction. Morgan & Claypool Publishers.
Noah A Smith and Jason Eisner. 2005. Contrastive Estimation: Training Log-Linear Models on Unlabeled Data. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop.
Ioannis Tsochantaridis, Thomas Hofmann, Thorsten Joachims, and Yasemin Altun. 2004. Support vector machine learning for interdependent and structured output spaces. Proceedings of the Twenty-First International Conference on Machine Learning.
Sam Wiseman and Alexander M Rush. 2016. Sequence-to-Sequence Learning as Beam-Search Optimization. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. 2018. A Semantic Loss Function for Deep Learning with Symbolic Knowledge. In Proceedings of the 35th International Conference on Machine Learning.
In this section we prove Theorem 1. The theorem and the relevant definitions are repeated here for convenience.
Define the rectifier (ReLU) activation function as  $\mathrm{R}(x) = \max(0, x)$ . Consider the following two-layer rectifier network:
$$
z (\mathbf {x}, \mathbf {y}) = \operatorname {s g n} \left(1 - \sum_ {k = 1} ^ {K} \mathrm {R} \left(\mathbf {w} _ {k} \cdot \psi (\mathbf {x}, \mathbf {y}) + b _ {k}\right)\right) \tag {10}
$$
The input to the network is still  $\psi (\mathbf{x},\mathbf{y})$ . There are  $K$  RLUs in the hidden layer, and one threshold unit in the output layer.
The decision boundary of this rectifier network is specified by a system of linear inequalities. In particular, we have the following theorem:
Theorem 2. Consider a two-layer rectifier network with  $K$  hidden ReLUs as in Eq. (10). Define the set  $[K] = \{1,2,\dots ,K\}$  . The network outputs  $z(\mathbf{x},\mathbf{y}) = 1$  if, and only if, for every subset  $\mathcal{S}$  of  $[K]$  , the following linear inequality holds:
$$
1 - \sum_ {k \in \mathcal {S}} \left(\mathbf {w} _ {k} \cdot \boldsymbol {\psi} (\mathbf {x}, \mathbf {y}) + b _ {k}\right) \geq 0
$$
Proof. Define  $a_{k} = \mathbf{w}_{k}\cdot \psi (\mathbf{x},\mathbf{y}) + b_{k}$ . We first prove the "if" part of the theorem. Suppose that for any  $\mathcal{S}\subseteq [K]$ ,  $1 - \sum_{k\in \mathcal{S}}a_k\geq 0$ . Thus for a specific subset  $\mathcal{S}^* = \{k\in [K]:a_k\geq 0\}$ , we have  $1 - \sum_{k\in \mathcal{S}^*}a_k\geq 0$ . By the definition of  $\mathcal{S}^*$ ,  $\sum_{k = 1}^{K}\mathrm{R}(a_k) = \sum_{k\in \mathcal{S}*}a_k$ , therefore  $1 - \sum_{k = 1}^{K}\mathrm{R}(a_k)\geq 0$ .
Next we prove the "only if" part of the theorem. Suppose that  $1 - \sum_{k=1}^{K} \mathrm{R}(a_k) \geq 0$ . For any  $\mathcal{S} \subseteq [K]$ , we have  $\sum_{k=1}^{K} \mathrm{R}(a_k) \geq \sum_{k \in \mathcal{S}} \mathrm{R}(a_k) \geq \sum_{k \in \mathcal{S}} a_k$ . Therefore, for any  $\mathcal{S} \subseteq [K]$ ,  $1 - \sum_{k \in \mathcal{S}} a_k \geq 0$ .
B Synthetic Integer Linear Programming Experiments
We first check if constraints are learnable, and whether learned constraints help a downstream task with a synthetic experiment. Consider framing structure prediction as an integer linear program (ILP):