SlowGuess/ACL_2020_OCR huggingface dataset:
Xingyuan Pan, Maitrey Mehta, Vivek Srikumar
{xpan,maitrey,svivek}@cs.utah.edu
Various natural language processing tasks are structured prediction problems where outputs are constructed with multiple interdependent decisions. Past work has shown that domain knowledge, framed as constraints over the output space, can help improve predictive accuracy. However, designing good constraints often relies on domain expertise. In this paper, we study the problem of learning such constraints. We frame the problem as that of training a two-layer rectifier network to identify valid structures or substructures, and show a construction for converting a trained network into a system of linear constraints over the inference variables. Our experiments on several NLP tasks show that the learned constraints can improve the prediction accuracy, especially when the number of training examples is small.
In many natural language processing (NLP) tasks, the outputs are structures which can take the form of sequences, trees, or in general, labeled graphs. Predicting such output structures (e.g. Smith, 2011) involves assigning values to multiple interdependent variables. Certain joint assignments may be prohibited by constraints designed by domain experts. As a simple example, in the problem of extracting entities and relations from text, a constraint could disallow the relation "married to" between two entities if one of the entity is not a "person". It has been shown that carefully designed constraints can substantially improve model performance in various applications (e.g., Chang et al., 2012; Anzaroot et al., 2014), especially when the number of training examples is limited.
Designing constraints often requires task-specific manual effort. In this paper, we ask the question: can we use neural network methods to
automatically discover constraints from data, and use them to predict structured outputs? We provide a general framework for discovering constraints in the form of a system of linear inequalities over the output variables in a problem. These constraints can improve an already trained model, or be integrated into the learning process for global training.
A system of linear inequalities represents a bounded or unbounded convex polytope. We observe that such a system can be expressed as a two-layer threshold network, i.e., a network with one hidden layer of linear threshold units and an output layer with a single threshold unit. This two-layer threshold network will predict 1 or  $-1$  depending on whether the system of linear inequalities is satisfied or not. In principle, we could try to train such a threshold network to discover constraints. However, the zero-gradient nature of the threshold activation function prohibits using backpropagation for gradient-based learning.
Instead, in this paper, we show that a construction of a specific two-layer rectifier network represents linear inequality constraints. This network also contains a single linear threshold output unit, but in the hidden layer, it contains rectified linear units (ReLUs). Pan and Srikumar (2016) showed that a two-layer rectifier network constructed in such a way is equivalent to a threshold network, and represents the same set of linear inequalities as the threshold network with far fewer hidden units.
The linear constraints thus obtained can augment existing models in multiple ways. For example, if a problem is formulated as an integer program (e.g., Roth and Yih, 2004, 2005; Riedel and Clarke, 2006; Martins et al., 2009), the learned constraints will become additional linear inequalities, which can be used directly. Alternatively, a structure can be constructed using graph search (e.g., Collins and Roark, 2004; Daumé et al., 2009; Doppa et al., 2014; Chang et al., 2015; Wiseman and Rush,
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4843-4858 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2016), in which case the learned constraints can filter available actions during search-node expansions. Other inference techniques that extend Lagrangian Relaxation (Komodakis et al., 2007; Rush et al., 2010; Martins et al., 2011) can also employ the learned constraints. Essentially, the learned constraints can be combined with various existing models and inference techniques and the framework proposed in this paper can be viewed as a general approach to improve structured prediction.
We report experiments on three NLP tasks to verify the proposed idea. The first one is an entity and relation extraction task, in which we aim to label the entity candidates and identify relations between them. In this task, we show that the learned constraints can be used while training the model to improve prediction. We also show that the learned constraints in this domain can be interpreted in a way that is comparable to manually designed constraints. The second NLP task is to extract citation fields like authors, journals and date from a bibliography entry. We treat it as a sequence labeling problem and show that learned constraints can improve an existing first-order Markov model trained using a structured SVM method (Tsochantaridis et al., 2004). In the final experiment we consider chunking, i.e., shallow parsing, which is also a sequence labeling task. We train a BiLSTM-CRF model (Huang et al., 2015) on the training set with different sizes, and we show that learned constraints are particularly helpful when the number of training examples is small.
In summary, the contributions of this paper are:
1. We propose that rectifier networks can be used to represent and learn linear constraints for structured prediction problems.
2. In tasks such as entity and relation extraction, the learned constraints can exactly recover the manually designed constraints, and can be interpreted in a way similar to manually designed constraints.
3. When manually designed constraints are not available, we show via experiments that the learned constraints can improve the original model's performance, especially when the original model is trained with a small dataset. $^{1}$
2 Representing Constraints
In this section, we formally define structured prediction and constraints. In a structured prediction problem, we are given an input  $\mathbf{x}$  belonging to the instance space, such as sentences or images. The goal is to predict an output  $\mathbf{y} \in \mathcal{Y}_{\mathbf{x}}$ , where  $\mathcal{Y}_{\mathbf{x}}$  is the set of possible output structures for the input  $\mathbf{x}$ . The output  $\mathbf{y}$  have a predefined structure (e.g., trees, or labeled graphs), and the number of candidate structures in  $\mathcal{Y}_{\mathbf{x}}$  is usually large, i.e., exponential in the input size.
Inference in such problems can be framed as an optimization problem with a linear objective function:
$$
\mathbf {y} ^ {*} = \underset {\mathbf {y} \in \mathcal {Y} _ {\mathbf {x}}} {\operatorname {a r g m a x}} \boldsymbol {\alpha} \cdot \phi (\mathbf {x}, \mathbf {y}), \tag {1}
$$
where  $\phi (\mathbf{x},\mathbf{y})$  is a feature vector representation of the input-output pair  $(\mathbf{x},\mathbf{y})$  and  $\alpha$  are learned parameters. The feature representation  $\phi (\mathbf{x},\mathbf{y})$  can be designed by hand or learned using neural networks. The feasible set  $\mathcal{V}_{\mathbf{x}}$  is predefined and known for every  $\mathbf{x}$  at both learning and inference stages. The goal of learning is to find the best parameters  $\alpha$  (and, also perhaps the features  $\phi$  if we are training a neural network) using training data, and the goal of inference is to solve the above argmax problem given parameters  $\alpha$ .
In this paper, we seek to learn additional constraints from training examples  $\{(\mathbf{x},\mathbf{y})\}$ . Suppose we want to learn  $K$  constraints, and the  $k^{\mathrm{th}}$  one is some Boolean function:  $c_{k}(\mathbf{x},\mathbf{y}) = 1$  if  $(\mathbf{x},\mathbf{y})$  satisfies the  $k^{th}$  constraint, and  $c_{k}(\mathbf{x},\mathbf{y}) = -1$  if it does not. Then, the optimal structure  $\mathbf{y}^*$  is the solution to the following optimization problem:
$$
\max  _ {\mathbf {y} \in \mathcal {Y} _ {\mathbf {x}}} \boldsymbol {\alpha} \cdot \phi (\mathbf {x}, \mathbf {y}), \tag {2}
$$
subject to  $\forall k, c_{k}(\mathbf{x},\mathbf{y}) = 1$
We will show that such learned constraints aid prediction performance.
2.1 Constraints as Linear Inequalities
Boolean functions over inference variables may be expressed as linear inequalities over them (Roth and Yih, 2004). In this paper, we represent constraints as linear inequalities over some feature vector  $\psi (\mathbf{x},\mathbf{y})$  of a given input-output pair. The  $k^{\mathrm{th}}$  constraint  $c_{k}$  is equivalent to the linear inequality
$$
\mathbf {w} _ {k} \cdot \psi (\mathbf {x}, \mathbf {y}) + b _ {k} \geq 0, \tag {3}
$$
<sup>1</sup>The scripts for replaying the experiments are available at https://github.com/utahnlp/learning-constraints
2We use 1 to indicate true and -1 to indicate false.
whose weights  $\mathbf{w}_k$  and bias  $b_{k}$  are learned. A Boolean constraint is, thus, a linear threshold function,
$$
c _ {k} (\mathbf {x}, \mathbf {y}) = \operatorname {s g n} \left(\mathbf {w} _ {k} \cdot \boldsymbol {\psi} (\mathbf {x}, \mathbf {y}) + b _ {k}\right). \quad (4)
$$
Here,  $\operatorname{sgn}(\cdot)$  is the sign function:  $\operatorname{sgn}(x) = 1$  if  $x \geq 0$ , and  $-1$  otherwise.
The feature representations  $\psi (\mathbf{x},\mathbf{y})$  should not be confused with the original features  $\phi (\mathbf{x},\mathbf{y})$  used in the structured prediction model in Eq. (1) or (2). Hereafter, we refer to  $\psi (\mathbf{x},\mathbf{y})$  as constraint features. Constraint features should be general properties of inputs and outputs, since we want to learn domain-specific constraints over them. They are a design choice, and in our experiments, we will use common NLP features. In general, they could even be learned using a neural network. Given a constraint feature representation  $\psi (\cdot)$ , the goal is thus to learn the parameters  $\mathbf{w_k}$ 's and  $b_{k}$ 's for every constraint.
2.2 Constraints as Threshold Networks
For an input  $\mathbf{x}$ , we say the output  $\mathbf{y}$  is feasible if it satisfies constraints  $c_{k}$  for all  $k = 1, \ldots, K$ . We can define a Boolean variable  $z(\mathbf{x}, \mathbf{y})$  indicating whether  $\mathbf{y}$  is feasible with respect to the input  $\mathbf{x}$ :  $z(\mathbf{x}, \mathbf{y}) = c_{1}(\mathbf{x}, \mathbf{y}) \wedge \dots \wedge c_{K}(\mathbf{x}, \mathbf{y})$ . That is,  $z$  is a conjunction of all the Boolean functions corresponding to each constraint. Since conjunctions are linearly separable, we can rewrite  $z(\mathbf{x}, \mathbf{y})$  as a linear threshold function:
$$
z (\mathbf {x}, \mathbf {y}) = \operatorname {s g n} \left(1 - K + \sum_ {k = 1} ^ {K} c _ {k} (\mathbf {x}, \mathbf {y})\right). \tag {5}
$$
It is easy to see that  $z(\mathbf{x}, \mathbf{y}) = 1$  if, and only if, all  $c_k$ 's are 1—precisely the definition of a conjunction. Finally, we can plug Eq. (4) into Eq. (5):
$$
z = \operatorname {s g n} \left(1 - K + \sum_ {k = 1} ^ {K} \operatorname {s g n} \left(\mathbf {w} _ {k} \cdot \psi (\mathbf {x}, \mathbf {y}) + b _ {k}\right)\right) \tag {6}
$$
Observe that Eq. (6) is exactly a two-layer threshold neural network:  $\psi (\mathbf{x},\mathbf{y})$  is the input to the network; the hidden layer contains  $K$  linear threshold units with parameters  $\mathbf{w}_k$  and  $b_{k}$ ; the output layer has a single linear threshold unit. This neural network will predict 1 if the structure  $\mathbf{y}$  is feasible with respect to input  $\mathbf{x}$ , and  $-1$  if it is infeasible. In other words, constraints for structured prediction problems can be written as two-layer threshold
networks. One possible way to learn constraints is thus to learn the hidden layer parameters  $\mathbf{w}_k$  and  $b_{k}$ , with fixed output layer parameters. However, the neural network specified in Eq. (6) is not friendly to gradient-based learning; the  $\mathrm{sgn}(\cdot)$  function has zero gradients almost everywhere. To circumvent this, let us explore an alternative way of learning constraints using rectifier networks rather than threshold networks.
2.3 Constraints as Rectifier Networks
We saw in the previous section that a system of linear inequalities can be represented as a two-layer threshold network. In this section, we will see a special rectifier network that is equivalent to a system of linear inequalities, and whose parameters can be learned using backpropagation.
Denote the rectifier (ReLU) activation function as  $\mathrm{R}(x) = \max(0, x)$ . Consider the following two-layer rectifier network:
$$
z = \operatorname {s g n} \left(1 - \sum_ {k = 1} ^ {K} \mathrm {R} \left(\mathbf {w} _ {k} \cdot \psi (\mathbf {x}, \mathbf {y}) + b _ {k}\right)\right) \tag {7}
$$
The input to the network is still  $\psi (\mathbf{x},\mathbf{y})$ . There are  $K$  ReLUs in the hidden layer, and one threshold unit in the output layer. The decision boundary of this rectifier network is specified by a system of linear inequalities. In particular, we have the following theorem (Pan and Srikumar, 2016, Theorem 1):
Theorem 1. Consider a two-layer rectifier network with  $K$  hidden RLUs as in Eq. (7). Define the set  $[K] = \{1,2,\dots ,K\}$  . The network output  $z(\mathbf{x},\mathbf{y}) = 1$  if, and only if, for every subset  $S$  of  $[K]$  , the following linear inequality holds:
$$
1 - \sum_ {k \in \mathcal {S}} \left(\mathbf {w} _ {k} \cdot \psi (\mathbf {x}, \mathbf {y}) + b _ {k}\right) \geq 0 \tag {8}
$$
The proof of Theorem 1 is given in the supplementary material.
To illustrate the idea, we show a simple example rectifier network, and convert it to a system of linear inequalities using the theorem. The rectifier network contains two hidden RLUs  $(K = 2)$ :
$$
z = \operatorname {s g n} \left(1 - \operatorname {R} \left(\mathbf {w} _ {1} \cdot \boldsymbol {\psi} + b _ {1}\right) - \operatorname {R} \left(\mathbf {w} _ {2} \cdot \boldsymbol {\psi} + b _ {2}\right)\right)
$$
Our theorem says that  $z = 1$  if and only if the following four inequalities hold simultaneously, one
per subset of  $[K]$ :
$$
\left\{ \begin{array}{l} 1 \geq 0 \\ 1 - \left(\mathbf {w} _ {1} \cdot \boldsymbol {\psi} + b _ {1}\right) \geq 0 \\ 1 - \left(\mathbf {w} _ {2} \cdot \boldsymbol {\psi} + b _ {2}\right) \geq 0 \\ 1 - \left(\mathbf {w} _ {1} \cdot \boldsymbol {\psi} + b _ {1}\right) - \left(\mathbf {w} _ {2} \cdot \boldsymbol {\psi} + b _ {2}\right) \geq 0 \end{array} \right.
$$
The first inequality,  $1 \geq 0$ , corresponding to the empty subset of  $[K]$ , trivially holds. The rest are just linear inequalities over  $\psi$ .
In general,  $[K]$  has  $2^{K}$  subsets, and when  $\mathcal{S}$  is the empty set, inequality (8) is trivially true. The rectifier network in Eq. (7) thus predicts  $\mathbf{y}$  is a valid structure for  $\mathbf{x}$ , if a system of  $2^{K} - 1$  linear inequalities are satisfied. It is worth mentioning that even though the  $2^{K} - 1$  linear inequalities are constructed from a power set of  $K$  elements, it does not make them dependent on each other. With general choice of  $\mathbf{w}_k$  and  $b_{k}$ , these  $2^{K} - 1$  inequalities are linearly independent.
This establishes the fact that a two-layer rectifier network of the form of Eq. (7) can represent a system of linear inequality constraints for a structured prediction problem via the constraint feature function  $\psi$ .
3 Learning Constraints
In the previous section, we saw that both threshold and rectifier networks can represent a system of linear inequalities. We can either learn a threshold network (Eq. (6)) to obtain constraints as in (3), or we can learn a rectifier network (Eq. (7)) to obtain constraints as in (8). The latter offers two advantages. First, a rectifier network has non-trivial gradients, which facilitates gradient-based learning<sup>3</sup>. Second, since  $K$  RLUs can represent  $2^{K} - 1$  constraints, the rectifier network can express constraints more compactly with fewer hidden units.
We will train the parameters  $\mathbf{w}_k$ 's and  $b_{k}$ 's of the rectifier network in the supervised setting. First, we need to obtain positive and negative training examples. We assume that we have training data for a structured prediction task.
Positive examples can be directly obtained from the training data of the structured prediction prob
lem. For each training example  $(\mathbf{x},\mathbf{y})$  , we can apply constraint feature extractors to obtain positive examples of the form  $(\psi (\mathbf{x},\mathbf{y}), + 1)$
Negative examples can be generated in several ways; we use simple but effective approaches. We can slightly perturb a structure  $\mathbf{y}$  in a training example  $(\mathbf{x},\mathbf{y})$  to obtain a structure  $\mathbf{y}'$  that we assume to be invalid. Applying the constraint feature extractor to it gives a negative example  $(\psi (\mathbf{x},\mathbf{y}^{\prime}), - 1)$ . We also need to ensure that  $\psi (\mathbf{x},\mathbf{y}^{\prime})$  is indeed different from any positive example. Another approach is to perturb the feature vector  $\psi (\mathbf{x},\mathbf{y})$  directly, instead of perturbing the structure  $\mathbf{y}$ .
In our experiments in the subsequent sections, we will use both methods to generate negative examples, with detailed descriptions in the supplementary material. Despite their simplicity, we observed performance improvements. Exploring more sophisticated methods for perturbing structures or features (e.g., using techniques explored by Smith and Eisner (2005), or using adversarial learning (Goodfellow et al., 2014)) is a future research direction.
To verify whether constraints can be learned as described here, we performed a synthetic experiment where we randomly generate many integer linear program (ILP) instances with hidden shared constraints. The experiments show that constraints can indeed be recovered using only the solutions of the programs. Due to space constraints, details of this synthetic experiment are in the supplementary material. In the remainder of the paper we focus on three real NLP tasks.
4 Entity and Relation Extraction Experiments
In the task of entity and relation extraction, we are given a sentence with entity candidates. We seek to determine the type of each candidate, as in the following example (the labels are underlined):
[Organization Google LLC] is headquartered in [Location Mountain View, California].
We also want to determine directed relations between the entities. In the above example, the relation from "Google LLC" to "Mountain View, California" is OrgBasedIn, and the opposite direction is labeled NoRel, indicating there is no relation. This task requires predicting a directed
3The output threshold unit in the rectifier network will not cause any trouble in practice, because it can be replaced by sigmoid function during training. Our theorem still follows, as long as we interpret  $z(\mathbf{x},\mathbf{y}) = 1$  as  $\sigma (\mathbf{x},\mathbf{y})\geq 0.5$  and  $z(\mathbf{x},\mathbf{y}) = -1$  as  $\sigma (\mathbf{x},\mathbf{y}) < 0.5$ . We can still convert the rectifier network into a system of linear inequalities even if the output unit is the sigmoid unit.
graph and represents a typical structured prediction problem—we cannot make isolated entity and relation predictions.
Dataset and baseline: We use the dataset from (Roth and Yih, 2004). It contains 1441 sentences with labeled entities and relations. There are three possible entity types: Person, Location and Organization, and five possible relations: Kill, LiveIn, WorkFor, LocatedAt and OrgBasedIn. Additionally, there is a special entity label NoEnt meaning a text span is not an entity, and a special relation label NoRel indicating that two spans are unrelated.
We used  $70\%$  of the data for training and the remaining  $30\%$  for evaluation. We trained our baseline model using the integer linear program (ILP) formulation with the same set of features as in (Roth and Yih, 2004). The baseline system includes manually designed constraints from the original paper. An example of such a constraint is: if a relation label is WorkFor, the source entity must be labeled Person, and the target entity must be labeled Organization. For reference, the supplementary material lists the complete set of manually designed constraints.
We use three kinds of constraint features: (i) source-relation indicator, which looks at a given relation label and the label of its source entity; (ii) relation-target indicator, which looks at a relation label and the label of its target entity; and (iii) relation-relation indicator, which looks at a pair of entities and focuses on the two relation label, one in each direction. The details of the constraint features, negative examples and hyper-parameters are in the supplementary material.
We compared the performance of two ILP-based models, both trained in the presence of constraints with a structured SVM. One model was trained with manually designed constraints and the other used learned constraints. These models are compared in Table 1.
We manually inspected the learned constraints and discovered that they exactly recover the designed constraints, in the sense that the feasible output space is exactly the same regardless of whether we use designed or learned constraints. As an additional confirmation, we observed that when a model is trained with designed constraints and tested with learned constraints, we get the same model perfor
Performance Metric	Designed	Learned
entity F-1	84.1%	83.1%
relation F-1	41.5%	38.2%
Table 1: Comparison of performance on the entity and relation extraction task, between two ILP models, one trained with designed constraints (Designed) and one with learned constraints (Learned).
mance as when tested with designed constraints. Likewise, a model that is trained with learned constraints performs identically when tested with learned and designed constraints.
Below, we give one example of a learned constraint, and illustrate how to interpret such a constraint. (The complete list of learned constraints is in the supplementary material.) A learned constraint using the source-relation indicator features is
$$
\begin{array}{l} - 1. 9 8 x _ {1} + 3. 5 3 x _ {2} - 1. 9 0 x _ {3} + 0. 1 1 x _ {4} \\ + 2. 6 6 x _ {5} - 2. 8 4 x _ {6} - 2. 8 4 x _ {7} - 2. 8 4 x _ {8} \tag {9} \\ + 2. 5 8 x _ {9} + 0. 4 3 x _ {1 0} + 0. 3 2 \geq 0 \\ \end{array}
$$
where  $x_{1}$  through  $x_{10}$  are indicators for labels NoEnt, Person, Location, Organization, NoRel, Kill, LiveIn, WorkFor, LocatedAt, and OrgBasedIn, respectively. This constraint disallows a relation labeled as Kill having a source entity labeled as Location, because  $-1.90 - 2.84 + 0.32 < 0$ . Therefore, the constraint "Location cannot Kill" is captured in (9). In fact, it is straightforward to verify that the inequality in (9) captures many more constraints such as "NoEnt cannot LiveIn", "Location cannot LiveIn", "Organization cannot WorkFor", etc. A general method for interpreting learned constraints is a direction of future research.
Note that the metric numbers in Table 1 based on learned constraints are lower than those based on designed constraints. Since the feasible space is the same for both kinds of constraints, the performance difference is due to the randomness of the ILP solver picking different solutions with the same objective value. Therefore, the entity and relation experiments in this section demonstrate that our approach can recover the designed constraints and provide a way of interpreting these constraints.
5 Citation Field Extraction Experiments
In the citation field extraction task, the input is a citation entry. The goal is to identify spans corresponding to fields such as author, title, etc. In the example below, the labels are underlined:
[Author A.M.Turing.] [Title Computing machinery and intelligence. ] [Journal Mind,] [Volume 59,] [Pages 433-460.] [Date October, 1950.]