SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\min  _ {\mathbf {z} \in \{0, 1 \} ^ {n}} \sum_ {i} c _ {i} \cdot z _ {i}, \tag {11}
$$
subject to  $\sum_{i}A_{ki}z_{i}\geq b_{k},\quad k\in [m]$
The objective coefficient  $c_{i}$  denotes the cost of setting the variable  $z_{i}$  to 1 and the goal of prediction is to find a cost minimizing variable assignment subject to  $m$  linear constraints in (11). We randomly generate a hundred 50-dimensional ILP instances, all of which share a fixed set of random constraints. Each instance is thus defined by its objective coefficients. We reserve  $30\%$  of instances as test data. The goal is to learn the shared linear constraints in Eq. (11) from the training set.
We use the Gurobi Optimizer (Gurobi Optimization LLC, 2019) to solve all the ILP instances to obtain pairs  $\{(\mathbf{c},\mathbf{z})\}$ , where  $\mathbf{c}$  is the vector of objective coefficients and  $\mathbf{z}$  is the optimal solution. Each  $\mathbf{z}$  in this set is feasible, giving us positive examples  $(\mathbf{z}, + 1)$  for the constraint learning task.
Negative examples are generated as follows: Given a positive pair  $(\mathbf{c},\mathbf{z})$  described above, if the  $i^{\mathrm{th}}$  coefficient  $c_{i} > 0$  and the corresponding decision  $z_{i} = 1$ , construct  $\mathbf{z}'$  from  $\mathbf{z}$  by flipping the  $i^{\mathrm{th}}$  bit in  $\mathbf{z}$  from 1 to 0. Such a  $\mathbf{z}'$  is a negative example for the constraint learning task because  $\mathbf{z}'$  has a lower objective value than  $\mathbf{z}$ . Therefore, it violates at least one of the constraints in Eq. (11). Similarly, if  $c_{i} < 0$  and  $z_{i} = 0$ , we can flip the  $i^{\mathrm{th}}$  bit from 0 to 1. We perform the above steps for every coefficient of every example in the training set to generate a set of negative examples  $\{(\mathbf{z}', -1)\}$ .
We trained a rectifier network on these examples and converted the resulting parameters into a system of linear inequalities using Theorem 2. The hyper-parameters and design choices are summarized in the supplementary material. We used the learned inequalities to replace the original constraints to obtain predicted solutions. We evaluated these predicted solutions against the oracle solutions (i.e., based on the original constraints). We
also computed a baseline solution for each test example by minimizing an unconstrained objective.
Table 5 lists four measures of the effectiveness of learned constraints. First, we want to know whether the learned rectifier network can correctly predict the synthetically generated positive and negative examples. The binary classification accuracies are listed in the first row. The second row lists the bitwise accuracies of the predicted solutions based on learned constraints, compared with the gold solutions. We see that the accuracy values of the solutions based on learned constraints are in the range from  $80.2 - 83.5\%$ . As a comparison, without using any constraints, the accuracy of the baseline is  $56.8\%$ . Therefore the learned constraints can substantially improve the prediction accuracy in the down stream inference tasks. The third row lists the percentage of the predicted solutions satisfying the original constraints. Solutions based on learned constraints satisfy  $69.8 - 74.4\%$  of the original constraints. In contrast, the baseline solutions satisfy  $55.3\%$  of the original constraints. The last row lists the percentage of the gold solutions satisfying the learned constraints. We see that the gold solutions almost always satisfy the learned constraints.
The hyper-parameter and other design choices for the synthetic ILP experiments are listed in Table 6.
C Entity and relation extraction experiments
C.1 Designed constraints
Table 7 lists the designed constraints used in the entity and relation extraction experiments. There are fifteen constraints, three for each relation type. For example, the last row in Table 7 means that the relation OrgBasedIn must have an Organization as its source entity and a Location as its target entity, and the relation in the opposite direction must be NoRel1.
C.2 Constraint features
We use the same example as in the main paper to illustrate the constraint features used in the entity and relation extraction experiments:
[Organization Google LLC] is headquartered in [Location Mountain View, California, USA].
In the above example, the relation from "Google LLC" to "Mountain View, California, USA" is OrgBasedIn, and the relation in the opposite direction is labeled NoRel, indicating there is no relation from "Mountain View, California, USA" to "Google LLC".
We used three constraint features for this task, explained as follows.
Source-relation indicator This feature looks at a given relation label and the label of its source entity. It is an indicator pair (source label, relation label). Our example sentence will contribute the following two feature vectors, (Organization, OrgBasedIn) and (Location, NoRel), both corresponding to positive examples. The negative examples contains all possible pairs of (source label, relation label), which do not appear in the positive example set.
Relation-target indicator This feature looks at a given relation label the label of its target entity. It is an indicator pair (relation label, target label). Our example sentence will contribute the following two feature vectors, (OrgBasedIn, Location) and (NoRel, Organization), both corresponding to positive examples. The negative examples contain all possible pairs of (relation label, target label), which do not appear in the positive example set.
Relation-relation indicator This feature looks at a pair of entities and focuses on the two relation labels between them, one in each direction. Therefore our running example will give us two positive examples with features (OrgBasedIn, NoRel) and (NoRel,OrgBasedIn). The negative examples contain any pair of relation labels that is not seen in the positive example set.
C.3 Hyper-parameters and design choices
The hyper-parameter and design choices for the experiments are in Table 8. Note that different runs of the SVM learner with the learned or designed constraints may give different results from those on Table 1. This is due to non-determinism introduced by hardware and different versions of the Gurobi solver picking different solutions that have the same objective value. In the results in Table 1, we show the results where the training with learned constraints seem to underperform the model that is trained with designed constraints. In other runs on different hardware, we found the opposite ordering of the results.
Number of RLUs
2	3	4	5	6	7	8	9	10
binary classification acc. (%)	85.1	87.3	92.1	90.3	95.0	94.3	94.1	97.7	98.0
bitwise solution acc. (%)	81.1	80.9	81.9	80.2	81.0	82.3	81.1	83.2	83.5
original constr. satisfied (%)	70.3	69.8	72.7	70.4	70.1	71.1	71.4	74.4	74.3
learned constr. satisfied (%)	95.6	98.6	98.7	99.1	97.4	98.9	99.9	99.1	99.4
Table 5: Effectiveness of learned constraints for the synthetic ILP experiments.
Description	Value
Total number of examples	100
Number of training examples	70
Number of test examples	30
Dimensionality	50
Range of hidden ReLU units considered for experiments	2-10
Learning rates for cross-validation while learning rectifier networks	{0.001, 0.01, 0.1}
Learning rate decay for cross-validation	{0.0, 10-7, 10-6}
Optimizer parameters for learning	β1 = 0.9, β2 = 0.999, ε = 10-7
Number of training epochs	1000
Table 6: Parameters used in the synthetic ILP experiments
Antecedents	Consequents
If the relation is	Source must be	Target must be	Reversed relation must be
Kill	Person	Person	NoRel
LiveIn	Person	Location	NoRel
WorkFor	Person	Organization	NoRel
LocatedAt	Location	Location	NoRel
OrgBasedIn	Organization	Location	NoRel
Table 7: Designed constraints used in the entity and relation extraction experiments
Description	Value
Structured SVM trade-off parameter for the base model	2-6
Number of hidden ReLU units
-for source-relation	2
-for relation-target	2
-for relation-relation	1
Learning rates for cross-validation while learning rectifier networks	{0.001, 0.01, 0.1}
Learning rate decay for cross-validation	{0.0, 10-7, 10-6}
Optimizer parameters for learning	β1 = 0.9, β2 = 0.999, ε = 10-7
Table 8: Parameters used in the entity and relation extraction experiments
C.4 Learned Constraints
We see in the main paper that  $2^{K} - 1$  linear inequality constraints are learned using a rectifier network with  $K$  hidden units. In the entity and relation extraction experiments, we use two hidden units to learn three constraints from the source-relation indicator features. The three learned constraints are listed in Table 9. A given pair of source label and relation label satisfies the constraint if the sum of the corresponding coefficients and the bias term is greater than or equal to zero. For example, the constraint from the first row in Table 9 disallows the pair (Location, Kill), because  $-1.90 - 2.84 + 0.32 < 0$ . Therefore, the learned constraint would not allow the source entity of a Kill relation to be a Location, which agrees with the designed constraints.
We enumerated all possible pairs of source label and relation label and found that the learned constraints always agree with the designed constraints in the following sense: whenever a pair of source label and relation label satisfies the designed constraints, it also satisfies all three learned constraints, and whenever a pair of source label and relation label is disallowed by the designed constraints, it violates at least one of the learned constraints. Therefore, our method of constraint learning exactly recovers the designed constraints.
We also use two hidden units to learn three constraints from the relation-target indicator features, and one hidden unit to learn one constraint from the relation-relation indicator features. The learned constraints are listed in Table 11 and Table 10. Again we verify that the learned constraints exactly recover the designed constraints in all cases.
D Citation field extraction experiments
D.1 Constraint Features
We use the same example as in the main paper to illustrate the constraint features used in the citation field extraction experiments:
We explore multiple simple constraint features  $\psi (\mathbf{x},\mathbf{y})$  as described below.
Label existence This features indicates which labels exist in a citation entry. In our above example, there are six labels. Suppose there are  $n_l$  possible labels. The above example is a positive example, the feature vector of which is an  $n_l$ -dimensional binary vector. Exactly six elements, corresponding to the six labels in the example, have the value 1 and all others have the value 0. To obtain the negative examples, we iterate through every positive example and flip one bit of its feature vector. If the resulting vector is not seen in the positive set it will be a negative example.
Label counts Label-count features are similar to Label-existence features. Instead of indicating whether a label exists using 1 or 0, label-count features records the number of times each label appears in the citation entry. The positive examples can be generated naturally from the training set. To generate negative examples, we perturb the actual labels of a positive example, as opposed to its feature vector. We then extract the label counts feature from the perturbed example, and treat it as negative if it has not seen before in the positive set.
Bigram labels This feature considers each pair of adjacent labels in the text. From left to right, the above example will give us feature vectors like (Author, Author), (Author, Title), (Title, Title), ..., (Date, Date). We then use one-hot encoding to represent these features, which is the input vector to the rectifier network. All these feature vectors are labeled as positive (+1) by the rectifier network, since they are generated from the training set. To generate negative examples for bigram-label features, we generate all positive examples from the training set, then enumerate all possible pair of labels and select those that were not seen in the positive examples.
Trigram labels This feature is similar to the bigram labels. From the training set, we generate positive examples, e.g., (Author, Author, Author), (Author, Author, Title) etc, and convert them into one-hot encodings. For negative examples, we enumerate all possible trigram labels, and select those trigrams as negative if two conditions are met: (a) the trigram is not seen in the positive set; and (b) a bigram contained in it is seen in the training set. The intuition is that we want negative examples to be almost feasible.
Source Labels	Relation Labels	Bias
NoEnt	Per.	Loc.	Org.	NoRel	Kill	Live	Work	Located	Based
-1.98	3.53	-1.90	0.11	2.66	-2.84	-2.84	-2.84	2.58	0.43	0.32
-1.61	-1.48	3.50	0.92	1.15	1.02	1.02	1.02	-3.96	-1.38	1.46
-3.59	2.04	1.60	1.03	3.81	-1.82	-1.82	-1.82	-1.38	-0.95	0.78
Table 9: Linear constraint coefficients learned from the source-relation indicator features
Forward Relation Labels	Backward Relation Labels	Bias
4.95	-1.65	-1.65	-1.65	-1.65	-1.65	5.06	-1.53	-1.53	-1.53	-1.53	-1.53	-2.41
Table 10: Linear constraint coefficients learned from the relation-relation indicator features. The order of the relation labels is: NoRel, Kill, LiveIn, WorkFor, LocatedAt, and OrgBasedIn
Part-of-speech For a fixed window size, we extract part-of-speech tags and the corresponding labels, and use the combination as our constraint features. For example, with window size two, we get indicators for  $(\text{tag}_{i-1}, \text{tag}_i, \text{label}_{i-1}, \text{label}_i)$  for the  $i^{\text{th}}$  token in the sentence, where tag and label refer to part-of-speech tag and citation field label respectively. For negative examples, we enumerate all four-tuples as above, and select it as negative if the four-tuple is not seen in the positive set, but both  $(\text{tag}_{i-1}, \text{tag}_i)$  and  $(\text{label}_{i-1}, \text{label}_i)$  are seen in the training set.
Punctuation The punctuation feature is similar to the part-of-speech feature. Instead of the POS tag, we use an indicator for whether the current token is a punctuation.
D.2 Hyper-parameters and design choices
The hyper-parameter and design choices for the experiments are in the Table 12.
E Chunking Experiments
E.1 Constraint Features
The two constraints which we discussed in the main paper for the chunking dataset are described below.
N-gram label existence This constraint is a general form of the label existence constraint mentioned in Section D.1. In fact, it is the n-gram label existence constraint with  $n = 1$ . The n-gram label existence constraint represents the labels of a sequence as a binary vector. Each feature of this binary vector corresponds to an n-gram label combination. Hence, the length of this constraint feature will be  $|l|^n$  where  $|l|$  is the total number of distinct labels. This means the vector size of
this constraint grows exponentially with increasing  $n$ . The binary vector indicates a value of 1 for all the n-gram label features present in the sequence tags. The positive examples are hence formed from the training set sequences. For the negative examples, we iterate through each positive example and flip a bit. The resulting vector is incorporated as a negative example if it doesn't occur in the training set.
N-gram part of speech (POS) This constraint is a general form of the part of speech constraint mentioned in Section D.1. POS tags of a token are converted to a indicator vector. We concatenate the indicator vectors of each gram in an n-gram in order and this vector is further concatenated with indicators of labels of each of these grams. Hence, for  $n = 2$ , we get the constraint vector as  $(\mathrm{tag}_{i - 1}, \mathrm{tag}_i, \mathrm{label}_{i - 1}, \mathrm{label}_i)$  where  $\mathrm{tag}_i$  and  $\mathrm{label}_i$  are indicators for POS tags and labels respectively for the  $i^{th}$  token. The positive examples enumerate vectors for all existing n-grams in the training sequences. The negative examples are creating by changing a label indicator in the constraint feature. The label to be perturbed and the perturbation both are chosen at random. The constraint vector hence formed is incorporated as a negative example if it doesn't occur in the set of positive examples.
E.2 Hyper-parameters and design choices
The hyper-parameter and design choices are summarized in Table 13.
Relation Labels	Target Labels
NoRel	Kill	Live	Work	Located	Based	NoEnt	Per.	Loc.	Org.	Bias
2.68	-3.17	-0.55	2.68	-0.55	-0.55	-1.58	3.15	0.53	-2.70	1.02
2.72	2.42	-1.39	-2.55	-1.39	-1.39	-2.51	-2.27	1.54	2.31	0.85
5.40	-0.74	-1.94	0.13	-1.94	-1.94	-4.10	0.88	2.08	-0.39	0.86
Table 11: Linear constraint coefficients learned from the relation-target indicator features
Description	Value
Structured SVM trade-off parameter for the base model	unregularized
Beam size	50
Number of hidden ReLU units for experiments	10
Learning rates for cross-validation while learning rectifier networks	{0.001, 0.01, 0.1}
Learning rate decay for cross-validation	{0.0, 10-7, 10-6}
Optimizer parameters for learning	β1 = 0.9, β2 = 0.999, ε = 10-7
Table 12: Parameters used in the citation field extraction experiments
Description	Value
Constraint Rectifier Network
Range of hidden ReLU units considered for experiments	{5,10}
Learning rates for development while learning rectifier networks	{0.001,0.005,0.01,10-4}
Number of training epochs	1000
Random Seeds	{1,2}
BiLSTM CRF Model
Learning rate for development while learning baseline model	{0.01,0.05,0.001,0.005}
Learning Rate Decay	{10-5,10-6}
Beam Size	10
Number of training epochs	300
Table 13: Parameters used in the chunking experiments