Lazy-k Decoding: Constrained Decoding for Information Extraction
Authors: Arthur Hemmer
Mickael Coustaty
Nicola Bartolo
Jerome Brachat
Jean-marc Ogier
Venue: EMNLP 2023 Main (2023)
Keywords: structured prediction
constrained decoding
token classification
information extraction
beam search
TL;DR: We present a flexible constrained information extraction method combined with probabilistic models

Abstract: We explore the possibility of improving probabilistic models in structured prediction. Specifically, we combine the models with constrained decoding approaches in the context of token classification for information extraction. The decoding methods search for constraint-satisfying label-assignments while maximizing the total probability. To do this, we evaluate several existing approaches, as well as propose a novel decoding method called Lazy-$k$. Our findings demonstrate that constrained decoding approaches can significantly improve the models' performances, especially when using smaller models. The Lazy-$k$ approach allows for more flexibility between decoding time and accuracy. The code for using Lazy-$k$ decoding can be found at https://github.com/ArthurDevNL/lazyk.

Lazy-k: Decoding for Constrained Information Extraction
Arthur Hemmer", Mickaël Coustaty, Nicola Bartolo; Jérôme Brachat, Jean-Mare Ogier'
Shift Technology, Paris, France
L3i La Rochelle, France
{arthur. hemmer, nicola bar bartcoooggghhfff--eeccooooooooo [arthur. hemmer, /moostta,ttttmmmmm jmogateeennjoiiieeeuiuun---- fr jerome www.haatgtgiiilccom..
Abstract
Subtota Total
54. 400
Tolel 54.000
We explore the possibility of improving proba- bilistic models structured prediction. Specifi- cally, we combine the models with constrained decoding approaches in the context to- ken classification for information extraction. The decoding methods search for constraint- satisfying lahel-asigsmmnnts while maximizing the total probability. To do this, evaluate several existing approaches, as well as propose a novel decoding method called Lazy-k. Our findings demonstrate that constrained decod- ing approaches can significantly improve the models' performances, especially when using smaller models. The Lazy-k approach allows for more flexibility between decoding time and accuracy. The code for using Lazy-k decod- ing can be found here https:/ //gi thub. com/ trthrDeeNL//azyk.
SH
Total
556.00
Mentalian
Charge
2.000
Figure 1: Crop of sample from the CORD dataset along with the highest probability predictions for the amounts. The model incorrectly predicts Total for the Cash amount.
allow for defining constraints that evaluate whether produced prediction adheres to the expected struc- ture, which can then be used to iterate multiple high probability predictions until satisfying solu- tion found.
A concrete example of such structure is in the case of invoice information extraction. In this task, the model is given the outputs of Optical Character Recognition (OCR) system and needs predict which parts of the correspond the various elements in an invoice. For example, Fig. 1, the model is expected to predict the total, cash and change amounts.
1 Introduction
Much of today's Information Extraction (IE) is done using probability-basee models such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), LayouLLM (Xu et al.. 2020b,a; Huang al., 2022) LiLT (Wang et al., 2022). These models aim for the best results by increasingly stacking large amounts parameters, which comes at the cost of increased computational requirements and training complexity. Typically, only the top-1 prediction is used, despite the fact that models produce probabilities for all token- label combinations.
However, the occlusion of the "CASH" text intro- duces noise into the model's predictions, causing it to incorrectly label the cash amount as another total amount. Using the arithmetic semantics of invoices, we know that the total amount should equal the cash amount paid minus the change amount. As such, we know that the model's best prediction is prob- ably incorrect. Alternative, high-probability label- assignments be explored to find constraint- satisfying solution instead.
Ideally, alternative, high-likelihood predictions are explored to improve predictions from existing models. This is especially interesting in structured- prediction tasks, where the model's predictions are parsed into predefined structures. These structures
Industrial document processing systems usually

---

have programmatic post-processing logic that de- tects and sometimes corrects aforementioned se- mantic constraints. These systems, however, rarely exploit the remaining information "hidden" in the produced probability distributions, and the custom correction code is often complex and hard to main- tain. Furthermore there is the possibility of OCR- induced errors, which go beyond the scope of the present work but remain important source of errors in document IE (Nguyen al., 2022).
Note that for the method proposed in Sec. 4, we make no further assumption about the con- straint. This important because many existing constrained decoding approaches require the con- straints to be expressed in linear form (Faghihi et al., 2023).
Some problems may consist of both linear and non-linear constraints. loken-classification models often use the BIO labeling scheme (Ramshaw and Marcus, 1999), where the labels are prefixed with B(eginning), I(nside) and Ouutside) to be able to classify spans of multiple tokens. In this formula- tion, an I label must always be preceded by B or another I label of the same class.
In short, we exploit task-specific structures to explore alternative high- high-likelihood predictions. Specifically, we
propose an efficient algorithm for iterating over high-likelhhood predictions and,
This labeling constraint can be expressed using linear constraints. However, the solution to the linear constraints is not guaranteed to also be a so- lution to the non-linear constraints. The semantic constraint cash = total + change cannot be ex- pressed linearly because in order to compute it, the text corresponding to the labelization must be parsed from text float, which non-linear operation. An example where the optimal solution satisfying the linear (BIO) constraints does not sat- isfy the non-linear (semantic) constraints is shown in Tab. 1(b) one line 4.
provide proof for the correctness of the al- gorithm and,
perform several experiments to evaluate the relevance of exploring alternative high- likelihood predictions in structured prediction tasks.
2 Background
To search over high-probabilityy predictions, we require probabilistic model that outputs in- dependent probabilities for a given sequence of tokens. Given an input sequence x = (21,,2,...,, xy C X' where X' is the token vo- cabulary, the goal is to estimate the probability of the output sequence y (n,, ....... ..,..,,,,,,, J. E ) where ) is the label vocabulary. As this probability quickly becomes intractable, it is usually estimated by factoring it as:
Related Work
Several decoding methods for the setting from Eq. (3) have been proposed. An excellent bench- mark for learning and decoding under constraints is provided in GLUECons (Faghihi al., 2023). For decoding, the work mostly explores the usage of Integer Linear Programming (ILP) for finding a constraint-satisfyingg solution given the model's probabilities.
p(y|x)= II p(ul)))
(1)
ILP problems can be solved using the branch and bound algorithm (Land and Doig, 1960), which is type of informed search algorithm; uses a linear formulation of the constraints guide it more effectively through the search space, Another example a method that uses knowledge about its constraints is Viterbi (Forney, 1973), which a dynamic programming approach that can also take into account specific constraints, although more restrictive than ILP it only works in Markovian setting. The advantage of these informed search methods is that they will always find the optimal answer within reasonable amount of time, should exist. However, they also have a non -iglnegliiible minimum running time and impose aforementioned
The decoding process refers to the way we ob- tain an estimate ÿ for from such a model. The simplest approach consists of taking the arg max
arg max p(y|x),
(2)
yeya
which is done for each y/ separately.
In addition, we introduce global, binary con- straint C y + 10 and formalize our problem of interest
y = arg max p(ylx) y(x,y). y
(3)

---

requirements on the constraints.
Label
"56"
"M"
"000"
Informed search methods have previously been applied to the task of information extraction. De- coding under constraints using ILP was inspired by the work from Roth and Yih who explored the application to entity and relation extraction (Roth and Yih, 2004, 2007). They formulate the decod- ing problem as a linear program with the objective to maximize the overall probability of sequence given a set of constraints.
Btotal
0.3
Hotol
0.4
0.4
Boash
0.5
Loush
0.3
0.3
(a)
"56"
"
"000"
BIO
Sem.
8.0%
Boash
Leotl
Lottl
No
6.0%
Beash
Trotal
Teash
No
6.0%
Beash
Teash
Itotll
No
In these programs, the decision variables are indicator variables 12, indicating the assignment of label j to token i. Using this, one can express the constraint "I label per token" V4 Σ}= 1! where / is the number of possible labels. Using this linear formulation for several other constraints, they observe 2-5% improvements in Fi-score on entity and relation classification tasks. Similarly, Viterbi has also been used for correcting structured predictions (Douzon et al., 2022).
4.8%
Biotal
Tental
Isoal
Yes
No
4.5%
Boash
Teah
I cash
Yes
Yes
3.6%
Biotal
Liotal
Lassh
No
(b)
Table I: (a) (partial) predicted probabilities for the red bounding box in Fig. 1, where the model splits the string "56.000"
three tokens "56", "" and "000". (b) most likely label assignments in order of probability. BIO the labels adhere to the BIO constraints, Sem. = the labels adhere to semantic structure (cash total + change).
For more complex constraints we can use unin- formed search methods they do not make any as- sumptions on the implementation of the constraints and simply iterate over the search space in greedy manner. The most widely known method for this is Beam Search (BS) (Bisiani, 1987). While does not impose any restriction on the type of con- straints, it not ideal to our global decoding setting as works a "left-to-right" manner.
label-assige aasignmnstt might violate future structure con- straints (Pan and Srikumar. 2018).
Our method follows similar approach to A x with Partial Expansion (Yoshizumi et al., 2000) which has previously been applied to the multiple sequence alignment problem.
To illustrate, BS takes as input a parameter and outputs the top-k sequences by computing the top-k beams at every token, based on the previ- ous top-k beams. In order to evaluate global con- straints, beam search first needs to compute all top-k sequences after which the constraint can be evaluated.
To our knowledge, we are the first to apply A* with partial expansion that allows for more gen- eral constraints than ILP for the global constraint decoding setting.
Lazy-k Decoding
Unfortunately, this means that if the constraint- validating prediction ends up being the most likely (arg max) sequence, beam scarch will have com- puted h: too many sequences. In addition, if the constraint-validating prediction is not in the top- k beams, a new search with an unknown, higher k needs to be run, which also includes recom- puting the initial previous & predictions. Several adaptations have been suggested in the context of natural language generation (Anderson et al., 2016; Hokamp and Liu, 2017: Post and Vilar, 2018; Lemons al., 2022), but none of which solve aforementioned problems for global constraints. Others propose extending beam search with learn- able heuristics that try to predict whether given
As the name suggests, the Lazy-k decoder allows for decoding the k most probable sequences in lazy manner. This means that it only iterates over the necessary number of sequences and stops once satisfying solution is found. The hypothesis that this decoder explores is that the constraint- satisfying sequence is somewhere among the other high probability sequences.
To do this efficiently, we exploit the fact that the k-th most probable sequence is always within "edit- distance" I from one of the k more probable sequences. This follows from the independence of each label as shown in Eq. 1. We put "edit- distance" in quotes here because we use slightly more strict definition of diittdis that also takes

---

into account the order between the various label probabilities. More details about this can be found App. A.
Algorithm 1 Lazy-k Decoding
Require: Input sequence:
Require: Cost function & from Eq. 4
At its core, is a variant of best-first search (Rus- sel et al., 1994), where the model's predictions are used to determine the order in which the possible label assignments are explored. Each state repre- sents full label assignment y {311 .2..... .. Vn for all tokens (211, 22,,..,,, Fn). The cost g(y) of state is defined as follows:
Require: Binary constraint C, max iterations k
function LAZY-K(X,,)) C, k)
2: y <- arg millyey" g(y)
3:
if (xx,y)) then return y
4:
Il + MinHcap(
5:
frontier + {y :1}
AddNextBestyy'. H, frontier)
count <-
sly) Σ log ((u)x)
(4)
while H not empty and count < k: do
y' < H.P.PMinin)
10:
y <-NextBestyy' frontier)
We use y to denote the k-th lowest cost label assignment, and define the starting point yt as:
11:
C(x,y") = then return y AddNex(Best(y H, frontier)
12:
13:
AddNex(Bestty" H, frontier)
y arg min g(y) y
(5)
14: count +=
15:
end while
The algorithm for Lazy-k decoding is given in Alg. 1. It works by maintaining heap of the k best states, prioritized by the score of the next best unexplored state within 1 edit distance. The heap is initialized with the starting state y'. Upon explor- ing a state, it is tested against the constraint and returns directly if it is satisfied. the constraint is not satisfied, the heap is extended with the newly explored state and the priority score of the origi- nating state y" is updated to reflect the score of the next best unexplored state.
16: return Failure
17:
end function
18:
19: function ADDNEXTBESTTy'S H, frontier)
20: y NexxBBsstyy frontier)
21:
while y * null and y c frontier do
22:
frontierly") +=1
23:
yd -NextBest(y,, frontier) end while
24:
25:
if y null then
26:
frontierfy +-
Different from best-first search, upon exploring a state, we do not add all the children the heap. Instead, we only add the next best state y* and update the priority key for y' to be the score of the next best state within edit distance I. This signif- icantly reduces the size of the heap, as classical search implementation adds n possible children at every iteration, whereas in this case, the number of states in the heap most equal to the number of iterations. This heap-size reduction in turn trans- lates in better run time complexity as all following heap operations become cheaper.
H.Add(y', .(y"))
27:
28: end
29: end function
to only add next best states to the frontier that are not alrcady in there. Sce App. for the pseudo- code for the NextBest function.
Given that the algorithm iterates over the possi- ble sequences in decreasing order of probability, it is trivial to prove that will always find the opti- mal solution should it exist. In practice however, the combinatorial growth in the number of states quickly renders exhaustive search infeasible. To prevent this, an additional stopping condition is used where the iteration stops if no satisfying so- lution has been found after a fixed number of k iterations. One could also the stopping condi- tion according to a cumulative probability mass p or some other measure; we leave this exploration for future work.
The NextBest function takes as input a state y and the frontier. The frontier is a dictionary that holds the explored states and next best states for all explored states. The values are integers that keep track of the i-th best change for given state. If i (the number of tokens) then the function returns null there is no next best change within edit-distance for this state. As the next best state may already exist in the frontier, the NextBest function is wrapped in Ned to make sure

---

4.1 Complexity Analysis
Table 2: Datasets used in the experiments. APL = Aver- age Page Length, CSR Constraint Satisfaction Ratio
Assuming n tokens, i labels and the requested top- A sequences. The space complexity of Lazy-k is O(kn), since for every k-best state we add at most new state of size n to the heap.
Dataset
Pages
APL
CSR
CORD 1,000
46
72%
WildReceipt
1,740
147
70%
The time complexity is slightly less obvious. For the top-k states, the outer while loop will run for k iterations. Inside this loop, there are two sources of complexity:
Docile
7,372
353
88%
very similar, the datasets have slightly different la- bels and different levels of granularity which means each dataset has its specific set of constraints. The exact constraints for each dataset are de- tailed in App. C. The models for all datasets are trained using BIO labels, for which the initial con- straint is that label-assignment should be valid BIO sequence.
1. H .AddO) which occurs at most twice in AddNextBest()
2. NextBest() which occurs in the outer loop and twice in AddNextBest in another while loop.
The other constraints depend on the specific la- bels available in each dataset. However, pos- sible for some labels not to be present in every sample. As such, we distinguish between manda- tory fields (ic total amount) and optional fields (ie service fee, discount) which are considered 0 if not found. A mandatory fields will be considered empty if is not present in the predictions. As such, any constraint involving this field is not evaluated (or automatically considered as satisfied).
The A.ddd) operation adds an element to the heap which is of logarithmic complexity with re- spect to the size of the heap. Since the heap holds exactly our top-k states at each iteration, the com- plexity of this operation is equal to Σi= log log I log k! which, using Stirling's approx- imation, is equivalent OOk lo)k). k).
The NextBest(y, frontier) function returns the frontier(yj-th next best state within 1 edit dis- tance of y. When expanding a state for the first time, we compute sorted list of next-best edits in O(n n) time. Using this, every NextBest call for this state can be computed in constant time. For every state, NextBest() is called at most n times. As the sorting takes n log n time, the total time complexity of the algorithm becomes: Olk(log k + n n)).
For each dataset, we apply the constraints to all samples and filter out any that do not satisfy the constraints (show counts in table). From these samples,
60% for training and validation (split 80-20), and 40% for testing. The samples not satisfying the constraints are added the training set. We purposefully choose a large percentage for the test as the small train set provided sufficient performance and we mostly wish to evaluate the decoding. Having the larger test set allows us to reduce the variance in our measurements and make stronger conclusions.
5 Experiments
To evaluate the relevance of the Lazy-k decoder, we perform invoice information extraction several datasets. The aim of the task is extract various amounts from invoices such that they satisfy their expected arithmetic structure. For each dataset, we train a token classificctnnn model and generate pre- dictions for the test set. The predictions are then fed into the different decoding algorithms along with the constraints, and return the highest-probabilg sequence satisfying the constraints.
Evaluation Metric Our primary evaluation met- ric Fl is the product of the micro-F; score and the percentage of samples satisfying all the constraints. We chose this metric as it allows us to measure the balance between the extraction performance and constraint satisfaction. Our filtering procedure ensures that all the test samples can completely satisfy the constraints.
Data We evaluate the decoders on total of three datasets shown Tab. 2: CORD (Park et al., 2019), WildReceipt (Sun et al., 2021) and DocILE (Šimsa et al., 2023). While the constraints are semantically
Models For each dataset, we fine-tune a Lay- outLM (Xu al., 2020b) model for a total of 20 epochs with a batch-size of 32 and learning rate 0.001. The models were trained using a NVIDIA

---

A40 (48GB) and the inference was done on an In- tel(R) Xeo Gold 6258R CPU @ 2.70GHz.
depending on the dataset. should be noted that the difference in running time between the different datasets can be primarily attributed to the average page lengths per dataset as shown in Tab. 2.
Methods The Lazy-k performance is compared both BS and ILP, as well as an Argmax baseline and vanilla Best-First implementation. Since ILP does not support non-lincar constraints, we propose a Lazy-ILP variant that works similar to Lazy-k. This method iteratively looks for the highest proba- bility solution satisfying the linear constraints and checks if it also satisfies the non-linear constraints. If it does not, the previous optimal solution is ex- plicitly excluded by adding a new constraint and a new solve is started. In our implementations we use the Python PuLP" package for solving the ILP problems. We chose various values for l for each method, such that it would approximately give the same total rumning time on different datasets. BS and Best-First are evaluated on less values for k: because of their likeness with - (exhaustive search) but slower already.
5.1 Smaller Models
A stated advantage is the possibility of using smaller models in combination with the constrained decoding methods to improve their performance. We devised a second experiment similar to the first one, but where we train several smaller models to evaluate the additional benefit of using con- strained decoding approaches. The smaller pre- trained BERT models were provided as part of pa- per on the importance on pre-training compact mod- els (Ture al., 2019), and are tiny (4.4M parame- ters), mini (11.3M), small (29.1M) and medium (41.7M) respectively. We also fine-tune a BERT base model counting 110.1M parameters, As Lazy- k gives the same results as BS and Best-First search but more efficiently, we only compare Lazy-k to ILP.
Results The results for the LayoutLM model for the different datasets shown in Tab. 3. It can be seen that constrained decoding approaches can significantly improve the Ff with respect to the Argmax baseline. For WildReceipt, the Ff score almost doubles from 44.5% to 82.1% %. CORD sees a relatively smaller improvement in Ff by going from 81.2% to 94.9% in the best case.
The results are shown in Fig. 2. We ob- serve the added value of constrained decoding in- creasing as the model gets smaller. In the extreme case of BERT tiny, the F score of the Argmax approaches 0%, but is increased significantly when combined with ILP. However, this can partially be explained by our choice of measuring the Ff as the product between the Fj and satisfaction ratio. Though not shown in the figures, most of the in- crease in FY can be attributed to the satisfaction ratio.
Results
The Lazy-ILP decoder achieves relatively high Fi after only one iteration, whereas BS, Lazy-k and Best-First grow more gradually with respect to k. This can be explained by the fact that Lazy-ILP directly finds the first sequence satisfying the linear (BIO) constraints whereas the other approaches might need to iterate over multiple sequences to find the sequences satisfying the linear constraints. Lazy-k achieves the F of LaPy-ILP (k: 1) at 1 R 21l for CORD, l x 28 for WildReceipt, and 1 ~ 27 for DocILE.
As the models get smaller, ILP gains in advan- tage with respect Lazy-k when keeping the num- ber of iterations constant. This means that in many cases the top-8 linear (BIO) constraint-sale -eattriittsatisfying solutions are outside of the 214 highest probability habel-assignments We wonder whether training the network to better predict correct BIO sequences would improve the overall performance, but we leave this for future work explore.
Lazy-ILP's high minimum Ff score also comes at a non-negligible average decoding time. For the same F score as LLP (k = 2º), Lazy- k is around 38x faster for CORD, 144x faster for WildReceipt and 182x faster for DocILE. However, k grows, the running time for Lazy-k becomes more significant. As expected, BS is much slower than the other methods. Because of the higher running time, we cut off the computation at k = 25. For the same k, Lazy-k is 150-500 times faster
5.2 Discussion
In the context of information extraction from in- voices, Lazy-k can be viable approaches for con- strained decoding. While ILP has the advantage of exactly computing optimal solutions to the linear constraints, also comes at an important mini- mum run time cost. Depending on the "spacing" between the solutions to the linear and non-linear
PPul.P

---

CORD
WildReceipt
DocILE
Decoder
k
F1
Time (s)
FT
Time (s)
F1
Time (s)
Argmax
81.2
0.000 = 0.000
44.5
0,000 = 0.000
48.2
0.000 = 0.000
BS
2
83.9
0.002 0.000
50.4
0.005 0.000
51.6
0.010 = 0.000
22
86.2
0.006 0.000
53.8
0.015 0.001
54.7
0.030 = 0.000
23
89.0
0.017 #.001
58.9
0.047 0.002
56.9
0.110 = 0.000
24
90.7
0.055 0.001
60.7
0.169 0.003
58.4
0.430 = 0.001
2
91.4
- =
65.3
0.907 = 0.025
59.4
2.969 = 0.012
Best-Fiirt
2'
90.7
0,005
0.000
60.7
0.061 0.012
58.4
0.088 = 0.001
25
91.4
0,007 = 0.001
65.3
0.103 0.001
59.4
0.161 = 0.003
26
92.2
0.009 # 0.000
68.8
0.184 + 0.001
60.3
0.289 = 0.001
27
92.2
0.015 = 0.000
70.4
0.340 0.016
60.9
0.520 = 0.002
2
92.5
0.025 0.000
72.0
0.604 0.003
61.2
0.948 + 0.003
LaZy-ILP
2
93.5
16188
67.9
1.592 0.004
60.7
2.188 = 0.002
21
94.5
0.629 0.009
72.2
1,902 0.006
61.6
2.664 + 0.005
22
94.5
0.641:00.00
74.3
2.434
0.006
62.7
3,889 = 0.007
23
94.5
0.666 = 0.009
75.8
3.526
0.010
63.3
7.283 = 0.010
24
94.9
0.725 = 0.009
77.6
5.926 0.013
63.6
16.024 0.018
Lazy-k
2
91.4
0.002 = 0.000
65.3
0.007 0.000
59.4
0.005 = 0.000
26
92.2
=
68.8
0.011 0,000
60.3
0.007 = 0.000
27
92.2
0
70.4
0.018
60.9
0.012 0.000 0.000
29
92.5
73.7
0.056
61.8
0.037 # 0.000
21
93.9
0.003
77.1
0.004
62.4
0.127 = 0.003
213
93.9
0.046
79.5
0.620 0.014
63.3
0.439 + 0.005
2
93.9
0.168 0.004
81.2
2.212 + 0.014
63.5
1.580 + 0.005
216
93.9
0,333 + 0,006
82.1
4.155 + 0.013
63.8
3,013 + 0.009
Table 3; Results of constrained decoding on different datasets. Time (s) = average decoding time per page in seconds averaged over 10 runs.
constraints, Lazy-k might be more suited to the problem. Although not measured in our experi- ments, Lazy-k also has a more significant memory usage than ILP because it needs to keep all previous solutions in the heap.
For the setting discussed in this paper, beam search is not recommended because of the limita- tion discussed in Sec. 3. However, it remains valid in the autoregressive decoding setting as this is not supported with the other methods.
On the smaller models observe larger im- pact from constrained decoding approaches. We find these results promising for resources con- strained applications and from an ecological point of view. We are able to achieve similar perfor- mance with significantly lighter models and less computational resources.
Conclusion
In summary, we have introduced novel and effi- cient decoding method called Lazy-k that allows for decoding under global, hard constraints. When applied in the context of invoice information extrac- tion, Lazy-k: is faster than existing, greedy search methods and allows for more flexibility in trad- ing off computing time and extraction performance compared to ILP. In addition, the possibility of us- ing programmatic constraints directly makes Lazy- an easy to use off-the-sheef solution for applying corrections to probabilistic models in the context of structured predictions.
Besides the performance one should also take into account the ease of implementation of the dif- ferent methods. The Lazy-k decoder is "plug-and- play" as it does not need any conversion of the constraints. While the linear constraints used in this paper were fairly trivial to implement, more complex problems will require more complex lin- car formulation which can be costly to implement correctly.
Future work could explore the application to other structured- prediccioonniii problems with non-

---

tured predictions tasks. It may not be possible specify concrete constraints for some tasks. We did not explore the integration of soft constraints, which are constraints that can have a degree sat- isfaction instead of the binary values considered in this paper.
Argmax Lazy-ILP Lazy-
60%
22" 40%
Ethies Statement
0%
We have not identified any direct ethical concerns with the presented methods and experiments. On the contrary, we believe that our method improves the verifiability of probabilistic predictions which allows for better control over opaque probabilistic methods. Furthermore, tial for extracting more performance out of smaller models which reduces the overall energy consump- tion required for training and inference.
(a) DocILE
80%
have shown the poten-
60%
40%
20%
0%
7 Acknowledgement
medium
(b) CORD
This work was supported by the French government in the framework the France Relance program.
60%
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Guided open vocabulary im- age captioning with constrained beam search. arXiv preprint arXiv.1612.00570
40%
0%
R Bisiani. 1987. Beam search: Encyclopedia artifi- cial intelligence, se shapiro (ed.): 56-58.
Liny
(c) WildReceipt
Jacob Devlin, Ming-Was Weig Chang. Kenton Lee, and
Figure Ff scores for constrained decoding on smaller models. Lazy-ILP is limited to 8 iterations and Lazy-k to 14
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arxiv:1810.04805.
Thibault Douzon, Stefan Duffner. Christophe Garcia. and Jérémy Espinas. 2022. Improving informa- tion extraction on business documents with specific pre-training tasks. Document Analysis Systems: 15th IAPR International Workshop, DAS 2022, La Rochelle, France, May 22-25, 2022, Proceedings. pages 111-125. Springer.
linear constraints besides information extraction. Additionally, the improvements in extraction per- formance using the decoding methods are promis- ing, which could also be explored in semi- supervised learning settings. Another interesting direction to explore would be the combination of Lazy-k decoding with confidence calibration meth- ods such as temperature scaling.
Hossein Rajaby Faghihi, Aliakbar Nafar, Chen Zheng. Roshanak Mirzaee, Yue Zhang, Andrzej Uszok. Alexander Wan. Tanawan Premsri. Dan Roth, and Parisa Kordjamshidi. 2023. Gluecons: generic benchmark for learning under constraints. arXiv preprint arXiv:2302 2002.10914.
Limitations
Most methods presented in this paper only apply the independent bubba-irobyyilityy setting whereas much of today's work in NL.P uses the autoregres- sive, generative setting. Furthermore, the methods only apply to tasks that can be formulated as struc-
David Forney. 1973. The viterbi algorithm. Proceed- ings of the IEEE, 616):268--77.
Chris Hokamp and Qun Liu. 2017. Lexically
strained decoding sequence generation using grid beam search. arXiv preprint arXiv: 174.0070..

---

Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutimv3; Pre-traine Pre-training for doe- ument ai with unified text and image masking. In Proceedings of the 30th ACM International Confer- ence on Multimedia, pages 4083-4091.
Hongbin Sun, Zhanghui Kuang, Xiaoyu Yue. Chen- hao Lin, and Wayne Zhang. 2021. Spatial dual- modality graph reasoning for key information ex- traction. arXiv preprint 2tt:::103..444
Julia Turc, Ming- Mein Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint 1rXiv:1908.06 08962.
H. Land and A. G. Doig, 1960. An automatic method of solving discrete programming problems. Econo- metrica, -87)5009
Sofia Lemons, Carlos Linares López, Robert C Holte. and Wheeler Ruml. 2022. Beam search: Faster and monotonic, In Proceedings of the International Con- ference on Automated Planning and Scheduling, vol- ume 32, pages 222-230.
Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022. Lilt: A simple yet effective language-inde language-indepeneen layout transformer for structured document understanding. arXiv preprint arXiv:2202.1366
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang. Wanxiang Che, et al. 2020a. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint (Xiv:2012.14740
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1162..
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020b. Layoutim: Pre-training of text and layout for document image understanding, In Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, pages 1192-1200.
Thi Tuyet Hai Nguyen, Adam Jatowt, Mickaël Cous- taty, and Antoine Doucet. 2022. Survey of post- ocr processing approaches. ACM Comput. Surv., 54(6): 14:11:22:11124:33
Xingyuan Pan and Vivek Srikumar. 2018. Learning to speed up structured output prediction. In Inter- national Conference on Machine Learning, pages 4905-4005. PMLR.
Takayuki Yoshizumi, Teruhisa Miura, and Toru Ishida. 2000. A" with partial expansion for large branching factor problems. In AAAI/IAA,, pages 923-929.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jacheung Surh, Minjoon Seo, and Hwalsuk Lec. 2019. Cord: consolidated receipt dataset for post-oer parsing. In Workshop on Document Intelligence at NeurlPS 2019.
Lazy-k Proof
We denote a label sequence using
y = {31 .NN..
(6)
Matt Post and David Vilar. 2018. Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. arXiv preprint
When obtaining the predicted probabilities from the model, we order the probabilities for each label in strictly decreasing order from j --1 to j |2% such that
Lance Ramshaw and Mitchell P Marcus. 1999. Text chunking using cansformation-basel learning. Nat- ural language processing using very large corpora, pages 157-176.
plu (x) p(y!"" [x).
(7)
While, in theory, it is possible for two labels to have the same probability, in practice any exact degeneracy is lifted by the numerical noise. Such edge cases could be included by fixing an order ar- bitrarily without significant impact on the outcome. For sake of simplicity, however, we will keep the strict incquality in Eq.( Once the order is fixed for each label, the sequence can be unequivocally represented by the indices j: as
Dan Roth and Wen-tau Yih. 2004. linear program- ming formulation for global inference natural language tasks. Technical report, Illinois Univ at Sibana-Champpiin Dept of Computer Science.
Dan Roth and Wen-tau Yih. 2007. Global inference for entity and relation identification via linear pro- gramming formulation. Introduction to statistical relational learning, pages 553-580.
Stuart Russel, Peter Norvig, et al. 1994. Artificial intel- ligence modern approach. 20:90020-9.
y {ii,-. ... ..N].
(8)
Stěpán Šimsa, Milan Sulc, Michal Uneaáf Yash Pa- tel, Ahmed Hamdi, Matěj Kocián, Matyáš Skalický. Jifí Matas, Antoine Doucet, Mickaël Coustaty, et al. 2023. Docile benchmark for document informa- tion localization and extraction. arXiv preprint arXiv:2302.06558.
We can now define the distance between two sequences as
N (yst(y, yy) ) Σ
(9)

---

Note that, by the definitions above, each iteration of our lazy-k method corresponds to increasing by only one of the indices Ji of the sequence considered the previous iteration.
float. next field indicates that the field optional and thus considered false if no value is predicted for a given document.
C.1 CORD
Following the independence assumption be- tween the labels, the probability of a sequence is given by
menu.sub.price sub gaa__oota.subbtta__ppg
N P(y p()sxx)
talb_total.tx_ppae 10%
(10)
x (sub Suub_total.uuttota__po
+ sub__olalseerice_prrre
Similarly the individual labels, can order all the sequences y by an index &o = 1,... , such that
total.cashprice =
total.total + total. changeereeee
p(y*))) > P
(11)
total.total_prit
where we neglect degencrate probabilitics for the same argument raised above.
sub tab__tttalstbbotal_piic
+ sub sub_totall.ax_priic
The ordering assumptions given in Eqs.(7) and (11), together with the definition of the dis- tance (9) imply that
+ sub sub_tootal.eerviee_rrcc
sub__total.disoouuttppic
C.2
WildReceipt
WKEA' < Dist(yy y²) = (12)
If we assume that condition (12) is not satisfied, it would mean that starting from y* and decreasing by any of its ju the sequence probability would increase. But this can only happen if condition (7) is violated.
total, value subtotal value + tax_valll value
subtotal_valuc- Σ prod_pricc_val value
C.3 DocILE
amount sttls_tttallllllll gross
NextBest
amount_ net + amount_totaa tax
amount_dee due
Algorithm 2 NextBest Implementation
amount_paid amount aomount_totalgrron
Require: Label assignment:
Require:
1: function NEXTBEST(y, frontier)
2: if frontier[y] y. Y.Lentth then return
null
3: diff's + (log ++ log v/lld E y
Notation from Eq. 7
4: i ArgSSorddifsssffommieee]]]]]]]] b Cached
+ y/ 5: y
y - y
7: return y
8: end function
Constraints
Below the constraints used for each dataset. All models are trained using the BIO labeling scheme and as such, the correct BIO constraint is used for all datasets. In addition, each numerical field in has the constraint that it needs to be parseable to


Reviews:
[{"review_id": "rPPhZYMRQP", "reviewer": "Reviewer_AmBy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to review our paper. While you find the paper well-written and the overall structure to be correct, it seems that the primary reasons not to accept the paper are for a lack of diversity and depth in the experiments and analysis of the algorithm, coupled with a title that is too broad.\n\nWe acknowledge that the receipt/invoice information extraction task is one-sided perspective of the algorithm. Having experimented with other tasks and datasets, we would be open to add additional results. For this paper, we chose not to include them because:\n1) there are but few Information Extraction (IE) tasks that can benefit from the type of constraints that interest us;\n2) the IE tasks that *do* allow for using constraints often contain too few variables to show significant differences between methods. For example, in entity relation extraction, one can use a constraint where the entity relation `WorksFor(x,y)` can only be applied between entities of types `person` and `organization`. The available datasets (ie CoNLL ‘03) for this contain only few relations and entity types which makes the problem space small, exhaustively searchable and thus would not show significant differences between different methods;\n3) non-IE structured prediction tasks such as Sudoku are often solvable with linear constraints, making ILP the obvious choice as acknowledged in the paper.\n\nWe created additional proof for a better understanding of the algorithm (see below) which we will include in the paper.\n\nAdmittedly, regarding the title, we are not sure to understand the objection. Perhaps the “constrained” adjective is better placed before the IE, changing the title to “Lazy-k: Decoding for Constrained Information Extraction”. The method is indeed not applicable to generative methods which we do not allude to in the article.\n\nHopefully this clarifies any concerns you might have had with our work.\n\nKind regards,\n\nAuthors\n\n---\n\n**Lazy-k Proof**\n\nLike in the paper, we'll denote a label sequence using\n1. \\begin{equation}\n    \\mathbf{y} = \\{y_1,\\cdots, y_N\\}.\n\\end{equation}\n\nWhen obtaining the predicted probabilities from the model, we order the probabilities for each label $y_i$ in strictly decreasing order from $j=1$ to $j=|\\mathcal{Y}|$, such that\n\n2. \\begin{equation}\n    p(y_i^{j}|\\mathbf{x}) > p(y_i^{j+1}|\\mathbf{x}).\n\\end{equation}\n\nWhile, in theory, it is possible for two labels to have the same probability, in practice any exact degeneracy is lifted by the numerical noise. Such edge cases could be included by fixing an order arbitrarily without significant impact on the outcome. For sake of simplicity, however, we will keep the strict inequality in Eq.(2).\nOnce the order is fixed for each label, the sequence can be unequivocally represented by the indices $j_i$ as\n\n3. \\begin{equation}\n    \\mathbf{y} = \\{j_1,\\cdots, j_N\\}.\n\\end{equation}\n\nWe can now define the distance between two sequences as\n\n4. \\begin{equation}\n    \\mathrm{Dist}(\\mathbf{y},\\mathbf{y}') = \\sum_i^N j'_i - j_i.\n\\end{equation}\n\nNote that, by the definitions above, each iteration of our lazy-$k$ method corresponds to increasing by $1$ only one of the indices $j_i$ of the sequence considered in the previous iteration.\n\nFollowing the independence assumption between the labels, the probability of a sequence $\\mathbf{y}$ is given by\n\n5. \\begin{equation}\n    P(\\mathbf{y}|\\mathbf{x}) = \\prod_i^N p( y_i|\\mathbf{x}).\n\\end{equation}\n\nSimilarly to the individual labels, we can order all the sequences $\\mathbf{y}$ by an index $k=1,\\cdots,|\\mathcal{Y}|^N$ such that\n\n6. \\begin{equation}\n    P( \\mathbf{y}^{k}|\\mathbf{x}) > P(\\mathbf{y}^{k+1}|\\mathbf{x}),\n\\end{equation}\n\nwhere we neglect degenerate probabilities for the same argument raised above.\n\nThe ordering assumptions given in Eqs.(2) and (6), together with the definition of the distance (4) imply that\n\n7. \\begin{equation}\n    \\forall\\,k\\,\\exists\\,\n    k'< k\\,|\\,\n    \\mathrm{Dist}(\\mathbf{y}^{k'}, \\mathbf{y}^{k}) = 1.\n\\end{equation}\n\nIf we assume that condition (7) is not satisfied, it would mean that starting from $\\mathbf{y}^k$ and decreasing by 1 _any_ of its $j_i$ the sequence probability would increase. But this can only happen if condition (2) is violated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-organized and easy to follow. \n2. The motivation is reasonable and is supported by experiments.", "reasons_to_reject": "1. Lazy-k gives larger improvements on smaller models and smaller improvements on larger models, thus it is still unknown whether Lazy-k can improves SoTA models.\n2. The title is over-claimed, this paper only conduct experiments on a small range of information extraction tasks and the proposed method can only used for generation based tasks. \n3. The experiment section is weak, there should be more experiments or deeper analysis to help readers to better understand why and how the proposed method works.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies constrained decoding for structured prediction, introducing a Lazy-k decoding method which maximizes the total probability while satisfying constraints. Experiments show that 1) lazy-k gives better results than baselines on three datasets; 2) Lazy-k allows for more flexibility in trading off computing time and extraction performance compared to baseline.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IZ0HOrxSqs", "reviewer": "Reviewer_fWdW", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to review our paper. You note the added value of the method, but would like to see a more formal explanation for supporting the statement that the ‘k-th most probable sequence is always within edit distance 1 from one of the k - 1 more probable sequences’.\n\nThe small but important detail in this statement is that the k-th most probable sequence is not within distance 1 of the k-1 most probable sequence, but from distance 1 from at least one of the up-to-k-1 most probable sequences. After reviewing your feedback, we also noted that our usage of the term 'edit distance' is not quite correct as we intend it slightly stricter than what is conventionally meant. Typically, the edit distance between two sequences indicates the number of differing elements between two sequences, but in our case we also take into account the order of the individual elements. For example, if we consider an alphabetical order, the strings \"AA\" and \"AC\" are within edit distance 1 of each other, but in our case we count it as 2 edits as we need to go through B to turn A in to C (making it simply a distance measure instead of edit distance). As such, we will revise our paper to clarify this distinction. In addition, we have formalized the proof (see below) which we will add to the paper.\n\nHopefully this clarifies any concerns you might have had with our work.\n\nKind regards,\n\nAuthors\n\n---\n\n**Lazy-k Proof**\n\nLike in the paper, we'll denote a label sequence using\n1. \\begin{equation}\n    \\mathbf{y} = \\{y_1,\\cdots, y_N\\}.\n\\end{equation}\n\nWhen obtaining the predicted probabilities from the model, we order the probabilities for each label $y_i$ in strictly decreasing order from $j=1$ to $j=|\\mathcal{Y}|$, such that\n\n2. \\begin{equation}\n    p(y_i^{j}|\\mathbf{x}) > p(y_i^{j+1}|\\mathbf{x}).\n\\end{equation}\n\nWhile, in theory, it is possible for two labels to have the same probability, in practice any exact degeneracy is lifted by the numerical noise. Such edge cases could be included by fixing an order arbitrarily without significant impact on the outcome. For sake of simplicity, however, we will keep the strict inequality in Eq.(2).\nOnce the order is fixed for each label, the sequence can be unequivocally represented by the indices $j_i$ as\n\n3. \\begin{equation}\n    \\mathbf{y} = \\{j_1,\\cdots, j_N\\}.\n\\end{equation}\n\nWe can now define the distance between two sequences as\n\n4. \\begin{equation}\n    \\mathrm{Dist}(\\mathbf{y},\\mathbf{y}') = \\sum_i^N j'_i - j_i.\n\\end{equation}\n\nNote that, by the definitions above, each iteration of our lazy-$k$ method corresponds to increasing by $1$ only one of the indices $j_i$ of the sequence considered in the previous iteration.\n\nFollowing the independence assumption between the labels, the probability of a sequence $\\mathbf{y}$ is given by\n\n5. \\begin{equation}\n    P(\\mathbf{y}|\\mathbf{x}) = \\prod_i^N p( y_i|\\mathbf{x}).\n\\end{equation}\n\nSimilarly to the individual labels, we can order all the sequences $\\mathbf{y}$ by an index $k=1,\\cdots,|\\mathcal{Y}|^N$ such that\n\n6. \\begin{equation}\n    P( \\mathbf{y}^{k}|\\mathbf{x}) > P(\\mathbf{y}^{k+1}|\\mathbf{x}),\n\\end{equation}\n\nwhere we neglect degenerate probabilities for the same argument raised above.\n\nThe ordering assumptions given in Eqs.(2) and (6), together with the definition of the distance (4) imply that\n\n7. \\begin{equation}\n    \\forall\\,k\\,\\exists\\,\n    k'< k\\,|\\,\n    \\mathrm{Dist}(\\mathbf{y}^{k'}, \\mathbf{y}^{k}) = 1.\n\\end{equation}\n\nIf we assume that condition (7) is not satisfied, it would mean that starting from $\\mathbf{y}^k$ and decreasing by 1 _any_ of its $j_i$ the sequence probability would increase. But this can only happen if condition (2) is violated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "the acceptation of this work will potentially facilitate the practical application of the NLP techniques with global decoding. This work offers a unified modeling for handling task-specific hard non-linear constraints, simplifies the post-processing procedure compared to programmatic methods.", "reasons_to_reject": "Providing a simple mathematical explanation for the observation mentioned in line 214 would contribute to the clarity of this paper", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. This work introduces a novel and efficient decoding method for decoding under global, hard, non-linear constraints.\n2. For the invoice information extraction task, the proposed constrained decoding approach exploits the probability distributions information, has a unified modeling of hard constraints and is easy to maintain, comparing to conventional programmatic post-processing method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gl1jmAduyR", "reviewer": "Reviewer_NKjo", "summary": "", "questions": "- Table 2: it would also be good to report the page length distribution for each dataset.\n- Table 3: why the k for vanilla best-first search stops at $2^8$? Is it because $2^9$ exceeds the time limit?\n- Table 3: what does k mean in ILP? Does it means how many times the ILP retries with new constraints (Line 378)? If so, why does the computing time for CORD remain almost the same?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to review our paper. You note the ease-of-use of the algorithm along with the possible applications outside of the ones presented in the paper, which we appreciate very much. Your primary concern seems to be regarding the novelty of the method, and have several questions regarding the results and the algorithm implementation.\n\nRegarding the novelty: the references you have provided are relevant and we will make sure to add them to the paper. You correctly point out that, on its own, the Lazy-k method is an exhaustive search method with problem-specific optimizations, much like A* with partial expansion. But we want to stress that the intended novelty with respect to the provided references is in its combination with \"modern-day\" probabilistic methods that are used to provide a heuristic for exploring the search space. Previous works [1, 2] on constrained decoding for structured prediction/NLP mention search but refer mostly to the more inefficient variants (Viterbi, A*, beam search).\n\nAnswers to the other questions:\n\n1. **`NextBest` implementation**: to compute NextBest we first sort the possible labels for each token in order of probability (can be done beforehand). Using this, for a given sequence $\\mathbf{y}^i$, we look up all the next best probabilities for each token from the pre-sort. From this we can simply find the j-th best change in linear time or, as we suggest in the article, sort and cache the obtained array to access the j-th best change by index later. We left out these details for brevity, but will add a more detailed implementation of this function to the paper.\n2. **Page length distributions**: We will add them when revising the paper. As you’re probably alluding to, this is an important factor for the differences in decoding times between different datasets.\n3. **Max-k, best-first**: the time limit cut-off was primarily for ILP and Lazy-k. Due to the likeness between best-first and lazy-k, we did not find it relevant enough to include further results as it would not impact the findings.\n4. **ILP-k:** your understanding is correct. The k indicates how many times we consider resolving another ILP with the additional constraints that exclude previous, non-satisfying solutions. The important factors in the running time of ILP are the setup of the problem and the number of variables and constraints. As the number of variables and constraints varies among samples, the ILP problem needs to be \"reformulated\" for every sample which is costly. Once created, however, it is relatively fast to add a single additional constraint. In addition, ILP achieves a high initial satisfaction ratio after 1 iteration on CORD which results in a lower impact on the average decoding time. The lower average decoding time for CORD ILP(k=1) is due to variance. We will update the results by averaging additional runs to reduce the variance and these inconsistencies. The conclusions will remain the same however.\n\nHopefully this clarifies any concerns you might have had with our work.\n\nKind regards,\n\nAuthors\n\n\n[1] Faghihi, Hossein Rajaby, et al. \"Gluecons: A generic benchmark for learning under constraints.\" arXiv preprint arXiv:2302.10914 (2023).\n\n[2] Roth, Dan, and Wen-tau Yih. \"Global inference for entity and relation identification via a linear programming formulation.\" Introduction to statistical relational learning (2007): 553-580.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is a simple plug-and-play algorithm. Its application is not limited to structured prediction tasks with global constraints. It can also be applied in other structured prediction tasks like sequence labeling, and parsing.\n\n- In general the paper is well motivated and easy to follow.", "reasons_to_reject": "- Although the speedup is impressive, I feel the proposed method can be categorized as an incremental technical improvement. The idea of partial/lazy expansion for a search algorithm has been explored in various problems in literature. E.g., [1] [2].\n- There are some details not explained well. In particular, it is not clear how `NextBest` function is computed. Given the problem is modeled as a sequence labeling task, how to calculate the next best edits given a label sequence?\n\n[1] A* with Partial Expansion for large branching factor problems. Yoshizumi et. al., 2000\n\n[2] Depth-first iterative-deepening: An optimal admissible tree search. Korf 1985", "questions_to_authors": "- Table 2: it would also be good to report the page length distribution for each dataset.\n- Table 3: why the k for vanilla best-first search stops at $2^8$? Is it because $2^9$ exceeds the time limit?\n- Table 3: what does k mean in ILP? Does it means how many times the ILP retries with new constraints (Line 378)? If so, why does the computing time for CORD remain almost the same?", "paper_topic_and_main_contributions": "This paper presents a technique, Lazy-k decoding, which is based on best-first search with global, non-decomposable constraints, for the structured prediction tasks in information extraction. In the best-first search, when a new state is examined, instead of directly adding all of its children states into the heap, the proposed Lazy-k decoding maintains a separate list of data structures for each state’s children states, and lazily adds them into the heap. With this change, the heap size is guaranteed to be the number of steps, and the heap can be maintained with less computation. Empirical evaluations show that the proposed improvement significantly outperforms vanilla best-first search without loss of accuracy.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: the authors focus on decoding for structured problems using probabilistic models, using a lazy-k decoding algorithm to accommodate global constraints while maintaining efficiency. In the discussion the authors added a proof which further enhanced the soundness. While the experiments are conducted on information extraction tasks (the authors agreed to make the title more specific), it has potential of being useful for more tasks.

There are two things to improve, including discussions on search algorithms for traditional probabilistic models, and more discussions on the tradeoff between decoding effectiveness and model size.
Decision: Accept-Main