Selective Labeling: How to Radically Lower Data-Labeling Costs for Document Extraction Models
Authors: Yichao Zhou
James Bradley Wendt
Navneet Potti
Jing Xie
Sandeep Tata
Venue: EMNLP 2023 Main (2023)
Keywords: information extraction
selective labeling
data efficiency
annotation efficiency
TL;DR: We propose Selective Labeling as a solution to reduce the cost of acquiring labeled data by 10X while achieving negligible loss in document extraction performance.

Abstract: Building automatic extraction models for visually rich documents like invoices, receipts, bills, tax forms, etc. has received significant attention lately. A key bottleneck in developing extraction models for new document types is the cost of acquiring the several thousand high-quality labeled documents that are needed to train a model with acceptable accuracy. In this paper, we propose selective labeling as a solution to this problem. The key insight is to simplify the labeling task to provide “yes/no” labels for candidate extractions predicted by a model trained on partially labeled documents. We combine this with a custom active learning strategy to find the predictions that the model is most uncertain about. We show through experiments on document types drawn from 3 different domains that selective labeling can reduce the cost of acquiring labeled data by 10× with a negligible loss in accuracy.

Selective Labeling: How to Radically Lower Data-Labeling Costs for Document Extraction Models
Yichao Zhou, James Bradley Wendt, Navneet Potti, Jing Xie, Sandeep Tata
Google Research Mountain View, USA
(yichaojoey, jwendt, navsan, lucyxie, tata)@google. com
Abstract
of the target document type. The annotation task can be tedious and cognitively taxing, requiring the annotator to identify and draw bounding boxes around dozens of target fields in each document.
Building automatic extraction models vi- sually rich documents like invoices, receipts, bills, tax forms, etc. has received significant at- tention lately. A key bottleneck in developing extraction models for new document types is the cost of acquiring the several thousand high- quality labeled documents that are needed to train model with acceptable accuracy. In this paper, we propose selective labeling as a so- lution this problem. The key insight is to simplify the labeling task to provide "yes/no" lahels for candidate extractions predicted by model trained partially labeled documents. We combine this with a custom active learning strategy to find the predictions that the model is most uncertain about. We show through ex- periments on document types drawn from different domains that selective laheling re- duce the cost of acquiring labeled data by 10x with a negligible loss in accuracy.
This data efficiency requirement has not gone un- noticed in the research literature on this topic. How- ever, even with model pre-training (Xu et al., 2020, 2021), transfer learning from out-of-domain la- beled corpus (Torre and Shavlik, 2010; Nguyen et al.. 2019), and data-programmiin (Ratner et al., 2017: Zhang al., 2022), empirical evidence sug- gests that performing well on a new target docu- ment type like procurement, banking, insurance, mortgage, etc. still requires thousands of anno- tated documents, amounting to hundreds of hours of human labor (Zhang, 2021).
The cost of acquiring high quality labeled data for hundreds of document types is prohibitively expensive and is currently key bottleneck. We could apply active learning strategies to select few but informative documents for human review (Set- tles, 2009), however the cost-reducing effect of this approach is limited, as requires annotating the span in every selected document for every field. Many of these annotations are repetitive, and of- ten not very informative if model can already extract those fields easily. In fact, our initial ex- periments with document-level active learning approach yielded modest results that cut down the number of documents required get to the same level of quality as random selection by approxi- mately 20%. this paper, we propose a technique called selective labeling that reduces this cost by 10x. The key insight is to combine two ideas: First, we redefine and simplify the task performed by the human annotators rather than labeling every tar- get field in every document by drawing bounding boxes around their values, we ask them simply verify whether proposed bounding box is cor- rect. This binary "yes/no" annotation task is faster and imposes a lighter cognitive burden on the an-
1 Introduction
Visually rich documents such as invoices, receipts, paystubs, insurance statements, tax forms, ctc. are pervasive in business workflows. The tedious and error-prone nature of these workflows has led to much recent research into machine learning meth- ods for automatically extracting structured infor- mation from such documents (Lee al., 2022; Gar- nearek al., 2021; Xu et al., 2021; Tata et al., 2021; Wu al., 2018; Sarkhel and Nandi, 2019). Given a target document type with an associated set of fields of interest, as well as a set of human- annotated training documents, these systems learn to automatically extract the values for these fields from documents with unseen layouts.
A critical hurdle in the development of high- quality extraction systems is the large cost of ac- quiring and annotating training documents belong- ing to the target types. The human annotators often require training not only on the use of the annota- tion tools but also on the definitions and semantics

---

used to dramatically cut down the primary bottle- neck in developing extraction models for visually rich documents.
INVOICE
Irvelie yuanberr
Inveice
2019061801
Number
INITECH
2
Background
compery
address billing
POOME Corporation
We first describe how typical annotation task is up acquire labeled documents. We point out two major deficiencies with this approach before outlining an alternative that takes advantage of the characteristics this domain. We then describe the assumptions underlying our approach.
jumbargh @imlesh.comm
123 Amiil Dr
Mountain View CA 94040
suppler
Description
Code
Price
111
TP5 Peport
10:00
$:30.0
Accounting
Total:
$70.00
Isc_anounk7
peyable amegg Tax 10%
$7.00
2.1
Annotation Workflow
Total Payable: $77.00
payment dae
All pocyreenta by
2012
Paymeerds made after will
2.1.1
Classic Annotation Workflow
decare aroic
sidoriinnt contanmed the instto desiaratio connect
Given a document type for which we want to learn an extraction model, we begin by listing out the fields that we want to extract, along with human- readable descriptions, viz., "labeling instructions". We provide these instructions human annotators and present them with various document images to label. The classic annotation task is to draw a bounding box around each instance of any of the target fields and label it with the corresponding field name (Figure Red Remarks). Typical docu- ment types like paystubs have dozens of fields, and each document may contain multiple pages.
Figure classic annotation task: even labeling fields in this toy invoice imposes heavy cognitive bur- den on the annotator, while real-world documents are significantly more complicated (Red). "yes/no" an- notation task: presenting proposed span and asking the annotator to accept or reject the label is simpler, quicker, and less prone errors (Green).
notator (Boim et al., 2012; Blog, 2020; Ganchev et al., 2007: Skeppstedt et al., 2017). Second, we adapt existing active learning strategies select the examples (i.e., candidate extraction spans) that the model is most uncertain in each round anno- tate. In other words, we consider leaiii-l at the (document, field)-pair granularity rather than at the document level granularity and choosing a modeling approach that can easily deal with the complexity resulting from the partially labeled doc- uments this approach produces.
The high cognitive burden of the classic annota- tion workflow leads to two major drawbacks. First, makes training data collection extremely expen- sive. In one annotation task for paystub-like docu- ments with 25 target fields, the average time to label each document was about 6 minutes. Scaling this to hundreds of document types with thousands of documents each would be prohibitively expensive. Second, the resulting annotation quality is often quite poor. We have observed systematic errors such as missing labels for fields that occur infre- quently in the documents or for instances that are in the bottom third of the page. To obtain acceptable training and test data quality, each document must be labeled multiple times, further exacerbating the annotation cost issue.
We interleave rounds of such human annotation with training model that is capable of consuming partially labeled documents. In combination, our proposed approach dramatically improves the effi- ciency of the annotation workflow for this extrac- tion task. In fact, through experiments on document types drawn from multiple domains, we show that selective labeling allows us to build models with 10x lower annotation cost while achieving nearly the same accuracy as a model trained on several thousand labeled documents. Note that our goal in this paper is not to advance the state-of-the-e thrt--rrt-t in active learning, nor to propose more data-efficient model for extraction from layout-heavy documents.
2.1.2 Proposed Annotation Workflow
We propose the following alternative to the classic annotation workflow:
We speed up labeling throughput by simpli- fying the task: rather than drawing bound- ing boxes, we ask annotators to accept or re- ject a candidate extraction. Figure (Green
Our main contribution is that we demonstrate that a novel combination of an existing active-lcrning strategy with an existing extraction model can be

---

Remarks) illustrates how much easier this "yes/no" task is compared to the classic one.
Candidate Generator
INITECH
Date
2. We further cut down annotation cost by only labeling a subset of documents and only a subset of fields in each document.
INITECH
3. We use model trained on partially labeled documents to propose the candidate extraction spans for labeling. This allows to interleave model training and labeling so that the model keeps improving as more labels are collected.
Scorer
$70.00
0.1
$7.00
0.9
$77.00
3th Juyy 2019
0.2
invoice date 18 June, 2018
18. 18.June, 2019
18X armunt
$7.00
4. We use a customized active learning strat- egy to identify the most useful labels to col- lect, viz., the candidate extraction spans about which the model is most uncertain. In succes- sive labeling rounds, we focus our labeling budget on the fields that the model has not yet learned to extract well, such as the more infrequent ones.
Figure 2: Architecture of the candidate generator and scorer of our document extraction model. The scoring done using a neural network model trained bi- nary classifier.
4. The analysis in this paper uses empirical measurements for labeling tasks on docu- ments with roughly 25 fields to model the costs of the traditional approach (6 minutes per document, details in Appendix) and the proposed approach (10 seconds per "yess/no" question (Blog, 2020)). For more complex documents the difference in the two costs may be significantly higher.
In Section 5, we show empirical evidence that this improved workflow allows us to get to nearly the same quality as model trained 10k docs by spending order-of-magnituce less data- labeling. Note that naively switching the labeling task to the "yes/no" approach does not cut down the labeling cost - we were to highlight every span that might potentially be an amount and present an "Is this the tax ao__amuut?? question, with the dozens of numbers that are typically present in an invoice, this workflow will be much more expensive than the classic one. A key insight contribute is that model trained on modest amount of data can be used to determine highly effective subset of "yes/no" questions to ask.
Throughout this work, we use an extraction sys- tem similar to the architecture described in (Ma- jumder al., 2020). As shown in Figure 2, this architecture consists of two stages: candidate gener- ation and candidate classification. In the first stage, we generate candidates for each field according the type associated with that field. For example, the candidates generated for the date of invoice field would be the set of all dates in that invoice. The can- didate generators for field types like dates, prices, numbers. addresses, etc. are built using off-the- shelf, domain agnostic, high-recall text annotation libraries. The recall of candidate generation varies across fields, e.g. high in dates and prices while relatively low addresses and names. Having a candidate generator with low recall indeed limits the recall of the final extractions for that field. In the second stage, we score each candidate's like- lihood of being the correct extraction span for the document and field it belongs to. This scoring is done using neural network model trained bi- nary classifier. The highest-scoring candidate for given document and field predicted as the extrac- tion output for the document and field if it exceeds a certain seedificc threshold.
2.2 Assumptions
We make the following four assumptions about the problem setting:
1. We assume access to a pool of unlabeled doc- uments. This is a natural assumption in any work on managing cost of acquiring labeled training data.
2. We assume the extraction model can be trained partially labeled documents.
3. We assume the model can generate candidate spans for each field and measure of un- certainty - this is used to decide the set of "yes/no" questions to present to the annotator.

---

Classically Labeled Candidates (Initial Set)
contained in S and we mark all the remaining unlabeled candidates in U" as US Our label- ing workflow proceeds in rounds. each round j. the model is used to select candidates Sj from U and have them reviewed by human annotators. The annotators answer a "yes/no" question either accepting or rejecting this proposed label. As a result, S = U Sj and U US, meaning the newly labeled examples are merged into the training set and removed from the unlabeled set. The model is retrained on S in each round and we repeat this itcrative labeling-ann and-training procedure until we exhaust our annotation budget or reach our target Fl score.
Unlabeled Candidates
Binary Classifier
Selective Labeled
Candidates
Initial training
Selective Laboling
Selective Labeler &
Re- Mee-trinning
The model training pipeline starts by inital training (blue) the binary classifier using the small clas- sically labeled dataset. We then selectively label (pur- ple) a fixed number of candidates according to the hud- get, which are then used re- re-train (orange) the model together with the initial dataset.
Figure 3:
3.2 Measuring Uncertainty
The ability to train on partially labeled docu- ments is trivially true for this modeling approach since it employs binary classifier trained on the la- beled candidates. This should be relatively straight- forward for sequence labeling approaches, such as (Xu al., 2021), as well. Identifying a potential span in the document to present as a "yes/no" ques- tion to an annotator is an exercise ranking the candidates for each field. We expect that sequence labeling approaches can be adapted to satisfy this requirement, e.g., by using beam search to decode the top few sequence labels. However, this is likely more complex than the aforementioned approach, and we leave this as an exercise for future work.
We select the candidates that the model is most un- certain about. In this work, we explored two met- rics to quantify a model's prediction uncertainty. Score distance. This method assigns a metric to each candidate based on the distance that the score is from some threshold (Li and Sethi, 2006). More formally, the uncertainty is 1 -essssorreereesllee For example, if the threshold is 0.5, this sug- gests that the model most uncertain of its pre- dictions of scores close to 0.5, in either direc- tion. This approach can also be interpreted as an entropy-based uncertainty method, where we find an optimal candidate x² so that x" arg max x
Σj P( |xr (o In our binary classifica- tion setting, y/: {0,, 1} and candidates with scores closer to 0.5 results in larger entropy.
3 Selective Labeling Methodology
We first provide an overview of the selective la- beling framework before describing various uncer- tainty measures and ways to deal with the unique characteristics of this setting, such varying diffi- culty for different fields.
Score variance. This method performs inference on a candidate multiple times with the dropout layer enabled and assigns the uncertainty metric as the variance of the scores (Gal and Ghahramani, 2016; Kirsch et al., 2019; Ostapuk et al., 2019). An alternative method trains multiple models indepen- dently from one another and assigns the uncertainty metric as the variance of the scores across all mod- els (Seung et al., 1992). Note that empirically, we observed this yields near identical results as the dropout-based approach, so we only present find- ings for the latter.
3.1 Overview
Figure provides a visual overview of our selec- tive labeling workflow. We assume a scenario in which corpus of several thousand unlabeled docu- ments Odd belonging to the target document type is available and we can request annotations from a la- beler for every unlabeled document d$ E Ud which consists of a set of candidates (co- G.
di c.. ...,
3.2.1 Score Calibration
We begin by fully labeling a small randomly sam- pled subset of documents Sdd Utd. say 50-250 documents, using the classic annotation workflow.
Our model's predicted scores tend be un- calibrated (as is very typical of neural networks (Guo et al., 2017)), particularly initial rounds and for infrequent fields due to training data scarcity. We calibrate scores in such a way that picking
We learn an initial document extraction model fa St), where S represents the candidate set

---

candidate with calibrated score of, say, 0.6 yields a 60% probability that it has a positive la- bel (Guo et al., 2017). Without loss of general- ity, we use a modified version of histogram bin- ning (Zadrozny and Elkan, 2001) and IsoRegC (Zadrozny and Elkan, 2002) to accommodate the highly non-uniform distribution of scores.
Domain
Fields
Splits
Docs
Candidates
Initial-50
50
118K
Initial-100
100
24.5K
Supply Chain
Initial-250 250
250
58.7K
Test
5,019
1.2M
Hidden-labee
10,000
2.4M
Initial-TO 100
100
76.0K
Retail Finance
Test
849
1.2M
Hidden-label
4,000
5.6M
Initial-100
100
13.4K
By calibrating the scores, threshold selection becomes much more intuitive for the score-based uncertainty metric. For example, if we specify a threshold of 0.5, we expect that to mean we will select candidates for which the model has 50% chance of classifying correctly across all fields.
Tax Forms
24
Test
1,498
1.0M
Hidden-label
7,500
5.1M
Table 1: Statistics of datasets three domains.
candidates selected from the same document and field. After ordering the candidates by the chosen uncertainty metric, if we were simply select the top-k candidates, we might end up selecting too many candidates for the same document and field. Instead, we select at most m candidates for each document and field, m being a tunable hyperpa- rameter we can adjust on a per-field basis. This ensures that we spread the annotation budget over more documents and fields.
3.3 Sampling Candidates
Once the uncertainty metric is calculated for each candidate in the unlabeled set, the next step is to se- lect subset of those candidates for human review. The most obvious method is to select the top-k can- didates, thereby selecting the candidates for which the model is most uncertain. In practice, this can lead to sub-optimal results when the model finds many examples for which it is uncertain but may in fact be very similar to one another. The most common approach to break out of this trap is to introduce some notion of diversity in the sampling methodology (Gao et al., 2020; Ishii et al., 2002). Combining Top-k and Random Sampling. A common method is to reallocate the budget in each round so that a portion of that budget goes towards the top candidates by uncertainty (ensur- ing we get labels for the most uncertain candidates) and the remaining budget goes towards random sample of candidates from the unlabeled set (en- suring that some amount of diversity is included in each round). We take a simple approach is to se- lect the top-k' candidates by the uncertainty metric, where k' < k; and then randomly sample k' candidates from the remaining unlabeled dataset. Capping Candidates for Each Document and Field. An important observation we make about the extraction problem is the following: While a given field typically has multiple candidates in ev- ery document, there are usually few positives per document compared to the number of negatives. For example, there are usually many dates in an invoice, and typically only one of them is the date of invoice. The uncertainty metrics we defined in Section 3.2 do not take into account this relation- ship between labels. We leverage this intuition to increase sample diversity by capping the number of
3.4 Automatically Inferring Negatives
After candidates have been selected and labeled, we merge the newly-labeled candidates into our training set. At this point, there is another oppor- tunity to draw additional value from the unlabeled corpus by utilizing the structure of the extraction problem, in particular, for fields that are defined in the domain's schema only have single value per document (such as a document identifier, statement date, amount due, etc.). The key insight here is that when a positive label revealed via selective labeling, we can infer negative labels for some re- maining candidates in the document. If the schema indicates that a particular field is non-repeating, automatically infer that all of that field's 'ss remaining candidates in the document are negative.
Datasets and Setups
To evaluate the performance of our proposed meth- ods, we use datasets belonging to three different domains, summarized Table The number of fields varies across domains, e.g., the Tax Forms dataset has more than twice the fields as the Retail Finance dataset. We use hidden-label datasets in- stead of real unlabeled datascts and simulate the la- beling procedure by revealing the labels of the can- didates from the hidden-label datasets. We leverage Average E2E Max F1 to evaluate the methods. Fur- ther explanations and experimental setups can be found in the Appendix.

---

Retail Finance
Supply Chain
Tex Forms
budget (i.e, 10% cost) and using the top-k sampling methodology. We observe that not only is com- puting score distances as the uncertainty indicator much more computationally efficient than variance- based metrics (10x faster), but it also significantly outperforms the latter as well. As we exhaust the budget over time, the advantage of score distance becomes more obvious.
0.836 0.644
0.8
0.7
0.887 0.705
0.6788.688
Avg
0.644
0.547
0.5
Full Labeling
Intia
Selective Labeling
5.3 Sampling Methodology
Figure 4: Best performing Selective Labeling com- pared to Initial which is trained on just 100 documents and Full Labeling in which the hidden-label dataset (used in Selective Labeling) is fully used in training.
Figure 5b compares performance across different sampling methodologies. As one might expect, pure random sampling far worse than any other approach - we believe the initial model is confident in predicting a large quantity of candidates (espe- cially the negatives), and randomly sampling from them does not obtain much useful knowledge.
5 Results
In this section, we provide evidence to prove that selective labeling reduces the annotation cost by 10X in different domains and analysis to support the design choices including number of selection rounds, selection and sampling strategies.
The top-k strategies produce much more im- pressive results. Furthermore, observe in later rounds that injecting some diversity via random- ness achieves slightly better performance than the vanilla top-k approach. We believe this mimics the aggregation of exploitation (top-k) and exploration (random) processes, proven to be beneficial in rein- forcement learning applications (Ishii et al., 2002). This also confirms our suspicion that top-k alone can lead into selecting many uncertain examples which are in fact very similar one another.
5.1 Best Performance on Different Domains We train three initial models randomly sam- pled and labeled set of 100 documents for each domain. For example, as shown in Figure 4, the initial model for the Supply Chain domain achieves 0.547 Fl on the test dataset. We fine-tune the ini- tial model on a fully labeled 10k document dataset (i.e., the hidden-label set from Table 1, in which for the purposes of this analysis we use its true labels), resulting in an FI score of 0.705. The performance gap between these two models is thus 0.158.
5.4 Multi-round Setting
In Figure 5c, we compare 5 learning curves, cach of which denotes selecting the same number of candidates in total (10% annotation cost) over a different number of rounds. For example, the 16- round experiment selects TO of the total budget in each round, while the 1 I-round experiment utilizes the entire budget in single round.
Starting from the same initial model, we apply our best selective labeling strategy (which we dis- cuss in the following sections) to reveal the labels from a subset of candidates that comprises only 10% of the annotation cost of fully labeling the hidden-label dataset. For the Supply Chain do- main, this achieves an F1 score of 0.687, which closes the performance gap by 89%. Similarly, close the gap by 88% and 92% for the Retail Fi- nance and Tax Forms domains, respectively. This demonstrates that our method can dramatically de- crease the annotation cost without sacrificing much performance and can be generalized well to other document types.
As we increase the total number of rounds, the model tends to yield better extraction performance until it peaks at about 12 rounds. This imer-grained strategy usually performs better than coarser ones but the gains become marginal at a higher number of rounds. Interestingly, we find that using up just half the budget in the first 8 rounds of a 16-round experiment achieves slightly better performance than exhausting the entire budget in the 1-round experiment. This comparison underscores the im- portance of employing a multi-round approach.
5.2 Selection Metries
Figure 5a we plot per-round performance of two selection metrics the Supply Chain domain given the same set of documents and annotation
5.5 Ablation Study
Table 2 presents an ablation study to understand the impact of different diversity strategies. SL rep-

---

0.68
0.88
0.88
0.64
Avg EZE Max F7
0.64
0.84
0.62
Sampling
0.52
0.82
Rendom
Tapk
0.60
0.00
Selection Metric
0.58
0.58
0.58
Score Variance
0.54
20
40
80
20
80
100
20
40
60 80
100
Completion Budget (%)
Complitton SL Budget
Completion SL Budget
(a)
(b)
(c)
Figure 5: Performance comparisons between (a) selection metrics, (b) sampling approaches, and (c) the rate at which exhaust the budget through different number of rounds of selective labeling. The x-axis denotes the percentage of the total selective labeling budget consumed. The results are about the Supply Chain dataset.
Models
Avg E2E Max EI (std.)
Initial Dataset Size
SL SL+CS SL+CC SL+AIN
0.671 (0.006)
100
250
0.679 (0.005)
+1.2%
0.683 (0.009)
0.675 (0.005)
+1.8%
+0.6%
0.70
0.687
0.705
0.894
SI+CS+CC+AIN
0.687 (0.005)
+2.1%
Avg E2E Max F1
0.55
0.667
0.521
Table 2: Ablation Study, SL denotes selective labeling utilizing the top-k sampling and score distance metric. CS, CC, and AIN represent calibrating scores, capping candidates and automatically inferring negatives.
0.547
0.50
0.45
Full Labeling
Intial
Selective Labeling
resents a 11 12-round selective labeling method using top-k sampling on the score distance metric. We separately add one feature at time test the effee- tiveness of calibrating scores (CS), automatically inferring negatives (AIN) and capping candidates (CC). Results show that every feature improves the model, but we achieve the largest improvement when applying all features in SL+CS+CC+AIN It is reasonable to conclude that increasing diversity intelligently helps us select more useful candidates than relying on the uncertainty metric alone.
Figure 6: Comparison among three initial dataset sizes in the Supply Chain domain. We present the same three approaches in Figure 4: Initial is trained on the ini- tial dataset alone, Selective Labeling selects the equiv- alent of 10% annotation cost in candidates, and Full Labeling fine-tunes from the initial model on the full hidden-label data.
by 77%, 89%, and 87%, for initial dataset sizes of 50, 100, and 250, respectively. We can conclude that selective labeling is capable of finding use- ful candidates to significantly improve the model performance even cost of only 10% of the an- notation budget. And it is not surprising that the selective labeling gains may suffer when the initial datasct is too small (c.g. 50).
5.6 Initial Labeled Dataset Size
Given the dependence of the selective labeling method on an initially labeled small dataset, it is imperative that we evaluate how the approach is affected by the number of documents in this initial dataset. We experiment with initial datasets 50, 100, and 250 documents in the Supply Chain do- main using our best selective labeling strategy and a budget equivalent of 10% cost of annotating the "unlabeled" dataset.
Note that the model can extrapolate to fields that are not present in the initial set of documents. For each document type, schema is defined to include all types of fields that users may be interested in. The model generate candidates matter if the field exists in the initial document set or not, long as the field included in the schema.
Figure indicates that the size of the initial dataset greatly impacts the performance of the model trained solely on those initial training sets, but has starkly less of an impact once we apply selective labeling. We close the performance gap
5.7 Per-field Extraction Analysis
We examine the extraction performances of eight fields from the Supplier Chain document type

---

Field
C8tn. at debeery
customer.cm nome
TIK amoOU
paytrert.doo
InvoloL rumber
GMd Jnvolced
customer 3ctress
suppl suppler.io le
0.0
0.6
0.4
vikui
Selecive Labelig
Figure 7: Per-field comparison among Initial, Selective Labeling and Full Labeling. Initial trained on the initial dataset alone, Selective Labeling selects the equivalent of 10% annotation cost in candidates, and Full Labeling fine-tunes from the initial model on the full hidden-label data.
Figure 7 (initial dataset size is 100) to better un- derstand where selective labeling works well. The recall of candidate generation for these fields varies from 30% 99% showing that selective labeling works even when candidate generation not per- fect. We observe that the big gap between Initial and Full Labeling is almost completely closed by selective labeling in fields such date_ož delivery and customer name. Unsurprisingly, the algorithm results in strong improvements for fields where the initial model hasn't seen enough examples. For frequent fields such as date date_invoice invoiced and in- voice number, the initial model performs well, and there is not much room for improvement. Conse- quently, few candidates are selected and the result- ing Selective Labeling model remains competitive on these fields.
mani, 2016; Kirsch et al., 2019; Bengar et al., 2021), Researchers seek increase the diversity by fore- ing the selection to cover a more representative set of examples (Yang al., 2017: Yin et al., 2017; Sener and Savarese, 2018) 8) or incorporating discrim- inative learning (Gissin and Shalev-Shwartz, 2019).
Researchers have studied combining active learn- ing with deep learning. Most of the advanced strate- gies such Coreset (Sener and Savarese, 2018), Dropout (Gal and Ghahramani, 2016), Discrimina- tive Active Learning (Gissin and Shalev-Shwartz, 2019). (Zhang et al., 2017; Prabhu et al., 2019; Sid- dhant and Lipton, 2018; Zhang and Plank, 2021; Yuan et al., 2020; Ash et al., 2020; Yin et al., 2017;
Shui et al., 2020) studies deep active learning on various NLP tasks. While we don't focus on invent- ing new active learning methods, we are the first to customize existing methods to reduce annotation costs in form-like document extraction.
6 Related Work
Form Extraction. There have been numerous re- cent studies on information extraction for form-like documents. Existing approaches either individually categorize every text span in the document (Ma- jumder et al., 2020; Zhang et al., 2021) l) or formulate the task into sequence modeling problem (Aggar- wal et al., 2020; Lee et al., 2022; Garncarek et al., 2021; Xu et al., 2021) and encode texts, layouts, and visual patterns into feature space. While these approaches produce state-of-the- extraction sys- tems, they require large amounts of labeled training data to do so. We do not propose a new model archi- tecture but instead, focus on the cost of acquiring labeled data for such extraction models.
Conclusion and Future Work
We propose selective labeling that dramatically cuts down the primary dataset annotation bottle- neck in developing extraction models for visually rich documents. There are several future avenues for investigation. First, we simplified the annota- tion task a binary "yes/no" question. Another approach is to allow the annotator to either accept the candidate annotation, or correct either by deleting it or by adjusting the bounding box. For certain text fields it can be valuable to adjust spans to include/exclude details like salutations from a name field ("Mr.", "Dr." etc.) or names from an address. The cost model for such option is more complex than "yes/no", but be used build on the results in this paper. Second, many recent approaches (Xu al., 2021; Lee et al., 2022) treat this as a sequence saqueece-labbling problem and use layout-awar language model. Adapting selec-
Active Learning. We refer to (Settles, 2009; Fu al., 2013; Ren al., 2021) for an extensive re- view of the literature. Two popular approaches for requesting annotation are uncertainty-bant selec- tion (Ko et al., 1995; Culotta and McCallum, 2005) and committee based selection (Gal and Ghahra-

---

tive labeling to a sequence-labeling model requires tackling several problems: a) getting uncertainty estimates for a given span from a sequence labeling model, b) training a sequence labeling model us- ing partially labeled documents, and c) optionally, eschewing menndidtte-gneraatee entirely and gen- erating both candidate-spans and their uncertainty estimates form the sequence labeling model. We hope explore the multiple ways to tackle each of these problems in future work.
Proceedings of theConference on Empirical Meth- ods in Natural Language Processing (EMNLP). pages 3830-3840.
Jordan T. Ash, Chicheng Zhang, Akshay Krishna- murthy, John Langford, and Alekh Agarwal. 2020. Deep batch active learning by diverse, uncertain gra- dient lower bounds. In 8th International Confer- ence on Learning Representations, ICLR 2020, Ad- dis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Javad Zolfaghari Bengar, Bogdan Raducanu, and Joost van de Weijer. 2021. When deep learners change their mind: Learning dynamics for active learning. In International Conference on Computer Analysis of Images and Patterns, pages 403-413. Springer.
8 Limitations
Within the scope of this paper, the proposed method is limited to utilizing combinations of candidate generators and scorers. As explained in Section 7, many recent attractive approaches treat document extraction sequence labeling problem using a layout-aware language model. This model family is attractive because it does not require a candidate generator. However, constructing selective labeling on sequence labeling models is not a simple task, as we must figure out how to obtain an uncertainty estimate for each span from sequence labeling model, how to define spans without candidate generator, and how to train the model with partially labeled documents, etc.
CloudResearch Blog. 2020. A Simple Formula for Pre- dieting the time to complete a study on mechanical Turk.
Rubi Boim, Ohad Greenshpan, Tova Milo, Slava Nov- gorodov, Neoklis Polyzotis, and Wang Chiew Tan, 2012. Asking the right questions in crowd data sourcing. In IEEE 28th International Conference on Data Engineering USA (Arlington, Virginia), 1-5 April, 2012, pages 1261-1264 IEEE Computer Society.
(ICDE (ICDE 2012), Washington, DC,
Aron Culotta and Andrew McCallum. 2005. Reduc- ing labeling effort for structured prediction tasks. In Proceedings of the National Conference on Artificial Intelligence (AAAI), volume 5, pages 746-751.
We understand the limitation of the availability of datasets. We are currently unable to open-source them since the datasets contain proprietary informa- tion (such as vendors and suppliers) that prevent us from sharing publicly. We use internal datasets in this work because they reflect the real-world needs of our institution and its customers better than pub- lic datasets. Compared the few available public datasets, such as FUNSD (Jaume et al., 2019) and CORD (Park et al., 2019), our internal datasets are reflective of real-world data set sizes, which are appropriate for model training and selective labeling, and have more realistic document com- plexity (considering the rich schema, layout-rich documents, diverse templates, high-quality OCR results and token level annotation). Additionally, we list all important information needed repro- duce the methods in the Appendix, including the annotation strategies, model dependencies, and se- lected hyperparameters.
Yifan Fu, Xingquan Zhu, and Bin Li. 2013. A survey on instance selection for active learning. Knowledge and information systems, 35(2):249-283.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning. In Proceedings of the Inter- national Conference on Machine Learning (ICML). pages 1050-1058 PMLR.
Kuzman Ganchev, Fernando Pereira, Mark Mandel, Steven Carroll, and Peter White. 2007. Semi- automated named entity annotation. In Proceedings of the Linguistic Annotation Workshop, pages 53- 56.
Mingfei Gao, Zizhao Zhang, Guo Yu, Serean Ö. Anık, Larry S. Davis, and Tomas Pfister. 2020. Consisteney- semi-supervised active learning: Towards minimizing labeling cost. In Proceedings of the European Conference on Computer Vision (ECCV), pages 510-526.
Lukasz Garcaark, Rafal Powalski, Tomasz Stanistawek, Bartosz Topolski, Piotr Halama, Michal Turski, and Filip Graliński. 2021. LAM- BERT: Layout-aware language modeling for information extraction. In Proceedings of the In- ternational Conference on Document Analysis and Recognition (ICDAR), pages 532-547, Springer.
References
Milan Aggarwal, Hiresh Gupta, Mausoom Sarkar, and
Balaji Krishnamurthy, 2020. Form2Seq : A frame-
work for higher-order form structure extraction. In

---

Daniel Gissin and Shai Shalev-Shwartz. 2019. Dis- criminative active learning. arXiv preprint
Natalia Ostapuk. Jie Yang, and Philippe Cudré- Mauroux. 2019. Activelink: deep active learning for link prediction in knowledge graphs. In The World Wide Web Conference, pages 1398-1408.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International Conference on Machine Learning, pages 1321-1330. PMLR.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jacheung Surh, Minjoon Sco, and Hwalsuk Lec. 2019. Cord: a consolidated receipt dataset for post- ocr parsing. In Workshop on Document Intelligence at NeurlPS 2019.
Shin Ishii, Wako Yoshida, and Junichiro Yoshimoto. 2002. Control of exploiaaion-eexllratiin meta- parameter in reinforcement learning. Neural Net- works, (((46))665-687.
Ameya Prabhu, Charles Dognin, and Mancesh Singh. 2019. Sampling bias in deep active classification: An empirical study, In EMNLP/JCNNP (1).
Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Philippe Thiran. 2019. Funsd: A dataset for form understanding noisy scanned documents. 2019 Internationaa Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1-6. IEEE
Alexander Ratner, Stephen H. Bach, Henry R. Ehren- berg, Jason Alan Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endow- ment, 1163)269-282.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. 2019. Batchbald: Efficient and diverse batch acqui- sition for deep bayesian active learning. Advances Neural Information Processing Systems, 32.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang. Zhihui Li. Brij B Gupta, Xiaojiang Chen, and Xin Wang. 2021. A survey of deep active learn- ing. ACM Computing Surveys (CSUR), 54(9):1-40.
Chun- Wa Ko, Jon Lee, and Maurice Queyranne. 1995. An exact algorithm for maximum entropy sampling. Operations Research, 634466684-6..
Ritesh Sarkhel and Amnb Nandi. 2019. Visual seg- mentation for information extraction from heteroge- neous visually rich documents. In Proceedings of the 2019 International Conference on Management of Data, pages 247-262.
Yu Lee, Chun-Liang Li, Timothy Dozat, Vin- cent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang. Yasuhisa Fujii, and Tomas Pfister. 2022. FormNet: Structural encoding beyond se- quential modeling in form document information ex- traction. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACI.).
Ozan Sener and Silvio Savarese, 2018. Active learn ing for convolutional neural networks: A core-set approach. In 6th International Conference on Learn- ing Representations, ICIR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
Mingkun Li and Ishwar K Sethi. 2006. Confidence- based active learning. IEEF Transactions on Pat- tern Analysis and Machine Intelligence, 28(8):1251- 1261.
Burr Settles. 2009. Active learning literature survey. Computer Sciences Technical Report 1648. Univer- sity of Wisconsin-Madison
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In P Proceedings of the International Conference Learning Representation (ICLR).
H Sebastian Seung, Manfred Opper, and Haim Som- polinsky. 1992. Query by committee. In Proceed- ings of the Fifth Annual Workshop on Computational Learning Theory, pages 287-294.
Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. 2020. Deep active learning: Unified and principled method for query and training. In Inter- national Conference on Artificial Intelligence and Statistics, pages 1308-11 10181318. PMLR.
Bodhisattwa Majumder. Navneet Potti, Sandeep Tata, James B Wendt, Qi Zhao, and Mare Najork. 2020. Representation learning for information extraction from like documents. In Proceedings of the Annual Meeting of the Association for Computa- tional Linguistics (ACL), pages 6495-6504.
Aditya Siddhant and Zachary C Lipton. 2018. Deep bayesian active learning for natural language pro- cessing: Results of large-scale empirical study. EMNLP.
Minh-Tien Nguyen, Viet-Anh Phan, Le Thai Linh, Nguyen Hong Son, Le Tien Dung, Miku Hirano, and Hajime Hotta. 2019. Transfer learning for informa- tion extraction with limited data. In Proceedings of the International Conference of the Pacific Associa- tion for Computational Linguisties, pages 469-482.
Maria Skeppstedt, Carita Paradis, and Andreas Ker- ren. 2017. PAL. a tool for pre-annotation and ac- tive learning. Journal for Language Technology and Computational Linguistics, 3 31(1):11-110.
Springer.

---

Sandeep Tata, Navneet Potti, James B Wendt, Lauro Beltrao Costa, Mare Najork, and Beliz Gunel. 2021. Glean: Structured extractions from templatic documents. Proceedings of the International Con- ference on Very Large Dutabases, 14(6):977-1055
Jieyu Zhang, Cheng- Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. 2022. survey on program- matic weak supervision.
Mengshi Zhang. Daniel Perelman, Vu Le, and Sumit Gulwami. 2021. An integrated approach of deep learning and symbolic analysis for digital pdf table extraction. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4062 4069. IEEE.
Lisa Torrey and Jude Shavlik. 2010. Transfer learn- ing. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, pages 242-264, IGI global.
Sen Wu, Luke Hsiao, Xiao Cheng, Braden Hancock, Theodoros Rekatsinas, Philip Levis, and Christopher Ré. 2018. Fonduer: Knowledge base construction from richly formatted data. Proceedings of the 2018 International Conference Management of Data, pages 1301-1316.
Mike Zhang and Barbara Plank. 2021. Cartography active learning. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 395-406.
Ye 7hang, Matthew Lease, and Byron Wallace. 2017. Active discriminative text representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Li- dong Zhou. 2021. Lay0utLMv2: Multi-modal pre- training for visually-rich document understanding. In Proceedings of the Anmual Meeting of the Asso- ciation for Computational Linguistics (ACL), pages 2579-2591.
Yiheng Xu, Minghao Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. LayoutLM: Pre- training of text and layout for document image derstanding.
Proceedings of the International Conference on Knowledge Discovery & Data Min- ing (KDD), pages 1192-1200. ACM.
Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. 2017. Suggestive annotation: deep active learning framework for biomedical im- age segmentation. In Proceedings of the Interna- tional Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 407. Springer.
Changchang Yin, Buyue Qian, Shilei Cao, Xiaoyu Li, Jishang Wei, Qinghua Zheng, and Tan David- son. 2017. Deep similarity-baac based batch mode active learning with exploration-exploitation In Proceed- ings of the International Conference on Data Mining (ICDM), pages 575-584, IEEE.
Michelle Yuan, Hsuan-Ties Lin, and Jordan Boyd- Graber. 2020. Cold-start active learning through self-supervised language modeling. In EMNLP (1).
Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Ieml, volume 1, pages 609-616. Citeseer.
Bianca Zadrozny and Charles Elkan. 2002. Transform- ing classifier scores into accurate multiclass proba- bility estimates. In Proceedings of the eighth ACM SIGKDD international conference Knowledge discovery and data mining, pages 694 699.
Cha Zhang. 2021. Visual document intelligence
wild. Document Intelligence Workshop at KDD.

---

A Appendix
Hyperparmmeter
Range explored
Best performer
learning rate dropout rate
0.0001-0.1
0.001
A.1 Experimental Setups
0,1-0.5
0.1
hatch size
[64, 128, 256]
128
To explore how the size of the initial labeled dataset impacts our methods, create three initial splits for the Supply Chain domain with 50, 100, and 250 documents. In all of our experiments, we split the train set into 80-20 training-validattin validation sets. The validation set is used to pick the best model by AUC-ROC, and we use the test split to report the performance metrics. We train using the Rectified Adam (Liu et al., 2020) optimizer and measure AUC-ROC on the validation set to decide whether trigger early stopping after 3 epochs of no im- provement. The binary classifier has 330k param- eters and each set of experiments trained within 4 hours on a NVIDIA Tesla PI00 GPU. We ap- ply grid search to tune the hyperparameters. The most performant hyperparameter values are listed in Table 3.
top-k' uncertain candidates candidutes each field doc
0.7-1.0k
0,9k
1-3
Table 3: Hyperparameter selection.
document types. We found it averaged 6-8 min for an annotator to label a single-page document with fewer than 20 fields while it averaged 10-30 min for an annotator to label a multi-page document with 25 fields. So we picked very conservative value (6 min) the estimated time of labeling document this paper. We employ two annotation methods: the classic annotation method, which is always applied to the initial training set, and the proposed "yes/no" method, which is applied during the selective labeling procedure on the unlabeled dataset. The annotation budget is computed based on the time needed to annotate full document and to answer yes/no question. Targeting 10% of the cost to fully label the unlabeled dataset via the classic annotation method, translates to selectively labeling 36k, 14k, and 27k "yes/no" questions for Supply Chain, Retail Finance, and Tax Forms do- mains according to the estimation of same amount of annotation hours. If we bootstrap the model using the classic annotation workflow number of documents, we simply subtract that cost from the budget for selective annotation.
A.2 Evaluation Metrics
We evaluate our methods by measuring the over- all extraction system's 's performance on the test set using the maximum averaged across all fields, denoted as "Average E2E Max F1" in (Majumder et al., 2020). Here we define the "Average E2E Max F1" metric in sufficient detail. "FI" is the harmonic mean of precision and recall. By varying the threshold which operate the binary field classifier, we vary precision and recall, and thus the FI. "Maximum F1" simply refers to the highest F1 that we can achieve with trained field classi- fier - put another way, this is the point along the precision-recall curve at which the Fl score is high- est. We take the "maximum F1" of each field in the schema and average them to compute the "Average E2E Max FI", thereby providing a single metric by which we evaluate overall performance. We optimize for macro Fl which weighs each field equally irrespective of frequency since macro Fl is the typical goal for most extraction applications. Every reported FI score is further averaged over 10 independent runs to account for variability. All Fl scores are generated by comparing the extrac- tions with the ground truth. If field has poor candidate generator, its final recall can obviously not exceed the recall of the candidate generator.
small
A.4 Imperfect Candidate Generation
We believe that the problem of imperfect candi- date generation requires more discussion. We build our Selective Labeling framework on the model architecture introduced in (Majumder et al., 2020; Tata et al., 2021) where they have already demon- strated that high-recall candidate generators built (and are in use) for many fields like numbers, prices, dates, names of people, places organiza- tions (using canned named-entity annotators), ad- dresses (using canned address detectors), alphanu- meric strings (canned regexes), etc. We denote the recall of candidate generation as coverage (to dis- tinguish this from recall of extractions) and present this for a few fields in Table 4. For simple fields, the coverage is indeed in the 90s (date delivery. purchase_order but for other fields is lower. Having a candidate generator with low coverage indced limits the recall of the final extractions and therefore the final Fl score for that field. A key
be
A.3 Annotation Budgets
We acquired stats from our team of annotators on how long the classic annotation takes for various

---

Field
Coverage
w/o
Fi
and 30% of the annotation budget. If our goal is close the gap by about 90%, then 10% of the label- 8,94%
A
date of d_delivrry
0.932
0.690
0.759
10.00%
purchase_orde order
0.992
0.884
0.963
customer address
0.484
0.293
0.363
23.89%
ing cost is enough to achieve the goal (for all three data sets). If our goal is to close the gap by 99%, then 20% of the labeling cost is sufficient for three datasets. The performance of selective labeling can even exceed the full labeling setup (see results for 30% of the labeling cost). We believe that's because noisy annotations exist in the full dataset and our Selective Labeling algorithm avoids select- ing a few incorrect annotations that can confuse the model especially when the model has already determined predictions with low uncertainty that are contrary to the ground-truth annotation.
customer name name
0.715
0.526
0.621
18.00%
Table 4: The Fl performance comparison between w/o Selective Labeling and w/ Selective Labeling on four fields. (Coverage: Candidate Generator Recall)
Supplier Chain
Retail Finance Eap closed
Tax Forms
Cost
pap clased
pop clossd
0.0%%
0.644
10%
0.687
88.6%
0.678
91.9%
0.836
88.7%
20%
0.704
99,498
0.682
102.7%
0.844
100.0%
30%
0.706
100.6%
0.686
113.5%
0.845
101,0%
FL.
100%
0.905
100.0%
0.681
100.0%
0.844
100.0%
Table 5:
Comparisons the extraction performance and gap closed by SL when consuming different anno- tation costs on three datasets. SL 0%, 10%. and FL 100% correspond to the Initial, Selective Labeling, and Full Labeling stats in Figure (SL: Selective Labeling, FL: Full Labeling)
A.6 Candidate Generator
As explained in Section 2.2, we use an extrac- tion system similar to the architecture described in (Majumder et al., 2020). As shown in Figure 2, this architecture consists of two stages: candidate generation and candidate classification. Candidate generators are used to identify potential values for a given field. The generators use cloud-based ser-
contribution of this paper is that even with imper- fect candidate generation, that is, on fields with low candidate coverage, Selective Labeling allows us to deliver big improvements to the final extraction Fl score for that field. See Fl scores before and after selective labeling for fields in the Supply Chain dataset in Table 4.
vice' to identify text spans in the OCR text that are instances of the corresponding type. The system leverages an internal library² of text annotators that was developed for web-search tasks. These text annotators are used to extract features from text that can be used to identify potential values for field. addition the internal text annotators, the system may also use several tection libraries. These libraries are used to detect common types of entities, such as names, dates, currency amounts, numbers, addresses, URLs, etc. Entities are identified by matching text against pat- terns that are known to represent these entities. For example, any date that is found in invoice is con- sidered possible value for any of the date fields in the target schema, such as invoice_ _ate, and delivery_dae _ate.
For simple extraction types such pur- chase_order (numbers) and late_of_dliveee (dates), Selective Sabelniv can improve their extrac- tion performance by 10%, thanks to high candidate generation coverage. For fields with low candidate generation coverage such as customer_name and customer mustomer_address, Selective Labeling is still able to find the uncertain candidates and dramatically improve their extraction Fl by 24% and 18% re- spectively. Finally, even with perfect candidate gen- eration (100% coverage), naively using candidate- level labeling (yes/no questions) instead of the clas- sic annotation workflow ends up being more ex- pensive. As we explain in Section 2.1.2, this is because we typically see dozens of candidates for many fields, and answering yes/no for each of these candidates ends up taking longer than the classic annotation workflow.
source entity de-
A.7 Neural Scoring Model
To predict whether an extraction candidate a valid value for a given target field, the scorer model takes the target field and the extraction candidate as input and outputs a prediction score. The model is trained and evaluated as binary classifier, which means that it predicts whether extraction can-
A.5 Increasing Annotation Budget
We gradually increased the annotation budget and observed the corresponding results in Table 5. Here we compared the extraction performance and gap closed using 10% (same as stats in Figure 4), 20%,
eloou.google somsallccmm/ntturalllaugggg 2NLTK: https: //www. nl org/

---

didate is valid invalid. The features of each extraction candidate used in the scorer model are its neighboring words and their relative positions. The model learns a dense representation for each extraction candidate using a simple self-attention based architccture. This representation captures the semantics of the extraction candidate. The model also learns dense representations for each field in the target schema. These representations capture the semantics of the fields. Based on the learned candidate and field representations, each extrac- tion candidate is scored based on the similarity to its corresponding field embedding. The model is trained as a binary classifier using cross-entropy loss. The target labels are obtained by compar- ing the candidate to the ground truth. Details of the model architecture can be found in (Majumder al., 2020).


Reviews:
[{"review_id": "lsC358HyiD", "reviewer": "Reviewer_Gbq2", "summary": "", "questions": "Have you considered releasing the codebase for your approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewers' time and effort in providing such detailed feedback.\n\n*Q1: While the authors acknowledge its limitations on dataset release, it would benefit from additional details regarding dataset creation and comparison with other baseline datasets. Have you considered releasing the codebase for your approach?*\n\nWe are grateful for your recommendation to open-source the codebase for our SL framework. We concur that this would be advantageous for the framework's reproducibility, and we are dedicated to making our research as reproducible as possible. We will try to obtain authorization to open-source the SL-related codes, which include how we quantify the uncertainty of candidates and how we calibrate the prediction scores. We will also include adequate experimental settings in the paper, such as hyperparameter choices, model training details, and evaluation metrics, to enhance the reproducibility of our findings.\n\n\n*Q2: Furthermore, a comparison with other state-of-the-art models would strengthen the paper's credibility and impact.*\n\nThe state-of-the-art models for document extraction tasks are FormNet (Lee et al., 2022) and LayoutLMv3 [1]. However, these models are transformer-based sequence labeling models, which are outside the scope of this paper. Our goal is not to build a new state-of-the-art document extraction solution, but rather to introduce a novel method for reducing the cost of annotating data for an existing document extraction solution. We propose to use a binary classifier-based scorer model (Majumder et al., 2020) to implement the selective labeling strategy. This is because (a) this smaller model can be trained more quickly and perform even better than large SOTA sequence labeling methods in low-data regimes; and (b) the candidate generation and scoring architecture enables a more efficient and fine-grained annotation style, in which annotators simply answer Yes/No questions to annotate each candidate. As we discussed in the conclusion section, it is non-trivial to adapt the selective labeling strategy to the SOTA sequence labeling methods. Therefore, it is not feasible to compare our method to these methods at this time. However, we believe that applying selective labeling to those sequence labeling methods is an interesting direction for future work.\n\n[1] Huang, Yupan, et al. \"Layoutlmv3: Pre-training for document ai with unified text and image masking.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper focuses on a practical problem concerning information extraction from visually rich documents while addressing cost issues with training and human annotations. Although it does not provide a SOTA model, it provides a novel combination to address the stated research problem.", "reasons_to_reject": "While the authors acknowledge its limitations on dataset release, it would benefit from additional details regarding dataset creation and comparison with other baseline datasets. Furthermore, a comparison with other state-of-the-art models would strengthen the paper's credibility and impact.", "questions_to_authors": "Have you considered releasing the codebase for your approach?", "paper_topic_and_main_contributions": "The paper presents two significant contributions to reducing data-labelling costs for document extraction models. The first contribution involves simplifying the human annotation task to a binary problem. Secondly, it introduces customised active learning at the document-field granularity level.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Syk1gHgXP", "reviewer": "Reviewer_b6MD", "summary": "", "questions": "- **A:** Can the model extrapolate to fields that are not present in the initial set of documents?\n- **B:** Could you provide clarification on how the score calibration is performed?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your thoughtful comments and suggestions, which have helped us to improve the quality of our work.\n\n*Q1. Felt there are details missing about the candidate generator, scorer model and the exploration of SL with different models.*\n\nWe appreciate the suggestion that the paper should become more self-contained by briefly addressing the candidate generator and scorer model. We will add more details of both candidate generator and scorer model into the paper. \n\n\n\n*Q2: Can the model extrapolate to fields that are not present in the initial set of documents? (Missing limitation: if a rare field is not present in the initial set of documents, then it would be impossible for the extraction model to generate candidates for that missing field.)*\n\nYes, the model can extrapolate to fields that are not present in the initial set of documents. For each document type, a schema is defined to include all types of fields that users may be interested in. **The model can generate candidates no matter if the field exists in the initial document set or not, as long as the field is included in the schema.** If it’s not in the initial document set, there’s actually a higher probability that more candidates of this field will be selected for annotation. So this is not a limitation and we will clarify it in the revision paper.\n\n\n*Q3: Hard reproducibility and no codebase.*\n\nThank you for your feedback on open-sourcing the codebase for our SL framework. We understand that this would be helpful for the reproduction of the framework, and we are committed to making our research as reproducible as possible. We will try to get approval to open-source the SL-related codes, including how we measure the uncertainty of candidates, and how we calibrate the prediction scores, etc. We have also provided detailed experimental settings in the paper, including hyperparameter choices, model training details, and evaluation metrics, to enhance the reproducibility of our results. In the meantime, the reproduction details for the candidate generator and scorer can be found in its original paper (Majumder et al., 2020). \n\n\n*Q4: Could you provide clarification on how the score calibration is performed?*\n\nWe use a modified version of histogram binning and IsoRegC to accommodate the highly non-uniform distribution of scores. We compute calibration curves using the labeled training dataset by bucketing the candidates based on score. Based on our knowledge of the score distribution, we made two design choices. (1) The vast majority ($>90\\%$) of our candidates are negative and most of them have very low scores ($<10^{-3}$), while the region of interest to us when calibrating the scores is the rest ($[10^{-3}, 1]$). In calculating bin edges, we exclude all candidates with scores that are smaller than a threshold ($10^{-3}$). All the scores below this threshold are placed in the first bin ($[0, 10^{-3})$). Since the vast majority of candidates get excluded by this filter, the remaining bins have a much higher resolution. (2) We use equal-frequency bins rather than equal-width bins because of the highly non-uniform distribution of scores, even within the score region of interest -- in other words, each bin has roughly the same number of scores, except the first bin. Once binned, calibration curves are computed for each field by interpolating between the curves prevalence (i.e., the proportion of candidates in each score bin that are positive) and the median scores for all the score bins. By calibrating the scores, threshold selection becomes much more intuitive for the score-based uncertainty metric. \n\n\n*Q5: Consider adding an algorithmic description of the SL steps. Consider adding in the legend of Fig 5 that the results are about the supply chain dataset.*\n\nThank you for these suggestions. We added the descriptions in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Simple and sound method:** The proposed method is straight forwarded and easily appliable.  \n\n- **Extensive evaluation:** The authors deeply investigated the behaviour of the proposed method under multiple circumstances, which revealed insightful results. ", "reasons_to_reject": "- **Felt there are details missing:** Although, the focus is on the SL method itself and not on the extractive models, it would be beneficial if the paper becomes more self-contained by briefly addressing the candidate generator and scorer model used during the experiments, instead of providing only a brief glimpse. Additionally, exploring the synergy of different models and the proposed SL method could add an interesting perspective to the paper.\n\n- **Missing limitation:** After reading the paper, I felt that there was one limitation that was not addressed. From what I understood, if a rare field is not present in the initial set of documents, then it would be impossible for the extraction model to generate candidates for that missing field, meaning that field would never receive an annotation.   \n\n- **Hard reproducibility and no codebase:** The paper's reproducibility is challenging due to the private nature of the chosen datasets (as correctly mentioned in the limitation section). To overcome this, conducting additional experiments on public datasets would enhance the broader dissemination of the work. Moreover, even though the datasets are private, releasing the codebase could facilitate a larger adoption of the method and encourage benchmarking on other datasets. ", "questions_to_authors": "- **A:** Can the model extrapolate to fields that are not present in the initial set of documents?\n- **B:** Could you provide clarification on how the score calibration is performed?\n", "paper_topic_and_main_contributions": "In this paper, the authors propose a method called “selective labelling” to more effetely produce human annotated labels for training models for visually extracting information from documents, like invoices, receipts, bills tax forms, etc. The main idea is to leverage the active learning paradigm and with the help of the training model simplify the complex labeling problem. Instead of directly asking humans to annotate fields in the documents, the training model first selects extraction candidates, and humans simply agree or disagree with the annotations, thereby reducing the mental burden of the annotation task. To validate their approach, the authors conducted experiments on three types of documents (supply chain, retail finance, and tax forms) and concluded that their method could reduce the cost of acquiring labeled data by up to 10 times while maintaining similar performance.\n\nOverall, I find the paper well-written, well-organized and well-motivated. I believe the main contribution in the paper is a simple but effective methodology for cheaply acquiring labeling data for training models in visual extraction tasks. \n", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Consider adding an algorithmic description of the SL steps. \n\nConsider adding in the legend of Fig 5 that the results are about the supply chain dataset.\n"}, "extra_text": {}}, {"review_id": "jZZCX9I7at", "reviewer": "Reviewer_5HoU", "summary": "", "questions": "(A)\tThe reviewer thinks similar approaches of this paper have been tackled in problems of sequence labeling such as named entity recognition. Are there any specific problems of document extraction not addressed in other sequence labeling problems?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thoughts and comments. In this rebuttal, we mainly respond to the two concerns of the reviewer:\n\n*1.  Are there any specific problems of document extraction not addressed in other sequence labeling problems?*\n\nYes, the problems of document extraction that were not addressed in other sequence labeling tasks are in three folds: \n\n(1) Paper shown in the “missing reference” (Active Learning with Subsequence Sampling Strategy for Sequence Labeling Tasks, Wanvarie et al., 2011) does not tackle the problem of turning the sequence labeling methods into answering Yes/No annotation questions, which is actually the key for annotation cost reduction in our approach. **Answering Yes/No questions, compared to labeling text spans (partial sequence), is much faster and adds less cognitive burden to the annotators**; \n\n(2) We believe that adapting selective labeling to work with sequence labeling models is an important problem, and indeed we're working on it, but that is not the focus of this paper (explained in more detail in the next question). In this paper, we're focused on showing that we can cut the cost of labeling for the two-stage binary classifier based approaches (Majumder et al., 2020) by 10x. We hope to demonstrate the same for sequence labeling approaches in a future paper. We focus on this binary classifier-based approach because it is less likely to underfit on small annotated datasets than prevailing sequence labeling models such as FormNet (Lee et al., 2022) or LayoutLM ( (Xu et al., 2021) and annotating each candidate is more natural to be converted to a Yes/No question; \n\n(3) Active learning methods designed in the “missing references” (such as Wanvarie et al., 2011) cannot fit the recently developed layout-aware sequence labeling methods, e.g. how to encode surrounding layout information from the visually complex document to select partial text spans for annotation remains an open question.\n\n$\\newline$\n\n*2. Why didn't the author mention the sequence labeling related papers (as recommended in the missing reference section)?*\n\nFirst of all, we did not mention the recommended references because the proposed annotation cost reduction strategy (selective labeling) is not built on top of any sequence labeling methods, but instead relying on a widely used binary classifier-based solution (Majumder et al., 2020) for visually complex document extraction. This extraction solution finds candidates of each field type for each document, and then applies a binary classifier to predict whether the candidate belongs to the type or not. This method has a relatively small number of parameters, which makes it easy to train on a limited amount of data but can still achieve good extraction performance, so it has been widely applied in the industry. **Most importantly, the two-stage binary classifier-based method can be naturally converted to a yes/no annotation task of low cognitive burden. This is the foundation of selective labeling.** There is an urgent need to explore efficient annotation strategies for the binary classifier-based methods, which motivates this paper. \n\nThough the sequence labeling method is also a prevailing technique for document extraction (it converts the document into a long sequence and applies a named entity recognition strategy to tag each token of the sequence), we did not explore how to reduce the cost for any sequence labeling methods in this paper. And it is non-trivial to adapt the annotation strategy designed for one type of document extraction method to another. For example, to enable the proposed selective labeling on a sequence labeling method, we need to further explore how to pick text spans from a long sequence and how to estimate their uncertainty. We also need to find ways to apply Yes/No annotation tasks to the selected partial sequences. As we discussed in Section 7 of this paper, we agree that reducing annotation cost for the state of the art layout-aware sequence labeling methods (e.g. Lee et al., 2022) is an interesting topic to explore in the future. We are grateful to the reviewer for providing us with some references, which can really be a good start of this future work. Actually, we have already made some preliminary progress in this direction, and we found that selective labeling on sequence labeling models can achieve similar cost reduction effects as what we observed in our paper. We would be happy to share more experimental results if the reviewer is interested in this thread.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Evaluation of two step annotation consisting of selection of candidate from extracted results and annotating labels of the selected candidates.", "reasons_to_reject": "In named entity recognition, which is categorized as sequential labeling problem as in document extraction of this paper, similar problems have been studied.However, they are not mentioned. Some of them are listed in the missing reference section. The reviewer feels that the combination of existing methods including the missing reference ones and active learning consist of this paper's method.", "questions_to_authors": "(A)\tThe reviewer thinks similar approaches of this paper have been tackled in problems of sequence labeling such as named entity recognition. Are there any specific problems of document extraction not addressed in other sequence labeling problems?", "paper_topic_and_main_contributions": "This paper proposes selective labeling as a solution to visually rich documents like invoices, receipts, bills, tax forms, and so on. The key insight is to simplify the labeling task to provide “yes/no” labels for candidate extractions predicted by a model trained on partially labeled documents. This paper combines the labeling task of \"yes/no\" with a custom active learning strategy to find the predictions that the model is most uncertain about. Experimental results on document types from different domains that the labeling can reduce the cost of acquiring labeled data with a negligible loss in accuracy.", "excitement": 2, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Two step extraction was proposed in the following paper.\n \n- Named Entity Extraction using AdaBoost (CoNLL’02)\n\n The following paper proposed to handle ambiguous annotation for sequence labelling, which can also be applied to partial annotation.\n\n- Training Conditional Random Fields Using Incomplete Annotations. (COLING 2008)\n\n A partial annotation-based active learning for sequence labeling was addressed in the following paper. \n\n- Active Learning with Subsequence Sampling Strategy for Sequence Labeling Tasks. (Information and Media Technologies 2011)"}}]

Author rebuttal:

Meta-review: The paper proposes a method for reducing the cost of labeling data for document-level extraction tasks via active learning by training a model to propose candidate regions and having annotators perform a binary annotation task where they agree or disagree with the proposed extractions. R1 and R2 agree that the method is simple and sound, addresses an important research problem, and is sufficiently backed up by appropriate experimentation. The noted limitations of the work include: the datasets used are private, so reproducibility is an issue and the paper could benefit from experimentation on a public dataset; some important details (namely about the candidate selection model) could be added. The main concern from reviewer 3 is that potentially similar approaches have been proposed on other tasks e.g. named entity recognition. The authors provide a detailed response clarifying many of the reviewers concerns about limitations related to reproducibility, generalizability, and model details. In particular, the though the spirit of the method is similar to two-step sequence annotation, the problem under study is starkly different as such tasks (e.g. NER) still require observing partial sequences, while in visually complex document extraction the authors are able to reduce the problem to individual binary decisions. Overall the paper appears to be sound and of interest to those working on this topic.
Decision: Accept-Main