SlowGuess/ACL_2020_OCR huggingface dataset:
We measured the performance of our model's scoring predictions using ROC AUC on the test split. We also analyzed its performance in the context of the overall extraction system using the accuracy of the end-to-end extraction results as measured by the maximum F1 score over all decision thresholds, averaged across all fields in the target schema shown in Table 2.
To demonstrate the benefits of our proposed neural architecture over a naive approach, we use two different baseline models for encoding a candidate and scoring it. The bag-of-words BoW baseline incorporates only the neighboring tokens of a candidate, but not their positions. The MLP baseline uses the same input features as our proposed model, including the relative positions of the candidate's neighbors, and encodes the candidate using 3 hidden layers. Both these baselines follow our representation learning approach, encoding the candidate and the field separately. Just as in our model, the final score is the cosine distance between the candidate and field encodings, normalized to [0, 1] using a sigmoid.
We chose the dimension size for each model architecture using a grid-based hyperparameter search. All the metrics we report were obtained from performing 10 training runs and picking the model with the best validation ROC AUC.
Table 2 summarizes the results of this performance comparison. On both our evaluation datasets, our model showed a significant improvement over the baselines by both metrics. For the invoice corpus, our model outperforms the BoW baseline by about 1 point in the scorer ROC AUC,
3rrc.cvc.uab.es/?ch=13
Corpus	Field	Field Type	Train # +ves	Test % +ves	Scorer ROC	End-to-End Max F1
BoW	MLP	Ours	BoW	MLP	Ours
Invoices	amount_due	currency	5,930	4.8%	0.967	0.968	0.973	0.800	0.789	0.801
due_date	date	5,788	12.9%	0.977	0.973	0.984	0.835	0.850	0.861
invoice_date	date	13,638	57.4%	0.983	0.986	0.986	0.933	0.939	0.940
invoice_id	alphanumeric	13,719	6.8%	0.983	0.988	0.993	0.913	0.937	0.949
purchase_order	alphanumeric	13,262	2.2%	0.959	0.967	0.976	0.826	0.851	0.896
total_amount	currency	8,182	12.5%	0.966	0.972	0.980	0.834	0.849	0.858
total_tax_amount	currency	2,949	7.5%	0.975	0.967	0.980	0.756	0.812	0.839
Macro-average	-	14.9%	0.973	0.974	0.982	0.842	0.861	0.878
Receipts	date	date	258	85.5%	0.748	0.792	0.737	0.885	0.885	0.854
total	currency	475	16.7%	0.834	0.796	0.889	0.631	0.607	0.813
Macro-average	-	51.1%	0.791	0.794	0.813	0.758	0.746	0.833
Table 2: Performance on the test set of unseen templates for Invoices and Receipts. The best-performing architecture in each case is highlighted.
which translates to about 3.6 points improvement in the end-to-end Max F1. In fact, our model beats the baseline in every field in our invoice target schema as well. This difference in performance clearly demonstrates the need to incorporate token positions to extract information accurately from form-like documents. Using neighbor position information, the MLP baseline is able to outperform the BoW baseline as well, but the improvement in end-to-end Max F1 is only about 2 points. This result demonstrates that our proposed architecture is better able to encode position information than a naive MLP.
Similarly, for the receipt corpus also, our model outperforms both the baselines. The improvement is much larger for the total field, more than 20 points. For the date field, since there are too few negative candidates in the dataset, all the models have comparable performance end-to-end.
A close examination of the per-field performance metrics in Table 2 reveals that model performance is greatly affected by both the number of positive training candidates, as well as by the ratio of positives to negatives. The best performance is observed for fields that occur frequently in invoices (e.g., invoice_id) and where the candidate generator emits only a small number of negatives for each positive (e.g., invoice_date). Conversely, the fields that are hardest to extract are those that are relatively rare and have low-precision candidate generators, viz., amount_due and total_tax_amount.
We also studied our model performance over various ablation setups and found that the relative order in which various features influence generalization performance is: neighbor text  $>$  candidate
position  $>$  neighbor position. This result is also borne out by the fact that the BoW baseline, which omits the last of these features, is quite competitive with the other approaches.
We also compared the performance of our proposed architecture with and without the self-attention layer applied to the neighbor encodings. We found that self-attention contributes greatly to model performance for the invoice corpus: not only did self-attention lead to a 1-point improvement in scorer ROC AUC and a 1.7 point improvement in end-to-end max F1, we also observed an improvement in every single field in our invoice schema.
6.2 Meaningful internal representations
We investigated the internal representations learned by our model by visualizing their 2-D projections using TSNE. Figure 4(a) shows the representations learned for date candidates. They are colored based on the ground truth data indicating if they belong to one of invoice_date, due_date, or delivery_date. The learned encodings clearly show three distinct (by color) coherent clusters matching the respective field labels.
Figure 4(b) shows the candidate encodings for a sample of positive and negative date candidates for the invoice_date field, along with the embedding for that field. It is apparent that the encodings of the positive examples are largely clustered together whereas the sampled negatives show a more uniform and sparse spatial distribution. Furthermore, the field embedding lies close to the cluster of positive examples. It is interesting to note that the field embedding lies not at the center of the cluster, but rather at its edge, as far away as possible from the clusters of positive examples for other
6501
Figure 4: TSNE visualizations for (a) positive candidate encodings for the date fields in the target schema for invoices, and (b) positive and negative candidate encodings for invoice_date field as well as its field embedding. (c), (d) and (e) show three cases of misclustered candidate encodings
fields. This pattern is predicted by the fact that the loss function is essentially trying to minimize the cosine distance between the field embedding and its positives, while maximizing its distance from its negatives, most importantly the positives for the other fields.
We also indicate three cases of misclustered candidate encodings in Figure 4(a), whose corresponding invoice candidates and their neighborhoods are excerpted below. Figure 4(c) shows a ground truth positive invoice_date example whose encoding is far from the invoice_date cluster. It is clear from examining the invoice that this is an error in the ground truth labels provided by the human annotator. In fact, this date is the date of purchase and not the invoice date. The candidate shown in Figure 4(d) has a candidate encoding that lies midway between due_date, its true label, and invoice_date. We believe this is explained by the fact that this date has both the terms "Due Date" and "date of invoice" nearby, which are usually indicative of due_date and invoice_date respectively. Finally, Figure 4(e) shows a true invoice_date example whose encoding is far away from all the field clusters. A closer examination of the features of this candidate showed that our OCR engine was unable to detect the word "Date" just above the date due to scanning noise. Since this crucial word was missing from the neighbors of this candidate, the learned neighborhood representation was clearly incorrect.
Information extraction from plain text documents for tasks like named entity recognition and relation extraction have benefited from recent advances in deep learning (Lample et al., 2016; Peng et al., 2017). However, these techniques are not directly applicable to our task on form-like documents. Palm et al. (2017) attempts to use RNNs to extract information from form-like documents. However, they treat each line as a vector of n-grams limiting the resulting accuracy.
The importance of understanding visual layout was recognized even in the context of information extraction of webpages in recent work (Cai et al., 2004; Yu et al., 2003; Zhu et al., 2006; Cai et al., 2003). The techniques developed by them are, however, not immediately applicable in our context since we do not have access to the source markup representation for the documents we deal with.
A common approach to solving the problem of extracting information from form-like documents is to register templates in a system, match new documents to an existing template, and use an extractor learnt from said template (Chiticariu et al., 2013; Schuster et al., 2013). The learning problem we tackle in this paper is more ambitious; we seek to generalize to unseen templates.
Our work is most closely related to recent attempts to combine layout features with text signals. Liu et al. (2019) use a document graph and intro
duce a graph combination model to combine visual and textual signals in the document. Katti et al. (2018) represent a document as a two-dimensional grid of text tokens. Zhao et al. (2019) show that using grid information can be useful for information extraction tasks. Denk and Reisswig (2019) combine the grid-based approach with BERT-based text encodings. While an apples-to-apples comparison with these approaches is difficult without a shared benchmark, our system has several advantages: in contrast to the graph-based approaches (Liu et al., 2019) we focus on the harder problem of generalizing to unseen templates rather than dealing with the variations within a template. Since we are not starting with raw pixels, our approach is computationally less expensive than grid-based approaches. Further, we do not require clever heuristics to construct a multi-scale grid that is required for the image-segmentation style abstraction to work well.
To the best of our knowledge, our approach of using representation learning for this task is the first of its kind. We gain many of the well-known benefits of this approach (Bengio et al., 2013), most notably interpretability.
In this paper, we presented a novel approach to the task of extracting structured information from templatic documents using representation learning. We showed that our extraction system using this approach not only has promising accuracy on unseen templates in two different domains, but also that the learned representations lend themselves to interpretation of loss cases.
In this initial foray into this challenging problem, we limited our scope to fields with domain-agnostic types like dates and numbers, and which have only one true value in a document. In future work, we hope to tackle repeated fields and learn domain-specific candidate generators. We are also actively investigating how our learned candidate representations can be used for transfer learning to a new domain and, ultimately, in a few-shot setting.
Acknowledgements We are grateful to Lauro Costa, Evan Huang, Will Lu, Lukas Rutishauser, Mu Wang, and Yang Xu on the Google Cloud team for their support with data collection, benchmarking, and continuous feedback on our ideas. We are also grateful for our research intern, Beliz Gunel, who helped re-run several experiments and fine-tune our training pipeline.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. TPAMI, 35(8):1798-1828.
Deng Cai, Shipeng Yu, Ji-Rong Wen, and Wei-Ying Ma. 2003. Extracting content structure for web pages based on visual representation. In Web Technologies and Applications, APWeb, pages 406-417.
Deng Cai, Shipeng Yu, Ji-Rong Wen, and Wei-Ying Ma. 2004. Block-based web search. In SIGIR, pages 456-463.
Laura Chiticariu, Yunyao Li, and Frederick R. Reiss. 2013. Rule-based information extraction is dead! long live rule-based information extraction systems! In EMNLP, pages 827-832.
Nilesh Dalvi, Ravi Kumar, and Mohamed Soliman. 2011. Automatic wrappers for large scale web extraction. In VLDB, volume 4, pages 219-230.
Brian L. Davis, Bryan S. Morse, Scott Cohen, Brian L. Price, and Chris Tensmeyer. 2019. Deep visual template-free form parsing. CoRR, abs/1909.02576.
Timo I. Denk and Christian Reisswig. 2019. Bert-grid: Contextualized embedding for 2d document representation and understanding. CoRR, abs/1909.04948.
iPayables. 2016. Why Automation Matters: A Survey Study of the Modern Accounts Payable Department. Technical report, iPayables.
Anoop R. Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. 2018. *Chargrid: Towards understanding 2d documents*. In *EMNLP*, pages 4459-4469.
Brian Kulis. 2013. Metric learning: A survey. Foundations and Trends in Machine Learning, 5(4):287-364.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL, pages 260-270.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In ICLR.
Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. 2019. Graph convolution for multimodal information extraction from visually rich documents. In NAACL, pages 32-39.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. JMLR, 9(Nov):2579-2605.
6503
Rasmus Berg Palm, Ole Winther, and Florian Laws. 2017. Cloudscan - A configuration-free invoice analysis system using recurrent neural networks. In IC-DAR, pages 406-413.
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. TACL, 5:101-115.
Sunita Sarawagi. 2008. Information extraction. Foundations and Trends in Databases, 1(3):261-377.
Daniel Schuster, Klemens Muthmann, Daniel Esser, Alexander Schill, Michael Berger, Christoph Weidling, Kamil Aliyev, and Andreas Hofmeier. 2013. Intellix - end-user trained information extraction for document archiving. In ICDAR, pages 101-105.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Shipeng Yu, Deng Cai, Ji-Rong Wen, and Wei-Ying Ma. 2003. Improving pseudo-relevance feedback in web information retrieval using web page segmentation. In WWW, pages 11-18.
Xiaohui Zhao, Zhuo Wu, and Xiaoguang Wang. 2019. CUTIE: learning to understand documents with convolutional universal text information extractor. CoRR, abs/1903.12363.
Jun Zhu, Zaiqing Nie, Ji-Rong Wen, Bo Zhang, and Wei-Ying Ma. 2006. Simultaneous record detection and attribute labeling in web data extraction. In KDD, pages 494-503.