SlowGuess/NAACL_2022_OCR huggingface dataset:
To better evaluate the system performance in the multi-key scenario, we relabel the dataset in key-value pairs format to tackle EE and EL tasks simultaneously. We named the new dataset FUNSD-R, which contains 1,421 keys for training and 397 keys for testing. Meanwhile, there are 267 keys in the test set that have not appeared in the training set. FUNSD-R will be released soon.
The CORD Dataset. The CORD (Park et al., 2019) dataset contains 800 receipts for the training set, 100 for the validation set and 100 for the test set. The dataset defines 30 fields under 4 categories and the task aims to label each word to the right field.
The SROIE Dataset. SROIE (Huang et al., 2019) dataset contains 626 receipts for training and 347 receipts for testing. Each entity of the receipt is annotated with pre-defined categories such as company, date, address, and total.
To further investigate the capacity of our proposed method under more challenging scenarios, we expand "SROIE" and "CORD" datasets to "SROIE-S" and "CORD-S" by shuffling the order of text lines and keep the box coordinates to simulate complex layouts. The evaluation metric is the exact match of the entity recognition results in the F1 score.
Model Pre-training. We initialize the weight of GMN model with the pre-trained UniLM base model except for the position embedding layer and visual embedding layer. Specifically, our BASE model has the same architecture: a 12-layer Transformer with 768 hidden sizes, and 12 attention heads. For the LARGE setting, our model has a 24-layer Transformer with 1,024 hidden sizes and 16 attention heads, which is initialized by the pre
trained UniLM LARGE model. For unidir-LM and bidir-LM methods, we select  $15\%$  of the input tokens of sentence A for prediction. We replace these masked tokens with the [MASK] token  $80\%$  of the time, a random token  $10\%$  of the time, and an unchanged token  $10\%$  of the time. For seq-to-seq LM and NER-LM, we select  $15\%$  tokens of the sentence B. The target of the token is the next token. Then, the model predicts the corresponding token with the cross-entropy loss.
In addition, we also add the two-dimensional position embedding and visual embedding for pretraining. Considering that the document layout may vary in different page sizes, we scale the actual coordinate to a "virtual" coordinate: the actual coordinate is scaled to have a value from 0 to 1,000, and rescale the images to the size of  $512 \times 512$ .
We train our model on 64 NVIDIA Tesla V100 32GB GPUs with a total batch size of 1,024. The Adam optimizer is used with an initial learning rate of 5e-5 and a linear decay learning rate schedule.
Task-specific Fine-tunings. We evaluate the model following the typical fine-tuning strategy and update all parameters in an end-to-end way on task-specific datasets. We arrange the source OCR result from top to bottom and left to right. In addition, we add the “[DSEP]” as the separator between text detection boxes. In SROIE and CORD datasets, we construct the target key-value pairs in a certain order due to the keys being limited. (i.e. company, date, address, total). In the FUNSD dataset, we organize the target key-value pairs from top to bottom and left to right. We add the “:” as the separator between key and value and “[DSEP]” as the separator between key-value pairs. The max source length parameter is set to 768 in the SROIE and CORD datasets and 1536 in the FUNSD-R datasets, so input sequences below max length will be padding to the same length. The model is trained for 100 epochs with a batch size of 48 and a learning rate of 5e-5. Note that, the annotations of all GMN results are the weakly-supervised label of sentence-level while other methods use word-level annotations.
4.3 Comparison to State-of-the-Arts
We compare our method with several state-of-the-art on the FUNSD-R, SROIE and CORD benchmarks. We use the publicly available PyTorch models for BERT, UniLM and LayoutLM in all the experiment settings. The results of PICK (Yu et al.,
Model	Precision	Recall	F1
BertLARGE	0	0	0
LayoutLMARGE	0	0	0
GMN_BASE	0.5264	0.4866	0.5057
GMN_LARGE	0.5568	0.5116	0.5333
Table 1: Model results on the FUNSD-R dataset, methods based on sequence labeling yield under scene with larger amount of keys.
Model	Precision	Recall	F1
BERTBASE	0.9099	0.9099	0.9099
UniLMBASE	0.9459	0.9459	0.9459
BERTLARGE	0.92	0.92	0.92
UniLMLARGE	0.9488	0.9488	0.9488
LayoutLMLARGE	0.9524	0.9524	0.9524
LayoutLMv2LARGE	0.9904	0.9661	0.9781
PICK	0.9679	0.9546	0.9612
MatchVIE	-	-	0.9657
BROS	-	-	0.9662
StrucTexT	0.9584	0.9852	0.9688
GMNBASE	0.9853	0.9633	0.9741
GMNLARGE	0.9956	0.9690	0.9821
Table 2: Model results on the SROIE dataset with Ground Truth Setting.
2020), MatchVIE (Tang et al., 2021), BROS (Hong et al., 2021), StrucTexT (Li et al., 2021b), SPADE (Hwang et al., 2021) and DocFormer (Appalaraju et al., 2021) are obtained from the original papers.
Results under scene with larger amount of keys. Table 1 shows the model results on the FUNSD-R dataset which is evaluated using entity-level precision, recall and F1 score. In the case of a large number of key categories, especially in the case that some categories have not appeared in the training set, the method based on sequence labeling yield, neither the Bert model, which only contains text modality nor the LayoutLM which also contains layout and visual modalities.
The best performance is achieved by the  $GMN_{LARGE}$ , where a significant improvement is observed compared to other methods. Note that,  $67.25\%$  of keys have not appeared in the training set, This illustrates that the generative method in GMN is suitable for scenes with a large number of keys.
Results with Ground Truth Setting. Under this setting, the ground truth texts are adopted as model input. As shown in Table 2 and Table 3, even using weakly supervised labels, our approach shows excellent performance on both SROIE and CORD, and yields new SOTA results, which indicates that GMN has a powerful representational capability and can significantly boost the performance on DIE tasks.
Model	Precision	Recall	F1
BERTBASE	0.8833	0.9107	0.8968
UniLMBASE	0.8987	0.9198	0.9092
BERTLARGE	0.8886	0.9168	0.9025
UniLMLARGE	0.9123	0.9289	0.9205
LayoutLMLARGE	0.9432	0.9554	0.9493
LayoutLMv2LARGE	0.9565	0.9637	0.9601
SPADE	-	-	0.925
DocFormer	0.9725	0.9674	0.9699
BROS	-	-	0.9728
GMNBASE	0.9547	0.9576	0.9562
GMNLARGE	0.9693	0.9798	0.9745
Table 3: Model results on the CORD dataset with Ground Truth Setting.
Model	SROIE-E2E
Precision	Recall	F1
BERTLARGE	0.4066	0.3876	0.3969
LayoutLMLARGE	0.4414	0.4236	0.4323
GMNBASE	0.7324	0.7161	0.7242
GMNLARGE	0.7543	0.7334	0.7437
Model	CORD-E2E
Precision	Recall	F1
BERTLARGE	0.6313	0.6724	0.6512
LayoutLMLARGE	0.6684	0.7086	0.6879
GMNBASE	0.7840	0.8133	0.7984
GMNLARGE	0.8165	0.8368	0.8265
Results with End-to-End Setting. We adopt Tesseract as OCR engine to get the OCR result of public datasets. It's worth noting that there are exist some OCR errors, the sequence labeling method can not handle, but in our GMN, the matching process between OCR results and ground truth is avoided thanks to the novel layout embedding method in an end-to-end training setting. The performances are shown in Table 4. Our method shows new state-of-the-art performance benefits from the ability to error correction. A detailed analysis of it is introduced in case studies B.
Results with Position Shuffle Setting. In order to verify the robustness of our two-dimensional embedding method, we apply a shuffling operation on boxes of the test dataset. As shown in Table 5, compared with models that have one-dimensional position embeddings, our method is more robust to input disruption with a big gap.
An ablation study is conducted to demonstrate the effects of different modules in the proposed model. We remove some components to construct several comparable baselines on the CORD dataset. The statistics are listed in Table 6.
Table 4: Model results on the SROIE and CORD datasets with End-to-End Setting.
Model	SROIE-S
Precision	Recall	F1
BERTBASE	0.0702	0.0490	0.0577
LayoutLMBASE	0.7169	0.6880	0.7022
GMNBASE	0.9679	0.9424	0.9550
Model	CORD-S
Precision	Recall	F1
BERTBASE	0.1235	0.1384	0.1305
LayoutLMBASE	0.6917	0.7139	0.7026
GMNBASE	0.9345	0.9488	0.9416
Table 5: Model results on the SROIE-S and CORD-S datasets with Position Shuffle Setting.
Model	Precision	Recall	F1
GMN	0.9693	0.9798	0.9745
GMN w/o Image	0.9623	0.9608	0.9616
GMN w/o NER-LM	0.96	0.9585	0.9593
GMN w/o MAMM	0.9576	0.9547	0.9562
Table 6: Model results of different components of our method on the CORD dataset.
The "GMN w/o MAMM" means using the same multi-modal feature as LayoutLM. Compared with LayoutLM, MAMM brings about  $1.82\%$  improvement of F1, which verifies the validation of MAMM. The "GMN w/o Image" means removing the image feature extraction. Experiment results show that visual modality can also improve the performance. Moreover, with NER-LM considered, the performance of information extraction increases to  $97.45\%$ . Extended experiments including attention visualization analysis and case studies can refer to Appendix A and B.
In this work, we propose a Generative Multi-modal Network (GMN) for practical document information extraction. Since GMN is a generation method including no pre-defined label category, it supports scenes that contain unknown similar keys and tolerates OCR errors, meanwhile requires no character-level annotation. We conduct extensive experiments on publicly available datasets to validate our method, experimental results demonstrate that GMN achieves state-of-the-art results on several public DIE datasets, especially in the practical scenarios. Though our GMN significantly outperforms other DIE models, there still exists potential to be exploited as regard to practical scenarios. In order to cope with complicated layout information as well as ambiguous semantic representations, we argue that more attention should be paid to the modality embedding and interaction strategy, which has more opportunity to handle such difficult cases.
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. 2021. Docformer: End-to-end transformer for document understanding. ArXiv preprint, abs/2106.11539.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13042-13054.
Daniel Esser, Daniel Schuster, Klemens Muthmann, Michael Berger, and Alexander Schill. 2012. Automatic indexing of scanned documents: a layout-based approach. In Document Recognition and Retrieval XIX, part of the IS&T-SPIE Electronic Imaging Symposium, Burlingame, California, USA, January 25-26, 2012. Proceedings, volume 8297 of SPIE Proceedings, page 82970H. SPIE.
Kaiming He, Georgia Gkioxari, Piotr Dólar, and Ross B. Girshick. 2017. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2980-2988. IEEE Computer Society.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Teakgyu Hong, Donghyun Kim, Mingi Ji, Won-seok Hwang, Daehyun Nam, and Sungrae Park. 2021. BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents. ArXiv preprint, abs/2108.04539.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. ICDAR2019 competition on scanned receipt OCR and information extraction. In 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019, pages 1516-1520. IEEE.
Wonseok Hwang, Jinyeong Yim, Seunghyun Park, Sohee Yang, and Minjoon Seo. 2021. Spatial dependency parsing for semi-structured document information extraction. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 330-343, Online. Association for Computational Linguistics.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. FUNSD: A dataset for form understanding in noisy scanned documents. In 2nd International Workshop on Open Services and Tools for Document Analysis, OST@ICDAR 2019, Sydney, Australia, September 22-25, 2019, pages 1-6. IEEE.
Anoop R Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Hohne, and Jean Baptiste Faddoul. 2018. Chargrid: Towards understanding 2D documents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4459-4469, Brussels, Belgium. Association for Computational Linguistics.
Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In Machine Learning: ECML 2004, 15th European Conference on Machine Learning, Pisa, Italy, September 20-24, 2004, Proceedings, volume 3201 of Lecture Notes in Computer Science, pages 217-226. Springer.
David D. Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David A. Grossman, and Jefferson Heard. 2006. Building a test collection for complex document information processing. In SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, USA, August 6-11, 2006, pages 665-666. ACM.
Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2021a. StructuralLM: Structural pre-training for form understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6309-6318, Online. Association for Computational Linguistics.
Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. 2021b. Structext: Structured text understanding with multi-modal transformers. In MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 1912-1920. ACM.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. ArXiv preprint, abs/1907.11692.
Bodhisattwa Prasad Majumder, Navneet Potti, Sandeep Tata, James Bradley Wendt, Qi Zhao, and Marc Najork. 2020. Representation learning for information
extraction from form-like documents. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6495-6504, Online. Association for Computational Linguistics.
Rasmus Berg Palm, Ole Winther, and Florian Laws. 2017. Cloudscan - A configuration-free invoice analysis system using recurrent neural networks. In 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan, November 9-15, 2017, pages 406-413. IEEE.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019. Cord: A consolidated receipt dataset for post-ocr parsing.
Ellen Riloff. 1993. Automatically constructing a dictionary for information extraction tasks. In Proceedings of the 11th National Conference on Artificial Intelligence. Washington, DC, USA, July 11-15, 1993, pages 811-816. AAAI Press / The MIT Press.
Clément Sage, Alexandre Aussem, Haytham Elghazel, Véronique Eglin, and Jérémy Espinas. 2019. Recurrent neural network approach for table field extraction in business documents. In 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019, pages 1308-1313. IEEE.
Daniel Schuster, Klemens Muthmann, Daniel Esser, Alexander Schill, Michael Berger, Christoph Weidling, Kamil Aliyev, and Andreas Hofmeier. 2013. Intellix - end-user trained information extraction for document archiving. In 12th International Conference on Document Analysis and Recognition, ICDAR 2013, Washington, DC, USA, August 25-28, 2013, pages 101-105. IEEE Computer Society.
Guozhi Tang, Lele Xie, Lianwen Jin, Jiapeng Wang, Jingdong Chen, Zhen Xu, Qianying Wang, Yaqiang Wu, and Hui Li. 2021. Matchvie: Exploiting match relevancy between entities for visual information extraction. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJ-CAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 1039-1045. ijcai.org.
Jiapeng Wang, Chongyu Liu, Lianwen Jin, Guozhi Tang, Jiaxin Zhang, Shuaiqiao Zhang, Qianying Wang, Yaqiang Wu, and Mingxiang Cai. 2021a. Towards robust visual information extraction in real world: New dataset and novel solution. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 2738-2745. AAAI Press.
Jiapeng Wang, Tianwei Wang, Guozhi Tang, Lianwen Jin, Weihong Ma, Kai Ding, and Yichao Huang. 2021b. Tag, copy or predict: A unified weakly-supervised learning framework for visual information
extraction using sequences. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 1082-1090. ijcai.org.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyls, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv preprint, abs/1609.08144.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2579-2591, Online. Association for Computational Linguistics.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutm: Pre-training of text and layout for document image understanding. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 1192-1200. ACM.
Wenwen Yu, Ning Lu, Xianbiao Qi, Ping Gong, and Rong Xiao. 2020. PICK: processing key information extraction from documents using improved graph learning-convolutional networks. In 25th International Conference on Pattern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021, pages 4363-4370. IEEE.
Peng Zhang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Jing Lu, Liang Qiao, Yi Niu, and Fei Wu. 2020. TRIE: end-to-end text reading and information extraction for document understanding. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, pages 1413-1422.
A Attention Visualization
To further explore what context information is focused by our GMN, we visualize the attention map of the multi-head Transformer, as shown in Figure 4. The input tokens of the model are marked in black and the decoding results are marked in orange, while X-axis represents the attended tokens.
Figure 4: Visualization of the attention map of the multi-head Transformer, representing semantic/X-coord/Y-coord/visual attention results in a row respectively. The decoding result is marked in orange. Best viewed in color.
Figure 5: Samples from SROIE/CORD/FUNSD datasets, key examples are highlighted by color boxes. Best viewed in color.
As shown in Figure 3, we use the Mask operator to control the attention between each token. The input OCR tokens can attend to each other but the output tokens can only be attended to the already decoded tokens. Consequently, the upper right area of the attention map has no active response, and the area in the lower right corner shows a stepped pattern.
We can observe that the semantic attention mechanism plays an important role in modeling local dependence. In semantic attention, the input OCR tokens mainly focus on themselves and their nearby semantically relevant parts. In contrast, decoded tokens mostly focus on the counterparts in the original tokens, showing a reasonable alignment. Meanwhile, layout and visual attention mechanisms focus on more global information, complementing the semantic attention mechanism.
B Case studies
The motivation behind GMN is to tackle the practical DIE tasks. To verify this, we show some examples of the output of LayoutLM and GMN, as shown in Figure 5. In the sub-figure A and B, GMN successfully corrects the recognition error of OCR results thanks to semantic learning on a large-scale corpus. In the sub-figure C~F, GMN accurately generates the key-value pairs with complex layouts and ambiguous contexts thanks to the novel position embedding method, in comparison LayoutLM is unable to merge the value entities correctly. It's noteworthy that the sub-figures G and H are failed cases, which are caused by semantic obfuscation and reasonable complement to missing character. These examples show that GMN is capable of correcting OCR errors and predicting more accurately in practical scenarios.