FormNetV2: Multimodal Graph Contrastive Learning for Form Document
  Information Extraction
Authors: name: Chen-Yu Lee; name: Chun-Liang Li; name: Hao Zhang; name: Timothy Dozat; name: Vincent Perot; name: Guolong Su; name: Xiang Zhang; name: Kihyuk Sohn; name: Nikolai Glushnev; name: Renshen Wang; name: Joshua Ainslie; name: Shangbang Long; name: Siyang Qin; name: Yasuhisa Fujii; name: Nan Hua; name: Tomas Pfister
Published: 2023-05-04 05:02:04

Abstract: The recent advent of self-supervised pre-training techniques has led to a
surge in the use of multimodal learning in form document understanding.
However, existing approaches that extend the mask language modeling to other
modalities require careful multi-task tuning, complex reconstruction target
designs, or additional pre-training data. In FormNetV2, we introduce a
centralized multimodal graph contrastive learning strategy to unify
self-supervised pre-training for all modalities in one loss. The graph
contrastive objective maximizes the agreement of multimodal representations,
providing a natural interplay for all modalities without special customization.
In addition, we extract image features within the bounding box that joins a
pair of tokens connected by a graph edge, capturing more targeted visual cues
without loading a sophisticated and separately pre-trained image embedder.
FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
and Payment benchmarks with a more compact model size.