Test-Time Adaptation for Visual Document Understanding
Authors: Sayna Ebrahimi
Sercan O Arik
Tomas Pfister
Venue: Accepted by TMLR ()

Abstract: For visual document understanding (VDU), self-supervised pretraining has been shown to successfully generate transferable representations, yet, effective adaptation of such representations to distribution shifts at test-time remains to be an unexplored area. We propose DocTTA, a novel test-time adaptation method for documents, that does source-free domain adaptation using unlabeled target document data. DocTTA leverages cross-modality self-supervised learning via masked visual language modeling, as well as pseudo labeling to adapt models learned on a \textit{source} domain to an unlabeled \textit{target} domain at test time. We introduce new benchmarks using existing public datasets for various VDU tasks, including entity recognition, key-value extraction, and document visual question answering. DocTTA shows significant improvements on these compared to the source model performance, up to 1.89\% in (F1 score), 3.43\% (F1 score), and 17.68\%  (ANLS score), respectively.

Published in Transactions on Machine Learning Research (06/2023)
Test- Time Adaptation for Visual Document Understanding
Sayna Ebrahimi
saynae
Google Cloud Al Research
Sercan Ö. Arik
soarik@google..o
Google Cloud Al Research
Tomas Pfister
ipfister Macklerrggoggee.cc com
Google Cloud Al Research
Reviewed on OpenReview: https: //
// openreview. net/ forun? id= zshemTAace
Abstract
For visual document understanding (VDU), self-supervised pretraining has been shown to successfully generate transferable representations, yet, effective adaptation of such represen- tations to distribution shifts at test-time remains to be an unexplored area. We propose DocTTA, novel test-time adaptation method for documents, that does source-free do- main adaptation using unlabeled target document data. DocTTA leverages cross-modality self-supervised learning via masked visual language modeling, as well as pseudo labeling to adapt models learned on a source domain to an unlabeled toryet domain at test time. We intuod new benchmarks using existing public datasets for various VDU tasks, including entity recognition, key-value extraction, and document visual question answering. DocTTA shows significant improvements on these compared to the source model performance, up to 1.89% in (Fl score), 3.43% (Fl score), and 17.68% (ANLS score), respectively. Our benchmark datasets are available at https://saynaebrahhin.. .ww..igg..gt.................... Cetthub.io/DcTTTI. html
1 Introduction
Visual document understanding (VDU) is on extracting structured information from document pages repre- sented in various visual formats. It has a wide range of applications, including tax/ max/invoiee/ /aaee//ao/////eeeeee /claims procossing, identity Seanttttyrisk///i /risk//acciin verification, medical records understanding, compliance management, etc. These applications affect operations businesses from major industries and daily lives of the general popu- lace. Overall, is estimated that there are trillions of documents the world.
Machine learning solutions for VDU should rely overall comprehension of the document content, extract- ing the information from text, image, and layout modalities. Most VDU tasks including key-value extraction, form understanding, document visual question answering (VQA) are often tackled by self-supervised pre- training, followed by supervised fine-tuning using human-labeled data (Appalaraju et al. 2021 Gu et al. 2021 Xu et al.] 2020bla Lee et al., 2022 Huang et al. 2022). This paradigm uses unlabeled data in a task-agnostic way during the pretraining stage and aims to achieve better generalization at various down- stream tasks. However, once the pretrained model finc-tuned with labeled data on source domain, a significant. performance drop might oceur if these models are directly applied to new unseen largel domain phenomenon known domain shift (Quinonero- Candela et al. 2008a Mloreno-Toree et al. 2012
CQuneeler-Cannella
The domain shift problem is commonly encountered in real-world VDU sccnarios where the training and test- time distributions are different, common situation due to the tremendous diversity observed for document data. Fig. ! exemplifies this, for key-value extraction task across visually different document templates and for visual question answering task on documents with different contents (figures, tables, letters etc.) for information. The performance difference due to this domain shift might reduce the stability and reliability of VDU models. This is highly undesirable for widespread adoption of VDU, especially given that the common

---

Published in Transactions on Machine Learning Research (06/2023)
Domain Erills Loters
VQA agrams
Laysul
the dete nentiooed letter?
Gistet next diasran iceeert azcaneinatio
<he botton the betdy
flest table?
228052
Indlennking
433155690
0.10883322-4:
Kap-iatee Extracion (Roceee Undwssar ding)
Kay Edrnction (Rooep Undersaading
Narred E-ily Macogribton iFor Undestaringg
Named Endlyy Racogniton (Fom Undersadd ng)
Source
Targat
Source
Target
Distribution shift examples for document samples from the proposed benchmark, DocVQA-TTA. Top row: shows documents from four domains: (i) Emails & Letters, (ii) Figures & Diagrams, (iii) Lay- out, (iv) Tables & Lists, from our VQA benchmark derived from DocVQA dataset (Mathew et al.) 2021
Figure 1:
Bottom left: documents from source and target domains for key-value information extraction task from SROIE (Huang et al., 2019) receipt dataset. Bottom right: documents from source and target domains
for named entity recognition task from FUNSD (Jaune et al. 2019) dataset.
use cases are from high-stakes applications from finance, insurance, healthcare, or legal. Thus, the methods to robustly guarantee high accuracy in the presence of distribution shifts would be of significant impact. Despile being a critical issue, to the best of our knowledge, no prior work has studied post-training domain adaptation for VDU.
Unsupervised domain adaptation (UDA) methods attempt to mitigate the adverse effect of data shifts, often by training a joint model on the labeled source and unlabeled target domains that map both domains into common feature space. However, simultancous access to data from source and target domains may not be feasible for VDU due to the concerns associated with source data access, that might arise from legal, technical, and contractual privacy constraints. In addition, the training and serving may be done in different computational environments, and thus, the expensive computational resources used for training may not be available at serving time. Test-time adaptation (TTA) (or sourco-free domain adaptation) has been introducdd to adapt a. model that is trained on the source to unseen larget data, without using any source data (Liang et al.) 22020; Wang et al. 2021b [Sun et al.) 20200; Wang et al., [2021a] Chen et al., 2022 Huang et al., 2.21). TTA methods thus far have mainly focused image classification and semantic segmentation tasks, while VDU remains unexplored, despite the clear motivations of the distribution shift besides challenges for the employment of standard UDA.
Since VDU significantly differs from other computer vision (CV) tasks, applying existing TTA methods in a straightforward manner is suboptimal. First, for VDU, information is extracted from multiple modalities (including image, text, and layout) unlike other CV tasks. Therefore, a TTA approach proposed for VDU should leverage cross-modal information for better adaptation. Second, multiple outputs (e.g. entities or questions) are obtained from the same document, creating the scenario that their similarity in some aspects (e.g. in format context) can be used. However, this may not be utilized in beneficial way with direct application of popular pscudo labeling or self training- based TTA approaches [Lee et al. 2013). which have gained a lot of attention in CV (Liang et. al. 2020 2021 Chen el al. 2022 Wang et al.) 2021a). Pseudo labeling uses predictions on unlabeled target data for training. However, naive pseudo labeling can result in accumulation of errors (Guo et al., 2017 Wei et al.) 2022 Meinke Hein 2020 Thulasidasan et al., 2019 Kristiadi et al. (020). Particularly in VDU, happens due to generation of multiple outputs at the same time that are possibly wrong in the beginning, as each sample can contain long sequence of
2

---

Published in Transactions on Machine Learning Research (06/2023)
words. Third, commonly-used self-supervised contrastive-basee TTA methods for CV (He et al. 2020 Chen et al.] 2020b Tian et. al. 2020) (that are known to improve generalization) employ a rich set of image augmentation techniques, while proposing data augmentation is much more challenging for general VDU.
In this paper, we propose DocTTA, a novel TTA method for VDU that utilizes self-supervised learning on text and layout modalities using masked visual language modeling (MVLM) while jointly optimizing with pseudo labeling. We introduce an uncertainty-aware per-batch pseudo labeling selection mechanism, which makes more accurate predictions compared to the commonly-usee pscudo labeling techniques in CV that use pseudo-labeling selection mechanism (Liang et al. 2020) in TTA or select pseudo labels based on both uncertainty and confidence (Rizve et al. 2021) in semi-supervised learning settings. To the best of our knowledge, this is the first method with a self-supervised objective function that combines visual and language representation learning as a key differentiating factor compared to TTA methods proposed for image or text data. While our main focus the TTA setting, we also showcase a special form of DocTTA where access to source data is granted at test time, extending our approach be applicable for unsupervised domain adaptation, named DocUDA. Moreover, in order to evaluate TTC diligently and facilitate future research in this direction, we introduce new benchmarks for various VDU tasks including key-value extraction, entity recognition, and document visual question answering (DocVQA) using publicly available datascts, by modifying them to mimic real-world adaptation scenarios. We show DocTTA significantly improves source model performance at test-time on all VDU tasks without any supervision. To our knowledge, our paper is first to demonstrate TTA and UDA for VDU, showing significant accuracy gain potential via adaptation. We expect our work to open new horizons for future research in VDU and real-world deployment in corresponding applications.
2 Related work
Unsupervised domain adaptation aims to improve the performance a different target domain, for model trained on the source domain. UDA approaches for closed-set adaptation (where classes fully overlap between the source and target domains) can be categorized into four categories: (i) distribution aligmment- based, (ii) reconstruction- based, and (fii) adversarial based, and (iv) pseudo-labeling based. Distribution alignment-base based approaches feature aligning mechanisms, such as moment matching (Peng et al., 2019) or maximum mean discrepancy (Long ct al.] [015 TTeng ct al. (004)). Reconstruction- based approaches reconst.ruct source and larget data with a shared encoder while performing supervised classification on labeled data GGhifary et al. 20066), use cycle consistency to further improve domain-specifer reconstruction Muurez
ct al.
2018 Hoffman et al. 2018). Inspired by GANs, adversarial lcarning based UDA approaches use two-
player games to disentangle domain invariant and domain specific features (Ganin & Lempitsky 2015 Long
2018 Shu et al.] 2018). Pseudo-labeling (or self-training approaches jointly optimize a model on the labeled source and pseudo-labeled target domains for adaptation (Kumar et al.) 2020 Liu et al. 2021 French et al., (0177). Overall, all UDA approaches need to access both labeled source data and unlabeled larget data during the adaptation which is a special case for the more challenging setting of TTA and we show how our approach can be modified to be used for UDA.
et al.
Test-time adaptation corresponds to source-free domain adaptation, that focuses on the more challenging setting where only source model and unlabeled target data are available. The methods often employ an unsupervised or self-supervised cost function. [ado et al. (2020) adopted the training time BN and used it at test time to adapt to test data efficiently. In standard practice, the batch normalization statistics are frozen to particular values after training time that are used to compute predictions. Nado al. (2020) proposed to recalculate batch norm statistics every batch at test time instead of re-using values from training time as mechanism to adapt to unlabeled test data. While this is computationally inexpensive and widely adoptable to different settings, we empirically show that it cannot fully overcome performance degradation issue when there is a large distributiio mismatch between source and target domains. TENT (Wang et al.) (021b) utilizes entropy minimization for test-time adaptation encouraging the model to become more "certain" " on target predictions regardless of their correctness. In the beginning of the training when predictions tend to be inaccurate, entropy minimization can lead to error accumulation since VDU models create long sequence of outputs per every document resulting in noisy training. SHOT [Liang et. al.
3

---

Published in Transactions on Machine Learning Research (06/2023)
2020) combines mutual information maximization with offline clustering-baser pseudo labeling. However, using simple offline pseudo-labeling can lead to noisy training and poor performance when the distribution shifts are large (Chen et al. 2022 Liu et al. 2021 Rizve et al., 2021; Mukherjee & Awadallah, 2020). We also use pscudo labeling in TOA but we propose online updates per batch for pscudo labels, as the model adapts to test data. Besides, we equip our method with pseudo label rejection mechanism using uncertainty, to ensure the negative effects of predictions that are likely to be inaccurate. Most recent TTA approaches in image classification use contrastive learning combined with extra supervision (Xia et al.] 2021; Huang et al., 2021 Wang et al. 2021a Chen et al. 2022]. In contrastive learning, the idea is to jointly maximize the similarity between representations of augmented views of the same image, while minimizing the similarity between representations of other samples). All these methods rely on self-supervised learning that utilize data augmenlation techniques, popular in CV while not yet being as effective for VDU. While we advocate for using SSL during TTA, we propose to employ multimodal SSL with pseudo labeling for the first time which is imore effective for VDU.
Self-supervised protraining for VDU aims to learn generalizable representations on large scale unlabeled data to improve downstream VDU accuracy (Appalaraju el al. 2021 Gu el al. 2021 Xu et al. 2020b
Lee et al. 2022 Huang et al. 2022). LayoutLM (Xu et al. 2020b) jointly models interactions between text and layout information using a masked visual-language modeling objective and performs supervised multi- label document classification on IIT-CDIP dataset (Lewis et al., 2006). LayoutLMv2 (Xu et al. 2020a) extends it by training on image modality as well, and optimizing text-image alignment, and text-image matching objective functions. DocFormer (Appalaraju et al. (2021] is another multi-modal transformer based architecture that uses text, vision and spatial features and combines them using multi-modal self-attention with multi-modal masked language modeling (MM-MLM) objective (as a modified version of MLM in BERT (Devlin et al. 2018)), an image reconstruction loss, and a text describing image loss represented as a binary cross-entropy to predict if the cut-out text and image are paired. FormNet (Lee et al. 2022 is a structure-aware sequence model that combines a transformer with graph convolutions and proposes rich attention that uses spatial relationship between tokens. UniDoc (Gu et al.] 2021) another multi-modal transformer based pretraining method that uses masked sentence modeling, visual contrastive learning, and visual language alignment objectives which unlike other methods, does not have a fixed document object detector (Li et al. 2021 Xu et al. 2020a). In this work, we focus novel TTA approach for VDU, that can be integrated with any pre-training method. We demonstrate DocTTA using the publicly available Lvy architecture pretrained on IIT-CDIP dataset.
3 DoCTTA: Test-time adaptation for documents
In this section, we introduce DocTTA, test-time adaptation framework for VDU tasks including key-value extraction, entity recognition, and document visual question answering (VQA).
3.1 DoCTTA framework
We define a domain as a pair of distribution D on inputs X' and a labeling function I: X y. We consider source and targel domains. In the source domain, denoted as (Da,la), we assume to have a model denoted (4) as fa and parameterized with 0, to be trained on source data {x19) wBs y where (9) X, and (1) V are document inputs and corresponding labels, respectively and n, is the mumber of documents in the source domain. Given the trained source model fa and leaving X, behind, the goal of TTA is to train Se on the target domain denoted as (De.4) where fe is parameterized with 0, and initialized with 0, and D, is defined over (2) 1 X2 without any ground truth label. Algorithm overviews our proposed DocTTA procedure.
Unlike single-modality inputs commonly used in computer vision, documents are images with rich textual in- formation. To extract the text from the image, we consider optical character recognition (OCR) is performed and use its outputs, characters, and their corresponding bounding boxes (dctails are provided in Appendix). We construct our input X' in either of the domains composed of three components: text input sequence X" of length denoted as (xf .. =xh") E [n(x)) image X 3x3 and layout X as 6-dimensional
4

---

Published in Transactions on Machine Learning Research (06/2023)
Diversity class predictions
8g 8,.........
Paoudo generation using model's predictions
Masked Linuuig Modaling
Visul/Token Ropresentation
[MASK]
T3
[MASK]
Transformer layers
Segment Emb.
B
A
A
A
1D Pos. Emb.
2D Pos. Emm.
Bux,
Box,
Box,
Boky
Box
Box,
Box,
Box.,
Box,
Box
Box
V2
V3
[CLS]
[MASK]
[MASK]
[SEP]
[MASK]
[MASK]
Visual Encoder
Parser
Input
Question, Words, Bounding
Words, Bounding
Bounding
boxss
Visual Document Understanding
VCA on Document
vey-value Extraction (Receipt Understancing)
Named Enttl Recognttoo (Form Understanding)
Figure 2: Illustration of how our approach, DocTTA, leverages unlabeled target data at test time to i) learn how to predict masked language given visual cues, ii) generate pseudo labels supervise the learning, and iii) maximize the diversity of predictions generate sufficient amount labels from all classes.
vector in the form of (Emin: Tmaz Mmin: Mmaz: w. h) representing bounding box associated with each word in the text input sequence. Note that for the VQA task, text input sequence is also prepended with the question. For the entity recognition task, labels correspond to the set of classes that denote the extracted text; for the key-value extraction lask, labels are values for predefined keys; and for the VQA task, labels are the starting and ending positions of the answer presented in the document for the given question. We consider the closed-set assumption: the source and target domains share the same class labels V, y with | C being the total number of classes.
Algorithm DocTTA for closed-set TTA in VDU
Input: Source model weights 0,, targel documents {=t) test-Lime training epochs ney test-Lime
training learning rate uncertainty threshold 7, questions for target documents in document VQA task
2: Initialization: Initialize target model fa, with 0, weights.
3: for epoch = 1 to n. do
4: Perform masked visual-language modeling in Eq.
Generate pseudo labels and accept a subset using criteria in Eq. 2 and fine-tune with Eq.
Maximize diversity in pseudo label predictions Eq. 4
0, - VDoeTTA
Update O, via total loss in Eq.
end for
5

---

Published in Transactions on Machine Learning Research (06/2023)
3.2 DoCTTA objective functions
In order to adapt Js in DocTTA, we propose three objectives to optimize on the unlabeled target data:
Objective I: masked visual language modeling (MVLM). Inspired by notable success of masked language modeling for NLP in architectures like BERT (Devlin et al. 2018), as well as the success of MVLM in (Xu et al.] 2020a) to perform sclf-supervised pretraining for VDU, we propose to employ MVLM at test time to encourage the model to learn better the lext represenlation of the test data given the 2D positions and other text tokens. The intuition behind using this objective for TTA is to enable the target model to learn the language modality of the new data given visual cues and thereby bridging the gap between the different modalities on the target domain. We randomly mask 15% of input text tokens among which 80% are replaced by special token [MASK] and the remaining tokens are replaced by a random word from the entire vocabulary. The model is then trained to recover the masked tokens while the layout information remains fixed. To do so, the output representations of masked tokens from the encoder are fod into a classifier which outputs logits over the whole vocabulary, to minimize the negative log-lieeihood of correctly recovering masked text tokens w/t given masked image tokens x. and masked layout s.:
LavL -Ez,EX, Σ log Po, (x. |0| xx
(1)
Objective I: self training with pscudo labels. While optimizing MVLM loss during the adaptation, we also generate pseudo labels for the unlabeled target data in an online way and treat. them as ground truth labels to perform supervised learning the target domain. Unlike previous pseudo labeling- based approaches for TTA in image classification which update pseudo labels only after each epoch (Liang et al,, 2020 2021 Wang et al., 2021a), we generate pseudo labels per batch aiming to use the latest version of the model for predictions. We consider a full epoch to be one training loop where we iterate over the entire dataset batch by batch. In addition, unlike prior works, we do not use a clustering mechanism to generate psoudo labels as they will be computationally expensive for documents. Instead, we directly use predictions by the model. However, simply using all the predictions would lead to noisy pseudo labels.
Inspired by (Rizve et al. 2021 ; in order to prevent noisy pscudo labels, we employ an uncortainty-awaro selection mechanism to select the subset of pseudo labels with low uncertainly. Note that in (Riave el al., 2021), pseudo labeling is used as a semi-supervised learning approach (not adaptation) and the selection criteria is based on both thresholding confidence and uncertainty where confidence is the Softmax probability and uncertainty is measured with MC-Dropout (Gal & Ghahramani] 2016). We empirically observe that raw confidence values (when taken as the posterior probability output from the model) are overconfident despite being right or wrong. Setting a threshold on pseudo labels' confidence only introduces a new hyperparameter without performance gain (sce Sec. 5.1. Instead, to select the predictions we propose to only use uncertainty, in the form of Shannon's 's entropy (Shannon 200T). We also expect this selection mechanism leads to reducing miscalibration due to the direct relationship between the ECEM and output prediction uncertainty, iee. when more certain predictions are selected, ECE is expected to reduce for the selected subset of pseudo labels. Assume ( be the output probability vector of the target sample x; (a) such that
(1) denotes the probability of class e being the correct class. We select a pseudo label v(( for a.(0) if the pe
uncertainty of the prediction u(pe"), measured with Shannon's entropy, is below specific threshold y and we update 6 weights with a cross-entropy loss:
d. [[u(p(") S 7]
(2)
Lox(0.) Ex, E Te log o (fe(zz)),
(3)
where a() is the softmax function. It should be noted that the tokens that are masked for the MVLM loss are not included in the cross-cntropy loss as the attention mask for them is zero.
Objective III: diversity objective. To prevent the model from indiscriminately being dominated by the most probable class based on pseudo labels, we encourage class diversification in predictions by minimizing
EExpected calibration error Naeini et
2015) which metric to measure calibration of model
6

---

Published in Transactions on Machine Learning Research (06/2023)
the following objective:
DDVV E Σ
c= Pe log Pe.
(4)
where p = E.,.x ((((e) is the output embedding of the target model averaged over target data, By obtain the full objective function in DocTTA below:
combining Eqs. B
and 4
DOCTTA = LMVLM + LCE + LDIV.
(5)
3.3 DoCTTA vs. DoCUDA
The proposed DoCTTA framework can be extended as an UDA approach, which we refer to DocUDA (sce Appendix for the algorithm and details). DocUDA is based enabling access to source data during adaptation to the larget. In principle, the availability of this extra information of source data can provide an advantage over TTA, however, as we show in our experiments, their difference is small in most cases, and even TTA can be superior when source domain is significantly smaller than the target domain and the distribution gap large, highlighting the efficacy of our DocTTA approach for adapting without relying on already-scen source data.
We note that the UDA version comes with fundamental drawbacks. From the privacy perspective, there would be concerns associated with accessing or storing source data in deployment environments, especially given that VDU applications are often from privacy-sensitive domains like legal or finance. From the com- putational perspective, UDA would yield longer convergence time and higher memory requirements due to joint learning from source data. Especially given that the stale-of-the-ar VDU models are large in size, this may become a major consideration.
4 DoCTTA benchmarks
To better highlight the impact of distribution shifts and study the methods that are robust against them, we introduce new benchmarks for VDU. Our benchmark datasets are constructed from existing popular and publicly-available VDU data to mimic real-world challenges. We have attached the training and test splits for all our benchmark datasets in the supplementary materials.
4.1 FUNSD-TTA: Entity recognition adaptation benchmark
We consider FUNSD, Form Understanding in Noisy Scanned Documents, dataset (Jaume et al. 2019 for this benchmark which is noisy form understanding collection consists of sparsely-filled forms, with sparsity varying across the use cases the forms are from. In addition, the scanned images are noisy with different degradation amounts due to the disparity in scanning processes, which can further exacerbate the sparsity issuc as the limited information might be based on incorrect OCR outputs. As a representative distribution shift challenge on FUNSD, we split the source and largel. documents based on the sparsity of available information measure. The original dataset has 9707 semantic entities and 31,485 words with 4 categories of entities question, answer, header, and other, where each category (except other) is cither the beginning or the intermediate word of sentence. Therefore, in total, we have 7 classes. We first combine the original training and test. splits and then manually divide them into two groups. We set aside 149 forms that are filled with more texts for the source domain and put 50 forms that are sparsely filled for the target domain. We randomly choose 10 out of 149 documents for validation, and the remaining 139 for training. Fig. (bottom row on the right.) shows examples from the source and target domains.
4.2 SROIE-TTA: Key-value extraction adaptation benchmark
We use SROIE [Huang et al. 2019) dataset with 9 classes in total. Similar to FUNSD, we first combine the original training and test splits. Then, we manually divide them into two groups based on their visual appearance source domain with 600 documents contains standard-looking receipts with proper angle of view and clear black ink color. We 37 documents from this split for validation, which we use to tune

---

Published in Transactions on Machine Learning Research (06/2023)
adaptation hyperparameters. Note that the validation split does not overlap with the target domain, which has 347 receipts with slightly blurry look, rotated view, colored ink, and large empty margins. Fig. 1 (bottom row on the left) exemplifies documents from the source and target domains.
4.3 DocVQA-TTA Document VQA adaptation benchmark
We use Doc (Mathew et al., 2021 a largo-scale VQA datasct with nearly 20 different types of documents including scientilic reports, letters, notes, invoices, publications, tables, etc. The original training and validation splits contain questions from all of these document types. However, for the purpose of creating an adaptation benchmark, we select 4 domains of documents: i) Emails & Letters (E), ii) Tables & Lists (T), iii) Figure & Diagrams (F), and iv) Layout (L). Since DocVQA doesn't have public meta-data to easily sort all documents with their questions, we use a simple keyword search to find our desired categories of questions and their matching documents. We use the same words in domains' names to search among questions (i.e., search for the words of "email" and "letter" for Emails & Letters domain). However, for Layout domain, our list of keywords is ["top", "bottom", "right", "left", "header", "page number"] which identifies questions that are querying information from a specific location in the document. Among the four domains, L and E have the shortest gap because emails/ /letters have structured layouts and extracting information from them requires understanding relational positions. For example, the name and signature of the sender usually appear at the bottom, while the date usually appears at top left. However, and T domains seem to have larger gaps with other domains, that we attributed to that learming to answer questions on figures tables requires understanding local information withing the list or table. Fig. (lop row) exemplifies some documents with their questions from each domain. Document counts in each domain are provided in Appendix.
5 Experiments
For entity rocognition and key-value extraction tasks, entity-level F1 score as the evaluation metric, whereas for the document VQA task, use Average Normalized Levenshtein Similarity (ANLS) introduced by (Biten et al.) 2019) (since it is recognized as a better measure compared to accuracy since it doesn't penalize minor text mismatches due to OCR errors).
Evaluation metrics:
Model architecture: In all of our experiments, we use LayoutLMv2BAsE architecture which has a 12-layer 12-head Transformer encoder with a hidden size of 768. Its visual backbone is based on Res.NeX111--FP, similar to that of MaskRCNN (He et al. 2017p. Overall, it has ~20M parameters. We note that approach is architecture agnostic, and hence applicable to any attention-based VDU model. Details training and hyper parameter tuning are provided in Appendix.
Basclincs: As our method is the first TTA approach proposed for VDU tasks, there is no bascline to compare directly. Thus, we adopt TTA and UDA approaches from image classification, as they can be applicable for VDU given that they do not depend on augmentation techniques, contrastive learning, or generative modeling. For UDA, we use two established and commonly used basclines DANN (Ganin & Lempitsky 2015) and CDAN (Long el al.] 2018) and for TTA, use the baselines batch normalization BN (lolle & Szegedy 2015 Nado et al. 2020), TENT (Wang et al. 2021b), and SHOT (Liang et al. 2020). Details on all the baselines are given in Appendix A.2.2] We also provide source-only, where the model trained on source and evaluated on target without any adaptation mechanism, and train-on-target, where the model is trained (and tested) on target domain using the exact same hyperparaneters used for TTA (which are found on the validation set and might not be the most optimal values for target data). While these two baselines don't adhere to any domain adaptation setting, they can be regarded the ultimate lower and upper bound for performance.
5.1 Results and Discussions
FUNSD-TTA: Table I shows the comparison between DocTTA and DocUDA with their corresponding TTA and UDA baselines. For UDA, DocUDA outperforms all other UDA baselines by large margin, and
8

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Fl score results for adapting source to target in FUNSD-TTA and SROIE-TTA benchmarks. Availability of the bbbeled/unlabllee data from source/target domains during adaptation in UDA and TTA settings and training phase in source-only only and train-on-targer settings are marked. Source-only and train-on- target serve lower and upper bounds, respectively for performance target domain. Standard deviations are in parentheses.
Labeled source data
Labeled target data
Unlabeled target data
DA category
Methods
FUNSD-TTA
SROIE-TTA
source-only
80.80 (0.12)
92.45 (0.08)
DANN
V
X
82.54 (0.14)
92.89 (0.13)
UDA
CDAN
X
83.72 (0.61)
93.36 (0.18)
DocUDA (ours)
V
X
V
89.76 (0.09)
97.38 (0.15)
BN
X
X
80.84 (0.93)
92.41 (0.45)
TTA
TENT
X
X
79.78 (1.28)
92.42 (0.87)
SHOT
X
X
80.89 (1.03)
92.78 (0.65)
DocTTA (ours)
X
X V
84.23 (0.88)
94.34 (0.43)
Brain-on-targe
99.89 (0.05)
100.0 (0.00)
improves 8.96% over the source-only. For the more challenging setting of TTA, DocTTA improves the F1 score of the source-only model by 3.43%, whereas the performance gain by all the TTA baselines is less than 0.5%. We also observe that DocTTA performs slightly better than other UDA baselines DANN and CDAN, which is remarkable given that unlike those, TocTTA does not have access to the source data at test time.
SROIE-TTA: Table 1 shows the comparison between UDA and TTA baselines vs. DocUDA and DocTTA on SROIE-TTA benchmark. Similar to our findings for FUNSD-TTA, DocUDA and DocTTA outperform their corresponding baselines, where DocTTA can even surpass DANN and CDAN (which use source data at test time). Comparison of DocUDA and DocTTA shows that for small distribution shifts, UDA version of our framework results in better performance.
DocVQA-TTA benchmark, where the ANLS scores are obtained by adapting each domain to all the remaining ones. The distribution gap between domains on this benchmark is larger compared to FUNSD-TTA and SROIE-TTA benchmarks. Hence, we also see a greater performance improvement by using TTA/UDA across all domains and methods. For the UDA setting, DoeUDA consistently outperforms adversarial- based UDA methods by a large margin, underlining the superiority of self-supervised learning and pseudo labeling leveraging labeled and unlabeled data at test time. Also in the more challenging TTA setting, DocTTA consistently achieves the highest gain in ANLS score, with 2.57% increase E + F and 17.68% on F E. Morcover, DocTTA significantly outperforms DANN on all domains and CDAN on 11 out of 12 adapiation scenarios, even though it does not utilize source data at test time. This demonstrates the efficacy of joint online pseudo labeling with diversity maximization and masked visual learning. Between DocUDA and DocTTA, it is expected that DocUDA performs better than DocTTA due to having extra access to source domain data. However, we observe three exceptions where DocTTA surpasses DocUDA by 1.13%, 0.79%, and 2.16% ANLS score on E - and T - F, and L F, respectively. We attribute this to: i) target domain (F) dataset size being relatively small, and ii) large domain gap between source and target domains. The former can create imbalanced distribution of source and target data, where the larger split (source data) dominates the learned representation. This effect is amplified due to (ii) because the two domains aren't related and the joint representation is biased in favor of the labeled source data. Another finding on this benchmark is that source model trained on a domain with a small dataset generalizes less compared to the one with sufficiently-laree dataset but has a larger domain gap with the target domain. Results for train-on-targot on each domain can shed light on this. When we use the domain with the smallest dataset (F) as the source, each domain can only achieve its lowest ANT.S score (39.70% on E, 24.77% on T, and 38.59% on L) whereas with T, second smallest. domain
DocVQA-TTA: Table 2 shows the results
9

---

Published in Transactions on Machine Learning Research (06/2023)
Table 2: ANLS scores for adapting between domains in Doc DoC-QA-TTA benchmark. Source-only and traret serve as lower and upper bounds, respectively for performance on target domain. Standard deviations are shown in Appendix.
Source:
Lettrs (E)
Figures&Diagrrant (F)
Tables& (T)
Layout (L)
Target:
F
T
L
E
T
L
E
F
L
E
F
T
sourcc-only
37.79
25.59
38.25
5.23
7.03
3.65
13.66
20.48
14.58
53.55
33.36
33.43
DANN
38.94
27.22
40.23
15.43
9.34
7.45
17.67
22.19
17.67
54.55
33.87
33.58
CDAN
39.08
29.33
41.29
16.99
11.32
10.23
27.87
25.23
27.66
56.82
34.27
34.81
DocUDA (ours)
39.23
43.54
57.99
24.21
15.76
20.45
53.19
29.91
47.81
61.09
34.85
41.80
BN
38.10
26.89
38.23
7.32
8.56
9.35
15.13
22.24
15.65
53.23
33.67
33.55
TENT
38.34
26.42
40.45
12.38
7.34
11.29
16.01
20.23
15.02
53.34
33.59
34.55
SHOT
38.98
27.55
39.15
14.34
10.10
13.21
22.56
24.33
19.15
56.23
34.56
35.65
DocTTA (ours)
40.36
35.28
49.35
22.91
15.67
16.01
35.67
30.70
26.32
59.84
37.01
39.10
train-on-target
95.28
93.54
95.01
39.70
24.77
38.59
84.59
70.66
83.73
92.32
91.36
93.41
Table 3: Ablation analysis on adapting from E to F, T,L. L in our DocVQA-TTA benchmark with different com- ponents including pseudo labeling, LMVLM, LDIV, and pseudo label selection mechanism using confidence only or together with uncertainty. Standard deviations are in parentheses.
Method
F
T
Source-only
37.79 (1.30)
25.59
(1.78)
38.25 (0.92)
DocTTA, conf.
32.67 (1.68)
21.50 (1.52)
6.71 (3.21)
DocTTA, conf. unc.
39.45 (0.87)
28.47 (0.72)
47.50 (0.51)
DocTTA, no LASLLM
35.66 (0.46)
25.72 (0.55)
45.88 (0.34)
DocTTA, no LDIV
34.32 (0.53)
25.17 (0.49)
46.36 (0.21)
DocTTA, no pseudo labeling
33.61 (1.65)
23.43 (0.87)
15.89 (1.35)
DocTTA
40.36 (0.53) 35.28 (0.76)
49.35 (1.20)
in dataset size in our benchmark (with 657 training documents), the scores obtained by train-on-target on E and increases lo 84.59% and 83.73% respectively. Thus, even we have access to entire larget labeled data, the limitation of source domain dataset size is still present.
Ablation studies: We compare the impact of different constituents of our methods on the Doe VQA-TAT benchmark, using model trained on Emails&Lotters domain and adapted to other three domains. Table 3 shows that pseudo labeling selection mechanism plays an important role and using confidence scores to accept pscudo labels results in the poorest performance, much below the source-only ANLS values and even worse than not using pseudo labeling. On the other hand, using uncertainty and raw confidence together to select pseudo labels yields the closest performance to that of the full (best) method (details are provided in A Appendix). MVLMM loss and diversity maximization criteria have similar impact DocTTA's performance.
6 Conclusions
We introduce TTA for VDU for the first time, with our novel approach DocTTA, along with new realistic adaptation benchmarks for common VDU tasks such as entity recoguition, key-value extraction, and docu- ment VQA. DocTTA starts from a pretrained model on the source domain and uses online pseudo labeling along with masked visual language modcling and diversity maximization on the unlabeled target domain. We propose an uncertainty-basee based online pseudo labeling mechanism that generates significantly more accurate pseudo labels in per-batch basis. Overall, novel TTA approaches result in surpassing the state-of-the-art
10

---

Published in Transactions on Machine Learning Research (06/2023)
TTA approaches adapted from computer vision, underlining the importance of VDU-specifie TTA approach to push the the state-of-the-art in VDU.
7 Reproducibility
We have included the details of our experimental setup such as the utilized compute resources, pretrained model, optimizer, learning rate, batch size, and number of epochs, etc. in Sec. A.2 of the Appendix. We have provided the details of our hyper parameter tuning and our search space in Section [A.2.3 of the Appendix. For our introduced benchmark datasets, the statistics of each dataset detailed in Section A.1.2 List of the all the training and validation splits for our proposed benchmarks are also provided in Supplementall/TTA Baaattaal/TAABBecmaaareee as json files. To ensure full reproducibility, we will release our code upon acceptance.
References
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 993-1003, 2021.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Minesh Mathew, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas. Icdar 2019 competition on scene text visual question answering. In 2019 International Conference on Document Analysis and Recognition (ICDAR), RJ, pp. 1563-1550 -1700 TEEE, 2019.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597 1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentun contrastive learning. arXiv preprint arxiv:2003. 200:2000300299,, 2020b.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12 22, 1983.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint 181004881, 2018.
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. an.ivv preprint arXio: 1066. 05208, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp. 1180-1189. PMLR, 2015.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjic Zhang, David Balduzzi, and Wen Li. Deep reconstruction- classification networks for unsupervised domain adaptation. In European conference on computer vision, pp. Springer, 2016.
Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unidoc: Unified pretraining framework for document understanding. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 39-50. Curran Associates, Inc., 2021. URL https::/pproeeddngg.

---

Published in Transactions on Machine Learning Research (06/2023)
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMTR, 2017.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-con. In Proceedings of the IEEE international conference on computer vision, pp. 2961 2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Judy Hoffman, Eric Tzeng, Tacsung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1989 1998. PMLR, 10 15 Jul 2018. URL httptttttt/:tteeeeetssss press/v80/hoffnne http://aa.maaaa....mmmmmmaaaaaa. htmml
mlr.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Information Processing Systems, 34, 2021.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layouthmv3: Pre-training for document ai with unified text and image masking, aixiv preprinl ar Xiv:22204 088877, 2022.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. ledar2019 competition on scanned receipt oct and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1516-1524 IEFF, 2019.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pp. 1-6. IEEE, 2019.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in relu networks. In International conference on machine learning, pp. 5436 5446. PMLR, 2020.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self- self-training for gradual domain adaptation. In International Conference on Machine Learning, pp. 5468-5479. PMLR, 2020.
Chen-Ye Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. Formnet: Struetural encoding beyond sequential modeling in form document information extraction. arXiv preprinl arXiv:2203.08. 08441, 2022.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, 1CML, volume 3, pp. 896, 2013.
David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David Grossman, and Jefferson Heard. Build- ing test collection for complex document information processing. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 665-666, 2006.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoe: Self-supervised document representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5652-5660, 2021.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pp. 6028 6039, 2020.
12

---

Published in Transactions on Machine Learning Research (06/2023)
Jian Liang Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEB Transactions on Pallern Analysis and Machine Intelligence (TPAMI), 2021. In Press.
Hong Liu, Jianmin Wang, and Mingsheng Long- Cycle self-training for domain adaptation. Advonces in Newral Information Processing Systems, 34, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp- 97-105. PMLR, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael T Jordan. Conditional adversarial domain adaptation. Advances in newral information processing systems, 31, 2018.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Xiv preprint 1rXi::1111.05100 2017.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2200-2209, 2021.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. In International Conference on Learning Representations, 2020. URL httpp:://pperreieew.net///////// nwt/forummm
id=ByxGkySKaH
Jose G Moreno-Tone Torens, Troy Raeder, Rocío Alaiz-Rodriguez, Nitesh V Chawla, and Francisco Herrera. A unifying view dataset shift in classification. Pattern recognition, 45(1):521-530) 2012.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. Uncertainty- self-training for text classification with few labels. arxiv preprint ar 1rXiv:2006.155 15315, 2020.
Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4500-4509, 2018.
Zachary Nado, Shreyas Padhy, Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. ar preprint orXiv:2006.10960 2020.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Alexandru Niculescu Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pp. 625 632, 2005.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, pirf-peeformance deep learning library. Advances in newral information processing systems, 32, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEFE International Conference Computer Vision, pp. 1406-1415, 2019.
Joaquin Canneea Masashi Sugiyama, Anton Schwaighofer, and N Lawrence. Covariate shift and local learning by distribution matching, 2008a.
Joaquin Quinonero-Candela Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learnóng. Mit Press, 2008b.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. In defense of pscudo-labeling: An uncertainty- pseudo-label selection framework for sumi-supe learning. In International Conference on Learning Representations, 2021. URL httppp//.ppppaneeeee..eee//e/e/a
13

---

Published in Transactions on Machine Learning Research (06/2023)
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communicatiins revieu, 5(1):3-55, 2001.
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A DIRT- approach to unsupervised domain adaptation. In International Conference Learning Representations, 2018. URL https://openrrevisi
net/foruu'iid-iiq-T-e
Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
Sunil Thulasidssan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Newral Information Processing Systems, 32, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In European conference on computer vision, pp. 776 194.794.22 Springer, 2020.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maxi- mizing for domain invariance. Xiv preprint 14 2014.
Dequan Wang, Shaoteng Liu, Sayna Ebrahimi, Evan Shelhamer, and Trevor Darrell. On-target adaptation. arXiv preprint Xrx:::2109.0010 01087, 2021a.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021b.
URL https::/openneviee wtt:
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning, pp. 23631- 23644. PMLR, 2022.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arv preprint (.Xix:66090084.. 2016.
Haifeng Xia, Handong Zhao, and Zhengming Ding- Adaptive adversarial network for source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 9010 9019, 2021.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, al. Layouthmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint ar Xiv:2001 14740, 2020a.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou, Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1192 1200, 2020b.
A Appendix
A.1 Dataset details
A.1.1
Licence
We use three publicly-vaailable datasets construct our benchmarks. These datasets can be downloaded from their original hosts under their terms and conditions:
FUNSD Jaune et al. (2019) License, instructions download, and term of use can be found at https://gui hta.//uulllllllmpee llaume github. httth///wo///////////o/////
14

---

Published in Transactions on Machine Learning Research (06/2023)
SROIE Huang et al. (2019) License, instructions to download, and term of use can be found at
https: //g com
DocVQA Mathow et al. [2021] License, instructions to download, and term of use be found at
https:// www. www.docvqa. org/
A.1.2 Dataset splits
We provide the list of the documents in the source and target domains for three benchmarks. Files located at Supplemental/TTA Berchaaencmmaks/... For FUNSD-TTA and SROIE-TTA, the validation splits have 10 and 39 documents, respectively, which are selected randomly using the seed number 42. Validation splits have a similar distribution as the source domain's training data. When performing TTA, we use the target domain data without labels - the labels are only used for evaluation purposes. Table 5 and Table t show the statistics of documents on source and larget domains FUNSD-TTA and SROIE-TTE TTA, respectively.
Table 4: Number of documents in the source and target domains in FUNSD-TTA and SROIE-TTA bench- marks. We use the validation set selected from the source domain to tune TTA algorithm's hyper parameters.
Table 5: FUNSD-TTA
Table 6: SROIE-TTA
Source Training 139
Source Training
600
Source Validation
10
Source Validation
39
Source Evaluation,
Source Evaluation,
Target Training,
50
Target Training, Target Evaluation
347
Target Evaluation
For DocVQA-TTA benchmark, we always choose 10% of source domain data for validation using the same seed (42).
Table 7: Number of documents in each domain of our DocVQA-TTA benchmark.
Layout (L)
Emails&LLetters (E)
Tables& (T)
Tigures&Diaarrann (F)
Source Training
1807
1417
592
150
Source Validation
200
157
65
17
Source Evaluation,
Target Training,
512
137
187
49
Target Evaluation
A.1.3 Text embeddings and OCR annotations
For all the benchmarks, we use dffcialll-proviider OCR annotations for each datasets. For the tokenization process, we follow Xu et al. (2020a) where they use WordPiece Wu al. (2016) such that each token in the OCR text sequence is assigned to certain segment of 8; {[A), [B] prepended by [CLS) if it is the starting token and/or appended by [SEP] if it is the ending token of the sequence. In order to have fixed sequence length in each document, extra [PAD] tokens are appended to the end, if the sequence exceeds a maximum length threshold (which is 512 in this work).
A.2
Experiments Details
A.2.1 Training.
We use PyTorch (Paszke al. 2019) on Nvidia Tesla V100 GPUS for all the experiments. For source training, use L pre-trained IIT-CDIP dataset. and fine-tune it with labeled source
15

---

Published in Transactions on Machine Learning Research (06/2023)
data our desired task. For all VDU tasks, we build task-specific classifier head layers over the text embedding of TayoutIMv2RASS outputs. For entity recognition and key-value extraction tasks, we use the standard cross-entropy loss and for DocVQA task, we use the binary cross-entropy loss on each token to prodict whether it is the starting/ /nnding position of the answer or not. We use AdamW (Loshchilov & Hutter 2017) optimizer and train source model with batch sizes of 32, 32, and 64 for 200, 200, and 70 epochs with a learning rate of 5 10-5 for entity recognition, key-value extraction, and DocVQA benchmarks, respectively with an exception of Figures & Diagrams domain on which we used a learning rate of 10-5, For BN and SHOT baselines, we followed SHOT implementation for image classification and added fully connected layer with 768 hidden units, followed by batch normalization layer right before the classification head. Note that we used the same backbone architecture (LayoutLMv2) as was used in DocATA and DocUDA for all the baselines methods. For example, ResNet backbone in DANN, CDAN, and SHOT methods was replaced with Lay architecture. Therefore all the baselines receive the inputs exactly similar to our approach.
Uncertainty and confidenco-aware pscudo labeling For uncertainty-aware pscudo labeling, set threshold (7) above which pseudo labels are rejected to be used for (raining. Likewise, for confidence-aware pseudo labeling, set a threshold for the output probability values for the predicted class below which pseudo labels are rejected. For the combination of the two, a pscudo label which has confidence (output probability) value above the threshold and uncertainty value (Shannon entropy) below the maximum threshold is chosen for self-training. We used confidence threshold of 0.95 and tuned the uncertainty threshold to be either 1.5 or (see below).
A.2.2 Baselines details
In this section we discuss more details about UDA and TTA baselines in our work and how they are different from our proposed approach, DocUDA and DocTTA, in both settings.
UDA bascliens. We compared DocUDA against DANN (Ganin & Lempitsky 2015 and CDAN (Long et al. 2018) methods. DANN was proposed as a deep architecture termed as Domain-Adversal Network (DANN) for unsupervised domain adaptation which consists of a feature extractor, a label predictor, and a domain classifier. The feature extractor is similar to a generator which tries to generate domain- independed fcatures for confusing the domain classificr. On the other hand, the domain classifier acts as the discriminator which tries to prediet whether the extracted features are from the source or target domain. Lastly, the label predictor which is trained on the extracted features of the labeled source domain, tries to predict the correct class for a target sample. DANN can be trained with a special gradient reversal layer (GRL) which authors claim gives superior performance compared to when GRL it is not used in the architecture. DANN is an adversarial adaptation method and hence is substantially different from DocUDA which uses pseudo labeling and self-supervised learning in the form of masked visual language modeling. CDAN (Long et al. 2018) or Conditional Domain Adversarial Network uses discriminative information conveyed in the classifier predictions to assist adversarial adaptation. The key to the CDAN approach is novel conditional domain discriminator conditioned the cross covariance of domain-specific the
Neural
feature representations and classifier predictions. They further condition the domain discriminator uncertainty of classifier predictions, prioritizing the discriminator on easy-to-trane to-ttansttrferr examples. CDAN is another adversarial learning- based adaptation method that tries to disentangle domain specific and domain invariant features for adaptation whereas DocUDA trics to leverage the model's knowledge on both domains at the same time without disentangling or conditional learning on discriminalive features.
TTA baselines. We compared DocTTA with BN [Iffe & Szegedy 2015 Nado et al. 2020 TENT WWagg
et al. 2021b) and SHOT (Liang et al. 2020). BN or batch normalization is widely used (training time)
technique proposed by Toffe & Szegedy 2015). Nado et al. (2020) adopted the training time BN and used it at test time which is a simple but surprisingly effective method and can be implemented with one line of code change. In standard practice, the batch normalization statistics are frozen to particular values after training time that are usod to compute predictions. Nado et al.) (2020) proposed to recalculate batch norm statistics on every batch at test time instead of re-using values from training time as a mechanism to adapt to unlabeled test data. While this is computationally inexpensive and widely adoptable to different settings, it is
16

---

Published in Transactions on Machine Learning Research (06/2023)
not enough to overcome performance degradation issue when there is large distribution mismatch between source and target domains. As shown in our experiments and also shown in the literature (Wang et al. 2021b a Chen et al. 2022) this method is usually outperformed by methods which rely on stronger adaptation mechanisms such as pscudo labeling. TENT (Wang et al. 2021b) proposed to adapt by test entropy minimization where the model is optimized for confidence as measured by the entropy of its predictions. TENT estimates normalization statistics and optimizes channel-wise affine transformations to update online on each batch. While this method was shown to improve BN which only updates BN statistics, it suffers from accumulation of error which can occur in the beginning of the training when predictions tend to be inaccurate especially because VDU models create a long sequence of outputs per every document resulting in noisy training. DocTTA's 's training is siguificantly different from TENT as it primarily uses pseudo labeling, MVLM and diversity losses for adaplation. Lastly, SHOT Liang et al. (020)) can be considered as the closest work to ours where combines mutual information maximization with offline clustering- based pseudo labeling. However, using simple offline pscudo-labeling can load to noisy training and poor performance when the distribution shifts are large (Chen et al. 2022; Liu et al., 2021 Rizve et al. 2021 Mukherjee & Awadallahi 2020j. In DocTTA as the model adapts to test data. Besides, we equip our method with pseudo label rejection mechanism using uncertainty, to ensure the negative effects of predictions that are likely to be inaccurate. DocTTA also leverages masked visual langugae modeling on target domain which contributes in learning the structure of the target domain whereas SHOT only depends on pseudo labeling.
also use pseudo labeling but we propose online updates per batch for pseudo labels,
A.2.3 Hyper parameter tuning
We use validation set (from source domain) in each benchmark for hyper parameter tuning. Although not optimal, it is more realistic to assume access to any labeled data in the target domain. We used simple grid search to find the optimal set of hyper parameters with the following search space:
Learning rate E {10 6,25 x 50-5,5 10 10-5}
2.5 Weight decay E 0.,0.01}
Batch size E (1,4,5,8, 8, 40, 68,
Uncertainty threshold y E [1.5,2]
A.3 Measuring confidence after adaptation with DoCTTA
For reliable VDU deployments, confidence calibration can be very important, as it desired identify when the trained model can be trusted so that when is not confident, a human can be consulted. In this section, we focus confidence calibration and analyze how DocTTA affects it. Figure 3 Gillustrates reliability diagrams for adapting from Emails & Tetters (E) to T, F, and L domains in DocVQA-TTA benchmark. We compares model calibration before and after DocA for 'starting position index of an extracted answer' in documents. We illustrate calibration with reliability diagram (DeGroot & Fienberg 1983 Niculescu- & Caruana 2005), confidence histograms (Guo et al. 2017). and the ECE metric. The reliability diagram shows the expected accuracy as a function of confidence. We first group the predictions on target domain into set of bins (we used 10). For each bin, we then compute the average confidence and accuracy and visualize them (top red plots in Fig. 3). The closer the bars are to the diagonal line, the more calibrated the model would be. Also, lower ECE values indicate better calibrations. It is observed that calibration improves with DocTTA. From this plot, we can also mcasure ECE as a summary metric (the lower, the better calibration). For instance, DocTTA on E + L yields significantly lower ECE, from 30.45 to 2.44. Although the reliability diagram can explain model's calibration well, it does not show the portion of samples at given bin. Thus, we use confidence histograms (sce bottom of Fig. 3) where the gap between accuracy and average confidence indicative of calibration. Before adaptation, the model tends be overconfident w whereas, after adaptation with Doc, the gap becomes drastically smaller and nearly overlaps.
17

---

Published in Transactions on Machine Learning Research (06/2023)
Table 8: Standard deviations (in parentheses) for ANLS scores shown in Table 2 of the main paper for adapting between domains in DocVQA-TTA benchmark (Part I).
Source:
Letter&&etteers (E)
Figures &Diggrams (F)
Target:
F
T
L
E
T
L
Source-only
37.79 (1.30)
25.59 (1.78)
38.25 (0.92)
5.23
(2.86)
7.03 (2.42)
3.65 (2.76)
DANN
38.94 (1.20)
27.22 (1.32)
40.23 (1.78)
15.43 (1.34)
9.34 (3.23)
7.45 (3.29)
CDAN
39.08 (1.59)
29.33 (0.82)
41.29 (2.80)
16.99 (1.56)
11.32 (2.43)
10.23 (2.54)
DocUDA (ours)
39.23 (1.42)
43.54 (0.91)
57.99 (0.12)
24.21 (0.72)
15.76 (0.67)
20.45 (0.43)
BN
38.10 (1.01)
26.89 (0.59)
38.23 (0.98)
7.32 (2.43)
8.56 (2.34)
9.35 (3:21)
TENT
38.34 (0.74)
26.42 (0.52)
40.45 (0.81)
12.38 (3.12)
7.34 (2.54)
11.29 (2.45)
SHOT
38.98 (0.89)
27.55 (0.81)
39.15 (1.23)
14.34 (3.87)
10.10 (1.34)
13.21 (2.54)
DocTTA (ours)
40.36 (0.53)
35.28 (0.76)
49.35(1.20)
(0.45)
15.66700778)
16.01 (1.18)
Train-on-target
95.28 (1.32)
93.54 (0.91)
95.01 (1.34) 39.70 (1.02)
24.77 (0.23)
38.59 (0.78)
Table 9: Standard deviations (in parentheses) for ANLS scores shown in Table 2 of the main paper for adapting between domains in DocVQA-TTA benchmark (Part IT).
Source:
Tables&Lists (T)
Layout (L)
Target:
E
F
L
E
F
T
Source-only
13.66 (1.67)
20.48 (1.54)
14.58 (1.30)
53.55
(1.76)
33.36 (1.84)
33.43 (1.94)
DANN
17.67 (1.49)
22.19 (2.34)
17.67 (2.58)
54.55 (0.72)
33.87 (1.02)
33.58 (0.28)
CDAN
27.87 (1.82)
25.23 (1.24)
27.66 (2.62)
56.82 (0.62)
34.27 (0.82)
34.81 (0.43)
DocUDA (ours)
53.19 (0.92)
29.91 (0.45)
(0.41)
61.09 (0.08)
34.85 (0.16)
41.80 (0.06)
BN
15.13 (2.04)
22.24 (1.29)
15.65 (2.43)
53.23 (1.08)
33.67 (1.17)
33.55
(1.55)
TENT
16.01 (2.83)
20.23 (2.61)
15.02 (2.83)
53.34 (0.93)
33.59 (1.82)
34.55 (1.44)
SHOT
22.56 (1.12)
24.33 (2.54)
19.15 (2.39)
56.23 (1.20)
34.56 (1.02)
35.65 (1.23)
DoCTTA (ours)
35.67 (1.10)
30.70 (0.80)
26.32 (1.27)
59.84 (0.04)
37.01 (0.05)
39.10 (0.06)
Train-on-target target
84.59 (1.02)
70.66 (0.82)
83.73 (0.23)
92.32 (0.01)
91.36 (0.04)
93.41 (0.05)
1.2
Source-only (no TTA)
Cop
1.2
-. test-time adaptation
Gap
1.0
Tafter tast-time adaptation
Gap
1.0
Gap
DoCTTA
Aec-racy
Accuracy
Accurary
Expected Accumnzy
0.8
Accuracy
Dxpeened Nerumey
0.5
0.6
0.4
0.4
0.2
33.-30.45
CCE=6,99
ECE=10.70
ECE -
0.0
0.0
0.0
Count
Count
Count
Aug, confidence
Accurscy
Accuracy Avg. confidance
50
confidence
0.0
0.2
0.4
0.6
0.8
1.0
0.0
02
0.4
0.8
1.5
0.8
02
6'4
0.6
088
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Confidence
Cont donce
Confidence
Figure 3: Comparing confidence calibration with and without DocTTA when adapting from Emails & Letters domain to other domains in DocVQA-TTA benchmark.
A.4 Standard deviations
Here we show the standard deviations obtained over 3 seeds for results reported in Table 2 of the main paper in Table 8 (part and g (part IT), respectively.
18

---

Published in Transactions on Machine Learning Research (06/2023)
A.5 DocUDA algorithm
In DocUDA, we use source data during training on target at test time. Therefore, DocUDA has an additional objective function which is a cross-cntropy loss using labeled source data:
LoEs (0.) = EB,, Σ Ve Volog Mllloo(eelen (fe(zs))
(6)
And the overall objective function of DocUDA is Eq. plus Eq.
DOCUDD LMVLM + Loe LDIV + LCES
(7)
Algorithm 2
shows how DocUDA is performed on VDU tasks.
Algorithm 2 DocUDA for closed-set UDA in VDU
1: Input: labeled source documents {a' Me target documents {af)" test-time training epochs Rer test-time training learning rate uncertainty threshold y
(6) 1.
2: Initialization: Initialize target model fo, with LayoutLMv2 weights trained on IIT-CDIP dataset.
for epoch 1 to ne do Perform masked visual-language modeling in Eq.
Generate pseudo labels and accept subset using criteria in Eq. 2 and fine-ture with Eq.
6: Maximize diversity in pseudo label predictions Eq. A
7: Perform supervised training using labeled source data with Eq.
8: Oe + - a
Update Og via total loss in Eq. 7
end for
A.6 Qualitative Results
Here show randomly selected document from our FUNSD-TTA benchmark. Comparing the results before and after using DocTTA shows that our method can refine some wrong predictions made by the unadapted model.
(7
L'
Belcen Adapiation
Aftee Artepratiinn with DocTTA
Ground trush lebels
Figure 4: From left to right show predictions made by (i) unadapted model, (ii) after using DocTTA iii) ground truth labels.
19

---

Published in Transactions on Machine Learning Research (06/2023)
A.7 Additional Baseline Results
Here show the performance of our model against AdaContrast (Chen et al. 2022) for TTA and SHOT- UDA OLiang et al. 2020) on DocVQA-TTA benchmark when adapting from Emails & Letters domain to F. T, and L.
Table 10
Source
Emails& Letters (E)
Target
F
T
L
SHOT (UDA)
39.02
31.35
48.87
DocUDA (ours)
39.23
43.54
57.99
AdaContrast. (TTA)
37.21
27.43
38.69
DoCTTA (ours)
40.36
35.28
49.35
A.8 Layout Illustration
In method, layout refers to bounding box associated with each word in the text input sequence and is represented with 6-dimensional vector in the form of (Zmin, Emary Mmnnn Uman, w. h). where (Xmin Umin) corresponds to the position of the lower left corner and (Imoz: Fmaz) represents the position of the upper right corner of the bounding box and w and h denote the width and height the box, respectively as shown below
Figure Illustaaioon of layout as a bounding box associated with each word the text input sequence.
20


Reviews:
[{"review_id": "xUmwGoK7Se", "reviewer": "Reviewer_hWWC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The empirical results show that the proposed , unsupervised approach  that doesn't use source data helps to improve the performance on the target domain.\n- Paper is generally well written\n\n### Weaknesses\n- The proposed approach is a trivial application of existing methods for domain adaptation in document image understanding. The proposed approach is essentially an in-domain self supervised  training using MVLM  on the target data and a supervised training using pseudo labels. Although these experiments are worthy contributions,  the technical contribution is limited.  MVLM based pretraining is quite popular in vision and language community. Self supervised learning using this objective, on the target domain is a a trivial contribution. The second objective is training using pseudo labels, which is widely used in domain adaptation literature.\n- For the FUNSD-TTA, what is the rationale behind the splitting criteria - sparse vs dense text. How does that make a \"shift\" from domain adaptation perspective ? Similarly for SROIE-TTA, the criteria for the split is more or less intuitive and not backed by any supporting evidence or reasoning.\n- ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This work introduces a new benchmark for domain adaptation for  document understanding tasks such as form understanding and VQA. \n- As the authors observe domain adaptation for document images understanding is not a well studied problem. Authors show that a simple in-domain self supervised training using MLVM  and retraining using pseudo labels can help to some extent.\n- \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In page 2 authors write \"pseudo-labelling can result in accumulation of errors..\" in the case of VDU. Can the authors please explain this in the context of VQA on document images. This reviewer could not understand what is written above it either.\n- Authors use a baseline from Nado et al. that uses batch normalization. But this work is not discussed anywhere in the related works. Request the authors to add details of of all the three baselines and how it compares with the approach proposed in this work.\n- In Fig2, align the text on the left with the  rows that show different embedding. For example, 1D pos. emb. is shown against the segment embedding\n\n-  It will be helpful if an illustration of DocTTA framework for DocVQA  is provided as it comes with an extra input in the form of a question. The current Figure only shows an image in the input.\n- In page 5, it is written that layout X^B is a 6 dimensional vector. When question tokens are introduced, what will be this vector be, for a question token?\n- One aspect that this reviewer is confused about is how the training happens for DocTTA. Is  MVLM and supervised training using MVLM happening concurrently ?  If that is the case, this reviewer has following questions in this regard\n\n1. For a token classification style task on datasets like FUNSD, if a token from the document is maksed, how can the model generate a label for it  (label as in the label corresponding to the semantic entity)\n\n2. For the VQA task, when some one tokens are masked does the model have enough information to find the answer to the question\n\n3. Would not it be better to perform the MVLM first, and then use pseudo-label based supervised training. Will it be possible for the authors to perform this experiment ?\n\n- In Algorithm 1, Input takes in only target documents. However for VQA, would not question also be a part of the input?\n- Page 6. in section 3.3, third sentence, there is a typo.\n-  How are the baselines used in this work modified/adapted for the problem of VQA. This reviewer feels that more details on what are these baselines and how are they used for the VDU problem needs to be added\n", "broader_impact_concerns": "The paper does not include any statement(s) on the ethical implications. This reviewer could not think of any.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "paKultoWt5", "reviewer": "Reviewer_2HCZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem is important and relatively under-explored in the field. I can see this work sparking more interest in this domain.\n- Most of the paper is clearly written and easy to read. For example, the explanation for how the pre-training happens (via the masked visual language modeling task, self-training with pseudolabels, and diversity objective), were straightforward.\n- The authors also introduce new benchmarks for the VDU domain adaptation task that are constructed from existing, publicly available data - FUNSD for entity recognition, SROIE for key-value extraction, and DocVQA for visual question-answer.\n - The authors provided extensive detail on how the datasets were constructed, their design decisions on how to construct the datasets, etc.\n\nWeaknesses:\n- Although there do not seem to exist relevant baselines to compare DocTTA against, I'm curious why the authors decided to adapt methods from quite a while back as their baselines (e.g. DANN, which is from 2015). \n- The authors mention in the paper (as well as in their ablation studies) that the way in which they handled the pseudo-labeling is quite important for achieving good performance, but it's not clear to me why this is the case.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes DocTTA, a test-time adaptation algorithm for visual document understanding (VDU) which effectively adapts representations learned from self-supervised learning algorithms to source-free domain adaptation settings in new target domains. The authors propose new benchmark datasets for this task and demonstrate the efficacy of their approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- When looking at Table 2, the discrepancies in ANLS scores between all the existing approaches and train-on-target are quite large. Would the authors address this in the text?\n- Elaborating a bit more on why they decided to adapt methods such as DANN and CDAN as their baselines would be helpful.\n- Talking a bit more in detail and clarifying the pseudo-labeling component of the adaptation procedure would be helpful. Section 3.2 was a bit difficult to parse.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8KzQFMuDZk", "reviewer": "Reviewer_JqyT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Interesting topic. Domain adaptation in visual document understanding has real frequent use cases in the real world.\n- The tasks build on prior datasets, and the source vs. target domain split is clever.\n- Uncertainty-based selection mechanism (the second part of the approach) seems very crucial. \n- It’s interesting that the ablation which removes the third objective (maximizing the diversity of pseudo-labels) has much poorer results. \n- It’s quite surprising to me that the ablation which removes masked visual LM training is better than the ablation that only removes pseudo labeling. \n- Writing is clear. \n\nConcerns\n\nQuestion: No coefficient for Equation (5) when combining different objectives. Would the authors expect that different coefficients would make the results better/worse? \n\nSection 3.2, objective II: “We empirically observe that raw confidence values are overconfident despite being right or wrong.” Is there evidence for this statement? Apologies I’m missing the details.\n\nAlgorithm 1: The equations don’t seem to correspond to the description. Should Equation 2 and Equation 3 be swapped?\n\nThe tasks have synthetically and manually designed splits, and it’s unclear whether a method that performs well on the proposed three tasks would imply that the method performs well on the noisier real-world test-time domain shifts. But this is not a huge concern imo. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to do domain adaptation (adapting models trained on a source main to a target domain) using unlabeled target-domain document data. The approach assumes that optical character recognition is already performed, so we already have information on text and bounding boxes. Then, the approach involves\n- (1) masked visual language modeling, \n- (2) pseudo labeling: specifically, first performing per-batch pseudo-labeling and then performing uncertainty-aware (based on thresholds according to Shannon entropy) selection according to Equations 2-3, \n- (3) maximizing diversity of pseudo-labels in case the pseudo-labels get dominated by one or very few classes only. \n\nThe tasks involve entity recognition, key-value extraction, and document visual question answering. The tasks are created by the authors using existing datasets: specifically, they split existing datasets to a source domain and a target domain. \n- For the entity-recognition adaptation task (“FUNSD-TTA”), an existing dataset is split into the source domain containing forms with more texts and the target domain containing forms that are only sparsely filled. \n- For the key-value extraction adaptation task (“SROIE-TTA”), the source domain contains standard-looking receipts (more description in the paper), and the target domain contains blurry or rotated or colored or other non-standard-looking receipts. \n- For document VQA adaptation (“DocVQA-TTA”), the dataset is split into different categories according to topic/domain, and then combined into source vs. target domains.\n\nResults are better than state-of-the-art domain adaptation approaches according to automatic metrics: the first two tasks use entity-level F1, and the third task uses average normalized Levenshtein. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Addressing the above concerns.\n\nOverall I think the work is quite sound. I'm also curious: what are the other potential impactful use cases of the authors' approach (other than the three described tasks)?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledged that the approach, which is based on self-supervised learning (via masked visual-language modeling) on the source domain followed by pseudo-labeling on the target domain, is reasonable and well-supported by convincing empirical evidence. One reviewer expressed concern that both self-supervised learning and pseudo-labeling have been extensively studied in the literature. The authors clarified that while these methods are well-established in the literature, their usage in the context of test-time adaptation for VDU is novel. The reviewers generally found that the overall contribution of this paper, which also includes a new benchmark, outweighs this concern, and they recommended acceptance. 

This AE agrees with the reviewers' assessment and recommendation for acceptance. The novel application of self-supervised learning and pseudo-labeling in the context of test-time adaptation adds value to the existing literature, and the new benchmark addresses an important research problem. Therefore, based on the reviewers' positive evaluations and the AE's own evaluation, the paper is recommended for acceptance.

Decision: Accept (poster)