Test-Time Adaptation for Visual Document Understanding
Authors: Sayna Ebrahimi
Sercan O Arik
Tomas Pfister
Venue: Submitted to ICLR 2023 (2023)
Keywords: Test-time adaptation
source data-free domain adaptation
visual document understanding

Abstract: Self-supervised pretraining has been able to produce transferable representations for various visual document understanding (VDU) tasks. However, the ability of such representations to adapt to new distribution shifts at test-time has not been studied yet. We propose DocTTA, a novel test-time adaptation approach for documents that leverages cross-modality self-supervised learning via masked visual language modeling as well as pseudo labeling to adapt models learned on a \textit{source} domain to an unlabeled \textit{target} domain at test time. We also introduce new benchmarks using existing public datasets for various VDU tasks including entity recognition, key-value extraction, and document visual question answering tasks where DocTTA improves the source model performance up to 1.79\% in (F1 score), 3.43\% (F1 score), and 17.68\%  (ANLS score), respectively.

Under review as a conference paper at ICLR 2023
TEST-TIME ADAPTATION FOR VISUAL DOCUMENT UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
For visual document understanding (VDU), self-supervised pretraining has been shown to successfully generate transferable representations, yet, effective adap- tation of such representations to distribution shifts lest-time remains be an unexplored area. We propose DocTTA, a novel rest-time adaptation method for documents, that does source-free domain adaptation using unlabeled target docu- ment data. DocTTA leverages cross moos-mto self- self-supprvised learning via masked visual language modeling as well as pseudo labeling to adapt models learned on a source domain to an unlabeled targer domain at test time. We also introduce new benchmarks using existing public datasets for various VDU tasks including entity recognition, key-value extraction, and document visual question answering tasks, at which DocTTA improves the source model performance up to 1.79% (F1 score), 3.43% (F1 score), and 17.68% (ANLS score), respectively.
INTRODUCTION
Visual document understanding (VDU) is on extracting structured information from document pages represented in various visual formats. It has a wide range of applications: including jux/invoicc/norrgagelllaini processing, identit//iisk/accine verification, medical records under- standing, compliance management, etc. These applications affect operations of businesses from major industries and lives of the general populace. Overall, it is estimated that there are trillions of documents in the world.
Machine learning solutions for VDU should rely on overall comprehension of the document content, extracting the information from text, image, and layout modalities. Most VDU tasks including key-value extraction, form understanding, document visual question answering (VQA) are often tackled by self-supervised pretraining, followed by supervised fine-tuning using human-labeled data (Appalaraju et al. 2021 I Gu et al.] 2021 Xu et al., 2020b a Lee et al. 2022 Huang al. 2022). This paradigm uses unlabeled data in task-agnostic way during the pretraining stage and aims to achieve better generalization at various downstream tasks. However, once the pretrained model is fine-tuned with labeled data on source domain, a significant performance drop might occur if these models are directly applied new unseen target domain phenomenon known domain shift Junonner--Cnnder et al. 2008a b. Moreno-Toree et al. 2012).
The domain shift problem is commonly encountered in real-world VDU scenarios where the training and test-time distributions are different, a common scenario due to the tremendous diversity observed for document data. Fig. D exemplifies this, for key-value extraction task across visually different document templates and for visual question answering task on documents with different contents (figures, tables, letters etc.) for information. The performance difference due to this domain shift might reduce the stability and reliability of VDU models. This is highly undesirable for widespread adoption of VDU, especially given that the common use cases are for high-stakes applications from finance, insurance, healthcare, or legal. Thus, the methods to robustly guarantee high accuracy in the presence of distribution shifts would be of significant impact. Despite being a critical issue, to the best of our knowledge, prior work has studied post-training domain adaptation for VDU.
Unsupervised domain adaptation (UDA) methods attempt to mitigate the adverse effect of data shifts, often by training a joint model the labeled souree and unlabeled target domains that map both domains into a common feature space. However, simultaneous access to data from source and target domains may not be feasible for VDU due to privacy concerns associated with source data access,

---

Under review as a conference paper at ICLR 2023
Olagame
@ cus des lester?
Grital
Seyire in Sctedion
vike Dituction
Newd brily tooppaoo ay!
Noreet brllyy Recepaaion
Searie
Surget
Aeane
Tagert
Figure I: Distribution shift examples for document samples from the proposed benchmark, Doc TTA. Top row: shows documents from four domains: (i) Emails & Letters, (ii) Figures & Diagrams, (iii) Layout, (iv) Tables& & Lists, from our VQA benchmark derived from DocVQA dataset (Mathew
et al. 202T). Bottom left: documents from source and target domains for key-value information extraction task from SROIE (Huang et al., (2099) receipt dataset. Bottom right: documents from source and target domains for named entity recognition task from FUNSD (Jaume et al., 2019)
dataset.
given legal, technical, and contractual constraints. In addition, the training and serving may be done in different computational environments, and thus, the expensive computational resources used for training may not be available. Test-time adaptation (TTA) (or source-free ffre domain adaptation) has been introduced to adapt a model that is trained on the source to unseen target data, without using any source data (Liang et al., 2020 Wang et al. 2021b Sun al. 2020, Wang et al. 2021 a Chen et al. 2022] Huang et al. 2021). Existing TTA methods have mainly focused on image classification and semantic segmentation tasks, while VDU remains unexplored, despite the clear motivations of the distribution shift besides challenges for the employment of standard UDA.
Since VDU significantly differs from other computer vision (CV) tasks, applying existing TTA methods in straightforward manner is suboptimal. First, in VDU, information is extracted from multiple modalities (including image, text, and layout) unlike other CV tasks. Therefore, a TTA approach proposed for VDU should leverage cross-modal information for better adaptation. Second, multiple outputs (e.g. entities or questions) are obtained from the same document, creating the scenario that their similarity in some aspects (e.g. in format or context) can be used. However, this may not be utilized in a beneficial way with direct application of popular pseudo labeling or self training- based TTA approaches (Lee et al.] 2013). which have gained lot of attention in CV (Liang ct al., 2020; 2021 Chen et al.] 2022 Wang et al. 2021a). Pscudo labeling uses predictions on unlabeled target data for training. However, in VDU, naive pseudo labeling can result in accumulation of errors due to generation of multiple outputs at the same time that are possibly wrong in the beginning, as each sample can contain long sequence of words. Third, commonly-used self- supervised contrastive-based TTA methods in CV (He et al. 2020 Chen et al.) 20200b a Tian et al.] 2020) (that are known to increase generalization) employ rich of image augmeentation techniques, while proposing data augmentation is much more challenging for general VDU.
In this paper, we propose DoCTTA, a novel TTA method for VDU that utilizes supe-supe learning on text and layout modalities using masked visual language modeling (MVLM) while jointly optimizing with pseudo labeling. We introduce a new uncertainty-aware per-batch pseudo labeling selection mechanism, which makes more accurate predictions compared to the commonly-used pseudo labeling techniques in CV that use no pseudo-labeling selection mechanism (Liang et al. 2020) in TTA or select pseudo labels based on both uncertainty and confidence (Rizve et al. 2021) in cemi-supervised learning settings. To the best of our knowledge, this is the first method that employs self-supervised objective function that combines visual and language representation learning key differentiating factor compared to TTA methods proposed for image or text data. While our main focus is the TTA setting, also showcase special form of TocTTA where access to source data is
2

---

Under review as a conference paper at ICLR 2023
granted at test time, extending our approach to be applicable for unsupervised domain adaptation, named DDcUDA. Moreover, in order to evaluate DocTTA diligently and facilitate future research in this direction, we introduce new benchmarks for various VDU tasks including key-value extraction, entity recognition, and document visual question answering (DocVQA) using publicly available datasets by modifying them to mimic real-world adaptation scenarios. We show DocTTA significantly improves source model performance at test-time on all VDU tasks without any supervision. To our knowledge, our paper first to demonstrate TTA and UDA for VDU applications, showing the significant accuracy gain potential via adaptation. We expect our work to open new horizons for future research in VDU and real-worke deployment in applications.
2 RELATED WORK
Unsupervised domain adaptation aims to improve the performance on a different target domain, for a model trained on the source domain. UDA approaches for closed-set adaptation (where classes fully overlap between the source and target domains) can be categorized into four categories: (i)
distribution alignment-based, (ii) reconstruction-bee based, and (iii) adversarial based, and (iv) pseudo- labeling based. Distribution alignment-based approaches feature aligning mechanisms, such as moment matching (Peng et al. (019) or maximum mean discrepancy (Long et al. 2015 Tzeng et al. 2014). Reconstruction-bat
approaches reconstruct source and target data with a shared encoder while performing supervised classification on labeled data (Ghifary et al. (016). or use cycle consistency to further improve domain-speciffi reconstruction (Murez et al. 2018 Hottman et al. 2018). Inspired by GANs, adversarial learning based UDA approaches
two two-player games
to disentangle domain invariant and domain specific features (Ganin & Lempitsky 2015 Long et al.
2018: Shu et al. 2018). Pseudo-labeling (or self-training) approaches jointly optimize model on
the labeled source and pseudo-labeled target domains for adaptation (Kumar et al. 2020) Liu et al.
2021; French et al. 2017). Overall, all UDA approaches need to access both labeled source data and unlabeled target data during the adaptation which is a special case for the more challenging setting of TTA and we show how our approach can be modified to be used for UDA.
Test-time adaptation corresponds to source-free domain adaptation, that focuses on the more challenging setting where only source model and unlabeled target data are available. The methods often employ an unsupervised or self-supervised cost function. TENT (Wang et al. (0211) utilizes entropy minimization for fully test-time adaptation which encourages the model to become more "certain" on target predictions regardless of their correctness. In the beginning of the training when predictions tend to be inaccurate, entropy minimization can lead to error accumulation since VDU models create a long sequence of outputs per every document resulting in noisy training. SHOT (Liang et al.) 2020) combines mutual information maximization with offline clustering-basee pseudo labeling. However, similar to TENT, using simple offline lscudo-labeling can lead noisy training and poor performance when the distribution shifts are large (Chen et al. 2022 Liu ct al. 2021 Rizve ett al. 2021 Mukherjee & Awadallah 2020). We also use pseudo labeling DoeT but we propose online updates per batch for pseudo labels, as the model adapts to test data. Besides, we equip our method with pseudo label rejection mechanism using uncertainty, of predictions that are likely to be inaccurate. Most recent TTA approaches in image classification
ensure the negative effects
use contrastive learning combined with extra supervision (iaa et al. 202T Huang et al., 202 Wang et al., 2021aa Chen et al.) (022). In contrastive learning, the idea is to jointly maximize the similarity between representations of augmented views of the same image, while minimizing the similarity between representations of other samples). All these methods rely on self-supervised learning that utilize data augmentation techniques, popular in CV while not yet being as effective for VDU. While we advocate for using SSL during TTA, we propose to employ multimodal SSL with pseudo labeling for the first time which is imore effective for VDU.
Self-supervised pretraining for VDU aims to learn generalizable representations on large scale unlabeled data to improve downstream VDU accuracy (Appalaraju et al., 2021 Gu et al. 202 Xu
et al. 2020b a: Lee et al. 2022 Huang et al. 2022). LayoutLM (Xu et al. 2020b) jointly models interactions between text and layout information using a masked visual- language modeling objective and performs supervised lubti-label document classification on IIT-CDI dataset (Lewis et al. 2006).
LayoutLMv2 (Xu et al. 2020ah extends it by training on image modality as well, and optimizing text-image alignment and text-image matching objective functions. DocFormer (Appalaraju et al.
3

---

Under review as a conference paper at ICLR 2023
B..
Diversily class predictions
8, 8.
B. Paeudo label generation
using model's predictions
Masked Languagg Modeling
V4
[MASK]
[MSKI
Sugment
Transformer layers
1D Pos. Emi.
B
B
3
B
2
3
2D Pos. Emb,
Box,
Box,,
Bok,
Box,
Bolgo
Bos,
Box.
Box,
Box,
Box.,
Bok
Vitual Test Emb
[MAC
[MSK
[MA54
[MA3K]
Te
Masked Tokens
Figure 2: Illustration of how our approach, DocTTA, leverages unlabeled target data test time
learn how to predict masked language given visual cues, ii) generate pseudo labels to supervise the learning, and iii) maximize the diversity of predictions generate enough labels from all classes.
2021 another modil transformer based architecture that uses text, vision and spatial features and combines them using multi-modaa self-attention with multi-modal masked language modeling (MM-MLM) objective (as modified version of MLM in BERT (Devlin al.) 2018)), an image reconstruction loss, and text describing image loss represented as binary cross-entropy to predict the cut-out text and image are paired. FormNet (Lee et al. 2022) is structure-awars sequence model that combines transformer with graph convolutions and proposed rich attention that uses spatial relationship between tokens. UniDoc (Gu et al., 202T) is another multi-modal transformer based pretraining method that uses masked sentence modeling, visual contrastive learning, and visual language alignment objectives which unlike other methods, does not have a fixed document object detector (Li al. 2021; Xu al. 2020a). In this work, focus novel TTA approach for VDU, that can be integrated with any pre-training method. We demonstrate DocTTA using the publicly available Mvy architecture pretrained on IIT-CDIP dataset.
DOCTTA: TEST-TIME ADAPTATION FOR DOCUMENTS
In this section, we introduce TocT a test-time adaptation framework for VDU tasks including key-value extraction, entity recognition, and document visual question answering (VQA).
3.1 DOCTTA FRAMEWORK
We define a domain as a pair of distribution D on inputs and a labeling function : + y. We consider source and target domains. In the source domain, denoted as (D., we assume to have model denoted as fa and parameterized with 0, to be trained on source data {a (() J= where ! X, and y Ve are document inputs and corresponding labels, respectively and n, is the number of documents in the source domain. Given the trained source model f. and leaving X, behind, the goal of TTA is train on the target domain denoted (Di,4) where fe is parameterized with 8, and is initialized with f, and D, is defined over {a, (4) ) E X, without any ground truth label. Algorithm overviews our proposed DocTTA procedure.
4

---

Under review as a conference paper at ICLR 2023
Unlike single-modalin inputs commonly used in computer vision, documents are images with rich textual information. To extract the text from the image, we consider optical character recognition (OCR) is performed and use its outputs, characters, and their corresponding bounding boxes (details are provided in Appendix). We construct our input X' in either of the domains composed of three components: text input sequence X" of length n denoted as (xf : ,xn) E g(nxd), image X E R3xW*x and layout X 6-dimensional vector in the form of (Amin Tmaz: Vmnin Ymazz w, h) representing a bounding box associated with each word in the text input sequence. For the entity recognition task, labels correspond to the set of classes that denote the extracted text; for the key-value extraction task, labels are values for predefined keys; and for the VQA task, labels are the starting and ending positions of the answer presented in the document for the given question. We consider the closed-set assumption: the source and target domains share the same class labels V = 3/. y with [>|] = C' being the total number of classes.
Algorithm 1 DocTTA for closed-set TTA in VDU
1: Input: Source model weights O,, target documents (=1) test-time training epochs ne. test- time training learning rate o, uncertainty threshold
2: Initialization: Initialize target model Sa, with 0g weights.
3: for epoch = 1 to n. do
4: Perform masked visual-language modeling in Eq.
5: Generate pseudo labels and accept a subset using criteria in Eq. B and fine-tune with Eq.
6: Maximize diversity in pseudo label predictions Eq.
7: Og < 02 aV LDoTTTA
Update O, via total loss in Eq.
8: end for
3.2 DOCTTA OBJECTIVE FUNCTIONS
In order to adapt Js in DocTTA, we propose three objectives to optimize on the unlabeled target data:
Objective I: masked visual language modeling (MVLM). Inspired by masked language modeling in BERT (Devlin et al. 2018) and MVLM used (Xu et al. 2020a) perform self-supervised pretraining, we propose to employ MVLM at test time to encourage the model to learn better the text representation of the test data given 2D positions and other text tokens. The intuition behind using this objective for TTA is to enable the target model to learn the language modality of the new data given visual cues and thereby bridging the gap between the different modalities on the target domain. To do so, we randomly mask 15% of input text tokens among which 80% are replaced by a special token [MASK] and the remaining tokens are replaced by a random word from the entire vocabulary. The model is then trained to recover the masked tokens while the layout information remains fixed. To do so, the output representations of masked tokens from the encoder are fed into a classifier which outputs logits over the whole vocabulary, to minimize the negative log-likelihood of correctly recovering masked text tokens xn given masked image tokens x' and masked layout BB
(MV = --E,EN, Σn,
log log Pe, (2,T, |2| x.).
(I)
Objective II: self training with pseudo labels. While optimizing MVLM loss during the adaptation, we also generate pseudo labels for the unlabeled target data in an online way and treat them as ground truth labels perform supervised learning on the target domain. Unlike previous pseudo labeling- approaches for TTA in image classification which update pseudo labels only after each epoch (Liang et al. 2020 202T: Wang al. 2021a). we generate hard pseudo labels per batch aiming to use the latest version of the model for predictions. In addition, unlike prior works, we do not use a clustering mechanism to generate pseudo labels as they will be computationally expensive for documents. Instead, we directly use predictions by the model. However, simply using all the predictions would lead to noisy pseudo labels. Inspired by (Rizve et al., 202T), order to prevent noisy pseudo labels, we employ an uncertainty-aware selection mechanism to select the subset of pseudo labels with low uncertainty. Note that in (Rizve et al. 2021), pseudo labeling is used as a semi- supervisd learning approach and the selection criteria is based both thresholding confidence and using MC-Dropou (Gal & Ghahramani] 2016) as the measure of uncertainty. We empirically observe that confidence values (when taken the posterior probability output from the model)
5

---

Under review as a conference paper at ICLR 2023
overconfident despite being right wrong. Setting threshold pseudo labels' confidence only introduces a new hyperparameter without a performance gain (see Sec. B.I.p Instead, to select the predictions we propose to only use uncertainty, in the form of Shannon's entropy (Shannon 200T). We also expect this selection mechanism leads to reducing miscalibration due to the direct relationship between the ECH and output prediction uncertainty, i.e. when selected, ECE is expected to reduce for the selected subset of pseudo labels. Assume (()) be the output probability vector of the target sample zt such that p! denotes the probability of class being the correct class. We select pseudo label (()) for x; if the uncertainty of the prediction u(pE)) measured with Shannon's entropy, is below a specific threshold y and we update 8, weights with cross-entropy loss:
certain predictions
Te = ==l((mp?") S $7],
(2)
LC ExEE Σ de log o (fc(z)));
where (-) is the softmax function.
Objective III: diversity objective. prevent the model from indiscriminately being dominated by the most probable class based on pseudo labels, encourage class diversification in predictions by minimizing the following objective:
DDIV E E Pe Pe,
(4)
where p = Ea, exx, ( is the output embedding of the target model averaged over target data. we obtain the full objective function in DocTTA as below:
By combining Eqs. and 1
LDETTA LAVLL + Loe LDIV.
(5)
3.3 DOCTTA VS. DOCUDA
The proposed DocTTA framework can be extended as a UDA approach, which we refer to as DocUDA (see Appendix for the algorithm and details), by enabling access to source data during adaptation to the target. In principle, the availability of this extra information of source data provides can provide an advantage over TTA, however, as we show in our experiments, their difference is small in most cases, and even TTA can be superior when source domain is significantly smaller than the target domain and the distribution gap is large, highlighting the efficacy of our DocTTA approach in adapting without relying on already-seen source data. The UDA version comes with fundamental drawbacks backs. From the privacy perspective, there would be concerns associated with accessing or storing source data in deployment environments, especially given that VDU applications are often from privacy-sensitive domains like legal or finance. From the computational perspective, UDA would yield longer convergence time and higher memory requirements due to joint learning from source data. Especially given that the state-of-the art VDU models are large in size, this may become a major consideration.
DOCTTA BENCHMARKS
To better highlight the impact of distribution shifts and study the methods that are robust against them, we introduce new benchmarks for VDU. Our benchmark datasets are constructed from existing popular and publicly-availkan training and test splits for all our benchmark datasets in the supplementary materials.
VDU data mimic real-world challenges. We have attached the
4.1 FUNSD- ENTITY RECOGNITION ADAPTATION BENCHMARK
We consider FUNSD Jaume et al. 2019) dataset for this benchmark which is a noisy form un- derstanding collection with 9707 semantic entities and 31,485 words with 4 categories of entities question, answer, header, and other, where each category (except other) is either the beginning the intermediate word of sentence. Therefore, in total, have 7 classes. We first
Expected calibration emror Nacini al. 2015 which metric measure calibration of model

---

Under review as a conference paper at ICLR 2023
combine the original training and test splits and then manually divide them into groups. We set aside 149 forms that are filled with more texts for the source domain and put 50 forms that are sparsely filled for the target domain. We randomly choose 10 out of 149 documents for validation, and the remaining 139 for training. Fig. (bottom row on the right) shows examples from the source and target domains.
4.2 SROIE-TTA: VAY-VALUE EXTRACTION ADAPTATION BENCHMARK
We use SROIE (Hang al. 2019) dataset with 9 classes in total. Similar to FUNSD, first combine the original training and test splits. Then, we manually divide them into two groups based on their visual appearance source domain with 600 documents contains standard-looking receipts with proper angle of view and clear black ink color. We use 37 documents from this split for validation, which we use tune adaptation hyperparameters. Note that the validation split does not overlap with the target domain, which has 347 receipts with slightly blurry look, rotated view, colored ink, and large empty margins. Fig. [| (bottom row on the left) exemplifies documents from the source and target domains.
4.3 DOCVQA-TTA DOCUMENT VQA ADAPTATION BENCHMARK
We use VQA Mathew et al. 2021> large-scale VQA dataset with nearly 20 different types of documents including scientific reports, letters, notes, invoices, publications, tables, etc. The original training and validation splits contain questions from all of these document types. However, for the purpose of creating an adaptation benchmark, we select 4 domains of documents: i) Emails Letters (E), ii) Tables & Lists (T), iii) Figure & Diagrams (F), and iv) Layout (L). Since DocVQA doesn't have public meta-data to easily sort all documents with their questions, we use a simple keyword search to find our desired categories of questions and their matching documents. We use the same words in domains' names to search among questions (i.e., we search for the words of "email" and "letter" for Emails & Letters domain). However, for Layour domain, our list of keywords is ['top", "bottom", "right", "left", "header", "page number"] which identifies questions that are querying information from specifie location in the document. Among the four domains, I. and E have the shortest gap because emails/letters have structured layouts and extracting information from them requires understanding relational positions. For example, the name and signature of the sender are usually at the bottom, while the date usually appears at top left. However, F and T domains seem to have larger gaps with other domains, that we attributed that learning to answer questions on figures or tables requires understanding local information withing the list or table. Fig. [ (top row) exemplifies some documents with their questions from each domain. Document counts in each domain are provided in Appendix.
5 EXPERIMENTS
For entity recognition and key-value extraction tasks, we use entity-level Fl score the evaluation metric, whereas for the document VQA task, we use Average Normalized Levenshtein Similarity (ANLSS) introduced by (Biten et al. 2019) (as it is recognized as better measure compared to accuracy since doesn't penalize minor text mismatches due to OCR errors).
Evaluation metrics:
Model architecture: In all experiments, we use LayoutLMV2BAsz architecture which has a 12- layer 12-head transformer encoder with a hidden size of 768. visual backbone is based on ResNeXt101-FPM similar to that of MaskRCNN (He et al.] 2017). Overall, it has ~200M parameters. We note that our approach architecture independent and hence applicable to any attention-based VDU model. Details training and hyper parameter tuning are provided in Appendix.
Baselines: As our method is the first TTA approach proposed for VDU tasks, there no baseline to compare directly. Thus, we adopt TTA and UDA approaches from image classification, as they can be applicable for VDU given that they do not depend on augmentation techniques, contrastive learning, or generative modeling. For UDA, we use the baselines DANN (Ganin & Lempitsky 2015) and CDAN Long et al. 2018) and for TTA, we use the baselines batch normalization BN (loffe & Szegedy 2015 Nado et al. 2020). TENT (Wang al. 2021b), and SHOT (Liang et al. 2020). We also provide source-only, where the model trained on source and evaluated on target without any adaptation mechanism, and train-on-target, where the model is trained (and tested) target domain
7

---

Under review as a conference paper at ICLR 2023
Table 1: Fl score results for adapting source to target in FUNSD-TTA and SROIE-TTA benchmarks. Availability of the labeled/unlabeled data from source/target domains during adaptation in UDA and TTA settings and training phase in source-only and train-on-targe settings are marked. Standard deviations are in parentheses.
Labeled source data
Labeled target data
Unlabeled target data
DA category
Methods
FUNSD-TTA
SROIE-TTA
Source-only
80.80 (0.12)
92.45 (0.08)
DANN
82.54 (0.14)
92.89 (0.13)
UDA
CDAN
83.72 (0.61)
93.36 (0.18)
DocUDA (ours)
89.76 (0.09)
97.38 (0.15)
BN
X
80.84 (0.93)
92.41 (0.45)
TTA
TENT
79.78 (1.28)
92.42 (0.87)
SHOT
80.89 (1.03)
92.78 (0.65)
DocTTA (ours)
X
X
84.23 (0.88)
94.34 (0.43)
Train-on on-target
99.89 (0.05)
100.0 (0.00)
using the exact same hyperparameters used for TTA (which found on the validation set and might not be the most optimal values for target data). While these two baselines don't It adhere to any domain adaptation setting, they be regarded the ultimate lower and upper bound for performance.
5.1 RESULTS AND DISCUSSIONS
FUNSD-TTA: Table D shows the comparison between DocTTA and DocUDA with their correspond- ing TTA and UDA baselines. For UDA, DocUDA outperforms all other UDA baselines by large margin, and improves 8.96% over the source-only. For the more challenging setting of TTA, DocTTA improves the F1 score of the source-only model by 3.43%, whereas the performance gain by all the TTA baselines is less than 0.5%. We also observe that DoeTTA performs slightly better than other UDA baselines DANN and CDAN, which is remarkable given that unlike those, DocTTA does not have access to the source data at test time.
Table I shows the comparison between UDA and TTA baselines vs. DocUDA and DocTTA on SROIE-TTA benchmark. Similar to our findings for FUNSD-TTA, DocUDA and DocTTA outperform their corresponding baselines, where DocTTA can even surpass DANN and CDAN (which use source data at test time). Comparison of DocUDA and DocTTA shows that for small distribution shifts, UDA version of our framework results in better performance.
SROIE-TTA:
Doc DOCVQA-TTA: Table 2 shows the results on our DocVQA-TTA benchmark, where the ANLS scores are obtained by adapting each domain to all the remaining ones. The distribution gap between domains on this benchmark is larger compared to FUNSD-TTA and SROIE- -TA benchmarks. Hence, we also a greater performance improvement by using TTA/UDA across all domains and methods. For the UDA setting, DocUDA consistently outperforms adversarial based UDA methods by large margin, underlining the superiority of self- sulf-supervised learning and pseudo labeling in leveraging labeled and unlabeled data at test time. Also in the more challenging TTA setting. DOCATT consistently achieves the highest gain in ANLS score of least 2.57% increase E F and up to 17. 17.68% F E. Moreover, DOATTA significantly outperforms DANN on all domains and CDAN on 11 out of 12 adaptation scenarios, even though it does not utilize source data at test time. This demonstrates the efficacy of joint online pseudo labeling with diversity maximization and masked visual learning. Between DocUDA and TTA, it is expected that DocUDA performs better than DocT'A due to having extra access to source domain data. However, we observe three exceptions where DocTTA surpasses DocUDA by 1.13%, 0.79%, and 2.1 in ANLS score on + and T + F, and L - F, respectively. We attribute this to: i) target domain (F) dataset size being relatively small, and ii) large domain gap between source and target domains. The former can create an imbalanced distribution of source and target data, where the larger split (source data) dominates the learned representation. This effect is amplified due to (ii) because the two domains aren't related and the joint representation is biased in favor of the labeled source data. Another finding this benchmark is that source model
8

---

Under review as a conference paper at ICLR 2023
Table 2: ANLS scores for adapting between domains in DocVQA-TTA benchmark. Standard deviations are shown in Appendix.
Souree:
Ematls&Letters (E)
Figures&DDiagrams (F)
Listes&Lists (T)
Layout (L)
Target:
E
T
Sourcc-only
37.79
25.59
38.25
5.23
7.03
3.65
13.66
20.48
14.58
53.55
33.36
33.43
DANN
38.94
27.22
40.23
15.43
9.34
7,45
17.67
22.19
17.67
54.55
33.87
33.58
CDAN
39.08
29.33
41.29
16.99
11.32
10.23
27.87
25.23
27.66
56.82
34,27
34.81
DocUDA (ours)
39.23
43.54
$7.99
24.21
15.76
20.45
53,19
29.91
47.81
61.09
34.85
41.80
BN
38.10
26.89
38.23
7.32
8.56
9.35
15.13
22.24
15.65
53.23
33.67
33.55
TENT
38.34
26.42
40.45
12.38
7.34
11.29
16.01
20.23
15.02
53.34
33.59
34.55
SHOT
38.98
27.55
39.15
14.34
10.10
13.21
22.56
24.33
19.15
56.23
34.56
35.65
DoCTTA (ours)
40.36
35.28
15.67
16.01
35.67
30.70
26.32
37.01
39.10
Train-on-target
95.28
93.54
95.01
39.70
24.77
38.59
84.59
70.66
83.73
92.32
91.36
93.41
Table 3: Ablation analysis on adapting from F, i our DocVQA-TTA benchmark with differ- ent components including pseudo labeling, LMVLM. CDrV, and pseudo label selection mechanism using confidence only or together with uncertainty. Standard deviations are in parentheses.
Method
F
T
L
Source-only
37.79 (1.30)
25.59 (1.78)
38.25 (0.92)
DocTTA, conf.
32.67 (1.68)
21.50 (1.52)
6.71 (3.21)
DocTTA, conf. & unc.
39,45 (0.87)
28.47 (0.72)
47.50 (0.51)
DoeTTA, no LMVLM
35.66 (0.46)
25.72 (0.55)
45.88 (0.34)
DocTTA, no CDsV
34.32 (0.53)
25.17 (0.49)
46.36 (0.21)
DocTTA, no pseudo labeling
33.61 (1.65)
23.43 (0.87)
15.89 (1.35)
DocTTA
40.36 (0.53)
35.28 (0.76)
49.35 (1.20)
trained on a domain with a small dataset generalizes less compared to the one with sufficiently-laree dataset but has a larger domain gap with the target domain. Results for train-on-tane on-taeett domain can shed light on this. When we use the domain with the smallest dataset (F) as the source, each domain can only achieve its lowest ANLS score (39.70% on E, 24.77% T, and 38.59% on L) whereas with T. second smallest domain in dataset size in our benchmark (with 657 training documents), the scores obtained by train-on-target on and increases to 84.59% and 83.73%, respectively. Thus, even if we have access to entire target labeled data, the limitation of source domain dataset size still present.
each
We compare the impact of different constituents of our methods on VOc-QA-TTA benchmark, using model trained on Emails&Letterr domain and adapted to other three domains. 3 shows that pseudo labeling selection mechanism plays an important role and using confidence scores accept pseudo labels results in the poorest performance, much below the source-only ANLS values and even worse than not using pseudo labeling. On the other hand, using uncertainty and raw confidence together to select pseudo labels yields the closest performance to that of the full (best) method (details provided in Appendix). MVLM loss and diversity maximization criteria have similar impact on DocTTA's performance.
Ablation studies:
Table
6 CONCLUSIONS
We introduce TTA for VDU for the first time, with our novel approach DocTTA, along with new realistic adaptation benchmarks for common VDU tasks such as entity recognition, key-value extraction, and document VQA. DocTTA starts from pretrained model on the source domain and uses online pseudo labeling along with masked visual language modeling and diversity maximization on the unlabeled target domain. We propose incertainty-based online pseudo labeling mechanism that generates significantly more accurate pseudo labels in a per-batch basis. Overall, novel TTA approaches result in surpassing the state-of-the-arr TTA approaches adapted from computer vision.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY
We have included the details of our experimental setup such as the utilized compute resources. pretrained model, optimizer, learning rate, batch size, and number of epochs, etc. in Section A.2 of the Appendix. We have provided the details of our hyper parameter tuning and our search space in Section A.2.2 of the Appendix. For our introduced benchmark datasets, the statisties of each dataset is detailed in Section A.1.2 List of the all the training and validation splits for our proposed benchmarks are also located at full reproducibility, we will also release our code upon acceptance.
a /TTAA Benchmarks as jaen files. To ensure
REFERENCES
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Doc- former: End-to-end transformer for document understanding, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 993-1003, 2021.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Minesh Mathew, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas. Iedar 2019 competition on scene text vi- sual question answering. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1563-1570. IEFF, 2019.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020b.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation forecasters. Journal of the Royal Statistical Society: Series D (The Stutistician), 32(1-2): 1983.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint wrXiv:1810.0400 2018.
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. arXiv preprint arXiv: orXiv:/706.05000 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189. PMLR, 2015.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstruction-llasfffaall networks for unsupervised domain adaptation. In European conference on computer vision, pp. 597-613. Springer, 2016.
Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unidoc: Unified pretraining framework for document under- standing. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp- 39-50. Curran As- sociates, Inc., 2021. URL httpp:://ppoccecenngg www.iinnnnn..uu.eeeeeeeeeeuuu..e neurips cttpppper//2.////cc/cccc 0084ae4bc24c) (ww2224440999ddl66e6 9ww.eeeeee454444.
Paper. pdf
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 2020.
Judy Hoffman, Eric Tzeng. Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Muchine Learning, volume 80 of Proceedings of Machine Learning Research, pp. fman 8a .tml
PMLR, 10-15 Jul 2018.
URL https::p///..... http:::/ppooeeeiiiep ngs. ml
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Information Processing Systems, 34, 2021.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutimv3: Pre-training for document ai with unified text and image masking. arXiv preprint urXiv:2204.08385 2022.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. lcdar2019 9 competition scanned receipt oer and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1516-1520. IEEE, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pp. 1-6. IEEE, 2019.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference Machine Learning, pp. 5468-5479. PMLR, 2020.
Chen-Yu Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. Formnet: Structural encoding beyond sequential modeling in form document information extraction. arXiv preprint arXiv:2203.08% 08441111,, 2022.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David Grossman, and Jefferson Heard. Building test collection for complex document information processing. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 665 665-666, 666,, 2006.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5652-5660, 2021.
Jian Liang. Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pp. 6028-6039, 2020.
Jian Liang, Dapeng Hu, Yunbo Wang. Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021. In Press.
Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. Advances in Neural Information Processing Systems, 34, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
11

---

Under review as a conference paper at ICLR 2023
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2200-2209, 2021.
Jose Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh Chawla, and Francisco Herrera. A unifying view dataset shift in classification. Pattern recognition, 45(1):521-530, 2012.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. Uncertainty-aware self-training for text classification with few labels. arXiv preprint arXiv:2006. 2020.
Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4500-4509, 2018.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pp. 625-632, 2005.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, reshhperformance deep learning library. Advances in neural information processing systems, 32, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Joaquin Quiñooneoo-Cndelaa Masashi Sugiyama, Anton Schwaighofer, and N Lawrence. Covariate shift and local learning by distribution matching, 2008a.
Joaquin Juinonero-Candela Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008b.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty- aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations, 2021. URL [tRl:sttpp///oeeeeeeee
ODN6SbiU0
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-I approach to unsupervised domain adaptation. In International Conference on Learning Representattons, 2018. URL fttps::/cpperret
net
Yu Sun, Xiaolong Wang. Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European
conference on computer vision, pp. 776-794. Springer, 2020.
12

---

Under review as a conference paper at ICLR 2023
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
Dequan Wang, Shaoteng Liu, Sayna Ebrahimi, Evan Shelhamer, and Trevor Darrell. On-target adaptation. arXiv preprint arXiv:2109.01087 2021a.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In International Conference on Learning Representations, 2021b. URL httptttts///ppeeeevieee/e wwwp:eewiiieeeeeerrrrrtee
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao,, Qin Gao, Klaus Macherey, al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint 160.08860008/444 2016.
Haifeng Xia, Handong Zhao, and Zhengming Ding. Adaptive adversarial network for source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9010-9019, 2021.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang. Wanxiang Che, et al. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv:2012.14740, 2020a.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1192-1200, 2020b.
13


Reviews:
[{"review_id": "pgxNkWrO0J", "reviewer": "Reviewer_tuFF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "summary_of_the_review": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "strength_and_weaknesses": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pEI3a781YiW", "reviewer": "Reviewer_F7R6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the task of domain shift in Visual Document Understanding (VDU) involving entity recognition, key-value extraction, and document visual question answering. DocTTA (document test time adaption) method is introduced which takes a pre-trained model from the source domain and adapts it to the target domain using self-supervised learning (masked visual language modeling and pseudo labelling).\n", "summary_of_the_review": "The proposed idea is well motivated and novel and the experiments show promising directions. However, the paper could be tweaked a bit to provide more detailed descriptions. Please see strengths, weaknesses, suggestions and comments.", "strength_and_weaknesses": "Strength:\n- The paper is well-motivated and clearly written in most parts. This is an interesting problem with the proposed approach very relevant to the community. \n- The authors promise the availability of corresponding code and a detailed description of hyperparameters which would be essential for reproducibility if open-sourced. \n- Ablation study on domain adaptation (Table 3) for different components of the system is interesting with pseudo labelling playing an important role. \n\nWeakness:\n\n- Some model-generated examples would provide more insight. \n- Human evaluation is not provided. \n- Even though the results are promising at the initial level, there is a huge performance difference compared to the models which are trained on target data (Table 1 and Table 2). In Table 2, the ANLS score is approx half compared to the train-on-target models.   \n- It would help to clarify what kind of OCR techniques were used (industrial like GCS or open-sourced models). How would the results compare when different grades of OCR models are used? An ablation study in this regard could definitely help. \n- It seems the relevant baselines haven’t been used where the DANN and BN are works from 2015 and this field has evolved a lot in recent years. \n\n\nQuestions:\n- Could the authors explain how the results compare with zero-shot models? Is the source-only model used in zero-shot settings in Tables 1 and 2? If yes, it would help to clarify that in the text. \n- In continuation of the point above, when it is mentioned on Page 5, that the closed-set assumption is used, how would the zero-shot model with no assumption like this compare to the proposed approach?  \n- Could the authors clarify what ECE means on Page 6?\n\n\nSuggestions/Comments:\n- It would help to explain DocTTA right at the beginning in the abstract itself so that the reader knows what to expect in the rest of the paper and not just Section 3.  \n- It is unclear where Train-on-target used the same model architecture (LayoutLMv2BASE) as the source-only and DocUDA/DocTTA methods. It would help to clarify this in the text.  \n- An example (maybe pictorial) of the layout XB (6-dimension vector) on Page 5 could definitely help articulate the method. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U7rS4W1hzXi", "reviewer": "Reviewer_eUpw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for performing test time adaptation of a model trained in a source domain to a target domain, without use or labeled data in the target domain. The main contribution of the paper is applying this framework to a model for document image representation, leveraging standard masked langage modeling (traditionally used in to learn models for document representation) with a pseudo-labelling selection strategy. To evaluate this new setting in document understanding, new datasets are created by splitting standard document datasets into a source and a target domain. Experiments are performed on these proposed datasets, comparing the proposed method with state-of-the-art domain adaptation and test-time adaptation approaches.\n", "summary_of_the_review": "The paper proposes an adaptation of TTA methods to the case of document understanding, where this issue has not been much explored. The proposed method seems to rely on standard tecniques for TTA and integrating the specific MVLM objective when retraining the model in the target domain. In this sense, novelty and signifficance of contribution seems limited. The rationale behind the splits for each dataset is not well motivated and comparison with SoA should take into account somehow the effect of specific MVLM task. Overall, I consider that it is not clear the contribution of the paper both methodologically and experimentally.", "strength_and_weaknesses": "Strengths\n- The paper addresses a setting which has not much explored in document understanding, transfering a model learned in a source domain to a target domain. This is a relevant problem in document understanding due to privacy issues or difficulty in getting annotation for new documents.\n- The proposed framework takes advantage of the specificity of document understanding integrating a masked language modeling task in addition to the generation and selectoin of pseudo-labels.\n- A new set of datasets specific to the task of test-time adaptation are created and released\n- Experiments show that the proposed method help to signifficantly improve the results in the target domain in all cases and that all components of the framework contribute to this improvement.\n\nWeaknesses\n- Apart from integrating masked language modelling, the difference of the proposed framework with previous TTA methods is not clear. The difference and contribution with respect to other methods to generate and select pseudo labels should be made more explicit, specially in relation to reference (Rizve et al. 2021). It seems that the contribution of the paper lies mainly on integrating documen specific MVLM.\n- The comparison with other methods in the state-of-the-art does not help to evaluate the actual contribution of the proposed method. It is not clear whether better results are due only to the MVLM strategy, to a better selection of pseudo labels, or to a combination of both. I understand that the proposed MVLM strategy could be integrated with most of the existing methods for TTA. Then, it would be more insightful either performing the comparison by also integrating MVLM into these existing methods or evaluating the proposed method without it, in order to decouple the effect of both strategies. Actually, results in tables 2 and 3 seem to show that DocTTA without the specific MVLM would perform worse thant SoA TTA methods\n- The dataset split between source and target domains seem, in some of the cases a bit arbitrary and not very realistic. The motivation behind such splits is not well motivated. Why is it appropriate to split FUNSD by text density and SROIE by blur and other artifacts. Why creating new categories in DocVQA that do not correspond to document types? In addition, details on how these splits were created are not provided. Were they created manually or automatically? Which criteria were used to filter or annotate documents in each split? Is there already any difference on the performance of the original full model on each of the splits?\n- The splits in the target domain do not contain training, validation and test sets. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject