Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for performing test time adaptation of a model trained in a source domain to a target domain, without use or labeled data in the target domain. The main contribution of the paper is applying this framework to a model for document image representation, leveraging standard masked langage modeling (traditionally used in to learn models for document representation) with a pseudo-labelling selection strategy. To evaluate this new setting in document understanding, new datasets are created by splitting standard document datasets into a source and a target domain. Experiments are performed on these proposed datasets, comparing the proposed method with state-of-the-art domain adaptation and test-time adaptation approaches.

strength_and_weaknesses: Strengths
- The paper addresses a setting which has not much explored in document understanding, transfering a model learned in a source domain to a target domain. This is a relevant problem in document understanding due to privacy issues or difficulty in getting annotation for new documents.
- The proposed framework takes advantage of the specificity of document understanding integrating a masked language modeling task in addition to the generation and selectoin of pseudo-labels.
- A new set of datasets specific to the task of test-time adaptation are created and released
- Experiments show that the proposed method help to signifficantly improve the results in the target domain in all cases and that all components of the framework contribute to this improvement.

Weaknesses
- Apart from integrating masked language modelling, the difference of the proposed framework with previous TTA methods is not clear. The difference and contribution with respect to other methods to generate and select pseudo labels should be made more explicit, specially in relation to reference (Rizve et al. 2021). It seems that the contribution of the paper lies mainly on integrating documen specific MVLM.
- The comparison with other methods in the state-of-the-art does not help to evaluate the actual contribution of the proposed method. It is not clear whether better results are due only to the MVLM strategy, to a better selection of pseudo labels, or to a combination of both. I understand that the proposed MVLM strategy could be integrated with most of the existing methods for TTA. Then, it would be more insightful either performing the comparison by also integrating MVLM into these existing methods or evaluating the proposed method without it, in order to decouple the effect of both strategies. Actually, results in tables 2 and 3 seem to show that DocTTA without the specific MVLM would perform worse thant SoA TTA methods
- The dataset split between source and target domains seem, in some of the cases a bit arbitrary and not very realistic. The motivation behind such splits is not well motivated. Why is it appropriate to split FUNSD by text density and SROIE by blur and other artifacts. Why creating new categories in DocVQA that do not correspond to document types? In addition, details on how these splits were created are not provided. Were they created manually or automatically? Which criteria were used to filter or annotate documents in each split? Is there already any difference on the performance of the original full model on each of the splits?
- The splits in the target domain do not contain training, validation and test sets. 

clarity,_quality,_novelty_and_reproducibility: The paper is clear and well written. The novelty of the paper is not clear specially with respect to existing TTA methods. Main novelty could be in integrating the specific MVLM for the case of TTA in documents. Code will be released upon publication for reproducibility. 
summary_of_the_review: The paper proposes an adaptation of TTA methods to the case of document understanding, where this issue has not been much explored. The proposed method seems to rely on standard tecniques for TTA and integrating the specific MVLM objective when retraining the model in the target domain. In this sense, novelty and signifficance of contribution seems limited. The rationale behind the splits for each dataset is not well motivated and comparison with SoA should take into account somehow the effect of specific MVLM task. Overall, I consider that it is not clear the contribution of the paper both methodologically and experimentally.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the task of domain shift in Visual Document Understanding (VDU) involving entity recognition, key-value extraction, and document visual question answering. DocTTA (document test time adaption) method is introduced which takes a pre-trained model from the source domain and adapts it to the target domain using self-supervised learning (masked visual language modeling and pseudo labelling).

strength_and_weaknesses: Strength:
- The paper is well-motivated and clearly written in most parts. This is an interesting problem with the proposed approach very relevant to the community. 
- The authors promise the availability of corresponding code and a detailed description of hyperparameters which would be essential for reproducibility if open-sourced. 
- Ablation study on domain adaptation (Table 3) for different components of the system is interesting with pseudo labelling playing an important role. 

Weakness:

- Some model-generated examples would provide more insight. 
- Human evaluation is not provided. 
- Even though the results are promising at the initial level, there is a huge performance difference compared to the models which are trained on target data (Table 1 and Table 2). In Table 2, the ANLS score is approx half compared to the train-on-target models.   
- It would help to clarify what kind of OCR techniques were used (industrial like GCS or open-sourced models). How would the results compare when different grades of OCR models are used? An ablation study in this regard could definitely help. 
- It seems the relevant baselines haven’t been used where the DANN and BN are works from 2015 and this field has evolved a lot in recent years. 


Questions:
- Could the authors explain how the results compare with zero-shot models? Is the source-only model used in zero-shot settings in Tables 1 and 2? If yes, it would help to clarify that in the text. 
- In continuation of the point above, when it is mentioned on Page 5, that the closed-set assumption is used, how would the zero-shot model with no assumption like this compare to the proposed approach?  
- Could the authors clarify what ECE means on Page 6?


Suggestions/Comments:
- It would help to explain DocTTA right at the beginning in the abstract itself so that the reader knows what to expect in the rest of the paper and not just Section 3.  
- It is unclear where Train-on-target used the same model architecture (LayoutLMv2BASE) as the source-only and DocUDA/DocTTA methods. It would help to clarify this in the text.  
- An example (maybe pictorial) of the layout XB (6-dimension vector) on Page 5 could definitely help articulate the method. 

clarity,_quality,_novelty_and_reproducibility: Please see strengths and weaknesses for more details. Availability of promised corresponding code would ideally make the work reproducible. The paper is written clearly in most parts while the proposed idea is original.  
summary_of_the_review: The proposed idea is well motivated and novel and the experiments show promising directions. However, the paper could be tweaked a bit to provide more detailed descriptions. Please see strengths, weaknesses, suggestions and comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
strength_and_weaknesses: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
clarity,_quality,_novelty_and_reproducibility: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
summary_of_the_review: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies test time adaptation of a model trained in a source domain to a target domain without labeled data. A method is proposed with components of masked language modeling and pseudo-labeling. For evaluation, the paper splits existing document datasets into source and target domains. Comparisons are made w.r.t SoTA domain adaptation and test-time adaptation approaches. Reviewers have concerns on the empirical results, in particular the contribution of pseudo-labeling (as one of the major novelties claimed in the paper), the performance gap compared to the SoTA approaches, and the discrepancy between the created datasets vs real settings.
justification_for_why_not_higher_score: The empirical studies raise concerns of reviewers as above
justification_for_why_not_lower_score: The test-time adaptation problem and perhaps the method is interesting
Decision: Reject