Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces an out-of-distribution "appendix" to the
well-known TVL-CDIP benchmark along with results obtained with a
number of state-of-the-art models.

strengths: * the authors prepared a very useful addendum to the RVL-CDIP to
  measure the quality of classifiers when the prediction is done on
  out-of-distribution documents/classes (frankly, this should have
  been done by the Document Understanding community long time ago…)
* the paper presents results for a wide range of types of classifiers
  on the data sets (and showing a significant drop in quality when
  compared to the in-distribution test sets)

weaknesses: * the paper is not that innovative (still, the work is quite valuable
  for the community)
* using synthetic scanner noise (3.2, it would be better to find real
  scanned/non-born-digital PDFs)

correctness: All the claims are correct.
clarity: The quality of presentation is high.
relation to prior work: Relation to prior work is clearly given.
additional feedback: Minor fixes:

* "oon the full RVL-CDIP" => "on the full RVL-CDIP"

* "LayoutLM-v2" => "LayoutLMv2" (that's the way it is spelled in
  https://arxiv.org/abs/2012.14740v1)

* "may have consist" => "may have consisted"
review_title: Out-of-Distribution Performance on Document Classifiers

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a dataset to evaluate document classifiers on out of distribution samples. They build a database that contains documents that are either not part of any class (out of domain) or in an existing class but sampled from a different distribution (out of distribution). The authors then propose metrics to evaluate classifiers on these out-of-distribution datasets, and compute them on multiple classical models, highlighting their weakness in this setting.
strengths: 1. This paper addresses a very interesting problem, that is relevant in practical (industrial) settings.
2. The proposed dataset covers two types of out-of-distribution samples, that are both very common in practical use cases.
3. Generated out-of-distribution samples have been carefully designed not to be too obvious to distinguish from real samples.
4. The evaluation metrics confirm the need of finer evaluation of classification models by showing that existing classifiers fail in the proposed setting.
5. Numerical experiments are rather extensive, and show consistent results on multiple models.
weaknesses: 1. In ODD samples, some classes are highly underrepresented. As authors mention, finding examples in this class may be difficult, but I am worried this could eventually lead to mis-use of this dataset as a training+validation set of data for handling OOD data.
2. Metrics are not fully convincing to me. I would like a more "down-to-earth" measure, for instance the true positive ratio for a fixed false positive ratio of 1%/10%/50%. I believe this would be more meaningful for industrial applications (which the benchmark seems to target).
3. The author's benchmark shows that existing methods indeed fail on the OOD data. But authors do not provide insights as to why this happens. They only mention that training networks to specifically recognize out-of-domain could help, but do not provide a (even very simple) method to serve as a baseline for future work.
4. I am afraid that making a classifier recognize out-of-domain samples may also make the classifier mistake out-of-distribution samples as out-of-domain. Is this a potential issue? I would have liked some comment on the link between these two types of OOD samples.
5. Authors report some examples of misclassified OOD samples in Figure 6. While this makes a good visual representation, this does not give insights on what are the real challenges: what are the classes that are often mixed together? I imagine giving a confusion matrix could help on building intuition.
6. OOD samples are not obviously distinguishible from real ones, but the "scan-like" generation procedure still looks quite artificial and could possibly be identified by a classifier.
7. Authors mention a third type of OOD data: corrupted data. But they do not include corrupted records without really motivating this decision.
correctness: The author's approach to benchmarking classification of OOD documents appears logical to me.
clarity: The paper is very clearly written, with insightful visualization of representative examples of the different OOD classes. Experimentsal results are clearly exposed.
relation to prior work: The authors mention existing benchmark on classification on OOD samples in different settings. While they discuss the specificities of each setting, it is not clear to which extent ideas are shared between the current paper and these other works. Multiple works on classifiers for classification of documents are properly cited.
additional feedback: The benchmark addresses a very relevant question. I believe better evaluation metrics would make it even more impactful.
review_title: This paper proposes an interesting benchmark to evaluate the performance of document classification on out of distribution samples, which is a very common problem in industrial applications. The benchmark seems solid, although I am not fully convinced by the chosen metrics. The benchmark is run on many existing models. I would have liked seeing more insights as to why these models fail, and some suggestions on how to improve them (potentially with simple baselines).

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors are enhancing the RVL-CDIP dataset with new out-of-distribution data. The authors consider both the out-of-domain (packaged in the new RVL-CDIP-N dataset) and the "new in-domain data" (packaged in the the RVL-CDIP-O dataset) aspect of the problem.

The paper is well written, and covers both the data acquisition and explains the usage of the new datasets with some relevant benchmarks.
As shown in the results, models trained on the current state-of-the-art RVL-CDIP dataset have a clear lower accuracy when tested on the new out-of-distribution data.
strengths: This new dataset allows to bechmark trained model for biases toward the training data. As highlighted in the paper, the state-of-the-art RVL-CDIP dataset is known to have some content bias, because of its origin (mostly related to tobacco industries, and documents older than 2006).

Thanks to this contribution, model developers will be able to better estimate their quality.
weaknesses: The dataset is still an order of magnitude smaller than the RVL-CDIP dataset, so it cannot be used for standalone training (which is anyway not the proposed purpose).

The data was acquired mostly via web searches, which would keep a proper licensing (for any purpose) unfeasible. The license states that the users of the dataset are responsible for acquiring proper licensing from the copyright holders for non-academic uses.
correctness: The benchmarks and claims are correct and clearly reproducible.

It must anyway be noted that the authors "reversed engineered" the original classification of the RVL-CDIP dataset, which could have caused in some inconsistencies compared to the original annotation guidelines. This is an aspect that the authors could improve in the paper, answering for example questions like "which rules were used to classify the new documents?", "could one quantify/measure a precision metrics for comparing to the original classification?".
clarity: The paper is well written and understandable.
relation to prior work: The authors clearly describe how their work is complementary to existing state-of-the-art datasets. Benchmarks are also referencing the previous work properly.
additional feedback: If the authors defined some annotation guidelines, it would be nice to include them as supplementary material or in the dataset link.
review_title: Relevant enhancement for Document Image Classifiers

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors present out-of-domain and distribution shift datasets for document classification. They claim that the original RVL-CDIP corpus only includes 16 document categories and focuses on the American tobacco industry, which can not evaluate prior classifiers’ ability to handle out-of-distribution inputs. They collect their new categories of pdf files from search engines and the DocumentCloud repository. They also do data processing to make their new files fit the RVL-CDIP corpus. Based on their experiments, they show that both CNN architectures and image transformers exhibit accuracy drops on their dataset. This facilitates future research on document classification on out-of-distribution documents.
strengths: 1. This paper proposes a new and out-of-distribution dataset for the classification of documents.
2. They analyze the original RVL-CDIP dataset and point out its limitation. They also provide thorough reasons to show their dataset value. 
3. Provide complete experiments and analysis.
weaknesses: 1. The paper collects the OOD files by searching the categories which are out of 16 RVL-CDIP categories, but they do not provide which categories they use.
2. They may need to explain how they familiarize themselves with the 16 in-domain categories. Besides, one document may belong to two categories. The authors may need to show their filter process.
3. It would be great if they could hire human annotators to confirm the effectiveness of this dataset. 
correctness: The evaluation procedures appear to be quite thorough. The results are also very well described and convincing. Only the data generation process may need more explanation for its filter process and the choice of categories. 
clarity: The paper is well-written and easy to follow.

In figure 5, the authors may need to add the vertical axis label.
relation to prior work: This paper does a good job explaining RVL-CDIP document classification and out-of-Distribution benchmark. Good summary.
additional feedback: Actually, I think the authors can publish a new dataset for document classifiers without accommodating to RVL-CDIP dataset. They can define their own categories and publish "born digital" documents.

It would be great if they can release their evaluation code.
review_title: Two kinds of document datasets for evaluating performance on document classifiers.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper collects two sets of data RVL-CDIP-N and RVL-CDIP-O, which serves as an domain shifted and out-of-distribution testing dataset for the RVL-CDIP dataset. Several models are evaluated on the new datasets and it is observed that models do not perform well on neither domain shifted or ood datasets.
strengths: The problems of both distributional shift and out-of-distribution are relevant to real-world applications of machine learning models. The collected dataset seems to be manually annotated (but not explicitly stated in the main text).
weaknesses: 1. The writing is not careful. e.g. in L9 of abstract, the RVL-CDIP-N is defined as ood, which is a mistake.
2. P7 line10 from bottom. "logits (typically done using the softmax function)". The authors confuses logits with probabilities.
3. P7 line3 from bottom. "report both macro and micro AUC". It is unclear how the "macro AUC" is computed considering that the decision rule of ood need to take the maximum over all classes.
4. P5 line3 from bottom. It is not wise to pollute the raw data with simple augmentations, because (a) augmentations can be also used in training, which makes the perturbation of testing images pointless (b) augmentation creates simple artifacts that can be exploited by ood algorithms, making the dataset less useful.
5. only one ood (MSP) algorithm is used in ood evaluation, so the conclusion "struggle on the task of out-of-domain document classification" is less convincing.
correctness: see previous section.
clarity: contains several typos and mistakes.
relation to prior work: yes
additional feedback: /
review_title: Marginal Contribution

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new image benchmark based on the RVL-CDIP corpus. The benchmark consists of two datasets: RVL-CDIP-N contains all documents which do not fall in any 16 domains of the RVL-CDIP corpus and RVL-CDIP-O contains all documents which fit in 16 domains of the RCL-CDIP corpus but comes from a different distribution. Experiments demonstrate that existing approaches perform 15%-30% lower on the new RVL-CDIP-N datasets than the original corpus due to the out-of-distribution.  The new benchmark is released.
strengths: 1. The dataset, to my knowledge, is the first out-of-distribution dataset for document image classification.
2. Experiments show the performance gap between the original corpus and the new dataset.
weaknesses: 1. The proposed dataset is not novel enough, which is obtained from the existing corpus.
3. The benchmark is an image dataset in nature, and it is unclear how it compares with other OOD image datasets.
2. The documentation of the benchmark is poor.

correctness: One major concern is the significance of the proposed benchmark. The authors claim that this is the first benchmark for the OOD document image benchmark. However, there exist other OOD image benchmarks such as all datasets used from the generalized ODIN paper.
Hsu, Yen-Chang, et al. "Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
It is much better if authors can discuss how their benchmark is different from them.
clarity: The paper is overall well-organized. clearly written, and easy to follow. But some acronyms should be introduced with their full names like OCR in Section 2.1
relation to prior work: The relation to previous work is well illustrated and easy to understand.
additional feedback: I consider raising the score if the authors can provide a satisfactory rebuttal to my concerns.
review_title: A New Out-of-Distribution Image Benchmark But Limited Novelty

Meta-review: metareview: This paper focus on evaluating dataset bias (out-of-distribution), particularly on document image classier. All reviewers find the evaluation technically sound thorough and insightful. All of them recommend acceptances. The authors have carefully addressed all reviewers concerns. Therefore this paper is suitable for accept.
confidence: 5: The area chair is absolutely certain
Decision: Accept