Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose an end2end benchmark for document understanding, providing the website, process, details, datasets, and including a baseline for this task. 
strengths: I really liked the paper. The problem with having a limited space to convey a lot of information automatically leads to issues down the line, where somebody says : hey, more details on the baselines, and somebody else: not enough descriptions on the datasets. I personally would dedicate most space to the datasets themselves and less on the baseline that needs to be broken anyway by future participants. 

Biggest strength: - The annex answers most foggy areas that couldn't be addressed in the paper that popped up when reading it.

- There are many datasets unified
- Format is pretty clear (thanks annex)
- Baseline seems reasonable
- There was a non-trivial amount of manual work poured in the datasets

weaknesses: - Like others before me, I wonder how useful in reality is a benchmark that packages to a high-level many not-quite-that-similar datasets. For example, I had a problem at work where I needed to understand a somewhat stable type of documents. I looked for tasks that are similar, and systems that solved this particularly close tasks. Looking back, I don't think I would have checked a unified benchmark (like this one) unless it really pointed out that particular set of tasks (or single task) that was close to my work task. Furthermore, I most likely wouldn't develop a system that worked on all tasks given that I was focused in particular on, say, one of them. Just like in the case of SUPERGLUE, where I'd just check, on average, which type of transformer works best, and not a particular system, as there are many variable tasks in there. But maybe that's particular to me.

- The website, while stylish, does not allow me to sort the best model by the WikiTable Questions dataset, for example. It is likely that in my work I would like to see the best performing systems sorted by one of the tasks that I find most similar to my current problem. I can't do that here, I have to manually check each system in turn. I know it "looks good" but a benchmark should focus on usability (e.g. like the results reported on paperswithcode.com)

- The website again, I can't see the date when a system was added. I would like to know that information (like, is it old, is it new?)

- The website, scores in the table are not sortable..

- https://duebenchmark.com/analysis gives me a nice 404

- I will not pick at some details of the paper. Overall this revision looks clear and good enough to read to me, there's a sufficient amount of info and a lot in the annex. 
correctness: I think the paper (benchmarks) checks all the tick-boxes to a sufficient degree. I have no major issues here.  
clarity: Well written. I would have structured it differently, with more focus on the datasets and less on the baselines, but there's always a mix that will not be to everybody's liking. 

Regarding the site (not the paper):
What I would improve on is the process of you (the authors) getting submissions on the benchmark. At the moment, for me, it seems I'd have to do quite some research to understand everything I'd need to do, from the data offered that's not explained at all on the website ( https://duebenchmark.com/data ), to the submission process which is not detailed - there are just some rules ( https://duebenchmark.com/rules ). In this case I would not take the time to make an account to see if there's anything more after the log-in screen.
 
relation to prior work: Minimal but given the space constraints, I think it's okay. 
additional feedback: N/A
review_title: Interesting benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a new end-to-end document understanding benchmark created using existing datasets by putting them all in the same format. The authors reformulated existing datasets on TableQA and Table verification and PWC to make it a document understanding problem. A lot of cleaning was done on another four more datasets before adding them to the benchmark. Authors showed how some of existing deep learning methods perform on this new benchmark.


strengths: - Document understanding is a an important problem in multiple industries and this benchmark will clearly help in creating new solutions to this problem.
- Additional annotations and reformulations of existing dataset is a good way to large good quality datasets.
- Seems to have answered the questions raised by reviewers in round 1.
weaknesses: - No experiments were run with computer vision component, which would have shown how multimodal the dataset actually is.
- What is the size of each of document in the dataset, as in how many words and when T5 truncates the input to 1024 tokens, how often does it remove the answer?
- The output of OCR isn't clear for the cases of Tables.
- The datasets used are all of single modality systems, which means only one modality is needed to get the answer to the question. Multimodal systems are more like TAT-QA, HybridQA and OTT-QA where tables and text in the document needs to be processed to find the answer.
correctness: Seems correct.
clarity: - Decent writing, but could be little more clearer on the algorithmic aspect. 
relation to prior work: - I think FinTabNet[1] work could be cited in the paper: 

[1] Zheng, Xinyi, et al. "Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.
additional feedback: NA
review_title: Document understanding benchmark created using existing datasets

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a new joint benchmark for end-to-end document understanding, combining several datasets and tasks. Distinct from prior work, this benchmark focuses on a very general setting where models only see a raw PDF version of the input data, along with bounding boxes etc. This significantly hinders common baselines -- for example, the performance of T5 on TabFact drops from near-SOTA to near-chance -- and as such provides a new and unresolved challenge. The authors accompany the dataset with several evaluative aids, including measurements of human performance on all datasets as well as manually annotated diagnostic datasets to identify problem cases for models.
strengths: The proposed benchmark joins several related tasks in an environment where models struggle (i.e. raw PDF reading), potentially stimulating research in a novel direction. The datasets are well selected, and accompanied by diagnostic tools that should encourage the development of models. There are large differences between baseline performance numbers on raw pdfs, baseline performance numbers on formatted text, and human performance; this is promising for the potential for this dataset to challenge models.
weaknesses: Potentially the largest weakness of the benchmark is that many or most examples only require reasoning over one form of data. Since the benchmark was stitched together from image datasets, table datasets, infographics datasets, and so on, few examples will benefit from reasoning that includes multiple modalities, e.g. tables *and* infographics. A smaller (related) weakness is the closed-domain nature of the benchmark, i.e. the fact that document retrieval is not required as the appropriate document to answer questions is given ahead of time. These are important problems for real-world document understanding scenarios, which models developed for this benchmark will not be tested on.
correctness: The dataset construction is sound.
clarity: The paper is clear and well-written.
relation to prior work: The relation between this dataset and prior work is clear, as is the selection process for the datasets which the authors included in the benchmark. One minor omission is [1], which given the otherwise thorough overview in Table 5 should be included there.

[1] Gupta et al., 2020. INFOTABS: Inference on Tables as Semi-structured Data. ACL.
additional feedback: N/A.
review_title: End-to-end document understanding dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper gathers 7 tasks in the area of document processing and understanding with the objective of creating a unified benchmark for evaluating model's ability to incorporate positional information in their predictions. The authors normalized the different tasks to have the same format and performed extensive data cleaning and added annotations for two of them. They also implemented baselines and measured human performance for datasets that were missing this information, and annotated certain features of examples to allow breakdown of the performance by categories.
strengths: * Processing all these datasets into a unified easy-to-use format and collecting all documents can be very valuable for any future research.
* The website and documentation seem intuitive and useful.
* The additional annotations can be very useful.

weaknesses: * The introduction discusses visual information and graphical elements which are included in some of the tasks but not in any of the evaluated baselines, and as the authors mention themselves in the results, it might in practice not be necessary. So it is unclear if this is part of what this benchmark is measuring or not.
* In general, the tasks are coming from different domains and and different qualities. The introduction reads as if doing better on this benchmark will result in models that perform better on any downstream document related task. But it is not clear to be that this correlation is true. It would be helpful to see some real tasks that succeeding on this benchmark correlates with doing better on them. Also in line 112: "it is crucial to consider these tasks together [...] may encounter in real=world applications" - I am not convinced by this paper that this is necessarily true specifically for these tasks.
* Following the previous point, it is unclear whether the spatial information is what needed in order to further improve performance on this benchmark or maybe it is just other NLP/vision/domain-specific capabilities. For example, maybe the T5+2D baseline already gets the most out of the document structure and the gap from the external best is due to other model improvements?
* Some descriptions to the baselines are missing (1) is T5+2D developed here or taken from a reference? How exactly is it implemented? (2) The OCR appears late in the paper without any explanation where in the process it is used (is it part of the preprocessing of documents?)
* The PWC task is very small (only 120 test examples) and the human performance is only 51. It feels a bit too noisy for evaluation. Also for WikiTableQ the human perf. is 77. 
* No error bars. I am not convinced by the authors response to the checklist that it's only a rough estimate and "multiple runs are not necessary". Also, line 226 says "the training is run three times for each..". So I am confused. Which run was picked for results? Are the results in Table 3 for Dev or Test?
* How are the diagnostic subsets distribute across tasks? There's a risk of confusing domain performance etc. if some categories are dominated by specific tasks.
correctness: * Missing error bars and unclear if results in Table 3 are dev or test, and how was the reported model chosen (out of three runs?)
clarity: * Overall the paper is pretty clear but it can use proofreading to correct a few typos.
* Table 1: align the dot to be in the same vertical position so, for example, .6, .12 and 1.7 will read correctly.
* figures 2 and 3 can use more human readable labels, and the alignment of the x-axis seems a bit off.
relation to prior work: * Seems good overall.
* Though, I believe that the KILT benchmark can be compared against as an example. In KILT the different tasks are performed over the same document (wiki articles). Therefore, the difference in perfomance between tasks cannot be attributed to different domains/ document quality/ etc. like in this paper.
additional feedback: -
review_title: Valuable work to process and normalize datasets.

Meta-review: The work presents a new end-to-end document understanding benchmark, which all reviewers think is interesting. Additionally, the authors have set up a leaderboard website with some useful features. Despite some concerns from reviewers, the authors make a clear and convincing rebuttal.
Decision: Accept