Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: **Update**
In the original review, I stated that the main weaknesses of the paper seem to be easily fixable, given the response from the authors I think my main concerns have been addressed. I've updated my score accordingly.

The paper introduces a new large dataset for the task of table extraction. The dataset is extracted from articles in the PubMed Central Open Access database. Further, the paper proposes a new way to create unambiguous ground truth for table extraction and a new performance metric for the task. The paper provides background for the underlying task and an overview of the creation of the dataset. Finally, the authors conduct some experiments using the new dataset, performance metric, and a new model. 

strengths: - providing a large dataset for a relevant task
- the overall way the dataset is created is documented 
- the dataset will be accessible (via Microsoft Research Open Data repository) and accountability is provided
- proposing a new performance metric and a canonical form of the label
weaknesses: - the actual steps of the dataset generation process are unclear 
- the experiments do not help to support the claims made in the paper
correctness: Seems good
clarity: Overall, the paper is well structured and provides enough background for a reader unfamiliar with the task of table extraction. The process of creating the dataset is also explained. However, the individual steps are a little unclear. Especially the canonicalization procedure used to create unique labels. The paper doesn't describe how the procedure works and does not provide any code for this step. Given that the procedure is mentioned as one of the major contributions by the authors, it would be good to explain it. The header correction step is also a little confusing. It mentions a "same assumption" (l. 151) and it's unclear what assumption that is. This can easily be clarified, by either providing a step-by-step description of the process or including the code used for these steps. 
relation to prior work: The paper mentions both prior datasets as well as performance metrics and describes their differences. It doesn't empirically compare to existing work.
additional feedback: ### Dateset generation process
Given the focus of this track, a clearer (i.e., reproducible) description of the data generation process would be desirable. You provide code for the experiments. Hence, you could easily give the code for the dataset generation or at least for the processing.  

### Experiments
The experiments use a new technique for training on a new dataset using a new performance metric. In the current form, it is impossible to put the results into perspective. You say you get good results, but given that this is a new metric, it is unclear what good results mean. Maybe the new metric is just close to 1 for everything. The experiments would be more meaningful if they would explore each new component individually. To demonstrate the metric's usefulness, you could show that it can differentiate between a good and a bad model. Additionally, you could demonstrate alignment with performance in a downstream task. To demonstrate the usefulness of DETR, compare it to existing techniques. Finally, to demonstrate the dataset's usefulness, show how training on existing datasets (or a smaller version of this dataset) leads to worse performance. 

### Potential bias
The dataset is trying to solve the table extraction task, but the creation process might introduce bias. First, all tables come from a specific domain (medical publications). Second, the original database required the submission of XML versions of the tables. This requirement might lead the authors of the papers to restrict themselves in what type of table they create. These factors might limit the generalization of models trained on this dataset to other domains. While these biases don't seem to raise ethical concerns, I recommend adding these potential problems in the datasheet and maybe clarify them in the paper.
review_title: Review (Updated 19th July 2021)

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper focuses on creating a dataset and metrics for better training and evaluation of methods for extraction table structures.

It claims that the dataset is larger (almost double) that of previously generated datasets and has consistent annotations. Though the methodology for creating the dataset seems identical to approaches used before.


strengths: The paper seems to represent an improvement in the state-of-the-art for training ML methods to infer and extract structure and content from tables in PDF files.
weaknesses: For people unfamiliar with prior literature, the paper does not do a great job of motivating why the problem of table extraction is challenging.

The contributions here seem very incremental. It appears that the dataset was generated using almost exactly the same procedure as before but over large publication data corpus. The big change appears to be in terms of creating ambiguous ground truth (via a canonicalization procedure), but here it is unclear what the key new insight behind the procedure is or what the drawbacks of such canonicalization might be. 

Similarly, for someone unfamiliar with prior literature on the topic, it is unclear how the proposed metrics have addressed the drawbacks of the existing metrics.
correctness: It is unclear how the increase (doubling) in size of the dataset benefitted the models trained. Similarly, it is unclear whether and how the new metrics helped address the drawbacks with existing metrics. 
clarity: At one level, the paper is easy to read and follow. But, the paper is written for audience already familiar with the literature on table extraction models. For example, as someone unfamiliar with the literature, the review found it is very hard to understand the drawbacks with existing metrics from the description here. 
relation to prior work: The authors do a good job of discussing related work and how the work improves on previous contributions. But, the contributions strike the reviewer as too incremental.
additional feedback: I would encourage the authors to more clearly demonstrate the benefits of the dataset, the canonicalization procedure, and the proposed metrics.
review_title: A dataset and metrics for training and evaluating table extraction models

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present a large-scale dataset and metrics for determining the structure of tables in unstructured documents. The authors tackle an important problem, and the dataset the propose, the metrics they propose, and the baseline model that they propose all seem like significant steps forward. 
strengths: The authors provide a large-scale dataset towards an important problem which seems relevant to the broader information extraction research community. The corresponding metrics seem reasonable measures of progress towards this task, and the baseline model established demonstrates high performance.
weaknesses: I would have liked to see more of a discussion on the practical implications of models that do better or worse on the metrics the authors propose. Are there certain types of data that are frequently lost? Are there certain types of tables that will be better or worse served, depending upon how a model does on the various metrics?

Given the baseline's apparent strong performance, I'd also like a better understanding of what the authors see as the most important remaining directions in this space.
correctness: The dataset and benchmark both seem sound, as does the baseline model proposes.
clarity: The writing is clear and well-organized, although dense at times.
relation to prior work: I'm not an expert in prior work in this area, but existing work seems well-cited and clear differentiations are made.
additional feedback: N/A
review_title: A solid step forward towards a useful benchmark and dataset

Meta-review: In this paper the authors offer an expanded dataset and evaluations for extracting tables from unstructured documents.  They apply this approach to PubMed and show that it works well, providing a large scale new dataset for the task.  Consistent feedback across reviewers was that while the results seemed impressive, the clarity of the approach and the significance could be significantly improved.  I'd encourage the authors to continue to polish the paper for clarity so that it can reach a broader audience.  I'd encourage the authors to revise and resubmit.
Decision: Reject