Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a first of a kind dataset for data-to-text generation task on "scientific tables". The dataset was created using computer science documents in arXiv mostly from "Computation and Language","Machine Learning" domains. From these documents, tables and parts of the text where the tables are explained is extracted to create the data-to-text instances. This annotation of which part of text explains the table is either done manually by the original authors of these documents or by a rule-based automated system. The annotated data which does not contain any arithmetical reasoning was removed. Initial experiments we conducted on these datasets.

Contributions:
- A new reasoning-aware table-to-text dataset on research papers in computer science field.
- An automated system to extract table-description pairs from LaTeX files of documents.
- Initial experiments to show the difficulty of the task.
strengths: - First dataset to work on scientific tables specifically from the computer science domain.
- There aren't much of reasoning-aware table-to-text dataset and this dataset will add value the subtopic.
weaknesses: - The dataset focuses only on tables in computer science domain. 
- Human annotation is there only for a small number of instances (1338). This happened because the authors of the documents were asked to annotates the parts of the document where tables are explained and it would be hard to get many authors to do this. Is it necessary to get the original authors to annotate? Getting non-author annotators might have been easier and they might have done more annotations with almost as accurate as the authors. As later in the paper it is shown that even the automated annotator is doing a decent job in identifying the descriptions of tables from document. 
- The applications of the dataset is little. Model trained on these dataset may only work on tables from ML and CL papers. However, it would be interesting to see if the automated table-description pair extractor can broaden the scope by creating datasets for other domains.
correctness: The paper is clear and concise.
clarity: The paper is well written.
relation to prior work: In relation to previous works, the ideas were clearly communicated.
additional feedback: In CS fields, especially ML and CL, the bold letters are often used to show the best performer in the results table. It might not be the case with other domains. It would be interesting to see without the [BOLD] tag how the performance drops, and if it does how generalizable is dataset to another domain where bold letters aren't the norm.
review_title: First data-to-text dataset on scientific domain

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: A new dataset from scientific domain with tables and corresponding textual description. Authors claim that SciGen has two unique properties 1) mostly contain tables with numerical values 2) Generating description require numerical reasoning over table values. To generate this dataset authors propose a technique to automatically extract table text pairs from the latex sources of scientific articles. So, the main contributions of this paper are two fold a) A new table to text dataset from scientific domain with tables containing mostly numeric values and require numerical reasoning to generate text b) a new technique to extract table text pair from latex sources of scientific articles.
 
strengths: - An interesting large scale  table to text dataset for scientific domain.
- Three settings of training data (which also includes few shot setup)
- automatic evaluation of baseline techniques on 5 metrics shows large gap between human performance and baseline models which will enable future research 

 
weaknesses: - How does the baseline model perform in zero shot setup? say the model trained on standard table to text dataset and tested on scigen. How does SOTA model perform in this setup is missing from the paper.
- The dataset is specific to particular topic/domain and the model trained on this dataset might not be generalisable to other topics/domains.

correctness: seems good.
clarity: The paper needs to be correctly structured and overall writing needs some improvement.
relation to prior work: Yes discussed. 
additional feedback: - Have a zero shot baseline setup.
review_title: Interesting dataset for table to text generation in scientific domain

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces SciGen, a new data-to-text generation dataset for generating arithmetic aware descriptions of scientific tables. SciGen is compared to prior work in the domain, most closely with LogicNLG, and describes how the increased length of the text descriptions and focus on arithmetic reasoning significantly increase the complexity of the problem. The paper compares BART and T5 performance to human baseline on SciGen, showing that ML models struggle heavily on the dataset and are very often producing wrong and factually incorrect statements. Furthermore, it is observed that common automated evaluation methods are not sufficient to differentiate the quality of generated descriptions.
strengths: 1. The dataset is very challenging while also being reasonably objective (from a human perspective) in terms of whether a description is correct or incorrect due to the focus on arithmetic reasoning.

2. Discussion of the validity of automatic evaluation metrics is useful and will hopefully lead to improved metrics being developed in future.

3. The usage of automatically extracted data to supplement the original training data is useful and will enable the size of the dataset to naturally grow over time as new papers are published.
weaknesses: [Update: The authors have provided a reply to the below concerns which addresses them. Because of this my score has been increased.]

1. The dataset is challenging to the point of it being hard to identify if any notable progress has been made in solving it. This is partially due to the lack of baselines. While T5 and BART were benchmarked, it would have been useful to benchmark older models to get a perspective on if the newer models are improving on this task.

2. An ablation on using even fewer than 200 training samples would have been useful to identify how rapidly performance improves with access to high quality expert descriptions.

correctness: The claims appear to be correct and the dataset seems to be soundly designed and constructed.
clarity: The paper flows nicely and is easy to understand.
relation to prior work: The paper mentions prior work such as LogicNLG and provides comparisons that indicate how SciGen significantly differs from prior datasets.
additional feedback: Given that evaluation with automatic metrics seem not sufficient, it appears difficult for future models to incorporate this dataset into their analysis without significant resource commitment in terms of expert human scorers (plus this opens up concerns of the scorers being different from those who scored the models in this work). This may be a problem for the general dataset-to-text generation domain, but I'm curious what the authors think of the difficulty of dataset adoption and evaluation.
review_title: An interesting and very challenging new dataset for data-to-text generation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduced a challenging dataset SciGen for scientific table-to-text generation. Generating the textual descriptions from their corresponding tables requires arithmetic reasoning over table values.  SciGen has three different settings including few-shot, medium, and large. The few-shot setting and the test set contain expert-annotated pairs. For constructing large-scale training data, they further developed a pipeline to extract table-description pairs from the corresponding latex files of scientific articles. The paper evaluates BART and T5 on the proposed datasets by using automatic metrics and manual efforts.
strengths: • This paper proposed a much more challenging dataset for table-to-text generation in terms of data complexity and text complexity. Three different settings are further provided.
•  Apart from expert annotated data, a pipeline method is proposed to extract table-description pairs from LaTeX files of scientific articles, which may facilitate future annotation studies for new domains.
• The data construction process seem thoughtful and reasonable.
weaknesses: My biggest concern is how to evaluate the models on this dataset. It seems to me that the automatic evaluations metrics used in this paper are not suitable for such a reasoning-aware dataset. These metrics are not reliable as they do not agree with each other (including the human evaluation). For example, I am wondering why BERTScore is so high for different settings.  

The most challenging problem of SciGen is how to evaluate the ability to perform arithmetic reasoning over the table, which is also the core problem of our paper. Other automatic metrics should be designed to evaluate models trained on the dataset. For example, prior effort LogicNLG used three additional metrics apart from perplexity and BLEU score to evaluate the logical fidelity of the generated sentences, including parsing-based, NLI-based and adversarial metrics. Are these evaluation metrics applicable for SciGen? Solely relying on human evaluation may hinder researchers from developing models for the dataset, as the human evaluations sometimes are not consistent and reliable as well. 
correctness: The construction of the dataset and baselines are sound, though I do have concerns related to the evaluation metrics mentioned above.
clarity: The paper is easy to read and follow. 
relation to prior work: The related work section gave a detailed comparison between Scigen and LogicNLG, in which the text generation also requires logical reasoning.
additional feedback: No
review_title: A challenging dataset for scientific table-to-text generation

Meta-review: This paper proposes a new dataset for generating texts that describe tables in CS articles. There was broad consensus among the reviewers that this work is interesting and contributes to the field of data-to-text generation in a new and challenging domain that requires performing reasoning over the tables.
Decision: Accept