Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a new dataset and benchmark called FEVEROUS, which consists of more than 87k claims. The claims are labeled as supporting, refuting and inconclusive, if there is support for the claim found in Wikipedia sentences and tables, if there is no support found in Wikipedia sentences and tables and, respectively, if there is not enough information to reach a verdict.
strengths: The use of the Grice’s maxims is commendable. I would encourage the authors, however, to verify the maxim of quantity, as it’s not only about sufficient information, but should also evaluate whether the evidence provides too much information. Furthermore, there is no actual reference back in 3.2 to show that the claims were of high quality.

The discussion section touches upon several important observations that shows possible avenues for future work.

weaknesses: In the Claim Generation section, 3.1.1, the authors mention that the annotators could highlight either four consecutive sentences or a table, which has at least 5 rows and at most 50 rows. Isn’t this a bit restrictive? How were all these thresholds chosen? At the moment, they seem fairly arbitrary. How were the restrictions implemented in the Claim using highlight only and beyond highlight - were there any checks implemented to make sure they don’t use paraphrases, or that they combine information from several rows in a table?

The link between the claim generation and claim verification is a bit ambiguous. In the claim generation, could the annotators create all three types of claims? Or only supported ones? Was the claim verification step performed by the same or different annotators? This detail comes too late in the paper and generates many unneeded questions. 

The calibration section discusses approval rates for annotators - are the numbers provided in the “annotators” section still applicable or should we consider that only half were employed to create the dataset? There are unclear references to two-way and three-way IAA: by how many annotators were the claims verified? Did they make a binary decision for each reason, i.e., ungrammatical = true/false, typographical errors = true/false? Did annotators also have to agree on the state of the claim and the highlight? Overall, there are too little details regarding the entire annotation process, the reliability of the dataset and how this was computed. There is no discussion provided regarding the suitability of a kappa score of 0.65. What is still not clear, is how did the annotators reach agreement? How was the claim, final label, and the evidence chosen?

Authors provide details regarding data imbalance, but only for NEI claims. What about the other two types of claims? It would be nice to give some more details regarding the annotated dataset, especially in terms of overview, as a summary of the annotated dataset (this is again a bit late in the paper). 

The results on classification should be compared with existing literature.

correctness: I would encourage the authors to compare their results with existing literature.
clarity: The paper is easy to read and follow. However, there are certain sections that could have an improved clarity, especially in terms of data annotation.
relation to prior work: The related work makes a nice overview of existing datasets for fact checking. The section, however, does not provide any details regarding the novelty of the approach presented in the current paper, which methods are extended or adapted from the literature. While Table 1 gives quite a nice overview of existing dataset and their characteristics, the Table is never described nor referenced. Also, not clear how these datasets were selected and how complete the comparison is.
additional feedback: Detailed comments are given in weaknesses section. The supplementary material does contain many of the issues that I raise. However, to improve the current manuscript, I strongly believe some details can be improved.
review_title: Review of "FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information"

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors release a relatively large-scale dataset and benchmark for fact verification that includes both unstructured and structured information. They also provide some baseline models, demonstrating performance over their new dataset. Overall I think this is a strong paper tackling an important problem. The authors claim that their proposed dataset is the first which considers both structured and unstructured evidence for verification, while explicitly requiring the retrieval of evidence. My primary concerns, discussed later in this review, are related to the ecological validity of the task. 
strengths: * Compared to the small number of datasets that tackle a similar problem, this dataset is much larger.
* The data collection process is well documented.
* The claim generation and evidence collection process seem thoughtful and reasonable.
* Tackles a timely and important problem with potentially large significant to the research community.
weaknesses: I'd have liked to see much more discussion about the ecological validity of this dataset, particularly focused on two questions:

1. How does the task design in this dataset compare to real-world fact-checking problems? To what extent could a high-performing model trained on this dataset be useful for real-world fact-checking tasks? To what extent might it be problematic?

2. The inter-rater agreement rate is quite low. The authors note that this rate is similar to that found in related datasets, and that the rate becomes very high when annotators consider the same evidence for a given claim. I'd like to have a better understanding of the  implications of this disagreements. For instance, is it often the case that different evidence results in entirely supporting or refuting the exact same claim? Or is it more often that some evidence supports a claim, while other evidence simply doesn't provide enough information? Are there clusters of annotators who tend to agree or disagree with each other, or does it appear to simply be noise?
correctness: The dataset's construction is relatively sound given their goals, though I do have concerns related to the high disagreement rate, mentions above. The baselines they establish for their benchmark seem sound.
clarity: The writing quality is good and the paper is well-written
relation to prior work: To my knowledge, the authors have sufficiently placed their work in context with related work.
additional feedback: I'd like to applaud the authors for the clear amount of effort and thought that went into collecting and describing this dataset!
review_title: A promising new dataset

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a new fact verification task by combining both structured (tables) and unstructured (text) data from Wikipedia. This dataset is collected by taking into account both the crawled text and table from Wikipedia with their self-built WikiMedia interface. They use highlighting to help the annotators to annotate diverse and grounded claims. They also develop different bias mitigation strategy to alleviate potential bias introduced in the annotation procedure. They develop different baselines using text-only and table-only algorithms. The experimental results show that the combined model perform better than the individual models on the proposed dataset. However, the existing models are still obtaining very low performance due to the challenging nature of reasoning over two forms of data. The model cannot deal with numerical reasoning, does not have a great representation of the table structure, etc. These weaknesses can serve as future research directions for researchers working on this dataset. In general, I think the paper is very well written, the proposed annotation pipeline is very professional and clear. One minor concern is that the paper missed many citations of text+table QA. I think QA and FEVER are quite similar tasks, their methodologies are quite shareable. Nevertheless, I still vote to accept the paper due to its valuable contribution to the community.
strengths: 1. Very clear motivation to complement missing piece in FEVER community.
2. Very professional data collection pipeline.
3. Very comprehensive experiments are performed to provide enough baseline systems.
weaknesses: The paper misses a few reference papers in the QA community.
correctness: Correct.
clarity: Clear.
relation to prior work: Reasonable, it would be great to add these papers to reference.

1. HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
2. Open Question Answering over Tables and Text
3. Unified Open-Domain Question Answering with Structured and Unstructured Knowledge
4. Representations for Question Answering from Documents with Tables and Text
additional feedback: no
review_title: Review

Meta-review: This paper introduces a dataset for fact-verification using both structured and unstructured data from WIkipedia.  Reviewers found the paper and dataset well motivated, documented, and novel.
Decision: Accept