Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces a dataset that helps to evaluate two major topics in NLP models:
1) Entity understanding;
2) Commonsense inference.

These two topics are among the most important issues in NLP, and could potentially help
with many other tasks, including question answering, commonsense reasoning, knowledge probing, etc.
strengths: 1) The authors put serious consideration into the design and collection of this solid dataset,
including the diverse distribution of entities and their types, diverse set of reasoning types,
as well as the combination of commonsense reasoning and knowledge.
The procedure of data collection is very cautiously designed with rigorous quality control.
2) The paper presents comprehensive evaluation of the dataset with various different settings
and various systems for evaluation and comparison. Human performance is included for further
analysis.
3) The dataset is carefully constructed with room for future improvement, which is good 
incentive for researchers to work on.
weaknesses: Overall, the dataset is good material for future research to be conducted on.
One minor issue might be that the study of bias in the dataset is still preliminary.
correctness: The paper looks correct. The review is not aware of any major issues.

clarity: The writing is clear and easy to follow.
relation to prior work: Literature review is comprehensive to the best knowledge of the reviewer.

additional feedback: N/A
review_title: A comprehensive dataset to study commonsense reasoning and entities

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors introduce CREAK, a dataset for investigating models' commonsense reasoning capabilities when involving entities. They assess the performance of two state-of-the-art pretrained language models against a strong SVM + TF-IDF and a human baseline. They find that current models fail to zero-shot generalize to CREAK. While fine-tuning on the dataset benefits performance, it still doesn't reach the human benchmark.
strengths: Since entities usually posit a very specific and hard challenge for language models, this dataset can help to start bridging the gap between more artificial commonsense queries and naturalistic reasoning. The dataset itself would therefore be of value for the NLP community. The authors discuss potential ethical and social implications. The construction of the dataset itself is detailed and explained well. I especially appreciated the analysis on potential token-based artifacts models could pick up on (3.3 Dataset Artifacts). 
weaknesses: I have two main concerns.
My first concern is with the authors' involvement in the analyses. It would be reassuring to see the performance and judgments of people that aren't directly involved in this work for assessing the human baseline since there is always a possibility that knowledge about the task provided to crowdworkers for instance can provide some advantage in evaluation performance. This holds similarly for the assessment of reasoning types in 3.3. Here, at least an estimate of interannotator agreement would already be informative.
Secondly, I'm wondering whether the gap between model performance and human performance is even theoretically possible to close. Specifically, models were trained at a specific point in time, which means they have no way of knowing certain celebrities that became important after they were trained, and presidents and other facts about the world change. Figure 4a and 4b already provide insights on model performance dependent on Wikipedia page views and entity types. It might be useful to include a measure for person and Wikipedia page creation date and relating that to the date of model training. There are probably other more sophisticated measures for dating claims and any would be useful to include.
correctness: The dataset creation and evaluation seem sound, keeping a potential implicit author bias in evaluation in mind (see Weaknesses).
clarity: The paper is clearly written. 

I only found that "Given how the contrast set was constructed, the fact that higher-capacity models work better suggests that having more entity knowledge is a key to doing better on this task." (line 293f) didn't follow from what I had read with respect to the creation of the contrast set. Was it specifically entities that were replaced to construct the contrast set?

typo: 
- line 236: "models are ask"
relation to prior work: Prior work is cited and discussed throughout the paper.
additional feedback: None
review_title: Interesting dataset bridging common sense and fact checking containing entities

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper presents a new dataset for testing commonsense reasoning, based on factual statements about entities. In contrast to previous fact-checking datasets (e.g. FEVER), the emphasis is on the commonsense aspect which is combined with relational statements about popular entities. Despite this, the results sugest that main driver of performance is still memorisation of information on the part of large LMs (the larger the better).
strengths: This work is well positioned as part of a growing movement of examination of the type of knowledge stored in large-scale LMs. The main appeal is use of commonsense knowledge during claim construction which isn't easily captured by existing pre-trained models. 
weaknesses: In addition to its relatively small size (compared e.g. to FEVER, LAMA), the main limitation is the degree of control over the elements of the dataset, namely the amount and type of commonsense reasoning needed for each claim, and the lack of any inspection/explanation mechanisms (e.g. evidence retrieval) - indeed the suggested vision of Figure 1, where a commonsense statement is appended to relational knowledge as evidence for a claim is not materialised in the paper (see Correctness section for more details). Finally, as the authors admit, as the claims in this dataset are artificial, it's not possible to use this dataset to examine how large-scale LMs will behave against real-world claims.
correctness: The manual analysis provided in Section 3.3 (classification into reasoning types) displays two methodological flaws with this work. As the authors admit, distinguishing between the different reasoning types (relational knowledge, commonsense or a mixture) is "somewhat subjective based on the background knowledge of the reader". As there is no attempt at an operational definition of what counts as commonsense knowledge and what counts as world (relational knowledge), this leaves any attempt at developing an automated solution without a solid foundation - optimising rather against the subjective judgements of annotators and adjudicators. For example, I find it very hard to believe that knowing how long it takes to fly from California (anywhere in CA?) to New York is commonsense (at least it's not if you're not American). Going beyond an operational definition however, there is the problem of defining the level of inference required to solve these claims. If we approach this task with a formal reasoning perspective, examples like the first claim in Figure 1 don't work (the commonsense part says that someone needs to be good at something before teaching it whereas we are never told that HP is good at riding on a broomstick). Another limitation is that the authors couldn't (or didn't) control for the distribution among the different types of reasoning required. 
clarity: The paper is generally very well written. A few minor issues:
- Missing closing bracket at the end of footnote 2
- "minimal information" (line 123) isn't defined
- Unnecessary use of "the authors" in paragraphs where "we" is also used in the rest of the paper (lines 170, 173) - especially the mention in line 265 ("we [...] ask the authors of this paper").
relation to prior work: There is good discussion of previous work and the comparison with FEVER is thorough and interesting (especially the use of the recently introduced artefact statistics and training dynamics analyses).
additional feedback: No additional feedback.
review_title: Timely work with a number of limitations

Meta-review: The authors introduce CREAK, which is a benchmark for commonsense reasoning about entity knowledge, collecting true and false information about entities with crowdsourcing. The reviewers agree that this dataset would be of a valuable contribution for various NLP tasks. I strongly recommend acceptance of this paper.
Decision: Accept