Title: Reading Books is Great, But Not if You Are Driving!  Visually Grounded Reasoning about Defeasible Commonsense Norms
Benchmark: NormLens
Summary: Commonsense norms are defeasible by context:  reading books is usually great, but not when  driving a car. While contexts can be explicitly  described in language, in embodied scenarios,  contexts are often provided visually. This type  of visually grounded reasoning about defeasible commonsense norms is generally easy for  humans, but (as we show) poses a challenge  for machines, as it necessitates both visual understanding and reasoning about commonsense  norms.  We construct a new multimodal benchmark  for studying visual-grounded commonsense  norms: NORMLENS.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: visually grounded reasoning about defeasible commonsense norms
Phenomenon defined by authors: Yes
Phenomenon definition: Defeasible commonsense norms: "Reasoning about commonsense norms highly depends on the context in which actions are performed. While an action reading a book is generally considered positive, the action is deemed to be wrong in the context of driving a car because the attention should be focused on the road. Understanding the defeasible commonsense norms - norms that could be further strengthened or attenuated based on the context, is crucial". 

Visual grounding: "real-world scenarios often lack explicit contextual information described in language. It is a more natural process to go directly from visual scene to judgment, but this is very understudied."
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given an image of a "situation context" (e.g someone sitting on the couch) along with an associated action written in text ("reading a book"). The model classifies this as either 1) action is wrong 2) action is okay or 3) action is impossible. 
Task item definition: Given an image of a "situation context" (e.g someone sitting on the couch) along with an associated action written in text ("reading a book"). The model classifies this as either 1) action is wrong 2) action is okay or 3) action is impossible. 
As ground truth, there are 5 human provided decisions about whether the action is wrong, okay and impossible, along with a written explanation for each one.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 2000 situations, 5 human labels per situation
Dataset metadata available: Yes
Dataset metadata detail: Split into problems with human annotator widespread agreement, and human annotator disagreement.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric definition detail: Measure alignment of model explanation to human explanation with ROGUE, which is pretty crap.
Metric aggregation: Weighted Mean
Metric subscores: Yes
Metric subscores detail: Situations with and without human annotator consensus
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding