Nera: Negation-Robust Plausibility Estimation Model for Commonsense Statements
Authors: Hao Xu, Jiacheng Liu


Reviewer paper summaries: The authors introduce NERA, an negation-enhanced version of VERA for plausibility estimation of commonsense statements.

Based on the observation that VERA is (largely) only trained with positive data lacking negations, the authors devise a way to augment the original training data by using LLMs to generate negated versions of the original data.
This negation-augmented data is used to train models in a way very similar to VERA, replacing the second-stage training with a multi-statement loss adapted to statement groups that can have multiple correct statements.

To evaluate the performance of NERA, two metrics are introduced: (i) Label Inconsistency Rate measures how often a model assigns the same label to a statement and its negation; (ii) Score Complementary Metric measures how far apart the assigned model scores are for a statement and its negation.

Experiments show that NERA significantly improves on these two metrics compared to VERA, while largely preserving its original performance on commonsense benchmarks.
Summary of strengths: - The paper is well-written and easy to follow.
- The data augmentation and training approaches are straight-forward yet effective.
- The introduced metrics are interesting
Summary of weaknesses: - The Score Complementary Metric seems a bit weak and "gameable"; to their credit, the authors point this out themselves.
- No "Conclusions" section, probably for lack of space.
Comments, suggestions and typos: - Tables 2 and 3, seem to indicate that overall performance is better on unseen data than on "seen" data? How could that be explained?
- Why is there no Vera-XL model for comparison in Table 3?
- Lines 163-164: "We generated 4k statements in 4k statement groups." Is this correct? This seems to imply there is 1 statement in each group. Or is it 4k statements in each of the 4k groups? That would b 16 million, which seems a lot.
- One finding (lines 220-223) is that "This result also demonstrates the challenge of negation faced by language models. If not specially training on negated statements, the model fails to distinguish negations over half of the time." However, it seems that even NERA, which trained on negations, suffers from this. Table 2 seems to show that it as well fails to distinguish negations over half of the time.
Overall assessment scores: 3