Adversarial Winograd Schema Challenge
Authors: Kaiqiao Han, Tianqing Fang, Zhaowei Wang, Yangqiu Song


Reviewer paper summaries: This paper proposes a new evaluation dataset for LLM's reasoning ability, based on the previous Winograd Schema Challenge (WSC) dataset. This dataset is constructed in the way of replacing entities associating with wrong answers while keeping the same reasoning rationale, which is harder for the LLMs. The authors also propose an Abstraction-of-Thought (AoT) prompt to improve the LLMs' performance. The paper is quite easy to follow.
Summary of strengths: * Propose a new evaluation dataset for LLM's reasoning ability, extending the previous WSC dataset
* The dataset is harder for the LLMs, as it requires the LLMs to understand the meaning of statements and attributes of the entities
* The proposed AoT prompt is also interesting
Summary of weaknesses: Lack of (remaining) error analysis of the AoT method
Comments, suggestions and typos: Add error analysis and discussions for further improvement
Overall assessment scores: 4