LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Authors: name: Sukannya Purkayastha, name: Zhuang Li, name: Anne Lauscher, name: Lizhen Qu, name: Iryna Gurevych

Peer review is a cornerstone of quality control in scientific publishing.
With the increasing workload, the unintended use of `quick' heuristics,
referred to as lazy thinking, has emerged as a recurring issue compromising
review quality. Automated methods to detect such heuristics can help improve
the peer-reviewing process. However, there is limited NLP research on this
issue, and no real-world dataset exists to support the development of detection
tools. This work introduces LazyReview, a dataset of peer-review sentences
annotated with fine-grained lazy thinking categories. Our analysis reveals that
Large Language Models (LLMs) struggle to detect these instances in a zero-shot
setting. However, instruction-based fine-tuning on our dataset significantly
boosts performance by 10-20 performance points, highlighting the importance of
high-quality training data. Furthermore, a controlled experiment demonstrates
that reviews revised with lazy thinking feedback are more comprehensive and
actionable than those written without such feedback. We will release our
dataset and the enhanced guidelines that can be used to train junior reviewers
in the community. (Code available here:
https://github.com/UKPLab/arxiv2025-lazy-review)