Clean First, Align Later: Benchmarking Preference Data Cleaning for
  Reliable LLM Alignment
Authors: name: Min-Hsuan Yeh; name: Yixuan Li
Published: 2025-09-28 01:44:05

Abstract: Human feedback plays a pivotal role in aligning large language models (LLMs)
with human preferences. However, such feedback is often noisy or inconsistent,
which can degrade the quality of reward models and hinder alignment. While
various automated data cleaning methods have been proposed to mitigate this
issue, a systematic evaluation of their effectiveness and generalizability
remains lacking. To bridge this gap, we introduce the first comprehensive
benchmark for evaluating 13 preference data cleaning methods in the context of
LLM alignment. PrefCleanBench offers a standardized protocol to assess cleaning
strategies in terms of alignment performance and generalizability across
diverse datasets, model architectures, and optimization algorithms. By unifying
disparate methods and rigorously comparing them, we uncover key factors that
determine the success of data cleaning in alignment tasks. This benchmark lays
the groundwork for principled and reproducible approaches to improving LLM
alignment through better data quality-highlighting the crucial but
underexplored role of data preprocessing in responsible AI development. We
release modular implementations of all methods to catalyze further research:
https://github.com/deeplearning-wisc/PrefCleanBench.