Less for More: Enhancing Preference Learning in Generative Language Models with Automated Self-Curation of Training Corpora
Authors: JoonHo Lee, JuYoun Son, Juree Seok, Wooseok Jang, Yeong-Dae Kwon


Reviewer paper summaries: This paper studies the potential for ambiguous annotations in feedback datasets for reward modeling, such as UltraFeedback. A proxy model is proposed, trained on preference data, to filter noisy data by using the delta rewards from the predictions of the proxy model and the original annotations. The experimental results demonstrate the effectiveness of the proposed self-curation methods. However, the paper lacks discussion on important related works, and the authors should provide more context about the relationship with previous research on this problem.
Summary of strengths: - This paper addresses an important issue and is well-motivated within the context of reward modeling.
- The proposed method is simple yet appears effective based on the experiments.
Summary of weaknesses: - The hypothesis that inconsistent samples based on the predictions from the proxy model should be removed is overly arbitrary. Previous works [7] indicate that ambiguous samples may include not only incorrectly labeled samples but also hard-to-learn samples. The authors should discuss these hard-to-learn samples, as some inconsistencies based on predictions might result from the proxy models’ weak capabilities.
- The paper lacks discussion on related works concerning over-optimization [1] [4] [5] and model soups. It should reference personalized soup [2] and rewarded soup [3], as preferences in the real world often come from different humans who may give contrasting preferences. This discrepancy cannot simply be removed but should be modeled in some way, as suggested in the literature [6].

References:
- [1] Rame, Alexandre, et al. “Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.” Advances in Neural Information Processing Systems 36 (2024).
- [2] Jang, Joel, et al. “Personalized soups: Personalized large language model alignment via post-hoc parameter merging.” arXiv preprint arXiv:2310.11564 (2023).
- [3] Gao, Leo, John Schulman, and Jacob Hilton. “Scaling laws for reward model overoptimization.” International Conference on Machine Learning. PMLR, 2023.
- [4] Eisenstein, Jacob, et al. “Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking.” arXiv preprint arXiv:2312.09244 (2023).
- [5] Coste, Thomas, et al. “Reward model ensembles help mitigate overoptimization.” arXiv preprint arXiv:2310.02743 (2023).
- [6] Wang, Haoxiang, et al. “Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards.” arXiv preprint arXiv:2402.18571 (2024).
- [7] Swayamdipta, Swabha, et al. "Dataset cartography: Mapping and diagnosing datasets with training dynamics." arXiv preprint arXiv:2009.10795 (2020).
Comments, suggestions and typos: - In Table 1, which dataset do you evaluate on? Why is the metric (2 x win + tie) / (number of prompts) x 100 used rather than (win + tie) /  (number of prompts) x 100?
Overall assessment scores: 2