Kati8/ICLR_2023_2025 huggingface dataset:
authors: Dongyoung Kim
authors: Kimin Lee
authors: Jinwoo Shin
authors: Jaehyung Kim
reviews: 
Comments: This paper proposes SPA, an approach to enhance the alignment performance of large language models by using minimal human-annotated preference data.
Introduce a confidence-based refinement of preference labels to reduce the risk of noise in preference learning with generated data.
It is experimentally verified that a tiny percentage of preference data (3.3%) achieves results comparable to or exceeding those obtained using the entire data and the existing optimal baseline method in the AlpacaEval 2.0 evaluation.
Strengths: 1. The overall idea is relatively simple, but the method achieves good performance and has great potential with limited human labeling.
2. The paper is well-structured and presents a rigorous methodology, with comprehensive experimental validation that supports the claims made about SPA’s effectiveness.
Weaknesses: This is a very solid piece of work. The proposed method is simple yet effective. I don't have any particular concerns or issues with it.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This paper introduces an efficient LLM alignment method, namely Spread Preference Annotation, aiming at reducing the demand for human-labeled preference data. By leveraging the inherent preference of current aligned model, this work generates new preference data points and conducts alignment iteratively. To reduce the potential noise caused by distribution shift, this work incorporates a self-refinement mechanism on preference labels, where this approximate a more strongly aligned model to better identify noise through a linearly extrapolated prediction method. Through experiments this paper proves that SPA can achieve better alignment performance with much less data.
Strengths: 1. This work inherits the idea of self-rewarding, but leverages the inherent preference of current aligned model in an intuitive way, which sounds novel to me.
2. The reduction on data usage seems promising, and the performance is robust.
Weaknesses: 1. 'De-coupled noise preference detection' is not stated clearly enough in section 4.2. Based on my understanding, $z_{\tilde{\theta}}$ is used to substitute for $z_{\theta}$ in the 'Self-refinement' part, which is also supported in Algorithm 1. If I am correct, I think it would be easier for readers to understand if the final usage of the approximated logits and labels are stated in the main text.
2. Lacks some explanatory discussion on why this method can work on such a small subsets and even perform better than DPO with the full dataset (Details in questions part).

If the author can address my concern in weakness/questions and provide some insightful discussion, I am willing to raise my score.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This work proposes SPA, a framework to lower the high costs of collecting large preference datasets for alignment. SPA uses an LLM's logits to generate pairwise preference data, without reward model learning or in-context learning, which is then used for preference learning for LLMs. The authors show the practical usefulness of SPA by aligning mistral 7B on 3.3% of Ultrafeedback preference data to achieve strong performance compared to state-of-the-art methods on AlpacaEval 2.0 and MT-Bench
Strengths: 1. The paper is well-written and very easy to follow throughout. The paper contextualizes itself within the alignment literature well, covering the fundamentals of pairwise preference learning (Bradley-Terry modeling) to direct alignment algorithms like DPO (Section 3)
2. SPA is able to use only a small seed preference dataset to then directly score preference labels using the implicit reward model learned by DPO (Section 4.1). Since these predictions can be noisy, the authors introduce a novel self-refinement denoising technique using a confidence prediction (eq 9) to smooth the preference label (eq 10)
3. Reproducibility: the authors provide implementation details and hyperparameters in Section 5 (*L311-321*). The authors will open-source the code and models after acceptance, which is appreciated. Lastly, because the modification to the DPO objective is minimal, the authors mention only a few lines of change to the DPO codebase, which is another advantage for practical utility of SPA
4. The authors compare to popular categories of baselines: iterative DPO methods, LLM-as-judge methods, and explicit reward-modeling + RLHF methods (*L288 - L291*) and achieve strong results on AlpacaEval 2.0 and MT-Bench
5. The authors show SPA extends beyond Mistral to other popular LLMs like Phi and Llama (Table 5) and is robust, in the win rate variance sense, to the seed of the initial preference data (Table 4)
Weaknesses: No major weaknesses, mainly minor clarifications:
1. Can the authors provide a little more description about the "length control" aspect of AlpacaEval 2.0 in the main paper? This setting is used in nearly all results, but is not explained clearly where first introduced (Section 5.2)
2. What is "gold label" (Table 1, 5)? Is this the Ultrafeedback preference data? Please make this explicit in the writeup
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4