Anyprefer: An Agentic Framework for Preference Data Synthesis
Authors: Yiyang Zhou
Zhaoyang Wang
Tianle Wang
Shangyu Xing
Peng Xia
Bo Li
Kaiyuan Zheng
Zijian Zhang
Zhaorun Chen
Wenhao Zheng
Xuchao Zhang
Chetan Bansal
Weitong Zhang
Ying Wei
Mohit Bansal
Huaxiu Yao
Venue: ICLR 2025 Poster (2025)
Keywords: synthetic preference data generation; preference fine-tuning; agent
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: High-quality preference data is essential for aligning foundation models with human values through preference learning. However, manual annotation of such data is often time-consuming and costly. Recent methods often adopt a self-rewarding approach, where the target model generates and annotates its own preference data, but this can lead to inaccuracies since the reward model shares weights with the target model, thereby amplifying inherent biases. To address these issues, we propose Anyprefer, a framework designed to synthesize high-quality preference data for aligning the target model. Anyprefer frames the data synthesis process as a cooperative two-player Markov Game, where the target model and the judge model collaborate together. Here, a series of external tools are introduced to assist the judge model in accurately rewarding the target model’s responses, mitigating biases in the rewarding process. In addition, a feedback mechanism is introduced to optimize prompts for both models, enhancing collaboration and improving data quality. 
The synthesized data is compiled into a new preference dataset, Anyprefer-V1, consisting of 58K high-quality preference pairs. 
Extensive experiments show that Anyprefer significantly improves model alignment performance across four main applications, covering 21 datasets, achieving average improvements of 18.55% in five natural language generation datasets, 3.66% in nine vision-language understanding datasets, 30.05% in three medical image analysis datasets, and 16.00% in four visuo-motor control tasks.

Published as a conference paper at ICLR 2025
ANYPREFER: AN AGENTIC FRAMEWORK FOR PREFERENCE DATA SYNTHESIS
Yiyang Zhou'
Zhaoyang Wang 1
Tianle Wang
Shangyu Xing
Peng Xia If
Bo Li
Kaiyuan Zheng St
Zijian Zhang
Zhaorun Chen
Wenhao Zheng'
Xuchao Zhang
Chetan Bansal
Weitong Zhang' Ying Wei Mohit Bansal
Huaxiu Yao
UUNC- Hill
"NTU Uniieerityy of Washington
UUChcaago Microsoft Research
(yiyangai, zhaoyang, huaxiu .unss.unc.dss
ABSTRACT
High-quality preference data is essential for aligning foundation models with hu- man values through preference learning. However, manual annotation of such data is often consucnng and costly. Recent methods often adopt self relf-rewa approach, where the target model generates and annotates its own preference data, but this can lead to inaccuracies since the reward model shares weights with the target model, thereby amplifying inherent biases. To address these issues, we pro- pose Anyprefer, a framework designed to synthesize high-quality preference data for aligning the target model. Anyprefer frames the data synthesis pro- cess cooperative two-player Markov Game, where the target model and the judge model collaborate together. Here, a series of external tools are introduced assist the judge model in accurately rewarding the target model's responses, mitigating biases in the rewarding process. In addition, a feedback mechanism is introduced to optimize prompts for both models, enhancing collaboration and improving data quality. The synthesized data is compiled into new preference dataset, Anyprefer-V1, consisting of 58K high-quality preference pairs. Ex- tensive experiments show that Anyprefer significantly improves model align- ment performance across four main applications, covering 21 datasets, achiev- ing average improvements of 18.55% in five natural language generation datasets, 3.66% in nine vision-language understanding datasets, 30.05% in three medical image analysis datasets, and 16.00% in four visuo-motor control tasks.
INTRODUCTION
Foundation models, including large language models (LLMs) and large vision-language models (LVLMs), have greatly enhanced AI model's ability to understand text, interpret images, and follow human instructions. Despite their impressive performance many tasks, they still face reliability issues such as hallucinations, stemming from misalignment with human instructions (Thakur et al.)
2024 Ouyang et al. 2022 Ye et al. 2023 Wang et al. 2023 Zhou et al. 2023) or different modality information (Zhou al. 2024a Wang et al.) 2024b Yu et al. 2024bb. To address these misalign- ment issues, recent studies have employed preference learning techniques such as reinforcement leamning from human feedback (RLHF) Yu et al.) [2024a] Sun et al., 2023) and direct preference optimization (DPO) (Deng et al. 2024a Rafailov et al., (024), to align the outputs of foundation models with human preferences L.MSs to harmonize multimodal knowledge in LVLMS.
The success of preference fine-tuning techniques hinges on the availability of high-quality, large- scale preference datasets. Researchers currently employ two main methods for constructing these datasets. The first involves human annotation, which yields high-quality data but is often limited in scale due to its labor-intensive nature (Yu al. 22224a] Di al. (024). The second method uses external AI models to generate preference data Li et al. (202cc): Zhou et al.] (2024a); however, this approach may fail to capture the inherent preferences of the target model being fine-tuned, render- ing the generated data less useful. Recently, the self-rewarding (Zhou et al.) 2024a] Yuan et al.) 2024 Wang al.) 2025) approach samples the target model's own outputs as responses and uses the model itself to reward these responses, constructing preference pairs. While promising, this
"Lad author, Core contributor

---

Published as a conference paper at ICLR 2025
Feedback for Prompt Optimization
Natural
Candidate Resporss
Languagge
Image
PUR
600
G
Target Model Judge Model
Reward Model
Data Source
Frowlldge Exptured
Web Search Grounded LLaVA-Med
External Tools
Libet dog, eet cxption
Preference Dataset
LLAMA
BLIP-2
The figure illustrates the Anyprefer framework. First, Anyprefer selects the nec- essary tools based on the input prompt to obtain supplementary information, which is then inte- grated into a knowledge base. Next, the target model generates several responses for the input data. The judge model then ranks these responses using the constructed knowledge hase. Subsequently, Anyprefez combines the best and worst-ranked responses into a preference pair. The reward model will then evaluate the quality of this preference pair, and all unqualified pairs will go through the optimization stage to refine its quality by using the proposed feedback mechanism.
Figure I:
method depends on the performance of the target model when serving as its own reward model. Inacourate rewarding can bias the generated preference pairs, seriously compromising data quality. Therefore, improving the process synthetic preference data synthesis is crucial for effective pref- erence fine-tuning, given the scarcity of high-quality preference data and the challenges associated with annotation.
In this paper, illustrated in Figure 1 propose Anyprefer, self-evolving synthetic pref- erence data synthesis framework designed to automatically curate high-quality preference datasets. Anypre fer models the preference data synthesis process a two-player cooperative Markov game between the Target Model and the Judge Model, parameterized by input prompts, to achieve universal goal: maximizing the quality of preference data, as reflected by feedback from the Reward Model. Here, the goal for the larget model is to generate high-quality pairwise preference data and the goal for the judge model is to provide robust and consistent ranking for the generated response. Achieving this universal goal requires collaboration between the target model and the judge model. Anypre fer supports various downstream applications, such as natural language generation, nat- ural vision-language understanding, medical image analysis, and visuo-motor control. Specifically, Anyprefe fer generates preference data following the process of (1) response sampling, (2) response rewarding, (3) data quality evaluation, and (4) prompt optimization. First, in sampling stage, the tar- get model generates set of candidate responses based on the input prompts. Next, the judge model leverages external tools to gather relevant knowledge for rewarding these responses. Once ranked, the responses are used to construct preference data, which is then fed into reward model to eval- uate whether the preference data meets general quality criteria. Finally, with the feedback from the reward model, we refine the policy of the target model and the policy for the judge model by improving the prompt for these two models. Throughout this process, the target model and judge model act as cooperative players, working together to enhance preference data quality.
Why Introducing Tools in Judge Model? The inclusion of external tools is essential for ensuring annotation accuracy. Anypre ferprefer:. fer strategically selects tools based the input data to extract valu- able information, mitigating bias during response rewarding. Additionally, the feedback mechanism introduced in the policy stage not only dynamically adjusts input prompts but also shares feedback with these tools, further enhancing their performance in supporting the judge model.
In summary, the primary contribution of this paper is Anypre fer, the first automatic framework for preference data synthesis. Experimental results across four key applications -ntural language gen- eration, vision-languaan understanding, medical image analysis, and visuo-motor control- spannnnn 21 datasets or tasks, demonstrate the effectiveness and advantages of anyprefez in generating high-quality preference data and facilitating effective preference fine-tuning. In these four applica-
2

---

Published as a conference paper at ICLR 2025
Table 1: Statistics comparison of Anyprefer-VI with existing preference datasets. The column "Scale" stands for the size of the generated dataset. In the column "Applications", NL stands for natural language tasks, IMG stands natural images tasks, MED stands for medical tasks and CTRL stands for visuo-motor control tasks. In the column "Data Type", Img-Txt stands for image-text, Img-Ctrl-Seq stands for image-control sequences. Column "Multi-iter" stands for if the generation process is multi-iteration process or not.
Dataset Name
Scale
Human Effort
Response Generator
Tasks
Data Type
Multi-iter.
HH-RLHF
161K
High
Human Label
NL
Text
No
Nectar
183K
Low
GPT-4
NL
Text
No
Paii-
13K
Low
GPT-4
Text
No
UltraFeedback
64K
Low
GPT-4
Text
No
LJaVA-RIHF
10K
High
Llava
TMG
Img-Txt
No
RLAIF-V
34K
Low
MLLM
IMG
Img-Txt
No
POVID
Low
Gar-4+Target Model
IMG
Img-Tx -xg-Txx
No
VLFeedback
80K
No
Open source LVLMs
IMG MED
Img-Txt
No
NL
IMG
Text; Img-Txt:
Anyprefer-Vi
58K
Target model
Yes
MED CTRL
Img-Ctrl-Seq
tions, Anyprefer achieves improvements of 18.55%. 3.66%, 30.05%, and 14.50%, respectively. Additionally, our experiments demonstrate the effectiveness of the tool-augmented judgment and feedback mechanism. Furthermore, we have compiled the synthesized data into new preference dataset, Anyprefer compared to previous synthesized preference data, Anyprefer-Ve includes broader range of application scenarios and data types. This will benefit the open-source community and further ad- vance AI alignment research.
comprising 58K high-quality preference pairs. As shown in Table [
ANYPREFER
To address the challenges of synthesizing high-quality preference data, propose automatic framework called Anyprefer, which models the preference data synthesis process as a two-player cooperative Markov game. As illustrated in Figure [ the target model and the judge model two collaborative players working together to perform preference data synthesis. The target model first generates response candidates based on the input prompt, while the judge model integrates information from various tools to accurately reward and rank the responses. The ranked candidates are then evaluated by a reward model to ensure they meet general data quality criteria. Feedback from the reward model is used to optimize both the input prompts and the tools employed, enhancing the quality of low-quality preference data pairs. Ultimately, qualified preference pairs are used as preference data for preference fine-tuning. In the following sections, we will first detail the problem formulation and then discuss how to generate the preference data for preference mne-tumnng.
as
2.1 PROBLEM FORMULATION
We discuss the formulation of the proposed Anyprefez framework. To begin with, we denote the input data prompt as x (e.g., natural image) and the of knowledge tools {M, Each knowledge tool M; (e.g., Grounded SAM (Ren et al. 2024)) takes the data x as the input and output a sequence q, Mx(x) extracting the information from x using model M, as delegate.
We model the preference data synthesis as a two-player cooperative Markov Game (MG). In par- ticular, the first player
target model m, which takes the data x as input and generate set of candidates {y=)e The second player is the judge model Tj. it takes the candidate set [y and the knowledge base model {q: M as input, then outputs the preference pair (y+. From the model selection perceptive, judde model n, actively aggregates the information from q, and rank the {y.} output by т., Since both and тΙ are language-based models, the input prompt Pa and Pj can be used to serve as their parameters, respectively. The goal of this MG is to generate a set of preference pair } by the collaboration between the judge model and the target model, that the collected preference data can improve the preference fine-tuning of the target model π., Gener- ally, it is costly and time-consuming to directly evaluate the preference fine-tuning performance in
3

---

Published as a conference paper at ICLR 2025
every step, we instead use reward model yy) to provide surrogate reward by evaluating whether the target model benefits from the preference data {y+.y-}. Therefore the goal of this framework can be formulated
arg max x [R(y Te(-Ip)), (5 [g:h],
Pt.p/
where the expectation is taken over y-) (-Kyyyyeeei {qrh: pу) and ye ~ 7 Pt). Ac- cording equation I in the preference data generation process, it is feasible to optimize prompt Pa and pj simultaneously using policy optimization with prompt-based gradient ascent (Pryzant et all 2023 We provide more detailed discussion and additional results in Appendix B to highlight the significance of the two-play cooperation framework.
2.2
RESPONSE SAMPLING AND REWARDING
To synthesize preference data using Anyprefer, the first stage is sampling several candidate re- sponses. Specifically, for a given input prompt x. we sample C' unique response candidates {y=)a= from the target model (- (Pt), where P. is initialized with the input prompt x. our experimental setup, C' is universally set to 5, balancing diversity of samples with sampling costs.
After sampling the candidate responses, the next step is to use the judge model to accurately re- ward and rank these responses {ya) To reduce potential bias from relying solely on the target model for evaluation (Yuan et al., 2024] Guo et al. 2024), we introduce a tool-augmented rewarding strategy for a more comprehensive evaluation. These knowledge tools gather relevant information from various perspectives to assist the judge model n in providing accurate rewards. Based on the input prompt and candidate response, along with its parameters (policy), i.c., the system prompt Pj. the judge model strategically aggregates information captured by external tools for evaluation. Specifically, the tools extract relevant information q: (x) from the input prompt x. The judge model nú then leverages this extracted knowledge qi to provide an overall score (-(- Jr P.)) for each candidate response ye- Finally, the candidates are ranked, and the top-scoring response is selected as the preferred response y+, while the lowest-scoring is selected as the dispreferred re- sponse y forming the preference pair y- The initial system prompt P, used in the judge model are detailed in Appendix [D] And note that this prompt part of the policy parameters can be constantly updated through the formulated two-player MG framework.
2.3 DATA QUALITY EVALUATION
Ideally, after identifying the preference pair {-+ we can directly use it to fine-tune the target model, collecting performance feedback to enhance the prompts p, and p. of both the judge model and target model. This, in turn, improves the data synthesis process. However, the fine-tuning process can be costly and tone-consuming, which prevents the immediate feedback for updating the judge model and the target model, setting barriers for effectively optimizing the policy. To address this issue, we instead adapt LLM-as-a-Judge strategy (Zheng et al., 2023) to LLM-based reward model R. judge the data quality. Here, the used LLMaass-a-udde prompt can be found in the Appendix [D This reward model can evaluate the quality of the generated preference pair {-+.y-} and return a reward R(y+.y-) that reflects the quality, and diversity of every preference pair. Generated preference pairs with high-quality rewards will be directly collected into the final preference dataset, while the others will be re-generated via the cooperation between the target model and judge model, using updated policy guided by the reward R(y+,y-).
2.4 LEARNING FROM THE FEEDBACK
To effectively refine and improve the filtered low-quality preference data, we can use the obtained reward y the feedback optimize the policy of the target model and judge model as illustrated in [ Specifically, for updating the policy of the target model Te, the input prompt p. can be optimized to increase the probability of sampling more high-quality and diverse responses from the target model Te. For updating the policy of the judge model Tj. the used system prompt p will be also optimized, which will finally affect the aggregation of the tools information. Motivated by Pryzant et al. (202)) and Yuksekkonul et al. (2024), this policy optimization process is similar normal gradient descent, where the feedback R(y+,y can be viewed as the gradients passing through the models to update their parameters Pa and Py- Thus, we formulae this process as follows:
Pe Pt + V [[((+
Pj < Pj v (R(y+,y
(2)
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Anyprefer Framework for Preference Data Synthesis
Require: Dataset D; Target model TT; Judge model Ty; Reward model R: Knowledge tools (Mi = Reward
threshold r
Ensure: A set of high-quality preference pairs and optimized prompts Pr. Pj
for each E D do repeat
1. Generate candidate responses (Yefe-1 using the target model æ, with prompt Pt
2. sy aggregates knowledge {q his from external tools {M, hies for each candidate response ye:
where S is the selected tools decided by the strategy of πt
3. Compute judge scores v o (:lye: ( (qilies: P.) for each candidate response Ve using the judge model
my with knowledge {q: hies
4. Rank candidate responses (y. based on judge scores
5. Select top-scoring and lowest-scoring responses form preference pairs (V++
6. Evaluate preference pairs using obtain reward R(y+,y-)
if ((y++y-)) < r then
Update prompts p. and pj using policy gradient ascent based on R(y+, +.y-)
until R(y,,y- y-)
where 7) the prompt adjustment step. The above policy gradient method aims at iteratively refining the input prompt (parameters) P: and Рj of the target model N, and judge model Tj. respectively. By iteratively updating these parameters, the updated players {Te, πT} are expected better coop- erate on generating preference pairs that meet criteria of the reward model and increase the reward. Finally, the proposed policy optimization are expected to effectively enhance the quality of the gen- erated preference data.
2.5 ITERATIVE PREFERENCE TUN-TUNNNG
In this section, after curating the high-quality preference data, we fine-tune the target model through Direct Preference Optimization (DPO) (Rafailov et al.) (024).. This process yields stronger model, which we then replace as the target model. Then, the enhanced target model collaborates with the judge model to generate preference data, which is subsequently used to fine-tune the target model. This iterative process can be repeated for multiple rounds. Details are in Appendix A.I
EXPERIMENT
In this section, empirically demonstrate how the preference data constructed by Anyprefer effec- tively enhances the performance of various foundation models across four downstream applications. We address the following key questions: (1) Does the preference data generated by Anyprefer improve model performance across diverse applications and benchmarks? (2) Can Anyprefer boost the capabilities of different foundation models through iterative preference learning? (3) Is there a positive correlation between the surrogate reward provided by the reward model and the performance of preference fine-tuning on the target model (i.e., the actual reward)? (4) What is the quality of the preference data automatically synthesized by Anyprefer?
3.1 APPLICATIONS AND EXPERIMENTAL SETUPS
This section provides an overview of the downstream applications along with their corresponding experimental settings, deployment details, evaluation benchmarks, and baselines. The downstream applications include natural language generation, vision-language understanding, medical image analysis, and visuo-motor control, which are detailed below:
Natural Language Generation. The first upplication is using large language models for natural language generation. We utilize LLaMA2-7B-on chat (Touvron et al. 2023) as the target model. We use GPT-4o as the judge model, which will utilize two tools: DuckDuckGo for web search (duc) and (Xiong et al.) (024) for response quality assessment. The GPT-4o also adopted as the reward model to provide the immediate feedback for the generated preference pair. For baseline methods, we include original LLaMA2 model, self-rewarding Yuan et al. (2024) and meta rewarding Wu et al. (2024a) for comparison. For evaluation, we use three natural language benchmarks: GSMSK (Cobbe et al. 2021). ERC-easy/challenge (Clark et al.) [2088), and AlpacaE- val (Li et al. 2023d), covering commonsense question answering, math reasoning and alignment domains. Implementation details provided in Appendix A.2
Fs
5

---

Published as a conference paper at ICLR 2025
RAD-O
PST Shap
PADD
POT Saccem
Succes
Apaia
SLAKE-C
ULEAAWW
SUNG-O
PCP_hwee
AAC: Chullnge
2001
10-02
Natural Language Generation
Natural VL Understanding
Medical Image Analysis
Visuo-Motor Control
Target Model
Rewarding
Anyprefer
VLFeedback
Meta Rewarding
We evaluated Anyprefer using benchmarks from four applications. The target model represents the original model before preference fine-tuning. For medical image analysis, "B" for BLEU, "R" for ROUGE-L, "M" for METEOR, "C" for closed, and "O" for open tasks. In medical iamge analysis, "RAD": VQA-RAD, "IU": IU-Xray.
Figure 2:
Natural Vision-Language Understanding. The second downstream application is using large Vision-Language Models (LVLMs) for natural vision-language understanding. In this application, we use LL. . 7B as the target model. For tool selection, we leverage several state-of-the- art vision models as external knowledge sources, including the visual detection model Florence-2- large (Xiao et al. 2023), the short captioning model BLIP-2 (Li et al. (023b), and the detection and segmentation model Grounded SAM (Ren et al., 2024). Additionally, we employ a powerful central multimodal model, GPT-4o, to integrate and interpret all the information for judgment and reward assessment. For baselines, we compare original LLLVA-115 7B model and L.L.VA-1.5 7B with the self-rewarding approach Yuan et al. (2024), vifeedback Li et al. (2024a) and meta reward- ing Wu et al. (2024a). For evaluation, we follow the setup from Zhou et al. (2024a) and validate Anypre fer three types of benchmarks: comprehensive benchmarks, general QA benchmarks, and hallucination benchmarks. For specific configurations, please refer to Appendix AA.3
Medical Image Analysis. Furthermore, we also evaluate Anyprefer in medical image analysis (MIA). Here, we use ILAVA--Med v1.5 (Li et al. 2023a) as the target model, which is a variant of LLaVA fine-tuned specifically for medical image understanding. For the tools and reward model selection, we use several powerful medical models in specific tasks (e.g.. detection, captioning) as external knowledge source, including MiniGPT-Med (Alkhaldi al. 2024), MedVInT (Zhang al., 2023), CheXagent (Chen et al., (202cc) and a powerful central multimodal model (i.e., GPT-40) for understanding and integrating all the information into judgment and rewarding. to noting that the current Med-LVLMs are unable to generate high-quality data as preferred re- sponses (Xia al. 2024). Therefore, unlike natural language generation and vision-language under- standing applications, we utilize the target model solely to synthesize dispreferred responses (Chen) ict al., 2024d), while the ground truth serves as the preferred responses. For evaluation, we con- duct experiments on two tasks using three datasets: VQA-RAD (Lau et al.) 2018) and SLAKE (Liu ctt al., 202T) for the medical VQA task, and IU-Xray generation task. Implementation details are provided in Appendix
worthwhile
et al. 2016) for the report
Visuo-Motor Control. The final application Anyprefer is using vision-languager language-action model for visuo-motor control (VMC). In this case, we employ OpenVLA VLA (Kim al. (024) as the target model. To implement Anyprefes fer, we use the image segmentation model Grounded SAM (Ren) al. 2024) as tool segment the objects involved in the tasks and obtain their pixel coordinates. We then employ GPT-4o as a judge model to generate trajectory cost functions based on the pixel co- ordinate information and task prompts, including path cost, grasp cost, and collision cost. Following feedback mechanism, the feedback generated by the scoring model is fed back to the judge model to produce prompts better suited for the current task, improving object segmentation and trajectory generation through multiple iterations. For baselines, we include several mainstream robotic mod- els, including RT-1 (Brohan et al., 2022). Octo-small (Team et al., 2024), Octo-base (Team et al., 2024), and Open VLA-SSFT (Open VL. fine-tuned on the Simpler-Eme Env (Li et al. (024b) dataset through SFT). We evaluate our model and the baseline models on four WidowX Robots tasks within the Simpler-Env (Li et al. 2024b): "placing the carrot
plate", "putting the spoon "stacking the green cube on top of the yellow cube", and "placing the eggplant into a basket". We compare the generated trajectories with the ground truth trajectories, evaluating the accuracy of task completion by the generated trajectories. See detailed implementations in Appendix A.5
towel",

---

Published as a conference paper at ICLR 2025
3.2 MAIN RESULTS
In Figure 2 we compare Anyprefer with four key baselines: the original target model, self- rewarding, meta-rewarding and vifeedback. Detailed results, along with values from additional baselines tailored to each specific application, are provided in Table 4 to 15 Appendix. Overall, Anyprefer demonstrates significant improvements across various applications, including natural language generation, vision-language understanding, medical image analysis, and visuomotor con- trol. Specifically, natural language generation, Anyprefer achieves up to a 10.92% increase in accuracy on the GSM8K and ARC datasets compared to the best baseline. On uision-languag understanding benchmarks, Anyprefer outperforms both the original LLaVA-1.5 and the self- rewarding approach, notably achieving 6.8% improvement on the Wis dataset. For medical im- age analysis, Anypre fer delivers the best performance, with an average improvement of 31.05% in medical VQA and report generation tasks. In visuomotor control, we observed success rate in- creases of up to 14.5% % across various tasks.
Additionally, the self-rewarding approach and newarrew also surpass the original target model, further demonstrating the effectiveness of synthesized preference data. By integrating tool infor- mation and feedback-guided policy optimization, Anyprefer significantly enhances the model's ability to generate more accurate and high-quality responses, making the constructed preference data more precise and effective. Moreover, in specialized domains like medical image analysis and visuomotor control, where data scarcity often leads to unstable performance in target models, the inclusion of additional tools and feedback mechanisms helps overcome the knowledge limitations of the original models, resulting in substantial performance gains.
3.3 ABLATION STUDY
We conduct ablation studies evaluate the effec- tiveness of incorporating tools for response judg- ment and the feedback mechanism for policy opti- mization. The results in Table [ introducing additional tools significantly improves overall model performance compared to the origi- nal model that only use GPT-4o as the judge model. This outcome aligns with expectations, as the ternal tools enhance the comprehensiveness of the judge model in rewarding and ranking candidate re- sponses, while also reducing bias in the ranking pro- cess to some extent, Moreover, incorporating the feedback mechanism to optimize the policy- both the prompts for the target model and the judge morth-ffuther boosts performance, with an average improvement of 21.51 across all applications. For more specific results, please refer to Tables and 15 in the Appendix. These findings indicate that the feedback mechanism elevates the qual- ity of preference data, thereby strengthening the target model. To further validate the role of tools in Anyprefer and the benefits of the joint two-player framework, we have conducted detailed ablation studies on these two aspects in Appendix specifically in Tables
Table Ablation study on the impact of tools and feedback. The table presents the average scores for each benchmark. "T" rep- resents tool-augmented judgment, and "F" represents feedback mechanism.
demonstrate that
T
TF
LLM
LVLM
Med-LVLM
VLA
56.88
67.90
23.35
28.0
59.88
68.82
30.5
61.03
69.61
30,60
40.5
and
T
89.6%
Natural Generation
Natural VL Understanding
67.2%
Staice
0%
20%
40%
60%
80%
100%
Medical Image Analysia
Vouo Motor Control
Judge Model Accuracy
Figure 3: Anyprefer at over all applications.
The performance of different iterations
Figure 4: Impact of tools (T) and feedback (F) on judge model.

---

Published as a conference paper at ICLR 2025
3.4 CAN ANYPREFER SUPPORT MODEL SELF-IMPROVEMENT
In this section, we validate if Anyprefer can continuously improve model performance across four applications through iterative updates, At each iteration, the Anyprefer framework gener- ate the preference data, and then use the data fine-tune the target model. As shown in Figure 3 we report the performance of Anyprefer in natural language generation, vision-language un- derstanding, medical image analysis, and visuomotor control. Through multiple iterative updates, Anyprefer exhibits significant performance improvements in all tasks. For instance, in natu- ral language generation, the model demonstrates a notable score increase on the GSM8K dataset compared to the baseline. Similarly, in vision-language understanding and medical image analy- sis, the model demonstrates significant progress, achieving improvements of 3.66% and 31.02%, respectively. In the visuo-motor control task, Anyprefer shows the most significant improvement in success rate, with a 16.00% increase compared to the base model. These results indicate that Anypre fer exhibits strong self-improvement capabilities across all four applications, improving the quality of preference data with each iteration, leading to better overall model performance.
3.5 ANALYSIS OF JUDGE MODEL
In this section, we use natural vision-language understanding as an example to analyze the scoring accuracy of the judge model with and without tools (T) and feedback mechanism (F). We man- ually selected 200 examples, consisting of 100 samples generated using tool-captured knowledge and feedback mechanisms, and 100 samples generated without them. A human evaluation was con- ducted following the criteria outlined in Appendix D The results, as shown in Figure demonstrate that the introduction of tools and feedhack mechanisms significantly improves the accuracy of the judge model: with tools and feedback mechanisms, the judge model's accuracy reaches 89.6%, whereas without them, it is only 67.2%, showing an absolute improvement of approximately 22.4%. This suggests that tools and feedback mechanisms can greatly enhance the judge model's evaluation accuracy, resulting in better ranking of responses generated by the target model.
3.6 ANALYSIS OF REWARD MODEL
Furthermore, we conducted experiments to evaluate whether the surrogate reward scores provided by the re- ward model in Anyprefer are highly correlated with the actual reward scores, i.e., the preference tine-tuning performance of the target model. We compared the corre- lation between the target model's performance over three preference fine-tuning iterations in Anypre fer and the surrogate reward scores corresponding to the preference data pairs generated by the target model during those iterations. As shown in Figure 5 the preference data produced by Anyprefez consistently improves the tar- get model's performance across all four applications over three iterations. Moreover, the iterations progress, the average surrogate reward score generated by our reward model increases in parallel with the target model's per- formance. This indicates a strong correlation between the surrogate reward scores and the direct evaluation results of preference tuning, demonstrating the effectiveness of our reward model in providing reliable surrogate rewards.
Natural Language Generation
Natural VL Understanding
Medical Image Analysaa
Visuo Motor Control
Figure 5: Impact of tools and feedback on judge model accuracy.
3.7 ANALYSIS OF SYNTHESIZED DATASET DIVERSITY AND QUALITY
In this section, we evaluate the preference data Anyprefer-Vi synthesized by Anypre comparing it against existing synthesized preference datasets to verify diversity and qualtiy. Di- versity is analyzed using methods from (Zhao et al. (024), while data quality are evaluated through manual annotations and GPT-4 scoring, which are detailed as follow:
Data Diversity. For diversity, we categorize the datasets in Table into two groups: natu- ral language datasets and multimodal datasets. We select two representative datasets from each group and randomly sample 2,000 instances from each. Specifically, HH-RLHF and Orca chosen for the natural language group. while LLaVA-RLHF and VLFeedback are selected for the multimodal group. The text data from both groups are mapped using the text encoder
8

---

Published as a conference paper at ICLR 2025
from ELIP-ViT-- Base, and the image data in the multimodal group are mapped using the tar- get model's image encoder. We apply t-SNE (Van der Maaten & Hinton 2008) to project these embeddings into two-dimensional space, as shown in Figure 6 (see more quantitative fer-Vi
analysis in Appendix C.. The results show that Anyprefer-Vi nearly covers the full range of other datasets, both for text-only and multimodal data. Moreover, it occupies regions of the embedding space that are not covered by other datasets, highlighting its greater diversity.
Data Quality, For quality assessment, we ran- domly sampled 800 examples for manual eval- uation. focusing primarily on two aspects: the difficulty of the data and the satisfaction level with the data. Specific scoring criteria and guide- lines are provided in Appendix [D3] The results, shown in Figure demonstrate that the difficulty of the preference data constructed by our frame- work mostly falls within the moderate range, with a reasonable distribution that avoids being too dif- ficult too simple. Moreover, the human evalua- tion results indicate that annotators are generally satisfied with the data generated by Anyprefer, which suggests that the preference data con- structed by Anypre fer is of high quality. Furthermore, we randomly selected 200 examples from the VLFeedback, Orca, and our constructed Anyprefer-VI datasets, and used GPT-4o score them on scale of to 10, with a higher score indicating higher data quality. The results are repre- sented as bar charts in part (b) of Figure 7 From the results we can see that it is clear that the data constructed by our framework received relatively higher scores, aligning with the manual validation results. This further demonstrates the high quality of the data generated by Anyprefer.
Natural Language
Multimodal
Figure 6: Comparison of Anyprefer-V1 and other representative datasets in 1-SNE mapping.
3.8 CASE STUDY
In this section, we present and analyze several cases from the dataset, Anyprefer-VL, constructed by
Bas Hard
Domittted
Anyprefer. We generated four cases, each correspond- ing to one application scenario: natural language gen- eration, vision- understanding, medical image analysis, and visuo-motor control, as shown in Figure g From the figure, we observe that the differences between the preferred and dispreferred responses in the prefer- ence pairs generated by Anyprefer are often quite subtle. For instance, in the vision-language understand- ing case, the dispreferred response mentions "kiwis and grapefruit," minor discrepancy. This aligns with our ex-
Very Der
Medun
Data Difficutty Distribution
Human Satisfaction Distribution
100
Anpporter
Vilrectec
Frequency
80
Ora
60
40
pectation that more similar answers make harder for the target model to differentiate between them. Further- more, even domains where preference data is scarce in literature, such as visuo-motor control, Anyprefer generates high-quality preference pairs. In one example, the preferred response successfully places the eggplant the plate, while the dispreferred response nearly grabs the eggplant but ultimately fails.
20
6
GPT Scores (1-10)
Figure 7: Data quality evaluation. (a) shows the results of manual evaluation from two aspects, and (b) represents the results of GPT-4o scoring.
RELATED WORK
Various empirical studies applying scaling laws (Kaplan et al., 2020, Hoffmann et al. 2022) to the training of foundation models have demonstrated the importance of the data size. To effectively scale the training data, synthetic data generation has emerged as a popular and cost-effective alternative, primarily leveraging advanced LLMs to produce high-quality data (Josifoski et al. 2023 Gunasekar et al. 2023) Taori et al., 2023; CCiang et al. 2023). In the post-training stage, especially for the preference training, high-quality preference data also faces the challenges in scaling.
Preference Data Generation. To effectively scale up the size of high quality preference data, self-play and self-rewarding methods have gained increasing attention practical method self-

---

Published as a conference paper at ICLR 2025
(a) Natural Language Generation
Prosptt
Wieee vis Creada colenteoo?
Ceradlls os cokonaatiin togra in the lisk eartary altn Famell explaoos, incted Auspes Corter 196,, claceed la de the sereof France
Vitary calertaaias Corak reflus airplex juunay kovobr bots and Brise influress datalaed aftoradda vatar sesoriis such hicees cxsperativd cuticl oe tipic
Thank you for ascent Caraat was out calambed by any taaopeas pineer dhoers traqpa cretie, the lead that is nsw Canaca was resutried excgemea peoples, mehenny bhe tree, Fins Naves and Mena
Canakl beane intigwemeng cemeuse Bosien, this infornitce helillit heso you would like keow?
Kangiess
wilh Quem tlzstech il serving is head
(20 Natural VL Understanding
(3) Medical Image Analysis
Prongee
What Fridts are ls the top right cumpartinntt of the lanchboa??
Proret
Doarth tallriing image
The sce iigtt paspartoent laasthos péscapple esdersage.
Cardac raadarzed comsareare wikka nermal
The image lea char X-ss than large raass right homittrrrr The mas appain so occeppin emare
land ceesins kinis and gropefrait
(4) Visuo-Motor Control
Take Put ezgplant in basket
Figure 8: Case study. A checkmark indicates the preferred response, while a cross represents the dispreferred response. Errors and hallucinations in the dispreferred response are highlighted in red.
generate the training data without external supervision and models (Yuan et al. 2024 Singh et al.)
2023 Chen et al. 2024d] Wu et al. 2024b; Cheng et al. 2024, Zhang et al. 2024). These methods usually consist of two steps: self gefeertnng data and tie-ttunng. And these two steps can be itera- tively proceeding. Another line of research is Reinforcement Learning from Al Feedback (RLAIF)
which utilizes advanced L.LMs to label response pairs (Bai et al. 2022 Lee al. 2023) for accurate rewarding and ranking. Meanwhile, the preference data generation for VLMS starts with CSR (Zhou al.) 2024b), which extends this concept VLMs, to generate high quality vision-language pref- erences. Following CSR, SIMA (Wang et al. 2024b) is proposed to self-generate responses and employ an in-context self-critic mechanism to select response pairs for preference tuning. Similarly, Deng et al. (2024b) successfully applied the self leff-training manner to image comprehension.
Though these methods have successfully applied synthetic data generation preference training, they commonly have the rewarding bias issue which means that their ranking annotations for those self-generated data are not accurate. For sela-rewa methods (Yuan et al. [0244 Singh et al.
2023 Chen et al.] 2024d: Wu et al.] 2024b; Cheng et al. 2024; |Wang et al. 2024a Chen et al.,
2024b, Wang et al. 2025), there are no explicit constraints on the rewarding function, resulting
in unreliable annotations. To mitigate this issue, our method introduces a series of external tools into the preference data rewarding process to ensure the rewarding accuracy. Existing works (Bai et al. 2022, Lee et al., 2023] Chen et al.. 20244 Tong et al.) (025) that use AI feedback to annotate preference data may alleviate the rewarding bias issue, however, they often overlook improving the quality of response sampling. To improve the quality of the sampled response, we introduce a two- player cooperative Markov Game framework to enable the immediate feedback for the policy model, which can help refine the response quality. Besides the proposed tool integration and feedback mechanism, we also apply preference data synthesis to multi domains including natural language generation, natural VL understanding, medical image analysis, and visuo-motor control, greatly benefiting the community.
5 CONCLUSION
This paper introduces the Anyprefer framework, an automatic system for synthesizing high- quality preference data across diverse applications. Anypre fer establishes a cooperative Markov game that synchronizes the target model with a judge model while incorporating external tools and feedback mechanisms. This approach improves both the quality and diversity of the generated preference data, Anyprefer-Ve V1, which can be used to fine-tune the target model and enhance its performance. Experimental results show that the proposed Anyprefer framework significantly boosts performance in various applications such as natural language generation, vision-language un- derstanding. medical image analysis, and risuo-motor control. Moreover, the experiments demon- strate the effectiveness of Anyprefer in enabling model self improomproeemnt,, and highlight the value of tool-augmented response judgment and feedback mechanisms.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This paper proposes the enyprefer framework for automatically generating preference datasets, applied across multiple domains. The constructed datasets strictly adhere to ethical guidelines, ensuring that no sensitive information is included and minimizing potential bias during the data construction process. All experiments and data usage in this research comply with ethical stan- dards. We acknowledge the potential issues related to fairness and bias that may arise when using automated tools for generating preference data. Therefore, we have adhered to relevant ethical stan- dards throughout the data creation and evaluation process to ensure faimess and transparency. No personally identifiable information was collected or processed in this study.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the results from Anyprefer, provide detailed experimental setups and dataset construction processes. In Section we explain the dataset creation, annotation guidelines, and data collection methods, with further elaboration in Appendix A Additionally, in Section A we provide a thorough description of the benchmark testing and evaluation procedures, with clearly defined metries to facilitate independent verification of our results. To further support research and application in the community, we have also made the generated Anyprefer prefer- ence dataset publicly available for download and use by other researchers.
REFERENCES
Duckduckgo search engine. httppss/daucxcuckkg..coo
Asma Alkhaldi, Raneem Alnajim, Layan Alabdullatef, Rawan Alyahya, Jun Chen, Deyao Zhu, Ahmed Alsinan, and Mohamed Elhoseiny. Minigpt-med: Large language model as a general interface for radiology diagnosis. arXiv preprint (4t0:24407,410066 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutiona ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time an- swers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 333-342, 2010.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arwivv2212.0688) 2022.
Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, et al. Mj-bench: Is your multimodal reward model really good judge for text-to-image generation? arXiv preprint arXiv:2407.044: 04.4422,4,4 2024a.
Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj, and Huaxiu Yao. Autoprm: Automating procedural supervision for multi-step reasoning via controllable question decomposition. arXiv preprint arXiv:2402. 1455,, 2024b.
Zhihong Chen, Maya Varma, Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.12208. 2024c.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arX1v:2401.01335.
2024d.
Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du. Self-playing adversarial language game enhances Ilm reasoning. arXiv preprint arXiv:2404. 10642, 2024.
11

---

Published as a conference paper at ICLR 2025
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-souree chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. htthsts::
og/ / mlmmys..rgg// ogggy//.rg/blg//2223333 -00---00330--0000 cunna/
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 180:::04044440 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Dina Funnerr-uushha,, Marc D Kohli, Mare B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George Thoma, and Clement J McDonald. Preparing collection of radiol- ogy examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304 -10, 2016.
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. En hancing large vision language models with self-training on image comprehension. arXiv preprint 2rXiv:2405.19716. 2024a.
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training image comprehension, 2024b. URL
attps::/arxxi org/ a25/ 22t.//255..111111
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404. 04445,, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. Uttphttps://axxi
/bs/
2306 13394
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, al. Textbooks are all you need. arXiv preprini arXiv:2306. 1644, 2023.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.0 04792, 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Drew A Hudson and Christopher D) Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of I1m via human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024.
Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction. arXiv preprint arXiv:2303.04133 04132, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arX.v.2001.08361, 2020.
12

---

Published as a conference paper at ICLR 2025
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.0022 2024.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement lennnnn from human feedback with ai feedback. arXiv preprint S.K.v::2300000266: 2023.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med med: training a large language- and-g---nddviiioon assis- tant for biomedicine in one day. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 28541-28564, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang. Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312. 10665, 2023c.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment. arXiv preprint arXiv:2410.09421, 2024a.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05944 2024b.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https:: / /qi thub. wwt.c/om/aaa.. labhhlllbbaaa haab/alpaca__evee
5 2023d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vasion-language models. arXiv preprint arXiv:2305.10333 2023e.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge- www.leeneeenaneedd enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1650-1654. IEEE, 2021.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. (6296-26306, 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng, Automatic prompt optimization with" gradient descent" and beam search. The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14158 2024.
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289-335 301, 2022.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.0658 2023.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv: /4525, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy, arXiv preprint 2rXiv:2405.12215 2024.
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-jude juddges arXiv preprini 2rXiv:2406.124 /2624, 2024.
Haibo Tong, Zhaoyang Wang, Zhaorun Chen, Haonian Ji, Shi Qiu, Siwei Han, Kexin Geng, Zhongkai Xue, Yiyang Zhou, Peng Xia, et al. Mj-video: Fine-grained benchmarking and re- warding video preferences in video generation. arXiv preprint arXiv:2502.01719 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshor, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
wrg.iii.orr//a...... /t/aaabs////../ 09288
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 9(11), 2008
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Fansen- Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
Chaoqi Wang, Zhuokai Zhao, Chen Zhu, Karthik Abinav Sankararaman, Michal Valko, Xuefei Cao, Zhaorun Chen, Madian Khabsa, Yuxin Chen, Hao Ma, et al. Preference optimization with multi- sample comparisons. 2024a.
Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Zhang. Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision- language models. arXiv preprint 2308 /5/26, 2023.
14

---

Published as a conference paper at ICLR 2025
Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou. Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual-language modality alignment in large vision language models via self-improveement arXiv preprint arXiv:2405. 15973, 2024b.
Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewa language models. In The Thirleenth International Conference on Learning Representations, 2025. URL https:
//openrvview. wwt. fortt//orm??io df
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with aun--s-a-meta-jucee arXiv preprint arXiv:2407. 19594, 2024a.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00677 00675, 2024b.
Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406.06007 2024.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks, arXiv preprint arXiv:231 1.0222311.00222,, 2023.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, In Forty-first International Conference on Machine Learning, 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 2xXx:2330414175, 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-graine -re-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang. Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning miims through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint 24X.::2405..7220,, 2024b.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.024:44 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401. 2024.
Mert Yuksekgonul, Federico Bianchi. Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: "utomatic" differentiation" via text. arXiv preprint 0774422406.07496 2024.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305. 10415, 2023.
Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao. Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:241 193009 2024.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild. arXiv preprint 27X.:22405.01470, 2024.
15

---

Published as a conference paper at ICLR 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination large vision-language models. arXiv preprint arX1v:2310.00754, 2023.
Yiyang Zhou, Chenhang Cui, Rafacl Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.1 11411, 2024a.
Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang. Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjum Zhang, and Huaxiu Yao. Calibrated self refaringg vision language models. arXiv preprint arXiv:2405. 240.:222.., 2024b.
A EXPERIMENTAL SETUP
A.1 TRAINING SETUP
For the training phase with preference data, after collecting each round of preference data, we use DPO to train for 3 epochs. The entire training process conducted a single A100 80G GPU. During training, we fine-tune the LoRA parameters for improved efficiency. Detailed training pa- rameters can be found in 3
Table 3: Training hyperparameters.
Hyperparameters
lora_r
128
lora_alpha
256
lora target
all
mm projector Ir
2e-5
Batch size
1
Learning rate
lc-7
model_max mex..engthhhhhhhhh
1024
A.2 NATURAL LANGUAGE GENERATION
A.2.1 DATASET AND BASELINES
To evaluate our method, we use three datasets that target different model capabilities: (1) GSM8K (Cobbe et al. 2021) focuses on primary school-level math problems, requiring 2-8 steps of basic arithmetic
solve. We evaluate based on exact final answer matching. (2) ARC-
easy/challenge (Clark et al. 2018) contains 7K grade-school science multiple-choice questions, split into an Easy Set and Challenge Set (questions hard for both retrieval and word co-occurrence al- gorithms). We also use exact answer matching for evaluation. (3) AlpacaEval (Li et al., 2023d)
tests general instruction-followingg where model responses are compared to reference answers us- ing GPT-4-based antooannntasors, with results reported length controlled win rate (Dubois al. 2024) and win rate.
As baselines, include the untrained LLaMA2 model, self-rewarding version of LLaMA2 fol- lowing the methodology of Yuan et al. (2024). and an improved meta-rewarding Wu et al. (024a)) version of LLaMA2 with the addition of providing correct answers during the self-rewarding pro- cess when possible. Additionally, we conduct ablation studies by disabling the tools and feedback modules evaluate their individual contributions Anyprefer.
16

---

Published as a conference paper at ICLR 2025
A.3 NATURAL VISION-LANUUAGE UNDERSTANDING
A.3.1 DATASET AND BASELINES
Besides the original LLaVA-1.5-7b model, its self-rewarding version and meta rewarding as base- lines, we also incorporate a wide range of other preference data construct method, including: Silkie:[Li et al., 2023c) Constructs VLFeedback dataset by generating responses from 12 LVLMs based on multimodal instructions. GPT-4V evaluates these responses on helpfulness, visual accu- racy, and ethical considerations. LLaVA-RLHER (Sun et al.) 2023) Introduces Factually Augmented RLHF, an algorithm that improves the reward model by incorporating factual data such as image captions and ground- trutt multi-choice answers. POVID: (Zhou al. (024a) Aligns VLLMMs pref- erences using external data from GPT-4 and the hallucination tendencies observed in noisy images. RLHE-V: (Yu et al., (2244)) Gathers human corrections on hallucinations at a paragraph level and applies dense direct preference optimization based on human feedback.
A.3.2 EVALUATION BENCHMARK
We conducted evaluations on three types of benchmarks: comprehensive benchmarks, general VQA and hallucination benchmarks. Specifically, this includes:
MME: (Fu et al. 2024) A broad benchmark for assessing LVLMs in multimodal tasks, focusing on both perception and cognition. It tests models across 14 subtasks that challenge their interpretative and analytical abilities.
LLaVAW; W; (Lu et al.
visual reasoning benchmark with 24 diverse images and 60 ques- tions, covering range of scenarios from indoor or outdoor environments abstract art.
2024
MMBench: [Liu al.
2023) Expands evaluation scope with curated dataset and introduces the CircularEval strategy, which uses ChatGPT to transform free-form predictions into struc- tured multiple-choice answers.
MM-Vet: eYu al. 2023) Assesses LVLMs through 16 multimodal tasks built from six core vision- language skills, providing detailed insights into model performance across various question types and response formats.
ScienceQA: (Saikh et al. 2022 A multimodal benchmark targeting multi-hop reasoning in science, containing 21K multiple-choice questions with associated explanations and lectures.
VizWiz: (Bigham et al.
2010) A VQA dataset with over 31,000 goal-oriented visual questions, featuring images taken by blind users and their spoken queries, along with crowdsourced answers.
GQA: (Hudson & Manning, 2019)
A visual reasoning dataset with 22 million semantically- generated questions based on scene graphs, designed to evaluate consistency, grounding, and plausibility in model responses.
POPE: Li et al. 2023e) A binary classification task detect object hallucination in LVLMs, using yes or no questions and diverse object sampling strategies to expose hallucination tenden- cies.
A.4 MEDICAL IMAGE ANALYSIS
A.4.1 DATASET AND BASELINES
We evaluate the performance of our method on three key datasets targeting medical image analysis
tasks: (1) VQA-RAD (Lau et al. 2018) contains 3,515 question-answer pairs and 315 radiology images, with questions categorized into types like abnormality, modality, and organ system. An- swers include both yes/no and open-ended responses. (2) SLAKE (Liu et al. 2021 consists of 642 radiology images and over 7,000 diverse QA pairs, requiring external medical knowledge and annotated with segmentation masks and bounding boxes. We only consider the English subset. (3) IU-Xray (Demner-Fushman Fushman et al. 2016) focuses medical report generation, containing chest X- ray images paired with detailed clinical reports, evaluating the model's ability to generate accurate medical text based images.
17

---

Published as a conference paper at ICLR 2025
As baselines, include the LLaVA-Med- 1.5 model (Li et al. 2023a), self-rewarding version of
LLa v1.5, and a meta-rewarding version of LLaVA-Meed- 1.5. Additionally, we adapt the VLFeedback method to LLaVA- for comparison. Additionally, perform ablation studies by disabling the tools and feedback modules to assess their individual contributions to Anypre fer.
A.5 VISUO-MOTOR CONTROL
A.5.1 DATASET AND BASELINES
We employ Simpler-Eny (Li et al. 2024b) as our experiment environment and dataset. SIMPLER (Simulated Manipulation Policy Evaluation for Real Robot Setups) is a suite of simulated environ- ments designed to evaluate real-world robot manipulation policies. SIMPLER utilizes simulated environments as an effective proxy for real-world testing, addressing the challenges of real robot evaluations, which are typically expensive, slow, and difficult to reproduce.
To comprehensively assess the performance of our proposed method, we conducted baseline com- parisons with several state-of-the-art robotic models. RT-1 (Bohhn et al.) 2022) is a sophisticated robotic control system designed to handle real-world tasks at scale. utilizes Transformer-based architecture trained on upproximately 130,000 demonstrations covering over 700 tasks, enabling it to generalize across a variety of tasks with minimal task specific data. OctorTeam et al.] 2024 is an open-source, generalist robot policy trained 800,000 diverse robot episodes from the Open X- Embodiment dataset. Employing a transformer-basee architecture, Octo demonstrates robust adap- tation to various tasks, robots, and environments; and base (93M parameters) versions. OpenVLA (Kim et al., 2024) is 7B-parameter open-source vision-language-actio model designed for generalist robot manipulation policies, trained on 970k robot demonstrations from the same dataset. Key features of Open VLA include its ability to con- trol multiple robots directly and its adaptability new robot domains through efficient fine-tuning. We used the Open VLA-baseline model, which was fine-tuned on the Simpler-Env dataset through supervised learning. These models were selected as baselines for comparison in our experiments to evaluate the effectiveness of our proposed method. Because Open VLA can not generate word, use LLaVA-1.5-7B for self-rewarding. Regarding the dataset, the Simpler-En -nv dataset was created by using the OpenVLA model fine-tuned the bridge-v Walkee al. (2023) data to generate 500 successful trajectories within Simpler-Enn Eny et al. (2024b).
evaluated both its small (27M parameters)
A.5.2 EVALUATION BENCHMARKS
All the baseline models were tested on four WidowX robot tasks within the Simpler-En Env:
1. Put the carrot on a plate
2. Put the spoon on a towel
3. Stack the green cube on the yellow eube
4. Put the eggplant in basket
For each task, executed 50 trials where the positions of the source and target objects were ran- domly generated. The evaluation was based on whether the objects could be continuously grasped and whether the tasks were successfully completed. We compared the generated trajectories from each model with the ground truth trajectories, assessing their performance in terms task success rate.
B SUPPLEMENTARY EXPERIMENTS
B. NATURAL LANGUAGE GENERATION
We present detailed results in Tables Anyprefer achieves substantial improvements across all datasets, particularly when combined with external tools and feedback mechanisms. For natural language, GSM8K and ARC datasets, approach improves the absolute accuracy by 10.92%, 5.81% % and 7.00% relative to the Pareto Optimal untrained and self-rewa baselines, clearly showcasing the strength of integrating external assistance. On AlpacaEval, method outperforms
18

---

Published as a conference paper at ICLR 2025
Table 4: Performance on text tasks. For GSM8K and ARC, we report the accuracy of the final answer. For Alpaca Eval, we report length controlled win rate / win rate (" indicates that the chosen response during the self-rewarding process uses the ground truth).
Method
GSM8K
ARC-Easy
ARC-Challenge
Alpaca Eval 2.0
T. Ilama-2
22.44
74.33
57.68
5.20 / 4.57
Self Rewarding
23.20
74.45
56.31
3.28 / 3.12
Self Rewarding"
27.22
73.53
56.66
Meta Rewarding
25.47
76.22
59.47
Anyprefer
38.14
80.26
64.68
19.25 15.14
Table 5: Ablation study of natural language generation.
Method
GSM8K
ARC-Easy
ARC-Challenge
Alpaca Eval 2.0
Feedback Mechanism Ablation
Anyprefer
30.10
78. 16
62.37
3.99 / 3.75
Anyprefer (tools)
37.53
78.70
63,40
18.96 14.40
Anyprefer (tools feedback)
38.14
80.26
64.68
19.25 15.14
Table 6: The multi-round preference iteration results of Llama2 and Anyprefer on the GSM8K dataset. The superscript "(" denotes LLaMA2, and the superscript "a" denotes Anyprefer (tools feedback).
Base'
Iter-1"
Iter-2"
Iter-3"
22.44
38.14
39.04
41.32
Table 7: Comparison of different methods on natural vision-language understanding.
Method
MME
MME
LLaVA
MMB
MMVet
SQA
VisWiz
GQA
POPE
LLaVA-1.5-7B
1510.7
348.2
64.3
66.8
50.0
62.0
85.90
Vfeedback
1432.7
3218
62.1
64.0
31.2
66.2
52.6
63.2
83.72
Paeman-Prefer
1490.6
335.0
63.7
63.4
31.1
65.8
51.7
61.3
81.50
+ POVID
1452.8
325.3
68.7
64.9
31.8
68.8
53.6
61.7
86.90
+ RLHE-V
1489.2
349.4
65.4
63.6
30,9
67.1
54.2
62.1
86.20
Rewarding
1505.6
362.5
69.6
53.9
86.88
Meta Rewarding
1498.3
357.4
64.0
64.2
69.1
53.5
62.0
86.70
+ Anyprefer 1510.1
362.9
69.2
65.1
33.0
70,9
54.0
62.2
86.98
simpler setups with more than threefold increase in win rates. In contrast, the self-rewarding mech- anism alone struggles to deliver meaningful improvements, with gains being marginal best. While self rewarding and meta rewarding offer some benefits, it alone cannot significantly enhance the per- formance of smaller models like LLaMA2-7B in complex tasks, indicating the need for additional support. Ablation studies further validate the effectiveness of each component in our approach. Dis- abling either the tools or feedback modules leads notable performance declines, confirming that both elements are crucial to maximizing the model's 's potential.
B.2 NATURAL VISION-LANUUAGE UNDERSTANDING
In this section, present detailed experiment results natural Dision-languse understanding.
Table compares the performance of Anypre fer against other methods. The results demonstrate that Anyprefer consistently outperforms prior approaches across most benchmarks, highlighting the effectiveness of our framework and the robustness of the constructed dataset.
To further investigate the impact of key components within Anyprefer, perform ablation stud- ies by systematically removing the tool utilization feature and varying the feedback iterations. The outcomes of these studies summarized in Table g Our findings reveal that integrating tools into
19

---

Published as a conference paper at ICLR 2025
Table The multi-round preference iteration results of LLaVA-115 on natural vision-language un-
derstanding.
Method
MME.
MME
LLaVA W
MMB
MMYet
SQA'
VisWix
GQA
POPE
LLaVA-1.5-7B
1510.7
348.2
64.3
50.0
Anyprefer Iter-1
1502.0
358.0
67.4
64.8
32.3
70.5
53.7
62.1
86.22
+ Anyprefer Iter-2
1506.5
360.3
67.2
64.9
32.4
70.7
53.6
62.0
86.95
+ Anyprefer Iter33
1510.1
362.9
69.2
65.1
33.0
70.9
54.0
62.2
86.98
Table 9: Ablation study of natural vision-language understanding. For the rank ablation, we default using lower-ranked responses as dispreferred data and higher-ranked responses preferred data.
Method
MME
MME
LLaVA
MMB
MMVet
SQA'
VisWiz
GQA
POPE
Feeelbuck Mechanism Ablution
Anyprefer
1488.5
340.4
64.3
64.7
31.7
69.9
53.4
62.0
86.92
Anypeefer (tools)
1498.2
357.5
66.8
64.6
32.1
70.3
53.6
62.1
86.90
Anypeefer ftools + feedbuck)
1510.1
362.9
69.2
65.1
33.0
70.9
54.0
62.2
86.98
Optimization Target Ablation
Optimize Target Model Only
1480.2
2
65.2
63.9
31.0
67.2
61.5
Optimize Judge Model Only
1485.6
353 1
66.1
64.1
31.5
68.0
53.2
61.8
86.45
Independenily Optimize Both Models
359.0
Rank Ablation
Anypeefer (nnnk3 rank5) Anypreter (rank3 + tank1)
1508.3
361.2
68.5
65.0
32.8
70.1
53.8
62.2
86.89
the framework enhances perceptual and cognitive capabilities, while increasing the number of feed- back iterations yields additional performance gains. These results underscore the critical role that tools and feedback mechanisms play in our framework.
B.3 MEDICAL IMAGE ANALYSIS
We evaluate the performance models benefited from Anyprefer across two tasks and three widely-used datasets. As demonstrated in Figure 2 Anypre performs the best overall pre- formance, with an average improvement of 31.0%. As shown in Table [0] for medical VQA and report generation, the performance increased by 13.14% and 67.8%, respectively. Interestingly, we can also observe that model performance is improved significantly on report generation task, which attributed Anyprefer enhancing the open- open-ended generation capability. Compared with self-rewarding method, Anyprefer significantly outperforms the basclinc method by 28 28.4%. By leveraging state-of-the-art medical models as external tools, we constructed an enhanced pref- erence dataset, which significantly outperformed the self-rewarding approach. This improvement attributed to the higher level of expertise and accuracy provided by specialized medical models in tasks such as VQA and medical report generation. Additionally, the integration of powerful central multimodal model (e.g., GPT-4o) for information synthesis and reward judgment further en- hances the model's ability to handle complex medical scenarios, resulting in significantly improved generation quality and accuracy.
Furthermore, the results indicate that increasing the number of external tools and incorporating feedback mechanisms both lead to notable improvements, particularly in medical report generation tasks. This suggests that approach is especially effective for open-ended generation tasks. The improvement can be attributed to the enhanced capacity of the model to integrate domain-specific knowledge from multiple tools, while the feedback mechanism allows for iterative refinement, en- abling the model to better capture the complexity and variability of medical reports, thereby produc- ing more accurate and contextually appropriate outputs.
B.4 VISUO-MOTOR CONTROL
The experimental results presented in Table 13 Anyprefer, performed notably well compared to other models. With the integration of tools and feedback mechanisms, the performance across all tasks further enhanced. The information provided by the tools improved the accuracy of
20

---

Published as a conference paper at ICLR 2025
Performance on medical VQA and report generation tasks. For open-set questions, report the recall in column Open. For closed-set questions, we report the accuracy in column Closed. indicates that the chosen response during the self-rewarding process uses the ground truth.
Table 10:
VQA-RAD
SLAKE
IU-Xray
Closed
Open
Closed
Open
BLEU-1
BLEU-2
BLEU-3
BLEU-4
ROUGE-L
METEOR
LLa Med
63.57
32.09
61.30
44.26
10.31
0.66
0.07
0.01
10.32
10.95
VI V..eeedback
64.33
32.38
61.52
44.03
10.65
0.67
0.10
0.03
10.78
11.36
Self Rewarding
64.17
33.29
61.30
42.63
9,71
0.97
0.10
10.38
10.52
Self Rewarding'
66.25
32.19
63.28
42.80
9.56
1.03
0.18
0.02
11.14
11.83
Meta Rewarding
45.12
Anyprefer
72.06
36.10
70.39
49.04
16.85
5,57
2.07
0.56
23.69
29.66
Table 11: The multi-round preference iteration results of medical image analysis.
VQA-RAD
SLAKE
IU-Xray
Closed
Open
Closed
Open
BLEU--
BLBU-2
BLEU-3
BLEU
ROUGE-L
METEOR
LLaVA-Med
63.57
32,09
61.30
44.26
10.31
0.66
0.07
0.01
10.32
10.95
Anyprefer Iter-1
70.96
35.58
67.40
47.69
9.30
2.85
1.12
0.31
19.36
22.24
Anyprefer Iter-2
71.47
35.72
69.22
48.
12.93
4.11
21.87
Anyprefer Iter3
72.06
36.10
70.39
49.04
16.85
5.57
2.07
0.56
23,69
29.66
Table 12: Ablation study medical image analysis. For the rank ablation, we default to using lower-ranked responses as dispreferred data and higher-ranked responses preferred data.
VQA-RAD
SLAKE Closed
IU-Xray BLBU-
Closed Open
BUBU-1
BLBU-2
ROUGEL
METEOR
Fendbock Mechanim Ablation
Anyprefer Anypeefer (tuok)
66.73
32.14
64.66
14.75
9.30
1.19
0.32
0.05
12.42
15.72
1.28
12.95
Anyprefer (tools + feedback)
72.06
36.10
70.39
49.04
16.85
5.57
2.07
0.56
23.69
29.66
Optimization Torgel Ablation
Optimize Target Only
Optimine Judge Model Only
68.71
34.99
67.48
46
13.37
3.79
1.42
0.36
17.89
19.60
Independently Optimize Both Models
70.21
34.89
68.37
47.65
14,57
4.18
1.68
0.43
20.66
23.74
Anyprefer (rank3 + rank5) Anypreter (rank3 + rank))
68.19
34.22
66.74
46 16.19
12.47
3.12
0.84
0.15
18.92
21.12
70.15
35.41
68.32
47,83
14,58
4,37
1.56
0.32
20.53
25.47
the judge model, enabling the model to generate more accurate prompts and trajectories. From the comparison, it is evident that Anyprefer with tool and feedback mechanisms achieved the highest success rates on all tasks, significantly outperforming the other baseline models.
To evaluate the specific contributions of key components in our method the overall perfor- mance, we conducted ablation experiments by removing the image segmentation model Grounded SAM (Ren et al. 2024) and the feedback mechanism. The experimental results presented in Table and Table
and
In the first ablation experiment, we assessed the performance of the model without using the im- age segmentation model Grounded SAM and feedback mechanism. This allowed us to understand the impact of the image segmentation model on object recognition and scene understanding The experimental results showed that without Grounded SAM, the model's accuracy in locating and recognizing target objects significantly decreased, leading to an increased failure rate in trajectory generation. Specifically, the average success rate across the four tasks increased by approximately 15.51%.
The
In the second ablation experiment, removed the feedback mechanism to observe how the ab- sence of detailed feedback affects model training and trajectory generation. The experimental results indicated that without the feedback mechanism, the model struggled to optimize the generated tra- jectories, resulting in lower success rate in task completion. The average success rate across the four tasks increased by approximately 17.46%, respectively.
As shown in Table 13 the integration of tools and feedback mechanisms led to relative improve- ments in the success rates of the four tasks by 26.67%. 38.89%, 38.46%, and 38,89% respectively.
21

---

Published as a conference paper at ICLR 2025
Table 13: Visuomotor-control:: success rates for different tasks and models ( indicates that Open-
VLA can not generate word, use LLaVA-1.5-7B as reward model).
Put Spoon
Towel Success
Put Carrot on Plate Success
Stack Cube
Put Eggplant in Basket
Grasp Spoon
Grasp Carrot
Grasp Cube
Success
Grasp Eggplant
Succeas
RT-1 Octo-small
0.10
0.05
0.20
0.12
0.22
0.02
0.06
0.00
0.42
0.28
0.30
0.16
0.42
0.10
0.48
0.32
Octo-base
0.38
0.20
0.22
0.24
0.04
0.46
0.32
OpeaVLA-SH AFF (baseline)
0.52
0.28
0.36
0,30
0.56
0.20
0.58
0.32
+Solf Rewarding"
0.54
0.28
0.38
0.56
0.22
0.54
0.34
+Anyprefer
0.60
0.38
0.72
0,50
0.70
0.36
0.84
0.50
Table 14: The multi-round preference iteration results of Visuomotor-control
Put Spoon Towel Success
Put Carrot Grasp Carrot
Plate Success
Stack Cube
Put Eggplant in Basket Grasp Eggplant
Grasp Spoon
Grasp Cube
Success
Success
OpenVLA
0.52
0.28
0.36
0.30
0.56
0.20
0.58
0.32
Anyprefer Iter-1
0.52
0.30
0.46
0.36
0.36
Anyprefar Iter-2
0.58
0.34
1166
0.46
0.62
01.34
0.80
0.46
Anyprefer Iter-3
0.60
0.38
0.72
0.50
0.70
0.36
0.84
0.50
Table 15: Ablation study of Visuomotor-control model.
Put Spoon Towel Success
Put Carrol Plate
Stack Cube Grasp Cube
Put Eggplant in Basket Grasp Eggplant
Grasp Spoon
Grasp Carrot
Success
Suceess
Succeas
Feedbook Mechanism Ablation
Anyprefer Anyprefer (tools)
0,52
0.30
0.46
0.36
0.52
0.26
0.70
0.36
0.60
0.42
0.30
0.80
Anyprefer ((ecbskee)back)
0.60
0.38
0.72
0.50
0.70
0.36
0.84
0,50
Anyprefer which combines tools and feedback, outperformed models that lacked either tools or feedback, and those with only tools.
C DIVERSITY EVALUATION
To further validate the diversity of the data, we selected the largest existing preference dataset, VLFeedback, as baseline for comparison. condition number-based approach was employed to evaluate the diversity of synthetic data (including VLFeedback and Anyprefer-vl). Specifically, we randomly sampled 500 examples from each dataset, constructed the covariance matrix of the data matrix (i.c., sample-by-featur matrix), and calculated its condition number quantify data diver- sity. smaller condition number indicates a more dispersed distribution in the embedding space, reflecting higher diversity. As shown in Table 16 the condition number of the Anyprefer dataset smaller, further supporting the conclusion that the Anyprefer dataset achieves higher diversity coverage.
D EVALUATION CRITERIA AND PROMPTS
In this section, list the prompts used in Anyprefer and some the rewarding criteria manually annotated during the experimental phase.
D.1 JUDGE MODEL
Judge Model Prompts
[Task] Suppose that you are an expert in (task_field), please rate the answers of some given questions.
22

---

Published as a conference paper at ICLR 2025
Table 16: Comparison of condition numbers for VLFeedback and Anyprefer-vl datasets.
Method
Condition Number
VLFeedback
1560.70
Anyprefer-vl 1390.15
[Guideline] Focus on correctness (whether the information provided in the answer is accurate according the context) and helpfulness (whether the response answers the question).
[Requirement] First provide analyses to all the answers, then assign each an integer between I and 10, where means the answer is worst and 10 means the answer is perfect.
(examples}
(context]
Query:
{query)
Answers:
(answers}
Aggregate Function Prompt
[Requirement] Based on the provided current knowledge base, the input, output, and the score from the previous round, reconsider the following:
1. Which information from the knowledge base is necessary to solve the current problem and optimize the output, and which information is redundant.
2. Are there any errors in the information from the knowledge base?
After your consideration, reorganize the necessary information you plan to use, and remove any incorrect information. Directly output the consolidated result without additional instructions.
(knowledge information]
fcontext]
Answers: (answers}
D.2 SURROGATE REWARD MODEL
Reward Model Prompts
[Task] Suppose that you are an expert in (task field}, please rate an RLHF data pair consisting of query, positive response and negative response.
[Guideline] Reference criteria:
1. The positive response should be coherent and correct possible;
23

---

Published as a conference paper at ICLR 2025
2. The negative response should worse than the positive one in certain way, but not wander off the topic diverge in too many aspects. For example, if the positive response is "The capital of France is Paris", a good negative response should be something like "The capital of France is London", but not "France is country in Europe" (diverge too much topic) or "Capital France London is" (diverge both knowledge and language).
[Requirement] Please provide an integer score between and 10 indicating the quality of the data pair used in RLHF. The higher the score, the better the data pair. Please first analyze the positive response and the negative response, and then give the score in the format of "score/10".
(examples}
[context]
Query: {query)
Positive Response: (positive)
Negative Response: (negative)
D.3 DETAILS OF MANUAL EVALUATION
For the evaluation of the difficulty of preference data pairs: We classified the difficulty of preference data pairs into four categories: very easy, easy, medium, and hard. The difficulty evaluation is mainly based on:
1. The difference between the preferred data and the dispreferred data in the preference pair. The smaller the difference, the higher the difficulty.
2. The difficulty of the question itself.
For the evaluation of the satisfaction level of the dataset: The evaluation is primarily based the correctness of the preference data pair. For preference data pair, both the preferred data is correct and the dispreferred data is incorrect, it is marked "Satisfied". If of them is incorrect, it is marked as "Okay". Otherwise, it is marked as "Dissatisfied".
24


Reviews:
[{"review_id": "Mq2aKq14bn", "reviewer": "Reviewer_jmiU", "summary": "The paper introduces a framework that automates the synthesis of high-quality preference data. It employs a target model and a judge model, and leverages external tools and a feedback mechanism to enhance data quality. The framework generates a diverse dataset, Anyprefer-V1, which contains 58,000 preference pairs, and improves model performance across applications in natural language generation, vision-language understanding, medical image analysis, and visuo-motor control.", "questions": "1. The authors indicate the the self-rewarding method uses the model itself to reward the responses, which would amplify inherent biases. In this paper, the authors use three different models (target model, judge model, and reward model) for this process. However, there can also be inherent biases in any of those three models, why won't this method amplify the inaccuracies and compromise data quality?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments on four downstream tasks demonstrate that the framework can generalize across various domains and tasks.\n2. Besides proposing a framework, the authors also provide their generated synthetic dataset, which would help future works.\n3. Their experimental results are promising in all the four downstream tasks.", "weaknesses": "1. The overall idea of tool-enhanced critiquing and self-correction is similar to the paper [1]. The authors should have some discussions here.\n2. Equation 2 of prompt optimization is misleading. I believe there should not be any gradient descent for the prompt optimization part.\n3. In line 196 in Section2.3, the authors should explain how is \"high-quality\" defined and what kind of reward R(y+,y-) would indicate high-quality reward.\n4. The authors only consider self-rewarding as the baseline. It's better to explore and include more baselines, such as [2].\n\n[1] Gou, Zhibin, et al. \"Critic: Large language models can self-correct with tool-interactive critiquing.\" arXiv preprint arXiv:2305.11738 (2023).\n\n[2] Wu, Tianhao, et al. \"Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge.\" arXiv preprint arXiv:2407.19594 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z8QlBUCd52", "reviewer": "Reviewer_xwba", "summary": "In this paper, the authors consider the problem of how to generate a preference data dataset synthetically, as current methods for generating high-quality preference-data currently require significant data annotation. As a solution, the authors propose Anyprefer a framework that frames the preference dataset production as a two-player game between the target and judge models, with a third method acting as a reward proxy for what the performance increase would be if the target (base) model was optimized through preference learning. Anyprefer functions by the target model producing a set of candidate responses to some prompt p_t, which are then ranked by a judge model that additionally gets inputs from expert tools that correspond to the kind of dataset that is being optimized for (vision, medical, natural language generation, or control in their case). They then use a reward model that determines if the preference pair, constructed from the best and worst of the ranking, was sufficiently good to be included. The authors then demonstrate the benefits of such generated pairs on a set of downstream tasks in comparison to the original target models, and both a self-rewarding scheme versus their proposed method that are both used to generate pairs and which are subsequently  used for fine-tuning.", "questions": "1. In line 47, its said that \"…this approach may fail to capture the inherence preferences of the target model being fine-tuned, rendering the data-generation less useful\". I found this phrasing quite confusing, as if your target model is imperfect, why would it matter that you aren't aligned with it. From my perspective the whole benefit of the proposed system is effectively a distillation of GPT-4o preferences into the target model.\n\n2. It would've been interesting to see ablations regardless the dataset generation process. For example, if the Judge model produces quality rankings, then doesn't it seem suboptimal to use only the best and worst generation? Couldn't one use many different pairs (preserving ranking order for + or -) from the ranking? Also it seems that 5 was chosen as the number of generations, but it's unclear the role that this plays in the final quality.\n\n3. Once the preference datasets are generated using any of the proposed techniques, what was the process for actually improving the target models? We are given the results but without any experimental setup for that tuning. I'm leaning on making this a weakness, as I think it's important to be included, but it could be that this should be a common knowledge process.\n\n4. Why is it the case that both the prompt of the target model AND of the judge model need to be optimized in this process? Is it not sufficient to optimize the prompt of the target model through the preferences of the judge? It would be interesting to see that this is insufficient.\n\n5. Where does so much improvement come from in the medical VQA task? It seems like 31.05% is a massive gain.\n\n6. In section 3.5, how were these samples manually selected? \n\n7. Was there an analysis done on how including different numbers of tools was helpful for different datasets, or which tools mattered the most?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The authors consider quite a broad set of target models and target domains to demonstrate their method on, showing that it's efficacious in different data scenarios and regimes. \n\n2. Anyprefer cleverly integrates other tools outside of the target and judge model, and because of the nature of the judge model being an incredibly powerful VLM like GPT-4o, this strategy seems like an extendable and scalable as more specialized tools become available.\n\n3. The improvements on downstream target models, in comparison to baselines, is clear. This technique effectively is able to take advantage of GPT-4o to improve the capabilities of weaker models.", "weaknesses": "1. It is unclear in the paper what the authors gain in perspective or methodology from framing their generation as a \"two-player game\". Given that this technical detail was referenced several times, I was expecting that it would be involved in some sort of modeling decision or theoretical result. Another point that I believe is important to explain well, and lacked clarity, was how one can update the policy of the target model p_t, by optimizing the input prompt p_t. This mixes together a few notions from language modeling and RL that I believe should be explained better.\n\n2. The authors should include their benchmarking on IMG/MED datasets with the synthetic dataset VLFeedback in the main text. If I understand correctly, it similarly does not require human effort, is larger than Anyprefer-V1, and could be compared against to demonstrate the claim made by Figure 6 that the diversity of Anyprefer is its strength over existing techniques. Additionally, I was not convinced by the t-SNE plots that Anyprefer seemed to cover all datasets, it is just in the distribution diagram spread that they appear either more sparse (Anyprefer) or more dense (VLFeedback). A quantitative analysis here consistent with what is done in the literature seems necessary.\n\n3. The authors claim in Lines 70 and 95 that the methods used as 'tools' for providing auxiliary information to the judge models are 'selected based on the input prompt' or 'Anyprefer strategically selects tools'. From what I understand in section 3.1, either the tools are selected manually corresponding to the task or it is not explained how 'Anyprefer' selects tools. This makes me believe that how the tools are chosen is ambiguous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0mRohLzppI", "reviewer": "Reviewer_zGJr", "summary": "This paper proposed Anyprefer, an automatic method to synthesize high-quality preference data to improve model alignment with human values. The preference data generation is set as a cooperative two-player Markov game between a target model and a judge model. Using external tools and feedback, the proposed Anyprefer claim to show substantial performance improvement across four applications: natural language generation, vision-language understanding, medical image analysis, and visuo-motor control.", "questions": "Please see the comments above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  This paper solves an important problem in synthetic data generation for model alignment. The proposed framework can automates preference data synthesis, which can reduce the reliance on human annotations.\n- The proposed model uses external tools in the reward phase, which may be able to make the data more diverse, reduce bias, and improve data quality in preference synthesis.\n- The paper conducted extensive experiments and ablation studies across multiple applications.", "weaknesses": "- The setting, a two-player cooperative Markov Game for preference data synthesis, is only vaguely justified. The paper could be better if adding some explanations about  how this approach explicitly improves alignment over simpler, existing methods.\n- External tools are incorporated to reduce bias. It would be good to add additional analysis or metrics to show that these tools effectively mitigate model bias or improve reliability beyond existing methods.\n- The proposed model’s effectiveness is evaluated on a narrow set of datasets within each application domain, which may not be able to show its overall generalizability. It would be good to perform on datasets or domains outside the four tested applications as well.\n- While the  feedback mechanisms are described,  the impact of these mechanisms on the process is not clearly analyzed. It would be good to add ablation study to study their specific contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LJgcGMHmO", "reviewer": "Reviewer_MDCN", "summary": "The paper proposes Anyprefer, a novel framework for generating preference data for aligning AI models. Anyprefer accomplishes the preference data generation process as a cooperative two-player Markov Game between a target model and a judge model. The judge model uses external tools to get rewards and rank the responses generated by the target model. Additionally, a feedback mechanism is adopted to learn the prompts for both models. Experimental results show the effectiveness of Anyprefer comparison to existing baselines such as an untrained model and self-rewarding. The author also conducts a series of analyses to investigate different components of Anyprefer.", "questions": "1. The method requires an additional reward model for refining the prompt, what's the expected performance if directly using the additional reward model as the judge model? \n2. Are there any insights on the adoption of AnyPrefer in different scenarios?\n3. If I understand correctly, the novelty of AnyPrefer lies in tool usage and prompt tunning from reward, these contribution looks more engineering-oriented. Is there any connection between the design of these two modules?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written.\n2. The experiments are good and provide positive results on AnyPrefer.\n3. The author claims to release a preference dataset that may help the community.\n4. The framework is novel with the feedback learning mechanism and tool argumentation.", "weaknesses": "1. The author may provide at least a short explanation on how you update the prompt for GPT-4o (a black box model) with policy gradient? I suspect that not all audients are not famaliar with TextGrad. It is better to provide more experimental details in the appendix on how TextGrad is accomplished in different datasets.\n2. The method requires an additional reward model for refining the prompt, what's the expected performance if directly using the additional reward model as the judge model? \n3. The proposed method is called AnyPreference and the author conducts experiments on various scenarios that need preference data. The comparison and insights on evaluations of different scenarios can be discussed. \n4. The author can consider more references regarding reward modeling, tool argumentation, and preference data construction, including but not limited to:\n   Li L, Chai Y, Wang S, et al. Tool-augmented reward modeling[J]. arXiv preprint arXiv:2310.01045, 2023.\n   Ye Z, Li X, Li Q, et al. Beyond Scalar Reward Model: Learning Generative Judge from Preference Data[J]. arXiv preprint arXiv:2410.03742, 2024.\n   Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland,\n   Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics,\n   pp. 4447–4455. PMLR, 2024.\n​  Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023\n  Wu T, Yuan W, Golovneva O, et al. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge[J]. arXiv preprint arXiv:2407.19594, 2024.\n5. The method can be compared with more strong baselines in addition to an untrained model and self-rewarding, such as meta-rewarding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)