Title: Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: AI Safety

AI Alignment
Abstract: Multimodal large language models (MLLMs) are essential for building general-purpose AI assistants; however, they pose increasing safety risks. How can we ensure safety alignment of MLLMs to prevent undesired behaviors? Going further, it is critical to explore how to fine-tune MLLMs to preserve capabilities while meeting safety constraints. Fundamentally, this challenge can be formulated as a min-max optimization problem. However, existing datasets have not yet disentangled single preference signals into explicit safety constraints, hindering systematic investigation in this direction. Moreover, it remains an open question whether such constraints can be effectively incorporated into the optimization process for multi-modal models. In this work, we present the first exploration of the Safe RLHF-V -- the first multimodal safety alignment framework. The framework consists of: (I) BeaverTails-V, the first open-source dataset featuring dual preference annotations for helpfulness and safety, supplemented with multi-level safety labels (minor, moderate, severe); (II) Beaver-Guard-V, a multi-level guardrail system to proactively defend against unsafe queries and adversarial attacks. Applying the guard model over five rounds of filtering and regeneration significantly enhances the precursor model’s overall safety by an average of 40.9%. (II) Based on dual preference, we initiate the first exploration of multi-modal safety alignment within a constrained optimization. Experimental results demonstrate that Safe RLHF effectively improves both model helpfulness and safety. Specifically, Safe RLHF-V enhances model safety by 34.2% and helpfulness by 34.3%.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Safe RLHF-V, the first comprehensive framework for safe alignment of multimodal large language models (MLLMs). The method integrates three key components: (1) a dual-preference dataset (BeaverTails-V) with separate annotations for helpfulness and safety, including fine-grained severity labels; (2) a guardrail system (Beaver-Guard-V) that performs multi-level filtering through rejection and regeneration; and (3) a constrained reinforcement learning algorithm that optimizes for helpfulness under safety constraints using a min-max Lagrangian framework with a budget-bound strategy. Experimental results across five MLLMs and multiple benchmarks show consistent improvements in both safety (+34.2%) and helpfulness (+34.3%).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis paper investigates the problem of aligning multimodal large language models (MLLMs) with both helpfulness and safety objectives. The authors propose a dual-objective alignment framework that separately models reward and cost, grounded in constrained optimization theory. A key contribution is the construction of the BeaverTails-V dataset, which includes diverse harmful content categories and high-quality dual annotations, offering a valuable benchmark for future research. In addition, the paper introduces Beaver-Guard-V, a multi-stage guardrail system that significantly improves robustness against adversarial queries via rejection and regeneration mechanisms. Extensive experiments and ablation studies across various models, evaluation metrics, and data scales comprehensively validate the effectiveness and stability of the proposed method.\nWeaknesses\n1. The generalization ability to other modalities (e.g., speech, video) is not evaluated. Although the current focus is on image-text, would the same framework apply in other multimodal contexts?\n2. The subjectivity and consistency of the safety labels, particularly for minor/moderate/severe categories, is not fully discussed. Did the authors measure annotator agreement or validate against human gold standards?\n3. While the Lagrangian min-max formulation is well-motivated, its comparison with other safety-constrained RL methods is missing?\n4. How computationally expensive is Safe RLHF-V, especially the multi-round Beaver-Guard-V moderation? Can it be efficiently integrated into real-time inference?\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article aims to investigate safety alignment fine-tuning techniques based on multimodal large models. Previous fine-tuning approaches based on Reinforcement Learning from Human Feedback (RLHF) have typically been applied to language-only large language models, with limited application to vision-language models, thus this paper partially addresses this gap. The contributions of this paper are threefold: first, it proposes a dataset for multimodal RLHF alignment; second, it designs a defense system against harmful outputs for multimodal large models; and third, it improves safe RLHF by dynamically updating the Lagrange coefficients in constrained optimization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1. The topic of multimodal alignment and the balance between its safety and usability has been underexplored and is of significant importance.\n\n2. The workload of this paper is substantial, and its results are comprehensive. Centered on the theme of multimodal alignment, it not only proposes a new dataset (along with a data collection methodology) but also designs a harmful output detection system and introduces algorithmic innovations for safe RLHF. Overall, the content is systematic and robust.\n\n**Weaknesses**\n1. This work improves the algorithm based on safe RLHF, resulting in safe RLHF-V. However, the novelty appears limited. The authors dynamically update the Lagrange coefficient in Equations 7 and 8, increasing the coefficient to impose greater penalties when the cost is high, and decreasing it otherwise. For such constrained optimization problems, a common approach is to fix the value of λ (i.e., treating it as a hyperparameter for the penalty term, such as in L1-based weight sparsity). The contribution of this algorithm seems to lie solely in the dynamic adjustment of this coefficient.\n\n2. There are some minor issues, such as the lack of explanation for the meanings of the variables in Equations 7 and 8. It is recommended to review the readability of the entire text and include additional details in the main body.\n\nQUESTIONS:\nPlease see weaknesses. I will increase my rating if the authors can demonstrate the novelty of the RLHF-V algorithm.\n\nLIMITATIONS:\nIt is recommended that the authors include a limitations section in the main text to provide a more detailed description of the study's limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Safe RLHF-V, the first comprehensive framework for safety alignment in multimodal large language models (MLLMs), which integrate visual and textual inputs. Traditional RLHF methods often optimize for helpfulness alone, neglecting safety, especially in the multimodal setting. To address this, the authors propose a dual-objective alignment approach that simultaneously optimizes for helpfulness and safety via constrained reinforcement learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Safe RLHF-V is the first to explicitly disentangle and optimize both helpfulness and safety in multimodal alignment.\n\n2. The authors construct BeaverTails-V, a well-annotated dataset that better reflects harmful multimodal interactions, and Beaver-Guard-V, a robust moderation system with fine-grained QA-pair severity filtering, improving both evaluation and real-world safety.\n\n3. The paper is well-written and easy to follow.\n\n**Weaknesses**\n\n1. The novelty is limited. While I acknowledge the effort to disentangle and optimize both helpfulness and safety in multimodal alignment, I do not consider this a strong novelty on its own.\n\n2. The authors claim that constrained optimization has shown effectiveness in language models, but that training becomes highly unstable in multimodal settings. This point is not well-justified. First, the issue that reward shaping tends to improve only a single dimension is not unique to the multimodal setting—it also exists in other domains. Thus, the motivation is not very convincing. Second, as shown in Figure 3, some static reward shaping settings actually lead to improvements in both metrics, which weakens the authors' claim.\n\n3. Much of the data, including prompts, responses, and annotations, is generated or evaluated by LLMs like GPT-4o, which may inherit biases or inconsistencies. The lack of human-annotated validation raises questions about reliability.\n\nQUESTIONS:\nplease see weakness\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework named Safe RLHF-V, designed for safety alignment of MLLMs. The authors tackle the challenge by balancing the often conflicting objectives of helpfulness and safety within a constrained optimization framework, which is a meaningful topic in safety alignment. \n\nThe main contributions of this work are threefold: (1) The authors created and open-sourced the BeaverTails-V dataset, which is the first multimodal dataset featuring both helpfulness and safety preference annotations, with multi-level safety labels in LLM; (2) The authors developed Beaver-Guard-V framework, which is a multi-level guardrail system for proactively defending against unsafe queries and attacks; (3) The authors proposed Safe RLHF-V, which is the first alignment algorithm for multimodal settings that balances safety and helpfulness based on constrained optimization. Experimental results show that this framework significantly enhances MLLM's safety while preserving its helpfulness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper is among the first comprehensive studies to explore safety alignment of MLLM. As the capabilities of MLLMs rapidly increase, their associated safety risks become ever more important. This work directly confronts this key challenge, with the proposed Safe RLHF-V framework establishing an important benchmark for the entire field. I found there are some other framework in LLM domain like PKU Safe-RLHF, but for MLLM, this work is the first.\n\n1. One of the major contribution of this work is the creation of the BeaverTails-V dataset, which innovatively introduces independent preference annotations for helpfulness and safety, and provides multi-level safety labels (“mild,” “moderate,” and “severe”). This design addresses existing shortcomings in current datasets, such as the coupling of preference signals and insufficient safety-related annotations, thus offering a valuable resource for future research.\n\n2. Rather than being limited to a single algorithm, the authors provide a comprehensive end-to-end framework—including dataset construction, model defense, and alignment optimization, which collectively comprise a systematic solution and greatly enhance the practical impact of this work.\n\n3. The work models the safety alignment problem as a min-max optimization task based on the Lagrangian method, providing solid theoretical foundations. To address the instability in multimodal training, the authors introduce the concept of Budget Bound. The experimental design is thorough, validating the effectiveness of the proposed approach across multiple mainstream MLLMs, with convincing results.\n\n4. The authors commit to open-sourcing all datasets, models, and code, an approach that is highly commendable. This offers great implementation value for both academia and industrial applications.\n\nWeaknesses:\n\n1. The authors correctly identify instability in multimodal alignment training as a critical challenge and propose the “Budget Bound” as a mitigation. However, the paper lacks a thorough analysis of the fundamental causes of such instability. Is it solely attributable to the updates of Lagrange multipliers? Or could it also stem from the visual encoder, methods of vision-language feature fusion, or inherent noise in multimodal data?\n\n2. While PPO is a classic algorithm in RLHF, its training complexity, high computational cost, and sample inefficiency are well known. Compared to lighter-weight alignment methods like DPO, does the performance improvement of Safe RLHF-V justify potential increases in resource consumption and deployment difficulty? It is recommended that the authors add further discussion on algorithmic efficiency and scalability to provide a more comprehensive perspective on the practicality of the proposed method.\n\n3. The evaluation relies primarily on GPT-4o as a judge to measure helpfulness and safety. While this is a common practice, the inherent bias in “LLM-as-a-Judge” approaches may not accurately reflect the full complexity of real human preferences. In the absence of any human evaluation to calibrate or validate the automatic assessment results, the reliability of the paper’s conclusions is somewhat limited. The authors are encouraged to provide supplementary analysis on the consistency between GPT- and human-based evaluation.\n\n4. The overall fluency of the manuscript could benefit from a thorough proofread. For example, layout and alignment issues in Table 3 detract from the clarity of data presentation, and there are some minor typos and grammatical errors (such as “+DPO (Helpful\")”). Correcting these will enhance the professionalism of the paper.\n\nIn summary, this is an good paper that addresses the critical issue of MLLM safety alignment in a timely manner and provides a thoroughly validated, systematic solution. The contributions are significant and the experiments are sufficient and robust. This framework meet the current needs of vision-language modalities, and paves the way for exciting new directions in future multimodal AI safety research. Extending these principles to more complex modalities such as video and audio will be a key next step for the field. This work lays a solid foundation for tackling these forthcoming challenges. \n\nI recommend acceptance of this paper and believe that, should the authors address the suggestions above, its quality will be further improved.\n\nQUESTIONS:\n1. Constructing the BeaverTails-V dataset is undoubtedly a major contribution, but since it contains diverse categories of harmful content, what specific ethical review measures were taken during data collection and annotation to protect annotators’ physical and mental well-being, and to prevent misuse of the data?\n2. The paper convincingly demonstrates that decoupling “helpfulness” and “safety” preferences leads to superior alignment. Beyond the technical improvements, what key insights have the authors gained from this dual-objective approach? Might this paradigm of decoupling competing values be generalized to other alignment challenges, such as honesty or fairness in AI?\n3. Beaver-Guard-V performs strongly across multiple benchmarks. However, how well does the model generalize? For subtle or novel attack types beyond the 20 categories covered in the BeaverTails-V dataset, does the model still provide robust defenses?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-06"}