Title: Robust SuperAlignment: Weak-to-Strong Robustness Generalization for Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Superalignment

weak-to-strong generalization

sdversarial robustness

vision-language model
Abstract: Numerous well-established studies have demonstrated the superhuman capabilities of modern Vision-Language Models (VLMs) across a wide range of tasks. However, growing is the doubt about the continuing availability of reliable high-quality labeling (supervision) from human annotators, leading to stagnation of the model's performance. To address this challenge, ``superalignment'' employs the so-called weak-to-strong generalization paradigm, where the supervision from a weak model can provide generalizable knowledge for a strong model. While effective in aligning knowledge for clean samples between the strong and weak models, the standard weak-to-strong approach typically fails to capture adversarial robustness, exposing strong VLMs to adversarial attacks. This inability to transfer adversarial robustness is because adversarial samples are normally missing in the superalignment stage. To this end, we are the first to propose the weak-to-strong (adversarial) robustness generalization method to elicit zero-shot robustness in large-scale models by an unsupervised scheme, mitigating the unreliable information source for alignment from two perspectives: alignment re-weighting and source guidance refinement. We analyze settings under which robustness generalization is possible. Extensive experiments across various vision-language benchmarks validate the effectiveness of our method in numerous scenarios, demonstrating its plug-and-play applicability to large-scale VLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introdces the robustness of vision-language models in the context of weak-to-strong generalization. The authors reveal that weak student models cannot provide robustness by simple distillation, and emphasize the importance of incorporating adversarial samples during training. In addition, this paper proposes Entropy-Guided Uncertainty Re-Weighting and Teacher Guidance Refinement via Inverse Adversarial Examples for improving robustness of student model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Clear and Logical Writing.\n2. To the best of my knowledge, this is the first paper to address the robustness of vision-language models specifically in the context of weak-to-strong generalization.\n3. The theoretical justification that increasing the teacher model’s confidence leads to greater robustness in the student model is highly logical, and the proposal of inverse adversarial examples is well-reasoned.\n\nWeaknesses:\nIn weak-to-strong generalization loss function, replacing the constant balancing factor $\\beta$ with an adaptive parameter was already explored in [1], which balances the weight utilizing the confidences of both the teacher and student models. I wonder what the advantages of EUR compared to the adaptive weight [1] and how the performance would change if EUR were replaced by the adaptive weight [1].\n\n\n[1] Guo, Jianyuan, et al. \"Vision superalignment: Weak-to-strong generalization for vision foundation models.\" arXiv preprint arXiv:2402.03749 (2024).\n\nQUESTIONS:\nAs describe in Weakness, I wonder how EUR compares to the adaptive weighting approach proposed in [1]. Specifically, what are the unique benefits of EUR, and how does the model's performance change when EUR is substituted with the adaptive weight from [1]?\n\n[1] Guo, Jianyuan, et al. \"Vision superalignment: Weak-to-strong generalization for vision foundation models.\" arXiv preprint arXiv:2402.03749 (2024).\n\nLIMITATIONS:\nYes, the authors report the limitations and broader impacts in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework called Adv-W2S (Adversarial Weak-to-Strong) to improve the robustness of VLMs against adversarial attacks. It makes several key contributions: it provides theoretical analysis that explains the shortcomings of traditional weak-to-strong generalization methods in terms of robustness, attributing this issue to the lack of adversarial samples in the learning objective; it introduces a new method based on adversarial sample generation, allowing robust knowledge to be extracted from strong student models in an unsupervised way; and it validates the effectiveness of Adv-W2S through extensive experiments, showing its superior performance in zero-shot classification tasks as well as its applicability to various downstream tasks such as image captioning, visual question answering, and COT reasoning. These contributions offer valuable insights for addressing current vulnerabilities in the security and robustness of VLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n① The writing is excellent, and the paper is logically structured.\n\n② New method design: The authors propose a weak-to-strong generalization framework called Adv-W2S, based on adversarial sample generation, which can extract robust knowledge from a strong student model in an unsupervised manner.\n\n③ Comprehensive experimental validation: The authors conduct extensive experiments across multiple network architectures and evaluation scenarios, demonstrating the effectiveness of Adv-W2S.\n\nWeaknesses：\n\n① The paper introduces the parameter β in the CLIP weak-to-strong formulation (i.e., Equation (4)) . However, there is limited discussion later in the paper regarding the specific value of β. It would be beneficial to include a brief analysis or ablation study on how β affects the experimental results.\n\n② The authors could refer to the PGR formulation used in Burns' work to evaluate the robust generalization performance of their method. Additionally, introducing a \"strong ceiling\" — that is, showing the upper bound of performance when directly training the strong model — would make the robustness comparison more convincing.\n\nQUESTIONS:\nWhen I first started reading the paper, a question came to mind: The term \"superalign\" usually refers to the field of AI alignment, where the focus is on aligning highly intelligent, complex, or enhanced models with human values and intentions. It primarily concerns how to ensure that superhuman-level AI systems (such as models that perform beyond human capabilities) remain consistent with human values and objectives.\n\nCLIP clearly does not fall into this category. However, in the later experimental sections, the paper integrates CLIP into large-scale VLMs such as LLAVA, which partially addresses this issue. It would be helpful if the authors could clarify this point more explicitly at the beginning of the paper. However, this does not detract from the fact that this is still a good paper.\n\nFurthermore, based on this point, since the authors likely have sufficient GPU resources, expanding the experiments to even larger models would better reflect the \"superalignment\" scenario.\n\nother comments see weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the weak-to-strong robustness generation problem is studied, and the key finding is that the strong teacher VLM is also vulnerable to adversarial attack as the training objective is not attack-aware. To this end alignment re-weighting and reference refining are proposed to enhance the weak-to-strong alignment and improve the model's adversarial robustness. In detail, besides using a fixed weights $\\beta$ to balance the tradeoff between teacher-student alignment and student self-refinement, a teacher entropy to measure the uncertainty is proposed and used to guide the adaptive weights. The proposed approach is validated in multiple datasets with multiple tasks like linear probing, image caption, VQA, hallucination, etc.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The paper is well formated and written, with both theoretical analysis and experiments.\n- Sufficient experiment coverage, the proposed method is validated in multiple vision-language tasks, like zero-shot classification, image caption, VQA, and hallucination.  \n- The pretrained vision encoder is validated on multiple open sourced base models, like llava 1.5 and openflanmingo.\n\n### Weakness\n- The studied adversarial attack seems to be limited: 1) only studied visual encoder. 2) limited to perturbation based attack\n- Did not test with more recent VLM models like llava 1.6 and Qwen VL\n\nQUESTIONS:\n- This work is mostly focusing on the perturbation based adversarial attack to the vision encoder, which is fine. But one follow up question would be: Can this approach be generalized to other adversarial attacks? E.g, the perturbation based attack is also studied in text only language models [1]. From principal, it seems this entropy based approach can be generalized to this case. Another possibility is extend the approach to adversarial attack that's not perturbation based, like prompt based attack [2]\n\n- Another hypothesis is that stronger VLM based model will be more robust, so it would be good to see how robust the more recent VLM models are, like the Qwen VL and llava-onevision, or llava 1.6(next).\n\n\n[1] Kim, Jaehyung, et al. \"RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training.\" EMNLP (Findings). 2023.\n[2] Xu, Xilie, et al. \"An LLM can Fool Itself: A Prompt-Based Adversarial Attack.\" ICLR. 2024.\n\nLIMITATIONS:\nYes, limitations are discussed in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of weak-to-strong (W2S) generalisation for vision-language models (VLMs) in the context of adversarial robustness, a capability that standard W2S approaches fail to transfer. The authors identify the root cause as a mismatch in the data used for alignment: while clean examples suffice for natural accuracy, they fall short for robustness. \n\nTo address this, the paper proposes Adv-W2S, an unsupervised, adversarially guided W2S generalisation framework that: 1) generates adversarial examples for the strong student and inverse adversarial examples for the weak teacher; 2) re-weights teacher guidance using an entropy-based uncertainty score; 4) aligns the student on adversarial inputs to the teacher’s refined predictions. The paper provides theoretical analysis (Theorems 1-3) to elucidate how low-entropy supervision encourages larger feature norms and margins, which underpin improved robustness of their proposed method. Extensive experiments on 20 datasets across diverse tasks, including zero-shot classification, image captioning, VQA, and hallucination show that Adv-W2S consistently improves both clean accuracy and adversarial robustness over strong baselines. Comprehensive ablations further validate the effectiveness of Adv-W2S.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n-\tThis paper demonstrates a clear problem formulation and empirical evidence, where vanilla W2S cannot transfer robustness - even with a robust.  \n-\tAdv-W2S is simple yet novel: coupling adversarial and inverse-adversarial views with entropy-based curriculum elegantly addresses noisy teacher signals.\n-\tA practical plug-and-play replacement vision encoder for downstream VLMs.\n-\tExtensive experiments and ablation empirically validate the efficacy of Adv-W2S across a wild range of tasks and benchmarks.\n-\tThe paper is clearly structured and easy to follow.\n\nWeaknesses:\n-\tThe authors have presented several theoretical results, it is suggested to provide intuitions following each result to improve clarity.\n-\tTable 11 shows that maximising the teacher-student gap outperforms student-only perturbations, but no intuition is provided beyond “better transfer”.\n-\tTheorem 2 establishes an $\\ell_2$ robustness radius of $ \\gamma / L_{\\max} || \\Psi_c - \\Psi_y ||_ 2$, however, it seems like the empirical evaluations rely on $\\ell_\\infty$ attacks. The theoretical bound may not be applicable to the evaluation setting without further clarification.\n-\tIt is suggested to provide computational cost analysis.\n-\tWriting can be improved (See questions).\n\nQUESTIONS:\n-\tHow was the one-hot labels based on strong model predictions derived in Eq. 4)? (I.e., how was the threshold determined to derive the one-hot labels given the predictions of the student model?).  \n-\tAll reported robustness results are against $\\ell_\\infty$ PGD/AutoAttack with $\\epsilon \\leq 4/255$. It remains unclear how Adv-W2S performs under other perturbation types, such as larger $\\epsilon$, $\\ell_2$-norm attacks, or text-prompt-based adversaries.\n-\tSince $κ_H$ in Eq.6) is set to the median teacher entropy, what happens when teacher predictions are close to uniform?\n-\tCan Adv-W2S handle open-set classes unseen by the teacher? How is $ \\mathbf{p}_T (x)$ defined in that case?\n-\tIt seems like Theorem 2 assume the image encoder $\\Phi$ is globally $L$-Lipschitz under the $\\ell_2$ norm. How does this impact in practice when the method is implemented with a standard CLIP backbone, which is not Lipschitz-controlled.\n-\tSeveral typos found: e.g., ‘Robutness’ in Section 3.2 Heading, ‘satifies’ in Line 202, ‘psedolabels’ in Line 222.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations. Some suggestion to consider is how the presented theoretical analysis relate practically.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-05"}