Title: Towards Robust Multimodal Learning via Adaptive Model Assembly
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Adversarial fine-tuning, adversarial robustness, adaptive model generation, multimodal attack, multimodal learning

Abstract: Adversarial fine-tuning is a widely used strategy to enhance the robustness of vision-language pre-trained models (VLPs) such as CLIP, ALBEF, and TCL. Traditional methods, however, typically fine-tune a single static model to defend against a specific attack type, limiting their ability to generalize to diverse or unseen adversarial threats. To address this, we propose Multimodal Adaptive Adversarial Fine-tuning (MAAF), a novel framework that achieves robust multimodal learning by adaptively assembling input-conditioned model parameters at inference time. MAAF starts from a shared base model and learns multiple defense vectors, which are dynamically fused through a lightweight, input-aware generation network to produce robust, sample-specific model parameters. This adaptive assembly allows the model to resist a wide range of adversarial attacks without retraining. Extensive experiments on standard vision–language benchmarks show that MAAF substantially enhances adversarial robustness while preserving clean accuracy, consistently outperforming existing fine-tuning methods. The results also provide insights into the distribution of defense vectors, the importance of adaptive fusion, and the optimal number of vectors for achieving a balance between robustness and stability. Code is available at \href{https://anonymous.4open.science/r/MAAF-63FC}{https://anonymous.4open.science/r/MAAF-63FC}.