MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness
  Against VLM-based Attacks
Authors: name: Zonglin Wu; name: Yule Xue; name: Xin Wei; name: Yiren Song
Published: 2025-06-06 11:02:01

Abstract: As automated attack techniques rapidly advance, CAPTCHAs remain a critical
defense mechanism against malicious bots. However, existing CAPTCHA schemes
encompass a diverse range of modalities -- from static distorted text and
obfuscated images to interactive clicks, sliding puzzles, and logic-based
questions -- yet the community still lacks a unified, large-scale, multimodal
benchmark to rigorously evaluate their security robustness. To address this
gap, we introduce MCA-Bench, a comprehensive and reproducible benchmarking
suite that integrates heterogeneous CAPTCHA types into a single evaluation
protocol. Leveraging a shared vision-language model backbone, we fine-tune
specialized cracking agents for each CAPTCHA category, enabling consistent,
cross-modal assessments. Extensive experiments reveal that MCA-Bench
effectively maps the vulnerability spectrum of modern CAPTCHA designs under
varied attack settings, and crucially offers the first quantitative analysis of
how challenge complexity, interaction depth, and model solvability interrelate.
Based on these findings, we propose three actionable design principles and
identify key open challenges, laying the groundwork for systematic CAPTCHA
hardening, fair benchmarking, and broader community collaboration. Datasets and
code are available online.