AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large
  Language Models on Harmfulness
Authors: name: Zixin Chen, name: Hongzhan Lin, name: Kaixin Li, name: Ziyang Luo, name: Zhen Ye, name: Guang Chen, name: Zhiyong Huang, name: Jing Ma

The proliferation of multimodal memes in the social media era demands that
multimodal Large Language Models (mLLMs) effectively understand meme
harmfulness. Existing benchmarks for assessing mLLMs on harmful meme
understanding rely on accuracy-based, model-agnostic evaluations using static
datasets. These benchmarks are limited in their ability to provide up-to-date
and thorough assessments, as online memes evolve dynamically. To address this,
we propose AdamMeme, a flexible, agent-based evaluation framework that
adaptively probes the reasoning capabilities of mLLMs in deciphering meme
harmfulness. Through multi-agent collaboration, AdamMeme provides comprehensive
evaluations by iteratively updating the meme data with challenging samples,
thereby exposing specific limitations in how mLLMs interpret harmfulness.
Extensive experiments show that our framework systematically reveals the
varying performance of different target mLLMs, offering in-depth, fine-grained
analyses of model-specific weaknesses. Our code is available at
https://github.com/Lbotirx/AdamMeme.