Title: Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: multi-modal language models, memorization, safety, unified representations

Abstract: We present *modal aphasia*, a systematic dissociation in which current unified multimodal models
accurately memorize concepts visually but fail to articulate them in writing, despite being trained on images and text simultaneously.
For one, we show that leading frontier models can generate near-perfect reproductions of iconic movie artwork, but confuse crucial details when asked for textual descriptions.
We corroborate those findings through controlled experiments on synthetic datasets in multiple architectures.
Our experiments confirm that modal aphasia reliably emerges as a fundamental property of current unified multimodal models, not just as a training artifact.
In practice, modal aphasia can introduce vulnerabilities in AI safety frameworks, as safeguards applied to one modality may leave harmful concepts accessible in other modalities.
We demonstrate this risk by showing how a model aligned solely on text
remains capable of generating harmful images.

TL;DR: We introduce modal aphasia, a surprising and systematic dissociation in which unified multimodal models demonstrate strong capabilities for generating visual content while simultaneously failing to access that same knowledge through text queries.