Reading Isn't Believing: Typographic Attacks on Multimodal Neurons (CLIP)
Date: 2021-03
Affected: CLIP, DALL-E, downstream multimodal models
Attack vector: adversarial-input
Category: research
Severity: Medium

Goh et al. (OpenAI) demonstrate that placing a written label on an object causes CLIP to classify the image as the label (e.g., an apple labeled 'iPod' is classified as iPod). The first systematic typographic attack on multimodal neurons.


Mitigations:

Architectural layers (MAESTRO):