FLIRT: Feedback Loop In-context Red Teaming
Authors: name: Ninareh Mehrabi, name: Palash Goyal, name: Christophe Dupuy, name: Qian Hu, name: Shalini Ghosh, name: Richard Zemel, name: Kai-Wei Chang, name: Aram Galstyan, name: Rahul Gupta

Warning: this paper contains content that may be inappropriate or offensive.
  As generative models become available for public use in various applications,
testing and analyzing vulnerabilities of these models has become a priority.
Here we propose an automatic red teaming framework that evaluates a given model
and exposes its vulnerabilities against unsafe and inappropriate content
generation. Our framework uses in-context learning in a feedback loop to red
team models and trigger them into unsafe content generation. We propose
different in-context attack strategies to automatically learn effective and
diverse adversarial prompts for text-to-image models. Our experiments
demonstrate that compared to baseline approaches, our proposed strategy is
significantly more effective in exposing vulnerabilities in Stable Diffusion
(SD) model, even when the latter is enhanced with safety features. Furthermore,
we demonstrate that the proposed framework is effective for red teaming
text-to-text models, resulting in significantly higher toxic response
generation rate compared to previously reported numbers.