Negative Token Merging: Image-based Adversarial Feature Guidance
Authors: name: Jaskirat Singh, name: Lindsey Li, name: Weijia Shi, name: Ranjay Krishna, name: Yejin Choi, name: Pang Wei Koh, name: Michael F. Cohen, name: Stephen Gould, name: Liang Zheng, name: Luke Zettlemoyer

Text-based adversarial guidance using a negative prompt has emerged as a
widely adopted approach to push the output features away from undesired
concepts. While useful, performing adversarial guidance using text alone can be
insufficient to capture complex visual concepts and avoid undesired visual
elements like copyrighted characters. In this paper, for the first time we
explore an alternate modality in this direction by performing adversarial
guidance directly using visual features from a reference image or other images
in a batch. In particular, we introduce negative token merging (NegToMe), a
simple but effective training-free approach which performs adversarial guidance
by selectively pushing apart matching semantic features (between reference and
output generation) during the reverse diffusion process. When used w.r.t. other
images in the same batch, we observe that NegToMe significantly increases
output diversity (racial, gender, visual) without sacrificing output image
quality. Similarly, when used w.r.t. a reference copyrighted asset, NegToMe
helps reduce visual similarity with copyrighted content by 34.57%. NegToMe is
simple to implement using just few-lines of code, uses only marginally higher
(<4%) inference times and generalizes to different diffusion architectures like
Flux, which do not natively support the use of a separate negative prompt. Code
is available at https://negtome.github.io