VLM-Guided Adaptive Negative Prompting for Creative Generation
Authors: name: Shelly Golan, name: Yotam Nitzan, name: Zongze Wu, name: Or Patashnik

Creative generation is the synthesis of new, surprising, and valuable samples
that reflect user intent yet cannot be envisioned in advance. This task aims to
extend human imagination, enabling the discovery of visual concepts that exist
in the unexplored spaces between familiar domains. While text-to-image
diffusion models excel at rendering photorealistic scenes that faithfully match
user prompts, they still struggle to generate genuinely novel content. Existing
approaches to enhance generative creativity either rely on interpolation of
image features, which restricts exploration to predefined categories, or
require time-intensive procedures such as embedding optimization or model
fine-tuning. We propose VLM-Guided Adaptive Negative-Prompting, a
training-free, inference-time method that promotes creative image generation
while preserving the validity of the generated object. Our approach utilizes a
vision-language model (VLM) that analyzes intermediate outputs of the
generation process and adaptively steers it away from conventional visual
concepts, encouraging the emergence of novel and surprising outputs. We
evaluate creativity through both novelty and validity, using statistical
metrics in the CLIP embedding space. Through extensive experiments, we show
consistent gains in creative novelty with negligible computational overhead.
Moreover, unlike existing methods that primarily generate single objects, our
approach extends to complex scenarios, such as generating coherent sets of
creative objects and preserving creativity within elaborate compositional
prompts. Our method integrates seamlessly into existing diffusion pipelines,
offering a practical route to producing creative outputs that venture beyond
the constraints of textual descriptions.