Title: Energy-Guided Prompt Optimization for Controllable Cross-Architectural Diffusion Models
Primary area: generative models
Keywords: Diffusion-based Generative Models, Controllable Image Synthesis, Energy-Guided Prompt Optimization, Cross-Architectural Attribution, Semantic Consistency Metrics

Abstract: Diffusion-based generative methods have fundamentally reshaped the process of converting textual descriptions into visual representations. However, consistently enforcing semantic constraints across a wide range of model architectures remains a significant challenge in practice. In this work, we present a unified framework that integrates diagnostic analysis and optimization strategies to enhance controllability within diffusion systems that span multiple architectural designs. The proposed framework incorporates latent space attribution for comparative architectural analysis, a differential formulation that isolates model specific behavioral dynamics, and an energy-guided optimization approach that adjusts the geometry of the representation space to promote constraint satisfaction, particularly in scenarios involving exclusion and negation.

By combining interpretability techniques with principles of constrained sampling, our method enables rigorous evaluation of image fidelity, stylistic characteristics, and adherence to user-defined constraints under varied prompting conditions. Experimental results, supported by perceptual metrics and human evaluations, demonstrate that modifying latent energy distributions leads to more reliable suppression of undesired attributes than conventional guidance heuristics. Furthermore, the diagnostic tools reveal persistent sensitivities tied to architectural design, offering actionable insights for targeted model refinement. This methodology advances the field of controllable image synthesis by integrating theoretical optimization principles, model introspection techniques, and practical constraint enforcement strategies applicable to both creative and industrial domains.

TL;DR: We propose an energy-guided prompt optimization framework that enhances controllability and constraint adherence across diverse diffusion model architectures.