Title: KB-DMGen: Knowlege-Based Global Guidance and Dynamic Pose Masking for Human Image Generation
Primary area: generative models
Keywords: Human Image Generation, knowledge base, dynamic pose masking

Abstract: Recent methods using diffusion models have made significant progress in Human Image Generation (HIG) with various control signals such as pose priors. In HIG, both accurate human poses and coherent visual quality are crucial for image generation. However, most existing methods mainly focus on pose accuracy while neglecting overall image quality, often improving pose alignment at the cost of image quality. To address this, we propose Knowledge-Based Global Guidance and Dynamic pose Masking for human image Generation (KB-DMGen). The Knowledge Base (KB), implemented as a visual codebook,  provides coarse, global guidance based on input text-related visual features, improving pose accuracy while maintaining image quality, while the Dynamic pose Masking (DM) offers fine-grained local control to enhance precise pose accuracy. By injecting KB and DM at different stages of the diffusion process, our framework enhances pose accuracy through both global and local control without compromising image quality. Experiments demonstrate the effectiveness of KB-DMGen, achieving new state-of-the-art results in terms of AP and CAP on the HumanArt dataset.  The project page and code will be available.

TL;DR: We introduce a knowledge-based global guidance diffusion and dynamic pose mask framework that generates high-quality human images conditioned on pose and text, achieving state-of-the-art results