CoSeC-LCD: Controllable Self-Contrastive Latent Consistency Distillation for Better and Faster Human Animation Generation

Authors: Ling Haotian, Wenzhang SUN, Sen Liang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Consistency Distillation; Controllable and Consistent Generation

Abstract:
Generating pose-driven and reference-consistent human animation has significant practical applications, yet it remains a prominent research challenge, facing substantial obstacles. A major issue with widely adopted diffusion-based methods is their slow generation speed, which is primarily due to multi-step iterative denoising processes. To tackle this challenge, we take the pioneering step of proposing the ReferenceLCM architecture, which utilizes latent consistency models (LCM) to facilitate accelerated generation. Additionally, to address hallucinations in fine-grained control, we introduce the Controllable Self-Contrastive Latent Consistency Distillation (CoSeC-LCD) regularization method. Our approach introduces a novel perspective by categorizing tasks into various classes and employing contrastive learning to capture underlying patterns. Building on this insight, we implement a hierarchical optimization strategy that significantly enhances animation quality across both spatial and temporal aspects. Comprehensive qualitative and quantitative experiments reveal that our method achieves results comparable to, or even surpassing, many state-of-the-art approaches, enabling high-fidelity human animation generation within just 2-4 inference steps.

Decision: reject

Reviews and discussion:

Metareview: