Multimodal Conditioning for Controllable Image and Video Generation
Creators: name: Cheong, Soon Yau
Publisher: name: University of Surrey
Year: 2025
Subjects: Computer Science - Artificial Intelligence
Computer Science - Computer Vision and Pattern Recognition

description: The field of generative AI has progressed at a rapid pace and can now produce high-quality images and
description: videos from text prompts. This evolution has also led to greater user demand for precise control over the
description: outcomes, posing new challenges in effectively directing generation processes. Standard conditioning
description: techniques, mainly using text and image inputs, have proven useful but remain limited in handling
description: more complex requirements, such as specific human pose, camera orientation or fine-grained visual
description: appearance. This PhD research enhances conditioning techniques by introducing a parametric approach
description: that emphasises multimodal conditioning for image and video generation models. It focuses on
description: developing methods to enable more comprehensive user control, incorporating various modalities such
description: as pose and spatial inputs to improve alignment between model capabilities and user intentions across
description: different aspects of generation. By refining the conditioning mechanisms, this research aims to bridge the
description: gap between user specifications and model outputs, ensuring greater flexibility, precision, and coherence
description: in generated content.
description: This thesis presents several key contributions. Traditional methods for human pose conditioning,
description: which rely on skeleton images, contain substantial redundancy and are computationally inefficient for
description: modern architectures. To address this, we proposed the concept of pose token, where raw pose parameters
description: are compressed into tokens that can be used as conditioning elements via attention mechanisms, a common
description: approach in advanced architectures. We validated this token-based approach with both 2D body keypoints
description: and 3D body parameters, demonstrating its effectiveness across multiple architectures, from transformers
description: to diffusion models. Additionally, our parametric approach introduces groundbreaking techniques for
description: human and camera pose interpolation within image generation.
description: A common approach for conditioning diffusion models involves incorporating adapters - lightweight
description: models to deliver control signals to pre-trained image models. However, our research has revealed that
description: this method often introduces a critical issue of mode conflict. This problem, worsened by cascading
description: multiple adapters, results from an imbalance in control signals: the model can become dominated by one
description: adapter, limiting the generative power of both the base model and other adapters. Despite its prevalence,
description: this issue remains largely unaddressed in existing research. To solve this, we devised a unified adapter
description: architecture that integrates both structural and visual conditioning within a single, harmonised control
description: pathway. This unified approach delivers balanced multimodal conditioning, avoiding the pitfalls of
description: adapter cascade and enabling greater model flexibility. As a result, our approach’s high controllability
description: empowers versatile human image generation and editing tasks.
description: Our research in 2D image generation was extended to video generation. Our study demonstrated that
description: the architectural differences in transformer-based diffusion models make existing camera control methods
description: for U-Net-based diffusion models ineffective. Through extensive experimentation, optimal architectures
description: and camera representations were identified. Combined with our novel camera motion guidance, camera
description: control was restored for video diffusion transformers, with motion boosted by over 400%. Our research
description: on human pose conditioning for images extends to video generation. Unlike existing methods that require
description: detailed camera pose input for every frame, our approach achieves smooth video motion with minimal
description: input. By specifying only the initial and final camera poses, our system interpolates between frames to
description: produce continuous camera movements, enabling consistent, controlled video generation with reduced
description: data requirements. This sparse video conditioning approach significantly simplifies the user interaction
description: while ensuring fluid transitions and stable pose dynamics across frames, pushing the boundaries of
description: efficient and user-friendly video generation.
description: Many of the challenges we aimed to address were novel, often lacking established evaluation methods.
description: As a result, we proposed new evaluation metrics to rigorously assess these areas. One of these, People
description: Count Error (PCE), identifies a unique type of error specific to AI-generated human images, such as
description: inaccurate body part generation. This metric has already gained traction in the research community and
description: is being adopted in image generation benchmarks, helping to set new standards for evaluating AI-driven
description: human image quality.