Towards Multiple Character Image Animation Through Enhancing Implicit Decoupling

## Abstract

Abstract. Pose-controllable character video generation is in high demand with extensive applications for fields such as automatic advertising and content creation on social media platforms.
While existing character image animation methods using pose sequences and reference images have shown promising performance, they tend to struggle with incoherent animation in complex scenarios, such as multiple character animation and body occlusion.
Additionally, current methods request large-scale high-quality videos with stable backgrounds and temporal consistency as training datasets, otherwise, their performance will greatly deteriorate.
These two issues hinder the practical utilization of character image animation tools.
In this paper, we propose a practical and robust framework Follow-Your-Pose v2, which can be trained on noisy open-sourced videos readily available on the internet. Multi-condition guiders are designed to address the challenges of background stability, body occlusion in multi-character generation, and consistency of character appearance.
Moreover, to fill the gap of fair evaluation of multi-character pose animation, we propose a new benchmark comprising approximately 4,000 frames.
Extensive experiments demonstrate that our approach outperforms state-of-the-art methods by a margin of over 35% across 2 datasets and on 7 metrics. Meanwhile, qualitative assessments reveal a significant improvement in the quality of generated video, particularly in scenarios involving complex backgrounds and body occlusion of multi-character, suggesting the superiority of our approach.

## 1. Introduction

Character image animation task targets animating a given static character image to a video clip using a sequence of motion signals, such as video, depth or pose, while preserving the visual appearance. It has attracted much attention and extensively explored in research (Hu et al., ; Xu et al., ; Wang et al., ; Zhang et al., ; Yoon et al., ; Yu et al., ). This technology has widespread applications in fields of the film industry, automatic advertising and content creation for social media platforms, etc.
As a data-driving task, cocurrent character image animation frameworks (Karras et al., ; Wang et al., ; Zhang et al., ) always optimize the model on high-quality video datasets, which request stable background and temporal consistency. The expensive collection of datasets restricts their utility in more common scenarios.

Existing character image animation can be divided into two categories based on the generative mechanism: GAN-based and diffusion-based frameworks. Previous methods (Siarohin et al., ; Wang et al., ) leverage a warping function to transfer the reference image into the target pose, and then GAN model is employed to generate the missing parts. In recent years, encouraged by the emergency of diffusion models (Song et al., ), the models use pose condition (Hu et al., ) and harness appearance (Xu et al., ) to generate natural video clips based on pretrained stable diffusion (Rombach et al., ).
Despite generating visually plausible video, these methods frameworks still face several challenges.
GAN-based methods fail to generate realistic character animation due to poor ability of motion transfer, such as hair, facial expressions, and other details.
It becomes more serious in cross-identity scenarios.
In contrast, diffusion-based methods alleviate this problem due to their powerful generative capability. However, they still synthesize incoherent animation in a frame-by-frame operation. These methods always have serious challenges in some complex pose sequences, such as multiple character pose sequences and body occluded pose sequences. Such approaches typically rely solely on pose as a condition, neglecting the benefits of considering various conditions, leading to flickering results.

In this work, to address the aforementioned limitations, we develop a general character image animation framework called Follow-Your-Pose v2, which employs multi-condition guided generation and optimizes our model on the large-scale noisy dataset from the internet.
Specifically, we design the optical flow guider to achieve better background stability, resulting in a significant improvement.
Secondly, to address the issue of occluded body parts in multiple character animation, we propose the use of the depth guider to assist the model in distinguishing between the foreground and background, thus maintaining the identification of different characters.
Thirdly, the reference pose guider is proposed for better appearance learning.
Finally, we collect a large number of video clips from the Internet for our training dataset. These videos encompass diverse elements such as backgrounds, clothing, races, and multiple character scenarios, all of which can enhance the generalization ability of the model. However, a notable drawback is that the majority of these videos exhibit camera shake or unstable backgrounds.
Thanks to the multiple condition guiders, our framework could generate temporally consistent and realistic detailed single and multiple character image animations using noisy video clips as our training set.
Additionally, to evaluate the performance of multi-pose sequence generation, we also propose a multi-pose benchmark including about 4000 frames, which empowers researchers to measure the generation ability in complex multiple character animations. To the best of our knowledge, we are the first to collect such a benchmark. We conduct plenty of quantitative and qualitative experiments to illustrate the superiority of our approach. Compared with cocurrent methods (Hu et al., ; Wang et al., ), our approach has achieved remarkable improvements in all 7 metrics across all experiments.

In summary, our contributions are four-fold:

- •
We propose a practical and multi-condition guided framework Follow-Your-Pose v2 to leverage large-scale noisy data for character image animation, which improves the performance significantly.
- •
To achieve efficient training using noisy data, technically, the optical flow guider ensures background stability, the depth guider addresses the occlusion among body parts, and reference pose guider enhances the learning of subject appearance.
- •
To address the lack of a benchmark in multiple character image animation, we introduce a new benchmark called Multi-Character Bench, which contains about 4,000 frames for fair evaluation.
- •
Extensive quantitative and qualitative evaluations are conducted using two public datasets and Multi-Character Bench. The experiments show the superiority of the proposed method.

## 2. Related Work

### 2.1. Diffusion Models for Video Generation

Compared with aforementioned GAN-based methods  (Mirza and Osindero, ; Siarohin et al., ; Zhao and Zhang, ), diffusion models (Song et al., ; Khachatryan et al., ; Guo et al., ; Chen et al., , ; Zhang et al., ; Wu et al., ; Molad et al., ; Esser et al., ; Guo et al., ; Ma et al., ; He et al., ; Ye et al., ; Zhang et al., ; Ma et al., , ) excel in generating high-quality contents and exhibit greater training efficiency. Diffusion models were first proposed to generate images (Ho et al., ; Song et al., ). Many studies adapt image synthesis pipelines for video generation (Khachatryan et al., ; Guo et al., ; Chen et al., , ; Wu et al., ). Text2Video-Zero (Khachatryan et al., ) incorporates motion dynamics in generated frames via cross-frame attention in zero-shot style. Animatediff (Guo et al., ) finetunes a plug-and-play motion module that can be integrated into any text-to-image models to obtain animations in a personalized style. VideoCrafter (Chen et al., , ) achieves both text-to-video and image-to-video generation and produces videos of high resolution. Besides text, I2V-Adapter (Guo et al., ) extracts semantics from images to condition T2V generation.

### 2.2. Pose-Controllable Character Animation

Generating realistic character video from the driving signal(e.g., key points, depth maps) has been extensively studied in recent years.
Early approaches employed GAN-based models for conditional video synthesis (Mirza and Osindero, ; Siarohin et al., ; Wang et al., ). Vid2vid (Wang et al., ) utilizes conditional GAN to produce diverse pose videos by incorporating optical flow, temporal consistency constraints, and multiple discriminators. Similar work (Chan et al., ) is proposed to transfer movements from a source person to a target person through keypoints. MRAA (Siarohin et al., ) extends this line of research by incorporating consistent regions that describe locations, shapes, and poses. To address the pose gap between objects in the source and driving images, TPSMM (Zhao and Zhang, ) effectively aligns object poses using thin-plate spline motion estimation, multi-resolution occlusion masks, and carefully designed loss functions.

With the development of diffusion model (Ho et al., ), existing methods incorporate pose sequences to guide the generation of the character video clips (Zhang et al., ; Ma et al., ; Karras et al., ; Wang et al., ; Chang et al., ; Xu et al., , ; Hu et al., ). DreamPose (Karras et al., ) proposes a finetuning framework to adapt the Stable Diffusion model (Rombach et al., ) to a pose-and-image guided video synthesis model.
AnimateAnyone (Hu et al., ) designs ReferenceNet to merge detailed human body features, ensuring character appearance consistency, while employing a pose guider and temporal modeling for controllable and continuous movements.
To enhance the faithfulness, generalizability, and compositionality of character animation, DisCo (Wang et al., ) disentangles the control conditions (i.e., character foreground, background, and poses) by introducing multiple ControlNets for different feature embeddings and proposes a human attribute pre-training framework.
Moreover, a two-stage training scheme (Ma et al., ) is proposed to address the lack of comprehensive paired video-pose datasets, leveraging available image pose pairs, pose-free videos, and a pre-trained text-to-image model to generate pose-controllable character videos. However, both GAN-based and diffusion-based methods fail to effectively address the issue of background stability and occlusion among body parts.

Figure: Figure 2. The overview of the proposed Follow-Your-Pose v2. The left half illustrates the data flow of the multiple condition guiders, with green and black arrows denoting training data flow, and red and black arrows indicating inference data flow. The right half shows the denoising U-Net and ReferenceNet.
Refer to caption: x2.png

Figure: Figure 3. The result of the model trained on the background unstable dataset. The red box highlights the phenomenon where the background of the generated image deviates from that of the inference image.
Refer to caption: x3.png

## 3. Preliminaries

Latent Diffusion Models.
The latent diffusion models (Rombach et al., ) (LDM) performs a denoising process within the latent space rather than the image pixel space (Ho et al., ; Song et al., ), cutting computational load while maintaining quality. Specifically, the encoder of variational autoencoder (Van Den Oord et al., ; Kingma and Welling, ; Razavi et al., ) (VAE) compresses the pixel space image $x$ to a latent space feature $z=\mathcal{E}(x)$, while the decoder reconstructs the generated latent space feature $z^{\prime}$ back into image $x^{\prime}=\mathcal{D}(z^{\prime})$. The denoising process is an iterative Markov process that progressively denoises the initial Gaussian noise $z_{T}\in\mathcal{N}(0,I)$ into the target latent space features $z_{0}$. The single-step iterative process involves predicting the noise of the forward process and subtracting it. LDM uses U-Net (Ronneberger et al., ) or Transformer (Vaswani et al., ; Peebles and Xie, ) to estimate the noise, with the loss function of

$$ (1) $\mathcal{L}=\mathbb{E}_{\mathcal{E}(x),c,\epsilon\sim\mathcal{N}(0,I),t}\left[ ||\epsilon-\epsilon_{\theta}(z^{t},t,c)||_{2}^{2}\right]\,,$ $$

where $\epsilon_{\theta}(\cdot)$ is the network for predicting noise. $c$ denotes the conditional information. $t$ represents the timestep of denoising process and $z^{t}$ represents the intermediate result of denoising at timestep $t$.

Temporal Motion Modules.
Video can be regarded as a sequence of moving images in the temporal dimension. Based on the powerful image generation capability of the text-to-image (T2I) model, the addition of temporal motion modules endows it with the ability to generate video (Guo et al., ), i.e., video latent diffusion model (VLDM). Specifically, inflating the T2I model enables it to generate video data, and insert a temporal motion module after spatial attention. The temporal motion modules run across temporal frames in each batch to enhance motion smoothness and content consistency in animation clips.
Besides, the initial input tensor $z\in\mathbb{R}^{c\times h\times w}$
of the T2I model should add a temporal dimension and repeat it $n$ times, becoming $z_{1:n}\in\mathbb{R}^{n\times c\times h\times w}$. The VLDM’s loss function is

$$ (2) $\mathcal{L}=\mathbb{E}_{\mathcal{E}(v_{1:n}),c,\epsilon_{1:n}\sim\mathcal{N}(0 ,I),t}\left[||\epsilon-\epsilon_{\theta}(z^{t}_{1:n},t,c)||_{2}^{2}\right]\,,$ $$

where $v_{1:n},\epsilon_{1:n},z^{t}_{1:n}$ respectively represent the corresponding variables after adding dimensions in Eq. (1).

Figure: Figure 4. The pipeline of calculating the skeletal dilation map and optical flow map. The optical flow map is obtained by superimposing the skeletal dilation map onto the optical flow vector map.
Refer to caption: x4.png

## 4. Method

Given a reference image $I_{0}$ and the pose sequences $\{P_{0},P_{1},P_{2},...,P_{N}\}$, the primary objective is to generate character video clip not only exhibits plausible motion but also faithfully preserves the visual appearance (foreground and background) of the reference image. By leveraging large-scale high-quality dataset training, recent methods have shown promising generation capacity. However, the collection of videos with stable backgrounds is costly and laborious. Meanwhile, existing methods offer limited performance over the multi-character generation and occluded body parts. In view of these limitations, we propose Follow-Your-Pose v2 (FYPv2), a novel framework equipped the multi-condition guiders, as shown in Fig. 2. The optical flow guider, detailed in Sec. 4.1, focuses on the stability control of the background. The depth guider, described in Sec. 4.2, tackles the occlusion of pose sequences in multi-character animation. We also present the details about the designed reference pose guider in Sec. 4.3. Finally, the base model architecture and objective loss are introduced in Sec. 4.4.

Figure: Figure 5. Bad cases of multi-character animation. The red boxes indicate the body occlusion regions of multiple characters that the model fails to generate accurately.
Refer to caption: x5.png

Figure: Figure 6. The pipeline of calculating the depth map. The yellow module “Skeletal Dilation” is shown in Fig. 4 (a).
Refer to caption: x6.png

### 4.1. Optical flow guider

Directly optimizing the model on a noisy dataset results in background instability, which may be attributed to training data with an unstable background. As shown in Fig. 3, we present the generated video frames during the training stage.
It is clear that the model tends to generate dynamic backgrounds similar to the ground truth rather than the reference image. This is a common issue when training on noisy datasets.
To deal with this problem, we utilize the guidance of optical flow to achieve stable background generation. Fig. 4 presents the pipeline of calculating the optical flow map in the training stage (Yang et al., ; Güler et al., ; Sun et al., ; Contributors, ). The optical flows from each frame $\{x_{1},x_{2}...,x_{N}\}$ are predicted by open-source optical flow estimator $\mathcal{E}_{{flow}}$ (Contributors, ). The dilation operation is then applied on pose skeletons to obtain binary mask sequences $\{\mathcal{M}_{1},\mathcal{M}_{2},\mathcal{M}_{3},...,\mathcal{M}_{N}\}$, which are used for definition of the control region. We blend optical flows with the binary mask and encode them using 8 layer inflated 3D convolution. Following the animate anyone (Hu et al., ), the representations are added into clear initial noise latent. In inference stage, the region in blended optical flows is set to 1.0. Formally, the optical flow guider is implemented as

$$ (3) $\begin{gathered}\mathcal{M}_{i}=\mathcal{D}\left(P_{i}\right),\\ c_{{flow}}^{i}=(1-\mathcal{M}_{i})\odot\mathcal{E}_{{flow}}\left(x_{i}\right) \end{gathered}$ $$

where $i=1,2,3,\ldots,N$, $\mathcal{D}$ is the dilation operation, $\mathcal{E}_{{flow}}$ represents the optical flow estimator, $\odot$ represents Hadamard product.

Figure: Figure 7. The potential impact of inputting reference image pose for model generation.
Refer to caption: x7.png

**Table 1. Quantitative comparison on Tiktok Dataset. The best and second-best results are indicated in red and blue respectively.**
| Method | FID↓ | SSIM↑ | PSNR↑ | LPIPS↓ | L1↓ | FID-VID↓ | FVD↓ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MRAA(Siarohin et al., ) | 54.47 | 0.672 | 29.39 | 0.296 | 3.21E-04 | 66.36 | 284.82 |
| TPSMM(Zhao and Zhang, ) | 53.78 | 0.673 | 29.18 | 0.299 | 3.23E-04 | 72.55 | 306.17 |
| DreamPose(Karras et al., ) | 79.46 | 0.509 | 28.04 | 0.450 | 6.91E-04 | 80.51 | 551.56 |
| DisCo(Wang et al., ) | 51.29 | 0.699 | 28.70 | 0.333 | 1.10E-04 | 61.41 | 379.56 |
| DisCo+(Wang et al., ) | 48.29 | 0.713 | 28.78 | 0.320 | 1.03E-04 | 52.56 | 334.67 |
| MagicAnimate(Xu et al., ) | 32.09 | 0.714 | 29.16 | 0.239 | 3.13E-04 | 21.75 | 179.07 |
| MagicPose(Chang et al., ) | 25.50 | 0.752 | 29.53 | 0.292 | 0.81E-04 | 46.30 | 216.01 |
| AnimateAnyone(Hu et al., ) | - | 0.718 | 29.56 | 0.285 | - | - | 171.90 |
| Ours | 27.70 | 0.760 | 29.70 | 0.272 | 0.73E-04 | 14.30 | 117.81 |

**Table 2. Quantitative comparison on TED-talks Dataset. The best and the second-best results are indicated in red and blue respectively. MRAA* and TPSMM* present these methods utilize ground-truth videos as driving signals, while gray metrics represent that they do not participate in quantitative comparisons.**
| Method | FID↓ | SSIM↑ | PSNR↑ | LPIPS↓ | L1↓ | FID-VID↓ | FVD↓ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MRAA*(Siarohin et al., ) | 25.76 | 0.807 | 32.62 | 0.216 | 0.36E-04 | 12.20 | 126.28 |
| TPSMM*(Zhao and Zhang, ) | 24.69 | 0.819 | 32.92 | 0.205 | 0.35E-04 | 9.04 | 87.76 |
| MRAA(Siarohin et al., ) | 50.36 | 0.762 | 31.90 | 0.266 | 0.50E-04 | 82.79 | 493.02 |
| TPSMM(Zhao and Zhang, ) | 23.71 | 0.771 | 32.30 | 0.252 | 0.49E-04 | 32.12 | 260.67 |
| DisCo(Wang et al., ) | 75.48 | 0.575 | 27.99 | 0.309 | 1.21E-04 | 66.18 | 393.04 |
| DisCo+(Wang et al., ) | 63.28 | 0.596 | 28.12 | 0.300 | 1.11E-04 | 55.81 | 343.20 |
| MagicAnimate(Xu et al., ) | 41.58 | 0.529 | 28.28 | 0.310 | 1.73E-04 | 33.61 | 223.54 |
| MagicPose(Chang et al., ) | 23.39 | 0.723 | 30.08 | 0.236 | 0.81E-04 | 27.53 | 214.23 |
| Ours | 18.21 | 0.779 | 30.88 | 0.198 | 0.46E-04 | 10.24 | 81.73 |

### 4.2. Depth guider

Cocurrent methods struggle to process scenarios with body occlusion. Insufficient understanding of spatial relations renders these methods incapable of accurately generating occluded body parts, as illustrated by the red boxes in Fig. 5.
The reason is that the absence of spatial information within the pose sequence results in various body occlusions mapping to the same pose frame, rendering the model incapable of learning the correct spatial relationships between foreground and background.

To solve this problem, we develop incorporating the depth map as a condition within the proposed framework (Yang et al., ). The aim is to leverage the spatial information in the depth maps to assist the model in learning spatial relationships between characters.
Fig. 6 illustrates the pipeline for incorporating the depth maps into the proposed framework, which can be divided into three processes.
The first process is indicated in the first row of Fig. 6. For the current frame, we extract the pose of each character individually, and then apply “Skeletal Dilation” (See Fig. 4 (a)) to each of them, resulting in Instance-wise Skeleton Dilation Maps, i.e., Fig. 6 (c).
In the second process, we first merge the (c) to take the union, and then subtract the union to get the Non-union Area. Subsequently, we split it by characters, resulting in the Instance-wise Non-union Area Maps, which represent the non-union region of each character, i.e., Fig. 6 (f). Lastly, we extract Depth Maps of the current frame, i.e., Fig. 6 (g).
In the third process, we first split (g) by characters, and then align them with (f). Consequently, we can compute the average depth of non-union region for each character, and then sort them by the average depth to obtain their position ranking. We set a value list to assign values to the region of each character based on its position ranking. For example, as shown in Fig. 6 (i), we assign “yellow value” to the frontmost character region and “red value” to the second front character region. Finally, we obtain the “character depth map”,i.e., Fig. 6 (j), containing the spatial relational information we desired.
Given a training video $v$, where the $i_{th}$ frame is denoted as $v_{i}$, and suppose that $J$ characters on $v_{i}$. These processes can be illustrated as

$$ (4) $a_{i,1},...a_{i,J}=f_{\rm e}\left(f_{\rm s}\left(v_{i}\right)\right),$ $$

$$ (5) $m_{i,j}=a_{i,j}-\left(1-\bigcup_{j\in\{1,\dots,J\}}\left(a_{i,j}\right)\right) ,{j\in\left\{1..J\right\}},$ $$

$$ (6) $rank_{j}=f_{\rm sort}\left(m_{i,1}\odot f_{\rm d}\left(v_{1}\right),...,m_{i,j }\odot f_{\rm d}\left(v_{i}\right)\right),$ $$

$$ (7) $c_{{\rm depth},j}=m_{i,j}\odot L_{rank_{j}}+\left(1-m_{i,j}\odot c_{{\rm depth },\left(j-1\right)}\right),c_{{\rm depth},0}=0,$ $$

$$ (8) $c_{{\rm depth}}=g_{\rm dp}\left(c_{{\rm depth},J}\right),$ $$

where $f_{{\rm d}}$ represents the depth extraction network, $a_{i,j}$ is the instance-wise skeleton dilation map, i.e., Fig. 6 (c). $\cup$ represents union operation and $m_{i,j}$ is the non-union region of each character, i.e., Fig. 6 (f). $f_{\rm sort}$ represents average depth sorting (descending order), $rank_{j}$ is the depth ranking of character $j$, and $L_{rank_{j}}$ denotes the value assigned to $j$ based on depth ranking. $c_{{\rm depth},j}$ represents the depth map of the farthest $j$ characters. $g_{\rm dp}$ represents the depth guider.

Figure: Figure 8. Qualitative comparisons between baselines and our approach on dataset Tiktok (Top three rows) and TED-talks (bottom two rows). MRAA* and TPSMM* present these methods utilizing ground-truth videos as driving signals.
Refer to caption: x8.png

### 4.3. Reference pose guider

The positions of characters in inference images and pose sequences often exhibit inconsistencies. As indicated by the red arrows in Fig. 7, the model employs spatial cross-attention to facilitate the interaction between the pose frame and reference image, enabling it to learn character texture information from the reference images and accurately map them to the corresponding pose positions. Therefore, the model is required to perceive the entirety of the reference image through spatial cross-attention in order to capture the position and texture information of the characters. This presents a significant learning challenge for the model.

We realize that this process involves the model implicitly learning character poses in the reference image and aligning them with the pose frame. Consequently, we can address this issue by introducing the reference pose as a prior. The green arrows in Fig. 6 show that the reference pose serves as a prior, to assist the model in accurately locating the character on the reference image. In this way, the model can efficiently accomplish mapping by focusing on the area specified by the reference pose within the spatial cross-attention. Besides, since the reference pose map and the pose frame are consistent in format, the model can directly align the pose and map the characters efficiently. Given the reference image $x$, the $c_{{\rm ref_pose}}$ can be formulated as

$$ (9) $c_{{\rm ref_pose}}=g_{\rm rp}\left(f_{{\rm s}}\left(x\right)\right),$ $$

where $f_{{\rm s}}$ represents the skeleton extraction network and $g_{\rm rp}$ denotes the reference pose guider.

### 4.4. Base Model Architecture

As is shown in Fig. 2.
FYPv2 contains some modules that are widely used in the pose-guided animation community.
Specifically, the reference image is compressed using VAE and then fed into referencenet (Cao et al., ). It interacts with the features in the U-net model through spatial cross-attention. Meanwhile, the CLIP-encoded reference image is used as the prompt to replace the text prompt in the U-net model. The pose sequence and the aforementioned three conditions are incorporated into the initial noise using the pose guider modules, which are composed of convolutional layers. FYPv2 is formulated as

$$ (10) $c_{\rm multi}=c_{\rm pose_seq}+c_{\rm flow}+c_{\rm depth}+c_{\rm ref_pose}\,,$ $$

$$ (11) $\mathcal{L}=\mathbb{E}_{\mathcal{E}(v_{1:n}),c,\epsilon_{1:n}\sim\mathcal{N}(0 ,I),t}\left[||\epsilon-\epsilon_{\theta}(z^{t}_{1:n},t,x,c_{\rm multi})||_{2}^ {2}\right]\,,$ $$

where $c_{\rm multi}$ denotes the overall control condition.
The definitions of $v_{1:n}$, $\epsilon_{1:n}$, and $z^{t}_{1:n}$ are analogous to Eq. (2).

## 5. Experiment

### 5.1. Implementations

Dataset.
The robustness of our model to noisy data enables direct training on large unfiltered videos. We collect 4000 character-action videos totaling 2000000 frames as our training set. The data come from public videos on TikTok, YouTube, and other websites. The dataset comprises a diverse range of characters and backgrounds, which facilitates the training of a highly generalizable model.

Training strategy.
Our model is trained in two stages. In the first stage, we freeze the weights of VAE (Van Den Oord et al., ) and CLIP (Radford et al., ) image encoder, and remove the temporal motion modules. U-Net, ReferenceNet, and multiple-condition guiders are trained to align their spatial generative capacities. In addition, we utilize the weights of Stable Diffusion v1.5 (Rombach et al., ) to initialize this training stage.
In the second stage, we incorporated temporal motion modules into training along with all parameters from Stage 1, aiming to endow the model with the ability of temporal smoothness. Besides, we use the weights of AnimateDiff v2 (Guo et al., ) to initialize this training stage.

Implementation Details.
We sample 16 frames of video, resize and center-crop them to a uniform resolution of 896x640 pixels.
Experiments are conducted on 8 NVIDIA A800 GPUs. Both stages are optimized with Adam with a learning rate set to $1\times 10^{-5}$. In the first stage, we train our model for 60k steps with a batch size of 4, and in the second stage, we conduct training for 60k steps with a batch size of 1.
At inference, we apply DDIM (Song et al., ) sampler for 50 denoising steps, with classifier-free guidance (Ho and Salimans, ) scale of 1.5.
More details can be found in the appendix.

**Table 3. Quantitative comparison on Multi-Character Bench. The best and the second-best results are indicated in red and blue respectively.**
| Method | FID↓ | SSIM↑ | PSNR↑ | FID-VID↓ | FVD↓ |
| --- | --- | --- | --- | --- | --- |
| DisCo(Wang et al., ) | 77.61 | 0.793 | 29.65 | 104.57 | 1367.47 |
| DisCo+(Wang et al., ) | 73.21 | 0.799 | 29.66 | 92.26 | 1303.08 |
| MagicAnime(Xu et al., ) | 40.02 | 0.819 | 29.01 | 19.42 | 223.82 |
| MagicPose(Chang et al., ) | 31.06 | 0.806 | 31.81 | 30.95 | 312.65 |
| Ours | 26.95 | 0.830 | 31.86 | 14.56 | 142.76 |

### 5.2. Comparisons

Dataset and metrics.
Following the previous methods (Wang et al., ), we evaluate our method on
TikTok-style videos (Wang et al., ) and TED-talks (Siarohin et al., ).
Additionally, due to the lack of benchmarks for multiple-character video generation, we collect 20 multiple-character dancing videos, totaling 3917 frames, from social media, named Multi-Character.
This dataset serves as a benchmark for evaluating models’ capabilities in generating pose-controllable videos with multiple characters.
Our evaluation metrics adhere to the existing research literature (Wang et al., ; Xu et al., ; Chang et al., ; Hu et al., ). Specifically, we employ conventional image metrics to assess the quality of individual frames, namely L1 error, PSNR (Hore and Ziou, ), SSIM (Wang et al., ), LPIPS (Zhang et al., ), and FID (Heusel et al., ). For the video evaluation metric FID-VID (Balaji et al., ) and FVD (Unterthiner et al., ), we form a sample by concatenating each consecutive 16 frames.

Figure: Figure 9. Qualitative comparisons between baselines and our approach on the Multi-Character bench.
Refer to caption: x9.png

**Table 4. Quantitative ablation results on TikTok Dataset.**
| Method | FID↓ | SSIM↑ | PSNR↑ | FID-VID↓ | FVD↓ |
| --- | --- | --- | --- | --- | --- |
| w/o Optical Flow | 52.94 | 0.710 | 28.21 | 34.35 | 178.48 |
| w/o Ref. Pose | 34.74 | 0.740 | 29.13 | 19.58 | 139.32 |
| w/o Depth | 27.43 | 0.754 | 29.98 | 14.50 | 119.26 |
| Ours | 27.70 | 0.760 | 29.70 | 14.30 | 117.81 |

Baselines.
We conduct a comprehensive comparison of our model with several state-of-the-art methods for character animation: (1) MRAA (Siarohin et al., ) and TPSMM (Zhao and Zhang, ) are state-of-the-art GAN-based animation methods. (2) DreamPose (Karras et al., ), Disco (Wang et al., ), MagicAnimate (Xu et al., ), MagicPose (Chang et al., ) and AnimateAnyone (Hu et al., ) are state-of-the-art VLDM-based animation methods. We notice that not all approaches adhere to a uniform generation size. As methods with different generation sizes yield different metrics results, potentially leading to unfair comparisons, we standardize the generation sizes by center-cropping and resizing to 512$\times$512. Under this unified standard, we reevaluate methods that do not conform to this generation size and directly reference relevant statistical data from the original literature of methods that do comply.
Note that MRAA (Siarohin et al., ) and TPSMM (Zhao and Zhang, ) utilize ground-truth videos as driving signals but not pose sequences. For fair comparisons, we evaluate two versions using two different driving signals: Visualization of pose sequence and ground-truth videos. As shown in Table 2, MRAA* and TPSMM* represent the evaluation metrics using ground-truth videos as driving signals, while MRAA and TPSMM represent the evaluation metrics using visualization of pose sequence as driving signals.

Figure: Figure 10. Qualitative comparison results of ablation variants without optical flow condition. Transparent and overlay the images to clearly see the changes in the background.
Refer to caption: x10.png

Evaluation on TikTok dataset.
Table 1 presents the quantitative comparison experimental results of the comparative methods on the TikTok dataset. Our model outperforms others in state-of-the-art performance on four metrics: SSIM, L1, FID-VID, and FVD, while ranking as the second best on the FID and LPIPS metrics. Notably, our model excels particularly in the realm of video metrics: FID-VID and FVD. Compared with MagicAnimate and AnimateAnyone, the second-best method in FID-VID and FVD, our model showcases a significant improvement of 39% and 35% respectively.

The visual comparison results of the TikTok dataset are shown in the first three rows of Fig. 8. Our approach obtains better performance in pose following and visual quality. The results of the first and third rows showcase the excellent visual quality of our method, which not only accurately follows the pose but also generates hand details that are absent in the ground truth. The second row showcases the powerful pose-following capability of our method, which stands as the sole approach capable of accurately generating the pose with the arm raised in reverse. These results confirm that our approach exhibits superior performance in generating pose-controllable, achieved through training with a large dataset to enhance generalization abilities.

Evaluation on TED-talks dataset.
The quantitative comparison results of the TED-talks dataset are depicted in Table 2. Our model achieves sota performance across six evaluation metrics except PSNR, and also demonstrates outstanding performance in the realm of video metrics. In contrast to MagicPose, the second-best of FID-VID and FVD, our model demonstrates a substantial enhancement of 49% and 48% respectively. Note that MRAA* and TPSMM* utilize ground-truth videos as driving signals, thus evaluating perfect yet unfair metrics. Nevertheless, our model manages to surpass or approach them in terms of FID, lpips, FID-VID, and FVD.

The last two rows in Fig. 8 illustrate the visual comparison results of the TED-talks dataset. Our approach achieves superior results in pose following and character consistency. In the fourth row, only our results successfully maintain the consistency of the character who holding white notes in the inference image, and in the last row, our results exhibit the best performance in the pose following, while also generating the fewest artifacts.

Evaluation on Multi-Character bench.
The quantitative comparison results in Table 3 show that our method significantly outperforms other methods across all metrics on Multi-Character bench. In terms of the video metrics FID-VID and FVD, we outperform the second-best method by 25% and 36% in each metric, respectively.

Fig. 9 illustrates the qualitative comparisons between baselines and our approach, highlighting the significant advantages of our approach in maintaining multi-character identity consistency, pose following, and preserving spatial relationships.
The top row demonstrates that our approach maintains multi-character identity consistency, whereas other methods fail to do so. As shown in the red boxes on the second and third rows, our approach achieves the most accurate pose following. In the last rows, our approach generates accurate multi-character spatial relationships without occlusion between characters like other methods.

Figure: Figure 11. Qualitative comparison results of ablation variants without depth condition.
Refer to caption: x11.png

### 5.3. Ablation Study

To investigate the roles of the proposed conditions, we examine three variants, each without Optical Flow, Depth, and Reference Pose, respectively.
Table 4 presents that the proposed full method (”Ours”) with three conditions outperforms other variants in terms of the evaluation metrics. This demonstrates that the three conditions have a positive impact on the model.

Optical Flow.
Table 4 presents that the variant without optical flow exhibits the most significant decline, suggesting that optical flow might have the most pronounced positive impact on the model.
The comparison results shown in Fig. 10 illustrate where its primary effects are: background stability. In Fig. 10, we overlay the transparent consecutive 6 frames to form the image on the right, enabling a clear depiction of the moving region of this video segment. As indicated by the red boxes, we can observe that the background in the results of the variant without optical flow shows noticeable shaking, whereas the background in the results of the full method with optical flow remains consistently stable. This validates our viewpoint: noisy data with unstable backgrounds leads to the generation of unstable backgrounds by the model, whereas the incorporation of the optical flow condition can solve this problem.

Depth.
We conduct a qualitative comparison of multi-character animation in Fig. 11 to validate the positive impact of depth condition. The red boxes highlight the overlapping areas of multiple characters. The variant without depth condition fails to generate the hand of the character on the left placed behind the right character, instead generating a malformed hand. Conversely, the full method with the depth condition can generate overlapping areas of multiple characters. As illustrated by the red boxes in Fig. 11, the hand of the left character is positioned behind the right one, and the hand of the right character is in front of the face of the left one.

Figure: Figure 12. Qualitative comparison results of ablation variants without reference pose condition.
Refer to caption: x12.png

Reference Pose.
Fig. 12 illustrates the qualitative comparison results when the character position of the inference image and pose sequence are not aligned. The comparison between the variant without reference pose and the full method illustrates that the full method performs better in terms of image quality and character consistency. In the results of the variant, consistency in the background is maintained, yet the clothing of the characters continuously changes, failing to maintain character consistency. In contrast, the full method incorporating the reference pose effectively achieves alignment between character and pose sequence, thereby maintaining character and background consistency.

## 6. Conclusion

In this paper, we propose a pose-controllable character animation framework called Follow-Your-Pose v2 that integrates multiple conditions into a VLDM, dedicated to addressing the issues of background stability and body occlusion of multiple characters.
This framework enhances the robustness of the model to noisy data. Within this framework, we design three guiders: optical flow guider, depth guider, and reference pose guider. These guiders effectively address issues of background stability and body occlusion, enhancing the learning of subject appearance. Moreover, we have curated and released a benchmark dataset of pose-controllable videos with multiple characters.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. I am curious about scenarios where two individuals have significant interactions.

2. I think that the results in Figure 2 should reference the specific previous methods used for these issues.
soundness: 2
strengths: 1. This paper proposes to integrate three additional guiders, optical flow, depth order, and reference pose map to improve the animation performance under complex backgrounds and multi-character situations (mostly two persons). 

2. The animation results are impressive and better than other methods, which demonstrate the efficiency of the proposed solutions.
summary: In this work, the authors propose solutions for challenges in image animation, such as instability under complex backgrounds and multi-person scenarios. To enhance quality in these situations, three additional guides are introduced: optical flow, depth order, and reference pose map. The network design is similar to previous methods which include ReferenceNet, denoising U-Net, and inserting a temporal module in the second stage. Experiments demonstrate that the proposed methods outperform existing approaches, but more experimental results should be provided.
weaknesses: 1. The claim that the depth order map provides positional information for multiple characters is somewhat confusing. In practice, you first use the dilate-skeleton map to determine the positions of individuals, and subsequently obtain the depth order using depth information. 

2. The results in the paper only involve two individuals. To support the claim of handling multiple characters, it would be beneficial to include examples with more than two characters.

3. The pose sequences for two persons in a video are all the same, I think more results in different pose sequences for different characters should be provided.

4. The network design follows the previous methods, like MagicAnimate, which includes ReferenceNet, denoising U-Net, and inserting a temporal module in the second stage. I think Figure 3 can be improved to better demonstrate the network architecture.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: I am overall positive about the submission. It would be nice if the authors can discuss a bit more about the following questions, in addition to the weaknesses raised above:
- What is the inference cost? Would it be able to run on commodity-level GPUs (e.g., Nvidia 2060, 3070, etc). I may have missed it, but it seems like only A100 is mentioned, and that is for training.
- How does the ```Ref Pose Guider``` handle multi-character skeleton maps. Do we assume there is a fixed number of character in the video?
soundness: 3
strengths: The core strength of the method lies in the attention to the details. Specifically, 
- By explicitly considering the depth order, the method disambiguates the spatial arrangement in multi-character settings, and therefore enables the model to learn better, and render with higher fidelity.
- By providing the pose for the reference image, the model can better decouple the texture and pose information (although it is still implicit). Despite being simple, this design appears to have a great positive impact on the generation quality (in Figure 10).
- The background optical flow mitigates the spurious correlations that hinder the model from focusing on character animation.

Overall, the presented method is sound, and the effectiveness of each proposed component is sufficiently verified through their experiments.
summary: The paper proposes a new image-based pose retargeting/character animation diffusion model. The key contributions lie in how the proposed method handles (1) noisy backgrounds, which may degrade the quality due to spurious correlations, (2) explicitly capture the depth-order to enable more accurate generation for multi-character retargeting, and (3) enhance decoupling of appearances and body pose representation to achieve better texture quality.

The experimental results show that the proposed method achieves higher perceptual quality, both quantitatively and qualitatively.
weaknesses: - The proposed method is computationally heavy (line 417-420, needs 8 A100 GPUs for training), making it less accessible to possible end users. This is, however, a common issue among diffusion-based methods, not particular to the paper.
- The depth-ambiguity is not completely resolved. One can see in the supplementary video 00:44, the top-left character has their hand rendered to the back incorrectly.
- The method focuses on front-facing poses. It is unclear how it performs on extreme poses with 360-degree rotation, and also poses like handstand, etc.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Can the authors provide animation results for more challenging scenarios, such as dense crowd animations or high occlusion ratios?
    
- In Table 1, the model trained on a noisy dataset outperforms the AnimateAnyone baseline trained on a clean TikTok dataset. Is this improvement due to the dataset or the proposed modules? If the improvement stems from the dataset, could the authors provide a more detailed analysis of dataset quality (e.g., pose diversity, data scale, character domain diversity)? If the improvement is due to the proposed modules, the authors should train their model solely on the TikTok dataset to directly compare with the baseline methods.
    
- In the inference stage, can the optical flow guider take a specific optical flow map as input to achieve results with camera rotation effects?
soundness: 3
strengths: - The paper identifies a key challenge in training human animation models on in-the-wild data—the instability of background motion—and proposes a pipeline to address this issue.
    
- An optical flow-based conditioning module is introduced, enabling human animation methods to train on data with background instability, which helps scale the approach to noisy, real-world datasets.
    
- The paper presents a heuristic depth-order map calculation pipeline, providing simple yet effective spatial information for multi-character generation tasks.
summary: Existing human animation methods struggle with multi-character animation tasks and are difficult to train on data with unstable, noisy backgrounds. This paper addresses these challenges with two main motivations: to decouple background motion from static elements and to separate multiple characters. To this end, the authors propose a novel framework that includes an optical flow guider and a depth-order guider.

The optical flow guider encodes background motion, while the depth-order guider captures the spatial relationships among multiple characters. Additionally, a reference pose guider is introduced to encode the pose of a reference image, simplifying the textual remapping process.

Evaluations on multi character benchmark demonstrate that this method generates high-quality character animations when trained on noisy web-collected datasets.
weaknesses: - Although the authors claim the method can handle multiple characters, most of the results feature only one or two characters.
    
- While the paper includes some analysis of background motion in the training dataset, it lacks in-depth analysis related to multiple characters, such as the effect of character count or occlusion ratios.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: In addition to the weakness, can the authors clarify how the depth order map of the target pose sequences is acquired during inference? More specifically, how the depth order is determined given only the skeletons of multi-character poses?
soundness: 3
strengths: * The proposed methods are well-motivated. The authors demonstrate the effectiveness of the proposed control signals by comprehensive ablation studies.
* The skeletal dilation map is novel to me as an alternative to the segmentation mask. The authors explain the choice and its advantage over the segmentation mask, which is sound.
* It shows better quality on multi-character animation compared to baselines.
summary: The paper tackles the problem of multiple-character image animation and addresses the failures in current approaches by introducing additional control signals: 1) To alleviate the influence of non-static background, it adds the optical flows as conditions; 2) To avoid identity mixup in multi-character animation, it adds depth order guidance that labels the identity id within the skeletal dilation map. It achieves comparable performance on single-character animation benchmarks and better quality on multi-character animation.
weaknesses: * The head poses are limited in all the qualitative results presented in the paper, including the figures and the supplementary videos. For example, in Fig. 1, the head poses of all characters are the same as the reference images while other parts of the body are retargeted into new poses. It makes it difficult to assess the animation quality without qualitative results showcasing more diverse head poses. This is my main concern since faces are probably the most difficult and important part of the animation. Comparing facial regions with the baselines will help the judgment.

* As mentioned in the limitations, the use of the skeletal dilation map restricts the animation of clothes that cover a significantly larger area than the skeletal dilation map. Is the kernel size of dilation a crucial factor for the method? How does it impact the animation of clothes?

Meta-review: The paper tackles challenges in character image animation with a multi-condition framework using optical flow, depth order, and reference pose guiders to handle complex backgrounds and multiple characters. It outperforms existing methods in quality and introduces a new benchmark for standardized evaluation.

Strengths:
Innovative Design: Introduces optical flow, depth order, and reference pose guiders to improve animation in complex and multi-character scenes.
High Quality: Outperforms baselines with better results in multi-character and noisy background scenarios.
Enhanced Accuracy: Depth-order and reference pose maps improve spatial precision and texture-pose decoupling.

Weaknesses:
Depth and Multi-Character Limitations: The depth order map’s claim of providing positional information is unclear, and the method's evaluation is limited to two characters, lacking examples with more diverse multi-character scenarios or varying pose sequences.
Interaction and Pose Diversity: The method focuses on front-facing poses with limited results on interactions or extreme poses (e.g., 360-degree rotation, handstands), and facial regions are underrepresented, leaving questions about animation quality in diverse scenarios.
Clothing Animation and Skeletal Dilation Map: The skeletal dilation map restricts animation of loose clothing, with insufficient analysis of how dilation kernel size impacts performance.

We have decided to (weakly) accept the paper based on the mixed reviews, but we encourage the authors to improve both the manuscript and the supplementary video. Additionally, it is noted that there is significant room for improvement in the quality of the video results.

Decision: Accept (Poster)