Durian: Dual Reference-guided Portrait Animation with Attribute Transfer
Authors: name: Hyunsoo Cha, name: Byungjun Kim, name: Hanbyul Joo

We present Durian, the first method for generating portrait animation videos
with facial attribute transfer from a given reference image to a target
portrait in a zero-shot manner. To enable high-fidelity and spatially
consistent attribute transfer across frames, we introduce dual reference
networks that inject spatial features from both the portrait and attribute
images into the denoising process of a diffusion model. We train the model
using a self-reconstruction formulation, where two frames are sampled from the
same portrait video: one is treated as the attribute reference and the other as
the target portrait, and the remaining frames are reconstructed conditioned on
these inputs and their corresponding masks. To support the transfer of
attributes with varying spatial extent, we propose a mask expansion strategy
using keypoint-conditioned image generation for training. In addition, we
further augment the attribute and portrait images with spatial and
appearance-level transformations to improve robustness to positional
misalignment between them. These strategies allow the model to effectively
generalize across diverse attributes and in-the-wild reference combinations,
despite being trained without explicit triplet supervision. Durian achieves
state-of-the-art performance on portrait animation with attribute transfer, and
notably, its dual reference design enables multi-attribute composition in a
single generation pass without additional training.