Title: VividFace: A Robost and High-Fidelity Video Face Swapping Framework
Venue: NeurIPS 2025
Accepted: yes
Keywords: video generation

face swapping

diffusion
Abstract: Video face swapping has seen increasing adoption in diverse applications, yet existing methods primarily trained on static images struggle to address temporal consistency and complex real-world scenarios.  To overcome these limitations, we propose the first video face swapping framework, VividFace, a robust and high-fidelity diffusion-based framework. VividFace employs a novel hybrid training strategy that leverages abundant static image data alongside temporal video sequences, enabling it to effectively model temporal coherence and identity consistency in videos.
Central to our approach is a carefully designed diffusion model integrated with a specialized VAE, capable of processing image-video hybrid data efficiently. To further enhance identity and pose disentanglement, we introduce and release the Attribute-Identity Disentanglement Triplet (AIDT) dataset, comprising a large-scale collection of triplets where each set contains three face images—two sharing the same pose and two sharing the same identity. Augmented comprehensively with occlusion scenarios, AIDT significantly boosts the robustness of VividFace against occlusions.
Moreover, we incorporate advanced 3D reconstruction techniques as conditioning inputs to address significant pose variations effectively. Extensive experiments demonstrate that VividFace achieves state-of-the-art performance in identity preservation, temporal consistency, and visual realism, surpassing existing methods while requiring fewer inference steps. Our framework notably mitigates common challenges such as temporal flickering, identity loss, and sensitivity to occlusions and pose variations.
The AIDT dataset, source code, and pre-trained weights will be released to support future research. The code and pretrained weights are available on the [project page](https://hao-shao.com/projects/vividface.html).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VividFace, a diffusion-based framework for high-fidelity video face swapping, leveraging both static images and video sequences for training. Key contributions include:\n1. A triplet dataset (same ID/different pose, same pose/different ID) to improve identity-pose disentanglement.\n2. An augmentation strategy for occlusions, and 3D reconstruction encoding for conditional input.\n3. Comprehensive evaluations on temporal smoothness, occlusion robustness, pose variation, and identity preservation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed framework are intuitive and straghtforward.\n- The results are natural-looking with improved temporal smoothness over prior work. The proposed method handles occlusions better than baseline methods.\n\n\nWeaknesses\n- The conditioning input is redundant. The motion frame is encoded three times, and the masked source image overlaps with the face region input. This may introduce inefficiency without clear justification.\n- Figure 2 is confusing. The blue block is ambiguously labeled (sometimes the temporal module, sometimes latent representation z_0). Missing tensor dimensions for latent representations (blue, yellow, gray blocks), making it harder to reproduce.\n- The methods works well when source/target faces are similar (e.g., Figure 1, all people have high checkbones) but struggles with large identity gaps (e.g., Figure 6). Results in Groups 1–6 either: Use overly similar identities, or Fail to preserve identity when faces differ significantly.\n- Fair Comparison Concerns: Comparisons with SimSwap/others use different examples. Testing on identical pairs would strengthen validity.\n\nQUESTIONS:\nHave you experimented with simplifying the conditioning pipeline (e.g., reducing motion frame encodings)? Does redundancy improve results, or is it an architectural constraint?\n\nIs the identity preservation issue in Figure 6/Group 1–6 due to dataset bias (e.g., AIDT lacks diverse pairs) or a fundamental trade-off in the framework?\n\nCould you include comparisons on the same source/target pairs as SimSwap or other methods to ensure fair evaluation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a diffusion-based video face-swapping framework that addresses temporal consistency and identity preservation. They design a VidFaceVAE, which consists of (2+1)D blocks and is trained on both static images and videos to improve temporal coherence. The authors also propose AIDT, a new dataset designed to disentangle identity and pose, which enhances robustness to occlusions. With additional 3D conditioning for pose variation, VividFace achieves state-of-the-art results in identity retention and visual quality, with fewer inference steps.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Propose to train their video face swapping network with both image and video data. And propose a new dataset, AIDT.\n2. Introducing 3DMM into the face swapping process, which guarantees the attributes (eg, pose, expression) to be unchanged.\n3. Propose AIDT, a new dataset designed to disentangle identity and pose, which enhances their method’s robustness to occlusions.\n\nWeakness: \n1. The authors claim they are the first video face-swapping network. However, several video face-swapping methods have been proposed recently, e.g., DynamicFace and HiFiVFS.\n2. Unsatisfying results. The generated videos still contain several artifacts, \\eg, large opening mouth, hair on the forehead.\n3. The comparison cases are not sufficient. Most of the test cases are man-man or woman-woman swapping. Including more cases under different genders might be better to show the superiority of the proposed method.\n4. Although 3DMM can effectively preserve the motions of the target video, it suffers from incorrect estimations under large headposes. Also, estimation 3dMM is quite slow.\n\nQUESTIONS:\nQuestions:\n1. Can you generate videos with arbitrary length while maintaining temporal consistency?\n2. What is the inference speed of this method?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VividFace, a diffusion-based framework for high-fidelity video face swapping. The authors identify key challenges in existing methods, namely the lack of temporal consistency, poor identity preservation, and vulnerability to real-world complexities like occlusions and large pose variations. To address these, VividFace proposes several techniques. The paper demonstrates through extensive experiments that VividFace achieves state-of-the-art performance on both video and image face swapping tasks, outperforming existing GAN-based and diffusion-based methods in terms of visual quality, identity preservation, and temporal consistency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper tackles a very relevant and challenging problem in generative AI: high-fidelity video face swapping. The release of the code, models, and the novel AIDT dataset is a major contribution that will undoubtedly facilitate future research in this area.\n\n2. The technical quality of the work is excellent. The proposed framework is well-designed, and each component's inclusion is well-motivated. The experimental evaluation is comprehensive and rigorous. The authors not only compare against a strong set of recent baselines on a new, challenging benchmark (VidSwapBench) but also conduct a thorough set of ablation studies that convincingly validate their design choices. The qualitative results shown in the paper are very impressive.\n\nWeaknesses:\n1. While the paper's focus is on generation quality, a practical limitation of diffusion models is their inference speed. The paper acknowledges this in the limitations section but does not provide a quantitative comparison of inference time against the baselines in the main experimental tables. For a video task, metrics like frames-per-second are highly relevant and would provide a more complete picture of the trade-offs involved.\n\n2. I notice that the eyes of the generated face are not the same as the target face, and the eye contact is not as vivid as human beings. This is a challenging problem. But humans are very sensitive to this.\n\nQUESTIONS:\nPlease refer to the weaknesses part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper focuses on robust video face swapping by adapting diffusion model with image-video hybrid training strategy. To improve the robustness of proposed method, a large-scale AIDT dataset is constructed and comprehensive occlusion augmentation is adopted. The writing is good in general. However, the overall framework, training strategy and other parts of Sec 3 may need proofreading for better readability. The experimental results show its effectiveness of proposed method.

STRENGTHS AND WEAKNESSES:
Strengthens:
(1)It is the first diffusion-based video face swapping framework;
(2)It constructs a large-scale AIDT dataset to significantly improve face feature identity-expression disentanglement;

Weaknesses:
(1)Diffusion-based method limits the practical applicability;
(2)It is better to separate the training and inference stages of Fig 2;
(3)For face encoder (Line 200-207), the 'source' and 'target' are different from the corresponding part of Fig 2.

QUESTIONS:
（1）Although robustness enhancement is adopted, why the performance in pose and expression only comparable to SOTA methods?
  (2)  Is the encoder-decoder is firstly trained on video data and secondly training on image data? Then random selection of previous video and image data is conducted for hybrid training?

LIMITATIONS:
(1) The performance improvement between baseline and static training is limited, which weakens the novelty of proposed method.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03