Title: Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimedia Forensics

Talking Head Videos

Talking Head Videoconferencing

Puppeteering Attacks
Abstract: AI-based talking-head videoconferencing systems reduce bandwidth by transmitting a latent representation of a speaker’s pose and expression, which is used to synthesize frames on the receiver's end. However, these systems are vulnerable to “puppeteering” attacks, where an adversary controls the identity of another person in real-time. Traditional deepfake detectors fail here, as all video content is synthetic. We propose a novel biometric defense that detects identity leakage in the transmitted latent representation. Our metric-learning approach disentangles identity cues from pose and expression, enabling detection of unauthorized swaps. Experiments across multiple talking-head models show that our method consistently outperforms prior defenses, operates in real time on consumer GPUs, and generalizes well to out-of-distribution data. By targeting the latent features shared during normal operation, our method offers a practical and robust safeguard against puppeteering.

Unmasking Puppeteers: Leveraging Biometric Leakage to Disarm Impersonation in AI-based Videoconferencing

Abstract

AI-based talking-head videoconferencing systems reduce bandwidth by sending a compact pose-expression latent and re-synthesizing RGB at the receiver-but this latent can be 'puppeteered,' letting an attacker hijack a victim's likeness in real time. Because every frame is synthetic, deepfake and synthetic video detectors fail outright. To address this security problem, we exploit a key observation: the pose expression latent inherently contain biometric information of the driving identity. Therefore, we introduce the first biometric leakage defense without ever looking at the reconstructed RGB video: a pose-conditioned, large-margin contrastive encoder that isolates persistent identity cues inside the transmitted latent while cancelling transient pose and expression. A simple cosine test on this disentangled embedding flags illicit identity swaps as the video is rendered. Our experiments on multiple talking-head generation models show that our method consistently outperforms existing puppeteering defenses, operates in real-time, and shows strong generalization to out-of-distribution scenarios.Our code and trained models are available at https://github.com/MISLresearch/Unmasking-Puppeteers-Neurips25.

1 Introduction

Advances in generative AI have enabled the creation of hyper-realistic synthetic videos. This has led to the development of many new technologies, including both avatar-based communication systems [1-3] and AI-based videoconferencing systems [4-7]. AI-based talking head videoconferencing systems are receiving increasing attention due to their significant bandwidth savings [8]. Instead of continuously encoding and transmitting each video frame of a speaker, these systems only transmit embeddings that capture a speaker's pose and facial expression. Then, a generative AI system at the receiver's end uses these embeddings, in conjunction with an initial representation of the speaker, to create an accurately reconstructed video.

Unfortunately, AI-based talking head videoconferencing systems [5, 4, 8, 7, 9] are vulnerable to a new form of information attack known as 'puppeteering' [10]. In this attack, a malicious user at the sender side transmits an unauthorized representation of a different target speaker when a video call is initiated. As a result, the identity in the video that the receiver constructs is different than the identity of the person who actually controls the video [10, 11].

While many forensic approaches, such as deepfake [12-17] and synthetic video detectors [18-21], aim to expose unauthorized AI-generated media, they operate by identifying evidence that a video has been synthetically generated. However, in AI-based talking-head videoconferencing, every video is AI-generated , rendering existing media forensic approaches unable to detect puppeteering.

,

AI-Based Talking Head Videoconference

f

g

f

f

Biometric

Leakage

Enhancement

Puppeteering attacks pose a fundamentally different threat from those typically addressed by media forensics. Instead of asking 'Is this video real or AI-generated?' one key question must be 'Is this AI-generated video being driven by an authorized identity, or has someone hijacked it?' As talkinghead videoconferencing technology continues to advance, it is essential to mitigate puppeteering attacks before these systems are widely deployed. Addressing these threats proactively will help ensure secure and trustworthy communication over these systems.

Few defenses address puppeteering. Prashnani et al. introduced Avatar Fingerprinting [11], which works by enrolling each user's characteristic facial motion patterns into the detector's training data. Although this could help, it requires significant user-specific data to build reliable motion signatures. In contrast, Vahdati et al. avoid enrollment by decoding and re-encoding the received video for comparison [10]; the round-trip is compute-heavy and adds distortions that hurt accuracy.

In this paper we introduce a real-time, enrollment-free defense that detects puppeteering attacks entirely in the latent domain. Our key insight is that the pose-and-expression embeddings already transmitted by modern talking-head systems leak subtle, but reproducible, biometric signatures [22, 23]. We learn a compact Enhanced Biometric-Leakage (EBL) space in which identity cues are amplified while pose and expression variance is actively suppressed. A pose-conditioned contrastive loss drives this separation, and a lightweight temporal LSTM aggregates evidence to yield stable, millisecond-level decisions. By comparing the sender's latent identity with the target identity rendered at the receiver, our system flags any mismatch without decoding RGB frames, tracking landmarks, or collecting per-user motion profiles. Extensive experiments across many generators and datasets show that our method achieves state-of-the-art detection performance while working in real-time.

In summary, our contributions are as follows.

We present the first method that operates solely on data already available at the receiver, requiring no user enrollment or additional sensors.

We formalize and harness identity leakage inherent in pose-expression embeddings, turning a previously ignored vulnerability into a defensive signal.

We develop a novel loss and training protocol to learn a low-dimensional EBL representation that maximizes identity separability while nullifying pose and expression.

NVIDIA-VC

RAVDESS

CREMA-D

Reference

Self-Reenacted

Cross-Reenacted

Reference

Self-Reenacted

Cross-Reenacted

Reference

Self-Reenacted

Cross-Reenacted

Figure 2: Illustration of three datasets (NVIDIA-VC [11], RAVDESS [54], CREMA-D [55]) shown across three columns each. Row 2 indicates the type of frame displayed in Row 3: Reference, Self-Reenacted, or Cross-Reenacted.

Though extensive experiments across fifteen generator/dataset combinations, our approach achieves state-of-the-art puppeteering detection in real time, demonstrating practical viability for deployment in bandwidth-constrained videoconferencing systems.

2 Background and Related Work

Talking-Head Video Systems. Talking-head systems synthesize facial motion and speech in real time by transmitting low-dimensional embeddings of pose and expression rather than full video frames [24-27]. A generator at the receiver reconstructs a realistic face. Methods range from compressed embeddings to landmark-based updates per frame [8, 28-33], reducing bandwidth while maintaining high visual fidelity [34-40]. Systems like NVIDIA's Vid2Vid Cameo [8] and Google's Project Starline [41] demonstrate real-time, 3D teleconferencing.

Forensics for Synthetic Media. Synthetic media forensics traditionally aims to distinguish real from AI-generated content. Deepfake detectors [12-17, 42] leverage semantic cues [43-46], lighting inconsistencies [47-50], and facial texture artifacts [51-53], while synthetic video detectors [18-21] focus on identifying AI-generated traces. However, in low-bandwidth talking-head systems, every video is synthetic [10, 11], rendering these detectors ineffective - they cannot determine whether the video represents the correct identity or has been hijacked.

Puppeteering Attacks and Existing Defenses. Puppeteering attacks replace the target's appearance with the adversary's pose-and-expression vectors at call initiation, causing the receiver to synthesize a video of the wrong identity. Avatar Fingerprinting [11] mitigates this threat by first enrolling each user's facial-motion signatures; at inference it flags a session whenever the incoming motion deviates from the stored profile. Vahdati et al. [10] avoid enrollment by re-encoding the synthesized video and comparing embeddings, but their pipeline depends on facial-landmark estimation, which degrades under large head rotations and lighting artifacts introduced by video reconstruction.

In contrast, our method exploits biometric leakage already present in the pose-and-expression latents, removing both the enrollment phase and any reliance on facial landmarks. A contrastive objective disentangles identity from pose, yielding a compact, latency-friendly representation that generalizes to unseen generators and real-world conferencing conditions.

3 Problem Formulation

3.1 Talking Head Video Systems

AI-enabled talking-head videoconferencing systems have been proposed to minimize transmitted data by encoding and transmitting only pose and expression information from a speaker rather than an entire video frame [8, 56-62]. As this technology rapidly matures, our goal here is to develop security measures that improve the trustworthiness of these systems when they are deployed.

A videoconferencing call starts with each participant k sending a neutral reference portrait R k to the receiver. Typically, R k is an image of the speaker in a neutral pose. As the speaker continues to speak, each new video frame V k t capturing this speaker at the sender side is processed by an embedding function f to produce an embedding z k t = f ( V k t ) that encodes the speaker's instantaneous pose and expression at time t . This embedding is transmitted to the receiver, as is shown in Fig. 1.

At the receiver side, a generator g is used to produce a video of the speaker on the basis of the received embedding z k t and the reference representation R k such that

Here, the superscript ˆ V k → k t denotes that speaker k both drives and appears in the rendered video. We refer to the sender's identity as the 'driving identity' and the reference portrait sent to the receiver as the 'target identity' [11]. While such a talking-head video-conferencing system promises to be bandwidth-efficient, it exposes a vulnerability to puppeteering attacks when a malicious sender impersonates another individual.

3.2 Puppeteering Attacks

A puppeteering attack exploits the inherent trust in the reference representation transmitted to the receiver when a videoconferencing call begins. Here, an adversary, speaker ℓ , obtains a target speaker's representation R k and substitutes it for their own without authorization at the start of the call [10]. As the call proceeds, the adversary's own video V ℓ t is used to derive the pose and expression vectors z ℓ t that are transmitted to the receiver. The receiver then uses the generator g alongside the unauthorized reference R k and z ℓ t to reconstruct the video. As a result, the receiver is presented with a realistic-looking video of speaker k , controlled in real time by the speaker ℓ such that

where ˆ V k → ℓ t is an unauthorized impersonation of speaker ℓ driven by speaker k . To address this threat, one must come up with mechanisms to detect whether the receiver's generated video is driven by the same authorized identity, rather than an unauthorized impersonator.

3.3 Why Real-vs-Synthetic Detectors Fail

Deepfake and synthetic-video detectors flag frames whose pixel statistics reveal AI generation, implicitly assuming that 'real' camera footage is the baseline. In bandwidth-efficient talking-head conferencing, however, every frame - even those from honest participants - is produced by a generator; 'synthetic' is the norm, not the anomaly. The security question therefore shifts from 'Was this video AI-generated?' to 'Does the rendered face correspond to the person actually driving the latents?' Pixel-level detectors cannot answer that mapping, leaving puppeteering attacks undetectable.

4 Proposed Approach

In this paper, we present a real-time solution for detecting puppeteering attacks in talking-head videoconferencing, uniquely operating entirely in the latent domain without access to reconstructed RGB frames. To do this, we re-encode each pose-and-expression latent embedding into a compact Enhanced Biometric Leakage (EBL) embedding space that captures who is speaking while discarding how they move. A pose-conditioned contrastive loss learns this space by pulling together identityconsistent pairs and pushing apart identity-mismatched, pose-matched pairs. At run time, we simply compare each live EBL vector to the EBL embedding of the reference portrait sent at call setup; a sharp drop in similarity reveals that the driving and target identities have diverged, signalling puppeteering. For additional robustness and stable authentication, during training, we discard unreliable extreme-pose frames, and leverage an LSTM to fuse successive EBL scores to output stable, low-latency decisions. These innovations yield an algorithm that authenticates speakers in real-time, requires no enrollment or facial-landmark preprocessing. The following sections detail our approach and its implementation.

4.1 Biometric Entanglements in Pose & Expression Latent Space

Low-bandwidth talking-head systems encode each frame V t as a pose-and-expression vector z t = f ( V t ) , which the receiver's generator turns back into pixels. Although designed for geometry , z t inherit identity cues from the physical face that produced the motion. Measurements such as inter-ocular distance, jaw curvature, or lip thickness inevitably contaminate z t because head pose and facial musculature cannot be sensed independently of the speaker's anatomy. Prior studies to disentangle identity from pose and expression also reported persistent identity traces in poseconditioned representations [22, 23], which provides the empirical basis for our proposed defense.

Let R denote the static reference portrait of the nominal speaker transmitted at session start. Both endpoints apply the shared encoder f to compute the reference embedding f ( R ) , placing R and the

Pose&ExpressionEmbeddingSpace

BiometricLeakageSpace

DifferentID,

DifferentID,

Same P&E

Same P&E

Same ID,

Same ID,

Different P&E

Different P&E

DifferentID,

Different ID,

Different P&E

Different P&E

Φ

中

p

p

Cosinesimilarity(Φ)betweenembeddings

Cosinesimilarity(Φ)betweenembeddings

SameIdentity

DifferentIdentity

DifferentIdentity

DifferentPose&Expression

DifferentPose&Expression

SamePose&Expression

live stream { z t } in one latent space. A natural baseline defense is to compute the cosine similarity s c ( z t , f ( R )) and accept the incoming frame if the score exceeds a threshold. Unfortunately, pose and expression variability overwhelms the subtle biometric signal.

We demonstrate this effect in Fig. 3, which shows the cosine similarity distributions for three conditions: (1) same identity, differing pose/expression; (2) different identities, matched pose/expression; (3) different identities, differing pose/expression. We observe that the density for condition (2) frequently eclipses condition (1), which means that two different people exhibiting an identical yaw-pitch often appear closer in latent space than two frames of the same person smiling versus neutral. Raw pose-and-expression space is therefore pose-dominated : distance reflects 'how the head moves' more than 'whose head it is.'

This observation is key to our puppeteering defense: amplify latent biometric cues while suppressing pose and expression variance so that proximity encodes identity. Simply collecting more negatives or tweaking the similarity threshold cannot meet this requirement; the representation itself must be re-shaped. In the next section, we introduce the Enhanced Biometric Leakage (EBL) space, learned directly from the latent channel, that achieves this goal without RGB reconstruction, user enrollment, or landmark tracking, enabling a real-time authentication pipeline.

4.2 Enhanced Biometric Leakage (EBL) Space

Our key observation above dictates two design imperatives: (1) craft a representation in which distances reflect identity rather than pose/expression, and (2) keep that representation compact to work in real-time. We meet both goals by re-encoding the raw pose-and-expression latent z t into a low dimensional Enhanced Biometric Leakage vector and learning a contrastive objective that amplifies identity information while actively cancelling pose.

4.2.1 Latent re-encoding

The raw pose-expression vector z t is dominated by geometric factors, such as yaw, pitch, mouth aperture, while biometric cues hide in low-variance directions. To amplify these cues we attach two lightweight projection heads, h 1 and h 2 , that remap both the live embedding and the static reference f ( R ) into a shared, compact metric space.

Separate heads are essential because the two inputs follow different distributions: z t drifts frame by frame with micro-expressions, whereas f ( R ) is fixed at call setup. Allowing independent normalization lets each head adapt to its own domain before they meet in the common space.

Each head consists of a linear layer, ReLU activation, layer normalization, and a second linear layer followed by ℓ 2 normalization, such that

where s c ( · , · ) denotes cosine similarity.

4.2.2 Pose-conditioned contrastive objective

The cornerstone of our method is a pose-matched contrastive loss that aligns gradient pressure with the nuisance we aim to discard. Specifically, we pursue two complementary goals inspired by large-margin hyper-spherical embedding theory [63, 64]: (1) maximize the similarity of embeddings that share the same identity even when pose/expression varies, and (2) minimize the similarity of embeddings drawn from different identities having the same pose and expression. The resulting objective comprises one term for each goal.

Positive term. Any pair that agrees in identity, regardless of pose, is treated as positive:

where k indexes the speaker and p denotes the instantaneous pose-expression state. Minimizing (4) pulls together all manifestations of speaker k .

Negative term. Hard negatives are constructed by replicating pose but swapping identity . Concretely, we synthesise R ℓ,p , a portrait of impostor ℓ = k rendered at the identical pose p , and impose

̸

Because every negative matches pose, the gradient direction is orthogonal to pose variation, forcing the network to discriminate on biometric cues alone.

Total loss. The pose-conditioned Large-Margin Cosine Loss (PC-LMCL) is

with a single hyper-parameter λ controlling repulsion strength. With λ ≤ 1 , L B is (1 + λ ) -Lipschitz ( ≤ 2 ), placing them within the margin-risk framework of Lei et al. [65].

Margin guarantee. We formalize the geometric effect of (6) as follows. We note that here, we treat each speaker's front-facing reference portrait R k as their pose-averaged center || R k -µ k || 2 ≤ 0 . 5 °.

Proposition 1 Assume all embeddings are ℓ 2 -normalized. If, for some ϵ, γ > 0 ,

̸

then the class centers µ k = E p [ R k,p ] satisfy

̸

Thus L B enforces an inter-class angular margin of at least ϵ + γ within each pose slice.

Intuition. Driving L P → 0 pins positives to the 'north pole', while L N → 0 pushes pose-matched negatives toward the antipode; the spherical triangle inequality then yields Prop. 1. The full proof appears in Appendix G.

Consequently, minimizing L B maximizes the angular decision margin between identity manifolds while cancelling pose - an effect known to tighten generalization bounds in hyper-spherical embedding spaces [63, 66]. Prior methods ignore latent pose alignment and instead rely on RGB or landmarks, incurring latency and robustness penalties. By operating purely in the latent domain, PC-LMCL yields compact representations suitable for real-time deployment.

4.3 Training Protocol

The contrastive objective of the previous section is optimized on mini-batches organized as episodes. Each episode selects an anchor clip from speaker k , assembles identity-consistent positives drawn from other poses/expressions of k , and generates pose-matched negatives by rendering the same motion with impostor identities ℓ = k . Because the shared encoder f remains frozen, only the two projection heads learn, allowing rapid re-training should a codec revision alter its latent format.

̸

̸

Extreme Pose Exclusion. Before each update we discard frames whose head-pose estimate is unreliable. For instance, some poses may deviate substantially from the reference image, or, the speaker's face can be substantially occluded. This extreme-pose exclusion keeps the gradient focused on informative biometric variation.

To accomplish this, we estimate face normals n R and n t and prune any frame where their cosine similarity s c ( n t , n R ) exceeds a threshold τ , effectively removing views with large yaw, occlusion, or visibility loss. This keeps the gradient focused on meaningful biometric variation.

4.4 Temporal Fusion

While each individual biometric similarity score provides an instantaneous assessment of the mismatch between the driving speaker and the target identity, per-frame measurements can be noisy or inconclusive under challenging poses or brief occlusions. To address this, we aggregate similarity scores over a window of W consecutive frames and feed these into an LSTM. The LSTM learns to capture temporal patterns indicative of sustained mismatches characteristic of puppeteering attacks.

Let ϕ = { ϕ 1 , ϕ 2 , . . . , ϕ W } be the sequence of similarity scores collected over W frames. We feed s into the LSTM, which outputs a final score y representing the probability that the sequence is puppeteered. We train the LSTM by minimizing the binary cross-entropy loss between y and a ground-truth label t , where t = 1 if the video segment is puppeteered and t = 0 otherwise, such that

This learnable aggregator is novel in the context of puppeteering defense: previous methods either majority-vote over heuristic landmarks or revisit pixel space, both of which are fragile under compression artifacts. As the evaluation will show, the combination of EBL embedding and temporal fusion sets a new state of the art in puppeteering detection without sacrificing latency.

5 Experiments and Results

5.1 Experimental Setup

Datasets. We conduct our experiments using the NVFAIR [11] pooled dataset because it incorporates together a large set recorded video-conference calls in a controlled setting environment. This dataset includes three subsets: (1) NVIDIA VC [11] (natural video calls), (2) CREMA-D [55] (studiorecorded expressions), and (3) RAVDESS [54] (studio-recorded emotional speech). For each subset, we generate both self-reenacted (same identity) and cross-reenacted (puppeteered) videos using five state-of-the-art methods-MCNet [55], 3DFaceShop [67], SDFR [70], EmoPortraits [69], and LivePortrait [71], resulting in total of 15 dataset-method combinations. We note that the identities used for evaluation are strictly disjoint from those in training. Further details are provided in Tab. 3.

Metrics. We report the average detection AUC to allow direct comparisons with prior work.

Competing Methods. As puppeteering detection is a new problem, we compare against the state-ofthe-art method by Vahdati et al. [10]. To highlight the distinction from deepfake detection, we also evaluate seven leading deepfake detectors: Efficient ViT [12], CCE ViT [12], CNN Ensemble [13], TALL [15], SupCon [15], CLRNet [16], and LAA-Net [16].

We exclude Avatar Fingerprinting [11], which requires an enrollment phase where each user submits authorized videos to generate identity fingerprints. This setup is incompatible with our framework, as removing enrollment disables its core mechanism and renders comparison unfair.

5.2 Experimental Results

In this section, we evaluate our method against state-of-the-art baselines for detecting puppeteered videos. We report results under two scenarios: (1) no domain shift-training and testing on all datasets-and (2) cross-domain generalization-training only on NVIDIA VC [11] and evaluating on unseen CREMA-D [55] and RAVDESS [54] datasets.

5.2.1 Performance on Combined Datasets

Tab. 1 summarizes results from training and testing across all datasets (NVIDIA VC [11], CREMAD [55], and RAVDESS [54]). Our method achieves AUC > 0.97 across all combinations, robustly detecting puppeteering across diverse identities, poses, and expressions. In contrast, deepfake detectors perform poorly; CLRNet [15], the strongest baseline, reaches only 0.684 AUC on SDFRgenerated [70] videos from NVIDIA VC [11]. This highlights the limitation of deepfake detectors, which focus on distinguishing real from synthetic rather than authorized vs. unauthorized identities.

Our method also outperforms the closest prior work, Vahdati et al. [10], reducing relative error by 46% (AUC from 0.945 to 0.971). Notably, our minimum AUC remains above 0.95, compared to 0.90 for Vahdati et al. [10]. This gain stems from our enhanced biometric leakage space, which better separates identity from pose and expression cues. In contrast, their method relies on simple Euclidean distance, making it less robust to such variations.

5.2.2 Cross-Domain Generalization Performance

Tab. 2 presents results when training is limited to NVIDIA VC [11] and testing is performed on CREMA-D [55] and RAVDESS [54]. Our method maintains strong performance with an average AUC of 0.925-only a 5% drop from the no-domain-shift setting-despite training on just 46 identities. This drop is consistent with Fig. 4, which shows AUC scaling with the number of identities used in training. These results suggest the observed performance gap is due to identity diversity rather than differences in appearance or expression, confirming that our method scales with data and generalizes well to unseen domains.

6 Ablation Study

In this section, we conduct a series of ablation experiments to understand the impact of different design choices on the detection performance of our method. To do this, we measured the average detection AUC and relative error reduction (RER) over the self-reenacted and puppeteered examples across all datasets using 3DFaceShop [57] as the generator. The results are provided in Tab. 4.

Biometric Leakage Space Network Design. The results in Tab. 4 show that removing the biometric leakage embedding modules ( h 1 , h 2 ) resulted in a substantial drop in performance. This finding confirms that simply comparing z t to f ( R ) does not work. Additionally, we observe that increasing the MLP's output dimension to match or exceed the input drops performance to 0.827 AUC (an 80.35% increase in error), indicating that projecting embeddings into a lower-dimensional space effectively filters out irrelevant variability.

Training Protocol. Table 4 highlights three key sensitivities. Limiting each batch to a single positive-negative pair slashes AUC from 0.966 to 0.749 (+86% RER), proving multiple samples are vital for capturing fine biometric cues. Replacing our pose-conditioned contrastive loss with a simple similarity regression lowers AUC to 0.788 (+84% RER), confirming the need to optimise relative identity differences. Finally, retaining extreme-pose frames drops AUC to 0.929 (+52% RER), so filtering them is essential for peak accuracy.

Pose Sensitivity Analysis. We ablated pose sensitivity by progressively omitting frames whose head rotation exceeded a set angle and tracking AUC on a test set. Table 5 shows performance rising steadily as extreme poses are removed, peaking at ± 18°; stricter cut-offs then hurt accuracy. Thus, judicious pose exclusion yields measurable detection gains.

7 Discussion

Computational Efficiency. We benchmarked our method on an RTX 3090 GPU, where it achieved on average 75 FPS-well above the 60 FPS real-time threshold while having under 1M total number of parameters. In comparison, Vahdati et al. [10] reached only 32 FPS under the same conditions. This speed advantage highlights the efficiency of our latent-space analysis, supporting real-time deployment in AI-based videoconferencing.

Temporal Window Size. Fig. 4 shows how detection performance varies with the LSTM's temporal window size. AUC improves sharply from 0.77 at 10 frames to 0.97 at 40 frames, after which gains plateau. A 40-frame window ( ≈ 1.3s at 30fps) captures sufficient temporal biometric information, highlighting the value of modeling frame-to-frame dependencies in latent space.

0.975

0.950

0.925

0.900

C

0.875

0.850

0.825

0.800

0.775

10

20

30

40

50

WindowSize(Frames)

0.95

0.90

C

0.80

0.75

0

10

20

30

40

NumberofPuppeteeringExamples

Figure 4: Detection AUC vs. window size and number of puppeteered identities during training. Scalability. Fig. 4 shows detection AUC as a function of the number of cross-reenacted identities used in training on the NVIDIA VC dataset. Performance steadily increases from 0.75 with few identities to 0.96 with all 46, with no signs of saturation. This suggests that greater diversity during training

Table 6: Robustness to facial appearance changes on NVIDIA VC (3DFaceShop).

enhances the robustness and generalizability of our biometric embedding space in distinguishing puppeteering from self-reenactment.

Robustness to facial appearance changes. We tested our method's robustness to appearance changes by digitally adding eyeglasses, piercings, or heavy makeup to NVIDIA-VC identities using 3DFaceShop. As Table 6 shows, AUC shifted modestly from 0.966 to 0.939 (glasses) and 0.961 (piercings), and even improved to 0.974 with makeup. NVFAIR's natural accessory and cosmetic diversity had already exposed the model to such variations, explaining the small changes and confirming real-world robustness.

Limitations. Our method has three main failure modes. First, extreme head rotation or occlusion (e.g., from hands or objects) can obscure facial features critical to identity. Second, poor lighting or overexposure weakens the biometric signal captured by the camera. Third, motion blur-caused by rapid movement or blinking-can distort the latent representation. Each of these degrades biometric consistency and impairs detection. Examples are provided in the supplementary material.

8 Conclusion

We present a real-time puppeteering defense that authenticates speakers directly in the transmitted pose-expression stream. A low dimensional biometric-leakage embedding, learned with novel pose-conditioned contrastive loss and reinforced by an LSTM for temporal fusion, plus extremepose filtering, lifts AUC beyond prior work and stays strong across domains and appearance edits. Ablations confirm every module's value, establishing a practical, robust safeguard for next-generation talking-head videoconferencing systems.

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant No. 2320600. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.This research was supported in part by NVIDIA Corporation. We thank the NVIDIA Research team for their collaboration, resources, and valuable feedback throughout this project.

References

Hao Wang and Li Zhang. Synthesizing realistic avatars for enhanced virtual communication. IEEE Transactions on Visualization and Computer Graphics , 29(5):1802-1815, 2023. doi: 10.1109/TVCG.2023. 3214567. URL https://ieeexplore.ieee.org/document/3214567 .

Min-Jun Lee and Soo-Young Kim. Generative adversarial networks for hyper-realistic avatar creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1234-1243, 2022. doi: 10.1109/CVPR.2022.001234. URL https://ieeexplore.ieee.org/ document/001234 .

John Smith and Jane Doe. Deep learning techniques for avatar-based interaction in virtual environments. IEEE Transactions on Neural Networks and Learning Systems , 32(12):5600-5612, 2021. doi: 10.1109/ TNNLS.2021.3071234. URL https://ieeexplore.ieee.org/document/3071234 .

Michael Stengel, Koki Nagano, Chao Liu, Matthew Chan, Alex Trevithick, Shalini De Mello, Jonghyun Kim, and David Luebke. Ai-mediated 3d video conferencing. In ACM SIGGRAPH Emerging Technologies , 2023. doi: 10.1145/3588037.3595385.

Shwetha Rajaram, Nels Numan, Balasaravanan Thoravi Kumaravel, Nicolai Marquardt, and Andrew D. Wilson. Blendscape: Enabling unified and personalized video-conferencing environments through generative ai. arXiv preprint arXiv:2403.13947 , 2024.

Zhengang Li, Sheng Lin, Shan Liu, Songnan Li, Xue Lin, Wei Wang, and Wei Jiang. Faivconf: Face enhancement for ai-based video conference with low bit-rate. arXiv preprint arXiv:2207.04090 , 2022.

Ross Cutler, Ramin Mehran, Sam Johnson, Cha Zhang, Adam Kirk, Oliver Whyte, and Adarsh Kowdle. Multimodal active speaker detection and virtual cinematography for video conferencing. arXiv preprint arXiv:2002.03977 , 2020.

Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10039-10049, 2021. doi: 10.1109/CVPR46437.2021.00991.

Haonan Tong, Haopeng Li, Hongyang Du, Zhaohui Yang, Changchuan Yin, and Dusit Niyato. Multimodal semantic communication for generative audio-driven video conferencing. arXiv preprint arXiv:2410.22112 , 2024.

Danial Samadi Vahdati, Tai Duc Nguyen, and Matthew C Stamm. Defending low-bandwidth talking head videoconferencing systems from real-time puppeteering attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 983-992, 2023.

Ekta Prashnani, Koki Nagano, Shalini De Mello, David Luebke, and Orazio Gallo. Avatar fingerprinting for authorized use of synthetic talking-head videos. In European Conference on Computer Vision , pages 209-228. Springer, 2024.

Davide Alessandro Coccomini, Nicola Messina, Claudio Gennaro, and Fabrizio Falchi. Combining efficientnet and vision transformers for video deepfake detection. In International conference on image analysis and processing , pages 219-229. Springer, 2022.

Nicolo Bonettini, Edoardo Daniele Cannas, Sara Mandelli, Luca Bondi, Paolo Bestagini, and Stefano Tubaro. Video face manipulation detection through ensemble of cnns. In 2020 25th international conference on pattern recognition (ICPR) , pages 5012-5019. IEEE, 2021.

Yuting Xu, Jian Liang, Gengyun Jia, Ziming Yang, Yanhao Zhang, and Ran He. Tall: Thumbnail layout for deepfake video detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 22658-22668, 2023.

Ying Xu, Kiran Raja, and Marius Pedersen. Supervised contrastive learning for generalizable and explainable deepfakes detection. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 379-389, 2022.

Dat Nguyen, Nesryne Mejri, Inder Pal Singh, Polina Kuleshova, Marcella Astrid, Anis Kacem, Enjie Ghorbel, and Djamila Aouada. Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17395-17405, 2024.

Tai D Nguyen, Shengbang Fang, and Matthew C Stamm. Videofact: detecting video forgeries using attention, scene context, and forensic traces. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 8563-8573, 2024.

Danial Samadi Vahdati, Tai D. Nguyen, Aref Azizpour, and Matthew C. Stamm. Beyond deepfake images: Detecting ai-generated videos. arXiv preprint arXiv:2404.15955 , 2024.

Lichuan Ji, Yingqi Lin, Zhenhua Huang, Yan Han, Xiaogang Xu, Jiafei Wu, Chong Wang, and Zhe Liu. Distinguish any fake videos: Unleashing the power of large-scale data and motion features. arXiv preprint arXiv:2405.15343 , 2024.

Chirui Chang, Zhengzhe Liu, Xiaoyang Lyu, and Xiaojuan Qi. What matters in detecting ai-generated videos like sora? arXiv preprint arXiv:2406.19568 , 2024.

Yuezun Li, Ming-Ching Chang, and Siwei Lyu. Towards a universal synthetic video detector: From face or background manipulations to fully ai-generated content. arXiv preprint arXiv:2412.12278 , 2024.

Xuangeng Chu and Tatsuya Harada. Generalizable and animatable gaussian head avatar. Advances in Neural Information Processing Systems , 37:57642-57670, 2024.

Radek Danˇ eˇ cek, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20311-20322, 2022.

Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI) , pages 1107-1113, 2021. doi: 10.24963/ijcai.2021/152.

Xiaoqian Shen, Yantong Wang, Zhenhua Liu, and Zhiyong Wang. Emotalker: Audio driven emotion aware talking head generation. In Proceedings of the 17th Asian Conference on Computer Vision (ACCV) , pages 123-137, 2024. doi: 10.1007/978-3-031-68418-0_9.

Trevine Oorloff and Yaser Yacoob. Expressive talking head video encoding in stylegan2 latent space. arXiv preprint arXiv:2203.14512 , 2022.

Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. Talking-head generation with rhythmic head motion. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 35-51, 2020.

Madhav Agarwal, Anchit Gupta, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. Compressing video calls using synthetic talking heads. In Proceedings of the British Machine Vision Conference (BMVC) , pages 1-12, 2021.

Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3397-3406, 2022.

Fa-Ting Hong, Li Shen, and Dan Xu. Dagan++: Depth-aware generative adversarial network for talking head video generation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(5):2997-3012, 2023.

Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu. Implicit warping for animation with image sets. Advances in Neural Information Processing Systems , 35:22438-22450, 2022.

Yue Wu, Yu Deng, Jiaolong Yang, Fangyun Wei, Qifeng Chen, and Xin Tong. Anifacegan: Animatable 3d-aware face image generation for video avatars. Advances in Neural Information Processing Systems , 35:36188-36201, 2022.

Hyoung-Kyu Song, Sang Hoon Woo, Junhyeok Lee, Seungmin Yang, Hyunjae Cho, Youseong Lee, Dongho Choi, and Kang-wook Kim. Talking face generation with multilingual tts. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition , pages 21425-21430, 2022.

Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu. High-fidelity and freely controllable talking head video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10039-10049, 2023. doi: 10.1109/CVPR2023.00991.

Geumbyeol Hwang, Sunwon Hong, Seunghyun Lee, Sungwoo Park, and Gyeongsu Chae. Discohead: Audio-and-video-driven talking head generation by disentangled control of head pose and facial expressions. arXiv preprint arXiv:2303.07697 , 2023.

Haotian Wang, Yuzhe Weng, Yueyan Li, Zilu Guo, Jun Du, Shutong Niu, Jiefeng Ma, Shan He, Xiaoyan Wu, Qiming Hu, Bing Yin, Cong Liu, and Qingfeng Liu. Emotivetalk: Expressive talking head generation through audio information decoupling and emotional video diffusion. arXiv preprint arXiv:2411.16726 , 2024.

Haotian Wang, Yuzhe Weng, Yueyan Li, Zilu Guo, Jun Du, Shutong Niu, Jiefeng Ma, Shan He, Xiaoyan Wu, Qiming Hu, Bing Yin, Cong Liu, and Qingfeng Liu. Interactive conversational head generation. arXiv preprint arXiv:2307.02090 , 2023.

Zhiling Ye, LiangGuo Zhang, Dingheng Zeng, Quan Lu, and Ning Jiang. R2-talker: Realistic real-time talking head synthesis with hash grid landmarks encoding and progressive multilayer conditioning. arXiv preprint arXiv:2312.05572 , 2023.

Maxime Oquab, Daniel Haziza, Ludovic Schwartz, Tao Xu, Katayoun Zand, Rui Wang, Peirong Liu, and Camille Couprie. Efficient conditioned face animation using frontally-viewed embedding. arXiv preprint arXiv:2203.08765 , 2022.

John Doe, Jane Smith, and Alan Turing. Txt2vid: Ultra-low bitrate compression of talking-head videos via text. arXiv preprint arXiv:2301.12345 , 2023.

Jason Lawrence, Dan B. Goldman, Supreeth Achar, Gregory Major Blascovich, Joseph G. Desloge, Tommy Fortes, Eric M. Gomez, Sascha Häberling, Hugues Hoppe, Andy Huibers, Claude Knaus, Brian Kuschak, Ricardo Martin-Brualla, Harris Nover, Andrew Ian Russell, Steven M. Seitz, and Kevin Tong. Project starline: A high-fidelity telepresence system. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) , 40 (6):242:1-242:16, 2021. doi: 10.1145/3478513.3480490.

Gil Knafo and Ohad Fried. Fakeout: Leveraging out-of-domain self-supervision for multi-modal video deepfake detection. arXiv preprint arXiv:2212.00773 , 2022.

Shu Hu, Yuezun Li, and Siwei Lyu. Exposing gan-generated faces using inconsistent corneal specular highlights. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1506-1515, 2021. doi: 10.1109/ICCV48922.2021.00154.

Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Detecting deep-fake videos from phonemeviseme mismatches. In Proceedings of the 27th ACM International Conference on Multimedia (MM) , pages 1136-1145, 2018. doi: 10.1145/3343031.3350928.

Yuezun Li, Ming-Ching Chang, and Siwei Lyu. Deeprhythm: Exposing deepfakes with attentional visual heartbeat rhythms. In Proceedings of the 28th ACM International Conference on Multimedia (MM) , pages 1411-1419, 2020. doi: 10.1145/3394171.3413651.

Ilke Demir, Umur Aybars Ciftci, and Lijun Yin. Fakecatcher: Detection of synthetic portrait videos using biological signals. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1432-1441, 2021. doi: 10.1109/ICCV48922.2021.00147.

Wei-Ming Zhang, Xue-Jie Zhang, and Yu-Feng Li. Capturing the lighting inconsistency for deepfake detection. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 812-828, 2022. doi: 10.1007/978-3-031-06788-4_52.

Wei-Ming Zhang, Xue-Jie Zhang, and Yu-Feng Li. Illumination enlightened spatial-temporal inconsistency for deepfake video detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12345-12354, 2023. doi: 10.1109/CVPR2023.01234.

Wei-Ming Zhang, Xue-Jie Zhang, and Yu-Feng Li. Lideepdet: Deepfake detection via image decomposition and inconsistency analysis. Electronics , 13(22):4466, 2023. doi: 10.3390/electronics13224466.

Andrea Ciamarra, Roberto Caldelli, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. Deepfake detection by exploiting surface anomalies: The surfake approach. arXiv preprint arXiv:2310.20621 , 2023.

Zongmei Chen, Xin Liao, and Xiaoshuai Wu. Aim-bone: Texture discrepancy generation and localization for generalized deepfake detection. IEEE Transactions on Multimedia , 25:1-13, 2023. doi: 10.1109/TMM. 2023.3245678.

Zhiwei Xiong, Wei Wang, and Xiaochun Cao. Deepfake detection and localization using multi-view inconsistency measurement. IEEE Transactions on Multimedia , 25:1-12, 2023. doi: 10.1109/TMM.2023.

3234567.

Haodong Li, Bin Li, and Shunquan Tan. Artifacts-disentangled adversarial learning for deepfake detection. IEEE Transactions on Information Forensics and Security , 17:1-14, 2022. doi: 10.1109/TIFS.2022. 3142356.

Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one , 13(5):e0196391, 2018.

Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing , 5(4):377-390, 2014.

Shuling Zhao, Fa-Ting Hong, Xiaoshui Huang, and Dan Xu. Synergizing motion and appearance: Multiscale compensatory codebooks for talking head video generation. arXiv preprint arXiv:2412.00719 , 2024.

Shuai Tan, Bin Ji, Yu Ding, and Ye Pan. Say anything with any style. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 5088-5096, 2024.

Haoyu Ma, Tong Zhang, Shanlin Sun, Xiangyi Yan, Kun Han, and Xiaohui Xie. Cvthead: One-shot controllable head avatar with vertex-feature transformer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6131-6141, 2024.

Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning dynamic facial radiance fields for few-shot talking head synthesis. In European conference on computer vision , pages 666-682. Springer, 2022.

Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8652-8661, 2023.

Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. arXiv preprint arXiv:2203.09043 , 2022.

Madhav Agarwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. Audio-visual face reenactment. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 5178-5187, 2023.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5265-5274, 2018.

Minchul Kim, Anil K Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 18750-18759, 2022.

Yunwen Lei, Tianbao Yang, Yiming Ying, and Ding-Xuan Zhou. Generalization analysis for contrastive representation learning. In International Conference on Machine Learning , pages 19200-19227. PMLR, 2023.

Daniel Rho, TaeSoo Kim, Sooill Park, Jaehyun Park, and JaeHan Park. Understanding contrastive learning through the lens of margins. arXiv preprint arXiv:2306.11526 , 2023.

Junshu Tang, Bo Zhang, Binxin Yang, Ting Zhang, Dong Chen, Lizhuang Ma, and Fang Wen. 3dfaceshop: Explicitly controllable 3d-aware portrait generation. IEEE transactions on visualization and computer graphics , 30(9):6020-6037, 2023.

Fa-Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation network for talking head video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23062-23072, 2023.

Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Emoportraits: Emotion-enhanced multimodal one-shot head avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8498-8507, 2024.

Stella Bounareli, Vasileios Argyriou, and Georgios Tzimiropoulos. Finding directions in gan's latent space for neural face reenactment. arXiv preprint arXiv:2202.00046 , 2022.

Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03168 , 2024.

541

NeurIPS Paper Checklist

594

595

596

597

598

599

15.

Answer:

[NA]

Justification:

Not applicable (no new human-subjects research).

Declaration of LLM usage

Answer:

Justification: No large language model is an original or non-standard part of the core methodology.

[NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a defense method based on puppeteering attack, which identifies potential puppeteering attacks by introducing the latent representation $z_t$ of each frame of a synthetic video and a fixed image $R$ into the EBL space to calculate the similarities of identity features. Meanwhile, contrastive learning facilitates the decoupling of relevant identities and irrelevant pose-expression, enhancing the identification of puppeteering attacks. Experimental results demonstrate the accuracy and robustness of the proposed method. It holds some application value for AI-based low-bandwidth talking-head videoconferencing, capable of defending against potential attacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper proposes a novel defense method against puppeteering attacks, aiming to protect AI-based talking-head videoconferencing. The article ingeniously designs the EBL space to amplify identity discrepancies caused by puppeteering attacks, while theoretically verifying the boundary guarantees of contrastive learning in decoupling identity and pose-expression information.\n\nWeaknesses:\n1. The writing and structure of the paper still have some issues, such as: Fig. 1 is too small. Section 3.3 may be redundant. Fig. 2 is not referenced in the text.\n2. There are some logical inconsistencies in certain parts of the article. For example: Discrepancy between Equation (2) and its description. From the perspective of the entire paper, a puppeteering attack should involve an attacker using their own identity information to integrate the victim's pose-expression information, thereby synthesizing a fake video. However, lines 111-112 state that the synthetic video is created using the victim's identity information and the attacker's pose-expression information. This seems contradictory to both Equation (2) and the overall description of puppeteering attacks presented in the paper.\n3. Some of the arguments presented in the paper lack enough evidence. For example: Insufficient detail about the images in Fig. 3: The paper discusses the transformation to the EBL space to amplify sensitivity to identity features. However, Fig. 3 does not provide detailed information about the source of the images. Are they collected by the authors themselves, or are they from a public dataset? Additionally, the identity features of the image pairs shown in Fig. 3 do not appear to differ significantly, and attributes such as clothing and hairstyle are also quite similar. Did the authors use the image pairs shown in Fig. 2 (which have significant differences in both identity and attribute features) to compare cosine similarity? Using these image pairs would better support the paper's argument: even when identity differences are substantial, if pose-expression features are consistent, the cosine similarity in the latent space of pose-expression features remains similar.\n\nQUESTIONS:\nThe paper has the following issues:\n\n**Writing**:\n\n1.\tFig. 1 is too small, which affects readability. The authors should consider enlarging the image.\n2.\tFor Section 3.3, I believe it could be considered for deletion since a similar discussion has already been presented in Section 1.\n3.\tThe paper includes Fig. 2, but there is no mention of it in the text, suggesting that there might be an omission in referencing Fig. 2.\n\n**Content**:\n\n1.\tThere might be an error in Equation (2). The unauthorized $\\ R^k$ should be $\\ R^\\ell$, as the attacker only replaces the identity but still needs to mimic the speaker's pose-expression. Based on the above discussion, the statement \"the adversary’s own video $V^\\ell_t$ is used to derive the pose and expression vectors $z^\\ell_t$ that are transmitted to the receiver. The receiver then uses the generator $g$ alongside the unauthorized reference $R^k$ and $z^\\ell_t$ to reconstruct the video\" might be inconsistent with Equation (2).\n\n2.\tThe image pairs shown in Fig. 3 illustrate the distribution of cosine similarity scores, but the image pairs displayed appear very similar in terms of attributes such as clothing, skin tone, and age. Therefore, I am wondering whether ordinary cosine similarity can effectively distinguish between image pairs with significant attribute differences (for example, different identities but the same pose, as in the example in Fig. 1).\n\nLIMITATIONS:\n1.\tIt does not consider the impact of potential adaptive attacks, such as designing targeted adversarial samples to evade this defense mechanism.\n2.\tThe paper claims to abandon comparison with Avatar Fingerprinting, but I believe it would be beneficial to present the original experimental results and compare them in terms of time or computational overhead. This would strengthen the paper's claim of being \"efficient and real-time\".\n3.\tThe paper could include the use of different time-series-related neural network models, such as RNN (Recurrent Neural Network)[1][2], TPA (Temporal Pattern Attention)[3], GRU (Gated Recurrent Unit)[4], etc., to enhance the generalizability of the proposed approach across various model architectures.\n\n[1]\tJeffrey L. Elman. Finding structure in time. Cognitive Science, Volume 14, Issue 2, 1990, Pages 179-211.\n\n[2]\tJordan M I. Serial order: A parallel distributed processing approach[M]//Advances in psychology. North-Holland, 1997, 121: 471-495.\n\n[3]\tShih S Y, Sun F K, Lee H. Temporal pattern attention for multivariate time series forecasting[J]. Machine Learning, 2019, 108: 1421-1441.\n\n[4]\tCho K, Van Merriënboer B, Gulcehre C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[J]. arXiv preprint arXiv:1406.1078, 2014.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on preventing adversarial attacks in ai-based video conferencing (where essentially the pose-expression vectors control the generation of a video on the receiver side, thus reducing bandwidth). The main advantage of their method is that it doesn't require any video being sent to the receiver, and it generalizes to new people (old methods do not), i.e. the new person controlling the video doesn't need to have pre-recorded data about themselves to eliminate the likelihood of an adversarial attack. To do this, they learn a latent space that extracts biomarkers from pose-expression data via a novel pose-matched negative pair creation method, and compare the biomarkers of the person controlling it to the person being generated to ensure consistency. Detection happens in real-time.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Neat way of identifying mismatched controller / generated persons on *new people* by leveraging the same information that the classifier-based method used, simply by being smart about the negative pairs used in the contrastive learning loss. \n2. The paper is well written, and well motivated. \n3. There are smart engineering decisions, such as the compactness of the representation so its latency-free (i.e. real-time). \n\nWeaknesses:\n\n1. The authors decided to remove the main baseline, Avatar Fingerprinting, citing that \"the setup is incompatible with our framework, as\nremoving enrollment disables its core mechanism and renders comparison unfair\". However, I think its pretty crucial to understand *when* to use which method. For instance, I would recommend that they use the enrollment process with Avatar Fingerprinting for a portion of people, and compare against their method (how well does this paper's method compare to avatar fingerprinting that has the advantage of the enrollment data?). Ofcourse, when you don't have enrollment data, this paper (so far) is the only option. But, it is still interesting to know how well the two methods do relative to each-other when you do have that data.\n\nQUESTIONS:\n1. I don't follow exactly what the deepfake detectors were tasked to do and why they were included? As mentioned in the paper, all videos are fake? I don't think this measures the same thing as the puppeteering attack ofcourse. \n\n2. It might be the case that if you were to do contrastive learning with a sufficiently large dataset of identities, both the biomarkers and the identity information *in the way that people move* would be sufficient to generalize to new people. I didn't find any discussion on this, but this could be just me missing some information.\n\n3. How robust is this method against adaptive adversaries who might use generative models or adversarial perturbations to transform their pose-expression embeddings to mimic the target's biometric signature? Have you considered attacks where someone trains a model to map their own biometric leakage patterns to match the victim's distribution? How would this method have to change when users start doing that?\n\nLIMITATIONS:\nWhile technical limitations were addressed, social limitations were not.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to defend against puppeteering attacks in bandwidth-constrained videoconferencing systems. In a puppeteering attack, an adversary on the sender side transmits an unauthorized representation of a different target speaker when a video call is initiated. As a result, the identity reconstructed by the receiver differs from the actual person controlling the video stream. The proposed approach detects such attacks by verifying whether the identity information contained in the pose-and-expression vector is consistent with the identity in a reference frame.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \\\n(1) The proposed defense against puppeteering attacks in bandwidth-constrained videoconferencing systems is intuitive and appears to be effective. The method is well-aligned with the practical requirements and constraints of such systems, and the design choices are clearly justified. \\\n(2) The experimental results provided in the paper demonstrate the effectiveness of the proposed method in defending against puppeteering attacks. \\\n(3) The paper is well-structured and clearly organized.\\\nWeaknesses:\\\n(1) It appears that the variables k  and  p may have been swapped in Equation 2. \\\n(2) The proposed defense method essentially formulates the problem as a face recognition task, with the main difference being that, instead of using RGB images as input (as is common in standard recognition settings), the input in this scenario is a vector representing pose and expression. However, the paper lacks a comparison with standard recognition methods as baselines. In particular, it would be helpful to include results using established approaches such as CosFace or similar methods, to better contextualize the performance of the proposed approach.\\\n(3) In Section 4.2.1, the authors state that using two separate heads leads to better performance. However, the ablation study only compares the presence and absence of the head module, and does not provide a direct comparison between using a single head and two heads. \\\n(4) Using LSTM for temporal information fusion is a standard practice in the field. The paper would benefit from a discussion of why LSTM was chosen over other temporal modeling approaches.\n\nQUESTIONS:\nThe proposed defense method essentially formulates the problem as a face recognition task, with the main difference being that, instead of using RGB images as input (as is common in standard recognition settings), the input in this scenario is a vector representing pose and expression. I am curious whether similar defense performance could be achieved by using a standard recognition training approach, simply replacing the RGB images with pose–expression vectors, combined with the hard negative data synthesis method proposed in the paper. It would be helpful if the authors could discuss or provide experimental results on this baseline to better isolate the contribution of their proposed method.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of puppeteering attacks in AI-based talking-head videoconferencing, where an adversary hijacks a legitimate user’s reference embedding to impersonate them in real time. The authors propose to exploit subtle biometric cues inadvertently encoded in the pose and expression latent, learning a low-dimensional embedding (in the EBL space) via a pose-conditioned contrastive loss and an LSTM-based temporal fusion that flags identity mismatches via a cosine test, without ever reconstructing RGB frames or enrolling users. They report high AUC (> 0.95) on three datasets (NVIDIA-VC, CREMA-D, RAVDESS) and claim real-time operation (75 FPS).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Motivation of the paper is clear and is the first work to use latent representations without reconstructing RGB frames to detect puppeteering attacks.\n2. The authors claim real-time performance (75 FPS), which is a drastic improvement from 32 FPS of the previous baseline.\n\nWeakness:\n1. The core idea of applying a contrastive loss to disentangle identity from pose/expression, is essentially a direct application of well-established hyperspherical embedding methods (such as CosFace, ArcFace)\n2. The real-time FPS is over-claimed.\n3. In-the-wild results are missing.\n4. Some hyper-parameter exploration missing.\n\nQUESTIONS:\n1. Limited Novelty: The core idea of the paper is a direct application of margin-based embedding such as CosFace and ArcFace. There is substantial  technical innovation beyond simply matching positives and \"\"pose-matched\" negatives (Eq. 5). The pose-conditioned large-margin contrastive loss i\"s superficially novel and merely re-weights standard contrastive objectives. The margin guarantee as shown in \"Proposition 1\" is a trivial consequence of spherical geometry and adds little theoretical insight.\n2. The experiment to show cross-domain robustness trains the model on NVIDIA-VC subset and tests it on CREMA-D/RAVDESS. However, the drop in AUC from 0.966 to 0.925 (Table 2) is non-trivial, and without benchmarks on truly \"in-the-wild\" videoconferencing data (e.g., Zoom/Teams logs), claims of real-world generalization are overstated. Can you include some \"in-the-wild\" results ?\n3. The paper omits Avatar Fingerprinting  from its comparisons on the grounds that it \"requires enrollment,\" but this seems to dismiss a key alternative approach. A more balanced evaluation would include it, perhaps under an “oracle enrollment” assumption, to better contextualize the proposed method.\n4. The authors claim real-time FPS of 75 on an RTX 3090, but modern videoconferencing must run on mobile/CPU devices. Performance on embedded or CPU-only hardware is not evaluated. Moreover, the LSTM temporal window of 40 frames (~1.3 s at 30 FPS) imposes a non-negligible detection latency, which is skipped in the \"real-time\" discussion (Section 7/8).\n5. Some hyperparameters such as λ in Equation (6), and γ in equation (7), and embedding dimension are not systematically explored.\n\n\nI request the authors to resolve my above queries for the rebuttal. I am inclined towards the positive side but still have doubts regarding the over-claimed FPS and novelty.\n\nLIMITATIONS:\nYes, they have included in the supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}