Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing
Authors: Danial Samadi Vahdati
Tai Duc Nguyen
Ekta Prashnani
Koki Nagano
david luebke
Orazio Gallo
Matthew Stamm
Venue: NeurIPS 2025 poster ()
Keywords: Multimedia Forensics
Talking Head Videos
Talking Head Videoconferencing
Puppeteering Attacks
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: AI-based talking-head videoconferencing systems reduce bandwidth by transmitting a latent representation of a speaker’s pose and expression, which is used to synthesize frames on the receiver's end. However, these systems are vulnerable to “puppeteering” attacks, where an adversary controls the identity of another person in real-time. Traditional deepfake detectors fail here, as all video content is synthetic. We propose a novel biometric defense that detects identity leakage in the transmitted latent representation. Our metric-learning approach disentangles identity cues from pose and expression, enabling detection of unauthorized swaps. Experiments across multiple talking-head models show that our method consistently outperforms prior defenses, operates in real time on consumer GPUs, and generalizes well to out-of-distribution data. By targeting the latent features shared during normal operation, our method offers a practical and robust safeguard against puppeteering.

Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AL-based Videoconferencing
Danial Samadi Vahdati', Tai Due Nguyen', Koki Nagano",
David Luebke", Orazio Gallo?, Ekta Prashnani
Matthew Stamm'
Drexel University, NVVIDIA
Abstract
Al-based talking-head videoconferencing systems reduce bandwidth by sending a compact pose-expression latent and re-syntheiizing RGB at the boeiver-but this latent can be "puppeteered," letting an attacker hijack a victim's likeness in real time. Because every frame is synthetic, deepfake and synthetic video detectors fail outright. To address this security problem, we exploit key observation: the pose expression latent inherently contain biometric information of the driving identity. Therefore, we introduce the first biometric leakage defense without ever looking at the reconstructed RGB video: cossicondiiioned encoder that isolates persistent identity cues inside the transmitted latent while cancelling transient pose and expression. A simple cosine test on this disentangled embedding flags illicit identity swaps as the video rendered. Our experiments multiple talking-heaa generation models show that method consistently outperforms existing puppeteering defenses, operates real-time, and shows strong generalization to oft-of-distribution scenarios. Our code and trained models are available at
contrastive
Introduction
Advances in generative AI have enabled the creation of hyper-realistic synthetic vidcos. This has led the development of many new technologies, including both avatar-based communication sys- tems [13 and Al-based videoconferencing systems 44 Al-based talking head videoconferencing systems are receiving increasing attention due to their significant bandwidth savings [8]. Instead of continuously encoding and transmitting each video frame of speaker, these systems only transmit embeddings that capture a speaker's 's pose and facial expression. Then, a generative Al system at the receiver's 's end uses these embeddings, in conjunction with an initial representation of the speaker, to create an accurately reconstructed video.
Unfortunately, Al-based talking head videoconferencing systems 5 g are vulnerable to new form of information attack known as "puppeteering" [TO]. In this attack, malicious user the sender side transmits an unauthorized representation of different target speaker when video call is initiated. result, the identity in the video that the receiver constructs different than the identity of the person who actually controls the video [10
While many forensic approaches, such as deepfake (12-17 and synthetic video detectors (1-
aim to expose unauthorized Al-generated media, they operate by identifying evidence that video has been synthetically generated. However, Al-based talking-head videoconferencing, every video is Al- Al-AAl--eee generated, rendering existing media forensic approaches unable to detect puppeteering.
Corresponding author: ds3729 @ drexel.edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Al-Based Talking Head Videoccon/erence
Puppeteering Attack
Binmetric Leakage Enhancement
Our Method
Figure I: Al-based talking-head generators transmit only a compact pose-and-expression embedding for low-bandwidth videoconferencing, but remain vulnerable to puppeteering attacks that swap in different identity for live impersonation. Our defense capitalizes on biometric signals inadvertently leaked in these embeddings to reveal mismatches between the driving speaker and the reconstructed identity in real time.
Puppeteering attacks pose fundamentally different threat from those typically addressed by media forensics. Instead of asking "Is this video real or Al-generated?" one key question must be "Is this Al-generated video being driven by authorized identity, or has someone hijacked it?" As talking- head videoconferencing technology continues to advance, it is essential to mitigate puppeteering attacks before these systems are widely deployed. Addressing these threats proactively will help ensure secure and trustworthy communication over these systems.
Few defenses address puppeteering. Prashnani al. introduced Avatar Fingerprinting [1]] which works by enrolling each user's characteristic facial motion patterns into the detector's training data. Although this could help, requires significant user-specific data build reliable motion signatures. In contrast, Vahdati et al. avoid enrollment by decoding and re-encoding the received video for comparison [IO; the round-trip is compute-heavy and adds distortions that hurt accuracy.
In this paper we introduce a real-time, enrollment-free defense that detects puppeteering attacks entirely in the latent domain. Our key insight is that the pose-and-expression embeddings already transmitted by modern talking-head systems leak subtle, but reproducible, biometric signatures [2
23 We learn compact Enhanced Chooetric-Leakagee (EBL) space in which identity cues amplified while pose and expression variance is actively suppressed. A pose-condittoned contrastive loss drives this separation, and lightweight temporal I.STM aggregates evidence yield stable, millisecond-leww decisions. By comparing the sender's 's latent identity with the target identity rendered the receiver, system flags any mismatch without decoding RGB frames, tracking landmarks, collecting per-user motion profiles. Extensive experiments across many generators and datasets show that our method achieves state-of-the-art detection performance while working in real-time.
In summary, our contributions are as follows.
We present the first method that operates solely on data already available at the receiver, requiring user enrollment additional sensors.
We formalize and harness identity leakage inherent in posc expresspne embeddings, turning a previously ignored vulnerability into defensive signal.
We develop a novel loss and training protocol to learn low-dimensional EBL representation that maximizes identity separability while nullifying pose and expression.

---

NVIDIA-VC
RAVDESS
CREMA-D Self-Reemmceed
Reference
Reeeneeed
Cross-Remacted
Reference
Self-Reenacee
Crov-Reenaced
Reference
Crus-Reemncee
Figure 2:
Illustration of three datasets (NVIDIA-VC [II. RAVDESS [54], CREMA-D (55) shown across three columns each. Row indicates the type of frame displayed in Row 3: Reference, Reff-Reenac,ed, or Cross-Reenacted.
Though extensive experiments across fifteen generator/dataset combinations, our approach achieves state-of-the-aa art puppeteering detection in real time, demonstrating practical viability for deployment in bandtrith videoconferencing systems.
Background and Related Work
Talking-Heaa Head Video Systems. Talking- head systems synthesize facial motion and speech in real time by transmitting low-dimensional embeddings of pose and expression rather than full video frames 27 A generator at the receiver reconstructs a realistic face. Methods range from compressed embeddings landmark- -based updates per frame g 28 reducing bandwidth while maintaining high visual fidelity [34. Systems like NVIDIA's Vid2Vid Cameo g and Google's [4]] demonstrate real-time, 3D teleconferencing.
Project Starline
Forensics for Synthetic Media. Synthetic media forensics traditionally aims to distinguish real from Al-generated content. Deepfake detectors [
17
42
leverage semantic cues 46 lighting
inconsistencies [4), and facial texture artifacts 51 53
while synthetic video detectors [1
focus on identifying Al Al-generated traces. However, low-bandwidth talking-head systems, every video is synthetic [TO I. rendering these detectors ineffective - they cannot determine whether the video represents the correct identity has been hijacked.
Puppeteering Attacks and Existing Defenses. Puppeteering attacks replace the target's appearance with the adversary's pose-and-expression vectors at call initiation, causing the receiver to synthesize a video of the wrong identity. Avatar Fingerprinting [1] mitigates this threat by first enrolling each user's facial-motion signatures; at inference it flags session whenever the incoming motion deviates from the stored profile. Vahdati er [TO] avoid enrollment by re-encoding the synthesized video and comparing embeddings, but their pipeline depends on facial-landmark estimation, which degrades under large head rotations and lighting artifacts introduced by video reconstruction.
In contrast, our method exploits biometric leakage already present in the pose-and-expression latents, removing both the enrollment phase and any reliance on facial landmarks. contrastive objective disentangles identity from pose, yielding compact, -riencyy representation that generalizes to unseen generators and yeal-world conferencing conditions.
3 Problem Formulation
3.1 Talking Head Video Systems
Al-enabled talking-head videoconferencing systems have been proposed to minimize transmitted data by encoding and transmitting only pose and expression information from a speaker rather than an entire video frame 8 56-621. As this technology rapidly matures, our goal here is to develop security measures that improve the trustworthiness of these systems when they are deployed.
A videoconfeerencing call starts with each participant l sending neutral reference portrait R* the receiver. Typically, R is an image of the speaker a neutral pose. As the speaker continues to speak, each new video frame V. capturing this speaker the sender side is processed by embedding function f to produce embedding =f f( that encodes the speaker's instantaneous pose and expression at time This embedding is transmitted to the receiver, as is shown in Fig.
At the receiver side, generator is used to produce video of the speaker on the basis of the received embedding 2f and the reference representation R² such that
k =[(2 R*)
(I)
3

---

Here, the superscript V. denotes that speaker & both drives and appears in the rendered video. We refer to the sender's identity the "driving identity" and the reference portrait sent to the receiver as the "target identity" [T]] While such a talking-heat video-conferencing system promises to be bandwidtt-efficient exposes a vulnerability to puppeteering attacks when a malicious sender impersonates another individual.
3.2 Puppeteering Attacks
puppeteering attack exploits the inherent trust in the reference representation transmitted to the receiver when a videoconferencing call begins. Here, an adversary, speaker L, obtains a target speaker's representation R and substitutes for their own without authorization at the start of the call [TO]. As the call proceeds, the adversary's own vidco V is used to derive the pose and expression vectors z that are transmitted to the receiver. The receiver then uses the generator alongside the unauthorized reference R² and z to reconstruct the video. As a result, the receiver is presented with realistic-looking video of speaker k;, controlled in real time by the speaker such that
V = ((((V)), R²) = R*)
(2)
where Vi is an unauthorized impersonation of speaker driven by speaker k. To address this threat, one must come up with mechanisms to detect whether the receiver's generated video is driven by the same authorized identity, rather than an unauthorized impersonator.
3.3 Why Real-vs-Syntheii Detectors Fail
Deepfake and synthetic-video detectors flag frames whose pixel statistics reveal AI generation, implicitly assuming that "real" camera footage is the baseline. In bandwidth-efficient talking-head conferencing, however, every frame even those from honest participants is produced by generator; "synthetic" is the norm, not the anomaly. The security question therefore shifts from "Was this video AI Al-generated?" to "Does the rendered face correspond to the person actually driving the latents?" Pixel-level detectors cannot answer that mapping, leaving puppetecring attacks undetectable.
Proposed Approach
In this paper, we present real-time solution for detecting puppeteering attacks in talking-head videoconferencing, uniquely operating entirely in the latent domain without access to reconstructed RGB frames. To this, we re-encode each mose-and-expression latent embedding into a compact Enhanced Biometric Leakage (EBL) embedding space that captures who is speaking while discarding how they move. posseccodiitoned contrastive loss learns this space by pulling together identity- consistent pairs and pushing apart ceentitymiimmtcched pose-matched pairs. At run time, we simply compare each live EBL vector to the EBL embedding of the reference portrait sent at call setup: sharp drop in similarity reveals that the driving and target identities have diverged, signalling puppeteering. For additional robustness and stable authentication, during training, we discard unreliable extreme-pose frames, and leverage an LSTM to fuse successive EBL scores to output stable, low-latency decisions. These innovations yield an algorithm that authenticates speakers in real-time, requires no enrollment or landaa-k preprocessing. The following sections detail our approach and its implementation.
4.1 Biometric Entanglements in Pose & Expression Latent Space
Low- www-bandwidth talking-head systems encode each frame VÚ as a pose-and-expresson vector o; fVV), which the receiver's generator turns back into pixels. Although designed for geometry, z inherit identity cues from the physical face that produced the motion. Measurements such as inter-ocular distance, jaw curvature, or lip thickness inevitably contaminate 2 because head pose and facial musculature cannot be sensed independently of the speaker's anatomy. Prior studies to disentangle identity from pose and expression also reported persistent identity traces in pose- conditioned representations 222 23 which provides the empirical basis for our proposed defense.
Let R denote the statie reference portrait of the nominal speaker transmitted at session start. Both endpoints apply the shared encoder f to compute the reference embedding (R), placing R and the

---

Expression Embedding Space
Biometric Leakage Space
Offere PSE
Same
Sanc
Offirurt Dffirrent P&E
Cefteent Oificrent PaL
Cosine similarity (#) between embeddings
Casine similarity (#) between embeddinng
Same Identity
Different Identity Different Pose Expression
Different Identity Same Pose & Expression
Different Pose Exoression
Figure 3: Similarity distributions in P&E space (left) and biometric leakage space (right). Red: same ID, diff. P&E; blue: diff. ID, same P&E; black: diff. ID, diff. P&E.
live stream {2e} in one latent space. A natural baseline defense to compute the cosine similarity s2(2 (R)) and accept the incoming frame if the exceeds threshold. Unfortunately. pose and expression variability overwhelms the subtle biometric signal.
We demonstrate this effect in Fig. which shows the cosine similarity distributions for three condi- tions: (1) same identity, differing posc/cxpression; (2) different identities, matched pose/expression, (3) different identities, differing pose/cxpression. We observe that the density for condition (2) frequently eclipses condition (1), which means that two different people exhibiting an identical yaw-pitch often appear closer in latent space than two frames of the same person smiling versus neutral. Raw pose-and-expresse
space is therefore pose-dominated: distance reflects "how the head moves" more than "whose head it is."
This observation key to our puppeteering defense: amplify latent biometric cues while sup- pressing pose and expression variance that proximity encodes identity. Simply collecting more negatives or tweaking the similarity threshold cannot meet this requirement; representation itself must be re-shaped. In the next section, we introduce the Enhanced Biometric Leakage (EBL.) space, learned directly from the latent channel, that achieves this goal without RGB reconstruction, user enrollment, landmark tracking, enabling real-time authentication pipeline.
4.2 Enhanced Biometric Leakage (EBL) Space
Our key observation above dictates two design imperatives: (1) craft a representation in which distances reflect identity rather than pose/expression, and (2) keep that representation compact to work in real-time. We meet both goals by re-encoding the raw pose-and-expressiv latent 2 into a low dimensional Enhanced Biometric Leakage vector and learning contrastive objective that amplifies identity information while actively cancelling pose.
4.2.1 Latent re-encoding
The raw pose-ent expressiee vector 2₁ is dominated by geometric factors, such yaw, pitch, mouth aperture, while biometric cues hide in low-variance directions. To amplify these cues we attach two lightweight projection heads, h, and h2, that remap both the live embedding and the static reference ((t) into a shared, compact metric space.
Separate heads are essential because the two inputs follow different distributions: 7, drifts frame by frame with expro-expresssons, whereas f(R) fixed at call setup. Allowing independent normalization lets each head adapt to its own domain before they meet in the common space.
Each head consists of linear layer, ReLU activation, layer normalization, and second linear layer followed by & normalization, such that
b(2,, R) (h1((2))) he h)(f)
(3)
where so(r.) denotes cosine similarity.
5

---

4.2.2 Pose-conditioned contrastive objective
The cornerstone of our method is pose-matched contrastive loss that aligns gradient pressure with the nuisance we aim discard. Specifically, we pursue two complementary goals inspired by large-margin hyper-spherical embedding theory that share the same identity even when pose/expression varies, and (2) minimize the similarity of embeddings drawn from different identities having the same pose and expression. The resulting objective comprises one term for each goal.
(1) maximize the similarity of embeddings
Positive term. Any pair that agrees in identity, regardless of pose, is treated as positive:
Lp 1111--220 b(z R²),
where & indexes the speaker and p denotes the instantaneous pose-expression state. Minimizing pulls together all manifestations of speaker k.
Negative term. Hard negatives are constructed by replicating pose but swapping identity. Concretely, we synthesise R a portrait of impostor + l rendered at the identical pose p, and impose
LN
1 N
bil R.P)
(S)
Because every negative matches pose, the gradient direction orthogonal to pose variation, forcing the network to discriminate on biometric cues alone.
Total loss. The pose-conditioned Large-Margin Cosine Loss (PC-LMCL)) is
LB +
(6)
with single hyper-parameter À controlling repulsion strength. With À < 1, LB is (1 + X)-Lipschitz (S 2), placing them within the margin-risk framework of Lei et al. 65
Margin guarantee. We formalize the geometric effect of (6) as follows. We note that here, we treat each speaker's 's front-facing reference portrait R as their pose- averaged center | Mell2 0.5%.
Proposition 1 Assume all embeddings are Ez-oormalized. for some e, 0.
cos(
k.P nea)) 2 1 - 2t
1
Σ Ex
cos( (z) k.p RRFPP) 5 -7
(7)
N
then the class centers [R satisfy
cos cos(pu)))) 1-
k #E
(8)
Thus Ln enforces an inter-class angular margin of at least e within each pose slice.
Intuition. Driving C. -> 0 pins positives to the "north pole". while LN - 0 pushes pose-matched negatives toward the antipode; the spherical triangle inequality then yields Prop. T The full proof appears in Appendix G.
Consequently, minimizing La maximizes the angular decision margin between identity manifolds while cancelling pose an effect known to tighten generalization bounds in hyper-spherical embedding spaces 663 66). Prior methods ignore latent pose aligmment and instead rely on RGB or landmarks, incurring latency and robustness penalties. By operating purely in the latent domain, PC-LMCL yields compact representations suitable for real-time deployment.
4.3 Training Protocol
The contrastive objective of the previous section is optimized on mini-batches organized as episodes. Each episode selects an anchor clip from speaker k. assembles identity-consic positives drawn from other poseslexpressions of k. and generates pose-matched negatives by rendering the same motion with impostor identities & Because the shared encoder / remains frozen, only the two projection heads learn, allowing rapid re- re-training should codec revision alter its latent format.

---

Table 1: Puppeteering attack detection performance measured in AUC across different dataset-
generation method pairs. NVC=NVIDIA VC [T] RAV-RAVDESS 54 and CRD-CREMA-D [55]
3DPaceShop
MCNET
[S
EMOPertraits
SDFR
LivePortrait
:1
Arp:
Method
NYC
RAY
CRD
NVC
RAY
CRD
NVC
RAV
CRD
NVC
RAY
CRD
NVC
RAV
CRD
NVC
RAV
CRD
Effciint VIT 12
.575
580
.563
.573
-573
.611
.560
547
.500
-569
-570
.505
.578
.578
.501
.571
.551
582
CCE
[P]
.322
6088
-518
/628
-618
.616
6624
6630
.507
662
6622
6006
6339
-615
.515
-830
.619
606
CNN Fnsemble
540
.599
-520
.570
.449
.50
TALL 14
.355
-511
.005
.331
.06
.044
.660
.009
.900
546
.505
.501
532
-00
515
541
506
505
SupCon
1+
.879
632
.40
651
.21
.322
676
669
.25
.631
.648
.199
676
639
615
.772
6.8
626
CLR Net 72
.820
633
.535
.529
.636
.684
A34
604
.641
A35
-334
.651
6339
.032
.000
.631
633
548
LAANe
Viludai
918
9133
Ours
.884
995
.958
.889
.978
.984
.962
.974
.98.
.970
.978
.922
.982
955
.962
.97
966
.968
Extreme Pose Exclusion. Before each update we discard frames whose head-pose estimate is unreliable. For instance, some poses may deviate substantially from the reference image, or, the speaker's face can be substantially occluded. This extreme-pos pose exclusion keeps the gradient focused on informative biometric variation.
To accomplish this, we estimate face normals n and n, and prune any frame where their cosine similarity se (e ng) exceeds threshold r, effectively removing views with large yaw, occlusion, or visibility loss. This keeps the gradient focused on meaningful biometric variation.
4.4 Temporal Fusion
While each individual biometric similarity score provides an instantancous assessment of the mis- match between the driving speaker and the target identity, per-frame mcasurements can be noisy inconclusive under challenging poses or brief occlusions. To address this, we aggregate similarity scores over window of W consecutive frames and feed these into LSTM. The LSTM learns to capture temporal patterns indicative of sustained mismatches characteristic of puppeteering attacks.
Let Ø = { .o,,..,, } be the sequence of similarity scores collected over W frames. We feed s into the LSTM, which outputs a final score representing the probability that the sequence puppeteered. We train the LSTM by minimizing the binary cross-entropy loss between y and ground-truth label 4, where = 1 if the video segment puppeteered and otherwise, such that
LISTM = -1 log(y) (1 () log(1 y)-
This learnable aggregator is novel in the context of puppeteering defense: previous methods either majority-vote over heuristic landmarks or revisit pixel space, both of which are fragile under com- pression artifacts. As the evaluation will show, the combination of EBL embedding and temporal fusion sets a new state of the art puppeteering detection without sacrificing latency.
5 Experiments and Results
5.1 Experimental Setup
Datasets. We conduct our experiments using the NVFAIR 11 pooled dataset because it incorporates together large set recorded video-conference calls in a controlled setting environment. This dataset includes three subsets: (1) NVIDIA VC [1] (natural video calls), (2) CREMA-D (55) (studio- recorded expressions), and (3) RAVDESS [54 (studio-recorded emotional speech). For each subset, we generate both self-reenacted (same identity) and cross-reenacted (puppeteered) videos using five sfate-of-the-art methods- LivePortrait (71), resulting in total of 15 dataset-method combinations. We note that the identities used for evaluation are strictly disjoint from those in training. Further details are provided in Tab. 3
(68), 3DFaceShop (67]. SDFR (70), EmoPortraits [69], and
Metrics. We report the average detection AUC to allow direct comparisons with prior work.
Competing Methods. As puppeteering detection a new problem, we compare against the state-of- the-art method by Vahdati et al. [10]. To highlight the distinction from deepfake detection, we also evaluate seven leading deepfake detectors: Efficient VIT [2]. CCE VIT 12, CNN Ensemble [3]
TALL [4). SupCon [15. CLRNet (72), and LAA-Net [16].
7

---

Table 2: Puppeteering attack detection performance measured in AUC of ours and competing methods when trained only on the NVIDIA-VC subset (In-domain) and tested on the CREMA-D [55 and RAVDESS [54] subsets
SDFnceshop 67
MUNET (s5
EMOPortralts (x])
SUFR 20
LivePortrait
7t]
Avg.
Method
In- domain
Cruse- demain
In-
Cruss domain
Cruss-
In-
Cross- dumain
In- demuin
Crees- domain
Cruse-
domain
thomain
domain
domain
domain
domain
Eflicient
12
.592
608
.059
.573
CNN Ensemble
614
629
605
589
.582
.516
600
.833
552
.395
.91
592
TALL [8]
.126
.799
.88
516
.562
.508
.003
.872
.61
.511
580
.81
SupCon 13
6.37
A69
.592
620
.599
.228
A62
.003
.644
.671
.22
A36
NET
A32
LAANNt
Vahdati
.920
.942
.890
903
Ours
.48
.914
.900
919
.036
.917
.951
.944
.989
.911
.945
.955
Table 3: Statistics of the datasets and generated data used this paper.
Table 4: Detection AUC of our proposed method and its alternative design choices.
Authoriizd Vidoos
# Puppeteered Videos
Component
Method
AUC
RER%
Dalaset
of
Proposed
0.966
IDs
No MLP Lower Dim. Pro-
NVIDIA-VC
46
41,761
45,212
Biometric Space
jection
(NVC)
Network Design
No Biometric Leakage Net- work
RAVDESS
10,560
1,300
11,880
(RAV]
Single Negutive Single Positive
CREMA-D
(CRD)
347,592
Training Proto-
0.749
86.45
No Biometric Contrastive Loss
No Extreme Pose Exclusion
0.929
52.11
We exclude Avatar Fingerprinting (11), which requires enrollment phase where each user submits authorized videos to generate identity fingerprints. This setup is incompatible with our framework, as removing enrollment disables its core mechanism and renders comparison unfair.
5.2 Experimental Results
In this section, we evaluate our method against state-of-the-az baselines for detecting puppeteered videos. We report results under scenarios: (1) domain shift- shift-training and testing all datasets-and (2) cross-domain generalization -ttraain only on NVIDIA VC [1] and evaluating
on unseen CREMA-D [55] and RAVDESS (44
datasets.
5.2.1 Performance on Combined Datasets
Tab. summarizes results from training and testing across all datasets (NVIDIA VC M. CREMA- [55], and RAVDESS [54)). Our method achieves AUC > 0.97 across all combinations, robustly detecting puppeteering across diverse identities, poses, and expressions. In contrast, deepfake detectors perform poorly; CLRNet (72), the strongest baseline, reaches only 0.684 AUC on MCNET- generated [68] videos from CREMA-D (55]. This highlights the limitation of deepfake detectors, which focus on distinguishing real from synthetic rather than authorized vs. unauthorized identities.
Our method also outperforms the closest prior work, Vahdati et al. 100 reducing relative error by 46% (AUC from 0.945 to 0.971). Notably, our minimum AUC remains above 0.95, compared to 0.90 for Vahdati et al. [10]. This gain stems from our enhanced biometric leakage space, which better separates identity from pose and expression cues. In contrast, their method relies on simple Euclidean distance, making it less robust such variations.
5.2.2
Generalization Performance
Tab. 2 presents results when training is limited to NVIDIA VC []] and testing is performed on CREMA-D (55) and RAVDESS [54]. Our method maintains strong performance with an average AUC of 0.925 only 5% drop from the no-domain-she setting-despite training on just 46 identities. This drop is consistent with Fig. which shows AUC scaling with the number of identities used in training. These results suggest the observed performance gap is due to identity diversity rather than differences in appearance or expression, confirming that our method scales with data and generalizes well to unseen domains.
8

---

6 Ablation Study
In this section, we conduct series of ablation experiments to understand the impact of different design choices on the detection performance of our method. To do this, we measured the average detection AUC and relative enror reduction (RER) over the self-reenacted and puppeteered examples across all datasets using 3DFaceShop [67 as the generator. The results are provided in Tab. 4
Biometric Leakage Space Network Design. The results in Jab.wshow that removing the biometric leakage embedding modules (h1, fi2) resulted in a substantial drop in performance. This finding confirms that simply comparing to f(It does not work. Additionally, we observe that increasing the MLP's output dimension to match or exceed the input drops performance to 0.827 AUC (an 80.35% increase in error), indicating that projecting embeddings into a lower-dimensional space effectively filters out irrelevant variability.
Training Protocol. Table 4 highlights three key sensitivities. Limiting each batch to a single positive-negative pair slashes AUC from 0.966 to 0.749 (+86% RER), proving multiple samples are vital for capturing fine biometric cues. Replacing our pose-conditioned contrastive loss with simple similarity regression lowers AUC to 0.788 (+84% RER), confirming the need optimise relative identity differences. Finally, retaining extreme-pont pose frames drops AUC to 0.929 (+52% RER), so filtering them is essential for peak accuracy.
Pose Sensitivity Analysis. We ablated pose sensitivity by progressively omitting frames whose head rotation exceeded set angle and tracking AUC on test set. Table 5 shows performance rising steadily as extreme poses are removed, peaking at 18 stricter cut-offs then hurt accuracy. Thus, judicious pose exclusion yields measurable detection gains.
Table Puppeteering detection AUC at increas- ing yaw thresholds. Performance peaks at 18%
Table 6: Robustness to facial appearance changes on NVIDIA VC (3DFaceShop).
Yaw (*) 133 18 113
118
Modification
AUC
AUC 0.932 0.938
0.953
0.966
None (baseline)
0.966
Eyeglasses
0.9393
Yaw (*) +:23
#:28
#33
+38
Piercings
0.9605
AUC 0,958 0.942 0.930 0,929
Makeup
0.9574
7 Discussion
Computational Efficiency. We benchmarked our method on an RTX 3090 GPU, where achieved on average 75 FPS-well above the 60 FPS real-time threshold while having under 1M total number of parameters. In comparison, Vahdati et al. [10] reached only 32 FPS under the same conditions. This speed advantage highlights the efficiency of our latent-space analysis, supporting real-time deployment in Al-based videoconferencingg
Temporal Window Size. Fig. 4 shows how detection performance varies with the LSTM's temporal window size. AUC improves sharply from 0.77 10 frames to 0.97 at 40 frames, after which gains plateau. A 40-frame window 1.33 at 30fps) captures sufficient temporal biometric information, highlighting the value of modeling frame-to-frame dependencies in latent space.
0.75
0.35
0.90
0375
0350
0825
0.80
0.75
10 20 30
Window Size (Frames)
30
Number of Puppeteering Examples
0
10
20 30 40
Figure 4: Detection AUC vs. window size and number of puppeteered identities during training. Scalability. Fig. 4 detection AUC as a function of the number of cross-reenacted identities used training on the NVIDIA VC dataset. Performance steadily increases from 0.75 with few identities to 0.96 with all 46, with no signs of saturation. This suggests that greater diversity during training

---

enhances the robustness and generalizability of our biometric embedding space in distinguishing puppeteering from self-reenactment.
Robustness to facial appearance changes. We tested our method's robustness to appearance changes by digitally adding eyeglasses, piercings, or heavy makeup to NVIDIA-VC identities using 3DoaceShop. As Table shows, AUC shifted modestly from 0.966 to 0.939 (glasses) and 0.961 (piercings), and even improved to 0.974 with makeup. NVFAIR's natural accessory and cosmetic diversity had already exposed the model to such variations, explaining the small changes and confirming real-world robustness.
Limitations, Our method has three main failure modes. First, extreme head rotation or occlusion (e.g., from hands objects) can obscure facial features critical to identity. Second, poor lighting or overexposure weakens the biometric signal captured by the camera. Third, motion blur-caused by rapid movement blinking consistency and impairs detection. Examples are provided in the supplementary material.
distort the latent representation. Each of these degrades biometric
Conclusion
We present real-time puppeteering defense that authenticates speakers directly in the transmitted pose-expression stream. A low dimensional biometric-leakage embedding, learned with novel pose-conditioned contrastive loss and reinforced by an LSTM for temporal fusion, plus extreme- pose filtering, lifts AUC beyond prior work and stays strong across domains and appearance edits. Ablations confirm every module's value, establishing a practical, robust safeguard for next-generation heai videoconferencingg systems.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No. 2320600. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This research was supported in part by NVIDÍA Corporation. We thank the NVIDIA Research team for their collaboration, resources, and valuable feedback throughout this project.
References
[1] Hang Chu, Shugao Ma, Fernando De la Torre, Sanja Fidler, and Yaser Sheikh, Expressive telepresence via modular codee avatars. European Conference on Computer Vision, pages 330 345. Springer, 2020.
[2] Guohao Li, Hongyu Yang. Yifang Men, Di Huang, Weixin Li, Ruijie Yang, and Yunhong Wang. Generating Editable Head Avatars with 3D Gaussian GANs. In 2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2025.
[3] Matthijs Van Der Boon, Lconor Fermoselle, Frank Ter Haar, Sylvic Biikstra-Sonddaiissaaae and Omar Niamut, Deep learning augmented realistic avatars for social VR human representation. In Proceedings of the 2022 ACM International Conference on Interactive Media Experiences, pages 31 311-318, 2022.
[4] Michael Stengel. Koki Nagano, Chao Liu, Matthew Chan, Alex Trevithick, Shalini De Mello. Jonghyun Kim, and David Luebke. Al-Mediated 3D Video Conferencing. In ACM SIGGRAPH Emerging Techmolo- gies, 2023, doi: 10. 14533588037.3555558
Shwetha Rajaram, Nels Numan, Balasaravanan Thoravi Kumaravel, Nicolai Marquardt, and Andrew D. Wilson. BlendScape: Enabling Unified and Personalized Vidco-Conferencing Environments through Generative AI. arXiv preprint 240.::2003.33444 2024.
Zhengang Li. Sheng Lin, Shan Liu. Songnan Li. Xue Lin, Wei Wang, and Wei Jiang. FAIVConf: Face Enhancement for Al-based Video Conference with Low Bit-rate. arXiv preprint arXiv:2207. 040000 2022.
[7] Ross Cutler, Ramin Mehran, Sam Johnson, Cha Zhang, Adam Kirk, Oliver Whyte, and Adarsh Kowdle. Multimodal Active Speaker Detection and Virtual Cinematography for Video Conferencing. arXiv preprint arXiv:2002.03977, 2020.
[8] Ting-Chun Chun Wang. Arun Mallya, and Ming- Yu Liu. One-Shot Free- View Neural Talking Head Synthesis for Video Conferencing, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10039- 10049, 2021. doi: 10.1109/CVPR44447.2200 2022144433.222.9999999 00991.
10

---

[9] Haonan Tong. Haopeng Li. Hongyang Du, Zhaohui Yang. Changchuan Yin, and Dusit Niyato. Mul- timodal Semantic Communication for Generative Audio-Driven Video Conferencing. arXiv preprint urXiv:2410.2111. 2024.
[10] Danial Samadi Vahdati, Tai Duc Nguyen, and Matthew C. Stamm. Defending www-Banwwidth Talking Head Videoconferencing Systems From Real-Time Puppeteering Attacks. Proceedings of the IEEF//VF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 983-992, June 2023.
[11] Ekta Prashnani, Koki Nagano, Shalini De Mello, David Luebke, and Orazio Gallo, Avatar fingerprinting authorized use of synthetic talking-hean videos. In European Conference on Computer Vision, pages 209-228. Springer. 2024.
[12] Davide Alessandro Coccomini, Nicola Mcssina, Claudio Gennaro, and Fabrizio Falchi. Combining efficientnet and vision transformers for video deepfake detection. In International conference on image analysis and processing, pages 219-229. Springer, 2022.
[13] Nicolo Bonettini, Edoardo Daniele Cannas, Sara Mandelli, Luca Bondi, Paolo Bestagini, and Stefano Tubaro. Video face manipulation detection through ensemble of cans. In 2020 25th international conference pattern recognition (ICPR), pages 5012-5019. IEEE, 2021.
[14] Yuting Xu, Jian Liang, Gengyun Jia, Ziming Vang, Yanhao Zhang, and Ran He. Tall: Thumbnail layout for deepfake video detection. In Proceedings of the IEEF/CVF international conference on computer vision, pages 22658-22668. 22662, 2023.
[15] Ying Xu, Kiran Raja, and Marius Pedersen. Supervised contrastive learning for generalizable and explainable deepfakes detection. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 379-389, 2022.
[16] Dat Nguyen, Nesryne Mejri, Inder Pal Singh, Polina Kuleshova, Marcella Astrid, Anis Kacem, Enjie Ghorbel, and Djamila Aouada. Laa-net: Localized artifact attention network for generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17395-17405, 2024.
and
[17] Tai D Nguyen. Shengbang Fang, and Matthew C Stamm. Videofact: detecting video forgeries using attention, scene context, and forensic traces. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8563-857 8573, 2024.
[18] Danial Samadi Vahdati, Tai D. Nguyen, Aref Azizpour, and Matthew C. Stamm. Beyond Deepfake Images: Detecting Al-Generated Videos. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pages 4397-4408
June 2024.
[19] Lichuan Yingqi Lin, Zhenhua Huang, Yan Han, Xiaogang Xu, Jiafei Wu, Chong Wang, and Zhe Liu. Distinguish Any Fake Videos: Unleashing the Power of Large-scale Data and Motion Features. arXiv preprint Xiv:::2405. 15343, 2024.
[20] Chirui Chang. Zhengzhe Liu, Xiaoyang Lyu, and Xiaojuan Qi. What Matters Detecting Al- Al-Generated Videos like Sora? arXiv preprint arXiv:2406. 19568, 2024.
[21] Rohit Kundu, Hao Xiong, Vishal Mohanty, Athula Balachandran, and Amit Joy-Chowdhury, Towards a universal synthetic video detector: From face or background manipulations to fully ai-generated content. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 28050-28060, 2025.
[22] Xuangeng Chu and Tatsuya Harada. Generalizable and animatable gaussian head avatar. Advances in Neural Information Processing Systems, 37:57642-556770 2024.
[23] Radek Dan&ček, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
pages
[24] Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2Head: Audio-driven Onc-shot Talking-head Generation with Natural Head Motion. In Proceedings of the 30th International Joint Conference Artificial Intelligence (IJCAI), pages 1107-1113, 2021. doi: 10.24963/ijcai.202111 2001//55.
[25] Xiaoqian Shen, Faizan Farooq Khan, and Mohamed Elhoseiny. Emotalker: Audio driven emotion aware talking head generation. In Proceedings of the Asian Conference Computer Vision, pages 1900-1917, 2024.
[26] Trevine Oorloff and Yaser Yacoob, Expressive Talking Head Video Encoding in StyleGAN2 Latent Space. arXiv preprint 2xXiv:2203.14512. 2022.

---

[27] Lele Chen, Guofeng Cui. Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. Talking-Head Generation with Rhythmic Head Motion. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35-51, 2020.
[28] Madhav Agarwal, Anchit Gupta, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. Compressing Video Calls using Synthetic Talking Heads. In Proceedings of the British Machine Vision Conference (BMVC), pages 1 1-12, 2021.
[29] Fa-Ting Hong. Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3397-3406, 2022.
[30] Fa- Fa-Ting Hong. Li Shen, and Dan Xu. Dagan++: Depth-aware generative adversarial network for talking head video generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):2997- 2023.
[31] Arun Mallya, Ting-Chun Wang, and Ming-Yo Yu Liu. Implicit warping for animation with image sets. Advances in Neural Information Processing Systems, 35:22438 22450, 2022.
[32] Yue Wu, Yu Deng. Jiaolong Yang. Fangyun Wei, Qifeng Chen, and Xin Tong. Anifacegan: Animatable 3d-aware face image generation for video avatars, Advances in Neural Information Processing Systems, 35:361 88-36201, 2022.
[33] Hyoung-Ky Kyu Song, Sang Hoon Woo, Junhycok Lec, Scungmin Yang. Hyunjac Cho, Youscong Lec, Dongho Choi, and Kang-wood Kim. Talking face generation with multilingual tts. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages (1425-21435 2022.
[34] Yue Gao, Yuan Zhou, Jinglu Wang. Xiao Li, Xiang Ming. and Yan Lu. High- fidelity and freely controllable talking head video generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5609-5619, 2023.
[35] Geumbycol Hwang. Sunwon Hong. Seunghyun Lee, Sungwoo Park, and Gycongsu Chac. DisCoHead: Audio-and- Video-Driven Talking Head Generation by Disentangled Control of Head Pose and Facial Expressions. arXiv preprint arXiv:2303.07663 2023.
[36] Haotian Wang. Yuzhe Weng, Yueyan Li, 7ilu Guo, Jun Du, Shutong Niu, Jiefeng Ma, Shan He, Xiaoyan Wu, Qiming Hu, Bing Yin, Cong Liu, and Qingfeng Liu. EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion. arXiv preprint arXiv:241 16722, 2024.
[37] Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, and Tiejun Zhao Interactive conversational head generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[38] Zhiling Ye, LiangGuo Zhang, Dingheng Zeng, Quan Lu,, and Ning Jiang. R2-Talker: Realistic Real- Talking Head Synthesis with Hash Grid Landmarks Encoding and Progressive Multilayer Conditioning. arXiv preprint arXiv:2312.05572 2023.
[39] Maxime Oquab, Daniel Haziza, Ludovic Schwartz, Tao Xu, Katayoun Zand, Rui Wang, Peirong Liu, and Camille Couprie. Efficient Conditioned Face Animation Using Frontally-Viewed Embedding. arXiv preprint 20X1v:2003.08765 2022.
[40] Pulkit Tandon, Shubham Chandak, Pat Pataranutaporn, Yimeng I. Liu, Anesu M Mapuranga, Pattic Maes, Tsachy Weissman, and Misha Sra. Tx12Vid Ultra-low bitrate compression of talking- videos via text. IEEE Journal on Selected Areas in Communications, 41(1):107-18,, 2022.
[41] Jason Lawrence, Dan B. Goldman, Supreeth Achar, Gregory Major Blascovich, Joseph G. Desloge, Tommy Fortes, Eric M. Gomez, Sascha Häberling, Hugues Hoppe, Andy Huibers, Claude Knaus, Brian Kuschak, Ricardo Martin-Brualla, Harris Nover, Andrew lan Russell, Steven M. Seitz, and Kevin Tong. Project Starline: A Hlgh-Fidelliy Telepresence System. ACM Transactions on Graphics (Proc. SIGGRAPH Asia). 40(6):242:1-244) 144:11, 2021. doi: 10.1145/3478513.34488
[42] Gil Knafo and Ohad Fried. FakeOut: Leveraging Out-of-domain Self-supervision for Deepfake Detection. arXiv preprint arXiv:2212..0777, 2022.
Video
[43] Shu Hu, Yuezun Li, and Siwei Lyu. Exposing GAN generraee faces using inconsistent corneal specular highlights. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 155002550 IEEE, 2021.
12

---

[44] Shruti Agarwal, Hany Farid, Ohad Fried, and Maneesh Agrawala. Delecting deep-fake videos from phoneme-viseme mismatches. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 660-661, 2020.
[45] Hua Qi, Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Wei Feng, Yang Liu, and Jianjun Zhao. Deepthythm: Exposing deepfakes with attentional visual heartbeat rhythms. In Proceedings of the 28th ACM international conference on multimedia, pages 4318 4327, 2020.
[46] Umur Aybars Ciftci, Ilke Demir, and Lijun Yin. Fakecatcher: Detection of synthetic portrait videos using biological signals. IEEE transactions on pattern analysis and machine intelligence, 2020.
[47] Wenxuan Wu, Wenbo Zhou, Weiming Zhang. Han Fang, and Nenghai Yu. Capturing the lighting inconsistency for deepfake detection. In International Conference on Artificial Intelligence and Security, pages 647. Springer, 2022.
[48] Kaiyue Tian, Chen Chen, Yichao Zhou, and Xiyuan Hu. Illumination enlightened spatial-temporal inconsistency for deepfake video detection. In 2024 IEEE International Conference Multimedia and Expo (ICME), pages 1-6. IEEE, 2024.
[49] Zhimao Lai, Jicheng Li, Chuntao Wang. Jianhua Wu, and Donghua Jiang. LIDeepDet: deepfake detection via image decomposition and advanced lighting information analysis. Electronics, 13(22):4466, 2024.
[50] Andrea Ciamarra, Roberto Caldelli, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. Deepfake detection by exploiting surface anomalies: the SurFake approach. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 1024-1033, 2024.
[51] Boyuan Liu, Xin Zhang, Hefei Ling, Zongyi Li, Runsheng Wang. Hanyuan Zhang, and Ping Li. Aim-bone: Texture discrepancy generation and localization for generalized deepfake detection. IEEE Transactions on Biometrics, Behavior, and ldentity Science, 7(3):222-433 2025.
[52] Zhiqing Guo, Gaobo Yang, Jiyou Chen, and Xingming Sun. Exposing deepfake face forgeries with guided residuals. IEEE Transactions on Multimedia,
2023.
[53] Xin Li, Rongrong Ni, Pengpeng Yang. Zhiqiang Fu, and Yao Zhao. Artifacts- disentangled adversarial learning for deepfake detection. IEEE Transactions on Circuits and Systems for Video Technology, 33(4): 1658-1670, 2022.
[54] Steven R Livingstone and Frank A Russo. The Ryerson Audio- Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set facial and vocal expressions in North American English. PloS one, ((5))c0196391 2018.
[55] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions affective computing. 5(4):377-390, 2014.
[56] Shuling Zhao, Fa-Ting Hong, Xiaoshui Huang, and Dan Xu. Synergizing Motion and Appearance: Multi- Scale Compensatory Codebooks for Talking Head Vidco Generation. arXiv preprint arXiv:2412.007/9 2024.
[57] Shuai Tan, Bin Ji, Yu Ding, and Ye Pan. Say anything with any style. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5088-5096, 2024.
[58] Haoyu Ma, Tong Zhang, Shanlin Sun, Xiangyi Yan, Kun Han, and Xiaohui Xie. CVTHead: One- shot controllable head avatar with vertex-feature transformer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6131 1131-6141, 2024.
[59] Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning dynamic facial radiance fields for few-shot talking head synthesis. In European conference on computer vision, pages 666-682. Springer, 2022.
[60] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang. Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion cocfficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pages 8652 8661, 2023.
[61] Yaohui Wang. Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. arXiv preprint arXiv:2203. 09043, 2022.
13

---

[62] Madhav Agarwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. Audio-visual face reenactment. In Proceedings the IEEE/CVF winter conference applications computer vision, pages 5178-5187, 2023.
[63] Hao Wang. Yitong Wang. Zheng Zhou, Xing Ji. Dihong Gong. Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018.
[64] Minchul Kim, Anil K Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattrrn recognition, pages
2022.
[65] Yunwen Lei, Tianbao Yang, Yiming Ying, and Ding- Xuan Zhou. Generalization analysis for contrastive representation learning. In International Conference on Machine Learning, pages 19200-19227 PMLR, 2023.
[66] Daniel Rho, TaeSoo Kim, Sooill Park, Jachyun Park, and JaeHan Park. Understanding Contrastive Learning Through the Lens of Margins, arXiv preprint arXiv: /526, 2023.
[67] Junshu Tang. Bo Zhang, Binxin Yang. Ting Zhang, Dong Chen, Lizhuang Ma, and Fang Wen. 3DFaceShop: Explicitly Controllable 3D-Aware Portrait Generation. IEEE Transactions Visualization and Computer Graphics, 30(9);6020 6037, 2024. doi: 10.1 N0.1009//CC.
[68] Fa-Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation network for talking head video generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23062-23072, 2023.
[69] Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis. and Maja Pantic. Emoportraits: Emotion-cnhanced multimodal one-shot head avatars. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8498-8507, 2024.
[70] Stella Bounareli, Vasileios Argyriou, and Georgios Tzimiropoulos. Finding Directions in GAN's Latent Space for Neural Face Reenactment. British Machine Vision Conference (BMVC), 2022.
[71] Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03166 2024.
[72] Shahroz Tariq, Sangyup Lee, and Simon S Woo. A Convolutional LSTTM based Residual Network for Deepfake Video Detection. arXiv preprint arXiv:2009.07488) 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Answer: [Yes]
Justification: All scientific claims are stated in Sees. 1-4; empirical claims are validated in Sec. 5 and Tab. 1-4.
2. Limitations
Answer: [Yss]
Justification: Sec. 7 "Limitations" details failure modes (extreme pose, lighting, motion blur).
3. Theory assumptions and proofs
Answer: [NA]
Justification: Work is empirical; no formal theorems are presented.
4. Experimental result reproducibility
Answer: [Yes]
Justification: Hyper-parameters, random seed, and training protocol given Sec. 5.1; code will be released.
5. Open access to data and code
Answer: Yess]
Justification: Public datasets (NVFAIR) are cited; inference/training code and weights will provided upon acceptance.
6. Experimental setting/details
Answer: [es]]
Justification: Dataset splits, generators, and evaluation metrics described in Sec. 5.2-5.3.
7. Experiment statistical significance
Answer: [Yes]
Justification:
8. Experiments compute resources
Answer: [Yes]
Justification: Sec. 5.1 states training cost (sx18 GPU-hours on a single RTX 3090) and runtime (75 FPS).
9. Code of ethics
Answer: [Yes]
Justification: Research follows the NeurlPS Code of Ethics; see Broader-Imppct section.
10. Broader impacts
Answer: [Yes]
Justificatio:: Societal impacts, misuse potential, and mitigation discussed in Broader- Impact paragraph.
Safeguards
Answer: [Yes]
Justification: Release plan includes watermarking detector outputs and open-soureing under non-commercial license.
12. Licenses for existing assets
Answer: [Yes]
Justification: We have properly attributed the work of others and have followed licensing and usage terms.
New assets
Answer: [NA]
Justification: No new dataset or proprictary model weights created: only derived metrics.
Crowdsourcing and research with human subjects
Answer: [NA]
Justification: Work uses publicly available datasets; no new human subject data collected.
Question: Does the paper describe potential risks incurred by study participants, whether
such risks were disclosed to the subjects, and whether IRB approvals were obtained?
15

---

Answer: [NA]
Justification: Not applicable (no new human-subjects research).
15. Declaration of LLM usage
Answer: [NA]
Justification: No large language model is original non-standard part of the core methodology.
16


Reviews:
[{"review_id": "JzBhhhddDa", "reviewer": "Reviewer_QoDj", "summary": "The paper addresses the problem of puppeteering attacks in AI-based talking-head videoconferencing, where an adversary hijacks a legitimate user’s reference embedding to impersonate them in real time. The authors propose to exploit subtle biometric cues inadvertently encoded in the pose and expression latent, learning a low-dimensional embedding (in the EBL space) via a pose-conditioned contrastive loss and an LSTM-based temporal fusion that flags identity mismatches via a cosine test, without ever reconstructing RGB frames or enrolling users. They report high AUC (> 0.95) on three datasets (NVIDIA-VC, CREMA-D, RAVDESS) and claim real-time operation (75 FPS).", "questions": "1. Limited Novelty: The core idea of the paper is a direct application of margin-based embedding such as CosFace and ArcFace. There is substantial  technical innovation beyond simply matching positives and \"\"pose-matched\" negatives (Eq. 5). The pose-conditioned large-margin contrastive loss i\"s superficially novel and merely re-weights standard contrastive objectives. The margin guarantee as shown in \"Proposition 1\" is a trivial consequence of spherical geometry and adds little theoretical insight.\n2. The experiment to show cross-domain robustness trains the model on NVIDIA-VC subset and tests it on CREMA-D/RAVDESS. However, the drop in AUC from 0.966 to 0.925 (Table 2) is non-trivial, and without benchmarks on truly \"in-the-wild\" videoconferencing data (e.g., Zoom/Teams logs), claims of real-world generalization are overstated. Can you include some \"in-the-wild\" results ?\n3. The paper omits Avatar Fingerprinting  from its comparisons on the grounds that it \"requires enrollment,\" but this seems to dismiss a key alternative approach. A more balanced evaluation would include it, perhaps under an “oracle enrollment” assumption, to better contextualize the proposed method.\n4. The authors claim real-time FPS of 75 on an RTX 3090, but modern videoconferencing must run on mobile/CPU devices. Performance on embedded or CPU-only hardware is not evaluated. Moreover, the LSTM temporal window of 40 frames (~1.3 s at 30 FPS) imposes a non-negligible detection latency, which is skipped in the \"real-time\" discussion (Section 7/8).\n5. Some hyperparameters such as λ in Equation (6), and γ in equation (7), and embedding dimension are not systematically explored.\n\n\nI request the authors to resolve my above queries for the rebuttal. I am inclined towards the positive side but still have doubts regarding the over-claimed FPS and novelty.", "limitations": "Yes, they have included in the supplementary.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Motivation of the paper is clear and is the first work to use latent representations without reconstructing RGB frames to detect puppeteering attacks.\n2. The authors claim real-time performance (75 FPS), which is a drastic improvement from 32 FPS of the previous baseline.\n\nWeakness:\n1. The core idea of applying a contrastive loss to disentangle identity from pose/expression, is essentially a direct application of well-established hyperspherical embedding methods (such as CosFace, ArcFace)\n2. The real-time FPS is over-claimed.\n3. In-the-wild results are missing.\n4. Some hyper-parameter exploration missing.", "was_revised": true, "final_justification": "The paper provides a real-time method for leveraging biometric leakage for detecting puppeteering. I had some queries which the authors have clarified in the rebuttal. I therefore, changed my rating accordingly.", "rebuttal": "_**Comment (Question 1):** Concerns regarding novelty.  Specifically, the reviewer suggests that the core idea closely follows existing margin-based embedding methods such as CosFace and ArcFace, with limited novelty in the proposed pose-conditioned large-margin contrastive loss._\n\nWe appreciate the reviewer’s question regarding novelty. We agree that our method leverages a margin-based embedding framework and that contrastive objectives have been used in other prior works (e.g., ArcFace, CosFace). However, as with those prior works, the contribution is not simply using a contrastive loss but rather _what we apply it to and how we make it work for this problem._\n\nOur novelty is not that we trivially adapt a margin-based embedding. It is that we identify a previously unrecognized biometric side channel in pose-expression embeddings, and we design a training protocol and inference pipeline that make it possible to reliably and efficiently exploit that side channel for real-time puppeteering detection.\n\nSpecifically, our key novel contributions are:\n\n**(1) Identifying and exploiting a new biometric side channel for protecting against puppeteering attacks.**\n\nThis is novel because such leakage had previously been considered a liability (a cause of identity entanglement), whereas we repurpose it as a security signal. In this paper, we first show for the first time that pose-and-expression embeddings used in AI-based talking-head videoconferencing systems unintentionally leak persistent biometric identity cues. We then demonstrate that this “biometric leakage” can be reliably extracted and used to detect puppeteering attacks - without decoding RGB frames or enrolling users.\n\n**(2) A training protocol that makes biometric leakage usable for authentication.**\n\nDirectly applying a standard contrastive loss to pose-expression embeddings fails because differences in pose/expression dominate the subtle biometric signal (see Figure 3). Our protocol introduces a _pose-matched positive/negative pairing strategy_ that actively controls for pose/expression during training.\n\nThis is technically important because it requires generating pose-matched negatives and positives in a way that is essentially impossible to achieve by simply visually matching real data. Without such control, neither our method nor standard ArcFace/CosFace-style training is effective in this domain.\n\nTo demonstrate this point, we conducted an experiment where we retrained ArcFace and CosFace directly on pose-expression embeddings from the NVIDIA‑VC dataset using both standard training, as well as our proposed training protocol.  Below, we present the AUCs obtained by averaging performance over 5 SOTA talking head generation systems.\n\n| Method | Standard Training | + Our Training Protocol |\n|---|---|:---:|\n| CosFace | 0.60 AUC | 0.92 AUC |\n| ArcFace | 0.56 AUC | 0.91 AUC |\n| Ours | 0.64 AUC| 0.97 AUC |\n\nThese results show that without our proposed training protocol, both ArcFace and CosFace perform poorly in detecting puppeteering. With our protocol, both improve substantially. _**This confirms that the novelty lies in how we adapt the margin-based embedding framework to this problem.**_ Our full method further outperforms these adapted versions because it incorporates additional design choices (e.g. extreme-pose filtering, temporal fusion) tailored to maximize extraction of biometric leakage.\n\n_**Comment (Question 2):** Request for benchmarking our system’s performance using “in-the-wild” videoconferencing data._\n\nThank you for this comment. We agree that demonstrating robustness on “in‑the‑wild” videoconferencing streams is important. \n\nTo this end, we conducted an additional “in-the-wild” benchmarking experiment in which four members of our lab each recorded five one‑minute Zoom calls under natural lighting and bandwidth fluctuations. For every call we produced both self‑reenactmed and puppeteered videos using five SOTA talking head generators (3DFaceShop, MCNet, EmoPortraits, SDFR, LivePortraits). This yielded 100 self-driven and 100 puppeteered videos that were never seen during training. \n\nNext, we used our system as described in Section 5.2.2 of our paper and evaluated its puppeteering detection performance on this new dataset.  Importantly, this was exactly the system used to produce the results in Table 2 of our paper and was trained solely on the NVIDIA VC subset.\nThe “in-the-wild” performance of our system is presented in the table below:\n\n| Generator | AUC |\n|---|---|\n| 3DFaceShop | 0.94 |\n| MCNet | 0.97 |\n| EmoPortraits | 0.92 |\n| SDFR | 0.97 |\n| LivePortraits | 0.98 |\n| Average | 0.96 |\n\nDespite codec differences, camera motion, and variable compression, our system retains ≥ 0.92 AUC for every generator and averages 0.955, outperforming the cross‑domain result (0.925) already reported in Table 2. These numbers bolster the claim that EBL features generalize to real videoconference traffic without fine‑tuning. We will incorporate this dataset description and the above table into the supplementary materials in the camera‑ready. \n\n_**Comment (Question 3):** Request for comparison to Avatar fingerprinting under “oracle enrollment” in which all users would be enrolled prior to evaluation._\n\nWe appreciate the reviewer’s suggestion to compare with Avatar Fingerprinting (AF). We agree that a side‑by‑side comparison can further clarify the strengths and benefits of our enrollment‑free approach.\n\nOur original decision to omit AF was motivated by problem‑setting asymmetry: AF requires a per‑user enrollment session during which a unique fingerprint is extracted and stored.  By contrast, our defense is designed to operate in “zero setup” conditions where new/unknown users can use videoconferencing without pre-enrollment. Including AF in Table 2 would therefore have been unfair in both directions - it would penalize the enrollment‑free baselines yet still undervalue AF by denying it the enrollment it requires.\n\nTo fully address the reviewer's concern, we have conducted an additional experiment in which we allowed AF to have an oracle enrollment period (meaning all users were enrolled ahead of the evaluation period).  Since AF is not publicly available or accessible, we cannot retrain it on our datasets presented in the main paper. However, we were able to make a fair comparison by benchmarking our method’s performance against AF on the identical set of data generated using LIA in AF’s main experiment.\n\nIn this experiment, our proposed method was able to achieve a detection AUC of 0.87.  This outperforms AF’s reported AUC of 0.84, demonstrating that our system can offer benefits over AF in terms of both detection performance and lack of an enrollment requirement.  We note that while our methods AUC in this experiment is lower than our average reported numbers in Table 1 and 2, LIA is an older generator. It only transmits a limited set of facial landmarks during operation, yielding noisier embeddings than the rich embedding vectors used by modern reenactment engines that were presented in our paper. We will amend the supplemental materials in the camera-ready version to reflect this comparison.\n\n_**Comment (Question 4):** Request to evaluate FPS performance on mobile/CPU devices and to address detection latency introduced by LSTM._\n\nThank you for your comment.  We agree that benchmarking our runtime on mobile/CPU devices will provide more insight into our method’s ability to run in real time.\n\nTo address this comment, we benchmarked our method (including the 40‑frame LSTM) on two consumer laptops released in 2021. One laptop had no dedicated GPU and utilized an AMD Ryzen 9 5900HX CPU. The other laptop had the same CPU, but was fitted with a mobile NVIDIA RTX 3070 Ti GPU. The results are presented in the table below: \n\n| Platform | Inference Speed (FPS) |\n|---|---|\n| CPU Only | 27 |\n| CPU + Mobile GPU | 64 |\n\nThese results show that our detector obtained 27 FPS on the CPU‑only machine, matching typical videoconference frame rates (≈ 25 FPS on low‑bandwidth Zoom). Using an older mobile‑grade GPU it reaches 64 FPS. This is within 15 % of the 75 FPS reported on a desktop RTX 3090, demonstrating favourable compute scaling.  We further note that our code has not been optimized for runtime, and further speed gains can likely be achieved using a more efficient implementation.\n\nWe agree that the latency introduced by the LSTM should be addressed. We will note in the paper that the LSTM incurs a one-time 1.33 s delay before detection begins, after which detection lags by 1.33 s due to the LSTM window. Since puppeteering typically persists throughout a video rather than occurring in isolated segments, this latency should not significantly impact real-world applicability.\n\n_**Comment (Question 5):** Request to systematically explore hyperparameters._\n\nThank you for your comment. We did perform a systematic exploration of our hyperparameters during system development to select values that yielded the best performance, but we regret that these results were not included in the original manuscript. We apologize for this oversight.\n\nTo identify optimal values for 𝜆, 𝛾, and d, we conducted experiments in which we fixed two of these hyperparameters while varying the third. This process was repeated for each hyperparameter, allowing us to evaluate their individual effects on performance. The results of these experiments are provided below and will be incorporated into the supplemental materials for the camera-ready version.\n\n| λ Value | AUC |\n|---|---|\n| 0.1 | 0.95 |\n| 0.3 | 0.96 |\n| 0.5 | 0.97 |\n| 0.7 | 0.97 |\n| 0.9 | 0.97 |\n\n| γ Value | AUC |\n|---|---|\n| 0.1 | 0.96 |\n| 1 | 0.97 |\n| 10 | 0.97 |\n| 50 | 0.96 |\n\n| d Value | AUC |\n|---|---|\n| 64 | 0.94 |\n| 128 | 0.96 |\n| 256 | 0.97 |\n| 512 | 0.96 |\n\nFrom these results, we found that the optimal performance of our system was achieved with the hyperparameter configuration reported in our submission (𝜆=0.5, 𝛾=1, 𝑑=256)."}, {"review_id": "4odis3M7XS", "reviewer": "Reviewer_CKta", "summary": "This paper proposes a method to defend against puppeteering attacks in bandwidth-constrained videoconferencing systems. In a puppeteering attack, an adversary on the sender side transmits an unauthorized representation of a different target speaker when a video call is initiated. As a result, the identity reconstructed by the receiver differs from the actual person controlling the video stream. The proposed approach detects such attacks by verifying whether the identity information contained in the pose-and-expression vector is consistent with the identity in a reference frame.", "questions": "The proposed defense method essentially formulates the problem as a face recognition task, with the main difference being that, instead of using RGB images as input (as is common in standard recognition settings), the input in this scenario is a vector representing pose and expression. I am curious whether similar defense performance could be achieved by using a standard recognition training approach, simply replacing the RGB images with pose–expression vectors, combined with the hard negative data synthesis method proposed in the paper. It would be helpful if the authors could discuss or provide experimental results on this baseline to better isolate the contribution of their proposed method.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \\\n(1) The proposed defense against puppeteering attacks in bandwidth-constrained videoconferencing systems is intuitive and appears to be effective. The method is well-aligned with the practical requirements and constraints of such systems, and the design choices are clearly justified. \\\n(2) The experimental results provided in the paper demonstrate the effectiveness of the proposed method in defending against puppeteering attacks. \\\n(3) The paper is well-structured and clearly organized.\\\nWeaknesses:\\\n(1) It appears that the variables k  and  p may have been swapped in Equation 2. \\\n(2) The proposed defense method essentially formulates the problem as a face recognition task, with the main difference being that, instead of using RGB images as input (as is common in standard recognition settings), the input in this scenario is a vector representing pose and expression. However, the paper lacks a comparison with standard recognition methods as baselines. In particular, it would be helpful to include results using established approaches such as CosFace or similar methods, to better contextualize the performance of the proposed approach.\\\n(3) In Section 4.2.1, the authors state that using two separate heads leads to better performance. However, the ablation study only compares the presence and absence of the head module, and does not provide a direct comparison between using a single head and two heads. \\\n(4) Using LSTM for temporal information fusion is a standard practice in the field. The paper would benefit from a discussion of why LSTM was chosen over other temporal modeling approaches.", "was_revised": true, "final_justification": "After reading the author's response, I decided to raise my score to accept", "rebuttal": "_**Comment (Weakness 1):** Equation 2’s variables may have been swapped_\n\nThank you for catching the mistake in Equation 2 of our main paper. We sincerely apologize for the confusion this oversight may have caused and will correct it in the camera-ready version.\n\nThe correct text in lines 108–109 should read:\n\n> *An adversary, speaker $\\ell$, obtains a target speaker’s representations $R^k$ and substitutes the driving video for the target speaker $k$, $V_t^k$, with their own video $V_t^\\ell$, thus replacing the target speaker’s pose and expression vectors $z_t^k$ with their own pose and expression vector $z_t^\\ell$.*\n\nAccordingly, the correct form of Equation 2 is:\n\n$$\n\\hat{V}_t^{k \\rightarrow \\ell} = g(f(V_t^\\ell), R^k) = g(z_t^\\ell, R^k)\n$$\n\nWe thank the reviewer for pointing out this issue and will ensure it is clearly corrected and explained in the camera-ready version.\n\n_**Comment (Weakness 2 & Question):** Direct comparison with standard facial recognition baselines_\n\nThank you for pointing out that a direct comparison with standard facial recognition baselines like CosFace can help in contextualizing our method’s performance and better highlight our paper’s novel contributions. \n\nTo accomplish this, we conducted an additional experiment in which we evaluate the puppeteering detection performance of CosFace and ArcFace in two scenarios: (i) applied directly to the same pose-and-expression embeddings that our method utilizes, and (ii) further adapted these methods to be compatible with our proposed training protocol. For both of these scenarios, we utilized the same experimental setup as in Table 2 of our main paper where we trained only on the NVIDIA-VC subset and evaluated on the CREMA-D and RAVDESS subsets. We then reported the average detection performance over both subsets. The result of this experiment is shown in the table below:\n\n| Method | Scenario | Avg. AUC |\n|:---|:---|:---:|\n| CosFace | Direct application | 0.61 |\n| ArcFace | Direct application | 0.56 |\n| CosFace | Adapted to use our training protocol | 0.93 |\n| ArcFace | Adapted to use our training protocol | 0.91 |\n| Ours | Our training protocol | 0.97 |\n\nFrom these results, we observe that a direct application of standard facial recognition baselines like CosFace and ArcFace obtains substantially lower performance than our proposed method. While their performance is improved when CosFace and ArcFace are adapted to utilize our training protocol, it is still lower (0.93 vs. 0.97) than our method. These results show that (1) facial recognition methods do not distinguish between identities when directly examining the pose-and-expression embeddings, (2) our proposed training protocol provides substantial performance gains when applied to multiple systems due to its ability to enable networks to control for pose and expression differences, and (3) our method’s Enhanced Biometric Leakage  space and network design are better at detecting puppeteering attacks than competing facial recognition approaches.\nThank you for requesting this additional validation. We will include these important results and discussions in the camera-ready version of our main paper.\n\n_**Comment (Weakness 3):** The ablation study only compares the presence and absence of the head module, and does not provide a direct comparison between using a single head and two heads._\n\nThank you for your comment. We believe this concern may stem from a misunderstanding due to a lack of clarity in how we described our ablation study. We address this below and will revise our camera-ready version to resolve the issue.\n\nOur method operates using two pose/expression embeddings that contain biometric information: Head 1 corresponds to the sender driving the video, and Head 2 corresponds to the reference frame used to reconstruct the video at the receiver. Puppeteering is detected by identifying biometric mismatches between these two heads. As stated in Section 4.2.1, two heads are essential—biometric mismatch cannot be identified without comparing biometric information between them. Information from a single head is insufficient to detect a mismatch between the speaker at the sender and the reconstructed identity at the receiver.\n\nThe confusion may have arisen from the way we described our ablation, specifically when we wrote: “The results in Tab. 4 show that removing the biometric leakage embedding modules (h1, h2) resulted in a substantial drop in performance.” This wording unintentionally implies that we removed the head modules entirely and attempted to measure biometric mismatch without using any heads, which could also be interpreted as suggesting that a single-head configuration could be evaluated.\n\nIn reality, our ablation did not remove the heads. Instead, it removed the two MLPs that project the information from the two heads into a common enhanced biometric leakage space. Without this projection module, the embeddings from the two heads were compared directly. The goal of this ablation was to demonstrate the importance of the enhanced biometric leakage space projection, not to imply that our method can operate without two heads.\n\nWe apologize for the confusion this wording caused and will clarify this point in the camera-ready version.\n\n\n_**Comment (Weakness 4):** Discussion regarding choice of temporal information fusion_\n\nWe appreciate the reviewer’s comment. The primary focus of our work is on the discovery of biometric leakage from the pose-and-expression embedding, the design of the enhanced biometric leakage (EBL) representation, and the associated training protocol and loss function. While temporal information fusion is an important component, it is not the core novelty of our approach.\n\nWe did evaluate different temporal fusion strategies during development and found that LSTM-based fusion provided stronger performance gains than simple temporal moving average methods. Importantly, the LSTM was able to achieve real-time performance on consumer hardware, aligning well with our design goals for practical deployment.\n\nGiven the strong overall system performance and our emphasis on the primary contributions noted above, we prioritized our research efforts toward those core components rather than conducting a comprehensive comparative study of temporal modeling approaches. We agree, however, that exploring alternative temporal modeling strategies could be an interesting direction for future work, and we will note this in the revised version of the paper."}, {"review_id": "Z9tHof1ef2", "reviewer": "Reviewer_8PG6", "summary": "This paper focuses on preventing adversarial attacks in ai-based video conferencing (where essentially the pose-expression vectors control the generation of a video on the receiver side, thus reducing bandwidth). The main advantage of their method is that it doesn't require any video being sent to the receiver, and it generalizes to new people (old methods do not), i.e. the new person controlling the video doesn't need to have pre-recorded data about themselves to eliminate the likelihood of an adversarial attack. To do this, they learn a latent space that extracts biomarkers from pose-expression data via a novel pose-matched negative pair creation method, and compare the biomarkers of the person controlling it to the person being generated to ensure consistency. Detection happens in real-time.", "questions": "1. I don't follow exactly what the deepfake detectors were tasked to do and why they were included? As mentioned in the paper, all videos are fake? I don't think this measures the same thing as the puppeteering attack ofcourse. \n\n2. It might be the case that if you were to do contrastive learning with a sufficiently large dataset of identities, both the biomarkers and the identity information *in the way that people move* would be sufficient to generalize to new people. I didn't find any discussion on this, but this could be just me missing some information.\n\n3. How robust is this method against adaptive adversaries who might use generative models or adversarial perturbations to transform their pose-expression embeddings to mimic the target's biometric signature? Have you considered attacks where someone trains a model to map their own biometric leakage patterns to match the victim's distribution? How would this method have to change when users start doing that?", "limitations": "While technical limitations were addressed, social limitations were not.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Neat way of identifying mismatched controller / generated persons on *new people* by leveraging the same information that the classifier-based method used, simply by being smart about the negative pairs used in the contrastive learning loss. \n2. The paper is well written, and well motivated. \n3. There are smart engineering decisions, such as the compactness of the representation so its latency-free (i.e. real-time). \n\nWeaknesses:\n\n1. The authors decided to remove the main baseline, Avatar Fingerprinting, citing that \"the setup is incompatible with our framework, as\nremoving enrollment disables its core mechanism and renders comparison unfair\". However, I think its pretty crucial to understand *when* to use which method. For instance, I would recommend that they use the enrollment process with Avatar Fingerprinting for a portion of people, and compare against their method (how well does this paper's method compare to avatar fingerprinting that has the advantage of the enrollment data?). Ofcourse, when you don't have enrollment data, this paper (so far) is the only option. But, it is still interesting to know how well the two methods do relative to each-other when you do have that data.", "was_revised": true, "final_justification": "I maintain my recommendation of accept.", "rebuttal": "_**Comment (Weakness 1):** Request inclusion of a comparison with Avatar Fingerprinting and the analysis of use cases for both Avatar Fingerprinting and proposed approach._\n\nWe appreciate the reviewer’s suggestion to compare with Avatar Fingerprinting (AF). We agree that a side‑by‑side comparison can further clarify the benefits (or potential weaknesses) of our enrollment‑free approach.\n\nTo fully address the reviewer's concern, we have conducted an additional experiment in which we allowed Avatar Fingerprinting to have an “oracle enrollment period,” meaning all users were enrolled ahead of the evaluation period.  We note that this experiment potentially unfairly penalizes our algorithm, since due to its problem setup, it is not allowed to exploit information that could be learned during this enrollment period.\n\nSince Avatar Fingerprinting is not publicly available or accessible, we could not retrain it on our datasets presented in the main paper to make a direct comparison. However, we were able to make a comparison by benchmarking our method’s performance against Avatar Fingerprinting on an identical dataset.  Specifically, we evaluated our method’s performance on NVIDIA-VC data generated using the generator LIA used in Avatar Fingerprinting’s main experiment.\n\nIn this experiment, our proposed method was able to achieve a detection AUC of 0.87.  This outperforms AF’s reported AUC of 0.84, demonstrating that our system can offer benefits over AF in terms of both detection performance and lack of an enrollment requirement.  We note that while our methods AUC in this experiment is lower than our average reported numbers in Table 1 and 2, LIA is an older generator. It only transmits a limited set of facial landmarks during operation, yielding noisier embeddings than the rich embedding vectors used by modern reenactment engines that were presented in our paper. We will amend the supplemental materials in the camera-ready version to reflect this comparison.\n\n_**Comment (Question 1):** Why are comparisons with deepfake detectors included considering the fact that all videos are fake._\n\nWe thank the reviewer for the question. We agree that traditional deepfake detectors are not designed to detect puppeteering attacks.\n\nWe included these experimental comparisons to help readers unfamiliar with this problem to understand the difference between puppeteering attack detection and deepfake detection.  As our experiments show, deepfake detectors perform roughly on par with a random guess when performing puppeteering detection.  This helps readers from outside this research area understand that differentiating between real videos and deepfake videos is a distinct problem from differentiating between self-reenacted and puppeteered videos, both of which are fully synthetic.\n\n_**Comment (Question 2):** If you were to do contrastive learning with a sufficiently large dataset of identities, would both the biomarkers and the identity information in the way that people move be sufficient to generalize to new people?_\n\nThank you for this comment. Given enough reference training data from a particular identity, it is in fact possible to learn the unique biomarkers and the identity information in how people move. This is the approach explicitly taken by Avatar Fingerprinting.  The experimental results presented in that paper demonstrate that movement biometrics can help expose puppeteering attacks.  \n\nHowever, this approach explicitly requires an enrollment period where a significant amount of movement data is recorded and used to train an identity-specific detector.  Without this enrollment period, Avatar Fingerprinting and techniques based on movement biometrics cannot work.\n\nOur method is designed to operate without an enrollment period requirement, thus enabling it to be applicable in a wider set of videoconferencing scenarios.\n\n_**Comment (Question 3):** How robust is the paper against adaptive adversarial attacks that specifically transform their pose-expression embeddings to mimic target biometric signature, or a model trained to map their own biometric leakage patterns to match victim’s distribution?_\n\nThank you for this comment.  We agree that understanding this system’s vulnerability to adversarial countermeasures is important.  However, this involves a detailed theoretical investigation, as well as thorough accompanying experimental analysis.  As a result, this is beyond the scope of this work.\n\nSpecifically, this would involve developing new attack strategies, creating new datasets, and running multiple experiments under several conditions to effectively understand this system’s vulnerability to attack. Unfortunately, we do not have the space to include this in this work.\n\nWe do, however, believe that this is an important topic of study and plan to address this in a separate future work."}, {"review_id": "PpTjibjqtE", "reviewer": "Reviewer_A3RD", "summary": "The paper proposes a defense method based on puppeteering attack, which identifies potential puppeteering attacks by introducing the latent representation $z_t$ of each frame of a synthetic video and a fixed image $R$ into the EBL space to calculate the similarities of identity features. Meanwhile, contrastive learning facilitates the decoupling of relevant identities and irrelevant pose-expression, enhancing the identification of puppeteering attacks. Experimental results demonstrate the accuracy and robustness of the proposed method. It holds some application value for AI-based low-bandwidth talking-head videoconferencing, capable of defending against potential attacks.", "questions": "The paper has the following issues:\n\n**Writing**:\n\n1.\tFig. 1 is too small, which affects readability. The authors should consider enlarging the image.\n2.\tFor Section 3.3, I believe it could be considered for deletion since a similar discussion has already been presented in Section 1.\n3.\tThe paper includes Fig. 2, but there is no mention of it in the text, suggesting that there might be an omission in referencing Fig. 2.\n\n**Content**:\n\n1.\tThere might be an error in Equation (2). The unauthorized $\\ R^k$ should be $\\ R^\\ell$, as the attacker only replaces the identity but still needs to mimic the speaker's pose-expression. Based on the above discussion, the statement \"the adversary’s own video $V^\\ell_t$ is used to derive the pose and expression vectors $z^\\ell_t$ that are transmitted to the receiver. The receiver then uses the generator $g$ alongside the unauthorized reference $R^k$ and $z^\\ell_t$ to reconstruct the video\" might be inconsistent with Equation (2).\n\n2.\tThe image pairs shown in Fig. 3 illustrate the distribution of cosine similarity scores, but the image pairs displayed appear very similar in terms of attributes such as clothing, skin tone, and age. Therefore, I am wondering whether ordinary cosine similarity can effectively distinguish between image pairs with significant attribute differences (for example, different identities but the same pose, as in the example in Fig. 1).", "limitations": "1.\tIt does not consider the impact of potential adaptive attacks, such as designing targeted adversarial samples to evade this defense mechanism.\n2.\tThe paper claims to abandon comparison with Avatar Fingerprinting, but I believe it would be beneficial to present the original experimental results and compare them in terms of time or computational overhead. This would strengthen the paper's claim of being \"efficient and real-time\".\n3.\tThe paper could include the use of different time-series-related neural network models, such as RNN (Recurrent Neural Network)[1][2], TPA (Temporal Pattern Attention)[3], GRU (Gated Recurrent Unit)[4], etc., to enhance the generalizability of the proposed approach across various model architectures.\n\n[1]\tJeffrey L. Elman. Finding structure in time. Cognitive Science, Volume 14, Issue 2, 1990, Pages 179-211.\n\n[2]\tJordan M I. Serial order: A parallel distributed processing approach[M]//Advances in psychology. North-Holland, 1997, 121: 471-495.\n\n[3]\tShih S Y, Sun F K, Lee H. Temporal pattern attention for multivariate time series forecasting[J]. Machine Learning, 2019, 108: 1421-1441.\n\n[4]\tCho K, Van Merriënboer B, Gulcehre C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[J]. arXiv preprint arXiv:1406.1078, 2014.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper proposes a novel defense method against puppeteering attacks, aiming to protect AI-based talking-head videoconferencing. The article ingeniously designs the EBL space to amplify identity discrepancies caused by puppeteering attacks, while theoretically verifying the boundary guarantees of contrastive learning in decoupling identity and pose-expression information.\n\nWeaknesses:\n1. The writing and structure of the paper still have some issues, such as: Fig. 1 is too small. Section 3.3 may be redundant. Fig. 2 is not referenced in the text.\n2. There are some logical inconsistencies in certain parts of the article. For example: Discrepancy between Equation (2) and its description. From the perspective of the entire paper, a puppeteering attack should involve an attacker using their own identity information to integrate the victim's pose-expression information, thereby synthesizing a fake video. However, lines 111-112 state that the synthetic video is created using the victim's identity information and the attacker's pose-expression information. This seems contradictory to both Equation (2) and the overall description of puppeteering attacks presented in the paper.\n3. Some of the arguments presented in the paper lack enough evidence. For example: Insufficient detail about the images in Fig. 3: The paper discusses the transformation to the EBL space to amplify sensitivity to identity features. However, Fig. 3 does not provide detailed information about the source of the images. Are they collected by the authors themselves, or are they from a public dataset? Additionally, the identity features of the image pairs shown in Fig. 3 do not appear to differ significantly, and attributes such as clothing and hairstyle are also quite similar. Did the authors use the image pairs shown in Fig. 2 (which have significant differences in both identity and attribute features) to compare cosine similarity? Using these image pairs would better support the paper's argument: even when identity differences are substantial, if pose-expression features are consistent, the cosine similarity in the latent space of pose-expression features remains similar.", "was_revised": true, "final_justification": "This paper proposes a novel defense method against puppeteering attacks, aiming to protect AI-based talking-head videoconferencing. The article ingeniously designs the EBL space to amplify identity discrepancies caused by puppeteering attacks, while theoretically verifying the boundary guarantees of contrastive learning in decoupling identity and pose-expression information. \n\nQuestions are well addressed by authors' rebuttal. I hold my opinion on accepting the paper.", "rebuttal": "_**Comment (Weakness 1 & Writing 1,2,3):** The writing and the structure of the paper have some issues such as Fig.1 being too small and section 3.3 having redundancies._\n\nWe thank the reviewer for pointing out the issues with the paper’s writing and structure. We will address these in the camera-ready version by ensuring that all figures, including Fig. 1, are resized and reformatted for clarity and that Section 3.3 is revised to remove redundancies and improve conciseness. We will also review the full manuscript to improve clarity, ensure consistent formatting, and enhance overall readability.\n\n_**Comment (Weakness 2):** There are logical inconsistencies in certain parts of the paper specially in relation to Equation 2 and the lines leading up to that equation._\n\nThank you for catching the mistake in Equation 2 of our main paper. We sincerely apologize for the confusion this oversight may have caused and will correct it in the camera-ready version.\n\nThe correct text in lines 108–109 should read:\n\n> *An adversary, speaker $\\ell$, obtains a target speaker’s representations $R^k$ and substitutes the driving video for the target speaker $k$, $V_t^k$, with their own video $V_t^\\ell$, thus replacing the target speaker’s pose and expression vectors $z_t^k$ with their own pose and expression vector $z_t^\\ell$.*\n\nAccordingly, the correct form of Equation 2 is:\n\n$$\n\\hat{V}_t^{k \\rightarrow \\ell} = g(f(V_t^\\ell), R^k) = g(z_t^\\ell, R^k)\n$$\n\nWe thank the reviewer for pointing out this issue and will ensure it is clearly corrected and explained in the camera-ready version.\n\n_**Comment (Weakness 3 - Part A):** Fig. 3 does not provide detailed information about the sources of the images. Are they collected by the authors themselves, or are they from a public dataset?_\n\nThank you for this comment.  We apologize for inadvertently omitting this information\nThe images shown in the Fig. 3 are taken from the NVFAIR dataset, which is publicly available.  Similarly, the data used to generate the distributions shown in Fig. 3 is also from the publicly available NVFAIR dataset.  We will revise the camera ready manuscript to include this information.\n\n_**Comment (Weakness 3 - Part B):** Additionally, the identity features of the image pairs shown in Fig. 3 do not appear to differ significantly, and attributes such as clothing and hairstyle are also quite similar. Did the authors use the image pairs shown in Fig. 2 (which have significant differences in both identity and attribute features) to compare cosine similarity?_\n\nThank you for raising this issue.  The specific images used in Fig. 3 were only chosen to visually illustrate pose/expression and identity matches or mismatches.  They are only a single datapoint, and as such do not capture the significant variation in facial attributes, clothing, hairstyle, gender, ethnicity, etc. that was present in the larger dataset used to generate the distributions shown in Fig. 3. \n\nTo create the distributions shown in Fig. 3, we used a dataset of 500 unique video-level pairs of self-reenacted and puppeteered videos from the NVFAIR dataset, corresponding to over 125,000 frame-level cosine similarity comparisons. These comparisons comprised a wide variety of facial attribute variation, thus ensuring that even when identity differences are substantial, if pose and expression are consistent, then cosine similarity between pairs of embeddings with closely aligned pose and expression remains similar.\n\nWe apologize for the confusion that has resulted due to us omitting this information from our paper.  We will revise the camera ready version to include this information to improve clarity.\n\n_**Comment (Limitation 1):** Paper doesn’t consider the impact of potential adaptive attacks such as adversarial examples to evade this defense mechanism_\n\nThank you for this comment.  We agree that understanding this system’s vulnerability to adversarial countermeasures is important.  However, this involves a detailed theoretical investigation, as well as thorough accompanying experimental analysis.  As a result, this is beyond the scope of this work.\n\nSpecifically, this would involve developing new attack strategies, creating new datasets, and running multiple experiments under several conditions to effectively understand this system’s vulnerability to attack. Unfortunately, we do not have the space to include this in this work.\n\nWe do, however, believe that this is an important topic of study and plan to address this in a separate future work. \n\n_**Comment (Limitation 2):** Lack of comparison with Avatar Fingerprinting in terms of time or computational overhead._\n\nThank you for your comment. We agree that a computational complexity comparison between our method and Avatar Fingerprinting (AF) would help the reader understand trade-offs and advantages of using each method, specifically with respect to computational complexity and runtime.\n\nUnfortunately, at the time of this writing, Avatar Fingerprinting’s code and trained model are not publicly available.  Furthermore, its authors did not report any results regarding its computational complexity or runtime. \n\nHowever, we note that AF’s proposed method used a deep 3D Temporal CNN to process facial data from a number of video frames. This neural architecture has significantly higher computational complexity than our method’s MLP layers. Additionally, AF can only perform detection post-hoc, meaning it produces a single score for an entire video clip. This means that this method is unlikely to be able to perform detection in real time.\n\n_**Comment (Limitation 3):** The paper would benefit from using different time-series-related neural networks such as RNNs._\n\nWe appreciate the reviewer’s comment. The primary focus of our work is on the discovery of biometric leakage from the pose-and-expression embedding, the design of the enhanced biometric leakage (EBL) representation, and the associated training protocol and loss function. While temporal information fusion is an important component, it is not the core novelty of our approach.\n\nWe did evaluate different temporal fusion strategies during development and found that LSTM-based fusion provided stronger performance gains than simple temporal moving average methods. Importantly, the LSTM was able to achieve real-time performance on consumer hardware, aligning well with our design goals for practical deployment.\n\nGiven the strong overall system performance and our emphasis on the primary contributions noted above, we prioritized our research efforts toward those core components rather than conducting a comprehensive comparative study of temporal modeling approaches. We agree, however, that exploring alternative temporal modeling strategies could be an interesting direction for future work, and we will note this in the revised version of the paper."}]

Author rebuttal:

Meta-review: This paper presents a real-time method to prevent adversarial attacks in AI-based video conferencing, where avatars are driven by pose-expression vectors. The core contribution is a technique that learns a latent space of "biomarkers" from this pose data. By comparing the biomarkers of the controller to those of the generated avatar, the system can detect a mismatch. A key innovation is a novel pose-matched negative pair creation strategy for contrastive learning, which allows the method to generalize to new users without requiring any pre-recorded data.

The paper is well-written, well-motivated, and proposes a clever solution to a practical problem. 

The initial submission had some weaknesses regarding comparisons with prior work, the extent of the ablation studies, and clarity on certain implementation details. However, these concerns were thoroughly and satisfactorily addressed by the authors during the rebuttal period.

The decision to accept is based on the paper's novel and practical contribution to securing AI-driven communication. The method's ability to generalize to unseen users is a key advance for the field. The technical approach is sound and elegant, and the authors successfully addressed all reviewer concerns in their rebuttal, resulting in a strong and well-supported paper.
Decision: Accept (poster)