DeepRhythm: Exposing DeepFakes with Attentional Visual
Heartbeat Rhythms
Hua Qi1∗ , Qing Guo2∗ , Felix Juefei-Xu3 , Xiaofei Xie2 , Lei Ma1†
Wei Feng4 , Yang Liu2 , Jianjun Zhao1
1 Kyushu University, Japan

2 Nanyang Technological University, Singapore

3 Alibaba Group, USA

4 Tianjin University, China

ABSTRACTAs the GAN-based face image and video generation techniques,
widely known as DeepFakes, have become more and more matured and realistic, there comes a pressing and urgent demand for
effective DeepFakes detectors. Motivated by the fact that remotevisual photoplethysmography (PPG) is made possible by monitoring the minuscule periodic changes of skin color due to blood
pumping through the face, we conjecture that normal heartbeat
rhythms found in the real face videos will be disrupted or evenentirely broken in a DeepFake video, making it a potentially powerful indicator for DeepFake detection. In this work, we propose
DeepRhythm, a DeepFake detection technique that exposes DeepFakes by monitoring the heartbeat rhythms. DeepRhythm utilizes
dual-spatial-temporal attention to adapt to dynamically changing
face and fake types. Extensive experiments on FaceForensics++
and DFDC-preview datasets have confirmed our conjecture anddemonstrated not only the effectiveness, but also the generalization capability of DeepRhythm over different datasets by various
DeepFakes generation techniques and multifarious challenging
degradations.

CCS CONCEPTS• Computing methodologies → Computer vision; • Security
and privacy → Social aspects of security and privacy.

KEYWORDS
DeepFake detection, heartbeat rhythm, remote photoplethysmography (PPG), dual-spatial-temporal attention, face forensics
ACM Reference Format:
Hua Qi1∗ , Qing Guo2∗ , Felix Juefei-Xu3 , Xiaofei Xie2 , Lei Ma1† and Wei
Feng4 , Yang Liu2 , Jianjun Zhao1 . 2020. DeepRhythm: Exposing DeepFakes
with Attentional Visual Heartbeat Rhythms. In Proceedings of the 28th ACM
International Conference on Multimedia (MM ’20), October 12–16, 2020, Seattle,∗ Both authors contributed equally to this research.

† Lei Ma is the corresponding author (malei@ait.kyushu-u.ac.jp).

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
MM ’20, October 12–16, 2020, Seattle, WA, USA© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-7988-5/20/10. . . $15.00
https://doi.org/10.1145/3394171.3413707

Manipulations:
DeepFakes
Face2Face
FaceSwap
NeuralTextures...
Rea Video

Fake Video

Sequential Signals ( Real vs. Fake )Figure 1: An example of a real video and its fake video generated by various
manipulations, e.g., DeepFakes, Face2Face, FaceSwap, etc. [57]. It is hard to
decide real/fake via the appearance from a single frame. The state-of-the-art
Xception [9] fails in this case. However, we see that the manipulations easily
diminish the sequential signals representing remote heartbeat rhythms.

WA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/
3394171.3413707

1

INTRODUCTION

Over the past decades, multimedia contents such as image andvideo have become more and more prevalent on various social
media platforms. More recently, with the advancement in deeplearning-based image and video generation techniques, i.e., generative adversarial networks (GAN) [22], anyone can now generate,e.g., a realistic-looking face that does not exist in the world, or
perform a face swap in a video with a high level of realism. The
latter is what the community refers to as the DeepFake [36–38].
Such a face swap used to require domain expertise such as theatrical visual effects (VFX) and/or high-speed tracking with markers(e.g., motion captures in the movie Avatar). But now, anyone cando it easily. The low barriers to entry and wide accessibility ofpre-trained DeepFake generator models are what the problem is.
DeepFakes are now a pressing and tangible threat to the integrityof multimedia information available to us. DeepFakes, e.g., whenapplied on politicians, fueled with targeted misinformation, canreally sway people’s opinions and can lead to detrimental outcomessuch as manipulated and interfered election without people evenknowing about it.

Therefore, the fight against DeepFakes is currently in dire need.
Although the detection of DeepFakes is a fairly new research frontier, there have been some attempts to spot DeepFake videos. Somemethods are based on traditional digital forensics techniques (seeSection 2.2), while others heavily rely on deep learning-based image
and video classification (real vs. fake) from the raw pixel-domain
DeepFake inputs. However, such detection methods based solely
on the raw pixel-domain input might become less effective when
the DeepFake images and videos become more and more realistic
as the deep image generation methods themselves become moremature in the near future. Consequently, a fundamentally different
DeepFake detection method is needed.
In this work, we present DeepRhythm, a novel DeepFake detection technique that is intuitively motivated and is designed fromground up with first principles in mind. Motivated by the fact that
remote visual photoplethysmography (PPG) [69] is made possible
by monitoring the minuscule periodic changes of skin color due
to blood pumping through the face from a video, we conjecturethat normal heartbeat rhythms found in the real face videos will be
disrupted or even broken entirely in a DeepFake video, making it apowerful indicator for detecting DeepFakes. As shown in Figure 1,
existing manipulations, e.g., DeepFakes, significantly change thesequential signals of the real video, which contains the primary information of the heartbeat rhythm. To further make our proposed
DeepRhythm method work more robustly under various degradations, we have devised and incorporated both a heart rhythm
motion amplification module as well as a learnable spatial-temporalattention mechanism at various stages of the network model.
Together, through extensive experiments, we have demonstrated
that our conjecture holds true and the proposed method indeed effectively exposes DeepFakes by monitoring the heartbeat rhythms.
More specifically, DeepRhythm outperforms four state-of-the-art
DeepFake detection methods including Bayer’s method [3], Inception ResNet V1 [59], Xception [9], and MesoNet [1] in the FaceForensics++ benchmark [57] and exhibits high robustness to JPEG
compression, noise, and blur degradations. To the best of our knowledge, this is the very first attempt to expose DeepFakes using heartbeat rhythms. Our main contributions are summarized as follows:• We propose DeepRhythm, the very first method for effective
detection of DeepFake with the heartbeat rhythms.
• To characterize the sequential signals of face videos, we propose
the motion-magnified spatial-temporal representation (MMSTR)
that provides powerful discriminative features for high accurate
DeepFake detection.
• To fully utilize the MMSTR, we propose dual-spatial-temporalattention network to adapt to dynamically changing faces andvarious fake types. Experimental results on FaceForensics++ andDeepFake Detection Challenge-preview dataset demonstrate that
our method not only outperforms state-of-the-art methods but
is robust to various degradations.

2 RELATED WORK
2.1 DeepFakes Generation
Recently, DeepFake techniques have gained widespread attention
and been used in generating pornographic videos, fake news, andhoaxes, etc. Some early studies use face-wrap-based methods togenerate fake videos. For example, Bregler et al. [4] track the movement of the speaker’s mouth and morph the input video. Dale et al.
[11] present a video face replacement via a face 3D model. Similarly,
Garrido et al. propose a face warp system while keeping the original face performance [18] and a photo-realistically replacement
method via high-quality monocular capture [19]. Thies et al. develop a real-time expression transfer for facial reenactment [61] andpropose the Face2Face method [62] that tracks target and source
facial expressions to build a face 3D model and re-renders source
face on the target model. In addition, Thies et al. [60] further use
neural textures and defer neural render to generate the forgeries.
Besides the above face-wrap-based methods, recent DeepFake
approaches, i.e., PGGAN [36], StyleGAN [37], and StyleGAN2 [38],employ the generative adversarial network (GAN) [22] for the nearrealistic face synthesis. Moreover, some methods can even alter
face attributes, e.g., changing or removing the color of the hair,adding glasses or scars [8, 21, 30, 63], and modifying persons’ facial
expression [41]. Overall, GANs have shown great potential in thisarea and are easy to use. However, current DeepFake methods, eventhose based on GANs, do not explicitly preserve the pulse signal,
inspiring us to capitalize on the pulse signal to distinguish the real
and manipulated videos.

2.2

Forgery and DeepFake Detection

DeepFake detection is challenging since GAN-based DeepFakes cangenerate near-realistic faces that are hardly detected even using
the state-of-the-art digital forensics. To alleviate this challenge,
researchers are exploring effective solutions to identify fake videos.
Early attempts focus on detecting forgeries via hand-craftedfeatures, e.g., [5, 17, 20, 53]. However, these hand-crafted models canbe strenuous due to the realistic faces generated by SOTA DeepFake
methods (e.g., FaceApp, Reflect, and ZAO). Later, researchers regard
the DeepFake detection as a classification problem by extracting
discriminative features, e.g., color cues [46], monitoring neuron
behaviors [43–45, 66, 68], and employing classifiers, e.g., support
vector machine (SVM), to tell whether a video is fake or real.
In addition, many researchers also employ SOTA deep neuralnetworks (DNNs) to detect forgery images. Cozzolino et al. [10] use
residual-based local features and achieve significant performance.
Bayar et al. [3] and Rahmouni et al. [56] propose novel DNNs todetect manipulated images. Zhou et al. [71] combine a DNN-based
face classification stream with a steganalysis-based triplet stream,yielding good performance. More recently, researchers are trying
to apply much more complex and advanced DNNs on video forgery
detection, such as Inception-ResNet [59], MesoNet [1], capsule
networks [47], and Xception [9].
Besides only adopting convolutional neural networks (CNNs),
some researchers use a combined recurrent neural network (RNN)
and CNN to extract image and temporal features to distinguish realand fake videos. For example, Güera et al. [28] use CNN features
to train an RNN to classify videos. Similarly, Sabir et al. [58] use asequence of spatio-temporal faces, as RNN’s input to classify thevideos. Furthermore, Dong et al. [12] utilize an attention mechanism to generate and improve the feature maps, which highlightthe informative regions. Different from existing methods, our technique initiates the first step of leveraging remote heartbeat rhythms

for DeepFake detection. To achieve high detection accuracy, we propose a motion-magnified representation for the heartbeat rhythms
and employ a spatial-temporal representation to improve the ability
in distinguishing real and fake videos.

2.3

Remote Photoplethysmography (rPPG)
Anti-Spoofing

Face spoof detection is similar to DeepFake detection, aiming to determine whether a video contains a live face. Since the remote heart
rhythm (HR) measuring techniques achieve quite a bit of progress[2, 6, 35, 39, 54, 55, 69], many works use rPPG for face spoofing detection. For example, Li et al. [40] use the pulse difference between
real and printed faces to defend spoofing attacks. Nowara et al. [49]compare PPGs of face and background to decide whether the face islive or not. Heusch et al. [32] use the long-term statistical spectralon the pulse signals and Hernandez-Ortega et al. [31] employ thenear infrared against realistic artifacts. Moreover, combining with
DNNs, Liu et al. [42] extract spatial and temporal auxiliary information, e.g., depth map and rPPG signal, to distinguish whether it
is a live face or spoofing face.
Overall, existing anti-spoofing methods also benefit from employing rPPG for liveness detection, which seems similar to ourwork. However, there are fundamental differences: the livenessdetection mainly relies on the judgment about whether the heart
rhythms exist or not; our work aims to find the different patterns between real and fake heart rhythms since fake videos may still havethe heart rhythms but their patterns are diminished by DeepFake
methods and are different from the real ones (see Figure 1).

3

METHOD

We propose DeepRhythm (Sec. 3.1) for effective DeepFake detection
by judging whether the normal HR in face videos are diminished.
Figure 2 (a) summarizes the workflow of DeepRhythm.

3.1

DeepRhythm for DeepFake Detection

Given a face video V = {Ii }Ti=1 that contains T frames, our goal isto predict if this video is real or fake according to the heart rhythm
signals. To this end, we first develop the motion-magnified spatialtemporal representation (MMSTR) (Sec. 3.2) for face videos, which canhighlight the heart rhythm signals and output a motion-magnified
spatial-temporal map (MMST map), i.e., X = mmstr(V) ∈ RT ×N ×C
where T is the number of frames, N is the N region of interest(ROI) blocks of the face in V (i.e., the regions marked by the blurgrid in Figure 2 (a)), and C means the number of color channels.
In the following, we formulate with single color channel for clear
representation but use RGB channels in practice. Intuitively, X
contains the motion-magnified temporal variation of N blocks inthe face video, i.e., highlighted heart rhythm signals.
We can simply design a deep neural network that takes the
MMST map as input and predict if the raw video is real. However,various interference, e.g., head movement, illumination variation,and sensor noises, may corrupt the MMST map. As a result, thecontributions of different positions in the MMST map are not always
the same (e.g., the three patches shown in Figure 2 (a) have different
heart rhythm strength), which definitely affects the fake detection
accuracy. To alleviate this challenge, we should assign different

weights to different positions of the MMST map before further
performing the fake detection
y = ϕ(A ⊙ X),(1)

y = ϕ((t · s⊤ ) ⊙ X),(2)

where ϕ(·) is a CNN for real/fake classification, ⊙ denotes theelement-wise multiplication, and y is the prediction (i.e., 1 for fakeand 0 for real). The matrix A ∈ RT ×N provides different weights
to different positions of X and is known as an attention mechanism.
We let RGB channels share the attention matrix.
We aim to produce A via a DNN. However, due to the diverse
types of fake and dynamic changing faces, it is difficult to get proper
A for different face directly. We handle this problem by further
decomposing A into two parts, i.e., spatial attention s ∈ RN ×1 andtemporal attention t ∈ RT ×1 , and reformulate Eq. (1) as
Intuitively, the two attentions indicate when (along the T ’s axis) andwhere (along the N ’s axis) of the input MMST map should be used
for better fake detection. Furthermore, the number of parameters ofs and t, i.e., N + T , is much smaller than that of A, i.e., N · T , whichallows the spatial-temporal-attention to be tuned more easily.
Then, the key problem is how to generate t and s to adapt todynamically changing faces and various fake types. In Sec. 3.3,
we propose the dual-spatial-temporal attention network to realize
Eq. (2) by jointly considering prior & adaptive spatial attention andframe & block temporal attention.

3.2

Motion-Magnified Spatial-Temporal
Representation

A straightforward way of employing heart rate (HR) signals for
DeepFake detection is to use existing HR representations that aredesigned for the remote HR estimation. For example, we can use
the spatial-temporal representation (STR) proposed by Niu et al.
[48] for representing HR signals and feed them to a classifier for
DeepFake detection. However, it is hard to achieve high fake detection accuracy with the STR directly since the differences between
real and fake videos are not highlighted, i.e., STR’s discriminative
power for DeepFake detection is limited.
To alleviate the problem, we propose the motion-magnified STR(MMSTR) where differences between real and fake face videos canbe effectively represented. Specifically, given in a face video, i.e., V
having T frames, we calculate MMSTR using the following steps:(i) Calculate landmarks1 of the faces on all frames of V andremove the eyes and background according to the landmarks,e.g., the faces shown in the left of Figure 2 (b).
(ii) Perform the motion magnification algorithm [50, 67]2 on the
background removed face video and obtain motion-magnified
face video with RGB space.
(iii) Divide the face areas of all frames into N non-overlapping
ROI blocks, i.e., regions marked by the blue grid in Figure 2 (b), and perform average pooling on each block andeach color channel for each frame. We then obtain the MMST
map, i.e., X, as the sub-figures shown in the right of Figure 2 (b). Each row of X represents the motion-magnified

1 https://github.com/codeniko/shape_predictor_81_face_landmarks.

2 We use its python implementation: https://github.com/flyingzhao/PyEVM.

(a) Pipeline of DeepRythm
MMST Map

T

N×T

Motion-Magniﬁed
STR (MMSTR)

Real
Dual-ST
Attentional Network

N=25 blocks
T=300 frames

Raw Video
Face Video

Fake
Motion-Magniﬁed Face Video

Background Removed
Real Face Video

Motion-Magniﬁed
Real Face Video

R3(b) MMSTR (Real vs. Fake)

Per-blockAveraging

MotionMagniﬁcation

Sequential signals of R3 on RGB channels: main diﬀerences between Real and Fake signals

R3

15×1, max poolbatch norm

15×1, conv,64

Motion-Magniﬁed
Fake Face Video

0

0

N

sa

N

sp

Adaptive Spatial Attention

Prior Spatial Attention

MMST Map

Background Removed
Fake Face Video

per-frame
classiﬁcation

Frame-level
TemporalAttention

tf

...

...

lstm

Real

...

0

A=(tb+tf)•(sa+sp)
T(c) Dual-ST AttenNet

Motion-Magniﬁed
Face Video

T

T

tb

Fake
...

...

...

0

T

sequential
input

ResNet18

Figure 2: The workflow of DeepRhythm, i.e., (a), and its two main modules: motion-magnified spatial-temporal representation (MMSTR), i.e., (b), and dual-spatialtemporal attentional network (Dual-ST AttenNet), i.e., (c). We also highlight the main differences of MMSTRs between real and fake videos in sub-figure (b).

temporal variation of one block on RGB channels, as the red,green, and blue curves shown in Figure 2 (b).
Figure 2 (b) shows examples of real and fake face videos and their
MMST maps, respectively. We have the following observations: 1)
it is difficult to judge which video is fake just by looking at the raw
frames. 2) differences between the real and fake videos can be easilyfound on our MMST maps that will provide effective information
for fake detection. The advantages of MMSTR over STR will befurther discussed in the experimental section.

3.3

Dual-Spatial-Temporal Attentional
NetworkIn this section, we detail the dual-spatial-temporal attentional network (Dual-ST AttenNet), with which we can realize accurate DeepFake detection through the MMST map and its spatial and temporalattentions, i.e., t and s defined in Eq. (2).
3.3.1

Dual-Spatial Attention. The dual-spatial attention iss = sa + sp ,(3)

where sp ∈ RN ×1 and sa ∈ RN ×1 are the prior and face-adaptive
spatial attentions, respectively. The prior attention sp is a fixed
vector whose six specified elements are set as one while others
are zero, which is to extract the HR signals from six specified ROI
blocks and ignore signals from other blocks. The six specified ROI
blocks are the four blocks under eyes and two blocks between eyes,
as shown in Figure 2 (c). The intuition behind this idea is that the

specified blocks are usually robust to various real-world interference while the HR signals of other blocks are easily diminishedwhen unexpected situations happen, e.g., head movement might
let the HR signals of blocks at face boundary disappear.
In addition to the prior spatial attention, we also need faceadaptive attention, i.e., sa , to highlight different blocks to adapt tothe environment variations since even the same face under different
situations, e.g., the illumination changes, has different effective ROI
blocks. To this end, we propose to train a spatial attention network
to generate adaptive spatial attention, which contains a convolution
layer that has 64 kernels with size being 15 × 1 followed by a batchnormalization layer and max-pooling layer. The CNN’s parameters
are jointly learned with the whole framework.
3.3.2 Dual-Temporal Attention. DeepFake methods usually add
different fake textures at different face locations to different frames,
which not only destroy the smooth temporal variation of a face but
lead to inconsistent fake magnitude among frames (i.e., some frames
contain obvious fake textures while others have few or no fakes).
We propose dual-temporal attention to consider above informationt = tb + tf ,(4)

where tb ∈ RT ×1 and tf ∈ RT ×1 indicate which frames are moresignificant for final fake detection. Specifically, we train an LSTM torepresent the temporal variation of a face, which is sequentially fed
with each row of the MMST map, i.e., X, and outputs tb that is denoted as the block-level temporal attention. The LSTM’s parameters
are jointly trained with the whole framework.

Real Video

Fake Video

Real Video

Fake Video

Figure 3: Two real-fake video pairs, their MMST maps and the colorful difference maps between real and fake.

To take full advantage of fake textures in each frame, we train
a temporal-attention network that takes each motion-magnified
frame as input and scores the fakeness of the frame independently
and get tf . The frames with higher probability to be fake contribute
more to the final classification and we denote tf as the frame-leveltemporal attention. In practice, we use the Meso-4 architecture[1] as the network containing a sequence of four convolution layers and two fully-connected layers. The Meso-4’s parameters are
independently trained for frame-level fake detection.
3.3.3 Implementation details. Our dual-spatial-temporal attentional
network is shown in Figure 2 (c) where an MMST map, i.e., X, is first
employed to produce the adaptive spatial attention, i.e., sa and theblock-level temporal attention, i.e., tb , through a spatial attention
network and an LSTM, respectively. The pre-trained Meso-4 is fed
with the motion-magnified face video and outputs the frame-leveltemporal attention, i.e., tf . Finally, the attentional MMST map, i.e.,(t · s⊤ ) · X, is fed to the ϕ(·) for the final DeepFake detection where
we use ResNet18 [29] for the ϕ(·).
We jointly train parameters of the spatial attention network, theLSTM, and the ResNet18 using the cross-entropy loss with Adam
optimizer. The learning rate and weight decay are set as 0.1 and 0.01
respectively. The max epoch number is set to 500, and training willstop if validation loss did not decrease in 50 epochs. For training theMeso-4, we use the same hyper-parameters. We use videos fromFaceForensics++ [57] as the training dataset which is introduced inSec. 4.1 and Table 1. Our implementation and results are obtainedon a server with Intel Xeon E5-1650-v4 CPU and NVIDIA GP102L.

4 EXPERIMENTS
4.1 Dataset and Experiment Setting
Dataset. We select FaceForensics++ [57] as our training and testingdatasets, and use DeepFake Detection Challenge preview (DFDCpreview) [15] as an additional testing dataset to evaluate our method’s
cross-dataset generalization capability.
FaceForensics++ dataset consists of thousands of videos manipulated with different DeepFake methods and contains four fake subdatasets, i.e., DeepFake Detection (DFD), DeepFake (DF), Face2Face(F2F), and FaceSwap (FS). However, the original FaceForensics++
dataset has the data imbalance problem. For example, the original
DFD subset contains 2728 fake videos, but only 268 real videos. To
solve this problem, we make the following improvements: 1) we
augment the the original 268 real videos by flipping them horizontally and get a total of 2,510 real videos. 2) To evaluate our methodon the whole FaceForensics++ dataset, we build an extra dataset, i.e.
‘ALL’ in Table 1, by concatenating the four subsets and augment

Table 1: Details of FaceForensics++ (FF++) and DFDC-preview datasets forboth testing and training

Dataset
DFD
DF
FF++
F2F
FS
ALL
DFDC-preview

total
5238
1959
1966
1971
10680
3310

real
2510
988
988
988
5020
578

fake
2728
971
978
983
5660
2732

train
4190
1567
1572
1582
8544
-

val.
524
195
196
197
1068
-

test
524
197
198
198
1068
1000

the real videos by flipping them horizontally and vertically, and rotating them 180 degrees, respectively. Table 1 summarizes the final
four subsets, the ALL dataset, and their partitions about training,validation, and testing. The dataset partition ratio is 8:1:1 and theaugmented videos are removed in the testing datasets.
For videos in DeepFake Detection Challenge, we directly use
it as the testing set. The details can be found in Table 1. Directly
using the DFDC-preview dataset as testing set will cause imbalance between the number of real and fake videos. So we randomly
sample 500 videos from the real part and 500 videos from the fakepart, and assemble them as the testing set. Then, we test every
method formerly trained on ALL’s training set on it to compare
their performance.
Pre-processing. For every video in the testing and training
datasets, we take the first 300 frames to produce the MMST map.
Specifically, while processing frames, we first use MTCNN [52] todetect face, and then use Dlib to get 81 facial landmarks [51]. If faces
are not detected in a frame, this frame will be abandoned. If morethan 50 frames were abandoned, this video will not be used to train
the network. If more than one faces were detected in one frame,
the one closer to the faces of previous frames will be retained.
Baseline. We choose the state-of-the-art DeepFake detection
methods, i.e., Bayer’s method [3], Inception ResNet V1 [59], Xception [9] and MesoNet [1], as baselines. All of them obtain highperformance on FaceForensics++’s benchmark [13] and are the
top-4 accessible methods therewithin. For Bayer’s method [3], we
do not find publicly available code, so we re-implement it on Keras.
For Inception ResNet V1 [59] and Xception [9], we directly use theKeras code provided and only add a Dense layer with one neuron after the final layer to get prediction. As for MesoNet [1], we directly
use the code provided by authors [14].
It should be noted that these baselines perform the fake detection
on an image instead of a video, i.e., estimating if a frame is real or
fake. We make the following adaptations to make them suitable toaddress videos: 1) For the testing setup, we use these baselines topredict every frame of the video and count the number of real or
fake frames. If the real frames are more than fake ones, we identified
this video as real, and vice versa. 2) In terms of the training setup,
we take the first five frames for every training video in the ‘ALL’
dataset in Table 1 and extract their facial region via MTCNN, all ofthese faces are divided into training, validation, and testing subsets.
We also employ Adam optimizer with batch-size of 32 and learning
rate of 0.001. The max epoch number is 500, and the training willstop if validation loss did not decrease in 50 epochs.

Table 2: Comparison with baseline methods on FaceForensics++ and DFDC-preview datasets with the models trained on sub-datasets and ALL dataset of FaceForensics++, respectively. We highlight the best and second best results with red and yellow.

test onBayer and Stamm [3]Inception ResNet V1 [59]
Xception [9]MesoNet [1]DeepRhythm (ours)

4.2

train on sub-datasets
DFD
DF
F2F
FS
0.52
0.503 0.505 0.505
0.794 0.783 0.788 0.778
0.98
0.995 0.985
0.98
0.804 0.979
0.985 0.995
0.987
1.0
0.995
1.0

train on ALL dataset
DF
F2F
FS
ALL
0.52
0.503 0.505
0.5
0.638 0.566 0.462 0.774
0.984 0.984
0.97
0.978
0.822 0.813 0.783 0.909
0.997 0.989 0.978
0.98

DFDC
0.5
0.597
0.612
0.745
0.641

Baseline Comparison on Accuracy

We test all methods on DFD, DF, F2F, FS, and ALL subsets (reported
in Table 2) and the DFDC-preview with the models trained on
FF++’s subsets (DFD, DF, F2F, FS, and ALL), respectively.
Results on FaceForensics++. Overall, in Table 2, our DeepRhythm achieves the highest accuracy on all datasets comparedwith the baseline methods. First, our method gets better results
than other methods across all cases, regardless of which training
dataset is used. This demonstrates the generalization capability of
our method across various DeepFake techniques. Second, although
we adopt the MesoNet in our framework for the frame-level temporal attention, our method significantly outperforms the MesoNet on
all cases, e.g., when trained on ALL dataset, DeepRhythm achieves
0.96 on the FS while MesoNet only has 0.719, which validates theeffectiveness of our MMST representation and other attention information, and also indicates the potential capability of our framework
for enhancing existing frame-level DeepFake detection methods.
Third, although the baseline method Xception has obtained significantly high accuracy, e.g., 0.985 and 0.995 on the testing datasetof F2F and FS, it is still exceeded by our method, confirming theadvantage of our method over the state-of-the-arts. We show twocases from the FaceForensics++ dataset in Figure 3, where all baseline methods fail to recognize the fake videos while our method
succeeds. The fake techniques diminish the sequential signal patterns of real videos (e.g., the waveform of the real video in the firstcase becomes flat in the fake video), which are effectively captured
by our MMST maps.
Results on DFDC-preview. According to the results on the
DFDC (see Table 2), Bayer’s still performs worse than others, achieving 0.5 accuracy. Inception ResNet V1 has worse performance thantheir results on ALL’s testing set, achieving 0.597. Xception obtains
0.612 accuracy on DFDC-preview, and is also much worse thanits performance on ALL’s testing set. Our DeepRhythm gets 0.641
accuracy and is better than Xception, being the second highest accuracy. Although MesoNet performs not as good on ALL’s testing
set, it achieves the highest accuracy (i.e., 0.745).

4.3

DFD
0.501
0.919
0.965
0.958
0.975

Ablation Study on Accuracy

To demonstrate the effectiveness of our motion-magnified spatialtemporal representation (MMSTR), dual-spatial-temporal attention
network, and end-to-end training, we conduct an ablation study by
first training the basic model with existing spatial-temporal (ST)map at the beginning and then add our contributions one by one.
DeepRhythm variants. We first train the bare model (DR-st),which only uses ST map as its input without motion magnification

MMST map of Real Video

Real Video

Fake Video

MMST map of Fake Video

Diﬀ. MMST map of Real & Fake

Diﬀ. Real& Fakes a : Adapt.
spatial att.

t b : Block-level temporal attention

Figure 4: An example of a real video, the corresponding fake video, the difference image between real and fake frames (Diff. Real & Fake), the MMST
maps of real and fake videos, the difference map between real and fake MMST
maps (Diff. MMST map of Real & Fake), the adaptive spatial attention (sa , i.e.,
adapt. spatial att.), and the block-level temporal attention (tb ).

and attention. Then, we use MMST map as inputs and re-train ourmodel (denoted as DR-mmst), still not using any attention. After
that, based on the pre-trained DR-mmst, we add adaptive spatialattention (A) and block-level temporal attention (B), respectively,(i.e., DR-mmst-A and DR-mmst-B) and perform fine-tuning. After
50 epochs, we observe that the validation loss does not decrease
further. Then, we add prior spatial attention (P) and frame-leveltemporal attention (F) on DR-mmst-A and DR-mmst-B, respectively,
then get DR-mmst-AP and DR-mmst-BF that are further fine-tuned.
Next, based on either DR-mmst-AP or DR-mmst-BF, we use four
attentions together (i.e., DR-mmst-APBF) and carry on training.
Finally, we compare DR-mmst-APBF with our final version (i.e.,
DR-mmst-APBF-e2e), where the adaptive spatial attention (A), blocklevel temporal attention (B), and the network are jointly or endto-end trained. For all the experiments, we use the same hyperparameters and datasets, as introduced in Sec. 3.3.3. The results are
summarized in Table 3.
Effectiveness of MMSTR. As shown in Table 3, our MMSTR
significantly improves the DR-st’s accuracy, e.g., 0.328 improvement
on ALL. The ST map from [48] has little discriminative power for
DeepFake detection since DR-st achieves about 0.5 accuracy on every testing dataset, which means it randomly guesses a video being
real/fake. After using our MMSTR, DR-mmst achieves 0.217 averaged accuracy increment over DFD, DF, F2F, FS, and ALL datasets.
Effectiveness of single attention. Based on the DR-mmst, we
add adaptive spatial attention (DR-mmst-A) and block-level temporal attention (DR-mmst-B), respectively. These two attentions
do help improve the model’s accuracy, as presented in Table 3

Raw

JPEG: 60

Blur: 11

Noise: 15

Sampling:10

MMST
maps of

Raw:
JPEG:
Blur:
Noise:
Sampling:

Figure 5: Frames and MMST maps of a video and their four degraded versions
with JPEG, blur, noise, and temporal sampling degradations whose degrees
are 60, 11, 15, and 10, respectively, which are the median values in the x-axis
ranges in Figure 7. Clearly, the JPEG, blur, and noise degradations do not affect the MMST maps of raw videos. The temporal sampling significantly diminishes the raw pattern of the MMST maps.

where DR-mmst-A and DR-mmst-B get average 0.061 and 0.0632
improvements over DR-mmst, respectively.
We further show an example of the adaptive spatial attention (sa )and the block-level temporal attention (tb ) in Figure 4. To validate
their effectiveness, we also present the difference image and MMST
map between real and fake cases. From the view of spatial domain,
the difference image indicates that the main changes caused by the
fake is around the nose, which is identical to the estimated adaptive
spatial attention. In terms of the temporal domain, the estimatedtemporal attention has high values at the peaks of the difference
MMST map.
Effectiveness of dual-spatial attention. In addition to theadaptive spatial attention (DR-mmst-A), we further consider theprior attention where the specified ROI blocks on faces are considered and realize the DR-mmst-AP. As validated in Table 3, DRmmst-AP outperforms DR-mmst-A on all compared datasets andobtains an average of 0.033 improvement, which demonstrates theadvantage of dual-spatial attention over single adaptive spatialattention.
Effectiveness of dual-temporal attention. The block-leveltemporal attention misses details among frames. To alleviate thisissue, we add the frame-level temporal attention (F) to DR-mmst-B
for the frame-level DeepFake detection and get the DR-mmst-BF. In
Table 3, DR-mmst-BF has much higher accuracy than DR-mmst-B
on all compared datasets. The average improvement is 0.178, whichshows the effectiveness of our dual-temporal attention.
Effectiveness of dual-spatial-temporal attention and endto-end training. We put DR-mmst-AP and DR-mmst-BF together
and get DR-mmst-APBF. Compared with DR-mmst-AP, DR-mmstAPBF has much higher accuracy on all datasets. However, whencomparing it with DR-mmst-BF, DR-mmst-APBF’s accuracy slightly
decreases on DFD, DF, and ALL while increasing on FS dataset.
Though, when we train DR-mmst-APBF in the end-to-end way and
get DR-mmst-SPTM-e2e, it achieves the highest accuracy on all
testing datasets, indicating that training four attention separately
might not mine the potential power of the four attention effectively,
and training them together helps get the maximum effect.

Table 3: Ablation study of DeepRhythm (DR) by progressively adding the MMSTR, adaptive (A) and prior (P) spatial attentions, block-level (B) and framelevel (F) temporal attentions, and end-to-end (e2e) training strategy.

test onDR-st
DR-mmstDR-mmst-A
DR-mmst-B
DR-mmst-AP
DR-mmst-BF
DR-mmst-APBF
DR-mmst-APBF-e2e

4.4

DFD
0.522
0.814
0.849
0.872
0.879
0.97
0.965
0.972

train on ALL sub-dataset
DF
F2F
FS
ALL
0.497 0.497 0.492 0.512
0.684 0.635 0.64
0.84
0.77 0.736 0.716 0.847
0.745 0.731 0.731 0.85
0.816 0.766 0.756 0.867
0.969 0.954 0.959 0.966
0.959 0.954 0.965 0.964
0.98 0.964 0.959 0.98

Baseline Comparison on Robustness

In this section, we study the robustness of our method and two baseline methods, i.e., Xception and MesoNet, which have the highest
accuracy among baselines. Their models are trained on the training
set of the ALL dataset. We consider four general degradations, i.e.,
JPEG compression, Gaussian blur, Gaussian noise, and temporal
sampling, and construct a degradation dataset by manipulating thetesting set of the ALL dataset. For the first three degradations, we
add the corresponding interference to each frame of the tested video
and use the compression quality, blur kernel size, and standard deviation of noise to control the degradation degree, respectively. We
show the degradation examples in Figure 5. The temporal samplingmeans that we do not use the raw continuous frames to get the
MMST map but select frame at every K frames. We use temporal
sampling to test if our method still works under the unsmooth
temporal variation. Please refer to the x-axis in Figure 6 and 7 forthe variation range of each degradation.
As shown in Figure 6, our method exhibits strong robustness on
JPEG compression and Gaussian noise, but do not perform well on
temporal sampling when compared with Xception and MesoNet.
However, we could mitigate the issue with the video frame interpolation techniques, which is yet to be explored as the future work.

4.5

Ablation Study on Robustness

We use the degradation dataset in Sec. 4.4 to analyze the robustnessof MesoNet (i.e., the frame-level temporal attention) and seven
DeepRhythm variants (see the legend of Figure 7). These methods
can be roughly divided into two clusters, one using MesoNet for
the frame-level temporal attention (donated as F-cluster), including
MesoNet, DR-mmst-BF, DR-mmst-APBF, and DR-mmst-APBF-e2e;the others do not employ the MesoNet (donated as non-F-cluster),including DR-mmst, DR-mmst-A, DR-mmst-B, and DR-mmst-AP.
As shown in Figure 7, our MMSTR helps the variants, i.e., DRmmst, DR-mmst-A, DR-mmst-B, and DR-mmst-AP, to keep at almost the same accuracy across all compression quality. The reason
is that the MMSTR is calculated by average pooling pixel values
in ROI blocks, thus is insensitive to local pixel variation caused by
JPEG compression, Gaussian blur, and Gaussian noise. As shown in
Figure 5, the MMST maps of JPEG compressed, noisy, and blurred
videos are almost the same to the raw video. On the other hand,
the MesoNet handles frames independently and relies on detailed

0.5

820
821

0.4

Accuracy

822

0.2

823

0.85

824

20

10

a) JPEG Compression

0.5

70

40

30

825Compression Quality

0.8

819

0.92

0.9

0.8

Xception
10
MesoNetDeepRhythm

0.3

0.75

50

40

30

822

20

0.2

(a) JPEG Compression
823
0

10

5

0.92

100

90

80
0.9510

0

0.7 5

0.9

6015

70

50

20

40

(b) Blur
30

823

010

20

0.2

5
0

105

100

1510
80

90

0.92

Accuracy

0.5

0.85

Sampling

0.4

0.2

875
5

10

200.7515

Sampling
interval
0.205(c) Sampling
20
6015

70

40 0 20

50

520

30

877

0.9

874

0.8

875

0.7

010

10

5 15
0

0.5

878

0.4

25

1510

877

879

0.3

0.92

20

880

0.88

0.2

20 15

881

25

882
0 20

5

0

Standard Deviation
Sampling interval
Kernel Size Length (pixel)

5 15

20 10

883

25

873

0.9

874

0.8

875

0.7

0.6

0.5

878

0.4

879

0.3

880

0.2

881

15

20

Standard Deviation
Sampling interval

882

0.8

10

1

877

879

Noise

872

876

878

0.9

(d) Gaussian881
Noise

20 105

876
0.94

880

0.9

875

0.96

0.6

15

(b)(a)Blur(c) Felix:
Sampling(d) Gaussian(c)
Noise
JPEG
Compression
Blur
Sampling(b) Blur(a)
JPEG
Compression
878
Figure2: Baseline Comparisonadd(b)
stuff,
need
to be more informative
0.4

873

Noise

871

874

0.98

Standard Deviation

876

0.96

10

1

0.88

Sampling

873

1

877

0.8

0.9

Noise

872

876

0.4

0.3

Sampling interval
Kernel Size Length Compression(pixel)Quality

824

0.6

0.3

Xception
0
MesoNet0.98
DeepRhythm

0.3

822

0.5

874

0.88

20

Kernel Size Length (pixel)
0.6

0.6

873

0.4

0.75
15

0.94

872

0.5

0.8

821

10
0.88

0.5
Kernel Size Length Compression(pixel)Quality

824

820

Blur

0.75

0.4

Compression Quality

20

0.85

871

0.6

0.9

821
60

818

0.92

0.7

0

5

10

15

Standard Deviation

882(d) Gaussian
Noise
(c)
Sampling
884(d) Gaussian Noise

883
883
825
879
885
884 Felix: add stuff, need to be more informative884
826
826
Table
4: be
JPEG
Compression
Figure 2: Baseline Comparison Felix: add stuff, needtoinformative
Figure2: moreBaseline
Comparison
to beinformative
Figure2: moreBaseline
Comparison
880 Felix: add stuff, need
886
885
885
827
827
881 4: JPEG Compression
887
829
Table 4: JPEG Compression
Table
Table 4: JPEG Compression
Kernel Size Length (pixel)
Sampling intervalStandard Deviation
886
886
828
828
828
Xception
MesoNetDeepRhythm
882
888
830
(b) Blur(d) Gaussian
887
887
829
829 (c) Sampling
829 Noise
0.67±0.283
0.852±0.133
0.871±0.145
889
831
Xception
MesoNetDeepRhythm
Xception883 MesoNet
DeepRhythm
Xception
MesoNetDeepRhythm
888
888
830
830
830
Xception890
832
0.67±0.283 0.852±0.133
0.871±0.145
0.67±0.283884 0.852±0.133
0.871±0.145
0.67±0.283 0.852±0.133
0.871±0.145
Figure 2: Baseline Comparison
Felix: add stuff, need to be more informative
Table 5: Blur
889
889
831
831
831
MesoNet885
891
833
DeepRhythm
890
890
832
832
832
Table 4: JPEG Compression
Table 5: Blur
Table 5: Blur
Table 5: Blur
886
892
834
Xception
MesoNetDeepRhythm
891
891
833
833
Compression Quality 833
Kernel Size Length (pixel)
Sampling intervalStandard Deviation
887
893
835
0.847±0.088 0.843±0.035
0.834±0.072
892
892
834
834
834
Xception
MesoNetDeepRhythm
Xception
MesoNetDeepRhythm
Xception888
MesoNetDeepRhythm
Xception
MesoNetDeepRhythm
894
836
893 0.843±0.035
893
835
835
835 Table 6: Sampling
0.67±0.283 0.852±0.133
0.871±0.145
0.847±0.088 0.843±0.035
0.834±0.072
0.847±0.088 0.843±0.035
0.834±0.072
0.847±0.088
0.834±0.072
889
895
837
894
894
836
836
836
890 Table 6: Sampling
Table 5: Blur
Table 6: Sampling
Table 6: Sampling
896
838
895
895
837
837
837
Xception
MesoNetDeepRhythm
891
897
839
896
896
838
838
838
0.978±0.003
0.913±0.007
0.906±0.031
Xception
MesoNetDeepRhythm
Xception
MesoNetDeepRhythm
Xception892
MesoNetDeepRhythm
Xception
MesoNetDeepRhythm
898
840
897
897
839
839
839
893 0.913±0.007
0.847±0.088
0.843±0.035
0.834±0.072
0.978±0.003 0.913±0.007
0.906±0.031
0.978±0.003
0.906±0.031
0.978±0.003 0.913±0.007
0.906±0.031
899
841
Table
7:
Gaussian
Noise
898
898
840
840
840
894
900
842
899
899
841
841
841
Table 6: Sampling
Table 7: Gaussian Noise
Table 7: Gaussian Noise
Table 7: Gaussian Noise
895
901
843
Xception
MesoNetDeepRhythm
900
900
842
842
842
896
0.336±0.291
0.402±0.271
0.436±0.273
902
844
901
843
843
843
Xception
MesoNetDeepRhythm
Xception
MesoNetDeepRhythm
Xception
MesoNetDeepRhythm
Xception901
MesoNetDeepRhythm
897
903
845
0.978±0.003 0.913±0.007
0.906±0.031
0.336±0.291 0.402±0.271
0.436±0.273
0.336±0.291 Sampling
0.402±0.271
0.436±0.273
0.336±0.291
0.436±0.273
902 0.402±0.271
902
844
844
844
JPEG Compression
Blur
Noise
JPEG Compression
Blur
Sampling
Noise
898
904
846
903
903
845
845
845
899 Compression
Table
7: Gaussian Noise
Compression
Blur
Sampling
Noise
JPEG Compression
Blur
Sampling
Noise 905 Sampling
JPEG
Blur
Noise
847 JPEG
904
904
846
846
846
900
906
848
905
905
847
847
847
901
Xception
MesoNetDeepRhythm
907
849
906
906
848
848
848
0.336±0.291
0.402±0.271
0.436±0.273
902
908
850
907
907
849
849
DR-mmst 849
903
DR-mmst-A
909
851
908
908
850
850
850
Blur
Sampling
Noise
DR-mmstDR-mmstDR-mmstDR-mmst 904
DR-mmst-B
910
852
DR-mmst-A DR-mmst-AP
DR-mmst-A
DR-mmst-A
DR-mmst-A
909
909
851
851
851
DR-mmst-B DR-mmst-B -----------------------DR-mmst-B
DR-mmst-B905
911
853
910
910
852
852
852
DR-mmst-AP
DR-mmst-AP
DR-mmst-AP
DR-mmst-AP MesoNet
906---------------------------------------------------------------------DR-mmst-BF
912
854
911
911
853
853
------------------------ 853
MesoNet
MesoNet
MesoNet 907
DR-mmst-APBF
913
855
MesoNetDR-mmst-BF
DR-mmst-BF
DR-mmst-BF
912
912
854
854
854
DR-mmst-APBF-e2e908
DR-mmst-APBF
DR-mmst-APBF
DR-mmst-APBF
DR-mmst-BF
914
856
913
913
855
855
855
Compression Quality
Kernel Size Length (pixel)
Sampling intervalStandard Deviation
DR-mmst-APBF-e2e
DR-mmst-APBF-e2e
DR-mmst-APBF-e2e
DR-mmst-APBF
909
915
857
914
914
856
856
856
(a) JPEG Compression(c)Compression
Sampling(d)Kernel
Gaussian
DR-mmst-APBF-e2eCompression
Quality
Kernel Size Length (pixel)
Sampling intervalStandard Deviation Sampling interval
Compression Quality (b) Blur
Kernel Size Length (pixel)
Standard Deviation Sampling interval
QualitySize LengthNoise(pixel)Standard Deviation
910
916
858
915
915
857
857
857
(a) JPEG Compression(b) Blur(d) interval
Gaussian Noise(a) JPEG Compression(c) Sampling (b) Blur(d) Gaussian Noise(a) JPEG(c) Sampling(d) Gaussian Noise
Compression Quality
Kernel Size Length (pixel) (c) Sampling (b) Blur
Sampling
Standard Deviation
911Compression
917
859
Figure 3: Ablation Study
916
916
858
858
858 Felix: add stuff, need to be more informative
912
918be more informative 917
860
917Felix: add stuff, need to859
859 Figure 3: Ablation Study Felix: add stuff, need859
to be
more3:informative
FigureAblation Study Felix: add stuff, need to be
more3:informative
FigureAblation Study
Table 8: Gaussian Noise 913
919
861
918
918
860
860
860
914
Table 8: Gaussian
Noise
Table 8: Gaussian Noise
Table 8: Gaussian Noise
Kernel Size Length (pixel)
Sampling intervalStandard
Deviation
920
862
919
919
861
861
861
DR-mmstDR-mmst-A (d)DR-mmst-B
DR-mmst-AP
MesoNetDR-mmst-BF DR-mmst-APBF DR-mmst-APBF-e2e
915
(b)
Blur
(c)
Sampling
Gaussian
Noise
921
863
920
920
862
862
862
JPEG
Compression
0.838±0.007
0.836±0.006
0.841±0.006
0.855±0.006
0.852±0.133
0.853±0.146
0.868±0.133
916 DR-mmst-AP
DR-mmstDR-mmst-A
DR-mmst-B
DR-mmst-AP
MesoNetDR-mmst-BF
DR-mmst-APBF
DR-mmst-APBF-e2e
DR-mmstDR-mmst-A
DR-mmst-B
MesoNetDR-mmst-BF
DR-mmst-APBF MesoNet
DR-mmst-APBF-e2e
DR-mmstDR-mmst-A
DR-mmst-B 0.871±0.145
DR-mmst-AP
DR-mmst-BF DR-mmst-APBF DR-mms
922
864
921
921
863
863
863
Blur need
0.838±0.004
0.845±0.003
0.856±0.004
0.843±0.035
0.84±0.067
0.832±0.068
Compression
0.838±0.007
0.836±0.006
0.841±0.006
0.855±0.006
0.852±0.133
0.868±0.133
0.871±0.145
JPEG
Compression
0.838±0.007
0.836±0.006
0.841±0.006
0.852±0.133
0.853±0.146
0.868±0.133 0.852±0.133
0.871±0.145
JPEG0.853±0.146
Compression
0.838±0.007
0.836±0.006
0.841±0.006 0.834±0.072
0.855±0.006
0.853±0.146
0.868±0.133
0.871
917 0.855±0.006
FigureJPEG
3: Ablation
Study Felix:
add stuff,
to be0.843±0.003
more
informative
923
865
922
922
864
864
864
Sampling 0.838±0.004
0.762±0.05
0.754±0.04
0.688±0.101
0.9±0.030.843±0.035
0.897±0.033
Blur
0.843±0.003
0.845±0.003
0.856±0.004
0.843±0.035
0.84±0.067
0.832±0.068
0.834±0.072
Blur0.668±0.103
0.843±0.003
0.838±0.004
0.845±0.003
0.84±0.067 0.906±0.031
0.832±0.068 0.843±0.035
0.834±0.072
Blur0.913±0.007
0.843±0.003
0.838±0.004
0.845±0.003
0.856±0.004
0.84±0.067
0.832±0.068
0.834
918 0.856±0.004
924
866 Table 8: Gaussian Noise
923
923
865
865
865
Gaussian Noise
0.842±0.002
0.834±0.011
0.84±0.006
0.855±0.006
0.402±0.271
0.425±0.274
0.444±0.265
0.436±0.273
Sampling
0.762±0.05
0.668±0.103
0.754±0.04
0.688±0.101
0.913±0.007
0.9±0.03
0.897±0.033
0.906±0.031
Sampling
0.762±0.05
0.668±0.103
0.754±0.04
0.913±0.007
0.9±0.03
0.897±0.033 0.913±0.007
0.906±0.031
Sampling
0.762±0.05
0.668±0.103
0.754±0.04
0.688±0.101
0.9±0.03
0.897±0.033
0.906
919 0.688±0.101
925
867
924
924
866
866
866
Gaussian Noise
0.842±0.002 0.834±0.011 Gaussian
0.84±0.006
0.425±0.274
0.444±0.265
0.436±0.273
Noise 0.855±0.006
0.842±0.002 0.402±0.271
0.834±0.011 Gaussian
0.84±0.006
0.425±0.274
0.444±0.265 0.402±0.271
0.436±0.273
Noise920 0.855±0.006
0.842±0.002 0.402±0.271
0.834±0.011
0.84±0.006
0.855±0.006
0.425±0.274
0.444±0.265
0.436
926
868
DR-mmst-A
DR-mmst-B
DR-mmst-AP
MesoNetDR-mmst-BF DR-mmst-APBF 867
DR-mmst-APBF-e2e
925
925
867
867
921
0.836±0.006
0.841±0.006
0.855±0.006
0.852±0.133
0.853±0.146
0.868±0.133
927
869
926
926
868
868
868 0.871±0.145922
0.838±0.004
0.845±0.003
0.856±0.004
0.843±0.035
0.84±0.067
0.832±0.068
928
870
927
927
869
869
869 0.834±0.072
8923
0.668±0.103
0.754±0.04
0.688±0.101
0.913±0.007
0.9±0.03
0.897±0.033
928
928
870
870
870 0.906±0.031
8
8
8
924
827

0

5

825

0.9

828

0.6

0.9

10

15

20

0.7

0.3

0.88

0

5

10

0.5

0.94

0.2

0.85
15

20

0

5

10

15

20

Accuracy

30

50

1

0.7

0.7

0.9

Accuracy

40

60

0.8

Accuracy

50

80
0.94

827

0.7

0.75

60

70

90

0.8

826

Compression Quality

80

100

817
Noise

0.94

0.6

0.95

0.96

0.8

872

871

0.8

0.9

0.94
0.8

0.9

0.96

0.2

(a) JPEG
826 Compression

0.8

Accuracy

70

90

825

ption
oNet
pRhythm
80

100

0.9

819

0.98

Xception
823
1
MesoNetDeepRhythm
824

0.3

0.9

0.85

Xception
JPEG Compression
820
MesoNetDeepRhythm
0.3

822

0.7

Sampling

818

0.4

821

0.95

817

0.5

0.98

Anon.

0.9

0.95

816

0.96

0.85

Blur

1

Anon.

871

Accuracy

820

Sampling

Noise

Accuracy

819

819

Noise

Blur JPEG Compression

0.96

Accuracy

818

Blur

0.6

815

0.9

0.8

0.9

Accuracy

Accuracy

0.7

0.9

0.95

816
Accuracy

817

JPEG Compression

815

0.6

1

Accuracy

0.7

818

0.98

Accuracy

817

0.8

1

0.9

814

1

Accuracy

0.9

Sampling

813

0.98

0.95

814

Accuracy

816

Accuracy815
media ’20, October 12–16, 2020, Seattle,
WA

Blur JPEG Compression

813

Accuracy

0.8

JPEG Compression

Anon.

Sampling

Accuracy

0.9

816

1

Accuracy815

814

Accuracy1

Accuracy

814

813

Blur

JPEG Compression

813

Accuracy

ACM Multimedia ’20, October 12–16, 2020, Seattle, WAACM Multimedia ’20, October 12–16, 2020, Seattle, WAACM Multimedia ’20, October 12–16, 2020, Seattle, WA

0.92

25

0.8

0.4

0.6

0.5

0.4

0.9

0.3

0.3

0.75

0.88

0.2

0.2

100

90

80

70

60

50

40

30

20

0

10

5

10

(a) JPEG

15

20

0

5

(b) Blur

10

15

20

0

5

(c) Sampling

10

15

20

25

(d) Gaussian Noise

Figure 6: Baseline comparison on robustness. We perform DeepFake detection through DeepRhythm and two state-of-the-art baselines, i.e., Xception and MesoNet,
on a degradation dataset. Four degradations, i.e., JPEG compression, Gaussian blur, temporal sampling, and Gaussian noise, are added to the testing set of the ALL
dataset. The average accuracy and corresponding standard deviation across all degradation degrees are presented at the bottom of each sub-figure.

1

1

1

0.9

0.7

0.7

0.85

0.6

0.8

0.9

0.8

0.6

0.8

0.5

0.6

0.75

90

80

70

60

0.8

100

90

80

70

100

60

90

50

40

80

0.890

80

70

60

50

0.5
30

40

10

0

0.75

30

20

10

0

5

90 10

100

80

70

0.7

70

5

0.75
15

10

0.7

0.85

0.4

0.6

60

50

20
40

30

5

10

10

0

0

5

0.7

0.6

0.5

0.4

0.4

0.6

0.3

0.3

0.75

0.5

20

5

10

15

0.2

0 20

0.6

0.4

0.2

0.3

0.6

0.7

0.8

0.5

0.5

0.80.3

0.4

0.5

0.6

15

0.7

0.8

0.7

0.5

0.6

0.7

20

Accuracy

0.5

0.85

Accuracy

Accuracy

100

0.5

0.6

0.5

0.75

0.5

100

15

90 10

80

20 70

15

60

20

0

5

10

0.3

15

20

0.5

0

50

20
405

30

25

0.2

01020

10

15

5

0

20

10

0.5

525

15

0.2

20

10

15

0.2

0.5

0

20 5

010

15

20

5

10

25

15

20

0

0.5

60

50

40

30

20

10

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

25

0.8

0.4

0.6

(a) JPEG

0.75

40

30

20

(a) JPEG Compression

10

0

5

10

15

20

(b) Blur

0.3(c) Sampling(d) Gaussian Noise

0.2

0.5

50

Compression Quality

mpression
lurpling
an Noise

0.7

0.6

0.9

DR-mmstDR-mmst-A
DR-mmst-B
DR-mmst-AP---------------------MesoNet
DR-mmst-BF
DR-mmst-APBF
DR-mmst-APBF-e2e

0.6

0.9
0.6

0.6

0.9

0.95

0.7

0.8

Accuracy

1

0.8

0.8

0.7

0.7

Accuracy

0.85

0.9

0.8

0.8

Accuracy

Accuracy
Accuracy

0.7

0.8

Accuracy

0.85

0.8

Accuracy

0.6

Accuracy

1

0.7

0.9

0.85

Accuracy

Accuracy

0.8

Accuracy

Accuracy

Accuracy

Accuracy

Accuracy

0.7

0.9

0.9

0.7

0.9

0.8

0.9

0.8

0.8

0.8

1

0.9

0.95

0.9

0.9

Accuracy

0.9

0.8

0.9

0.95

0.9

1

1

0.9

Accuracy
Accuracy

0.95

0.8

JPEG Compression

0.9

1

1

0.9

0.9

1

1

0.9

0.95

Accuracy

1

0.95

1

0.9

Accuracy

1

1

1

0

5

10

15

20

0

5

10

15

20

25

(e) Average accuracy ± standard deviation across degradation degrees
DR-mmst0.838±0.007
0.843±0.003
0.762±0.05 Figure 7: Ablation Study on robustness. We perform DeepFake detection through MesoNet and DeepRhythm’s seven variants on a degradation dataset. The
are clustered
two types, i.e.,
F-cluster0.444±0.265
using MesoNet0.436±0.273
for frame-level temporal attention (i.e., MesoNet itself, DR-mmst-BF, DR-mmst-APBF,
0.842±0.002compared
0.834±0.011methods
0.84±0.006
0.855±0.006 to 0.402±0.271
0.425±0.274

925
and DR-mmst-APBF-e2e), and non-F-cluster that does not employ MesoNet (i.e., DR-mmst,
DR-mmst-A, DR-mmst-B, and DR-mmst-AP). For each degradation,926
the average accuracy and corresponding standard deviation across all degradation degrees
are presented at the bottom of figure.
927

8

information within frames. As a result, it helps our methods berobust to temporal sampling and achieve the best performance but
is sensitive to local pixel variation. Clearly, the advantages anddisadvantages of MMSTR and MesoNet are complementary. Our
final version combining these two modules shows comprehensive
robustness across all degradations.

5

CONCLUSIONSIn this work, we have proposed DeepRhythm, a novel DeepFake detection technique. It is intuitively motivated by the fact that remotevisual photoplethysmography (PPG) is made possible by monitoring
the minuscule periodic changes of skin color due to blood pumping
through the face. Our extensive experiments on FaceForensics++
and DFDC-preview datasets confirm our conjecture that normal
heartbeat rhythms in the real face videos are disrupted in a DeepFake video, and further demonstrate not only the effectivenessof DeepRhythm, but how it generalizes over different datasets by
various DeepFake generation techniques. One interesting future
direction is to study the combined effort of DeepRhythm with other928

DeepFake detectors [33, 34, 65, 66]. Beyond DeepFake detection,
the investigation of how DeepRhythm can be applied further todomains such as countering non-traditional adversarial attacks[7, 26, 27, 64] is also potentially viable. In addition, the possibility ofusing tracking methods [16, 23–25, 70] to mine more discriminative
spatial-temporal features would also be further studied.

ACKNOWLEDGMENTS
This research was supported by JSPS KAKENHI Grant No. 20H04168,
19K24348, 19H04086, JST-Mirai Program Grant No. JPMJMI18BB,
Japan. It was also supported by Singapore National Cybersecurity
R&D Program No. NRF2018NCR-NCR005-0001, National Satelliteof Excellence in Trustworthy Software System No. NRF2018NCRNSOE003-0001, NRF Investigatorship No. NRFI06-2020-0022, andthe National Natural Science Foundation of China under contracts
Nos. 61871258 and U1703261 and the National Key Research andDevelopment Project under contracts No. 2016YFB0800403. We
gratefully acknowledge the support of NVIDIA AI Tech Center(NVAITC) to our research.

5

10

15

20

REFERENCES[1] D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen. 2018. MesoNet: a Compact
Facial Video Forgery Detection Network. In 2018 IEEE International Workshop on
Information Forensics and Security (WIFS).
[2] G. Balakrishnan, F. Durand, and J. Guttag. 2013. Detecting Pulse from Head Motions in Video. In 2013 IEEE Conference on Computer Vision and Pattern Recognition.
3430–3437.
[3] Belhassen Bayar and Matthew Stamm. 2016. A Deep Learning Approach to
Universal Image Manipulation Detection Using a New Convolutional Layer. 5–
10.
[4] Christoph Bregler, Michele Covell, and Malcolm Slaney. 1997. Video Rewrite:
Driving Visual Speech with Audio. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). ACM
Press/Addison-Wesley Publishing Co., USA, 353–360. https://doi.org/10.1145/
258734.258880[5] P. Buchana, I. Cazan, M. Diaz-Granados, F. Juefei-Xu, and M.Savvides. 2016. Simultaneous Forgery Identification and Localization in Paintings Using Advanced
Correlation Filters. In ICIP.
[6] Giovanni Cennini, Jeremie Arguel, Kaan Akşit, and Arno Leest. 2010. Heart ratemonitoring via remote photoplethysmography with motion artifacts reduction.
Optics express 18 (03 2010), 4867–75. https://doi.org/10.1364/OE.18.004867[7] Yupeng Cheng, Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Shang-Wei Lin, Weisi
Lin, Wei Feng, and Yang Liu. 2020. Pasadena: Perceptually Aware and Stealthy
Adversarial Denoise Attack. arXiv preprint (2020).
[8] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, andJaegul Choo. 2017. StarGAN: Unified Generative Adversarial Networks for MultiDomain Image-to-Image Translation. arXiv:cs.CV/1711.09020[9] Francois Chollet. 2017. Xception: Deep Learning with Depthwise Separable
Convolutions. 1800–1807.
[10] Davide Cozzolino, Giovanni Poggi, and Luisa Verdoliva. 2017. Recasting Residualbased Local Descriptors as Convolutional Neural Networks: an Application to
Image Forgery Detection. 159–164.
[11] Kevin Dale, Kalyan Sunkavalli, Micah K. Johnson, Daniel Vlasic, Wojciech Matusik, and Hanspeter Pfister. 2011. Video Face Replacement. ACM Trans. Graph.
30, 6 (Dec. 2011), 1–10. https://doi.org/10.1145/2070781.2024164[12] Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil Jain. 2019. On the
Detection of Digital Face Manipulation. arXiv:cs.CV/1910.01717[13] Vincent Nozick Darius Afchar. [n. d.]. FaceForensics Benchmark. http://kaldir.vc.
in.tum.de/faceforensics_benchmark/.
[14] Vincent Nozick Darius Afchar. [n. d.]. MesoNet. https://github.com/DariusAf/
MesoNet/.
[15] Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. 2019. The Deepfake Detection Challenge (DFDC) Preview Dataset.
arXiv:cs.CV/1910.08854[16] Wei Feng, Ruize Han, Qing Guo, Jianke Zhu, and Song Wang. 2019. Dynamic
Saliency-Aware Regularization for Correlation Filter-Based Object Tracking. IEEE
TIP 28, 7 (2019), 3232–3245.
[17] Jessica Fridrich and Jan Kodovsky. 2012. Rich Models for Steganalysis of Digital
Images. IEEE Transactions on Information Forensics and Security 7 (06 2012),
868–882.
[18] Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormählen, Patrick Pérez,and Christian Theobalt. 2014. Automatic Face Reenactment. 2014 IEEE Conference
on Computer Vision and Pattern Recognition (2014), 4217–4224.
[19] P. Garrido, Levi Valgaerts, Hamid Sarmadi, I. Steiner, Kiran Varanasi, P. Pérez,and C. Theobalt. 2015. VDub: Modifying Face Video of Actors for Plausible Visual
Alignment to a Dubbed Audio Track. Computer Graphics Forum 34 (05 2015).
https://doi.org/10.1111/cgf.12552[20] Miroslav Goljan and Jessica Fridrich. 2015. CFA-aware features for steganalysis ofcolor images. Proceedings of SPIE - The International Society for Optical Engineering
9409 (03 2015).
[21] E. Gonzalez-Sosa, J. Fierrez, R. Vera-Rodriguez, and F. Alonso-Fernandez. 2018.
Facial Soft Biometrics for Recognition in the Wild: Recent Works, Annotation,
and COTS Evaluation. IEEE Transactions on Information Forensics and Security
13, 8 (2018), 2001–2014.
[22] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative
Adversarial Networks. arXiv:stat.ML/1406.2661[23] Qing Guo, Wei Feng, Ce Zhou, Rui Huang, Liang Wan, and Song Wang. 2017.
Learning Dynamic Siamese Network for Visual Object Tracking. In ICCV. 1781–
1789.
[24] Qing Guo, Wei Feng, Ce Zhou, Chi-Man Pun, and Bin Wu. 2017. StructureRegularized Compressive Tracking With Online Data-Driven Sampling. IEEE TIP
26, 12 (2017), 5692–5705.
[25] Qing Guo, Ruize Han, Wei Feng, Zhihao Chen, and Liang Wan. 2020. Selective
Spatial Regularization by Reinforcement Learned Decision Making for Object
Tracking. IEEE TIP 29 (2020), 2999–3013.

[26] Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Jian Wang, Bing Yu, Wei Feng, andYang Liu. 2020. Watch out! Motion is Blurring the Vision of Your Deep Neural
Networks. arXiv preprint arXiv:2002.03500 (2020).
[27] Qing Guo, Xiaofei Xie, Felix Juefei-Xu, Lei Ma, Zhongguo Li, Wanli Xue, Wei
Feng, and Yang Liu. 2020. SPARK: Spatial-aware Online Incremental Attack
Against Visual Tracking. European Conference on Computer Vision (ECCV) (2020).
[28] D. Güera and E. J. Delp. 2018. Deepfake Video Detection Using Recurrent Neural
Networks. In 2018 15th IEEE International Conference on Advanced Video andSignal Based Surveillance (AVSS). 1–6.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual
Learning for Image Recognition. In The IEEE Conference on Computer Vision and
Pattern Recognition (CVPR).
[30] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen. 2019. AttGAN: Facial Attribute
Editing by Only Changing What You Want. IEEE Transactions on Image Processing
28, 11 (2019), 5464–5478.
[31] Javier Hernandez-Ortega, Julian Fierrez, Aythami Morales, and Pedro Tome. 2018.
Time Analysis of Pulse-Based Face Anti-Spoofing in Visible and NIR. 657–6578.
[32] Guillaume Heusch and Sébastien Marcel. 2018. Pulse-based Features for Face
Presentation Attack Detection. 1–8.
[33] Yihao Huang, Felix Juefei-Xu, Run Wang, Qing Guo, Lei Ma, Xiaofei Xie, Jianwen
Li, Weikai Miao, Yang Liu, and Geguang Pu. 2020. FakePolisher: Making DeepFakes More Detection-Evasive by Shallow Reconstruction. ACM International
Conference on Multimedia (ACM MM) (2020).
[34] Yihao Huang, Felix Juefei-Xu, Run Wang, Qing Guo, Xiaofei Xie, Lei Ma, Jianwen Li, Weikai Miao, Yang Liu, and Geguang Pu. 2020. FakeLocator: Robust
Localization of GAN-Based Face Manipulations. arXiv preprint arXiv:2001.09598(2020).
[35] Kenneth Humphreys, Tomas Ward, and Charles Markham. 2007. Noncontact
simultaneous dual wavelength photoplethysmography: A further step towardnoncontact pulse oximetry. The Review of scientific instruments 78 (05 2007),
044304. https://doi.org/10.1063/1.2724789[36] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive Growing of GANs for Improved Quality, Stability, and Variation.
arXiv:cs.NE/1710.10196[37] Tero Karras, Samuli Laine, and Timo Aila. 2018. A Style-Based Generator Architecture for Generative Adversarial Networks. arXiv:cs.NE/1812.04948[38] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, andTimo Aila. 2019. Analyzing and Improving the Image Quality of StyleGAN.
arXiv:cs.CV/1912.04958[39] X. Li, J. Chen, G. Zhao, and M. Pietikäinen. 2014. Remote Heart Rate Measurementfrom Face Videos under Realistic Situations. In 2014 IEEE Conference on Computer
Vision and Pattern Recognition. 4264–4271.
[40] Xiaobai Li, Jukka Komulainen, Guoying Zhao, Pong-Chi Yuen, and Matti
Pietikainen. 2016. Generalized face anti-spoofing by detecting pulse from facevideos. 4244–4249.
[41] Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, andShilei Wen. 2019. STGAN: A Unified Selective Transfer Network for Arbitrary
Image Attribute Editing. arXiv:cs.CV/1904.09709[42] Yaojie Liu, Amin Jourabloo, and Xiaoming Liu. 2018. Learning Deep Models forFace Anti-Spoofing: Binary or Auxiliary Supervision. 389–398.
[43] Lei Ma, Felix Juefei-Xu, Jiyuan Sun, Chunyang Chen, Ting Su, Fuyuan Zhang,
Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems. In The 33rd
IEEE/ACM International Conference on Automated Software Engineering (ASE).
[44] Lei Ma, Felix Juefei-Xu, Minhui Xue, Bo Li, Li Li, Yang Liu, and Jianjun Zhao.
2019. DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems.
Proceedings of the IEEE International Conference on Software Analysis, Evolution
and Reengineering (SANER) (2019).
[45] Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie,
Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepMutation: Mutation
Testing of Deep Learning Systems. In The 29th IEEE International Symposium on
Software Reliability Engineering (ISSRE).
[46] Scott McCloskey and Michael Albright. 2018. Detecting GAN-generated Imagery
using Color Cues. arXiv:cs.CV/1812.08247[47] Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. 2019. Use of a Capsule
Network to Detect Fake Images and Videos. arXiv:cs.CV/1910.12467[48] X. Niu, S. Shan, H. Han, and X. Chen. 2020. RhythmNet: End-to-End Heart Rate
Estimation From Face via Spatial-Temporal Representation. IEEE Transactions on
Image Processing 29 (2020), 2409–2423.
[49] Ewa Nowara, Ashutosh Sabharwal, and Ashok Veeraraghavan. 2017. PPGSecure:
Biometric Presentation Attack Detection Using Photopletysmograms. 56–62.
[50] Tae-Hyun Oh, Ronnachai Jaroensri, Changil Kim, Mohamed Elgharib, Frédo
Durand, William T. Freeman, and Wojciech Matusik. 2018. Learning-based Video
Motion Magnification. arXiv:cs.CV/1804.02684[51] Online. [n. d.]. 81 Facial Landmarks Shape Predictor. https://github.com/codeniko/shape_predictor_81_face_landmarks/.
[52] Online. [n. d.]. Face Recognition Using Pytorch. https://github.com/timesler/
facenet-pytorch/.

[53] Xunyu Pan, Xing Zhang, and Siwei Lyu. 2012. Exposing image splicing withinconsistent local noise variances. 2012 IEEE International Conference on Computational Photography, ICCP 2012 (04 2012).
[54] M. Poh, D. J. McDuff, and R. W. Picard. 2011. Advancements in Noncontact,
Multiparameter Physiological Measurements Using a Webcam. IEEE Transactions
on Biomedical Engineering 58, 1 (2011), 7–11.
[55] Ming-Zher Poh, Daniel McDuff, and Rosalind Picard. 2010. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics express 18 (05 2010), 10762–74. https://doi.org/10.1364/OE.18.010762[56] Nicolas Rahmouni, Vincent Nozick, Junichi Yamagishi, and I. Echizen. 2017.
Distinguishing computer graphics from natural images using convolution neuralnetworks. 1–6.
[57] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies,
and Matthias Nießner. 2019. FaceForensics++: Learning to Detect Manipulated
Facial Images. In International Conference on Computer Vision (ICCV).
[58] Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, andPrem Natarajan. 2019. Recurrent Convolutional Strategies for Face Manipulation
Detection in Videos. arXiv:cs.CV/1905.00582[59] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. 2016.
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. AAAI Conference on Artificial Intelligence (02 2016).
[60] Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred Neural
Rendering: Image Synthesis using Neural Textures. arXiv:cs.CV/1904.12356[61] Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015. Real-time Expression Transfer for Facial Reenactment. ACM Transactions on Graphics 34 (10 2015), 1–14. https://doi.org/10.1145/2816795.2818056[62] J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. 2016.
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. In 2016

IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2387–2395.
[63] Paul Upchurch, Jacob Gardner, Kavita Bala, Robert Pless, Noah Snavely, andKilian Weinberger. 2016. Deep Feature Interpolation for Image Content Changes.
(11 2016).
[64] Run Wang, Felix Juefei-Xu, Qing Guo, Yihao Huang, Xiaofei Xie, Lei Ma, and Yang
Liu. 2020. Amora: Black-box Adversarial Morphing Attack. ACM International
Conference on Multimedia (ACM MM) (2020).
[65] Run Wang, Felix Juefei-Xu, Yihao Huang, Qing Guo, Xiaofei Xie, Lei Ma, and Yang
Liu. 2020. DeepSonar: Towards Effective and Robust Detection of AI-Synthesized
Fake Voices. ACM International Conference on Multimedia (ACM MM) (2020).
[66] Run Wang, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yihao Huang, Jian Wang, andYang Liu. 2020. FakeSpotter: A Simple yet Robust Baseline for Spotting AISynthesized Fake Faces. International Joint Conference on Artificial Intelligence(IJCAI) (2020).
[67] Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, andWilliam T. Freeman. 2012. Eulerian Video Magnification for Revealing Subtle
Changes in the World. ACM Transactions on Graphics (Proc. SIGGRAPH 2012) 31,
4 (2012).
[68] Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun
Zhao, Bo Li, Jianxiong Yin, and Simon See. 2019. DeepHunter: A CoverageGuided Fuzz Testing Framework for Deep Neural Networks. In ACM SIGSOFTInternational Symposium on Software Testing and Analysis (ISSTA).
[69] Zitong Yu, Wei Peng, Xiaobai Li, Xiaopeng Hong, and Guoying Zhao. 2019. Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-toend Deep Learning Solution with Video Enhancement. arXiv:eess.IV/1907.11921[70] Ce Zhou, Qing Guo, Liang Wan, and Wei Feng. 2017. Selective object and context
tracking. In ICASSP. 1947–1951.
[71] Peng Zhou, Xintong Han, Vlad I. Morariu, and Larry S. Davis. 2018. Two-Stream
Neural Networks for Tampered Face Detection. arXiv:cs.CV/1803.11276