Delving into Sequential Patches for Deepfake Detection
Authors: Jiazhi Guan
Hang Zhou
Zhibin Hong
Errui Ding
Jingdong Wang
Chengbin Quan
Youjian Zhao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deepfake Detection
Digital Forensics

Abstract: Recent advances in face forgery techniques produce nearly visually untraceable deepfake videos, which could be leveraged with malicious intentions. As a result, researchers have been devoted to deepfake detection. Previous studies have identified the importance of local low-level cues and temporal information in pursuit to generalize well across deepfake methods, however, they still suffer from robustness problem against post-processings. In this work, we  propose the Local- & Temporal-aware Transformer-based Deepfake Detection (LTTD) framework, which adopts a local-to-global learning protocol with a particular focus on the valuable temporal information within local sequences. Specifically, we propose a Local Sequence Transformer (LST), which models the temporal consistency on sequences of restricted spatial regions, where low-level information is hierarchically enhanced with shallow layers of learned 3D filters. Based on the local temporal embeddings, we then achieve the final classification in a global contrastive way. Extensive experiments on popular datasets validate that our approach effectively spots local forgery cues and achieves state-of-the-art performance.

Delving into Sequential Patches for Deepfake Detection
Jiazhi Guan 1.2, Hang Zhou?, Zhibin Hong?, Errui Ding" Jingdong Wang". Chengbin Quan', Youjian Zhao 1.3*
1 *Department of Computer Science and Technology, Tsinghua University
DDpartment of Computer Vision Technology (VIS), Baidu Inc. *Zhongguancun Laboratory {guan ja2ammal2ss quancb0, zhaoyoujjaanD):singnna edu. cn
(zhouhang09, dingerrui, vangjingdoong/Oaaiuu.
Abstract
Recent advances in face forgery techniques produce nearly visually untraceable deepfake videos, which could be leveraged with malicious intentions. As a re- sult, researchers have been devoted to deepfake detection. Previous studies have identified the importance of local low-level cues and temporal information in pur- suit to generalize well across deepfake methods, however, they still suffer from robustness problem against post-processings. In this work, we propose the Local- & Temporal-aware Transformer-based Deepfake Detection (LTTD) framework, which adopts local-to-globa learning protocol with a particular focus on the valuable temporal information within local sequences. Specifically, we propose a Local Sequence Transformer (LST), which models the temporal consistency on se- quences of restricted spatial regions, where low-level information hierarchically enhanced with shallow layers of learned 3D filters. Based on the local temporal embeddings, we then achieve the final classification Extensive experiments on popular datasets validate that our approach effectively spots local forgery cues and achieves state-of-the-ar performance.
global contrastive way.
Introduction
With the development of face forgery methods [
an enormous amount of fake videos (a.k.a deepfakes) have raised non-neglectable concerns privacy preservation and information security. To this end, researches have been devoted to the reliable tagging deepfakes in order to block the propagation of malicious information. However, it is still an open problem due to the limited generalization of detection methods and the continuous advances deepfake creation.
25 26
45 69 devote efforts to enhance general Convolutional Neural Networks (CNN) for identifying clear semantic distortions deepfakes. Recent methods pay attention to the temporal inconsistency problem 6
Earlier studies 7 34 13 14
50, 5l yet most of them fall into semantic motion understanding (e. g., detecting abnormal eye blinking, phoneme-viseme mismatches, aberrant landmark fluctuation). These methods are able to learn specific forgery patterns, however, the remarkable visual forgery cues are expected to be gradually eliminated during the continuous arms race between forgers and detectors. As result, the generalizability of previous methods is typically unsatisfactory when encountering deepfakes generated by unseen techniques.
33
21
58
51
60
On the other hand, the importance of low-level information is identified for tackling the generalization problem. A group of studies are developed using hand-made low-level filters (c. g.. DCT
44 19 29),
steganalysis fcatures [67]. SRM [40) to better capture subtle differences between generated textures and the natural ones. However, methods depending on recognizable low-level patterns would become less effective on degraded data with commonly applied post-prost post-processing procedures like visual compression
[39 21 65 which indicates the lack of robustness.
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this work, we rethink the appropriate representation that can ensure both generalizability and robustness in deepfake detection. The inspiration is taken from Liu al. 39, which indicates that deepfakes can be directly distinguished from pieces of skin patches. Such practice prevents network from overfitting to global semantic cues, making learned patterns more generalizable. the creation of deepfakes inevitably relies on frame re- re-assembling. the substantial temporal differences also arise locally during the independent local modifications of forged frames. As the underlying temporal patterns are less affected by spatial interference, such temporal inconsistency will not be easily erased during common perturbations, making the low-level modeling more robust.
addition, as
Motivated by the observations above, we propose the Local- & Temporal-awee aware Transformer based Deepfake Detection (LTTD) framework, which particularly focuses on patch sequence modeling in deepfake detection with Transformers [17]. Detailedly, we divide the 3D video information spatially into independent local regions (as shown in Fig. []. Encouraged by the recent success of vision transformers [17], we formulate the patch sequence modeling problem in self-attention style and propose a Local Sequence Transformer (L.ST) module, which operates on sequences of local patches. Benefited from the attention mechanism, LST is not constrained by the receptive field in CNNs, allowing for better learning of both long and short span temporal patterns. In addition, we hierarchically inject low-level results from shallow 3D convolutions after self-attention layers to enhance the low-level feature learning at multiple scales. Such design particularly emphasizes on the low-level information from a temporal consistency perspective.
After modeling each sequence of local patches in LST, two questions remain to be solved: 1) how to model their inherent relationships and 2) how to aggregate their information for final prediction. We explicitly impose global contrastive supervision on patch embeddings with a Cross-Patch Inconsis- tency (CPI) loss. Then the final predictions is given in the Cross-Patch Aggregation (CPA) module with follow up Transformer blocks. Ablations on these designs show non-trivial improvements.
Our contributions can be summarized as follows: 1) We propose the Local- & Temporal-aware Transformer-based Deepfake Detection (LTTD) framework, which emphasizes low-level local tem- poral inconsistency by modeling sequences of local patches with Transformer. 2) We design the Cross-Patch Inconsistency (CPI) loss and the Cross-Patch Aggregation (CPA) module, which ef- ficiently aggregate local information for global prediction. 3) Quantitative experiments show that our approach achieves the tate-of-the--a generalizability and robustness. Qualitative results further illustrate its interpretability.
2 Related work
2.1 Deepfake detection
In recent years, we witness great progress in deepfake detection, where numerous forgery spotting models are proposed successively to address the practical demands of the application. In the earlier stage, methods 7Z) 34| 4 14 45 69] are built with a major emphasis on spotting semantic visual artifacts with sophisticated model designs. Dang et al. [13 design segmentation task optimized with the classification backbone simultancously to predict the forgery regions. Zhao et al. [63] regard the deepfake detection task as a fine fine-grained classification problem. While these methods achieve satisfied in na-aataet results, the cros--ataseet evaluation shows poor generalizability.
Nevertheless, generalizability is considered be the first point should it be designed for practical application scenarios, since we never have a chance to foresee the attackers' movements. More works [56] 10 49 begin notice the vital problem. Wang et al. [56] argue that the lack of generalizability is due to overfitting to significant semantic visual artifacts, and propose a dynamic data argumentation schema to relieve the issue. But considering the CNN always takes the downsampled semantic representation for final classification, their method tends to only focus on more semantic visual 39 D 20 dig deeper into the fundamental differences of deepfakes from the generation process. With the expectation that forgery methods will gradually improve, those works propose to identify deepfakes from low-level image features rather than semantic visual clues, as the latter are disappearing in the latest deepfakes.
artifacts. Another group of works [3] 10 64 44 19
29
40 36
31 10 64 both notice the content-indepe innteentdindeeeneen low-level features that can uniquely identify their
sources and the identity swapping will destroy the origin consistency. Li et al. [31 propose to detect those low-level features across facial boundary. Zhho et al. (64] and Chen al. [10 turn to
learn the spatial-local inconsistencies. Other methods look for clues in the frequency domain. With

---

fuse the low-level frequency pattern learning into CNN to improve the instead theoretically analyze that phase information are more sensitive to upsampling, therefore, such low-level feature is more crucial than high-level semantic information for our task. The finds of [39] further encourage us to detect deepfakes from local patches, where the generated images are easy to be isolated even only one small skin patch is given. However, there is still a vital issue that low-level features should be even less robust to real-world distortions. Experiments in [39] show that simple smoothing could impair the performance of more than 20% Drastic performance drops of [31] on suspected videos with degradation also indicate the weakness of low-level pattern learning. Different from current works, propose to rely temporal low-level changes with learnable shallow filters to better cope with real-world degradations.
DCT transform, 44 19 29
generalizability. Liu et al. 36
Although temporal methods 6 33 21 58 50 65 are also studied, most of they fall into the visual anomalous pattern learning such as abnormal eye blinking [3], non-synchronized lip movements 211 inconsistent facial movements [65] etc. In the foreseeable further or even good cases of current arts, we could hardly find these significant patterns. In contrast, we explore the substantial temporal inconsistency of independently frame-wise generation at local patches, which is more generalizable and also more robust to common perturbations.
2.2
Vision transformer
Vaswani et al. 55 first propose to using only self-attention, multilayer perceptron, and layer norm to establish a new canonical form, coined Transformer, for natural language processing (NLP). Promoted by the great advancement NLP, researchers of vision communities also start explore the potential of transformer designs. VIT [17] could be the first success to apply pure typical transformer in image classification. After that, many researches also extend transformer into different vision tasks, (e. g..
semantic segmentation [8] action recognition 45 32 video understanding [38] g 62 scene graph
generation 12 object detection [68 59 and achieve exceptional performance.
Recently, transformer structure is also adopted by few works [65] 16 57] in deepfake detection. [57] stack multiple ViTs [77 with different spatial embedding size. With the motivation learn identity information, a transformer with an extra id-token is proposed in [16]. Besides, the closest work to us, [65] adopt few self aiff-attention layers to aggregate successive frames' features, which are completely downsampled semantic embeddings. While in our method, transformer is introduced to achieve patch- seeueeseuentill temporal learning a restricted spatial receptive field with totally different purpose to identify low-level temporal inconsistency.
3 Approach
In this section, we will elaborate the proposed Local- & Temporal- aware Transformer-bast based Deepfake
Detection (LTTD) framework after formally define the problem in section 3.1] We describe Local Sequence Transformer (LSTT) in section 3.2 Then, the Cross-Patch Inconsistency (CPI) loss is introduced with the Cross-Patch Aggregation (CPA) module in section 3.3
3.1 Problem statement
Given one image E RCCHXW we reshape it into a sequence of flattened 2D patches fxt e RCPP1 | 1, NJ. where C is the number of image channel, P is the patch size, and N HW/P² 22 is the patch number. In the original setting of ViT [17], the separated patches can be directly sent into the Transformer blocks for semantic understanding. Differently, in task, take as input, where the extra T indicates the clip length, i. e., with T successive frames. We also split each frame into independent patches for better low-level forgery pattern learning, but with an additional temporal dimension. The flattened patch set is represented
video clip v E XTXXXXXx
as s = fx E R It 1, .... T; = 2,.. NN. where x stands for the flattened patch at the i-th spatial region of the 1-th frame. Thus the total patch number becomes T N. After that, our proposed LTTD leverages the patch locality with learnable filters and powerful self-attention operations to explore the low-level temporal inconsistency of deepfakes, and finally give a prediction (fake or real) based on global contrast across the whole spatial region.
3

---

Data Flow
Gruup
EST
CPA
Widgs W
Real or Fake
LST
CPI
Real
Weights
[leed
Heigg
CPA
Fake
Transtamar Enooder Layer
Num. of Sequenocss IIW
x
LST
Sequenoel Length
[eland]]
Local Sequence Embedding
LET-stage x
Convadi Maxpeoling
Convädi Maxpoeling
Fiatlan& Linear
Flathens Lineer
FlattenSLinear
CC
Transformer Encoder Layer x
indicator
(10)
sim
Tempural Augoooling
tomporal
Lopt
embedding after LST
taken after LST patch embedslng
(g((2)
simngt
Figure 1: The overall pipeline and details of the proposed Local- & cemporal-awar Transformer- based Deepfake Detection (LTTD). Top left: divide the whole process into three cascaded parts as Group Patch Sequences. LST forward, and CPI&CPA. Others: we illustrate the details of Local Sequence Transformer (LST), Cross-Patch Inconsistency (CPI), and Cross-Patee Aggregation (CPA).
3.2 Local Sequence Transformer
As we discussed previously, in order to learn the low-level temporal patterns, we feed the local patches at the same spatial position into the proposed Local Sequence Transformer (LST) for further temporal encoding, i. the input of our LST is set like s' {x** E gC-r² 1,2, ...," T', where
i 2,,2,.., N), and s² includes frame patches at the spatial location of
We show the details of the LST in the middle of Fig. [ which is divided into two parts: the Local Sequence Embedding and the Low-level Enhanced Transformer stages. both parts, low-level temporal enhancement is consistently introduced by 3D convolutions given three intuitions: 1) Learnable shallow filters would be better than hand-crafted filters in capturing low-level information in complex situations. 2) Voxels at one specific spatial location may not be well aligned temporally considering camera movements. Thus 3D filters which covers neighborhood structures are more suitable than 2D ones when handling this kind of situation. 3) The patch embedding in Transformers always projects the patch as a whole without more fine-grained locality emphasis. We make up for this using shallow convolutions to enhance low-level feature learning multiple scales.
Local Sequence Embedding. In addition commonly used linear patch embedding, we involve 3D convolutions at the beginning to enhance low-level temporal modeling, More formally, we define this part follows:
2sh Fx 1., E,x21 :
F.x E, E (X.C-) :...:y" yo T (Convol(onnv3d((x''
[yo Yo 2
x xx;;;..; (.));)))
(2)
zt Eavl' Eoyo. 2.1, EayT E. (P/A(()
z tempi 2 (1) o (2t) (1)) (2 a (zt (2)); 2s) (T) - (2t)(T))) + + Epps
piempi z 11: 23 [..;
Xiemp E D. Epos E R (T+))x
where in Eq. ①. F., is trainable linear projection with dimension of D. and so Eg in Eq. but the difference that Eg acts patches after temporal filtered in Eq. 2 i.c., the enhanced patch

---

representations at the initial stage y. The k in Eq. 2 denotes the pooling kernel size, which is always set to 2 for
locality emphasis as discussed previously. The pooling also leads to more efficient training. And C2 in Eq. (3) is the number of the used temporal filters in Conv3d, which set to 64. Finally, the embedding is given by Eq. ( where Xiemp is a learnable temporal token, a indicates sigmoid function, z = = (1) o (zt) represents the patch sequence embedding of spatial location i timestamp L. The learnable position embedding Epos is also kept, but with different meaning of temporal indicator.
Low-level Enhanced Transformer stage. Then, patch sequence embeddings different spatial locations are independenly fed into multiple Low-level Enhanced Transformer stages (LET-stages) for further temporal modeling. Similar to the 3D convolutions we used before, we propose to enhance the temporal modeling of patch sequence with aids of shallow spatial-temporal convolution at multiple scales. Given one Transformer block (Trans) [17] defined as:
Trans(e) = MLP(LN(s (s)) +'', E' MSA(LN(()) +
(5)
where MSA and LN denote multiheaded self-attention and LayerNorm, respectively. With the input embeddings at stage / - 1: z 1 = izempi ; i :...; ==."), and the enhanced patch representations:
23 (y yi i-ii.. y yi-il we formally define one LETT-ttage as:
28| ( 2, " 2
2. ....; T Trans (xi-1); Trans"(zi_1)
(6)
[y
1, :1 j... = Maxpool (Con FAll;)))
2
(7)
zt = [Ent" Esy 2,1 :..; Eyy T. *1 E, E (X(CC-(P)
(8)
2 LET-stage( = iiempi (1) (z)) (2) (2); (|| (zt((T))
Izcempi ztd 1
J.
where Trans in Eq. (6) indicates cascaded stacking three blocks together. Overall, the proposed LST is formed by a Local Sequence Emebedding stage and three cascaded Low-level Enhanced Transformer stages.
Compared with our designs, one straight thought might be "just leave the work to self-attention", since theoretically patches can progressively find the most relative patches at the same spatial room for temporal modeling. However, this is nearly impracticable considering both short and long span temporal information is important to our task [65]. Also, the self-attention operation has a quadratic complexity with respect to the number of patches. In contrast, by independently modeling the patch sequences with weight-shared LST, we not only reduce the time complexity of the straight throught from O .N²) to O(T N²), but also explicitly avoid semantic modeling of features like facial structure.
Discussion.
3.3 Cross-Patch Inconsistency loss and Cross-Patch Aggregation
After modeling the temporal relation of sequential patches in LST, we have a set of temporal embeddings of all spatial locations {z'li = 1, 2, .... N]. However, how to give a final decision is still nontrivial, since the embeddings at different spatial location would represent different levels of temporal changes. For example, pixel variation in the background region is usually less dramatic than that of the mouth region. In addition, there normally exist non-edited regions (usually the background) deepfakes. Thus, directly adopting an overall binary classification loss on all patches is not the best choice. Instead, we propose to identify the inconsistency by global contrast, because forgery parts should retain heterogencous temporal patterns compared with the real ones. Concretely, we achieve this goal through the Cross-Patch Inconsistency loss and the proposed Cross-Patch Aggregation.
Cross-Patch Inconsistency loss. Given the feature set after LST [z = 1,2, ..., N). we leave the first temporal token at all spatial locations (fziemp E R P| = 1,2, ..., NN) for latter classification, and use the remaining patch embeddings ((z TR: 1,2,..., T; 2.., .N, in this part
(see Fig. lp. We first reduce the temporal dimension of all local regions as: = + Σt th,, Then we calculate the dense cosine similarities of all regions as:
(PP f?)
sim
p == 1,2, =: 1,2,.., N.
(10)
|
5

---

The sequence of real regions will certainly depict "natural" variation, while the sequence of fake regions formed by re-assembling will be different. According to the simplest thought that temporal features of real regions should be similar to the real ones, and vice versa. We propose to impose intra-frame contrastive supervision as:
LCPT = Σ max (/sim=4 sim?e' - u4, 0)²
(11)
P-7
where ji is a tolerance margin which we set it to 0.1, that allows more diverse feature representations in binary classes. And simgt E RR NNN is the ground truth similarity matrix that generated from the modification mask sequence m. E Txx
follows:
1 Σ m! E RHx
mg Interpolate(ma)) E RVNNVN m Flatten(ms) E R
Ma
(12)
sim P. 2. |m - gt
1,2,..,N m1)) 1, p= 1.2, 1, N.
(13)
The modification masks m. are generated by simply subtracting the fake frame from the corre- sponding real one, which should be normalize to range of (0, 1) in advance. Thus, the value range of simgt is consistent with cosine similarity, i.e., (-1. 1). As for the real clip without mask, the simgt N
Cross-Patch Aggregation. For the final classification, we want to determine the final prediction based on the overall consistency. We first insert a class token Xolasa E ER" in to temporal tokens of LST output as [Xelnss; Zlempi hempi ...; zi z. Based on those location-specifie temporal tokens, we then adopt additional three Transformer blocks (similar to Eq. (6) to achieve cross-patch temporal information aggregation. Then the final prediction is given by fully connection layer taking the [class] embedding as input. With such practice, our model is empowered to identify the different correlations between real and fake sequences. For the classification loss, we simply use the binary cross-entropy loss denoted as LBCE
And finally, we can train our LTTD in end-to-end manner with the two losses as:
L LBCE + A LCPI,
(14)
where the parameter À is used to balance the two parts and empirically set to 10-3.
4 Experiments
4.1 Setups
Datasets. Our experiments are conducted based on several popular deepfake datasets including FaceForensics++ (FF++) (55, DeepFake Detection Challenge dataset (DFDC) [15]. CelebDF-V2 VV (CelebDF) (35), FaceShift dataset (FaceSh) [30], and DeeperForensics dataset (DeepFo) 224]. FF++ (HQ) used as train set and the remaining four datasets are used for generalization evaluation. FF++ is one of the most widely used dataset in deepfake detection, which contains 1000 real videos collected from Youtube and 4000 fake videos generated by four different forgery methods including Deepfakes [1. FaceSwap 22) Face2Face [53] and NeuralTextures [52]. To simulate the real- world stream media environment, FF++ also provides three versions with different compression rates, which are denoted by raw (no compression), HQ (constant rate quantization parameter equal to 23), and LQ (the quantization parameter is set to 40), respectively. Based on the 1000 real videos of FF++, FaceSh is a later published dataset containing 1000 fake videos, which are generated by a more sophisticated face swapping technique. DeepFo is a large-scale dataset for real-world deepfake detection. To ensure better quality and diversity, the authors make the source videos in controlled scenario with paid actors. More impressively, a new face swapping pipeline considering temporal consistency is proposed to generate deepfakes with more "natural" low-level temporal features. DFDC is million-scale dataset used in the most famous deepfake challenge B Following previous works 21], more than 3000 videos in the private test set for cross- cross-dataset dataset evaluation in this paper. In addition, CelebDF is one of the most challenging dataset, which is generated using an improved deepfake technique based on videos of celebrities.
Data preprocessing. All the used datasets are published in a full-frame format, thus, most of the deepfake detection methods will crop out the face regions in advance. However, with a motivation

---

to learn the low-level temporal patterns, cropping face regions in advance will lead to artificially introduced jittering due to the independent face detection. Therefore, our method, we crop the face regions using the same bounding box (including all facial regions) after randomly determine the clip range on-the-fly, where the box detected by MTCNN [1].
Implementation details. The spatial input size И x W and patch size P is set to 224 x 224 and 16, respectively. is worth noting that the division into such small patches has considerably suppress the overall semantic features. The embedding dimension D is set to 384. In Local Sequence Embedding stage, we use "convl, conv2 x" of [22] for low-level feature embedding. And we use one Conv3d layer with kernel size of 3 x 3 x 3 in each LST. For temporal dimension T, we empirically set it to 16 and provide more discussion in ablations. We use only the first 128 frames of each video in the experiments, thus the final prediction is averaged from 8 clips. For optimizer, we use Adam 2 with the initial learning rate of 10 10-4 When the performance no longer improve significantly, we gradually decay the learning rate. Four NVIDIA A100 GPUs are used in our experiments.
we use binary classification accuracy (ACC) and Area Under the Receiver Operating Characteristic Curve (AUC) as evaluation metrics.
Evaluation metrics. Following previous works 45 21 311,
4.2 Generalizability evaluation
For practical application, generalizability should be one of the most concerned properties. Never- theless, it is usually the Achilles' heel of most deepfake detectors. Since deepfakes generated by different forgery methods (in different datasets) hold different kinds of forgery cues, and overfitting on semantic visual artifacts of the train set can easily lead to cross cross-dataset evaluation collapse. As we show the performance of models trained on FF++ and tested on four unseen datasets in Table. many methods do not perform satisfactorily. In contrast, our approach outperforms all the recently published novel detectors, and achieves a new state of the of 99.9 AUC% averaged from the four datasets. Note PatchForensics also focuses on local patches, but only narrows the perceptive field by truncating the CNN without considering the relations between patches globally, it shows limited generalizability. In addition, FTCN-TT re- ports comparable results, where they also leverage the self-attention mech- anism. But different from our low-level tem- poral view, they use the semantic features of the whole frame for predic- tion. Thus, when the vi- sual artifacts less dis- tinguishable in CelebDF and DFDC, our method surpasses it by a clear mar- gin. A similar situation in LipForensics, which is proposed to perform deep- fake detection using pretrain priors 41 of high-level semantic understanding. With temporal regularity considered, they also achieve comparable results but show suboptimal performance on CelebDF and DFDC. Moreover, PCL+I2G and Face X-ray also focus on low-level learning. However, without considering temporal properties, they exhibit insufficient robustness and perform poorly in DFDC, where the videos are filmed under very different circumstances and have been perturbed to some extent.
Table Generalizability evaluation. Models are trained on FF++, and test on remaining four datasets. We show the metric of video-level AUC% comparing with the state of the arts.
Method
CelebDF
DFDC
FaceSh
DeepPo
Average
CNN-GRU 46
69.8
68.9
80.8
74.1
73.4
Multi-task 42
75.5
68.1
66.0
77.7
71.9
PatchForensics FWA 34
69.6
65.6
57.8
81.8
68.7
69.5
67.3
65.5
50.2
63.1
Face X-ray
31
79.5
65.5
92.8
86.8
81.2
PCL+12G
90.0
67.5
99.4
85.6
SBI+EB4 48
89.9
74,9
97.4
77.7
85.0
LipForensics 22
82.4
73.5
97.1
97.6
87.7
FTCN-TT [65]
86.9
74.0
98.8
98.8
89.6
LTTD (ours)
89.3
80.4
99.5
98.5
91.9
4.3 Robustness to perturbations
Another vital property for practical application is robustness. For network transmission, videos are always compressed. Also, consider possible attacking against the detectors, we evaluate our approach on different types of perturbed videos.
7

---

space, which is Color Saturation
Contratt
Blad-viia
Prddido
Compresdion
Ausrae
Eigbe
030
Figure 2: Robustness evaluation. Models are trained on the the clean train set of FF++ and tested on perturbed lest sets, respectively. Averuge indicates the results averaged from types perturbations.
Following previous works [21 24]]. we use the script with FF++ and generate seven types of perturbations at five levels. As show the diagram in Fig. (others refer to (2]]), the seven perturb methods include color saturation change, color contrast change, block-wise noise, gaussian noise, gaussian blur, pixelation, and video compression. Examples be found here
Comparing with other methods, our approach shows excellent robustness against most the pertur- bations. As for the type of gaussian noise at relatively higher level, the underlying low-level patterns are severely disturbed that our method can not work properly. We also present the comparison with two state of the arts in Table. 2 where the results are averaged from all five levels. In general, Face X-ray detects low-level boundary artifacts, and LipForensics focuses on high-level semantic features. Thus, the former performs better with block-wise noise type and the latter shows better results with others. In contrast to Face X-ray, we detect low-level patterns in better resistant perturbations, and thus achieves better performance.
Table 2: Robustness evaluation. Average performance evaluated on perturbed videos at five levels. Clean: origin videos, CS: color saturation, CC: color contrast, BW: block-wise noise, GNC: gaussian noise, GB: gaussian blur, PX: pixelation, VC: video compression, Avg: averaged performance on distored videos, Drop: performance drop comparing to Clean. The gray numbers do not reflect robustness, and metrics of video-level AUC% is reported.
Method
Clean
CS
CC
BW
GNC
GB
PX
VC
Avg/Drop
Face X-ray [3
99.8
97.6
88.5
99.1
49.8
63.8
88.6
55.2
77.5/-22.3
LipForensics 21
99.9
99.9
99.6
87.4
73.8
96.1
95.6
95.6
92.6/-7.3
LTTD (ours)
99.4
98.9
96.4
96.1
82.6
97.5
98.6
95.0
95.0/-4.3
4.4 Ablations
Module effects. We first compare our method with related baselines and several alternative designs. Xception the commonly used backbone in deepfake detection; 2) ViT is the most famous vision transformer backbone, where we use the "small" version with embedding dimension of 384; 3) ViVIT is a recently published work developed in the self-attention style with spatio-temporal modeling ability for action recognition. We use the best model, "Model 1", evaluated in their paper with comparable backbone, "VIT small", with our method; 4) LTTD w/o LSST indicates the model that we replace the proposed LST with commonly used Patch Embedding and Transformer blocks [17] 5) LTTD w/o CPI is our LTTD framework trained without using LCPT (Eq, [3)): 6) LTTD w/o CPA represents the model that we replace the CPA module with a simple fully connected classification layer after average pooling the temporal embeddings from all spacial locations. From Table. 3 all the models achieve nearly perfect results on FF++. While in the cross- cross-dttaeet setting, our model with elaborately designed modules exhibits much stronger generalizabilty, and the three components consistently boost the
best results.
Visualization. To intuitively demonstrate the reason that our approach more generalizable, we compare the feature representations before the final classification in Fig. 3 Models are trained on FF++ and tested on four subsets of FF++ (Deepfakes, Face2Face, FaceSwap. and NeuralTextures) respectively, Due to the abundant inductive bias of the convolution, Xception clearly split the four

---

Table 3: Module effect. Models are trained on FF++. Gray numbers reflect in-dataset effectiveness, and others represent cross-dataset generalization. Cross-Avg: average from unseen datasets.
F+++
FaceSh
DFDC
DeepFo
Cross-Avg
Method
ACC%
AUC%
ACC%
AUC%
ACC%
AUC%
ACC%
AUC%
ACC%
AUC%
Xception
96.08
99.38
72.47
78.60
60.47
67.36
69.21
83.28
67.38
76.41
VIT 17
95.00
97.92
62.86
65.56
64.81
72.89
71.85
83.24
66.51
73.90
VIVIT R
94.71
97.92
63.21
77.40
67.52
74.16
60.12
82.86
63.62
78.14
LITD w/o LST
95.57
98.57
90.71
97.44
60.40
70.06
87.68
97.94
79.60
88.48
LTTD w/o CPI
97.29
99.23
95.00
98.86
67.95
77.03
90.62
97.68
84,52
91.19
LTTD w/o CPA
96.14
99.00
91.79
99.35
65.29
70.64
87.39
96.62
81.49
88.87
LTTD
97.72
99.52
96.55
99.51
71.34
80.39
92.53
98.50
86.81
92.80
kinds of deepfakes into four clusters, even only binary labels are used in training. This shows that strong CNN tends to overfit on method-specific artifacts generated by different deepfake methods, although plausible results are achieved in in-dataset testing, it is harmful to generalize to unseen deepfakes. In addition, similar phenomena can found in the results of ViT. However, the divison of the four clusters is not as clear in Xception due to the lesser inductive biases introduced in VIT. In contrast, the features of our approach show a completely different outline, where the real videos are also clearly separated, but the remaining four types of deepfakes We attribute this phenomenon to our low-level temporal learning, which can distinguish deepfakes from real depending more fundamental low-level temporal inconsistencies, thus achieving the best generalizability.
compacted in unified manifold.
Xception
ViT
LTTD (ours)
beepttkes Fack
Dieplaes
Neural lodtatt
Moura redured
Peoothars
Feature visualization. In the t-SNE [54] visualization, every dot represents compacted features of the corresponding test video, and different color indicates different class.
Figure 3:
Sequential input. As we mentioned previously, empirically set the clip length to 16. On the one hand, too short a clip does not ensure enough temporal information, e. g., con- secutive frames are almost identical to each other. On the other hand, too long clip is not necessarily better: 1) A longer clip costs more memory; accordingly, a smaller batch size also slow down training convergence worse. 2) We found some of the videos in the test datasets contained scene switching. Considering the FPS of 24, a video clip in greatly avoiding the scene-switching problem. Here, we further analyze how clip sampling will affect the low-level temporal learning of the pro- posed LTTD. Two hyper-parameters are inves- tigated: clip length and frame sampling space. From Table 4 the first 4 rows show that clip length has little effect on the results. We believe this is closely related to the idea of low-level temporal learning, which does not require the video clip to last long enough (e. 3 seconds) but only adequate frames to extract the low-level temporal patterns. Sparse sampling is another practice to aggregate more content in a single clip. When we expand the frame interval, the performance degrades considerably. This phenomenon suggests that sparse sampling is detrimental to the learning of low-level temporal patterns even when more motion content is included. These
experiments will not last second, thus
Table 4: Sequential input ablation. CL: clip
length, FSS: frame sampling space.
CL.
FSS
FF++
DFDC
DeepFo
Average
8
99.26
77.25
96.82
91.11
16
99.52
80.39
98.50
92.80
32
99.47
80.92
98.41
92,93
64
99.38
79.23
98.49
92.37
16
99.15
76.17
97.32
90.88
16
98.51
73.02
91.33
87.62

---

findings demonstrate that LTTD is distinctly different from related temporal-basee models, since the low-level temporal learning is specially designed for deepfake detection.
Forgery localization. Our model enjoys a local-to-global learning protocol, where differences between real and fake regions naturally explored. Here we investigate this property by visualizing the CAM [47] responses of our model. Considering the input size of 224 x 224 and the patch size of 16 16, the spatial space is divided into a 14 x 14 grid. We use the CAM responses of the Xelass token to draw localization map by bilinear interpolation.
Deepfakes
Face2Facc
FaceSwap
NeureeTexturee
Figure 4: Forgery localization.
As shown in the Fig. 4 our model is able identify the local inconsistencies. addition, we can learn some different characteristics of each forgery type. Most distinctively, regions of mouth and eyes in FaceSwap are not modified, thus showing patterns that distinguish from other f'acial parts. Moreover, regions relating to the mouth are reenacted in NeuralTexturrs, just like the localization results shown in the figure.
Despite the good intuitive demonstrations, it remains future works to determine whether the localiza- tion results credible.
Conclusion and discussion
Conclusion. In this paper, we propose reliable framework to address the practical problems of deepfake detection, which emphasizes the low-level temporal patterns of sequential patches in the restricted spatial region with a whole-range temporal receptive field using Transformer blocks. In addition, we make the final classification generalizability and robustness are achieved to better support deepfake detection in real-world scenes. Moreover, qualitative results further verify that low-level temporal information can lead to stronger generalizability, which could also be a guideline for developing better approaches in the future.
more general global-contrastive way. Thus better
Linitation. Considering the continuous advances in deepfake creation and adversarial training, the performance our approach when encountering low-level and temporal adversarially enhanced deepfakes in the future is yet unclear. Moreover, although our method shows favorable performance compared to recent works, it still requires intensive labor in order to handle real-world scenarios. This is commonly shared limitation that do not know if the detectors are calibrated well for real-world ment In addition to identifying deepfakes, how we can ensure the predictions are credible remains an open problem, hindering the application of all deepfake detectors.
References
[1] Deepfakes github.
2022. Accessed: 2022-02-28.
[2] Deepfakes github. httppp///tthhccccm////aaaaaaaaaa 2022. Accessed: 2022-02-28.
[3] Drde ottccallaae..pss///..g.goooooooo bttps//www..ggggg.ooooooo//ooeed www.ooonnccaaanggggeeo 2022. Accessed:
2022-04-04.
10

---

[4] Darius Afchar. Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Mesonet: compact facial video forgery detection network. In 2018 IEEE International Workshop on Information Forensicy and Security (WIFS), pages 1-7. IEBE. 2018.
[5] Shruti Agarwal and Hany Farid. Detecting deep-fake videos from aural and oral dynamics. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 981-989, 2021.
[6] Shruti Agarwal, Hany Farid, Tarek El-Gaaly, and Ser-Nam Lim. Detecting deep-fake videos from appearance and behavior. In 2020 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1-6, IEEE, 2020.
[7] Irene Amerini, Leonardo Galteri, Roberto Caldelli, and Alberto Del Bimbo. Deepfake video detection through optical flow based can. In Proceedings of the IEEE/CVF international conference on computer
vision workshops, pages 0-0, 2019.
[8]
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEF/CVF International Conference on Computer Vision,
pages 1836-6846, 2021.
[9] Lucy Chai, David Bau. Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? understanding properties that generalize. In European Conference on Computer Vision, pages 103-120. Springer, 2020. [10] Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, Jilin Li, and Rongrong Ji. Local relation learning for face forgery detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35,
pages 1081-1088, 2021.
[11] François Chollet. Xception:
Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1251-1258, 2017.
[12] Yuren Cong. Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. Spatial- temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pages 16372-16382, 2021.
[13] Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain. On the detection of digital face manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 5781-5790. 2020.
[14] Sowmen Das. Selim Seferbekov, Arup Datta, Md Islam, Md Amin, et al. Towards solving the deepfake problem: An analysis on improving deepfake detection using dynamic face augmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3776-3785, 2021.
[15] Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang. and Cristian Canton Ferrer. The deepfake detection challenge (dfde) dataset. arXiv preprint arXiv:2006.073::
2020. [16] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Ting Zhang, Weiming Zhang, Nenghai Yu, Dong Chen, Fang Wen, and Baining Guo. Protecting celchrities with identity consistency transformer. arXiv preprint arXiv:2203.0133/8 2022.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / //929, 2020.
[18] Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, and Graham W Taylor. Sstvos: Sparse spatiotemporal transformers for video object segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 5912-59211 12211, 2021.
[19] Qiqi Gu, Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, and Ran Yi. Exploiting ine-grained face forgery clues via progressive enhancement learning. arXiv preprint ar.
2021.
[20] Jiazhi Guan, Hang Zhou, Mingming Gong, Youjian Zhao, Errui Ding, and Jingdong Wang. Detecting deepfake by creating spatio-temporal
p [21] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don't lie: A
disruption. arXiv preprint XXiv:2207.10400 2022.
generalisable and robust approach to face forgery detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5039-5049, 2021.
[22] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the IEEE International Conference on Computer Vision
Workshops, pages 3154-3166 2017.
[23] Xinya Ji. Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio- driven emotional video portraits. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2021.
[24] Liming Jiang, Ren Li, Wayne Wu, Chen Qian, and Chen Change Loy. Deeperforensies-..0 1.:: A large-scale datasct for real-world face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2889-2898, 2020.
[25] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[26] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10191 2020.
[27] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
1411 66800, 2014.
[28] Mohammad Rami Koujan, Michail Christos Doukas, Anastasios Roussos, and Stefanos Zafeiioon.
Head2head: Video based neural head synthesis. arXiv preprint ArXiv:2005. 2020.
11

---

[29] Jiaming Li, Hongtao Xie, Jiahong Li. Zhongyuan Wang, and Yongdong Zhang. Frequency-aware discrimi- native feature learning supervised by single-center loss for face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6458-6467, 2021.
[30] Lingzhi Li, Jianmin Bao, Hao Yang. Dong Chen, and Fang Wen. Advancing high fidelity identity swapping for forgery detection. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5074-5083,
[31] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang. Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more general face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages. 5001-5010, 2020.
[32] Xiangyu Li, Yonghong Hou, Pichao Wang. Zhimin Gao. Mingliang Xu, and Wanqing Li. Trear: Transformer-based rgb-il egocentric action recognition. IEEE Transactions Cognitive and Devel-
opmental Systems, 2021.
[33] Yuezun I., Ming- Chang, and Siwei Lyu. In ictu oculi: Exposing ai created fake videos by detecting eye blinking. In 2018 IEFE International Workshop on Information Forensics and Security (WIFS), pages
1-7. IEFF, 2018.
[34] Yuezun Li and Siwci Lyu. Exposing deepfake vidcos by detecting face warping artifacts. arXiv preprint
arXiv:18/1 (0w,181100066, 2018.
[35] Yuezun Li, Xin Yang. Pu Sun, Honggang Qi, and Siwei Lyu, Celeb-df: A large-scale challenging dataset for deepfake forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3207-3216, 2020.
[36] Honggu Liu, Xiaodan Li, Wenho Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai
Yu. Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In Proceedings
2021. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 772-781, 2021.
[37] Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-aware implicit
neural oudio-driven video portrait generation. ECCV, 2022.
[38] Ze Liu, Jia Ning. Yue Can, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer.
Xrviv preprint arXiv:2 /3230, 2021.
[39] Zhengzhe Liu, Xiaojuan Qi, and Philip HS Torr. Global texture enhancement for fake face detection in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3060-8069, 2020.
[40] Yuchen Luo. Yong Zhang, Junchi Yan, and Wei Liu. Generalizing face forgery detection with high- frequency features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 13317-16326, 2021.
[41] Pingehuan Ma, Brais Martinez, Stavros Petridis, and Maja Pantic. Towards practical lipreading with distilled and efficient models. In ICASSP 2021-2021 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 7608-7612. IEEE, 2021.
[42] Huy H Nguyen, Fuming Fang. Junichi Yamagishi, and Isao Echizen. Multi-task Icarning for detecting and segmenting manipulated facial images and videos. In 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS), pages 1-8. IEEE, 2019.
[43] Chiara Plizzari, Marco Cannici, and Matteo Matteucei. Spatial temporal transformer network for skeleton- based action recognition. In International Conference on Pattern Recognition, pages 694-701- Springer, 2021.
[44] Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in frequeney: Face forgery detection by mining frequency- aware clucs. In European Conference on Computer Vision, pages 86-103. Springer, 2020.
[45] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensicst+: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 1 11. 2019.
[46] Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, and Prem Natarajan. Recurrent convolutional strategies for face manipulation detection in videos. Interfaces (GUT), 3(1):00-87.
2019.
[47] Ramprasaath R Selvaraju, Michael Cogswell, Abhishck Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017_
[48] Kaede Shiohara and Toshihiko Yamasaki. Detecting deepfakes with self-blended images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 18720-18729 2022.
[49] Ke Sun, Taiping Yao, Shen Chen, Shouhong Ding, Rongrong Ji, et al. Dual contrastive learning for general face forgery detection. arXiv preprini arXiv:2112. 13522, 2021.
[50] Zekun Sun. Yujie Han. Zeyu Hua, Na Ruan, and Woijia Jia. Improving the efficiency and robustness of deepfakes detection through precise geometric features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3609-36 3618, 2021.
[51] Shahroz Tariq. Sangyup Lee, and Simon S Woo. A convolutional 1stm based residual network for deepfake video detection. arXiv preprint arXiv:2009. 07449007000000 2020.
[52]
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG), -81 2019.
[53] Justus Thies, Michael Zollhofer, Mare Stamminger, Christian Theobalt, and Matthias NieBner. Face2face: Real-tine face capture and reenactment of rgb videos. In Proceedings of the IEEE conference on computer
12

---

vision and puttern recognition, pages 2387-2395, 2016.
[54]
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 9(11), 2008.
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008. 2017.
[56] Chengrui Wang and Weihong Deng. Representative forgery mining for fake face detection. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14923-14932, 2021. [57] Junke Wang, Zuxuan Wu, Jingjing Chen, and Yu- Yu-Gang Jiang. M2tr: Multi-modal mulli-scabe transformers for deepfake detection, arXiv preprint arXiv:2104 09770, 2021.
[58] Xin Yang. Yuezun Li, and Siwei yu. Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2261-8265. IEEE, 2019.
[59] Junbo Yin, Jianbing Shen, Chenye Guan, Dingfu Zhou, and Ruigang Yang. Lidar-based online 3d video object detection with graph-based message passing and spatiotemporal transformer attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11495-
11504, 2020.
[60]
Daichi Zhang, Chenyu Li, Fanzhao Lin, Dan Zeng, and Shiming Ge. Detecting deepfake videos with temporal dropout 3denn. LICAL, 2021.
[61]
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23 23(10):1499-1500 2016. Yanyi Zhang. Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13577- -1587,, 2021.
[62]
[63]
Hanqing Zhao, Wenho Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi- attentional deepfake detection. arXiv preprint Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, and Wei Xia. Learning self-consisteney for deepfake detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15023-15033, 2021.
Skdi:21033024000 2021.
[64]
[65] Yinglin Zheng. Jianmin Bao, Dong Chen, Ming Zeng, and Fang Wen. Exploring temporal coherence for more general video face forgery detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15044-15054 2021.
[66] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiangang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In Proceedings of the IEEE Conference on Computer Vision and Patiern Recognition (CVPR). 2021.
[67] Peng Zhou, Xintong Han, Vlad Morariu, and Larry S Davis. Two-stream neural networks for tampered face detection. 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pages 1831-1839, IEEE, 2017.
[68] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li. Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.0413
2020.
[69] Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu- Yu-Gang Jiang. Wilddeepfake: A challenging real-world dataset for decpfake detection. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2382-2390, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type
of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] The FF++ dataset and FaceSh dataset are released under the FaceForensics Terms Use The CelebDF dataset is released under the Terms to Use Celeb-DF] The use of DeepFo dataset is bounded by the Terms
of Use: DeeperForensics-I... Dataset
(c) Did you include any new assets either in the supplemental material or as URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] The datasets we use are face-related open- source datasets that contain personally identifiable information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "zAVQlCO2DrC", "reviewer": "Reviewer_DdUQ", "summary": "The paper proposes a method for DeepFake Detection. The method focuses on a particular way of modeling the temporal aspect of the detection: instead of learning first spatio-temporal downsampled embedding with 3D convolution and then aggregating the embedding with transformers [62], the proposed method immediately breaks the image into patches and adds the temporal dimension. Unlike prior work, temporal modeling is done directly over patches with the motivation of identifying low-level temporal inconsistencies and standing on the shoulders of insights from [35]. This is achieved by moving the transformer architecture immediately upstream in the pipeline contrary to previous methods in which usually temporal aggregation is performed downstream. Though the method models patches in this way still use part of 3D convolutions applied to patches. The method is supervised with standard cross-entropy loss and another loss that enforces that real patches should be similar to each other in the learned embedding space.\nExperimental results are provided on recent benchmarks for Deepfake detection along with ablation studies. The robustness of the method to common perturbation is also tested.", "questions": "- Tab. 3 maybe improve if an average is presented.\n- No detail of training and testing time.\n- What are the parameters of the 3D Convolution used? Kernel size over spatial dimension and temporal.\n\n- There are several parts that are not clear:\n   > L198: the simplest thought is that the real region should be similar to the real one to some extent\n\nFor me, this is not enough to justify the loss and is not a very clear sentence. The sentence says that real regions should be similar to real ones but this is not true in the image space; maybe this is the requirement that the loss is imposing in the learned feature space though is not clear. Furthermore, it is not clear what the loss why the loss matches the cosine similarity between all patches with the ground-truth similarity matrix. Which min and max values take the ground-truth similarity matrix?\nThe process of generating the ground-truth similarly matrix is displayed in Eq. (12,13) but seems over complicated with the notation ($m_\\alpha,m_o,m_\\beta,$). Why does interpolation have to be $\\pi$? Per my understanding, the min and max values for sim_gt are $\\{-1, 1\\}$. Is this correct? Anyway, the motivation for the loss is that real features should be basically similar to each other but it is not clear how this is attained given the current loss formulation since there is no selection of the real embedding in the loss unless this is achieved with the $sim_git$ though how? \n\n L:208 $\\mathbf{x}_{class}$ is mentioned without explaining what is it.\n\nL:242, In the same way, I mentioned in the weakness the part on data processing is hard to understand. Especially this sentence:\n>  Therefore, in our method, we crop the face regions using the same bounding box after randomly determining the clip range in-the-fly,\n\n\n\n", "limitations": "The limitations presented are somewhat related to the generalization of the model. It says _now it is good (at least a bit superior to the state-of-the-art) but we will be unsure in the future._ \n\nI think an overall direction for the future that would be nice for Deepfake Detection is working toward ensuring that the prediction of the Detection is really calibrated well. To the best of my knowledge, there is no detection system yet deployed that can be left free of running on video on social networks and flagging videos. I am not sure if there are implementations with humans in the loop that check highly confident video marked by the detector. Anyway, the point is that, given that an erroneous highly confident prediction as fake for an authentic video could be even more problematic of a deepfake itself, all these deepfake detectors have the limitation that we do not know if they are _calibrated well_.  Perhaps adding a general discussion on this may improve the paper: i.e. as of now, the method has good generalization accuracy, but if we run it in the wild on videos for which we do not have labels, how much can we trust the prediction?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strenghts\n- The method is overall motivated when proposing something and takes a direction that is different than the state-of-the-art [62] of directly modeling patch sequences bringing transformers upstream in the pipeline.\n- Overall the innovative claims in the intro are well balanced with the paper's experimental evidence, especially in the robustness part of perturbations.\n- I was able to grasp the overall idea of the method quickly though often parts feel overcomplicated in the way they are written.\n- The experimental validation is promising and uses recent benchmarks for deepfake detection. Ablation studies are provided to support design choices. In some cases, the ablation improvement is clear in other cases less clear. In Tab. 3 an overall summary using average with standard deviation across unseen manipulation is needed.\n\n\n### Weaknesses\n\n- The average improvement over the competing method [62] `(89.6% vs 91.8%) `is appreciated but looks a bit marginal without comparing the timings of the approach. Also, it is unclear if the improvement `+2.2` video-level AUC is statistically significant. What happens if you train multiple **LTTD** for different initialization and multiple **FTCN-TT** and report the average and standard deviation? Is the improvement statistically significant, given the standard deviation of the two approaches? (Assuming authors have code to run FTCN-TT). I ask this because, it is clear that from the distribution of performance the gap is obtained thanks to the DFDC dataset, all the others are basically the same. What makes LTTD works well on DFDC? This small improvement questions the overcomplexity of the method and I have the idea that the paper will be hard to reproduce and re-implement.\n-  Though the method is well-motivated, it feels a bit the method is riding the success of transformers and applied to the idea of modeling temporal discrepancy with patches for DeepFake Detection, changing the idea of [62]. Also, though the method seeks a changing paradigm does still apply 3D convolution as in [62]. This part where the method applies 3D convolution on patches is less clear. Convolution works on images because of the assumption that patches can be highly correlated but here the method breaks the image by splitting into patches, so more explanation on this 3D convolution is needed for the way it handles the spatial component over patches.\n- The idea to break the image in patches does not account for rigid and non-rigid deformation that the face may even in 3D, convolution suffers from this too but is more resistant to jittering of translation by definition. The paper says something on this matter at `L149` but is not clear.\n- `L242` gives details about the face cropping which is related to alignment, motion compensation, and breaking the image into patches. It is not clear overall how the face cropping of the system works. The method does not do frame-by-frame face detection and cropping but uses a single crop. The text says using `the same bounding box` but same respect to which frame? It is not clear also what is randomly determined. If a single crop is used and the face move slightly this is even worse than the random jittering that you get by frame-by-frame processing.\n- More on this: In section 3.2 it is not clear if the Conv3D still slides over the patches or if the spatial kernel size of the convolution is big as the patch itself. This part spends a lot of space with equations (1-4) with tons of super and subscripts but it does not make the reading easier.\n- It is ok to be formal with the notation yet very often the article abuses notation and this makes the article not easily readable.\nThere are multiple of these remarks e.g. usage of subscript that is not necessary $ \\mathbf{x}_{p}^{i}$. Why $p$ is used in L.131, L144 I believe the patch is indexed by $i$. Same remarks for Eq. (5-8).\n- Given that the title includes _Delving into Sequential Patches_, the method misses an instructive study of what happens when the temporal dimension $T$ varies from different values than 16; also, given a fixed $T$, the paper misses a study to understand whatever it is better to have a dense sampling of the frames (e.g. 1:1 wrt to the original video) and a small window or maybe more sparse sampling of the frames (1:20) but to aggregate more content of the video.\n- Fig. 1 is overall nice and I appreciated but it is pretty dense and not easy to digest immediately.\n- I found some typos and sometimes the text use adjective a bit off: e.g. `L17 ` `enormous fake videos`? This means that a video is huge, big in size. `L81` data aRgumentation. `L199` vise versa (vice versa). `L308` a unified manifolds. \n\n### Justification of the rating\nMy opinionf the paper is that it is a good system paper for deepfake detection with marginal improvement over the state-of-the-art, assuming these are statistically significant. The paper has some strength in showing more robustness than other methods. The overall claims are decently supported. Besides that, the paper needs to be improved by simplifying the notation by making it more clear for the community since as of now it will be hard to reproduce it. Overall it is a good paper but I believe that if the paper is not accepted, the deepfake community is not going to lose innovative ideas. According to this, I sugged a borderline reject score before rebuttal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ul9Sk2FIvVa", "reviewer": "Reviewer_Yygr", "summary": "The proposed method combines 3D conv and vision transformer to extract forgery artifacts in Deepfakes videos, the experiments prove that the proposed method achieves outstanding generalization performance on multiple Deepfakes datasets. \n\n", "questions": "1. In the discussion part for Section 3.2, the authors claim that the proposed structure which combines 3D conv feature and self-attention feature can explicitly avoid semantic modeling of features like facial structure and always focus on low-level temporal learning. But no explanations are given to prove this statement.\n2. Comparing Eq.4 and Eq.8, the formats of these two equations are not unified. In addition, the meaning of the symbols used in the equations is not clearly given, making the equations hard to understand. Also, it is strange to use t (lower letter) to represent the features after the pooling operation while the upper letter T represents the time sequence. Similar problems occur in all the equation parts.\n3. In equation 12, in my understanding, the m should equal Flatten(mβ)\n4. The authors mention that the temporal dimension T is set to 16. However, no selection details are given. Suppose we have a 300 frames Deepfakes video with the last 150 frames tampered. The frames selection strategy would be crucial in detecting this Deepfakes video. In addition, no discussions for frame number selection are given. The authors should present more analysis on how frame number influences the proposed structure.\n5. The following method is not compared in generalizability evaluation:\nKaede Shiohara, Toshihiko Yamasaki: Detecting Deepfakes with Self-Blended Images. CVPR 2022. ", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe overall performance of the proposed method is outstanding.\nWeaknesses:\nThe writing of the manuscript should be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jo2urd0juA_", "reviewer": "Reviewer_nqV7", "summary": "The authors propose a framework to improve the generalization and robustness of deepfake detection, which relies on local low-level and temporal information, and transformer-based model. In particular, the Local Sequence Transformer (LST) is used to identify low-level temporal inconsistency and the Cross-Patch Inconsistency loss (CPI) is used to model spatial inconsistency. The Global Contrastive Classification (GCC) is used for final classification, which inserts temporal tokens and adopts additional three Transformer blocks.\nQuantitative experiments show better results on four datasets in generalization evaluation benchmark and on seven perturbations in robustness evaluation benchmark, compared with some recent works. \n", "questions": "Confusion:\n1. In Sec 3.3, it seems that the supervision information has not only binary labels but also masks, so the prediction may be determined by other simpler structures like fine-grained classification layers, instead of the proposed complex “temporal tokens + three Transformer blocks” structure.\n2. Based on Table 3, the result line from the ‘LTTD w/o CPI’ shows that CPI contributes far less than the LST or GCC module in DFDC and DeepFo. So is the CPI module significance?\n\nLimitation:\n1. For the authors’ motivation, using low-level features instead of semantic features benefits a lot in Deepfake Detection. It seems that the middle-level features, and the suitable mixture of low-& middle-level features may also contribute to Deepfake Detection. It is a pity that the paper does not include this exploration.\n2. For Sec 4.4, the robustness evaluation seems not strong enough, the authors could refer to the benchmarks in Improving [45] and ForgeryNet [CVPR 2021, Forgerynet: A versatile benchmark for comprehensive forgery analysis]，whose robustness evaluation contains more types of perturbations and mix perturbation test sets.\n", "limitations": "The authors have done it properly. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written, easy to follow and provides adequate experimental results.\n2. The proposed framework successfully integrates temporal information and low-level features, which are separate from improved vision transformer and shallow CNN, and could be of some contributions to the community.\n3. There are some novel ideas in the LST module: the first is the input type: ‘local patches with same spatial position’, and the second is the low-level temporal enhancement module, corresponding with the research that the low-level artifact is more suitable than the semantic artifact for deepfake detection. What’s more, the CPI module is well designed, and the motivation of the GCC module shows good understanding of deepfake datasets.\n4. The evaluation experiments of generalization and robustness are well done.\n\nWeakness：\n1. In Sec 3.2, the authors use ‘shallow 3D convolution’ in the LST module for ‘align’ as the first reason (Ignoring the second reason temporally). In my opinion, it is similar to face alignment, so if face-alignment operation plays the same role，why not just do it in the pre-processing stage.\n2. In Sec 3.3, sim_gt is calculated by interpolation operation, how about max or average pooling operation? The authors may provide ablation study about how to calculate it.\n3. In Sec 4, the experiment part is lacking intra-evaluation, for example, both training and testing on FF++.\n4. In Sec 4.1, how many frames or clips are sampled from videos in each deepfake dataset. \n5. In Sec 4.3, the result analysis about robustness evaluation seems not enough.\n6. In Sec 4.4, the ablation study is only done for generalization evaluation, that for robustness evaluation is also required.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KgHskORqAw8", "reviewer": "Reviewer_zPSW", "summary": "Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.\n\nTo achieve generalizability across deepfake methods and robustness towards image post-processings, this paper proposes the Local- & Temporal-aware Transformer-based Deepfake Detection (LTTD) framework. The experiments show that the proposed approach achieves the state-of-the-art generalizability and robustness.", "questions": "See above.", "limitations": "This work aims to detect deepfake videos, which is important to prevent the spread of fake information.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper attempts to ensure both generalizability and robustness in deepfake detection, which are two important problems in deepfake detection. The proposed method focuses on low-level temporal learning and prevents overfitting to global semantic cues.\n\n+ The experiments are comprehensive.\n\n+ The paper is clear to understand and the writing quality is relatively good.\n\n\nWeaknesses:\n- The paper claims that the method crops the face regions using the same bounding box randomly determine the clip range in-the-fly (L246), which may affect the practicality. It may be suitable for the videos in datasets because many of these videos often have a single scene. However, some videos in datasets and most videos in wild often have many scenes. If the clip contains scene switching, it may cause face misalignment and affect the detection results.\n\n- Some details of selecting video clips are missing. How many clips extracted from one video? In FaceForensics++, one video may have less than 300 frames or more than 1000 frames. For short videos and long videos, if the number of clips different? The paper report video-level AUC Why the temporal dimension is 16? It is necessary to have an ablation study to see the effect of different choices of this parameter.\n\n- The method splits the input image into local patches and use 3D convolutions. It seems that the framework needs complex computing. What about the efficiency of inference?\n\n- It is not clear that how to compute the ground truth similarity matrix and how to generate the mask sequence (L201-205), especially the fake clip.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JUqIE2pcn8Q", "reviewer": "Reviewer_vJQN", "summary": "This paper proposes a Local- and Temporal-aware Transformer-based Deepfake Detection (LTTD) framework to capture temporal cues from local sequences. The authors design a Local Sequence Transformer (LST) models the temporal consistency of restricted spatial regions to learn local-to-global features. The proposed LTTD framework achieves outstanding generalizability and robustness.", "questions": "1. Novelty. \nMany studies prove that the low-level patterns and temporal inconsistency are effective clues for Deepfake detection. What is the motivation of this search? Does the LTTD framework address specific problems that previous methods have neglected? The authors are supposed to underline the motivation and advantage of the proposed method.\n\n2. Analyses. \n1) The novel Low-level Enhancement is designed by shallow 3D convolutions. It can be regarded as an 3D extension of textural enhancement [60]. The authors should validate whether the 3D convolutions have the major contribution of improving feature learning. Can it be implemented with 2D convolutions to have comparable performance gain and less computational overhead?\n2) The LTTD focuses on the low-level temporal patterns of restricted spatial region. What is the restricted spatial region? Please introduce the definition of restricted region. Furthermore, is it necessary to restrict specific regions? The authors should analyze the difference between restricted region and unrestricted region.\n3) As shown in Table 3, the ViT [18] has inferior performance to the CNN baseline Xception. What is the reason for this phenomenon? How do the ViT-based methods FTCN [62] and LTTD improve the performance of ViTs on Deepfake detection? What is the key to this significant improvement?\n4) Complexity. The comparisons on model complexity and GFLOPs between LipForensics, FTCN, and LTTD are desirable.\n5) Localization. Can the LTTD localize the forged regions? It will be explicit to visualize the localization of face forgery.\n", "limitations": "Adequately claimed. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Clear description of method.\n2. Remarkable generalization towards unseen deepfakes and strong robustness against post-processing operations. \n\nWeakness:\n1. Novelty.\n2. Analyses.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept