Title: Multitask Learning for Face Forgery Detection: A Joint Embedding Approach
Venue: NeurIPS 2023
Accepted: no
Keywords: face forgery detection

multitask learning

joint embedding

vision-langauge correspondence
Abstract: Multitask learning for face forgery detection has experienced impressive successes in recent years. Nevertheless, the semantic relationships among different forgery detection tasks are generally overlooked in previous methods, which weakens knowledge transfer across tasks. Moreover, previously adopted multitask learning schemes require human intervention on allocating model  capacity to each task and computing the loss weighting, which is bound to be suboptimal.  In this paper, we aim at automated multitask learning for face forgery detection from a joint embedding perspective. We first define a set of coarse-to-fine face forgery detection tasks based on face attributes at different semantic levels. We describe the ground-truth for each task via a textural template, and  train two encoders to jointly embed visual face images and textual descriptions in the shared feature space. In such a manner,  the semantic closeness between two tasks is manifested as the distance  in the learned feature space. Moreover, the capacity of the image encoder can be automatically allocated to each task through end-to-end optimization. Through joint embedding, face forgery detection can  be performed by maximizing the feature similarity between the test face image and candidate textual descriptions. Extensive experiments show that the proposed method improves face forgery detection in terms of generalization to novel face manipulations. In addition, our multitask learning method renders some degree of model interpretation by providing human-understandable explanations.

Abstract

Multitask learning for face forgery detection has experienced impressive successes in recent years. Nevertheless, the semantic relationships among different forgery detection tasks are generally overlooked in previous methods, which weakens knowledge transfer across tasks. Moreover, previously adopted multitask learning schemes require human intervention on allocating model capacity to each task and computing the loss weighting, which is bound to be suboptimal. In this paper, we aim at automated multitask learning for face forgery detection from a joint embedding perspective. We first define a set of coarse-to-fine face forgery detection tasks based on face attributes at different semantic levels. We describe the groundtruth for each task via a textural template, and train two encoders to jointly embed visual face images and textual descriptions in the shared feature space. In such a manner, the semantic closeness between two tasks is manifested as the distance in the learned feature space. Moreover, the capacity of the image encoder can be automatically allocated to each task through end-to-end optimization. Through joint embedding, face forgery detection can be performed by maximizing the feature similarity between the test face image and candidate textual descriptions. Extensive experiments show that the proposed method improves face forgery detection in terms of generalization to novel face manipulations. In addition, our multitask learning method renders some degree of model interpretation by providing humanunderstandable explanations.

1 Introduction

The emergence of deep generative models [1, 34, 67, 71] has significantly simplified and automated 22 the process of generating realistic counterfeit face images, popularly known as DeepFake. The 23 prevalence of falsified face images can erode the reliability and credibility of digital visual information. 24 Additionally, the exploitation and manipulation of such technologies pose a threat to individual rights 25 and national security. 26

Traditional DeepFake detectors were largely influenced by classic photo forensics [21] to expose 27

forgery traces by examining statistical anomalies [51, 58], visual artifacts [32, 46, 50, 51, 59], 28

and physical and geometric inconsistencies [15, 33, 35, 56]. With the rapid development of deep 29

learning, there has recently been a growing consensus on exploiting multitask learning for face 30

forgery detection [8, 10, 19, 41, 55, 80, 81]. The underlying assumption is that the primary task 31

( i.e. , global face forgery classification) is likely to benefit from other highly relevant auxiliary tasks 32

through knowledge transfer. Representative auxiliary tasks include manipulation type (and degree) 33

classification [10], manipulation parameter estimation [75], blending boundary detection [41], spatial 34

forgery localization [28], face reconstruction [8], and face segmentation [55]. 35

The prevailing multitask learning paradigm for face forgery detection follows a discriminative 36 approach, predicting multiple target outputs, one for each task, directly from the input face image. 37 Such a paradigm suffers from two main drawbacks. First, semantic relationships across tasks are 38 overlooked, which weakens knowledge transfer. For example, irrelevant information ( e.g. , every 39 detail of the face image in face reconstruction [8]) may be transferred across tasks. Second, extensive 40 human expertise should be involved, when determining task-agnostic (and task-specific) model 41 parameters and the loss weightings. 42

In this paper, we explore multitask learning for face forgery detection from a joint embedding 43 perspective [38]. In the joint embedding architecture, both the input and the target output are encoded 44 into latent representations in the shared feature space such that the irrelevant information can be 45 discarded from feature encoding. More importantly, the semantic closeness between two tasks can 46 be naturally modeled as the distance in the learned feature space, which is subsequently end-to47 end optimized to facilitate knowledge transfer across multiple tasks. Meanwhile, joint embedding 48 gives us a great opportunity to automate multitask learning in terms of allocating model capacity 49 ( i.e. , specifying task-agnostic and task-specific model parameters). In the context of face forgery 50 detection, the parameters of the face image encoder are shared across all tasks, whose capacity is 51 dynamically adjusted through end-to-end optimization. In addition, the multitask loss weightings can 52 be automatically computed in either theoretical [45, 65] or empirical [13, 36, 47] ways. 53

54

More concretely, we first introduce three coarse-to-fine face forgery detection tasks based on face

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

(2) A photo of a face with the global attribute of expression altered (1) A photo of a fake face (3) A photo of a face with the local attribute of mouth altered

attributes at different semantic levels. Leveraging the recent advances in vision-language correspondence as joint embedding [61], we encode the binary labels of the three tasks via textural prompts, and thus the semantic dependencies among tasks can be represented with the textual embeddings in the representation space. Fig. 1 shows an example, in which we describe a fake face image with a set of coarseto-fine textual descriptions: 1) ' A photo of a fake face ,' 2) ' A photo of a face with the global attribute of expression altered ,' and 3) ' A photo of a face with the local attribute of mouth altered .' By jointly embedding the face image and all its associated textural prompts through a popular vision-language model - CLIP [61], face forgery detection can then be performed by maximizing the vision-language correspondence.

Our contributions are threefold. First, we formulate multitask face forgery detection from a joint embedding perspective. Second, we define a set of coarse-to-fine face forgery detection tasks with corresponding textural templates to describe (fake) face images. Compared to previous multitask learning schemes, our instantiation gives rise to a more interpretable face forgery detector. Third, we conduct extensive experiments on five popular face forgery detection datasets, and show that our method performs favorably against state-of-the-art (SOTA) detectors in terms of generalization to novel face manipulations.

2 Related Work

In this section, we briefly review the literature on face forgery detection, multitask learning, and joint embedding architectures.

2.1 Face Forgery Detection

Many face forgery detection methods usually explore the specific clues to detect the forgery inspired 80 by the traditional photo forensics [15, 32, 33, 35, 46, 50, 51, 56], in which they detect eye blink81 ing [42], head pose [77], pupil shape [24], lipreading [26], statistical anomalies [43, 60, 66, 81], 82 corneal specularity [29], and idiosyncratic behavioral patterns of a well-known person [3]. In 83 recent years, there is a growing consensus of exploiting multitask learning on face forgery detec84 tion [8, 10, 41, 55, 81]. Besides the main face forgery classification task, these methods include 85 auxiliary tasks to get performance improvement by knowledge transfer across tasks, such as manipula86 tion type (and degree) classification [10], manipulation parameter estimation [75], blending boundary 87 detection [41], spatial forgery localization [28], face reconstruction [8], and face segmentation [55]. 88

With the development of deep learning, some advanced networks are employed to facilitate the face 89 forgery detection based on multiple tasks, such as two-stream CNN [82], self-attention model [80], 90 and vision transformers [19]. Additionally, more advanced training strategies are also utilized to 91 enhance the forgery detectors, including adversarial learning [10], reconstruction learning [8], and 92 meta learning [11]. However, the previous learning paradigm and human intervention are sub-optimal 93 for multitask learning on face forgery detection. In this paper, we explore an automated multitask 94 learning method for face forgery detection from the joint embedding perspective, where multiple 95 tasks are encoded into the language prompts, and vision-language correspondence is transferred 96 across tasks as the primary knowledge. 97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

2.2 Multitask Learning

Multitask learning aims to jointly learn multiple related tasks to improve the generalization performance of all tasks by leveraging the knowledge contained in each [79]. Two main groups are model parameter sharing and loss weighting. The former involves both manual specifications of shared parameters [4, 22, 37, 54] and learning to determine parameters for specific tasks [52, 64, 68, 74]. Loss weighting is typically divided as follows: Pareto Optimization (PO) methods and weight adaption methods. PO methods formulate multitask learning as a multi-objective optimization [45, 65], and find a Pareto stationary solution for the optimal loss weighting. Weight adoption methods adaptively adjust the loss weights during training based on pre-defined heuristics, such as uncertainty [36], gradient normalization [13], and loss descending rate [47]. In this paper, we consider multitask learning from the joint embedding perspective, in which the semantic closeness between tasks can be manifested as the distance in the learned feature space. Moreover, we assume all parameters in the image encoder are shared, whose capacity is dynamically allocated to each task during end-to-end optimization. We also adopt the method in [47] for dynamic loss weighting.

2.3 Joint Embedding Architectures

Joint embedding architectures (JEA) [38] aim at learning to output similar embeddings for compatible inputs, x and y , and dissimilar embeddings for incompatible inputs, which is different from the discriminative approaches that predict y directly from x . Becker et al. [6] propose the first JEA for maximizing mutual information between representations from two views of the same scene. Later on, Bromley et al. [7] propose a contrastive method of JEA for signatures verification. After a long hiatus, JEA has been re-explored in face verification [14] and recognition [69], dimensionality reduction [25], and video feature learning [70]. With the emergence of self-supervised learning, the use of JEA has explored in recent years with methods training on contrastively ( e.g. , PIRL [53], MoCo [27], and SimCLR [12]) or non-contrastively ( e.g. , BYOL [23], Barlow Twins [78], and I-JEPA [5]). More recently, the emerging vision-language foundation models [30, 61] can also be grouped into JEA, in which two separate encoders encode the compatible visual ( i.e. , x ) and textual ( i.e. , y ) inputs into similar embeddings and contrast incompatible visual and textual embeddings. In this paper, we use CLIP [61], a joint vision-language model pretrained on massive image-text pairs, to implement the JEA to aid DeepFake detection by vision-language correspondence in the embedding space. Moreover, we end-to-end fine-tune the CLIP in the context of automated multitask learning.

3 Method

In this section, we present multitask learning for face forgery detection using a joint embedding approach, including preliminaries of the problem formulation, language prompts over multiple tasks, and specifications of loss functions. The main joint embedding framework for face forgery detection is shown in Fig. 2.

3.1 Preliminaries

Given a face image x ∈ R N , a face forgery detector f θ : R N ↦→ R aims to predict a binary label 134 y for the authenticity of x , i.e. , 0 as the real or 1 as the fake. Considering that existing forged face 135 images are mainly generated by modifying face components/attributes, we include two other related 136 tasks - global face manipulation detection and local face manipulation detection. We consider three 137

A photo of a {b} face.

A photo of a {b} face.

A photo of a face with

{a} attribute

modified.

Contrastive Prompt Pairs

A photo of a {b} face.

A photo of a {b} face.

A photo of a face with

{a} attribute

modified.

Text

Encoder

Textual

Embeddings

Textual

Embeddings

Textual

Embeddings

Textual

Embeddings

Input Face Image

Image

Encoder

Binary Category

Probability

Global Category

Probability

Local Category

Probability

Fidelity

Loss

Softmax

Visual

Embedding

A photo of a  face with

the global attribute of

{g}

altered

A photo of a face with

the global attribute of

{g}

unaltered

Textual

Embeddings

Cosine

Similarity

face attributes ( i.e. , expression, identity, and physical consistency 1 ) for global face manipulations, and 138 four face attributes ( i.e. , eye, illumination, mouth, and nose) for local face manipulations. Notably, a 139 face image may contain multiple attribute labels. 140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

3.2 Multitask Language Prompts

For each face attribute label from multiple tasks, we encode the ground-truth labels via language prompts. In specific, we design textual templates as follows. 1) binary level : a photo of a { c } face , where c ∈ C = {real, fake}; 2) global-attribute level : A photo of a face with the global attribute of { g } altered , where g ∈ G = {expression, identity, physical consistency}; and 3) local-attribute level : A photo of a face with the local attribute of { l } altered , where l ∈ L = {eye, illumination, mouth, nose}. Inspired by contrastive methods [27, 53] in the joint embedding architecture, we also introduce contrastive language prompts, which are opposite in meaning to the original textual templates. Thus, we can have a contrastive prompts pair for each attribute label, as follows: global-attribute level : {(1) A photo of a face with the global attribute of { g } altered , (2) A photo of a face with the global attribute of { g } unaltered }; local-attribute level : {(1) A photo of a face with the local attribute of { l } altered , (2) A photo of a face with the local attribute of { l } unaltered }. Notably, the binary level prompts naturally have the property of contrastive prompt pairing. In this way, multiple tasks are encoded into a text corpus T , where each language prompt represents a ground-truth label y of the corresponding task, and their semantic closeness can be learned through joint embedding.

3.3 Multitask Learning via Joint Embedding

Joint Embedding Formulation . Given the input face image x and the set of possible outputs Y , we predict the output by minimizing an energy-based model [39], i.e. , ˆ y = arg min y ∈Y E ( x , y ) , in the joint embedding architecture. In this paper, we construct E by two encoders: one image encoder f ϕ : R N ↦→ R K for encoding the face image and one text encoder g φ : T ↦→ R K for encoding the language prompts, parameterized by ϕ and φ , respectively.

The ideal energy landscape of joint embedding satisfies that the energy is low for similar embeddings 162 of compatible inputs, while energy is high for dissimilar embeddings [39]. Thus, we calculate 163 the probability of similarity ˆ p ( ·| x ) between the visual embedding and textual embeddings for the 164 following optimization. Let u ∈ R K be the visual embedding, and let v ∈ R K and ¯ v ∈ R K be the 165 textual embeddings from the two prompts opposing in meaning, we then estimate ˆ p ( ·| x ) as 166

where 167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

⟨· , ·⟩ denotes the inner product and ∥ · ∥ represents the ℓ 2 -norm. The probability ˆ p ( ·| x ) is the abbreviation of ˆ p ( c | x ) , ˆ p ( g | x ) , and ˆ p ( l | x ) according to a specific task, and a larger probability indicates a closer match to the corresponding semantic meaning of v .

Losses for Multitask Learning . We use the statistical distance measure in the form of fidelity loss [73] to calculate the losses for multitask learning. Given the predicted category probability ˆ p ( c | x ) , we design the loss at the binary level as

where θ = { ϕ , φ } indicates the learnable parameters in image and language encoders, and p ( c | x ) = 1 if x belongs to the c category or otherwise we have p ( c | x ) = 0 . In our setting, a face image can be assigned with labels regarding one or more global face attribute manipulations, which forms a typical multi-label classification problem. Therefore, the averaged loss at the global-attribute level can be defined as follows,

where p ( g | x ) = 1 if x belongs to the g category, otherwise we have p ( g | x ) = 0 . Since the manipulations over different local face attributes may appear in one face image, we also consider it as a multi-label classification task, and the loss at the local-attribute level is:

where p ( l | x ) = 1 if x belongs to the l category.

Given a minibatch of training data B at the t -th iteration, we evaluate the overall loss function via the weighted sum of the individual losses in different levels as follows,

Here, the weighting vector λ ( t ) = [ λ 1 ( t ) , λ 2 ( t ) , λ 3 ( t )] ⊺ at the t -th iteration is automatically computed according to the relative descending rate [47]:

and τ is a fixed temperature parameter.

4 Experiments

4.1 Experimental Setup

Datasets. We adopt the widely used FF++ [63] dataset for training. It contains 1 , 000 real videos, among which 720 and 140 are used for training and validation, respectively, and the remaining 140 are reserved for testing. All videos are manipulated by four face forgery methods, including Deepfakes [1], Face2Face [72], FaceSwap [2], and NeuralTexures [71], with three compression levels, i.e. , no compression (denoted as Raw), slight compression with quantization parameter QP = 23 (denoted as C23 ), and severe compression with QP = 40 (denoted as C40 ). Following [10, 11, 26], C23 version is adopted by default in our experiments. We evaluate the generalizability of the proposed method on four popular DeepFake benchmarks, including FaceShifter (FSh) [40], Celeb-DF (CDF) [44], DeeperForensics-1.0 (DF-1.0) [31], and DeepFake Detection Challenge (DFDC) [18].

Implementation Details. To facilitate the multitask learning via joint embedding paradigm, we need face images associated with the proposed textual templates. In this paper, we adopt FF++ [63] to enrich the training data. Following the general generation procedures ( i.e. , detecting face and then blending two faces according to the region-of-interest mask) in [10, 41], we focus on supplementing the tampering of 'expression' on 'eye' and individual face attribute that is linked to 'physical consistency', i.e. , 'eye', 'illumination', 'mouth', and 'nose'. Face attribute manipulations associated with other textual prompts are already included in FF++.

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

As for face pre-processing, we use RetinaFace [17] to detect faces and save the aligned face images as input with a size of 317 × 317 . As in [63], we only extract the largest face and use an enlarged crop, 1 . 3 × the tight crop produced by the face detector.

As for the training, we use CLIP [61] to implement the joint embedding architecture, where we adopt ViT-B/32 [20] as the visual encoder and GPT-2 [62] with a base size of 63 M-parameter as the text encoder. We then train the model by minimizing the loss using AdamW [49] with a decoupled weight decay of 1 × 10 -3 . The initial learning rate is set to 1 × 10 -7 , which changes following a cosine annealing schedule [48]. The model is optimized for 36 epochs with mini-batches of 32 . Data augmentation strategy is also applied during training, which is a common trick in the face forgery detection [41, 76, 80], and details can be found in Sec. 4.3. A single NVIDIA RTX 3090 GPU is used during training.

4.2 Comparison with SOTA Methods

We compare our method with the several SOTA methods, including Face X-ray [41], PCL [81], MADD [80], LipForensics [26], RECCE [8], SBI [66], ICT [19], SLADD [10], and OST [11], to demonstrate its superiority. The test performance on five datasets are listed in Table 1.

Table 1 shows that many methods do not perform satisfactorily on face forgery detection, while the proposed method outperforms all the recent SOTA, achieving 92 . 33% of AUC averaged from five test datasets and surpassing the second best, i.e. , LipForensics, by 2 . 79% in the term of Mean AUC over datasets including FF++ [63]. For

cross-dataset generalizability comparison, the proposed method also surpasses the second best ( i.e. , ICT) and third best ( i.e. , LipForensics) by 2 . 79% and 3 . 14% , respectively. In addition, we also have several interesting observations. First , all the methods can achieve saturated performance in FF++ [63], while underperform in the rest datasets, such as CDF [44] and DFDC [18]. This suggests that the forgery cues in FF++ are easier to spot and overfit by these forgery detectors. Second , SBI reports a very high AUC of 93 . 18% on CDF, while performing unsatisfactorily on DF-1.0 [31] and DFDC. Similar results are also demonstrated by PCL, which exhibits an exceedingly high AUC of 99 . 40% on DF-1.0 but underperforms in DFDC. This may arise due to the overfitting on the low-level features, such as statistical inconsistency ( e.g. , landmark and color mismatch). Third , all methods obtain relatively low scores on DFDC, which we attribute to the domain shift caused by significantly different filming conditions. However, our method achieves a relative satisfactory result with a score of 82 . 06% , surpassing the second best by 4 . 33% . In summary, the remarkable results validate the effectiveness and superiority of the proposed joint-embedding-based multitask learning for DeepFake detection.

4.3 Robustness Analysis

In this subsection, we study the robustness performance of the proposed method. Following [31], we consider four popular perturbations ( i.e. , Patch Substitution (Patch-Sub), additive white Gaussian Noise contamination (Noise), Gaussian Blurring (Blur), and pixelation), and only four severity levels

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

( i.e. , from level 1 to level 4 ) are considered in the experiments 2 . Two different models are evaluated in this section, i.e. , our model training without data augmentation (denoted as Ours (w/o Aug)) and our model training with data augmentation strategy (denoted as Ours). In specific, when training with data augmentation strategy, each training data is augmented with a probability of 0 . 3 by one randomly chosen perturbation during training, in which severity level is randomly applied at level 1 or 2 .

To begin, we first evaluate the robustness for the model without data augmentation. We find that the CLIP-based model is sensitive to the perturbations to images, which we argue that the vision-language correspondence is corrupted by perturbations. We then evaluate the model training with data augmentation. In Table 2, we find that training with a slight data augmentation can alleviate the model sensitivity to the perturbations, and achieve a satisfactory performance on average. Moreover, the model of Ours also maintains a satisfactory performance on pixelation and Blur. It is noteworthy that CNND [76] and Face X-ray [41] also augment their training data by compression and blurring during training, thus leading to good robustness to perturbations of pixelation and Blur. Fig. 3 demonstrates the effect of increasing the severity for each perturbation, where we compare with Xception [63], CNND, PatchForensics [9],























































































































Face X-ray, and LipForensics [26]. It can be observed that the proposed method maintains a good performance against the perturbations by Patch-Sub and Noise, while other methods suffer from the Noise, and LipForensics also suffers from the Patch-Sub.

4.4 Ablation Studies

Joint Embedding Framework. We conducted a series of ablations to verify the instantiated joint embedding framework by CLIP [61]. We first (1) evaluate the pretrained CLIP, and then (2) fine-tune it with the frozen text encoder on FF++ [63]. The following ablations adopt the same training procedure, while differing in two alternatives: (3) using equal task weights for multiple tasks instead of dynamic loss weighting; (4) training without the contrastive prompt pairs, i.e. , no contrastive textual descriptions are used during training. From Table 3, we can observe that freezing language encoder negatively affects the generalization performance, which we believe is because forgeryrelated concepts have not been sufficiently captured during the pretraining stage of CLIP. We also find that utilizing contrastive prompts can improve generalization, further indicating the contrasting

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

operation can benefit the joint embedding methods [12, 27]. Moreover, including the dynamic loss weighting scheme is advantageous as it not only yields a slight improvement compared to using equal task weights but also frees us from the burdensome task of hyper-parameter tuning.

Textual Templates. In this subsection, we investigate how the textual template design affects the model performance. We try three different alternatives from single task to three tasks: (5) binary-level text templates, i.e. , single task formulation only considering the label of real or fake; (6) two-level separate text templates, i.e. , two-level-task formulation, where we consider the separate templates describing the overall authenticity and global face attributes; and (7) the joint text templates putting together labels from three tasks, e.g. , ' A photo of a { fake } face with the global attribute of { expression } and the local attribute of { mouth } are altered '. The joint probability over multiple tasks can be computed from the similarities between the image embedding and all candidate tex-

tual embeddings. Then, we marginalize the joint distribution to obtain the marginal probability for each task. From Table 3, we can observe that the performance of the model using joint templates is inferior to that of the model using separate templates ( i.e. , Ours (Default)), indicating that separate templates for each task are more conducive for learning the semantic closeness between two face forgery detection tasks in joint embedding. On the other hand, less tasks ( i.e. , single task and two tasks) result in the inferior performance. Notably, benefiting from the joint embedding, the model using binary templates also achieves comparable results on generalization, though it only classifies the overall authenticity of the face.

Encoder Architecture. In this subsection, we investigate other visual encoders with different settings and model sizes. In specific, we choose (8) ViT-B/16 [20] and (9) ViT-L/14 [20]. As shown in Table 3, two alternative ViT-based architectures achieve better results on generalization. However, the larger model will result in both computationally more expensive and time-consuming.

Multitask Objective. In this subsection, we study how different optimization objectives affect the performance. As a reference, we first replace the fidelity loss functions with (10) binary cross entropy loss (BCE Loss). We also adopt the (11) hierarchical probabilistic loss [16] to jointly formulate multi-level classification tasks under a hierarchical label semantic graph. The relative similarity score ( i.e. , s -¯ s ), as a raw score, for each node in the label hierarchy, will be converted into marginal probabilities for loss computation. From Table 3, we observed that the proposed method outperforms the variant trained with BCE loss, thus providing evidence for the effectiveness of the designed fidelity losses. Furthermore, Table 3 shows that fidelity loss yields better performance than the hierarchical probabilistic loss, suggesting that implicitly learning the semantic dependencies may be better than explicitly encoding the prior knowledge in the label hierarchy graph in advance.

4.5 Discussion: Vision-Language Correspondence

Human-Understandable Interpretation . The proposed joint embedding approach enjoys the vision-language correspondence, which naturally provides model interpretations by providing humanunderstandable explanations. Fig. 4 shows some examples of FF++ [63], in which Deepfakes [1] indicate the identity swap, leading all local parts of the face are fake; and NeuralTextures [71] modify the expression in the mouth part. Take an example of NeuralTextures, the texts with a probability over 50% include 'fake', 'expression', and 'mouth'. Hence, we consider this face image to be fake

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

[Fake]faceimage

ecause...

fake

99.05

ssion0.59%

entity

88.43%

tency

0.13%

eye

96.88%

lation0.03%

nouth

100.0

nose

81.54%

[Fake] face image

ecause...

fake

98.87

ssion

99.9

entity

1.12%

tency

1.13%

eye

33.47%

lation0.00%

nouth

96.87

nose

1.04%

Deeplakes

and w

phy

andw

phy

1%

because the model's prediction relies on the following three textual prompts: ' a photo of a fake face ', ' a photo of a face with the global attribute of expression altered ', and ' a photo of a face with the local attribute of mouth altered '. More examples can be found in Appendix.

Semantic Closeness across Tasks . We show the semantic closeness across tasks by a correlation matrix in Fig. 5, in which each entry is represented by the cosine similarity between two textual embeddings from the language prompts depicting the specific tasks. From Fig. 5, we can observe that the text encoder of the pretrained CLIP has not sufficiently captured the semantic closeness across tasks and treats most tasks equally, further verifying the results of the variant with frozen text encoder in Table 3. After joint embedding learning on the forged faces, the semantic closeness across tasks can be sufficiently learned, e.g. , the concept of 'identity' forgery is more related to the 'nose', 'mouth', and 'eye', thus improving the performance of multitask learning for face forgery detection.

(a) By Text Encoder of Pretrained CLIP

(b) By Text Encoder of Fine-tuned CLIP

- 1.00

- 1.00

fake-1.00

0.87 0.91 0.92 0.91

0.84

0.900.89

fake -1.00

0.250.190.510.280.220.380.24

expression-0.87

1.000.94 0.93 0.920.850.92 0.90

-0.98

expression-0.25

1.00

0.80

identity -0.91 0.94

1.00

0.950.930.860.930.92

0.95

identity -0.19 0.18

1.000.050.430.180.41

0.64

physical inconsistency-0.92 0.93 0.95

1.000.930.850.930.92

physical inconsistency-0.51

0.070.05

1.000.080.280.150.22

0.60

0.93

eye-0.91 0.92 0.93 0.93

1.00

0.880.950.96

eye -0.28 0.25 0.43 0.08

1.000.230.360.46

0.90

0.40

illumination-0.84 0.85 0.86 0.85 0.88

1.00

0.87 0.87

illumination-0.220.09 0.180.28 0.23

1.00

0.200.23

mouth -0.90 0.92 0.93 0.93 0.950.87

1.00

0.96

0.88

mouth-0.38 0.36 0.410.150.36 0.20

1.00

0.40

0.20

nose-0.89 0.90 0.92 0.92 0.960.87

0.96

1.00

0.85

nose -0.24 0.10 0.64 0.22 0.46 0.23 0.40

1.00

fake

identity

mouth

nose

fake

identity

mouth

nose

expression

illumination

expression

illumination

physical inconsistency

physical inconsistency

5 Conclusion and Limitations

Conclusion . In this paper, we consider multitask learning for face forgery detection from the joint embedding perspective. We have designed a set of coarse-to-fine language prompts to represent multiple tasks for face forgery detection. We then take an automated multitask learning scheme to train two encoders to joint embed visual face images and textual descriptions. Thus, semantic closeness across tasks is manifested as the distance in the learned feature space, thus improving multitask learning. From extensive experiments, vision-language correspondence after joint embedding shows great promise to support better face forgery detection by maximizing the feature similarity between the face image and candidate textual prompts, verifying the effectiveness and superiority of the proposed method. Moreover, the joint embedding scheme also renders some degree of model interpretation in a human-friendly way.

Limitations . The proposed method relies on the assumption that the forged faces are generated with the blending operation [41]. Thus, it may perform unsatisfactorily when fake face images are totally synthesized by GAN- or diffusion-model-based methods. Additionally, our model is image-based, though it can handle video-based DeepFake by sampling frames for prediction, it may fail when encountering the fake video manipulated by only lowering the frame rate [57].

References 383

Deepfakes. https://github.com/deepfakes/faceswap . 384

FaceSwap. https://github.com/MarekKowalski/FaceSwap . 385

S. Agarwal, H. Farid, Y. Gu, M. He, K. Nagano, and H. Li. Protecting world leaders against deep fakes. In 386 CVPRW , pages 38-45, 2019. 387

A. Argyriou, T. Evgeniou, and M. Pontil. Multi-task feature learning. In NIPS , pages 1-13, 2006. 388

M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas. 389 Self-supervised learning from images with a joint-embedding predictive architecture. arXiv preprint 390 arXiv:2301.08243 , 2023. 391

S. Becker and G. E. Hinton. Self-organizing neural network that discovers surfaces in random-dot 392 stereograms. Nature , 355(6356):161-163, 1992. 393

J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah. Signature verification using a 'Siamese' time 394 delay neural network. In NIPS , pages 737-744, 1993. 395

J. Cao, C. Ma, T. Yao, S. Chen, S. Ding, and X. Yang. End-to-end reconstruction-classification learning for 396 face forgery detection. In CVPR , pages 4113-4122, 2022. 397

L. Chai, D. Bau, S.-N. Lim, and P. Isola. What makes fake images detectable? Understanding properties 398 that generalize. In ECCV , pages 103-120, 2020. 399

L. Chen, Y. Zhang, Y. Song, L. Liu, and J. Wang. Self-supervised learning of adversarial example: Towards 400 good generalizations for DeepFake detection. In CVPR , pages 18710-18719, 2022. 401

402

L. Chen, Y. Zhang, Y. Song, J. Wang, and L. Liu. OST: Improving generalization of DeepFake detection

403

via one-shot test-time training. In

NIPS

, pages 1-14, 2022.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual 404 representations. In ICML , pages 1597-1607, 2020. 405

Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. GradNorm: Gradient normalization for 406 adaptive loss balancing in deep multitask networks. In ICML , pages 794-803, 2018. 407

S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to 408 face verification. In CVPR , pages 539-546, 2005. 409

V. Conotter, J. F. O'Brien, and H. Farid. Exposing digital forgeries in ballistic motion. IEEE TIFS , 410 7(1):283-296, 2011. 411

J. Deng, N. Ding, Y. Jia, A. Frome, K. Murphy, S. Bengio, Y. Li, H. Neven, and H. Adam. Large-scale 412 object classification using label relation graphs. In ECCV , pages 48-64, 2014. 413

J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou. RetinaFace: Single-shot multi-level face 414 localisation in the wild. In CVPR , pages 5203-5212, 2020. 415

B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer. The DeepFake detection 416 challenge (DFDC) dataset. arXiv preprint arXiv:2006.07397 , 2020. 417

X. Dong, J. Bao, D. Chen, T. Zhang, W. Zhang, N. Yu, D. Chen, F. Wen, and B. Guo. Protecting celebrities 418 from DeepFake with identity consistency transformer. In CVPR , pages 9468-9478, 2022. 419

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min420

421

derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers

422

for image recognition at scale. In

ICLR

, pages 1-12, 2020.

H. Farid. Image forgery detection: A survey. IEEE SPM , 26(2):16-25, 2009. 423

Y. Gao, J. Ma, M. Zhao, W. Liu, and A. L. Yuille. NDDR-CNN: Layerwise feature fusing in multi-task 424 cnns by neural discriminative dimensionality reduction. In CVPR , pages 3205-3214, 2019. 425

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, 426 M. Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: A 427 new approach to self-supervised learning. In NIPS , pages 21271-21284, 2020. 428

H. Guo, S. Hu, X. Wang, M.-C. Chang, and S. Lyu. Eyes tell all: Irregular pupil shapes reveal GAN429 generated faces. In ICASSP , pages 2904-2908, 2022. 430

R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 431 CVPR , pages 1735-1742, 2006. 432

A. Haliassos, K. Vougioukas, S. Petridis, and M. Pantic. Lips don't lie: A generalisable and robust approach 433 to face forgery detection. In CVPR , pages 5039-5049, 2021. 434

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation 435 learning. In CVPR , pages 9729-9738, 2020. 436

Y. He, B. Gan, S. Chen, Y. Zhou, G. Yin, L. Song, L. Sheng, J. Shao, and Z. Liu. ForgeryNet: A versatile 437 benchmark for comprehensive forgery analysis. In CVPR , pages 4360-4369, 2021. 438

S. Hu, Y. Li, and S. Lyu. Exposing GAN-generated faces using inconsistent corneal specular highlights. In 439 ICASSP , pages 2500-2504, 2021. 440

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig. Scaling up 441

visual and vision-language representation learning with noisy text supervision. In ICML , pages 4904-4916, 442 2021. 443

L. Jiang, R. Li, W. Wu, C. Qian, and C. C. Loy. DeeperForensics-1.0: A large-scale dataset for real-world 444

face forgery detection. In CVPR , pages 2889-2898, 2020. 445

[11]

M. K. Johnson and H. Farid. Exposing digital forgeries through chromatic aberration. In ACM MM&Sec , 446 pages 48-55, 2006. 447

M. K. Johnson and H. Farid. Exposing digital forgeries in complex lighting environments. IEEE TIFS , 448 2(3):450-461, 2007. 449

T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image 450 quality of StyleGAN. In CVPR , pages 8110-8119, 2020. 451

E. Kee, J. F. O'brien, and H. Farid. Exposing photo manipulation from shading and shadows. ACM TOG , 452 33(5):165:1-165:21, 2014. 453

A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry 454 and semantics. In CVPR , pages 7482-7491, 2018. 455

I. Kokkinos. UberNet: Training a universal convolutional neural network for low-, mid-, and high-level 456 vision using diverse datasets and limited memory. In CVPR , pages 6129-6138, 2017. 457

Y. LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. 2022. 458

Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy-based learning. 459 Predicting Structured Data , 1(0):1-59, 2006. 460

L. Li, J. Bao, H. Yang, D. Chen, and F. Wen. Advancing high fidelity identity swapping for forgery 461 detection. In CVPR , pages 5074-5083, 2020. 462

L. Li, J. Bao, T. Zhang, H. Yang, D. Chen, F. Wen, and B. Guo. Face X-ray for more general face forgery 463 detection. In CVPR , pages 5001-5010, 2020. 464

Y. Li, M.-C. Chang, and S. Lyu. In Ictu Oculi: Exposing AI created fake videos by detecting eye blinking. 465 In WIFS , pages 1-7, 2018. 466

Y. Li and S. Lyu. Exposing DeepFake videos by detecting face warping artifacts. In CVPRW , pages 46-52, 467 2019. 468

Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu. Celeb-DF: A large-scale challenging dataset for DeepFake 469 forensics. In CVPR , pages 3207-3216, 2020. 470

X. Lin, H.-L. Zhen, Z. Li, Q.-F. Zhang, and S. Kwong. Pareto multi-task learning. In NIPS , pages 471 12037-12047, 2019. 472

Z. Lin, R. Wang, X. Tang, and H.-Y. Shum. Detecting doctored images using camera response normality 473 and consistency. In CVPR , pages 1087-1092, 2005. 474

S. Liu, E. Johns, and A. J. Davison. End-to-end multi-task learning with attention. In CVPR , pages 475 1871-1880, 2019. 476

I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR , pages 1-13, 477 2017. 478

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR , pages 1-10, 2019. 479

S. Lyu. Estimating vignetting function from a single image for image authentication. In ACM MM&Sec , 480 pages 3-12, 2010. 481

S. Lyu, X. Pan, and X. Zhang. Exposing region splicing forgeries with blind local noise estimation. IJCV , 482 110(2):202-221, 2014. 483

A. Mallya, D. Davis, and S. Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning 484 to mask weights. In ECCV , pages 72-88, 2018. 485

I. Misra and L. v. d. Maaten. Self-supervised learning of pretext-invariant representations. In CVPR , pages 486 6707-6717, 2020. 487

I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In CVPR , 488 pages 3994-4003, 2016. 489

H. H. Nguyen, F. Fang, J. Yamagishi, and I. Echizen. Multi-task learning for detecting and segmenting 490 manipulated facial images and videos. In BTAS , pages 1-8, 2019. 491

J. F. O'brien and H. Farid. Exposing photo manipulation with inconsistent reflections. ACM TOG , 492 31(1):4:1-4:11, 2012. 493

D. O'Sullivan. Doctored videos shared to make Pelosi sound drunk viewed millions of times on social me494 dia. https://edition.cnn.com/2019/05/23/politics/doctored-video-pelosi/index.html , 495 2019. Date of access: May 12, 2023. 496

A. C. Popescu and H. Farid. Exposing digital forgeries by detecting traces of resampling. IEEE TSP , 497 53(2):758-767, 2005. 498

A. C. Popescu and H. Farid. Exposing digital forgeries in color filter array interpolated images. IEEE TSP , 499 53(10):3948-3959, 2005. 500

Y. Qian, G. Yin, L. Sheng, Z. Chen, and J. Shao. Thinking in frequency: Face forgery detection by mining 501 frequency-aware clues. In ECCV , pages 86-103, 2020. 502

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, 503 J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language 504 supervision. In ICML , pages 8748-8763, 2021. 505

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised 506 multitask learners. 2019. 507

A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner. FaceForensics++: Learning to 508 detect manipulated facial images. In ICCV , pages 1-11, 2019. 509

S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard. Latent multi-task architecture learning. In AAAI , pages 510 4822-4829, 2019. 511

O. Sener and V. Koltun. Multi-task learning as multi-objective optimization. In NIPS , pages 525-536, 512 2018. 513

K. Shiohara and T. Yamasaki. Detecting deepfakes with self-blended images. In CVPR , pages 18720-18729, 514 2022. 515

Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In NIPS , page 516 11918-11930, 2019. 517

X. Sun, R. Panda, R. Feris, and K. Saenko. AdaShare: Learning what to share for efficient deep multi-task 518 learning. In NIPS , pages 8728-8740, 2020. 519

Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. DeepFace: Closing the gap to human-level performance 520 in face verification. In CVPR , pages 1701-1708, 2014. 521

G. W. Taylor, I. Spiro, C. Bregler, and R. Fergus. Learning invariance through imitation. In CVPR , pages 522 2729-2736, 2011. 523

J. Thies, M. Zollhöfer, and M. Nießner. Deferred neural rendering: Image synthesis using neural textures. 524 ACM TOG , 38(4):1-12, 2019. 525

J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. Face2Face: Real-time face capture 526 and reenactment of RGB videos. In CVPR , pages 2387-2395, 2016. 527

M.-F. Tsai, T.-Y. Liu, T. Qin, H.-H. Chen, and W.-Y. Ma. FRank: A ranking method with fidelity loss. In 528 ACM SIGIR , pages 383-390, 2007. 529

M. Wallingford, H. Li, A. Achille, A. Ravichandran, C. Fowlkes, R. Bhotika, and S. Soatto. Task adaptive 530 parameter sharing for multi-task learning. In CVPR , pages 7561-7570, 2022. 531

S.-Y. Wang, O. Wang, A. Owens, R. Zhang, and A. A. Efros. Detecting Photoshopped faces by scripting 532 Photoshop. In ICCV , pages 10072-10081, 2019. 533

S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros. CNN-generated images are surprisingly easy 534 to spot...for now. In CVPR , pages 8695-8704, 2020. 535

X. Yang, Y. Li, and S. Lyu. Exposing Deep Fakes using inconsistent head poses. In ICASSP , pages 536 8261-8265, 2019. 537

J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny. Barlow Twins: Self-supervised learning via redundancy 538 reduction. In ICML , pages 12310-12320, 2021. 539

Y. Zhang and Q. Yang. A survey on multi-task learning. IEEE TKDE , 34(12):5586-5609, 2021. 540

H. Zhao, W. Zhou, D. Chen, T. Wei, W. Zhang, and N. Yu. Multi-attentional deepfake detection. In CVPR , 541 pages 2185-2194, 2021. 542

T. Zhao, X. Xu, M. Xu, H. Ding, Y. Xiong, and W. Xia. Learning self-consistency for deepfake detection. 543 In ICCV , pages 15023-15033, 2021. 544

P. Zhou, X. Han, V. I. Morariu, and L. S. Davis. Two-stream neural networks for tampered face detection. 545

In CVPRW , pages 1831-1839, 2017. 546

Peer reviews:
comments: SUMMARY:
The proposed method introduces a novel approach to deepfake detection by integrating natural language and image information. Moreover, it attains state-of-the-art performance on several contemporary deepfake datasets and can generate explanatory sentences that justify the authenticity or falsity of the input image, which is crucial in the field of deepfake detection.

STRENGTHS:
See Questions Section in detail.

WEAKNESSES:
See Questions Section in detail.

QUESTIONS:
Although the proposed method demonstrates remarkable performance on contemporary deepfake datasets, the supplemental materials reveal that without the proposed dataSup scheme, the method can only achieve 80.76 and 75.94 AUC on Celeb-DF and DFDC, respectively, which is inferior to state-of-the-art methods.

Based on this, I have the following reservations about the submitted manuscript:

1. It would be preferable for the authors to compare the performance of their method with the proposed dataSup scheme and previous methods such as Face X-Ray, SBI and SLADD using a unified backbone and then analyze the impact of different dataSup schemes.
2. The data augmentation scheme is essential according to Table 2 in the supplemental materials. Therefore, I recommend that the authors elaborate on their dataSup scheme in detail as the current description of the scheme is obscure and difficult to replicate.

LIMITATIONS:
The limitations are well illustrated in the manuscript.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multitask learning framework for video deepfake detection. The idea is to rely on a joint embedding architecture and define a set of coarse-to-fine face forgery detection tasks with corresponding textual descriptions for fake face images (binary level, global-attribute level and local-attribute level). This helps to obtain understandable explanations and hence a more interpretable forensic detector. CLIP is used to implement the joint embedding architecture, while ViT-B/32 is adopted as the visual encoder and GPT-2 as the text encoder. Experiments are carried out on several publicly available datasets and show better performance in terms of generalization compared with SOTA methods.\n\nSTRENGTHS:\n- It is very relevant to design a deepfake detector that is able to generalize to different types of manipulations since often current deepfake methods perform poorly on forgeries not seen during training.\n- It is also very valuable to design a detector that can provide explanations about the manipulations. \n- The idea to encode the ground-truth labels via language prompts is interesting and not explored yet in the context of deepfakes.\n\nWEAKNESSES:\n- The technical description of the method based on multitask learning (Section 3.3) is very generic and not related at all with the problem of deepfakes. In addition, the technical contribution seems to come from already published work: the joint embedding formulation is inspired by minimizing an energy-based model as in [39] and the losses for multitask Learning are inspired by [73]. \n\n- The section on Multitask Language Prompts (3.2) is more related to the specific application, but it is not justified why it is important to consider a coarse-to-fine approach and above all it is not clear how the ground-truth labels via language prompts have been generated. It is said 'Face attribute manipulations associated with other textual prompts are already included in FF++.', but this is new to me. FF++ is only labelled using four different manipulations but does not include the global-attribute level and local-attribute level as described in Section 3.2. This is absolutely not clear. \n\n- In this same section there is a reference to a face attribute called 'physical consistency', which is explained in the supplemental material. However, Section 1 of the appendix is very confusing and I was not able to understand it clearly. \n\n- The ablation study is confused. There are several variants that perform well as the proposal in Table 1 and this is puzzling.\n\n- Comparison with SOTA methods should be enlarged including also other methods, such as [66] and [19].\n\n- The experiments that show that the explanations provided by the detector are correct are too limited. They have been shown only on FF++ (Section 4.5 and Appendix). What is more interesting is the ability to generalize to other datasets different from the training one and these are not present in the paper. This is very limiting and does not help to show the relevance of the proposal as stated in the Introduction.\n\n- The paper needs a major re-writing. The presentation is poor and hence not adequate for NeurIPS.\n\nQUESTIONS:\nPlease, refer to the weaknesses section.\n\nLIMITATIONS:\nAuthors have presented the limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper appears to be about a method for detecting manipulated facial images, specifically deepfakes. The authors have used a model that employs a joint embedding architecture, using ViT-B/32 as the visual encoder and GPT-2 as the text encoder. The model is trained using AdamW with a decoupled weight decay of 1 × 10−3 and an initial learning rate set to 1 × 10−7, which changes following a cosine annealing schedule. The authors compare their method with several state-of-the-art (SOTA) methods, including Face X-ray, PCL, MADD, LipForensics, RECCE, SBI, ICT, SLADD, and OST. The results show that their proposed method outperforms all the recent SOTA.\n\nSTRENGTHS:\n1.\tThis paper is well written and easy to follow and will be of interest to researchers from the community of multitask learning in deepfake.\n\n2.\tFinding the semantic dependencies among tasks using texture prompts is clear and places the previous work very well in context of this framework.\n\n3.\tFinally, the authors provide experimental results to demonstrate the effectiveness of the proposed objective function and algorithms.\n\nWEAKNESSES:\n1.\tThe majority of the contributions in this study are essentially modifications of existing work. Additionally, the significance of the main contribution appears to involve identifying similarities among previous work and proposing a comprehensive generalization that encompasses a significant portion of the existing research. While this contribution may enhance understanding, it seems to be primarily pedagogical in nature rather than being a novel research finding.\n\n2.\tThe complexity of the proposed method seems high (impractical). How effectively does it handle large datasets? Is it possible to use it in conjunction with sparse variational inference approaches?\n\n3.\tIt would be great if the authors could extend the proposed algorithm to adapt to other types of loss functions ( from eq.3 – eq.7) such as exp-concave and strongly convex functions.\n\nQUESTIONS:\n1.\tHow was the drop calculated in Table 2 ?\n\n2.\t Contrastive textual pairing is not defined well? \n\n3.\t How did the minimize energy based model for joint embedding ?\n\n4. What is the performance of the model when dealing with different convex loss functions?\n\n5. How does the energy-based model function within the GPT-2 and CLIP-based embedding framework?\n\n6. What are the challenges and potential solutions when applying your method to face images fully synthesized by GANs or diffusion models?\n\nLIMITATIONS:\nAs described in the manuscript, the proposed method may perform unsatisfactorily when encountering fake face images generated by diffusion-model-based methods. \n\nAlso, see the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an automated multitask learning framework for face forgery detection from a joint embedding perspective. The central idea is to utilize the multi-modality of visual and textural features to enhance blending-based face forgery detection with the global and local semantic face attributes. Experiments demonstrate the effectiveness of this proposed framework.\n\nSTRENGTHS:\nThe new paradigm of multitask learning strategy from a joint embedding perspective is introduced into the face forgery detection field. The work trains two encoders to jointly embed visual face images and textual descriptions in the shared feature space. Thus, one can guide the forgery detection that is mainly based on visual content with textural descriptions. This work successfully explored the feasibility of using multi-modality data with a multi-task learning framework. Extensive results on the ablation studies verified the effectiveness of the proposed framework.\n\nWEAKNESSES:\nThe majority of technical components of this work are borrowed from existing works, e.g., multitask learning, embedding space representation (latent space), textural space and etc. The technical contributions that inspire the following research are quite limited.\n\nQUESTIONS:\n1) The authors suggest using a textual template, where the critical description such as real/fake are instantiated as needed. It is no doubt about the effectiveness of this treatment. However, does it really necessary to handle the textural description using templates? What if directly using the attributes (e.g., real or fake) rather than a textural template (one can see that the differences between the textural descriptions on real images and fake images are only the keyword “real” and “fake”.)\n2) For Section 4.3 Robustness Analysis, the considered distortion of perturbations is necessary. However, more distortions such as image/video compression (JPEG or HEVC) are missing. In practical scenarios, the forged images are often communicated via online social networks, which are typically applied compression.\n3) The authors stated that the proposed method can only be applied to the scenarios where the forged faces are generated with blending operations. What if the blended image is further processed with some other harmonization techniques (e.g., with illumination correction on the faces or the edges between the beguine and fake regions)\n4) The adopted FF++ dataset contains video data for training, validation, and testing. As a well-known fact, one critical information of the faked video is temporal information. However, it seems that the authors neglect the temporal information directly.\n\nLIMITATIONS:\nThe authors clearly stated the limitation of this work. The proposed method cannot be applied to totally generated AI-generated images such as GAN-generated or diffusion-based model generated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a joint embedding approach for multitask learning in face forgery detection. The method defines a set of coarse-to-fine face forgery detection tasks based on face attributes at different semantic levels, and describes the ground truth for each task via a textual template. CLIP is used to implement the joint embedding architecture, and multi-level fidelity losses are used for multitask learning. The proposed method outperforms state-of-the-art detectors in terms of generalization ability.\n\nSTRENGTHS:\n1. This paper proposes a joint-embedding-based multitask learning method for face forgery detection. It could probably be the first work to apply the language prompts on the task of face forgery detection.\n2. This paper defines a set of coarse-to-fine face forgery detection tasks based on face attributes at different semantic levels to facilitate the multitask learning.\n3. The proposed method achieves better performance than the SOTA schemes in terms of generalization ability.\n\nWEAKNESSES:\n1. The authors apply the existing technologies including CLIP and fidelity loss for joint-embedding-based multitask learning. The technical contribution is rather limited. \n2. It lacks of explanation of why the authors use CLIP for joint learning. For the same token, it also lacks of analysis regarding the use of fidelity loss for multi-task learning. \n3. The works for comparing the robustness do not include the SOTA schemes.\n\nQUESTIONS:\n1. In Table 1, the AUC of OST on DFDC is lower than that is reported in reference [11] (77.73% vs 83.30%), while the AUC of OST on CDF is exactly the same as that is reported in [11]. The authors may want to explain why. \n2. The authors may want to explain the purpose of reporting the performance of w/o Aug  in Table 2 and 3. \n3. In section 1, lines 71-72, “textural templates” should be “textual templates”.\n\nLIMITATIONS:\nPlease refer to weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}