Title: Tracing Hyperparameter Dependencies for Model Parsing via Learnable Graph Pooling Network
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Forensics

Forgery Attack Defense

Low-level Vision
Abstract: \textit{Model Parsing} defines the task of predicting hyperparameters of the generative model (GM), given a GM-generated image as the input. 
Since a diverse set of hyperparameters is jointly employed by the generative model, and dependencies often exist among them, it is crucial to learn these hyperparameter dependencies for improving the model parsing performance. 
To explore such important dependencies, we propose a novel model parsing method called Learnable Graph Pooling Network (LGPN), in which we formulate model parsing as a graph node classification problem, using graph nodes and edges to represent hyperparameters and their dependencies, respectively. 
Furthermore, LGPN incorporates a learnable pooling-unpooling mechanism tailored to model parsing, which adaptively learns hyperparameter dependencies of GMs used to generate the input image. 
Also, we introduce a Generation Trace Capturing Network (GTC) that can efficiently identify generation traces of input images, enhancing the understanding of generated images' provenances.
Empirically, we achieve state-of-the-art performance in model parsing and its extended applications, showing the superiority of the proposed LGPN.

Tracing Hyperparameter Dependencies for Model Parsing via Learnable Graph Pooling Network

Abstract

Model Parsing defines the task of predicting hyperparameters of the generative model (GM), given a GM-generated image as the input. Since a diverse set of hyperparameters is jointly employed by the generative model, and dependencies often exist among them, it is crucial to learn these hyperparameter dependencies for improving the model parsing performance. To explore such important dependencies, we propose a novel model parsing method called Learnable Graph Pooling Network (LGPN), in which we formulate model parsing as a graph node classification problem, using graph nodes and edges to represent hyperparameters and their dependencies, respectively. Furthermore, LGPN incorporates a learnable pooling-unpooling mechanism tailored to model parsing, which adaptively learns hyperparameter dependencies of GMs used to generate the input image. Also, we introduce a Generation Trace Capturing Network (GTC) that can efficiently identify generation traces of input images, enhancing the understanding of generated images' provenances. Empirically, we achieve state-of-the-art performance in model parsing and its extended applications, showing the superiority of the proposed LGPN. The source code is available at link.

1 Introduction

Generative Models (GMs) [22, 73, 11, 44, 33, 60, 32, 34, 50], e.g. , Generative Adversarial Networks (GANs), Variational Autoencoder (VAEs), and Diffusion Models (DMs), have gained significant attention, offering remarkable capabilities in generating visually compelling images. However, the proliferation of such Artificial Intelligence Generated Content (AIGC) can inadvertently propagate inaccurate or biased information. To mitigate such negative impact, various image forensics [52] methods have been proposed [62, 54, 20, 6, 24, 64, 31, 76, 56, 71, 1]. Alongside these defensive efforts, the recent work [2] defines a novel research topic called ' model parsing ', which predicts 37 GMhyperparameters using the generated image as the input, as detailed in Fig. 1a.

Model parsing requires analyzing GM hyperparameters and gaining insights into origins of generated images, which facilitate defenders to develop effective countermeasures. For example, one can reasonably determine if there exists coordinated attacks [2] - two images are generated from the same GM that is unseen during the training, using predicted hyperparameters from the model parsing algorithm. In light of this, the previous method [2] introduces a clustering-based approach that achieves effective model parsing performance. However, this approach neglects the learning of hyperparameter dependencies. For instance, an inherent dependency exists between GM's layer number and parameter number, as GM's layer number is positively proportional to its parameter number. A similar proportional relationship exists between the number of convolutional layers and convolutional filters. In contrast, the use of the L1 loss is negatively correlated with the use of the L2 loss, as GMs typically do not employ both losses as objective functions simultaneously. We believe the neglect of such dependencies might cause a suboptimized performance. Therefore, in

NF

4% DM

19%

AR

5%

GAN

57%

AA

5%

VAE

10%

L1

MSE

ReLu

tanh

Batch

Norm.

Param.

Num.

Layer

Num.

(a)

(b)

(c)

NF

4% DM

19%

AR

5%

GAN

57%

AA

5%

VAE

10%

L1

MSE

ReLu

tanh

Batch

Norm.

Param.

Num.

Layer

Num.

(a)

(b)

(c)

L2

L1

ReLU

Params. Num.

(0, 180M)

Params. Num.

(360M, 540M)

Params. Num.

(180M, 360M)

L1

L2

Params. Num.

(360M, 540M)

KL

divergence

Sigmoid

Batch

Norm

NF

4% DM

19%

AR

5%

GAN

57%

AA

5%

VAE

10%

L1

MSE

ReLu

tanh

Batch

Norm.

Param.

Num.

Layer

Num.

(a)

(b)

(c)

NF

4% DM

19%

AR

5%

GAN

57%

AA

5%

VAE

10%

L1

MSE

ReLu

tanh

Batch

Norm.

Param.

Num.

Layer

Num.

(a)

(b)

(c)

L2

L1

ReLU

Params. Num.

(0, 180M)

Params. Num.

(360M, 540M)

Params. Num.

(180M, 360M)

L1

L2

Params. Num.

(360M, 540M)

KL

divergence

Sigmoid

Batch

Norm

L

1

L

2

ReLu

Param. Num.

Hyperparameters

Generative

Models

Generated

Image

(a)

(b)

(c)

this work, we propose to use graph nodes and edges to explicitly represent hyperparameters and their dependencies, respectively, and then propose a model parsing algorithm that utilizes the effectiveness of Graph Convolution Network (GCN) [35, 61, 18, 51] to capture dependencies among graph nodes.

Specifically, we first use training samples in the RED dataset [2] to construct a directed graph (Fig. 1b). The directed graph, based on the label co-occurrence pattern, illustrates the fundamental correlation between different categories and helps prior GCN-based methods achieve remarkable performances in various applications [10, 7, 68, 46, 16, 58]. In this work, this directed graph is tailored to the model parsing - we define discrete-value and continuous-value graph nodes to represent hyperparameters, shown in Fig. 1c. Then, we use this graph to formulate model parsing as a graph node classification problem, in which the discrete-value graph node feature decides if a given hyperparameter is used in the given GM, and the continuous-value node feature decides which range the hyperparameter resides. This formulation helps obtain effective representations of hyperparameters and dependencies among them for the model parsing task, detailed in Sec. 3.1.

To this end, we propose a novel model parsing framework called Learnable Graph Pooling Network (LGPN), which contains a Generation Trace Capturing Network (GTC) and a GCN refinement block (Fig. 2). Our GTC differs from neural network backbones used in State-of-The-Art (SoTA) forgery detection methods [62, 74, 63, 4, 63], using down-sampling operations ( e.g. , pooling) gradually reduce the learned feature map resolution during the forward propagation. This down-sampling can cause the loss of already subtle generation traces left by GMs. Instead, our GTC leverages a high-resolution representation that largely preserves generation traces throughout the forward propagation (Sec. 3.2.1). Therefore, the learned image representation from the GTC deduces crucial information ( i.e. , generation trace) of used GMs and benefits model parsing. Subsequently, this representation is transformed into a set of graph node features, along with the pre-defined directed graph, which are fed to the GCN refinement block. The GCN refinement block contains trainable pooling layers that progressively convert the correlation graph into a series of coarsened graphs by merging original graph nodes into supernodes. Then, the graph convolution is conducted to aggregate node features at all levels of graphs, and trainable unpooling layers are employed to restore supernodes to their corresponding children nodes. This learnable pooling-unpooling mechanism helps LGPN generalize to parsing hyperparameters in unseen GMs and improves the GCN representation learning. In summary, our contributions are:

⋄ We innovatively formulate model parsing as a graph node classification problem, using a directed graph to help capture hyperparameter dependencies for better model parsing performance.

⋄ A learnable pooling-unpooling mechanism is introduced with GCN to enhance representation learning in model parsing and its generalization ability.

⋄ We propose a Generation Trace Capturing Network (GTC) that utilizes high-resolution representations to capture generation traces, facilitating a deeper understanding of the image's provenance.

GCN block

GCN block

L-Pooling

GCN block

L-Pooling

GCN block

Unpooling

GCN block

Unpooling

ℎ

%

ℎ

+

.

-

Feature

Extractor

GCN refinement

+

%

+

+

2

Model

Parsing

/

!

/

%

/

+

/′

+

/′

%

/′

!

0

!

0

%

0

!

/

0

%

/

Generation

Trace

Capturing

Network

!

!

!

"

!

!

#$!

…

"

#

%

"

%

#$!

…

%

!

!

GCN Refinement Block

Model

Parsing

⋄ Extensive empirical results demonstrate the SoTA performance of the proposed LGPN in model parsing and identifying coordinated attacks. Additionally, the GTC's effectiveness is validated through CNN-generated image detection and image attribution tasks.

2 Related Works

Model Parsing Previous model parsing works [59, 30, 17, 3, 47] require prior knowledge of machine learning models and their inputs to predict model hyperparameters, and such predictions are primarily limited to architecture-related hyperparameters [17, 47]. In contrast, Asnani et al. [2] recently propose a technique to estimate 37 pre-defined hyperparameters covering loss functions and architectures by only leveraging generated images as inputs. Specifically, this work [2] designs FEN-PN that uses a clustering-based approach to estimate the mean and standard deviation of hyperparameters for network architecture and loss function types. Then, FEN-PN uses a fingerprint estimation network that is trained with four constraints to estimate the fingerprint for each image. These fingerprints are employed to predict hyperparameters. However, FEN-PN overlooks dependencies among hyperparameters, which cannot be adequately captured by the estimated mean and standard deviation. In contrast, we propose LGPN, using GCN to model dependencies among different hyperparameters, improving overall model parsing performance.

GCN-based Method Graph Convolution Neural Network (GCN) shows effectiveness in encoding dependencies among different graph nodes [35, 61, 18, 26, 28], and in the computer vision community, directed graphs based on label co-occurrence patterns are used with GCN in tasks such as multi-label image recognition [10, 7, 68], semantic segmentation [9, 29, 41, 16], and person ReID and action localization [46, 65, 8, 58]. These works rely on original graph structures, whereas our method uses a pooling algorithm to modify this graph structure, which can benefit GCN's representation learning. Also, our GCN refinement block differs from Graph U-Net [19] in that it reduces the graph size by removing certain nodes. However, we formulate model parsing as a graph node classification task, where each node represents a specific hyperparameter, meaning no nodes should be discarded.

Learning Image Generation Traces GMs leave particular traces in their generated images [43, 12], which can be visual artifacts [72, 4] or through evident peaks in the frequency domain [74, 62]. These traces serve as important clues for image forensic tasks such as detection [12, 74, 62], attribution [70, 49] and model parsing [2, 66, 67, 21]. Current methods [62, 74, 63, 4, 63, 57] use backbones like ResNet and XceptionNet that have high generalization abilities, and vision-language foundation models [55, 75, 23] also show effectiveness in detecting unseen forgeries. However, these prior methods often use backbones that rely on downsampling operations, such as convolutions with large strides and pooling, which capture global semantics but discard high-frequency details that contain critical generation traces. In contrast, we propose a Generation Trace Capturing network that mainly operates on a high-resolution representation for capturing such generation traces. Its effectiveness is shown in our experiment by comparing against recent detection methods that use pre-trained CLIP features [48] and the novel representation [63], and competitive image attribution methods.

Conv

(3,3)

ResNet

Block #1

Conv

(5,5)

Conv

(7,7)

Fusion Layer

ResNet

Block #2

ResNet

Block #3

ResNet

Block #4

⊕

Φ

!

⊕

Φ

"

⊕

Φ

#

⊕

Φ

$

Fusion Layer

I

ResNet branch

High-res branch

𝐅

𝐡

𝐟

3 Method

In this section, we first revisit some fundamental preliminaries in Sec. 3.1, and then introduce the proposed Learnabled Graph Pooling Network (LGPN) in Sec. 3.2. Lastly, we describe training and inference procedures in Sec. 3.3.

3.1 Preliminaries

This section provides the problem statement of model parsing and its formulation as a graph node classification task with the Graph Convolutional Network (GCN).

Revisiting Model Parsing Given an input image I ∈ R 3 × W × H generated by a GM ( i.e. , G ), the model parsing algorithm generates three vectors ( y d ∈ R 18 , y c ∈ R 9 , and y l ∈ R 10 ) as predictions of G different hyperparameters used in the G . Specifically, defined by the previous work [2], these predictable G hyperparameters include discrete and continuous architecture hyperparameters, as well as loss functions, which are denoted as y d , y c , and y l , respectively. For y d and y l , each element is a binary value representing if the corresponding feature is used or not. Each element of y c is the value of certain continuous architecture hyperparameters, such as the layer number and parameter number. As these hyperparameters are in different ranges, we normalize them into [ 0 , 1 ], same as the previous work [2]. Predicting y d and y l is a classification task while the regression is used for y c . Detailed definitions are in Tab. 4, 5, and 6 of the Supplementary. We augment the previous model parsing dataset ( e.g. , RED 116 [2]) with different diffusion models such as DDPM [27], ADM [15] and Stable Diffusions [50], increasing the spectrum of GMs. Also, we add real images on which these GMs are trained into the dataset. In the end, we collect 140 GMs in total and denote such a collection as the RED 140 dataset. Details are in the Supplementary Sec. D.

Correlation Graph Construction We design a graph structure of model parsing, where graph nodes and edges represent hyperparameters and their dependencies, respectively. We first use conditional probability P ( L j | L i ) to denote the probability of hyperparameter L j occurrence when hyperparameter L i appears. We count the occurrence of such pairs in the RED 140 to construct the matrix G ∈ R C × C , where C and G ij denotes the number of graph nodes and the conditional probability of P ( L j | L i ) , respectively. Next, we apply a fixed threshold τ to remove edges with low correlations in G and then obtain a directed graph, denoted as A ∈ R C × C , where A ij indicates if there exists an edge between node i and j .

Specifically, as depicted in Fig. 1c, each discrete hyperparameter ( e.g. , discrete architecture hyperparameters and loss functions) is represented by one graph node of A , denoted as a discrete-value graph node. For continuous hyperparameters, we first divide its value range into n different intervals, and each interval is represented by one graph node of A denoted as a continuous-value graph node. In other words, C is larger than G since each continuous hyperparameter is represented by n continuous-value graph nodes. Subsequently, we use discrete-value graph nodes to decide if given hyperparameters are present, and the continuous-value node decides which range the hyperparameter resides. Therefore, we denote the constructed graph as A and apply stacked graph convolution on A as follows:

where h l i represents the i -th node feature in graph A . W l and b l are weight and bias terms.

3.2 Learnable Graph Pooling Network

In this section, we detail the Generation Trace Capturing Network (GTC) and GCN refinement block-two major components used in the proposed LGPN, as depicted in Fig. 2.

3.2.1 Generation Trace Capturing Network

As shown in Fig. 3, GTC uses one branch ( i.e. , ResNet branch) to propagate the original image information. Meanwhile, the other branch, denoted as high-res branch , harnesses the high-resolution representation that helps detect high-frequency generation artifacts stemming from various GMs. More formally, three separate 2 D convolution layers with different kernel sizes ( e.g. , 3 × 3 , 5 × 5 and 7 × 7 ) are utilized to extract feature maps of I . We concatenate these feature maps and feed them to the fusion layer - the 1 × 1 convolution for the channel dimension reduction. Then, we obtain the feature map F h ∈ R D × W × H , with the same resolution as I . After that, we proceed F h to a dual-branch backbone. Specifically, we upsample intermediate features output from each ResNet block and incorporate them into the high-res branch, as depicted in Fig. 3. The high-res branch also has four different convolution blocks ( e.g. , Φ b with b ∈ { 1 . . . 4 } ), which do not employ down-sampling operations, such as the 2 D convolution with large strides or pooling layers. Then, intermediate feature maps throughout the high-res branch possess the same resolution as F h .

The ResNet branch and high-res branch output feature maps are concatenated and then passed through an AVGPOOL layer. Then, we obtain the final learned representation, f ∈ R D , that captures generation artifacts of the input image I . Subsuqently, we learn C independent linear layers, i.e., Θ = { θ C -1 i =0 } to transform f into a set of graph node features H = { h 0 , h 1 , ..., h ( C -1) } , where H ∈ R C × D and h i ∈ R 1 × D ( i ∈ { 0 , 1 , ..., C -1 } ). We use H to denote graph node features of the directed graph ( i.e., graph topology) A ∈ R C × C .

3.2.2 GCN Refinement Block

The GCN refinement block has a learnable pooling-unpooling mechanism that progressively coarsens the original graph A 0 into a series of coarsened graphs, i.e. , A 1 , A 2 ... A n , and graph convolution is conducted on graphs at all different levels. Specifically, such a pooling operation is achieved by merging graph nodes, namely, via a learned matching matrix M . Also, correlation matrices of different graphs, denoted as A l 2 , which are learned using MLP layers, which are also influenced by the GM responsible for generating the input image. This further emphasizes the significant impact of GMon the correlation graph generation process.

Learnable Graph Pooling Layer First, A l ∈ R m × m and A l +1 ∈ R n × n denote directed graphs at l th and l +1 th layers, with m and n ( m ≥ n ) graph nodes, respectively. An assignment matrix M l ∈ R m × n converts A l to A l +1 as:

Also, we use H l ∈ R m × D and H l +1 ∈ R n × D to denote graph node features of A l and A l +1 , respectively. Therefore, we can use M l to perform the graph node aggregation operation via:

For simplicity, we use f GCN to denote the mapping function imposed by a GCN block that has multiple GCN layers. Assuming the q -th and l -th graph layer are the first and last layer of the GCN block, then we have:

Ideally, the learnable pooling operation should be dependent on the learned representation of the input image, and such representation is converted into H l . Therefore, we employ a trainable weight W m to transform H l into the assignment matrix M l :

where α is set as 1 e 9 . Values of the resultant M l are approximately equal to 0 or 1 . It is worth noting that prior works [39, 40] also adopt techniques similar to Eq. 5 for making the thresholding operation differentiable.

1

!

!

"

Used or Not

Used or Not

Params. Num.

Value

!

!

"

A

B

C

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

!

!"

#

=

(!

!

"

)

#

(a)

Presence v.s. Not

Presence v.s. Not

concat

Params. Num.

Value

FC layers

MSE

ReLU

Params. Num.

Params. Num.

Params. Num.

(0, 180M)

(180M, 360M)

(360M, 540M)

(b)

L1

MSE

Sig.

Relu

BN

LN

L1

Batch

Norm

Sigmoid

MSE

B

Layer Norm

Relu

Learnable Unpooling Layer We perform the graph unpooling operation that progressively restores pooled graphs to the graph at the original resolution for the graph node classification task. As shown in Fig. 2, to avoid confusion, we use H and V to represent the graph node feature on the pooling and unpooling branches, respectively. The correlation matrix on the unpooling branch is denoted by A ′ ,

where A ′ l and A ′ l -1 are the l th and l -1 th layers in the unpooling branch, respectively. Finally, we use the refined feature V for model parsing.

Discussion This learnable pooling-unpooling mechanism offers three distinct advantages. First, each supernode in the coarsened graph serves as the combination of features from its children nodes, and graph convolutions on supernodes have a large receptive field for aggregating features. Secondly, the learnable correlation models hyperparameter dependencies dynamically based on generation artifacts of input image features ( e.g. , f ). Lastly, learned correlation graphs A vary across different levels, helping address the over-smoothing issue commonly encountered in GCN learning [38, 45, 5].

3.3 Training and Inference

We jointly train our method with three objective functions. Graph node classification loss ( e.g. , L graph ) encourages each graph node feature to predict the corresponding hyperparameter label. Artifacts isolation loss ( e.g. , L iso ) helps the LGPN only parse the hyperparameters for generated images, and hyperparameter hierarchy constraints ( e.g. , L hier ) imposes hierarchical constraints among different hyperparameters while stabilizing the training.

Training Samples We denote a training sample as { I , y } , in which y = { y d , y c , y l } = { y 0 , y 1 , ..., y ( C -1) } is annotations of C graph nodes for G parsed hyperparameters as introduced in Sec. 3.1. Specifically, y c is assigned as 1 if the sample has c -th hyperparameter and 0 otherwise, where c ∈ { 0 , 1 , ..., C -1 } . Details are in Supplementary Sec. B.

Graph Node Classification Loss Given the image I , we convert the refined feature V into the predicted score vector, denoted as s = { s 0 , s 1 , ..., s ( C -1) } . We employ the sigmoid activation to retrieve the probability vector p = { p 0 , p 1 , ..., p ( C -1) } , namely, p c = SIGMOID ( s c ) . Then, we have:

Hyperparameter Hierarchy Prediction Fig. 4 shows that different hyperparameters can be grouped, so we define the hyperparameter hierarchy assignment M s to reflect this inherent nature. More details of such the assignment are in Supplementary Sec. A. Suppose, at the l th layer, we minimize the L 2 norm of the difference between the predicted matching matrix M l and M s l ,

Artifacts Isolation Loss We denote the image-level binary label as y img and use p img as the probability that I is a generated image. Then we have:

In summary, our joint training loss function can be written as L all = λ 1 L graph + λ 2 L hier + λ 3 L iso , where λ 1 and λ 2 equal 0 when I is real.

Inference As Fig. 4b, we use the discrete-value graph node feature to perform the binary classification to decide the presence of given hyperparameters. For the continuous architecture hyperparameter, we first concatenate n corresponding node feature and train a linear layer to regress it to the estimated value. Empirically, we set n as 3 and show this concatenated feature improves the robustness in predicting the continuous value (see Supplementary Tab. 7).

4 Experiment

4.1 Model Parsing

Setup Our experiment utilizes RED 140 dataset. In RED 140 , each GM contains 1 , 000 images, resulting in a total of 140 , 000 generated images that encompass a wide range of semantics, including objects, handwritten digits, and human faces. Also, RED 140 has real images on which these GMs are trained, such as CelebA [42], MNIST [14], CIFAR10 [36], ImageNet [13], facades [77], edges2shoes [77] and, apple2oranges [77]. We follow the protocol of [2]: we divide samples into 4 disjoint sets, each of which comprises different GM categories such as GAN, VAE, DM, etc . Next, we do leave-one-out testing, i.e. , train on 125 GMs from three sets, and test on GMs of the remaining set. The performance is averaged across four test sets, measured by F 1 score and accuracy for discrete hyperparameters (loss function and discrete architecture hyperparameters) and L 1 error for continuous architecture hyperparameters. Implementation details are in Supplementary Sec. B.

Main Performance We report model parsing performance in Tab. 1, where our proposed LGPN (line #6 ) largely outperforms previous model parsing algorithms. We first employ commonly used backbones to set up competitive model parsing baselines for a more comprehensive comparison. More formally, four baselines in lines #1 -4 that use ResNet-50, ViT-B, HR-Net, and GTC as backbones with 2 layers MLP as the model parsing head, respectively. Baseline 4 (line #4 ) achieves the best performance, which indicates that GTC is the most suitable backbone for model parsing. Specifically, Baseline 4 has 1 . 8% and 3 . 5% higher F1 score over Baseline 2 that uses HR-Net on predicting hyperparameters of loss functions and discrete architecture hyperparameters. After that, we report FEN-PN's performance, which already proves the effectiveness on the model parsing task since it has specific model parsing architectures containing a fingerprint estimate network (FEN) and a parsing network that predicts hyperparameters. Surprisingly, although FEN-PN has achieved competitive results on RED 116 , it only has comparable performance to Baseline 2 . This indicates that FEN-PN reduces its effectiveness in predicting hyperparameters of diffusion models, as RED 140

Method

Loss

Function

Dis. Archi.

Para.

Backbone

MP Head

F1

↑

Acc.

↑

F1

↑

Acc.

↑

1

GTC

MLP

82

.

5

80

.

9

75

.

7

70

.

9

2

Stacked GCN

83

.

2

81

.

3

76

.

9

73

.

8

3

Att-GCN [61]

83

.

4

82

.

7

78

.

0

74

.

5

4

82

.

2

82

.

0

74

.

8

70

.

2

5

GCN refinement

84

.

6

83

.

3

79

.

5

77

.

5

(a)

FC

Num

Param.

Num

Pooling

Num

Conv.

Num

Norm

Num

Layers.

Num

Filters

Num

0.08

0.10

0.12

0.14

0.16

0.18

L1 Error

.077

.081

.091

.137

.138

.147

.149

.093

.148

.111

.151

.155

.149

.161

GTC+MLP

(b)

1.0

0

1.0

0

(a)

Training Objectives

F1 score

↑

L

graph

L

iso

L

hier

Loss

Fun.

Dis. Archi.

Para.

✔

83

.

7

77

.

0

✔

✔

84

.

0

78

.

1

✔

✔

83

.

9

79

.

0

✔

✔

✔

84

.

6

79

.

5

(b)

Method

Loss

Function

Dis. Archi.

Para.

Para.

F1 score

↑

L1 error

↓

FEN-PN [2]

81

.

3

71

.

8

0

.

149

GTC

w

MLP

77

.

8

68

.

9

0

.

169

GTC

w

S-GCN

79

.

0

69

.

8

0

.

145

LGPN

84

.

1

74

.

3

0

.

130

(c)

contains more images from diffusion models than RED 116 . The complete performance comparison on RED 116 is reported in Tab. 5c. Lastly, we replace MLP with the GCN refinement block, which is the full model of LGPN (line #6 ) and performs better on all metrics, demonstrating that the GCN refinement block indeed refines graph node features and makes a more effective model parsing head than MLP layers.

Hyperparameter Dependency Capturing Tab. 2a reports model parsing performance using different GCN variants as the model parsing head. Overall, the proposed GCN refinement block, which refines graph node features to better capture hyperparameter dependencies, helps achieve the best model parsing performance. By comparing lines #1 and #2 , we conclude that replacing MLP layers with the GCN refinement benefits the model parsing task. After all, GCN leverages structural information of the pre-defined graph, improving the learning of correlations among different hyperparameters. Next, we use Graph attention networks [61] (Att-GCN) at line #3 , which employs the attention mechanism to update the graph node feature. As a result, Att-GCN achieves 1 . 1% higher F1 than stacked GCN ( e.g. , line #2 ) on predicting discrete architecture parameters. Lastly, line #4 uses Graph U-Net [19], which has a similar pooling-unpooling mechanism to our GCN refinement block, but its pooling operation discards graph nodes in the previous layer for forming a smaller graph. Using the Graph U-Net as the model parsing head produces the worse performance on discrete architecture hyperparameters 4 . 3% lower than Att-GCN ( #3 ). We believe this is because dropping graph nodes is not optimal for model parsing, in which all graph nodes represent corresponding hyperparameters and are important for the final performance. In contrast, LGPN merges children nodes into the supernode, so all node information in the previous layer remains in the smaller pooled graph, helping achieve the best performance on discrete hyperparameters in Tab. 2a. On the other hand, continuous hyperparameter prediction can also benefit from the GCN refinement block. In Tab. 2b, LGPN shows L1 errors of 0 . 147 and 0 . 081 on Layers Num. and Param. Num. respectively, whereas the model with MLP layers only achieves 0 . 149 and 0 . 148 , respectively. This is because the GCN refinement learns the dependency between Param. Num. and Layer Num., which aligns with the observation that models with more layers typically have more parameters. Therefore, modeling such dependencies ultimately decreases the L1 error in Param. Num. prediction. Likewise, when predicting Conv. Layer Num. and Filters Num., the GCN refinement achieves 0 . 137 and 0 . 149 L1 error, whereas GTC with MLP layers have 0 . 151 and 0 . 161 L1 error.

GM-dependent Graph Fig. 5a shows that learned correlation graphs ( A ′ 0 ) from image pairs exhibit significant similarity when both images belong to the same unseen GM. This result demonstrates that our correlation graph largely depends on the GM instead of image contents, given that we have

Method

Acc.

AUC

Pd@

5

%

HiFi-Net [24]

72

.

3

75

.

4

30

.

4

FEN-PN [2]

83

.

0

92

.

4

61

.

2

GTC

w

MLP

83

.

9

92

.

2

62

.

5

GTC

w

S-GCN

84

.

3

94

.

2

68

.

6

LGPN

85

.

9

95

.

7

77

.

2

(a)

Same Seen

Diff. Seen

Same Unseen

Diff. Unseen   Seen & Unseen

50

60

70

80

90

84

82.3

63.4

52.2

50.8

82

80

70.9

71.4

71.7

92.3

85.1

75.4

81.6

83.7

HiFi-Net [CVPR2023]

FEN-PN [PAMI2022]

LGPN [Ours]

[24]

[2]

(b)

100

different contents ( e.g. , human face and objects) in unseen GMs from each test. In addition, we empirically observe our method remains robust when using different thresholds to construct the graph, which is detailed in the Supplementary Tab. 9. However, the performance declines more when the threshold increases to 0 . 65 , which causes the correlation graph to have very sparse connectivities, hindering the learning of hyperparameter dependencies.

Objective Functions Analysis Fig. 5b shows the ablation of different training objective functions introduced in Sec. 3.3. Line #1 only optimizes the LGPN with L grpah , producing results comparable to simply stacking GCN with the attention mechanism ( e.g. , line #3 in Tab. 2a). Lines #2 and #3 show contributions from L iso and L hier , which improve the performance by 1 . 1% and 2 . 0% than only using L grpah , on predicting discrete architecture hyperparameters, respectively. This is because L iso and L hier make the LGPN concentrate on learning generation traces from generated images and impose the hierarchical constraints, respectively.

RED116 Results Fig. 5c reports that LGPN achieves the best performance on all metrics in RED116 dataset. Interestingly, LGPN obtains 79 . 5% F1 score on predicting discrete architecture hyperparameters in RED140 (Tab. 1), whereas only 74 . 3% in RED116, which does not contain diffusion model generated images. We believe this is because all diffusion models share similar architectures, and such similarities make the prediction of their architecture hyperparameters easier.

4.2 Coordinate Attack Detection

We evaluate the proposed LGPN on coordinated attacks detection [2], which aims to classify whether two fake images are generated from the same GM or not. This is achieved by computing the cosine similarity between predicted hyperparameters from given images. Specifically, we evaluate coordinated attack detection on RED 140 in a 4 -fold cross-validation. In each fold, the train set has 125 GMs, and the test set has 30 GMs where 15 GMs are exclusive (unseen) from the train set and 15 GMs are seen in the train set. We generate 89 , 000 training image pairs from train-set GMs for training 1 , 000 image pairs for validation. We generate 25 , 000 test image pairs from test-set GMs, and the average of the 4 folds is used as the final result. For the measurement, we use accuracy, AUC, and detection probability at a fixed false alarm rate (Pd@FAR) e.g. , Pd@ 5% as metrics. Specifically, aside from FEN-PN [2], we also compare with the recent work HiFi-Net [24] that show SoTA results in attributing different forgery methods. Specifically, we train HiFi-Net to classify 125 GMs and take learned features from the last fully-connected layer for coordinated attack detection. The performance is reported in Tab. 3a, which demonstrates that our proposed method surpasses both prior works. We observe the HiFi-Net performs much worse on AUC than the model parsing baseline ( e.g. , GTC w S-GCN) and FEN-PN. Furthermore, Tab. 3b shows the FEN-PN and LGPN perform comparably as HiFi-Net on seen GMs (first two bar charts), yet much better than HiFi-Net when images are generated by unseen GMs (last three bar charts). Lastly, LGPN has a better Pd@ 5% performance than other methods.

4.3 Capturing Generation Traces

To study GTC's ability to identify generation traces, we adopt it for CNN-generated image detection and image attribution. For a fair comparison, no model parsing dataset is used for the pre-training.

CNN-generated Image Detection We append fully-connected layers at the end of GTC to obtain a binary detector that distinguishes CNN-generated images from real ones (detailed in supplementary

Average Precision (AP)

[62]

[24]

[53]

[48]

[57]

(a)

Method

CelebA

LSUN

Metric:

Acc(

%

)

PRNU [43]

86

.

61

67

.

84

Attr. [70]

99

.

43

98

.

58

FEN-PN [2]

99

.

66

99

.

84

Ours

99

.

79

99

.

73

(b)

Method

CelebA

LSUN

Metric:

Acc(

%

)

PRNU [43]

74

.

23

67

.

92

Attr. [70]

84

.

16

81

.

63

FEN-PN [2]

81

.

09

78

.

28

Ours

88

.

11

86

.

23

(c)

Fig. 8). We follow the experiment setup from prior works [62, 48, 57], which trains the model on images generated by ProGAN [32], and test it on images generated by 11 unseen forgery methods, using average precision (AP) and accuracy for the measurement. Fig. 6a reports that our method achieves premium detection performance compared to prior methods. The second-best method, NPR [57], focuses on learning local up-sampling artifacts from pixels, helping detect images from unseen GMs. Instead, GTC's high-resolution representation more effectively exploits both local and global traces left from generation processes, obtaining a better performance.

Image Attribution Tab. 6b and Tab. 6c report the image attribution performance in two different protocols. Specifically, we define the protocol 1 based on the previous work [2], which trains methods on 100 , 000 real and 100 , 000 images generated from four different GMs ( e.g. , SNGAN, MMDGAN, CRAMERGAN, and ProGAN), conducting a five-way classification ( i.e. , 4 GMs and real). In protocol 2 , we add two more generative methods ( e.g. , styleGANv2, styleGANv3), resulting in a more challenging task: a 7 -way classification task, classifying whether the image is real samples or generated by which one of 6 GMs. As depicted in Supplementary Fig. 8, we apply fully-connected layers on the top of GTC, which leverages the final representation of the generation trace for the multi-category classification task, e.g. , image attribution. Our proposed method achieves the best image attribution performance on CelebA and competitive results on LSUN, indicating that GTC has a promising ability to capture the generation trace.

5 Conclusion

In this study, our focus is model parsing , which predicts pre-defined hyperparameters of a GM given an input image. We propose a novel method that incorporates a learnable pooling-unpooling mechanism devised for the model parsing task. This mechanism serves multiple purposes: modeling GM-dependent hyperparameter dependencies, expanding the receptive field of graph convolution, and mitigating the over-smoothing issue in GCN learning. In addition, we provide the Generation Trace Capturing network to capture generation artifacts, which proves effective in two different image forensic applications: CNN-generated image detection and coordinated attacks detection.

Limitation We empirically observe two limitations in our proposed method, both of which can be interesting directions for future research. First, while our model parsing approach delivers excellent performance on the RED 140 dataset, it is worth exploring its effectiveness on specific GMs that fall outside our dataset scope. This investigation would provide valuable insights into the generalizability of our method to a broader range of GMs. Secondly, we formulate the model parsing task as a closed-set classification problem, which limits its ability to predict undefined hyperparameters, e.g. , LeakyReLU. One interesting solution could be adding a few new graph nodes for undefined hyperparameters while keeping original graph nodes - the learned dependency between graph nodes representing ReLU and LeakyReLU should be high.

Broader Impact We strongly advocate for the machine learning and computer vision community to actively work towards mitigating potential negative societal implications of research. It is possible that generated face images used in training could leak the identity information of subjects who have not provided consent forms. We shall strive to work on real face imagery whose collection is reviewed by an Institutional Review Board (IRB).

Ackonwledge This work was supported by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR00112090131 to Xiaoming Liu at Michigan State University.

References

Vishal Asnani, Xi Yin, Tal Hassner, and Xiaoming Liu. Malp: Manipulation localization using a proactive scheme. In CVPR , 2023.

Vishal Asnani, Xi Yin, Tal Hassner, and Xiaoming Liu. Reverse engineering of generative models: Inferring model hyperparameters from generated images. PAMI , 2023.

Lejla Batina, Shivam Bhasin, Dirmanto Jap, and Stjepan Picek. CSI NN: Reverse engineering of neural network architectures through electromagnetic side channel. In USENIXSS , 2019.

Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? understanding properties that generalize. In ECCV , 2020.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In AAAI , 2020.

Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection. In CVPR , 2022.

Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin. Learning semantic-specific graph representation for multi-label image recognition. In ICCV , 2019.

Ying Chen, Shixiong Xia, Jiaqi Zhao, Yong Zhou, Qiang Niu, Rui Yao, Dongjun Zhu, and Dongjingdian Liu. Rest-reid: Transformer block-based residual learning for person re-identification. Pattern Recognition Letters , 2022.

Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalantidis. Graph-based global reasoning networks. In CVPR , 2019.

Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In CVPR , 2019.

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR , 2018.

Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva. On the detection of synthetic images generated by diffusion models. In ICASSP , 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE signal processing magazine , 2012.

Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. In NeurIPS , 2021.

Henghui Ding, Hui Zhang, Jun Liu, Jiaxin Li, Zijian Feng, and Xudong Jiang. Interaction via bi-directional graph of semantic region affinity for scene parsing. In ICCV , 2021.

Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of functions. In AISTATS , 2021.

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In WWW , 2019.

Hongyang Gao and Shuiwang Ji. Graph u-nets. In ICML , 2019.

Candice R Gerstner and Hany Farid. Detecting real-time deep-fake videos using active illumination. In CVPR , 2022.

Yifan Gong, Yuguang Yao, Yize Li, Yimeng Zhang, Xiaoming Liu, Xue Lin, and Sijia Liu. Reverse engineering of imperceptible adversarial image perturbations. In ICLR , 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.

Xiao Guo, Xiaohong Liu, Iacopo Masi, and Xiaoming Liu. Language-guided hierarchical fine-grained image forgery detection and localization. IJCV , 2024.

Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Iacopo Masi, and Xiaoming Liu. Hierarchical fine-grained image forgery detection and localization. In CVPR , 2023.

Xiao Guo, Yaojie Liu, Anil Jain, and Xiaoming Liu. Multi-domain learning for updating face anti-spoofing models. In ECCV , 2022.

Zhijiang Guo, Yan Zhang, and Wei Lu. Attention guided graph convolutional networks for relation extraction. In ACL , 2019.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

I Hsu, Xiao Guo, Premkumar Natarajan, and Nanyun Peng. Discourse-level relation extraction via graph pooling. In AAAI DLG workshop , 2021.

Hanzhe Hu, Deyi Ji, Weihao Gan, Shuai Bai, Wei Wu, and Junjie Yan. Class-wise dynamic graph convolution for semantic segmentation. In ECCV , 2020.

Weizhe Hua, Zhiru Zhang, and G Edward Suh. Reverse engineering convolutional neural networks through side-channel information leaks. In DAC , 2018.

Kaixiang Ji, Feng Chen, Xin Guo, Yadong Xu, Jian Wang, and Jingdong Chen. Uncertainty-guided learning for improving image manipulation detection. In ICCV , 2023.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR , 2018.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR , 2019.

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. DiffusionCLIP: Text-guided diffusion models for robust image manipulation. In CVPR , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998.

Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgcns: Can gcns go as deep as cnns? In ICCV , 2019.

Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In CVPR , 2018.

Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Guided attention inference network. PAMI , 2019.

Xia Li, Yibo Yang, Qijie Zhao, Tiancheng Shen, Zhouchen Lin, and Hong Liu. Spatial pyramid based graph reasoning for semantic segmentation. In CVPR , 2020.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV , 2015.

Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. Do GANs leave artificial fingerprints? In MIPR , 2019.

Safa C. Medin, Bernhard Egger, Anoop Cherian, Ye Wang, Joshua B. Tenenbaum, Xiaoming Liu, and Tim K. Marks. MOST-GAN: 3D morphable StyleGAN for disentangled face image manipulation. In AAAI , 2022.

Yimeng Min, Frederik Wenkel, and Guy Wolf. Scattering GCN: Overcoming oversmoothness in graph convolutional networks. In NeurIPS , 2020.

Binh X Nguyen, Binh D Nguyen, Tuong Do, Erman Tjiputra, Quang D Tran, and Anh Nguyen. Graphbased person signature for person re-identifications. In CVPR , 2021.

Seong Joon Oh, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural networks. Explainable AI: interpreting, explaining and visualizing deep learning , pages 121-144, 2019.

Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. In CVPR , 2023.

Yongyang Pan, Xiaohong Liu, Siqi Luo, Yi Xin, Xiao Guo, Xiaoming Liu, Xiongkuo Min, and Guangtao Zhai. Towards effective user attribution for latent diffusion models via watermark-informed blending. arXiv preprint arXiv:2409.10958 , 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In ESWC , 2018.

Husrev Taha Sencar, Luisa Verdoliva, and Nasir Memon. Multimedia Forensics . Springer Nature, 2022.

Zeyang Sha, Zheng Li, Ning Yu, and Yang Zhang. De-fake: Detection and attribution of fake images generated by text-to-image generation models. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , pages 3418-3432, 2023.

Kaede Shiohara and Toshihiko Yamasaki. Detecting deepfakes with self-blended images. In CVPR , 2022.

Xiufeng Song, Xiao Guo, Jiache Zhang, Qirui Li, Lei Bai, Xiaoming Liu, Guangtao Zhai, and Xiaohong Liu. On learning multi-modal forgery representation for diffusion generated video detection. In NeurIPS , 2024.

Zhihao Sun, Haoran Jiang, Danding Wang, Xirong Li, and Juan Cao. Safl-net: Semantic-agnostic feature learning network with auxiliary plugins for image manipulation detection. In ICCV , 2023.

Chuangchuang Tan, Huan Liu, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Rethinking the up-sampling operations in cnn-based generative network for generalizable deepfake detection. In CVPR , 2024.

Praveen Tirupattur, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. Modeling multi-label action dependencies for temporal action localization. In CVPR , 2021.

Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction APIs. In USENIXSS , 2016.

Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning GAN for pose-invariant face recognition. In CVPR , 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR , 2017.

Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images are surprisingly easy to spot... for now. In CVPR , 2020.

Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. Dire for diffusion-generated image detection. In ICCV , 2023.

Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features. In CVPR , 2019.

Yingmao Yao, Xiaoyan Jiang, Hamido Fujita, and Zhijun Fang. A sparse graph wavelet convolution neural network for video-based person re-identification. Pattern Recognition , 2022.

Yuguang Yao, Xiao Guo, Vishal Asnani, Yifan Gong, Jiancheng Liu, Xue Lin, Xiaoming Liu, Sijia Liu, et al. Reverse engineering of deceptions on machine-and human-centric attacks. Foundations and Trends® in Privacy and Security , 6(2):53-152, 2024.

Yuguang Yao, Jiancheng Liu, Yifan Gong, Xiaoming Liu, Yanzhi Wang, Xue Lin, and Sijia Liu. Can adversarial examples be parsed to reveal victim model information? In WACV , 2024.

Jin Ye, Junjun He, Xiaojiang Peng, Wenhao Wu, and Yu Qiao. Attention-driven dynamic graph convolutional network for multi-label image recognition. In ECCV , 2020.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Ning Yu, Larry S Davis, and Mario Fritz. Attributing fake images to GANs: Learning and analyzing GAN fingerprints. In ICCV , 2019.

Yuanhao Zhai, Tianyu Luan, David Doermann, and Junsong Yuan. Towards generic image manipulation detection with weakly-supervised self-consistency learning. In ICCV , 2023.

Lingzhi Zhang, Zhengjie Xu, Connelly Barnes, Yuqian Zhou, Qing Liu, He Zhang, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, and Jianbo Shi. Perceptual artifacts localization for image synthesis tasks. In ICCV , 2023.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , 2023.

Xu Zhang, Svebor Karaman, and Shih-Fu Chang. Detecting and simulating artifacts in gan fake images. In WIFS , 2019.

Yue Zhang, Ben Colman, Ali Shahriyari, and Gaurav Bharaj. Common sense reasoning for deep fake detection. In ECCV , 2024.

Jizhe Zhou, Xiaochen Ma, Xia Du, Ahmed Y Alhammadi, and Wentao Feng. Pre-training-free image manipulation localization through non-mutually exclusive contrastive learning. In ICCV , 2023.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV , 2017.

In this supplementary, we provide:

⋄ Predictable hyperparameters introduction.

⋄ Training and implementation details.

⋄ Additional results of model parsing and CNN-generated image detection.

⋄ The construction of RED 140 dataset.

⋄ Hyperparameter ground truth and the model parsing performance for each GM

A Predictable Hyperparameters Introduction

We investigate 37 hyperparameters that exhibit the predictability according to Asnani et al. [2]. These hyperparameters are categorized into three groups: (1) Loss Function (Tab. 4), (2) Discrete Architecture Hyperparameters (Tab. 5), (3) Continuous Architecture Hyperparameters (Tab. 6). We report our proposed method performance of parsing hyperparameters in these three groups via Fig. 7a, Fig. 7b, and Fig. 7c, respectively. Moreover, in the main paper's Eq. 8 and Fig. 4a, we employ the assignment hierarchy M s to group different hyperparameters together, which supervises the learning of the matching matrix M . The construction of such the M s is also based on Tab. 4, 5, and 6, which not only define three coarse-level categories, but also fine-grained categories such as pixel-level objective (loss) function ( e.g. , L1 and MSE ) in Tab. 4, and normalization methods ( e.g. , ReLu and Tanh ) as well as nonlinearity functions ( e.g. , Layer Norm. and Batch Norm. ) in Tab. 5.

B Training and Implementation Details

Training Details Given the directed graph A ∈ R C × C , which contains C graph nodes. We empirically set C as 55 , as mentioned in the main paper Sec. 3.3. In the training, LGPN takes the given image I and output the refined feature V ∈ R 55 × D , which contains learned features for each graph node, namely, V = { v 0 , v 1 , ..., v 54 } . As a matter of fact, we can view V as three separate sections: V l = { v 0 , v 1 , ..., v 9 } , V d = { v 10 , v 11 , ..., v 27 } , and V c = { v 28 , v 29 , ..., v 54 } , which denote learned features for graph nodes of 10 loss functions ( e.g. , L1 and MSE ), 18 discrete architecture hyperparameter ( e.g. , Batch Norm. and ReLU ), and 9 continuous architecture hyperparameter ( i.e. , Parameter Num. ), respectively. Note V c represents learned features of 9 continuous architecture hyperparameters because each continuous hyperparameter is represented by 3 graph nodes, as illustrated in Fig. 1c of the main paper. Furthermore, via Eq. 7 in the main paper, we use V to obtain the corresponding probability score p = { p 0 , p 1 , ..., p 54 } for each graph node. Similar to V , this p can be viewed as three sections: p l ∈ R 10 , p d ∈ R 18 and p c ∈ R 27 for loss functions, discrete architecture hyperparameters, continuous architecture hyperparameters, respectively. In the end, we use p to help optimize LGPN via the graph node classification loss (Eq. 7). After the training converges, we further apply individual fully connected layers on the top of frozen learned features of continuous architecture hyperparameters ( e.g. , V c ). via minimizing the L 1 distance between predicted and ground truth value.

In the inference (the main paper Fig. 4b), for loss function and discrete architecture hyperparameters, we use output probabilities ( e.g. , p l and p d ) of discrete value graph nodes, for the binary 'used v.s. not" classification. For the continuous architecture hyperparameters, we first concatenate learned features of corresponding graph nodes. We utilize such a concatenated feature with pre-trained, fully connected layers to estimate the continuous hyperparameter value.

Model Parsing Implementation Details Denote the output feature from the Generation Trace Capturing Network as f ∈ R 2048 . To transform f into a set of features H = { h 0 , h 1 , ..., h 54 } for the 55 graph nodes, 55 independent linear layers ( Θ ) are employed. Each feature h i is of dimension R 512 . The H is fed to the GCN refinement block, which contains 5 GCN blocks, each of which has 2 stacked GCN layers. In other words, the GCN refinement block has 10 layers in total. We use the correlation graph A ∈ R 55 × 55 (Fig. 10) to capture the hyperparameter dependency and during the training the LGPN pools A into A 1 ∈ R 18 × 18 and A 2 ∈ R 6 × 6 as the Fig. 2 of the main paper. The LGPN is implemented using the PyTorch framework. During training, a learning rate of 3e-2 is used. The training is performed with a batch size of 400 , where 200 images are generated by various GMs and 200 images are real.

Implementation Details for Detection and Attributions We validate GTC's effectiveness in capturing the generation trace in Fig. 6. Specifically, Fig. 8 shows the detailed implementation. We employ FC layers to convert output feature f ∈ R 2048 to f det. ∈ R 2 and f att. ∈ R 5 for CNN-generated image detection and image attribution respectively.

C Additional Results

We report detailed performance on RED 116 via Tab. 8, demonstrating that our proposed LGPN achieves the best performance on both datasets. Also, in Fig. 5a of the main paper, we visualize

L1

L2

MMD

MSE

LS

Adv.

WGAN

KL

Hinge

CE

0.0

0.2

0.4

0.6

0.8

1.0

.655

.83

.968

.748

.752

.861

.901

.939

.927

.752

Loss Function Types

(a)

Non-linear.

last layer

Norm type

None-linear

last block

Skip connect.

Upsampling

Downsampling

0.0

0.2

0.4

0.6

0.8

.787

.897

.727

.815

.701

.792

Discrete Architecture Hyperparameter

(b)

Layers

Conv

FC

Pooling

Norm

Filters

Block

Layers

Block

Num

Param. Num

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

L1 Error

.147

.137

.077

.091

.138

.149

.083

.168

.081

(c)

the correlation graph similarities among different GMs in the first test set. In this section, we offer a similar visualization ( e.g., Fig. 9 of the supplementary) for other test sets. In the main paper's Sec 3.3, we use n graph nodes for each continuous hyperparameter and n is set as 3 . Tab. 7 shows the advantage of choice, which shows the lowest L 1 regression error is achieved when n is 3 .

FC

Real or CNN-

FC

Classification

GTC

layers

generated?

GTC

layers

among5

categories.

1*2048

1*2

1*2048

1*5

1

(a)CNN-generated image detection

(b)Image attribution.

AAE

CRGAN_C

GFLM

ADAGAN_C

FACTOR_VAE

IMAGE_GPT

BEGAN

FGSM

LSGAN

BETA_H

ICRGAN_C

MADE

LOGAN

PROG_GAN

CARM_GAN

PIXELSNAIL

RSGAN_REG

DEEPFOOL

STARGAN_2

SEAN

STYLEGAN

DRIT

SURVAE_MAX

SURVAE_NON

FVBN

VAE_FIELD

WGAN_DRA

STAB_DM_15

CONTROLNET

SDEDIT

DDPM_32

STAB_DM_21

PNDM_128

GLIDE

ADM_G_512

STAB_DM_XL

0

1.0

二

L1

1.0

L2

MSE

MMD

LS

WGAN

KL

Adv

Hinge

3D

layer# [250,500) -

layer# [0,250)

0.8

layer# [500,) -

conv# [0,90)

conv# [90,180)

conv#[180.)

FC#[0,60)

FC# [60,120)

FC# [120,)

Pool# [0,15)

Pool# [15,30) -

Nor# [0,70)

Pool# [30,)

0.6

Nor# [70,140)

Nor# [140,)

Filter# [0,3k]

Filter# [3k,6k]

Filter# [6k,9k]

Block# [0,50)

Block# [50,100)

Block# [100.)

L/Block# [0,7)

L/Block# [7,14)

- 0.4

L/Block# [14,)

Para.# [0,180M) -

Para. # [180M,360M) -

Para. # [360M,) -

Insta. Norm.

Ada. Norm.

Group Norm.

ReLU

TanH

Sigmoid

0.2

SiLU

ELU/block

ReLU/block

LReLU/block

SiLU/block

5udues-dn NN

Deconvup-sampling

skip-connection

down-sampling

0.0

Table 8: The model parsing performance on RED 116 . In the last row, our proposed LGPN that contains GTC and GCN Refinement block, which achieves the best model parsing performance in all metrics. [ Key : GCN refine.: GCN refinement block; Bold : best.].

(a)

(b)

(c)

0.9

1.1

Object Acc.

Dis. Archi. Acc.

0.8

1.0

0.7

60

0.6

0.5

0.8

0.4

0.7

0.3

0.6

0.2

0.1

0.5

ADV_FACES

BETA_BBETA_TCVAE

RSGAN_HALF

0.0

0.9

1.1

Object Acc.

Dis. Archi. Acc.

0.8

1.0

0.7

0.9

0.6

0.5

0.8

0.4

0.7

0.3

0.6

0.2

0.1

0.5

AAE

ADAGAN_C

BEGAN

DEEPFOOL

DRIT

FVBN

STA_DM_15

DDPM_32

GLIDE

0.0

0.9

1.1

Object Acc.

0.8

Dis.Archi.Acc.

1.0

0.7

60

0.6

0.5

0.8

0.4

0.7

0.3

0.6

0.2

0.1

0.5

CRGAN_C

FACTOR_VAE

FGSM

ICRGAN_C

LOGAN

VAE_FIELD

LDM

CONTROL

STA_DM_21

0.0

0.9

1.1

Object Acc.

Dis.Archi. Acc.

0.8

1.0

0.7

0.9

0.6

0.5

0.8

0.4

0.7

0.3

0.6

0.2

0.1

0.5

GFLMIMAGE_GPT

LSGAN

MADEPROG_GAN

RSGAN_REG

SEAN

WGAN_DRA

0.0

(d)

LSUN

4.7%

ImageNet

4.7%

MNIST

25.2%

CIFAR10

15.0%

Face

50.4%

Semantics

(a)

CIFAR10

16.7%

SiWMv2

8.3%

ImageNet

16.7%

celebA (face)

8.3%

FFHQ (face)

16.7%

MNIST

16.7%

LSUN

16.7%

REAL

(b)

32*32

23.9%

512*512

4.2%

256*256

17.6%

224*224

7.0%

1024*1024

4.2%

64*64

20.4%

128*128

22.5%

Resolution

(c)

D RED 140 Dataset

In this section, we provide an overview of the RED 140 dataset, which is used for both model parsing and coordinated attack detection. Note that, for the experiment reported in Tab. ?? of the supplementary, we follow the test sets defined in RED 116 [2]. When we construct RED 140 , we use images from ImageNet, FFHQ, CelebHQ, CIFAR10, and LSUN as the real-images category of RED 140 . We exclude GM that is not trained in the real-image category of RED 140 . In addition, both RED 116 and RED 140 contain various image content and resolution, and the details about RED 140 are uncovered in Fig. 12 of the supplementary. For test sets (Tab. 12 of the supplementary), we follow the dataset partition of RED 116 , excluding the GMs that are trained on real images, which RED 140 does have. For example, JFT300 Mis used to train BigGAN, so we remove BIGGAN_128 , BIGGAN_256 and BIGGAN_512 in the first, second, and third test sets.

E GMHyperparameter Ground Truth

In this section, we report the ground truth vector of different hyperparameters of each GM contained in the RED 140 . Specifically, Tab. 13 and Tab. 14 report the loss function ground truth vector for each GM. Tab. 15 and Tab. 16 report the discrete architecture hyperparameter ground truth for each GM. Tab. 17 and Tab. 18 report the continuous architecture hyperparameter ground truth for each GM. The detailed model parsing performance on each GM is reported in the supplementary's Fig. 11.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose a model parsing algorithm, which is claimed in both the abstract and introduction sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of our proposed method at the end of the paper, which helps the deeper understanding.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide extensive empirical results and detailed insights that justify each of our contribution statements.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes. We include an extensive supplementary that includes the dataset description, implementation details, and detailed performance on each generative model and hyperparameters.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes. We will release the source code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We offer the detailed description regarding the experimental setups in this work.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our proposed method uses different metrics, such as ROCAUC, F1 score, accuracy, and average precision, to report the performance. This helps show the statistical significance of the performance difference between the previous work and ours. Also, considering the large number of experiments, including error bars for all of them is computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We indicate the computation resources and architecture implementations in the supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We discuss the potential harms and societal impact at the end of our work, following NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We include societal impact contents at the end of our work, following NeurIPS code of ethics.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The dataset consists of digital images, which do not need safeguards. This is not applicable to our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Both the RED116 and RED140 datasets used in this work are publicly available and can be used for research purposes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will release the dataset and code based on the acceptance. They are well-documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does involve crowdsourcing and human subjects' biometric information.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [No]

Justification: Due to the page limit, we provide societal impact here: We strongly advocate for the machine learning community to actively work towards mitigating the potential negative societal implications of research. It is possible that generated face image may leak the identity information of subjects in the training set of a GM who do not sign the consent form. We shall strive to work on real face imagery whose collection is reviewed by an Institutional Review Board (IRB).

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the model parsing (MP) task [1], which predicts the hyperparameters of the generative model with a generated image input. The authors propose a Learnable Graph Pooling Network (LGPN) method to explore hyperparameter dependencies in the generative model. The LGPN consists of two core designs: the Generation Trace Capturing Network (GTC) and the learnable pooling-unpooling graph mechanism called the GCN Refinement Block. Extensive experiments validate the effectiveness of the method.\n\nSTRENGTHS:\n1. A new idea of graph pooling mechanisms to model hyperparameter dependency in the MP task.\n2. A new dataset, RED140, extended from RED116 [1].\n3. Outstanding performances on both the RED116 and RED140 datasets.\n\nWEAKNESSES:\nDespite the strengths mentioned above, there are some concerns listed as follows:\n1. The authors provide a new dataset, RED140, which removes and collects some samples. Please clarify the distinction between these two datasets. Is the ground-truth feature vector for Discrete Architecture hyperparameters newly added to supervise the model training? \n2. The work mainly compares with FEN-PN. [1]. The experimental results on the RED116 dataset in Fig. 5(c) just report the F1 score and L1 error performance. [1] provides more evaluation metrics (P-value, Corr. Coef, Slope, Acc., etc.) and tests more dataset types and GM models, such as continuous type, discrete type, and loss type datasets, as well as face and non-face GMs. These analyses provide more evidence to show the method's generalizability and robustness. By comparison, the proposed method needs more experimental evidence.\n3. From Fig. 6 (b), the performance of the proposed method has a slight drop compared with FEN-PN [1]. What happens?\n4. In lines 30-32, there is a motivation statement for hyperparameter dependency modeling: ”For instance, an inherent dependency exists between GM’s layer number and parameter number.” The authors do not provide such analysis and discussion in the experiments. The discussion on hyperparameter dependency analyses (what and how dependents?) is insufficient. \n5. About the correlation graph construction (lines 125–126), “we count the occurrence of such pairs in the RED140 to retrieve the matrix G\\in R^{C \\times C}, where C is the hyperparameter number.” The method approach appears to overuse the dataset information in advance and is unfair when compared to FEN-PN [1].\n6. There are some parameters, such as the thresholds to remove relations in the graph or binary classification; how about the sensitivities of these parameters?\n\nQUESTIONS:\nPlease see the comments in Weakness. I'm willing to change my rating if the authors address my concerns.\n\nLIMITATIONS:\nThe authors introduce the performance improvement, but the discussions on hyperparameter dependency analyses (what and how dependents?) and the compatibility on different GM models are insufficient.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper examines the challenge of predicting the hyperparameters of a generative model from its output, specifically a generated image. The proposed approach introduces a learnable graph pooling technique, framing the problem as a node classification task. Experiments across several tasks demonstrate the effectiveness of this method.

STRENGTHS:
- This paper explores the interesting problem of predicting hyperparameters and their dependencies.

- The proposed graph-based method is reasonable and appears to be easy to implement.

- The experimental results in Table 1 clearly demonstrate the effectiveness of the proposed method for model parsing. Additionally, ablation studies or case studies provide further valuable insights.

WEAKNESSES:
-  It remains unclear how the proposed method aids in the detection of coordinated attacks.

- Additional demonstrations or examples of predicted hyperparameters would provide clarity on the effectiveness of the proposed method.

QUESTIONS:
See weakness.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of model parsing, which aims to analyze the hyperparameters of the model that generates a given image. Different from existing model parsing methods, the proposed Learnable Graph Pooling Network (LGPN) exploits the dependencies among hyperparameters to better perform model parsing. In LGPN, the features are extracted by Generation Trace Capturing Network (GTC), which fuses the outputs of the high-res branch and the ResNet branch to preserve high-frequency generation artifacts. The performance of LGPN is compared with baseline methods and a recently proposed model parsing method on the RED140 dataset. The effectiveness of the proposed method is verified by the experimental results.\n\nSTRENGTHS:\n1. The considered problem setting is novel and practical.\n2. The dependences among hyperparameters are exploited to improve the model parsing performance.\n3. The proposed method outperforms the compared methods.\n\nWEAKNESSES:\n1. The effectiveness of GTC can be further verified if different backbones are also compared with GCN refinement as the MP head.\n2. It is unclear whether the ground truth of the hyperparameter hierarchy assignment is constructed based on the entire dataset or just the training data.\n3. The quality of the paper can be improved if the expressions are polished more carefully, especially in the experimental part.\n\nQUESTIONS:\n1. Is the ground truth of the hyperparameter hierarchy assignment constructed based on the entire dataset or just the training data?\n2. It has been shown that the hyperparameter hierarchy prediction loss and artifacts isolation loss is important to the performance. Will the performance of the proposed method drop noticeably if: 1) the hyperparameter hierarchy in the test set differ greatly from the training set; 2) some generated images are mistakenly treated as real images in the training set.\n\nLIMITATIONS:\nThe limitations are discussed in the conclusion part. There is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper solves the model parsing task that requires models to predicting hyperparameters of the generative models (GMs) through one input image. The proposed model LGPN predict hyperparameters and their dependencies via directed graphs. LGPN incorporates a learnable pooling-unpooling mechanism to convert this task as a node classification problem. As stated in paper, LGPN achieves state-of-the-art results in model parsing and some extended applications like coordinate attack detection.\n\nSTRENGTHS:\n1. As stated in the paper, LGPN consider more generative models in the training set, e.g., diffusion models. Therefore, LGPN can be adapted to the test environments where the images can be generated from diffusion model family.\n2. Modeling model parsing as node classification on directed graphs makes sense. The proposed graph pooling network can group similar type of hyperparameters into supernodes, which indicates the model’s understanding on hyperparameter dependences.\n3. LGPN show outstanding performance in experiments and has practical applications like coordinate attack detection.\n\nWEAKNESSES:\n1. The extensibility of LGPN can be limited in some aspects. For example, if a class of generative models that include new activation functions is proposed, LGPN may not be able to be directly applied to the new task environment.\n2. Efficiency of the proposed method is an aspect that needs to be discussed. As the types of hyperparameters increase, the time cost of the algorithm may increase as well.\n\nQUESTIONS:\n1.\tOne concern is about the relation to prior works. The authors can better illustrate the contributions by discussing the relationship and difference between LGPN and the papers [1][2]. The paper [1] uses a meta model to predict model attributes from input data, and the paper [2] regards generative models as distributions of functions and captures model hyperparameters and architectures through the latent vector.\n2.\tHow can we predict the continuous parameters of activation functions like leaky ReLU?\n3.\tQuestion about the problem setting. Will using more input images as context improves the accuracy in results?\n\n[1] Oh S J, Schiele B, Fritz M. Towards reverse-engineering black-box neural networks[J]. Explainable AI: interpreting, explaining and visualizing deep learning, 2019: 121-144.\n\n[2] Dupont, Emilien, Yee Whye Teh, and Arnaud Doucet. \"Generative models as distributions of functions.\" arXiv preprint arXiv:2102.04776 (2021).\n\nLIMITATIONS:\nThe ability of LGPN to infer input images generated with unseen or undefined model architectures is still unclear. For example, would LGPN predicts the leaky ReLU activation in the test environment if it is not appeared in the training data?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-25"}